Table of contents<\/h2>\n\t- \n\t\tNotes before we start<\/a>\n\t<\/li>\n\t
- \n\t\tBuilding Infrastructure<\/a>\n\t\t
\n\t\t\t- \n\t\t\t\tAdding ELB to the mix<\/a>\n\t\t\t<\/li>\n\t\t<\/ul>\n\t
- \n\t\tConclusion<\/a>\n\t<\/li>\n<\/ul>\n<\/ul>\n<\/div>\n
\nTerraform<\/a> is a tool for automating infrastructure management. It can be used for a simple task like managing single application instance or more complex ones like managing entire datacenter or virtual cloud. The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features and others. It is a great tool to have in a DevOps environment and I find it very powerful but simple to use when it comes to managing infrastructure as a code (IaaC). And the best thing about it is the support for various platforms and providers like AWS, Digital Ocean, OpenStack, Microsoft Azure, Google Cloud etc. meaning you get to use the same tool to manage your infrastructure on any of these cloud providers. See the Providers<\/a> page for full list.<\/p>\nTerraform uses its own domain-specific language (DSL) called the Hashicorp Configuration Language (HCL): a fully JSON-compatible language for describing infrastructure as code. Configuration files created by HCL are used to describe the components of the infrastructure we want to build and manage. It generates an execution plan describing what it will do to reach the desired state, and then executes it to build the terraform.tfstate<\/code> file by default. This state file is extremely important; it maps various resource metadata to actual resource IDs so that Terraform knows what it is managing. This file must be saved and distributed to anyone who might run Terraform against the very VPC infrastructure we created so storing this in GitHub repository is the best way to go in order to share a project.<\/p>\nTo install terraform follow the simple steps from the install web page Getting Started<\/a><\/p>\nNotes before we start<\/span><\/h2>\nFirst let me mention that the below code has been adjusted to work with the latest Terraform and has been successfully tested with version 0.7.4<\/code>. It has been running for long time now and been used many times to create production VPC’s in AWS. It’s been based on CloudFormation<\/code> templates I’ve written for the same purpose at some point in 2014 during the quest of converting our infrastructure into code.<\/p>\nWhat is this going to do is:<\/p>\n
\n- Create a multi tier VPC (Virtual Private Cloud) in specific AWS region<\/li>\n
- Create 2 Private and 1 Public Subnet in the specific VPC CIDR<\/li>\n
- Create Routing Tables and attach them to the appropriate subnets<\/li>\n
- Create a NAT instance with ASG (Auto Scaling Group) to serve as default gateway for the private subnets<\/li>\n
- Create Security Groups to use with EC2 instances we create<\/li>\n
- Create SNS notifications for Auto Scaling events<\/li>\n<\/ul>\n
This is a step-by-step walk through, the source code will be made available at some point.<\/p>\n
We will need an SSH key and SSL certificate (for ELB) uploaded to our AWS account and awscli<\/code> tool installed on the machine we are running terraform before we start.<\/p>\nBuilding Infrastructure<\/span><\/h2>\nAfter setting up the binaries we create an empty directory that will hold the new project. First thing we do is tell terraform which provider we are going to use. Since we are building Amazon AWS infrastructure we create a .tfvars<\/code> file with our AWS IAM API credentials. For example, provider-credentials.tfvars<\/code> with the following content:<\/p>\nprovider = {\n access_key = \"<aws_access_key>\"\n secret_key = \"<aws_secret_key>\"\n region = \"ap-southeast-2\"\n}\n<\/code><\/pre>\nWe make sure the API credentials are for user that has full permissions to create, read and destroy infrastructure in our AWS account. Check the IAM user and its roles to confirm this is the case.<\/p>\n
Then we create a .tf<\/code> file where we create our first resource called provider<\/code>. Lets name the file provider-config.tf<\/code> and put the following content:<\/p>\nprovider \"aws\" {\n access_key = \"${var.provider[\"access_key\"]}\"\n secret_key = \"${var.provider[\"secret_key\"]}\"\n region = \"${var.provider[\"region\"]}\"\n}\n<\/code><\/pre>\nfor our AWS provider type.<\/p>\n
Then we create a .tf<\/code> file vpc_environment.tf<\/code> where we put all essential variables needed to build the VPC, like VPC CIDR, AWS zone and regions, default EC2 instance type and the ssh key and other AWS related parameters:<\/p>\n\/*=== VARIABLES ===*\/\nvariable \"provider\" {\n type = \"map\"\n default = {\n access_key = \"unknown\"\n secret_key = \"unknown\"\n region = \"unknown\"\n }\n}\n\nvariable \"vpc\" {\n type = \"map\"\n default = {\n \"tag\" = \"unknown\"\n \"cidr_block\" = \"unknown\"\n \"subnet_bits\" = \"unknown\"\n \"owner_id\" = \"unknown\"\n \"sns_topic\" = \"unknown\"\n }\n}\n\nvariable \"azs\" {\n type = \"map\"\n default = {\n \"ap-southeast-2\" = \"ap-southeast-2a,ap-southeast-2b,ap-southeast-2c\"\n \"eu-west-1\" = \"eu-west-1a,eu-west-1b,eu-west-1c\"\n \"us-west-1\" = \"us-west-1b,us-west-1c\"\n \"us-west-2\" = \"us-west-2a,us-west-2b,us-west-2c\"\n \"us-east-1\" = \"us-east-1c,us-west-1d,us-west-1e\"\n }\n}\n\nvariable \"instance_type\" {\n default = \"t1.micro\"\n}\n\nvariable \"key_name\" {\n default = \"unknown\"\n}\n\nvariable \"nat\" {\n type = \"map\"\n default = {\n ami_image = \"unknown\"\n instance_type = \"unknown\"\n availability_zone = \"unknown\"\n key_name = \"unknown\"\n filename = \"userdata_nat_asg.sh\"\n }\n}\n\n\/* Ubuntu Trusty 14.04 LTS (x64) *\/\nvariable \"images\" {\n type = \"map\"\n default = {\n eu-west-1 = \"ami-47a23a30\"\n ap-southeast-2 = \"ami-6c14310f\"\n us-east-1 = \"ami-2d39803a\"\n us-west-1 = \"ami-48db9d28\"\n us-west-2 = \"ami-d732f0b7\"\n }\n}\n\nvariable \"env_domain\" {\n type = \"map\"\n default = {\n name = \"unknown\"\n zone_id = \"unknown\"\n }\n}\n<\/code><\/pre>\nI have created most of the variables as generic and then passing on their values via separate .tfvars<\/code> file vpc_environment.tfvars<\/code>:<\/p>\nvpc = {\n tag = \"TFTEST\"\n owner_id = \"<owner -id>\"\n cidr_block = \"10.99.0.0\/20\"\n subnet_bits = \"4\"\n sns_email = \"<sns -email>\"\n}\nkey_name = \"<ssh -key>\"\nnat.instance_type = \"m3.medium\"\nenv_domain = {\n name = \"mydomain.com\"\n zone_id = \"<zone -id>\"\n}\n<\/code><\/pre>\nTerraform does not support (yet) interpolation by referencing another variable in a variable name (see Terraform issue #2727<\/a>) nor usage of an array as an element of a map. These are couple of shortcomings but If you have used AWS’s CloudFormation you would have faced similar “issues”. After all these tools are not really a programming language so we have to accept them as they are and try to make the best of it.<\/p>\nWe can see I have separated the provider stuff from the rest of it including the resource so I can easily share my project without exposing sensitive data. For example I can create GitHub repository out of my project directory and put the provider-credentials.tfvariables<\/code> file in .gitignore<\/code> so it never gets accidentally uploaded.<\/p>\nNow is time to do the first test. After substituting all values in <><\/code> with real ones we run:<\/p>\n$ terraform plan -var-file provider-credentials.tfvars -var-file vpc_environment.tfvars -out vpc.tfplan \n<\/code><\/pre>\ninside the directory and check the output. If this goes without any errors then we can proceed to next step, otherwise we have to go back and fix the errors terraform has printed out. To apply the planned changes then we run:<\/p>\n
$ terraform apply vpc.tfplan\n<\/code><\/pre>\nbut it’s too early for that at this stage since we have nothing to apply yet.<\/p>\n
We can start creating resources now, starting with a VPC, subnets and IGW (Internet Gateway). We want our VPC to be created in a region with 3 AZ’s (Availability Zones) so we can spread our future instance nicely for HA. We create a new .tf<\/code> file vpc.tf<\/code>:<\/p>\n\/*=== VPC AND SUBNETS ===*\/\nresource \"aws_vpc\" \"environment\" {\n cidr_block = \"${var.vpc[\"cidr_block\"]}\"\n enable_dns_support = true\n enable_dns_hostnames = true \n tags {\n Name = \"VPC-${var.vpc[\"tag\"]}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_internet_gateway\" \"environment\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-internet-gateway\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_subnet\" \"public-subnets\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index)}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-subnet-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n map_public_ip_on_launch = true\n}\n\nresource \"aws_subnet\" \"private-subnets\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index + length(split(\",\", lookup(var.azs, var.provider[\"region\"]))))}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n Network = \"private\"\n }\n}\n\nresource \"aws_subnet\" \"private-subnets-2\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index + (2 * length(split(\",\", lookup(var.azs, var.provider[\"region\"])))))}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-2-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n Network = \"private\"\n }\n}\n<\/code><\/pre>\nThis will create a VPC for us with 3 sets of subnets, 2 private and 1 public (meaning will have the IGW as default gateway). For the private subnets we need to create a NAT instance to be used as internet gateway. We can create a new .tf<\/code> file vpc_nat_instance.tf<\/code> lets say where we create the resource:<\/p>\n\/*== NAT INSTANCE IAM PROFILE ==*\/\nresource \"aws_iam_instance_profile\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-profile\"\n roles = [\"${aws_iam_role.nat.name}\"]\n}\n\nresource \"aws_iam_role\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-role\"\n path = \"\/\"\n assume_role_policy = < <EOF\n{\n \"Version\": \"2008-10-17\",\n \"Statement\": [\n {\n \"Action\": \"sts:AssumeRole\",\n \"Principal\": {\"AWS\": \"*\"},\n \"Effect\": \"Allow\",\n \"Sid\": \"\"\n }\n ]\n}\nEOF\n}\n\nresource \"aws_iam_policy\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-policy\"\n path = \"\/\"\n description = \"NAT IAM policy\"\n policy = <<EOF\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:DescribeInstances\",\n \"ec2:ModifyInstanceAttribute\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeRouteTables\",\n \"ec2:CreateRoute\",\n \"ec2:ReplaceRoute\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\nEOF\n}\n\nresource \"aws_iam_policy_attachment\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-attachment\"\n roles = [\"${aws_iam_role.nat.name}\"]\n policy_arn = \"${aws_iam_policy.nat.arn}\"\n}\n\n\/*=== NAT INSTANCE ASG ===*\/\nresource \"aws_autoscaling_group\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-asg\"\n availability_zones = \"${split(\",\", lookup(var.azs, var.provider[\"region\"]))}\"\n vpc_zone_identifier = [\"${aws_subnet.public-subnets.*.id}\"]\n max_size = 1\n min_size = 1\n health_check_grace_period = 60\n default_cooldown = 60\n health_check_type = \"EC2\"\n desired_capacity = 1\n force_delete = true\n launch_configuration = \"${aws_launch_configuration.nat.name}\"\n tag {\n key = \"Name\"\n value = \"NAT-${var.vpc[\"tag\"]}\"\n propagate_at_launch = true\n }\n tag {\n key = \"Environment\"\n value = \"${lower(var.vpc[\"tag\"])}\"\n propagate_at_launch = true\n }\n tag {\n key = \"Type\"\n value = \"nat\"\n propagate_at_launch = true\n }\n tag {\n key = \"Role\"\n value = \"bastion\"\n propagate_at_launch = true\n }\n lifecycle {\n create_before_destroy = true\n }\n}\n\nresource \"aws_launch_configuration\" \"nat\" {\n name_prefix = \"${var.vpc[\"tag\"]}-nat-lc-\"\n image_id = \"${lookup(var.images, var.provider[\"region\"])}\"\n instance_type = \"${var.nat[\"instance_type\"]}\"\n iam_instance_profile = \"${aws_iam_instance_profile.nat.name}\"\n key_name = \"${var.key_name}\"\n security_groups = [\"${aws_security_group.nat.id}\"]\n associate_public_ip_address = true\n user_data = \"${data.template_file.nat.rendered}\"\n lifecycle {\n create_before_destroy = true\n }\n}\n\ndata \"template_file\" \"nat\" {\n template = \"${file(\"${var.nat[\"filename\"]}\")}\"\n vars {\n cidr = \"${var.vpc[\"cidr_block\"]}\"\n }\n}\n<\/code><\/pre>\nWe create the NAT instance in a Auto Scaling group since being a vital part of the infrastructure we want it to be highly available. This means that in case of a failure, the ASG will launch a new one. This instance then needs to configure it self as a gateway for the public subnets for which we create and attach to it an IAM role with specific permissions. Lastly the instance will use the userdata_nat_asg.sh<\/code> file (see the variables file) given to it via user-data<\/code> to setup the routing for the private subnets. The scipt is given below:<\/p>\n#!\/bin\/bash -v\nset -e\necho iptables-persistent iptables-persistent\/autosave_v4 boolean true | debconf-set-selections\necho iptables-persistent iptables-persistent\/autosave_v6 boolean true | debconf-set-selections\nrm -rf \/var\/lib\/apt\/lists\/*\nsed -e '\/^deb.*security\/ s\/^\/#\/g' -i \/etc\/apt\/sources.list\nexport DEBIAN_FRONTEND=noninteractive\napt-get update -y -m -qq\napt-get -y install conntrack iptables-persistent nfs-common > \/tmp\/nat.log\n[[ -s $(modinfo -n ip_conntrack) ]] && modprobe ip_conntrack && echo ip_conntrack | tee -a \/etc\/modules\n\/sbin\/sysctl -w net.ipv4.ip_forward=1\n\/sbin\/sysctl -w net.ipv4.conf.eth0.send_redirects=0\n\/sbin\/sysctl -w net.netfilter.nf_conntrack_max=131072\necho \"net.ipv4.ip_forward=1\" >> \/etc\/sysctl.conf\necho \"net.ipv4.conf.eth0.send_redirects=0\" >> \/etc\/sysctl.conf\necho \"net.netfilter.nf_conntrack_max=131072\" >> \/etc\/sysctl.conf\n\/sbin\/iptables -A INPUT -m conntrack --ctstate INVALID -j DROP\n\/sbin\/iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP\n\/sbin\/iptables -A INPUT -p tcp --syn -m limit --limit 5\/s -i eth0 -j ACCEPT\n\/sbin\/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT\n\/sbin\/iptables -t nat -A POSTROUTING -o eth0 -s ${cidr} -j MASQUERADE\nsleep 1\nwget -P \/usr\/local\/bin https:\/\/s3-ap-southeast-2.amazonaws.com\/encompass-public\/ha-nat-terraform.sh\n[[ ! -x \"\/usr\/local\/bin\/ha-nat-terraform.sh\" ]] && chmod +x \/usr\/local\/bin\/ha-nat-terraform.sh\n\/bin\/bash \/usr\/local\/bin\/ha-nat-terraform.sh\ncat > \/etc\/profile.d\/user.sh <<END\nHISTSIZE=1000\nHISTFILESIZE=40000\nHISTTIMEFORMAT=\"[%F %T %Z] \"\nexport HISTSIZE HISTFILESIZE HISTTIMEFORMAT\nEND\nsed -e '\/^#deb.*security\/ s\/^#\/\/g' -i \/etc\/apt\/sources.list\nexit 0\n<\/code><\/pre>\nIt configures the firewall and the NAT rules and executes the ha-nat-terraform.sh<\/code> script fetched from a S3 bucket.<\/p>\nIn the same time we create Security Groups, or instance firewalls in AWS terms, to attach to the subnets and the NAT instance we are going to create:<\/p>\n
\/*=== SECURITY GROUPS ===*\/\nresource \"aws_security_group\" \"default\" {\n name = \"${var.vpc[\"tag\"]}-default\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n ingress {\n from_port = -1\n to_port = -1\n protocol = \"icmp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-default-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = -1\n to_port = -1\n protocol = \"icmp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 123 \n to_port = 123\n protocol = \"udp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 80\n to_port = 80\n protocol = \"tcp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n ingress {\n from_port = 443 \n to_port = 443 \n protocol = \"tcp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n egress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = -1\n to_port = -1\n protocol = \"icmp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 80\n to_port = 80\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 443 \n to_port = 443 \n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 123 \n to_port = 123\n protocol = \"udp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 53 \n to_port = 53\n protocol = \"udp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 53\n to_port = 53\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-nat-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"public\" {\n name = \"${var.vpc[\"tag\"]}-public\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 80 \n to_port = 80 \n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"private\" {\n name = \"${var.vpc[\"tag\"]}-private\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"${aws_subnet.private-subnets.*.cidr_block}\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"private-2\" {\n name = \"${var.vpc[\"tag\"]}-private-2\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"${aws_subnet.private-subnets-2.*.cidr_block}\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-2-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n<\/code><\/pre>\nNext we need to sort out the VPC routing, create routing tables and associate them with the subnets. Create a new file vpc_routing_tables.tf<\/code>:<\/p>\n\/*=== ROUTING TABLES ===*\/\nresource \"aws_route_table\" \"public-subnet\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n route {\n cidr_block = \"0.0.0.0\/0\"\n gateway_id = \"${aws_internet_gateway.environment.id}\"\n }\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-subnet-route-table\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_route_table_association\" \"public-subnet\" {\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n subnet_id = \"${element(aws_subnet.public-subnets.*.id, count.index)}\"\n route_table_id = \"${aws_route_table.public-subnet.id}\"\n}\n\nresource \"aws_route_table\" \"private-subnet\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-route-table\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_route_table_association\" \"private-subnet\" {\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n subnet_id = \"${element(aws_subnet.private-subnets.*.id, count.index)}\"\n route_table_id = \"${aws_route_table.private-subnet.id}\"\n}\n\nresource \"aws_route_table\" \"private-subnet-2\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-2-route-table\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_route_table_association\" \"private-subnet-2\" {\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n subnet_id = \"${element(aws_subnet.private-subnets-2.*.id, count.index)}\"\n route_table_id = \"${aws_route_table.private-subnet-2.id}\"\n}\n<\/code><\/pre>\nTo wrap it up I would like to receive some notifications in case of Autoscaling events so we create vpc_notifications.tf<\/code> file:<\/p>\n\/*=== AUTOSCALING NOTIFICATIONS ===*\/\nresource \"aws_autoscaling_notification\" \"main\" {\n group_names = [\n \"${aws_autoscaling_group.nat.name}\"\n ]\n notifications = [\n \"autoscaling:EC2_INSTANCE_LAUNCH\", \n \"autoscaling:EC2_INSTANCE_TERMINATE\",\n \"autoscaling:EC2_INSTANCE_LAUNCH_ERROR\",\n \"autoscaling:EC2_INSTANCE_TERMINATE_ERROR\"\n ]\n topic_arn = \"${aws_sns_topic.main.arn}\"\n}\n\nresource \"aws_sns_topic\" \"main\" {\n name = \"${lower(var.vpc[\"tag\"])}-sns-topic\"\n\n provisioner \"local-exec\" {\n command = \"aws sns subscribe --topic-arn ${self.arn} --protocol email --notification-endpoint ${var.vpc[\"sns_email\"]}\"\n }\n}\n<\/code><\/pre>\nAs we further build our infrastructure we will use more Auto Scaling configurations and we can add those to the above resource under group_names<\/code>.<\/p>\nAt the end, some outputs we can use if needed:<\/p>\n
\/*=== OUTPUTS ===*\/\noutput \"num-zones\" {\n value = \"${length(lookup(var.azs, var.provider[region]))}\"\n}\n\noutput \"vpc-id\" {\n value = \"${aws_vpc.environment.id}\"\n}\n\noutput \"public-subnet-ids\" {\n value = \"${join(\",\", aws_subnet.public-subnets.*.id)}\"\n}\n\noutput \"private-subnet-ids\" {\n value = \"${join(\",\", aws_subnet.private-subnets.*.id)}\"\n}\n\noutput \"private-subnet-2-ids\" {\n value = \"${join(\",\", aws_subnet.private-subnets-2.*.id)}\"\n}\n\noutput \"autoscaling_notification_sns_topic\" {\n value = \"${aws_sns_topic.main.id}\"\n}\n<\/code><\/pre>\nAt the end we run:<\/p>\n
$ terraform plan -var-file provider-credentials.tfvars -var-file vpc_environment.tfvars -out vpc.tfplan\n<\/code><\/pre>\nto test and create the plan and then:<\/p>\n
$ terraform apply vpc.tfplan\n<\/code><\/pre>\nto create our VPC. When finished we can destroy the infrastructure:<\/p>\n
$ terraform destroy vpc.tfplan --force\n<\/code><\/pre>\nAdding ELB to the mix<\/span><\/h3>\nSince we are building highly available infrastructure we are going to need a public ELB to put our application servers behind it. Under assumption that the app is listening on port 8080 we can add the following:<\/p>\n
variable \"app\" {\n default = {\n elb_ssl_cert_arn = \"\"\n elb_hc_uri = \"\"\n listen_port_http = \"\"\n listen_port_https = \"\"\n }\n}\n\n<\/code><\/pre>\nto our vpc_environment.tf<\/code> file and set the values by putting:<\/p>\napp = {\n instance_type = \"<app-instance-type>\"\n host_name = \"<app-host-name>\"\n elb_ssl_cert_arn = \"<elb-ssl-cert-arn>\"\n elb_hc_uri = \"<app-health-check-path>\"\n listen_port_http = \"8080\"\n listen_port_https = \"443\"\n domain = \"<domain-name>\"\n zone_id = \"<domain-zone-id>\"\n}\n<\/code><\/pre>\nin our vpc_environment.tfvars<\/code> file. Now we can create the ELB by creating the vpc_elb.tf<\/code> file with following content:<\/p>\n\/*== ELB ==*\/\nresource \"aws_elb\" \"app\" {\n \/* Requiered for EC2 ELB only\n availability_zones = \"${var.zones}\"\n *\/\n name = \"${var.vpc[\"tag\"]}-elb-app\"\n subnets = [\"${aws_subnet.public-subnets.*.id}\"]\n security_groups = [\"${aws_security_group.elb.id}\"]\n listener {\n instance_port = \"${var.app[\"listen_port_http\"]}\"\n instance_protocol = \"http\"\n lb_port = 443\n lb_protocol = \"https\"\n ssl_certificate_id = \"${var.app[\"elb_ssl_cert_arn\"]}\"\n }\n listener {\n instance_port = \"${var.app[\"listen_port_http\"]}\"\n instance_protocol = \"http\"\n lb_port = 80\n lb_protocol = \"http\"\n }\n health_check {\n healthy_threshold = 2\n unhealthy_threshold = 3\n timeout = 5\n target = \"HTTP:${var.app[\"listen_port_http\"]}${var.app[\"elb_hc_uri\"]}\"\n interval = 10\n }\n cross_zone_load_balancing = true\n idle_timeout = 960 # set it higher than the conn. timeout of the backend servers\n connection_draining = true\n connection_draining_timeout = 300\n tags {\n Name = \"${var.vpc[\"tag\"]}-elb-app\"\n Type = \"elb\"\n }\n}\n\n\/* In case we need sticky sessions\nresource \"aws_lb_cookie_stickiness_policy\" \"app\" {\n name = \"${var.vpc[\"tag\"]}-elb-app-policy\"\n load_balancer = \"${aws_elb.app.id}\"\n lb_port = 443\n cookie_expiration_period = 960\n}\n*\/\n\n\/* CREATE CNAME DNS RECORD FOR THE ELB *\/\nresource \"aws_route53_record\" \"app\" {\n zone_id = \"${var.app[\"zone_id\"]}\"\n name = \"${lower(var.vpc[\"tag\"])}.${var.app[\"name\"]}\"\n type = \"CNAME\"\n ttl = \"60\"\n records = [\"${aws_elb.app.dns_name}\"]\n}\n<\/code><\/pre>\nThe ELB does not support redirections so the app needs to deal with redirecting users from port 80\/8080 to 443 for fully secure SSL operation.<\/p>\n
Finally, the Security Group for the ELB in vpc_security.tf<\/code> file:<\/p>\nresource \"aws_security_group\" \"elb\" {\n name = \"${var.vpc[\"tag\"]}-elb\"\n ingress {\n from_port = 80\n to_port = 80\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 443\n to_port = 443\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-elb-security-group\"\n Environment = \"${var.vpc[\"tag\"]}\"\n }\n}\n<\/code><\/pre>\nand we are done.<\/p>\n
To get some outputs we are interested in from the ELB resource we can add this:<\/p>\n
output \"elb-app-public-dns\" {\n value = \"${aws_elb.app.dns_name}\"\n}\n\noutput \"route53-app-public-dns\" {\n value = \"${aws_route53_record.app.fqdn}\"\n}\n<\/code><\/pre>\nto the outputs.tf<\/code> file.<\/p>\nConclusion<\/span><\/h2>\nAs we add more infrastructure to the VPC we can make some improvements to the above code by creating modules for the common tasks like Autoscaling Groups and Launch Configurations, ELB’s, IAM Profiles etc., see Creating Modules<\/a> for details.<\/p>\nNot everything can be done this way though. For example the repetitive code like:<\/p>\n
resource \"aws_subnet\" \"public-subnets\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index)}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-subnet-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n map_public_ip_on_launch = true\n}\n<\/code><\/pre>\nis a great candidate for a module except Terraform does not (yet) support count<\/code> parameter inside modules, see Support issue #953<\/a><\/p>\nIt’s graphing feature might come handy in obtaining a logical diagram of the infrastructure we are creating:<\/p>\n
![\"\"](\"https:\/\/icicimov.com\/blog\/wp-content\/uploads\/2016\/09\/graph.png\")
<\/a><\/p>\nTo generate this run:<\/p>\n
$ terraform graph | dot -Tpng > graph.png\n<\/code><\/pre>\n
- \n\t\t\t
- \n\t\t\t\tAdding ELB to the mix<\/a>\n\t\t\t<\/li>\n\t\t<\/ul>\n\t
- \n\t\tConclusion<\/a>\n\t<\/li>\n<\/ul>\n<\/ul>\n<\/div>\n
\nTerraform<\/a> is a tool for automating infrastructure management. It can be used for a simple task like managing single application instance or more complex ones like managing entire datacenter or virtual cloud. The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features and others. It is a great tool to have in a DevOps environment and I find it very powerful but simple to use when it comes to managing infrastructure as a code (IaaC). And the best thing about it is the support for various platforms and providers like AWS, Digital Ocean, OpenStack, Microsoft Azure, Google Cloud etc. meaning you get to use the same tool to manage your infrastructure on any of these cloud providers. See the Providers<\/a> page for full list.<\/p>\nTerraform uses its own domain-specific language (DSL) called the Hashicorp Configuration Language (HCL): a fully JSON-compatible language for describing infrastructure as code. Configuration files created by HCL are used to describe the components of the infrastructure we want to build and manage. It generates an execution plan describing what it will do to reach the desired state, and then executes it to build the
terraform.tfstate<\/code> file by default. This state file is extremely important; it maps various resource metadata to actual resource IDs so that Terraform knows what it is managing. This file must be saved and distributed to anyone who might run Terraform against the very VPC infrastructure we created so storing this in GitHub repository is the best way to go in order to share a project.<\/p>\nTo install terraform follow the simple steps from the install web page Getting Started<\/a><\/p>\n
Notes before we start<\/span><\/h2>\n
First let me mention that the below code has been adjusted to work with the latest Terraform and has been successfully tested with version
0.7.4<\/code>. It has been running for long time now and been used many times to create production VPC’s in AWS. It’s been based onCloudFormation<\/code> templates I’ve written for the same purpose at some point in 2014 during the quest of converting our infrastructure into code.<\/p>\nWhat is this going to do is:<\/p>\n
- \n
- Create a multi tier VPC (Virtual Private Cloud) in specific AWS region<\/li>\n
- Create 2 Private and 1 Public Subnet in the specific VPC CIDR<\/li>\n
- Create Routing Tables and attach them to the appropriate subnets<\/li>\n
- Create a NAT instance with ASG (Auto Scaling Group) to serve as default gateway for the private subnets<\/li>\n
- Create Security Groups to use with EC2 instances we create<\/li>\n
- Create SNS notifications for Auto Scaling events<\/li>\n<\/ul>\n
This is a step-by-step walk through, the source code will be made available at some point.<\/p>\n
We will need an SSH key and SSL certificate (for ELB) uploaded to our AWS account and
awscli<\/code> tool installed on the machine we are running terraform before we start.<\/p>\nBuilding Infrastructure<\/span><\/h2>\n
After setting up the binaries we create an empty directory that will hold the new project. First thing we do is tell terraform which provider we are going to use. Since we are building Amazon AWS infrastructure we create a
.tfvars<\/code> file with our AWS IAM API credentials. For example,provider-credentials.tfvars<\/code> with the following content:<\/p>\nprovider = {\n access_key = \"<aws_access_key>\"\n secret_key = \"<aws_secret_key>\"\n region = \"ap-southeast-2\"\n}\n<\/code><\/pre>\nWe make sure the API credentials are for user that has full permissions to create, read and destroy infrastructure in our AWS account. Check the IAM user and its roles to confirm this is the case.<\/p>\n
Then we create a
.tf<\/code> file where we create our first resource calledprovider<\/code>. Lets name the fileprovider-config.tf<\/code> and put the following content:<\/p>\nprovider \"aws\" {\n access_key = \"${var.provider[\"access_key\"]}\"\n secret_key = \"${var.provider[\"secret_key\"]}\"\n region = \"${var.provider[\"region\"]}\"\n}\n<\/code><\/pre>\nfor our AWS provider type.<\/p>\n
Then we create a
.tf<\/code> filevpc_environment.tf<\/code> where we put all essential variables needed to build the VPC, like VPC CIDR, AWS zone and regions, default EC2 instance type and the ssh key and other AWS related parameters:<\/p>\n\/*=== VARIABLES ===*\/\nvariable \"provider\" {\n type = \"map\"\n default = {\n access_key = \"unknown\"\n secret_key = \"unknown\"\n region = \"unknown\"\n }\n}\n\nvariable \"vpc\" {\n type = \"map\"\n default = {\n \"tag\" = \"unknown\"\n \"cidr_block\" = \"unknown\"\n \"subnet_bits\" = \"unknown\"\n \"owner_id\" = \"unknown\"\n \"sns_topic\" = \"unknown\"\n }\n}\n\nvariable \"azs\" {\n type = \"map\"\n default = {\n \"ap-southeast-2\" = \"ap-southeast-2a,ap-southeast-2b,ap-southeast-2c\"\n \"eu-west-1\" = \"eu-west-1a,eu-west-1b,eu-west-1c\"\n \"us-west-1\" = \"us-west-1b,us-west-1c\"\n \"us-west-2\" = \"us-west-2a,us-west-2b,us-west-2c\"\n \"us-east-1\" = \"us-east-1c,us-west-1d,us-west-1e\"\n }\n}\n\nvariable \"instance_type\" {\n default = \"t1.micro\"\n}\n\nvariable \"key_name\" {\n default = \"unknown\"\n}\n\nvariable \"nat\" {\n type = \"map\"\n default = {\n ami_image = \"unknown\"\n instance_type = \"unknown\"\n availability_zone = \"unknown\"\n key_name = \"unknown\"\n filename = \"userdata_nat_asg.sh\"\n }\n}\n\n\/* Ubuntu Trusty 14.04 LTS (x64) *\/\nvariable \"images\" {\n type = \"map\"\n default = {\n eu-west-1 = \"ami-47a23a30\"\n ap-southeast-2 = \"ami-6c14310f\"\n us-east-1 = \"ami-2d39803a\"\n us-west-1 = \"ami-48db9d28\"\n us-west-2 = \"ami-d732f0b7\"\n }\n}\n\nvariable \"env_domain\" {\n type = \"map\"\n default = {\n name = \"unknown\"\n zone_id = \"unknown\"\n }\n}\n<\/code><\/pre>\nI have created most of the variables as generic and then passing on their values via separate
.tfvars<\/code> filevpc_environment.tfvars<\/code>:<\/p>\nvpc = {\n tag = \"TFTEST\"\n owner_id = \"<owner -id>\"\n cidr_block = \"10.99.0.0\/20\"\n subnet_bits = \"4\"\n sns_email = \"<sns -email>\"\n}\nkey_name = \"<ssh -key>\"\nnat.instance_type = \"m3.medium\"\nenv_domain = {\n name = \"mydomain.com\"\n zone_id = \"<zone -id>\"\n}\n<\/code><\/pre>\nTerraform does not support (yet) interpolation by referencing another variable in a variable name (see Terraform issue #2727<\/a>) nor usage of an array as an element of a map. These are couple of shortcomings but If you have used AWS’s CloudFormation you would have faced similar “issues”. After all these tools are not really a programming language so we have to accept them as they are and try to make the best of it.<\/p>\n
We can see I have separated the provider stuff from the rest of it including the resource so I can easily share my project without exposing sensitive data. For example I can create GitHub repository out of my project directory and put the
provider-credentials.tfvariables<\/code> file in.gitignore<\/code> so it never gets accidentally uploaded.<\/p>\nNow is time to do the first test. After substituting all values in
<><\/code> with real ones we run:<\/p>\n$ terraform plan -var-file provider-credentials.tfvars -var-file vpc_environment.tfvars -out vpc.tfplan \n<\/code><\/pre>\ninside the directory and check the output. If this goes without any errors then we can proceed to next step, otherwise we have to go back and fix the errors terraform has printed out. To apply the planned changes then we run:<\/p>\n
$ terraform apply vpc.tfplan\n<\/code><\/pre>\nbut it’s too early for that at this stage since we have nothing to apply yet.<\/p>\n
We can start creating resources now, starting with a VPC, subnets and IGW (Internet Gateway). We want our VPC to be created in a region with 3 AZ’s (Availability Zones) so we can spread our future instance nicely for HA. We create a new
.tf<\/code> filevpc.tf<\/code>:<\/p>\n\/*=== VPC AND SUBNETS ===*\/\nresource \"aws_vpc\" \"environment\" {\n cidr_block = \"${var.vpc[\"cidr_block\"]}\"\n enable_dns_support = true\n enable_dns_hostnames = true \n tags {\n Name = \"VPC-${var.vpc[\"tag\"]}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_internet_gateway\" \"environment\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-internet-gateway\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_subnet\" \"public-subnets\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index)}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-subnet-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n map_public_ip_on_launch = true\n}\n\nresource \"aws_subnet\" \"private-subnets\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index + length(split(\",\", lookup(var.azs, var.provider[\"region\"]))))}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n Network = \"private\"\n }\n}\n\nresource \"aws_subnet\" \"private-subnets-2\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index + (2 * length(split(\",\", lookup(var.azs, var.provider[\"region\"])))))}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-2-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n Network = \"private\"\n }\n}\n<\/code><\/pre>\nThis will create a VPC for us with 3 sets of subnets, 2 private and 1 public (meaning will have the IGW as default gateway). For the private subnets we need to create a NAT instance to be used as internet gateway. We can create a new
.tf<\/code> filevpc_nat_instance.tf<\/code> lets say where we create the resource:<\/p>\n\/*== NAT INSTANCE IAM PROFILE ==*\/\nresource \"aws_iam_instance_profile\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-profile\"\n roles = [\"${aws_iam_role.nat.name}\"]\n}\n\nresource \"aws_iam_role\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-role\"\n path = \"\/\"\n assume_role_policy = < <EOF\n{\n \"Version\": \"2008-10-17\",\n \"Statement\": [\n {\n \"Action\": \"sts:AssumeRole\",\n \"Principal\": {\"AWS\": \"*\"},\n \"Effect\": \"Allow\",\n \"Sid\": \"\"\n }\n ]\n}\nEOF\n}\n\nresource \"aws_iam_policy\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-policy\"\n path = \"\/\"\n description = \"NAT IAM policy\"\n policy = <<EOF\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Action\": [\n \"ec2:DescribeInstances\",\n \"ec2:ModifyInstanceAttribute\",\n \"ec2:DescribeSubnets\",\n \"ec2:DescribeRouteTables\",\n \"ec2:CreateRoute\",\n \"ec2:ReplaceRoute\"\n ],\n \"Resource\": \"*\"\n }\n ]\n}\nEOF\n}\n\nresource \"aws_iam_policy_attachment\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-attachment\"\n roles = [\"${aws_iam_role.nat.name}\"]\n policy_arn = \"${aws_iam_policy.nat.arn}\"\n}\n\n\/*=== NAT INSTANCE ASG ===*\/\nresource \"aws_autoscaling_group\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat-asg\"\n availability_zones = \"${split(\",\", lookup(var.azs, var.provider[\"region\"]))}\"\n vpc_zone_identifier = [\"${aws_subnet.public-subnets.*.id}\"]\n max_size = 1\n min_size = 1\n health_check_grace_period = 60\n default_cooldown = 60\n health_check_type = \"EC2\"\n desired_capacity = 1\n force_delete = true\n launch_configuration = \"${aws_launch_configuration.nat.name}\"\n tag {\n key = \"Name\"\n value = \"NAT-${var.vpc[\"tag\"]}\"\n propagate_at_launch = true\n }\n tag {\n key = \"Environment\"\n value = \"${lower(var.vpc[\"tag\"])}\"\n propagate_at_launch = true\n }\n tag {\n key = \"Type\"\n value = \"nat\"\n propagate_at_launch = true\n }\n tag {\n key = \"Role\"\n value = \"bastion\"\n propagate_at_launch = true\n }\n lifecycle {\n create_before_destroy = true\n }\n}\n\nresource \"aws_launch_configuration\" \"nat\" {\n name_prefix = \"${var.vpc[\"tag\"]}-nat-lc-\"\n image_id = \"${lookup(var.images, var.provider[\"region\"])}\"\n instance_type = \"${var.nat[\"instance_type\"]}\"\n iam_instance_profile = \"${aws_iam_instance_profile.nat.name}\"\n key_name = \"${var.key_name}\"\n security_groups = [\"${aws_security_group.nat.id}\"]\n associate_public_ip_address = true\n user_data = \"${data.template_file.nat.rendered}\"\n lifecycle {\n create_before_destroy = true\n }\n}\n\ndata \"template_file\" \"nat\" {\n template = \"${file(\"${var.nat[\"filename\"]}\")}\"\n vars {\n cidr = \"${var.vpc[\"cidr_block\"]}\"\n }\n}\n<\/code><\/pre>\nWe create the NAT instance in a Auto Scaling group since being a vital part of the infrastructure we want it to be highly available. This means that in case of a failure, the ASG will launch a new one. This instance then needs to configure it self as a gateway for the public subnets for which we create and attach to it an IAM role with specific permissions. Lastly the instance will use the
userdata_nat_asg.sh<\/code> file (see the variables file) given to it viauser-data<\/code> to setup the routing for the private subnets. The scipt is given below:<\/p>\n#!\/bin\/bash -v\nset -e\necho iptables-persistent iptables-persistent\/autosave_v4 boolean true | debconf-set-selections\necho iptables-persistent iptables-persistent\/autosave_v6 boolean true | debconf-set-selections\nrm -rf \/var\/lib\/apt\/lists\/*\nsed -e '\/^deb.*security\/ s\/^\/#\/g' -i \/etc\/apt\/sources.list\nexport DEBIAN_FRONTEND=noninteractive\napt-get update -y -m -qq\napt-get -y install conntrack iptables-persistent nfs-common > \/tmp\/nat.log\n[[ -s $(modinfo -n ip_conntrack) ]] && modprobe ip_conntrack && echo ip_conntrack | tee -a \/etc\/modules\n\/sbin\/sysctl -w net.ipv4.ip_forward=1\n\/sbin\/sysctl -w net.ipv4.conf.eth0.send_redirects=0\n\/sbin\/sysctl -w net.netfilter.nf_conntrack_max=131072\necho \"net.ipv4.ip_forward=1\" >> \/etc\/sysctl.conf\necho \"net.ipv4.conf.eth0.send_redirects=0\" >> \/etc\/sysctl.conf\necho \"net.netfilter.nf_conntrack_max=131072\" >> \/etc\/sysctl.conf\n\/sbin\/iptables -A INPUT -m conntrack --ctstate INVALID -j DROP\n\/sbin\/iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP\n\/sbin\/iptables -A INPUT -p tcp --syn -m limit --limit 5\/s -i eth0 -j ACCEPT\n\/sbin\/iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT\n\/sbin\/iptables -t nat -A POSTROUTING -o eth0 -s ${cidr} -j MASQUERADE\nsleep 1\nwget -P \/usr\/local\/bin https:\/\/s3-ap-southeast-2.amazonaws.com\/encompass-public\/ha-nat-terraform.sh\n[[ ! -x \"\/usr\/local\/bin\/ha-nat-terraform.sh\" ]] && chmod +x \/usr\/local\/bin\/ha-nat-terraform.sh\n\/bin\/bash \/usr\/local\/bin\/ha-nat-terraform.sh\ncat > \/etc\/profile.d\/user.sh <<END\nHISTSIZE=1000\nHISTFILESIZE=40000\nHISTTIMEFORMAT=\"[%F %T %Z] \"\nexport HISTSIZE HISTFILESIZE HISTTIMEFORMAT\nEND\nsed -e '\/^#deb.*security\/ s\/^#\/\/g' -i \/etc\/apt\/sources.list\nexit 0\n<\/code><\/pre>\nIt configures the firewall and the NAT rules and executes the
ha-nat-terraform.sh<\/code> script fetched from a S3 bucket.<\/p>\nIn the same time we create Security Groups, or instance firewalls in AWS terms, to attach to the subnets and the NAT instance we are going to create:<\/p>\n
\/*=== SECURITY GROUPS ===*\/\nresource \"aws_security_group\" \"default\" {\n name = \"${var.vpc[\"tag\"]}-default\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n ingress {\n from_port = -1\n to_port = -1\n protocol = \"icmp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-default-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"nat\" {\n name = \"${var.vpc[\"tag\"]}-nat\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = -1\n to_port = -1\n protocol = \"icmp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 123 \n to_port = 123\n protocol = \"udp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 80\n to_port = 80\n protocol = \"tcp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n ingress {\n from_port = 443 \n to_port = 443 \n protocol = \"tcp\"\n cidr_blocks = [\"${var.vpc[\"cidr_block\"]}\"]\n }\n egress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = -1\n to_port = -1\n protocol = \"icmp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 80\n to_port = 80\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 443 \n to_port = 443 \n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 123 \n to_port = 123\n protocol = \"udp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 53 \n to_port = 53\n protocol = \"udp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 53\n to_port = 53\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-nat-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"public\" {\n name = \"${var.vpc[\"tag\"]}-public\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 80 \n to_port = 80 \n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"private\" {\n name = \"${var.vpc[\"tag\"]}-private\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"${aws_subnet.private-subnets.*.cidr_block}\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_security_group\" \"private-2\" {\n name = \"${var.vpc[\"tag\"]}-private-2\"\n ingress {\n from_port = 22\n to_port = 22\n protocol = \"tcp\"\n cidr_blocks = [\"${aws_subnet.private-subnets-2.*.cidr_block}\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-2-security-group\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n<\/code><\/pre>\nNext we need to sort out the VPC routing, create routing tables and associate them with the subnets. Create a new file
vpc_routing_tables.tf<\/code>:<\/p>\n\/*=== ROUTING TABLES ===*\/\nresource \"aws_route_table\" \"public-subnet\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n route {\n cidr_block = \"0.0.0.0\/0\"\n gateway_id = \"${aws_internet_gateway.environment.id}\"\n }\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-subnet-route-table\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_route_table_association\" \"public-subnet\" {\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n subnet_id = \"${element(aws_subnet.public-subnets.*.id, count.index)}\"\n route_table_id = \"${aws_route_table.public-subnet.id}\"\n}\n\nresource \"aws_route_table\" \"private-subnet\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-route-table\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_route_table_association\" \"private-subnet\" {\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n subnet_id = \"${element(aws_subnet.private-subnets.*.id, count.index)}\"\n route_table_id = \"${aws_route_table.private-subnet.id}\"\n}\n\nresource \"aws_route_table\" \"private-subnet-2\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-private-subnet-2-route-table\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n}\n\nresource \"aws_route_table_association\" \"private-subnet-2\" {\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n subnet_id = \"${element(aws_subnet.private-subnets-2.*.id, count.index)}\"\n route_table_id = \"${aws_route_table.private-subnet-2.id}\"\n}\n<\/code><\/pre>\nTo wrap it up I would like to receive some notifications in case of Autoscaling events so we create
vpc_notifications.tf<\/code> file:<\/p>\n\/*=== AUTOSCALING NOTIFICATIONS ===*\/\nresource \"aws_autoscaling_notification\" \"main\" {\n group_names = [\n \"${aws_autoscaling_group.nat.name}\"\n ]\n notifications = [\n \"autoscaling:EC2_INSTANCE_LAUNCH\", \n \"autoscaling:EC2_INSTANCE_TERMINATE\",\n \"autoscaling:EC2_INSTANCE_LAUNCH_ERROR\",\n \"autoscaling:EC2_INSTANCE_TERMINATE_ERROR\"\n ]\n topic_arn = \"${aws_sns_topic.main.arn}\"\n}\n\nresource \"aws_sns_topic\" \"main\" {\n name = \"${lower(var.vpc[\"tag\"])}-sns-topic\"\n\n provisioner \"local-exec\" {\n command = \"aws sns subscribe --topic-arn ${self.arn} --protocol email --notification-endpoint ${var.vpc[\"sns_email\"]}\"\n }\n}\n<\/code><\/pre>\nAs we further build our infrastructure we will use more Auto Scaling configurations and we can add those to the above resource under
group_names<\/code>.<\/p>\nAt the end, some outputs we can use if needed:<\/p>\n
\/*=== OUTPUTS ===*\/\noutput \"num-zones\" {\n value = \"${length(lookup(var.azs, var.provider[region]))}\"\n}\n\noutput \"vpc-id\" {\n value = \"${aws_vpc.environment.id}\"\n}\n\noutput \"public-subnet-ids\" {\n value = \"${join(\",\", aws_subnet.public-subnets.*.id)}\"\n}\n\noutput \"private-subnet-ids\" {\n value = \"${join(\",\", aws_subnet.private-subnets.*.id)}\"\n}\n\noutput \"private-subnet-2-ids\" {\n value = \"${join(\",\", aws_subnet.private-subnets-2.*.id)}\"\n}\n\noutput \"autoscaling_notification_sns_topic\" {\n value = \"${aws_sns_topic.main.id}\"\n}\n<\/code><\/pre>\nAt the end we run:<\/p>\n
$ terraform plan -var-file provider-credentials.tfvars -var-file vpc_environment.tfvars -out vpc.tfplan\n<\/code><\/pre>\nto test and create the plan and then:<\/p>\n
$ terraform apply vpc.tfplan\n<\/code><\/pre>\nto create our VPC. When finished we can destroy the infrastructure:<\/p>\n
$ terraform destroy vpc.tfplan --force\n<\/code><\/pre>\nAdding ELB to the mix<\/span><\/h3>\n
Since we are building highly available infrastructure we are going to need a public ELB to put our application servers behind it. Under assumption that the app is listening on port 8080 we can add the following:<\/p>\n
variable \"app\" {\n default = {\n elb_ssl_cert_arn = \"\"\n elb_hc_uri = \"\"\n listen_port_http = \"\"\n listen_port_https = \"\"\n }\n}\n\n<\/code><\/pre>\nto our
vpc_environment.tf<\/code> file and set the values by putting:<\/p>\napp = {\n instance_type = \"<app-instance-type>\"\n host_name = \"<app-host-name>\"\n elb_ssl_cert_arn = \"<elb-ssl-cert-arn>\"\n elb_hc_uri = \"<app-health-check-path>\"\n listen_port_http = \"8080\"\n listen_port_https = \"443\"\n domain = \"<domain-name>\"\n zone_id = \"<domain-zone-id>\"\n}\n<\/code><\/pre>\nin our
vpc_environment.tfvars<\/code> file. Now we can create the ELB by creating thevpc_elb.tf<\/code> file with following content:<\/p>\n\/*== ELB ==*\/\nresource \"aws_elb\" \"app\" {\n \/* Requiered for EC2 ELB only\n availability_zones = \"${var.zones}\"\n *\/\n name = \"${var.vpc[\"tag\"]}-elb-app\"\n subnets = [\"${aws_subnet.public-subnets.*.id}\"]\n security_groups = [\"${aws_security_group.elb.id}\"]\n listener {\n instance_port = \"${var.app[\"listen_port_http\"]}\"\n instance_protocol = \"http\"\n lb_port = 443\n lb_protocol = \"https\"\n ssl_certificate_id = \"${var.app[\"elb_ssl_cert_arn\"]}\"\n }\n listener {\n instance_port = \"${var.app[\"listen_port_http\"]}\"\n instance_protocol = \"http\"\n lb_port = 80\n lb_protocol = \"http\"\n }\n health_check {\n healthy_threshold = 2\n unhealthy_threshold = 3\n timeout = 5\n target = \"HTTP:${var.app[\"listen_port_http\"]}${var.app[\"elb_hc_uri\"]}\"\n interval = 10\n }\n cross_zone_load_balancing = true\n idle_timeout = 960 # set it higher than the conn. timeout of the backend servers\n connection_draining = true\n connection_draining_timeout = 300\n tags {\n Name = \"${var.vpc[\"tag\"]}-elb-app\"\n Type = \"elb\"\n }\n}\n\n\/* In case we need sticky sessions\nresource \"aws_lb_cookie_stickiness_policy\" \"app\" {\n name = \"${var.vpc[\"tag\"]}-elb-app-policy\"\n load_balancer = \"${aws_elb.app.id}\"\n lb_port = 443\n cookie_expiration_period = 960\n}\n*\/\n\n\/* CREATE CNAME DNS RECORD FOR THE ELB *\/\nresource \"aws_route53_record\" \"app\" {\n zone_id = \"${var.app[\"zone_id\"]}\"\n name = \"${lower(var.vpc[\"tag\"])}.${var.app[\"name\"]}\"\n type = \"CNAME\"\n ttl = \"60\"\n records = [\"${aws_elb.app.dns_name}\"]\n}\n<\/code><\/pre>\nThe ELB does not support redirections so the app needs to deal with redirecting users from port 80\/8080 to 443 for fully secure SSL operation.<\/p>\n
Finally, the Security Group for the ELB in
vpc_security.tf<\/code> file:<\/p>\nresource \"aws_security_group\" \"elb\" {\n name = \"${var.vpc[\"tag\"]}-elb\"\n ingress {\n from_port = 80\n to_port = 80\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n ingress {\n from_port = 443\n to_port = 443\n protocol = \"tcp\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n egress {\n from_port = 0\n to_port = 0\n protocol = \"-1\"\n cidr_blocks = [\"0.0.0.0\/0\"]\n }\n vpc_id = \"${aws_vpc.environment.id}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-elb-security-group\"\n Environment = \"${var.vpc[\"tag\"]}\"\n }\n}\n<\/code><\/pre>\nand we are done.<\/p>\n
To get some outputs we are interested in from the ELB resource we can add this:<\/p>\n
output \"elb-app-public-dns\" {\n value = \"${aws_elb.app.dns_name}\"\n}\n\noutput \"route53-app-public-dns\" {\n value = \"${aws_route53_record.app.fqdn}\"\n}\n<\/code><\/pre>\nto the
outputs.tf<\/code> file.<\/p>\nConclusion<\/span><\/h2>\n
As we add more infrastructure to the VPC we can make some improvements to the above code by creating modules for the common tasks like Autoscaling Groups and Launch Configurations, ELB’s, IAM Profiles etc., see Creating Modules<\/a> for details.<\/p>\n
Not everything can be done this way though. For example the repetitive code like:<\/p>\n
resource \"aws_subnet\" \"public-subnets\" {\n vpc_id = \"${aws_vpc.environment.id}\"\n count = \"${length(split(\",\", lookup(var.azs, var.provider[\"region\"])))}\"\n cidr_block = \"${cidrsubnet(var.vpc[\"cidr_block\"], var.vpc[\"subnet_bits\"], count.index)}\"\n availability_zone = \"${element(split(\",\", lookup(var.azs, var.provider[\"region\"])), count.index)}\"\n tags {\n Name = \"${var.vpc[\"tag\"]}-public-subnet-${count.index}\"\n Environment = \"${lower(var.vpc[\"tag\"])}\"\n }\n map_public_ip_on_launch = true\n}\n<\/code><\/pre>\nis a great candidate for a module except Terraform does not (yet) support
count<\/code> parameter inside modules, see Support issue #953<\/a><\/p>\nIt’s graphing feature might come handy in obtaining a logical diagram of the infrastructure we are creating:<\/p>\n
<\/a><\/p>\nTo generate this run:<\/p>\n
$ terraform graph | dot -Tpng > graph.png\n<\/code><\/pre>\n
- \n\t\tConclusion<\/a>\n\t<\/li>\n<\/ul>\n<\/ul>\n<\/div>\n