Networking in AWS

Recently I spent about a month going over building a VPC in AWS. I went in with a beginner's knowledge and came out knowing a lot more, and today I'd like revisit my experiences with you, so you can architect a secure, maintainable network for yourself or your company.

To start out, let's put forth a hypothetical design at a high level. Let's say you have some database instances and some compute instances. The compute instances need to communicate with both the database as well as the public internet, however you want connections to be outbound only. Perhaps something like this.

Let's make our VPC. If you navigate in the AWS Console to the VPC service page and click Create you will get a popup to make your VPC. I'll just be using a 10.0.0.0/16 cidr block and no IPv6 for this example.

Notice in the example above, there is a tag being populated with the key "Name" and the value "My-VPC". This will be populated for you when you fill out the input box for your VPC name. The reason for this is that VPCs don't actually have a name metadata property. The name of the VPC is nothing more than the value of the tag named "Name". This means that the VPC name can be changed at any time without requiring modifications to the VPC (although if other resources are depending on a specific VPC name they will need to be modified as well).

Now we can begin creating resources. Let's start by splitting our network into subnets. A subnet in an AWS VPC is exactly what the name suggests, it's a sub-network with the full set of features available to a VPC as a whole. Each subnet can be given it's own route table, network ACLs and cidr blocks. This is the first thing we will be looking at to solve our hypothetical problem. We will use 3 subnets, lets call them "public", "private" and "isolated". There's a reason we need 3 here. We want to be able to give each subnet its own routes and network ACLs, as we're designing this with security and maintainability in mind. We would place our compute in the private subnet, and the database into the isolated subnet. We will use the public subnet to route traffic out to the internet, the private subnet to route traffic to resources within our vpc and the isolated subnet will only be able to route to the local network.

To create our subnets we first need to know what cidr blocks to use. This is where we need to do some math to figure out how to define those blocks.

We used 10.0.0.0/16 as our cidr block for the VPC, and we want 3 subnets. Unfortunately since everything with computers is in binary, we can't divide the network into 3 subnets. The closest we can get is 4 subnets. That /16 at the end of our IP block is the number of bits in the subnet mask. We can increment this to create subnets and since we want 3 we need to increment this by 2, because 2^2 is 4, and 4 is the smallest value we have that is greater than the number of subnets we want. So our subnet mask will be /18 and with 4 subnets we get the cidr blocks 10.0.0.0/18, 10.0.64.0/18, 10.0.128.0/18 and 10.0.192.0/18. Unfortunately we are only using 3 of these so the last cidr block will just be untouched.

On the sidebar navigate to the "subnets" section and click "Create subnet" and you can create you subnets there. Multiple subnets can be created on this page. Notice that once again we have the "Name" tag appearing here. VPC subnets follow the same naming process as VPCs.

Your subnet table should look similar to this now.

Notice that the route table and network ACL are the same for all of these subnets. This is because they are not actually bound to these subnets, but rather are the default route table and default network ACL created for your VPC and were automatically assigned to your subnets. We don't want this for our use case, we want to create separate route tables and network ACLs for each subnet, so let's go ahead and do that.

Click the "Route tables" button in the sidebar and click "Create route table". This will give you a very simple form asking for the route table name and a dropdown where you will select the VPC to place it in. Once again the name is nothing more than a tag.

You should now have 4 route tables. 3 that you just created and 1 that was created with your VPC. The default route table created with your VPC will not have a name unless you explicitly name it.

Navigating to the Public route table we will now associate the route table with our Public subnet. Click the "Subnet Associations" button and then click "Edit subnet associations". You should be presented with a table of all of your subnets, we can now select the Public subnet and click "Save associations"

Once you repeat this process for your Private and Isolated subnets, each subnet will route traffic differently based on the contents of their route tables. This means that the default route table associated with your VPC is no longer used, as we only have 3 subnets and each has an explicitly associated route table. Unfortunately the default route table cannot be deleted, so it will have to just sit there unused. You may want to name it something to remind you of this, or notify your teammates that this route table is unused.

Now we need to create 2 resources to help route our traffic. We need an Internet Gateway to send packets out to and receive packets from the internet. We will also need a NAT Gateway to allow resources inside our network to communicate outside of the VPC.

Internet Gateways do not attach to subnets, only to VPCs. Lets make that first. Click the "Internet gateways" button in your sidebar and click "Create internet gateway". All it requires is a name, which as you may have guessed by now is just a tag.

You should see this once you create your internet gateway, and you now need to attach it to your VPC. Simply click the "Attach to VPC" button, or click Actions -> Attach to VPC. Select your VPC from the dropdown and attach it.

Now we need a way to send traffic from inside our subnet where we use private IP addresses, to the outside world with a public IP address. In order to do this we need a NAT Gateway, but the gateway will need to know what public IP address is allocated to it. Let's make an elastic IP address. In the sidebar click Elastic IPs and click Allocate Elastic IP address. A network border group will be pre-selected for you, though there is no name field. If you want to name this elastic IP go ahead and add a new tag, name it "Name" and the value will be whatever you want the name to be.

Now we can create our NAT Gateway. Navigate to the NAT Gateway creation page and give your gateway a name. For our subnet, we want to choose our Public subnet. This is because we will be creating a route in this subnet's route table later and that route will point to the internet gateway. This means that all traffic coming out of the NAT Gateway will take the route to the Internet Gateway and go out of your network, and vice versa for inbound connections if you allow any. Connectivity type here will be set to Public, as private NAT Gateways can't reach the internet. Now simply select your elastic IP from the drop down and hit Create.

Now we are ready to create our routes. We only need 2 routes for this example, one to the Internet Gateway in our public subnet and one to the NAT Gateway in our private subnet. The isolated subnet will not have any special routes here.

Navigate to your route tables and add a route to the public route table. The destination will be 0.0.0.0/0 (everywhere outside of this network) and the target will be your Internet Gateway. When you click the target input field you will be given a dropdown to select from.

We will do the same thing for our Private route table, however instead of the internet gateway the target will be our NAT Gateway.

You may be thinking now, why did we make a route to the NAT Gateway in the private subnet route table if the gateway is attached to the public subnet? It's because we plan to place our compute inside of the private subnet. Let's say you have some Lambda running that needs to send an HTTP request to Google. That Lambda will resolve Google's public IP address and emit packets directed to that address. Since the target IP is not within our VPC it needs to go out to the internet and this can only happen if the packet is processed by our NAT Gateway. Since our 0.0.0.0/0 route points there, that packet will go to the gateway and be processed. Once that is done the NAT Gateway will emit a new packet targeted at the Google public IP address. Since our NAT Gateway is in the public subnet our 0.0.0.0/0 route points to the Internet Gateway and the packet can now exit our VPC and go to Google.

We now have our subnets and route tables set up, we are able to run our applications now. However there is still a major security hole here in that we have not modified our network ACLs. Each subnet has a network ACL attached to it but the one attached to our subnets currently is the default ACL that came with our VPC. We need to have separate ACLs for our subnets. Lets navigate to the Network ACLs page and click Create network ACL. You'll need to select your VPC from the dropdown, and lets name our ACLs Public, Private and Isolated.

We will now remove the ALLOW rule from our default network ACL. Select the default ACL (the one without a name) and under the Inbound rules tab, click edit inbound rules and remove the allow rule.

This will block all inbound network traffic to your network until we add our rules to the other ACLs and associate them with your subnets. If you need to prevent downtime you may want to do that first and then come back to this step.

Now lets go add the appropriate inbound rules to our ACLs. For our public ACL let's just say that the only inbound traffic will be responses for our HTTPs requests. This means that the only port being used will be 443, so we can add that as an allow.

Now we need to associate this ACL with our Public subnet. In the ACLs table select the Public ACL and click on the Subnet associations tab, then select your Public subnet and associate it. This will allow all inbound traffic on port 443 to enter your Public subnet and hit your NAT Gateway.

For our Private ACL we need to know what port the NAT Gateway will emit packets on when it sends packets into the Private subnet. AWS has a list of ephemeral ports we can refer to so that we can allow the correct ports in our Private network ACL. A NAT Gateway will emit packets on the port range 1024-65535, so we will allow those ports in our ACL but since this is the Private subnet we only want to allow traffic that comes from within our VPC, so we specify our VPC cidr block in the source.

For our Isolated subnet we will only be allowing a single port so that our compute instances can connect to our database instances. Lets just say we are using PostgreSQL databases and so we need to allow incoming traffic on port 5432, as with our Private subnet we only want traffic from our network, so we specify our VPC cidr as the source.

Now we save this rule and associate this ACL with our Isolated subnet.

At the end of this all we now have a (relatively) well architected VPC. We did waste an entire cidr block, but in practice you would likely be using 6 or 9 subnets instead of 3 and would have more efficient IP distribution. Thanks to our subnetting we know exactly how traffic is routed for our applications and if something goes wrong we will be able to debug the issue fairly quickly.

Edit: For those of you who would prefer infrastructure as code I have a post for that too