DEV Community: Jeff Loughridge

First Class Container Networking with EC2 IP Prefix Assignments

Jeff Loughridge — Thu, 05 Aug 2021 01:47:05 +0000

Container networking in the AWS VPC is now much simpler following AWS’s announcement of EC2 IP prefix assignments. Want to get rid of overlays and bridge networking? Let’s examine how the new IP prefix assignment functions and how it can be used to enable containers as first class citizens on the network.

Feature Overview

Prior to the announcement of this feature, there was no way to reserve a contiguous block of IPv4 or IPv6 prefixes for use by an EC2 instance. The new feature provides this functionality for multiple IPv4 and IPv6 prefixes. The maximum prefix size for IPv4 is /28 and IPv6 is /80.

The number of prefixes you can assign is limited by the number of IP addresses supported by the instance type as documented at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html. The new limits are based on the cumulative number of addresses and prefixes. The documentation gives the example of a c5.large instance which has a limit of 10 IPv4 addresses and 10 IPv6 addresses on a network interface. Therefore, the total number of /28 and /80 prefixes must be less than 10.

A major benefit of this approach is the cost savings associated with additional IP addresses on less expensive instances. Previously the requirement for a large number of IP addresses drove the selection of more expensive instances. Now the small instances can support more IP addresses.

The EC2 IP prefix assignment feature–as of August 2021–is only supported by the latest release of the AWS v2 CLI. You cannot assign prefixes to elastic network interfaces (ENI) in the Console or CloudFormation. The prefix assignment can be automatic or manual. With automatic assignment, a /28 or /80 is selected from the subnet in which the ENI resides. The user can request a /28 or /80. With either mode, the prefix can be assigned at the time of ENI creation or at a later time.

Container Networking

In the ideal scenario for container networking, each container gets a unique IP address on the primary network substrate. NAT, bridges, and overlays are obviated. There are a number of ways to accomplish this including the docker macvlan and ipvlan network modes. AWS’s VPC networking implementation doesn’t play nicely with multiple MAC addresses so macvlan is a nonstarter. In the ipvlan approach, multiple IP addresses share the same MAC address of the host’s ENI.

Since the IPv4 prefix assignment is for private IPv4 addresses only, the EC2 IP prefix assignment feature really shines for IPv6 as you can assign globally reachable IPv6 addresses. Offering an IPv6 service may not be viable for serving content to end users although many use cases exist for IOT and other machine-to-machine communication. All is not lost for IPv4 because you could put a load balancer in front of the containers and communicate with them using private IPv4 addresses.

It's Demo Time

Let’s configure a docker host in a public subnet and run webservers in containers with IPv6 addresses. You can follow along with this demo using the terraform at https://github.com/jeffbrl/terraform-examples/tree/master/networking-with-ec2-prefixes.

Here is the network configuration used in my templates (note that I am using the private IPv4 block from RFC6598).

VPC IPv4 CIDR - 100.64.0.0/16

VPC IPv6 CIDR - Assigned dynamically from Amazon IPv6 space

Public “1A” Subnet - 100.64.0.0/24

Because the AWS provider for terraform currently lacks support for EC2 IP prefixes, I use the local-exec provisioning on a null resource to call the AWS CLI, specifically ‘aws ec2 assign-private-ip-addresses’ and ‘aws ec2 assign-ipv6-addresses”.

Once the docker host is online, I request a /80 and /28 to be associated with the host’s ENI using the AWS CLI.

resource "null_resource" "ec2_prefix_assignment_v6" {
  provisioner "local-exec" {
    command = "aws ec2 assign-ipv6-addresses --network-interface-id ${aws_network_interface.docker_eni.id} --ipv6-prefix-count=1 --region us-east-1"
  }
}

resource "null_resource" "ec2_prefix_assignment_v4" {
  provisioner "local-exec" {
    command = "aws ec2 assign-private-ip-addresses --network-interface-id ${aws_network_interface.docker_eni.id} --ipv4-prefix-count=1 --region us-east-1"
  }
}

Now we’ll configure the docker network on the host. We want the docker host to dole out IPs from the ranges obtained above while the container should recognize that they are attached to the 100.64.0.0/24 and IPv6 /64 associated with the public “1A” subnet.

sudo docker network create --ipv6 -d ipvlan -o parent=eth0 --subnet 100.64.0.0/24 \
--ip-range=$ipv4_prefix --subnet $public_subnet_ipv6 --ip-range $ipv6_prefix dockernet

The “ipv4_prefix” and “ipv6_prefix” are the /28 and /80 obtained from the request for EC2 IP prefixes.

We’ll execute three containers based on a custom container image I built that displays the IPv4 and IPv6 addresses of the container.

sudo docker run --rm -d --net dockernet gcr.io/stately-minutia-658/jweb
sudo docker run --rm -d --net dockernet gcr.io/stately-minutia-658/jweb
sudo docker run --rm -d --net dockernet gcr.io/stately-minutia-658/jweb

Docker hands out IPv6 consecutively from the base of the IPv6 prefix we used in the “--ip-range” parameter. We can observe the IPv6 address of a given container using “docker inspect ”.

Navigating to three URLs using the bracket syntax for IPv6 literals yields the following.

Voila! We have a docker container host in which each container has a globally unique IPv6.

Conclusion

This post described the new AWS VPC feature for assigning contiguous IPv4 and IPv6 prefixes to EC2 instances (the ENIs to be specific). I demonstrated how this drastically simplifies container networking, making containers first class citizens on the network rather than being stuck behind NAT or other kludges. I’m eager to experiment further with this feature particularly in its uses for networking appliances in the cloud.

VPC Ingress Routing - Reducing the Friction in Integrating Security Appliances

Jeff Loughridge — Sun, 17 Jan 2021 15:42:32 +0000

Designing cloud infrastructures that include security appliances for traffic inspection is easier following Amazon's introduction of a feature called VPC Ingress Routing. In this article and accompanying video, I'll examine the feature and how you can use it to force inbound traffic though security appliances.

Companies that operate in highly regulated environments frequently require security appliances to meet compliance requirements. These appliances might take the form of firewalls, intrusion prevention systems (IPS) and intrusion detection systems (IDS). The AWS Marketplace has a variety of appliances you can choose from.

Prior to VPC ingress routing, there were several models for traffic inspection for a service intended for the public Internet. The first--which I've covered in a previous post--uses Elastic IPs attached to the appliances. Your service is implemented on EC2 instances in the private subnets. Traffic is "drawn" into the VPC with the two EIPs. Here is a diagram.

The firewalls may perform a NAT function is routing symmetry is needed. Otherwise, return traffic will go directly to the Internet Gateway (IGW).

This architecture does not scale as it relies on EIPs, which are limited to five per region. While this limit can be increased, I advise against building a service that relies on EIPs. There are better design such as the load balancer sandwich. Check out Palo Alto Network's reference design for more details.

If you're like me, you may get the impression that using these appliances designed for the data center can feel like "going against the grain" of cloud. I think was definitely true in the early days of the VPC. AWS's introduction of VPC ingress routing in 2019 and its more recent launch of Gateway Load Balancing are significant improvements.

What functionality does VPC ingress routing provide and how does it help in the integration of security appliances? Using VPC ingress routing, we can launch our instances in the private subnet using ephemeral public IPs. Note that these are not EIPs. This fact alone improves the scaling properties of the design.

Using public IPs in private subnets may seem counterintuitive. How does traffic from the Internet get routed to the EC2 instances that power your service? The "secret sauce" is the ability to associate a routing table with the IGW. On ingress, the IGW translates the public IP to the private IP in the VPC, this is always the case and isn't specific to VPC ingress routing. What's different in this scenario is that the destinations for the private subnet prefixes are the Elastic Network Interfaces (ENI) of the security appliances. Let's go to the diagram!

The routing table depicted is linked to the IGW using what AWS calls an "edge association." This is how inbound traffic is forced to traverse the security appliance.

Here's how this looks in the AWS console.

Now we can launch multiple services in the private subnets without being limited by the number of EIPs per region.

That's VPC ingress routing in a nutshell. The feature is designed for a very specific use case that may be applicable to your workloads if you need to inspect North-South traffic for service offered to the public Internet.

Check out the video version of this blog post on the YouTube.

A Taxonomy of IP Addresses in the AWS VPC

Jeff Loughridge — Wed, 30 Dec 2020 20:53:17 +0000

Whether you are a developer or network engineer, designing and implementing services in the AWS VPC requires an understanding of IP networking. My objective in this post about classifying IP addresses in the AWS VPC is to teach you one thing you didn't previously know. Let's dive in.

The first way that IP addresses can be classified is by protocol. At the time of launch, the VPC supported only IPv4 addresses. AWS has incrementally added support for IPv6 although it currently lacks parity with the legacy version of the protocol. (For more on IPv6 in the VPC, check out this post.) In this article, I will reference the protocol version if it is relevant to the point I'm making. If I use "IP" by itself, the statement applies to both protocol versions.

If you've studied for AWS certification, you may expect me to jump into discussing public and private IP addresses. I'm going to suggest that you think about classifying IP addresses differently. Let's go with internal and external.

Internal IP Addresses

Internal IP addresses are tied to your VPC at the time of creation. They are used to uniquely identify endpoints of the components in the VPC from a networking perspective. You will see the internal IP addresses when you log into an EC2 instance and display the state of the network interfaces.

[ec2-user@ip-100-64-15-154 ~]$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 12:31:da:d4:36:47 brd ff:ff:ff:ff:ff:ff
    inet 100.64.15.154/20 brd 100.64.15.255 scope global dynamic eth0
       valid_lft 3151sec preferred_lft 3151sec
    inet6 2600:1f18:5cc:2900:a18b:129d:3082:4fa0/128 scope global dynamic
       valid_lft 395sec preferred_lft 95sec
    inet6 fe80::1031:daff:fed4:3647/64 scope link
       valid_lft forever preferred_lft forever
[ec2-user@ip-100-64-15-154 ~]$

In the above output, there is one internal IPv4 address, 100.64.15.154, that is tied to the Elastic Network Interface (ENI) of this EC2 instance.

AWS places few restrictions on what internal addresses can be used within the VPC. The most common choice is one that's been carried over from traditional "big iron" networking: private IPv4 addresses documented in RFC1918. These IPv4 prefixes are 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16.

I'm a proponent of using RFC6598 IPv4 addresses as internal addresses. This block was reserved by the IETF--the Internet standards body--for carrier-grade NAT but there's no harm in using it for the VPC. I like that these addresses stand out from IPv4 addresses in corporate networks, data centers and home networks. For companies that maintain hybrid infrastructure, there's no concern for IPv4 address overlap as you don't see RFC6598 IPv4 addresses outside of service provider networks.

Get this. You can use public IPv4 address to number your VPC. This has nothing to do with Bring Your Own IP (BYOIP), which I'll touch on later. Don't believe me? Try it. Since this address space is internal to your VPC, AWS allows you to use these addresses. You don't have to own the IPv4 address space although "squatting" on other companies' IPv4 addresses is ill-advised.

External IP Addresses

External IP addresses are addresses capable of being routed on the public Internet. Typically these addresses are owned by Amazon and tied to a particular AWS region.

I classify external IPv4 addresses into ephemeral and Elastic IPs. You won't see the term ephemeral in AWS documentation. Instead, you'll see public used in AWS documentation and the reader is expected to know from the context whether these are EIPs or not. I like to make this distinction for clarity's sake.

Ephemeral IP addresses are assigned by AWS for VPC components such as EC2 instances. These IPv4 addresses can change and should not be expected to be fixed. For many use cases, that these IP address can change is acceptable. We use the indirection that is DNS for a reason.

EIPs are fixed IPv4 addresses that are assigned to VPC components when a static address is needed. You may have used these for EC2 instances or load balancers. EIPs were previously used exclusively in public subnets. Following the introduction of VPC ingress routing, you can assign EIPs to instances in private subnets in advanced architectures in which services are protected by a firewall.

You can port public IPv4 addresses that you own to be used as EIPs in the VPC. AWS refers to this as BYOIP, which is available in most but not all AWS regions. Once AWS verifies that you own this IPv4 address space, the pool will show up in your account to be used as EIPs. BYOIP isn't relevant to the majority of AWS customers although it has interesting use cases for some enterprises. You can also port IPv6 addresses but this an advanced topic that I'm going to omit from this article.

External IPv4 addresses will not show up when examining the state of network interfaces on an EC2 instance. The instance is not aware of these addresses. There are discovery mechanisms available, the most simple being the use of curl at the command line to sites such as https://ifconfig.me.

External IP addresses are the key to outbound Internet access. To perform basic tasks on an EC2 instance such as downloading OS packages, python modules or git repos, the instance must have an external IP or use another component in the VPC that has a public IP (e.g., NAT Gateway, NAT instance, firewall) as transit device.

You may be wondering how an EC2 instance with an internal IPv4 address communicates with the IPv4 Internet. The VPC enables this communication using Network Address Translation (NAT) bindings at the Internet Gateway (IGW). The internal address is converted to the external address for outbound traffic and vice versa for the return traffic.

IPv6

Let's return to our discussion of IPv6. IPv6 is not easily classified into internal or external. To understand why, consider the availability of IP addresses. Public IPv4 addresses are scarce and can no longer be obtained from the regional Internet registries such as ARIN and RIPE. We've consumed the approximately four billion addresses at a fast clip and the only reason IPv4 is still viable is that we've gotten really good at NAT.

IPv6 addresses are plentiful given the 128 available bits for addressing. When you create a VPC with an IPv6 prefix, AWS provides a globally unique /56 IPv6 prefix. For those not familiar with Classless Interdomain Routing (CIDR) notation, the "/56" denotes the number of bits that specify the network portion of the addresses. Unlike IPv4, IPv6 has a standard subnet size of /64. This means a /56 can accommodate 2**8, or 256, /64s. If your VPC requires more than 256 subnets, you can assign a secondary IPv6 prefix although you may instead want to contact me to discuss your infrastructure design. ;)

IPv6 addresses are used for communication within the VPC and the Internet. There is no concept of EIPs with IPv6. The IGW does not do address translation. If you are not comfortable with using security groups and NACLs as the sole protection from inbound Internet access, you may want to use AWS's Egress-Only Gateway. It is analogous to a NAT Gateway as it is horizontally-scaled and redundant. Internet end systems will not be able to initiate communication with instances behind an Egress-Only Gateway.

Conclusion

When I was throwing around ideas about writing about IP addresses in the VPC, I considered using a title along the lines of "Everything you need to know about IP addresses in the VPC" but I decided to scrap that as a comprehensive guide to IP addresses would be a very long blog article. Instead, I took the route of classifying IP addresses in a way that you may not have considered. My hope is that the classification I've presented gives you a mental model of IP addressing that provides a foundation for understanding networking in the VPC.

The Path to IPv6-Only in the AWS VPC

Jeff Loughridge — Thu, 26 Nov 2020 20:03:55 +0000

The recent announcements on IPv6 support for the Network Load Balancer and Fargate reminded me of AWS's steady progress towards more widespread coverage for the current version of IP. The EC2 networking team deserves credit for the advances made since IPv6 was first supported on the Classic Load Balancer back in 2011. I believe that success should not be measured in IPv4/IPv6 parity alone. The end state that we need is a fully viable IPv6-only VPC.

Why IPv6-Only

A dual stack network configuration is an intermediate step. This gives customer experience with IPv6; however, that the IPv4 stack remains is a crutch. Can you download OS packages from an obscure Ubuntu PPA over IPv6? Can you ship your logs to your favorite third-party logging service? You probably don't know in a dual stack environments. These IPv4 dependencies are cloaked when you have an IPv4 stack to fall back on.

Dual stack represents added operational overhead and complexity. Now your security groups, NACLs, WAF rules and S3 bucket policies must permit access from the appropriate IPv4 and IPv6 prefixes. Your automation that processes LB logs must handle both IP versions. Your network security tools must include policies for IPv4 and IPv6. Your Route 53 domains require A and AAAA records for every resource. During troubleshooting, Ops has to take into account two protocol stacks and possibly unforeseen and browser-dependent interaction between them (think the Happy Eyeballs algoritm).

Dual stack does nothing to solve IPv4 depletion. For workloads running in AWS VPCs, AWS is well-positioned to shield its customers from IPv4 address depletion. According to a recent blog post by Andree Toonk, Amazon owns approximately 100M IPv4 addresses. That's a ton! One could argue that running workloads on the AWS VPC platform helps extend the lifetime of IPv4 with its massive collection of public IPv4 addresses and the ease of using multiple private IPv4 addresses from the RFC1918 and RFC6598 address blocks. This argument has validity; however, the abundance of IPv4 addresses within AWS won't help your workloads communicate with end systems that only have an IPv6 address. The regional Internet registries do not have additional IPv6 prefixes to allocate. New deployments would need to find IPv4 addresses from the transfer market or must be able to scale NAT.

Required Enhancements

AWS doesn't currently provide the capability to launch an IPv6-only VPC. It can't. Crucial gaps exist in IPv6 support.

Here are a few examples.

EC2 metadata
CloudFront fetch from origin
~~API service endpoints~~
Load balancer to instance traffic including health checks
Route 53 resolver
Lambda ENIs

Update 1/18/2021 - AWS added support for IPv6 for the EC2 API in January 2021. Nice work!

I want to address the lack of support for IPv6 on the Lambda ENIs that are instantiated when Lambda is launched in a VPC. This is an important one for me as I frequently us Lambda to monitor and maintain VPC infrastructure. From health checks to reporting, I see Lambda as crucial in keeping the infrastructure functioning.

There are many other services that currently lack IPv6 support, and I'm sure all IPv6 proponents have their respective "want" lists.

Getting to an IPv6-only VPC involves more than enabling IPv6 on all AWS services. We need a way to set a flag such that the EC2 mapping service is completely turned off for IPv4. I believe this is a better approach than attempting to use SGs or NACLs to prevent IPv4 communication. This would eliminate the IPv4 stack as an attack vector.

Even though modern OSes will prefer IPv6, I'd like to see popular AMIs tweaked such that they do not configure IPv4 addresses. Sure, you can bake your own AMIs that do this. Why not make it easy?

The Use Cases for IPv6-Only VPCs

While an IPv6-only service is not suitable for public web applications, there are services that do not need IPv4. The Internet of Things (IoT) is one example. IPv6 allows IoT devices to overcome the limitations of NAT and restores end-to-end reachability. This means any device can communicate directly with any other device and IPv6 servers on the Internet can initiate connections. Recall that NAT forces the devices with private IPv4 addresses to initiate the communication. Inbound connectivity in the IPv4 world requires port forwarding that has very poor scaling properties. If are building services in AWS that need to communicate with IoT devices, considering architecting a solution based on IPv6 transport.

Telecoms have been leaders in IPv6 deployments. The benefits of running IPv6 workloads in the VPC for managing mobile infrastructure mirror that of IOT: end-to-end reachability and the avoidance of NAT complexity. Network monitoring, compliance, performance analytics and handset management are functions that can be performed more elegantly and at reduced cost over IPv6 when compared to IPv4. The alternative is often re-using RFC1918 IPv4 addresses to create confederations that cannot communicate with one another without NAT.

Conclusion

The legacy IP protocol plays an outsized role in the VPC platform. As the world continues it march toward a ubiquitous IPv6 Internet, I expect AWS to continue to IPv6-enable its services. For workloads that do not need to communicate with the long tail of IPv4-only endpoints, AWS customers would benefit from the ability to instantiate IPv6-only VPCs. AWS prides itself for adding features based on customer demand so let's encourage the company to put resources behind this enhancement.