EC2 real network bandwidth

EC2 instances does not always provide exact numbers on their network capacity. In some cases where we need to optimize our network traffic we might need to know the exact numbers for our own instances.

iperf & Terraform

One way to achieve this is using iperf3. This tool uses a client-server model, which is ideal for testing network performance between two hosts. The server listens for incoming test requests, and the client initiates the test, making it easy to measure the capacity of the network link between them.

The next step involves setting up two EC2 instances. The first will act as a server, which will help us determine its actual network capacity. The second will act as a client. The client instance should have a higher network capacity than the server to ensure that the server's network link is fully utilized during the test.

To implement this, we need to create a pair of EC2 instances along with their corresponding resources, such as roles and security groups, in our AWS account. Doing this manually for every EC2 instance type we need to measure could be tedious, so we'll use Terraform for this task.

Following are the important parts of terraform code to setup this test environment to measure network capacity of m3.medium instances.

resource "aws_instance" "iperf_server" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "m3.medium"

  subnet_id       = data.aws_subnet.current.id
  security_groups = [aws_security_group.iperf.id]

  iam_instance_profile = aws_iam_instance_profile.iperf.name

  tags = {
    Name = "${var.name}-iperf-server"
  }

  key_name = aws_key_pair.iperf.key_name
  connection {
    host        = coalesce(self.public_ip, self.private_ip)
    type        = "ssh"
    user        = var.ssh_user
    private_key = tls_private_key.iperf.private_key_pem
  }

  provisioner "remote-exec" {
    inline = [
      "sudo yum -y update",
      "sudo yum -y install iperf3"
    ]
  }
}

resource "aws_instance" "iperf_client" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "m5.12xlarge"

  subnet_id       = data.aws_subnet.current.id
  security_groups = [aws_security_group.iperf.id]

  iam_instance_profile = aws_iam_instance_profile.iperf.name

  tags = {
    Name = "${var.name}-iperf-client"
  }

  key_name = aws_key_pair.iperf.key_name
  connection {
    host        = coalesce(self.public_ip, self.private_ip)
    type        = "ssh"
    user        = var.ssh_user
    private_key = tls_private_key.iperf.private_key_pem
  }

  provisioner "remote-exec" {
    inline = [
      "sudo yum -y update",
      "sudo yum -y install iperf3"
    ]
  }
}

resource "aws_security_group" "iperf" {

  name = "${var.name}-iperf-seg"
  description = "Allow inbound traffic for iperf"

  vpc_id = data.aws_vpc.current.id

  ingress {
    from_port = 0
    to_port   = 0
    protocol  = "-1"
    self      = true
  }

  ingress {
    from_port   = 5001  # Assuming iperf server listens on port 5001
    to_port     = 5001
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "port 22"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = var.ssh_cidr_ingress_blocks
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    "Name" : "${var.name}-iperf-seg"
  }

}

output "server_private_ip" {
  value       = aws_instance.iperf_server.private_ip
}

output "server_public_ip" {
  value       = aws_instance.iperf_server.public_ip
}

output "client_public_ip" {
  value       = aws_instance.iperf_client.public_ip
}

Conducting the test

Once we have created our test env, we just need to ssh into the server instance and start iperf in server mode

$ iperf -s

Then we’ll ssh into the client and start the test attaching to the server using its private ip to reduce extra network overhead.

$ iperf -c <server_private_ip>

If we plan to test multiple instance types, we can create a script to run the tests as follows.

terraform_path=.

vars=$(terraform -chdir=$terraform_path output -json)
client_public_ip=$(echo $vars | jq -r '.client_public_ip.value')
server_public_ip=$(echo $vars | jq -r '.server_public_ip.value')
server_private_ip=$(echo $vars | jq -r '.server_private_ip.value')

keypair="$terraform_path/iperf-keypair.pem"

ssh -i $keypair \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    ec2-user@$server_public_ip "iperf3 -s -D"

ssh -i $keypair \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    ec2-user@$client_public_ip -t "iperf3 -c $server_private_ip -t 150 -i 60"

Real figures

I ran this test for some instance types. The results are as follows:

Instance Type	Bandwidth
m3.medium	681 Mbits/sec
c4.large	628 Mbits/sec
m4.large	768 Mbits/sec
r4.large	9.53 Gbits/sec
i3.large	4.97 Gbits/sec
t2.nano	1.04 Gbits/sec
t3.medium	4.52 Gbits/sec

Remember that burstable instances, such as the t2 and t3 families, incorporate the concept of network credits. These credits permit them to "burst" their network performance beyond the baseline level for short periods, as needed. Therefore, the figures in the table are not directly comparable between burstable and non-burstable instance types.

To establish the baseline network performance of burstable instances, we need to conduct the test for a duration sufficient to use up the accumulated network credits.