Setting Up a NAT Gateway for Outbound Calls to a Model API

#ai #machinelearning #llm #tutorial

Creating a NAT gateway is four commands and takes about three minutes, most of which is waiting. The interesting part is what happens two months later when a fan-out job opens tens of thousands of connections to one provider and the gateway starts dropping packets without telling anybody.

Provisioning it

A public NAT gateway needs an Elastic IP and a public subnet — meaning a subnet whose route table already sends 0.0.0.0/0 to an internet gateway. It does not go in the subnet it serves.

# 1. allocate an Elastic IP
ALLOC=$(aws ec2 allocate-address --domain vpc --query AllocationId --output text)

# 2. create the gateway in the PUBLIC subnet
NAT=$(aws ec2 create-nat-gateway \
  --subnet-id subnet-0public1a \
  --allocation-id "$ALLOC" \
  --connectivity-type public \
  --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=egress-1a}]' \
  --query NatGateway.NatGatewayId --output text)

# 3. wait for it
aws ec2 wait nat-gateway-available --nat-gateway-ids "$NAT"

One creation-time failure is worth knowing in advance because the error is opaque: AWS documents that the network border group of the Elastic IP must match the network border group of the Availability Zone you are launching into, and that the gateway will fail to launch if it does not. If create-nat-gateway succeeds and the state later goes to Failed rather than Available, that mismatch is the first thing to check.

The same thing in Terraform, since this is rarely a one-off:

resource "aws_eip" "egress_1a" {
  domain = "vpc"
}

resource "aws_nat_gateway" "egress_1a" {
  allocation_id     = aws_eip.egress_1a.id
  subnet_id         = aws_subnet.public_1a.id
  connectivity_type = "public"
  tags              = { Name = "egress-1a" }

  depends_on = [aws_internet_gateway.main]
}

resource "aws_route" "private_1a_default" {
  route_table_id         = aws_route_table.private_1a.id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.egress_1a.id
}

The depends_on is not decorative. Without an internet gateway attached to the VPC, the NAT gateway comes up and does nothing, and Terraform’s implicit dependency graph does not otherwise connect the two.

The route, and which table it goes in

The route belongs in the private subnet’s route table, pointing at the gateway. Adding it to the public subnet’s table is the classic inversion and produces a loop rather than an error.

aws ec2 create-route \
  --route-table-id rtb-0private1a \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id "$NAT"

If a 0.0.0.0/0 route already exists in that table you must replace-route rather than create-route; a route table holds exactly one route per destination and the create call will fail with RouteAlreadyExists.

Confirming it reaches the provider

Test from inside the private subnet, not from your laptop. The most useful single check gets you both the reachability answer and the source address the provider will see:

# from a task or instance in the private subnet
curl -sS -o /dev/null -w 'status=%{http_code} connect=%{time_connect}s\n' \
  https://api.anthropic.com/v1/messages

# what the outside world sees as your source IP
curl -sS https://checkip.amazonaws.com

A 401 from the first command is the result you want: the connection completed, TLS negotiated, and the API answered — it simply did not like your (absent) credentials. The second command should return the Elastic IP you allocated. If it returns something else, the task is not routing through the gateway you think it is, and the route table association is where to look.

The address that comes back is also the one you hand a provider that supports IP allowlisting on their side, and the one your own outbound firewall rules should reference. Keep it stable: releasing and reallocating the Elastic IP silently invalidates whatever agreements were built on it. Allowlisting in the other direction is covered in allowlisting a model provider’s IP ranges.

The port allocation ceiling

This is the failure this page exists for, because it appears only under load and it does not look like a networking problem. AWS documents that each IPv4 address on a NAT gateway supports up to 55,000 simultaneous connections to each unique destination, where a unique destination is the combination of destination IP, destination port and protocol — see AWS on working with NAT gateways.

Model API traffic is unusually good at reaching that ceiling. Every request goes to the same hostname on the same port, so every connection you open counts against the same 55,000. A batch job that fans out across thousands of workers, each holding a long-lived streaming connection open for tens of seconds, concentrates load in exactly the dimension the limit is measured in. What you observe when you cross it is intermittent connection failures and latency spikes that correlate with concurrency and with nothing else.

The two CloudWatch metrics that tell you this is what is happening:

ErrorPortAllocation — the gateway could not allocate a source port. Any non-zero value here is the diagnosis.
PacketsDropCount — packets dropped by the gateway. Useful corroboration.

The remedy AWS documents is more addresses: a NAT gateway can carry up to 8 IPv4 addresses (one primary and seven secondary). For a public gateway those are Elastic IPs, and the default quota is 2 Elastic IP addresses per public NAT gateway, adjustable up to 8. Each address adds another 55,000 ports against the same destination.

aws ec2 associate-nat-gateway-address \
  --nat-gateway-id "$NAT" \
  --allocation-ids eipalloc-0secondary1 eipalloc-0secondary2

The 55,000 figure, the 8-address maximum and the default quota of 2 Elastic IPs per public NAT gateway are what AWS publishes at the time of writing. Quotas move; check the VPC quotas page before designing against the number rather than the mechanism.

Before adding addresses, check whether connection reuse fixes it more cheaply. Most SDKs will pool HTTP connections if you let them, and a worker that creates a new client per request is opening a new connection per request for no reason. Fewer, longer-lived connections is both the cheaper fix and the faster one.

What it costs and what it does not survive

Amazon’s Amazon VPC pricing page lists a NAT gateway in US East (N. Virginia) at $0.045 per hour and $0.045 per GB processed. The hourly charge is about $32 a month per gateway and is unavoidable; the per-GB charge applies in both directions, which for a streaming workload means the model’s response is metered on the way in even though inbound internet transfer itself is free. Estimating egress cost for streaming responses works that through with an actual byte count.

Prices and Regions change. Both figures above are quoted from the AWS VPC pricing page at the time of writing and apply to US East (N. Virginia) only.

Two operational facts to design around. First, a NAT gateway is zonal: it exists in one Availability Zone and it goes away with that zone. Run one per AZ and point each private route table at the gateway in its own zone — routing across zones both costs cross-AZ transfer and converts one zone’s problem into every zone’s problem. Second, deleting a NAT gateway does not clean up after itself: AWS notes that routes pointing at a deleted gateway remain in blackhole status until you delete or update them, and that deleting the gateway disassociates the Elastic IP without releasing it, so you carry on paying for an unattached address. Both are easy to miss because neither produces an error.