Build an AWS Network Cost Agent: Explain NAT Gateway and Cross-AZ Spend with VPC Flow Logs

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

The line item nobody can attribute

Open Cost Explorer in any multi-AZ account and you'll find them: NatGateway-Bytes, DataTransfer-Regional-Bytes, DataTransfer-Out-Bytes. Together they're routinely 10–20% of an EKS bill, and they are the least attributable money on the invoice. Cost Explorer tells you the VPC and the dollar amount. It cannot tell you that 60% of your NAT processing fee is one CI runner pulling container images from Docker Hub every four minutes, or that two chatty services pinned in different availability zones are billing you twice for every gigabyte they exchange.

The cost anomaly agent from earlier in this series can detect that DataTransfer-Regional-Bytes spiked — that was literally its example — but it stops at what got expensive. This post builds the AWS network cost agent that answers who and why: it aggregates VPC Flow Logs deterministically, maps IPs to subnets, AZs, and pods, classifies each traffic pattern against a short list of known fixes (gateway endpoints, ECR endpoints, topology-aware routing, NAT placement), and posts a ranked plan with dollar arithmetic attached. It holds read-only credentials; every fix ships as a PR.

What data transfer actually costs

The agent's dollar figures come from list prices (us-east-1, on-demand) applied to measured bytes — arithmetic, not model guesses:

Path	Price	The catch
NAT gateway processing	$0.045/GB + $32.85/mo per gateway	Charged even when the destination (S3 in-region) is free
Cross-AZ within a region	$0.01/GB each direction	Both sides billed, so a chatty pair pays ~$0.02/GB
Internet egress	$0.09/GB after the free 100 GB/mo	Also pays NAT processing if it leaves via NAT
S3/DynamoDB gateway endpoint	Free	Removes NAT processing for that traffic entirely
Interface endpoint (PrivateLink)	~$7.30/mo per AZ + $0.01/GB	Cheaper than NAT's $0.045/GB — if volume is high enough
Same-AZ traffic	Free	The reason topology-aware routing exists

Two consequences fall out of that table and drive everything the agent recommends. First, in-region S3 traffic through a NAT gateway costs $0.045/GB for no reason at all — a gateway endpoint makes it free. Second, an interface endpoint only beats NAT when measured volume covers its hourly cost, which is exactly the kind of per-case arithmetic an agent should show rather than assert.

Step 1: A role that can only read

Same discipline as the waste reclamation agent: IAM is the guardrail, the prompt is documentation. This role reads flow logs, network topology, and cost data. It cannot create endpoints, modify routes, or delete a NAT gateway.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadNetworkTopology",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeNatGateways",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeVpcEndpoints",
        "ec2:DescribeRouteTables"
      ],
      "Resource": "*"
    },
    {
      "Sid": "QueryFlowLogs",
      "Effect": "Allow",
      "Action": [
        "logs:StartQuery",
        "logs:GetQueryResults",
        "logs:StopQuery"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/vpc/flow-logs*"
    },
    {
      "Sid": "ReadCost",
      "Effect": "Allow",
      "Action": ["ce:GetCostAndUsage"],
      "Resource": "*"
    }
  ]
}

Prove the boundary before trusting it:

# Must succeed
aws ec2 describe-nat-gateways \
  --query 'NatGateways[].{id:NatGatewayId,subnet:SubnetId,ip:NatGatewayAddresses[0].PrivateIp}'

# Must fail with UnauthorizedOperation — the whole point
aws ec2 create-vpc-endpoint --vpc-id vpc-123 \
  --service-name com.amazonaws.us-east-1.s3 --vpc-endpoint-type Gateway

Credentials arrive as a short-session assumed role — no long-lived keys in env vars.

Step 2: Aggregation is code, not a prompt

A busy VPC produces millions of flow log records a day. No model reads that; the sweeper reduces it to a few dozen labeled rows first. Two Logs Insights queries do the heavy lifting.

Who is pushing bytes through the NAT gateway, and to where:

filter interfaceId = 'eni-0nat1234567890abc'
| stats sum(bytes) as totalBytes by srcAddr, dstAddr
| sort totalBytes desc
| limit 50

Which internal pairs are talking across the VPC (AZ resolution happens in code, since flow logs don't carry it):

filter srcAddr like /^10\./ and dstAddr like /^10\./
| stats sum(bytes) as totalBytes by srcAddr, dstAddr
| sort totalBytes desc
| limit 100

The Python around them turns raw IP pairs into evidence records: it maps every internal IP to a subnet and AZ by CIDR containment, tags external destinations that belong to AWS services using the published ip-ranges.json, and — on EKS with the VPC CNI, where pod IPs are real VPC IPs — joins against a pod-IP snapshot so a row says checkout-worker instead of 10.0.41.7.

import boto3, ipaddress, json, urllib.request

ec2 = boto3.client("ec2")

def subnet_map(vpc_id):
    subs = ec2.describe_subnets(
        Filters=[{"Name": "vpc-id", "Values": [vpc_id]}])["Subnets"]
    return [(ipaddress.ip_network(s["CidrBlock"]), s["SubnetId"],
             s["AvailabilityZone"]) for s in subs]

def az_of(ip, smap):
    addr = ipaddress.ip_address(ip)
    for net, sid, az in smap:
        if addr in net:
            return az
    return None  # not in this VPC

def aws_service_of(ip, ranges):   # ranges = parsed ip-ranges.json prefixes
    addr = ipaddress.ip_address(ip)
    for p in ranges:
        if addr in p["net"]:
            return p["service"]   # "S3", "EC2", "CLOUDFRONT", ...
    return "internet"

def classify_nat_row(row, smap, ranges, pod_index):
    dest = aws_service_of(row["dstAddr"], ranges)
    gb = row["totalBytes"] / 1e9
    return {
        "kind": "nat_traffic",
        "source": pod_index.get(row["srcAddr"], row["srcAddr"]),
        "source_az": az_of(row["srcAddr"], smap),
        "destination_service": dest,
        "gb_per_day": round(gb, 2),
        "usd_per_month": round(gb * 30 * 0.045, 2),   # NAT processing only
        "extra_egress_usd": round(gb * 30 * 0.09, 2) if dest == "internet" else 0.0,
    }

def classify_crossaz_row(row, smap, pod_index):
    src_az, dst_az = az_of(row["srcAddr"], smap), az_of(row["dstAddr"], smap)
    if not src_az or not dst_az or src_az == dst_az:
        return None
    gb = row["totalBytes"] / 1e9
    return {
        "kind": "cross_az",
        "pair": [pod_index.get(row["srcAddr"], row["srcAddr"]),
                 pod_index.get(row["dstAddr"], row["dstAddr"])],
        "azs": [src_az, dst_az],
        "gb_per_day": round(gb, 2),
        "usd_per_month": round(gb * 30 * 0.02, 2),    # $0.01/GB, billed both sides
    }

Everything here is deterministic and testable without a model. The per-namespace attribution plays the same role showback plays in the cost allocation setup: a number nobody can argue with, attached to a name somebody owns.

Step 3: The agent explains and ranks — nothing else

Two tools. get_traffic_summary(kind) returns the records above, capped at 50 rows so a week of flow logs never turns into a six-figure token bill. propose_fixes(plan) writes a plan to the review queue. There is no create_vpc_endpoint tool, for the same reason the anomaly agent has no stop_instances.

The system prompt is a pattern catalog, because network cost has a short list of shapes that cover most real bills:

You are a FinOps engineer explaining AWS network spend. For each traffic
record, match it to exactly one pattern:

- s3_via_nat: destination_service is S3 and traffic leaves via NAT.
  Fix: S3 gateway endpoint (free). Savings = current NAT processing fee.
- registry_via_nat: destination is ECR, or internet destinations that
  resolve to container registries. Fix: ecr.api + ecr.dkr interface
  endpoints plus the S3 gateway endpoint for layers; show the endpoint
  hourly cost against measured volume before claiming a saving.
- chatty_cross_az: a cross_az pair above $50/month. Fix: topology-aware
  routing if replicas exist in the caller's zone; otherwise co-schedule.
- nat_placement: heavy NAT traffic whose source_az differs from the NAT
  gateway's AZ. Show both options with arithmetic: add a per-AZ gateway
  ($32.85/mo each) vs keep paying the cross-AZ toll.
- legit_egress: real internet traffic to users or third-party APIs.
  Say so explicitly. Not every line item is waste.

Rules:
- Dollar figures come only from usd_per_month fields. Never estimate.
- A fix that adds a fixed cost must show break-even volume.
- Rank by net monthly saving, descending. Cap at 15 items.

A plan item comes back with the arithmetic a reviewer can check in their head:

{
  "pattern": "registry_via_nat",
  "source": "ci-runner (namespace ci, az us-east-1a)",
  "evidence": "38.4 GB/day to ECR via nat-0abc, $51.84/mo NAT processing",
  "fix": "ecr.api + ecr.dkr interface endpoints in us-east-1a",
  "arithmetic": "endpoints cost 2 x $7.30 = $14.60/mo + $11.52 processing; net saving $25.72/mo",
  "confidence": 0.85
}

Step 4: Fixes are pull requests

Every recommendation maps to a small, reviewable diff — the GitOps-for-agents rule applies to network plumbing more than anywhere, because a wrong route table entry is an outage, not a cost bug. The two highest-frequency fixes:

The S3 gateway endpoint — free, and the single most common finding in accounts that grew organically:

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = module.vpc.private_route_table_ids
}

Topology-aware routing for the chatty cross-AZ pair, one annotation on the callee's Service:

metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto

Kubernetes then prefers same-zone endpoints when every zone has capacity — the caveat being when: with two replicas across three zones the hints stay off and nothing changes, so the agent's fix text must say "requires at least one replica per zone," not just "add the annotation."

Where it will be wrong

Run it in propose-only mode for a couple of weeks, the shadow mode habit, and expect these failure modes:

• Flow logs don't name buckets. They show an S3 IP, not which bucket or whose requests. Attribution to a team comes from the source side (pod, namespace) — good enough for the fix, not good enough to bill a bucket owner. CUR data fills that gap if you need it.
• The analysis has its own bill. Flow logs into CloudWatch Logs cost ~$0.50/GB ingested, and Logs Insights charges ~$0.005/GB scanned per query. On a high-traffic VPC, deliver flow logs to S3 (~$0.25/GB) and run the same aggregations with Athena — otherwise the agent's own telemetry becomes a line item it should be flagging.
• Interface endpoints can lose the arithmetic. Two ECR endpoints in three AZs is ~$43.80/mo before a byte moves. Below roughly 1 TB/mo of registry traffic the NAT fee is cheaper. This is why the prompt forbids recommending an endpoint without showing break-even volume.
• Topology-aware routing shifts load, not just cost. Zone-local preference means a zone with fewer healthy pods works harder. If the callee is latency-sensitive, that saving can reappear as a paged SLO. Treat the annotation as a deployment change with a rollout plan, not a config flip.
• Short windows lie. A seven-day flow log window misses the monthly batch job that moves 4 TB on the 1st. Same lesson as every sweep in this series: widen the window before trusting a "safe to change" conclusion.

Reconcile after each merged fix against Cost Explorer's NatGateway-Bytes and DataTransfer-Regional-Bytes for the affected VPC — measured deltas, not projected ones, are the only optimization claims worth reporting.

Takeaway

Network spend is the last unattributed corner of most AWS bills because the attribution work is genuinely tedious: flow log aggregation, CIDR-to-AZ joins, pod-IP lookups, and price arithmetic across five different rate cards. All of that is deterministic code. The judgment layer — recognizing that this NAT traffic is really ECR pulls, that this cross-AZ pair is one annotation away from free, that this egress is legitimate product traffic — is a pattern-matching task an LLM does well when the evidence is pre-chewed and the prices are pinned. Keep the role read-only, ship fixes as Terraform PRs with break-even math in the description, and the agent earns the only reputation that matters for a cost tool: its numbers survive contact with the next invoice.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.