NAT gateway hours were flat while NAT gateway data processing was up 4.6x, and that pair was the whole diagnosis. Something inside the VPC had started pushing bulk traffic through NAT that should never have touched it. It was same-region S3 reads from a new EKS node group whose subnets were never added to the S3 gateway endpoint's route tables. The nightly rollup job moved 2.7 TB a day down that path across two shards, 1.9 TB of it on the largest, for 22 days before anyone looked, at about $2,860 above baseline.
Problem signals:
- NAT gateway data-processing bytes climb 4.6x while NAT gateway hours stay perfectly flat
- The EC2 - Other line on the monthly preview runs several times its forecast with no new instances and no new AZ
- A VPC Flow Logs top-talker query puts one node IP at 1.9 TB in a single day against the NAT ENIs
- A private subnet’s route table ID is missing from the S3 gateway endpoint’s RouteTableIds in describe-vpc-endpoints
- A CronJob that never appeared in a cost report starts appearing right after a nodeSelector change
Where does a 4.6x NAT gateway data processing spike usually come from?
The CRM sync was innocent, and so was the retry loop
There was no page. There was a FinOps close-out preview on 2026-07-28 showing EC2 - Other running at about $5,050 a month against a baseline near $1,150. The analyst drilled the sub-lines and it all sat in one usage type. Data processed by the NAT gateways had gone from roughly 760 GB/day to roughly 3,470 GB/day, starting 2026-07-06, part of a day at first because the rollup only moved that evening, then holding at full height every day since. Gateway hours had not moved by a minute.
Flat hours means the same NAT gateways ran for the same number of seconds. Nobody added a gateway, nobody added an AZ, nobody changed the topology. Only the volume per gateway changed. That one fact killed a whole class of explanation before we opened a terminal against the cluster, and it is the first thing we check now when a bill moves and the architecture did not.
aws ce get-cost-and-usage \
--time-period Start=2026-06-25,End=2026-07-28 \
--granularity DAILY \
--metrics UnblendedCost UsageQuantity \
--filter '{"Dimensions":{"Key":"SERVICE","Values":["EC2 - Other"]}}' \
--group-by Type=DIMENSION,Key=USAGE_TYPE \
--output json
Two usage types dominate the result: gateway hours, dead flat across 33 days, and gateway bytes, a step that starts on 2026-07-06 and squares up from the 7th. The hours ruled out half our hypotheses for the price of one API call.
The platform lead's first read was a new external integration behaving badly. Two candidates had shipped inside the window: a CRM sync service on 2026-07-04 and an SMS verification service on 2026-07-11. The SMS service was the wrong date, five days late for an anomaly that started on the 6th. The CRM sync was documented at around 200 MB/day of payload, four orders of magnitude short of a 2.7 TB/day delta. Both vendor dashboards agreed with their own docs. Dead end.
Second guess was a runaway HTTP client, because a retry loop hammering a third-party API and dragging response bodies back is the classic version of this bill. The on-call SRE grepped two weeks of application logs in Loki for elevated retry counters and for 429 and 503 responses. Two services came back hot. One was the CRM sync, already accounted for. The other was an internal service calling an in-cluster endpoint, which never touches NAT at all. Dead end.
Third guess was a DNS leak: a service resolving an internal name against a public resolver, getting a public address back, and hairpinning out through NAT and back in. CoreDNS forward counts to the upstream resolver were flat and there was no NXDOMAIN storm in the logs. Dead end, and about two hours gone.
All three guesses shared an assumption. Each one took for granted that expensive NAT bytes were bytes that genuinely belonged outside the VPC. Nobody had yet considered that the traffic was going to a bucket 15 milliseconds away in the same region. That inversion, where the cost spike is in the path rather than the payload, is the shape we hit most often in cloud cost spike work.
How do you find the top talker through a NAT gateway in VPC Flow Logs?
One node moved 1.9 TB to read its own region's bucket
At 11:20 the SRE stopped interrogating the cluster and started interrogating the network. VPC Flow Logs had been enabled at the VPC level since the VPC was built, landing in S3 on a 90-day lifecycle, queried by Athena approximately never. They were already paid for, so the marginal cost of the answer was one Athena scan, and they were the only place with a per-source byte count.
SELECT
CASE WHEN srcaddr IN ('10.42.0.219', '10.42.32.87', '10.42.64.140')
THEN dstaddr ELSE srcaddr END AS node_ip,
sum(bytes) / 1e9 AS gb,
count(*) AS flows
FROM vpc_flow_logs
WHERE day = '2026/07/27' -- slash form: the projected partition
-- format is yyyy/MM/dd, and a hyphenated
-- value matches nothing and returns zero
-- rows without erroring
AND (srcaddr IN ('10.42.0.219', '10.42.32.87', '10.42.64.140')
OR dstaddr IN ('10.42.0.219', '10.42.32.87', '10.42.64.140'))
-- Keep only the node-to-NAT legs, where both addresses are VPC-private.
-- That drops the NAT-to-internet hop, which carries the same bytes again.
AND srcaddr LIKE '10.42.%'
AND dstaddr LIKE '10.42.%'
GROUP BY 1
ORDER BY gb DESC
LIMIT 10;
The NAT ENI private addresses come from describe-nat-gateways. The CASE is what makes this work: on a NAT ENI the download leg arrives as srcaddr = the NAT address and dstaddr = the node, so a plain GROUP BY srcaddr tops out at an S3 public address and the node never appears. Keying on whichever side is not the NAT sums both directions against the workload that caused them. It is the node rather than the pod because the VPC CNI SNATs pod traffic to the node primary ENI for anything outside the VPC CIDR.
The top row was 10.42.174.83 at 1.9 TB for the day, and almost all of it inbound. Read that address correctly, because the obvious reading wastes an hour: it is a NODE, not a pod. The VPC CNI translates a pod address to the primary private address of its node primary ENI for any destination outside the VPC CIDR, and S3 prefixes are outside it, so what reaches a NAT gateway ENI, and therefore the flow log, is the node. Grep a pod listing for it and you find nothing. Only a cluster running AWS_VPC_K8S_CNI_EXTERNALSNAT=true hands SNAT to the NAT gateway and preserves pod addresses that far, and this one was not. So kubectl get nodes -o wide first, which placed it in the new memory group, and then the pods scheduled on that node, which gave a nightly rollup pod in the reporting namespace, one shard of a CronJob that had been running daily for months and had never shown up in a cost conversation. Second row, 800 GB, was another node in the same group running a sibling shard. Together the two shards account for 2,700 of the 2,710 GB/day the meter had gained, and the last 10 GB is image pulls on the new pool (more on that below). Third row, 400 GB, was a general-pool node whose traffic had always been there and sat inside the old baseline.
The rollup job reads parquet from a bucket in the same region and the same account, and writes aggregates to the warehouse. That traffic should never be metered by a NAT gateway, and for months it had not been. Something had changed underneath a job whose own manifest had not been touched in eleven weeks.
Why does a new EKS node group lose the S3 gateway endpoint?
Nine subnets, six route tables on the endpoint
S3 traffic stays inside the region's gateway endpoint only if the subnet's route table carries a route for it. This account had had an S3 gateway endpoint since the VPC was built in 2024, and it had always just worked, which is exactly why nobody had a mental model of it. Describing the endpoint returned six associated route tables. Those six covered every subnet the platform team had provisioned since 2024: three private application subnets and three private data subnets. The three public subnets, one per zone and each holding a NAT gateway, never needed it.
Then we mapped node private IPs to subnet CIDRs. The rollup pod's node sat in 10.42.174.0/24, which belonged to a route table created on 2026-07-05 and named for a memory-optimised pool. It was not one of the six.
Git history on the EKS module and a Slack thread from the same afternoon filled in the rest. On 2026-07-05 the model team asked for r6i-class nodes because their feature-store rebuild kept getting OOM-killed on the general pool. The platform team was heads-down on a certificate rotation, so a model-team engineer wrote the PR themselves: new node group, a new subnet per AZ, new route tables, default route to the NAT gateway. A peer on the same team reviewed it. It merged, and it was correct as far as it went.
The subnet module had never been built to touch the gateway endpoint. The endpoint's route table associations live in a separate networking module with its own state file, owned by a different team, and nothing in the PR diff pointed at it. For one day it cost nothing, because nothing in the new subnets read S3. On 2026-07-06 the model team's validation job pulled a slice of the analytics data. Those were the first S3 bytes to leave via NAT.
The second change landed that same evening, 2026-07-06, and did the real damage. The platform team let the rollup CronJob schedule onto the memory pool as well, because those nodes sat idle at night while the model jobs were dormant. One line in a nodeSelector and tolerations block, no ticket, a rubber-stamp review from a lead who was still mid certificate rotation. From that night on, a job reading 1.9 TB of parquet a day landed on nodes in a subnet with no route to the endpoint.
Same pod, same bucket, same region, same account. One line in a nodeSelector decided which of these two paths it took, and one of them has a meter on it.
Neither change was wrong on its own. The PR was small and the reviewer was competent. Bin-packing a nightly batch job onto idle memory nodes is advice we would give. The failure is that two Terraform state files present subnet creation and endpoint association as unrelated concerns, so the coupling between them exists only in the head of whoever built the VPC. That seam shows up constantly in migration recovery work, where a landing zone gets stood up by one team and the endpoint policy by another, and nothing surfaces the gap until a bill or an outage finds it.
What actually proves S3 traffic is using the gateway endpoint?
Checking the remote IP told us nothing
By 14:15 both engineers were on, with roughly three days before the billing cycle closed. There were two ways out and only one of them was the fix.
| Step | What it does |
|---|---|
| 1. Pin the job back to the old pool | One line, in place inside 20 minutes, though the bleeding would not have stopped until that night: the job is nightly, so the next run is the earliest either fix could show anything. We rejected it. The model team's validation job had already read S3 from those same subnets, and every future workload scheduled onto that pool would have re-armed the trap. Scheduling was where the cost showed up, not where the bug lived. |
| 2. Add the three route tables to the endpoint | Three new route-table associations in the networking module the platform lead owned. The plan read 3 to add, 0 to change, 0 to destroy, which is the diff you want to see at 14:30 with three days of the cycle still to run. We applied that one. |
We applied at 14:40, deliberately between rollup runs. Changing the path a multi-terabyte transfer is currently using is not an experiment worth running during an incident, and waiting cost us nothing we were not already spending.
Then we nearly declared victory on a check that proved nothing. Someone shelled into a node on the new pool, made a request against the regional S3 endpoint, and read back the remote address. It came back a public S3 address, and it comes back as a public S3 address either way. A gateway endpoint does not hand S3 a private address. It installs a route for the S3 managed prefix list into your route table and points that route at the endpoint. The destination on the packet is identical on both paths, so the remote IP is not evidence of anything.
# 1. which route tables does the gateway endpoint actually cover?
aws ec2 describe-vpc-endpoints \
--filters Name=service-name,Values=com.amazonaws.eu-west-1.s3 \
--query 'VpcEndpoints[].{Id:VpcEndpointId,Type:VpcEndpointType,RouteTables:RouteTableIds}'
# 2. does THIS route table carry the prefix-list route to the endpoint?
aws ec2 describe-route-tables \
--route-table-ids rtb-0c9a1f4e7b2d5a613 \
--query 'RouteTables[0].Routes[].{Dest:DestinationCidrBlock,Prefix:DestinationPrefixListId,Gateway:GatewayId,Nat:NatGatewayId}'
On a covered route table the second command returns a route whose prefix-list ID is the S3 managed list with the vpce ID as its target. Project both target fields or the output lies to you: a NAT route reports under NatGatewayId and leaves GatewayId null, so a query that reads only GatewayId blanks the target on exactly the subnet you are diagnosing. Before the fix, the memory subnets returned one usable route, 0.0.0.0/0 with the NAT gateway under Nat and no prefix-list row at all.
The confirming evidence was the meter, not the address. We reran the top-talker query that same afternoon and both rollup shards were gone from the top ten, and on its own that proved nothing. The job is nightly. An afternoon window is empty whether or not the route landed, which makes it exactly the kind of check that cannot fail. The one that could was the next night's run window, and the meter behind it. Overnight the NAT gateways settled at about 750 GB/day against a pre-incident baseline near 760. Nothing further accrued before the cycle closed. The damage was already done: 22 days at about $130/day above baseline, roughly $2,860 of pure waste, every dollar of it avoidable by three route-table associations in a module nobody knew to open.
Three controls went in that week, and they are not equally important. The load-bearing one took two attempts, and the first is worth describing because it is the version most teams reach for. We wrote roughly 40 lines of rego against the Terraform plan JSON, in CI, on every PR touching the networking directory, failing when a route table attached to a private subnet was absent from a gateway-type endpoint's associations. It would never have fired. The PR that caused this created subnets and route tables in the EKS module and never touched the networking directory, so the check does not run on the one PR shape that matters. And the two concerns live in separate state files, so a plan JSON for either workspace cannot see the other side: the networking plan holds the endpoint's association list and no knowledge of the new route tables, the subnet plan holds the route tables and no endpoint. The predicate has no left-hand side and passes on an empty set.
What went in instead reads the account rather than a plan. A scheduled conformance check calls describe-route-tables and describe-vpc-endpoints and fails when a route table associated with a private subnet is missing from a gateway-type endpoint's associations. Because it queries the account it sees both state files' resources, and because it runs on a timer it does not care which directory a PR touched. The cost is real and worth stating: it catches the gap after the merge rather than before it, within an hour rather than at review time. An hour of NAT charges is about $5.40 at this volume. The plan-time version would have caught nothing at all.
The second control is a tighter cost alert on NAT data processing, and getting there needs one step people skip. Cost Anomaly Detection cannot watch a usage type: a monitor's dimension is one of SERVICE, LINKED_ACCOUNT, TAG or COST_CATEGORY, and nothing else. So we created a Cost Category whose rule matches the NAT data-processing usage type and pointed a COST_CATEGORY monitor at that, with a subscription threshold well under the old one. Match it with CONTAINS on NatGateway-Bytes rather than an exact string. Usage types carry a Region billing code, and only us-east-1 line items appear bare, so in this account (eu-west-1) the value is EU-NatGateway-Bytes, and Ireland uses the legacy EU code rather than the EUW1 you would guess from the Region name. An exact rule on the unprefixed form categorises nothing and the replacement monitor never fires, which is the original failure wearing a new hat. An AWS Budget filtered on UsageType works too and is quicker to stand up. Be careful about why the old monitor stayed quiet, because the obvious reading is wrong and it changes how you size the new one. A Cost Anomaly Detection threshold is not a daily rate: it is the anomaly’s total cost impact, actual minus expected spend accumulated over the anomaly’s whole duration, so a $500 threshold against a $130/day drift is four days away, not unreachable. So why did nothing fire? Not because the step was small: EC2 - Other sat near $1,150 a month, about $38 a day, and $130 a day on top of that is more than triple the pool, which is the 4.6x the preview showed. A service-level monitor is exactly what catches that. It did not fire because there was no subscription on that monitor anyone read: the alerts were addressed to a shared mailbox the platform team had stopped opening when it filled with RI-expiry notices. The lesson is not about thresholds at all, it is that an unread alert and no alert cost the same. Scoped to the category, the same pattern surfaces inside about three days. The third is CODEOWNERS on the node group and subnet modules, routing review to the platform engineer who owns the VPC rather than to any platform engineer who is free. The change looked EKS-shaped. The risk was VPC-shaped. We also updated the runbook, which is not a control at all: documentation explains why, it does not block a merge. If you are auditing this class of coupling across your own modules, it sits in the same family as the Terraform and IaC debt problems that only surface on the invoice.
FAQ: S3 gateway endpoints, NAT charges and new subnets
The four questions the retro kept circling
- Does an S3 gateway endpoint cost anything to run? Gateway endpoints carry no hourly and no per-GB charge, which is why the gap between the two paths in this story is the entire NAT data-processing bill. Interface endpoints are the ones with hourly plus per-GB pricing, and they are a different decision.
- We attached the endpoint but S3 still resolves to a public address. Did it work? Yes, that is expected. The gateway endpoint changes routing, not addressing. Verify by reading the route table for a prefix-list route targeting the vpce, then watch the NAT gateway bytes metric fall. The remote IP will look the same on both paths.
- Does this hit ECR image pulls too? Yes. Layer downloads come from S3-backed storage, so pods cold-starting in a subnet without the route pull their images through NAT. On the new pool that was about 20 cold starts a day at roughly 500 MB an image, near 10 GB/day. Small next to 3.5 TB, but it is the signal that arrives before your batch job does.
- Do interface endpoints have the same route table problem? Different failure mode. An interface endpoint is an ENI placed in specific subnets plus a private DNS name, so it does not depend on a route table association. Create a new subnet and leave it off the endpoint's subnet list, and resolution still points at an ENI in another AZ. It works, and it will not show on the bill: since April 2022 AWS does not charge inter-AZ transfer for traffic through an interface endpoint. What you get instead is every call from that subnet crossing a zone boundary, which is added latency and a dependency on another zone that nobody designed. Check it whenever you add an AZ.
When the bill moves and nothing in the cluster did
If your EC2 - Other line just doubled
The hard part of this class of incident is that every signal you normally trust reads clean. Node count normal, pod count normal, error rates normal, mesh RPS normal. The expensive traffic is going somewhere your dashboards consider boring, and the evidence sits in flow logs nobody queries and endpoint associations nobody reads, one Terraform state file away from the change that caused it. Teams lose weeks to it because there is nothing to debug, only something to notice.
We come at this from the network side rather than the cluster side. Route tables, endpoint associations and flow logs first, then the workloads: which of your subnets are paying per gigabyte for traffic that should be free, and which module owns the line that has to change. The finding is usually smaller than the invoice that prompted the call.
If your NAT data processing has stepped up and the cluster looks healthy, book an infrastructure review and we will start on your flow logs and gateway endpoint associations the same business day.
Originally published at https://infraforge.agency/insights/s3-traffic-through-nat-missing-endpoint-route/.
If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.
Top comments (0)