DEV Community: Bharath Nadar

The Network Bill Nobody Owned: Finding Hidden AWS Data Transfer Cost with eBPF

Bharath Nadar — Fri, 09 Oct 2026 09:55:25 +0000

On a typical Kubernetes-on-AWS bill, 15–20% of the cost is network: cross-AZ data transfer, VPC peering, cross-region transfer and NAT Gateway bytes. On a $1M monthly AWS bill, that's $150–200k every month. It's serious money, and it's almost always hidden. It sits on a few line items that everybody sees and nobody owns, and every review ends the same way: "it's network, it's the platform's problem".

This post is about how we turned those lines into a list of named workloads with a cost next to each, and what we found once we could see it. The short version: within days of having per-app network data, we had more concrete savings opportunities than in months of looking at Cost Explorer.

The pain: you can see the bill, but not the cause
The approach: count bytes where the pod still has a name
What we found
How FinOps and OBI solved it together
Why network-level observability matters
Takeaways

The pain: you can see the bill, but not the cause

Cost Explorer tells you what you pay for, never who caused it. DataTransfer-Regional-Bytes is one number per account. It doesn't know about pods, deployments or teams.
Tag-based showback gets network cost wrong. Most FinOps reports assign cost by the EC2 instance's team tag. That works for compute. For network, it charges every byte leaving a node to whoever owns the node, including bytes sent by shared DaemonSets like log shippers and agents.
VPC Flow Logs see IPs, not workloads. Pods churn, IPs get reused, and NAT and peering traffic is often SNATed to the node IP before Flow Logs see it. Joining Flow Logs to Kubernetes metadata after the fact is a project of its own, and keeping them on for a large cluster isn't cheap.
So the conversation goes nowhere. "Your team's network cost went up." "We don't do anything network-heavy." Nobody can prove otherwise, so nothing changes.

The missing piece wasn't another dashboard of totals. It was attribution: this workload talks to that workload, across this billable path, costing this much.

The approach: count bytes where the pod still has a name

We used OBI (OpenTelemetry eBPF Instrumentation). A small eBPF program on each node counts bytes per network flow in the kernel, with no sidecars and no code changes. OBI then enriches each flow with Kubernetes metadata: namespace, owner workload, and pod labels like application and team.

Two placements made it work for cost:

On the node NIC (egress only): every byte is counted once, on the sender. This is the source of truth for cross-AZ.
On the pod's veth interface: the packet still carries the pod IP before SNAT, so NAT and peering traffic can still be attributed to the pod that sent it.

We gave OBI a CIDR map generated from the AWS API: every subnet with its AZ, every peered VPC (same region and cross region), the gateway endpoints, and 0.0.0.0/0 as "internet via NAT". An OpenTelemetry Collector turns that into one
metric with a cost class:

netcost_bytes_total{net_class="cross_az|nat|same_region_peering|cross_region",
                    net_team, net_app, net_workload,
                    net_remote_app, net_remote_workload, net_zone, ...}

Multiply by the AWS price per class and you get **cost per app, per team, and per "who talks to whom" pair*you already use.

The dashboard that mattered most was a set of plain tables, not charts: "Cross-AZ: who talks to whom", "alks to each peer VPC". Each row is a sender workload, a receiver workload, bytes, and estimated cost.

What we found

None of these were visible in Cost Explorer. All of them showed up in the first few days of data.

Logs shipped uncompressed over VPC peering. The biggest peering sender wasn't an app. It was the logng every node's logs uncompressed to a search cluster in a peered VPC. Its cost was spread across everyteam's node bill. Fix: compress, drop noisy logs at the source, keep the log path local.
Pods that aren't topology-aware. With random endpoint choice across three AZs, about two-thirds of sossed a zone. Fix: topologySpreadConstraints plus zone-aware routing (trafficDistribution: PreferCloseor mesh locality load balancing).
Single-replica Redis. One app's biggest cross-AZ flow was to its own Redis: a single pod in one AZ, others. Fix: a read replica in every AZ and nearest-replica reads, which also removes a single point offailure.
Unwanted cross-region transfer. Workloads calling another region when an in-region endpoint already d configs. Fix: use the in-region endpoint, and make cross-region traffic a reviewed decision.
Ingress responses crossing zones. Responses (the big half of the traffic) went back to an ingress gateway in another AZ. Fix: locality-aware load balancing at the gateway.

How FinOps and OBI solved it together

FinOps on its own gives you a bill. OBI on its own gives you traffic metrics. The value came from using them in that order:

FinOps answers "how much" and "where to look". The FinOps reports, built on the AWS bill (CUR), show each team's network spend and flag which team or account is growing. That's the starting point.
OBI answers "who" and "why". Once FinOps points at a team, OBI digs in: every byte has a sender worklnd a billable path. And because AWS prices turn those bytes into cost, we can check the result adds upagainst the bill.
Together, they turn a cost line into a ticket. Instead of "network cost went up", each finding names es, the cost, and the fix, and it goes to the team that can act on it.
They also correct showback. When a team's bill includes traffic they didn't send, like shared log shipping on their nodes, the data shows it, and the cost moves to the owner who can fix it.

Why network-level observability matters

Most teams have good visibility into compute: CPU, memory, requests per second, cost per node pool. Network is usually a blind spot, even though it's often the fastest-growing part of the bill. Without network-level observability:

Architecture decisions carry hidden cost. A single-replica cache, a service without zone awareness or a default cross-region endpoint all look fine in a design review. The cost only shows up on the bill, months later, with no owner attached.
The cost grows silently. Every new service, replica or AZ adds traffic. Nothing alerts on it, and nobody sees it until the monthly bill.
Showback is wrong. Tag-based attribution charges node owners for traffic they didn't send. Teams lose the real owners never find out.
Troubleshooting is slower. The same data that explains cost also answers "which app suddenly started sending gigabytes to another region?" during an incident.

With network-level observability, network cost becomes like any other engineering metric: measured per workload, owned by a team, and reviewed when architecture changes. That's what lets a platform team push for zone-aware design, per-AZ
replicas and sensible log shipping with evidence instead of guidelines.

Takeaways

Network is 15–20% of a typical AWS bill, and it's the part nobody owns. On a $1M bill, that's $150–200k without attribution.
FinOps tells you which team's network spend to look at. eBPF (OBI) tells you which workload sends each byte, and why. You need both to turn the bill into action.
The usual suspects are worth checking first: uncompressed logs over peering, pods that aren't topology-as, unwanted cross-region calls, and ingress responses crossing zones.
Treat network as a first-class part of observability, alongside CPU and memory. It's where a lot of the hidden cost lives.

If your bill has a network line that nobody owns, you probably have the same findings waiting for you.

One Default CloudTrail Setting Cost Us $200,000 — Fixed in 5 Minutes

Bharath Nadar — Wed, 11 Mar 2026 14:47:30 +0000

We had a $200,000 problem hiding in plain sight.

Not a breach. Not a runaway workload. Not even the security tool we had just onboarded. A single default setting, accepted during that tool's setup, that nobody revisited as the company scaled.

The Numbers That Didn't Add Up

Late 2025. Reviewing AWS Cost Explorer. CloudTrail — a line item that used to sit quietly around $1,000 a month — had crept to $10,000 in one prod account and $30,000 at the org level.

It never triggered an alert because it didn't spike. It crept.

Around the same time, other costs had come down — reserved instance discounts, some workloads rightsized. The overall bill looked reasonable. CloudTrail was hiding in the noise.

By the time I caught it: roughly $200,000 spent on audit logs nobody had asked for.

What Was Causing It

Breaking down CloudTrail by usage type in Cost Explorer showed the bill was driven entirely by data event recording — 8.98 billion S3 events in us-east-1 alone in December, at $0.10 per 100,000 events.

The org trail config explained why:

{
  "ReadWriteType": "All",
  "DataResources": [
    { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::"] }
  ]
}

Every S3 API call. Every read, every write. Every bucket. Every account in the org.

S3 reads account for 85–95% of all S3 data events in a production environment. ML pipelines, CDN origins, build artifact stores — all logging every single fetch. Billions of them. Every month.

Where That Config Came From

One of the onboarding steps for a new security tool was to enable CloudTrail S3 data events. The setup offered a default: log read and write events for all S3 buckets. An engineer accepted it. Completely reasonable — it was the default, and a security tool was asking for it.

That's the trap. The security tool wasn't reading our buckets. Our own workloads were, the same way they always had. The difference was that CloudTrail now recorded every one of those GetObject and HeadObject calls as an audit event, at $0.10 per 100,000. As traffic grew, so did the bill.

Does Compliance Actually Require This?

Before touching anything, I checked.

Framework	S3 Read Logging Required?
SOC 2 Type II	No
ISO 27001	No
GDPR	No
CIS AWS Benchmark	Recommended (Level 2), not mandatory

The CIS benchmark does recommend object-level logging for both reads and writes. That's often why security tools ask for it — it's an easy check to pass. But passing a check isn't the same as needing the data for every bucket.

Read auditing earns its cost on buckets that hold customer data, PII, or credentials — there you want to know who accessed what. For everything else, writes and deletes are what matter.

Nobody needs to know a Lambda function fetched a static asset at 3am. They do need to know if someone deleted a production backup.

The Fix

Phase 1 — Switch to WriteOnly

aws cloudtrail put-event-selectors \
  --trail-name <your-org-trail-arn> \
  --event-selectors '[{"ReadWriteType":"WriteOnly","IncludeManagementEvents":true,"DataResources":[{"Type":"AWS::S3::Object","Values":["arn:aws:s3:::"]}]}]' \
  --region us-east-1

"All" → "WriteOnly".

One word. ~$300,000 in annual savings.

Phase 2 — Add Reads Back Only Where They Matter

Switching to WriteOnly eliminates ~90% of the cost. The next step is turning read logging back on — but only for buckets that genuinely need it: customer data, PII, credentials, or compliance data.

[
  {
    "ReadWriteType": "WriteOnly",
    "IncludeManagementEvents": true,
    "DataResources": [
      { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::"] }
    ]
  },
  {
    "ReadWriteType": "ReadOnly",
    "IncludeManagementEvents": false,
    "DataResources": [
      {
        "Type": "AWS::S3::Object",
        "Values": [
          "arn:aws:s3:::your-pii-bucket/",
          "arn:aws:s3:::your-customer-data-bucket/"
        ]
      }
    ]
  }
]

Writes are audited everywhere. Reads are audited only on the handful of buckets where access itself is sensitive. Full coverage where it matters, at a fraction of the cost.

At org scale, tag sensitive buckets (security-scan: enabled) and use that tag to drive both CloudTrail scoping and security scanner configuration.

Phase 3 — Make Sure It Can't Creep Again

The original problem wasn't the spend — it was that nobody saw it grow. Set an AWS Budget or Cost Anomaly Detection monitor on the CloudTrail service so a slow climb gets flagged before it becomes a six-figure number.

The Real Lesson

The security tool wasn't the problem. Neither was the engineer who set it up. The problem was a CloudTrail default — log every read and every write on every bucket — accepted as part of a setup checklist without anyone asking what it would cost.

Read logging is valuable on buckets that hold customer data, PII, or credentials. On everything else, it's billions of events nobody will ever look at.

When a new tool asks you to enable CloudTrail data events as part of its setup, check what the default actually covers:

Read, write, or both? Writes are usually enough.
All buckets, or only the ones that hold sensitive data?
What does that cost at your request volume? At $0.10 per 100,000 events, a busy bucket adds up fast.
Who owns the monthly bill for this tool's footprint?

Security and FinOps need to be in the same conversation when a new tool lands.

Not because security is the enemy of cost — but because the most expensive line on your bill might be a default checkbox a well-intentioned engineer left ticked during a tool rollout.

The config that cost $1,000 a month in 2024 cost $30,000 a month in 2025. Our traffic scaled. The default didn't care.