DEV Community

Krishna Garg
Krishna Garg

Posted on

Your Kubernetes cluster is paying a cross-AZ tax. ZoneTax shows you exactly which pod.

Your Kubernetes cluster is probably paying a tax you cannot see.

On AWS, traffic between availability zones costs about $0.01/GB in each direction — so a GB crossing a zone boundary costs you $0.02. Nothing in your cluster tells you it is happening. It arrives days later on the bill as "EC2-Other / DataTransfer-Regional-Bytes", with no hint of which pod caused it.

ZoneTax makes that visible in real time, and attributes every dollar to the workload generating it.

What it does

  • Detects cross-AZ traffic live, via an in-cluster DaemonSet
  • Prices it the way AWS actually bills, counting both directions
  • Shows you who is paying it — an interactive map of which namespaces and workloads talk across zones
  • Exports Prometheus metrics so you can alert with the tooling you already run
  • Ships a CLI for zonetax top and zonetax report without opening a dashboard

Install it in two minutes

helm repo add zonetax https://gargkrishna730.github.io/zonetax
helm install zonetax zonetax/zonetax -n zonetax --create-namespace
Enter fullscreen mode Exit fullscreen mode

Or with plain manifests:

kubectl apply -f https://github.com/gargkrishna730/zonetax/releases/latest/download/install.yaml
Enter fullscreen mode Exit fullscreen mode

Then open the dashboard:

kubectl -n zonetax port-forward svc/zonetax-collector 8080:8080
# http://localhost:8080
Enter fullscreen mode Exit fullscreen mode

Cloud and region are auto-detected from your node labels. There is nothing to configure.

What it found on a real cluster

Within minutes of installing on a 3-AZ EKS cluster, one route stood out: 97% of all cross-AZ spend came from a single telemetry collector writing ~4.4 GB/hour into its database in another zone.

The database was pinned to its zone by an EBS volume. The collector was stateless and had simply landed wherever the scheduler put it. The two had never been co-located because nothing had ever made the cost visible.

One nodeSelector:

Cross-AZ GB/hour Cost/day
Before 4.83 $2.32
After 0.06 $0.03

Roughly $70/month from a one-line change, on a cluster nobody thought had a network cost problem. The route disappeared from the map within one scrape interval.

Two patterns worth checking in your own cluster:

  1. Central logging or metrics backends. Collectors that write a lot are the usual culprit — check where they sit relative to their storage.
  2. Stateful workloads whose volume pins them to one zone while their clients spread across all of them.

Accuracy

This is a cost tool, so the number has to be trustworthy. Measured on a real 3-AZ cluster by sending a known 1 GiB between pods:

Test Reported Error
Upload across zones 1.0834 GB +0.90%
Download (reply direction) 1.0780 GB +0.4%
Through a ClusterIP Service 1.0816 GB +0.7%
Same-zone control no cross-AZ entry correct

The small positive error is TCP/IP header overhead — conntrack counts bytes on the wire, which is exactly what AWS bills.

ZoneTax attributes pod-to-pod traffic. Anything with an endpoint outside the cluster (managed databases, S3, load balancers, internet egress) is reported separately as unattributed rather than guessed at, so you always know what is and is not being counted.

How it works

agent (DaemonSet, every node)
  └─ samples /proc/net/nf_conntrack
  └─ resolves IP → pod → node → availability zone
  └─ exposes raw byte counters

collector (one pod)
  └─ scrapes agents, applies a versioned AWS pricing table
  └─ serves the REST API, the dashboard and /metrics
Enter fullscreen mode Exit fullscreen mode

Zones come from the standard topology.kubernetes.io/zone node labels, so there are no cloud credentials and no IMDS calls. The agent reads a local kernel file and generates no network traffic of its own.

Alerting

ZoneTax exports costed Prometheus metrics, so you alert with Prometheus and Alertmanager rather than yet another notifier:

- alert: CrossAZSpendHigh
  expr: zonetax_cross_az_cost_usd > 1 and zonetax_coverage_complete == 1
  for: 15m
Enter fullscreen mode Exit fullscreen mode

That zonetax_coverage_complete gauge is worth knowing about: it is 1 only when every node in the cluster is actually being measured. Gate your alerts on it and a missing agent can never look like a drop in spend.

Running cost

The agent requests 50m CPU / 64Mi memory per node; the collector 100m / 128Mi. The agent reads a local kernel file and sends nothing over the network. In practice it costs far less than the first misplaced pod it finds.

Try it

Apache 2.0, multi-arch images (amd64 + arm64), AWS/EKS:

github.com/gargkrishna730/zonetax

If you run a multi-AZ cluster, install it and look at the top route. I would genuinely like to know what you find.

Top comments (2)

Collapse
 
manojkagitha profile image
Manoj Kumar Kagitha •

Found the same pattern on our clusters: a telemetry collector turned out to be the loudest cross-zone talker, and co-locating it with its storage wiped most of the charge. The zonetax_coverage_complete guard is the detail I appreciate most. A silent agent looks like a saving, not a blind spot.

Collapse
 
gargkrishna730 profile image
Krishna Garg •

Thanks — that collector-vs-storage pattern seems to be the common one. The
storage gets pinned to a zone by its volume, the stateless writer lands wherever
the scheduler puts it, and nothing surfaces the cost until the bill arrives.

If you're up for sharing, what did the before/after look like on your side? A
data point from a cluster I don't control would be genuinely useful.

The coverage guard came from getting bitten. An agent sat Pending for 8 days on
a memory-full node and the reported cost quietly dropped — looked like an
improvement until I checked why. Worth alerting on
zonetax_coverage_complete == 0 separately; otherwise, a dead agent just
silences your spend alert.