Kafka Compute Is Cheap. Network Is Not

Elad Eldor — Wed, 20 May 2026 14:28:42 +0000

Cross-AZ network transfer often costs more than compute. Here's why it's invisible and what to do about it

Your most expensive Kafka topic probably isn't the one with the most data. It's the one with the most consumers, because cross-AZ network transfer often costs more than compute in real Kafka deployments - sometimes by 5–10x when fan-out is high and pod placement is unlucky.

While the Data Transfer cost shows up in cloud billing, the line items don't point back to Kafka topics. The AWS CUR (Cost and Usage Report) shows EC2 / Data Transfer, Kafka dashboard shows producer and consumer metrics, and nobody looks at both at once. That gap is why Data Transfer cost persists at companies that are otherwise rigorous about infrastructure spend.

This article is about this hidden cost and what to do about it, and it's relevant when:

Kafka brokers span multiple AZs (Availability Zones)
Producers and Consumers run in different AZs than the Brokers
You run Kafka on AWS or GCP (Azure doesn't charge on cross-AZ networking)

Quick Diagnostic

If you already have bytes_in and bytes_out metrics per topic, you can estimate fan-out.
For a topic with 200 GB/hour in and 600 GB/hour out at RF=2:

Producer throughput ≈ 200 ÷ 2 = 100 GB/hour
Replication outbound ≈ 100 GB/hour
Consumer outbound ≈ 600–100 = 500 GB/hour
Fan-out ≈ 500 ÷ 100 = 5x

That estimate is enough to rank topics by cost impact. It's not exact enough for chargeback, because compression, retries, and internal broker traffic can distort the numbers.
If bytes_out is much larger than bytes_in, the gap is usually fan-out.

How Data Transfer Billing Works

In AWS CUR, AZ-to-AZ traffic in the same region appears under DataTransfer-Regional-Bytes. On AWS, this is typically about $0.01 per GB (before discount) for data leaving an AZ within a region. GCP is similar, but exact rates vary by region and discount agreement.
This means a single GB can be charged multiple times as it moves through Kafka:

producer to leader broker
leader to follower broker
broker to each consumer group

Kafka also generates extra bidirectional traffic from fetches, acknowledgments, heartbeats, and recovery activity, so the effective cost of a topic is usually a bit higher than the raw payload size suggests.

The Three Paths

Kafka traffic has three cost-bearing paths.

Producer to broker: Producers write to partition leaders. If the producer pod and the leader live in different AZs, that write crosses an AZ boundary. Producers must reach the leader, so this cannot be avoided by configuration alone.
Replication between brokers: Leaders replicate to followers. RF=2 copies each write once. RF=3 copies it twice. In a multi-AZ cluster, replication is part of durability.
Broker to consumer: Consumers fetch data from brokers. Each consumer group reads the topic independently, so this path scales with fan-out. The mental model is:

billable transfers=1+(RF−1)+fan-out

This is a worst-case upper bound, but it is a useful one. It explains why a topic with many consumers can cost far more than a topic with more writes.

Kafka transfers compressed batches, so Data Transfer cost is based on bytes on the wire, not logical message size - better compression and batching reduce every term in the model. Consumer-side filtering doesn't reduce network cost, since Kafka still ships the full records - filtering saves CPU, not bandwidth.

Fan-Out Drives Cost

The main surprise is that consumer count often matters more than producer throughput - a topic with one consumer group has far less cross-AZ cost than the same topic with five consumer groups, even when producer traffic is identical. The extra cost comes entirely from the broker-to-consumer path.

Fan-out is often understated in steady-state measurements. Rebalances, pod restarts, backfills, and offset resets can replay old data and temporarily amplify Data Transfer cost. That's why optimization should target throughput×fan-out instead of throughput alone.

Placement Matters

Compute optimization and network cost optimization pull in opposite directions, since Kubernetes autoscalers usually optimize for compute, not Kafka topology. When pods are rescheduled, they land wherever capacity is available, not wherever Kafka brokers happen to be.

That matters because a pod in a non-broker AZ pays extra on every Kafka interaction:

Producer-heavy services are affected on writes
Consumer-heavy services are affected on fetches
Mixed services pay on both

In a three-AZ cluster with brokers in two of them, a randomly placed pod has a baseline ~67% chance of landing outside a broker AZ. K8s autoscalers can push that higher since they bin-pack into whatever AZ has spare capacity, so in practice the effective cross-AZ exposure for consumer pods can run 73–90%+ on some clusters.

RF=3 Can Be Cheaper Than RF=2

When looking only at the replication path, RF=2 is cheaper on storage and replication compared to RF=3. Counter-intuitively, it's not always true for total network cost. On high fan-out topics, RF=3 can reduce cross-AZ consumer traffic because each AZ has a replica available for local reads. The extra replication cost is fixed per write, while the read-side savings scale with fan-out.

This requires client.rack on consumers, a rack-aware assignor, and reasonably balanced AZ distribution.

KIP-392 enables consumers to fetch from the closest replica when rack-aware selection is configured. KIP-881 improves rack-aware consumer assignment. However KIP-405 is different since it moves cold log segments to remote storage, which reduces storage cost but doesn't remove broker-mediated Data Transfer cost.
If those conditions are met, RF=3 can lower total cross-AZ traffic even though it stores more data. On read-heavy topics, that can make it cheaper overall.
The right question isn't is RF=3 more expensive? - Instead it's which costs more on this topic: extra replication or repeated cross-AZ reads?

What To Do With This

The biggest savings usually lie within these steps:

Start with the top throughput topics
Derive fan-out from bytes_in and bytes_out
Sort topics by throughput × fan-out
Map pod distribution against broker AZs
Audit client.rack on all consumers
Revisit RF on high fan-out topics
Check for AZ mismatches across clusters

A few things worth keeping in mind as you work through that list:

Services with significant pod presence outside broker AZs are paying a topology tax in the form of Data Transfer cost
RF=3 with proper rack configuration may be cheaper than RF=2 on read-heavy topics
The conventional Data Transfer cost ranking assumes single-consumer topics and doesn't generalize for high fanout ones
Compression is an important lever - better batching and better codecs reduce bytes on the wire, and that lowers cross-AZ cost directly
For non-real-time consumers, another option is to remove them from Kafka entirely and serve them from S3 instead - one Kafka consumer writes to S3, and many analytical readers consume from S3 over a VPC gateway endpoint. That avoids Kafka fan-out for workloads that can tolerate seconds to minutes of latency

Closing

Kafka often looks expensive because the bill is driven by network topology, consumer fan-out, and placement - not just EC2 compute.

The fix is usually the same:

measure fan-out
align placement with broker topology where possible
improve compression
reconsider RF or S3 indirection where read traffic dominates

The cost is rarely where people first look. Once you can see the fan-out, the leverage is obvious.

This is Part 1 of the Kafka Network Cost series. Part 2: Kafka's Real Compression Problem Is Batch Depth. Part 3: Fix the Codec Before You Touch the Schema. Part 4: The Cheapest Kafka Consumer Is One That Doesn't Read From Kafka. Part 5: The S3 GET Limit Nobody Plans For

Keywords: Kafka cross-AZ cost, AWS DataTransfer-Regional-Bytes, Kafka network optimization, fan-out cost model, Kafka VPC topology, KIP-392, KIP-881, KIP-405.

DEV Community: Elad Eldor