Kafka compression waste is usually a batch depth problem, not a codec problem. Better batching improves producer compression, which reduces consumer CPU and cross-AZ cost downstream.
In one production deployment, changing batch sizing and linger settings cut the consumer fleet in half and moved compression from under 10% to over 50% - with no codec change. The cause wasn't the codec. It was batch depth.
Why batch depth controls what the codec sees
Kafka producers compress batches, not individual messages. The compression codec sees whatever the producer has accumulated by the time it flushes. linger.ms sets how long the producer waits to accumulate records. batch.size caps how large that accumulation can grow.
Both settings are conservative by default. When per-producer throughput is low - because traffic is light, or because it's spread across too many producer instances - the linger window closes before much data has arrived.
That matters because compression ratio is a function of (1) how much data the codec can see at once and (2) how much redundancy exists across that data. A compressor working on a single JSON record finds repetition only within that record. Working on a hundred records from the same schema, it finds the same field names, the same value patterns, and the same structural redundancy repeated across every record.
At shallow batch depth, redundancy is limited to a single record. At depth, the compressor finds the same field names, value patterns, and structural repetition across every record in the batch - a qualitatively different input. This batch shape problem doesn't stay at the producer.
Small producer batches create a consumer CPU tax
When producer batches are small, the broker stores small compressed record batches. Consumers fetching from that topic receive small responses, so to get more data they issue more fetch requests to Kafka brokers.
Each fetch request carries fixed overhead: a network round trip, broker-side processing, client-side dispatch, metadata handling, bookkeeping. When responses are small, that overhead is paid repeatedly on little data. The consumer fleet burns CPU on round-trip mechanics rather than on processing records.
In one production deployment, a high-throughput topic had batch.size at 16KB (the default) and fetch.min.bytes at 1 byte (also the default). Tuning batch.size to 80KB and fetch.min.bytes to 512KB cut the consumer fleet from 60 to 30 pods. Per-pod CPU increased by roughly 30%, but the fleet was processing the same volume of data with half the pods - it had stopped spending the majority of its time on fetch overhead. Compression ratio on the same topic improved from 10% to 50% with no codec change.
The overhead is fixed per fetch. What changes is how much data it buys you.
The producer's batch decision bills every consumer group
In cloud deployments, data crossing availability zone boundaries is billed per byte - producer-to-broker, inter-broker replication, and broker-to-consumer are all billable paths. Batch depth affects all three paths simultaneously:
- Smaller wire size from better compression reduces the bytes in the producer-to-broker path.
- Replication copies those same bytes, so smaller compressed batches reduce replication traffic proportionally.
- Every consumer group fetches its own copy of those bytes - fan-out multiplies the savings across every downstream reader automatically.
A meaningful reduction in compressed batch size propagates through producer ingress, replication, and every consumer fan-out stream.
The prioritization rule follows directly: throughput × fan-out. A 20% wire-size reduction on a topic with 8× fan-out matters more than a 50% reduction on a topic with 1× fan-out. The highest ROI comes from fixing the topics where the multiplier is largest.
Diagnosing the problem
The following queries use metric names common to the standard JMX exporter - verify names against your specific client library and exporter version before relying on them.
Batch fill rate:
kafka_producer_batch_size_avg / kafka_producer_batch_size_max
Values consistently below 0.3 indicate that batches are flushing before they are meaningfully filled.
Compression ratio by topic:
rate(kafka_producer_compression_rate_avg[5m])
This metric reports the ratio of compressed to uncompressed size - lower is better. A value near 1.0 means the codec is doing nothing. On a zstd-configured producer with structured data, sustained values well below 1.0 are achievable with proper batch depth - if you're seeing values near 1.0 consistently, batches are too shallow.
Consumer fetch size:
rate(kafka_consumer_fetch_size_avg[5m])
Consistently small values indicate consumers are issuing many small fetches - a downstream symptom of small producer batches.
These three metrics, read together, identify whether the problem is at the producer (batch fill), at the codec (compression rate), or propagated to the consumer (fetch size). They also identify which topics to fix first: sort by bytes_out_per_sec × consumer_group_count.
What to fix, in order
For each prioritized topic:
Batch depth: Increase linger.ms to 20–50ms. This adds a hard latency floor - every message waits up to that window before flushing. On latency-sensitive paths - fraud detection, ad bidding, synchronous request-reply over Kafka - this is unacceptable. Apply only where end-to-end latency tolerance is measured in seconds, not milliseconds.
Increase batch.size to 64–256KB depending on message size and throughput and measure batch fill rate before and after.
One constraint before raising batch.size: Kafka producers allocate memory pools per partition from a shared buffer.memory budget (default 32MB). On a producer writing to many partitions simultaneously, large batch.size values can exhaust this budget under load, causing blocked send() calls or client-side exceptions. Check partition count per producer instance and raise buffer.memory proportionally before making the change.
Codec: Switch to compression.type=zstd with compression.zstd.level=1, not zstd-3. If the topic is already on zstd, check the level - the Kafka default is not optimal for structured data.
Consumer fetch settings: Align fetch.min.bytes and fetch.max.wait.ms with the new batch sizes. Without this, consumers issue small fetches against larger broker batches, negating part of the gain.
Broker disk usage drops as a side effect - Kafka stores compressed record batches on disk, so whatever reduces wire size reduces storage without additional work.
Closing
Kafka compression waste is usually a batch depth problem. Once the batch is deep enough, the codec does its job; until then, the producer is starving it of useful input.
This is part 2 of the Kafka Network Cost series. Part 1: Kafka Compute Is Cheap. Network Is Not. Part 3: Fix the Codec Before You Touch the Schema. Part 4: the S3 indirection pattern for analytical consumers.
Keywords: Kafka batch tuning, Kafka compression zstd, linger.ms batch.size optimization, Kafka producer tuning, cross-AZ network cost, fetch.min.bytes.
Top comments (0)