How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself.
Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub, and the Appendix has the commands.
The problem
We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape.
Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet.
topic activity over 24h
msgs ▲
│ ██ nightly import / CDC burst
│ ██
│______________██______________ flat, idle ~22h/day
└───────────────────────────────▶ time
That shape creates two problems at once:
Idle waste. When the topic is quiet, each sink still runs — it polls Kafka, holds connections, emits metrics, and occupies CPU and memory. Multiply one "small" sink across dozens of them and several regions, and you're paying around the clock for work that happens for a couple of hours a night.
Spike lag. When the burst lands, a backlog builds fast. If consumers can't drain it quickly enough, consumer lag — the gap between what's been produced and what's been processed — climbs, and downstream reads start serving stale data.
We want two things that sound contradictory: cost almost nothing when idle, and absorb the spike fast when it hits.
Why the obvious autoscaler doesn't help
The reflex is a Kubernetes Horizontal Pod Autoscaler (HPA) on CPU or memory. For sinks, that's the wrong signal.
Sink work is I/O-bound: the consumer spends its time waiting on Kafka polls and database writes, not burning CPU. So when a backlog builds, CPU stays flat while lag climbs — and an HPA watching CPU concludes everything is fine and never scales.
during a burst:
lag ▲ ███████ ← climbing fast (real work waiting)
CPU ▲ ▁▁▁▁▁▁▁ ← flat (blocked on I/O, not compute)
→ HPA sees low CPU, never scales, lag grows silently
The question a sink autoscaler must answer isn't "is this pod using CPU?" — it's "is there work waiting to be processed?" The only signal that answers it directly is Kafka consumer lag.
That reframes the whole solution. We need to scale on lag, we need each pod to actually drain fast, and we need scaling itself not to get in the way. Three moves.
The solution, in three moves
Move 1 — Scale on lag, all the way to zero, with KEDA
KEDA (Kubernetes Event-Driven Autoscaling) lets Kubernetes scale on external signals — here, Kafka consumer-group lag — and, crucially, can scale a workload to zero. A plain HPA can't go from one replica to zero; KEDA manages the HPA and the scale-to-zero around it.
You describe the workload and the trigger in a ScaledObject:
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sink-scaler
spec:
scaleTargetRef:
kind: StatefulSet
name: sink
minReplicaCount: 0 # scale-to-zero — the whole point
maxReplicaCount: 12 # = partition count (see "The ceiling")
cooldownPeriod: 60 # stay up until the backlog is fully drained
triggers:
- type: kafka
metadata:
bootstrapServers: hotel-cluster-kafka-bootstrap:9092
consumerGroup: foo-consumer-group
topic: hotel-topics
lagThreshold: "1000" # desired replicas ≈ totalLag / 1000
activationLagThreshold: "0" # wake from zero the instant lag appears
KEDA polls the lag, turns it into an autoscaling metric, and drives replicas roughly as desiredReplicas ≈ totalLag / lagThreshold, capped at maxReplicaCount.
What we measured. Idle, the sink sits at 0 replicas — no pods, no cost. We fired a one-million-message burst and watched KEDA react:
| time | replicas | total lag | what KEDA did |
|---|---|---|---|
| idle | 0 | 0 | scaled to zero — no pods, no cost |
| +6s | 12 | 654,364 | saw lag, jumped straight to 12 |
| +18s | 12 | 357,452 | 12 pods draining in parallel |
| +36s | 12 | 0 | backlog cleared |
| ~+96s | 0 | 0 | cooldown elapsed → back to zero |
That 0 → 12 → 0 cycle is the "always-on to on-demand" shift in one table: you pay for sink compute only when there's work.
One honest caveat about this run: all 12 pods wrote to a single shared database in the lab, which capped the aggregate drain rate — so this isn't "12× one pod." The point here is the scaling shape (zero → full → zero on lag). Per-pod throughput is a separate measurement, and it's where Move 2 comes in. (Scaling the write path itself is a topic for a follow-up post.)
Move 2 — More pods isn't enough: raise each pod's drain rate
Here's the trap most KEDA write-ups skip: KEDA gives you pods, but a single pod's throughput is set by your consumer configuration. If each pod drains slowly, a fleet of slow pods still drains slowly. We proved it by running the same burst through the same single consumer, twice — once naive, once tuned.
The naive baseline. One database round-trip per record, one record per poll():
NAIVE — one upsert per row, max.poll.records = 1
≈ 175 records/s per pod
on a 40k burst: lag sits around ~37k, dropping only ~500 every 3 seconds
a 1M backlog would take ≈ 1.5 hours
The consumer isn't "busy" in CPU terms — it's just doing tiny, chatty units of work. Two changes fix that.
Lever 1 — fetch fatter batches per poll. Three consumer settings control how much a single poll() returns:
| setting | what it does |
|---|---|
fetch.min.bytes |
broker waits until it has at least this many bytes before replying |
fetch.max.wait.ms |
…but never waits longer than this (stays responsive when idle) |
max.poll.records |
caps how many records one poll() hands your code |
Under a flood, fetch.min.bytes is satisfied instantly, so the broker returns a big batch and max.poll.records lets your code process a fat chunk at once — far fewer, far larger round-trips.
Lever 2 — write the whole batch in one transaction. Instead of N single upserts, do one bulk, idempotent upsert per poll:
INSERT INTO materialized_view (id, payload, updated_at)
VALUES (?, ?, now()), (?, ?, now()), ... -- the entire poll() batch
ON CONFLICT (id) DO UPDATE
SET payload = EXCLUDED.payload, updated_at = now();
ON CONFLICT keeps it idempotent, so reprocessing after a rebalance is safe and can't corrupt the view.
The tuned result. With the two levers on:
# the "after" consumer profile
KAFKA_MAX_POLL_RECORDS: "500"
KAFKA_FETCH_MIN_BYTES: "1048576" # ~500 × 2KB → the broker returns a full batch
SINK_BULK: "true" # one bulk upsert per poll
TUNED — same single consumer, same burst
≈ 19,000 records/s
a 1M backlog drains in ≈ 50 seconds (vs ≈ 1.5 hours naive)
lag ▲ peak ╲
╲____
╲______ same partitions, same single pod
0 ─────────────────────╲_____▶ ~50s
That's about 100× the per-pod drain rate, purely from reading fatter and writing in bulk — no extra pods. This is the lever KEDA can't pull for you. And since sinks are usually database-bound, batching the write is where most of the real gain lives.
Move 3 — Don't let scaling sabotage the drain: rebalances
Now the subtle failure mode. When KEDA scales 1 → 6 → 12 mid-burst, every change in group membership triggers a Kafka consumer group rebalance — a renegotiation of which consumer owns which partition. With the default eager rebalancing, that's stop-the-world: every consumer gives up all its partitions, re-joins, and only then resumes. Consumption pauses across the whole group exactly when you're trying to drain a spike, so scaling up can briefly make lag worse.
EAGER (default): add a pod → WHOLE group pauses → lag ticks up
▐███ pause ███▌ → reassign → resume
Two settings defuse it.
Cooperative rebalancing makes a rebalance incremental — only the partitions that actually move pause; every other partition keeps being consumed:
CooperativeStickyAssignor: add a pod → only moved partitions pause
the rest keep draining the entire time
Static membership gives each pod a stable identity, so a restart (a rolling deploy, a node move) skips the rebalance entirely — as long as the pod returns within session.timeout.ms. That needs a stable pod name, which is why the sink runs as a StatefulSet — its pods are named sink-0, sink-1, … and keep that ordinal across restarts:
# StatefulSet pod → stable ordinal name via the downward API
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name # sink-0, sink-1, … (survives a restart)
# consumer config:
group.instance.id: ${POD_NAME} # static member → a restart skips the rebalance
Add to those a scale-up policy that jumps to full width in one or two steps rather than climbing 1 → 2 → 3 → 4 (each step is a rebalance), and a generous cooldownPeriod so a draining burst doesn't make the fleet flap up and down. Now scaling adds throughput instead of pausing it. (All four settings, with values, are in the config glossary below.)
The ceiling: you cannot outscale your partitions
One hard limit ties it together. A Kafka partition is consumed by at most one consumer in a group. If the topic has 12 partitions, the 13th pod gets no assignment and sits idle. So:
effective max consumers ≤ number of partitions
That's why maxReplicaCount is set to the partition count. If bursts are so large that 12 fully-tuned consumers still can't keep up, the fix is more partitions (raising the ceiling) or a faster per-pod write path — not more replicas. Once you've hit the ceiling, adding pods is pure waste:
lag still climbing after replicas == partitions?
→ the bottleneck is no longer pods.
→ repartition, or make each pod drain faster — not more replicas.
Putting it together
nightly burst ──▶ partitions fill ──▶ lag rises
│
▼
┌──────────────────────────────────────────────────────────────┐
│ KEDA (autoscaler) │
│ scales 0 → N on lag, N ≤ partitions, in big steps │
└──────────────────────────────────────────────────────────────┘
│ + N pods
▼
┌──────────────────────────────────────────────────────────────┐
│ CONSUMERS (your config) │
│ fat polls: max.poll.records + fetch.min.bytes │
│ bulk idempotent upsert │
│ cooperative + static membership → scaling never pauses │
└──────────────────────────────────────────────────────────────┘
│ drain rate = N × per-pod rate
▼
lag peaks, then decays to 0 ──▶ cooldown ──▶ KEDA scales to 0
The system stays stable — lag bounded, then decaying — when #partitions × per-pod-drain-rate > peak produce rate, and you scale in few enough steps that rebalances don't eat the throughput you just added.
Takeaways
- Scale on work, not on resource usage. For event-driven sinks, consumer lag is the only signal that reflects real demand. CPU and memory stay flat while lag climbs.
- KEDA gives you elasticity and scale-to-zero — not throughput. A fleet of slow pods is still slow. Tune each pod: fatter polls, bulk idempotent writes. We saw about 100× the per-pod drain rate from tuning alone.
- Don't let autoscaling pause the drain. Cooperative rebalancing, static membership, and big scale-up steps keep consumption flowing while KEDA adds pods.
- Respect the partition ceiling. Consumers can't exceed partitions. Past that, repartition or speed up the write path — more replicas do nothing.
The net effect: a large set of always-on deployments became true on-demand workers — zero replicas when idle, a full-width tuned fleet within seconds of a burst, and lag that peaks and decays instead of climbing.
In a follow-up post: what happens on the database side when all those consumers write at once — connection pooling, write throughput, and when to shard.
Config glossary — every setting, and why it's there
The settings fall into five groups, each doing a distinct job. Read it as: autoscaling decides how many pods, the fetch and write knobs decide how fast each one drains, and the rebalance knobs make sure scaling doesn't pause that drain.
1. Autoscaling — how many pods (KEDA ScaledObject)
| setting | lab value | why it's there |
|---|---|---|
minReplicaCount |
0 |
enables scale-to-zero; this is what kills the idle-time cost |
maxReplicaCount |
12 |
caps replicas at the partition count — more pods can't get a partition (the ceiling) |
lagThreshold |
1000 |
target lag per replica; desiredReplicas ≈ totalLag / lagThreshold — lower = scale more aggressively |
activationLagThreshold |
0 |
the wake-from-zero trigger; any lag at all brings up the first pod |
cooldownPeriod |
60s |
how long to wait after lag clears before scaling down; stops the fleet flapping on a bursty topic |
pollingInterval |
10s |
how often KEDA checks lag; smaller reacts faster, at more query load |
scale-up behavior policy |
big steps | jump to full width in 1–2 steps instead of 1→2→3→4 → fewer rebalances |
fallback.replicas |
1 |
if the lag metric is unavailable, hold this many pods instead of dropping to 0 (safety net) |
2. Fetch — how fat each poll() is (consumer)
| setting | lab value | why it's there |
|---|---|---|
max.poll.records |
500 |
records handed to your code per poll; sets the bulk-write batch size |
fetch.min.bytes |
1 MB |
broker waits for this much data before replying → fewer, bigger round-trips under load |
fetch.max.wait.ms |
500 |
caps the wait above, so an idle topic still returns promptly |
3. Write path — turn a fat batch into a cheap write (sink)
| setting | lab value | why it's there |
|---|---|---|
bulk upsert (multi-row INSERT) |
on | one transaction per poll instead of N single writes — the biggest per-pod gain |
ON CONFLICT DO UPDATE |
on | idempotency; safe to reprocess a batch after a rebalance without corrupting the view |
4. Rebalance & stability — keep scaling from pausing the drain (consumer)
| setting | lab value | why it's there |
|---|---|---|
partition.assignment.strategy |
CooperativeStickyAssignor |
incremental rebalance; only moved partitions pause, the rest keep consuming |
group.instance.id |
pod name | static membership; a pod restart skips the rebalance entirely |
session.timeout.ms |
45s |
grace window a static member's partitions are held before reassignment on a restart |
max.poll.interval.ms |
300s |
max time between polls before the broker evicts you; must exceed how long a batch takes to process |
5. Topic — the parallelism ceiling (Kafka)
| setting | lab value | why it's there |
|---|---|---|
partitions |
12 |
one consumer per partition, so this is the hard cap on useful consumers — size it for peak parallelism |
Appendix — Reproduce it yourself
The whole setup runs on a laptop: a local Kubernetes cluster (kind), Kafka via Strimzi, real KEDA, a tunable Spring Kafka sink, and a live lag dashboard. No cloud account needed.
Prerequisites: docker, kind, kubectl, helm, java 17, mvn (~6 GB of free RAM for Docker).
# 0. Get the code
git clone https://github.com/Ajeethkumar-r/kafka-lag-keda-lab.git
cd kafka-lag-keda-lab
# 1. Stand up cluster + operators + build image + deploy (naive profile)
make up # kind + Strimzi + KEDA + build + deploy (~5-8 min)
# 2. Let KEDA drive replicas from lag
make keda-on
# 3. Watch it live in a second terminal — http://localhost:8088
make dashboard # lag, replicas, drain rate, ETA
# 4. Fire a burst and watch the naive sink struggle (it stays stuck — no need to wait)
make burst NUM=1000000 # 1M messages → lag climbs, drains slowly
# 5. Switch to the tuned consumer and fire the SAME burst
make tuned
make burst NUM=1000000 # lag peaks, then collapses; replicas 0 → 12 → 0
# 6. Tear it all down
make down
What the dashboard shows:
NAIVE profile, after a burst:
Consumers 0 → 12 (KEDA scaled up on lag)
Total lag climbs, drains ~175/s → stuck high for a long time
Trend ▲ climbing
TUNED profile, same burst:
Consumers 0 → 12 → 0 (scaled up, drained, scaled back to zero)
Total lag peaks, drains ~19k/s → clears in under a minute
Trend ▼ draining
Where each knob lives:
| concept | file |
|---|---|
| scale-to-zero on lag | k8s/40-scaledobject.yaml |
lagThreshold / maxReplicaCount
|
k8s/40-scaledobject.yaml |
| partitions = the ceiling |
k8s/10-kafka.yaml (partitions: 12) |
max.poll.records / fetch.*
|
k8s/32-sink-tuned.configmap.yaml |
CooperativeStickyAssignor |
k8s/32-sink-tuned.configmap.yaml |
| static membership (pod name) |
k8s/30-sink.yaml (POD_NAME via the downward API) |
| bulk idempotent upsert | sink/.../SinkWriter.java |
Flip a single ConfigMap between the NAIVE and TUNED profiles, re-run the same burst, and the difference in drain behavior shows up on the dashboard in real time.
Top comments (0)