DEV Community: AjeethKumar_Ramesh

Taming Kafka Lag Spikes with KEDA Scale-to-Zero

AjeethKumar_Ramesh — Sat, 15 Aug 2026 09:23:53 +0000

How we turned always-on Kafka sinks into on-demand workers that shrug off nightly bombardments — by scaling on the right signal, tuning per-pod drain rate, and keeping autoscaling from sabotaging itself.

Every number in this post is measured from a local lab you can run yourself — the full code is on GitHub, and the Appendix has the commands.

The problem

We run a fleet of Kafka sinks — consumer services that read change events from Kafka, apply business logic, and write the result into a service-local database as a query-friendly materialized view. It keeps reads fast and independent from upstream systems, and it's a great pattern. But the workload has an awkward shape.

Most sinks are idle most of the day, then buried in minutes. Traffic isn't steady: changes arrive in bursts, usually from nightly imports or CDC jobs. The rest of the day the topic is quiet.

topic activity over 24h
  msgs ▲
       │              ██  nightly import / CDC burst
       │              ██
       │______________██______________  flat, idle ~22h/day
       └───────────────────────────────▶ time

That shape creates two problems at once:

Idle waste. When the topic is quiet, each sink still runs — it polls Kafka, holds connections, emits metrics, and occupies CPU and memory. Multiply one "small" sink across dozens of them and several regions, and you're paying around the clock for work that happens for a couple of hours a night.
Spike lag. When the burst lands, a backlog builds fast. If consumers can't drain it quickly enough, consumer lag — the gap between what's been produced and what's been processed — climbs, and downstream reads start serving stale data.

We want two things that sound contradictory: cost almost nothing when idle, and absorb the spike fast when it hits.

Why the obvious autoscaler doesn't help

The reflex is a Kubernetes Horizontal Pod Autoscaler (HPA) on CPU or memory. For sinks, that's the wrong signal.

Sink work is I/O-bound: the consumer spends its time waiting on Kafka polls and database writes, not burning CPU. So when a backlog builds, CPU stays flat while lag climbs — and an HPA watching CPU concludes everything is fine and never scales.

during a burst:
   lag   ▲  ███████   ← climbing fast (real work waiting)
   CPU   ▲  ▁▁▁▁▁▁▁   ← flat (blocked on I/O, not compute)
             → HPA sees low CPU, never scales, lag grows silently

The question a sink autoscaler must answer isn't "is this pod using CPU?" — it's "is there work waiting to be processed?" The only signal that answers it directly is Kafka consumer lag.

That reframes the whole solution. We need to scale on lag, we need each pod to actually drain fast, and we need scaling itself not to get in the way. Three moves.

The solution, in three moves

Move 1 — Scale on lag, all the way to zero, with KEDA

KEDA (Kubernetes Event-Driven Autoscaling) lets Kubernetes scale on external signals — here, Kafka consumer-group lag — and, crucially, can scale a workload to zero. A plain HPA can't go from one replica to zero; KEDA manages the HPA and the scale-to-zero around it.

You describe the workload and the trigger in a ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sink-scaler
spec:
  scaleTargetRef:
    kind: StatefulSet
    name: sink
  minReplicaCount: 0            # scale-to-zero — the whole point
  maxReplicaCount: 12           # = partition count (see "The ceiling")
  cooldownPeriod: 60            # stay up until the backlog is fully drained
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: hotel-cluster-kafka-bootstrap:9092
        consumerGroup: foo-consumer-group
        topic: hotel-topics
        lagThreshold: "1000"          # desired replicas ≈ totalLag / 1000
        activationLagThreshold: "0"   # wake from zero the instant lag appears

KEDA polls the lag, turns it into an autoscaling metric, and drives replicas roughly as desiredReplicas ≈ totalLag / lagThreshold, capped at maxReplicaCount.

What we measured. Idle, the sink sits at 0 replicas — no pods, no cost. We fired a one-million-message burst and watched KEDA react:

time	replicas	total lag	what KEDA did
idle	0	0	scaled to zero — no pods, no cost
+6s	12	654,364	saw lag, jumped straight to 12
+18s	12	357,452	12 pods draining in parallel
+36s	12	0	backlog cleared
~+96s	0	0	cooldown elapsed → back to zero

That 0 → 12 → 0 cycle is the "always-on to on-demand" shift in one table: you pay for sink compute only when there's work.

One honest caveat about this run: all 12 pods wrote to a single shared database in the lab, which capped the aggregate drain rate — so this isn't "12× one pod." The point here is the scaling shape (zero → full → zero on lag). Per-pod throughput is a separate measurement, and it's where Move 2 comes in. (Scaling the write path itself is a topic for a follow-up post.)

Move 2 — More pods isn't enough: raise each pod's drain rate

Here's the trap most KEDA write-ups skip: KEDA gives you pods, but a single pod's throughput is set by your consumer configuration. If each pod drains slowly, a fleet of slow pods still drains slowly. We proved it by running the same burst through the same single consumer, twice — once naive, once tuned.

The naive baseline. One database round-trip per record, one record per poll():

NAIVE — one upsert per row, max.poll.records = 1
  ≈ 175 records/s per pod
  on a 40k burst: lag sits around ~37k, dropping only ~500 every 3 seconds
  a 1M backlog would take ≈ 1.5 hours

The consumer isn't "busy" in CPU terms — it's just doing tiny, chatty units of work. Two changes fix that.

Lever 1 — fetch fatter batches per poll. Three consumer settings control how much a single poll() returns:

setting	what it does
`fetch.min.bytes`	broker waits until it has at least this many bytes before replying
`fetch.max.wait.ms`	…but never waits longer than this (stays responsive when idle)
`max.poll.records`	caps how many records one `poll()` hands your code

Under a flood, fetch.min.bytes is satisfied instantly, so the broker returns a big batch and max.poll.records lets your code process a fat chunk at once — far fewer, far larger round-trips.

Lever 2 — write the whole batch in one transaction. Instead of N single upserts, do one bulk, idempotent upsert per poll:

INSERT INTO materialized_view (id, payload, updated_at)
VALUES (?, ?, now()), (?, ?, now()), ...          -- the entire poll() batch
ON CONFLICT (id) DO UPDATE
  SET payload = EXCLUDED.payload, updated_at = now();

ON CONFLICT keeps it idempotent, so reprocessing after a rebalance is safe and can't corrupt the view.

The tuned result. With the two levers on:

# the "after" consumer profile
KAFKA_MAX_POLL_RECORDS: "500"
KAFKA_FETCH_MIN_BYTES:  "1048576"   # ~500 × 2KB → the broker returns a full batch
SINK_BULK:              "true"       # one bulk upsert per poll

TUNED — same single consumer, same burst
  ≈ 19,000 records/s
  a 1M backlog drains in ≈ 50 seconds  (vs ≈ 1.5 hours naive)

   lag ▲  peak ╲
               ╲____
                    ╲______        same partitions, same single pod
   0  ─────────────────────╲_____▶  ~50s

That's about 100× the per-pod drain rate, purely from reading fatter and writing in bulk — no extra pods. This is the lever KEDA can't pull for you. And since sinks are usually database-bound, batching the write is where most of the real gain lives.

Move 3 — Don't let scaling sabotage the drain: rebalances

Now the subtle failure mode. When KEDA scales 1 → 6 → 12 mid-burst, every change in group membership triggers a Kafka consumer group rebalance — a renegotiation of which consumer owns which partition. With the default eager rebalancing, that's stop-the-world: every consumer gives up all its partitions, re-joins, and only then resumes. Consumption pauses across the whole group exactly when you're trying to drain a spike, so scaling up can briefly make lag worse.

EAGER (default): add a pod → WHOLE group pauses → lag ticks up
   ▐███ pause ███▌ → reassign → resume

Two settings defuse it.

Cooperative rebalancing makes a rebalance incremental — only the partitions that actually move pause; every other partition keeps being consumed:

CooperativeStickyAssignor: add a pod → only moved partitions pause
   the rest keep draining the entire time

Static membership gives each pod a stable identity, so a restart (a rolling deploy, a node move) skips the rebalance entirely — as long as the pod returns within session.timeout.ms. That needs a stable pod name, which is why the sink runs as a StatefulSet — its pods are named sink-0, sink-1, … and keep that ordinal across restarts:

# StatefulSet pod → stable ordinal name via the downward API
env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name      # sink-0, sink-1, … (survives a restart)
# consumer config:
group.instance.id: ${POD_NAME}        # static member → a restart skips the rebalance

Add to those a scale-up policy that jumps to full width in one or two steps rather than climbing 1 → 2 → 3 → 4 (each step is a rebalance), and a generous cooldownPeriod so a draining burst doesn't make the fleet flap up and down. Now scaling adds throughput instead of pausing it. (All four settings, with values, are in the config glossary below.)

The ceiling: you cannot outscale your partitions

One hard limit ties it together. A Kafka partition is consumed by at most one consumer in a group. If the topic has 12 partitions, the 13th pod gets no assignment and sits idle. So:

effective max consumers ≤ number of partitions

That's why maxReplicaCount is set to the partition count. If bursts are so large that 12 fully-tuned consumers still can't keep up, the fix is more partitions (raising the ceiling) or a faster per-pod write path — not more replicas. Once you've hit the ceiling, adding pods is pure waste:

lag still climbing after replicas == partitions?
   → the bottleneck is no longer pods.
   → repartition, or make each pod drain faster — not more replicas.

Putting it together

   nightly burst  ──▶  partitions fill  ──▶  lag rises
        │
        ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ KEDA  (autoscaler)                                           │
   │   scales 0 → N on lag,  N ≤ partitions,  in big steps        │
   └──────────────────────────────────────────────────────────────┘
        │  + N pods
        ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ CONSUMERS  (your config)                                     │
   │   fat polls: max.poll.records + fetch.min.bytes              │
   │   bulk idempotent upsert                                     │
   │   cooperative + static membership → scaling never pauses     │
   └──────────────────────────────────────────────────────────────┘
        │  drain rate = N × per-pod rate
        ▼
   lag peaks, then decays to 0  ──▶  cooldown  ──▶  KEDA scales to 0

The system stays stable — lag bounded, then decaying — when #partitions × per-pod-drain-rate > peak produce rate, and you scale in few enough steps that rebalances don't eat the throughput you just added.

Takeaways

Scale on work, not on resource usage. For event-driven sinks, consumer lag is the only signal that reflects real demand. CPU and memory stay flat while lag climbs.
KEDA gives you elasticity and scale-to-zero — not throughput. A fleet of slow pods is still slow. Tune each pod: fatter polls, bulk idempotent writes. We saw about 100× the per-pod drain rate from tuning alone.
Don't let autoscaling pause the drain. Cooperative rebalancing, static membership, and big scale-up steps keep consumption flowing while KEDA adds pods.
Respect the partition ceiling. Consumers can't exceed partitions. Past that, repartition or speed up the write path — more replicas do nothing.

The net effect: a large set of always-on deployments became true on-demand workers — zero replicas when idle, a full-width tuned fleet within seconds of a burst, and lag that peaks and decays instead of climbing.

In a follow-up post: what happens on the database side when all those consumers write at once — connection pooling, write throughput, and when to shard.

Config glossary — every setting, and why it's there

The settings fall into five groups, each doing a distinct job. Read it as: autoscaling decides how many pods, the fetch and write knobs decide how fast each one drains, and the rebalance knobs make sure scaling doesn't pause that drain.

1. Autoscaling — how many pods (KEDA `ScaledObject`)

setting	lab value	why it's there
`minReplicaCount`	`0`	enables scale-to-zero; this is what kills the idle-time cost
`maxReplicaCount`	`12`	caps replicas at the partition count — more pods can't get a partition (the ceiling)
`lagThreshold`	`1000`	target lag per replica; `desiredReplicas ≈ totalLag / lagThreshold` — lower = scale more aggressively
`activationLagThreshold`	`0`	the wake-from-zero trigger; any lag at all brings up the first pod
`cooldownPeriod`	`60s`	how long to wait after lag clears before scaling down; stops the fleet flapping on a bursty topic
`pollingInterval`	`10s`	how often KEDA checks lag; smaller reacts faster, at more query load
scale-up `behavior` policy	big steps	jump to full width in 1–2 steps instead of `1→2→3→4` → fewer rebalances
`fallback.replicas`	`1`	if the lag metric is unavailable, hold this many pods instead of dropping to 0 (safety net)

2. Fetch — how fat each `poll()` is (consumer)

setting	lab value	why it's there
`max.poll.records`	`500`	records handed to your code per poll; sets the bulk-write batch size
`fetch.min.bytes`	`1 MB`	broker waits for this much data before replying → fewer, bigger round-trips under load
`fetch.max.wait.ms`	`500`	caps the wait above, so an idle topic still returns promptly

3. Write path — turn a fat batch into a cheap write (sink)

setting	lab value	why it's there
bulk upsert (multi-row `INSERT`)	on	one transaction per poll instead of N single writes — the biggest per-pod gain
`ON CONFLICT DO UPDATE`	on	idempotency; safe to reprocess a batch after a rebalance without corrupting the view

4. Rebalance & stability — keep scaling from pausing the drain (consumer)

setting	lab value	why it's there
`partition.assignment.strategy`	`CooperativeStickyAssignor`	incremental rebalance; only moved partitions pause, the rest keep consuming
`group.instance.id`	pod name	static membership; a pod restart skips the rebalance entirely
`session.timeout.ms`	`45s`	grace window a static member's partitions are held before reassignment on a restart
`max.poll.interval.ms`	`300s`	max time between polls before the broker evicts you; must exceed how long a batch takes to process

5. Topic — the parallelism ceiling (Kafka)

setting	lab value	why it's there
`partitions`	`12`	one consumer per partition, so this is the hard cap on useful consumers — size it for peak parallelism

Appendix — Reproduce it yourself

The whole setup runs on a laptop: a local Kubernetes cluster (kind), Kafka via Strimzi, real KEDA, a tunable Spring Kafka sink, and a live lag dashboard. No cloud account needed.

Prerequisites: docker, kind, kubectl, helm, java 17, mvn (~6 GB of free RAM for Docker).

# 0. Get the code
git clone https://github.com/Ajeethkumar-r/kafka-lag-keda-lab.git
cd kafka-lag-keda-lab

# 1. Stand up cluster + operators + build image + deploy (naive profile)
make up                          # kind + Strimzi + KEDA + build + deploy (~5-8 min)

# 2. Let KEDA drive replicas from lag
make keda-on

# 3. Watch it live in a second terminal — http://localhost:8088
make dashboard                   # lag, replicas, drain rate, ETA

# 4. Fire a burst and watch the naive sink struggle (it stays stuck — no need to wait)
make burst NUM=1000000           # 1M messages → lag climbs, drains slowly

# 5. Switch to the tuned consumer and fire the SAME burst
make tuned
make burst NUM=1000000           # lag peaks, then collapses; replicas 0 → 12 → 0

# 6. Tear it all down
make down

What the dashboard shows:

NAIVE profile, after a burst:
  Consumers  0 → 12         (KEDA scaled up on lag)
  Total lag  climbs, drains ~175/s → stuck high for a long time
  Trend      ▲ climbing

TUNED profile, same burst:
  Consumers  0 → 12 → 0     (scaled up, drained, scaled back to zero)
  Total lag  peaks, drains ~19k/s → clears in under a minute
  Trend      ▼ draining

Where each knob lives:

concept	file
scale-to-zero on lag	`k8s/40-scaledobject.yaml`
`lagThreshold` / `maxReplicaCount`	`k8s/40-scaledobject.yaml`
partitions = the ceiling	`k8s/10-kafka.yaml` (`partitions: 12`)
`max.poll.records` / `fetch.*`	`k8s/32-sink-tuned.configmap.yaml`
`CooperativeStickyAssignor`	`k8s/32-sink-tuned.configmap.yaml`
static membership (pod name)	`k8s/30-sink.yaml` (`POD_NAME` via the downward API)
bulk idempotent upsert	`sink/.../SinkWriter.java`

Flip a single ConfigMap between the NAIVE and TUNED profiles, re-run the same burst, and the difference in drain behavior shows up on the dashboard in real time.

Everything you need to know about Git

AjeethKumar_Ramesh — Wed, 29 Dec 2021 13:34:25 +0000

Move fast and break things. Unless you are breaking stuff, you are not moving fast enough. - Mark Zuckerberg

Hope you all doing well, by the end of this post you will have complete knowledge to start with git and also familiar on it,

⚡ Git
It is a version control tool in short this maintains your code history, you can take a real time scenario , Browser is a tool which stores your Browsing history

Git Bash

we are going to use Git Bash which is a terminal, while installing git enable linux commands to get the full features.

keep reading and you will know everything you need to know...

Folder creation
Create a Newfolder give it a name as you like and open in Git Bash or you can go with Command propmt,

your bash will look something like this

If you are in command prompt make sure your are working in right directory.
Initialize git repository
Initialize git in this folder git init, it will create a empty repository in your local device

File creation
Create a new file and open it with vscode ,it is my opinion you can also do it with atom, sublime, it's completely up to you, you will see master, It is a branch no need to worry about this now, I will explain in a while,
To create a file in command prompt use echo >fileName

It will look similar to this,

Make some changes in the file and don't forget to save it

Clean terminal
sometimes we are the situation to make our bash looks clean, type clear and hit enter. For command prompt use cls and give enter

Check status of commit
To check the status of our commits ,the info of file use git status it will show you the file which is tracked or untracked,
tracked denotes our file is staged and untracked denotes the file is not yet staged, Obviously we didn't commit anything so no commits will appear.

Eye and vision
Now the real part begins, to make our changes as a history there are two steps:

staging the change
commit the change

Stage
To bring our changes to the staging area use git add ., you can replace period simple . with your file name to be staged, generally the period simple includes all the changes in to staging area

Commit
To commit the changes of the staged file use git commit -m "your commit message" and it will show the number of changes and type of changes that are insertion or deletion to the file

This is similar to photo taking scenario in marriage reception, the people who want to take photo should come to stage , after taking photo it will save in the marriage album, changes we need to save as history should be staged first and then commit .

Again we check our status it show you working tree clean ,

Create another file and make changes check the status

stage the changes

Dump area
Type git stash, check the status of your commits again , it shows nothing to commit working tree clean,

Take a realtime scenario, if you are working on some kind of logical code in project and you commit the changes that might have issues, so you decide not to lose the changes but also keep it for future purpose.
We use git stash, stash is a hidden place where we can store our commits until it really needs.

Incase now you need the commits from stash area now, we need to bring it in staging area again using git stash pop

Now we can commit those changes and check status of our commits if you want,

To see all the commits which are recorded as history use git log

so far you are doing great !!!

Host on Github
To add our local folder or a local repository to Github, there are two steps

1.Create a Repository
2.Add local repo to github

Github
To create a new repository on github, repository(repo) is simply a storage area of all our file along with the commits respectively.

Create Repository
For creating a repo you should have an account in github, so signUp and then follow along with me, if you already have an account login to your account , go to repository click new.

Give a name for your repo and click Create repository

and now your screen will appear like this, github show you the steps here we need last two steps , because already we did up to commit,

Add local repo to github
Need to copy HTTPS or SSH link and work on it, word remote denotes you are working with URL, origin is basically your github account.

To push the changes that we made in our local repo to github main repo

Now go to github tab and refresh the browser and the magic happens, you are able to see the changes that we made in our local repo in main repo

Restore commits
Change something in file1 or file2 and stage the change , now if we want to restore changes we made stage it and check status

and now use git restore --staged fileName , leave --staged flag to restore local changes too and check status

Remove file
To remove the file from working tree use git rm --cached fileNameand check status

Tip
Stage the file, commit it and check status, it shows your branch is ahead of origin/master by 1 commit and inform us to push local commits, whenever these kind of situation happens do as your command line said. I know that's a lot, move forward you need few more concepts to complete.

As the status shows we need to push the changes and check status

Hosting on github
Go to your browser and refresh, you will be amazed, give an applause to yourself. Note the commit back to normal that is pushed from our local branch.

Create another file make some changes, stage that and commit

Commit records
See all the commit messages using git log.

Structure of Commit messages:
Commit messages are align on top of one another, To delete commit messages you need to pick the commitID and paste in git reset commitID, this will delete all the commits above the one which you pick.

Commit messages are like a bread slices

I want this to be clean and understandable , for that delete file3.txt in my case , you may leave it.

To push our changes to github main repo git push origin master

Pointer
It's time to know HEAD, this is simply a pointer which points to the active branch , you will understand more in a bit.
Branch
Now we are going to work on branches, to create a branch use git branch branchName and checkout this branch by git checkout branchName, after checking out the HEAD is now point to the new branchName, here after the commits we will made is from branchName

Create new branch , checkout then create a file and change something ,stage it and commit

Make some more commits may be two is good to go and git log now

we need to push our change in local branch to github

Newbie-our branch will be on github
Now your github screen automatically shows you Compare & pull request , click on that

Now click on create pull request it will show you

Merge conflict
Here if there is no conflict on your change can simply click on Merge pull request, conflict is popularly known as merge conflict , assume that two persons working on same change and commit it, github doesn't know which change to take in this case we manually make the change on github and then we are allowed
merge pull request

I know we are doing tons of things but it's worth .
Now click confirm merge, there you can delete the branch if it is no longer needed

Go to your code now and refresh the page you can see there will second branch and the file along with it's commit

Hope you feel confidence on yourself .
Rebase- new base
There is a little more to do , for that need some changes in master branch may be three commits are good to go , now git log, In sometimes we need to reduce number of commits as one commit, assume that you are working in login form of your project you will commit each changes and push it, those commits are points to single work so we squash all three to one commit, here we use git rebase, before that we need to pick the commit id , this has same hierarchy as git reset

use this command git rebase -i CommitID, i is interactive most of the time we need -i flag, give enter this will show you like this

Squash
you can see that Pick and s ,we don't need to change the first pick it remains same, now if we want to squash the commit we change it to s, at least the commits should have one pick ,the s commits combine with the nearest pick commit , it doesn't matter how many commits to be squashed , you should have one pick which will be above s commit . To know more you can refer , to type in this click i.

By default Vim is the text editor for git, this below lines will help the non vim users,
to escape from vim editor click ESC, colon : and x after that you have screen like this

remove the # which is used to comment the line then type you commit message, after that click ESC , : and x to exit edit mode

we finished rebasing now time to push it to github

here is the Squashed commit

Go to github now you can see the change in commit message

git push -push the changes along with commit messages.
git fetch -fetch the recent commit messages.
git pull -pull the code with commit messages, it internally operate git fetch .

To know more you can refer git atlassian

Learning never fulfilled untill you practice on your own and share it with other people .

Alright now you can add git skill in profile and be happy for what you have done by reading this post.

querySelector() and querySelectorAll()

AjeethKumar_Ramesh — Wed, 28 Jul 2021 04:01:11 +0000

Let me explain the most basics of querySelector() by name.

Don't skip while reading these lines

$-CREATIVE EXPLANATION:

Generally querySelector() and querySelectorAll() from same family which is "API".

query: His most popular name is "question", he is a toddler in API family

Selector: She is the mother of our toddler(query), without her(Selector) query don't know how to work or what to do.

If you understand the pretty lines above go ahead to read real concept,

$-SUBJECT:

querySelector(): To get the particular element from your html document,and make fancy using css or you can use it in your js codes, Which only select the first instance of the element.

querySelectorAll(): Only difference between querySelector() and querySelectorAll() is it(querySelectorAll()) can select as many as instances specifically a particular group of elements having same name (.classNames) or by the tags(h1,p,div etc.,)

This is my first blog, I hope it has little fun with explaining concept, //if it has any mistakes feel free to comment.