Kafka Consumer Group Internals: Rebalance Protocols, Static Membership & Cooperative Sticky

A Kafka consumer group is the abstraction that lets a fleet of consumer instances share the partitions of a topic without ever reading the same record twice — and it is the single component streaming engineers get wrong most often, because "just add more consumers" quietly triggers a rebalance protocol that can stop the world, drop every partition, replay uncommitted offsets, and stall a pipeline for seconds at exactly the moment traffic spikes. Every partition of every subscribed topic must be owned by exactly one member of the group at any instant; when membership changes — a pod deploys, a consumer misses a heartbeat, an operator scales out — the group coordinator has to recompute that ownership and hand it back to each member, and the way it does that recomputation is the difference between a 50 ms hiccup and a 30-second outage.

This guide is the senior-streaming walkthrough you wished existed the first time an interviewer asked "walk me through what happens between JoinGroup and SyncGroup," or "your consumers rebalance on every rolling restart — how do you stop it," or "explain the cooperative sticky assignor and why it beats the eager range assignor." It walks through the group coordinator and generation-fencing model, the eager versus incremental rebalance protocols, static membership via group.instance.id, the four partition assignment strategies (range, round-robin, sticky, cooperative-sticky), and the offset commit semantics against the __consumer_offsets topic that decide whether your delivery guarantee is at-least-once, at-most-once, or exactly-once. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse on the event-processing practice library →, and sharpen the pipeline axis with the real-time-analytics practice library →.

---

On this page

• Why the consumer group model determines everything
• The rebalance protocol — JoinGroup, SyncGroup, generations
• Static membership and stable groups
• Cooperative sticky assignor and incremental rebalance
• Offset commit semantics and delivery guarantees
• Cheat sheet — Kafka consumer group recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why the consumer group model determines everything

Four moving parts — coordinator, membership, assignment, generation — decide whether scaling out is a hiccup or an outage

The one-sentence invariant: a Kafka consumer group is a set of members that collectively subscribe to one or more topics, where the group coordinator guarantees that every partition is owned by exactly one live member at any instant, and any change to membership forces a rebalance that recomputes that ownership — so the pattern you pick for assignment, membership stability, and offset commit decides whether a deploy is invisible or a stall. The group is not a load balancer you configure once; it is a live consensus over "who owns which partition" that is renegotiated every time a member joins, leaves, or misses a heartbeat, and every downstream latency SLO you promise is really a promise about how cheap that renegotiation is.

The four moving parts every senior answer names.

• The group coordinator. One broker is elected coordinator for a given group.id (the broker that leads the __consumer_offsets partition the group hashes to). It tracks membership, receives heartbeats, drives rebalances, and stores committed offsets. Every consumer talks to its coordinator, not to each other.
• Membership and heartbeats. Each member sends a heartbeat every heartbeat.interval.ms; if the coordinator sees no heartbeat within session.timeout.ms, it declares the member dead and rebalances. A member that stops calling poll() for longer than max.poll.interval.ms is also evicted — a slow processing loop looks identical to a dead consumer.
• Partition assignment. The partition.assignment.strategy (range, round-robin, sticky, or cooperative-sticky) decides how the group leader maps partitions to members. The coordinator does not compute the assignment — it elects a leader consumer, the leader computes it, and the coordinator distributes it via SyncGroup.
• Generation (epoch). Every completed rebalance bumps a monotonic generationId. Any request carrying a stale generation is rejected with ILLEGAL_GENERATION, fencing "zombie" members that were partitioned away and came back thinking they still own partitions.

The four axes interviewers actually probe.

• Who owns which partition, and can two members own the same one? The invariant is exactly-once ownership per partition per generation. A correct answer explains that duplicate ownership is impossible within a generation and that generation fencing is what prevents a lagging member from double-consuming after a rebalance.
• What triggers a rebalance? Member join, member leave (graceful LeaveGroup or session timeout), max.poll.interval.ms breach, topic-metadata change (partition count grows), and subscription change. Naming all five separates the fluent from the memorised.
• Eager vs cooperative revocation. Eager (the classic protocol) revokes every partition from every member at the start of a rebalance — stop-the-world. Cooperative revokes only the partitions that actually change hands, so members keep processing the ones they retain.
• Where and when are offsets committed? Offsets live in the __consumer_offsets compacted topic. The commit point relative to the rebalance and to record processing is what sets the delivery guarantee — at-least-once, at-most-once, or exactly-once.

The 2026 reality — cooperative sticky + static membership are the defaults that tame rebalances.

• Cooperative sticky is the default assignor recommendation for any new consumer deployment. It combines stickiness (keep partitions with the member that had them) with incremental cooperative revocation (only move what must move), which turns a scale-out from a stop-the-world event into a background reassignment.
• Static membership (KIP-345, group.instance.id) is the standard fix for rebalance storms on rolling restarts. A member with a stable instance id can leave and rejoin within session.timeout.ms and reclaim its exact prior assignment without triggering a rebalance at all.
• Manual offset commit with enable.auto.commit=false is the senior default for anything that needs an at-least-once guarantee stronger than "whatever auto-commit happened to flush." Auto-commit is fine for idempotent sinks and lossy telemetry; it is a subtle bug for anything that must not reprocess or drop.
• Kafka Streams and the transactional producer layer exactly-once semantics (EOS) on top of the group by committing offsets inside the same transaction as the output writes — the group primitives are the same, but the commit is atomic with the produce.

What interviewers listen for.

• Do you say "exactly one member owns each partition per generation" rather than "consumers share the topic"? — required answer.
• Do you name the group coordinator as a specific broker and distinguish it from the partition leader? — senior signal.
• Do you explain that the leader consumer, not the coordinator, computes the assignment? — senior signal.
• Do you name generation fencing as the zombie-defense, not just "Kafka handles it"? — senior signal.
• Do you tie the offset-commit point to the delivery guarantee instead of treating commits as bookkeeping? — required answer.

Worked example — the four-part mental model of a consumer group

Detailed explanation. The single most useful artifact for a consumer-group interview is a four-box mental model: coordinator, membership, assignment, generation. Every senior Kafka discussion converges on these four within the first ten minutes; having them in your head is what separates a fluent answer from a stumbling one. Walk through building the model for a hypothetical orders topic with 12 partitions and a group of 3 consumers.

• Topic. orders with 12 partitions, replication factor 3.
• Group. order-enrichment with 3 consumer instances (C1, C2, C3).
• Target. Each consumer owns 4 partitions; scaling to 4 consumers should move 3 partitions, not all 12.
• Coordinator. The broker leading the __consumer_offsets partition that hash("order-enrichment") % 50 lands on.

Question. Map the four moving parts onto this group and state the ownership invariant that must hold at every instant.

Input.

Moving part	Concrete value for order-enrichment
Group coordinator	one broker (owner of the group's offsets partition)
Members	C1, C2, C3 (heartbeat every 3 s; session 45 s)
Assignment strategy	cooperative-sticky
Generation	starts at 1; +1 per completed rebalance

Code.

Consumer group "order-enrichment" — steady state, generation 5
==============================================================

  topic: orders  (12 partitions: P0..P11)

  Group coordinator (broker 2)
    ├─ tracks members: C1, C2, C3
    ├─ receives heartbeats every 3s
    ├─ stores committed offsets in __consumer_offsets
    └─ current generationId = 5

  Assignment (computed by the LEADER consumer, C1):
    C1 → P0  P1  P2  P3
    C2 → P4  P5  P6  P7
    C3 → P8  P9  P10 P11

  Invariant: every partition P0..P11 is owned by exactly ONE of
  {C1, C2, C3} in generation 5. No partition is unowned; no
  partition is double-owned.

Step-by-step explanation.

1. The coordinator is a broker, chosen by hashing the group.id to one of the 50 (default) __consumer_offsets partitions and taking that partition's leader. This is why the coordinator is stable for a given group but different groups spread across brokers.
2. Membership is a live set maintained by heartbeats. C1/C2/C3 each send a heartbeat on a background thread every heartbeat.interval.ms; the coordinator expires any member it has not heard from within session.timeout.ms.
3. The assignment is computed by the leader consumer (here C1, the first to join the current generation), not by the coordinator. The coordinator only ships each member's subscription to the leader and ships the leader's computed assignment back out.
4. The generation id (5) stamps every request. If C3 were network-partitioned during a rebalance to generation 6 and then sent an offset commit tagged generation 5, the coordinator rejects it with ILLEGAL_GENERATION — C3 must rejoin.
5. The ownership invariant — exactly one owner per partition per generation — is the whole point of the group. Everything else (heartbeats, generations, assignors) exists to preserve it across membership changes without ever letting two members process the same partition simultaneously.

Output.

Question an interviewer asks	One-line senior answer
Who assigns partitions?	The leader consumer computes; the coordinator distributes
Can two consumers read P0?	Not within a generation; fencing blocks stale owners
What is the coordinator?	A broker — leader of the group's __consumer_offsets partition
What bumps the generation?	Every completed rebalance

Rule of thumb. Never describe a consumer group as "consumers sharing a topic." Describe it as "exactly-once partition ownership per generation, renegotiated on every membership change" — that framing makes every follow-up (rebalance cost, static membership, sticky assignment) fall out naturally.

Worked example — what interviewers actually probe

Detailed explanation. The senior Kafka consumer-group interview has a predictable escalation: the interviewer opens with an ambiguous scaling question ("how do you add throughput to a consumer?"), then narrows to test whether you understand rebalances, membership, and offsets. Candidates who mention rebalance cost unprompted score highest; candidates who say "just add consumers" score lowest. Walk through the grading rubric.

• Ambiguous opener. "Your consumer can't keep up — what do you do?" — invites you to name partitions and the group.
• Follow-up 1. "What happens when you add the 4th consumer?" — probes the rebalance.
• Follow-up 2. "Why did latency spike during your last deploy?" — probes stop-the-world eager rebalance and static membership.
• Follow-up 3. "How do you guarantee you don't reprocess?" — probes offset-commit timing.
• Follow-up 4. "How do you replay yesterday?" — probes offset reset and __consumer_offsets.

Question. Draft a 5-minute senior consumer-group answer that covers scaling, rebalance cost, membership stability, and delivery guarantees without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Scaling unit	"add more consumers"	"add consumers up to the partition count; partitions are the parallelism ceiling"
Rebalance	"Kafka rebalances"	"adding a member triggers a rebalance; eager stops the world, cooperative moves only what changes"
Deploy spike	"restarts are slow"	"rolling restart rebalances twice per pod unless static membership holds the assignment"
No reprocess	"we commit offsets"	"commit after processing for at-least-once; commit on revoke so a rebalance doesn't replay"
Replay	"reset the consumer"	"seek or reset the committed offset in __consumer_offsets; auto.offset.reset only fires with no committed offset"

Code.

Senior consumer-group answer template (5 minutes)
=================================================

Minute 1 — parallelism ceiling
  "Throughput scales with consumers up to the partition count. 12
   partitions => at most 12 useful consumers in the group; a 13th
   sits idle. If I need more parallelism I add partitions first."

Minute 2 — the rebalance
  "Adding a consumer triggers a rebalance. With the classic eager
   protocol every member revokes ALL partitions, then the leader
   reassigns — stop-the-world. With cooperative-sticky only the
   partitions that actually move are revoked; the rest keep flowing."

Minute 3 — deploy stability
  "A rolling restart rebalances on the pod leaving AND on it
   rejoining — two rebalances per pod. Static membership
   (group.instance.id) lets the pod rejoin within session.timeout.ms
   and reclaim its exact partitions with NO rebalance."

Minute 4 — delivery guarantee
  "Offsets live in __consumer_offsets. Commit AFTER processing for
   at-least-once (a crash replays uncommitted records). Commit BEFORE
   processing for at-most-once. For exactly-once I commit offsets
   inside the producer transaction and read with read_committed."

Minute 5 — rebalance + commit interplay
  "On revoke I commit synchronously in onPartitionsRevoked so the
   next owner starts from a durable offset. Without that, a rebalance
   replays everything since the last auto-commit."

Step-by-step explanation.

1. Minute 1 anchors the parallelism ceiling: consumers in a group cannot exceed partitions usefully. Weak candidates promise linear scaling; senior candidates name the partition count as the hard limit and reach for repartitioning when they need more.
2. Minute 2 is the rebalance-cost framing. Naming eager stop-the-world versus cooperative incremental up front signals you have felt the pain of a stalled pipeline, not just read the quickstart.
3. Minute 3 pre-empts the deploy-spike probe. "Two rebalances per pod on a rolling restart, unless static membership" is the exact senior sentence — it shows you know why deploys hurt and the one config that fixes it.
4. Minute 4 ties commit timing to the guarantee. The order — commit after vs before processing — is the guarantee; saying "we commit offsets" without the timing is the weak answer.
5. Minute 5 covers the subtle interaction: a rebalance that fires between the last commit and now will replay. Committing on revoke is the senior detail that most candidates miss.

Output.

Grading criterion	Weak score	Senior score
Names partition ceiling	rare	mandatory
Names eager vs cooperative	rare	senior signal
Names static membership for deploys	rare	senior signal
Ties commit timing to guarantee	occasional	mandatory
Commits on revoke	rare	senior signal

Rule of thumb. The senior consumer-group answer is a 5-minute monologue that covers parallelism ceiling, rebalance cost, deploy stability, and commit timing without waiting for the follow-ups. Rehearse it once; deploy it every time.

Senior interview question on consumer group fundamentals

A senior interviewer often opens with: "You run a consumer group of 3 instances on a 12-partition topic. During deploys latency spikes to seconds, and after a crash you sometimes reprocess an hour of data. Walk me through what the group coordinator is doing, why the deploy hurts, and the two configuration changes you'd make first."

Solution Using the group coordinator model, static membership, and commit-on-revoke

// Consumer configuration that fixes the two reported problems
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-enrichment");

// Fix 1 — stable identity so a rolling restart does NOT rebalance
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "order-enrichment-pod-1");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "45000");     // survive a fast restart
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "3000");

// Fix 2 — incremental rebalance so scaling is not stop-the-world
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

// Manual commit for a real at-least-once guarantee
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() {
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
        // Commit synchronously BEFORE we lose ownership so the next
        // owner starts from a durable offset — no reprocessing window.
        consumer.commitSync();
    }
    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> assigned) {
        // Newly assigned partitions resume from their committed offset.
    }
});

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> r : records) {
        process(r);                      // side effect must be idempotent
    }
    consumer.commitSync();               // commit AFTER processing => at-least-once
}

Step-by-step trace.

Step	Before (defaults)	After (this config)
Rolling restart	2 rebalances per pod (leave + rejoin)	0 rebalances if rejoin < 45 s
Scale-out rebalance	eager: revoke all 12 partitions	cooperative: revoke only the ~3 that move
Deploy latency spike	seconds (stop-the-world)	tens of ms (retained partitions keep flowing)
Crash recovery	replay since last auto-commit (up to 5 s)	replay only uncommitted batch
Reprocess-an-hour bug	possible if auto-commit lagged	eliminated by commit-on-revoke + commit-after-process

After the change, a rolling deploy no longer rebalances because each pod keeps its group.instance.id and rejoins inside the session window; a scale-out moves only the partitions that must change hands; and a crash replays at most the current uncommitted batch instead of an hour of data.

Output:

Metric	Before	After
Rebalances per rolling deploy	2 × pods	0 (static, within timeout)
Partitions moved on scale-out	all 12	~3
Deploy p99 consumer latency	seconds	tens of ms
Max reprocessed window after crash	~1 h (worst case)	1 batch
Delivery guarantee	"whatever auto-commit did"	at-least-once, explicit

Why this works — concept by concept:

• Group coordinator ownership — one broker owns the group's membership and offsets; every heartbeat, join, and commit flows through it, so a single authority enforces exactly-once partition ownership per generation.
• Static membership — group.instance.id gives the pod a durable identity; leaving and rejoining within session.timeout.ms reclaims the exact prior assignment with no rebalance, killing the two-rebalances-per-pod deploy tax.
• Cooperative sticky assignor — incremental revocation moves only the partitions that change hands, so retained partitions keep flowing during a scale-out instead of stopping the world.
• Commit-on-revoke plus commit-after-process — committing synchronously in onPartitionsRevoked and after each processed batch pins the durable offset just behind the work actually done, so recovery replays a batch, never an hour.
• Cost — one extra config block and a synchronous commit on revoke (a few ms per rebalance). The eliminated cost is O(deploys) stop-the-world stalls and an unbounded reprocessing window. Net: O(1) rebalance cost per real membership change instead of O(pods) per deploy.

Streaming
Topic — streaming
Streaming consumer-group and partitioning problems

Practice →

Design Topic — design Design problems on distributed consumers

Practice →

---

2. The rebalance protocol — JoinGroup, SyncGroup, generations

JoinGroup then SyncGroup — the two-round handshake the group coordinator runs to reassign partitions, fenced by a monotonic generation

The mental model in one line: a rebalance protocol is the two-round handshake — every member sends JoinGroup to the coordinator, the coordinator picks one member as leader and forwards it every member's subscription, the leader computes the assignment and returns it inside SyncGroup, and the coordinator fans that assignment back to each member — all stamped with a generationId that bumps by one and fences any member still operating on the old generation. Every rebalance you have ever watched in a consumer log is this handshake; understanding it is what lets you reason about why a rebalance stalls, why a zombie can't double-consume, and why the leader (not the broker) owns assignment logic.

The two rounds — JoinGroup and SyncGroup.

• JoinGroup (round 1). Every member sends JoinGroup carrying its subscription and its list of supported assignors. The coordinator waits until all known members have joined (or the rebalance.timeout.ms expires), then designates the first member to join as the group leader and returns to it the full member-to-subscription map. Non-leader members receive an empty response and just wait.
• SyncGroup (round 2). The leader runs the chosen assignor over the member/subscription map, produces a per-member partition assignment, and sends the whole assignment to the coordinator inside its SyncGroup request. Every other member sends an empty SyncGroup and blocks. The coordinator stores the assignment and returns each member its own slice in the SyncGroup response.
• Why the leader, not the coordinator, assigns. Assignment logic (range, sticky, cooperative) lives in the client library, so different client versions can ship new assignors without a broker upgrade. The broker stays a dumb, durable orchestrator; the smarts live client-side.

The generation id — the zombie fence.

• What it is. A monotonically increasing integer stamped on the group at the end of every successful rebalance. Members learn their generation in the SyncGroup response.
• What it fences. Any request (heartbeat, offset commit, sync) carrying a generation lower than the coordinator's current one is rejected with ILLEGAL_GENERATION. A member that was slow, paused, or partitioned re-joins rather than acting on stale ownership.
• Why it matters. Without generations, a member that missed a rebalance could keep consuming and committing partitions that now belong to someone else — double processing and offset corruption. The generation makes stale ownership unrepresentable.

The four rebalance triggers.

• Member joins. A new consumer sends JoinGroup; the coordinator triggers a rebalance to fold it in.
• Member leaves. Graceful shutdown sends LeaveGroup (immediate rebalance). A crash or hang is detected by missed heartbeats after session.timeout.ms, or by a poll() gap exceeding max.poll.interval.ms.
• Topic metadata change. A subscribed topic gains partitions, or a pattern subscription (subscribe(Pattern)) matches a newly created topic. New partitions must be assigned to someone.
• Subscription change. A member changes the set of topics it subscribes to, forcing a recompute.

Common interview probes on the rebalance protocol.

• "Who computes the assignment?" — required answer: the leader consumer, not the coordinator.
• "What are the two rounds?" — JoinGroup (elect leader, gather subscriptions) then SyncGroup (distribute assignment).
• "What does the generation id protect against?" — zombie members acting on stale ownership.
• "Why is a slow poll() loop a rebalance trigger?" — exceeding max.poll.interval.ms looks like a dead member.

Worked example — tracing a JoinGroup → SyncGroup rebalance

Detailed explanation. The canonical rebalance trace: a group of two consumers is joined by a third, and you follow the exact request sequence the coordinator drives. Being able to narrate this trace out loud is the difference between "Kafka rebalances" and a senior answer. Walk through the handshake for orders (6 partitions) as C3 joins C1/C2.

• Before. Generation 7; C1 owns P0-P2, C2 owns P3-P5.
• Trigger. C3 sends JoinGroup.
• Rounds. JoinGroup (leader elected, subscriptions gathered) then SyncGroup (assignment distributed).
• After. Generation 8; balanced 2-2-2 assignment.

Question. Narrate the JoinGroup/SyncGroup request sequence and the resulting assignment when C3 joins.

Input.

Step	Request	Sent by	Coordinator action
1	JoinGroup	C3 (new)	mark rebalance needed; bump target generation
2	JoinGroup	C1, C2	rejoin required; gather subscriptions
3	JoinGroup resp	→ leader C1	send full member/subscription map
4	SyncGroup	C1 (leader)	receive computed assignment
5	SyncGroup resp	→ C1, C2, C3	each gets its own partitions

Code.

Rebalance trace: C3 joins group "order-enrichment" on topic orders(6)
=====================================================================

Generation 7 (before)
  C1: [P0, P1, P2]   C2: [P3, P4, P5]

── Round 1: JoinGroup ────────────────────────────────────────────
  C3 --JoinGroup(subscribe=orders)--> Coordinator
  Coordinator: "rebalance in progress" -> C1, C2 must rejoin
  C1 --JoinGroup--> Coordinator
  C2 --JoinGroup--> Coordinator
  Coordinator elects LEADER = C1 (first to (re)join)
  Coordinator --JoinGroupResp(members={C1,C2,C3}, subs)--> C1
  Coordinator --JoinGroupResp(empty)--> C2, C3   # they just wait

── Leader computes assignment (client-side assignor) ─────────────
  C1 runs CooperativeStickyAssignor over {C1,C2,C3} x orders(6):
     C1 -> [P0, P1]   C2 -> [P3, P4]   C3 -> [P2, P5]

── Round 2: SyncGroup ────────────────────────────────────────────
  C1 --SyncGroup(assignment=<the map above>)--> Coordinator
  C2 --SyncGroup(empty)--> Coordinator
  C3 --SyncGroup(empty)--> Coordinator
  Coordinator stores assignment; stamps generation 8
  Coordinator --SyncGroupResp([P0,P1], gen=8)--> C1
  Coordinator --SyncGroupResp([P3,P4], gen=8)--> C2
  Coordinator --SyncGroupResp([P2,P5], gen=8)--> C3

Generation 8 (after): C1:[P0,P1] C2:[P3,P4] C3:[P2,P5]

Step-by-step explanation.

1. C3's JoinGroup flips the group into the "preparing rebalance" state. The coordinator responds to the other members' next heartbeats with REBALANCE_IN_PROGRESS, forcing C1 and C2 to re-send JoinGroup.
2. The coordinator waits for all known members to rejoin (bounded by rebalance.timeout.ms, which equals max.poll.interval.ms). Members that don't rejoin in time are dropped from the new generation.
3. The coordinator elects the first member to join as leader (C1) and hands it the complete member-to-subscription map. Everyone else gets an empty JoinGroup response and waits — they contribute subscriptions, not assignment logic.
4. C1 runs the configured assignor locally and produces the full assignment. It ships that map to the coordinator via SyncGroup. This is the step people misattribute to the broker; the broker never computes assignments.
5. The coordinator persists the assignment, bumps the generation to 8, and returns to each member only its own slice. From this instant, any request stamped generation 7 is fenced with ILLEGAL_GENERATION.

Output.

Member	Gen 7 (before)	Gen 8 (after)	Partitions moved
C1 (leader)	P0, P1, P2	P0, P1	lost P2
C2	P3, P4, P5	P3, P4	lost P5
C3 (new)	—	P2, P5	gained P2, P5

Rule of thumb. Memorise the handshake as "JoinGroup elects the leader and gathers subscriptions; SyncGroup distributes the leader's assignment; the generation fences the past." If you can narrate that in an interview, every rebalance follow-up becomes trivial.

Worked example — generation fencing stops a zombie consumer

Detailed explanation. A consumer pauses for a long GC (or a network partition isolates it) past session.timeout.ms. The coordinator evicts it and rebalances to a new generation; its partitions move to another member. When the paused consumer wakes, it still believes it owns those partitions. Generation fencing is what stops it from double-processing and corrupting offsets. Walk through the failure and the fence.

• Symptom. A consumer resumes after a 60 s stall and tries to commit offsets for partitions it no longer owns.
• Root cause. The stall exceeded session.timeout.ms; the coordinator rebalanced without it.
• Fence. Its offset commit carries generation 8; the coordinator is on generation 9 → ILLEGAL_GENERATION; the client must rejoin.

Question. Show what the coordinator does when the zombie's stale-generation commit arrives, and why no double-processing results.

Input.

Actor	Generation it thinks it has	Coordinator's current generation	Outcome
Zombie C2 (woke from stall)	8	9	commit rejected
Live C3 (new owner of P4)	9	9	commit accepted

Code.

// Inside the poll loop, a commit after a long stall
try {
    consumer.commitSync();                       // offsets tagged with the member's generation
} catch (CommitFailedException e) {
    // Thrown when the coordinator reports ILLEGAL_GENERATION or
    // REBALANCE_IN_PROGRESS: this member was fenced out.
    log.warn("Fenced — lost partition ownership during a stall; rejoining", e);
    // Do NOT retry the commit. The records processed during the stall
    // must be treated as "maybe reprocessed by the new owner".
    // The client will automatically rejoin on the next poll().
}

// On the next poll(), the client re-sends JoinGroup and gets a
// fresh (possibly smaller) assignment in the current generation.
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));

Step-by-step explanation.

1. During the 60 s stall, C2 sent no heartbeats. After session.timeout.ms (45 s) the coordinator declared C2 dead, rebalanced to generation 9, and moved C2's partition P4 to C3.
2. When C2 wakes, its in-memory state still says "I own P4, generation 8." It attempts commitSync() for P4 with generation 8 stamped on the request.
3. The coordinator compares 8 against its current 9 and rejects with ILLEGAL_GENERATION. The client surfaces this as CommitFailedException. Crucially, the offset is not written — C3's view of P4 is untouched.
4. C2 must not retry the commit; retrying can't succeed and would only risk clobbering C3's progress if the fence didn't exist. It logs, drops its stale assignment, and rejoins on the next poll().
5. Because the commit was fenced, the durable offset for P4 reflects only C3's committed progress. C2's work during the stall is either redundant (C3 reprocessed the same records — at-least-once) or discarded — never silently committed over the live owner.

Output.

Event	Committed offset for P4	Who owns P4 now
Before stall	1000 (by C2, gen 8)	C2
Coordinator rebalances	1000 (unchanged)	C3 (gen 9)
C3 processes, commits	1200 (by C3, gen 9)	C3
Zombie C2 commits gen 8	rejected — stays 1200	C3

Rule of thumb. Treat CommitFailedException as "I was fenced; abandon this batch and rejoin," never as "retry the commit." The generation is doing its job — it made the zombie's stale ownership unrepresentable, which is exactly what protects offset integrity.

Worked example — the max.poll.interval.ms self-eviction trap

Detailed explanation. The most common accidental rebalance is not a crash — it is a processing loop that takes longer than max.poll.interval.ms between poll() calls. The heartbeat thread keeps beating (so session.timeout.ms is satisfied), but the coordinator still evicts a member that hasn't called poll() in time, because a live heartbeat with a stuck processing loop is worse than a dead one. Walk through the trap and the fix.

• Symptom. Consumers rebalance every few minutes with no deploys and no crashes; logs show "member ... leaving group because it exceeded max.poll.interval.ms."
• Root cause. A batch of 500 records where each record does a 2 s external call = 1000 s of processing, far past the 300 s default.
• Fix. Lower max.poll.records, raise max.poll.interval.ms, or move slow work off the poll thread.

Question. Diagnose the self-eviction and give the three levers that stop it.

Input.

Lever	Default	Effect
max.poll.records	500	fewer records per poll = shorter processing gap
max.poll.interval.ms	300000 (5 min)	more time allowed between polls
session.timeout.ms	45000	heartbeat liveness (separate thread)
Processing model	inline	move slow I/O to a worker pool

Code.

// BEFORE — a slow inline loop self-evicts under max.poll.interval.ms
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 300000);   // 5 min
while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) slowExternalCall(r);   // 2s each × 500 = 1000s  ✗
    consumer.commitSync();
}

// AFTER — bound the batch so the gap between polls stays well under the limit
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50);           // 50 × 2s = 100s
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 600000);   // 10 min headroom
while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) slowExternalCall(r);   // 50 × 2s = 100s  ✓
    consumer.commitSync();
}

Step-by-step explanation.

1. The heartbeat runs on a background thread and keeps session.timeout.ms satisfied, so the member looks alive. But liveness is not enough — Kafka also requires progress, measured by the interval between poll() calls.
2. With 500 records at 2 s each, the processing loop takes ~1000 s before the next poll(). That exceeds the 300 s max.poll.interval.ms, so the coordinator evicts the member and rebalances — even though heartbeats were flowing.
3. The primary fix is to shrink max.poll.records so each batch finishes well within the interval. 50 records × 2 s = 100 s, comfortably under a 300 s (or raised 600 s) limit.
4. Raising max.poll.interval.ms buys headroom but is a blunt instrument: it also delays detection of a genuinely stuck consumer. Prefer bounding the batch first, then raise the interval only for legitimately long batches.
5. The durable fix for very slow work is to decouple: poll() fast, hand records to a bounded worker pool, pause() partitions while the pool is saturated, and commit only offsets whose work has completed. This keeps the poll thread responsive regardless of downstream latency.

Output.

Config	Poll gap	Under limit?	Rebalances
500 records, 300 s limit	~1000 s	no	every few minutes
50 records, 300 s limit	~100 s	yes	none
50 records, 600 s limit	~100 s	yes (large margin)	none

Rule of thumb. When a group rebalances with no deploys and no crashes, suspect max.poll.interval.ms first. The fix is almost always "process fewer records per poll," not "raise the timeout" — bounding the batch keeps genuine-failure detection fast.

Senior interview question on the rebalance protocol

A senior interviewer might ask: "Draw the request sequence a consumer group runs when a new member joins — who sends JoinGroup, who computes the assignment, what SyncGroup carries, and what the generation id protects against. Then explain how a consumer that pauses for a minute is prevented from double-consuming when it wakes up."

Solution Using the JoinGroup/SyncGroup handshake with generation fencing and a rebalance listener

// A consumer that logs the handshake outcome and commits safely across rebalances
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

var listener = new ConsumerRebalanceListener() {
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
        // SyncGroup is about to hand these away — pin durable offsets first.
        if (!revoked.isEmpty()) consumer.commitSync(currentOffsets(revoked));
    }
    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> assigned) {
        // We are now the owner in the CURRENT generation for these.
        log.info("Assigned in new generation: {}", assigned);
    }
    @Override
    public void onPartitionsLost(Collection<TopicPartition> lost) {
        // Cooperative protocol: we were fenced (generation bumped without us).
        // Do NOT commit — those partitions already moved. Just drop state.
        clearInFlight(lost);
    }
};

consumer.subscribe(List.of("orders"), listener);

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) process(r);
    try {
        consumer.commitSync();               // stamped with our current generation
    } catch (CommitFailedException fenced) {
        // ILLEGAL_GENERATION / REBALANCE_IN_PROGRESS: rejoin on next poll.
        log.warn("Fenced during commit; rejoining");
    }
}

Step-by-step trace.

Phase	What happens	Generation
Steady state	C1/C2 own P0-P2 / P3-P5	7
C3 sends JoinGroup	coordinator marks rebalance; C1/C2 rejoin	7 → (preparing)
Leader elected	C1 gets full subscription map	preparing
Leader assigns	C1 computes 2-2-2 split	preparing
SyncGroup	assignment stored; each member gets its slice	8
Zombie commit (gen 7)	rejected with ILLEGAL_GENERATION	8

After the handshake completes, the group is on generation 8 with a balanced assignment; onPartitionsRevoked committed durable offsets before ownership moved, so the new owner resumes cleanly; and any request still tagged generation 7 is fenced, making double-consumption impossible.

Output:

Concern	Mechanism	Result
Who assigns	leader consumer via SyncGroup	client-side assignor, broker stays dumb
Clean handoff	onPartitionsRevoked commitSync	next owner starts from durable offset
Zombie defense	generation stamped on every request	stale-generation requests rejected
Cooperative loss	onPartitionsLost (no commit)	dropped state, no clobber

Why this works — concept by concept:

• JoinGroup leader election — the coordinator picks one member to compute the assignment and forwards it every subscription, keeping assignment logic in the client so new assignors ship without a broker upgrade.
• SyncGroup distribution — the leader returns the whole assignment; the coordinator persists it and hands each member only its slice, which is the single source of truth for the new generation.
• Generation fencing — a monotonic id stamped on every request makes stale ownership unrepresentable; a fenced member gets ILLEGAL_GENERATION and must rejoin rather than double-consume.
• Rebalance listener commit-on-revoke — committing in onPartitionsRevoked pins the durable offset before ownership moves, so the next owner never replays committed work.
• Cost — two coordinator round-trips per rebalance plus one synchronous commit on revoke. The eliminated cost is silent double-processing and offset corruption. Net: O(members) messages per rebalance, and rebalances happen only on real membership change.

Streaming
Topic — streaming
Streaming rebalance and coordination problems

Practice →

Event processing Topic — event-processing Event-processing problems on partitioned streams

Practice →

---

3. Static membership and stable groups

group.instance.id turns a dynamic member into a static one — a restart within the session window reclaims its exact partitions with no rebalance

The mental model in one line: static membership is the feature where assigning each consumer a stable group.instance.id lets the coordinator remember that member across disconnects, so a member that leaves and rejoins within session.timeout.ms is handed back its exact previous assignment without triggering a rebalance at all — which is the single most effective cure for the rebalance storms that plague rolling restarts and autoscaling churn. A dynamic member is anonymous: every disconnect and reconnect is a stranger, so the group rebalances twice per pod on every deploy. A static member is recognised, so the deploy is invisible to the group.

Dynamic vs static membership.

• Dynamic (default). A member has no persistent identity; the coordinator assigns it an ephemeral memberId on join. When it disconnects, that identity is gone. Reconnecting is a brand-new member → rebalance on leave and rebalance on rejoin.
• Static (KIP-345). Setting group.instance.id gives the member a durable identity that survives disconnects. The coordinator keeps the member's slot reserved for session.timeout.ms; a rejoin with the same id reclaims the slot and the prior assignment with no rebalance.
• The graceful-shutdown twist. A static member that shuts down does not send LeaveGroup by default (it wants to keep its slot). So the coordinator holds the assignment until the session timeout — deliberately trading a brief unavailability for zero rebalances during a fast restart.

How static membership skips the rebalance.

• On leave. The static member disconnects (deploy, restart) but its slot is retained. No rebalance — the partitions are simply not being consumed for a moment.
• On rejoin within the window. The member reconnects with the same group.instance.id before session.timeout.ms expires. The coordinator recognises it, returns its exact prior assignment, and no JoinGroup/SyncGroup reassignment runs.
• On rejoin after the window. If the restart takes longer than session.timeout.ms, the coordinator has already evicted the slot and rebalanced. The rejoin is then a normal new member → rebalance. So the session timeout must exceed the worst-case restart time.

The tuning triangle — session, heartbeat, poll.

• session.timeout.ms. Must be long enough to cover a rolling restart (pod termination + scheduling + JVM warmup) but short enough to detect a genuine crash promptly. Common static-membership value: 45–120 s (vs a 10–45 s dynamic value). Bounded above by broker group.max.session.timeout.ms.
• heartbeat.interval.ms. Typically one-third of the session timeout. It only governs liveness signalling; it does not affect static reclaim.
• max.poll.interval.ms. Independent of static membership — still evicts a member whose processing loop stalls. Static membership fixes restart churn, not stuck-processing churn.

Common interview probes on static membership.

• "What does group.instance.id change?" — required answer: durable identity → restart within the session window skips the rebalance.
• "What's the trade-off?" — brief unavailability of the member's partitions during the restart window (they're reserved, not reassigned).
• "How do you size session.timeout.ms?" — longer than worst-case restart, shorter than acceptable crash-detection lag.
• "Does static membership stop max.poll.interval.ms rebalances?" — no; that's a separate, processing-side trigger.

Worked example — static vs dynamic on a rolling restart

Detailed explanation. A group of 3 consumers on a 12-partition topic gets a rolling deploy: pods restart one at a time. With dynamic membership the group rebalances 6 times (leave + rejoin per pod); with static membership it rebalances 0 times as long as each restart finishes within the session window. Walk through both.

• Deployment. 3 pods, restarted sequentially, ~20 s each.
• Dynamic. Each pod's leave and rejoin each trigger a rebalance → 6 rebalances.
• Static. group.instance.id per pod + session.timeout.ms=45000 → 0 rebalances.

Question. Count the rebalances and the partition movement for a 3-pod rolling restart under each membership model.

Input.

Model	group.instance.id	session.timeout.ms	Rebalances per deploy
Dynamic	unset	45000	6 (2 per pod)
Static	set per pod	45000	0 (if restart < 45 s)

Code.

// Static membership — one stable id per deployment slot.
// In Kubernetes, derive it from the StatefulSet ordinal:
//   group.instance.id = "order-enrichment-" + HOSTNAME.split("-").last()
String podOrdinal = System.getenv("POD_NAME").replaceAll(".*-", "");   // "0","1","2"
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "order-enrichment-" + podOrdinal);
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "45000");           // > worst-case restart
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "15000");        // ~ session/3
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-enrichment");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

Rolling restart of 3 pods (topic orders, 12 partitions)
=======================================================

DYNAMIC membership:
  pod-0 stops   -> rebalance (P0-P3 reassigned to pod-1, pod-2)
  pod-0 starts  -> rebalance (P0-P3 handed back... maybe)
  pod-1 stops   -> rebalance
  pod-1 starts  -> rebalance
  pod-2 stops   -> rebalance
  pod-2 starts  -> rebalance
  => 6 rebalances, up to 12 partitions shuffled repeatedly

STATIC membership (restart < session.timeout.ms):
  pod-0 stops   -> slot reserved (no rebalance); P0-P3 idle ~20s
  pod-0 starts  -> reclaims P0-P3 (no rebalance)
  pod-1 stops   -> slot reserved; P4-P7 idle ~20s
  pod-1 starts  -> reclaims P4-P7 (no rebalance)
  pod-2 stops   -> slot reserved; P8-P11 idle ~20s
  pod-2 starts  -> reclaims P8-P11 (no rebalance)
  => 0 rebalances, brief per-pod idle instead of churn

Step-by-step explanation.

1. Under dynamic membership each pod is anonymous. Stopping pod-0 looks like a permanent departure, so the coordinator rebalances P0-P3 onto the survivors. Starting pod-0 is a new member, so the coordinator rebalances again.
2. Six rebalances per deploy means six stop-the-world stalls (with the eager assignor) or six incremental reassignments (with cooperative). Either way, partitions bounce between owners repeatedly, cache/state is discarded, and consumer lag sawtooths.
3. Under static membership, group.instance.id makes pod-0 recognisable. When it stops, the coordinator reserves its slot for session.timeout.ms rather than reassigning P0-P3. The partitions are briefly not consumed, but nobody else is disrupted.
4. When pod-0 restarts (within 45 s) with the same id, the coordinator hands back its exact prior partitions with no JoinGroup/SyncGroup reassignment. The other pods never even observe the event.
5. The net trade is "~20 s of idle on 4 partitions per pod" instead of "6 rebalances that disrupt all 12 partitions." For most workloads a brief, isolated idle beats repeated global churn — especially for stateful consumers (Kafka Streams) whose local state would otherwise be rebuilt on every move.

Output.

Metric	Dynamic	Static
Rebalances per deploy	6	0
Partitions disturbed	up to 12, repeatedly	only the restarting pod's 4, briefly
State store rebuilds (Streams)	many	none
Worst-case per-pod unavailability	rebalance stall	~restart duration (idle)

Rule of thumb. For any deploy-frequently or autoscaling consumer, set a stable group.instance.id and a session.timeout.ms comfortably above the worst-case restart time. You trade a short, isolated idle for the elimination of deploy-time rebalance storms.

Worked example — sizing session.timeout.ms against restart time

Detailed explanation. Static membership only skips the rebalance if the restart finishes inside the session window. Size the window too small and you get the worst of both worlds — an idle slot and an eventual rebalance. Size it too large and genuine crashes take too long to detect. Walk through choosing the value for a Kubernetes StatefulSet.

• Restart budget. pod terminationGracePeriod (10 s) + scheduling (5 s) + JVM + client warmup (15 s) ≈ 30 s worst case.
• Session timeout. Must exceed 30 s with margin → 45–60 s.
• Crash detection. A real crash is now detected in ≤ 60 s (acceptable for most SLOs); tighten only if faster detection is required.

Question. Pick session.timeout.ms, heartbeat.interval.ms, and the broker ceiling, and state the failure mode of each wrong choice.

Input.

Parameter	Value	Constraint
Worst-case restart	30 s	measured from rollout
session.timeout.ms	60000	> restart, < crash-detection SLO
heartbeat.interval.ms	20000	≈ session/3
group.max.session.timeout.ms (broker)	300000	client value must be ≤ this

Code.

# Client — sized so a 30 s restart stays inside the window
group.instance.id        = order-enrichment-0
session.timeout.ms       = 60000        # 2× the 30 s restart budget
heartbeat.interval.ms    = 20000        # session/3
max.poll.interval.ms     = 300000       # unrelated to static reclaim; keep sane

# Broker — must allow the client's session timeout
# (server.properties)
# group.min.session.timeout.ms = 6000
# group.max.session.timeout.ms = 300000   # client 60000 is within range

Step-by-step explanation.

1. Measure the actual worst-case restart from real rollouts, not the happy path. Termination grace, image pull on a cold node, JVM JIT warmup, and first-poll metadata fetch all count — the clock starts when the old process dies and stops when the new one heartbeats.
2. Set session.timeout.ms to roughly 2× that budget (60 s here). The 2× margin absorbs a slow node or a GC pause during startup without falling out of the window.
3. Set heartbeat.interval.ms to about one-third of the session timeout so the coordinator sees multiple heartbeats per window; a single missed heartbeat should not risk eviction.
4. Confirm the client value is within [group.min.session.timeout.ms, group.max.session.timeout.ms] on the broker — a client value above the broker ceiling is rejected at join with INVALID_SESSION_TIMEOUT.
5. The failure modes: too small a session timeout → the restart overruns the window, the slot is evicted, and you rebalance anyway (idle and churn). Too large → a genuinely crashed pod's partitions sit unconsumed until the long timeout expires, inflating lag.

Output.

session.timeout.ms	30 s restart fits?	Crash detection	Verdict
15000	no (overruns)	15 s	rebalances on every deploy — too small
60000	yes (2× margin)	60 s	recommended
250000	yes	~4 min	crash lag too high — too large

Rule of thumb. Set session.timeout.ms to about twice the measured worst-case restart, heartbeat.interval.ms to a third of that, and verify both against the broker's min/max ceilings. The window must comfortably contain the restart, or static membership silently degrades to dynamic.

Worked example — combining static membership with cooperative sticky

Detailed explanation. Static membership and the cooperative sticky assignor solve different halves of the churn problem and compose cleanly. Static membership removes restart rebalances; cooperative sticky makes the rebalances that do happen (real scale-out, genuine crash) cheap and incremental. Together they turn a noisy group into a quiet one. Walk through the combined behaviour on a genuine scale-out.

• Baseline. 3 static members, cooperative-sticky, steady state.
• Event. Operator adds a 4th member (real scale-out, not a restart).
• Result. One incremental rebalance moves ~3 partitions; the 3 existing members keep the rest flowing.

Question. Describe what happens on a real scale-out when both static membership and cooperative sticky are configured.

Input.

Config	Value
group.instance.id	per-pod (static)
partition.assignment.strategy	CooperativeStickyAssignor
Existing members	C1, C2, C3 (4 partitions each)
New member	C4

Code.

// Both features together — the recommended production baseline
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "order-enrichment-" + podOrdinal);
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "60000");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

// Scale-out: C4 starts with a fresh group.instance.id "order-enrichment-3"
// -> one cooperative rebalance; C1/C2/C3 each give up ~1 partition to C4

Static + cooperative-sticky on a genuine scale-out (12 partitions)
==================================================================

Before (gen 12):  C1:[P0-P3]  C2:[P4-P7]  C3:[P8-P11]

C4 joins (new static id "order-enrichment-3"):
  Phase 1 (revoke only movers): C1 revokes P3, C2 revokes P7, C3 revokes P11
  Phase 2 (assign): C4 <- [P3, P7, P11]
  C1/C2/C3 KEEP P0-P2 / P4-P6 / P8-P10 flowing the whole time

After (gen 13):   C1:[P0-P2] C2:[P4-P6] C3:[P8-P10] C4:[P3,P7,P11]
  Partitions moved: 3 of 12   Stop-the-world: none

Step-by-step explanation.

1. On a deploy (restart), static membership means no rebalance at all — C4's absence here is a real new member, so a rebalance is correct and necessary.
2. The cooperative sticky assignor computes the target 3-3-3-3 balance and notices that only 3 partitions need to change owners. It revokes only those 3 (P3, P7, P11) in phase 1.
3. During phase 1, C1/C2/C3 keep consuming the 9 partitions they retain. There is no stop-the-world pause — the group processes at near-full throughput throughout.
4. In phase 2, the 3 revoked partitions are assigned to C4. C4 begins consuming them from their committed offsets. The rebalance completes in a single incremental cycle.
5. The combination is the production sweet spot: static membership makes deploys free, and cooperative sticky makes real membership changes cheap. Neither alone is sufficient — static membership still allows eager stop-the-world on genuine scale-out unless you also choose cooperative sticky.

Output.

Scenario	With static only	With static + cooperative sticky
Rolling restart	0 rebalances	0 rebalances
Genuine scale-out	1 eager (stop-the-world) rebalance	1 incremental (no stall) rebalance
Partitions moved on scale-out	up to all 12 (eager)	~3
Throughput during scale-out	drops to 0 briefly	near-full throughout

Rule of thumb. Ship static membership and the cooperative sticky assignor together. Static membership eliminates restart rebalances; cooperative sticky makes the unavoidable rebalances incremental. The pair is the modern default for any nontrivial consumer group.

Senior interview question on static membership

A senior interviewer might ask: "Your consumer group rebalances on every rolling deploy, and each rebalance forces your Kafka Streams state stores to rebuild, adding minutes of lag. Walk me through static membership — what group.instance.id changes, how to size session.timeout.ms, the trade-off you're accepting, and how you'd combine it with the assignor to also make genuine scale-outs cheap."

Solution Using static membership with a tuned session window and cooperative sticky

// Production consumer: static identity + tuned window + incremental rebalance
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-enrichment");

// Static identity derived from the StatefulSet ordinal (stable across restarts)
String ordinal = System.getenv("POD_NAME").replaceAll(".*-", "");
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "order-enrichment-" + ordinal);

// Session window sized to 2× the measured 30 s worst-case restart
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "60000");
props.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, "20000");   // session / 3

// Incremental rebalance for the genuine scale-outs
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

// Explicit at-least-once
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"), new CommitOnRevokeListener(consumer));

Step-by-step trace.

Event	Static behaviour	Assignor behaviour	Rebalances
Rolling restart (< 60 s)	slot reserved, reclaimed	not invoked	0
Restart overrun (> 60 s)	slot evicted	cooperative reassign	1 (incremental)
Genuine scale-out (+1 pod)	new static id	revoke only movers	1 (incremental)
Genuine crash	detected at 60 s	cooperative reassign	1 (incremental)
Slow poll loop	unaffected	evicted on max.poll.interval	1 (fix the loop)

After deployment, rolling restarts produce zero rebalances (each pod reclaims its slot inside the 60 s window); the only rebalances that occur are for genuine membership changes, and those are incremental — the surviving members keep their partitions flowing while only the movers change hands.

Output:

Metric	Before (dynamic + eager)	After (static + cooperative)
Rebalances per deploy	2 × pods	0
State-store rebuilds per deploy	many	0
Scale-out throughput dip	to zero (stop-the-world)	negligible
Crash detection	~session timeout	~60 s
Delivery guarantee	implicit	explicit at-least-once

Why this works — concept by concept:

• Stable group.instance.id — a durable identity lets the coordinator reserve the member's slot across a disconnect, so a fast restart reclaims the exact prior assignment with no reassignment.
• Session window sized to 2× restart — the reclaim only happens inside session.timeout.ms; sizing it to twice the measured restart absorbs slow nodes and GC pauses without falling out of the window.
• Heartbeat at one-third of session — multiple heartbeats per window mean a single missed beat never risks eviction, keeping liveness detection robust.
• Cooperative sticky for genuine changes — static membership does not help real scale-outs or crashes; the incremental assignor makes those cheap by moving only the partitions that must move.
• Cost — a slightly longer crash-detection window (seconds) and a brief per-pod idle during restart. The eliminated cost is O(pods) rebalances and state rebuilds per deploy. Net: rebalances scale with real membership change, not with deploy frequency.

Streaming
Topic — streaming
Streaming stability and static-membership problems

Practice →

Design Topic — design Design problems on resilient deploy pipelines

Practice →

---

4. Cooperative sticky assignor and incremental rebalance

partition.assignment.strategy picks how the leader maps partitions — cooperative sticky revokes only the partitions that move, in a two-phase incremental rebalance

The mental model in one line: the partition assignment strategy is a pluggable, client-side algorithm the leader runs to map partitions to members, and the cooperative sticky assignor is the modern choice because it combines stickiness (keep each partition with the member that already had it) with an incremental rebalance protocol that revokes only the partitions changing owners — eliminating the stop-the-world revoke-everything pause of the classic eager assignors. The assignor is not just about balance; it is about how much churn rebalancing causes, and cooperative sticky minimises both the imbalance and the movement.

The four built-in assignors.

• Range (default legacy). For each topic independently, sort partitions and members, and assign contiguous ranges. Simple, but co-locates the same partition numbers of different topics on one member and rebalances eagerly. Uneven when partitions do not divide evenly.
• Round-robin. Lay all partitions of all subscribed topics in one list and deal them round-robin across members. Better balance than range across multiple topics, still eager.
• Sticky. Round-robin-balanced, but on rebalance it preserves as many existing assignments as possible to minimise movement. Still uses the eager protocol (revoke all, then reassign) even though the final assignment barely changes.
• Cooperative sticky. Sticky balance plus the cooperative (incremental) protocol: revoke only the partitions that actually change owners, and do it in two phases so retained partitions never stop.

Eager vs cooperative — the protocol difference.

• Eager protocol. At the start of every rebalance, every member revokes all its partitions (onPartitionsRevoked for everything), then the leader computes a fresh assignment, then everyone gets their new partitions. Between revoke and assign, the whole group processes nothing — stop-the-world.
• Cooperative protocol. Rebalance runs in two rounds. Round 1: the leader computes the target assignment and each member revokes only the partitions it must give up; it keeps the rest. Round 2: a follow-up rebalance assigns the freed partitions to their new owners. No member ever revokes a partition it gets to keep.
• Why two rounds. A partition must be fully revoked by its old owner before its new owner can safely take it (exactly-once ownership). The cooperative protocol splits "revoke the movers" and "assign the movers" into two rebalances so retained partitions keep flowing throughout.

Stickiness — why minimal movement matters.

• Stateless consumers. Moving a partition means re-seeking to its committed offset — cheap, but any in-flight batch is reprocessed. Less movement = less reprocessing.
• Stateful consumers (Kafka Streams). Each partition may back a local state store (a RocksDB instance). Moving a partition means rebuilding that store from the changelog topic — potentially minutes of restore. Stickiness is the difference between a cheap and an expensive rebalance.
• Cache locality. Consumers often cache per-partition context (dedup sets, enrichment lookups). Keeping partitions put preserves that cache.

Migrating from eager to cooperative — the one-time two-step upgrade.

• Why it's careful. All members must agree on the protocol. You cannot flip half the group to cooperative while the other half is eager.
• The two-step rolling upgrade. Deploy once with partition.assignment.strategy listing both the old assignor and CooperativeStickyAssignor (old one first). Once every member is on that build, deploy again with only CooperativeStickyAssignor. The dual-list intermediate step lets the group negotiate a common protocol during the transition.

Common interview probes on assignors.

• "What's the difference between sticky and cooperative sticky?" — required answer: both minimise movement; cooperative also uses the incremental protocol (no stop-the-world).
• "Why is range the default and why is it not great?" — historical default; uneven balance and eager revocation.
• "How do you migrate to cooperative without downtime?" — the two-step dual-assignor rolling upgrade.
• "What does the leader do that the broker doesn't?" — runs the assignor algorithm.

Worked example — range vs cooperative sticky on the same scale-out

Detailed explanation. The clearest way to feel the difference is to run the same scale-out under range (eager) and cooperative sticky. Range revokes everything and reshuffles; cooperative sticky moves only what must move and never stops the retained partitions. Walk through both on orders (6 partitions) as C3 joins C1/C2.

• Before. C1 owns P0-P2, C2 owns P3-P5.
• Range (eager). Both revoke all, leader recomputes 2-2-2, everyone reassigned — stop-the-world.
• Cooperative sticky. Only P2 and P5 move; C1/C2 keep the rest flowing.

Question. Compare the revocation set, the partition movement, and the stall for range vs cooperative sticky when C3 joins.

Input.

Assignor	Protocol	Revoked on rebalance	Movement
RangeAssignor	eager	all 6 partitions	up to 6 reassigned
CooperativeStickyAssignor	incremental	only P2, P5	2 moved

Code.

Same scale-out, two assignors (orders: P0..P5; C3 joins C1,C2)
==============================================================

RANGE (eager):
  Round 1: C1 revokes [P0,P1,P2], C2 revokes [P3,P4,P5]   <-- all revoked
           (group processes NOTHING here — stop-the-world)
  Leader recomputes: C1->[P0,P1] C2->[P2,P3] C3->[P4,P5]
  Round 1 assign: everyone gets partitions back
  Net: 6 partitions revoked; P2,P3,P4,P5 changed hands

COOPERATIVE STICKY (incremental):
  Phase 1: leader computes target C1->[P0,P1] C2->[P3,P4] C3->[P2,P5]
           C1 revokes [P2] only; C2 revokes [P5] only
           (C1 keeps P0,P1 flowing; C2 keeps P3,P4 flowing)
  Phase 2: assign freed partitions: C3 <- [P2, P5]
  Net: 2 partitions revoked; only P2,P5 changed hands; no stall

Step-by-step explanation.

1. Under range (eager), the protocol requires every member to revoke all partitions before reassignment — even partitions that will end up back with the same owner. During the revoke-to-assign gap the group consumes nothing.
2. The leader then computes a fresh range assignment. Range assigns per topic, so with 6 partitions and 3 members it lands 2-2-2, but 4 of the 6 partitions changed hands relative to before — needless movement.
3. Under cooperative sticky, the leader first computes the target assignment and diffs it against the current one. Only P2 (leaving C1) and P5 (leaving C2) need to move; those are the only revocations in phase 1.
4. Crucially, C1 keeps P0/P1 and C2 keeps P3/P4 throughout — they never revoke partitions they retain, so those partitions keep being consumed with no pause.
5. Phase 2 is a quick follow-up rebalance that hands P2 and P5 to C3. The total disruption is 2 partitions briefly paused instead of 6 partitions stopped — and for a stateful app, that is 2 state-store restores instead of 6.

Output.

Metric	Range (eager)	Cooperative sticky
Partitions revoked	6	2
Partitions that changed owner	4	2
Stop-the-world pause	yes	no
State restores (Streams)	up to 4	2

Rule of thumb. Prefer cooperative sticky for any group where a rebalance is more than free — which is nearly all of them. Range is acceptable only for tiny, stateless, rarely-rebalancing groups; everywhere else the incremental protocol pays for itself the first time you scale out.

Worked example — the two-phase revoke in a rebalance listener

Detailed explanation. With the cooperative protocol, your ConsumerRebalanceListener sees a different pattern than under eager: onPartitionsRevoked receives only the partitions actually moving (often empty), and a new callback onPartitionsLost handles the fenced case. Writing the listener correctly is what makes the incremental protocol safe. Walk through the callbacks.

• onPartitionsRevoked. Called with only the partitions this member is giving up (a subset, often small or empty). Commit their offsets here.
• onPartitionsAssigned. Called with only the newly added partitions (not the full set). Initialise state for these.
• onPartitionsLost. Called when partitions were taken without a clean revoke (fencing). Do not commit — they already moved.

Question. Write a cooperative-safe rebalance listener that commits only revoked partitions and never clobbers a lost one.

Input.

Callback	Cooperative payload	Correct action
onPartitionsRevoked	only the movers	commitSync for those offsets
onPartitionsAssigned	only the new arrivals	seed per-partition state
onPartitionsLost	fenced partitions	drop state, no commit

Code.

class CooperativeListener implements ConsumerRebalanceListener {
    private final KafkaConsumer<String, String> consumer;
    private final Map<TopicPartition, OffsetAndMetadata> pending;   // per-partition progress

    CooperativeListener(KafkaConsumer<String, String> c, Map<TopicPartition, OffsetAndMetadata> p) {
        this.consumer = c; this.pending = p;
    }

    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
        // Cooperative: 'revoked' is ONLY the partitions moving away.
        Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();
        for (TopicPartition tp : revoked) {
            if (pending.containsKey(tp)) toCommit.put(tp, pending.remove(tp));
        }
        if (!toCommit.isEmpty()) consumer.commitSync(toCommit);   // durable handoff
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> assigned) {
        // Cooperative: 'assigned' is ONLY the newly added partitions.
        for (TopicPartition tp : assigned) seedState(tp);
    }

    @Override
    public void onPartitionsLost(Collection<TopicPartition> lost) {
        // Fenced: these already belong to someone else. Never commit.
        for (TopicPartition tp : lost) pending.remove(tp);
    }
}

Step-by-step explanation.

1. Under the cooperative protocol, onPartitionsRevoked no longer means "I'm giving up everything." It means "here are the specific partitions moving away." Committing offsets for only those partitions pins a durable handoff point without touching retained ones.
2. onPartitionsAssigned similarly carries only the new partitions. This is a behavioural change from eager (where it carried the entire assignment); seeding state for only the arrivals avoids redundant re-initialisation of partitions you already held.
3. onPartitionsLost is the fenced case — a member discovered its partitions were reassigned without a clean revoke (e.g. after a long stall). Because someone else already owns them, committing would either fail (fencing) or, worse, race the new owner. The correct action is to discard local state silently.
4. Keeping a pending map of per-partition offsets lets the listener commit exactly the moving partitions. This is more precise than a blanket commitSync(), which would also commit partitions you keep — harmless but unnecessary.
5. The net effect: retained partitions never pause, moving partitions hand off durably, and fenced partitions never corrupt the new owner. That is the whole safety contract of the incremental protocol, expressed in three callbacks.

Output.

Callback firing	Partitions in payload	Commit?
onPartitionsRevoked (scale-out)	just the 1-2 movers	yes, those only
onPartitionsAssigned (scale-out)	just the new arrivals	n/a (seed state)
onPartitionsLost (after stall)	fenced partitions	no

Rule of thumb. When you switch to cooperative sticky, audit your rebalance listener: onPartitionsRevoked and onPartitionsAssigned now carry deltas, not the full set, and you must implement onPartitionsLost to drop state without committing. A listener written for the eager protocol will over-commit and mishandle fencing.

Worked example — the two-step migration from eager to cooperative

Detailed explanation. You cannot flip a running group from an eager assignor to cooperative sticky in one deploy — the members must negotiate a common protocol, and a half-eager/half-cooperative group is invalid. The safe path is a two-step rolling upgrade using a dual-assignor list as the bridge. Walk through both deploys.

• Deploy 1. Set partition.assignment.strategy to a list: the current assignor first, then CooperativeStickyAssignor. Roll it out to every member.
• Deploy 2. Once all members carry the dual list, set the strategy to only CooperativeStickyAssignor. Roll it out.
• Why two. During deploy 1 the group keeps using the old (eager) protocol because it's the common denominator; only after every member supports cooperative can deploy 2 switch the protocol.

Question. Give the two consumer configs and explain what the group uses during each phase.

Input.

Phase	partition.assignment.strategy	Protocol in effect
Start	RangeAssignor	eager
Deploy 1	Range, CooperativeSticky	eager (common denominator)
Deploy 2	CooperativeSticky	cooperative

Code.

// ---- Deploy 1: dual list, old assignor FIRST ----
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    "org.apache.kafka.clients.consumer.RangeAssignor," +
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
// Roll this to ALL members. The group still rebalances EAGERLY,
// because Range is the highest strategy every member supports.

// ---- Deploy 2: cooperative only ----
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
// Now every member supports cooperative; the group upgrades the
// protocol to incremental on the next rebalance.

Migration timeline
==================
  t0: all members on Range (eager)
  t1: rolling Deploy 1 (Range, CooperativeSticky) — mixed builds coexist
      -> group selects Range (only common strategy) -> still eager, still SAFE
  t2: all members on the dual list -> still eager
  t3: rolling Deploy 2 (CooperativeSticky only)
      -> as members update, cooperative becomes the common strategy
      -> next rebalance runs the incremental protocol
  t4: all members cooperative -> incremental rebalances from here on

Step-by-step explanation.

1. The assignor list is a preference order: the group picks the highest-priority strategy that every member supports. Listing Range first in deploy 1 guarantees the mixed-build group keeps using the eager protocol, which both old and new builds understand.
2. During deploy 1, old members (Range only) and new members (Range + Cooperative) coexist. Their common denominator is Range, so the group rebalances eagerly — exactly as before. No behaviour change, no risk.
3. Only after deploy 1 has reached every member does the group universally support cooperative. That is the precondition for switching the protocol.
4. Deploy 2 drops Range from the list. As each member updates, cooperative becomes the common strategy; on the next rebalance the group upgrades to the incremental protocol.
5. Skipping the dual-list step — going straight from Range to Cooperative — risks a window where some members expect eager and others expect cooperative, which Kafka rejects. The two-step bridge is the only safe path.

Output.

Timeline point	Members' config	Protocol used
t0	Range	eager
t1-t2	Range, Cooperative (rolling)	eager (safe)
t3	Cooperative (rolling)	upgrading
t4	Cooperative	incremental

Rule of thumb. Migrate to cooperative sticky in two deploys: first add CooperativeStickyAssignor after the current assignor in the list and roll it everywhere, then remove the old assignor and roll again. Never jump directly — the dual-list intermediate is what keeps the mixed-version group valid.

Senior interview question on partition assignment

A senior interviewer might ask: "Your Kafka Streams app rebuilds its state stores on every scale event because you're on the range assignor. Explain the four assignors, why cooperative sticky minimises both imbalance and movement, how the two-phase incremental rebalance keeps retained partitions flowing, and the exact rolling-upgrade steps to switch without an invalid mixed-protocol group."

Solution Using the cooperative sticky assignor with a delta-aware listener and a two-step rollout

// Target state after migration — cooperative sticky, delta-aware listener
Properties props = new Properties();
props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-enrichment");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
          "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, "order-enrichment-" + ordinal);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);

var pending = new HashMap<TopicPartition, OffsetAndMetadata>();
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(List.of("orders"), new CooperativeListener(consumer, pending));

while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) {
        process(r);
        pending.put(new TopicPartition(r.topic(), r.partition()),
                    new OffsetAndMetadata(r.offset() + 1));
    }
    consumer.commitAsync(pending, null);     // retained partitions never pause
}

Step-by-step trace.

Phase	Assignor action	Member experience
Deploy 1 (dual list)	Range remains common strategy	eager, unchanged, safe
Deploy 2 (cooperative only)	cooperative becomes common	protocol upgrades on next rebalance
Scale-out	revoke only movers (phase 1)	retained partitions keep flowing
Assign movers (phase 2)	new member gets freed partitions	one incremental cycle
Fenced after stall	onPartitionsLost	drop state, no commit

After the two-step rollout, the group runs the incremental protocol: a scale-out revokes only the partitions that change owners, the surviving members never pause on their retained partitions, and Kafka Streams rebuilds only the state stores for the few partitions that actually moved.

Output:

Metric	Range (before)	Cooperative sticky (after)
Partitions moved per scale event	up to all	only the delta (~1/N)
Stop-the-world pause	yes	none
State-store restores per scale event	many	few
Migration risk	n/a	zero (dual-list bridge)
Rebalance throughput dip	to zero	negligible

Why this works — concept by concept:

• Sticky assignment — the leader diffs the target against the current assignment and preserves as many existing owner-partition pairs as possible, minimising both movement and state rebuilds.
• Incremental cooperative protocol — splitting the rebalance into revoke-the-movers and assign-the-movers means retained partitions never stop, replacing the eager stop-the-world pause with a background reassignment.
• Delta-aware rebalance listener — onPartitionsRevoked/onPartitionsAssigned carry only the changed partitions and onPartitionsLost handles fencing, so commits are precise and no live owner is clobbered.
• Two-step dual-list rollout — listing the old assignor first keeps a mixed-version group on the eager protocol until every member supports cooperative, then the second deploy flips the protocol safely.
• Cost — a two-phase rebalance costs one extra coordinator round-trip per scale event and a slightly more complex listener. The eliminated cost is O(partitions) stop-the-world pauses and state rebuilds. Net: rebalance work is proportional to the partitions that actually move, not the whole group.

Streaming
Topic — streaming
Streaming partition-assignment and sticky problems

Practice →

Optimization Topic — optimization Optimization problems on rebalance and throughput

Practice →

---

5. Offset commit semantics and delivery guarantees

__consumer_offsets stores the committed position — and when you commit relative to processing and to the rebalance sets at-least-once, at-most-once, or exactly-once

The mental model in one line: an offset commit records, in the internal compacted __consumer_offsets topic, the position up to which a group has durably processed each partition, and the timing of that commit — before or after you process a record, and before or after a rebalance revokes the partition — is the entire mechanism that decides whether your pipeline is at-least-once, at-most-once, or exactly-once. The committed offset is not the same as your current read position; the gap between them is your reprocessing (or loss) window on a crash.

Committed offset vs current position.

• Current position. The in-memory offset of the next record poll() will return for a partition. Advances as you consume; lost on crash.
• Committed offset. The durable offset stored in __consumer_offsets — where a new owner (after a rebalance or restart) resumes. Advances only when you commit.
• The gap. Records between the committed offset and the current position are "processed but not yet committed." On a crash they are reprocessed (at-least-once) — or, if you committed ahead of processing, they are lost (at-most-once).

Where offsets live — the __consumer_offsets topic.

• The topic. A compacted internal topic (default 50 partitions). Each commit appends a record keyed by (group.id, topic, partition) with the committed offset as the value; compaction keeps only the latest per key.
• Why compacted. Only the newest committed offset per (group, topic, partition) matters, so log compaction garbage-collects older commits and keeps the topic bounded.
• The coordinator connection. A group's offsets live on the partition of __consumer_offsets that its group.id hashes to — the same partition whose leader is the group coordinator. That is why one broker owns both membership and offsets for a group.

The three delivery guarantees, by commit timing.

• At-least-once (commit after processing). Process the record, then commit. A crash between processing and commit replays the uncommitted records — no loss, possible duplicates. Requires idempotent side effects. The most common production choice.
• At-most-once (commit before processing). Commit the offset, then process. A crash after commit but before processing loses those records — no duplicates, possible loss. Acceptable only for lossy telemetry.
• Exactly-once (transactional). The consumer's offset commit is written inside the same producer transaction as the output records, and downstream reads with isolation.level=read_committed. Either both the output and the offset commit are visible, or neither. This is what Kafka Streams processing.guarantee=exactly_once_v2 does.

Auto-commit vs manual commit.

• Auto-commit (enable.auto.commit=true). The client commits the current position every auto.commit.interval.ms (default 5 s) on the poll thread. Simple, but the commit is decoupled from your processing — a crash can replay up to 5 s, and a commit can fire for records you haven't finished processing.
• Manual sync (commitSync). Blocking commit; retries on retriable errors; throws on fencing. Use for a durable checkpoint (e.g. on revoke, or end of batch) where you need certainty.
• Manual async (commitAsync). Non-blocking; higher throughput; no automatic retry (a later commit supersedes a failed earlier one). Use for the hot path, with a final commitSync on shutdown/revoke as the safety net.

Commit interplay with rebalance.

• The risk. If a rebalance revokes a partition and you have not committed since the last auto-commit, the new owner resumes from a stale committed offset and replays everything since — a reprocessing spike on every rebalance.
• The fix. Commit synchronously in onPartitionsRevoked (eager) or for the revoked subset (cooperative) so the durable offset is current at the moment of handoff.

Common interview probes on offsets.

• "Committed offset vs position?" — required answer: durable resume point vs in-memory next-read; the gap is your crash window.
• "How do you get at-least-once?" — commit after processing, idempotent sinks.
• "How does exactly-once actually work?" — offset commit inside the producer transaction + read_committed.
• "Why does a rebalance sometimes replay data?" — uncommitted gap at handoff; commit on revoke.

Worked example — auto-commit replay vs manual commit-after-process

Detailed explanation. The default enable.auto.commit=true silently commits every 5 s, decoupled from your processing. That produces two surprises: a crash replays up to 5 s of records, and a rebalance can hand off a stale offset. Switching to manual commit-after-process makes the guarantee explicit and the window a single batch. Walk through both.

• Auto-commit. Commits current position every 5 s on the poll thread, regardless of processing progress.
• Manual after-process. Commit only after the batch's side effects have durably completed.

Question. Show the reprocessing window for a crash under auto-commit vs manual commit-after-process.

Input.

Model	Commit trigger	Crash window
Auto-commit (5 s)	timer on poll thread	up to 5 s of records
Manual after-process	end of processed batch	one batch

Code.

// BEFORE — auto-commit: the commit is a timer, not tied to your work
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, 5000);
while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) process(r);      // if we crash here, up to 5s replays
    // (no explicit commit — the client commits on its own timer)
}

// AFTER — manual commit after processing: window shrinks to one batch
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) process(r);      // durable side effects first
    consumer.commitSync();                 // then pin the offset => at-least-once
}

Step-by-step explanation.

1. Under auto-commit, the client commits the current position on a 5 s timer from the poll thread. That position may be ahead of records whose side effects have not durably landed — so auto-commit can commit work you have not finished.
2. A crash under auto-commit replays every record since the last timer commit — up to auto.commit.interval.ms (5 s). For a 10k-records/s partition that is 50k records reprocessed, which is only safe if process() is idempotent.
3. Manual commitSync() after the processing loop pins the offset to exactly the records whose side effects completed. The crash window shrinks to the current uncommitted batch (bounded by max.poll.records).
4. commitSync blocks and retries on retriable errors, giving a durable checkpoint; the trade is throughput. For hot paths, commitAsync in the loop with a commitSync on shutdown/revoke balances speed and safety.
5. The guarantee is now explicit: process, then commit, means at-least-once with a one-batch replay window. Nothing is committed that was not processed; nothing processed is lost.

Output.

Metric	Auto-commit	Manual after-process
Crash replay window	up to 5 s	one batch
Commit tied to processing?	no (timer)	yes
Duplicates on crash	up to 5 s of records	one batch
Requires idempotent sink	yes	yes (smaller blast radius)

Rule of thumb. Turn off auto-commit for anything with real side effects. Commit after processing for at-least-once; the replay window becomes one batch instead of a 5 s timer interval you don't control.

Worked example — commit-on-revoke to stop rebalance replay

Detailed explanation. Even with manual commit, a rebalance that fires between your commits hands the partition to a new owner at a stale offset — replaying everything since your last commit. Committing in the rebalance listener at the moment of revocation closes this gap. Walk through the replay bug and the fix.

• Symptom. Every rebalance triggers a spike of reprocessed records, proportional to how long since the last commit.
• Root cause. The revoked partition's committed offset lags the position; the new owner resumes from the lag point.
• Fix. Commit synchronously in onPartitionsRevoked (or the cooperative revoked subset) before the handoff.

Question. Add commit-on-revoke so a rebalance hands off a current offset and the new owner does not replay.

Input.

Without commit-on-revoke	With commit-on-revoke
new owner resumes at last periodic commit	new owner resumes at revoke-time offset
replay = records since last commit	replay = 0 (clean handoff)

Code.

class CommitOnRevokeListener implements ConsumerRebalanceListener {
    private final KafkaConsumer<String, String> consumer;
    private final Map<TopicPartition, OffsetAndMetadata> pending;

    CommitOnRevokeListener(KafkaConsumer<String, String> c,
                           Map<TopicPartition, OffsetAndMetadata> p) {
        this.consumer = c; this.pending = p;
    }

    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> revoked) {
        // Pin durable offsets for the partitions leaving us, right now.
        Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();
        for (TopicPartition tp : revoked)
            if (pending.containsKey(tp)) toCommit.put(tp, pending.get(tp));
        if (!toCommit.isEmpty()) consumer.commitSync(toCommit);
    }

    @Override public void onPartitionsAssigned(Collection<TopicPartition> a) { }
    @Override public void onPartitionsLost(Collection<TopicPartition> lost) {
        for (TopicPartition tp : lost) pending.remove(tp);   // already gone; no commit
    }
}

// Poll loop keeps 'pending' current so revoke has an accurate offset to commit
while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) {
        process(r);
        pending.put(new TopicPartition(r.topic(), r.partition()),
                    new OffsetAndMetadata(r.offset() + 1));
    }
    consumer.commitAsync(pending, null);
}

Step-by-step explanation.

1. The poll loop maintains a pending map of the next offset to commit per partition, updated as each record is processed. This gives the listener an accurate, per-partition offset to commit at revoke time.
2. Without commit-on-revoke, the durable offset only advances on the periodic commitAsync. A rebalance between two of those commits hands the partition off at the older offset, so the new owner replays everything processed since.
3. onPartitionsRevoked commits synchronously for exactly the revoked partitions, using the pending offsets. This blocks the rebalance briefly but guarantees the handoff offset reflects all completed work.
4. commitSync (not async) is essential here: the revoke callback runs on the poll thread during the rebalance, and the partition ownership is about to move — the commit must complete before the handoff, so it must block.
5. onPartitionsLost deliberately does not commit — those partitions were fenced and already belong to someone else; committing would fail or race. The net result is zero replay on a clean rebalance and no clobber on a fenced one.

Output.

Rebalance type	Without commit-on-revoke	With commit-on-revoke
Clean (revoke fires)	replay since last periodic commit	0 replay
Fenced (lost)	replay (unavoidable)	drop state, no double-commit
Handoff offset accuracy	stale	current

Rule of thumb. Always commit synchronously on revoke. Manual commit alone still replays on every rebalance if the periodic commit lags; committing at the moment of handoff is what makes a rebalance replay-free.

Worked example — exactly-once with transactional offset commits

Detailed explanation. True exactly-once across a consume-process-produce pipeline requires the offset commit to be atomic with the output write. Kafka does this by committing the consumer's offsets inside the producer's transaction via sendOffsetsToTransaction, and having downstream consumers read with isolation.level=read_committed. Walk through the transactional loop.

• Producer. Transactional (transactional.id set, enable.idempotence=true).
• Offset commit. Sent into the transaction, not via consumer.commitSync().
• Downstream. Reads read_committed so it never sees aborted output.

Question. Write the consume-transform-produce loop that makes the output and the offset commit atomic.

Input.

Component	Setting
Producer	transactional.id=enrich-1, idempotence on
Consumer	enable.auto.commit=false, isolation.level=read_committed
Offset commit	producer.sendOffsetsToTransaction(...)
Downstream	isolation.level=read_committed

Code.

producer.initTransactions();

while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    if (records.isEmpty()) continue;

    producer.beginTransaction();
    try {
        var offsets = new HashMap<TopicPartition, OffsetAndMetadata>();
        for (var r : records) {
            var out = transform(r);
            producer.send(new ProducerRecord<>("orders-enriched", out.key(), out.value()));
            offsets.put(new TopicPartition(r.topic(), r.partition()),
                        new OffsetAndMetadata(r.offset() + 1));
        }
        // Commit the CONSUMER offsets INSIDE the producer transaction:
        producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
        producer.commitTransaction();          // output + offsets commit atomically
    } catch (KafkaException e) {
        producer.abortTransaction();           // neither output nor offsets are visible
    }
}

Step-by-step explanation.

1. The producer is transactional: transactional.id gives it a stable identity so the broker can fence a zombie producer, and initTransactions() recovers any in-flight transaction from a previous incarnation.
2. Inside beginTransaction()/commitTransaction(), every send() to orders-enriched and the consumer's offset advance are part of one transaction.
3. sendOffsetsToTransaction(offsets, consumer.groupMetadata()) writes the consumer offsets into the transaction — not via consumer.commitSync(). This is the crux: the offset commit and the output records commit or abort together.
4. commitTransaction() atomically makes both the enriched output and the new committed offset visible. A crash before commit leaves the transaction open; on recovery it aborts, so neither the output nor the offset advance is visible — the input records will be reprocessed cleanly.
5. Downstream consumers set isolation.level=read_committed, so they never read records from an aborted transaction. End to end, each input record affects the output exactly once, even across crashes and rebalances.

Output.

Failure point	Output visible?	Offset advanced?	Net effect
Before commitTransaction	no (aborted)	no	input reprocessed cleanly
After commitTransaction	yes	yes	processed exactly once
Downstream read_committed	only committed	—	never sees aborted output

Rule of thumb. Exactly-once is not a commit flag — it is committing offsets inside the producer transaction with sendOffsetsToTransaction and reading downstream with read_committed. If the offset commit is separate from the output write, you have at-least-once at best.

Senior interview question on offset commit semantics

A senior interviewer might ask: "Your consumer sometimes reprocesses an hour of data after a crash and sometimes drops records; and every rebalance triggers a reprocessing spike. Explain committed offset vs position, where offsets are stored, how commit timing sets at-least-once vs at-most-once, why rebalances replay, and how you'd implement true exactly-once for a consume-transform-produce pipeline."

Solution Using manual commit-after-process, commit-on-revoke, and transactional exactly-once

// At-least-once baseline (idempotent sink): commit after process + on revoke
Properties c = new Properties();
c.put(ConsumerConfig.GROUP_ID_CONFIG, "order-enrichment");
c.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);          // explicit commits
c.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");  // ignore aborted upstream
c.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
      "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

var pending = new HashMap<TopicPartition, OffsetAndMetadata>();
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(c);
consumer.subscribe(List.of("orders"), new CommitOnRevokeListener(consumer, pending));

while (running) {
    var records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) {
        process(r);                                              // durable, idempotent
        pending.put(new TopicPartition(r.topic(), r.partition()),
                    new OffsetAndMetadata(r.offset() + 1));
    }
    consumer.commitSync(pending);                                // after process => at-least-once
}
// For exactly-once, replace process()+commitSync with the transactional
// beginTransaction / sendOffsetsToTransaction / commitTransaction loop.

Step-by-step trace.

Concern	Mechanism	Result
Crash replay	commit after process	window = one batch
Rebalance replay	commitSync on revoke	0 replay on clean handoff
Aborted upstream	read_committed	never consumes aborted records
Exactly-once	offsets in producer txn	output + offset atomic
Storage	__consumer_offsets (compacted)	latest offset per (group, topic, partition)

After the change, a crash replays only the current uncommitted batch, a rebalance hands off a current offset with no reprocessing spike, and the optional transactional variant makes the enriched output and the offset commit atomic for true exactly-once.

Output:

Metric	Before	After
Crash replay window	up to hours	one batch (or 0 with EOS)
Rebalance replay	spike every rebalance	0 (clean handoff)
Dropped records	possible	none
Delivery guarantee	accidental	at-least-once, or exactly-once
Offset durability	__consumer_offsets	__consumer_offsets, current

Why this works — concept by concept:

• Committed offset in the offsets topic — the durable resume point per (group, topic, partition) lives in the compacted internal offsets topic; a new owner reads it, so keeping it current bounds every replay window.
• Commit after process — pinning the offset only after side effects land makes the guarantee at-least-once with a one-batch window, instead of an uncontrolled auto-commit timer.
• Commit-on-revoke — a synchronous commit at handoff closes the rebalance replay gap so the next owner never reprocesses committed work.
• Transactional offsets plus read_committed — committing offsets inside the producer transaction makes output and progress atomic, and downstream read_committed hides aborted output, delivering true exactly-once.
• Cost — a synchronous commit per batch and per revoke, plus transactional overhead for EOS. The eliminated cost is unbounded reprocessing and silent loss. Net: the replay window is O(batch), and with transactions it is zero.

Streaming
Topic — streaming
Streaming offset-commit and delivery-guarantee problems

Practice →

Event processing
Topic — event-processing
Event-processing problems on exactly-once pipelines

Practice →

---

Cheat sheet — Kafka consumer group recipes

• The ownership invariant. Every partition of every subscribed topic is owned by exactly one live member per generation. The group coordinator (a broker — leader of the group's __consumer_offsets partition) enforces it; the leader consumer computes the assignment; the generation id fences zombies. Consumers in a group cannot usefully exceed the partition count — partitions are the parallelism ceiling.
• The rebalance handshake. JoinGroup (coordinator elects the first joiner as leader, gathers all subscriptions) → leader runs the assignor client-side → SyncGroup (leader returns the full assignment, coordinator distributes each member's slice and stamps the new generation). Any request on an old generation is rejected with ILLEGAL_GENERATION.
• The four rebalance triggers. Member joins; member leaves (LeaveGroup or session.timeout.ms miss); processing stall past max.poll.interval.ms; topic-metadata or subscription change. A rebalance with no deploys and no crashes is almost always the max.poll.interval.ms self-eviction — process fewer records per poll before you raise the timeout.
• Static-membership config block. group.instance.id=<stable-per-pod> (derive from the StatefulSet ordinal), session.timeout.ms= ~2× the measured worst-case restart (commonly 45–120 s), heartbeat.interval.ms= ~session/3. A restart within the window reclaims the exact prior assignment with no rebalance; a static member does not send LeaveGroup on shutdown, trading a brief idle for zero deploy rebalances.
• Assignor selection. Range (legacy default, eager, uneven), round-robin (eager, better multi-topic balance), sticky (eager but minimal movement), cooperative-sticky (minimal movement and incremental protocol — the modern default). Cooperative sticky revokes only the partitions that change owners in a two-phase rebalance, so retained partitions never stop.
• Eager-to-cooperative migration. Two rolling deploys: (1) partition.assignment.strategy = RangeAssignor, CooperativeStickyAssignor (old first) rolled to every member — group stays eager, safe; (2) partition.assignment.strategy = CooperativeStickyAssignor only — group upgrades to incremental on the next rebalance. Never jump directly; the dual-list bridge keeps the mixed-version group valid.
• Cooperative listener contract. Under the incremental protocol, onPartitionsRevoked and onPartitionsAssigned carry only the changed partitions (deltas, not the full set); implement onPartitionsLost to drop state without committing. A listener written for the eager protocol over-commits and mishandles fencing.
• Offset storage. Commits land in the compacted __consumer_offsets topic (default 50 partitions), keyed by (group.id, topic, partition); compaction keeps only the latest. Committed offset = durable resume point; current position = in-memory next read; the gap is your crash window.
• Delivery-guarantee decision matrix. Commit after process = at-least-once (idempotent sink required; replay a batch on crash). Commit before process = at-most-once (lossy telemetry only). Offsets inside a producer transaction via sendOffsetsToTransaction + downstream read_committed = exactly-once. Auto-commit is a 5 s timer decoupled from your work — turn it off for anything with real side effects.
• Rebalance-safe commit. Keep a pending map of the next offset per partition; commit after processing for a one-batch window; commit synchronously on revoke (or the cooperative revoked subset) so a rebalance hands off a current offset with zero reprocessing spike. Never commit in onPartitionsLost.
• Tuning triangle. session.timeout.ms (liveness / static reclaim window) > worst-case restart, < crash-detection SLO; heartbeat.interval.ms ≈ session/3; max.poll.interval.ms (progress) > worst-case batch processing time. Keep max.poll.records small enough that a batch finishes well inside max.poll.interval.ms.
• Monitoring. Alert on rebalance rate (should track real membership change, not deploys), consumer lag per partition (records behind the log end), commit latency, and max.poll.interval evictions. A rising rebalance rate with stable membership means a stuck poll loop; rising lag with no rebalances means slow processing.

Frequently asked questions

What is a Kafka consumer group in one sentence?

A Kafka consumer group is a set of consumer instances that share a common group.id and collectively consume one or more topics such that every partition is owned by exactly one live member at any instant, giving you horizontal scale-out with no double-reading. The group coordinator (a broker) tracks membership via heartbeats, drives a rebalance whenever membership changes, and stores each partition's committed offset in the internal __consumer_offsets topic. Because partitions are the unit of ownership, the number of useful consumers in a group is capped at the topic's partition count — a consumer beyond that sits idle. Every senior streaming interview probes consumer groups because they are the load-bearing abstraction for scaling Kafka consumption.

What triggers a consumer group rebalance?

Four things trigger a rebalance: a member joins the group, a member leaves (either gracefully via LeaveGroup or by missing heartbeats past session.timeout.ms), a member's processing loop stalls beyond max.poll.interval.ms and is evicted, or the topic metadata changes (a subscribed topic gains partitions, or a pattern subscription matches a new topic). The most common accidental rebalance is the max.poll.interval.ms self-eviction — the heartbeat thread keeps beating so the member looks alive, but a slow processing loop that doesn't call poll() in time is evicted anyway. When a group rebalances with no deploys and no crashes, suspect that first, and fix it by lowering max.poll.records rather than blindly raising the timeout.

Eager vs cooperative sticky rebalance — which do I pick?

Pick the cooperative sticky assignor for essentially every new consumer group. The eager protocol (used by range, round-robin, and the plain sticky assignor) revokes all partitions from all members at the start of every rebalance, so the whole group processes nothing until reassignment completes — a stop-the-world pause. The cooperative sticky assignor combines stickiness (keep partitions with the member that already had them) with an incremental two-phase protocol that revokes only the partitions actually changing owners, so retained partitions keep flowing throughout. The one caveat is migration: you cannot flip a running group directly, so roll out RangeAssignor, CooperativeStickyAssignor (old first) to every member, then roll out CooperativeStickyAssignor alone.

What does static membership actually change?

Static membership, enabled by setting a stable group.instance.id per consumer, gives each member a durable identity that survives disconnects, so a member that restarts and rejoins within session.timeout.ms reclaims its exact prior partition assignment with no rebalance at all. Without it, every rolling restart rebalances twice per pod (once when the pod leaves, once when it rejoins as an anonymous new member), which for stateful apps like Kafka Streams forces expensive state-store rebuilds. The trade-off is that a static member's partitions are briefly unconsumed during the restart (its slot is reserved, not reassigned) and a genuinely crashed member is detected only after the session timeout — so size session.timeout.ms to about twice the worst-case restart. Static membership fixes restart churn only; it does not stop max.poll.interval.ms evictions.

Where are consumer offsets stored and when should I commit?

Committed offsets are stored in the internal compacted __consumer_offsets topic, keyed by (group.id, topic, partition), on the same broker that acts as the group's coordinator. The committed offset is the durable point a new owner resumes from after a rebalance or restart; it is distinct from the in-memory current position, and the gap between them is your reprocessing window on a crash. Commit after processing for at-least-once (a crash replays only the uncommitted batch, so side effects must be idempotent), before processing for at-most-once (lossy telemetry only), or inside a producer transaction with sendOffsetsToTransaction for exactly-once. Turn off enable.auto.commit for anything with real side effects — the auto-commit timer is decoupled from your processing and can both replay up to 5 s on a crash and hand off a stale offset on a rebalance.

How do I stop rebalance storms?

Attack the two root causes. First, eliminate restart rebalances with static membership: set a stable group.instance.id per pod and a session.timeout.ms comfortably above the worst-case restart, so deploys reclaim assignments instead of reshuffling them. Second, make the rebalances that do happen cheap by switching to the cooperative sticky assignor, which revokes only the partitions that move rather than stopping the world. Then rule out the silent trigger: if the group rebalances with stable membership and no deploys, it is almost certainly the max.poll.interval.ms self-eviction from a slow poll loop — lower max.poll.records, move slow I/O off the poll thread, and only then consider raising the interval. Together, static membership plus cooperative sticky plus a bounded poll loop turn a churning group into a quiet one.

Practice on PipeCode

• Drill the streaming practice library → for the consumer-group, rebalance, partition-assignment, and offset-commit problems senior interviewers love.
• Rehearse on the event-processing practice library → for partitioned-stream ownership, delivery-guarantee, and exactly-once pipeline scenarios.
• Sharpen the design axis with the design practice library → for resilient-consumer, deploy-stability, and coordination system-design questions.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the rebalance, static-membership, and cooperative-sticky decisions against real graded inputs.

Lock in Kafka consumer group muscle memory

Docs explain the config flags. PipeCode drills explain the decision — when a rolling restart storms the group, when eager rebalance stops the world, when static membership earns its session window, when commit-after-process is the difference between at-least-once and an hour of replay. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior streaming engineers actually face.

Practice streaming problems →
Practice design problems →