🔀 Your Kafka Group Freezes on Every Deploy

You have a Kafka consumer group with four consumers, humming along. Traffic climbs, so you add a fifth. For a few seconds, every consumer stops. Lag spikes. Then it recovers and nobody thinks about it again.

That pause is not a bug. It is eager rebalancing, and most groups still run it by default.

Rebalancing is not the enemy

A consumer group rebalances whenever membership changes: a consumer joins, leaves, crashes, or the topic gains partitions. The group has to agree on who owns which partitions. That part is unavoidable.

The cost is not the reassignment. It is how the group gets there. That is controlled by partition.assignment.strategy, and the choice you make there decides whether a deploy costs you a few milliseconds or a few seconds of frozen consumption.

The four assignors

RangeAssignor (the historical default) assigns each consumer a contiguous range of partitions, per topic.

6 partitions, 3 consumers:
A -> P0 P1   B -> P2 P3   C -> P4 P5

Clean when the numbers divide. The moment they do not, the lower-id consumers get the extra partitions, and if you subscribe to several topics they pile up on the same consumers.

RoundRobinAssignor deals every partition out one by one across all consumers.

6 partitions, 4 consumers:
A -> P0 P4   B -> P1 P5   C -> P2   D -> P3

Even spread. But it has no memory. Every rebalance recomputes the whole assignment from scratch, so partitions jump around even when they did not need to.

StickyAssignor keeps the spread even and tries to preserve the previous assignment, so it moves as few partitions as possible on each rebalance.

CooperativeStickyAssignor does the same assignment as sticky, but over a different rebalance protocol. This is the one that matters.

Eager vs cooperative: the real difference

Here is the part that trips people up. Sticky reduces how many partitions move. It does not change what happens to the ones that stay.

Range, round-robin, and plain sticky all use the eager protocol. On every rebalance the group does a full stop-the-world:

1. Every consumer revokes all of its partitions.
2. Nobody consumes anything.
3. The new assignment is computed.
4. Consumers pick their partitions back up.

So even a partition that ends up on the exact same consumer still gets revoked and paused. Add one consumer to a group of ten and all of them stop, for every partition, until the dust settles.

Cooperative-sticky (KIP-429, Kafka 2.4+) rebalances incrementally:

1. Only the partitions that actually need to move are revoked.
2. Every other partition keeps being consumed, right through the rebalance.
3. The moved partitions get reassigned in a second, short round.

One consumer joins, one or two partitions pause for a moment, and the rest of the group never notices. Stop-the-world becomes stop-one-partition.

props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

You cannot just flip the config

This is the trap. You cannot take a running group on the eager protocol and switch it to cooperative in a single deploy. A group with some members speaking eager and some speaking cooperative will not rebalance correctly.

The supported path is a two-phase rolling upgrade:

// Phase 1: deploy every instance with BOTH strategies listed.
// The group stays on the old protocol until all members support the new one.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName(),
            RangeAssignor.class.getName()));

// Phase 2: after every instance is running phase 1, deploy again
// with only the cooperative strategy. Now the group flips protocols.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

Skip phase 1 and you get a broken rebalance in production. Two deploys, in order, with the whole fleet on phase 1 before phase 2 starts.

The honest trade-off

Cooperative-sticky is not free.

• The migration is a two-step rolling upgrade you have to get right, not a one-line change.
• Incremental rebalancing can take more rounds than a single eager rebalance, so the total rebalance can be longer even though nobody is fully stopped.
• Your onPartitionsRevoked and onPartitionsLost callbacks need to be correct, because partitions now come and go without a global reset to lean on.

For a group that rebalances rarely and briefly, the eager pause may be cheap enough to ignore. For a group that scales, deploys, or loses instances often, the stop-the-world pause is a tax you pay on every event, and cooperative-sticky removes it.

Which assignor does your consumer group run today, and have you ever actually measured how long a rolling deploy freezes it?