🩸 acks=all Still Lost My Data: Kafka Unclean Leader Election

You did everything the durability docs told you to. acks=all. Replication factor 3. You even remembered min.insync.replicas. Then a broker failover ate a handful of records that your producer had already been told were committed.

acks=all did not lie to you. A different setting did.

What acks=all actually promises

When you produce with acks=all, the leader does not acknowledge the write until every replica in the in-sync replica set (the ISR) has it. Combined with min.insync.replicas=2, that means at least two replicas hold the record before your producer sees success. That is a real, strong guarantee.

The word doing all the work is in-sync. The guarantee is only as durable as the membership of that set, and that membership is not fixed.

Step one: the ISR shrinks

A follower has one job: keep pulling from the leader and stay caught up. When it cannot keep the pace, for a long enough window (replica.lag.time.max.ms), the leader removes it from the ISR.

Followers fall behind for boring reasons. A GC pause. A slow disk. A saturated network link. A node under memory pressure. None of these are exotic; they happen in normal operation.

Here is the subtle part. When the ISR shrinks from three replicas to two, writes keep committing. As long as the surviving set still meets min.insync.replicas, the producer never notices. The topic looks healthy. But the redundancy you were counting on is quietly gone.

Step two: the leader dies

Now the leader for that partition fails. Kafka must elect a new leader. It has two options.

1. Promote a replica that was in the ISR. That replica has every committed record, so nothing is lost. This is a clean election.
2. If no in-sync replica is available, either wait, or promote a replica that was not in sync.

That decision is controlled by a single broker/topic setting:

unclean.leader.election.enable=false

Step three: the flag pulls the trigger

With unclean.leader.election.enable=true, when there is no in-sync replica to promote, Kafka picks a lagging one anyway to keep the partition available.

That replica is missing the newest committed records, the ones that only lived on the ISR members that are now gone. The moment it becomes leader, it is the source of truth. Every other replica that rejoins truncates its log to match the new leader. The records that were ahead of the new leader's high-water mark are discarded across the whole partition.

Those were committed writes. Your producer got a success response for them. They are now gone, and nothing in the client will ever tell you.

ISR = {1, 2, 3}   acks=all commits, all good
broker 3 lags     ISR = {1, 2}, writes still commit
leader (1) dies   no in-sync replica free
unclean election  broker 3 (stale) becomes leader
                  brokers truncate to broker 3's offset
                  -> committed records 4,5 lost

The fix

unclean.leader.election.enable=false
min.insync.replicas=2
# producer
acks=all

With unclean election off, if there is no in-sync replica to promote, the partition goes offline and waits for one to come back. Producers get an error and retry or buffer. You lose availability for that partition, but you never serve or acknowledge data you cannot back up.

Keep min.insync.replicas at 2 (with RF 3) so a single lagging follower cannot drop you below the durability floor. Produce with acks=all so the two guarantees actually compose.

The honest trade-off

This is not free. You are explicitly choosing consistency over availability for that partition. If enough replicas are unhealthy at once, the partition stops accepting writes until Kafka has an in-sync replica to trust. For a payments ledger or an event source that must never lose a record, that is exactly what you want. For a firehose of metrics where a short gap is fine and downtime is not, you might genuinely prefer the opposite.

The mistake is not picking availability. The mistake is picking it by accident, because the setting defaulted that way on an old cluster and nobody revisited it. Durability you did not choose on purpose is not durability.

Takeaway

acks=all protects you while the ISR is healthy. Unclean leader election is what happens when it is not. Decide, deliberately, whether a partition should go dark or go stale when the replicas you trusted are gone.

Which way is your most important topic configured, and did someone actually choose it?