🗑️ Kafka Keeps Data Longer Than retention.ms

You set retention.ms to one hour. Six hours later the data is still on disk. That is not a bug, and it is not your cleanup thread being slow.

Kafka deletes data very differently from how most people picture it.

Kafka deletes segments, not records

A partition is not one big file. It is an ordered sequence of segments, each a file on disk. Retention operates at the segment level, never at the record level.

Kafka will never open a segment and rewrite it to drop a handful of expired records. That would be expensive and would break the append-only design the whole system is built on. Instead it waits until an entire segment is eligible, then deletes the whole file in one move.

When a segment is actually eligible

A closed segment becomes eligible for deletion only when both are true:

1. The segment is closed (rolled). A segment rolls when it fills up (segment.bytes, default 1 GB) or ages out (segment.ms, default 7 days).
2. Its newest record is older than retention.ms (or the partition is over retention.bytes).

Note it is the newest record in the segment that has to age out, not the oldest. One young record keeps the whole segment, and every older record in it, alive.

The active segment never dies

Here is the part that surprises people, and the reason your data outlives its retention.

The active segment, the one currently being written, is never eligible for deletion, no matter what. Retention only ever considers closed segments.

So on a low-traffic topic, the active segment fills slowly and rolls rarely. The oldest record sitting in it can be hours or days past retention.ms, just waiting for the segment to finally roll so it can even be considered for deletion.

[ segment 0 ][ segment 1 ][ segment 2 ][ segment 3 (active) ]
   closed       closed       closed        being written
   deletable    deletable    deletable     NEVER deletable
   once past retention.ms                  regardless of age

Your retention is effectively retention.ms plus however long it takes the active segment to roll.

The honest trade-off

The fix is not to lower retention.ms further. That does nothing while the segment has not rolled. Lower segment.ms so segments roll more often and become deletable sooner.

But smaller segments are not free: more files, more open file handles, more frequent rolls, and more index overhead. On a high-throughput topic the default 1 GB / 7 day segments are fine and you will never notice this. It only bites on low-traffic topics with a tight retention expectation: compliance windows, PII deletion SLAs, "we only keep 24 hours" promises. Those are exactly the cases where the gap matters, so size the segment to the retention you actually need.

This is separate from log compaction (cleanup.policy=compact), which keeps the latest value per key instead of deleting by age. Same segment mechanics underneath, different eligibility rule.

Have you ever had data outlive its configured retention on a quiet topic, and traced it back to a segment that just never rolled?