🧹 Kafka Is Deleting the Wrong Data: Retention vs Compaction

Kafka is a log, and logs grow. At some point the broker has to reclaim disk, and it has two completely different ways to do it. They are not variations on a theme. They delete on different rules, and confusing them is a quiet data-loss bug that looks like corruption weeks later.

Retention: delete by age

Retention is a stopwatch. You set a window, and once data is older than that window it is deleted, segment by segment, regardless of what is inside.

log.retention.hours=168      # delete anything older than 7 days
log.retention.bytes=-1       # or cap by size per partition

Kafka stores a partition as a sequence of segment files. Retention works at segment granularity: when the newest record in a segment is older than the window (or the partition is over its byte cap), the whole segment is dropped. It never looks at record keys. It does not care whether a key still matters. Old is old.

This is exactly right for a stream of events. A click from last month, a sensor reading from an hour ago, a request log from yesterday. The value is in the recent flow; the old entries can leave.

Compaction: keep the latest per key

Compaction is a dedupe. Instead of asking "how old is this?", it asks "is there a newer record with the same key?" If yes, the older one can go.

cleanup.policy=compact

With compaction, Kafka guarantees that for every key, the latest record is retained. Older records for that same key are eventually removed by the log cleaner. A key that is written once and never updated survives forever, no matter how old it is. That is the crucial difference from retention.

This is exactly right for state. A changelog topic, a Kafka Streams KTable, a database outbox, a topic that materializes "current value per entity". The whole point is that a consumer can replay the topic from the beginning and rebuild the current state of every key.

You can also combine them:

cleanup.policy=compact,delete

Now the latest value per key is kept, but truly ancient records can still age out. Useful for a changelog where very old, untouched keys are acceptable to drop.

The incident this prevents

Here is the failure I have watched happen more than once.

Someone models current entity state as a Kafka topic: key is the entity id, value is its latest state. A downstream service rebuilds a local view by consuming the topic from offset zero. Works perfectly in testing.

The topic was left on the default cleanup.policy=delete with a 7-day retention. Most entities update often, so nobody notices. Then a slow-moving entity goes untouched for eight days. Its only record ages out. Now when the downstream service rebuilds, that entity is simply missing. No delete event, no error, no log line. It looks like data corruption, and you will spend a day chasing a bug that is really one config line.

The fix is one property: that topic should have been compact.

The honest trade-off

Compaction is not free, and it is not a drop-in for retention.

• The log cleaner spends CPU and I/O rewriting segments. On a high-churn topic that is real, ongoing work.
• Compaction only bounds size by number of distinct keys, not by time. A topic with an unbounded key space (say, keying by request id) under compact will grow without limit, because every key is unique and nothing is a duplicate. That is a disk-filling trap in the opposite direction.
• Deletes in a compacted topic need tombstones: a record with the key and a null value. Miss that and a "deleted" key lives forever.

So the rule is not "compaction is better". It is: retention for events, compaction for state, compact,delete when state has a long tail you are willing to drop, and never compact on an unbounded key space.

Takeaway

Retention deletes by time. Compaction keeps the latest per key. They solve different problems, and the default (delete) is the wrong choice for any topic that represents state.

Go look at your most important topic. Is it delete or compact, and is that on purpose or just the default nobody changed?