Two different systems. The same two lines of Kafka topic configuration.
retention.ms = -1
log.cleanup.policy = compact
Most Kafka topics are streams of events with a shelf life. Data lands, consumers read it, and after a few days or weeks the retention window deletes it. These two settings build the opposite thing. Infinite retention says records never expire by time. Log compaction says the broker keeps only the latest record for each key. Together they turn a topic into a permanent table: one current record per key, kept forever, replayable from the beginning at any moment.
I have used this combination in two production systems that look nothing alike. The pattern behind both is the same, and it is the subject of this article.
What the configuration actually does
With log.cleanup.policy=compact, the log cleaner works through the topic's older segments and drops any record whose key has a newer record later in the log. The newest value for every key always survives. With retention.ms=-1, no time-based deletion ever removes what compaction keeps. The result is a topic whose size tracks the number of distinct keys, not the volume of writes. A million updates to one key still leave one record.
That gives the topic a property normal topics do not have: a consumer that reads it from the beginning reconstructs the complete current state. Not the history of how the state evolved — the state itself, exactly as it stands now. Any service can rebuild its view of the world by replaying the topic, and a brand-new consumer catches up to the same picture as everyone else.
Project one: connector configs as a compacted topic
The first system is a self-service tool for Kafka connectors. It is a Flask application. Users sign in with SSO and see cards for the connector types they can create — Salesforce, Google, MirrorMaker 2, and other Camel sources and sinks. Picking a card opens a form with the fields that connector needs. The backend is an open-source Kafka Connect distributed cluster running on EKS, loaded with the Camel connector JARs.
When a user submits the form, the tool produces one record to a Kafka topic. The key is the connector name. The value is the connector config. That topic runs with infinite retention and compaction.
This makes the topic the registry of every connector in the system. A connector's config can never age out and disappear, because retention is infinite. An updated config does not pile duplicates on top of the old one, because compaction keeps exactly one live record per connector name — the latest write wins. The full set of connector definitions lives in one durable place, in the same system that runs them, with no separate database to keep in sync.
Project two: user data for log enrichment
The second system starts with an identity tool that emits user data logs — records for new users and updated records for existing users. Every value carries one unique user with all of their details. This data has to be kept forever, and there must never be duplicates for the same user.
Same configuration. The key is the user. The value is the full user record. Infinite retention keeps every user's current record permanently. Compaction guarantees that no matter how many updates a user accumulates, exactly one record per user survives — the current one.
This topic feeds a bigger pipeline: a log aggregation system that processes server logs from across the platform. As server logs stream through, the pipeline enriches them with user data, joining each log against the latest user record from the topic. Enrichment is only trustworthy if the reference data is complete and current, and that is exactly what the compacted topic provides. It is the full user base, deduplicated by construction, and replaying it from the beginning rebuilds that user base exactly as it stands today.
The rule behind both
Configs and user profiles share one trait: they describe state, not history. The useful question they answer is "what does this look like right now." Nobody needs the fourteen previous versions of a connector config or a user's profile. They need the current one, guaranteed present, guaranteed unique.
That is the decision test I apply. Data that describes events — things that happened, in order, where the history itself is the value — belongs in a normal topic with time-based retention. Data that describes current state belongs in a compacted topic with infinite retention. Events get retention windows. State gets compaction.
Two operational notes complete the picture. Compaction is a background process, so a superseded record can sit in the log for a while before the cleaner removes it; the guarantee is about what survives, not about instant cleanup. And a compacted topic still grows with the number of keys, so it fits state with a bounded key space — connector names, user IDs — rather than unbounded event streams.
Used this way, Kafka stops being only a pipe between systems. A compacted, infinitely retained topic is a durable, replayable store of current state that any consumer can rebuild from at any time. Two lines of configuration, and the topic never forgets what things look like right now.
Top comments (0)