Every few months a team I talk to decides they need exactly-once delivery in Kafka. They read the docs, discover processing.guarantee=exactly_once_v2, wire up transactional producers, and pat themselves on the back for building something bulletproof.
Most of them never needed it.
Exactly-once is one of the most over-reached-for features in event-driven systems. It sounds like the obvious, responsible choice (who wants duplicates?) but in practice it buys a lot of complexity and performance cost to solve a problem that a much simpler pattern usually solves better. In three years of building Kafka consumers in production, I've reached for it exactly zero times. Here's why.
Why exactly-once is so tempting
The pull toward exactly-once is completely understandable. "Duplicates" sounds like a bug, and exactly-once sounds like the grown-up, production-ready answer. When you're staring at a double-charged customer, a config flag that promises the problem simply won't happen is the most attractive thing in the world.
There's also the framing in the docs. Kafka lists its guarantees as at-most-once, at-least-once, and exactly-once, and read top to bottom, it looks like a difficulty ladder where exactly-once is the top rung, the "best" one. Nobody wants to build the lesser system, so teams reach for the rung labelled strongest.
| Guarantee | Duplicates? | Data loss? | Cost |
|---|---|---|---|
| at-most-once | No | Possible | Cheap |
| at-least-once | Possible | No | Cheap |
| exactly-once | No | No | Expensive, and Kafka-only |
But that ladder is misleading. These aren't "worse" and "better" versions of the same thing, they're different trade-offs, and the cheap middle option is the one most systems actually want.
And to be fair, the instinct isn't wrong. Duplicates are a real problem. Left unhandled, they double-charge customers, send emails twice, and corrupt counters. The disagreement isn't about whether duplicates matter. It's about where you solve them. Exactly-once tries to solve them at the transport layer, before your code ever sees them, and that's where the hidden costs start.
The hidden cost
Exactly-once isn't free, and the price shows up in three places.
First, performance. Exactly-once in Kafka is built on transactions, and transactions add coordination overhead on every message. You pay for it in throughput and latency. For a lot of systems that's an acceptable trade, but you're paying it whether or not you actually had a duplicate problem worth solving.
Second, complexity. Turning on exactly_once_v2 isn't a single switch either. Under the hood it's really several moving parts working together: an idempotent producer so retries don't create duplicates, transactional writes, and a transactional consumer to tie the read and write into one unit. You inherit all of that coupling and complexity, and every extra moving part is a part that can break at 2 a.m.
Third, and this is the one that changes the whole argument: Kafka's exactly-once only works Kafka to Kafka. It covers the read, process, and write cycle when everything stays inside Kafka. The moment your consumer does something to the outside world, calling a payment gateway, writing to a separate database, sending an email, that action is outside the transaction. Exactly-once will not protect it.
So if you're charging a customer, you still have to make that charge idempotent yourself. Which means you did all the work of setting up exactly-once, and you still wrote the dedup logic you were trying to avoid. You paid twice and solved the problem once.
What to do instead
Start from the other end. Assume duplicates will happen, because with at-least-once they will, and make your consumer not care.
In practice that means one thing: make your side effects idempotent. Give every event a stable ID, keep track of the ones you've already handled, and skip anything you've seen before. The charge runs once no matter how many times the event arrives. That's it. No transactions, no coupling, no throughput tax. Just a consumer that shrugs at a redelivery and moves on.
@KafkaListener(topics = "orders", groupId = "billing-service")
@Transactional
public void handle(OrderCreatedEvent event) {
if (processed.existsById(event.getId())) return; // seen it, skip
charge(event);
processed.save(new ProcessedEvent(event.getId()));
}
Five lines. No pipeline-wide mode, no transactional producer, nothing to reason about at 2 a.m. The event can arrive ten times and the customer is charged once.
The nice part is that this scales down as well as up. A tiny service and a twelve-service saga need the exact same discipline: every step has to be safe to retry. Idempotency is a property you build into each handler, so it composes naturally as your system grows, while exactly-once is a mode you switch on for a whole pipeline and hope nobody steps outside of.
So my rule of thumb is simple. Reach for idempotency first. It solves the duplicate problem where the duplicates actually cause damage, in your code, right next to the thing that touches the outside world.
If you want the concrete implementation, with the dedup table and the Spring Boot consumer, I wrote up the full how-to here:
When exactly-once actually earns its place
None of this means exactly-once is useless. It's a real feature that solves a real problem, and there's a case where it genuinely fits: pure Kafka to Kafka pipelines with no outside world involved. If you're reading from a topic, transforming, and writing back to another topic, and correctness of the result matters, like a running aggregation or a stateful stream processor, exactly-once is doing exactly what it was designed for. There's no external side effect to make idempotent, so the transactional model covers the whole job.
The point isn't that the feature is bad. It's that it's a specialized tool, not the default. The moment a real side effect enters the picture, and in most business systems it does almost immediately, idempotency is the pattern carrying the weight, with or without exactly-once on top.
So before you turn it on, ask one question: does my consumer actually touch anything outside Kafka? If the answer is yes, and it usually is, start with idempotency. Make your consumer not care how many times an event arrives, and the duplicate problem quietly disappears where it was actually going to hurt you.
Top comments (0)