Exactly-once delivery is a lie: what your broker actually guarantees
Your message broker's docs say "exactly-once semantics." You read that, nod, and assume your consumer handler runs exactly one time per message. Ship it.
Then you get duplicate charges in production. Or a user's order gets created twice. And you're staring at the logs thinking: I thought this was supposed to be exactly-once?
Here's the thing. The broker isn't lying to you. But it's not promising what you think it's promising. And the gap between what "exactly-once" means on the marketing page and what it means in the protocol spec is where your bugs live.
⚡ The three delivery semantics
Before we untangle the confusion, let's get precise about what the three guarantees actually mean. They look different depending on whether you're the producer or the consumer.
At-most-once. Producer fires a message and doesn't retry on failure. If the network drops it, it's gone. On the consumer side, you commit your offset before processing. If your process crashes after committing but before finishing work, that message is skipped forever. No duplicates, but you lose data.
At-least-once. Producer retries until it gets an acknowledgment. If the ack gets lost but the broker already wrote the message, you get a duplicate in the log. Consumer side: you process the message first, then commit the offset. Crash after processing but before committing? You'll reprocess that message on restart. Duplicates happen. Guaranteed.
Exactly-once. Each message appears in the log once and its effect is applied once. Sounds perfect. But there's a catch.
Why exactly-once delivery is impossible
Think about what happens at the network level. Your producer sends message M to the broker. The broker writes it. The broker sends an ACK back. The ACK gets lost.
Now your producer is stuck. It can't tell the difference between "the broker never received M" and "the broker received M but the ACK didn't make it back." Two completely different situations, identical from the producer's perspective.
So it has to choose. Retry and risk a duplicate? Or don't retry and risk losing the message? There's no third option. This is the Two Generals Problem (1975): no finite protocol can guarantee two parties reach agreement over an unreliable channel. The sender can never know its last message arrived.
Not a Kafka limitation. Not a broker limitation. A mathematical impossibility for any system communicating over a network that can drop packets.
So when someone says "exactly-once delivery," they're either wrong or they're talking about something else.
🎯 Exactly-once processing: the achievable goal
Here's the pivot. You can't guarantee a message is delivered exactly once. But you can guarantee its effect is applied exactly once. Different thing entirely.
The industry sometimes calls this "effectively-once." Your consumer might receive the same message three times. Doesn't matter, as long as the business result only happens once. Two ways to get there:
Idempotent processing. Design your writes so that applying the same operation twice produces the same result. An upsert keyed on a deterministic message ID. A dedup check before inserting. If the same message shows up again, the second write is a no-op.
Atomic offset + effect commit. In a single database transaction, write your business result AND advance your consumer's offset/cursor. On restart, you read the last committed offset from your DB. You might reprocess a message, but the dedup check inside the transaction rejects it.
Here's what that looks like in practice. An at-least-once consumer with a PostgreSQL dedup table:
async function handleMessage(msg: { id: string; payload: unknown }, pool: Pool) {
const client = await pool.connect();
try {
await client.query("BEGIN");
// Insert message ID — ON CONFLICT means we've seen this before
const { rowCount } = await client.query(
`INSERT INTO processed_messages (message_id) VALUES ($1) ON CONFLICT DO NOTHING`,
[msg.id]
);
if (rowCount === 0) {
await client.query("ROLLBACK");
return; // duplicate, skip it
}
// Business logic: only runs once per message ID
await client.query(
`INSERT INTO orders (id, data) VALUES ($1, $2)`,
[msg.id, JSON.stringify(msg.payload)]
);
await client.query("COMMIT");
} catch (e) {
await client.query("ROLLBACK");
throw e;
} finally {
client.release();
}
// ACK to broker ONLY after commit succeeds
}
The consumer uses at-least-once delivery from the broker. Messages might arrive more than once. But the ON CONFLICT DO NOTHING clause on processed_messages means the business write only happens once. Exactly-once processing on top of at-least-once delivery.
What Kafka's "exactly-once" actually covers
Kafka does offer exactly-once semantics. But with a hard boundary.
The idempotent producer (enable.idempotence=true, default since Kafka 3.0) assigns each producer a PID and attaches monotonic sequence numbers per partition. The broker deduplicates by rejecting messages with a sequence number it's already seen. Scope: one producer session, one partition.
Transactions (transactional.id) go further. They atomically write to multiple partitions and commit consumer offsets in a single operation. Consumers with isolation.level=read_committed only see committed messages. Zombie producers with the same transactional ID get fenced off.
Here's a transactional produce-and-commit using KafkaJS:
const producer = kafka.producer({
transactionalId: "my-app-topicA-0",
maxInFlightRequests: 1,
idempotent: true,
});
await producer.connect();
const txn = await producer.transaction();
try {
await txn.send({ topic: "output", messages: [{ value: "result" }] });
await txn.sendOffsets({
consumerGroupId: "my-group",
topics: [{ topic: "input", partitions: [{ partition: 0, offset: "42" }] }],
});
await txn.commit();
} catch (e) {
await txn.abort();
throw e;
}
This gives you exactly-once from input topic to output topic. Read a message, produce a result, commit the offset, all atomically. Powerful stuff.
But here's the boundary everyone misses. This only works when both source and sink are Kafka topics. The moment your consumer writes to an external database, calls an HTTP API, sends an email, anything outside Kafka's transaction fence, you're back to at-least-once. The transaction can't wrap your PostgreSQL insert or your Stripe API call.
So yes, Kafka has exactly-once. For Kafka-to-Kafka pipelines. For everything else, you need the patterns from the previous section.
The practical playbook
You've got a consumer that writes to a database. Messages will arrive more than once. Here's what actually works:
1. Dedup table. Store (message_id, processed_at) in the same transaction as your business write. Reject on conflict. Simplest pattern, works everywhere.
2. Idempotent upserts. If your write is naturally idempotent (setting a user's email to a value, not incrementing a counter), just use ON CONFLICT DO UPDATE or equivalent. No separate dedup table needed.
3. Atomic offset commit. Store the consumer offset in your application database, not in Kafka's __consumer_offsets topic. On restart, seek to the stored offset. Combined with a dedup check, this closes the gap completely.
4. Dedup windows. Keep a TTL-bounded set of seen message IDs in Redis or memory. Cheaper than a DB check, but only works within the window. Messages replayed after the TTL expires will be processed again. Partial fix.
And honestly? Sometimes at-most-once is the right call. Metrics counters where a missing data point is fine but a duplicate inflates your graphs. Fire-and-forget telemetry. Log shipping. Not everything needs exactly-once processing.
If you're running Kafka with consumer groups, the rebalancing protocol already causes redelivery on partition reassignment. Your consumers need to handle duplicates regardless of what guarantee you think you've configured.
📌 Key takeaways
- Exactly-once delivery is impossible over an unreliable network. The Two Generals Problem proves it. Your broker can't fix physics.
- Exactly-once processing is achievable. Idempotent handlers + dedup tables turn at-least-once delivery into effectively-once effects.
- Kafka's exactly-once covers Kafka-to-Kafka only. External writes (databases, APIs, emails) fall outside the transaction boundary.
- Design for at-least-once. Assume every message arrives more than once and make your handlers safe for it.
- The dedup table pattern works everywhere. Single transaction, message ID check, business write. Hard to mess up.
For how Kafka's partitioning and consumer groups handle parallelism and rebalancing, see Kafka partitions and consumer groups. And if you're routing traffic to your consumers through an API gateway, the retry behavior at the gateway layer adds another source of duplicates to plan for.
More writing
If this was useful, there's more where it came from at arnavsharma.dev.
Top comments (0)