DEV Community

Aliasgar
Aliasgar

Posted on

Exactly-Once Semantics in Kafka: Promise vs. Reality

#dataengineering #distributedsystems #streaming #systemdesign

"We're using Kafka with exactly-once semantics, so we don't have to worry about duplicates."

I've heard this in architecture reviews, design docs, and postmortem explanations. It represents a misunderstanding of what Kafka's exactly-once guarantee actually covers, and the gap between the promise and the reality has caused real production incidents.

What Kafka's Exactly-Once Actually Covers

Kafka's exactly-once semantics (EOS), introduced in 0.11.0, operates at two levels:

Producer idempotence (enable.idempotence=true): The producer assigns each message a sequence number. The broker deduplicates messages with the same producer ID and sequence number. This prevents duplicates caused by producer retries — the message lands in the Kafka partition exactly once, regardless of retry count.

Transactions (transactional.id): Allows a producer to write to multiple partitions atomically. Either all writes commit or none do. Combined with isolation.level=read_committed on consumers, readers only see committed transactions.

Together, these give you exactly-once message delivery within the Kafka cluster.

What Exactly-Once Does Not Cover

Here's the boundary that engineers miss: Kafka's exactly-once guarantee is scoped to the Kafka cluster. The moment your consumer does anything outside Kafka — writes to a database, calls a REST API, publishes to a cloud queue — you're outside the transaction boundary.

Consider a typical consumer:

consumer.poll(records);
for (record : records) {
    database.save(process(record));  // External write — outside Kafka transaction
}
consumer.commitSync();
Enter fullscreen mode Exit fullscreen mode

If the application crashes after database.save() but before commitSync(), Kafka re-delivers the message. The consumer reprocesses it. The database now has two writes for the same event.

Enabling producer idempotence on the consumer's Kafka writes does not fix this.

The Patterns That Actually Give You End-to-End Safety

Idempotent Consumers

Design consumer processing logic to be idempotent: processing the same message twice produces the same result as once.

public void processPaymentEvent(PaymentEvent event) {
    if (paymentRepository.existsByEventId(event.getId())) {
        return; // Already processed
    }
    paymentRepository.save(toPayment(event));
}
Enter fullscreen mode Exit fullscreen mode

Use a unique constraint on event_id and handle the constraint violation rather than relying solely on the pre-check — this closes the concurrent processing race condition.

Transactional Outbox

Write to your database and record the outgoing Kafka message in an outbox table in the same local transaction:

BEGIN;
INSERT INTO payments (id, amount, status, event_id) VALUES (...);
INSERT INTO outbox (event_id, topic, payload) VALUES (...);
COMMIT;
Enter fullscreen mode Exit fullscreen mode

A separate process (Debezium CDC or a polling publisher) reads the outbox and publishes to Kafka. This makes the database write and Kafka publish effectively atomic from a business perspective.

Offset-Aware Idempotency

Store the consumed offset alongside the processed result in the same database transaction. On restart, start consuming from the last successfully committed offset. This requires managing offsets manually rather than using auto-commit.

Performance Considerations

Kafka transactions have a cost:

  • min.insync.replicas=2 is required for EOS correctness — forcing synchronous replication, which increases write latency.
  • Transaction coordinators add a round-trip per transaction commit.

In payment event pipelines where I've implemented EOS, throughput dropped roughly 15-20% compared to at-least-once, with higher p99 latencies. In the context of financial correctness, that was an acceptable tradeoff — but it's a tradeoff you should measure, not assume.

The Honest Summary

Kafka's exactly-once semantics is real and valuable — for the Kafka-to-Kafka case. Read from a topic, process, write results to another topic: that chain can be exactly-once if you configure it correctly.

For consumers that write to databases or call external services, exactly-once at the Kafka layer removes the easy duplicate vector (broker-level retries) but doesn't remove the hard one (consumer restarts after partial processing). Your consumer still needs to be idempotent.

Design idempotent consumers. Use the transactional outbox for end-to-end atomicity. Use Kafka EOS as one layer in a defense-in-depth approach, not as a magic bullet.

Top comments (0)