Your Kafka Consumer Will See Every Event Twice - Here's How to Handle It

Nazrin Suleymanli — Sat, 19 Sep 2026 15:52:30 +0000

It's 2 a.m., and your on-call phone buzzes. A customer was charged twice for the same order. You dig into the logs - the OrderCreated event was processed, then processed again seconds later. Nothing crashed. No exception was thrown. Kafka did exactly what it promised: it delivered the message at least once. The bug isn't in Kafka. It's an assumption that your consumer would ever see each event only once.

In this tutorial, we'll fix that assumption: why duplicates are normal, why the obvious consumer code fails silently, and how to make a Spring Boot Kafka consumer idempotent - safe to run any number of times with the same result as running it once.

Why does this happen?

Kafka gives you an at-least-once delivery guarantee. Not exactly once - at least once. That word "least" is doing a lot of work, and it's the root of our problem.

Here's the normal flow of a consumer:

Kafka hands your consumer a message.
Your code processes it (charges the customer, writes to the DB).
Your consumer commits the offset - essentially a bookmark saying "I've handled everything up to here."

The trouble lies in the gap between steps 2 and 3. If your service crashes, is redeployed, or the consumer group rebalances after the work is done but before the offset is committed, Kafka has no idea step 2 ever happened. So when the consumer restarts, it reads from the last committed offset and delivers that same OrderCreated event again.

Think of the offset as a bookmark in a book. You read a page (process the event), then move the bookmark forward (commit the offset). Now imagine your phone dies after you read the page but before you move the bookmark. When you reopen the book, you start from the old bookmark and read that page again.

Nothing is broken. Kafka is doing exactly what it promised. The double charge comes from our code assuming each event arrives once.

Rebalancing makes this worse in production. Every deploy, every scale-up, every pod restart in Kubernetes triggers a rebalance, so the more you scale, the more often you hit this window.

The naive approach (and why it fails)

Most consumers start out looking like this:

@KafkaListener(topics = "orders", groupId = "billing-service")
public void handleOrderCreated(OrderCreatedEvent event) {
    paymentService.charge(event.getCustomerId(), event.getAmount());
    log.info("Charged customer {} for order {}", event.getCustomerId(), event.getOrderId());
}

This is clean, readable, and works perfectly in every demo. Ship it, and it will happily process thousands of orders without complaint.

Then one day, a pod restarts mid-processing, Kafka redelivers the last event, and paymentService.charge(...) runs a second time. There's no error, no stack trace, just a customer with two charges and a support ticket.

The reason this is so easy to miss is that the code is correct in isolation. The bug isn't in any single line. It's in the assumption baked into the whole method: that it will run only once per order. Kafka never promised that.

So the fix isn't "handle the exception" or "add a try/catch." There's nothing to catch. The fix is to make the operation idempotent, so it is safe to run any number of times with the same result as running it once.

A quick way to picture idempotency: pressing an elevator call button once and pressing it five times gives the same result. The elevator still comes once. A light switch is not idempotent: every press changes the state. Our charge() method is currently a light switch. We're going to turn it into an elevator button.

The fix: an idempotency key + a dedup store

The idea is to give every event a stable identity and remember which ones we've already handled. Kafka can redeliver all it wants. We just refuse to act on the same ID twice.

One thing to get right before any code: the event ID must be a stable ID that travels in the payload (an eventId or orderId that the producer sets), not a UUID.randomUUID() you generate at the time of consumption. If you mint a fresh ID on each consume, every redelivery looks "new" and the whole scheme is pointless.

First, a table whose only job is to remember processed events. The event_id is the primary key, so the database itself guarantees we can never store the same one twice:

CREATE TABLE processed_events (
    event_id     VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP NOT NULL DEFAULT now()
);

@Entity
@Table(name = "processed_events")
public class ProcessedEvent {
    @Id
    private String eventId;
    private Instant processedAt = Instant.now();
    // constructors, getters
}

Now the consumer. Two things must happen together, or not at all: recording the event as processed, and doing the actual work. So we wrap them in a single @Transactional method:

@KafkaListener(topics = "orders", groupId = "billing-service")
@Transactional
public void handleOrderCreated(OrderCreatedEvent event) {
    if (processedEvents.existsById(event.getEventId())) {
        log.info("Duplicate event {} — skipping", event.getEventId());
        return;
    }

    paymentService.charge(event.getCustomerId(), event.getAmount());
    processedEvents.save(new ProcessedEvent(event.getEventId()));
}

Why the single transaction matters: the charge and the "I've processed this" marker now commit as one unit. If charge() throws halfway through, the transaction rolls back the marker as well, so Kafka will redeliver and we'll retry cleanly. And if a rare concurrent duplicate slips past the existsById check, the primary key on event_id rejects the second insert, the transaction fails, and no second charge is committed.

So there are really two layers here: the existsById check is for speed (skip the obvious duplicates cheaply), and the primary key is for correctness (the database is the last line of defense when two duplicates race). They work together.

What about Redis?

Redis is a popular, faster alternative - a single atomic SET billing:event:<eventId> 1 NX EX 86400 does the "write only if absent" check (NX) and auto-expires the key after a day (EX). But Redis can't join your database transaction: if you mark the event seen in Redis and then the DB write fails, you're left with a "processed" marker and no work done, and the redelivery gets skipped too. So use Redis when the side effect isn't critical (send a notification once, dedupe analytics), and keep the event ID in the same database as your work when correctness really matters, like billing.

Edge cases worth knowing

Offset commit strategy

A nice consequence of the dedup store: you no longer need heroics around offset committing. Even if the offset commit fails after your transaction commits, the redelivered event just hits the dedup check and gets skipped.

That said, don't leave Spring Kafka on naive auto-commit (enable.auto.commit=true), which commits offsets on a timer regardless of whether your work has finished. If the consumer dies after a commit but before the work completes, the event is lost. Instead, commit after the listener succeeds. The simplest way is AckMode.RECORD: Spring commits each record's offset automatically when the listener method returns without throwing an exception, with no extra code on your part.

factory.getContainerProperties().setAckMode(AckMode.RECORD);

If you want finer control, switch to manual acks (AckMode.MANUAL_IMMEDIATE) and acknowledge explicitly at the point you decide the work is done:

@KafkaListener(topics = "orders", groupId = "billing-service")
public void handle(OrderCreatedEvent event, Acknowledgment ack) {
    // dedup check + charge ...
    ack.acknowledge(); // only now: "this offset may be committed"
}

Either way, the principle is the same: commit the offset after the work, never before. The dedup store is your safety net against redelivery, not an excuse to commit early.

The dedup table grows forever

Duplicates only ever arrive within a short window (a rebalance, a restart), so you don't need to keep event IDs around for months. Add a small scheduled job that deletes rows older than a few days:

DELETE FROM processed_events WHERE processed_at < now() - INTERVAL '7 days';

Duplicates on the producing side

This tutorial fixes the consumer. If your service also publishes events while writing to its own database, you have a related-but-separate problem: the DB write can succeed while the Kafka publish fails, or vice versa. The standard fix there is the transactional outbox pattern: write the event to an outbox table in the same transaction as your data, and relay it to Kafka separately. Worth a follow-up read once the consumer side is solid.

When you need this and when you don't

Reach for a dedup store whenever the side effect isn't naturally idempotent: charging money, sending an email, incrementing a counter, or calling an external API that performs an action. These are the operations that hurt when they run twice.

You may not need one when the operation is inherently idempotent. If your consumer just sets the absolute state UPDATE orders SET status = 'PAID' WHERE id = ? running it twice changes nothing the second time. In those cases, making the write itself idempotent (an upsert keyed by ID, setting absolute rather than relative values) can be enough on its own.

The mental shift is the whole point: stop trying to make Kafka deliver exactly once; it won't. Instead, make your consumer not care how many times it's called. Once each event carries a stable ID and your side effects are idempotent, that 2 a.m. double charge simply can't happen. The redelivery still arrives. Your code just shrugs and moves on.

DEV Community: Nazrin Suleymanli