OrderHub Day 32: exactly-once Kafka consumers — a dedup row in the same DB transaction as the effect, acked only after commit

Kafka only promises at-least-once delivery to a consumer. If the app commits its database work and then crashes before its offset is committed, the broker redelivers the exact same record on restart. That's not a bug — it's the contract. For OrderHub that means a redelivered OrderPlaced reserves stock twice, and the same window would double-charge a card or send two confirmation emails. Day 32 ships the production-shaped fix, and the interesting part is where Kafka's guarantees stop. Here is what I built.

Kafka's exactly-once ends at the broker

It's worth being precise about what Kafka does give you, so you add exactly what's missing and no more. Inside Kafka, exactly-once is real: the idempotent producer stamps sequence numbers so a retried publish is de-duplicated by the broker; Kafka transactions tie a batch of writes to the consumer-offset commit so "consume → transform → produce → advance offset" is atomic; and isolation.level=read_committed means a consumer only ever sees records from producer transactions that committed.

But every one of those guarantees lives between Kafka clients and the broker. None of them reach into your database. The moment a consumer reserves stock in Postgres, that write is outside the transaction that commits the offset — two systems, not one — and that seam is exactly where an at-least-once redelivery can double the effect.

The durable dedup store

Day 26 used an in-memory idempotency guard, but it has a fatal gap: it lives in the JVM heap, so the very crash that causes the redelivery also wipes the guard. Day 32 makes the guard durable — a processed_events table with one row per record we've fully processed, keyed by the record's physical coordinates: topic, partition, offset. That triple is globally unique across the log, and a post-crash redelivery of that record carries the same coordinates, which is what lets us recognise it.

CREATE TABLE processed_events (
    topic        VARCHAR(128) NOT NULL,
    partition_id INT          NOT NULL,
    kafka_offset BIGINT       NOT NULL,   -- (topic, partition, offset) is globally unique
    order_id     VARCHAR(64),
    outcome      VARCHAR(32)  NOT NULL,
    processed_at TIMESTAMP    NOT NULL,
    CONSTRAINT pk_processed_events PRIMARY KEY (topic, partition_id, kafka_offset)
);

Read-process-write in one transaction

This is the whole idea in one method. Marked @Transactional, it runs three steps in a single local DB transaction. READ: is this key already in processed_events? If so it's a redelivery — skip, touch nothing. PROCESS: otherwise apply the effect (reserve stock). WRITE: insert the marker.

@Transactional
public ProcessOutcome process(String topic, int partition, long offset, OrderPlacedEvent event) {
  var key = new ProcessedEventKey(topic, partition, offset);
  if (processedEvents.existsById(key))          // 1. READ — redelivery?
      return ProcessOutcome.DUPLICATE_SKIPPED;
  var stock = stockLevels.findById(event.item()).orElseThrow();  // 2. PROCESS
  stock.reserve(event.quantity()); stockLevels.save(stock);
  processedEvents.save(ProcessedEvent.of(key, event.orderId(), RESERVED));  // 3. WRITE
  return RESERVED;                              // marker + reservation commit TOGETHER
}

Because the marker and the reservation commit atomically, the marker is trustworthy: if it's present, the reservation definitely happened; if a crash rolled the transaction back, neither exists and the record is legitimately reprocessed. There is no window where the effect applied but the marker is missing. That is what makes it correct rather than merely likely — and it's why the business effect has to be a transactional DB write that can join the same transaction, not an in-memory map.

Ack after commit — the order is the whole game

The transaction only helps if the offset is committed at the right moment: by us, after the DB commits, never on a timer. The listener is deliberately thin, and the order of its two lines is everything.

public void onOrderPlaced(ConsumerRecord<String, OrderPlacedEvent> record, Acknowledgment ack) {
  processor.process(record.topic(), record.partition(), record.offset(), record.value());
  ack.acknowledge();   // commit the offset ONLY AFTER the DB transaction committed
}

If the app dies in the gap between the DB commit and the ack, the offset was never committed, so Kafka redelivers — and the dedup finds the marker and skips. The opposite ordering (ack first) would be a disaster: a crash before the DB commit would advance the offset past an unprocessed record and the reservation would be lost. This runs on a dedicated factory with enable.auto.commit=false, ackMode=MANUAL_IMMEDIATE, isolation.level=read_committed, and it's gated behind orderhub.exactly-once.enabled (default false) so the Day-26 path is untouched when off.

An @EmbeddedKafka + H2 test delivers the same (topic, partition, offset) twice — precisely what a redelivery looks like, and what two producer sends could never reproduce — and asserts the first returns RESERVED while the second returns DUPLICATE_SKIPPED with stock unchanged. Delivered twice, reserved once, against a real broker and real JPA, no Docker. It's the natural partner of the outbox (which re-emits) and the DLT (which replays) — both deliberately redeliver, so the consumer must handle each exactly once.

Walk the crash-and-redeliver flow live:
https://dev48v.infy.uk/orderhub.php
Repo: https://github.com/dev48v/order-hub-from-zero