Every event-driven service I've built in the OrderHub series shared one silent flaw, and it took me until now to close it. Since the day placeOrder started publishing events, it did a dual write: save the order to the database, then separately publish OrderPlaced to Kafka. Two systems, no shared transaction — so a crash in the window between them leaves state and events diverged: the order is saved, the event is lost, and nothing downstream ever hears about it. The transactional outbox is the pattern that removes the dual write entirely, and it's built from one extra table and a scheduled poll.
The bug: two writes, no transaction spanning them
public Order placeOrder(...) {
Order saved = repository.save(order); // ① DB write — commits
// 💥 a crash here leaves the order saved but the event never sent
events.publishOrderPlaced(saved); // ② Kafka write — separate system, no tx
return saved;
}
On the happy path this is fine. But ① and ② are separate writes to separate systems with no transaction across both. Crash after ① and the order exists with no event; publish-first instead and you can emit an event for an order that then rolls back. Crucially, this divergence happens before any consumer or saga runs — so no amount of choreography or orchestration can repair it. The fix isn't better coordination downstream; it's removing the dual write at the source.
The fix: turn the second write into a first write
The outbox turns the event into a row, written into an outbox table inside the order's own transaction. State and the intent-to-publish now commit atomically — both land, or neither does.
@Transactional // ← binds the order INSERT + the outbox INSERT into ONE commit
public Order placeOrder(String customer, String item, int quantity) {
reserveStock(item, quantity);
Order saved = repository.save(new Order(UUID.randomUUID().toString(), customer, item, quantity));
if (!outbox.append(saved)) { // outbox ON -> recorded in THIS tx; relay publishes later
events.publishOrderPlaced(saved); // outbox OFF -> old direct publish (fallback)
}
return saved;
}
append() deliberately has no @Transactional of its own — a new transaction there would defeat the whole pattern by committing the outbox row independently. It joins the caller's transaction, serialises the same event to JSON, and stores it verbatim so the relay later re-sends the exact bytes committed with the order.
public boolean append(Order order) {
if (!properties.enabled()) return false; // opt-in; off -> keep direct publish
OrderPlacedEvent event = OrderPlacedEvent.from(order); // same event, minted once
String payload = objectMapper.writeValueAsString(event);
repository.save(OutboxEvent.pending(
UUID.randomUUID().toString(), "Order",
order.getId(), // aggregateId = the Kafka message key (per-order ordering)
event.eventId(), // stable, redelivery-safe dedup key
"OrderPlaced", properties.topic(), payload, Instant.now()));
return true;
}
The relay: publish unsent rows, then mark them sent
A background @Scheduled poller drains the oldest unsent rows, publishes each to Kafka, and — the guarantee — only flips processed=true after the broker acknowledges.
private int publishPending() {
var batch = repository.findByProcessedFalseOrderByCreatedAtAsc( // oldest unsent first
PageRequest.of(0, properties.relayBatchSize()));
for (OutboxEvent row : batch) {
outboxKafkaTemplate.send(row.getTopic(), row.getAggregateId(), row.getPayload())
.get(10, TimeUnit.SECONDS); // block for the ACK before marking sent
row.markSent(Instant.now()); // managed entity -> flushed on commit
}
// a send that throws leaves the row processed=false -> retried next poll (nothing lost)
}
That ordering is the reliability: a row stays unsent until Kafka has acked it, so a crash before the ack just leaves it to be re-sent next tick — the event is never silently dropped, the exact flaw the dual write had.
The trade you're accepting: at-least-once
The outbox gives at-least-once, not exactly-once. A crash in the narrow window after the ack but before markSent re-publishes the same event next poll. That's the correct trade — never lost — and it's harmless because every event carries a stable eventId that survives redelivery unchanged, so idempotent consumers dedup on it. The demo lets you place an order in outbox mode vs the old dual-write mode, toggle a crash, and watch exactly who loses the event: dual-write strands the order with no event; the outbox recovers on restart and publishes it.
I proved it end to end against a real Postgres + an embedded Kafka. Three assertions: placing an order writes exactly one unsent outbox row in the same transaction; the relay publishes it keyed by the order id then marks it sent (and a re-run publishes zero — idempotent); and a transaction that saves the order, appends the outbox row, then throws leaves neither — the atomicity guarantee, demonstrated directly.
The whole feature is gated behind orderhub.outbox.enabled (default false), so flipping it on is a one-line config change with no recompile — off, the service behaves byte-for-byte as before. Reliable production messaging, built from one extra table and a scheduled poll.
Place an order (with or without a crash) and watch the divergence check:
https://dev48v.infy.uk/orderhub.php
Top comments (0)