The Outbox Pattern Explained (Complete Guide)

#distributedsystems #systemdesign #database #backend

Your service writes an order to the database, then publishes an event to Kafka. Two lines. Nobody flags it in review.
In production, it quietly loses events.

The failure nobody catches until it's a support ticket: the DB commit succeeds, then the process dies before the Kafka publish lands. A network blip, a pod restart, a broker that's down for four seconds. Now the order exists in your database, but the shipping service never heard about it. Your data says one thing, your event stream says another, and nothing threw an error.

This is the dual-write problem. You're writing to two systems that don't share a transaction, and no amount of reordering makes "save then publish" atomic. Flip the two lines and you get the mirror bug: event out, DB write fails.

The outbox pattern fixes it by refusing to do two writes at all.
Instead of publishing to Kafka, you insert the event into an outbox table inside the same local transaction as your business data. One transaction, one database. Both rows commit or neither does. Your database already guarantees that. A separate relay then reads the outbox and forwards events to the broker.

That's the whole trick. You turn a distributed-transaction problem into a boring local one, and databases are very good at boring local transactions.

Now the part most "complete guides" skip.
The outbox does not give you exactly-once delivery. It gives you at-least-once. The relay can publish an event and then crash before it marks the row as sent, so it sends again on restart. Log-based CDC has the same gap: it ships the change, dies before the offset is acknowledged, resends on recovery. If your consumers aren't idempotent, you didn't kill the bug. You just moved it downstream.
Two ways to run the relay, and neither is the "right" one:
Polling: a loop that queries the table for unsent rows. Easy to reason about. It also hammers the database, ties your latency to the poll interval, and needs locking so two instances don't grab the same row.

CDC: Debezium tailing the Postgres WAL or MySQL binlog. Near real-time, no query load, keeps commit order. You pay for it in operations: replication slots, WAL retention, and another connector to babysit.

Polling is completely fine to start with. You can switch to CDC later without changing the outbox schema.
One thing everyone forgets: that table grows forever unless you delete processed rows. You find this out somewhere around a few hundred million rows, usually at the worst time.
There's no reliability magic here. Just an honest admission that you can't touch two systems atomically, so you stop pretending and design around it.

If you've run this in production: polling or CDC, and what pushed you one way?

Full end-to-end breakdown here: https://www.youtube.com/watch?v=BJvQdS0m-Kw