Transactional Outbox Pattern: Prevent Lost Events in EDA

The failure that taught me to stop trusting the network

Yesterday a user updated an order status and the database showed the change — but downstream services never received the event. Payments and fulfillment were out of sync for hours. The root cause was simple and common: a DB-then-publish flow. We committed the DB, then published the event. A network blip (or a thread getting killed during send) produced silent divergence.

If you ship events that other teams or systems depend on, the core guarantee you need is that events can't magically disappear. The transactional outbox pattern gives you that guarantee without two-phase commit or exotic infrastructure.

What the transactional outbox pattern buys you

At its core, the transactional outbox pattern ensures the producer's state change and the event that announces that change are committed together in a single local transaction. That means there is no window where the DB has changed but no event exists. It converts the dual-write problem into a local, ACID-backed write plus an out-of-band relay that publishes reliably.

Benefits:

• Guarantees that an event exists if and only if the corresponding DB change committed.
• Eliminates the most dangerous failure mode: lost events caused by partial failure between DB commit and broker publish.
• Keeps producer latency predictable (no remote broker call on the request critical path).

Tradeoffs:

• Adds a table and a relay process (poller or CDC connector).
• Creates at-least-once delivery; consumers must be idempotent.
• Requires monitoring and outbox cleanup to avoid table bloat.

Recipe: a practical, step-by-step implementation

1) Persist the event in an outbox table inside the same DB transaction as your state change (or use CDC to stream committed inserts).

2) Relay reliably: run a dedicated relay worker that reads the outbox and pushes to your broker. Use row-level locking hints — e.g. PostgreSQL's FOR UPDATE SKIP LOCKED — so multiple workers can run in parallel without stepping on each other.

3) Make consumers idempotent so retries are safe. Record processed event IDs in the consumer's DB and make the dedup insert share the same transaction as the side effect.

4) Add lightweight observability: counters for init/send_success/send_fail, lag metrics (age of oldest unsent row), and alerts for rising stuck rows.

Minimal outbox schema (Postgres)

CREATE TABLE outbox (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_type TEXT NOT NULL,
aggregate_id TEXT NOT NULL,
event_type TEXT NOT NULL,
payload JSONB NOT NULL,
status TEXT NOT NULL DEFAULT 'init',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
sent_at TIMESTAMPTZ
);

Create an index for pending rows:

CREATE INDEX idx_outbox_pending ON outbox(created_at) WHERE status = 'init';

Example: claim a batch safely (Postgres)

SELECT id, payload
FROM outbox
WHERE status = 'init'
FOR UPDATE SKIP LOCKED
LIMIT 100;

This simple query is the core of a poller that allows horizontally scaling relay workers: each worker claims rows and others skip locked rows.

Concrete engineering example

A high-throughput Ledger Service used the outbox plus SKIP LOCKED and hit 6,500 RPS with effectively 0% send errors by allowing multiple relay workers to claim rows without contention. At larger scale, some teams shard local SQLite WALs or switch to CDC; those options work but add operational complexity.

Three-state model and why it helps

I favor a three-state model: init, send_success, send_fail. This handles races and interruptions cleanly:

• init: just-written, ready to be claimed
• send_success: published — safe to delete or archive
• send_fail: permanent or retryable failures; surface to DLQ or operator

If a worker crashes while publishing, the row remains init (or is rolled back to init) and another worker will retry. If publishing succeeds but marking the row fails, you get a harmless at-least-once re-publish; consumers must deduplicate.

CDC vs polling relayer: pick based on scale and ops appetite

Polling relayer (FOR UPDATE SKIP LOCKED)

• Pros: simple, dependency-free, easy to reason about
• Cons: adds read load to the primary DB and has poll interval latency

CDC (Debezium → Kafka)

• Pros: near real-time, low DB load, scalable
• Cons: operational overhead (replication slots, WAL retention, connector management)

If polling becomes expensive, offload to CDC. Debezium's EventRouter SMT can convert outbox rows into clean broker messages while honoring aggregate_id as the message key (preserving per-aggregate order).

Consumer idempotency (the other half of correctness)

The outbox pattern gives you at-least-once delivery. Exactly-once side effects still live on the consumer side. The recommended approach is a consumer-side inbox / processed-events ledger keyed by event_id:

1. Consumer reads message.
2. Within a DB transaction: INSERT INTO processed_events(event_id) ON CONFLICT DO NOTHING; then if inserted, apply side effect(s) and commit.
3. If the insert did not affect a row, skip the side effect — it's a duplicate.

This guarantees that consumers never apply the same business change twice, even if messages are redelivered.

Observability and housekeeping

Key metrics to emit and alert on:

• Outbox lag: now() - min(created_at WHERE status='init')
• Counts: init / send_success / send_fail
• Stuck rows: rows pending > X minutes
• CDC replication lag (Debezium connector metrics)

Cleanup strategies:

• Periodic batched delete of sent rows (DELETE ... LIMIT 10000) or partitioning the outbox by date and drop old partitions.
• If using CDC, ensure you tune WAL retention and monitor replication slot lag so WAL doesn't pile up.

Failure modes you still need to accept

• At-least-once delivery = duplicates are possible. Consumer idempotency is required.
• Per-aggregate ordering is preserved only if you use aggregate_id as message key/partition key. Global ordering across aggregates is not guaranteed.
• Relayer latency creates a small window between DB commit and event arrival — surface that in SLAs.

Conclusion — low-risk, high-reward

The transactional outbox pattern doesn't eliminate complexity, but it shifts the most dangerous failure mode — lost events — off the request path and into a manageable, observable pipeline. For services where downstream consistency matters (billing, fulfillment, inventory), the pattern is low-risk and high-reward: you get atomicity for free using your existing DB.

If you're shipping events other systems depend on, start with an outbox table, a SKIP LOCKED relay, idempotent consumers, and a couple of dashboards. How do you ensure events never vanish in your system?