The Outbox Pattern: Solving the Dual-Write Problem in Distributed Systems

Introduction

In a microservices architecture, a single business operation often needs to update a database and notify other services through a message broker, REST API, or another integration mechanism.

For example, when an order is created, the Order Service must save the order and publish an OrderCreated event.

The challenge is that these are two separate operations involving two different systems. The database update may succeed while event publishing fails—or the event may be published while the database transaction rolls back.

This is the dual-write problem.

The Outbox Pattern solves this by storing the business data and the event in the same database transaction, and publishing the event asynchronously from the outbox.

In this article, we'll see how the dual-write problem occurs, how the Outbox Pattern solves it, and how to handle retries, duplicate events, idempotency, and scalability.

Note: Throughout this article, we’ll use event publishing as the primary example for simplicity. The same underlying problem can occur whenever a database change needs to trigger an interaction with another system—for example, a REST API call, webhook, message broker, or other external operation. The examples will focus on events, but the core principles apply more broadly.

---

The Problem Statement

The Dual-Write Problem

Imagine an OrderService that, when an order is placed, must:

1. Insert a row into the orders table.
2. Publish an OrderCreated event so that InventoryService, NotificationService, and BillingService can react.

The naive implementation looks like this:

sequenceDiagram
    participant Client
    participant OrderService
    participant Database
    participant MessageBroker as Message Broker

    Client->>OrderService: POST /orders
    OrderService->>Database: INSERT INTO orders
    Database-->>OrderService: OK
    OrderService->>MessageBroker: publish(OrderCreated)
    MessageBroker-->>OrderService: ACK
    OrderService-->>Client: 201 Created

This looks fine — until you ask: what happens if step 2 succeeds but step 3 fails, or vice versa?

---

Failure Scenarios

The dual-write problem becomes clearer when we look at what can go wrong.

Scenario A — Database commit succeeds, but event publishing fails
The order is successfully saved in the database, but the broker is unavailable, the network fails, or the application crashes before publishing the event.
The order exists, but downstream services such as InventoryService and BillingService may never know about it.

Scenario B — Event publishing succeeds, but the database transaction rolls back
Downstream services may start processing an OrderCreated event even though the order was never successfully committed to the Order Service's database.

This can lead to inconsistent or invalid state in downstream services.
Now you have phantom orders in every consuming service.

Scenario C — The application crashes between the two operations
Even if you reorder operations (publish first, write DB second, or wrap both in a "try/catch and retry"), a crash between the two non-atomic operations always leaves a window where state is inconsistent. You cannot make two independent I/O calls to two independent systems (a database and a broker) atomic using application-level logic alone. There is no distributed transaction spanning both by default.

Scenario D — Retries can create duplicate events
To defend against Scenario A, an engineer adds a retry: "if publish fails, try again." But now imagine the publish actually succeeded, only the ACK was lost. The retry sends a second OrderCreated event. Now inventory gets reserved twice.

If the consumer is not idempotent, it could perform the same business operation twice—for example, attempting to reserve inventory twice.

This is why reliable event-driven systems typically combine the Outbox Pattern with retries and idempotent consumers.

---

flowchart LR
    A[Business Operation]

    A --> B[(Database)]
    A --> C[Message Broker]

    B -->|Write| D{Atomic?}
    C -->|Publish| D

    D -->|No| E[⚠️ Dual-Write Problem]

    E --> F[Database succeeds<br/>Event fails]
    E --> G[Event succeeds<br/>Database rolls back]
    E --> H[Retry / crash<br/>causes duplicates]

---

Where This Bites in Practice

The dual-write problem can appear anywhere a business operation must persist state and publish an event.

Scenario	What can go wrong without an Outbox
E-commerce checkout	Order is saved, but OrderCreated is not published → inventory may not be reserved and downstream services never learn about the order.
Payments	Payment is recorded, but the event sent to fraud detection is lost → the transaction may not be evaluated by the fraud service.
User signup	User is created, but UserRegistered is not published → the email or onboarding service never receives the event.
Saga / distributed workflows	A saga step commits its local state but fails to publish the event that triggers the next step → the workflow can get stuck waiting for an event that never arrives.
CQRS / read-model synchronization	The write-side database is updated, but the event used to update the read model is lost → the read model can become stale or inconsistent.
Audit / compliance	A state change is committed, but the corresponding audit event is lost → the audit trail may contain gaps.

Despite the different use cases, the underlying problem is the same:

A business operation changes local state and publishes an event, but those two actions cannot be committed atomically as part of the same local transaction.

That is the problem the Outbox Pattern is designed to address.

---

The Outbox Pattern — The Solution

Core Idea

Instead of writing to the database and separately publishing to the broker, you do only one atomic operation: write your business data and the event into the same local database, in the same ACID transaction.

A dedicated background process then reads that "outbox" table and reliably relays the events to the message broker, retrying as needed, after the transaction has safely committed.

flowchart LR
    subgraph "Step 1: Single Atomic Transaction"
        A[OrderService] -->|"BEGIN TX"| B[(orders table)]
        A -->|"same TX"| C[(outbox table)]
        B --> D["COMMIT"]
        C --> D
    end
    D --> E[Message Relay / Poller]
    E -->|"reads unpublished rows"| C
    E -->|"publishes"| F[Message Broker]
    F --> G[Inventory Service]
    F --> H[Billing Service]
    F --> I[Notification Service]
    E -->|"marks row as published"| C

Because orders and outbox live in the same database, a single local ACID transaction guarantees: if the order is saved, the outbox event is saved too — and if the transaction rolls back, neither exists. There is no window where one happens without the other.

The Outbox Table

A typical outbox table looks like this:

erDiagram
    OUTBOX_EVENT {
        bigint id PK
        string aggregate_type
        string aggregate_id
        string event_type
        json payload
        timestamp created_at
        timestamp published_at
    }

The important part is that the order record and the outbox record are created in the same database transaction.

Sequence Diagram: End-to-End Flow

sequenceDiagram
    participant Client
    participant OrderService
    participant DB as Database (orders + outbox)
    participant Relay as Message Relay
    participant Broker as Message Broker
    participant Consumer as Downstream Service

    Client->>OrderService: POST /orders
    OrderService->>DB: BEGIN TX
    OrderService->>DB: INSERT INTO orders
    OrderService->>DB: INSERT INTO outbox (event)
    OrderService->>DB: COMMIT
    DB-->>OrderService: OK
    OrderService-->>Client: 201 Created

    loop Poll / CDC stream
        Relay->>DB: SELECT unpublished outbox rows
        DB-->>Relay: rows
        Relay->>Broker: publish(event)
        Broker-->>Relay: ACK
        Relay->>DB: mark published
    end

    Broker->>Consumer: OrderCreated event
    Consumer->>Consumer: reserve inventory / charge / notify

Notice: the client gets a response the instant the local transaction commits. Publishing to the broker happens asynchronously and reliably in the background — the client is never blocked on the broker being available.

Two Ways to Implement the Relay

Once the transaction commits, we need a relay to move events from the outbox table to the message broker. There are two common approaches.

1. Polling Publisher
A background process periodically queries the outbox table for events that have not yet been published.

SELECT * FROM outbox_events WHERE published_at IS NULL ORDER BY created_at LIMIT 100;

This approach is simple and easy to implement, but frequent polling can add database load and introduce some publishing latency.

It is often a good choice when the event volume is moderate and simplicity is more important than achieving very low latency.

2. Change Data Capture (CDC) — e.g., Debezium
A CDC tool tails the database's transaction log (e.g., MySQL binlog, Postgres WAL) and streams outbox inserts directly to the broker in near real-time, without polling the table at all.

CDC can provide lower latency and better scalability than frequent polling, particularly for high-throughput systems. However, it also introduces additional infrastructure and operational complexity.

flowchart TD
    subgraph Polling Approach
        P1[Scheduled Job] -->|"SELECT ... WHERE published_at IS NULL"| P2[(Outbox Table)]
        P1 --> P3[Broker]
    end
    subgraph CDC Approach
        C1[Debezium / CDC Connector] -->|"tails WAL/binlog"| C2[(Outbox Table)]
        C1 --> C3[Broker]
    end

CDC is generally preferred in high-throughput systems: no polling overhead, lower latency, and it captures changes at the storage engine level so nothing is missed even under load.

---

How This Achieves Loose Coupling

The Outbox Pattern isn't just a reliability trick — it's structurally what makes event-driven microservices loosely coupled:

• Producer doesn't know who's listening. OrderService never calls InventoryService or BillingService directly. It just writes an event to its own outbox. Any number of new consumers can subscribe later with zero changes to OrderService.
• No synchronous dependency chain. If BillingService is down, OrderService is completely unaffected — it already committed its transaction and moved on. Compare this to direct service-to-service calls, where a downstream outage cascades upstream.
• Producer and consumer can evolve independently. As long as the event schema is respected (ideally versioned), each service can be deployed, scaled, and changed on its own timeline.
• Failure isolation. A bug or outage in the message broker or in a consumer doesn't threaten the producer's data integrity — the outbox table is durable, local, and transactionally consistent regardless of what's happening downstream.

flowchart LR
    O[OrderService] -->|writes event, no direct call| Outbox[(Outbox)]
    Outbox --> Relay
    Relay --> Broker((Broker))
    Broker --> S1[Inventory Service]
    Broker --> S2[Billing Service]
    Broker --> S3[Notification Service]
    Broker --> S4[Analytics Service - added later, zero changes upstream]

---

How This Achieves Scaling

• Producer scaling is decoupled from consumer scaling. OrderService throughput is governed only by its own DB, not by how fast BillingService can process events. Consumers can scale horizontally (more partitions, more consumer instances) independently, at their own pace.
• Backpressure absorption. If a downstream consumer is slow or temporarily down, events simply queue up in the broker/outbox — they aren't lost, and the producer keeps accepting new requests without slowing down.
• Horizontal scaling of the relay itself. Multiple relay instances can process the outbox table in parallel by sharding on aggregate_id, with row-level locking (SELECT ... FOR UPDATE SKIP LOCKED) to avoid double-publishing.
• CDC-based relays scale near-linearly with database throughput since they read the transaction log rather than executing repeated table scans.
• New consumers scale the system's capabilities, not its coupling. Adding a 5th, 10th, or 20th downstream service to react to OrderCreated costs nothing on the producer side — it's purely an additive subscription to the broker topic.

---

Trade-offs to Be Aware Of

No pattern is free. Be honest about these costs when adopting Outbox:

• At-least-once delivery, not exactly-once. A crash between publishing and marking a row as published can cause the same event to be sent twice. Consumers must be idempotent (e.g., dedupe using an event ID).
• Added latency. Events aren't published in the same millisecond as the DB write — there's a small delay (milliseconds with CDC, potentially seconds with polling).
• Extra infrastructure. You now need a relay process (or CDC pipeline like Debezium + Kafka Connect) to operate and monitor.
• Ordering guarantees need care. If ordering matters (e.g., events for the same order must be processed in sequence), you need to partition by aggregate_id so all events for the same entity go to the same broker partition/consumer.
• Outbox table growth. Published rows need to be archived or deleted periodically to avoid unbounded table growth.

---

Summary

Without Outbox	With Outbox
Two independent, non-atomic writes (DB + broker)	One atomic local transaction (DB + outbox row)
Silent event loss or duplication under failure	Reliable, at-least-once delivery via relay/CDC
Producer implicitly coupled to broker availability	Producer only depends on its own database
Hard to add new consumers safely	New consumers subscribe freely, no upstream changes
Scaling is tangled between producer and consumer	Producer and consumer scale independently

The Outbox Pattern converts a distributed atomicity problem (which is hard) into a local atomicity problem (which databases already solve well), then hands off reliable delivery to a purpose-built relay. That's the whole trick — and it's why it underpins so much of reliable event-driven architecture today.