Transactional Outbox Pattern: Solving the Dual-Write Problem with Atomicity
When building distributed systems, we often need to perform multiple operations as part of a single business action.
Imagine you're developing an e-commerce application using Node.js, PostgreSQL, and Kafka.
When a customer creates an order, your application needs to:
- Save the order in PostgreSQL.
- Publish an
OrderCreatedevent to Kafka so other services can react.
Sounds simple, right?
async function createOrder(orderData) {
const order = await db.orders.create(orderData);
await kafka.publish("OrderCreated", {
orderId: order.id
});
return order;
}
But what happens if PostgreSQL successfully creates the order and Kafka becomes unavailable?
We now have an order in our database, but the other services don't know about it.
This is where things start getting interesting.
Understanding the Dual-Write Problem
The dual-write problem happens when an application needs to write to two independent systems as part of the same logical operation.
In our example:
- PostgreSQL stores the order.
- Kafka distributes the event.
The problem is that these systems don't share the same transaction.
Let's consider two scenarios.
Scenario 1: Database first
Create order in PostgreSQL ✅
Publish event to Kafka ❌
The order exists, but the event was never published.
Scenario 2: Kafka first
Publish event to Kafka ✅
Create order in PostgreSQL ❌
Now other services might receive an event about an order that doesn't exist.
Reversing the operations doesn't solve the problem. It just changes which inconsistency we might encounter.
So how do we guarantee that our business operation and the intent to publish its event succeed or fail together?
Before answering that, we need to understand atomicity.
What Is Atomicity?
Atomicity is one of the four ACID properties of database transactions:
- Atomicity: All operations within a transaction succeed, or none of them do.
- Consistency: Transactions preserve the database's integrity constraints.
- Isolation: Concurrent transactions follow defined rules about how their operations interact.
- Durability: Once committed, changes survive failures.
Let's focus on atomicity.
Imagine transferring $100 between two bank accounts.
Account A: -$100
Account B: +$100
Both operations belong to the same logical transaction.
We cannot accept a situation where Account A loses $100 but Account B never receives it.
That's why databases support transactions:
BEGIN;
UPDATE accounts
SET balance = balance - 100
WHERE id = 1;
UPDATE accounts
SET balance = balance + 100
WHERE id = 2;
COMMIT;
If an operation fails before the transaction commits, the database can roll back the entire transaction.
But here's the catch:
A PostgreSQL transaction cannot normally control a Kafka publication.
Even if we write:
await db.transaction(async (tx) => {
await tx.orders.create(orderData);
await kafka.publish("OrderCreated", event);
});
Kafka is still an independent system.
If Kafka accepts the event but the database transaction later fails to commit, we have an inconsistent state.
We need another approach.
Introducing the Transactional Outbox Pattern
The Transactional Outbox Pattern solves this problem with a simple idea:
Instead of publishing the event directly to Kafka, persist the event in the same database transaction as the business operation.
We introduce a new table called outbox.
Our transaction becomes:
BEGIN;
INSERT INTO orders (id, status)
VALUES (42, 'CREATED');
INSERT INTO outbox (
id,
event_type,
payload
)
VALUES (
'evt-123',
'OrderCreated',
'{"orderId": 42}'
);
COMMIT;
Now both operations happen inside PostgreSQL.
That means we have two possible outcomes:
Order created ✅
Outbox event created ✅
Or:
Order created ❌
Outbox event created ❌
We no longer have a committed order without its corresponding outbox record.
That's the fundamental benefit of the pattern.
Notice that we didn't make PostgreSQL and Kafka atomic.
Instead, we moved the atomicity boundary to something PostgreSQL can actually guarantee.
But How Does the Event Reach Kafka?
Good question!
We introduce another component, usually called an Outbox Relay or Outbox Publisher.
Its responsibility is to read unpublished events from the outbox table and forward them to our messaging system.
Our architecture becomes:
Node.js Application
|
v
PostgreSQL Transaction
/ \
v v
Orders Table Outbox Table
|
v
Outbox Relay
|
v
Kafka
|
v
Other Services
A simplified relay could look like this:
async function publishPendingEvents() {
const events = await db.outbox.findPending();
for (const event of events) {
await kafka.publish(event.event_type, event.payload);
await db.outbox.markAsPublished(event.id);
}
}
This is intentionally simplified. A production implementation needs to handle concurrent workers, failures, retries, and message ordering.
The important part is that the event is persisted before we attempt to publish it.
If Kafka becomes unavailable, the event remains in our database and can be retried later.
Polling vs. Event-Driven Relays
A common implementation uses a background worker that periodically queries the outbox table.
For example:
SELECT *
FROM outbox
WHERE published_at IS NULL;
But what if we don't want to wait for the next polling interval?
With PostgreSQL, we can use LISTEN/NOTIFY.
When an outbox event is inserted, the database can emit a notification to wake up the relay.
The relay then queries the outbox and publishes pending events.
However, there's an important distinction:
The notification is not the event itself. It's just a signal that new work might be available.
Notifications aren't a durable queue. If the relay is disconnected, it may miss them.
That's why the outbox table remains our source of truth, and the relay should also query for pending events on startup, reconnection, and periodically.
Another approach is Change Data Capture (CDC), using tools such as Debezium to observe committed database changes and forward outbox events without application-level polling.
What About Duplicate Events?
Here's another interesting problem.
Imagine the relay performs these operations:
1. Read event from outbox ✅
2. Publish event to Kafka ✅
3. Mark event as published ❌
What if the relay crashes immediately after Kafka accepts the message?
The event was published, but the database still considers it pending.
When the relay restarts, it might publish the same event again.
This is why the Transactional Outbox Pattern commonly provides at-least-once delivery, assuming the relay continues retrying and the underlying infrastructure eventually recovers.
That means an event can be delivered more than once.
To handle this safely, consumers should be idempotent.
For example, a consumer can track processed event IDs and avoid applying the same business operation twice.
Receive evt-123
|
v
Already processed?
/ \
Yes No
| |
Ignore Process
In practice, the consumer should atomically record the event ID and apply its business changes, so concurrent duplicate deliveries don't create duplicate effects.
What About Multiple Relay Workers?
As our system grows, we might want multiple relay workers processing events concurrently.
But imagine two workers querying the same pending event.
Worker A → evt-123
Worker B → evt-123
Without coordination, both workers might publish it.
PostgreSQL provides a useful mechanism called FOR UPDATE SKIP LOCKED.
SELECT *
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at, id
LIMIT 100
FOR UPDATE SKIP LOCKED;
When used inside a transaction, this allows workers to lock available rows while skipping rows locked by other transactions.
This helps distribute work across workers.
For longer publishing operations, we can instead claim rows in a short transaction and use leases or timeouts to recover work from crashed workers.
There's also message ordering to consider. If OrderCreated and OrderPaid belong to the same order, we may need to preserve their relative order, even while processing different orders concurrently.
These are important implementation decisions, but they don't change the fundamental purpose of the pattern.
When Should We Use Transactional Outbox?
Transactional Outbox is especially useful when:
- A business database change must reliably produce an event.
- Losing an event could leave other services inconsistent.
- We need reliable asynchronous communication between services.
- We want to tolerate temporary messaging infrastructure failures.
However, the pattern also introduces additional complexity: outbox storage, relays, retries, monitoring, cleanup, idempotency, and potentially ordering guarantees.
Not every event requires that complexity.
For example, losing an occasional non-critical analytics event may be acceptable.
As with most architectural patterns, the decision should depend on the business requirements.
Final Thoughts
The biggest lesson I've learned about the Transactional Outbox Pattern is that we don't need to make every component of a distributed system participate in one giant transaction.
Instead, we can use the atomicity guarantees of a local database transaction to persist both our business changes and the intent to publish an event.
Then we rely on asynchronous delivery, retries, and idempotent consumers to propagate that change safely.
The mental model I like is:
Atomic locally.
Eventually propagated globally.
Duplicates handled safely.
Transactional Outbox doesn't eliminate every challenge in distributed systems, but it gives us a reliable foundation for handling one of the most common problems: keeping business state and event publication consistent.
And sometimes, the most valuable architectural patterns aren't about introducing something complex. They're about moving the problem into a boundary where we already have strong guarantees.
Top comments (1)
The polling-publisher variant is underrated for smaller systems: one row written in the same transaction, a relay tailing it, and Kafka downtime just delays events instead of losing them. Per-aggregate ordering is the detail most first implementations get wrong.