DEV Community

Cover image for Dual write: the bug hiding between your database and your message broker
Dev May Cry
Dev May Cry

Posted on

Dual write: the bug hiding between your database and your message broker

You save an order to the database and then publish an order.created event. Two lines of code. What could go wrong?

DB::transaction(function () use ($order) {
    $order->save();
});

Kafka::publish('order.created', $order); // what if this fails?
Enter fullscreen mode Exit fullscreen mode

This is a dual write: one operation writing to two different systems (database + broker, database + cache, database + search index) with no transaction covering both.

Why it breaks:

  • Partial failure: the order is committed, but the publish fails (network blip, broker down, process killed mid-request). The order exists, and no downstream service ever hears about it.
  • Swapping the order doesn't help: publish first and you can end up with an event for an order that was never saved.
  • Race conditions: two concurrent requests can write in a different order to each system, leaving them out of sync.

It even hides in familiar patterns. Dispatching a queued job with afterCommit() in Laravel is still a dual write: if the process dies between the commit and the push to Redis, the job is gone.

The fix: Transactional Outbox

Write the event to an outbox table inside the same database transaction:

DB::transaction(function () use ($order) {
    $order->save();

    OutboxMessage::create([
        'topic'   => 'order.created',
        'payload' => $order->toJson(),
    ]);
});
Enter fullscreen mode Exit fullscreen mode

A separate worker reads pending outbox rows, publishes them, and marks them as sent. Now it's atomic: either both the order and the event are saved, or neither is.

One catch: the relay can publish a message and crash before marking it as sent, so delivery is at-least-once. Your consumers need to be idempotent.

Other options:

  • Change Data Capture (CDC): tools like Debezium read the database's transaction log and turn changes into events. The database stays the single source of truth.
  • Event sourcing: the event log is the source of truth, and everything else is a projection.
  • Two-phase commit: works in theory, but it's heavy, couples your services, and most brokers don't support it.

Ever been bitten by a dual write in production? How did you handle it?

Top comments (0)