The notification that fired before the rollback

Here is a bug that passes every test you'll write for it. A customer's order is confirmed, the restaurant gets a "new order" ping, and a moment later the transaction that confirmed it rolls back. The restaurant is now cooking food for an order that doesn't exist. Nothing crashed. Two things that were supposed to happen together simply didn't. Building Saturdays, a food delivery platform where one order is shared by a customer, a restaurant, a rider and the platform itself, this was the class of bug I most wanted to make impossible.

Two writes pretending to be one

The naive version looks correct:

with transaction.atomic():
    order.confirm()
    order.save()
    notify_restaurant(order)   # HTTP call, queue publish, websocket push...

It reads like one unit of work. It isn't. The database write is transactional; the notification is not. They fail independently, and every combination is a production incident:

• Notify succeeds, commit fails. The restaurant hears about an order the database never kept.
• Commit succeeds, notify fails. The order is confirmed and nobody who has to act on it knows.
• Notify fires while the transaction is still open. A consumer reacting to the event reads the row and sees the old state.

Moving the call after the atomic() block fixes the first case and leaves the second fully in place. There is no ordering of two independent operations that makes them atomic. You have to turn them into one.

Write the intent, not the effect

The transactional outbox does exactly that. Instead of sending the event, you record it — in a table, in the same transaction as the state change that caused it:

with transaction.atomic():
    order.confirm()
    order.save()
    OutboxEvent.objects.create(
        kind="order.confirmed",
        payload={"order": str(order.public_id)},
    )

Now the question "did we tell the restaurant?" has the same answer as "did the order get confirmed?" — because both are rows in one commit. Either both exist or neither does.

A separate worker drains the table and does the actual delivery. In Saturdays that's a scheduled Celery task, and it's where all the unglamorous reliability lives:

• Attempt counts. Every delivery try is recorded, so a flaky receiver is visible instead of silent.
• Backoff. A failing destination is retried on a widening interval, not hammered in a loop.
• Dead-lettering. After enough failures the event is parked for a human — not dropped, and not retried forever.

What you're really buying

The outbox doesn't make delivery instant, and it doesn't make it exactly-once. What it gives you is a guarantee that matters more: no event is ever sent for a state that didn't commit, and no committed state change is ever silently left unannounced. The cost is a few seconds of delay. The alternative is a class of inconsistency you can't reproduce on your laptop and can't explain to a restaurant owner.

It does push one responsibility onto the receiving side. Because a retry can deliver an event twice, consumers should be idempotent — handling the same order.confirmed a second time must be a no-op. That's a far easier property to build than distributed atomicity, and it's the right trade.

The takeaway

Any time you see a database write and a network call in the same function, ask what happens if exactly one of them succeeds. If the answer is "someone acts on something that isn't true," the call doesn't belong there. Record the intent inside the transaction; deliver it afterwards.

The rest of how Saturdays keeps four kinds of user in agreement about one order — the state machine, the payment path, the POS integration — is written up in the case study.

👉 Read it: Saturdays — Food Delivery Platform

---

Divyakush Punjabi — Full-Stack & AI Systems Engineer

🌐 https://www.divyakush.com · 💼 LinkedIn · 💻 GitHub