Distributed transactions: why BEGIN/COMMIT can't save you across services
Someone splits the monolith into services. Yesterday, orders, payments, and inventory all lived in one database. You wrapped everything in a transaction, committed, done. ACID handled the rest.
Now the order lives in one database, the payment in another, inventory in a third. You still write BEGIN and COMMIT. But there's no shared transaction manager anymore. That COMMIT only applies to the database you're talking to right now. The other two? They don't know. They don't care.
This is where things get ugly.
🛠️ The dual-write problem
Here's the situation. Your Order Service needs to do two things when an order is created: write to its database, and publish an event so downstream services (payment, inventory, notifications) can react. Two systems. Two writes. No shared transaction between them.
Three ways this blows up:
DB commits, event publish fails. Your database has the order. But the message broker never got the event. Payment never gets charged. Inventory never gets reserved. The systems silently diverge and nobody notices until a customer complains.
Event publishes, DB rolls back. The broker accepted your event. Downstream services start processing. But your database transaction failed. Now payment is charging for an order that doesn't exist.
Process crashes between the two writes. One write landed. Which one? Depends on the order you wrote them. Either way, you're inconsistent.
No retry logic fixes this. Retrying the publish doesn't help if the DB already rolled back. Retrying the DB write doesn't help if the event already fired. The problem isn't failure handling. The problem is that two independent systems can't commit atomically.
🧠 Two-phase commit: the textbook answer nobody uses
The academic solution is two-phase commit (2PC). A coordinator asks every participant "can you commit?" in Phase 1. Each participant acquires locks, writes to durable storage, and votes YES or NO. In Phase 2, if everyone voted YES, the coordinator sends COMMIT. Otherwise, ABORT.
Sounds clean. In practice, teams avoid it for good reasons.
It blocks. If the coordinator crashes between phases, every participant that voted YES holds its locks indefinitely. They can't commit (they don't know the decision). They can't abort (maybe the coordinator will come back and say COMMIT). They just wait. With locks held. Blocking every other transaction that touches those rows.
It requires XA. Every participant needs to implement the XA interface (prepare, commit, rollback). Your PostgreSQL supports it. Your message broker probably doesn't. Your managed cloud datastore? Almost certainly not. So you can't even use 2PC across the systems you actually have.
It kills availability. All participants must be online simultaneously. One slow service and the entire transaction is stuck. In a distributed system where partial failures are the norm, this is a non-starter.
So 2PC works in theory and in tightly controlled environments (like a single vendor database cluster). But for services communicating over a network? Not practical. 3PC exists as an academic improvement that adds a pre-commit phase to reduce blocking, but it still can't handle network partitions and almost nobody implements it.
⚡ The transactional outbox
Here's the trick. You can't atomically write to a database and a message broker. But you can atomically write to a database twice. Same database, same transaction.
Instead of publishing the event directly, you write it to an outbox table inside the same transaction as your business data:
BEGIN TRANSACTION;
INSERT INTO orders (id, customer_id, status)
VALUES ('ord-123', 'cust-1', 'PENDING');
INSERT INTO outbox (id, aggregate_id, event_type, payload, created_at)
VALUES (gen_random_uuid(), 'ord-123', 'OrderCreated',
'{"orderId":"ord-123","customerId":"cust-1"}', NOW());
COMMIT;
One transaction. Both writes succeed or both fail. No dual-write problem.
A separate relay process picks up committed outbox rows and publishes them to your broker, Kafka, SQS, whatever you're using. The relay can poll the table on an interval, or you can use change data capture (CDC) to tail the database's transaction log directly.
If the relay crashes mid-publish, it restarts and re-publishes. This means consumers might see the same event twice. That's fine. They need to be idempotent anyway (a topic for its own post).
Simple pattern. Genuinely reliable. And it works with whatever broker you already have.
Sagas and compensating actions
The outbox solves "write + publish" atomicity. But what about operations that actually span multiple services? An order that needs to reserve inventory and charge payment and confirm the order?
That's where sagas come in. A saga is a sequence of local transactions, each in its own service. Each step publishes an event or command that triggers the next step. No global transaction. No distributed locks.
CreateOrder saga (orchestrated):
1. OrderService.createOrder(PENDING)
→ on failure: OrderService.rejectOrder()
2. PaymentService.authorizePayment()
→ on failure: PaymentService.refundPayment()
3. InventoryService.reserveStock()
→ on failure: InventoryService.releaseReservation()
4. OrderService.confirmOrder(CONFIRMED)
→ terminal step, no compensation needed
If Step 3 fails:
→ run compensate(Step 2): refund the payment
→ run compensate(Step 1): reject the order
Two flavors exist. Choreography: services react to each other's events with no coordinator. Works well for 2-3 steps, gets tangled fast after that. Orchestration: a central saga orchestrator sends commands and decides what to do next. Easier to trace and reason about.
But here's the thing people miss about sagas. Compensation is not rollback.
A database rollback erases the write. It never happened. A compensating action is a new forward operation. A refund is not an un-charge — the money moved to the merchant, now it moves back. There are processing fees, accounting entries, customer notifications. It takes time. The original charge still shows up in logs.
You have to design compensating actions explicitly for every step. And they have their own failure modes. What if the refund fails? Now you need retry logic for your compensations too.
I'll write a full saga implementation walkthrough in a future post. For now, the key insight: sagas give you eventual consistency across services, but they trade the simplicity of ACID for a lot of explicit failure handling.
The best first move is often the simplest
Before you reach for outbox patterns or saga orchestrators, ask yourself: does this operation actually need to span multiple services?
If Order Service and Payment Service always transact together, maybe they shouldn't be separate services. Maybe you split too early. A single database transaction is simpler, faster, and more reliable than any distributed pattern.
Redraw your service boundaries so the transaction stays local. That's not a failure of architecture. That's good design.
When you genuinely need cross-service coordination:
- Outbox pattern for reliable event publishing from a single service
- Saga when the operation truly spans multiple autonomous services with independent datastores
And whatever pattern you pick, idempotent consumers aren't optional. Every message will be delivered at least once. Design for it.
📌 Quick reference
- Dual-write problem: you can't atomically write to two systems without a shared transaction
- 2PC: blocks on coordinator failure, requires XA, reduces availability. Avoid for service-to-service coordination
- Transactional outbox: write the event to the same DB transaction as the business data; relay publishes it later
- Sagas: sequence of local transactions with compensating actions on failure. Compensation is a new operation, not a rollback
- Best first move: keep the transaction local by redrawing service boundaries
If you're working with event-driven services, the post on Kafka partitions and consumer groups covers how the messaging layer handles ordering and parallelism. And for routing requests to the right service in the first place, see API gateways.
Where else to find me
My writing and side projects all live at arnavsharma.dev.
Top comments (0)