Imagine you’re building an e-commerce checkout. A customer clicks “Buy.” Two things need to happen: their balance has to go down, and the item’s stock has to go down. If the money moves but the stock doesn’t, you’ve charged someone for nothing. If the stock moves but the money doesn’t, you’ve promised someone a product you can’t deliver.
In a single database, you’d wrap both updates in one transaction and move on. But the moment those two pieces of data live on different machines, which is common in microservice architectures, that guarantee disappears. You need a way for multiple machines to agree that a transaction either happened everywhere or nowhere.
The classic answer is Two-Phase Commit (2PC). It works, it’s been around since the late 1970s, and it’s still used in plenty of real systems. It also has a famous flaw: it can freeze. This post walks through how 2PC works, why it blocks, how Google Spanner solved that problem, and how you can run a working 2PC demo yourself.
Part 1: The Problem Is Atomicity Across Machines
Atomicity means a transaction is all-or-nothing. A single Postgres instance handles this for you. Once you split data across services, you have to coordinate.
Our running example is a purchase that touches two independent pieces of state:
- The money shard holds
users(id, name, balance). - The inventory shard holds
products(id, name, price, quantity).
Both have to change together or not at all.
Part 2: How Two-Phase Commit Works
2PC uses two roles:
- The Coordinator drives the transaction and makes the final decision.
- The Participants hold the data and can each vote on whether the transaction can proceed.
The protocol runs in two phases.
Phase 1: Prepare (voting). The coordinator asks each participant, “Can you commit this if I tell you to?” Each participant checks its constraints (enough balance, enough stock), takes locks, and writes its intent to durable storage. Then it votes YES or NO. A YES is a promise: “I will commit if asked, even if I crash and restart.”
Phase 2: Commit or Abort. If every vote is YES, the coordinator sends COMMIT to everyone. If any vote is NO, it sends ABORT. The decision goes to all participants.
The rule is simple: all YES means commit everywhere, and any NO means abort everywhere.
Part 3: The Architectural Trap: Why 2PC Blocks
Here’s where things get uncomfortable.
Suppose both participants vote YES. They’ve locked their rows and are waiting. Then the coordinator crashes right after collecting the votes, before it sends the COMMIT message to Participant B.
Participant B is now in a state called PREPARED. It has promised to commit, so it can’t abort on its own. Other participants might have already committed, so it can’t safely guess. It also can’t commit, because it doesn’t know whether the coordinator decided COMMIT or ABORT before it died.
So Participant B waits. It keeps its exclusive locks on the product row, and every other purchase touching that product queues up behind it.
This is the cost of blocking:
- Locks held hostage. A single dead coordinator can freeze rows across the system.
- Thread saturation. Requests pile up waiting on those locks, and connection pools fill.
- Cascading latency. Services that depend on the stuck rows slow down, then time out, then fail in turn.
Recovery usually needs a human to look at the coordinator’s logs and push the decision through by hand. In the middle of an incident, that’s a bad place to be.
Part 4: A Quick Detour: Consensus in 60 Seconds
Before we fix 2PC, we need a building block: consensus. You don’t need a PhD to understand the core idea.
The problem with the coordinator is that it’s a single machine. If it dies, its memory dies with it. The fix is to stop trusting one machine and instead replace it with a group of machines that agree on what happened, using an algorithm like Paxos or Raft.
A few terms are enough to follow the rest of this post:
- Leader and followers. One node in the group is the leader. It accepts writes and proposes decisions. The followers copy those decisions so they have the same history.
- The log. This is an append-only sequence of state changes. Every node that has the same log has the same state. The log is the source of truth.
- Quorum (N/2 + 1). A decision is official only when a majority of nodes have recorded it. Quorums stop split-brain problems: if the network splits, only one side can hold a majority, so only one side can make decisions.
With those pieces, you can replace a fragile single machine with a cluster that survives individual failures.
Part 5: How Google Spanner Fixed It
Spanner’s key idea is to make both the coordinator and the participants consensus groups instead of single machines.
The shift looks like this:
- Old model: One coordinator node. One participant node per shard.
- Spanner model: Each coordinator and each participant is a Paxos group with a leader and replicas.
Here’s what changes in practice. When the coordinator’s leader decides to commit, it writes that decision to its replicated log before telling any participant. The commit decision is now durable across a majority of replicas.
If the coordinator’s leader crashes mid-commit, the group elects a new leader. The new leader reads the log, sees the decision (or sees that the transaction was prepared but undecided), and finishes the job. No human has to dig through logs, and no one has to guess.
That’s why Spanner doesn’t block the way naive 2PC does. A single machine failure no longer strands a transaction. The protocol’s state survives because it lives in a replicated log, not in one process’s memory.
The tradeoff is cost. Every decision now requires a round of consensus, which adds latency. Spanner accepts that cost because the alternative is a system that freezes when a machine fails.
Part 6: Real-World Edge Cases and Gotchas
Even with consensus, there are things to watch for.
Participant failures. What if a participant’s leader crashes while its rows are locked in the PREPARED state? Its replicas hold the prepared record. A new leader takes over, reads that record, and asks the coordinator’s replicated log for the outcome. The lock is released once the decision is known.
Network partitions. Quorums handle splits. If the coordinator group loses contact with part of the network, the side without a majority can’t make decisions. It waits rather than guessing. This is the right behavior, but it means some transactions will wait during a partition instead of finishing.
The performance tax. 2PC adds network round trips. Every write transaction pays for both phases plus consensus. Spanner has a shortcut for the most common workload: read-only transactions don’t need 2PC at all. They read a consistent snapshot at a chosen timestamp. Spanner’s TrueTime API provides tightly bounded clock uncertainty, which lets it pick timestamps that are safe across machines without coordinating. For read-heavy traffic, that removes most of the overhead.
Conclusion: The Tradeoffs at a Glance
Here's how the three approaches compare:
| Naive 2PC | 3PC | Paxos-Backed 2PC (Spanner) | |
|---|---|---|---|
| Rounds of messages | 2 | 3 | 2 plus consensus |
| Blocks if coordinator dies | Yes | Mostly no, under strict assumptions | No (a new leader takes over) |
| Behavior during network partitions | Can block | Can produce wrong decisions | Waits safely for quorum |
| Complexity | Low | Medium | High |
| Typical use | Single-datacenter systems, careful ops | Rarely used in practice | Globally distributed databases |
Three-phase commit adds an extra round so participants can make progress after a coordinator crash. It works only if the network never partitions, which real networks do. That's why it's mostly a textbook idea.
If you remember one thing from this post, make it this:
Your coordinator's decision is data. Treat it like data: replicate it, persist it, and never let it live only in one process's memory.
2PC is not a bad protocol. It's a protocol that exposes a hard truth about distributed systems: agreement costs something, and the cheapest version breaks when you need it most. Understanding where it blocks is the first step toward designing systems that don't.
Code and crash-injection demos:
2PC Hands-On: E-Commerce Purchase (Go + Postgres)
Learn Two-Phase Commit (2PC) by running it. A purchase must update two logical shards atomically:
-
users(id, name, balance)— money shard (participant A) -
products(id, name, price, quantity)— inventory shard (participant B),price= cost per unit
POST /purchase runs a real 2PC coordinator with durable votes
(Postgres PREPARE TRANSACTION / COMMIT PREPARED / ROLLBACK PREPARED)
Structured logs (slog) narrate every phase so you can watch 2PC happen.
0. Mental model (60 seconds)
Client -> Coordinator -> Participant A (users) Participant B (products)
| | |
|--- BEGIN ---->|--- BEGIN ------------------->|
| Phase 1: PREPARE / VOTE |
| "can you commit if I ask you to?" |
|<-- YES/NO ----| <-- YES/NO -|
| Phase 2: COMMIT or ABORT (one decision, sent to ALL) |
|--- COMMIT ---->|--- COMMIT ----------------->|
Rules: all YES → COMMIT everywhere; any NO →…
Top comments (0)