Hello, I'm Maneshwar, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Your database has one primary and two replicas, and for about two years that is the best decision you ever made.
Reads scale out across the replicas. Writes all go to the one primary.
There is exactly one copy of the truth, and it lives on a machine you could point at.
Then sales closes a deal in Sydney.
Physics sends you an invoice
Your primary is in Virginia. Your new users are in Sydney.
Every write from Sydney now has to cross the Pacific, get acknowledged, and come back.
That is roughly a 200 millisecond round trip, and no amount of clever indexing touches it, because the delay is not your database being slow.
It is light being inconveniently finite.
Two hundred milliseconds does not sound fatal until you notice how many writes hide inside one user action.
A session token. An analytics event. A "last seen" bump. Suddenly a page that feels instant in Boston feels like dialup in Sydney.
And then there is the other problem, the one that shows up on a Saturday.
If us-east-1 has a bad day, nobody writes anything. Not Sydney, not Boston, not the replicas sitting right there with a full copy of the data.
They have the bytes. They are just not allowed to touch them.
One leader means one place to be slow and one place to fail.
So you give each region its own leader
This is multi-leader replication, and the idea is exactly as blunt as it sounds.
Put a leader in each region. Let each one accept writes from the users nearest to it. Then have the leaders ship their changes to each other in the background.
Sydney writes land in Sydney at 8 milliseconds. Virginia writes land in Virginia at 8 milliseconds.
A few seconds later, each change has made its way to the other side of the world and both leaders hold the same data.
You just bought two genuinely good things.
Write latency collapses for everyone who is not sitting next to your original region. Writes are now local.
Losing a region is survivable. Sydney goes dark, Virginia keeps taking writes, and when Sydney comes back it catches up from the log.
Your write availability is no longer the uptime of a single building.
This is the same shape as single-leader replication, just with the "single" removed. Which is a sentence that should make you slightly nervous, and it should.
Because the thing that made single-leader so pleasant to reason about was never the leader. It was the one.
The bill arrives as a row number
Alan is in Sydney. He opens row 1,000,000 and sets x = 7.
At more or less the same moment, John is in Virginia. He opens the same row and sets x = 9.
Sydney accepts Alan's write and returns a 200. Virginia accepts John's write and returns a 200.
Neither leader hesitated, because at the moment each of them decided, the other write did not exist yet as far as they could tell.
Now the replication streams cross in the middle of the ocean, and both leaders simultaneously learn that they were lied to.
So what is x?
There is no answer to that question hiding in the data. The database has already promised both values to two different humans.
Somebody is about to be wrong and nobody has told them yet.
This is the problem with multi-leader replication. Not a problem. The problem. Everything else in this post is a response to it.
And notice what single-leader gave you for free here.
With one primary, these two writes would have arrived at the same machine, one would have gone second, and it would have overwritten the first while knowing it was overwriting something. A SELECT in between would have seen a consistent story.
That is not a feature anybody shipped. It is a side effect of there being one queue.
"At the same time" is the wrong mental model
It is worth being precise here, because this is where people's intuition quietly breaks.
Alan and John did not have to write at the same instant for this to happen. They could have been forty seconds apart.
Two writes are concurrent if neither one knew about the other. That is the entire definition.
It has nothing to do with wall clock time and everything to do with what information had reached whom.
Leslie Lamport worked this out in 1978 in Time, Clocks, and the Ordering of Events in a Distributed System, and it is still the single most useful thing you can read on the subject.
If Alan's write had reached Virginia before John started typing, John's write would be a normal overwrite. Not a conflict, just an edit.
Replication lag is what turns an edit into a conflict.
Which means your conflict rate is not a property of your users. It is a property of how long your replication takes, multiplied by how much your users overlap.
The default answer is "last write wins", and it is worse than it sounds
Most systems that offer multi-leader replication ship with a default conflict resolution policy, and that default is almost always last write wins.
DynamoDB global tables do it. Cassandra does it. It is the policy you get when nobody chose a policy.
Each write carries a timestamp. When two versions of a row collide, the one with the bigger timestamp survives. The other one is discarded.
Not flagged. Not queued for review. Discarded.
Here is the part that gets people. LWW is a guaranteed data loss mechanism, and that is not a criticism, that is its job description.
If two writes conflict and you keep exactly one, you have deliberately thrown away a write that the database told a user it had accepted.
Sometimes that is completely fine. If the field is "last known GPS position", dropping the older one is correct and you should feel nothing.
If the field is "account balance", you have just invented a very exciting kind of bug.
And then there is the detail that turns a tolerable policy into a haunted one.
Whose clock?
LWW compares timestamps generated by two different machines, and those machines do not agree on what time it is.
NTP keeps them roughly in sync, where "roughly" means tens of milliseconds on a good day and considerably worse on a VM whose host is under load or whose clock just stepped.
So if Sydney's clock has drifted 120 milliseconds ahead, Alan's earlier write gets stamped with a later timestamp.
LWW dutifully keeps it and throws away John's write, which actually happened second.
The system did exactly what it was configured to do. It also reverted a value, told the user it had succeeded, and left no trace anywhere that it happened.
This is the reason Google built TrueTime for Spanner, with GPS receivers and atomic clocks in every datacenter, and then had Spanner deliberately wait out the clock uncertainty interval before committing.
They did not solve clock skew. They bounded it, paid for the bound in hardware, and then paid for it again in commit latency.
If you are doing LWW on timestamps from now() across regions, you are making the same correctness claim Spanner makes, with none of the equipment.
What you can do instead of picking a loser
The good news is that LWW is a default, not a law.
Once you accept that conflicts are going to happen, you get to choose what happens next, and there are better options than a coin flip weighted by NTP drift.
Keep both and let the application decide. CouchDB does this properly: a conflicting document keeps every version as a sibling, picks one deterministically so reads do not break, and leaves the rest sitting there for you to resolve.
Riak called these siblings too. It is the most honest option, and it is the one that forces you to actually write resolution logic, which is why people avoid it.
Write a merge function. For a lot of real data, "pick a winner" is just the wrong operation. Two users adding different items to a shared cart is a union, not a race.
Two users editing different fields of the same profile row is two independent edits that a column-level merge handles without anyone losing anything.
Change the data type so conflicts cannot exist. This is the good one.
Replicate the operation, not the value
Here is the move that makes a whole class of conflicts evaporate.
A conflict happens because two leaders made competing claims about what a value is. Sydney said the cart has 4 items.
Virginia said the cart has 4 items. They both read 3 and both added 1.
Last write wins picks one of those two 4s, and the answer is 4. The customer added two things and the cart grew by one.
Now watch what happens if the leaders stop shipping the result and start shipping the operation.
Sydney does not say "the cart is 4". It says "Sydney has contributed 1". Virginia says "Virginia has contributed 1".
The value of the cart is the sum of everyone's contributions, and both leaders compute 5 without ever disagreeing about anything.
Nothing had to win, because the two writes were never competing claims in the first place.
That is a CRDT, and the counter version of it is so simple you can build one in plain SQL with no special database at all:
-- one row per (cart, replica). each leader only ever touches its own row,
-- so two leaders can never write the same row and never conflict.
CREATE TABLE cart_counter (
cart_id bigint NOT NULL,
replica text NOT NULL, -- 'syd' | 'iad' | 'fra'
delta bigint NOT NULL DEFAULT 0,
PRIMARY KEY (cart_id, replica)
);
-- Sydney's leader, adding an item. it writes replica = 'syd', always.
INSERT INTO cart_counter (cart_id, replica, delta)
VALUES (1841, 'syd', 1)
ON CONFLICT (cart_id, replica)
DO UPDATE SET delta = cart_counter.delta + 1;
-- the actual value, which every region computes identically
SELECT sum(delta) FROM cart_counter WHERE cart_id = 1841;
The write set of each leader is disjoint by construction. Replication can arrive in any order, duplicated, delayed by a week, and every region still converges on the same number.
This is the same principle behind Automerge and Yjs, which is how collaborative editors let twelve people type into one paragraph without a central referee.
And it is worth noticing that a Google Doc or a Figma file is a multi-leader system. Every laptop is a leader.
The offline mode you shipped last quarter made your mobile app one too.
The catch is that CRDTs are not free. They carry per-replica metadata, which grows, and they are only convenient for data types somebody has already worked out the merge semantics for: counters, sets, registers, sequences, maps.
The moment your business rule is "the total must never exceed the credit limit", you are back to needing coordination, because that is a global invariant and CRDTs specifically do not give you those.
The conflicts nobody warns you about
So far this has all been about one row, one column. The conflicts that actually ruin a migration weekend are the structural ones.
Auto-increment primary keys. Two leaders, both confidently handing out id 4817 to different rows. The fix is for each leader to generate ids from a disjoint space: offset sequences, UUIDs, or a Snowflake-style id with the region baked into the bits. If you are turning on a second leader and have not solved this, solve it first.
Unique constraints. Two users in two regions both register alan@example.com. Each leader checks its local table, sees nothing, and allows it.
The constraint held locally and is violated globally, which is a fun error to discover during replication rather than during signup. Uniqueness is a global invariant and asynchronous replication fundamentally cannot enforce one.
Foreign keys and deletes. Sydney deletes a parent row. Virginia inserts a child pointing at it. Both are valid locally.
Together they produce an orphan, and now your replication stream is stuck trying to apply something your schema refuses.
Triggers and defaults that call now(). Any logic that is not a pure function of the row will produce different results on different leaders. Replicate the computed value, not the computation.
Notice the pattern. Everything on that list is a constraint that spans rows your leaders do not coordinate on.
Single-leader hid all of them behind one serialisation point, for free, and you never knew they were being handled.
The strategy that actually ships
Here is what teams running this in production actually do, and it is less glamorous than any of the above.
They make the conflict impossible.
Give every record exactly one home leader that owns its writes. All writes for a given customer, tenant, or account are routed to that customer's home region, no matter where the request physically landed.
Every leader still replicates everything to everywhere, so reads stay local and failover stays possible.
Now the two leaders have disjoint write sets. Sydney writes rows Virginia never writes. There is no conflict to resolve because the two leaders are never making claims about the same data.
Conflict resolution logic you never run is the only kind that never has a bug.
This is why multi-leader works beautifully when your data has a natural ownership boundary. Per-customer SaaS. Per-tenant workspaces. Per-region marketplaces.
It works much less beautifully for a global leaderboard or a shared inventory count, where every write touches data everybody else touches too.
The routing itself is usually unexciting, which is the point:
def route_write(account_id, request_region):
home = accounts.home_region(account_id) # cached, changes rarely
if home == request_region:
return local_leader() # the fast path, ~8ms
# cross-region write: slower, but still exactly one leader owns this row,
# so it is an ordinary overwrite rather than a conflict waiting to converge
return leader_in(home)
Note what that else branch is buying you. A European user acting on a US-homed account pays the 100 millisecond hop.
You have traded some users' write latency for never having to explain to your CFO why a balance went backwards.
That is usually the right trade, and it is a trade, not a free lunch.
flowchart TD
A[Write arrives in Frankfurt] --> B{Which region<br/>owns this account?}
B -->|Frankfurt| C[Local leader, 8 ms]
B -->|Virginia| D[Forward to Virginia, 100 ms]
C --> E[Commit, ack the user]
D --> E
E --> F[Async replication to every other leader]
F --> G[All regions can serve reads<br/>and take over on failover]
B -->|Home region is down| H{Can you wait<br/>for it to come back?}
H -->|Yes| I[Reject the write, stay conflict-free]
H -->|No| J[Reassign the home, accept<br/>that you may now conflict]
classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
classDef warn fill:#ff9a5c,stroke:#c25f20,color:#1a1a1a
class A start
class B,H decision
class C,D,E,F,G good
class I,J warnThe honest caveat: homes move
Ownership routing is clean right up until the moment a region fails, which is the exact scenario you built multi-leader for in the first place.
Sydney goes down. Its accounts have no home.
You either reject their writes, which defeats the point, or you reassign their home to Virginia, which means accepting writes for rows Sydney might also have accepted writes for in the seconds before it died but not yet replicated.
That window is where your conflicts live. Not during normal operation, during failover.
So the real design is: disjoint write sets almost always, plus a conflict resolution policy you have genuinely thought about for the minutes per year when they are not disjoint.
Teams that skip the second half discover they needed it during an incident, which is the worst possible time to be reading about version vectors.
Two smaller things that bite, worth knowing before they bite you.
Reading your own writes gets harder. A user writes to their home leader in Virginia, then their next request hits a read replica in Frankfurt that is 400 milliseconds behind.
Their change is gone. You fix this by pinning a user's session to one region for a while after a write, or by having the client carry the version it last saw and the read wait for it.
Replication topology matters. All-to-all is the usual choice and is the most robust, but it lets messages overtake each other, so an update can arrive before the insert it depends on.
Circular and star topologies are cheaper on connections and much worse on failure, because one dead node breaks the chain. MySQL's circular replication is famous for exactly this.
So should you turn this on?
The uncomfortable truth is that most teams reaching for multi-leader replication do not need it.
If your problem is read latency, you want read replicas, and you already have them. Multi-leader only pays off when writes are the thing crossing an ocean.
If your problem is regional failover, a single leader with a well-rehearsed promotion runbook gets you most of the way there, with a fraction of the semantics to reason about.
flowchart TD
A[Users in multiple regions] --> B{Are WRITES the<br/>slow part?}
B -->|No, reads are| C[Read replicas per region<br/>Keep the single leader]
B -->|Yes| D{Does the data split<br/>cleanly by owner?}
D -->|Yes, per tenant or account| E[Multi-leader with home regions<br/>Disjoint write sets]
D -->|No, everyone writes everything| F{Can you model it<br/>as a CRDT?}
F -->|Yes, counters sets text| G[CRDTs, converge without picking]
F -->|No, you need real invariants| H[Distributed SQL<br/>Spanner, CockroachDB, YugabyteDB]
E --> I[Still need a policy for<br/>the failover window]
classDef decision fill:#f4d35e,stroke:#b8991f,color:#1a1a1a
classDef start fill:#e9ecef,stroke:#6c757d,color:#1a1a1a
classDef good fill:#5ee6c8,stroke:#1f9c86,color:#1a1a1a
classDef alt fill:#9d8cff,stroke:#5b4bcc,color:#1a1a1a
classDef warn fill:#ff9a5c,stroke:#c25f20,color:#1a1a1a
class A start
class B,D,F decision
class C,E good
class G,H alt
class I warnAnd if you genuinely need strong consistency across regions, that is what CockroachDB and Spanner are for.
They run consensus under the hood and charge you for it in write latency, openly, instead of handing you an asynchronous system that looks consistent until it is not.
That is the actual choice. Not "multi-leader or single-leader", but where you would like to pay.
Single-leader charges distant users on every write. Distributed SQL charges everyone a smaller amount on every write.
Multi-leader charges nobody anything on the happy path and sends you one enormous invoice on the day two regions disagree about a row.
Pick the bill you would rather receive at 3am.
The one-line version
Multi-leader replication does not create conflicts.
It reveals that your data model always had them, and that a single primary was quietly doing your reconciliation for free.
The moment you add a second leader, that work moves to you.
You can do it with a timestamp comparison and hope, or with disjoint write sets and intent.
Only one of those lets you sleep.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production secure and reliable without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)