A switch upgrade dropped the link between our two sites for a little under five minutes. Nothing crashed. Both halves of the system stayed up, kept serving, and kept writing. That was the problem: for those five minutes we had two primaries, and the reconciliation took the rest of the week.
Our failover logic was a health check with a timeout. The standby site could not reach the primary's database, concluded it was gone, promoted its own replica and started accepting writes. The primary could not reach the standby either, saw no reason to stop, and carried on. The load balancer in front of both had health checks that passed on each side, because each side was locally healthy. There was no third opinion anywhere in the design, and "I cannot see you" was treated as proof of "you are dead" by a system that had no way to distinguish a partition from a failure.
When the link came back we had two divergent write sets. Around nine hundred orders existed on one side and not the other. Sequence-generated invoice numbers overlapped, which meant two different customers held documents with the same identifier. We wrote a reconciliation script by hand, and the parts that could not be resolved automatically went to a spreadsheet and three people for two days.
The design fix is old and dull: promotion requires a quorum, so a minority partition cannot promote itself. We moved the decision to an etcd-backed manager with witnesses in a third location, and the losing side now fences itself, refusing writes rather than guessing. Fencing before promotion is the part people skip, and it is the part that makes the guarantee real.
We also made the data less catastrophic to merge. Identifiers that used to come from a per-site sequence now come from a scheme that cannot collide across sites. That alone turned the worst class of divergence from a manual investigation into an automatic merge.
And we test it. A quarterly game day drops the inter-site link on purpose and we watch which side stops. The first time we ran it, it stopped the wrong side.
– Sergey Shinder
Top comments (0)