At 02:03:12, a sync script I'd written months earlier — it ran on a six-hour cycle and had been dependable up until that night — deleted a pile of private modules that were never committed anywhere. The log line was unremarkable: [02:03:12] sync start target=test-cluster. The rsync command had --delete, the exclusion list had gone stale, and everything not on that list — dozens of modules, some of them only ever existing on that one disk — was gone. Two of us spent the next two and a half hours reconstructing what we could, and we never got all of it back.
That incident was about a dumb file sync script. But it's the same failure mode I watch for in multi-agent fleets, and it's closer than most people think. When ten agents share the same MCP tools but each holds a different memory of what was already executed, the result is duplicated payments, overwritten records, and silent rollbacks. Nobody even disagrees; the mismatches just surface as weird one-off failures. The log looks fine. The damage is already done.
The part that took me three failed attempts and way too many late nights to learn is that consensus in a multi-agent fleet is a memory-safety problem. MCP gives agents tools and a context window, but no governance plane. Sessions are isolated; each agent's context window is its own private truth, and those truths drift. In my experience, most production fleet failures trace to memory divergence, not weak model reasoning. And no amount of prompt engineering fixes it. The fix has to be structural.
The first thing we tried was the obvious one: make the agents vote. Proposal goes out, agents respond, majority wins. It failed in the most boring way possible — overloaded or just plain agreeable agents said yes to everything. A majority of yeses just means nobody was paying attention.
So we stopped treating it like an election and started treating it like a ledger. Every irreversible action — external payment, data deletion, anything that produces a side effect we can't roll back — has to be written to an append-only log on the MCP memory server before it can execute. The proposing agent writes a proposal entry with a monotonic sequence number. Every peer that can see the log fetches that entry and appends ACK or NACK. The action proceeds only when a quorum of ACKs is observed. No entry in the log, no action. It sounds heavy, and it is. That's the point.
Here's what we got wrong the first time. We treated the quorum as a count, let agents ACK in whatever order they felt like. That turned the log into a tangle of out-of-order entries, and we couldn't tell whether an action had genuinely cleared quorum or had just collected ACKs from everyone who happened to be awake. The sequence number is what lets a peer reconstruct what actually happened. Without it you're back to guesswork. People skip it because it feels like overhead, and then spend a week chasing phantom mismatches.
The trade-off is real. Latency grows with fleet size, and a partitioned minority can stall an action indefinitely. For high-stakes tools, that's the correct price. If you're not willing to pay it, don't give the agent the tool.
Quorum alone still isn't safe, because a majority of peers can be wrong in the same way at the same time — that sync script's exclusion list was "up to date" for everyone who looked at it. So we designated a small set of governance agents and gave them a veto channel. If no veto arrives within a bounded TTL — we run 30 seconds — the quorum stands. That converts majority opinion into majority minus explicit objection.
The first version had a five-minute TTL, because we wanted to "give agents time to think." What actually happened: the fleet stalled for five minutes waiting for a veto that was never coming, and operators got paged for nothing. We cut it to 30 seconds and never looked back. If a governance agent has something to say, it says it fast.
The failure mode here is as predictable as it is annoying: the veto agent becomes a bottleneck. We track veto response time as a first-class metric, right next to quorum wait time. If you don't measure both, you won't know which one is eating your latency until it's already eating your latency.
Somewhere in the middle of this, we tried a global peer lock on shared memory. Deadlock factory. Agent A holds a lock and waits for agent B; agent B is looping on a bad tool result; the whole fleet is stuck behind a lock that is never released. The fix was leases: an agent that will mutate a shared namespace holds a lease for 10 seconds, renewable. If the agent dies or starts looping, the lease expires and peers proceed. Lease expiry is your recovery mechanism. We built release-of-lease into the tool layer so a crashed agent doesn't leave a zombie lease behind.
The lease duration turned out to be a bargaining chip. Too short, and agents are constantly renewing, which is its own kind of overhead. Too long, and a dead agent blocks mutations far longer than it should. We settled on 10 seconds as a default and made it configurable per tool, not globally — a payment mutation and a log-read mutation have very different blast radii.
For the low-stakes coordination memory — which search ran, which tool returned last, what the previous agent in the chain decided — we use CRDTs with last-writer-wins. The pragmatic route. You converge on some order, not necessarily the one you'd have chosen with perfect information. That's fine. The blast radius of a wrong which search ran is small; the blast radius of a wrong payment is not. Don't put the same machinery on both.
The lease renewal path is still shaky. If an agent's context window runs out mid-renewal, the lease expires and the action aborts. That's the safe failure mode, but it produces the most confusing errors in practice, because the agent that initiated the mutation doesn't know its lease died. It just sees a failed tool call.
The veto channel has a blind spot too. If a governance agent is down, the TTL still runs, and a missing veto and a considered veto look identical in the logs. We haven't found a way to tell them apart without adding a second acknowledgment round, and that doubles the latency. So we live with the ambiguity and make sure the veto agents are the most monitored processes in the fleet.
And then there's the organizational failure mode, which no protocol can fix. The veto TTL and the lease duration live in runtime configuration. People tune things. One afternoon, someone bumped the lease duration from 10 seconds to 60 because they saw renewals in the logs and figured it was noise. They weren't wrong that it was noise. They were wrong that it was harmless noise. It took a day and a near-miss with a stuck mutation to notice. If your governance parameters can be adjusted silently, the governance is decorative.
Back to the rsync. The sync script didn't fail. The exclusion list didn't fail. The 02:03:12 was just a clock. What failed was that everyone who looked at that exclusion list saw a current version of the truth. A fleet full of healthy nodes agreeing on a stale truth — that's the failure. If that sync script had had a dry-run gate — a pre-flight pass that simulated the rsync before running it and intercepted any deletion of files that currently exist on disk — the private modules would still be there. The command would have been marked blocked, and the alert would have fired before the damage. That's exactly what we added: --dry-run first, compare the deletion list against existing files, and if anything that exists today would be deleted, abort and mark it blocked.
It took us a while to get there. A dry-run gate first, then a second pair of eyes for deletions above a certain count, then a backup job that finally ran on the right schedule. None of it happened in one go. The dry-run gate caught the next incident. The review step caught the one after that. The backup job saved us the third time. Each layer was a reaction to a specific failure, and it had to be, because the failure modes kept changing.
If you're building agent federation on MCP today, think about what your equivalent of --delete is. For most of you it's a payment call, a record deletion, a write that overwrites something — the tool that can't be undone. Give it a log that every peer can verify. Give it a veto that has a deadline. Give it a lease, not a lock. And measure how long those mechanisms take to act, because when something goes wrong, you'll want to know exactly how much time you have before the damage is done.
If I had to start over, here's the shortcut: clone a state you can afford to lose, let your exclusion list go stale on purpose, and run the thing that used --delete. Watch whether your log, your veto, and your lease catch it before it burns. If they do, good. If they don't, you just saved yourself a 02:03:12.
The good news is you don't have to get it right all at once. We didn't. We got a little less wrong, one incident at a time. That's the honest version of battle-tested — scar tissue, not a badge.
Top comments (0)