Originally published on Loop & Retry — field notes on building LLM agents that survive production.
In distributed systems, split-brain is what happens when a network partition convinces two halves of a cluster that each is the sole leader. Both sides keep serving traffic. Both sides accept writes. Neither one is malfunctioning — each is behaving exactly as designed for the information it has, which is the problem, because the information each side has is now wrong. Reconciliation after the partition heals either loses data or needs a human to pick a winner.
Multi-agent systems have never had a real network partition, and most of them still build this exact topology. Give two or more agents read-write access to a "shared" memory — a blackboard, a scratchpad file, a vector store of learned facts, a task-state document — with no leader, no lock, and no version check, and you have the split-brain preconditions without needing a single dropped packet. The partition isn't in the network. It's in time: agent A reads the store, does five minutes of work, and writes back, while agent B read the same store at second three of those five minutes and is now reasoning over a version of reality A has already changed. That gap is enough. Nothing needs to break for split-brain to happen — everything needs to work at slightly different times.
This is the companion to failure modes in multi-agent teams: that post covered four ways a crew of agents breaks down — correlated collapse, diffused responsibility, context fragmentation, livelock. Split-brain is closely related to the third but sharper and more specific, and it earns its own post because it doesn't look like the others. Correlated collapse is agents converging on the same wrong answer. Split-brain is agents converging on different right answers — each one correct relative to the state it saw, incompatible with what another agent saw, and nobody notices because no single agent's transcript contains both halves of the contradiction.
"Shared" memory usually isn't
The word "shared" is doing a lot of unearned work. In almost every multi-agent framework, agents don't share memory the way threads share RAM with a cache-coherence protocol enforcing what "current" means. Each agent pulls a snapshot of the store into its own context window at read time, works from that snapshot for the rest of its turn or task, and writes back whatever it changed — usually without checking whether the store moved underneath it in the meantime. The context window is a cache, and a cache that's never invalidated on a write from somewhere else is exactly the setup that produces stale reads. Multiply that by N agents polling and writing on their own schedules and the "shared" store is really N loosely synchronized copies, reconciled by whichever write happens to land last.
A concrete failure
Three agents run an automated incident-response crew: a detector watching error rates, a remediator that applies fixes, and a verifier that confirms a fix actually worked. All three read and write a shared incident-state document — status: active | resolved, plus a timestamp.
The detector flags an error-rate spike and opens the incident at status: active. The remediator claims it, restarts the affected service, and writes status: resolved back to the store. Meanwhile the verifier had pulled its own copy of the incident document a few seconds earlier, before the remediator's write landed, and is now running its check loop against a local status: active that's already stale. The verifier's check comes back showing elevated error rates — the restart hasn't finished propagating yet — and it does exactly what it's supposed to do with the state it has: it re-escalates, paging the remediator to try again.
If the restart isn't safely repeatable — and most operations aren't idempotent by default — a second restart during an already-recovering rollout is its own disruption, one that looks to the detector like a new incident. Now you have three agents in a loop, each one locally correct, and a debugging session that starts by asking which agent is wrong. None of them are. They were never looking at the same variable at the same time, and nothing in any single transcript shows that, because the mismatch lives in the space between two agents' copies of one document, not inside either one.
Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article
Top comments (0)