Stale Memory, Stale Execution, and How Loom Tackles StaleBench

Thirty-six AI agents each finished a coding task. Thirty-six test runs came back green.

Then I merged each agent's work with the change another agent had made in parallel. Git reported no conflicts, not once.

I ran the tests again. Zero of thirty-six passed.

This post is about those thirty-six green lights that lied, the graph-backed coordination layer I built on FalkorDB to catch them, and the twelve cases where even that wasn't enough. Those twelve turned out to be the most interesting part. (The demo video is at the bottom, if you would rather watch first.)

The short version: with no warning, 0 of 36 agent pairs merged cleanly. With Loom, 24 of 36 did. The other 12 are the interesting part.

The bug that git cannot see

Here is the whole problem in two sentences.

Agent A renames a helper function and updates every place that calls it. Agent B, working at the same time in its own copy of the repo, writes a brand new function that calls the helper by its old name.

Each agent is correct. Each agent's tests pass. A merge produces no conflict, because the two agents never touched the same lines. And the merged code crashes, because B is calling a function that no longer exists.

I think of this as two different failures stacked together:

• Stale memory. B believes something that stopped being true: "the helper is called escape." It was true when B read the file. It isn't any more.
• Stale execution. B then acts on that belief. It writes code, confidently, on top of a fact that expired minutes ago.

People avoid this with a hallway conversation: "heads up, I'm renaming that." Parallel agents don't have a hallway. They have git, and git only understands text. It can tell you two agents edited the same line. It cannot tell you that one agent changed what a function means while another was busy depending on it.

Running more tests doesn't help either, because each agent's tests are green. The failure only exists in the combination.

Why this is the hard case, not an edge case

Anthropic's engineering team wrote up how they built Claude's multi-agent research system. They note that domains with many dependencies between agents are a poor fit for this style today, and that most coding tasks have fewer truly parallel pieces than research does. Coding is the awkward case, because the pieces are tangled together through the code they share (source).

That matches what I kept seeing. Parallel agents are great right up until two of them touch the same web of dependencies. So the question I wanted to answer was narrow and testable: can you keep parallel coding agents out of each other's way without making them take turns?

My answer was to give them the hallway. But it had to be a hallway that only speaks to the right person.

The idea: a shared map that talks

Loom keeps one live graph of the codebase and of everyone working in it, stored in FalkorDB.

• Every function is a node. "This function calls that one" is an edge.
• Every agent is a node too, linked to the functions it has read, edited or locked.
• When an agent edits a function, Loom records it as a change in the graph, right then.

Then comes the part that makes the graph worth having. Loom asks one question: who depends on what just changed? Here is the real query, from the daemon:

MATCH p = (c:Symbol)-[:CALLS*1..3]->(s:Symbol {id: $sid})
MATCH (b:Agent)-[r:READ|EDITED|LEASED]->(c)
WHERE b.id <> $agent
RETURN b.id AS agent, type(r) AS rel, length(p) AS hops,
       [n IN nodes(p) | n.qname] AS path

In plain words: walk up to three calls away from the changed function, and find every other agent who is reading, editing or holding a lock on something in that chain. A relational database would need a pile of recursive joins for this. In a graph it is one pattern.

Only those agents get a message, and it arrives while they are still working:

The message is plain text that appears in the agent's context, just after one of its tool calls. Agent C, which depends on something else entirely, hears nothing. This is a real notice that an agent received during the benchmark, trimmed for length:

[loom] agent A changed capfirst
  in django/utils/text.py
  (contract, in progress, not in your checkout yet)
  before: def capfirst(x)
  after:  def capfirst(x)
  body change:
    -     if not x: ...
    +     return _CoordT(_capfirst_impl(x))
  you depend on it: title_words → capfirst
  Both changes will be merged: make your code work
  with the AFTER version.

Look closely at the before and after lines. They are identical. The function's signature did not change at all. Only what it returns changed, from a string to a small wrapper object. Keep that detail in mind, because it comes back later.

(Loom also stops collisions before they start. An agent can take a lock on a single function instead of a whole file, and the check-and-grab is one atomic Cypher statement. In a stress test of 50 agents making 200 attempts each, there were no double grants. That deserves its own post, so I'll leave it there.)

How do you prove a thing like this works?

A demo where it works proves nothing. I wanted a number. Luckily, this exact failure already has a benchmark: STALE-bench, "Benchmarking Silent Semantic Merge Failures in Parallel LLM Agents", released by the illinoisdata group. The task pairs are theirs, not mine. Building valid pairs out of real Django history is the hard part, and I'm grateful they did it and published it. I'll call it StaleBench from here on, as the Loom repo does.

It runs on Django, a codebase large enough to be real. The tier I used has 12 real helper functions, each with three kinds of breaking change:

• rename: the function gets a new name
• signature: its arguments change
• return type: what it gives back changes shape

Twelve helpers times three changes is 36 pairs. In my harness, agent A makes the breaking change and agent B is a real Claude Haiku agent asked to write a new function using that helper. A is a scripted rewrite that I wrote, not a second LLM. That was deliberate: it makes A's change identical every time, so the only thing that varies is what B knows.

Each pair runs three ways:

1. Blind. B is told nothing. This is how parallel agents work today.
2. Oracle. B is told exactly what A changed, up front. This is the ceiling: the best any warning system could possibly do.
3. Live. B runs with Loom and gets a notice only when the graph says it is affected.

That is 36 × 3 = 108 real agent runs.

Each is graded offline twice: does B's code pass its tests alone, and does it still pass merged with A's change?

When the answer is "yes alone, no merged", I call it interference. It is the bug from the top of this post.

Most of the runs went beautifully

The blind and oracle runs were easy to read. Blind was exactly as bad as I feared: all 36 passed alone, none survived the merge. Oracle was perfect. When B knew what A had changed, all 36 merged cleanly, so the pairs were fair and solvable.

Then the live runs, the real test of the graph. The terminal scrolled, one finished run after another, each line a pair graded and recorded. It felt like it was working. About a hundred records landed.

Then this:

<pair id>  live  SKIPPED
  (RemoteDisconnected: Remote end closed connection
   without response)

And again. Four times. Four live pairs refused to run, and every one failed the same way.

The HTTP 500

I opened the daemon logs and found the cause was not in my code at all.

FalkorDB had refused a write. Under the hood it is Redis, and Redis has a safety rule: if its background snapshot fails, it stops accepting writes entirely. It does this to protect you from silently losing data. The daemon asked for a write, FalkorDB said no, and the daemon passed that failure along as an HTTP 500.

It got worse. A second Loom daemon from another clone of the repo was still running in the background on the same port, so for a while I was talking to the wrong program. I also had a relative file path that broke when the process changed directory.

Three unrelated bugs, one symptom. The fix was to tell Redis not to stop writes when a snapshot fails, kill the stray daemon, make the path absolute, and rerun only the four missing pairs. The benchmark has a resume flag for this, so nothing already finished was redone. I now start FalkorDB with that setting by default.

I tell this story for a reason beyond the war story. A coordination layer sits in the middle of every agent's work, so when it falls over, it takes the whole team with it. Reliability isn't a feature of a system like this. It is the system.

With all 108 records in, I read the table.

The result

Condition	Passes alone	Passes merged	Interference
Blind	36 / 36	0 / 36	100%
Oracle	1 / 36	36 / 36	0%
Live (Loom)	13 / 36	24 / 36	33%

Blind: perfect on its own, a total failure merged. Every agent that wrote correct code failed anyway.

Oracle looks odd at first, with only 1 of 36 passing alone. That is correct behavior. Those agents coded against A's new version, so their code can't work against the old one by itself. It only works once merged, which is the whole point.

Live is Loom: from 0 of 36 to 24 of 36 merged cleanly. Same model, same tasks, same prompts. The only addition was Loom: the notices, plus one wake-up nudge if a notice arrived after the agent had already finished. That closes two thirds of the distance between "blind" and the best possible score.

The notices were also not a firehose. Loom delivered 43 in total, and 34 of them (79%) were precise. I counted a notice as precise when it named the helper that pair really conflicts on. Every blind run broke, so a notice naming the right helper was always a correct warning.

Now break it down by kind of change:

Kind of change	Merged cleanly with Loom
Signature	12 / 12
Rename	9 / 12
Return type	3 / 12

Signatures were solved. Renames mostly. And then there was return type: three out of twelve.

The twelve that didn't work

My first guess was easy and flattering to the system: the notice must be getting lost, or the agent must be ignoring it.

Neither was true for most of them. In two of the twelve, B never received a notice at all. Those are real delivery misses and they count against Loom. But in the other ten, the notice arrived.

So I opened the transcripts.

Here is what Haiku wrote at the end of one of them, after being told that capfirst was about to start returning a wrapper object:

If it lands, " ".join(...) will raise a TypeError, because it expects strings. … title_words will need to unwrap each result (for example via .text) before joining. Tell me when it's in, and I'll make that adjustment.

Read that again. The agent understood the warning. It predicted the exact error that would happen. It even named the exact fix.

And then it did not apply it.

Another agent in the set wrote that it "did not write a workaround" because the notice only showed part of the new class, and a version that works with both would be a hack. A rename case ended with the agent offering two options and waiting: wait for the merge, or switch now and accept a broken checkout.

I read the closing message of each of the ten. In eight, the agent says outright that it will wait or asks which way to go. In the other two it either argues its code already works with either version, or asks me to confirm the final shape first. None of them applied the fix. This wasn't a memory failure. The agent wasn't blind to the change. It was careful. Told that the new code is "not in your checkout yet", it treated that as a reason to stop and ask, the way a good engineer would when a colleague's change hasn't landed.

That caution is the right call when you are alone in a repository. In a team of parallel agents there is no one to say "okay, it's in now." The agent that waits politely is an agent whose work will break at merge.

It also explains why return types fall so far behind. A rename gives the agent a clear next move: switch to the new name. A changed return type leaves the signature untouched, as in the notice above, and asks the agent to code against behavior it cannot run yet. Haiku wouldn't take that leap.

So the benchmark did something I didn't expect. I started out trying to fix stale memory, and Loom does fix it: the agent knew. What remains is stale execution, the gap between knowing and acting, and my best read is that this gap is about trust and nerve, not information.

What I'm still not sure about

Here is what these numbers can and can't support.

• One run per pair, so 36 samples per condition with no confidence intervals. Treat 24/36 as a strong signal and not a precise rate.
• One model, Haiku. A bigger model may be bolder, or more careful. I haven't tested it.
• One codebase, Django, and Python only. Other languages are on the roadmap below.
• The pairs are STALE-bench's, but the harness around them (scripted A, Haiku, the three conditions, the grading) is mine, not the authors' protocol. Don't compare my numbers with theirs.
• Agent A is a script. A real agent making a change halfway through its own reasoning would be messier.
• My claim about why the return-type runs stalled comes from reading a handful of transcripts, not from a controlled experiment. It is a well-supported hypothesis, not a result.

The benchmark and every record are in the repo so you can check, argue, or beat it. At list prices the full 108 runs cost about $0.58.

Where this goes next

The bug at the heart of this post is that agents carry a stale picture of a shared world. The usual answer to "agent memory" is to remember the past: logs, summaries, notes. What these agents were missing was the present tense: what is true right now, in the code we're all editing, that I don't know yet?

A graph is a good home for that because the useful answer is a relationship ("who depends on this, and who's touching it?"), not a document. The same structure already answers other questions I didn't have room for: who changed this code and why, and which tests have gone stale because the code they cover moved.

The next experiments follow directly from the twelve failures:

1. Change the wording. Tell the agent plainly that the other change will land, and give it the full new body instead of a fragment, so there is nothing left to wait for.
2. Close the delivery gaps. Two runs got no notice. That is the cheapest improvement available.
3. Try a bolder model, and a more careful one, and see which way the gap moves.

Those three are small. The larger plan has two parts, and I want to be straight that both are plans, not features.

Beyond Python

Today Loom's index reads Python only, through tree-sitter. That is why this post is about Django and why I list "Python only" above as a limit. It is a choice I made to ship in time, not something about the idea.

The idea needs one thing from a language: a reliable answer to "who calls this?" Typed languages give a much stronger one. TypeScript's compiler can resolve a call to the exact declaration it lands on, across files, through imports and re-exports, with the types attached. A tree-sitter pass gets the same shape from JavaScript, only with less certainty about which function a name refers to. So the plan is:

• TypeScript and JavaScript first. Use the compiler's own resolution for the call graph, so a changed return type can be flagged because the compiler can see it, not because a heuristic guessed.
• Then other typed languages with a compiler you can ask, such as Go, Rust and Java.
• Let the compiler be the check. In Python, the only way I know a change broke something is to run the tests. In a typed language, "does this still compile against the other agent's change?" is a question a machine can answer in seconds, before anyone merges anything. That could also help with the stale-execution gap, because an agent can try the new version in a scratch copy instead of waiting.

Beyond pairs

StaleBench uses two agents because two is the smallest case that shows the bug and the easiest to measure. Real work does not look like that. A bigger project has several agents on several branches at once, each on a different piece, with changes that overlap in ways nobody planned.

What I want to build next is a team of agents, each on its own branch, working on one complex project, with the graph as the shared picture between them. Each agent's changes would flow into the graph as they happen. Each branch's agent would hear about the changes that touch what it depends on, and only those. The same query that finds one affected reader today would find five of them, across branches.

That raises questions the pair benchmark cannot answer. What happens when three changes land on the same function? Who decides the order? Do notices pile up and drown the agent? And does a more capable model change how much of this it can handle? I would measure each of these the way I measured this one, with a benchmark published before the results.

If agents are going to work in parallel, they need more than a shared codebase. They need to know what's true in it right now, and permission to act on it. I think that is a solvable problem. On this benchmark, the data says we're two thirds of the way there.

Try it yourself

Here is Loom in action, if you would rather see it first:

git clone https://github.com/18Abhinav07/Loom.git
cd Loom
./scripts/demo.sh

That checks your setup, starts FalkorDB and the dashboard, and offers the live demo or the benchmark. The full steps are in localsetup.md, and the dashboard's Benchmark tab shows all 108 runs side by side.

My challenge: run your own parallel agents on your own repo in the blind setup. On changes like these, I would be surprised if you get anything but zero.

Code: github.com/18Abhinav07/Loom

Let your agents bloom with Loom.

The benchmark pairs come from STALE-bench by the illinoisdata group; the Loom system and the evaluation harness are mine. Built for the Graph Hacks hackathon (WeMakeDevs × FalkorDB), track: Agent Memory and Coordination. I used Claude Code as an AI coding assistant to build it.