Introduction:
A quick thanks to @WeMakeDevs and FalkorDB for hosting this awesome hackathon Graph Hacks.
1. The problem🤷♂️
Upgrading a Python codebase from Pydantic v1 to v2 is a classic long-tail job. The official codemod, bump-pydantic, rewrites the easy parts and leaves # TODO[pydantic] markers on the rest: validators that read values, .dict() and .json() calls it can't see, removed config keys, custom types built on __get_validators__, and behaviour that quietly changed (unions, coercion, JSON output). That tail is exactly the kind of work you'd like to hand to several agents at once.
Problems that occur while several LLM agents work:
- They don't share memory
- They attempt the same method and try to fix the already fixed library
- Keep running into same failures
2. Stigmergy (A fancy term)😎
Ants don't hold meetings. An ant that finds food lays a pheromone trail on the way back; other ants are more likely to follow strong trails; trails nobody reinforces evaporate. Coordination emerges from the environment, not from messages. Biologists call this stigmergy.
Pheromone applies that to coding agents. The environment is a FalkorDB graph.
- A failing test points to a Failure; each Failure has candidate fixes (Hypotheses);
- The edge between them carries a pheromone value. Agents pick hypotheses by a weighted roulette over pheromone, claim one with a single atomic query, try it, and report.
- Only test results change pheromone: a fix that passes reinforces its trail, a fix that fails decays it, and everything evaporates a little every ten seconds.
- The idea got motivated from
LLM agents in MIT's SwarmWorldwork; here it runs on real code with tests as the only judge.
3. Why a graph, not a key-value blackboard📊
I have worked on Agentic memory for sometime and also have had the experience of building an agentic open source memory named Engram-Causal, which mainly answers why did a particular function call passed/failed instead of what the agent did.
A shared dictionary of "who is doing what" would cover claims. The graph earns its place in three other ways.
Blast radius. Every repo file is a node with IMPORTS edges to the files it imports. Before a patch merges, the merge gate collects every test that could notice the change with one variable-length match, (f:File {path:$path})<-[:IMPORTS*0..3]-(dep:File)<-[:DEFINED_IN]-(t:Test), and requires them to stay green. That's a multi-hop question a flat store can't answer cheaply.
The trail behind a fix. Every attempt is a node linked to the agent that made it, the hypothesis it tested and the tests it fixed or broke. "Why is this test green?" is one path query from Test through Failure, Hypothesis and Attempt to Agent, and the dashboard renders that path with the diff.
Memory that outlives a run. When the same kind of fix wins in several places, it becomes a Recipe in a separate recipes graph. A run on a different repo seeds its hypotheses from Recipes whose signature matches. Signatures (dict-call, validator-values, json-encoders) never contain file paths, which is what lets them transfer.
Graph algorithms help the agents too: PageRank over the import graph raises the prior on central modules, and weakly connected components group failures into independent clusters.
4. Two FalkorDB lessons (and two lessons about tests)👨💻
There are no multi-query transactions, so every coordination step is one query. FalkorDB serializes writes to a graph and each query is atomic, but you can't wrap a read and a write from Python in a transaction. A claim is therefore a compare-and-set in a single statement: match the hypothesis where it's open or its lease has expired, set the new owner, return the id. An empty result means another agent won. Twenty concurrent claims on the same hypothesis produce exactly one winner, and that's a test in the repo.
SET runs on every matched row, even if a LIMIT comes later. A "pick the top candidate and claim it" query with SET ... LIMIT 1 claims every candidate. So the frontier read and the claim are separate queries, and the claim always targets one id.
We also hit a server crash: a read returning node properties together with collect() on a hypothesis that had just been claimed panicked inside reply_compact. Splitting it into plain-row queries fixed it.
Two lessons came from the tests, not the database:
-
Tests decide, so tests can be fooled. An agent once made a frozen-model test pass by overriding
__eq__and__setattr__instead of settingfrozen=True. Pydantic's warning about the leftoverallow_mutationkey was only aUserWarning, so the suite didn't catch it. TreatingUserWarningas an error, and telling agents not to override dunder methods, fixed it. -
Real test suites hide the real error. On robusta-dev/krr the swarm stalled with 13 red tests. The CLI under test caught the actual exception (a deprecated
.dict()call in the runner), logged it and exited with code 1, so the test only reported an exit code. And krr's tests depend on run order: a subset of tests can fail where the full suite passes, so correct fixes were rejected as regressions. Reading the logged traceback as the failure's cause, and judging regressions against the same test selection run before the patch, took krr from 42 to all 55 tests green. Along the way the swarm also fixed a deprecatedresult.json(indent=2)that the human migration PR had left in place.
5. The scoreboard🧮
I ran the same six agents four ways, with the same model (gpt-6-luna), budget, prompts, hypothesis templates and merge gate. Fifty runs, shuffled, on frozen code, every one ending with all tests green:
- Baseline: independent agents, no shared graph.
- Coordinated: shared graph, claims, leases and attempt history, with pheromone switched off.
- Swarm: Coordinated plus pheromone-weighted selection.
- Swarm + Recipes: Swarm plus fixes learned on earlier runs (krr only).
| Tokens | Time to green | Collisions | |
|---|---|---|---|
| gym, Baseline (10 runs) | 150k | 166 s | 124 |
| gym, Coordinated (10) | 47k | 72 s | 7 |
| gym, Swarm (10) | 48k | 73 s | 8 |
| krr, Baseline (5) | 178k | 243 s | 88 |
| krr, Coordinated (5) | 121k | 207 s | 11 |
| krr, Swarm (5) | 119k | 205 s | 13 |
| krr, Swarm + Recipes (5) | 101k | 253 s | 5 |
Shared state is what works. Against independent agents, the graph-coordinated modes used about two thirds fewer tokens on the gym and a third fewer on krr, and collided roughly ten times less often. On the gym they were also more than twice as fast. krr's failures form a chain where each fix reveals the next layer, so the time gain there is smaller.
Where it didn't win: pheromone. Turning pheromone-weighted selection off changed nothing measurable on either repo. The gains come from shared claims, leases and attempt history. That's partly because the model fixes most of these failures on the first try, which leaves pheromone little to choose between; harder targets, where most attempts fail, are where reinforcement would have to prove itself. Recipes are inconclusive too: they produced 80% of the merged krr fixes and lowered median tokens by 15%, but the ranges overlap and one run was slow.
The coordination itself held up under load. Thirty agents hammering the claim and report queries for a minute, with short leases and agents that stall on purpose, produced no double claims, no reports accepted after another agent took over, and pheromone values identical to an exact replay. FalkorDB's default queue cap pushed back at that load, which the agents absorbed with backoff.
6. What's next⏭️
Find out if Recipes compound: Recipes produced most of the fixes on krr but didn't measurably cut cost over five runs, partly because the hand-written templates already covered what krr needed. The interesting experiment is a run of ten or more real repositories in a row. If the tenth migration is clearly cheaper than the first, the shared memory is worth something no single run can show.
Isolate the graph itself Our baseline compares coordinated agents with independent ones. It doesn't tell you how much of the gain needs a graph rather than any shared store. Rebuilding the claims, leases and history on a key-value store, with the same scheduler, would answer that directly. My bet is that the graph's value lies in the questions you can ask afterwards (blast radius, provenance, why a fix won) more than in raw speed, and I'd like to measure it.
Leave the gym: The gym was written for this project and krr is one repository. Next is a handful of real open-source repos with pending Pydantic v2 migrations, opening the swarm's diffs as pull requests and counting how many maintainers merge. Other long-tail upgrades (SQLAlchemy 2.0, Django, NumPy 2) fit the same loop.
Top comments (1)
With evaporation every ten seconds, how do you keep a slow test run from letting the trail die before its result comes back? Do the IMPORTS edges get rebuilt when one agent edits a file, or do the other agents keep working from a stale graph?