On 12 August one of our benchmark runs built its repository inside a sandbox where the poetry binary could not start. Forty-two commands were issued to it. Thirty-three of them never ran at all. Thirteen agents walked into that wall one at a time over an hour and forty-two minutes, and not one of them was told by any of the twelve before it.
The run finished. It shipped a working repository and it scores 77.6 on our public board today.
Nothing failed, no alert fired, and no number moved. The cost was real, and it was paid entirely in turns that no longer exist, inside branches we threw away on purpose.
What happened
The first agent asked for a poetry command at 17:21 and got nothing back. The last one did the same thing at 19:04. The run lasted 128 minutes, so the wall stood through most of it and was exactly as undiscovered for the last agent as it had been for the first.
The counts are small enough to check by hand. The run executed 76 shell commands. Forty-two started with poetry, and thirty-three of those returned an exit code of -1, which in our harness, the program that runs the agents and owns the tools they call, means the process was never started at all. Thirty-four commands in the whole run failed to start; thirty-three of them were this one, and the odd one out was a powershell call that also never started.
The thirteen agents that hit it were thirteen separate instances filling four roles, nine of them coding agents, two develop, one build and one test. The run page counts roles rather than instances, so the number there is smaller.
Every one of them recovered, which is exactly what made this invisible. A coding agent that cannot run poetry run pytest tries python -m pytest, gets a result, and carries on with its task. Recovery is the behavior we want. It is also the behavior that turned a one-time environment problem into a tax collected thirteen times.
What we ruled out
The obvious first suspect is the model. We had a control we had not planned for. A different vendor's model ran against the same specification the same day. That run executed 137 commands, 59 of them starting with poetry, and 44 of those never started. Nineteen agents hit the wall across four hours and thirteen minutes.
Two models from two vendors produced the same behavior, which makes the model the least likely explanation. It is a check rather than a proof, because both runs sit inside the same harness and a harness-level defect reproduces whatever model is in the seat.
Root cause
Our agents work in two modes. An agent plans in a strategy conversation, its running planning thread. When it takes on a concrete piece of work it opens a branch, a separate short-lived conversation holding only that job, executes inside it, and closes it. The design document for that architecture states the isolation as a success criterion. Work branch conversations are fully isolated from the strategy conversation, and only completion reports cross the boundary.
The branch is then discarded. What returns is the completion report, and the next agent's branch starts from the strategy conversation, which never saw the previous branch's contents.
The architecture requires that report to be structured against the work contract the agent was handed, the written spec of the one job it took on. That makes the report a statement about the deliverable, and the boundary exists to protect that shape, because isolation keeps an agent's window small enough to reason in and cheap enough to run at this scale.
A dead binary is not part of any deliverable. Nothing in the contract the report answers asks about it. It is friction met on the way, worked around in a single turn, and never mentioned again, so it goes in the half of the branch that gets thrown away. The next agent arrives with a clean window and the same wrong assumption.
The system remembers what it built and forgets what it learned. What it built has a route out of the branch. What it learned has none.
Why nothing caught it
This sat for a month, and the reason it sat is the same reason it is worth writing down. Every signal we watch was pointed somewhere else. The repository was fine, the score came out fine, and the only trace of the dead binary was thirty-three lines in a log nobody had a reason to open.
We do have a layer with the cross-agent view. A supervising agent inspects every live agent while the run is unattended, awards merits and demerits against a trust score, and can pull an agent off duty. It watches the whole fleet. What it watches for is agent behavior, and a dead binary is not agent behavior, so it had no reason to look.
An agent working around a broken environment looks identical to an agent doing its job well. Watching harder does not close that gap, because there is nothing anomalous to see.
The fix
The signal was already in the exit codes. We were not reading it.
Across the whole run, thirty-one commands succeeded, eleven ran and returned a nonzero status, and thirty-four never started. Those last two look similar in a log and mean opposite things.
A command that runs and exits nonzero is a statement about your code. It belongs to the agent that asked for it, because the next agent is on a different file at a different moment and the answer does not transfer.
A command that never starts is usually a statement about the machine rather than about one agent's code. That was the case here for all thirteen, and where it holds it stays true until somebody changes the environment. It is not automatic. A malformed invocation, or a permission scoped to one sandbox, can fail to start too, which is why the routing below is a default rather than a law.
Exit -1 is our own convention, so the portable part is the question rather than the number. Did this command run and disagree with you, or did it never run at all? Split the two where the process returns, and give only the second kind a way out of the branch. Resolve the toolchain once at the start of a run and write the answer where every agent reads. When something fails to start mid-run anyway, file it as an environment fact rather than a task failure and put it in the same place. Everything else stays local.
The restraint matters as much as the routing. Sharing every failure would rebuild the sprawling common context the branch design exists to avoid. Facts about the machine are a bounded set, and they stop accumulating as soon as the machine stops surprising you.
Status
Not fixed. The run above is from August, the behavior is unchanged, and wiring the supervising layer to environment facts is the obvious next move that we have not made.
If you run a fleet of agents against a shared machine, your agents can already recover from a broken environment. They will do it one at a time, and they will hide the cost while they do it. What to check this week is whether anything in your system has a place to put what one of them worked out.
The receipts
Two of the claims here are checkable to different depths. The grok-4.6 run and the second vendor's run on the same specification are public, and the pages carry the scores and the published run statistics.
Favur itself is closed-source and invite-only, and the repositories its runs produce are open. The board holds every run we have scored, if you want to pick one apart yourself.
Top comments (0)