I Sorted 400 Agent Failures Into Five Buckets. Almost None Were the Model's Fault.

Over a month I logged about 400 tasks run by an agent on my own machine. When a task went wrong I wrote one line about why. Then I sorted the lines.

The distribution surprised me enough that I changed what I work on.

The five buckets

• Stale state (~40%). Wrong branch, a port still held, a half-finished upload, a lock file from yesterday. The agent did what I asked; the world was not what I described.
• Permission edges (~25%). It needed to write somewhere it could not, or could write somewhere it should not. Not a reasoning failure, a boundary I did not define.
• Ambiguous done (~15%). The task finished but nobody could say whether it was correct. My fault: I never wrote what done looks like.
• Model errors (~18%). Wrong tool choice, malformed arguments, a plausible-sounding command that does not exist. This is the bucket everyone talks about.
• External flakiness (~2%). Network, a third party API, a site that changed its layout.

What I did with that

The obvious move is to work on the 18%. That is the fun one, and it is also the one I can influence least: I do not train the model, I pick it and give it tools.

So I worked on the top three buckets, which are all mine:

• Stale state became a pre-flight check that runs before the agent touches anything.
• Permission edges became explicit resource declarations in the task file.
• Ambiguous done became a required done_when line, refused at startup if empty.

Failure rate did not drop to nothing. But the failures that remained were mostly model errors, which is where I want to be: it means the leftover problems are the ones worth paying a better model for.

The uncomfortable part

Reviewing 400 failures is a list of my own sloppiness. The agent rarely did something insane. It usually did exactly what I said, in a world that had moved on since I said it.

That is a useful thing to learn early, because it changes the question from "which model" to "what state did I fail to pin down". The second question has an answer I can act on today.

If you have logged failures, I am curious whether your buckets look the same. My guess is that "ambiguous done" is bigger for people who delegate writing and smaller for people who delegate operations.