We read all 1,186 failed agent runs. Half failed on the same thing.
Most agent write-ups show you the demo that worked. This one shows the ones that did not — every one of them, because the system records each run as a row with a verdict and a reason, and that file is boring enough to be true.
As of this writing the ledger holds 1,737 runs: 551 passed, 1,186 failed — a 31.7% pass rate. (Window: 2026-09-28 to 2026-10-12, measured live before publishing.) A pass rate under a third sounds like a broken agent. Read the reasons and it turns out to be the opposite.
The ledger is one JSONL file
Each run appends a single line:
{"ts": "2026-10-12T05:35:31Z", "task": "t632203732", "goal": "…", "agent": "a293925", "passed": false, "why": "exit=1 … AssertionError"}
passed is a boolean. why is the agent's own account of what happened. That second field is the whole point: a boolean alone tells you whether a run worked, never why it didn't, and without the why you cannot tell a broken agent from a strict one.
What the 1,186 failures actually are
Grouping the why strings of every failed run by failure mode:
| Failure mode | Count | Share |
|---|---|---|
| Assertion failed | 583 | 49.2% |
| Other (mostly generated code that crashed) | 360 | 30.4% |
| Missing file | 230 | 19.4% |
| Empty result | 8 | 0.7% |
| Import error | 4 | 0.3% |
| Timeout | 1 | 0.1% |
Two of these deserve the space.
49% of all failures are assertion failures — the agent's own verification gate refusing to declare success. The run wrote an artifact, then checked it against a condition it had set itself, the condition failed, and the run was recorded as failed rather than rounded up to "done". The single largest bucket of failure in this system is the system declining to lie. If the gate had been softer, this number would be prettier and mean less.
19% are missing files: the artifact the run was supposed to produce never landed. This is the failure mode that hides in a success-shaped system — the run exits 0, the UI shows green, and the file the next step depends on is simply not there. Recording it as its own category means it can be counted, and counted means it can be fixed. It already fell from a larger share of earlier windows.
The "other" bucket is instructive too: sampling it, most entries are Python the agent wrote for itself that crashed on the first line (IndentationError, a missing environment variable). The agent's own scaffolding is a bigger source of failure than any external dependency — which is only visible because the reason is written down.
Why write this down instead of just fixing it
An agent that reports "success" is unfalsifiable. You cannot improve a metric you refuse to record, and you cannot tell a regression from a quiet, growing pile of lies. The ledger makes the agent's own performance a first-class artifact: the pass rate is a number, the failure taxonomy is a distribution, and both move over time.
That is the entire trick, and it is not a clever one. Every run writes a line; nothing gets to skip the line.
How to check this yourself
The system that produces this ledger is open source and runs locally on a Windows laptop, CPU-only, no GPU:
- Repository and full source: https://github.com/openamer/openamer
- Apache-2.0 licensed, 30+ releases, 20 topics, contributions welcome.
The honest caveats: this is a small project (single-digit stars), the pass rate above is from one laptop's own runs and is not a benchmark against any other agent, and "1,186 failures" is a description of this system's strictness, not a claim of superiority over anyone else's.
If you run agents and don't keep the why, that is the one line I would add first.
Top comments (0)