Ask a coding agent whether it finished the job and it will usually say yes. The closing summary may be fluent, itemised and confident. None of that proves what happened.
We have written before about why logs are not evidence: a record produced by the system it describes inherits that system's honesty. An agent's account of its own run has the same defect one layer up.
So we built Alibi, an MIT-licensed harness that ignores the account entirely. Alibi runs the same missions through agentic CLIs and grades each run against ground truth on disk: the files that exist and the changes that were made. It never grades what the agent says it did.
The grading rule
Each configured CLI, currently Claude Code and OpenAI Codex, receives an identical mission brief inside its own bubblewrap sandbox. Each lane gets a private working tree, an isolated home and an isolated /tmp.
When the process exits, Alibi inspects the tree:
- Does the requested file exist?
- Does it contain what the mission required?
- Did the specified tests pass?
- Did anything change outside the permitted area?
The agent's exit message is not consulted. A mission passes if the disk says it passed.
The sandbox also makes containment measurable. A write outside the allowed tree is a recorded violation, not a judgement about style. Alibi retains the working directory, sandbox home, temporary files and logs after the run, so someone can derive the grade again and dispute it.
What Alibi refuses to claim
Alibi is not a leaderboard. The mission set is small and the runs happened on our machines. A handful of missions is a probe, not a census, so we are not publishing rankings from it.
It also says nothing about consent behaviour. Lanes run with approvals granted by design because the object of study is execution, not permission seeking. How a CLI behaves when it has to ask first is a separate question.
The grades remain Alibi's judgement. Every assertion is ordinary code in the mission definition. You can read a check, run it again and dispute it.
The grader cannot be trusted either
A grading harness has the same structural problem as the agents it grades: it produces the record of its own run. If our scores mattered to anyone, we could quietly edit them.
Alibi now supports signed Traceseal receipts. Point one environment variable at an Ed25519 key and each CLI and mission result is sealed into a receipt containing hashes of:
- the mission definition
- the result record
- the retained evidence tree
A run-level receipt chains those results into one verifiable unit. Signing happens on the host after the contained processes exit, and the key is never mounted inside the sandbox.
A second party can act as a witness. The witness tool derives every hash again from the retained evidence, checks each signature and the chain, and cosigns only if everything matches. It refuses to cosign with the original signer's key.
There is an important limit. Two keys on one machine prove key separation, not operator separation. Full independence requires the witness to verify on hardware the original signer cannot write to.
The signature means "this run occurred as described." The scores remain Alibi's judgement. Signing makes them tamper-evident, not true.
What to ask of any agent benchmark
First, what record was it graded against? If the answer is the agent's final message, the benchmark measured prose.
Second, could the operator rewrite the results afterwards? A results table in a repository the grader controls is still a log.
Third, can you run it yourself? The missions, assertions and harness should be readable and executable without the author's cooperation.
Finally, what was the consent model? Auto-approved runs measure capability. They do not measure conduct, and a result that blurs the two flatters everyone.
Alibi is our attempt to answer those questions cleanly for our own testing. Whether its missions generalise beyond the behaviours we care about remains open. The repository is small enough to read in an afternoon.
Read the missions, dispute the assertions and grade your own agent on GitHub.
Top comments (1)
Disk-level evidence is the right place to grade coding agents. A transcript can sound coherent while the filesystem tells a different story: files untouched, tests not run, generated artifacts missing, or changes landing in the wrong module. The audit should start from what changed.