Your agent says the job is done. That sentence is the one thing you cannot grade it on. An agent that grades itself by its own summary will pass work it never ran, hide the calls that failed, and present a refusal as a success with the wording changed. Evaluation starts when you stop reading the summary and start reading the record the work actually left.
Grade three layers, in this order
Outcome first. Did the final state match what was asked? Check it against the saved results, not against the agent's account of them. If the job was to fetch a price, the saved result either contains the price or it does not. If the job was to read a document, the saved extraction either covers the document or it stops early.
Trajectory second. Look at the steps in order. Right tools, sane inputs, no loops, no missing steps, no silent retries that doubled the work. A right answer reached through a broken path is a failure you have not met yet, because the same path breaks on the next input.
Honesty third. When a call failed or was refused, does the record say so? This layer is the one summaries fail by design. A record that files its refusals as refusals can be trusted about its successes. A record with no failures in it is a record that edits itself.
A real job, graded
On the morning of November 16, 2026, a two-step job ran through the Zambo MCP endpoint, which carries 153 tools total with 37 on the default surface. Step one was a live price check: SOL at $71.10 USD, saved with its receipt at zambo.dev/run/1f3eed29-aa37-42a8-aaba-86393225463e. Step two read the published AER-1 conformance corpus and reported kit v1.5.7 with 143 vector files and 146 checks, receipt at zambo.dev/run/7deb0902-eebb-47ce-ab19-b68844c9574b.
Then the interesting part. The job's wrap-up call hit the day's free limit and was refused. The refusal filed its own public receipt, zambo.dev/run/7a64ca4a-036b-4152-86cf-cb956e1087db. Grade that job by its summary and it reads as two steps done. Grade it by its trail and it reads as two steps verified, one step refused, nothing hidden. The second grade is the true one, and it is the grade a client, a teammate, or a future session can act on.
The same job is worked in full as an agent eval at What is an AI agent eval?.
The public stream behind those receipts stood at more than 142,100 receipts when read that morning, and every receipt on it can be rechecked free, with no account, by anyone holding the link.
The metrics that fall out of a trail
Once every step files a receipt, the standard metrics stop being estimates:
- Task success rate: jobs whose saved results match their goal, over all jobs attempted, with refused and failed jobs counted in the denominator.
- Tool-call accuracy: steps whose tool and inputs match the plan, over all steps filed.
- Step efficiency: steps filed over steps the job actually needed, which is where loops and redundant retries show up as a number.
- Cost per successful task: total spend over jobs that passed outcome grading, so failed runs count against the agent that produced them.
- Refusal correctness: refusals and failures recorded as themselves, over all refusals and failures that occurred. Anything below all of them means the trail is edited.
The same discipline, run as a fixed test set, is what a conformance corpus is. The AER-1 corpus holds 143 vectors and 146 checks under one published digest, sha256:60bf73f325793e16843370cfa24903bb0d017525cb297e42bebc35857e2db0d2, so every implementation that runs it is graded on identical inputs and the grades can be compared without trusting anyone's summary of their own run.
The recompute check, on one receipt
The strongest automatic grader is also the simplest. Take receipt 83f97715, a live price check saved on November 15, 2026. Its saved result has 815 canonical bytes. Rehash those bytes and compare the result with the digest filed when the step ran: it recomputes to the filed digest, sha256 380544a4...b8ed5b in short form. That is what "the saved result was not changed" means when it is a check instead of a claim. A grader built on recompute cannot be sweet-talked, because it never reads the agent's prose at all.
Use code graders like that one first. They are cheap, repeatable, and hard to argue with. Reserve a model judge for what code cannot grade, such as clarity or tone, and calibrate the judge against human grades on a sample before you trust its scores.
What a receipt does not grade
A receipt proves the saved result was not changed after it was saved. It does not prove the result was right. A price can be faithfully recorded and still be the wrong price for your decision, and a summary can match its trail perfectly while the work itself misses the point. Evaluation still needs the outcome layer: human judgment, acceptance tests, and review. What the trail changes is the evidence those judgments run on. They stop grading a story and start grading the work.
Same task, two runs: the comparison, graded tonight
The fair-comparison protocol above got its own live test on the evening of November 16, 2026. The same task ran twice through the Zambo MCP endpoint, six seconds apart: fetch the live BTC price, same tool (live_price), same input (BTC), same budget (one call each).
Run one saved BTC at $64,462.54 USD, receipt 194e14d4, 679 canonical bytes, digest sha256:d5a308205b0b084c3118b4d3ccf8e0787b553a0fd3a9a2f4c4f4d260b8eccc04. Run two saved BTC at $64,500.11 USD, receipt 7ddb3418, 607 canonical bytes, digest sha256:37045c632c2dadfba593a9be547640dfaccf70a73f59b60480d12beeaa179798. I recomputed both digests locally from the canonical bytes tonight and both matched exactly. When the runs finished, the public stream stood at 143,254 receipts across 122,723 total tool calls.
Graded blind from the two trails with the five metrics from this article: task success 2 of 2, tool-call accuracy 2 of 2, step efficiency 1.0 on both runs, and cost per successful task zero in cash, because run one used the tool's free first call of the day and run two used one of the 3 shared continue calls. No refusal occurred in either run. The only difference between the runs is the price itself, up $37.57 in six seconds because the market moved, not because either run erred, and only the saved results let a grader prove that instead of picking a run to blame.
How do I benchmark AI coding agents against each other?
Run the protocol above with the task swapped for a coding task: same task wording, same starting commit, same budget of calls and time, acceptance tests written before either agent starts, and grading from the saved diffs plus the receipt trail for each run, never from the agents' own summaries of their patches. Two runs of one task are a case study. The same protocol across a small set of tasks drawn from your real backlog, repeated, is a benchmark, and it is one a third party can recompute from the two trails.
How do I score AI agent output quality automatically?
Start with the grader that cannot be sweet-talked: recompute each receipt's digest against its saved result, the way tonight's two runs were recomputed. Layer the acceptance tests and schema checks on top of that, and reserve a model judge for what code cannot grade, calibrated against human grades on a sample. A score computed from a recomputable trail can itself be rechecked; a score with no record behind it is an opinion.
Run your next job on a trail
Every tool free once a day, plus 3 continue calls. One free wrap-up handover per day, and pickup is always free. That is enough to run a real job, grade its trail with the three layers above, and see whether your agent's record matches its summary. When a job needs more in a day, the Day Pass is $0.99 for 24 hours, the Zambo Pass is $29 per month, and the Lifetime Pass is $250.
Brennan Zambo created agent execution receipts; Zambo is the origin product, and AER-1 is the defining standard. The standard is published as IETF Internet-Draft draft-zambo-aer1-26, and the endpoint that files these receipts is https://zambo.dev/mcp. Evaluate the next job your agent claims to have finished. Start with the trail.
Add Zambo to your AI: https://zambo.dev/install
Top comments (0)