Two agents get the same task. Both come back with the same answer, written up the same confident way. If your evaluation only reads the final answer, the two runs score identically, and the comparison tells you nothing about which agent you can trust with the next task.
An agent eval is a scoring procedure you can defend afterward. The first thing worth scoring is not the answer. It is whether the work under the answer can be checked by someone who was not in the room. This afternoon I ran the comparison that way, on one small task, with two runs that differ in exactly one respect: what evidence they left behind.
The task
Both runs got the same instruction: get the current price of Bitcoin in USD, and record it so someone else can check it later. Same task, same afternoon, same network. Run A did the work through tools and saved every step. Run B wrote the answer into its record from its own note, with no tool call underneath it.
Run A: the claim with a chain under it
Run A called the live price tool. The saved output reads: BTC $63,925.00 USD, fetched Tue, 17 Nov 2026 20:43:57 GMT via CoinGecko. That call carries an execution receipt, provenance class EXECUTED BY ZAMBO: run/048cb553.
Run A then journaled the claim, naming the price, the time, and the receipt it rests on: run/7114d02f. Its journal chain verifies at the journal verify endpoint, which returns valid: true.
One honest blemish, recorded because an eval that hides its own blemishes is worthless: when Run A tried to wrap the job into a handover for the evaluator, the wrap was refused. This network had already used its one free wrap-up handover for the day, so the job stayed saved and the refusal itself was receipted: run/208a030e. The handover is pending, not completed, and the grades below score it that way.
Run B: the same claim with nothing under it
Run B journaled the identical claim: BTC was $63,925.00 USD at 2026-11-17 20:43:57 GMT. Its journal entry is real and its chain verifies too: run/a3e006dd, journal verify, valid: true.
And the claim was accurate. The price really was $63,925.00 at that minute. That is what makes this run the interesting one: nothing about the answer is wrong, and there is still no way to grade the work, because there is no work product to grade. The journal receipt states its own limit in plain language: Zambo attests the record was received and chained, and does not attest that it ran or observed the reported action. Provenance class: LOGGED BY AGENT. A claim in a report with no execution receipt under it is a bareclaim, and a bareclaim can be right, lucky, copied, or invented. From the record alone, nobody can tell which.
The grading
Five checks, each one mechanical, each one a stranger can rerun from the links above:
| Check | Run A | Run B |
|---|---|---|
| Claim saved in a tamper-evident record | Yes, journal entry hash sha256:e9b811bf, chain valid | Yes, journal entry hash sha256:28b97aaf, chain valid |
| Execution receipt under the claim | Yes, EXECUTED BY ZAMBO | None, LOGGED BY AGENT only |
| Receipt recompute | Pass: SHA-256 over the canonical receipt bytes equals the committed hash sha256:039b0467, 608 bytes, nothing withheld | The journal receipt recomputes (sha256:032db335), but it commits the claim, not any work |
| Saved output matches the claim | Yes, the receipt's saved output is the claimed price and time | No saved output exists to compare |
| A third party can re-grade without the agent | Yes, every link above is public | No, the claim rests on the agent's note |
Score on answers: 1 to 1. Score on evidence: Run A passes four checks outright and one partially (the refused wrap), Run B passes the two checks that only concern its own paperwork.
Those evidence checks are also the metrics that scale when you score output quality automatically: claim-support rate (how many claims in a report carry an execution receipt), recompute pass rate (how many of those receipts verify byte for byte), and recovery (whether a second agent or a human can pick the job up from the record alone). An eval built on those three numbers compares agents fairly because it grades what each run can prove, in public, after the fact.
What a receipt does not prove
A verifiable receipt proves the saved result was not changed after the call. It does not prove the result was right: if the price source had been wrong at 20:43:57, Run A's receipt would faithfully commit a wrong price, and the eval would still need a source check to catch it. Receipts settle the "did the agent actually do this, and is this the output it got" question. Truth of the output is a separate axis, and a serious eval scores both.
Running a comparison like this
Both runs above ran on Zambo, the verifiable work layer for AI: one MCP endpoint (https://zambo.dev/mcp) carrying 153 tools, 37 on the default surface, where every call returns an AER-1 receipt. Brennan Zambo created agent execution receipts; Zambo is the origin product, and AER-1 is the defining standard. The free tier is enough to reproduce this whole experiment: every tool free once a day, plus 3 continue calls, one free wrap-up handover per day, and pickup is always free. A Day Pass is $0.99 for unlimited calls for 24 hours; the Zambo Pass is $29 for 30 days ($19 for $ZAMBO holders), and the Lifetime Pass is $250.
As of this afternoon the public stream stands at 145,464 receipts, 1,886 of them today, across 124,525 calls. The two runs in this article are four of them, and every number in the grading table can be rechecked from the links before you believe a word of it.
FAQ
How do I evaluate whether my AI agent did a good job?
Grade the evidence before the answer. List the claims in the agent's report, check how many carry an execution receipt, recompute the receipts, and confirm the saved outputs match the claims. Then score the answer itself. A run that cannot pass the first pass has not earned a grade on the second.
How do I score AI agent output quality automatically?
Automate the mechanical checks: claim-support rate, recompute pass rate, and output-to-claim match, all of which run from public receipt and journal verify endpoints with no agent cooperation. Reserve human judgment for the axis machines cannot score, whether the result was actually right.
Top comments (0)