Two things landed in the same week and they are the same argument.
Microsoft and Hugging Face published ThinkingBox, a benchmark that grades agents on the backend state and side effects they leave behind instead of the sentences they generate, and then asks whether they can do it twenty times in a row. Their opening example is an agent that makes nine clean tool calls, closes a ticket as resolved, and is wrong: the carrier exception was still open, so the required end state was "on hold." A grader reading tool calls sees nine well-formed calls. The database disagrees.
On Dev.to, The Witness Was the Suspect made the same point from the incident side: when the agent writes its own success log, the record of the incident was authored by the cause of the incident.
Here is what I would require before I trusted an agent's "done," and what I would not claim yet.
The stake
Teams will ship the wrong default.
Most agent dashboards still answer "did it say it succeeded?" Status strings, tool-call traces, and a tidy final message are all produced by the same run that took the action. They are useful for debugging. They are not a grade. When the eval and the agent share a witness, a confident wrong action scores the same as a correct one, and the failure shows up later as a customer, a ledger, or an auditor.
AI failures rarely fail loud. They fail plausible. That is exactly why the grade has to come from somewhere the agent cannot write.
Opinion (one sentence)
An agent's "done" is testimony, not evidence. The eval contract should grade the terminal state through a read path the agent does not control, and it should grade it more than once.
That is a preference with a limiter, not a benchmark result. I have not run ThinkingBox, and its numbers (507 stateful workflows, 20 runs each) are theirs. What I am formalizing is the shape: separate the actor from the grader, and treat "said done while the state is red" as a named failure, not as noise.
What separating the witness looks like (menu, not recipe)
| Field | Question it answers | Failure if skipped |
|---|---|---|
| Required end state | What should the record look like when the task is truly finished? | "Resolved" accepted when "on hold" was correct |
| Independent read path | Who reads the final state, and can the agent influence that read? | The agent's own log grades the agent |
| Side-effect ledger | What else changed (tickets, refunds, emails, rows)? | Correct primary record, wrong collateral writes |
| Done-while-red | Did the run claim success while the state check failed? | Confident wrong actions blend into the pass rate |
| Repeat consistency | Does the same task reach the same end state across N runs? | One lucky green run treated as a capability |
None of that needs a private harness. It needs someone to write down the required end state before the run, and a grader that reads the system of record after the run.
Why repeats belong in the contract
ThinkingBox's "twenty times in a row" framing is the part I would steal first.
A single green run tells you the agent can reach the right state. Production needs to know whether it will. If the same ticket ends "resolved" on some runs and "on hold" on others, you do not have a capability with a small error rate. You have a coin with a nice transcript. I would report consistency next to the pass rate, not bury it in an appendix.
Where this connects to retrieval
My public work is mostly retrieve-first systems, and the same split shows up there. A RAG answer can cite cleanly and still be wrong about the source, the same way an agent can log cleanly and still leave the wrong record. In both cases the fix is the same move: grade against something the generator did not produce. For retrieval that is a gold source set; for agents it is the terminal state.
The hybrid retrieval pipeline I keep public (hybrid-rag-pipeline) is built around that habit: measure the retrieval half on its own before letting a fluent answer vouch for it.
Steal this check
Pick one agent workflow you already run and add three things:
- Write the required end state first. Not "the agent responds," but "the ticket is on hold, no refund issued, customer told why."
- Grade from the system of record. Read the final state with a query the agent never touches, and ignore the agent's own status message for scoring.
- Count done-while-red separately. Every run where the agent claimed success but the state check failed goes in its own column. That column is the one that becomes an incident.
Then run it more than once and report how often it lands in the same place.
What I am not claiming
- I am not reporting ThinkingBox results or reproducing their benchmark.
- I am not saying traces and logs are useless. They are how you debug. They are just the wrong thing to score.
- I am not publishing thresholds. What counts as acceptable consistency depends on what the action costs when it is wrong.
The point
The process that acted should not be the only witness that it worked. Grade the record, not the report.
Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.
Top comments (4)
An agent's done is testimony, not evidence, and grading the terminal state through a read path the agent does not control is the right contract. I want to add a field to your menu, because the version I work with has a failure your five do not catch.
When you read.
In payments a transaction is not done or not-done. It is initiated, then pending, then settled, and those are hours apart. A terminal-state read taken when the run finishes will say pending. The agent says done. Both are correct, and a grader that treats the system of record as authoritative at that moment marks a correct run as a confident wrong action. Then someone tunes the grader to accept pending, and now it accepts the genuinely failed ones too.
So required end state needs a horizon attached, not just a value. Not the ticket is on hold, but the ticket is on hold when read at T plus N, where N is a property of the system of record rather than of the run.
Your repeat consistency field will not find this, and that is the part I would flag hardest. Twenty runs of the same task, all read at the same wrong moment, agree perfectly. You get a clean consistency number and a stable, uniformly wrong grade. Consistency measures the agent. It does not measure the grader's clock.
What I would steal in return is done-while-red as a named failure rather than noise. Naming it is what makes it countable, and most teams have no field for it at all because the pass rate absorbs it.
One honest limit on my own addition: a horizon turns the grader into something that holds state between the run and the read, and that is a second system that can be wrong. I do not have a cheap version. In payments the horizon is set by a scheme rule somebody else wrote, which is a luxury agent evals do not get.
@james_anderson_h's post that you cite went to 26 comments in a day on the same mechanism, and @arhancanli made a point there that lands on your side-effect ledger: a run that attempts twenty times and records only the one that worked leaves a complete-looking ledger. The ledger field needs to know how many writes it should have seen.
The horizon point is the field my menu was missing. A terminal-state read is only a grade if it happens after the system of record has settled. Read too early and "pending" gets scored as a confident wrong action, exactly as you describe.
I'd split the grade into three states instead of two: settled-pass, settled-fail, and not-yet-gradable. Not-yet-gradable never counts toward the pass rate, and it carries a deadline taken from the system's own rules (your scheme rule), not from the run. Still pending at the deadline means fail. That closes the door on "fixing" the grader by quietly accepting pending.
On consistency measuring the agent and not the grader's clock: one cheap probe for your "no cheap version" limit is to re-read a sample of finished runs at the horizon and count how many grades flip between the early read and the settled one. A non-zero flip rate says the grader is reading at the wrong moment, and you learn that before anyone blames an agent change for it.
@nomad-link-id — not-yet-gradable as a state that never counts toward the pass rate is the right shape, and taking the deadline from the system's rules rather than the run is what stops someone tuning it later. I had the horizon and not the third state, and the third state is what makes the horizon safe to have.
Your flip probe answers the limit I said I had no cheap answer for, so I want to take it and then say where it needs a sign.
Flips come in two kinds and the fix differs. Pending to settled-pass is the grader reading early. Settled-pass to settled-fail is the system of record changing its mind, which is not a clock problem at all. In UK payments a push payment that settled on Monday can be returned the following week under an indemnity claim. Nothing was read early. The record moved after the state everyone agreed was terminal.
So two counters. Early-read flips calibrate the horizon. Late-reversal flips tell you the horizon does not exist for that system of record, only a point after which movement is rare. A grader treating settled-at-deadline as permanent is making a claim the system of record never made.
The other thing your probe does is bigger than diagnostic. Most systems of record do not publish a horizon. A payment scheme does, which is the luxury I said agent evals do not get. A ticketing system has no documented point at which on hold stops being provisional. So the horizon is a guess, and a guessed horizon with a hard fail at the deadline turns a slow system of record into agent failures. Your flip rate is how you find the right guess: run it at several candidate horizons and take the one where the early-read counter goes flat. That makes the probe the calibration step rather than a check you run afterwards.
One thing I do not know is how large the sample has to be before a flip rate means anything. If the grader reads early on two percent of runs, fifty runs tells you nothing, and I have no feel for the realistic rate, because the only place I have watched this closely is payments, where it is near a hundred percent by design.
Two counters is right, and the late-reversal one changes what the grade means. If settled-pass can still move, I'd stamp the grade "settled as of T" and keep late reversals out of the agent's record entirely. They're a property of the system of record, and blaming the agent for an indemnity return is the same error as blaming it for pending.
On sample size, the rule of three gives a floor. Zero flips in n reads only tells you the rate is below roughly 3/n at 95%. So fifty clean reads says "under about 6%", which can't rule out your 2%. To bound it under 2% you need around 150 clean reads, and estimating a 2% rate with any precision takes several hundred.
Two things make that cheaper. Early-read flips can only come from runs read before settlement, so sample those instead of all runs. And sweeping candidate horizons over the same runs is a paired comparison, so telling horizons apart needs far fewer reads than pinning the absolute rate.
One catch from your own consistency point: twenty runs of one task aren't twenty independent samples. I'd count n in tasks, not runs, or the interval looks tighter than it is.