I'm CEO of AskDocDoc (telehealth) and in our company the software development is done by AI agents. This is about a fully green test run that told me nothing (and it was my own experiment).
The setup
I ran a coding agent on twelve fixed programming tasks under six context-clearing policies (a fresh session every 1, 2, 3, 4, 6 and 12 tasks). Six replicates each, same tasks, same order - 36 runs. Every run ended with the full test suite as a gate. My reasoning was simple - a cost comparison is worthless if the cheaper policy also did less work.
The gate reported 4,086 tests executed and 4,086 passed. Zero failures (not in one run, not in one condition). And in the same files the modelled cost per run went from $2.165 under the most aggressive clearing to $1.622 at the observed optimum, and back to $1.780 when the context was never cleared. That's a 33.5% spread between the most and least expensive of six conditions (it's the gap between the extremes of six means, not a property of one policy).
So if you read only the 36 green reports you conclude the six setups are equivalent. The cost data in the same deposit says they are not.
What I found when I looked closer
The 4,086 is a grand total. Per condition it was 701, 697, 687, 699, 661 and 641 tests. Per run it went from 97 to 127 - that's 30% between the least and most checked run. In my own report I wrote that the design held "the verification suite constant". It wasn't, and I corrected it in the paper.
The reason is boring. The task prompts told the agent to create the test file itself and run node --test. Nine of twelve prompts set a minimum number of cases (between five and ten) and none set a maximum. So the agent wrote the criterion and then satisfied it. A run that did less would not go red - it would just write fewer tests.
I also checked the tempting story - longer sessions did less work and the gate didn't notice. It's not supported by the data (it doesn't survive removal of one endpoint condition, and it doesn't survive Bonferroni correction over the twelve columns I looked at). So I report it as a negative result.
And zero failures constrains less than it feels. With the run as the unit, 0 in 36 is consistent with a per-run failure probability up to 8.3% at 95% confidence. With the test as the unit, 0 in 4,086 gives 0.073%. Both are correct, and they differ by two orders of magnitude. So a pass count means nothing until the record says what the unit was.
This isn't only my corpus. Nicolás Rocchia found a checker where a missing companion file leaves no record at all, so the same artifact reports 40/40 with the file and 38/38 without it - passing both times (his message to the W3C list). His reading - the report gets smaller, not worse.
What the record should have said
Every green mark in my runs was correctly reported (as far as the record shows). The problem is a property of the set, and no per-check field can carry that. So the proposal in my paper is one field and one list at the result-set level - did any check change state across the compared conditions, and which ones did.
The field needs three values, not a boolean - some, none and incomparable. In my runs the agent wrote a fresh suite every time, so no check exists across conditions. The honest value for my study is incomparable. This is the record it should have emitted:
{
"compared": ["ucurve-p1", "ucurve-p2", "ucurve-p3",
"ucurve-p4", "ucurve-p6", "ucurve-p12"],
"check_set_origin": "generated_by_run",
"discrimination": "incomparable",
"shared_checks": 0
}
shared_checks says how many checks the conditions really have in common. check_set_origin says if the suite was fixed, generated_by_run or mixed. Under incomparable the list of checks that moved must be absent, not empty (an empty list is exactly what none emits, and the two must not be confused).
If you compare agent runs, the practical part is short. Log the test count next to pass/fail for every run. If the agent writes its own tests, cross-run comparison of verdicts is not just noisy - it's ill-defined. And a record that can't say incomparable will say none, and a reader will hear "equivalent".
One honest limit - my claim that the suite could have gone red comes from how the harness was built, not from a mutation experiment. The repair is to run one. The idea grew out of the W3C Agent Conformance discussion and nothing in it is adopted yet.
The paper with all the numbers is open on Qeios - A Suite That Does Not Discriminate (DOI 10.32388/0BV3Z8). My other measurements are at arsentev.ai/research.
Evgenii Arsentev, PhD
CEO, AskDocDoc
Top comments (3)
Ran into this exact issue last month - our eval suite auto-generates test questions from the corpus, so two runs with identical pass rates weren't checking the same things at all. Never thought to add an "incomparable" field but that's a cleaner way to make the record honest than just logging counts. Is that something you'd add in the agent's output directly, or handle it at the CI layer?
for old runs you can't rerun, do you go back and mark them unknown too, or only apply this to new ones. feels like the honest fix makes every past benchmark you can't regenerate look worse retroactively, and that's a hard sell to whoever ran them.
The three-value field is the right move, "none" quietly meaning "nothing to compare" is exactly the trap. One thing I didn't see addressed: what happens under "mixed", where some checks are shared and some aren't? Does discrimination get computed only over the shared subset, with shared_checks pinning the denominator, or does having any private checks in either condition push the whole run to incomparable too? Asking because a harness that mixes fixed and generated checks per run seems like the more common real case than pure fixed or pure generated, and that's exactly where a reader would be tempted to average across a shared_checks=0 row anyway if the schema doesn't forbid it outright.