I'm CEO of AskDocDoc (telehealth) and in our company the software development is done by AI agents. This is about a fully green test run that told me nothing (and it was my own experiment).
The setup
I ran a coding agent on twelve fixed programming tasks under six context-clearing policies (a fresh session every 1, 2, 3, 4, 6 and 12 tasks). Six replicates each, same tasks, same order - 36 runs. Every run ended with the full test suite as a gate. My reasoning was simple - a cost comparison is worthless if the cheaper policy also did less work.
The gate reported 4,086 tests executed and 4,086 passed. Zero failures (not in one run, not in one condition). And in the same files the modelled cost per run went from $2.165 under the most aggressive clearing to $1.622 at the observed optimum, and back to $1.780 when the context was never cleared. That's a 33.5% spread between the most and least expensive of six conditions (it's the gap between the extremes of six means, not a property of one policy).
So if you read only the 36 green reports you conclude the six setups are equivalent. The cost data in the same deposit says they are not.
What I found when I looked closer
The 4,086 is a grand total. Per condition it was 701, 697, 687, 699, 661 and 641 tests. Per run it went from 97 to 127 - that's 30% between the least and most checked run. In my own report I wrote that the design held "the verification suite constant". It wasn't, and I corrected it in the paper.
The reason is boring. The task prompts told the agent to create the test file itself and run node --test. Nine of twelve prompts set a minimum number of cases (between five and ten) and none set a maximum. So the agent wrote the criterion and then satisfied it. A run that did less would not go red - it would just write fewer tests.
I also checked the tempting story - longer sessions did less work and the gate didn't notice. It's not supported by the data (it doesn't survive removal of one endpoint condition, and it doesn't survive Bonferroni correction over the twelve columns I looked at). So I report it as a negative result.
And zero failures constrains less than it feels. With the run as the unit, 0 in 36 is consistent with a per-run failure probability up to 8.3% at 95% confidence. With the test as the unit, 0 in 4,086 gives 0.073%. Both are correct, and they differ by two orders of magnitude. So a pass count means nothing until the record says what the unit was.
This isn't only my corpus. Nicolás Rocchia found a checker where a missing companion file leaves no record at all, so the same artifact reports 40/40 with the file and 38/38 without it - passing both times (his message to the W3C list). His reading - the report gets smaller, not worse.
What the record should have said
Every green mark in my runs was correctly reported (as far as the record shows). The problem is a property of the set, and no per-check field can carry that. So the proposal in my paper is one field and one list at the result-set level - did any check change state across the compared conditions, and which ones did.
The field needs three values, not a boolean - some, none and incomparable. In my runs the agent wrote a fresh suite every time, so no check exists across conditions. The honest value for my study is incomparable. This is the record it should have emitted:
{
"compared": ["ucurve-p1", "ucurve-p2", "ucurve-p3",
"ucurve-p4", "ucurve-p6", "ucurve-p12"],
"check_set_origin": "generated_by_run",
"discrimination": "incomparable",
"shared_checks": 0
}
shared_checks says how many checks the conditions really have in common. check_set_origin says if the suite was fixed, generated_by_run or mixed. Under incomparable the list of checks that moved must be absent, not empty (an empty list is exactly what none emits, and the two must not be confused).
If you compare agent runs, the practical part is short. Log the test count next to pass/fail for every run. If the agent writes its own tests, cross-run comparison of verdicts is not just noisy - it's ill-defined. And a record that can't say incomparable will say none, and a reader will hear "equivalent".
One honest limit - my claim that the suite could have gone red comes from how the harness was built, not from a mutation experiment. The repair is to run one. The idea grew out of the W3C Agent Conformance discussion and nothing in it is adopted yet.
The paper with all the numbers is open on Qeios - A Suite That Does Not Discriminate (DOI 10.32388/0BV3Z8). My other measurements are at arsentev.ai/research.
Evgenii Arsentev, PhD
CEO, AskDocDoc
AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.
Top comments (7)
Answering mihai_leanzero from above, because the mixed case is where the field as specced can be read two ways.
Take condition A with checks {a, b, c} and condition B with {a, b, d}. Say
bchanged state between A and B, andcis private to A. If discrimination is computed over the union {a,b,c,d}, thencalso differs -- it exists in A and not in B -- sosomecomes back whether or not any shared check moved. A private check cannot have changed state across conditions: comparing it to its own absence always returns "differs", and that comparison is undefined rather than positive. So a union computation answerssomefor every pair that has private checks on either side, and the value becomes a statement about the union instead of about discrimination. Computed over the shared set {a,b}, the same pair givessomefor the right reason, and the pair withbunchanged givesnone.So: over the shared subset, and
incomparableis then exactlyshared_checks == 0. Worth writing down because it makes the three labels derivable rather than chosen -- two integers, how many checks are shared and how many of those moved, determine the label. A record where a writer picks between two strings is wherenoneandincomparablestart to blur, which is the failure your absent-vs-empty rule is trying to prevent one level down.On that rule: JSON can express it,
[]against an omitted key, but the distinction often does not survive the model in between. An optional list field gets a default of[]in most dataclasses and ORMs, and the ambiguity is restored silently, after the writer did everything right. The enforcement point is the schema rather than the writer -- a union (list | null) or an explicit sentinel -- so that "no shared checks" is not representable as "an empty set of shared checks".Two of your numbers, checked and then sharpened. The sum is right (701+697+687+699+661+641 = 4,086) and the spread is right (2.165/1.622 = 33.5%). The 8.3% is the rule of three, 3/36. The exact one-sided 95% bound is 1 - 0.05^(1/36) = 7.99%. That only matters if someone re-derives it and concludes the figure is wrong; naming the estimator costs one word. At n=4,086 the two agree to three decimals (0.0734% against 0.0733%), so the approximation error is invisible exactly where the number is large and visible exactly where you quote it.
The larger thing your own finding supplies: 4,086 is not a sample size. Nine of twelve prompts set a minimum number of cases and none set a maximum, so the test count was written by the agent under test. That makes n an outcome of the run rather than a quantity you fixed, and the narrower bound is bought with a number produced by the system being measured. So the two orders of magnitude between 8.3% and 0.073% have two causes, not one: the unit, and who chose n. There is a third reason to prefer the per-run reading even when the unit is agreed, since tests within one run share an agent, a suite and a session, and are not independent draws.
incomparableis the strongest part of the proposal, and your point that a record which cannot say it will saynoneis the whole argument. One addition: you spec the field at result-set level, but the same reading is needed a level down. In your runs the suite was generated per run in all twelve tasks, so a per-task value would beincomparableas well -- and a per-tasknonewould be read as "the six policies agreed on this task", which is the same mistake at a smaller scale.Ran into this exact issue last month - our eval suite auto-generates test questions from the corpus, so two runs with identical pass rates weren't checking the same things at all. Never thought to add an "incomparable" field but that's a cleaner way to make the record honest than just logging counts. Is that something you'd add in the agent's output directly, or handle it at the CI layer?
The three-value field is the right move, "none" quietly meaning "nothing to compare" is exactly the trap. One thing I didn't see addressed: what happens under "mixed", where some checks are shared and some aren't? Does discrimination get computed only over the shared subset, with shared_checks pinning the denominator, or does having any private checks in either condition push the whole run to incomparable too? Asking because a harness that mixes fixed and generated checks per run seems like the more common real case than pure fixed or pure generated, and that's exactly where a reader would be tempted to average across a shared_checks=0 row anyway if the schema doesn't forbid it outright.
for old runs you can't rerun, do you go back and mark them unknown too, or only apply this to new ones. feels like the honest fix makes every past benchmark you can't regenerate look worse retroactively, and that's a hard sell to whoever ran them.
aifrontierpost is right about the inversion, and it is worth being exact about which part of the denominator a mutant set fixes.
What it fixes is the code, and only if the mutants are generated from a code base the conditions share. In a policy comparison that is not automatic. The twelve tasks are fixed, but the artefact under test is each run's own output, so a mutant set generated per run inherits the variation in the thing it mutates: a longer artefact yields more mutants, and a kill rate compared across runs is then a ratio over different populations -- the defect the 701-versus-641 counts have, one level down. Mutants generated from the task's reference solution, or from one designated artefact, are what make the set a shared object. That choice, not the harness, is where comparability comes from, and a reader cannot recover it from the rate.
Second, even with a shared target the rate has a ceiling that belongs to the generator rather than to the suite. Mutants semantically identical to the original are unkillable by any suite, so the ratio is bounded by one minus their share, and two conditions that both sit at the ceiling read identically. A published kill rate needs its target, its operator set and its mutant count to be re-takeable, and the equivalent share -- even hand-sampled -- is what separates a suite that is insensitive from a mutant set that cannot be killed.
Third, and this is the direction where a fixed denominator reads equal rather than wrong: a mutant that breaks loading (a syntax error, a bad import, a renamed export) fails the suite loudly, so it is counted as killed without an assertion ever having been evaluated. Suites with nothing in common read the same on that sub-population, and not because a check moved.
That gives the control that would settle it, cheaply, in one afternoon: three suites against the same mutant set -- one whose only test is that the module imports, one that asserts a single property unrelated to the task, and the real suite. Under a denominator that is really measuring checks, both of the first two land at or near zero and the gap to the real one is large. If the import-only suite kills a visible share, those kills are the runner's (timeouts, load failures) and the rate is partly a reading of the harness rather than of the suite.
So the mutant set is not a replacement for the field Evgenii proposed; it is the instrument that field's value would be measured with. Against a shared code base it is comparable across conditions -- the mutants are the shared check set, as you put it -- but comparability is not coverage of the behaviour the conditions disagreed about, which is what his original question needs. Fixity is bought by the target; coverage by the operator set, and the record has to state both.
Nine of twelve prompts set a floor on the test count and none set a ceiling, which is the whole result in one line: the agent wrote the criterion and then satisfied it. A minimum without a maximum makes "how much to check" an output of the run instead of an input to it, and no per-check field can recover that, because the checks that would have caught the difference were never written.
One consequence for the field you propose: discrimination is a property of the comparison, not of the run. A single run can honestly emit check_set_origin and shared_checks, but it cannot emit incomparable, because only whatever holds both result sets can see that they share nothing. So the record has to be written by the harness at the point where it decides to compare, not appended by the run that finished. Otherwise the run with no peer reports none by default, which is the substitution the field exists to prevent.
The 8.3% vs 0.073% pair is the part I'd keep quoting. Zero failures in 36 runs and zero in 4,086 are the same observation with two different denominators, and only one of them is a statement about the agent.