A test that only checks the type of a return value passed three of the five bugs I planted underneath it. A test whose only assertion was that an empty list comes back as an empty list passed four of five. Both count as success under the metric most agent benchmarks use, because both pass the reference implementation.
I reproduced TestPrism's joint-success metric (arXiv 2610.12289v1) on a small corpus I could enumerate completely, and then split the failures by cause. The split is the part I had not seen quantified before: over-specification and under-assertion are about equally common, and the metric cannot tell you which one you are looking at.
The metric, in the paper's own terms
TestPrism scores a generated test suite per task. For task i there is a buggy initial state b_i, five valid candidate implementations V_i, and five invalid ones I_i. Joint Success Function is
JSF = (1/N) * sum_i r_i(b_i) * prod_{p in V_i} a_i(p) * prod_{p in I_i} r_i(p)
where r_i(b_i) is the suite failing on the initial state, a_i(p) accepts a valid candidate, and r_i(p) rejects an invalid one. Single-reference success is the same shape over one implementation. The paper reports 59.67% single-reference against 28.00% joint across fourteen agent configurations, on 300 tasks and 3000 labelled implementations.
Their corpus is not published — the full text of the HTML version carries no artifact link for it — so nobody outside can recompute those two numbers on their panel. That is why I built my own.
What you can run
The whole harness is this. CORPUS is a dict of tasks, each holding one buggy state, five valid candidates, five invalid ones, and six suites; every candidate defines f.
def load(src):
ns = {}
exec(compile(src, "<candidate>", "exec"), ns)
return ns["f"]
def passes(suite_src, impl):
try:
exec(compile(suite_src, "<suite>", "exec"), {"f": impl})
return True
except Exception:
return False
# per (task, suite)
fails_buggy = not passes(suite, load(buggy))
accepted = [n for n, src in valid if passes(suite, load(src))]
rejected = [n for n, src in invalid if not passes(suite, load(src))]
ref = fails_buggy and passes(suite, load(valid[0][1]))
jsf = fails_buggy and len(accepted) == 5 and len(rejected) == 5
Six tasks, six suites each: 36 measured pairs. Every task is a small pure function whose spec leaves one dimension open — rounding, case, punctuation, container type, key order, transliteration — and the valid candidates differ only on those open dimensions. One of them, whole_minutes, is spec'd as "return the number of whole minutes in a duration given in seconds"; seconds // 60 and round(seconds / 60) are both legal readings of it.
I wrote the candidates first, choosing what seemed natural for the spec, and only then wrote the suites the way a suite gets written when its author has the reference open: by asserting what that implementation does on edge inputs. No candidate was edited afterwards to break a suite. That ordering is the only thing standing between this and a rigged result, and you should hold it against me if the numbers look too clean.
The result
| Reading | Value |
|---|---|
| single-reference success, all pairs | 20/36 = 55.56% |
| joint success, all pairs | 1/36 = 2.78% |
| single-reference, pairs that fail the buggy state | 20/22 = 90.91% |
| joint success, same pairs | 1/22 = 4.55% |
The direction matches the paper and does not match its magnitude. My valid candidates differ more aggressively than theirs, so joint success collapses further. Treat the ratio as not reproduced; treat the direction as reproduced.
The part I did not expect: two failure modes, indistinguishable
Of the 20 suites that pass the reference implementation, one also passes jointly. The other 19 split like this:
| How the suite failed | Count |
|---|---|
| rejected at least one valid candidate | 8 |
| accepted at least one invalid candidate | 6 |
| both | 5 |
Eight suites were over-fitted to the reference: palindrome/case_sensitive asserts f("Aba") is False because the reference is case-sensitive, and three of my valid candidates lowercase first. Six suites were too weak: the same suite accepted four of five planted defects. Five did both at once.
A single joint-success number folds all nineteen into "failed". If you are using that number to drive an agent's repair loop — generate a suite, score it, tell the model to try again — the score tells the model that something is wrong and nothing about what. Over-specification wants the assertion loosened; under-assertion wants it strengthened. Those are opposite edits, and the metric cannot distinguish the states that call for them.
A third of my suites never failed the buggy state
Fourteen of the 36 suites passed the buggy initial state, which sets r_i(b_i) = 0 and scores them zero under both metrics. That is correct per the definition, and it hides something: percentile/median_odd asserts f([1, 2, 3, 4, 5], 50) == 3, and my buggy implementation — which forgets to sort — happens to return 3 for that input. The suite is not over-specified. It is a test that does not test.
So a zero under joint success means at least three different things: the suite rejects a legal implementation, the suite accepts a broken one, or the suite cannot see the bug at all. Thirds of my corpus, roughly: 14, 8+5, 6+5. Three repairs, one score.
Type assertions and empty-input assertions are not tests
The two most contentless shapes I found, both of which pass a reference implementation:
-
assert isinstance(f(120), int)accepted three of five defects, including one that returns hours instead of minutes and one that is off by one. -
assert f([], 2) == []accepted four of five, including a chunker that drops the remainder and one that pads the last chunk withNone.
Neither assertion can fail for a wrong answer on any non-empty input, so neither constrains the behaviour the task is about. If you are writing or generating tests, these are the two lines to delete first. They are also the two lines most likely to survive an agent's self-review, because they pass.
The number depends on the panel
The paper's own analysis notes that joint success depends on which implementations the panel contains, and studies growing panels. My corpus is small enough to enumerate every panel, so here is the same suites scored on every sub-panel that can be drawn from five valid and five invalid candidates:
| Panel | Evaluations | Mean joint success |
|---|---|---|
| 1 valid + 1 invalid | 900 | 35.44% |
| 2 + 2 | 3,600 | 23.22% |
| 3 + 3 | 3,600 | 14.61% |
| 4 + 4 | 900 | 8.00% |
| 5 + 5 | 36 | 2.78% |
Same suites, same implementations, a factor of 12.7 between the smallest and largest panel. A joint-success figure quoted without its panel size and composition is not portable between corpora. That is not a flaw in the paper — it reports one panel and studies growth on it — but it does mean the headline ratio cannot be compared across papers unless the panel is held fixed, and here the panel is what moves.
What I would change in an agent loop
- Score the two directions separately. Rejecting a valid candidate and accepting an invalid one are different defects; report
valid acceptance rateandinvalid rejection ratealongside the joint number, as the paper also does, and let the repair prompt see which one moved. - Keep a few candidates that differ only in format and type. If a suite fails those, it is over-fitted and the fix is to loosen it — cheap to detect, and impossible to see from a single number.
- Never let a type-only or empty-input assertion count as coverage. Both passed a majority of planted defects here.
- Report the panel. If the number moves 12.7x with panel size on six hand-built tasks, it moves on three hundred real ones too.
Limits
Six tasks, 36 suites, all authored by me, on specs I wrote to be deliberately under-determined — which makes this the hard case, not the average case. I authored both the corpus and the suites, and the only defence against a rigged result is the authoring order stated above plus the corpus being short enough to read. The single-reference figure of 90.91% on eligible pairs is not comparable to the paper's 59.67%: different corpus, different tasks, and my valid candidates are more diverse than theirs by construction. Nothing here disputes their finding; it adds the decomposition and the measured panel sensitivity.
If you want to run it yourself, the harness above plus one task's five valid and five invalid candidates is enough to reproduce a single row of the first table, and that is where I would start. The interesting number is not the joint success rate — it is how often a suite passes the reference while being unable to fail.
I build EMRG, a self-evolving coding agent, at github.com/argszero/emrg; the reason this measurement exists is that its gates were passing when they should have failed, which is the same defect one layer up.
Top comments (0)