When a Test Suite Rejects a Correct Program

I started with a narrow question: could something without a GPU replace a language model for writing small programs? The answer arrived by elimination, and then the question turned into a different one, because the part I assumed was weak turned out to be fine and the part I had never measured turned out to be broken.

The setup is twenty-one small operations and a property oracle for each, written before any implementation existed. The operations are what an exercise list holds: reverse a string, build a minimum spanning tree, validate a card number with the Luhn checksum (double every second digit from the right, sum the digits of the results, and a valid number totals a multiple of ten), print the first eight happy numbers (replace a number by the sum of the squares of its digits, repeat, and see whether it reaches 1 or falls into a cycle).

A property oracle is the part worth being precise about, because everything below turns on it. It is not a stored list of expected outputs, which would only work for an implementation somebody had already run; it is a program that asserts properties of whatever it is handed, so it can score an implementation it has never seen. The ones here mix four kinds, and the mix matters later. A known value, where the answer is external and settled. eulerian is checked against the actual Seven Bridges of Königsberg: four nodes, seven edges, every vertex of odd degree, and an answer that has not moved since Euler settled it in 1736. A metamorphic property, which needs no correct answer of its own: negating a complex number twice returns the original, so the check holds whatever the program computes. A structural one, where the answer is re-checked rather than its cost: the oracle re-walks a printed spanning tree edge by edge, so a program printing the right total weight beside a set of edges that is not a tree fails. And an independent brute force over every spanning tree, on graphs small enough for that to be exhaustive. A second copy of the same algorithm would agree with a wrong first copy about anything the two misunderstood together.

A blind generator gets the task statement and never sees the oracle. Then the oracle scores it. In the first round of that, the generator passed 9 of 12 operations against frozen suites, where retrieval from a pool of 48 human solutions had managed 2 of 12; a behavioural index did worse still, recognising 0 of 11 operations even when relevance was guaranteed by construction. So the generator was never the scarce thing.

What stayed scarce was the residue. Across four independent generation runs, 11 of 21 operations never passed in any run, and the failing checks were identical run to run for nine of the eleven. I wrote that up twice as capability: graph algorithms are genuinely harder than reversing a string, so of course mst and eulerian sat in the residue.

That was an inference from which operations failed, and it was wrong.

The code lives in a private research repo, so there is no link to it here, and nothing below names a file inside it. Every figure comes from a recorded run rather than from memory.

A wrapper that only re-expresses stdout

The test that settled it is blunt. I wrapped each candidate program in a shim that runs it, rewrites its standard output into the notation its oracle turned out to demand, and hands that to the unmodified oracle. The shim inherits stdin, so it never reads the input, it opens no files, and it computes nothing about the problem. Every transform in it is a rename, a token map, or a re-serialisation of a value the program had already printed.

If a check goes from failing to passing under that wrapper, the program had computed the answer and the contract could not carry it.

For the five deepest failures in the residue, across all four runs:

operation	run1	run2	run3	run4
complex_algebra	3/9 → 9/9	3/9 → 9/9	3/9 → 9/9	3/9 → 9/9
connected	2/7 → 7/7	2/7 → 7/7	2/7 → 7/7	2/7 → 7/7
eulerian	6/11 → 11/11	4/11 → 11/11	6/11 → 11/11	6/11 → 11/11
mst	4/9 → 9/9	4/9 → 9/9	4/9 → 9/9	4/9 → 9/9
graph_from_links	6/9 → 8/9	6/9 → 8/9	4/9 → 4/9	4/9 → 4/9
total	21/45 → 44/45	19/45 → 44/45	19/45 → 40/45	19/45 → 40/45

78 of 180 checks passing became 168 of 180. 90 failures repaired, 12 still failing, and none broken by a wrapper. The remaining two operations in the residue's middle band went the same way: all 16 of their failing checks repaired without touching a line of any program, 12 by re-formatting and 4 by lifting an interpreter limit I will come back to.

Each operation failed for its own reason, and every reason is the same kind of thing.

connected publishes the key. The oracle demands connected=yes; all four generations printed connected=true. The vocabulary was never stated, and the only value check that passes is the one reading a different key.

eulerian does not publish the key at all. The oracle scrapes a path= prefix and splits on whitespace; the task statement named odd_degree and result. Across four generations the trail arrives as trail=A,B,C,D,A, route=A->B->C->D->A and tour=A D C B A: four key names and three separators, none published, every trail correct. One run also answers Königsberg correctly and calls it result=impossible where the oracle reads none.

mst publishes the key and not its type. The oracle parses edges as a comma-separated A-B list; every generation used it for a count and put the tree under tree= or mst_edges=. The same key means a count in another operation's oracle in the same suite, so the contract is not merely silent, it is internally inconsistent. The four checks that pass are the ones reading weight, which includes an independent brute-force optimality check: the oracle could already prove the tree was minimal while being unable to read it.

That spread across four independent runs is the part I would not have got from one. One program using an unpublished notation is a coincidence; four programs inventing four different notations for the same unpublished value is what an underspecified contract does.

The complexity check that field order decides

The sharpest case is fast_expt, whose task is to output a^b in O(lg n) time. Two of its six checks failed in every run, and both are complexity claims, which is a different kind of thing from a notation: if a program computes the right answer by naive repeated multiplication, no re-expression of its output can make the count logarithmic. I predicted, in writing and in advance, that at least one of those two would survive the wrapper for exactly that reason.

It survives. The reason has nothing to do with complexity.

The task statement published value as the only key the checker reads. The oracle reads mults. All four generations print their multiplication count under steps= or multiplications=, and every count is comfortably inside the bound (5, 7 and 10 to 15, against bounds of 11, 15 and 21). Renaming the key repairs one of the two checks in all four runs.

The other check also probes 3**65536. That number is 31,269 decimal digits, and CPython refuses to render an integer past 4,300 digits by default, a limit added as a denial-of-service mitigation and configurable through sys.set_int_max_str_digits. Every generation prints value before its count:

a=2   b=10   value=1024   steps=4
base=2   exponent=10   value=1024   multiplications=5

So every generation computes the right answer with correct square-and-multiply and then dies inside print, and the count the check wants is never emitted. Setting PYTHONINTMAXSTRDIGITS=0 and changing no line of any program, one of them emits steps=17 against a bound of 35.

The consequence is worse than the failure. A program printing its count before its value would pass this check while crashing in exactly the same place, and would pass it whatever its algorithm did, because the count is read and the crash is ignored. The one check in this registry that tests a complexity bound is decided by the order of fields in the output. It is also interpreter-dependent: on CPython 3.10.6 that row never fails, which means the pass rates I published are tied to the interpreter they ran on.

Counting the whole residue, 11 of 21 operations failed on the specification and not one on capability. Two of the twelve unrepairable checks are still the specification rather than the program: every generation collapses two identical input links into one edge, because multigraph support was never published, and two generations refuse the oracle's own weighted A B 4 line as malformed, because the task required a malformed-line refusal without ever publishing the input grammar that refusal is the complement of.

So I built a linter, and it did not survive contact

If a checker demands things it never publishes, and those demands are visible in the checker's source, then the fix is mechanical: read each oracle and emit the contract it actually enforces. Keys, value shapes, vocabularies, the separator a list is split on, the largest magnitude probed. Publish shape and never a value bound to an input, so an alphabetic token like yes is fair game (which one is correct still depends on the input) while anything carrying a digit appears only as its shape.

That works well enough to be worth showing. As a detector for "this operation's contract is missing something load-bearing," measured against which operations actually failed, it scored 91% precision and 62% recall on the twenty-one: ten true positives, one false positive, six missed.

Then I wrote down why that number could not be trusted, which is the part I would want a reader to check hardest. The taxonomy of contract defects was derived by examining those same operations, so 91% is a fit statistic. Worse, the discovery curve had not flattened: the first round of four operations found four defect classes, the second round of five found seven more, and the third round of two still found one new class. Twelve classes from eleven operations, with a new one in the final round, means recall on a suite the taxonomy was not fitted to is unknown and probably much lower.

There was no holdout in the repo to settle it. The other oracle suites there check HTTP routes and database state, so a stdout-contract linter cannot apply; the operations never examined are all single-value positional ones that cannot exercise the classes that matter; and per-check ground truth exists only for the same twenty-one.

One thing worth saying before that number gets any weight: the prompt side of this is already well studied, and there is a literature on ambiguous, contradictory and incomplete task descriptions in exactly these benchmarks. What I had been looking at is the other side of the same seam, where the task statement is fine and the checker is the thing demanding something unstated.

An external one exists, and it is somebody else's work. Datacurve's DeepSWE audit ran ten agent configurations three times over thirty sampled tasks from SWE-Bench Pro, had an independent judge review every rollout against the task definition, reference solution and verifier output, and published the labels. Their finding is that the verifier rejected 24% of correct solutions while accepting 8.5% of wrong ones, and their headline example is the defect class above in another costume: a test suite carrying an import that never appeared in the task prompt, so a functionally identical patch failed to compile and was rejected.

Their labels give 14 tasks with at least one checker-side false negative and 10 with only genuine failures. Before running anything I wrote down the number that decides the experiment, because it is the easiest one to leave out of a favourable table: 14 of 24 tasks are positive, so a predictor that flags every task scores 58% precision at 100% recall. Beating that by ten points at 50% recall or better was the bar.

Against the published spec for each task (the problem statement plus the requirements and interface fields that SWE-bench Pro already ships, since diffing against the prose alone would measure a gap its authors have closed), and demands taken from the added lines of each test patch:

predictor	precision	recall	against baseline
flag every task	58%	100%	n/a
any identifier absent from the spec	58%	100%	+0 points
newly imported symbols only	50%	36%	−8 points
newly called public symbols only	53%	57%	−5 points
imports or called symbols	53%	64%	−5 points

Nothing clears the bar, and two of the three restrictions score below chance. The naive predictor landing exactly on the base rate is the result in one number: in a real repository every test patch names fixtures, helpers and internal APIs that no prose statement mentions, so "something here is unstated" is true of all 24 tasks and separates none of them.

The single thing that explains the gap between 91% and 58% is what the linter was allowed to assume. On the twenty-one operations it could read kv(), the one function every oracle used to parse output, and learn that values are split on whitespace, which is precisely what made the luhn defect statable (a correct vendor=American Express cannot survive that parser, so the right answer is unrepresentable in the published format). Twenty-four repositories in three languages have no single parser to read. An earlier round had already noticed that all 34 oracle functions in the repo take exactly one program path, and that stereotypy is what the 91% was measuring.

A per-harness implementation, one that parses each repository's test framework and knows its assertion conventions, is not ruled out by this. It is also exactly the cost that made a general tool doubtful in the first place, and there is now no cheap-version evidence to justify paying it.

What generalises

The claim I would carry elsewhere is not "check your test suites," which is advice rather than a finding. It is narrower and it has a mechanism.

A pass rate against a fixed checker measures the checker's format discipline and the generator's capability together, and nothing in the number says which. Four independent programs solved eulerian correctly and all four scored between 5 and 7 of 11, entirely on where they put the answer. Had I stopped at the pass rate, the write-up would have said graph algorithms are hard, which is both plausible and false here.

The instrument that made it visible is worth stating too, because it is cheap and it is not a language model: let something generate blind, then repair only the notation and re-score. A person writing to an underspecified contract silently patches it from context and the defect never surfaces; a blind generator does exactly what it is told, which is what makes the gap between what a checker demands and what it publishes measurable at all.

Two things I could not settle. Whether a derived contract actually raises a blind pass rate is unanswerable by me, because it needs a session that has never read these oracles, and having spent two rounds reading them line by line I am no longer that; anything I generated now would be teaching to the test. And the whole series rests on twenty-one operations of a size where the hard case is a minimum spanning tree over six nodes, with one model. The failure mode it found is real and independently corroborated at a much larger scale by the audit above. The rate at which it occurs, on anything bigger than this, I have not measured.