Nine Bugs in My Own Evaluation Harness. Every One Made My Results Look Better.

I built an evaluation harness. I found nine bugs in it. Every single one would have made my results look better than they were.

Nine out of nine, all pointing the same way. That is not a coincidence, and I do not think it is specific to me or to my project. I think it is a structural problem with any evaluation you build for yourself.

Here is the mechanism, before any of the evidence.

Debugging is triggered by surprise

You do not audit numbers. You audit numbers that bother you.

When a result comes back disappointing, you go looking for the reason. You check the setup, you re-run it, you add logging, you find the bug. The bug gets fixed and the number moves.

When a result comes back good, none of that fires. Nothing feels wrong. There is no surprise to investigate. You write it up.

So the filter that removes measurement bugs from your work is applied unevenly. Hard against results you dislike. Softly against results you like. Every pass through that filter removes more unflattering bugs than flattering ones.

Run that loop for a few days and your instrument has drifted in one direction, and nothing in your process is designed to notice. The bugs that survive to publication are disproportionately the ones that helped you. Not because anyone was dishonest. Because they never triggered the thing that catches bugs.

I knew this argument in the abstract before I started. It did not stop me from writing nine of them.

What the harness did

Only enough context for the bugs to make sense.

Mutation testing changes your source code in small ways. Flip a comparison. Change a constant. Delete a raise. Then it runs your test suite and checks whether anything failed.

  def withdraw(balance, amount):
-     if amount <= 0:
+     if amount < 0:
          raise ValueError("amount must be positive")

If the suite stays green, that is a fault your tests cannot detect.

Coverage tells you a line ran. This tells you whether anything would have complained if the line were wrong. Those are very different questions. A toy module with one happy-path test sits at 47% line coverage and a 9.5% mutation kill score.

The harness generated mutations, ran suites against them, and scored how many got caught. Roughly 40 hours of work.

The nine bugs, and which way each one pushed

1. Editable installs made mutations invisible. pip install -e on src-layout packages resolved imports back to the original checkout, so mutations written to a temp copy never executed. Three targets scored 0.000. Direction: reads as "these test suites are terrible" rather than "my harness is broken." It made the problem I was solving look bigger.

2. Parallel execution corrupted one target. Running mutants concurrently produced three different results across four runs on the one target doing real async I/O. A recorded improvement of 0.27 to 0.77 was noise. Spurious failures get counted as the mutation being detected, and detection was the number I was maximising. Direction: inflated the result.

3. A file picker chose an unrelated test file for the hardest target, feeding the model irrelevant context exactly where context mattered most. Direction: made a baseline look worse than it was.

4. A classifier categorised batches instead of individual tests. One good test in a batch of 69 would have marked all 69 as good. Direction: inflated quality.

5. A reconstruction step dropped shared imports and manufactured test failures that were not real. Direction: understated a baseline's capability.

6. An extractor only scanned top-level functions, so a valid unittest.TestCase response was discarded as "no test found." The retry loop then received a harness error instead of real pytest output, which disabled the exact mechanism I was measuring. Direction: understated the agent.

7. self.assertEqual(...) was classified as "no assertion." I was testing a hypothesis about models writing assertion-free tests at the time. Direction: would have manufactured my own hypothesis and handed it back to me.

8. A pre-registered metric was not computable on dunder-dispatched code. It read as a real near-zero rate instead of as undefined. Direction: false signal.

9. A documentation figure that had already survived two audits. One arm was recorded as having zero clean-pass failures when it had three. Direction: flattered a comparison.

Note that they do not all inflate the headline number. Three of them understate a baseline or an arm. That still counts as favourable, because a worse baseline makes the thing I built look better by comparison. "Favourable" means favourable to the story, not favourable to one metric.

Bug 9 is the one I find hardest to be relaxed about. It had been looked at twice. Two audits, both of which read past it, because the figure was consistent with what we expected to see and nothing about it invited a third look.

None of them was found by reading code

This is the part I would most want someone to take away.

Not one of those nine was caught by re-reading the function. I had already read the functions. Reading code that you wrote, looking for a bug you do not yet believe exists, is close to useless.

Every single one was caught the same way: by running a check whose outcome I had predicted in advance, and getting a different answer.

The clearest case was bug 5. I knew, independently and from earlier output, that one specific generated test was broken. So the prediction was simple. Remove that one test, and the suite goes green.

I removed it. The suite did not go green.

That contradiction is the only reason I found the dropped-imports bug before its numbers went into anything. There was no other signal. The scores it produced looked entirely plausible. They were plausible in a direction I liked, which is why nothing else would have prompted me to look.

The prediction is what does the work. A check you run without a prediction just produces another number, and you will interpret that number the same way you interpret all the others.

Two checks worth stealing

Both are cheap. Both caught things.

The canary. Overwrite the file under test with unparseable garbage and assert that the suite fails. This proves your mutation actually reaches the interpreter. If the suite passes while the file is syntactically invalid, you are not testing what you think you are testing. This is what would have caught bug 1 on day one.

The determinism gate. Run the same scoring three times, serially, and require byte-identical output. This proves execution is isolated. This is what catches bug 2.

The important part is that these prove different properties, and neither substitutes for the other.

The canary passes happily while concurrency silently corrupts your results. Your mutation reached the interpreter, so the canary is satisfied, and the numbers are still garbage.

The determinism gate passes happily while imports resolve to the wrong file. Three identical runs of the wrong thing are still perfectly deterministic. The gate is satisfied and the score is meaningless.

You need both, and you need to write down what each one actually proves, so you do not talk yourself into believing one covers the other.

The check that caught one of my own targets

Three hours before my deadline I did a clean-clone reproduction run to verify the reproducibility claim.

The determinism gate quarantined one of the project's own targets. Eleven of twelve reproduced exactly. The twelfth varied.

I reported it in the README instead of fixing it.

A check that has never caught anything is indistinguishable from a check that cannot catch anything. Mine had just caught something, and the something was mine. Removing that from the record would have made the project look better and the instrument look worse, which is exactly the trade this entire post is about.

What to actually do

Before you measure anything you built:

Write down what your instrument would look like if it were lying to you. Then build the check that catches specifically that, and run it before you have any results you are attached to.

Then write down which direction each possible lie would push your result.

That second list is the one that matters, because it is a list of the checks you will be least motivated to run. Every item on it is a place where a bug will feel like a finding. You will not notice those on your own. Nobody does. That is the whole reason the nine came out nine for nine.