I built a measurement harness. I found ten bugs in it. Every one made my results look better than they were, and not one was found by reading the code.
Here is the checklist that came out of that. It is meant to be used rather than admired, so each check comes with what it proves, what it does not prove, and roughly what it costs to write.
Why the bugs point one way
Debugging is triggered by surprise. A pleasing result is not surprising.
So the filter that removes measurement bugs from your work is applied unevenly. Hard against results you dislike, softly against results you like. Every pass removes more unflattering bugs than flattering ones, and after a few days your instrument has drifted in one direction while nothing in your process is designed to notice.
That is the whole mechanism. It does not require anyone to be dishonest, which is why being careful does not fix it.
I wrote that argument up in full, with all ten bugs and the direction each one pushed, here. The rest of this post is the checklist rather than the case for it.
Check 1: the canary
Replace the thing under test with something unparseable and assert that your harness notices.
def test_canary_reaches_interpreter(target):
with swapped_source(target, "this is not valid python"):
result = run_suite(target)
assert result.failed, "harness did not notice a broken module"
Proves: your change actually reaches the interpreter.
Does not prove: that the change is isolated, or that your scoring is correct, or that a valid change reaches the interpreter. Mine passed happily while a same-length mutation was being silently ignored, because unparseable garbage has a different byte length and therefore invalidates Python's bytecode cache. Vinh Nguyen (@vinhnguyenthanhdn) found that one. A byte-size-preserving variant of the canary closes it.
Caught in my project: editable installs. pip install -e on src-layout packages resolved imports back to the original checkout, so modifications written to a temp copy never executed. Three targets scored 0.000, which reads as "these test suites are terrible" rather than "my harness is broken."
Write this before you write any measurement at all. It is the cheapest check on the list and it catches the most embarrassing class of failure.
Check 2: the determinism gate
Run the same scoring three times, serially, and require byte-identical output.
runs = [score(target) for _ in range(3)]
assert runs[0] == runs[1] == runs[2], f"nondeterministic: {runs}"
Proves: execution is isolated between runs.
Does not prove: that you are executing the right thing. Three identical runs of the wrong file are still perfectly deterministic.
Caught in my project: parallel execution. Running mutants concurrently produced three different results across four runs on the one target doing real async I/O. I had already recorded an improvement off that data. It was noise.
The important thing is that checks 1 and 2 do not substitute for each other. The canary passes happily while concurrency corrupts your results. The determinism gate passes happily while your imports resolve to the wrong file. You need both, and you need to write down what each one actually proves so you do not talk yourself into believing one covers the other.
Check 3: the negative control
Build one case where the score should be near zero.
# a suite that imports, calls, and asserts nothing meaningful
def test_negative_control():
score = run_scoring(vacuous_suite, target)
assert score < THRESHOLD, f"vacuous suite scored {score}"
Proves: your scoring is not generous in a way you would otherwise never look for.
Does not prove: anything about the top of the range.
This is the one I did not have, and it came from Ahmet Özel (@ahmetozel). The logic is worth stating carefully. Everywhere else in a harness, high is good and low prompts investigation. On a negative control, high is the alarm. That gives you exactly one place where a flattering failure is the surprising one, which is the only condition under which debugging reliably fires.
There is a critical design constraint. The control must fail loudly if it passes for a boring reason. A near-zero score is also what you get when nothing loaded, which is precisely what my first bug produced. So the control has to assert separately that the suite passes on clean source, that the module actually imported and executed, and that your other checks still fire.
Mine did not come back at zero. It came back at 7 of 51, and the explanation was a genuine calibration fact: assert x is not None is a real detector, just an extremely narrow one. It catches exactly one class of fault. All seven kills were that one operator and zero were anything else.
So a suite of vacuous tests has a non-zero floor rather than a zero one. A control that came back clean would have taught me nothing.
Check 4: assert the property, not the proxy
When you write a regression test, check the thing you care about rather than a signal correlated with it.
# proxy: correlated with correctness, until it isn't
assert count_cache_files(workdir) == 0
# property: the thing you actually need to be true
before = observe(target)
with mutated_source(target):
after = observe(target)
assert after != before, "source changed but behaviour did not"
Proves: the behaviour you depend on actually holds.
Does not prove: that you picked the right property. But it removes a failure mode the proxy has for free.
Caught in my project: my fix for the bytecode problem asserted a cache-file count of zero in the temp copy. Set PYTHONPYCACHEPREFIX and bytecode goes to a central tree keyed on the copy's absolute path. Zero cache files arrive in the copy and the stale read happens anyway. My test would have passed while the harness lied. Vinh found that too, and ran three branches on his own machine to isolate which variable mattered.
The proxy held under my configuration and failed under a supported environment variable. The property costs the same to assert and does not have that failure mode. That is the entire argument.
Add a meta-test that reproduces the broken case, so you know the regression test has teeth. A regression test nobody has ever seen fail belongs in the same category as a canary that cannot detect what it claims to.
Check 5: predict the outcome before you run it
This is not a check you write. It is a habit, and it caught four of my ten.
Before running any diagnostic, write down what you expect to see. One line is enough. Then run it, and treat any contradiction as stop-and-investigate rather than as something to rationalise on the spot.
There is no code for this one, which is precisely why it is easy to skip. The prediction has to exist before the output does, and nothing in your tooling will remind you.
Proves: nothing on its own.
Does what nothing else does: it manufactures surprise where none would otherwise exist. A check you run without a prediction just produces another number, and you will interpret that number the same way you interpret all your other numbers.
Caught in my project: I knew, independently, that one specific generated test was broken. So I predicted that removing it would make the suite go green. It did not. That contradiction is the only reason I found a bug where a reconstruction step was dropping shared imports and manufacturing failures that were not real. The scores it produced looked entirely plausible, and plausible in a direction I liked.
Check 6: run it where you don't develop it
Install into a fresh environment, outside your own tree, and point it at a repository you did not build around.
cd $(mktemp -d)
pip install <your-tool>
<your-tool> run --target some-repo-you-never-tested-on
Proves: the thing works for someone who is not you.
Does not prove: correctness. It proves that your correctness is reachable by someone else, which is a separate problem. Your checks can all be sound and still never run for a user, because the path they take through your tool is not the path you take.
Caught in my project: my tenth bug, and the one I find most instructive. I had fixed a problem across twelve curated repositories. Then I shipped a CLI where the check that catches that problem was not wired into the command my own quickstart tells people to run first. A stranger pointing it at a normal project would get a confident 0.0000 with no warning at all.
Everything looked correct from inside the repository where I develop it. That is the point. Your development environment is the one configuration you have accidentally optimised for, and it is the one your users are least likely to reproduce.
What this checklist does not cover
Every check above validates the execution path. That your change reached the interpreter, that runs are isolated, that the behaviour actually differs. None of them validates the scorer: the code that reads a result and decides what it means, then aggregates that across everything else.
That gap is not theoretical. Three of my ten bugs were scorer-level. A classifier run on the wrong unit, a reconstruction step dropping shared imports, an assertion style the classifier couldn't see. Not one was caught by a check on this list. All three came from check 5, predicting an outcome and hitting a contradiction, which is a habit rather than a control.
Zain Dana Harper (@zaindanaharper) put the general version of this better than I can: an intact artifact tells you nothing about whether the thing interpreting it is correct. A report can be perfectly well-formed, every byte verified, and carry a wrong number.
I'm building three checks for this now. Known-outcome fixtures that assert the pipeline reports an answer you know by construction. Conservation invariants so counts have to reconcile at every stage. Explicit unit metadata on anything carrying a count, so a consumer can assert what it's being handed rather than assume.
The first one found a bug on its first run. My outcome classifier decided a result was a collection error only if the output contained a specific phrase, one that the pytest version I'm on never actually emits. So that branch had never fired, for any target, ever. Worse than the bug: I had published the empty bucket as a finding. "Zero error outcomes across all twelve targets" read as reassurance about data quality when it was the signature of dead code. A uniform zero across twelve independent targets should have been suspicious on its own. Real data is rarely that clean.
That one doesn't fit the pattern in the rest of this post, and it's worth naming why. It didn't inflate a metric, and the primary number is unaffected. What it did was make a null result look like evidence. Until the scorer checks are finished, treat this list as covering execution and leave interpretation to check 5.
The order to do these in
- Canary first, before any measurement exists at all.
- Determinism gate before you trust any number that comes out.
- Negative control before you interpret a good result.
- Fresh-environment test before anyone else runs your tool.
- Property-not-proxy every single time you write a regression test.
- Predict-then-run on every diagnostic, forever.
The ordering matters because each one gets harder to run honestly once you have results you are attached to.
Three traps that no check catches
These are scoring decisions rather than checks. Nothing on the list above will catch them, so you have to decide them deliberately and then say what you decided.
Batch versus per-test scoring. If one broken item invalidates a whole batch, your number is a floor rather than a measurement. Both are defensible. Publishing one while implying the other is not. Say which you have.
Budget matching. When you compare approaches that spend resources differently, there is no neutral unit. Matching calls starves one arm of output. Matching output tokens effectively rebuilds a different arm. Every choice advantages someone. Report the full resource vector, and say plainly that the unit was chosen before you saw the results, assuming it was.
Controls die at small n. I built two versions of a held-out control and abandoned both. One ended up with a denominator of 1. The right move is to state that you have no control, not to dress up something weaker and call it one. A weak control that nobody flags is worse than an admitted absence, because it transfers confidence you have not earned.
What this cost and what it bought
Ten bugs, all flattering, none found by inspection.
Then two readers found two more by pointing at my checks rather than at my numbers. Neither time did a result move. Nobody disputed a single finding. All of the scrutiny landed on the instrument, and both times the instrument was wrong in a way that could not be seen by reading it.
That is the argument for publishing the instrument rather than the finding. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works.
I am considering putting the full version of this together, with the checks as drop-in code, the pre-registration and commit-ordering templates, and the eval set as a worked example.
If you build evaluations: what would you want in it that isn't here, and what would you not bother with? The second question is the more useful one to me.
Top comments (0)