Last week I published a post about nine measurement bugs I found in my own evaluation harness, all of which happened to favour my results. Two readers replied with things I hadn't found.
Both were about my checks rather than my numbers. I verified both and fixed both. My results did not change at all.
That turned out to be the interesting part.
Reader one: a canary that could not detect what it claimed to
Some setup. The harness mutates a source file in a temp copy and runs the test suite against it. To prove that mutations actually reach the interpreter, I had a canary: overwrite the module with unparseable garbage and assert the suite fails.
Vinh Nguyen pointed out that CPython's default bytecode invalidation keys on source mtime in whole seconds plus source size. A mutation flipping < to > is byte-size identical. If a stale .pyc is sitting there with a matching header, Python skips recompilation and imports the original bytecode.
The mutation is on disk. The original behaviour executes. The suite passes. The mutant gets recorded as surviving.
Here is why my canary could not catch that. Unparseable garbage has a different byte length. So the pyc genuinely does invalidate, so the canary fires, so it reports success. It was answering a question I wasn't asking.
I checked the mechanism first.
# compile a pyc from this
a == b
# overwrite with a byte-identical mutation, force mtime back
a != b
# import: stale bytecode executes, original comparison runs
Reproduced on the first try.
Then I checked whether it reaches anything in my harness. It doesn't. Every temp-copy call site passes ignore_patterns("__pycache__", "*.pyc", ...), so no pyc ever arrives in the copy. I verified it by copying a target with 19 real pyc files and confirming zero arrive.
Here is the part I keep thinking about. I wrote that line to avoid copying junk. It turned out to be load-bearing for correctness, and I had no idea until someone pointed at the mechanism.
The reason an instrument is correct is often not the reason you wrote the code that makes it correct. A line added for tidiness that silently does safety work is a line someone removes during a cleanup, with no way to know what it was holding up.
I added a byte-size-preserving canary alongside the original. Eight of twelve targets are eligible and all pass. The other four report N/A rather than being silently skipped.
Reader two: the negative control I didn't have
Ahmet Özel came at it from a different direction. His reading of the original post was that "debugging is triggered by surprise" explains a one-directional bias without anyone needing to be dishonest, which is exactly the argument I was making.
His counter-measure was one I didn't have. Keep a deliberate negative control. A case where the score should be near zero.
Everywhere else in a harness, high is good and low prompts investigation. On a negative control, high is the alarm. That gives you one place where a flattering failure is the surprising one, which is the only condition under which debugging reliably fires.
He also made a sharper point about pre-registration, which I was doing. Writing down your expected number helps when you're wrong. It does nothing when the instrument is broken in a way that produces the number you predicted.
One of my original nine was exactly that. A classifier misread self.assertEqual as "no assertion" while I was testing a hypothesis that models write assertion-free tests. It would have handed me my own prediction with a confirmation attached.
So I built the control. A suite of deliberately vacuous tests. Import the module, call things, assert nothing meaningful. I expected near zero.
It killed 7 of 51 mutants. 13.7%.
That is not a bug. assert x is not None is a real detector, just a very narrow one. It catches exactly one class of mutation: a function that started returning None. All seven kills were that operator. Zero were constant mutations and zero were comparison mutations.
So a suite of vacuous tests has a non-zero floor rather than a zero one. That is a calibration fact about my scoring that I would never have learned from a control that came back clean.
The design constraint is the practical part, and it took me a second attempt to get right. A negative control has to fail loudly if it passes for a boring reason. A near-zero score is also what you get when the module never loaded, which is precisely what bug 1 in my last post produced. Three targets at 0.000 that read as a finding rather than as a fault.
So the control asserts separately that the suite passes on clean source, that the module actually imported and executed (coverage-confirmed, 37 lines), and that both canaries still fire.
Round two: my fix asserted a proxy
Vinh came back after I described the fix and found a hole in it.
My regression test asserted pyc_count == 0 in the temp copy. But set PYTHONPYCACHEPREFIX and bytecode goes to a central tree keyed on the copy's absolute path. Zero pyc files arrive in the copy and the stale read happens anyway.
My test would have passed while the harness was lying.
He ran three branches on his own machine to isolate the real dependency:
- no prefix, fixed work path: fresh
- prefix, same fixed work path: stale, zero pyc in the copy
- prefix, unique directory per mutant: fresh
I reproduced all three. What actually protects me is a second accidental thing. Every copy site uses tempfile.TemporaryDirectory(), which gives a unique path per mutant, which is branch three. I verified it across every call site and ran a full twelve-target sweep with the prefix set. It diffs byte-identical against the committed results.
Two accidental protections. Neither written for the reason it works. ignore_patterns to avoid junk. TemporaryDirectory because it cleans itself up. Both are now commented as load-bearing and name the environment variable explicitly, because anyone adding a --work-dir flag would remove one of them and have no way to know.
The rule underneath this is the most useful thing I got out of the whole exchange.
My test asserted a proxy. Pyc count is a proxy. "The observed behaviour actually changed" is the property. The proxy held under my configuration and failed under a supported environment variable. The property costs the same to assert and does not have that failure mode.
So I rewrote the test to push a byte-size-preserving mutant through the real code path and assert the observed outcome isn't "survived." I added a meta-test that reproduces the stale branch to prove the check has teeth. A regression test nobody has ever seen fail sits in the same category as the canary that couldn't detect this in the first place.
One result that came along the way
While I was in there I closed something I had deferred: hand-labelling the nine mutants the agent couldn't kill. Seven are provably equivalent, six of them type-hint mutations inside TYPE_CHECKING blocks that are never evaluated at runtime. Two are genuine misses, traced to the agent testing the wrong call pattern and the wrong function. So the claim moves from 44 of 53 with an unbounded equivalence caveat to 44 of 46 killable, 95.7%, with the per-mutant arguments published.
Update: the tenth bug, three weeks later.
After publishing this, I packaged the harness as a CLI so other people could run it, then field-tested it against three repos I hadn't built it around: pytest, attrs, and jsonschema.
It returned 0.0000 on pytest. Zero of 35 mutations detected, reported confidently, with no warning that anything was wrong.
It's the src-layout bug from item one of this list. The editable install resolves imports back to the original checkout, so the mutated copy is never executed. Exactly the bug I said I'd found and fixed, which I had - for the twelve repositories I curated by hand.
What I hadn't done was wire the check into the command people actually run. The canary exists. It works. It catches this every time. But it only ran under verify, and score is the command my own quickstart puts first. So a stranger installing the tool, pointing it at a normal src-layout project, would get a confidently wrong number reading "your test suite detects nothing" when the truth is "my harness never executed your code."
The shape is worth sitting with. Both reader findings in this post were about checks that couldn't detect what they claimed. This one is a check that could, and simply wasn't called. Same outcome for the user. And I found it the same way as all the others - not by reading the code, but by running it somewhere I hadn't run it before and getting an answer I couldn't explain.
score and harden now abort on canary failure rather than returning a number. There's a --skip-canary for people who know what they're doing, and it says loudly that the results are unverified. The error message names src-layout as the likely cause and suggests the fix, because "mutations are not reaching this target's test process" tells a user nothing.
One more thing that came out of it. pytest is a hard case - it imports itself before test collection runs, so the usual -o pythonpath=src fix doesn't work. Setting PYTHONPATH before the interpreter starts does. With that, pytest scores 0.657 instead of aborting. So the tool needed a way to set environment variables, not just command-line flags, which I'd never have discovered without a target that broke in an unusual way.
Nine bugs became ten. The tenth is the one I'd have been least likely to find on my own, because everything looked correct from inside the repository where I develop it.
What I actually took from this
Two rounds. Two readers. Four checks improved. Zero numbers moved.
Nobody disputed a finding. All of the scrutiny landed on the instrument, and both times the instrument was wrong in a way I could not have seen by reading it.
That is the argument for publishing the harness rather than the headline. A result is a claim people can take or leave. An instrument is something they can attack, and the attacks are what tell you whether it works. I got more out of two comment threads than out of forty hours of my own review.
Two things worth stealing:
Assert the property, not the proxy. Same cost. One fewer failure mode.
Build one check where a good number is the alarm. Everywhere else, surprise is what triggers debugging, and a flattering bug never surprises you.
Top comments (3)
This is a genuinely humbling follow-up - most people would have quietly patched the canary and moved on. The mtime-granularity detail is a nice trap: any mutation check that changes size is exactly the kind of mutation your fuzzer is least likely to generate. Did you end up seeding your mutator with a size-preserving flip generator after this, or did you switch the invalidation check entirely (hash-based pyc validation via PYTHONPYCACHEPREFIX tricks or source hashing)? Curious which fix held up.
Neither, and the reason is the interesting bit, because your premise is the one I'd have assumed too.
My mutator can't produce size-identical output. Mutants go through ast.unparse, which reformats the whole file, so 0 of 455 come out byte-length identical to the original. The staleness path was never reachable through a normal mutant. Seeding a size-preserving flip generator would have made the harness more vulnerable, not less. It would have created the exact input class that slips past invalidation.
So the size-preserving thing stays on the canary side. It's a deliberately constructed adversarial case whose only job is to prove the harness would notice, kept as a separate artifact from the mutants I actually score. The vulnerability is latent rather than absent, which matters: if someone later swaps ast.unparse for a byte-patching mutator to preserve formatting, which is a reasonable thing to want, the path opens up. The canary is there so that change fails loudly instead of quietly.
One correction on the mechanism, since it caught me out in the same direction. PYTHONPYCACHEPREFIX isn't hash-based validation; it just relocates the cache tree. Hash-based invalidation is compileall --invalidation-mode checked-hash. In my case, the prefix was the attack rather than the mitigation. It's what lets a stale read happen while zero pyc files land in the copy, which is precisely what defeated my first fix.
What actually held was unrelated to either: unique temp directories per mutant, plus not copying pycache. Both accidental, written for cleanup and tidiness rather than correctness. Both are now commented as load-bearing and naming the env var, because the next person adding a --work-dir flag would remove one and have no signal.
The regression test now asserts the property (observed behaviour changed when the source changed) instead of the proxy (pyc count), with a meta-test that reproduces the stale case so the check is known to have teeth.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.