Between 20 August and 31 August I published ten small packages, eight of them on the last day. The first of the eight repositories was created at 06:05 UTC and the last at 18:34. They are small verification and measurement tools, each in Python or JavaScript or both, installable from PyPI and npm, and each encodes one way a green check can mean nothing. Nothing about the code was written in a day. The packages are extractions from the measurement work behind much of what is on this site. The recurring failure there was almost never the thing being measured; it was the instrument reporting success without having checked anything. A memory profiler that watched the wrong process for eight months and a rate limiter that reported requests it never sent are both that failure. The research codebase these packages fell out of is, like the ones in those posts, not public. The packages are the public form of what it kept teaching.
The failure has a family resemblance wherever it appears. go test ./... on a tree with no test files prints [no test files] and exits 0. A pytest suite in which every test is skipped prints 2 skipped and exits 0 (measured on pytest 9.1.1). A JUnit report can read tests="50" skipped="50", which is a green run of nothing wearing a total. A lint rule disabled by a config merge is green forever, and green is what everyone was looking for. In each case a check stopped checking, and nothing about the repository looks wrong.
The rule all ten packages are built on is stated in didrun's README: a check must answer separately whether it ran, whether it failed, and whether the failure was the right one. Collapsing any two of those three is how every defect in this family happens. The corollary is the title of this post. A guard that has never failed may be incapable of failing, and the only way to know is to give it something it must fail on.
So here is the claim, and a stranger can check it one repository at a time: every one of the ten commits at least one test that would report the package's own premise wrong. Not a test that the tool works; a test that the naive alternative fails where the tool succeeds, or that fixtures can force the tool's verdict in both directions. If any repository below lacks such a test, this post is wrong about it. Two terms recur, because the family shares its probing machinery: a ladder is a fixed list of probe inputs a tool walks in order, and a witness is an input that produced two different answers. And assay-checks appears twice, because it carries two unrelated questions in one binary and each needs its own control, so the table has eleven rows for ten packages.
| package | the way a check lies | the committed control |
|---|---|---|
| assay-checks, a duplicate-function finder that executes code | two functions maintained separately answer identically | pairs come from executed outcome vectors, never names, and functions that genuinely differ must not group |
| nondet, a determinism prober | an in-process repeat calls a function deterministic when hash order only moves between processes | the in-process check must find nothing in 20 calls while fresh processes find the witness |
assay's runners half, an auditor of mutation harnesses |
no failures reported and no test executed look identical; a crash scored as a catch | seven properties, each shipped as a mutation its own runner must catch |
| restore-verified, a guard for tools that break files on purpose | a restore that ran is read as a restore that worked | a control asserts try/finally really does leave the tree broken on SIGTERM |
| didrun, a command wrapper demanding evidence of work | exit 0 is read as evidence that something happened |
0 passed matching the expected pattern must still score as did-not-run |
| canfail, a breaker of guarded files | a CI guard is green because it cannot go red | the example config must yield a catch, a blind guard and two refusals in one run, and CI asserts that tally line |
| undetermined, a curve fitter that can refuse | a constant is fitted to a drift and reported without the error bar that decided it | the demo's second observable must come back UNDETERMINED while the first does not |
| zerocase, a reader of the reports runners already write | a zero denominator reports clean | a full report and an empty one from the same command shape must get opposite verdicts |
| countfn, an operation counter for scaling questions | a complexity class is named because a curve fit looks close | three functions must produce three answers at once: n², log n, and a refusal |
| ladderpin, a freezer of measured behaviour | behaviour drifts under tests that stay green; a flaky pin blames the pinning tool | with the determinism gate off, a pin on an unchanged tree must go red |
| lexindex, a code completion engine over repository statistics | a completion accuracy is quoted with no baseline beside it | the shipped harness must exit 2 when its scorer was never observed producing both a hit and a miss |
Three of the controls are worth reading in full, because each one tests the reason its package exists rather than any feature of it. test_in_process_repetition_would_have_missed_it asserts that calling a function 20 times in one interpreter finds no variation, and that nondet finds it anyway in fresh processes, where Python's per-interpreter hash randomisation actually varies. If in-process repetition ever catches it, fresh processes are expensive theatre and the test says so. test_SIGTERM_leaves_a_try_finally_harness_broken spawns a real child running an ordinary try/finally harness, kills it, and asserts the mutated file stayed mutated. restore-verified exists because finally does not run on SIGTERM, and this is the test that would report that premise obsolete. And test_without_the_gate_the_same_function_is_pinned_and_the_pin_is_flaky turns off ladderpin's determinism gate, pins a function whose output depends on hash order, changes not one byte of the tree, and asserts the next check reports a change that nobody made.
The tools were then turned on themselves, and the results that mattered were the failures. Seven of the ten READMEs report a deliberate-mutation pass over their own source, from five mutations in restore-verified to 193 in assay, and the survivors were worth more than the catches. In canfail, the mutation that re-enabled mtime restoration survived, which proved that an earlier fix did nothing. mtime invalidation has one-second granularity, and a tool that edits, runs and restores in milliseconds defeats it whichever way the flag is set; the load-bearing guard is PYTHONDONTWRITEBYTECODE, and the README had to be corrected. In zerocase, two mutations survived their first run. One exposed a branch reachable only through --allow-stale that nothing exercised. The other exposed that no fixture held a test case carrying both a skip and a failure, a shape real runners emit. Two packages also hit the same third category: a mutation that cannot change behaviour at all still scores as survived and reads as a test gap. A mutation suite can lie in the flattering direction as well as the other one.
Building the packages against each other found defects none of them found alone. countfn is built on undetermined, and its parity suite compares the rendered report between the Python and JavaScript halves. On the first run after that comparison was added it caught the two halves of undetermined formatting any integer of one million or more differently, truth=1e+06 against truth=1000000. undetermined's own parity suite asserts those strings agree, and had passed forever, because its test ladder tops out at 512. The checker's checker needed a checker, and the general lesson is the one the mutation-testing post reached from the other side: an instrument is only as trustworthy as the thing that has been shown to break it.
The same discipline shaped the dependency graph. canfail originally carried 78 lines of inline restore logic, about a quarter of the module, so that it could claim no dependencies. That was right while restore-verified was unpublished and wrong afterwards: a second copy of a guarantee is a second thing to get wrong, and the copy is the one that does not get the upstream's tests. undetermined went the other way and rejected a proposed dependency on nondet, since its observables are closures with no file-and-name address for nondet to probe. A dependency that looks like a guarantee and is not is worse than no dependency.
The cost of all this refusing is coverage, and the numbers are small and printed rather than hidden. nondet, pointed at a real 283-function tree, probed 127 of them with the safety gate on when that census was taken, and found 2 genuinely nondeterministic. assay's census on a 41-function tree probed 9, and ladderpin can only pin what assay probes. lexindex, the one package here that is a product rather than an instrument, states the condition under which it is worth installing as a single number, the recital rate (how often a short context in a held-out file already appears in the index), which ran from 13.5% to 72.9% across the nine corpora measured. Its harness reports its own nulls instead of suppressing them. Every census (each tool's printed count of what it refused and why) names its refusals, because a report that stays quiet about what was never examined is reporting we never looked as we found none.
What generalises is the asymmetry the whole family is built around. A witness is a fact: an input and two different answers to it, a file that did not come back, a zero where a denominator belongs. Agreement is only the absence of a witness, and a check that has never disagreed with anything has not yet demonstrated that it can. Before trusting any green light, mine included, the question worth a minute of anyone's time is the one in the title: what, concretely, would make this check fail, and has that ever been watched happening?
Top comments (0)