The tool has a self-test. It plants two positive fixtures that must be found and four negative fixtures that must stay quiet. Then it prints a control line, and you read the real numbers only after that line is green.
It printed the control line. All four negative fixtures were clean. Both positives were reported as seen.
The skip list contained fixtures.
Zero files were scanned. Every fixture directory was excluded by name before the walk reached it, so the four negatives were clean in the way that a locked room is quiet. The control passed on an empty set.
The fix is one line and one idea
The line: a fixture run where filesScanned === 0 is untestable rather than ok, and it exits non-zero.
The idea took longer. A control that reports only pass or fail cannot separate "the thing did not happen" from "the thing happened and was fine". Those need different spellings, and folding the first into the second is a reliable way to lose a bug forever.
I added two mutation controls as well: deliberately broken builds, one with a term removed from the vocabulary and one with the pattern damaged. Each must miss the planted defect and exit 1. If a broken build still passes, the control is measuring nothing. That is the check on the check, and it stops there. Two stages is where I decided the regress ends.
The self-test now prints which signal saw which positive fixture, so a fixture passing for the wrong reason is visible in the output.
What I got wrong
The whole thing, obviously. The specific failure is worse than a bug, though.
I had already written the principle down. Do not fold a measurement you could not take into a failure, and do not fold a measurement of nothing into a success. I have argued that in public. I have made it the pitch for a piece of software. Then I wrote a self-test that scanned zero files, read the silence as agreement, and did not notice until I went looking for something else entirely.
Writing the rule down is not a defence against breaking it. It may have made this worse, because part of why I trusted the tool was that I knew about this failure mode.
What I did not check
The mutation controls cover two of the five signals. The other three have no planted-failure control, so for those the self-test proves only that something matched. Passing for the right reason is visible there, not enforced.
Nothing here measures false negatives in general. A fixture suite tests the cases you thought of. The defect that started this whole line of work was found by a person reading a file after the machine returned zero, which is exactly the case no fixture covers.
I have not run the self-test on another machine. Skip lists interact with path separators, and every path in this session was Windows-shaped.
Trace: ledger bands/decisions/01_RULINGS.md entry D-G012, defect 1 of the 3 recorded there, with the mutation controls recorded in D-G015.
Repository: crates/gx-substrate-conformance is the suite that exists so an implementation cannot pass by doing nothing.
Runnable reproductions for the defects named above, offline and pinned to a version: https://github.com/mahirhir/unanswered-approval
Top comments (0)