TL;DR: A test can pass without executing the control it names; mutation and path coverage found 59 refusal paths no test reached. The parent slice log carries the failure.
Your safety check has a test with a reassuring name. Does that test execute the safety check? You cannot answer by reading the name, the coverage total, or a green suite. One cleanup control had never worked on any Python the project supports, while its test replaced the very function it claimed to cover.
In this note
- The test that proved less than its name promised
- Measure the general form, not your favorite example
- Go hunt these shapes in your own safety net
- What the recorded proof says
- Break the check before it blesses the code
That is how a green suite becomes theater. The test is present. The control is present. The binding between them is absent.
When the general form was measured, 59 refusal paths had no test execution at all. None of those 59 are failures or weak assertions. 59 refusals that no test reached.
The test that proved less than its name promised
The named test never executed the cleanup function it claimed to cover. It replaced _remove_materialisation with a stub, so it proved only the surrounding precedence logic.
SLICE-004 isolated the runner, its toolchain, and its environment so Ranex could observe a materialisation of the subject commit rather than a tree chosen by the party being measured. Its first close claimed that its controls had been mutation-checked.
Then a cleanup path broke the claim open.
The materialisation must be removed after a run, including after a refusal. The intended cleanup error handling failed on Python 3.11, 3.12, 3.13, and 3.14. On Linux, the removal path could call an error handler with os.open, which needs a second argument. The handler called it with one. That raised TypeError, escaped from a finally block, and replaced the original refusal.
The control had never worked on any supported Python. This was not a new regression. The first closure had declared a fixed behavior that did not exist.
The named test was the worse part. It monkeypatched _remove_materialisation out and replaced it with a stub that raised SubjectError. So it exercised precedence logic in materialise_subject. It never ran the cleanup function it was named for.
That is a shape you should learn to fear: a test whose setup removes the thing its title says it proves. The test can be impeccably formatted. It can pass for years. It can still certify a path no real run takes.
The original mutation check was also run by hand by the actor who wrote the code. The slice says exactly why that matters: it was a self-report, and it missed the defect. This is not an accusation of bad intent. It is a statement about the limits of a workflow that asks one actor to build the control, choose how to break it, and summarize the result.
Measure the general form, not your favorite example
One broken cleanup handler was a symptom. Measuring refusal paths across src/ranex/ found 59 raise statements and except bodies no test executed.
That is the move. Do not stop at “did this one example fail?” Ask the larger question that could embarrass the entire pattern.
The unreached paths included kernel input validation, five pinned-toolchain refusals, unsafe-path and duplicate-entry guards in the materialiser, and one branch of the journal chain check. Any one of them could be wrong in the same way the cleanup handler was wrong.
After reopening, SLICE-004 added mutmut and diff-cover. diff-cover prevents a future change from adding a line no test executes. mutmut replaced the hand-run claim with recorded output.
The whole-package mutation run produced 2,596 mutants: 1,636 killed, 73 with no tests, 7 timeouts, and 880 survivors. The larger figure is a map of debt, not a victory lap.
Fifteen of the 59 unreached refusals were closed. Forty-four remain. Some of the 880 surviving mutants are in the kernel. They are recorded instead of being rounded into a tidy conclusion.
A check nobody has tried to break is a check nobody knows works.
Mutation testing does not make a test suite omniscient. The slice documents that mutmut excludes several subprocess-driven tests, which makes its signal for parts of the CLI noise rather than evidence. That limitation is a reason to say where the measurement applies, not to abandon it.
Go hunt these shapes in your own safety net
You can start without adopting Ranex or redesigning your test stack. Look for the distance between a control, the test named after it, and the behavior your system actually takes.
- Choose a refusal, validation, cleanup, or recovery path that matters. Confirm a test reaches that exact function rather than a stub standing in for it.
- Delete or invert the control in a disposable branch. Watch the covering test fail. If it stays green, the test is not bound to the control.
- Search for tests that monkeypatch the function named in the test title. Read what remains after the patch replaces it.
- Measure all error and refusal paths in the relevant package, not only the one you planned to fix.
- Add a changed-lines coverage ratchet so a newly added path cannot enter unreached.
- Record mutation survivors, timeouts, and exclusions. Do not call a tool’s blind area coverage.
Yes, this is slower than saying “the suite is green.” A safety net is supposed to slow a fall, not decorate the floor.
What the recorded proof says
SLICE-004 closed with the cleanup behavior made version-independent and proven against a real mode-0 directory. It replaced a hand-run mutation claim with tool output, and diff-cover reported the newly added cleanup at 100%.
It also says what remains unclosed. The kernel’s verdict.py had 47 surviving mutants with zero unreached mutants. One inspected survivor inverted the success comparison inside the contradiction check, and no repository test detected it. That is named as a test gap, not treated as proof the behavior is wrong.
This distinction matters. A surviving mutant is evidence that a test did not distinguish a code change. It is not automatically evidence that the system fails in production. A green test suite is not automatically evidence that the relevant branch ran either. Keep the claims separate.
Ranex is pre-release. Its README says it has a working verdict path and very little else, while significant production hardening remains unstarted. The accountability apparatus is not a slogan if it hides its own blind spots. The slice records are in the repository under docs/slices/done/ precisely so the unresolved counts stay visible.
Questions people actually ask
These questions help you check whether a named test reaches the safety control it claims to cover.
What can mutation testing reveal that line coverage misses?
Mutation testing found 59 raise statements and except bodies in src/ranex that no test executed at all.
Why was SLICE-004 reopened?
A cleanup control had never worked on supported Python, and its named test monkeypatched out the function it claimed to cover.
What changed after reopening SLICE-004?
The slice added mutmut and diff-cover, closed 15 of 59 unreached refusals, and recorded 44 remaining refusals plus 880 surviving mutants.
Break the check before it blesses the code
Pick one safety check this week. Make the guarded condition happen. Remove the control in a disposable branch. Verify its named test turns red. Then measure the rest of that control family, because one passing example cannot tell you the family is covered.
Try it. Break it. Tell me what broke. Star Ranex on GitHub if the record helped, and send the honest critique. Especially if you find a measurement that is only pretending to measure.
Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records, the same fact gate the product enforces on code. It ships only after Anthony’s own review.
Top comments (0)