A Detector That Only Ever Says "Clean" Proves Nothing

Volodymyr — Sun, 16 Aug 2026 17:09:09 +0000

A few days ago I asked my agent to count how many of my tooling scripts carry a self-test. It grepped and answered: 12 of 13.

The number was wrong. One script labels its control НЕГАТИВНИЙ КОНТРОЛЬ — uppercase — and the probe's regex was lowercase with no -i flag. The real answer was 13 of 13.

A probe written to find blind detectors was blind. It returned a clean, specific, entirely plausible number, and nothing in its output hinted that it had missed anything. I caught it only because the total felt one short and I opened the file by hand.

That is the whole problem, and it took ninety seconds to demonstrate on myself.

The detectors are multiplying

If you work with a coding agent, you are accumulating detectors far faster than you notice. Not tests — detectors. Pre-commit hooks. Custom lint rules. Audit scripts. "Check that no doc references a deleted file." "Check that every rule in the project has an actual mechanism behind it." "Check that the test count claimed in this commit message matches reality."

They cost one sentence to request, so you request them constantly. I have 29 in a single project. They run on every commit and they almost always print nothing, which is exactly what you want them to print.

And there is the trap:

A detector that found nothing and a detector that cannot see produce byte-identical output.

Silence is the success state. Silence is also the total-failure state. You cannot tell them apart by looking — and the longer a detector stays quiet, the more you trust it, which is precisely backwards. A broken detector is silent more reliably than a working one.

Test suites have a defence against this. Mutation testing perturbs your production code, re-runs the suite, and reports any mutant that survived — a change nothing caught. It exists because you can reach 100% line coverage with tests that assert nothing at all.

But mutation testing points at suites, in CI, over application code. Nobody mutation-tests the 200-line script their agent wrote on Tuesday to check something about their docs. And the guardrail tooling that grew up around LLM coding — pre-call and post-call interceptors — is built to stop the model from doing something dangerous, not to prove that a bespoke checker can still see.

So the fastest-growing category of quality machinery in your repo is the one with no soundness check at all.

Lab science solved this in the 1800s

Every assay ships with controls. A negative control is the assay run with everything except the sample: no signal is expected, and if a signal shows up, the run is contaminated and its results are void. A positive control is the mirror — a known-present sample that must produce a signal. If it doesn't, the instrument is dead, and every clean reading it gave you today means nothing.

The translation to software is direct. Give every detector a --self-test flag. Behind it, paired controls:

positive — a case the detector exists to catch. It must fire.
vacuum — an invented case that resembles a real one but isn't. It must stay silent.

Run the controls before trusting the report. If any control fails, the tool does not print a verdict at all. It prints that it is unsound.

Here is the real thing, trimmed, from a tool that audits whether every rule in my project has an enforcement mechanism behind it:

// A detector that only ever says "clean" proves nothing.
function selfTest(S) {
  const ok = [], bad = [];
  const t = (name, got, want) =>
    (String(got) === String(want) ? ok : bad).push(`${name}: got ${got}, want ${want}`);

  // Can it still read its inputs at all?
  t("sources · routing table found",  S.table.length > 500,    "true");
  t("sources · test classes parsed",  S.testClasses.size > 50, "true");

  // Positive: a rule that IS in the table must resolve.
  t("positive · slug fully in table",
      slugHit(norm(S.table), "smart-design"),            "full");

  // Vacuum: a rule that exists nowhere must resolve to nothing.
  t("negative · absent slug",
      slugHit(norm(S.table), "zzz-nonexistent-rule"),    "null");

  // Vacuum: an invented filename must not be accepted as a witness.
  t("rules-witness · vacuum — invented file is not",
      S.rulesTests.has("no-such-file.test.js"),          "false");

  console.log(bad.length
    ? "❌ DETECTOR UNSOUND — do not trust its report"
    : "✅ controls pass — detector may be trusted for this run");
  return bad.length === 0;
}

Look at the last line. The tool never claims the codebase is clean. It claims that for this run, its own verdict is worth reading. Those are different statements, and keeping them apart is most of the value.

The trap inside the fix

Controls go blind too. This is where it stops being trivial.

One of my audits reads a corpus of documentation and reports orphans — guards that exist in code but that no document explains. It needed a positive control: some token guaranteed to be present, so that a false would mean "the reader is broken," not "this guard is undocumented."

The first control I wrote picked a guard that, as it turned out, genuinely was undocumented. So it returned false. And false there is indistinguishable from a completely broken corpus reader. A control designed to prove the reader worked would have quietly certified a reader that had stopped reading.

The comment in that file now reads:

// Positive control for the orphan check: EtalonChainGuardTest is the best-explained
// guard in the project (6 memory files + 5 docs). If the corpus is being read at all,
// this token is present — so a `false` here means the reader is broken, not that the
// guard is an orphan. (An earlier control used a guard that is in fact undocumented,
// which would have made a broken reader look correct.)

The rule I took from it: anchor a positive control to the most redundantly-present fact you have, never to a convenient example. If the anchor is marginal, its absence is ambiguous — and an ambiguous control is not a control.

There is a floor below that one

I found it the day after publishing these tools as a public repository, in the one script that grades all the others.

That harness has a mutation mode: it blinds a hook on purpose and requires every positive case to go silent. A green run means the controls are capable of failing. I ran it and got a flawless score for two of three hooks — and the score meant nothing. The mutation expression replaced the first line of a multi-line pipeline and orphaned its continuations, so the mutant no longer parsed. Bash never started the hook. Every case "went silent" for a reason that has nothing to do with blindness, and the report concluded: every EXPECT went silent — these controls can actually fail.

The harness already carried a control for this family. It refuses a mutation that changes nothing, on the grounds that such a mutation looks identical to one that worked. That control ran, and passed: the file had changed.

Changed and still executable are two different questions, and only the second one makes the silence mean what the report says it means. One line separates them — run the mutant through bash -n before grading anything. It is now control ⑨, with a paired control ⑨b, because a parse check that rejected every mutant would have passed ⑨ alone.

I keep relearning the same shape. The detector is a hypothesis about the subject; the control is a hypothesis about the detector; and the thing that runs the controls is a hypothesis nobody had written down.

Copy it in ten minutes

Minimum viable version, any language:

Add a --self-test flag that runs before anything else and exits non-zero on failure.
For every rule the detector enforces, write two fixtures: one that must trip it, one that looks similar and must not.
Add at least one input-parse control — assert that what you're reading is non-empty and shaped as expected. Most silent blindness isn't subtle logic error; it's a path that moved, and now you're matching against an empty string.
Anchor positive controls to the most redundant, least removable fact available.
On any control failure, print "unsound — do not trust this report" and refuse to emit a verdict. Never a partial one.
Every time the detector is wrong in real use, don't just fix it — turn that exact case into control #N+1.
If any control works by breaking the thing it grades, assert the broken version still runs. Changed and still working enough to be measured are different questions.

Step 6 is where it compounds. My detectors' control lists now read like a diary of every way each one has previously been wrong. That list is the actual asset; the detector is just the thing it's attached to.

If you would rather start from working code than from a list, the same practice is packaged in negative-control — MIT, three bash hooks plus two harness scripts and one .mjs, no install. The snippets above are from a private project; the repository holds generalized copies, each with its own controls. bash scripts/probe.sh --all runs them, and --mutate blinds a guard on purpose so you can watch its controls catch it.

What this is not

It is not a replacement for tests. It is not mutation testing — mutation perturbs the subject to grade the checker, which is stronger and considerably more expensive. Self-tests with controls perturb nothing. They just refuse to let a checker report until it has shown, on fixed fixtures, that it can still tell signal from noise.

Nor does it escape the regress. Who controls the controls? Nothing does. The controls are hand-written fixtures and they can rot right alongside the code. What the practice buys is a floor, not certainty: a detector without controls can be blind from birth and never say so, while a detector with controls has to survive a named list of things it must catch and must ignore — and when one of those breaks, it breaks loudly instead of printing a reassuring nothing.

The numbers, measured rather than remembered

In a solo-built production Android app — 73,411 lines of Kotlin, 1,683 unit tests, all measured 2026-08-16:

29 custom tooling scripts
15 carry a --self-test
15 of those 15 carry at least one explicit vacuum control

Fourteen tools have none. They are the ones I trust least, which is the correct amount.

The opening anecdote counted 13 of 13. That was a few days earlier, and two more tools have grown controls since — which is precisely why every number here carries a date and none of them is worth repeating without one.

Limits, stated plainly. These counts come from grepping my own repository — the same method that was wrong in the opening anecdote before I checked it by hand. I haven't surveyed how common the practice is elsewhere; my impression that it's rare comes from searching, not from reading other people's code, and absence from a search result is not absence from the world. And a passing control suite proves only that the detector could see at the moment the controls ran, on the fixtures it was handed. Nothing beyond that.

Which is still a great deal more than green.

If you have a checker in your pipeline that has been quietly green for six months, run the experiment: hand it something it should catch. Five minutes, and the result is always interesting.

DEV Community: Volodymyr