Twelve Checks That Could Not Fail

Four days ago I published a piece about a release pipeline that installed the package it had just published, confirmed the version, and reported success against a version the registry was not serving. A reader called build996 left a comment that went one level below anything I had written:

The 2>&1 in that npx line is doing quiet damage too: it merges npm's error output into the same variable the case statement reads, so the failure text is handed straight to the matcher that decides success. Without it, $out would have been empty on a miss and the pattern could not have matched.

I had described the check. He had found the mechanism. So I measured it, because a claim that good deserves a run rather than a nod.

The line, as it shipped:

out=$(npx --yes "$PKG@$v" --version 2>&1)
case "$out" in
  *"$v"*) echo "smoke ok" ;;
  *) echo "smoke failed"; exit 1 ;;
esac

Asking npm for a version that does not exist, with the redirect in place:

captured bytes: 346
first line:     npm error code ETARGET
VERDICT:        smoke ok

The same check, with stderr left where it was:

captured bytes: 0
VERDICT:        smoke FAILED

Three characters. npm's error text quotes the package and version back at you, proactive-gate@99.99.99, and the pattern was looking for the version string anywhere in the output. Merging stderr into stdout handed the matcher a sentence that contains exactly what it was searching for, at the moment the thing being tested had failed. Take the redirect away and the variable is empty, the pattern misses, and the check that was supposed to work does.

I did not put that in the article because I had not understood it. The fix I shipped happened to remove the problem, by separating "did it install" from "what did it print", but I shipped it for a different reason than the one that made it work.

That comment sent me back through the week, and the count is worse than I thought.

The shape

A check that can only report agreement is not a check.

The question that finds them is short, and I have started asking it of everything: what does this do when it finds nothing, and is that distinguishable from finding everything in order?

In one week I found twelve in my own repositories. Not in code I inherited. In checks I wrote, on purpose, to catch exactly the thing they were failing to catch. Here is what they looked like, because the shapes repeat and the shapes are learnable.

A proxy standing in for the condition

My CV build skipped regeneration when the PDF was newer than its source. That is not the question. The question is whether the source changed, and mtime is a rumour about it: git checkout sets it, cp sets it, an editor that saves without changing a byte sets it. The build was answering a cheaper question and presenting the answer as if it were the expensive one.

It now hashes the source and compares the hash, and writes the stamp last, after the page check and the copies, so an interrupted run cannot leave a stamp claiming work it did not finish.

The general form: when a check reads something correlated with the condition rather than the condition, it passes for reasons unrelated to what it is about.

The failing direction, which looks like diligence

A test compared a rendered link against a literal string. It had passed for two months. It passed because the three items it happened to compare were all external links, where the rendered form and the literal form agree. The first internal link would have broken it, and the breakage would have been the test noticing a difference that did not matter.

A check can be wrong in the direction of passing or in the direction of failing, and the second kind is more comfortable to live with, because a test that goes red looks like a test doing its job. Nobody audits a check that has never fired.

Three of the twelve were repairs, not checks

This is the sub-pattern I did not see until the count got high enough, and it is the one I would tell you about first.

A fixture I added to cover a gap turned out to duplicate coverage that already existed. A verification rule I wrote, to confirm a generated file matched its source by hashing it, could not verify anything, because the generator embeds a timestamp and produces three different hashes from three runs of unchanged input. A revert I ran succeeded, reported success, and changed nothing, because the string I was reverting to and the string I was reverting from were identical; the actual bad change was in the commit before the one I looked at.

The repair is where the thinking is finished and the attention has left. You have found the bug, you know what to do, and the part where you confirm you did it is the part that feels like paperwork. Three times in one week, that was where the hole was.

For a revert the concrete rule is: confirm the restored content differs from what was there. A no-op revert and a real one produce identical output at exactly the point where people stop looking.

"Could not reproduce" wears the same costume

I failed to reproduce a reported bug, and my inability to reproduce it was the finding rather than the absence of one.

A run that does not reproduce a defect is a measurement of your setup, not of the defect. It is only evidence when you can say what would have shown up if the bug were present, and confirm that your run could have shown it. Otherwise "I ran it and nothing happened" and "I ran it wrong" produce the same sentence.

Deriving is necessary and not sufficient

I fixed a class of brittle test by deriving expected values from the source instead of typing them in. That is the right move and it does not finish the job. Where a value crosses a boundary that transforms it, rendering a link, formatting a date, normalising a path, deriving it still leaves the question of which form you are asserting on. Both sides can be derived and still disagree, and then the test is pinning a coincidence.

Availability is not identity

A deploy check fetched a URL and treated 200 as proof the right file was there. A 200 proves a file is served. It says nothing about which file. The site sat a week behind a green check, because the alias kept serving the previous good deployment and the check kept confirming that something answered.

Read the artefact's own content, not its status code. For a PDF that meant pdftotext | grep for the claim that was supposed to have changed, which is slower and is the only version that can fail.

The twelfth was the checker built to catch the other eleven

I have a script that compares every published number about my work, package versions, contribution counts, essay totals, against a snapshot collected from the registries. It exists precisely because those numbers were all wrong at once, on a page that said it kept version numbers off it because they would rot.

Yesterday it printed ok, every claim in 7 surfaces matches while six of them were stale. The snapshot had been collected twenty minutes before a merge that changed the count.

It was not wrong. It was silent about the only thing that could make it wrong. "The surfaces match the snapshot" and "the surfaces are correct" are the same sentence only while the snapshot is current, and nothing bounded its age.

It now refuses a snapshot older than six hours, treats an unreadable timestamp as a failure rather than skipping it, prints the age on the success line so ok carries its own scope, and exits with a distinct code for staleness so the commit hook can refresh and ask again instead of blocking.

That is the general lesson from a checker that compares against a cached reference: ask what it is comparing against, and what proves that reference is still true.

How to find out whether a check can fail

Break the thing it watches and confirm it goes red. That is the whole method, and it costs a minute.

But there is a distinction I got wrong this week and had to correct, so I will state it plainly.

I had a set of passing tests and I mutated one assertion inside each, in a throwaway copy, to expect a value the code does not produce. All of them failed. I wrote that this proved the tests were not vacuous. Someone reading my notes pointed out that it proves only that the chosen assertion is reached and evaluated. Nothing about the behaviour had changed, so nothing had been shown about whether the test would catch a regression.

Mutating an assertion tests the assertion. Mutating behaviour tests the test. They are different experiments and only one of them is evidence about a check's usefulness. When I did change behaviour, in a single source file, three of four tests caught it and the fourth did not, because it did not depend on that boundary. One behaviour mutation is not expected to break every test, and expecting that would be its own mistake.

What I actually take from twelve

Not that I am careless. I wrote all twelve deliberately, several of them in direct response to being burned by the previous one.

What I take is that the moment of writing a check is the moment you are most convinced the condition is understood, and that conviction is what removes the step where you confirm the check can fail. The checks in this list were not lazy. They were confident.

So the habit I am keeping is smaller than a process. Every time I write something that reports a verdict, I break the thing it watches once, and watch it go red before I trust it green. And when a check tells me everything is in order, I ask what it is comparing against.

Twelve is what one week of asking produced in code I already trusted. I would not bet on your number being zero.

---

Thanks to build996 for the mechanism I had missed, and to beusebiu and raknaos, whose comments on the previous piece are in this one.