Eight checks lied to me in one day, and none of them threw an error

In one working day I wrote or relied on eight checks that were wrong. Not one of them threw an
error. Every single one returned a plausible answer, which is exactly why each cost me time.

Here they are, with what they actually returned.

The four ways a check can lie

1. It never runs

I wrote assertions that validate prices against the strings shown to users, then deliberately
corrupted one to confirm the build would fail.

sabotaged price string
BUILD EXIT=0     <- the guard saw nothing

The assertions run on import. Nothing imported the module yet. After wiring it into a page that
is actually built, the same sabotage gave EXIT=1.

A check nothing calls protects nothing, and it is worse than no check, because you believe you
are covered.

2. It caches a stale verdict

A crawler wrote sitemap_absent: true on the first pass. I fixed the fetch, the data came back
correctly, and the report still printed "SITEMAP NOT SERVED" — because the flag was never
cleared.

The JSON on disk said 445 URLs. The screen said absent. I only caught it by opening the artefact
instead of reading the summary.

3. It asks for the wrong shape

Counting rendered questions on a page returned 0. I nearly reported that a paid feature shipped
nothing.

My selectors looked for DOM classes. The items were props of a client component, serialised into
the framework payload as {"q": …}. The page was 16,908 bytes in both measurements. The content
was always there; only my way of counting was wrong.

The fix was not a better guess. It was reading the type definition and counting what the code
actually emits.

4. It includes itself in its own sample

Two on the same day, and this is my favourite.

pgrep -af "next start"
# returns only the bash -c wrapper that contains the pattern

The old server was still holding the port under a different process name. I found it with
ss -ltnp — searching for who holds the port always works; searching for a name you assume it has
does not.

Then, checking whether a new slug collided with existing ones, I globbed the directory — including
the file I had just written. It saw itself and answered "already taken."

A probe that includes itself in its own sample always confirms what it is looking for.

The two failure modes, and why calibration is the whole job

A check can fail in exactly two directions, and I hit both:

too weak   counting accented characters       -> a text with a few accents passes while being wrong
too strong requiring a fixed list of words    -> cries on correct text that had no reason to use them

The German page was correct. My check demanded words the article never needed, and shouted anyway.

A false alarm that repeats gets ignored, and an ignored check protects nothing. That is the
quietest way to lose a safeguard: it stays in the code, it goes green in the reader's head, and it
stops saying anything.

The calibrated version turned out to be: a non-zero count of language-specific characters AND a
few witness words present — not all of them.

What I do now, and it costs three minutes

Test the guard by making it fail on purpose. Not once written, but every time it is written.
Sabotage the input, confirm it screams, restore. Two of my checks passed that test only on the second
attempt, and I would never have known otherwise.

Two other habits that came out of the same day:

• Read the artefact, not the summary. The JSON, the manifest, the served HTML.
• Prove identity, not just presence. "The server responds" is not "my version responds." A deployment I thought I was testing had never started; the old process answered 200 all along.

That last one is the same shape as a thing I ended up writing about elsewhere: a VPN kill switch
toggle being on tells you what the app intends, not what the operating system will do when the tunnel
dies at an awkward moment. I wrote up the four tests that actually settle it
here.

Eight wrong checks, zero errors raised. A check that crashes gets fixed. A check that answers
slightly the wrong question gets believed.