Proof over claim
You run the tests, they pass, you ship. Now try the other direction: build a check that must say no, and prove you have seen it say no. Most of us have never watched our guards refuse anything.
Four rejections, one shape
In one working day my own tooling stopped me four times. I kept the list, because by the third one the pattern was too clean to ignore.
One: I scanned a network range to find a service, found something answering, and concluded the service lived there. It was a leftover process serving an older model. The thing I needed was on a machine my scan never reached.
Two: I read an error message from a stored record and changed code based on it. The message came from a version that had been repaired three days earlier. I nearly rebuilt a fix that already existed.
Three: my own verification loop reported two passing test files as failing, because it searched output text for the word red and the German word for nevertheless contains those three letters.
Four: I added a file to a build without adding it to the file list. The build reported success, and the packaged result crashed on the first command.
The shape: an output is not a proof
Every one of the four was the same mistake wearing different clothes. Something produced output, and I treated the output as evidence about a different question.
A service answering proves something is listening. It does not prove what. A stored error message proves what a tool once said. It does not prove what the tool does now. A build that lists files proves the list is satisfied. It does not prove the result runs.
This is uncomfortable because output is exactly what we have. We read logs, exit codes and dashboards all day, and each of them answers a narrower question than the one in our head.
The gap between those two questions is where a whole day of work quietly goes wrong.
Why the checks caught it and I did not
Not because they were clever. Because each one asserted the narrow question in writing, and I could not talk them out of it.
The file-list check now starts the packaged tool and fails if it does not respond. It stopped caring whether the list was satisfied, which was never the interesting question.
The service check now asserts which model answers, not that something answers. One extra string comparison, and the leftover process becomes visible instead of convincing.
And every verification now reads exit codes instead of searching text. A word inside another word cannot fake an exit code.
The test most guards never get
Here is the part that took me longest to accept. For every guard I have written, I can tell you that it passes. For most of them, I cannot tell you that I have ever watched them fail.
A guard that has only ever been green is not a guard. It is decoration that happens to be the right colour, and it will keep being the right colour after it stops working.
So the acceptance for a guard needs two cases, and the order matters: first prove it says no when it should, then prove it says yes when it should. Written in that order, because the yes case is the one you will remember to write anyway.
When I applied this to my own package, the very first thing it caught was that the loop brake and the frozen-acceptance check both fired correctly. My tooling was fine. My test was wrong. That is the good outcome.
Add the refusal case to one guard today
Pick the guard you trust most. That is the right one, because trust is the thing being tested.
Step one: write a case that should be rejected, and assert the rejection. Not just a non-zero exit — assert the message names the reason, so a future failure for a different reason cannot pass as this one.
Step two: break the guard on purpose in a scratch copy and confirm your new case turns red. If it stays green, your case proves nothing and you have just learned that for free.
Step three: keep the broken-on-purpose command in a comment next to the case. Six months from now, that is the only way anyone can re-verify the test still bites.
The pattern in fifteen lines:
#!/usr/bin/env bash
# Guard acceptance: prove NO first, then YES.
set -u
# 1. The refusal case — the one almost nobody writes.
if out=$(./my-guard bad-input 2>&1); then
echo "FAIL: guard accepted bad input"; exit 1
fi
grep -q 'reason:' <<<"$out" || { echo "FAIL: refusal without a stated reason"; exit 1; }
# 2. The happy case.
./my-guard good-input >/dev/null || { echo "FAIL: guard rejected good input"; exit 1; }
echo "OK: guard says no AND yes"
# Counter-check (run by hand once, keep the line):
# sed -i 's/exit 1/exit 0/' my-guard && ./this-test # must FAIL
That last commented line is the whole discipline. A test you have never seen fail is a rumour.
What changes for you
Before: your suite is green, and you cannot say which of those checks would notice if the thing they watch quietly stopped working. Green means nothing broke loudly.
After: for each guard you can point at the case that proves it refuses, and at the command that proves the case bites. Green starts meaning something specific.
In our own setup this is a hard rule: every finding becomes a check, and every check ships with the run that shows it failing. The record of those runs is what we keep in cachly, so the next session inherits the reason and not just the rule. Free tier, hosted in the EU.
Being rejected by your own tooling four times in a day is not a bad day. It is the day the tooling finally earned its keep.
I build cachly — persistent memory for AI coding assistants, over MCP. ChatGPT and Claude remember your conversations. cachly remembers your codebase: the bug you fixed, why you chose Postgres, the deploy step that always breaks — including what your teammates learned. And every assistant you use reads the same memory.
Free tier, hosted in the EU: cachly.dev
Top comments (0)