Every one of these was a check that came back clean while the thing it was checking was broken. None of them threw an error. That is the whole problem: a false negative looks exactly like a pass.
Four from the last two weeks, on a small static site I run alone.
1. The bare curl that could not tell "absent" from "not served to you"
I wanted to confirm an analytics script was live in production. I curled the page, grepped for it, found nothing, and concluded the deploy had never happened. I was about to go re-do the setup.
It was live. The script is injected at the edge by the hosting platform, not baked into the built HTML, and the platform does not bother injecting it for a request that does not look like a browser. My curl had no user agent, no Accept header, nothing.
The tool did not lie. I asked "is this string in the response" when my actual question was "does a real visitor receive this script", and those two questions have different answers.
Now anything that might be edge-injected gets checked with browser headers, and cross-checked against the platform's own dashboard before I declare it missing.
2. The attribute that was on another line
I check outbound links on directory listings to see whether they are dofollow or nofollow. So I grep the HTML for the href and read the rel next to it.
Except HTML does not promise you that. Attributes get wrapped, minifiers move things around, and a template can perfectly well emit this:
<a class="btn btn--primary"
href="https://example.com"
rel="nofollow"
target="_blank">
A pattern that expects href and rel on the same line finds the href, sees no rel, and reports dofollow. Confidently. Wrongly.
The fix is one pipe:
curl -s "$url" | tr '\n\r\t' ' ' | grep -o '<a[^>]*example\.com[^>]*>'
Normalize whitespace first, then match the whole tag. I had three wrong verdicts recorded before I caught this, and the wrong ones were all in my favor, which is exactly how a bias survives.
3. The Content-Security-Policy that would have eaten an image
I was about to add a third-party badge to a page. Copy the snippet, point the img at their CDN, done.
The build would have passed. The page would have rendered. The image would have been blocked in every visitor's browser and I would have had no signal at all, because my site sends this:
Content-Security-Policy: default-src 'self'; img-src 'self' data:; ...
I caught it before shipping, and only because I happened to read the headers file for an unrelated reason. There is no test in a static build that fails on this. The asset now lives in my own /public folder.
If you have a strict CSP, every third-party snippet you paste is a silent failure waiting for a deploy. Grep your own policy before you paste, not after.
4. The causal story I invented from one snapshot
This one is not a tool failure, it is me.
I pulled five directory listings and read their outbound links. Four were nofollow. Mine was dofollow. Mine was also the only one sitting on a paid trial that had just expired. Obvious conclusion: the dofollow came from the paid tier and I was about to lose it.
I had started writing that down as a finding when I went back through my own notes. I had measured that same link as dofollow the day before the paid tier was activated. The correlation was real. The cause was fiction.
The four other listings were probably just never claimed by their owners, which explains the difference without any money changing hands.
Nothing errored here either. I simply had a same-day snapshot and no memory, and a snapshot with no memory will hand you a causal story every single time.
The common thread
None of these were bugs in a tool. Every one was a question that could not distinguish the two cases I cared about:
- absent vs not served to you
- no rel vs rel I did not look for
- rendered vs allowed to render
- correlated vs caused
The check said green because the check was answering an easier question than the one in my head. The only defenses I have found that work are boring: send requests that look like the real client, normalize before you match, read your own config, and write down what you measured with the date on it so that yesterday can argue with today.
That last one has caught more of my mistakes than any test I have written.
I write about HR software for a living at wiserstaff.com, where the whole premise is testing things on real paid accounts instead of trusting the demo. Turns out the same discipline applies to my own stack, and I am worse at it than I would like.
Top comments (0)