I have a rule for anything my sites tell a reader. Can someone check it. Will someone make a decision based on it. Would a bigger version of me be embarrassed by it.
I had never once pointed that rule at my own tooling.
On the night of 16 August I finally did, in one codebase, looking for controls I had never actually watched fail. I found eight. Eight instruments whose whole job was to tell me the truth about something else, every one of them reporting success, not one of them doing anything.
A config file that said it blocked backups was inert on the host it shipped to. Never blocked a single request in its life. The 404s that made it look like it was working were files that had never existed.
A SQL revoke came back "Success. No rows returned." A column revoke does nothing underneath a table-level grant. Twenty-seven member email addresses had been sitting public the whole time.
An export gate printed ok 81 samples and ok 45 frames. Printed both. Compared neither.
A fade test asserted on Math.max(alphas), which is 1.0 for every fade ever written and also for a hard cut.
A deploy reported complete directly above a log line reading "Failed to publish your Function."
Five of those had been sitting there for days, believed. Three I had written that same evening and caught within the hour, which tells you the count is not the interesting part.
What got me was that this failure runs backwards from every other kind of bug. A control you never wrote gets found by the incident. A control that lies gets found by the incident and then eats the incident review too, because everybody stares at the green light and goes looking somewhere else.
Two of the eight were worse than bugs, and they were the same shape as each other. Something real went wrong. We responded by writing a rule. The rule did nothing. We went back to work feeling protected.
The twenty-seven addresses leaked for real. The fix was a list of backup globs that never blocked one request.
A signup rework shipped its front end while its migration sat unapplied in the repo, so every new member logged into an empty room. That migration had a section in its own header titled "THE ONE THING THAT WOULD HAVE BROKEN." It was right. Nobody ran it.
A control written in the hours after an incident is the least trustworthy code in the repo and gets treated as the most. You are tired, you are angry, writing it feels like doing something, and the second it exists the incident feels closed. Nobody goes back.
An instrument is not a control until something has been observed to bounce off it.
That is a procedure, not a slogan. If you want to know whether a rule fires, make the thing it is supposed to catch and watch what happens to it. Reading the config only tells you what somebody intended. That backup file was articulate, dated, carried its own incident history in a comment block, and did nothing.
Three weeks later I found the version of this that is genuinely hard to see.
Five checks, five different programs, one session, every one of them green while structurally incapable of going red. No broken code anywhere. Each was a correct program answering a question that had no failing branch. A mutation test told me four of four mutants were killed. Every mutant had died on "no properties found" before it ever reached the canary, because the script resolved its root relative to its own location and the mutants ran from a scratch directory. Four for four. Zero of four.
Another asserted that a marker string was absent after a fix. The string never matched the tool's own output casing, so "not found" and "never findable" came back identical. That same check passed while the scanner it depended on had crashed, because it went looking for the marker inside the crash text and did not find it there.
Green is not evidence. Green is evidence only if you know what red would have looked like.
Four questions killed all five. What input makes this fail. Did the failing case actually run, and fail for the reason I meant rather than a path error. Is absence proven or just unobserved. Does this thing report differently for "nothing here" and "I never looked."
I hit two more today, in a tool I built this evening.
Twelve controls, all green, including a deliberately broken copy to prove the suite could go red at all. Then I pointed it at the live API and it threw on the first article. The list endpoint returns an article's tags as a comma-separated string. The single-article endpoint returns the same field as an array. Every fixture I had written used the array. Twelve green tests against a shape production does not send, and three of those twelve now exist only because of it.
The second one I nearly shipped, and it is worse. The whole premise of the tool is that a settled page beats a fresh one, so age scores positively. But the endpoint I was harvesting from only returns the latest feed. There were no settled pages in the pool. None. The age term, the entire reason the thing exists, was sorting fresh posts against an empty set and handing me a clean ordered list while it did it. Widened the harvest and a page 340 days old came out at rank three, carried there by the term that had been doing nothing at all an hour earlier.
An instrument whose scan root excludes the thing it was built to find will report full coverage forever, and from outside the checker that looks exactly like a clean bill of health.
None of this is an argument for more gates. Most of the damage came from instruments that already existed, and a ninth unverified one makes it worse. What I would actually defend is fewer instruments, each proven to bite, which means deleting checks that currently pass. People hate doing that. It feels like taking down a smoke alarm.
Here is the problem with everything I just told you: you cannot check any of it. Private codebase, my logs, my word, which is worth about what anybody's word is worth on the internet. That is a real hole in a piece arguing that a specific finding beats a general claim.
So take the instrument instead of the stories. The seven shapes and the four questions are at deepsynthesis.org/instrument-integrity. It runs against your own repository in about twenty minutes and it needs nothing from me. Point it at a check you have never seen go red.
Every fix above came from mutation or from curiosity. Not one came from the suite that was supposed to cover it. Today's second one surfaced because I glanced at a column of dates and thought, huh, these all look new.
I have no way to systematise that. It is just the only thing that has ever worked, and I could not tell you how much of my tooling is lying to me right now.
Top comments (0)