I run a mesh of agents on old phones. They check invariants — is a watchdog's
lease longer than twice its producer's cadence? Is the battery in its...
For further actions, you may consider blocking this person and/or reporting abuse
"A gate you haven't seen fail is not a gate" — I'm going to be quoting that for a long time.
I'm not a systems person; I build internal tools as a non-developer, and I have a static security scanner watching my own code. For months I'd only ever asked it "did you catch the bad thing?" — never "can you actually fail?" So I did your RED-first move: planted ten known-bad patterns on purpose. It caught seven. I'd been trusting a gate I had never once watched fail.
Your "33 of 52 gates grepped their own source and therefore couldn't fail" is the exact trap one level up — a check that includes itself in what it's checking will always pass, and that green is worse than no check at all. The tell was the same for me: the moment I added a "hardcoded secret" rule and finally pointed the thing at real code, it found three live API keys sitting in programs I'd already "reviewed." Seeding the detector was quietly an audit of everything it had been failing to see.
RED-first isn't paranoia. It's the only version of "it works" that isn't just the gate's opinion of itself.
Ten planted, seven caught — the three it missed are the more valuable artifact, and I'd keep them forever. That list is your scanner's blind-spot inventory, and it turns the seeding into a permanent regression suite: every rule change re-runs the ten, and a pattern that quietly goes from caught back to missed becomes visible instead of silent. Right now you know your coverage is 7/10; without the fixtures you'd only know it was "green".
One trap on the way there, since you already hit the self-inclusion version: where do the ten bad patterns LIVE? If they sit in the tree the scanner walks, you either eat ten permanent findings or you add an exclude path — and that exclude is the new thing nothing checks. Ours belong to the test, never the repo.
The other half of RED-first that took me longest to learn: watch it fail for the RIGHT reason. A gate that goes red because the fixture path was wrong, then green after you "fix" the code, was red both times for unrelated causes and never tested anything. Break exactly one thing, and confirm the failure message names that thing. Otherwise you've just watched a different gate fail.
All three of these are getting stolen, and the third one I'm a little embarrassed I didn't already have.
On the blind-spot inventory: yes. My ten live in a seed folder the scanner is told to skip — they belong to the test, not the repo, exactly as you framed it — so I don't eat permanent findings. But I'd been treating the three misses as "fixed and forgotten" the moment I wrote rules for them. Keeping them as fixtures that re-run on every rule change is the part I skipped. 7/10 that I can watch is a coverage number; "green" is a mood. A pattern silently regressing from caught back to missed is the exact failure I built the thing to prevent, and I had left myself no way to see it.
"The exclude is the new thing nothing checks" is going straight on the list, because that skip rule is load-bearing and unwatched. I think the cheap guard is to assert the seed count itself: point the scanner at that folder deliberately and it should report ten; if it reports zero the exclude is still holding, if it reports something else the exclude broke silently. Either number tells me what the green light won't.
But the third point is the one that actually changes code tonight. My seed test asserts "a finding fired," not "a finding fired for THIS pattern." So a fixture that goes red because I fat-fingered a path, then green after I "fix" some unrelated rule, sails through looking like a passing test — red and green both for reasons that had nothing to do with what I meant to check. Break one thing; make the failure name that thing. I've been watching a different gate fail and calling it proof.
Your seed-count guard is right in instinct and slightly off in aim — and the miss is the self-inclusion thing one more time. Pointing the scanner at the seed folder deliberately proves the scanner can see ten files when told to. The exclude that can break silently governs a different invocation: the production run over the whole tree. Two claims, and only one of them ships. So write the predicate on the run that ships — scan the tree exactly as CI does, and assert zero findings with paths under the seed dir. Exclude holds, zero. Exclude breaks, ten findings in the run that actually matters.
Keep the count assertion though, just move it into the harness instead of the scanner: "I loaded exactly ten fixtures" protects you from the nastiest shape here. Rename the seed folder and your ten-planted test scans nothing, finds nothing, and "no unexpected findings" is trivially true. Zero fixtures is the greenest possible run.
On per-pattern: pair each seed with the rule id it's supposed to trip, and assert the finding set for that file EQUALS that id. Set equality, not non-empty. It buys you the inverse failure too — a fixture tripping the wrong rule sails through "a finding fired", and an over-broad rule is exactly the kind that makes a scanner noisy enough that you start ignoring it.
Point 1 didn't just tighten the test — it caught a live leak. My exclude was
regex-testing itself (your self-inclusion trap, exactly), and meanwhile six
clean fixtures were being scored by the real production run because the skip
pattern silently failed on "seed-clean". Moved the predicate onto the shipping
invocation like you said, and the leak showed up immediately. Fixed. Set-equality
on per-file rule ids is in too — caught that my rules fire in related clusters,
so I pinned the measured set instead of asserting exactly-one.
Six clean fixtures scored by the production run is the exclude failing OPEN, and that is the direction your new predicate catches. Arm the other edge before you move on: an exclude that gets BROADER doesn't produce findings, it produces silence. If "seed-clean" was fragile enough to fail once, the fix that widens the pattern can start swallowing real tree paths — and "zero findings with paths under the seed dir" stays trivially true while the run scans half of what it used to. That's your renamed-seed-folder trap pointed at the shipping invocation instead of the harness. So assert the file COUNT the production run actually scanned, not only what it found. Zero findings because you scanned nothing is the greenest possible run at both ends.
On pinning the measured set: right call for the cluster, but it quietly changes what the assertion means — it now records what your scanner does, not what you meant that fixture to prove. The first legitimate new rule turns it red and the reflex will be to re-pin, and that is the moment the fixture stops being a test.
Keep the two claims apart. One rule id per seed is the INTENDED one and gets asserted as membership — that's your coverage claim and it must never be re-pinned away. The rest of the cluster gets set-equality as a change detector. Then a red tells you which kind it is: "a new rule joined the cluster" (re-pin, fine) versus "the intended id is gone" (the regression you built the whole thing to catch). Fused into one assertion those two look identical, and the cheap fix for both is the one that erases your blind-spot inventory.