One capital letter made my verifier call a failing score clean

I build a thing that checks whether AI evaluation claims are true. Someone broke it, I fixed it, and then I broke it four more times myself.

Here's the cheapest one.

A bundle carries security test results. Each case has a severity, and the score weighs failures by it:

_WEIGHTS = {"none": 0, "low": 1, "med": 2, "high": 4, "critical": 8}
total_w  = sum(_WEIGHTS.get(c.get("severity"), 0) for c in graded)
failed_w = sum(_WEIGHTS.get(c.get("severity"), 0) for c in graded if not c["passed"])

Take a real bundle with three failing cases. Change critical to Critical, on the failing rows only. Their weight drops to zero and leaves the numerator. The passing rows keep theirs. The score divides cleanly:

severities: [critical, critical, high, med, med]
        ->  [critical, Critical, high, Med, Med]

vulnerability_score:  0.0    (honest value: 0.4545)
sha256:               re-pinned honestly
verifier:             exit 0, "structural verification: PASS"

No exception. No fallback branch. Ordinary arithmetic, all the way down. .get(sev, 0) treats an unknown label as harmless, so a weight table I called "frozen" was actually one the issuer controlled.

And my own spec had written the hole down as a decision: "an unknown severity weighs 0."

Three more, same shape

• A lie typed as a string. The summary comparison walked the JSON and returned early on anything that wasn't a number. Retype every headline as "9999" and nothing is ever compared. No artifact touched, no hash re-pinned.
• Delete the check instead of breaking it. Nothing required a listed artifact to be covered by a check. Earlier hardening made breaking a check a named failure; deleting one stayed free.
• Delete the stamp instead of faking it. Four comparisons were guarded on the artifact-side key existing. Corrupt those fields and you get refused by name. Remove them and nothing happens.

That last one generalizes: a comparison guarded on both operands existing is not a check, it's a suggestion. The party supplying one operand decides whether the comparison happens at all.

Then I stopped collecting anecdotes

Four bugs is a story, not a measurement. So I asked a different question: of every place this verifier can say no, how many are actually protected by a test?

The verifier rejects bundles by appending named reasons. I disabled them one at a time, replacing each failures.append(...) with pass, and asked whether anything noticed: the unit suite, a liveness control, or any of the 16 committed tamper fixtures.

112 refusal sites
 37 caught
 75 SURVIVED, deletable in silence, everything still green
mutation score 0.330

Two thirds of the ways this thing could say no were unprotected.

Worse: I had a CI job named invalidation-liveness, written specifically to prove the verifier can block. It caught zero of the 75. Each fixture exercises one refusal path, so deleting any other refusal leaves all sixteen verdicts unchanged. The gate built to prove the gate works had, against this operator, a score of zero.

The number I didn't expect

I fixed the four bugs. Added a regression fixture for each, the responsible post-audit move. Re-measured:

before fixing   37/112 = 0.330
after fixing    39/119 = 0.328

Fixing the four defects I'd actually found, each pinned against return, did not meaningfully improve coverage. It closed four real holes, which is real value, but it left the rest of the gates exactly as unprotected as before, because four regressions sample the failures you discovered rather than the population you own.

Testing every refusal instead:

0.330  ->  0.941  ->  1.000

The defects you find are a sample. The gates you own are the population.

The part where my tools lied

Six times, during this work. Twice inside scripts I wrote to hunt this bug. The worst:

The first run after hardening scored a perfect 1.000. I almost wrote that down. It was false. The harness ran pytest -x against a baseline that was already red, so pytest exited nonzero for every mutant, every mutant scored "caught," and nothing was measured at all.

A tool built to detect checks that pass without checking produced a check that passed without checking. And it presented as the best possible result.

The fix is the rule the whole exercise is about, turned on itself: the sweep now aborts unless the clean baseline is green. And when it finally did reach a real 1.000, I planted a refusal on an unreachable branch and confirmed the sweep still reported SURVIVED before believing the number.

What I'm not claiming

I sent this to an outside reviewer specifically to get the framing attacked, and several claims came back too big:

• 1.000 is not "the verifier is correct." It's a complete score against one narrow operator (disable a refusal) over an enumerated population. Mutated comparisons, boundaries, and control flow are unmeasured.
• The honest name is refusal-append liveness coverage. The denominator is a source-level proxy that misses raises, early returns, and every fail-open behavior that happens before a refusal is reached.
• "Vacuous pass" is a label, not a discovery. This is well-trodden ground: vacuity detection in model checking, the oracle problem, mutation adequacy, fail-open validation.
• Every number here is self-measured on a system I wrote, verifying a registry whose entries are all my own repos. That's one multi-repo fixture, not independent evidence.

The one that surprised me

The closure fix, every listed artifact must be read by some check, was written to close one hole. Pointed at my live registry, it refused every remaining issuer I had.

And in every single case, the unchecked artifact was the human-readable one. The rendered report. The contract. The .md file. The machine-readable JSON was always bound. One contract announced "6/6 seeded defects fixed". True, and verified by nothing.

I don't have the sample to call that a law. Three repos, one author. But as a hypothesis it's cheap to test on your own work: find the document your users actually read, and ask what binds its numbers to the data.

Everything's public: verifier, spec, the mutation tool, and the write-up with the parts that came back wrong. github.com/egnaro9/vac-protocol

If you run something similar: what does your suite do if you delete one of your gates? Not break it. Delete it. I'd genuinely like to know whether 0.330 was unusual or ordinary.