A Check That Has Never Run Is Not Passing

Originally published on hexisteme notes.

I run a video pipeline where every shot carries a contract: a must_have list, a must_not_have list, and a serves_line field — the narration line the shot exists to support. One rule in that pipeline, which I call the enactment rule, checks each contract for self-contradiction: if must_not_have forbids the very thing serves_line requires, the shot can't honor its own contract and support its sentence at the same time, so the rule fails it hard at author time.

It's a good rule to want. It had never once run.

One layer deeper than the usual dead gate

This adds another entry to the same line of investigation: Grep won't find your dead gates. A fill-rate query will. found rules nobody was calling — the wiring itself was absent; The Guard Passed on an Empty Table found a correctly-called rule whose target population was empty at runtime, so it passed on nothing left to check; and The check that cannot fire found a rule that ran against real data and silently disabled itself because its threshold was an absolute constant that didn't match the input's scale. This one sits a layer deeper than all three: the callers existed, the rule fired on every invocation, and what was actually missing was a producer for the one field the rule needed to read.

The field nothing writes

serves_line is a derived field — by definition, it's the narration of the beat the shot is attached to. I checked that derivation against reality before touching anything else: every stored value that existed matched its beat's narration exactly, 6 of 6, zero drift. Wherever the field existed, it was right.

But almost nowhere did it exist. A repo-wide search for anything that writes serves_line returned zero producers. The archetype templates that generate contracts don't set it. So every contract created through the supported authoring path had serves_line = None, and the rule — which needs that field as its input — silently evaluated nothing. A published episode had all five of its shots sitting in exactly that state.

The rule wasn't failing. It wasn't passing either. It was unmeasured, and unmeasured looks exactly like clean from the outside.

Wiring it in looked like the fix

The obvious move is to derive the field at author time from the beat it's attached to. The derivation is provably right, the plumbing is small, and it turns a dormant rule live. I wired it. Tests passed.

Then, before trusting that, I ran it retroactively against every existing contract the rule could now evaluate.

outcome	count
hard failure	6
exempted (declared metaphor)	2
true positive	0

Eight firings. Zero of them were catching a real contradiction.

The rule was measuring the wrong thing

serves_line is narration — audio. must_not_have governs pixels — what's allowed to appear on screen. Those are orthogonal channels. A voiceover saying "one million dollars" while the frame shows no digits at all is not a contradiction; it's the ordinary craft of the format. The rule was firing on the co-occurrence of a spoken quantity and a visual numeral ban, and calling that a conflict.

The clearest proof is the best-authored contract in the repository. Its must_have reads, literally, "blank faces free of pips and numerals" — the dice are deliberately blank while the narration speaks the numbers aloud. The author had even declared that relationship as metaphorical, specifically so the tooling would know not to flag it. The rule flagged it anyway.

Had I shipped the derivation as-is, six correct contracts would have hard-failed at author time. Three of those six belong to an episode that had already shipped.

What I actually did

I reverted the wiring. What stays is a comment at the rule itself, recording the measurement — eight firings, zero true positives — where the next person who goes to wire a producer for serves_line will read it before they do.

I kept one separate piece in place: something that reports the unmeasured state by name. It doesn't enforce anything. It just refuses to let "no exemptions recorded" be confused with "never checked" — which is the exact confusion that let this rule sit dormant and clean-looking for as long as it did.

What transfers

• Ask who produces each input, not just who calls the rule. A rule can have real callers and fire on every invocation and still evaluate nothing, if one of the fields it reads has no writer anywhere in the codebase. Wiring absence isn't only "nobody calls this" — it can be "nobody ever fills in the thing it reads."
• Silence from a check is not a verdict. Before trusting a clean record, ask whether the check has ever actually fired. A long spotless history can mean the input was never supplied.
• Verify the rule's positive class before wiring its producer. Getting that order backwards means correct inputs start getting rejected the day the plumbing lands. The derivation being provably right — 6 of 6 — said nothing about whether the rule built on top of it was right.
• A retroactive dry run against real data is part of the wiring, not a follow-up. The change was correct and the tests were green; only running it against everything the rule could evaluate showed the rule underneath was wrong.
• Two lists that describe different media can't contradict each other. Audio and pixels are orthogonal. A rule that compares across that boundary is measuring co-occurrence and calling it contradiction.
• The plausible true positive was never in the missing field. The contradiction this rule could actually have caught — must_have fighting must_not_have — sits in two lists that do govern the same medium. A rule aimed at the wrong pair produces false positives at exactly the rate the wrong pair happens to co-occur.

This rhymes with something I wrote up before: The Known-Good Sample Was Not Known-Good. There, a threshold was derived from a control sample nobody had actually verified belonged to its class. Here, a rule's entire positive class had never been observed at all. Different mechanism, same artifact: a number or a rule that feels validated because something was measured, when the thing measured was never the thing being claimed.

More notes at hexisteme.github.io/notes.