When You Delete a Rule, Hunt the Tests That Went Quiet

Originally published on hexisteme notes.

I deleted a validation rule from a video pipeline that had never once been right. Twenty tests turned red immediately, and fixing them was mechanical — one assertion at a time. Then I went looking for the tests that hadn't turned red, and found four that had quietly stopped testing anything at all.

The twenty red tests were the safe ones. Failing loudly, they told me exactly what they were coupled to. The four quiet ones told me nothing — they just kept passing, for reasons that had stopped having anything to do with the feature they were named after.

The rule I deleted

The pipeline builds video shots. Each shot carries a contract: a must_have list, a must_not_have list, and serves_line — the sentence of narration the shot exists to depict. One validation rule checked a shot's contract against itself, and it had two clauses.

The first was lexical: if the narration and a forbidden item share a content word — the line says "three doors," the contract forbids "doors" — nothing can legally be drawn, so it's a hard failure. Reasonable.

The second banned numeral class: if a forbidden item rules out numeric notation (number, digit, numeral) and the narration states a number, that also counted as a conflict.

The second clause was a category error. serves_line is narration — it goes in your ears. must_not_have governs what's on screen — it goes in your eyes. A shot can say "twenty thousand games" out loud while drawing no digits at all; that isn't a contradiction, it's just house style. Run against real contracts, this clause fired eight times. It was wrong eight times.

So I deleted it. That part really was easy.

Twenty red tests, and the four that weren't

The twenty failures were exactly what you'd expect from removing a rule: assertions that a numeral-only input produces a conflict, now producing none. Delete the assertion, rewrite it against the clause that survived, move on. Bookkeeping.

But a deleted rule doesn't only break the tests that assert on it directly. It rewrites the meaning of every fixture that was chosen because it happened to trigger that rule — without touching the fixture's type, shape, or name. Nothing in the toolchain reports this. The test still runs. It still passes. It has simply stopped measuring anything.

I had a module-level constant holding a real narration line whose only conflict ran through the deleted clause. Four tests were built on it:

# "declaring metaphor exempts the conflicts"
conflicts = find_conflicts(LINE, FORBIDDEN, relation="metaphor")
assert conflicts == []

That test passed before the deletion and passed after it, and the two green checkmarks meant opposite things. Before, it proved the metaphor exemption actually suppressed a real conflict. After, there was no conflict left to exempt — the assertion had quietly become [] == []. It was green because the input had gone inert, not because the exemption still worked.

The same rot had reached three more:

• A test asserting that omitting the relation argument behaves like literal — it was comparing two empty lists.
• A test asserting that the exemption preview stays empty unless the relation is metaphor — it was empty for every relation, metaphor included, so the boundary the test was named for no longer existed.
• An integration test asserting that a literal contract still hard-fails. This one did go red, with DID NOT RAISE — and it's the only reason I opened this group of tests at all.

Three of the four would have sat there passing indefinitely, reading as coverage, in a file that no longer tested the thing its name promised.

How to hunt them

The audit itself is mechanical, and took about five minutes once I knew to run it:

• Grep every use of any fixture the deleted rule depended on.
• Open all of them — the ones that just went red included, not only the survivors.
• For each one, ask a single question: does this input still trigger anything? If the answer is no, the test is asserting a tautology.
• Replace the fixture with one that exercises the rule that's still alive.

The second step is the one that takes deliberate effort, because the test runner has already told you those files are fine. You're not chasing a failure. You're second-guessing a pass.

Two more things the deletion taught me

Check the spec before you check reality. I'd falsified the numeral clause empirically — prototypes, sweeps across real contracts, a false-positive tally that came out eight for eight. Only afterward did I read the domain glossary, which had defined the hard-failure rule the whole time as "must_not_have forbids a core noun of serves_line." A core noun. Numerals were never in the specification. The clause was an expansion nobody had asked for, and the cheapest possible refutation had been sitting in a glossary file the entire time.

Measure the replacement before you ship it. My own earlier notes had proposed where the true positives really lived: contradictions between must_have and must_not_have on the same shot. It sounded right, so I prototyped it and ran it across the real corpus. Four firings, all four false positives. The two lists routinely describe the same subject from opposite sides by design — must_have: "the door remaining closed" and must_not_have: "door fully opening" share the word door while fully agreeing with each other. Plain word overlap can't tell contradiction from paraphrase. So I deleted a rule and shipped no replacement for it, and wrote down why, so the next person doesn't rebuild it.

The general shape

Deleting code silently rewrites the semantics of the data your tests feed it. A test suite only measures what its inputs manage to provoke, and removing a rule can leave an input provoking nothing at all, while every signature — the fixture's type, shape, and name — stays exactly as valid-looking as before.

So the completion criterion for a deletion isn't "the suite is green." It's narrower:

For every test that touched the deleted rule, does its input still reach live code?

A red test is a question the suite is asking you. A test that went quiet is one that stopped asking — and silence reads exactly like success.

Three related traps, and how this one differs

This isn't the only way a green suite has lied to me, and it's worth being precise about which failure this is:

• Green tests prove behavior, not reachability is about whether production ever calls the code under test at all — these four tests were reachable and ran fine, it was the assertions inside them that had gone hollow.
• The patch and the test were wrong in the same direction is two artifacts written together, sharing one shallow reading of a requirement — these four tests were correct the day I wrote them, and only went vacuous later, when an unrelated deletion pulled the ground out from under them.
• Unit tests passed, the feature never ran in production is a wiring gap between a tested component and its live call site — nothing here was ever unwired, the tests just kept exercising a code path that no longer had anything to do.

Twenty red tests were never the risk. The four quiet ones were — and the only reason to go looking for them is that nothing in the toolchain will ever ask you to.

More notes at hexisteme.github.io/notes.