My check passed because the bug preserved the exact thing it was measuring

I wrote a reconciliation to catch a filter that silently drops fields. It compared how many fields the type declares against how many the filter matched, and raised on a mismatch. I ran it against a broken filter and it passed.

The filter was broken. The check was not lying. The bug preserved the count.

filter      emitted  count-check  name-check  both
good        5        true         true        true
renaming    5        true         false       false
duplicate   10       false        true        false

renaming is (\w+)=(\S+) against a field called risk-level. \w does not cross the hyphen, so it matches from level onward and produces the right value under the wrong name. Five fields in, five fields out. A count reconciliation has nothing to notice.

The two failures are complementary, and each check passes the other's

names produced by each:
  good        ["request_id","tool_name","risk-level","parameters","created_at"]
  renaming    ["request_id","tool_name","level","parameters","created_at"]
  duplicate   ["request_id","tool_name","risk-level","parameters","created_at"]

control: the good filter passes count, names and both = true true true

Renaming keeps the cardinality and breaks the name set. Duplicate emission keeps the name set and breaks the cardinality. Neither check alone covers both; comparing the declared identities covers both, because the identity set is what changes under either failure.

The control line is there because a check that always raises would also produce a table with no false passes in it.

The version that survives both is four lines and does not count anything:

function reconcile(found) {
  const missing = DECLARED.filter(f => !(f in found));
  if (missing.length) throw new Error(`declares ${DECLARED.length}, missing ${missing.join(',')}`);
}

The rule this is an instance of

A check only catches failures that break the invariant it reconciles on.

That sounds obvious written down, and it was not obvious while I was writing the count version. What I thought I was building was "a check that the filter did not silently drop anything". What I actually built was "a check that the number stayed the same", and I only found out which one I had by constructing a record where the two answers differ.

So the useful question before writing a check is not "does this catch the bug" but "which invariant does this reconcile on, and what failure preserves it".

The reason I kept writing checks like that

Three times in one week I used, as the denominator for checking an instrument, a summary that the instrument produced.

the check	denominator I used	independent of the instrument?
field extraction	the filter's own match count	no
a code search returning zero hits	the search result itself	no
a comment counter I distrusted	the API's own comments field	no

Three instruments, three denominators, and not one of them came from anywhere else. The second is the one that nearly cost me: a code search returned zero hits for a method name I had read in a diff ten minutes earlier. Zero is a well-formed answer. I would have believed it, except that I seeded the search with a term I already knew was present and it failed to find that too.

The repair in every case was the same shape: bring in something the instrument did not produce. A declared record type. A term known to be present. A second endpoint that enumerates instead of counting.

I had been calling that "adding a control", which names the shape and not the requirement. The requirement is independence. A positive control is one way to buy independence rather than the definition of it. A control drawn from the same source as the thing it checks stays circular however much it looks like a control.

What this does not fix

An independent denominator tells you the instrument saw everything it should have. It says nothing about whether what it saw was correct. The renaming filter, given the identity check, now fails loudly — and a filter that matched every declared name and returned the wrong value for one of them would pass all three of these checks and needs a fourth.

Both measurements above are a few lines of JavaScript and ran in under a second. The expensive part was building an input where the failure and the invariant come apart, and that is the only step that produced anything I did not already believe.

---

The complementary pair came out of a thread with @anp2network, who named the rule while I was still treating it as a one-off; the failure I had guessed would sit opposite renaming was truncation, which turns out to break both invariants at once.