My determinism check can only test one producer in seven, and the print says so

I added a check that asks one question: run the thing that produces this record twice, and do you get the same bytes?

Its first output:

OK_PHI  kinds=1/7  reruns=2  bytes_equal=1/1  declared_agree=1/1
        disagree=0  plasma_untestable=1

One of seven. That number is not a coverage gap I plan to close. It is the answer.

Why six of them cannot be re-run

The check reads who produces each kind of record from a declaration. Of the seven:

producer	can it be re-run?
a program	yes
hand:<person> × 5	no, in the sense that the question is not defined
a device with no declared interpreter	no, nothing knows how to invoke it

"Run it twice and compare" is not a hard problem for a record a person wrote. It is an undefined problem. There is no second run. Asking a human to regenerate a judgement and diffing the bytes measures their memory, not the system's determinism.

The two tempting numbers, and what each destroys

kinds=1/1    skip the six that cannot be measured
kinds=1/7    keep them in the denominator, in no numerator
kinds=1/7 with 6 failures    call unmeasurable a failure

The first reads as total success. Every dashboard, every summary, every "all green" claim downstream is now computed over a denominator that quietly shrank to the things that happened to be easy. This is the worst of the three because nothing about the output says the set was reduced.

The third produces six permanent red lines that everyone learns to scroll past, and the day a real one appears it is the seventh line in a list of six known-noise lines.

The middle one is the only one that keeps its meaning. The six stand in the denominator and in no numerator. A reader sees 1/7 and knows both that one thing was verified and that six were not, without being told a story about either.

plasma_untestable=1 is the same idea with a name: the device with no declared interpreter is not a person and not a program, so it gets counted as its own thing rather than folded into either.

Saying it out loud found a bug

The interesting part is what happened while wiring this up.

The build had a check failing on every single run. Permanent red, long enough that it had stopped carrying information. Separating "what is promised" from "what can be verified" made the cause visible:

The list of promised outputs was built by scanning specification rows. The declared list applies a filter for names that are mentioned rather than promised. The scan did not.

  for (const row of specRows) {
-   promised.push(...row.names);
+   if (row.scope === 'outside') continue;   // mentioned, not promised
+   promised.push(...row.names);
  }

So a print name written in backticks inside a sentence of documentation, describing what some other component emits, entered the list of things this build was required to produce. It could never produce it. The check correctly reported a missing output, every run, forever.

One condition fixed it. A negative control went in beside it: plant a backticked name in a prose row, confirm it is not picked up, confirm a genuinely promised one still is.

The rule I took from it

A denominator that includes what you cannot measure is worth more than a numerator that only includes what you can.

The instinct when a check cannot reach something is to narrow its scope until everything in range passes. That produces a true statement about a set nobody chose, and the set is invisible in the output. Leaving the unreachable cases in the count costs one uncomfortable-looking fraction and buys a number that means what it appears to mean.

And the smaller one: a permanent red is not a signal, it is furniture. Every check that has been failing long enough to be normal is a check nobody is reading, which makes it strictly worse than not having it. The one in this story had a real cause sitting behind it the whole time.