The Known-Good Sample Was Not Known-Good

Originally published on hexisteme notes.

I set a threshold from measurement instead of guessing. The measurement was clean: zero overlap between the two clusters, a 33x gap between them. I wrote the numbers into a comment with their sample sizes, feeling good about not having guessed.

It was wrong, because the sample I had labelled "known good" was one of the bad ones.

I've written before about checks that cannot fire — guards whose thresholds were miscalibrated for the scale of their input, so nothing you fed them ever tripped the line. This is a different animal. My threshold was calibrated from data. That's exactly what made it convincing, and it's why the calibration itself is where the bug lived.

The check

A video pipeline burns captions onto a rendered preview. A gate then diffs the burned output against the preview and treats every changed pixel as "text we drew," so it can ask whether our captions intrude into the platform's UI safe area.

That reading only holds if the two files are a pair — if this output was burned from this preview. Nothing verified that.

The only guard compared the number of sampled frames. Sampling is time-uniform, so two generations whose durations differ by 0.1s both yield exactly 60 samples. The guard was structurally incapable of noticing the thing it was nominally there to notice.

Setting the threshold

I wanted a statistical backstop: if the whole-frame difference between the two files is too large, they probably aren't a pair, so refuse to render a content verdict at all.

Exactly one episode in the repo had both files sitting on disk. I used it as my positive control.

sample	median whole-frame abs diff
"correctly paired" episode	19.51
known-mismatched pair	98.65

Threshold: 55.0. Zero overlap, a 33x gap. Two clusters, cleanly separated. Done.

The control was a negative

That episode's preview file had an mtime nine hours later than its output — and later than the gate run that had already approved it. The preview on disk had been re-rendered after the burn. It was never a pair.

So 19.51 wasn't "what a pair looks like." It was "what a non-pair looks like." I had drawn my line using a negative sample as my positive control, and every statistic downstream inherited that.

Manufacturing a real control

I couldn't find a verified pair anywhere, so I made one: ran the burn and recorded a content-hash link between the preview and the output at the moment the output was produced. Then I measured again.

A genuine pair: 2.94. On a second episode, 3.37. Both land exactly on the background re-encode noise figure — 3 to 4 — that the same comment file had documented long before. The evidence had been sitting in my own repo disagreeing with me the whole time.

The real picture had three bands, not two:

band	values	character
verified pair	2.94, 3.37	normal
same episode, different burn generation	19.31, 19.51	looks normal, produces false verdicts
entirely different content	62.88, 97.68, 98.65	obviously wrong

What it cost

The dangerous band is the middle one, and it sat below my threshold.

So the backstop passed the exact class of failure it existed to stop. It had already recorded a "212px safe-area violation" against an episode that was published. Re-run against the genuine pair, that episode is clean. The 212px blob covered a quarter of the frame — it was two unrelated regions merged by a comparison of two different pictures, not caption creep. A defect that never existed, filed against an artifact that was already live.

Band three gets caught by any threshold you pick; you don't need measurement to separate 3 from 98. The only place a threshold does real work is band two — and I had never measured band two at all. I drew a line between what I believed was band one and what I knew was band three, and the entire region where the check actually operates fell inside the pass zone.

Re-deriving

Log-midpoint of the worst verified pair (3.37) and the best-known bad pair (19.31): sqrt(3.37 * 19.31) = 8.07, so 8.0. Symmetric in log space, 2.4x of headroom in each direction.

Verified pairs: n = 2. A 2.4x margin on two samples is not statistics, and I said so in the comment. What decided the direction wasn't confidence, it was cost asymmetry:

• A false positive says "re-render this." The affected population is finite and closed — only artifacts predating the new bookkeeping.
• A false negative ships a fabricated violation into a publish decision, and can mask a real one.

When the sample is too thin to tell you where the line goes, it can still tell you which side to be wrong on.

The threshold was never the fix

A threshold is a proxy for a fact you failed to record. The actual repair was to record it: at the end of every burn, write one line linking preview_sha256 to output_sha256, and have the gate use that line to prove the pairing instead of inferring it from pixels.

It has to be a content hash rather than a filename, because the last step of this pipeline is a human copying a candidate file over the canonical name. A filename-keyed link dies at that copy. A copy preserves bytes, so a hash-keyed one survives — and it did: immediately after promotion, the gate reported the pair as proven rather than estimated.

The statistical backstop still exists, but it's been demoted to a fallback for artifacts produced before the bookkeeping did.

What I'd tell myself

1. For every control sample, write one line: what guarantees this sample is in this class? A filename, an mtime, or "it's the only candidate I have" is not a guarantee.
2. If there's no guarantee, manufacture the control. Run the process once and capture an artifact whose identity is established by construction rather than by inference.
3. Look for three bands, not two. Normal, obviously-broken, and the middle band that looks normal while producing wrong answers. If the middle band sits below your threshold, your threshold is decoration.
4. With a thin sample, pick the direction by cost asymmetry, and write the n and the blind spot into the comment next to the number.
5. Prefer recording the fact over inferring it. Thresholds are what you reach for when the provenance wasn't written down.

The failure mode I want to flag hardest is the one in my second paragraph: zero overlap, a 33x gap. Separation metrics tell you two clusters differ. They say nothing about which cluster is which — and they are extremely good at supplying the feeling of having verified something.

More notes at hexisteme.github.io/notes.