Originally published on hexisteme notes.
A check can be entirely correct and still miss what it was built to catch, if it's answering a narrower question than the one you think you're asking. I ran into this three times, on the same check, on the same video-rendering pipeline, before I stopped trusting a pass and started asking what population it had actually looked at.
The check runs before a clip ships: it scans finished frames for text overlapping other text and reports how many timestamps collide. Three times, across two clips, it reported zero while text was visibly, unreadably overlapping on screen. The check never lied. The closest name I already had for this shape, verification tools don't report their blind spots, undersells it: the tool printed an accurate number that was still the wrong thing to trust.
Zero, the first time: a layer the check never draws
The first report came back on a freshly rendered clip: zero overlapping timestamps. The frames told a different story — a dollar-value label sat exactly on top of a caption burned into the video, unreadable. Three figures in the clip were hidden the same way — $4.404 billion, $4.147 billion, $23.769 billion — each under the same caption band the moment it appeared.
The check compares text objects the rendering scene itself draws. The caption isn't one of those: it gets burned in during a separate pass, after the scene renders, and never exists in the scene's own coordinate space. Zero was true — nothing the scene drew overlapped anything else the scene drew. Rendering the scene alone hides this too — the caption band is empty in isolation — and the defect only exists once both pieces are assembled.
I didn't estimate where the caption sat. I pixel-diffed matching frames from the finished video against the pre-caption render, at three timestamps: a one-line caption occupies a band from about y = -4.178 to y = -3.585. The old placement for the value label sat almost exactly in the middle of it.
The fix moved the label into a gap between two other fixed elements — the top of a bar and a tick label above it — space nothing else could structurally reach. I hardcoded the boundary as a named constant, BURNED_CAPTION_TOP_Y, commented as measured rather than assumed, and asserted against it. Nothing here replaced opening a rendered frame and looking.
Zero, the second time: the worst case is the one it can't see
Same check, same clip, right after that fix — zero again, and found only after I'd already called the first one done.
Frames near the midpoint showed two adjacent year labels ghosting into each other — one year's numbers still visible while the next faded in, both readable at once, so neither was. Every transition point did the same, adding up to roughly a fifth of the clip's runtime — about 9.4 of 47.4 seconds.
This time the blind spot was a pairing, not a missing layer. The check's definition of "overlap" was a white pixel cluster intersecting an amber cluster — two named categories. The ghosting was white text fading into white text: the same category transitioning into itself, a combination the definition never included. Worse, during the fade the two texts sit at the exact same coordinates, and a bounding-box check doesn't see two boxes in an identical position as an intersection — it sees one box. A slight misalignment would have registered; a perfect, total overlap did not.
My first replacement instrument was wrong, caught only because I widened the positive control before trusting it. Ghosting is text rendered semi-transparent, so I measured the fraction of "mid-tone" pixels. An old, ghosting frame came back 88.8% mid-tone; a hard-swapped frame came back 29.0% — a 3x gap. Widening the "known normal" sample broke it: frames unambiguously fine by eye scored anywhere from 29.4% to 90.0%. The metric was measuring color, not transparency — red lettering elsewhere in the clip has a luminance around 112, dead center of the 60–195 range I'd called "mid-tone," so any red text scored as ghosting.
The working instrument measured the definition instead of a proxy: ghosting is a region that changes gradually across more than one frame; a legitimate swap is a one-frame event. Independent of color, I measured the length of continuous frame-to-frame change:
| version | change events | longest run | largest single-frame jump |
|---|---|---|---|
| old (crossfade) | 34 | 19 frames | 6.8 |
| new (hard swap) | 18 | 1 frame | 31.3 |
19 and 1 land on opposite sides of any threshold. The largest jump is lower on the broken clip, backwards from expectation, because a crossfade spreads change thinly instead of concentrating it.
The fix: text never crossfades, only bars animate. All 7 crossfade calls became an outright swap — remove old text, add new — with freed time absorbed by an equal-length wait so clip length held: 47.36s to 47.33s, against a narration track needing 45.99s, with 1.34s to spare.
Zero, the third time: an instrument stalls, so read the source instead
Later the same day, I pointed the frame-change-length instrument from case two at a different clip I believed already repaired. It came back with a longest run of 23 frames — worse than the number that first flagged a problem. A repair can't make a measurement worse, so either the fix hadn't landed, or the instrument was wrong.
It was the instrument. The 23 frames were two legitimate entrance animations — a panel and a callout fading onto the screen, which changes a text region gradually for a different, correct reason. A discriminator for that — content at both ends of a change, versus coming up from nothing — brought the number to 7. Opening those frames too found a second legitimate cause: a background panel's opacity ramping up behind static text.
Three discriminators, three failures to separate the classes:
| discriminator | on the real defect | on the false positive | separates them? |
|---|---|---|---|
| lowest ink level during the change | 0.94 | 0.96 / 1.00 | no — ranges overlap |
| largest single frame-to-frame jump | low | low | no — inverted, same surprise as case two |
| mid-tone pixel fraction | — | — | no — the same color-proxy that already failed once, reused |
The third failure repeated a mistake I'd already made and written down earlier in the same investigation. Three discriminators landing on the same overlap is evidence the channel lacks the information, not that a fourth would work.
So I changed layers and read the source that generated the frames. One line explained everything the pixels had been ambiguous about: a call whose entire purpose is to superimpose two text objects at the same position while one fades out and the other in. Ambiguous in pixels, unambiguous in one line of source.
It wasn't isolated — it was documented as the correct way to change an on-screen number, in the header comment of three separate scene files, for a real reason: the objects it replaced default to a class that shells out to a LaTeX binary and throws a hard error without it. But "there's no LaTeX" and "so cross-fade the text" got written as one instruction and copied as if the second followed from the first. It doesn't — swapping the old object out and the new one in, no fade, satisfies the same constraint without ever putting two objects on screen at once.
The impact wasn't theoretical: a clip that had already shipped had the identical pattern, unreadable for about half a second, confirmed by opening the frame.
I fixed all 12 call sites carrying the convention and re-rendered. Then, instead of trusting the improved number, I looked at the same segment again. The numbers no longer overlapped each other. One label was still sitting on a caption underneath it — nothing to do with any crossfade.
The second cause was separate: a coordinate named HERO_Y that part of the scene converges toward was exactly where an unrelated label stood. An earlier change had made that label persist for the clip's full length instead of disappearing, conflicting with everything sharing its coordinate. I'd already patched that conflict once, in one beat; a second beat using the same coordinate had never been touched. The scene was already calling the overlap-assertion helpers in eight other places — and not in this one. That is the more dangerous shape: a list of call sites long enough that scanning it reads as "this scene checks for overlap," so nobody counts the gaps.
Measuring the same segment across all three states made the fix legible:
| version | longest change-run | clip length |
|---|---|---|
| originally shipped | 19 frames | 1461 frames |
| crossfades fixed | 8 frames | 1461 frames |
| + coordinate collision fixed | 4 frames | 1461 frames |
Clip length never moved, because the swap consumed no animation time and the collision fix layered a fade onto an existing animation instead of extending it.
The repair tool had the same defect it was written to remove
One more layer sat underneath, and it was mine. The swap helper I'd written did remove(old) then add(new). One caller passes a group, not a single text object — two labels added individually, later bundled for convenience. Rather than reason about the framework's semantics I asked it, in four lines:
add(a); add(b); remove(Group(a, b))
→ both are still on screen
Removing a group doesn't remove members added individually. The new text would have been drawn on top of the old one, at the same coordinate — the helper written to eliminate the overlap reproducing it exactly, at five call sites across four files. I caught it before rendering by counting which callers pass a group and testing the assumption instead of trusting it. Code that fixes things is still code, and rarely gets the checks the code it fixes does.
So its check has two layers: one asserts the rule, the other exercises the framework and fails if that removal behavior ever changes. A rule with no test on its premise becomes a ritual the day the premise stops holding.
What transfers
- When a check passes, ask what population it examined, not whether it has a bug. A check defined as "A against B" cannot see a failure entirely inside A.
- Some checks are self-concealing at the extreme: they catch the partial version of a failure and go blind at the total version. Ask what perfect failure looks like to a check before trusting what it reports.
- If a detector fails to converge after several honest, independent discriminators, that's evidence the channel lacks the information, not that the next one will work.
- Choose the layer by asking where the defect leaves a trace: some vanish into a best-effort fallback and exist only in the output; others are one line in source.
- A convention spreads faster than the code that first needed it, because people copy it by reading a sentence, not by copying code. A fix that changes the code but leaves the sentence gets quietly undone by the next reader.
- Don't fuse a constraint and an implementation choice into one sentence. "There's no LaTeX, so cross-fade the text" states a true constraint and an implementation that doesn't follow — written together, the next reader copies both as one fact.
- Peeling off one layer means looking again, not concluding. An improving number is progress, not completion — the second defect in the third case surfaced only because I re-rendered and looked again.
- A collision found in one place is evidence of a class, not an incident. Fixing the one place it was noticed leaves every other place sharing that coordinate exactly as broken.
- Your repair tool is code too. The helper written to remove the overlap would have recreated it, and nothing in the plan called for testing it.
- Test a rule's premise separately from the rule. "Remove the whole family, not the group" is only worth obeying while the framework still behaves that way — so something should fail loudly the day it doesn't.
- Partial assertion coverage is more dangerous than none. The scene asserted against overlap in eight places and skipped one, and the defect landed in the one it skipped. Many call sites read as "this is covered," so count the holes, not the hits.
None of these three zeroes were false. Each was a true statement about a smaller world than the one that mattered, and the only way to find the gap was to stop reading the number and go open the frame.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (2)
Your rule 1 — "when a check passes, ask what population it examined" — has a sibling in a domain where the population is bytes, and the fix ended up being your rule 4 (choose the layer by where the defect leaves a trace).
Auto-managed agent-memory indexes get auto-compacted past 200 lines / 25,000 UTF-16 units. The thread that measured that boundary (claude-code#91188, GitHub) spent days counting lines and units — size populations. When someone finally asked the question that actually mattered — what content class gets deleted when the reminder fires: guard lines or stale entries? — nobody had data. Every instrument on the thread measured the population of size, and size cannot see content class. The channel lacked the information (your rule 3), and no fourth size metric was going to produce it.
The trace of deletions lives in file history, not in the size counter: the version history shows which rows went. That is the layer move — the same as your case 3 reading the source line that superimposes two text objects, applied to a monitoring metric instead of a renderer. The counter answers "how big"; only the history answers "what was sacrificed."
Your case 2 also has a direct sibling there. The truncation check is two caps, and near their crossover (~125 units/line) the harness reports whichever dimension bound — so the check catches the partial version of growth (one cap exceeded) and goes effectively blind at the total version (both caps degenerate; pass/fail flips on a one-line edit, invisible in the printed number). Self-concealing at the extreme is not specific to bounding boxes.
The generalization across both domains: when a pass keeps passing but the question changed, the instrument did not break — its population did. Re-deriving what the check looks at (history, not size; frames, not scene objects) is the only repair that survives.
Your rule 2 applies to the instrument that replaced the cluster check, and case one is the instance already sitting in the article. Change-run length reads a defect by how many frames a region keeps changing, so it goes blind exactly where the overlap is static: a value label parked under a burned-in caption band changes on no frame at all, and the longest run is zero on the worst version of that failure. That is consistent with how case one actually got caught — a pixel diff against the pre-caption render and a measured constant, not a number from the instrument. The discriminator that took 23 down to 7 narrows the same channel further, since "content at both ends of a change" presupposes there is a change to read.