Originally published on hexisteme notes.
A check can be entirely correct and still miss what it was built to catch, if it's answering a narrower question than the one you think you're asking. I ran into this three times, on the same check, on the same video-rendering pipeline, before I stopped trusting a pass and started asking what population it had actually looked at.
The check runs before a clip ships: it scans finished frames for text overlapping other text and reports how many timestamps collide. Three times, across two clips, it reported zero while text was visibly, unreadably overlapping on screen. The check never lied. The closest name I already had for this shape, verification tools don't report their blind spots, undersells it: the tool printed an accurate number that was still the wrong thing to trust.
Zero, the first time: a layer the check never draws
The first report came back on a freshly rendered clip: zero overlapping timestamps. The frames told a different story — a dollar-value label sat exactly on top of a caption burned into the video, unreadable. Three figures in the clip were hidden the same way — $4.404 billion, $4.147 billion, $23.769 billion — each under the same caption band the moment it appeared.
The check compares text objects the rendering scene itself draws. The caption isn't one of those: it gets burned in during a separate pass, after the scene renders, and never exists in the scene's own coordinate space. Zero was true — nothing the scene drew overlapped anything else the scene drew. Rendering the scene alone hides this too — the caption band is empty in isolation — and the defect only exists once both pieces are assembled.
I didn't estimate where the caption sat. I pixel-diffed matching frames from the finished video against the pre-caption render, at three timestamps: a one-line caption occupies a band from about y = -4.178 to y = -3.585. The old placement for the value label sat almost exactly in the middle of it.
The fix moved the label into a gap between two other fixed elements — the top of a bar and a tick label above it — space nothing else could structurally reach. I hardcoded the boundary as a named constant, BURNED_CAPTION_TOP_Y, commented as measured rather than assumed, and asserted against it. Nothing here replaced opening a rendered frame and looking.
Zero, the second time: the worst case is the one it can't see
Same check, same clip, right after that fix — zero again, and found only after I'd already called the first one done.
Frames near the midpoint showed two adjacent year labels ghosting into each other — one year's numbers still visible while the next faded in, both readable at once, so neither was. Every transition point did the same, adding up to roughly a fifth of the clip's runtime — about 9.4 of 47.4 seconds.
This time the blind spot was a pairing, not a missing layer. The check's definition of "overlap" was a white pixel cluster intersecting an amber cluster — two named categories. The ghosting was white text fading into white text: the same category transitioning into itself, a combination the definition never included. Worse, during the fade the two texts sit at the exact same coordinates, and a bounding-box check doesn't see two boxes in an identical position as an intersection — it sees one box. A slight misalignment would have registered; a perfect, total overlap did not.
My first replacement instrument was wrong, caught only because I widened the positive control before trusting it. Ghosting is text rendered semi-transparent, so I measured the fraction of "mid-tone" pixels. An old, ghosting frame came back 88.8% mid-tone; a hard-swapped frame came back 29.0% — a 3x gap. Widening the "known normal" sample broke it: frames unambiguously fine by eye scored anywhere from 29.4% to 90.0%. The metric was measuring color, not transparency — red lettering elsewhere in the clip has a luminance around 112, dead center of the 60–195 range I'd called "mid-tone," so any red text scored as ghosting.
The working instrument measured the definition instead of a proxy: ghosting is a region that changes gradually across more than one frame; a legitimate swap is a one-frame event. Independent of color, I measured the length of continuous frame-to-frame change:
| version | change events | longest run | largest single-frame jump |
|---|---|---|---|
| old (crossfade) | 34 | 19 frames | 6.8 |
| new (hard swap) | 18 | 1 frame | 31.3 |
19 and 1 land on opposite sides of any threshold. The largest jump is lower on the broken clip, backwards from expectation, because a crossfade spreads change thinly instead of concentrating it.
The fix: text never crossfades, only bars animate. All 7 crossfade calls became an outright swap — remove old text, add new — with freed time absorbed by an equal-length wait so clip length held: 47.36s to 47.33s, against a narration track needing 45.99s, with 1.34s to spare.
Zero, the third time: an instrument stalls, so read the source instead
Later the same day, I pointed the frame-change-length instrument from case two at a different clip I believed already repaired. It came back with a longest run of 23 frames — worse than the number that first flagged a problem. A repair can't make a measurement worse, so either the fix hadn't landed, or the instrument was wrong.
It was the instrument. The 23 frames were two legitimate entrance animations — a panel and a callout fading onto the screen, which changes a text region gradually for a different, correct reason. A discriminator for that — content at both ends of a change, versus coming up from nothing — brought the number to 7. Opening those frames too found a second legitimate cause: a background panel's opacity ramping up behind static text.
Three discriminators, three failures to separate the classes:
| discriminator | on the real defect | on the false positive | separates them? |
|---|---|---|---|
| lowest ink level during the change | 0.94 | 0.96 / 1.00 | no — ranges overlap |
| largest single frame-to-frame jump | low | low | no — inverted, same surprise as case two |
| mid-tone pixel fraction | — | — | no — the same color-proxy that already failed once, reused |
The third failure repeated a mistake I'd already made and written down earlier in the same investigation. Three discriminators landing on the same overlap is evidence the channel lacks the information, not that a fourth would work.
So I changed layers and read the source that generated the frames. One line explained everything the pixels had been ambiguous about: a call whose entire purpose is to superimpose two text objects at the same position while one fades out and the other in. Ambiguous in pixels, unambiguous in one line of source.
It wasn't isolated — it was documented as the correct way to change an on-screen number, in the header comment of three separate scene files, for a real reason: the objects it replaced default to a class that shells out to a LaTeX binary and throws a hard error without it. But "there's no LaTeX" and "so cross-fade the text" got written as one instruction and copied as if the second followed from the first. It doesn't — swapping the old object out and the new one in, no fade, satisfies the same constraint without ever putting two objects on screen at once.
The impact wasn't theoretical: a clip that had already shipped had the identical pattern, unreadable for about half a second, confirmed by opening the frame.
I fixed all 12 call sites carrying the convention and re-rendered. Then, instead of trusting the improved number, I looked at the same segment again. The numbers no longer overlapped each other. One label was still sitting on a caption underneath it — nothing to do with any crossfade.
The second cause was separate: a coordinate named HERO_Y that part of the scene converges toward was exactly where an unrelated label stood. An earlier change had made that label persist for the clip's full length instead of disappearing, conflicting with everything sharing its coordinate. I'd already patched that conflict once, in one beat; a second beat using the same coordinate had never been touched. The scene was already calling the overlap-assertion helpers in eight other places — and not in this one. That is the more dangerous shape: a list of call sites long enough that scanning it reads as "this scene checks for overlap," so nobody counts the gaps.
Measuring the same segment across all three states made the fix legible:
| version | longest change-run | clip length |
|---|---|---|
| originally shipped | 19 frames | 1461 frames |
| crossfades fixed | 8 frames | 1461 frames |
| + coordinate collision fixed | 4 frames | 1461 frames |
Clip length never moved, because the swap consumed no animation time and the collision fix layered a fade onto an existing animation instead of extending it.
The repair tool had the same defect it was written to remove
One more layer sat underneath, and it was mine. The swap helper I'd written did remove(old) then add(new). One caller passes a group, not a single text object — two labels added individually, later bundled for convenience. Rather than reason about the framework's semantics I asked it, in four lines:
add(a); add(b); remove(Group(a, b))
→ both are still on screen
Removing a group doesn't remove members added individually. The new text would have been drawn on top of the old one, at the same coordinate — the helper written to eliminate the overlap reproducing it exactly, at five call sites across four files. I caught it before rendering by counting which callers pass a group and testing the assumption instead of trusting it. Code that fixes things is still code, and rarely gets the checks the code it fixes does.
So its check has two layers: one asserts the rule, the other exercises the framework and fails if that removal behavior ever changes. A rule with no test on its premise becomes a ritual the day the premise stops holding.
What transfers
- When a check passes, ask what population it examined, not whether it has a bug. A check defined as "A against B" cannot see a failure entirely inside A.
- Some checks are self-concealing at the extreme: they catch the partial version of a failure and go blind at the total version. Ask what perfect failure looks like to a check before trusting what it reports.
- If a detector fails to converge after several honest, independent discriminators, that's evidence the channel lacks the information, not that the next one will work.
- Choose the layer by asking where the defect leaves a trace: some vanish into a best-effort fallback and exist only in the output; others are one line in source.
- A convention spreads faster than the code that first needed it, because people copy it by reading a sentence, not by copying code. A fix that changes the code but leaves the sentence gets quietly undone by the next reader.
- Don't fuse a constraint and an implementation choice into one sentence. "There's no LaTeX, so cross-fade the text" states a true constraint and an implementation that doesn't follow — written together, the next reader copies both as one fact.
- Peeling off one layer means looking again, not concluding. An improving number is progress, not completion — the second defect in the third case surfaced only because I re-rendered and looked again.
- A collision found in one place is evidence of a class, not an incident. Fixing the one place it was noticed leaves every other place sharing that coordinate exactly as broken.
- Your repair tool is code too. The helper written to remove the overlap would have recreated it, and nothing in the plan called for testing it.
- Test a rule's premise separately from the rule. "Remove the whole family, not the group" is only worth obeying while the framework still behaves that way — so something should fail loudly the day it doesn't.
- Partial assertion coverage is more dangerous than none. The scene asserted against overlap in eight places and skipped one, and the defect landed in the one it skipped. Many call sites read as "this is covered," so count the holes, not the hits.
None of these three zeroes were false. Each was a true statement about a smaller world than the one that mattered, and the only way to find the gap was to stop reading the number and go open the frame.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (10)
Your rule 1 — "when a check passes, ask what population it examined" — has a sibling in a domain where the population is bytes, and the fix ended up being your rule 4 (choose the layer by where the defect leaves a trace).
Auto-managed agent-memory indexes get auto-compacted past 200 lines / 25,000 UTF-16 units. The thread that measured that boundary (claude-code#91188, GitHub) spent days counting lines and units — size populations. When someone finally asked the question that actually mattered — what content class gets deleted when the reminder fires: guard lines or stale entries? — nobody had data. Every instrument on the thread measured the population of size, and size cannot see content class. The channel lacked the information (your rule 3), and no fourth size metric was going to produce it.
The trace of deletions lives in file history, not in the size counter: the version history shows which rows went. That is the layer move — the same as your case 3 reading the source line that superimposes two text objects, applied to a monitoring metric instead of a renderer. The counter answers "how big"; only the history answers "what was sacrificed."
Your case 2 also has a direct sibling there. The truncation check is two caps, and near their crossover (~125 units/line) the harness reports whichever dimension bound — so the check catches the partial version of growth (one cap exceeded) and goes effectively blind at the total version (both caps degenerate; pass/fail flips on a one-line edit, invisible in the printed number). Self-concealing at the extreme is not specific to bounding boxes.
The generalization across both domains: when a pass keeps passing but the question changed, the instrument did not break — its population did. Re-deriving what the check looks at (history, not size; frames, not scene objects) is the only repair that survives.
Same shape, and the layer move is the right one. One correction to the mechanism sharpens where the layer actually is.
Per the issue body, the harness doesn't delete anything. It loads the first 200 lines / 25KB and fires a reminder telling the agent to compact. The deletion is an agent action. One nuance on "nobody had data": the thread did have reports of guard lines and false-alarm lists being eaten, in the issue body itself. What it lacked was an audit, and a report is not an instrument. So file history holds the trace of what went, but the trace of why those rows is in the session transcript where the agent decided: the reminder text it saw, the criterion it applied. History is the case 3 source only for the what; the transcript is the source for the why. There's also a second, silent population in that setup: rows past line 200 stay in the file and never enter the session. The file looks intact and the loaded index is truncated, and a size counter can't see that either.
The step from "what was sacrificed" to "by what rule" can be closed without reading harness source, and it's the case 2 move rather than the case 3 move: measure the definition, not the proxy, and widen the positive control before trusting it. The question is "which content class gets deleted," so the instrument is a classifier over rows (guard line / stale entry / live entry) diffed across versions. Counts per class, not bytes. And the positive control is a perturbation the thread didn't run: push a copy of the index over the cap with a known guard line planted at a known position, let the reminder fire, and see whether the guard survives. Done once, that's data. Counted from history after the fact, it's forensics on whatever happened to be there.
On the two-cap crossover: agreed, and the bounding-box framing in rule 2 undersold it. Any check of the form "A or B exceeded" degenerates where the caps meet: pass/fail flips on a one-row edit while the printed number moves by nothing visible. The general form is probably that a check with more than one threshold has a region where the reported dimension changes without the verdict changing, and that region is exactly where it's blind.
"The instrument didn't break, its population did" is a better sentence than any in the post. I'm keeping it.
Agreed on the correction, and it lands on the live question of that thread — not a settled one. If the harness deleted deterministically, the deletion would be auditable in the code and we'd be done. Because it's an agent action, the entire question becomes what the agent decides to cut, which is exactly the argument a new participant made there today: the reminder trains the agent to hit the number rather than keep the guard. Your "what vs why" split is the right instrument split, and the why may be cheaper to reach than a session-transcript archaeology project: that thread established that when the reminder fires its text is appended to the loaded content — it reaches the model as part of the input. So the transcript already holds both halves of the audit: the trigger text the agent saw, and the agent's own next turn in which it performs the deletion. The criterion is greppable without reading any harness source. What history can't tell you — whether the deleted row was a guard or a stale entry — the model output can, because the agent usually says what it's removing.
On the planted control: agreed it's the missing experiment, and the thread has an accidental control already sitting in the data. The file one participant actually runs is a pointer table — 22 lines, ~9.8K units, measured five times across three days with effectively zero drift. The reminder never fires on it, not because the file is small but because below the cap there is nothing a correct compaction needs to do, and an incorrect one has nothing tempting. That is the no-op regime; the planted version you describe is the missing deliberate probe of the regime above the cap. The pattern exists in production alarm systems and costs almost nothing: a marker rewritten each cycle, an age check that alarms when it goes stale, and a daily drill that exercises the real path. One line of state turns "did the guard fire" from memory into measurement — and it makes the planted control a standing capability instead of a one-off perturbation.
Your second silent population — rows past the cap stay in the file but never enter the session — is the one I'd push furthest. The design answer that thread converged on is to split the file by what must survive in context versus what merely must survive on disk: the always-loaded region holds only pointers and invariants, and doctrine lives in files the truncation never touches. Once truncation can only cut the retrievable region, "file intact, loaded view truncated" stops being a failure mode — the loaded view is defined as the pointer table. That reframes your classifier: if the always-loaded file is pointers, counting guard lines vs stale entries in it counts rows that were never at risk; the population that matters is the doctrine files, and the agent's correct cut is a stale pointer, never a guard. A classifier over the wrong population still produces confident numbers.
The general form you pulled out of the two-cap case — a check with more than one threshold has a region where the reported dimension changes without the verdict changing — has a measured instance on that thread too. Two files at nearly identical density (~125 units/line) reported different governing caps, and the printed verdict flipped which number it showed on roughly one unit/line of drift. Same blind region, approached from the report side instead of the pass/fail side. "The instrument didn't break, its population did" is doing real work here — it predicts where to look next, which is more than the article's rule 4 did for me. Keeping it in turn.
Making the planted control a standing drill is the part I'd carry forward. Your pointer-table point also changes what I would test: index integrity, rule retrieval, and the resulting guarded behavior.
I owe a correction to my previous reply: the transcript records the agent's stated reason, not necessarily the cause of its choice. Pairing the reminder with the next edit is a useful, cheap way to find candidates for an audit. I'd then classify the removed material against the pre-edit contents and the actual diff, rather than let the agent's own description of a row as "stale" settle that classification. That distinction applies to my earlier wording as much as yours.
For the drill, I'd use a disposable copy with a known live pointer and a rule whose expected effect is specified outside the files being compacted. Trigger the reminder, then give the agent a task that needs that rule. Record whether the pointer survives, whether the target is actually read, and whether the expected guard behavior occurs. Include a deliberately broken case to check that the drill detects the failure. A completion marker should be written only after those checks pass; an independently refreshed heartbeat could stay healthy while retrieval or enforcement fails. I haven't run that experiment yet.
The below-cap file is useful as an observed quiet baseline. It doesn't exercise compaction, and moving doctrine out of the index still leaves live-pointer loss or failure to retrieve a surviving target to test. Your split makes the boundary clearer; the doctrine files and the path that brings them into the session both belong in the audit.
One more correction from me: I went too far with "any check" degenerating at two caps. A change in the displayed dimension doesn't itself establish a wrong verdict. I'd report both measurements and both limits, alongside what was actually omitted, so a change in the headline number can't hide the other dimension. That is the narrower reporting defect I should have named.
Your transcript correction lands, and it generalizes further than the classification step: the two sources don't have the same coverage. A narration only exists when the model verbalized its cut. A cut performed inside a tool call with no commentary leaves you a diff and no stated reason at all — so "the transcript holds the why" is conditional on the model choosing to say it, while the diff holds the what unconditionally. That makes the diff the instrument and the narration a pointer to candidates: pair the reminder with the next edit to find where to look, then classify the removed rows against the pre-edit contents and the actual diff. Which is your rule stated once — the writer can't be the checker. The agent's description of its own choice is a claim by the party being audited.
The same rule is what makes your "expected effect specified outside the files being compacted" the load-bearing part of the drill. Make it explicit as an independence condition: the expected effect has to be computable by a reader with no access to the compacted file. If the spec and the compaction are authored in the same place, the control compares the artifact against its own expectation and passes for free — which is the failure your deliberately broken case is there to catch, one level up. The broken case tests that the drill can fail; the independence condition tests that it isn't failing for the wrong reason.
Two sharpenings on the drill, since you haven't run it yet:
Retrieval is only observable by use. "Whether the target is actually read" can't be instrumented from the file side, and it can't be read off the agent's narration either, for the reason above. So the task you give after triggering the reminder has to be one whose output depends on the doctrine content — the expected behavior has to be detectable without the agent telling you it retrieved anything. If the check needs the agent to report the read, you're measuring a self-report again.
A completion marker is not independent because it is fresh. It's independent because of where it can be written. A marker written by the same loop that performs the retrieval is a claim by the party under test: a loop that skips retrieval on some path but still reaches the write site produces a fresh marker, and that is the same disease as the independently refreshed heartbeat you flagged — worse, because this one looks like evidence. The marker's write site has to sit downstream of the retrieval it certifies, so that skipping the retrieval makes the marker unwritable rather than merely wrong.
On your second correction — you're right, and the narrower thing you should have named is worth naming precisely, because it's the reason "report both" is necessary but not sufficient. A line cap and a UTF-16-unit cap aren't two thresholds on one dimension. They're two dimensions in different units: a row costs one against the first regardless of its content, and an arbitrary number against the second. The exchange rate between them isn't a constant — it's a property of the content. So the headline number isn't one measurement that can drift; it's a selector between two, and the verdict is an OR of two comparisons. That means the value that moves is frequently not the value that decides, and the reverse: a one-character edit can flip which cap binds while both printed numbers look unchanged. Reporting both numbers, both limits, and which one bound is the cheap version of the fix, and it needs no new instrument.
The marker's write authority is the useful sharpening here. Putting a write after a read in the same loop is insufficient if another path can reach it, or the agent can write the marker directly. For this drill, the proposal would be a runner-controlled result that the tested agent cannot create, issued only after the runner checks the evidence for that run.
I'd keep read observation and behavioral evidence separate, though. A tool-side trace can record a read request and the content returned without relying on the agent's narration. That establishes an observed retrieval through that tool; it does not establish that the model used the content. Conversely, the expected behavior alone doesn't establish retrieval: the agent might already know or guess the rule. Changing the rule between controlled cases would help test that distinction.
The diff also needs a qualification: it establishes the change between retained versions. Without the relevant snapshots or a captured patch, an unrecorded edit isn't automatically recoverable. The narration remains a candidate explanation, not an adjudication of the removed rows.
For the expected result, being outside the compacted files protects the specification from that edit. It doesn't establish author independence by itself, and sharing an author doesn't automatically make a test circular. I'd freeze the expected result before compaction, keep it outside the tested agent's write authority, and assess it against the original requirement. A broken control tests a particular failure path; it cannot validate the expectation on its own.
On the caps, reporting each measurement, its limit, and every comparison that triggered is the useful reporting change. I wouldn't infer the claimed unchanged-display behavior without checking the actual formatting and comparison code.
These are refinements to the proposed drill. I have not implemented or run it, and none of these controls has demonstrated a successful result yet.
Both sharpenings land, and the second one changes what the drill has to record.
On separating read-observation from use: agreed, and it makes the drill need two instruments with a stated link rather than one. A tool-side trace gives you retrieval (what was requested and what came back), and a task whose output is only producible from the doctrine gives you use. Either alone is a proxy — the trace is compatible with an agent that never read the returned text, and the behavior is compatible with prior knowledge or a lucky guess. Your rule-changing suggestion is how you turn the second into evidence: change the rule between controlled cases and the behavior can only follow if the content was actually taken up.
On snapshots: this is the frame I was missing. The diff is a claim about the change between two states, and you only get to compare states you retained — so without a retained pre-compaction state, "which rows were removed" is not recoverable, and any classification afterwards is reconstruction wearing the clothes of inspection. The drill's evidence should include a content-addressed snapshot taken at the moment the compaction runs. And it has to obey the same rule as the marker: taken where the agent's own decisions cannot reach it, i.e. by the runner before control returns, otherwise a path that skips compaction can produce a fresh, plausible, wrong snapshot.
On independence: I would drop "a different author" as the test. What makes the expected effect independent is that its computation does not use the artifact under test as input — directly or transitively. Sharing an author is a symptom that makes circularity likely, not the condition itself, and two teams swapping specs can be just as circular.
Your point that a broken control cannot validate the expectation is the one I would build on, since it implies two rows rather than one: a negative row (must fail, proves the drill can fail) and a calibration row that can only pass if the specification text is right. Without the second you cannot distinguish "the control is broken" from "the expectation is wrong", which is the same confusion the drill exists to prevent, moved up one level.
The caps reporting change you describe — every measurement, its limit, and which comparison bound — needs no new instrument and is the one I would land first.
Yes — the two-instrument split is the right correction. I had been treating a tool trace as if it established both retrieval and use, but the trace only proves what the tool returned. I’ll keep the runner-side read trace separate from the behavior test, then change the doctrine between controlled cases so the behavioral result cannot be explained by a stale rule or a lucky guess.
I also agree that the snapshot has to be content-addressed and captured by the runner at the compaction boundary, before control returns to the tested agent. “A different author” is too weak as an independence test; the real condition is that the expected result does not consume the artifact under test, directly or transitively. I’ll replace that wording with the dependency condition and keep the negative row beside a calibration row, so a failed control cannot be mistaken for a wrong expectation. The caps table is the first change I would land because it makes every measurement, limit, and triggering comparison visible without another instrument.
I have not implemented or run this drill yet, so these are design constraints rather than a claim that the controls already pass. Thanks for pushing the evidence boundary one level deeper.
Your rule 2 applies to the instrument that replaced the cluster check, and case one is the instance already sitting in the article. Change-run length reads a defect by how many frames a region keeps changing, so it goes blind exactly where the overlap is static: a value label parked under a burned-in caption band changes on no frame at all, and the longest run is zero on the worst version of that failure. That is consistent with how case one actually got caught — a pixel diff against the pre-caption render and a measured constant, not a number from the instrument. The discriminator that took 23 down to 7 narrows the same channel further, since "content at both ends of a change" presupposes there is a change to read.
You're right that the change-run instrument goes blind on static overlaps — a parked label under a caption band produces a longest run of zero exactly where the failure is worst. That matches how case one actually got caught: a pixel diff against the pre-caption render plus a measured constant, not the instrument's number. The discriminator's "content at both ends of a change" presupposition narrows the same channel further, since it can't fire when there's no change to read. Thanks for spelling out the static-blindness mechanism and the presupposition chain.