DEV Community: John

Same Video File, Same Threshold, Three Answers: Caption Safe-Zone Frames Are Matched by Grid, Not by Count

John — Sat, 12 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

I run a gate that checks whether a short-form video's burned-in captions crossed into the frame's safe zone. It works by diffing two renders of the same clip: preview.mp4, rendered before captions get burned in, against final.mp4, rendered after. It samples sixty frames at equal intervals from each file, subtracts the i-th sample of one from the i-th sample of the other, and whatever's left over is supposed to be the caption and nothing else. If that leftover crosses outside the safe zone, the clip fails.

It failed a clip that looked completely fine. The report was specific: t=34.97s, right edge exceeded by 170px. I went to reproduce it and got three different answers from three different ways of asking what should have been the identical question. Re-running the gate's own function against the saved sample: zero violations. Calling the underlying comparison function directly: a violation, but at a different timestamp, on a different patch of pixels. Running the clip through the gate's actual production entry point: the original violation, at the original timestamp. Same file, same function, same threshold — three answers.

Three notes already sit near this one on this site, and it's worth being exact about why this isn't a repeat of any of them. One is about a check whose "zero" was correct for a narrower question than the one that mattered — a caption layer the check couldn't see, a same-color overlap its category system had no way to represent. Another is about three checks passing because each one watched the input or an intermediate structure instead of the rendered artifact. A third is about a detector whose zero-false-positive record was indistinguishable from one that never fires, until a separate positive-control test showed it could. This is none of those three. The check here was pointed at the right two artifacts, comparing the right kind of thing, and fully capable of firing. It still returned a confident, specific, wrong answer — because the two samples it was comparing were never the same moment to begin with.

Two numbers, a tenth of a millisecond apart

The gate derives its sampling interval from each clip's actual measured duration. For this pair, the computed value was 0.736117 seconds. What got written to the log was the rounded version: 0.736. When I reproduced the failure, I read the log instead of the code, and sampled at the rounded number.

On its own, a difference of 0.000117 seconds shouldn't matter to anything. But the sampling isn't continuous — under the hood, ffmpeg's fps filter is choosing one discrete frame per requested sample time, by presentation timestamp. When a requested time lands within roughly a millisecond of a 30fps frame boundary, a fractional difference that small is enough to flip which side of that boundary a sample resolves to: one computed interval selects frame n, the other selects frame n+1.

That alone would have just been a rounding curiosity, except one of the two files had already drifted off its own grid before any of this. preview.mp4 is assembled by concatenating segments, and one cut point didn't land cleanly on a frame boundary. Two frames near that cut carry a presentation timestamp of 502 ticks instead of the expected 512. From that point on, every frame in preview.mp4 sits 0.65 to 1.3 milliseconds earlier than the frame it's supposed to correspond to in final.mp4 — small on its own, but exactly the kind of small that decides which way a boundary-adjacent sample falls.

Combine the rounding with the drift, and one of the sixty sample pairs picked frame n from final.mp4 and frame n+1 from preview.mp4. The difference between two different frames of a moving scene isn't a caption — it's the entire frame. That whole-frame difference crossed the safe-zone boundary at twenty-two separate points near the edge, and the gate surfaced the largest of them as a 170-pixel intrusion at t=34.97s. The other fifty-nine samples, correctly paired, showed zero violations, because there was nothing actually wrong with the caption.

Three backstops, all green

This gate doesn't take it on faith that preview.mp4 and final.mp4 are comparable. Three checks stand in front of it specifically to catch a broken pairing: a sha256 match on the render receipt, a check that both files have the same frame count, and a check that the median of the sixty per-sample differences stays at or under 8.0. All three were green on this clip.

None of the three could have caught this — not because they're weak, but because none of them tests the thing that actually mattered. A matching frame count says both renders are the same length; it says nothing about whether frame N in one lines up with frame N in the other after two independent, floating-point-driven sampling passes. A matching receipt hash confirms the inputs to the render were the ones intended — a fact about provenance, not alignment. And a median is close to the worst statistic you could pick for a one-sample failure: fifty-nine correctly paired samples pull a single misaligned one toward the center of the distribution and bury it there. The failure was never in the bulk. It was sitting in one tail, and a statistic built to describe the bulk has no way to see into a tail.

"Same count" and "same grid" are different claims, and only the second one is what a pairwise diff actually depends on.

Putting both files back on the same grid

The fix doesn't touch the safe-zone threshold at all — it changes what "sample 34" means. Putting setpts=N/FRAME_RATE/TB, ahead of the fps=… filter re-times both files onto a frame-index grid before either one gets sampled, so "sample 34" resolves to "the 34th frame" — identically defined on both sides — instead of "whichever frame happens to sit closest to 34 times some computed number of seconds," a question the two files can answer differently the moment either timeline drifts even slightly off an exact multiple of the frame duration.

Before shipping that change, I checked what should hold rather than trusting that the fix was obviously right:

Question asked before shipping the fix	Answer
Does `final.mp4` (constant frame rate) produce byte-identical samples across repeated runs?	Yes — confirms a separate instrument that only reads `final.mp4` wasn't what was moving
Does the misaligned pair disappear using both the rounded and the exact interval?	Yes, at both
Does a normal, correctly-paired clip's median move at all?	No — 1.73 before, 1.73 after

That third row matters as much as either of the first two. A fix to a comparison instrument only earns trust if it leaves the comparisons that were already correct exactly where they were.

There's one more question worth asking directly, because it's the one a fix like this could get quietly wrong: could re-aligning the grid hide a real burned-in caption defect, by happening to pick a preview.mp4 frame that erases a genuine discrepancy? It can't, and the reason is structural rather than empirical. The caption exists in every frame of final.mp4 once it's burned in, and in none of preview.mp4's frames — that asymmetry is the entire premise of comparing the two files at all. Choosing an index-matched frame from preview.mp4 only changes which instant of scene motion gets subtracted out. It has no mechanism for making a caption that's actually present in final.mp4 disappear from its side of the subtraction.

Where this stops holding

The fix depends on a frame index meaning the same thing as a point in time, and that's only true if the frame rate is genuinely constant. If final.mp4's stored frame rate and its actual average frame rate ever diverge, "sample k sits at time (k + 0.5) × step" stops being a true statement, and the index grid and the time grid come apart again — just somewhere else. A clip like that needs a direct check comparing those two frame-rate values before its samples can be trusted at all.

It's also worth being precise about how narrow this particular drift is. Two earlier episodes built from a single simulated render, with no concatenation step involved, showed zero instances of a timestamp landing off the expected 512-tick grid. This isn't a general property of the renderer — it's specific to the path that assembles a clip out of separately-produced segments, which is exactly where a cut point can land off-grid in the first place. A clip that never goes through that assembly step has no known reason to trigger it.

What generalizes

If a check's entire job is pairing sample i of one artifact with sample i of another, sample both by index, not by an independently computed timestamp. Matching counts and matching grids are different claims, and a diff built on the wrong pairing will confidently report a defect that was never there.
When the backstop guarding a pairing is a sum or a median, look at the single largest sample separately. A statistic built to summarize a distribution is, by construction, insensitive to one outlier sitting inside it — this kind of false alarm lives in the tail, not in the shape of the bulk.
Reproduce with the value the checker actually computed, not the value it rounded for the log. A record that keeps only the rounded number quietly sends every future reproduction onto a slightly different grid than the one that actually ran. If a computed parameter gets logged, log where it came from too, not only its rounded value.
Before trusting a fix to a comparison instrument, measure what should not move. Zero threshold changes, plus byte-identical output on a file that had no reason to be affected, is stronger evidence than a clean run on the one case you set out to fix.
Fixing the consumer's tolerance for a drifted input doesn't close the producer defect that caused the drift. The off-grid timestamp in the concatenated file is still sitting there; it's simply no longer able to fool this particular gate. That belongs on the list as a separate, still-open item — not as something the fix already resolved.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Your Per-Edit Test Hook Is the Cost You Can't See

John — Fri, 11 Sep 2026 00:00:05 +0000

Originally published on hexisteme notes.

I audited a 24-day session on a short-form video pipeline I run with a small fleet of coding agents. On paper the headline numbers were unremarkable for a project that size: 52,188 main-thread turns, roughly 1,065 turns per episode, and 1,873 calls to pytest sitting right there in the Bash tool-call log. Then the person actually using the pipeline flagged something that didn't match any of those numbers at all: the worker, they said, kept trying tests it didn't need to. That complaint didn't trace back to Bash. It traced back to a hook I'd wired in during an earlier session and half-forgotten.

Not another measurement-artifact story

It's worth being precise about what kind of failure this is, because it sits right next to a few others I've written up that look similar from a distance and aren't. A measurement proxy inserted to observe a system can quietly change what that system does, inflating the very number you added it to see. Pooling behavioral metrics across two different roles in a fleet — a long interactive session and a short one-shot worker — without separating them first can turn two nearly-identical within-role ratios into a misleading pooled headline. Two different models can bill different token counts for nearly identical input, because the token meter itself is scoped to whichever model is doing the counting. All three of those are stories about a number that comes back wrong, or a number that's right but not comparable to the number sitting next to it.

This one has no number to begin with, wrong or otherwise. The cost wasn't measured incorrectly — it was never inside the measurement's field of view. A tool-call audit counts what runs through the tools it's watching, and a hook doesn't run through Bash; it runs through a layer that audit was never pointed at. Worse, when the hook succeeds it produces no output at all, so there's nothing for even a hook-aware audit to add up except the failures. This isn't an instrument distorting a reading. It's a cost source sitting on a wire nobody had an instrument on.

The mechanism

The hook itself was simple, which is exactly why it had been running unexamined. Every time an agent edited src/<mod>.py, a PostToolUse hook ran the entire corresponding suite, tests/test_<mod>.py — a run costing anywhere from two to three and a half minutes — and only spoke up, waking the model with a failure message, if something in that suite came back red. A clean run produced nothing: no log line, no transcript entry, no evidence that anything had happened.

Why a silent, per-edit hook is worse than it sounds

Three things stacked on top of each other here, and any one alone would have been tolerable.

First, it's invisible to the audit method you'd normally reach for. An audit that counts tool calls counts Bash invocations, API calls, background processes it already knows to watch — not hooks, and especially not a hook whose success path is silent. All you can ever recover after the fact is a lower bound, built out of whichever failures were loud enough to leave a trace.

Second, and worse than the blind spot, the hook was scoring the wrong moment. A change that touches multiple files is, by construction, broken partway through: you edit the first file, and until you've also edited the second, the tests covering the first file are correctly red. That redness is exactly what wakes the hook into failure mode, and the pressure it creates is "fix this now." Faced with that pressure, the model reverts or routes around the file it just touched — the one already edited — rather than moving on to the second file it hasn't gotten to yet, the one that would have actually made the suite pass.

Third, it was a cost nobody had asked for. Neither I nor the model triggered these runs on purpose — the hook fired on its own, off an edit event — which means nobody was ever in a position to ask "why is this running," the question that would normally catch a wasteful process before it repeats hundreds of times.

The numbers

Counting up what the transcripts actually showed over the 24-day window: 971 failed hook runs, totaling 5.0 hours. Edits in the main session alone were enough to trigger the hook 533 times — before adding in whatever the sub-agent workers triggered on their own edits, which ran through separate transcripts not folded into that count. The true number of times the hook fired, successes and failures combined, isn't something I can reconstruct after the fact; silence doesn't leave a receipt.

The per-file cost behind those numbers: test_cli took 216 seconds, test_map_scenes took 160, test_compliance_gate took 134 — every one of those running in full, after every single edit to the file it covered.

After the fix, a smoke test told the other side of the story: a queue of 3 pending files drained down to pytest running against 2 files, in 4 seconds.

The fix: move the trigger from the editor to the author

The fix keeps the same suite and the same enforcement intent — verification still has to happen — and only changes when it fires.

The PostToolUse hook now does one thing: it appends the path of the file that was just edited to a queue file. That's a write of a few milliseconds, not a multi-minute test run.

A Stop/SubagentStop hook — the one that fires when a turn actually ends — reads that queue, de-duplicates it, and runs pytest exactly once, covering every file touched during the turn. It returns a summary to the model only if something in that single combined run fails. A turn with no edits costs nothing, because the queue is empty and there's nothing to drain. A lock keeps two drains from overlapping, and an empty queue is its own stopping condition, so there's no path to the drain looping on itself.

One part of this was easy to get backwards: hook configuration is snapshotted at session start. Change the hook definition mid-session, and the session you're in keeps running on the old snapshot — the new behavior only takes effect starting the next session. That's worth flagging on its own, because it's exactly the kind of fix that looks broken if you test it in the same session where you wrote it.

What generalizes

If you've established a rule like "verification happens once per unit of work," the first thing to go looking for isn't more verification — it's the automation that's already quietly breaking that rule. A written rule changes what a person or a model does next. It does nothing to a hook, because a hook doesn't read rules; it reads trigger events, and it keeps firing on the event it was configured for regardless of what the team has since agreed the right cadence should be.

The practical corollary is about how you audit time and cost in an agent fleet at all: tool calls are not the whole cost surface. Hook output and background processes are part of it too, and a hook whose success path is silent means any total computed from logs is a floor, not a figure — bounded below by whichever failures happened to be loud enough to record themselves, and unbounded above by however many quiet successes never got the chance.

And the timing question generalizes past testing specifically. The right moment to verify a change is almost never "immediately after this one edit." It's "after this one unit of change is finished" — and knowing where that boundary sits isn't something an editor or a file-save event can know. Only whoever is actually authoring the change, and deciding when a turn is done, is in a position to say so.

Where this would turn out to be wrong

The falsifier here is specific enough to check for directly: in the next batch of work, if the end-of-turn drain still comes back red at roughly the same rate the old per-edit hook did — because of the same mid-edit, still-incomplete state — then the fix didn't actually solve the problem it looks like it solved. That result would mean the defect was never about when the check runs. It would mean the test files themselves are too coarse a unit, bundling too much unrelated behavior into one suite for any single trigger point to time correctly. At that point the next move isn't to move the hook again — it's to split the suite.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Five Automated Gates Passed Three PayPal and Stripe-vs-Square Fee Pages. One Human Reader Rejected All Three.

John — Thu, 10 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

I run a small pipeline that turns a shared brief into published pages, and it won't let a draft go live until it clears five separate automated checks. I recently used it to build three pages at once — a PayPal Goods-and-Services fee explainer, a general merchant-fee anatomy page, and a Stripe-vs-Square comparison — written in parallel by three separate workers from the same brief, each working against the same set of published rate pages from the providers themselves.

Every one of the three pages passed every one of the five gates on the first try — all five check one document at a time, and all five ran. warrant_gate confirmed that every number cited in the prose actually appears there. numeral_gate confirmed that every digit on the page traces back to a declared piece of evidence. expression_gate checked each page's prose against the sources it cited, for near-duplicate sentences. disclosure_gate and revenue_axis_gate passed too. On top of the five machine gates, each worker ran its own 7-item reader checklist before calling itself finished, and each reported 7 out of 7. The step that decides whether a draft is clean enough to put in front of a human exited clean — three times over.

Then one read-through of the actual rendered text — one script pulls all three pages down to roughly 5,500 words of running prose for a person to read in one sitting — rejected all three pages.

This isn't a gate that got lied to

It's worth being precise about what this wasn't. All five publishing gates had actually run, against real rendered text, checking real evidence entries, on each page on its own. Nothing here was a checklist item nobody executed, and no page asserted a number without a source for it. Every one of those five gates did exactly the job it was built to do, correctly. What got through lived in three narrower places instead: in the word standing next to a correctly sourced number, in what one page denied about a source its own sibling page was busy quoting, and — this is the one that stings — in a check that lives one layer downstream of those five, in the site builder rather than the publishing pipeline, whose entire job is comparing sibling pages against each other. For three pages each built on its own, that check never ran at all.

Three ways to be wrong while citing your sources correctly

1. The right number, the wrong owner

PayPal charges an additional 1.50% on what it calls an "international commercial transaction" — its term for a cross-border sale. Two of the three pages described that same 1.50% figure as triggering "when the buyer's card was issued outside the United States." That's a real trigger condition. It just belongs to Stripe and Square, not PayPal. The number was correct, it was sourced, and it was attached to the right provider's fee line — it was just carrying someone else's rule for when that fee applies.

A gate that checks "does this number have a citation" has no opinion on that sentence. The citation is real. The digit is real. The predicate wrapped around the digit was copied by pattern from a different company's rate page rather than re-read from the specific source sitting in front of the worker that used it.

2. A denial one page made that its own sibling disproved

The merchant-fee anatomy page stated that "neither PayPal nor Square publishes a keyed-in rate on the pages cited here." The Stripe-vs-Square page, built in the same run, quoted "3.5% + 15¢" for keyed-in cards — from the very same Square URL the first page had just declared silent on the subject.

Both pages were internally correct about their own citations. Both passed every gate that checks a page against its own sources. Set side by side, they flatly contradict each other, and nothing in the pipeline ever put them side by side while it still mattered.

3. Wording with nothing to check it against

The rest were smaller, but the same shape: "buyer protection," asserted with no evidence entry behind it, on a source page that never uses those words. "Each row below," describing a table that actually rendered above the sentence. "A flat 1.5% surcharge" — a percentage described with a word that means it doesn't scale. "Four tenths of a percentage point," a number spelled out in words in exactly the place no gate is looking for digits.

None of these needed a wrong citation. They needed no citation at all to be wrong — a non-numeric claim, a spatial reference, an adjective, a number written as words instead of digits. A pipeline built to check digits against sources has nothing to say about any of them, because none of them is a digit.

Why five green gates didn't add up to one correct page

Two structural gaps and one process failure, and the process failure is the one worth remembering.

warrant_gate and numeral_gate both check that a number is sourced. Neither reads the sentence wrapped around the number closely enough to know whose rule it's describing. "Sourced" and "attributed to the right actor" are different properties, and only the first one had a check.

A negative claim like "does not publish" has no number and no evidence cell to anchor to. A per-document gate can verify what a page asserts with a value. It has no mechanism for verifying what a page denies about a source it didn't fully quote — and even less way to know that a sibling page, built the same hour, was about to quote exactly that.

Catching the second gap at all would have needed something outside those five gates entirely. One layer downstream, in the site builder rather than the publishing pipeline, a separate check named NEAR_DUPLICATE_PAGE measures sentence overlap between sibling pages in the same build. It never ran for these three workers: each built its page alone, in its own single-page run, and a single-page run has no siblings to measure against, so the check is skipped — what the build reports for that situation is, in substance, "no sibling pages to compare". It first actually ran later, when the coordinator built the full site across nine pages, and passed. Every worker had still reported "gate PASS" for a check that, for them, had never once run.

The self-check missed it for a more human reason. The checklist item that should have caught the third defect class was worded as "no claim beyond the evidence," illustrated with a single example. None of the three workers generalized from that one example out to buyer protection, table position, or a spelled-out fraction. A checklist item is only ever as good as the example riding along with it.

What changed, and why it isn't a sixth gate

The fix went into the brief, not into the gate pipeline. Three new checklist items: each provider keeps its own trigger term, so a fee's condition gets re-read from that provider's own page instead of pattern-matched off a sibling's; never write that a source "does not publish" something — write that "this page does not compute" the number, which is a claim about what got fetched rather than a claim about what exists; and non-numeric claims need an evidence entry exactly the way numeric ones do, which also covers direction words, "flat," and spelled-out fractions. Separately, it's now written down that a single-page build skips the site builder's sibling-comparison check, and a report of "gate PASS" has to name which gates actually ran rather than asserting a clean run in general.

A sixth gate was the other option on the table, and it's worth explaining why it lost. Catching an unfalsifiable "does not publish" mechanically would mean fetching and fully indexing the entire text of every source cited anywhere in a batch, not just confirming that a cited number appears somewhere in it — a much bigger, slower, and more fragile system to maintain than the five gates already running. The brief rule removes the same class of defect for the cost of one sentence in a document a worker was already required to read.

What generalizes

The generic caution here is a familiar one: a passed check only tells you about whatever it actually looked at. What's worth keeping isn't that line by itself, but the specific shape it took across three independently gated documents. A fact can ride on a citation that is completely correct and still describe the wrong actor's rule, because "this number has a source" and "this number's condition belongs to the company it's attributed to" are different properties, and only the first one anywhere had a check. A claim can be false in a way nothing catches simply because nothing else in the batch was compared against it — until a sibling document happens to quote the very source it denied. And a check built to compare documents against each other is worthless the moment it runs on a batch of one, because from the outside, "ran and found nothing wrong" and "never ran" report themselves the same way.

None of that called for a sixth gate. It called for three sentences added to a brief three workers were already reading, and a rule that a report of "gate PASS" has to name what actually ran.

How this would be shown wrong

If the next batch, briefed with these three new items, still produces a mechanism-drift or an unfalsifiable-negative-claim defect, the brief wasn't the fix, and something more structural is needed — a per-vendor trigger-term table the export step enforces, say, rather than a paragraph a worker is trusted to have absorbed. And if someone builds a per-document gate that catches the negative-claim class at a reasonable cost, without fetching and indexing every cited source in full, then "this isn't a gate problem" was wrong for that class specifically, and it should become one.

What it cost

One reply per worker fixed everything in the next round. The real registration then passed all five gates three times, picked up the human signature, and the batch went out clean. Start to fixed, the whole detour cost one extra round-trip per page. That's cheap, as long as somebody actually reads the rendered result before the first clean run gets treated as done.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Five Fact-Check Gates and a Signature Passed a Page Whose Source URL Never Renders the Number

John — Wed, 09 Sep 2026 09:00:06 +0000

Originally published on hexisteme notes.

I run a small pipeline that drafts comparison pages and won't publish one until it clears five machine gates and I sign a fact-check sheet by hand. One page already live under that pipeline — a comparison of online card-payment fees across Stripe, PayPal, and Square — had cleared all five gates and already carried my signature. Then I ran the pipeline's dossier-export step against a real cluster and real fetching for the first time, and it rejected a citation on that already-signed, already-live page. The number in the citation was true. The page it pointed to had never rendered that number at all.

That distinction is worth holding onto, because it's tempting to read this as "the fact-check missed something" and stop there. The fact-check hadn't missed the fact — the underlying claim was correct, and the sheet recorded it correctly. What was wrong was the identity between the URL in the footnote and the text a browser actually shows when it opens that URL. Confirming that a claim is true and confirming that a specific page currently displays that claim feel like the same act once you've done the first one. They are not the same act, and nothing downstream of my signature had ever been built to do the second.

What a real fetch found that a fixture never would

The dossier-export step rejected three of the page's evidence entries outright the first time it ran against real data: their recorded quotes could not be found anywhere in the text the exporter actually fetched. One of the three was Square's entry for its international-card surcharge — an extra 1.5% on cards issued outside the United States — and the rejection reason was blunt: the quote itself was not on the page.

I opened the cited URL myself to see why. Square's fee table on that page renders client-side; a plain fetch of the page pulled back roughly ten thousand characters of navigation and FAQ text and nothing resembling a rate table. The number wasn't missing from the response, exactly — it was sitting inside the page's client-side hydration payload, the data blob a framework ships down so it can take over already-served HTML without a second round trip. The page's actual visible text ran about 3,300 characters. No shadow DOM held the number. No <template> element held it either, which would at least have been a recognized route to eventual rendering. It was present in what the server sent and absent from what a browser ever paints.

A second page on the very same site — a plain pricing page, not the one the footnote named — rendered the identical claim as ordinary static text: "An additional fee applies to payments made on credit or debit cards issued outside of the United States. 1.5%." Same fact, same number, sitting in the open. It just wasn't the page the citation pointed to.

Why five gates and a signature both said yes

None of this was a gap left open on purpose; it was a gap in what "checked" had ever meant here. The five machine gates I run check whether numbers in the body match the fact-check sheet, whether every claim in the sheet is grounded in an evidence entry, whether phrasing overlaps across pages, and whether revenue and disclosure wording follow the site's rules. Every one of those compares the drafted page against the sheet, or compares the sheet's own entries to each other. Not one of them re-opens the cited URL and asks whether the page, right now, contains what the sheet says it contains. Nothing in the five had a job that pointed outward.

I hadn't caught it by hand either, for the same underlying reason. Signing that fact-check sheet meant verifying the claim — does Square really charge more on foreign-issued cards. It does. It didn't mean following the footnote's URL and checking that the number was actually sitting there, visibly, on the page it named. Once a claim checks out as true, there's very little pull toward reopening its citation. Verifying that a source is real and verifying that the specific page you pointed to currently shows what you say it shows read, in the moment, like one task. They only turn out to be two the moment a framework's client-side rendering quietly pulls them apart, which is exactly what had happened here without anyone deciding it should.

Changing what gets compared

The fix changed the unit of comparison. Evidence used to carry a value — the number itself — checked against the drafted page's body text. Now every piece of evidence has to carry a quote, a verbatim excerpt a human can actually read, and that quote has to exist word-for-word inside whatever text got fetched from the citation's URL; the value then has to be a token drawn from inside its own quote, not a separately typed number that merely resembles it. A citation no longer passes because its number looks right. It passes because its quote is provably sitting in the page it names.

For a citation target that only renders its numbers through JavaScript, I didn't add a bypass flag. A flag that says "trust me, this one renders client-side" is a lock with no key on the other side — six months later, nobody can tell a legitimate exception from a stale one. Instead there's now an explicit option to supply the already-rendered text for a given URL directly, with the sheet recording what stood in for the live fetch: which text, its length, its hash. Whatever let a citation through stays reproducible after the fact instead of becoming someone's unverifiable say-so. For Square specifically I didn't even need that path — I repointed the footnote at the plain pricing page that already renders the number as static text. The corrected entry now reads plainly: source squareup.com/us/en/pricing, quote verification found in fetched text.

The second crack the first fix exposed

Requiring a verbatim quote-in-fetched-text match immediately broke a second citation on the same page that had been getting a free pass. PayPal's international fee, quoted at 1.50%, turned out to be two non-adjacent fragments of the source page joined with an ellipsis — someone had spliced together two separate sentences into what read like one continuous quote. Verbatim matching doesn't forgive that. The joined string, ellipsis included, exists nowhere in the fetched text.

The fix was narrower than banning ellipses outright, since a real quote sometimes does skip filler mid-sentence honestly. Each fragment on either side of an ellipsis now has to be at least fifteen characters long, has to appear verbatim in the fetched text, and the fragments have to appear in the source in the same order they appear in the quote. A splice stitching together two sentences from opposite ends of a page no longer has an ellipsis to hide behind.

Not overwriting a signed dossier

The number on the live page never changed — Square still charges exactly what the page always correctly said. What changed was which URL backs that number up, and this pipeline treats a citation swap as no smaller an edit than a change to the claim itself. I didn't patch the existing, signed dossier in place. I built a new one and ran it through export, registration, and judgment again, and the judge answered exactly the way it does for any new page: awaiting a human signature, mine, again, before it can replace what's live. There's no tier of edit small enough to skip that step.

The same day, a different corner, the same shape

A second failure surfaced that same day, in a different part of this pipeline, with nothing to do with citations — except that it was the identical failure wearing different clothes. A worker process had reported the export pipeline's exit clean, but that green result had come from a fixture cluster and a stand-in fetcher, not the real ones. Pointed at the path the tool is actually meant to run on — a real cluster, real fetching turned on — the same pipeline failed immediately, rejecting on its very first check: the real cluster had none of the revenue-axis data, source licenses, or evidence values that the fixture had always quietly supplied.

That worker's green light had been real. It just hadn't been a green light about the thing anyone actually needed it to be true of. A passed check only tells you about whatever it actually looked at, and if what it looked at was a stand-in, the pass is a statement about the stand-in. Structurally, that's the same defect as a citation whose quote had only ever been checked against the sheet that wrote it, never against the page it names — a comparison dressed up as verification that never touched the thing it claimed to verify.

What generalizes

If a check exists to verify a citation, it's worth naming, explicitly, which question it actually answers: is it comparing the source string against itself, or against what the target currently, actually renders? A check that only does the former will wave through a citation whose number is completely true and whose pointer is completely wrong, and it will do it every time, because truth and pointer accuracy are independent facts that happen to usually travel together.

Second, a green result is only as trustworthy as the path that produced it. Before treating a pass as reassurance, it's worth confirming it came from the real path — real data, a real fetch — rather than a fixture standing in for one.

Third, when a citation target only renders through client-side code, the fix isn't a flag that waives the check. It's an explicit, recorded substitute for the fetch, detailed enough to reproduce the decision later, so that whatever let a citation through stays checkable instead of becoming a permanent, silent exception.

Where this would be overkill

I wouldn't defend this at every scale. If a set of citation targets is served as plain static pages that change rarely, the URL and its rendered text are going to agree almost all the time, and forcing a fresh quote-in-fetched-text comparison on every citation could easily cost more in fetches and friction than the defects it would ever catch. This case produced a real defect specifically because the target mixed client-side hydration and static rendering across pages on the same site — a common pattern, and an inconsistent one, which is exactly the combination that lets a citation be true and its pointer be wrong at the same time without anything about the page looking obviously broken.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

A Detector That Never Fires Scores Perfect on False Positives

John — Wed, 09 Sep 2026 00:00:05 +0000

Originally published on hexisteme notes.

Two rework incidents this week had the identical shape: a label chip rendered on top of the exact thing the episode was about. In one episode, chile, a ruler's label chip covered the ruler's own body and tick marks — the episode's conclusion rested on comparing the length of two rulers, and the horizontal one wasn't visible anywhere in the frame. In another, siliguri, a scene label chip sat above a narrow corridor of Indian territory and severed it, and that corridor was the episode's subject. Both episodes rendered without error. Schema validation passed. Compliance gates passed. Regression tests passed. The defect was visible on the screen and nowhere else.

This is the fourth attempt at an automatic detector for it, and the first three failed the same way. I've written about checks that go quiet before, each a different failure: a probe whose own prep procedure made a pass the only possible outcome, no matter what had happened to the file under test; a threshold written as an absolute constant, tuned to fixtures at one scale, silently disabled at another; a checker whose zero, four separate times in one audit, meant "I didn't look there" rather than "nothing is there"; a check whose "zero overlaps" was right about a narrower question than the one I was asking. This one is none of those four. Every attempt below ran, at full aperture, with a live path to failure — and still couldn't tell a broken frame from a clean one. What was missing wasn't a working check. It was a test for whether a check was working at all.

Three failures, one blind spot

I built and discarded three separate detectors before this one, and all three failed for the same underlying reason.

The first checked a safe area: the four edge extremes of whatever sat at the frame's border. Both defects sit dead center, so the extremes never moved. For siliguri, the broken frame and the repaired frame produced identical edge extremes.

The second counted a "focus-color run" around the label, tried under six different definitions of what a run is. Under every one, the real defect scored no higher than a clean episode: 26 against 27 under one definition, tied at 28 apiece under another, next to lesotho, a normal episode. All six definitions failed the same way.

The third isolated the amber-colored label blob down to just its glyph. It separated seven samples cleanly. Checked retroactively against 27 published episodes, 10 of the 11 frames it flagged were false positives — 91%.

All three inferred geometry from the color of an already-rendered PNG. The renderer, meanwhile, knew exactly what it had drawn and where, and threw that information away the moment the pixels hit disk. All three failures pointed at the same missing thing.

Teaching the renderer to keep a receipt

The fix wasn't a fourth color heuristic. I made the renderer log its own output. src/shorts_factory/render_receipt.py tags each matplotlib artist with set_gid() to record its role, then reads back the measured bounding box with get_window_extent(renderer) right after savefig and writes it to render_geometry.json.

Two properties made this cheap to add. set_gid is read only by the SVG backend — the Agg backend that rasterizes the frame ignores it — so tagging is provably pixel-neutral: a test, test_tagging_is_pixel_neutral_for_map_frames, monkeypatches the tag call to a no-op and confirms the rendered sha256 still matches. And savefig has already done the drawing by the time the receipt gets collected, so reading the extent back costs nothing extra.

The receipt only records the occluding side — chips, lines, cards. Map polygons don't get logged; one country can be tens of multipolygons across roughly a thousand frames, and the receipt would end up bigger than the video it documents. Instead it records camera state — camera_bbox, axes_rect, highlight, geo — precise enough to reproject the occluded side offline, since the projection is equirectangular — one linear formula.

Overlap stopped being a heuristic at that point. It became arithmetic: does one bounding box intersect another.

Two traps that would have been quiet false negatives

Two matplotlib behaviors nearly undid this, both failing the same direction: evidence quietly missing, downstream code reading the absence as "no overlap."

A Collection artist's get_window_extent returns (inf, inf, -inf, -inf) even after drawing, confirmed on matplotlib 3.11.1. Left alone, ruler_ticks (a LineCollection) and density_dots (a scatter) would silently drop out of every receipt — the exact false negative this module exists to prevent. The fix: get_datalim(ax.transData), both corners transformed through transData, padded by marker radius (√s / 2, s in points squared).

Axes built with ax.inset_axes() don't show up in fig.axes — they live in ax.child_axes. A Monte Carlo minimap's tagged elements were structurally uncollectable as a result: only 3 of its 4 sim_particles ever made it into a receipt. The fix makes the axis-walking function, _iter_axes, recurse into child_axes.

I pinned both down with regression tests.

Measuring before wiring anything

The repository runs on a standing rule: measure retroactive false positives against published output before wiring anything into a gate — the rule that caught the third attempt's 91% false-positive rate, and the same discipline applied here.

The instrument, scripts/measure_render_occlusion.py, defaults to exit code 0 — a verdict threshold activates only when explicitly requested — and it ran against all 25 published episodes.

Re-rendering all 25 for that measurement would take 2.7 hours. Instead I built a probe mode, geometry_probe_step: it samples every N frames and writes nothing at all — no frame PNGs, no render_events.json, no receipt. Read-only is structural here, not a promise, so it can't touch published output no matter what it finds. Sampling every 1.0 seconds across 25 episodes took 12 minutes; a directory listing taken before and after confirmed nothing on disk had moved.

Two predicates, one split

Two detectors came out of the receipt data, and I didn't treat them the same way.

instrument_swallowed — the chile-type defect — fires when a tool line (ruler body, tick marks, leader line, path line, arrow) sits entirely inside an opaque chip with higher z-order. I didn't invent a threshold; it only counts a containment ratio of exactly 1.0.

Axis	Result
Specificity (retroactive false positives)	0 across 25 published episodes
Sensitivity (true defect)	Disabling the fix (`_ruler_label_offset_pt`) reproduces the defect episode: 8 firings (t=17.5s, `ruler_ticks` fully contained in `ruler_label` — "101 km" — covered=1.0). The repaired version: 0

I wired that into compliance_gate.check_instrument_not_swallowed.

subject_severed — the siliguri-type defect — measures the longest contiguous run of pixels in a row where an overlay chip cuts across the highlighted territory.

Chip role	Episodes it fires on / 25	Max run
`judgment_card`	5	144px
`scene_label` (same role as the actual defect)	13	56px
`ruler_label`	4	40px
`marker_label`	4	36px

It fires on 18 of the 25 episodes. The top-level judgment_card is an outro card designed to cover the map, so its hits aren't defects — the count is real, the alarm isn't. Narrow to scene_label, the same role the actual defect used, and the top three episodes by run length are southamerica-east, lesotho, and russia-nk. All three are normal episodes, tied with the corridor defect itself at 56 pixels. There's no threshold that sits between them.

I didn't wire that one anywhere. It stayed an instrument, not a gate.

The only test that tells a detector apart from a stub

Here's the part that generalizes past matplotlib and video frames. instrument_swallowed's "0 false positives across 25 published episodes" is not, on its own, distinguishable from a function that always returns an empty list. Both produce the identical report: zero findings, every time. The only way to tell them apart is to check whether the detector can fire at all — hold out a sample labeled known-bad, not just a pile of presumed known-good ones, and confirm it fires.

That's what the sensitivity row above really is. _ruler_label_offset_pt is the line of code that fixes the chile defect, so turning it off reproduces the defect exactly, on demand: eight firings with the fix removed, zero with it back in place. instrument_swallowed wasn't verified by its clean run against 25 real episodes. It was verified by being made to fail on purpose, and doing so.

Zero false positives is not evidence that a detector works. It's the score a detector that never fires gets for free. The only way to tell the two apart is a positive control: hold out a known-bad sample and confirm the detector actually fires on it.

subject_severed never earned that kind of confirmation. The wall it hit is the same one the second, color-based attempt hit: whether a chip cutting 56 pixels off a territory is a defect depends on what the episode is arguing, not on the pixels. The identical run length is a defect in siliguri and completely normal in three others. Switching coordinate systems from color to exact geometry fixed instrument_swallowed's precision problem — the false-positive rate that sank the third attempt. It did nothing for subject_severed, because what an episode is trying to say was never a precision problem.

An exemption rule worth keeping

One design choice here is worth keeping. The gate guarding instrument_swallowed passes automatically when render_geometry.json doesn't exist — renders made before the receipt system existed are supposed to lack one. That contrasts with a hardcoded list of episode names the same repository uses to exempt the editor_note evidence system, written the same day: a list has to be maintained by hand and can be forgotten after a snapshot, while re-rendering an old episode here turns the check back on with nothing to remember. A list is still the right call sometimes — file presence alone can't always tell missing evidence apart from the defect itself. Either way, the reason has to live in the code, not in memory.

What's still unsolved

siliguri-type defects still aren't caught automatically. The current evidence guarantees a chip cut 56 pixels off a territory; it says nothing about whether those pixels were the point of the episode. That claim exists — in script.yaml's camera_bbox and emphasis fields — but it never makes it into the rendered pixels or the receipt describing them. Closing that gap means putting what the script claims into the evidence itself — schema work, not instrument work, and it's next.

Both predicates came from the same receipt data, the same week, checked against the same 25 episodes. One runs in the gate today. The other is still a script I run by hand, because it never earned the right to run unattended — and the only reason I know the difference is that I asked both to prove it, not just report it.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

The Renderer Dropped What the Script Ordered. Every Check Watched the Script.

John — Tue, 08 Sep 2026 09:00:05 +0000

Originally published on hexisteme notes.

I run a renderer that turns geographic data into short map-animation episodes — country polygons filled in over a basemap, driven by a script that lists, per scene, which countries to focus on, which are neighbors, and which get highlighted. On a single day, it produced two defects, and both had the exact same shape: the renderer silently dropped part of what the script had ordered it to draw. Both had passed everything I had running at the time.

Two ways to drop what was ordered

The interior ring that got dropped

The code that draws a country's polygon was only using the outer boundary:

exterior = polygon[0]  # simplification: ignore interior holes

South Africa's polygon carries a real interior ring at Lesotho's location — an actual hole, because Lesotho sits entirely inside South African territory. Drop that ring and South Africa paints as one solid shape. Countries draw in alphabetical order, so Lesotho — drawn first — got painted, and then South Africa, drawn afterward with no hole cut into it, covered it over. The episode this happened in is titled "A COUNTRY INSIDE A COUNTRY." The subject of the episode had disappeared from its own frame.

The countries that only glowed

The code that decides which countries get a filled polygon at all was built from two of the scene's three country lists:

target_codes = set([*scene.focus, *scene.neighbors])   # no highlight

The glow effect around a highlighted country is generated from the separate highlight field, so a country that's only in highlight — not in focus or neighbors — never enters target_codes. It gets the glow. It gets no polygon. One episode's label was "Three, side by side"; only one of the three countries actually had a filled shape, the other two were a blur of red light around nothing. In a different episode, the same gap meant all five highlighted countries came out that way.

Three checks were green. None of them looked at the picture.

At the time, this renderer had three separate verification passes, and all three were green on both episodes.

Check	What it actually asserts
Schema validation	The input is well-formed — country codes exist, fields are populated
Compliance gate	Policy was followed — sourcing, AI disclosure, label conventions
501 unit tests	The renderer's internal behavior matches expectations — patch counts, the color at a given coordinate, label-placement rules

Not one of the three asks what the rendered artifact is actually saying. All three either look at the input or at an intermediate structure the renderer builds on the way to a frame. So "the label claims three countries and the picture shows one" got produced, cleanly, underneath three green signals — because none of the three signals was ever pointed at the thing that was wrong.

The structural reason input checks can't see this

This is worth stating plainly, because it isn't a case of the checks being lazy or thin — it's structural. In both bugs, the input was exactly correct. The script ordered the right countries; scene.focus, scene.neighbors, and scene.highlight all had the right values; the country codes existed; the fields validated. What failed was delivery: code that ran precisely as written and, by design, dropped something on the way to the frame. exterior = polygon[0] is not a defect a schema can see, because the schema was never told a polygon might carry more than one ring. A discard bug like this is invisible to an input assertion by construction — there is no wrong input to catch, because the input was never wrong.

I've written before that you should verify the surface the consumer actually sees instead of the payload you built — that's the general principle, and this is a sharp case of exactly why the substitution fails: the input check wasn't weaker evidence, it was evidence about a different fact. A separate note covers a renderer measured against the wrong source geometry entirely, upstream of rendering — a different failure from a mid-render discard, where the source was right and the render dropped part of it anyway. And another one is about an agent's inability to perceive its own render at all; this isn't a perception problem, it's a check aimed at the wrong target from the start.

Looking at the frame doesn't fix it either

I opened the frame and saw a black shape sitting inside South Africa's outline — easy to look at and conclude "there's the hole" and move on.

Back-projecting those pixels through the camera transform put them at longitude 30.8–32.2, latitude −27.4 to −25.6: Eswatini, not Lesotho. Eswatini borders Mozambique, so it was never an enclosed country to begin with — whatever put a dark shape there was unrelated to the ring-dropping bug. Looking at the frame produced a confident answer. It just wasn't the right one.

The first attempt to check this programmatically was also wrong. matplotlib's Path.contains_point returns True for a point inside an interior ring, same as for a point inside the fill — true as of matplotlib 3.11, and make_compound_path doesn't change that. Ask the geometry object whether a point is "inside the polygon," and a hole doesn't register as an exception to that; it's still inside the outer boundary.

What actually answered the question was connectivity, not color and not the eye: flood-fill from the frame's outer border, then count the pixels that fill can never reach.

Before the fix: 0 enclosed pixels
After the fix: 3,900 px, x644-731 / y878-955 — matches the back-projected location of Lesotho

The rule

"Look at the output" is not a verification method. "Ask the output the same question the claim is making" is.

When an artifact is something a person has to interpret — an image, a PDF, a chart, a rendered document, an audio track — a test that inspects intermediate structure cannot, by construction, catch that the artifact stopped saying what it was supposed to say. Those tests are all answering "did the pipeline run the way I coded it." The question that actually matters is "does the artifact say what the claim says," and the two questions do not overlap. A pipeline can run exactly as coded and still discard part of the order, because "ran as coded" was never a promise that nothing gets dropped.

Three rules came out of fixing this:

Write one question per claim, and take the question from the claim, not the code. "Is Lesotho a hole?" becomes: flood-fill from the frame border, count the enclosed components. "Are three countries highlighted?" becomes: count the connected components in the highlight color. The question has to come from what the episode is claiming, because the code will happily tell you it's doing exactly what it's doing.
Answer the question in the same medium as the artifact. An image's claim gets asked of pixels. A sound's claim gets asked of the waveform. Ask a data structure instead and you get the data structure's answer — contains_point will tell you a hole is "inside," because that's a true fact about the geometry object and a false one about the picture.
Don't use what you saw with your eyes as evidence. A human eye is reliable up to "something is there." It is not reliable for "that is what it is" — mine confirmed a dark shape and had nothing to say about whether it was the hole the bug produced or something else entirely. It only becomes evidence once you can back-project it to coordinates or count it into a named, located component.

A side effect: the question is also a blast-radius scanner

Once a claim becomes a precise, mechanical question, it can be run against everything you've already shipped, not just the thing in front of you. The second defect's condition writes exactly as highlight ⊄ focus ∪ neighbors — a highlighted country the focus/neighbor set doesn't already cover. Running that condition against the ten episodes already published took about a minute and confirmed none of them triggered it — the condition happened not to be satisfied anywhere in what was already out the door, so nothing already published needed a fix. Turning a claim into a question about the artifact doesn't just catch the next instance of the bug. It tells you, mechanically, how far the last one reached.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

The Checkpoint Remembered the Result, Not the Request

John — Tue, 08 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

A few days before this, I'd fixed a bug in the same YouTube upload stage of my pipeline: the local record marking an episode as uploaded was written only after verification succeeded, so a verification failure left the video live on YouTube with nothing on disk that knew about it — and re-running the command would have uploaded it a second time. I wrote about that separately in The Upload Succeeded, the Record Did Not; the fix was to move the checkpoint earlier, writing the video ID to disk the instant it came back — marked verified: false — so a resume would finish the verification instead of re-uploading.

That design was correct. It also got exercised for real, on a live publish run — and the resume failed anyway.

The incident

I was publishing five episodes as public. One of them failed verification. The cause was mundane: right after upload, re-querying the video through videos.list hits eventual consistency — snippet.tags can read back empty for a moment — and it didn't propagate within the retry budget (5 attempts at 5 seconds each, 25 seconds total). This is exactly the situation the earlier fix was built for. The checkpoint was written as designed:

{ "video_id": "fzxvS2S3I7I", "privacy_status": "public", "verified": false }

I re-ran the command to resume. I didn't pass the visibility flag this time — it's a resume, why would I need to:

$ agent-youtube upload --episode EP-...-bolivia-navy
Upload verification failed (downgrade detected): requested privacyStatus='unlisted' but actual is 'public'.

"Downgrade detected" is about the most serious alarm this stage can raise — it means the platform silently overrode what I asked for and made the video more private than intended. Except nothing had happened. The video was exactly public, exactly as requested. The alarm was wrong.

A false alarm, not a real one

The resume path had re-derived what "correct" means from the current moment instead of from the checkpoint. The requested value — public — was sitting right there in upload.json. The code didn't read it. It recomputed the expected value from the current CLI flags and the config.yaml default, which is unlisted. The first run had passed --status public; the resume run hadn't, because a resume isn't supposed to need it. Same video, same code, a different yardstick.

Why this one was hard to catch

First, the symptom doesn't look different from a real alarm. "Downgrade detected" is the heaviest signal this stage can raise, and a false positive breaks two things at once: it blocks a perfectly good resume, and it trains you to shrug off the next real downgrade as "probably that same false alarm again." An alarm that cries wolf is worse than no alarm at all.

Second, it can't reproduce on the first run. When the request and the verification live inside the same process, they read the same variables — there's nothing to diverge. The mismatch only exists on the resume path, and the resume path only runs after something else has already failed. This bug needs a different bug to fire first before it can even show up.

Third, the tests were green. There was a test for exactly this case — if a checkpoint exists, does re-running resume verification instead of re-uploading? That test called the first and second run with the same config. When the config doesn't change, it doesn't matter where the yardstick comes from; the checkpoint and "now" already agree, so the result is identical either way. The test confirmed that resume runs. It never asked what resume uses to judge.

The fix: pin the request, not just the result

On resume, the verification target now gets overwritten with what the checkpoint recorded — privacy_status, title, tags, category_id, made_for_kids, video_language — instead of being re-derived from whatever the CLI flags and config happen to say today.

The regression test had to be built to deliberately disagree with itself: upload as public to force the original failure, then resume with no flags at all — meaning the config default of unlisted is now in play — and it still has to pass. A test that hands the same config to both runs is structurally incapable of catching this, so I pinned the mismatch into the test itself: assert config["default_status"] != "public". I also removed the fix to confirm the test goes red without it.

What generalizes

A checkpoint has to hold not just what happened, but what was asked for.

The earlier fix established that the moment you receive an identifier is the moment you record it. This is the layer on top of that one: a record holding only the identifier can't later tell you whether the thing it points at is correct, because correctness is only ever defined relative to the request. A result-only checkpoint makes resume half-possible — you know what got built, not whether it's what you meant to build.

The moment a resume path goes back to current config to fill that gap, it starts judging a past result by today's rules. Anything that shifts in between — a flag you didn't think to repeat, a config default someone changed, a deploy that landed while the job was sitting there — turns a perfectly fine piece of work into a reported failure. The gap between two runs isn't a precondition your resume logic gets to assume away. It's a variable.

The same shape shows up anywhere a checkpoint exists:

A payment idempotency key that stores only the key, not the amount or currency, ends up comparing a retry against whatever the cart holds now. If the cart changed in between, the comparison means nothing.
Infra provisioning that records only the instance ID, not the requested spec, has its resume report "drift" against whatever the current Terraform files say — not against what was actually asked for the first time.
A deploy retry that never recorded the target version re-reads whatever HEAD happens to be when it wakes back up.

One question covers all of it: of the values this resume path reads, which ones could have changed since the first attempt? Anything that can belongs in the checkpoint, not in "look it up again." And the only way to test for it is to deliberately make the first attempt and the resume disagree on purpose — a test that calls both with identical conditions is structurally blind to this exact defect.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

The Confidence Interval Was [0, 0]. That Was Not Precision.

John — Mon, 07 Sep 2026 09:00:04 +0000

Originally published on hexisteme notes.

This is the third note I've written about the same measurement pipeline turning up a bootstrap confidence interval of exactly [0, 0] that doesn't mean what it looks like. What's actually new this time: the pipeline has caught up with the first two notes. It now names this exact failure mode in its own documentation — and its most recent weekly snapshot still shipped thirty of these intervals anyway.

The pipeline is PAMSL, the same one behind Your Agent Telemetry Ranks Your Routing Policy, Not Your Models and Sub-Agent Metrics Are Not Comparable to Main-Thread Metrics. The snapshot here is dated 2026-08-31 — seven model epochs crossed with two roles, association-only per its own header, graded no higher than "C (exploratory)." Nothing below ranks a model. It's about what the report's Bootstrap 95% CI table still does, next to what the report's own footnotes now say about it.

Thirty rows out of ninety-five

The Bootstrap 95% CI table compares model-epoch × role cells pairwise, restricted to pairs where both sides clear n=20, resampled 2,000 times from a fixed seed (42), no p-values anywhere by design. A pair filter compares only within the same role — main against main, sidechain against sidechain — and skips any pair where both sides are already flat: median 0, interquartile range 0 to 0.

The two Stop hooks behind this note are on GitHub under MIT: hexisteme/hard-gate-hooks. They ship with their tests and a read-only scanner that prints what they did on **your* machine, not mine — including the case where it tells you they aren't worth wiring up yet. No email, no signup.*

I counted the rows that survive it. Ninety-five, across six metrics. Thirty of them — just under a third — report a 95% CI of exactly [0, 0]: 12 of 22 tool_error_rate rows, 6 of 10 validation_run_count rows, 12 of 22 error_recovery_seq_count rows. None of same_file_reedit_rate, output_tokens_total, or completion_proxy land on a point.

Three concrete ones:

tool_error_rate, claude-haiku-4-5-20251001/base×sidechain vs claude-opus-4-8/base×sidechain: Δ = 0, CI = [0, 0]. Haiku sidechain: n=267, median 0, IQR 0–0.0526. Opus-4-8 sidechain: n=2781, median 0, IQR 0–0. The intuitive case — both medians sit at zero, and one side's IQR is a literal point.
validation_run_count, claude-opus-4-8/base×main vs claude-sonnet-5/base×main: Δ = 0, CI = [0, 0]. The first group: n=211, median 0, IQR 0–4, mean 3.9573. The second: n=61, median 0, IQR 0–0, mean 0.3607. One group runs validation an average of almost four times per thread, with real spread up to its third quartile — and the interval on the difference of medians still reads as a single point.
error_recovery_seq_count, claude-sonnet-4-6/base×sidechain vs claude-sonnet-5/base×sidechain: Δ = 0, CI = [0, 0]. The second group: n=3478, median 0, IQR 0–1, mean 0.4963 — real dispersion, not a constant — and the paired CI is still a point.

A guard I've already written up twice

Look at the last two examples again. In each pair, one side is exactly flat — claude-sonnet-5/base×main has median 0 and IQR 0–0; claude-sonnet-4-6/base×sidechain has median 0 and IQR 0–0 — and the other side is not, with IQR reaching 4 and 1 respectively and means that are clearly nonzero. The filter's skip rule needs both sides flat to fire. One flat side isn't enough, so neither pair gets skipped. They still collapse to [0, 0], because the non-flat side's median is 0 too: at least half of its own values are exactly zero, which is a narrower condition than "the whole IQR is a point," and it's the one the bootstrap actually runs on.

I've derived this mechanism in detail twice before, on this same pipeline. In July I named it directly: a [0, 0] interval is "tie-degeneracy: so many identical zeros that every resample returns zero". Ten days later, working through this same metric — validation_run_count, means then ranging 0.60 to 4.11 across model pairs, much like the 0.36-to-3.96 pair above — I found the pipeline's existing guard sailing a pair through and degenerating anyway, and called the guard "necessary and insufficient". That's not a new finding here. It's the same mechanism, five weeks later, still doing the same thing to the same metric. An interval that cannot move isn't precise. It's just not moving.

Documented is not fixed

Between the second of those two notes and this snapshot, the pipeline changed — but not by fixing the thing I flagged.

The report's Coverage & Limitations section now carries item F14: for the two binary proxy metrics, completion_proxy and abandonment_proxy, a median-difference CI is explicitly forbidden and replaced with a proportion-difference CI, on the arithmetic grounds that a 0/1 metric's median difference can only ever be -1, 0, or 1. (That it's not much of an interval to build a statistic on is my gloss, not the report's wording.) That is a real fix — narrower than what I'd called for in July, which was checking whether the resampled statistic itself is constant, for any metric. F14 patches the binary metrics specifically. It doesn't reach tool_error_rate, validation_run_count, or error_recovery_seq_count, which is exactly where all thirty zero-width rows above come from.

It also carries item F15, translated from the original: "many cells are nearly constant (mostly 0), so the percentile bootstrap CI collapses to [0, 0] — this is not evidence of precision, but a coverage limitation of the median bootstrap under heavy ties." That is not a fix. It is a sentence in a Limitations section, naming a failure mode, attached to no specific row. The same document that carries it still prints all thirty zero-width rows above with nothing in the table itself distinguishing "genuinely nothing here" from "tie-degenerate, see the footnote."

The self-measurement problem has the same shape. The second of the two July notes found the pipeline measuring its own build-and-audit sessions and promised "the next iteration gets an explicit exclusion stratum." Five weeks later, this snapshot's own Limitations section still marks that stratum as planned for M5 — flagged, not built.

None of this is a complaint that the report is dishonest. It is unusually candid: this section alone also names same_file_reedit_rate conflating iterative editing with rework, a completion_proxy heuristic broken by the exact harness-logging convention I unpacked in the first of these two notes, 37 main and 790 sidechain threads dropped from every comparison for missing model attribution, and historical dispatch-policy versions folded into an explicit UNKNOWN stratum rather than guessed at. Candor at the bottom of a document and a clean number in the middle of a table are two different deliverables. A reader only ever sees one of them without going looking for the other.

What to do the next time you see [0, 0]

Don't read a zero-width CI as "checked, no difference." Check the same cell's IQR and mean first — they sit one column over in the Core Metrics table and cost nothing to look up. If the IQR is a point on both sides, the interval is telling the truth. If the IQR or mean shows real spread on either side while the CI still reads [0, 0] — the validation_run_count case above, median 0 next to a mean of 3.9573 — the interval isn't confirming anything. It's tie-degenerate, and this report, at least, will tell you so if you scroll to the bottom and go looking.

That last part is the part worth fixing, if you own the pipeline rather than just reading its output. A limitation that lives only in prose at the end of the document protects a reader who already knows to look for it, and nobody else. The distance between F15 existing and F15 doing something is the distance between a footnote and a flag on the row: a tied-value share printed next to every median-based interval, or a suppressed cell instead of a manufactured zero — the same gap between citing a number correctly and reading it right, just moved from a model's output to a pipeline's own table. Naming a failure mode is a start. It is not the same work as making the specific row that failed impossible to misread.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

The Overlap Check Said Zero Overlaps on the Same Clip Three Times. Each Time It Answered the Wrong Question.

John — Mon, 07 Sep 2026 00:00:14 +0000

Originally published on hexisteme notes.

A check can be entirely correct and still miss what it was built to catch, if it's answering a narrower question than the one you think you're asking. I ran into this three times, on the same check, on the same video-rendering pipeline, before I stopped trusting a pass and started asking what population it had actually looked at.

The check runs before a clip ships: it scans finished frames for text overlapping other text and reports how many timestamps collide. Three times, across two clips, it reported zero while text was visibly, unreadably overlapping on screen. The check never lied. The closest name I already had for this shape, verification tools don't report their blind spots, undersells it: the tool printed an accurate number that was still the wrong thing to trust.

Zero, the first time: a layer the check never draws

The first report came back on a freshly rendered clip: zero overlapping timestamps. The frames told a different story — a dollar-value label sat exactly on top of a caption burned into the video, unreadable. Three figures in the clip were hidden the same way — $4.404 billion, $4.147 billion, $23.769 billion — each under the same caption band the moment it appeared.

The check compares text objects the rendering scene itself draws. The caption isn't one of those: it gets burned in during a separate pass, after the scene renders, and never exists in the scene's own coordinate space. Zero was true — nothing the scene drew overlapped anything else the scene drew. Rendering the scene alone hides this too — the caption band is empty in isolation — and the defect only exists once both pieces are assembled.

I didn't estimate where the caption sat. I pixel-diffed matching frames from the finished video against the pre-caption render, at three timestamps: a one-line caption occupies a band from about y = -4.178 to y = -3.585. The old placement for the value label sat almost exactly in the middle of it.

The fix moved the label into a gap between two other fixed elements — the top of a bar and a tick label above it — space nothing else could structurally reach. I hardcoded the boundary as a named constant, BURNED_CAPTION_TOP_Y, commented as measured rather than assumed, and asserted against it. Nothing here replaced opening a rendered frame and looking.

Zero, the second time: the worst case is the one it can't see

Same check, same clip, right after that fix — zero again, and found only after I'd already called the first one done.

Frames near the midpoint showed two adjacent year labels ghosting into each other — one year's numbers still visible while the next faded in, both readable at once, so neither was. Every transition point did the same, adding up to roughly a fifth of the clip's runtime — about 9.4 of 47.4 seconds.

This time the blind spot was a pairing, not a missing layer. The check's definition of "overlap" was a white pixel cluster intersecting an amber cluster — two named categories. The ghosting was white text fading into white text: the same category transitioning into itself, a combination the definition never included. Worse, during the fade the two texts sit at the exact same coordinates, and a bounding-box check doesn't see two boxes in an identical position as an intersection — it sees one box. A slight misalignment would have registered; a perfect, total overlap did not.

My first replacement instrument was wrong, caught only because I widened the positive control before trusting it. Ghosting is text rendered semi-transparent, so I measured the fraction of "mid-tone" pixels. An old, ghosting frame came back 88.8% mid-tone; a hard-swapped frame came back 29.0% — a 3x gap. Widening the "known normal" sample broke it: frames unambiguously fine by eye scored anywhere from 29.4% to 90.0%. The metric was measuring color, not transparency — red lettering elsewhere in the clip has a luminance around 112, dead center of the 60–195 range I'd called "mid-tone," so any red text scored as ghosting.

The working instrument measured the definition instead of a proxy: ghosting is a region that changes gradually across more than one frame; a legitimate swap is a one-frame event. Independent of color, I measured the length of continuous frame-to-frame change:

version	change events	longest run	largest single-frame jump
old (crossfade)	34	19 frames	6.8
new (hard swap)	18	1 frame	31.3

19 and 1 land on opposite sides of any threshold. The largest jump is lower on the broken clip, backwards from expectation, because a crossfade spreads change thinly instead of concentrating it.

The fix: text never crossfades, only bars animate. All 7 crossfade calls became an outright swap — remove old text, add new — with freed time absorbed by an equal-length wait so clip length held: 47.36s to 47.33s, against a narration track needing 45.99s, with 1.34s to spare.

Zero, the third time: an instrument stalls, so read the source instead

Later the same day, I pointed the frame-change-length instrument from case two at a different clip I believed already repaired. It came back with a longest run of 23 frames — worse than the number that first flagged a problem. A repair can't make a measurement worse, so either the fix hadn't landed, or the instrument was wrong.

It was the instrument. The 23 frames were two legitimate entrance animations — a panel and a callout fading onto the screen, which changes a text region gradually for a different, correct reason. A discriminator for that — content at both ends of a change, versus coming up from nothing — brought the number to 7. Opening those frames too found a second legitimate cause: a background panel's opacity ramping up behind static text.

Three discriminators, three failures to separate the classes:

discriminator	on the real defect	on the false positive	separates them?
lowest ink level during the change	0.94	0.96 / 1.00	no — ranges overlap
largest single frame-to-frame jump	low	low	no — inverted, same surprise as case two
mid-tone pixel fraction	—	—	no — the same color-proxy that already failed once, reused

The third failure repeated a mistake I'd already made and written down earlier in the same investigation. Three discriminators landing on the same overlap is evidence the channel lacks the information, not that a fourth would work.

So I changed layers and read the source that generated the frames. One line explained everything the pixels had been ambiguous about: a call whose entire purpose is to superimpose two text objects at the same position while one fades out and the other in. Ambiguous in pixels, unambiguous in one line of source.

It wasn't isolated — it was documented as the correct way to change an on-screen number, in the header comment of three separate scene files, for a real reason: the objects it replaced default to a class that shells out to a LaTeX binary and throws a hard error without it. But "there's no LaTeX" and "so cross-fade the text" got written as one instruction and copied as if the second followed from the first. It doesn't — swapping the old object out and the new one in, no fade, satisfies the same constraint without ever putting two objects on screen at once.

The impact wasn't theoretical: a clip that had already shipped had the identical pattern, unreadable for about half a second, confirmed by opening the frame.

I fixed all 12 call sites carrying the convention and re-rendered. Then, instead of trusting the improved number, I looked at the same segment again. The numbers no longer overlapped each other. One label was still sitting on a caption underneath it — nothing to do with any crossfade.

The second cause was separate: a coordinate named HERO_Y that part of the scene converges toward was exactly where an unrelated label stood. An earlier change had made that label persist for the clip's full length instead of disappearing, conflicting with everything sharing its coordinate. I'd already patched that conflict once, in one beat; a second beat using the same coordinate had never been touched. The scene was already calling the overlap-assertion helpers in eight other places — and not in this one. That is the more dangerous shape: a list of call sites long enough that scanning it reads as "this scene checks for overlap," so nobody counts the gaps.

Measuring the same segment across all three states made the fix legible:

version	longest change-run	clip length
originally shipped	19 frames	1461 frames
crossfades fixed	8 frames	1461 frames
+ coordinate collision fixed	4 frames	1461 frames

Clip length never moved, because the swap consumed no animation time and the collision fix layered a fade onto an existing animation instead of extending it.

The repair tool had the same defect it was written to remove

One more layer sat underneath, and it was mine. The swap helper I'd written did remove(old) then add(new). One caller passes a group, not a single text object — two labels added individually, later bundled for convenience. Rather than reason about the framework's semantics I asked it, in four lines:

add(a); add(b); remove(Group(a, b))
→ both are still on screen

Removing a group doesn't remove members added individually. The new text would have been drawn on top of the old one, at the same coordinate — the helper written to eliminate the overlap reproducing it exactly, at five call sites across four files. I caught it before rendering by counting which callers pass a group and testing the assumption instead of trusting it. Code that fixes things is still code, and rarely gets the checks the code it fixes does.

So its check has two layers: one asserts the rule, the other exercises the framework and fails if that removal behavior ever changes. A rule with no test on its premise becomes a ritual the day the premise stops holding.

What transfers

When a check passes, ask what population it examined, not whether it has a bug. A check defined as "A against B" cannot see a failure entirely inside A.
Some checks are self-concealing at the extreme: they catch the partial version of a failure and go blind at the total version. Ask what perfect failure looks like to a check before trusting what it reports.
If a detector fails to converge after several honest, independent discriminators, that's evidence the channel lacks the information, not that the next one will work.
Choose the layer by asking where the defect leaves a trace: some vanish into a best-effort fallback and exist only in the output; others are one line in source.
A convention spreads faster than the code that first needed it, because people copy it by reading a sentence, not by copying code. A fix that changes the code but leaves the sentence gets quietly undone by the next reader.
Don't fuse a constraint and an implementation choice into one sentence. "There's no LaTeX, so cross-fade the text" states a true constraint and an implementation that doesn't follow — written together, the next reader copies both as one fact.
Peeling off one layer means looking again, not concluding. An improving number is progress, not completion — the second defect in the third case surfaced only because I re-rendered and looked again.
A collision found in one place is evidence of a class, not an incident. Fixing the one place it was noticed leaves every other place sharing that coordinate exactly as broken.
Your repair tool is code too. The helper written to remove the overlap would have recreated it, and nothing in the plan called for testing it.
Test a rule's premise separately from the rule. "Remove the whole family, not the group" is only worth obeying while the framework still behaves that way — so something should fail loudly the day it doesn't.
Partial assertion coverage is more dangerous than none. The scene asserted against overlap in eight places and skipped one, and the defect landed in the one it skipped. Many call sites read as "this is covered," so count the holes, not the hits.

None of these three zeroes were false. Each was a true statement about a smaller world than the one that mattered, and the only way to find the gap was to stop reading the number and go open the frame.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

It Fit in Memory and Was Still Unusable — Do the Bandwidth Arithmetic First

John — Sun, 06 Sep 2026 09:00:08 +0000

Originally published on hexisteme notes.

"Will it fit on our hardware?" is the wrong first question. It's the one everyone asks, because
it's free to answer — the thing either loads or it doesn't.

Throughput costs you a measurement. So the capacity gate passes, and it feels like the
decision is made.

The measurement

Mac Mini M4, 24GB unified memory, ~120GB/s memory bandwidth. A 27B model, IQ4_XS quantized,
15GB on disk.

Capacity gate: pass. Metal's recommendedMaxWorkingSet is 17.76GB, the model is 15GB,
ollama ps reports 100% GPU resident. No swap, no spillover. By every "does it fit" criterion
this is a clean win.

Generation: 5.6 tokens/second.

That's not a usable interactive worker. It's barely a usable batch worker. And nothing about
the capacity check hinted at it.

The arithmetic that would have told me in advance

Autoregressive generation reads the entire model's weights once per token. So:

ceiling ≈ memory bandwidth ÷ bytes touched per operation
        = 120 GB/s ÷ 15 GB
        = 8 tokens/second

Measured 5.6 against a ceiling of 8. Ratio 0.70.

That ratio is the whole verdict. When measured throughput is a large fraction of the arithmetic
ceiling, you are bandwidth-bound, and you now know something concrete: the bottleneck is
not your configuration, not memory pressure, not thermal throttling. It's how fast bytes move.

Rule of thumb I now use: ratio ≥ 0.5 → bandwidth-bound, and size-reduction fixes are dead.

Why "just quantize harder" doesn't work

The natural move when capacity is tight is to shrink. Lower quantization, smaller batch,
heavier compression. It's the reflex, and in a bandwidth-bound regime it's close to useless.

I was considering Q3_K_M at 13.8GB. Run the same division:

120 ÷ 13.8 = 8.7 tokens/second      (up from 8)

Under 9% more throughput. For a real drop in output quality, because quantization error
doesn't scale linearly with size the way bandwidth does — you give up more than you get, every
time, in this regime.

I killed that plan without downloading anything. That's the saving this rule buys: an
arithmetic rejection instead of an afternoon of benchmarking a model that couldn't have won.

The deeper reason is that capacity and throughput are governed by different resources.
Capacity is bytes of memory. Throughput is bytes per second across a bus. Pulling the capacity
lever moves the capacity number. It touches the throughput ceiling only through the incidental
fact that a smaller model has fewer bytes to stream — a weak, strictly linear coupling, and you
pay for it non-linearly.

Naming the right bottleneck tells you which lever works

This is the part that makes the arithmetic worth doing. Once you know it's bandwidth, the same
model on different hardware is a division away:

Machine	Bandwidth	Ceiling for a 15GB model
M4	~120 GB/s	8 tok/s
M4 Pro	~273 GB/s	18 tok/s
M4 Max	~546 GB/s	36 tok/s

The verdict flips on hardware, not on model size. "This model is too slow" was never true —
"this model is too slow on 120GB/s" was. Those lead to completely different purchase
decisions, and only one of them is correct.

When you correctly name the constrained resource, the set of interventions that can possibly
work falls out of it. Naming the wrong one sends you tuning things that were never the
problem.

Anti-patterns, all of which I've done

"It loaded, so we can use it." Loading is the capacity gate. Passing it leaves throughput entirely unknown.
"It's slow, so quantize lower." In a bandwidth-bound regime, size and speed are linearly coupled and quality degrades faster. This is the classic symptom of conflating the two gates.
"It must be swap / memory pressure." Measure warm, at least once, with the model already resident. Otherwise you're mixing the ceiling with transient congestion — and if you go clean up applications before separating them, you'll never find the actual cause. Here there was no pressure at all and the ceiling was exactly where the arithmetic put it.
"Theoretical bandwidth isn't effective bandwidth, so the calculation is meaningless." Effective is typically 60–80% of theoretical. Include that and the estimate is still good to an order of magnitude — and order of magnitude is the entire decision. 8 vs 5.6 is the same answer. 8 vs 36 is a different one.

The procedure

Before checking capacity, compute the throughput ceiling: ceiling ≈ bandwidth ÷ bytes touched per operation.
Measure at least once warm, so cold-load and swap pressure don't contaminate the number.
If measured / ceiling ≥ 0.5, declare bandwidth-bound and reject size-reduction fixes.
Choose from what's actually left: (a) restrict to latency-tolerant uses, (b) change hardware, (c) move the work off this machine.
Write the falsifier as a bandwidth number. If you've named the constrained resource, the point at which that resource changes is the condition that overturns your verdict.

That last step is why this generalizes past local models. The same shape applies to any
streaming bottleneck — disk-bound ETL, network-bound sync, cache-line-bound inner loops.
"It fits" and "it's fast enough" are separate gates, and the first one is free to check, which
is exactly why it gets mistaken for the second.

What would change my mind

Bandwidth is the named bottleneck, so the falsifier is a bandwidth number: on a 273GB/s
machine this model clears 18 tok/s and the "not usable as a live worker" verdict is void. Any
architecture change that stops reading all weights per token — heavy MoE sparsity, aggressive
speculative decoding — also breaks the bytes touched = model size assumption the ceiling is
built on, and the division has to be redone with the real figure.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Four MCP Tools Died the Same Day. One Unpinned SDK Dependency Killed Three.

John — Sun, 06 Sep 2026 00:00:06 +0000

Originally published on hexisteme notes.

Four of my MCP servers were dead. All four reported the same useless thing:
Connection closed.

It looked like four incidents. It was two — and one of them accounted for three.

Server	Symptom	Actual cause
A	`Connection closed`	Corrupted package-runner cache — a partial install left a dependency missing
B	`Connection closed`	SDK 2.0.0: `McpError` renamed to `MCPError`
C	`Connection closed`	SDK 2.0.0: the `…server.fastmcp` module removed
D	`Connection closed`	Same as C

Simultaneous death is a signal, not a coincidence

B, C, and D were different repos, different authors, different purposes. They died the same
morning.

Three independent failures landing on the same day is possible. One shared thing moving is
overwhelmingly more likely. And the shared thing was sitting in the launch commands:

uv run --with 'mcp[cli]'  mcp run .../server-c.py     # no upper bound
uv run --with 'mcp[cli]'  mcp run .../server-d.py     # no upper bound
uvx mcp-server-b                                      # pulls the SDK transitively

A dependency with no upper bound is a different program every time you run it. The SDK cut
2.0.0 and three deterministic imports became undefined simultaneously.

The diagnostic mistake I nearly made was going server by server. That path reads the same
stack trace three times and calls it three bugs. The moment I saw …server.fastmcp in the
first trace, the right question wasn't "how do I fix this server" — it was "what else shares
this SDK?"

When several components fail at the same time, stop looking at the components and look at
the shared dependency graph.

That's a cheap habit and it collapses an afternoon into ten minutes. Same-timestamp failures
across unrelated systems are usually one upstream event.

Pinning everything is the wrong fix

The reflex is "pin all of it." That's worse than the disease.

An upper bound freezes security patches along with breaking changes. Pinning all 22 of my
servers would trade three outages for twenty-two units of staleness debt, forever, most of it
on servers that were never going to break.

What I adopted:

Pin only what actually broke (three of twenty-two).
Record the pin date in a pin ledger.
Flag any pin older than 90 days for "can this bound come off yet?"
Leave the other sixteen unpinned. Deal with breakage when it happens.

The principle underneath:

A pin is a response to an incident, not a prevention. The prevention is detection.

You cannot pin your way out of upstream churn — you can only choose whether you find out from
a health check or from a user. So I built the health check.

What the health check actually looks for

Two things, and the second one is the point:

Retrospective — which servers are dead right now, plus the exact command to reproduce each failure, so diagnosis starts at second zero rather than after ten minutes of reconstructing the invocation.
Prospective — which servers could die the same way. This is the real product.

Scoping that second list took a revision. My first pass flagged 19 servers and the signal was
mush. The correct population is narrower: servers that re-resolve upstream on every launch
through a package runner (npx, uvx, uv, npm). An absolute-path binary — a venv
Python, a built Node script — has no version to pin; neither does a URL transport. Including
them wasn't cautious, it was noise. Sixteen, not nineteen.

A watchlist that flags things you can't act on trains you to ignore the watchlist.

The side finding: enabled ≠ used

While I was in there, I compared 60 days of measured usage against what was actually switched
on.

A language server for the platform accounting for ~13% of my activity (310 prompts): off.
A language server for a language I'd mentioned once in five months, with no session history in any relevant directory: on.

I swapped them.

Nobody chose that configuration. There was a good reason for each toggle at the moment it was
set, the reason expired, and the toggle stayed. Plugin state rots silently because nothing
in the system ever asks whether the original justification still holds.

Worth a periodic diff: what's enabled, against what you actually use. Both halves are
measurable. Almost nobody measures them together.

Mirroring configs, and the tool I deliberately left out

I also mirrored the server set into a second client. Two things worth stealing:

Package-runner servers need an explicit PATH. The desktop client launches processes
without a login shell, so a server invoked as uvx --from git+https://… can't find git. It
fails in a way that looks nothing like a PATH problem.

I excluded the trading and wallet servers — the ones exposing order placement, transfers,
and token mint/burn. That client had a permission-bypass mode enabled, meaning a single
misinterpretation on a general chat surface could place a real order or move real funds. My
project rules define a safety hierarchy for exactly those operations, and that surface was the
one place the hierarchy had no enforcement point.

The rule I'd write from that: when a capability's guardrail lives in one surface's config,
that capability doesn't belong in surfaces that don't read that config. Copying a tool list
across clients copies the tools; it doesn't copy the constraints.

(And: that client reads its config only at startup. A full quit and relaunch, or you're
debugging a file the process never opened.)

The incident inside the incident

A subagent I'd delegated the mirroring to ran a jq '… | tojson' over the config's value
objects while diagnosing — and printed eight live API keys into its own transcript. It
self-reported.

Remediation was straightforward: replace the values in that transcript, chmod 600. Residual
risk is real, because the values passed through a model API; whether to rotate is the owner's
call.

The lesson is about the delegation brief, and it's uncomfortable:

My brief said "don't print secret values." The agent complied with that. It never intended to
print a secret. It serialized a container that happened to hold them.

The ban has to name forbidden operators, not forbidden intentions.

tojson, to_entries, dump, repr, console.log(obj), print(vars(x)) — bulk
serialization of any structure that might transitively contain a credential. "Don't print
secrets" is a rule about goals, and careless disclosure isn't goal-directed. Since then my
delegation briefs name the operators.

What would falsify all this

If the SDK diagnosis were wrong, the three servers would keep dying after pinning. Falsified: all three exit 0 and reconnect.
If the selective-pin policy is wrong, a security patch lands within 90 days that exists only above the bound and I miss it. The ledger's re-check reminder is designed to catch that — and if the reminder fires and nobody reads it, the policy failed, not the tooling.
If leaving sixteen servers unpinned is wrong, another synchronized death from an upstream breaking change happens within 90 days. If it recurs, "pin what broke" becomes "pin the critical N preemptively."

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

The Dedicated OCR Engine Lost to the General-Purpose Model — 300 Slower

John — Sat, 05 Sep 2026 09:00:06 +0000

Originally published on hexisteme notes.

I had a 27B vision model running locally (IQ4_XS quantized, 15GB resident) and needed to decide
whether it was worth using for OCR. The comparison was macOS's built-in Vision framework
(VNRecognizeTextRequest) — a dedicated text-recognition engine, free, zero memory
footprint.

My expectation going in: the specialist wins on character accuracy, and the general-purpose
model is reserved for when you need semantic understanding. Slow and expensive, use sparingly.

That expectation was wrong, and it was wrong in a way that would have been invisible in
production.

Method: an image whose answer I already knew

The usual mistake in an OCR comparison is measuring against real documents, where you don't
have ground truth. Then you can't distinguish plausible output from correct output — and
plausible output is exactly what both engines produce when they fail.

So I rendered a 1100×720 test image with the answer fixed in advance:

A title and date in Korean, a 4-column × 3-row table (model / memory / speed / status), four lines of prose
One adversarial line: A0-1lO9-B8 — digit 1 next to lowercase l, capital O next to digit 0
Two empty table cells containing -

Then I looked at it. The first render was wrong — a label came out as tofu boxes (□□),
because the monospace font had no Korean glyphs. If ground truth is broken at the moment you
fix it, every measurement afterwards is void. That check costs thirty seconds and it's the
whole experiment.

The results

	Local 27B VLM	Apple Vision (dedicated)
Character errors	2	8
Reading order	preserved	destroyed
Table cells dropped	0	2 (the `-` cells)
Wall clock	82.8s (cold)	0.27s

Vision's eight errors: IQ4_XS→I04_XS, cloud→cLoud (twice), tok/s→tok/5, two
characters inside a code string, em dash —→-, arrow →→->.

300× faster. On accuracy alone, four times the error rate on a document of this size is
arguably a fine trade.

Accuracy alone is not what decided it.

The difference was structural, not lexical

Vision returned the table decomposed by column. Three model names in a row, then three
memory figures, then the speed and status columns appended at the end of the document.

Which means: you cannot recover which speed belongs to which model from the output. The
row associations are gone. Not garbled — gone. The characters are all there, correctly
grouped, in a well-formed sequence, and the relation between them has evaporated.

The VLM kept the rows. Next to that, 2 errors versus 8 is a rounding difference.

What this actually says is narrower than "the general model is better":

"A dedicated tool beats a general one" depends entirely on where you cut the task.

Vision is dedicated to character recognition. It is not dedicated to document
understanding. My task needed the second and I was picking tools by the first one's benchmark.
The specialist was genuinely better at the thing it specializes in — I had just mislabeled what
I needed.

Why this breaks the cheap-first fallback

The obvious architecture is: run the cheap engine, detect failure, escalate to the expensive
one. Almost everyone reaches for this.

It requires failure to be detectable.

Column-shredded output is syntactically perfect. It has plausible text, plausible structure,
no error signal of any kind. Downstream, it is indistinguishable from a correct read. The
information didn't get corrupted — it got dropped, and dropped information leaves no
residue to detect.

This generalizes past OCR. Any escalation ladder — cheap model then expensive model, cache then
origin, heuristic then solver — is only sound when the cheap tier's failure mode is
observable at the boundary. If the cheap tier can fail by silently discarding a relation
rather than producing a wrong value, "cheap first" isn't an optimization. It's an undetected
data loss path with a cost saving attached.

Both engines failed in exactly the same place

The adversarial string A0-1lO9-B8:

Attempt	Output
VLM, full image	`A0-1109-B8` — 2 misreads
VLM, that line at 4× with an explicit "distinguish 0/O and 1/l" instruction	`A0-1l09-B8` — recovered `l`, still lost `O`
Vision, on the enlarged crop	`A0-1109-B8` — 2 misreads, unchanged

I inspected the enlarged image myself. The font renders digit 0 with a slash through it and
capital O as a plain oval. The two glyphs are visibly different. This isn't image
ambiguity that more pixels would resolve — it's both engines genuinely misreading a
distinguishable character, and resolution doesn't touch it.

So:

No OCR engine can be trusted on strings where homoglyphs change the meaning — codes, IDs,
hashes, addresses, license keys.

That's not a tool-selection problem. It's a property of the entire tool class, which means the
remedy isn't a better engine. It's human confirmation or a checksum. If you're about to build
an OCR path for identifiers, build the checksum first.

What I adopted

Vision as the first pass; escalate to the VLM only for documents where reading order carries
meaning — tables, forms.

With one correction to that rule, from the paragraph above: since column-shredding isn't
detectable downstream, "escalate on failure" doesn't work for tables. If the corpus is mostly
tables, go to the VLM first and eat the 300×. Cheap-first is only valid when failure is
visible.

What would change my mind

If the target documents are mostly prose, Vision alone is sufficient and the VLM is a 300× waste.
If Vision gains layout analysis and starts preserving table structure, this verdict is dead.
The homoglyph failure is common to both engines, so improving one doesn't touch that part of the conclusion.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.