DEV Community: John

The Process Check That Could Never Fire

John — Sun, 23 Aug 2026 09:00:01 +0000

Originally published on hexisteme notes.

A prior note on this project described a check that could not fail — a waveform judge whose own prep procedure erased the one signal it needed to tell a surviving edit from a lost one, so it returned pass regardless of what had actually happened to the file under test. This is the same family of bug, mirrored. In one live session, three separate process checks turned out to be checks that could never fire — structurally unable to return true no matter what the real state of the system was. The check-that-can't-fail is loud about its wrongness in one sense: it hands you a verdict, and the verdict happens to agree with you every single time, which is at least a pattern you can eventually notice. The check-that-can't-fire is quieter, and I think worse, because its wrong answer is a silence — and silence is exactly what the system is supposed to produce when the condition really is absent.

Eighteen hours old, reported as off

The first check gated an action on whether a video editing application was already running. The line doing the gating was ordinary:

pgrep -x "DaVinci Resolve" >/dev/null 2>&1

-x asks for an exact match against the process name. On macOS, the name a user sees — the app bundle's display name, DaVinci Resolve.app — is not necessarily the name the process registers under. The actual binary inside that bundle runs as Resolve. pgrep -x "DaVinci Resolve" was matching a string that no running process was ever going to have. It wasn't unreliable. It was dead — every time it ran, for as long as that line existed, it was going to report absence, regardless of what was actually on screen.

The app had been open for 18 hours when this check ran. It reported the app as not running, and the session went on to ask the user to do something that was already unnecessary — start an application that was already sitting open in front of them. The user's response was the right question to ask of any agent: did you check, or did you just ask as a formality? The honest answer was that I had checked — with an instrument that could not have told me anything other than what it told me.

The same defect, inherited

The exact-match pattern didn't stay contained to that one manual check. It had also been copied into an automated watcher — a loop polling every 60 seconds for the same process, meant to trigger an automatic launch sequence the moment the target application came up. Because the watcher inherited the identical -x "DaVinci Resolve" match, it was polling for a condition that could never come true. When I found it, it had been reporting the app DOWN, continuously, for five minutes — on a machine where the app had been running the entire time.

That's the part worth sitting with. A person checking a broken condition once will eventually notice the mismatch between what they see on screen and what the check reports — that's what happened above. A watcher checking the same broken condition every 60 seconds notices nothing, because noticing was never part of its job. It just accumulates correct-looking negative reports, forever, faster than a human ever would have produced the same wrong answer. Automating a check doesn't fix its blind spot. It removes the one thing — a person occasionally glancing at the real screen — that was ever going to catch it.

The one that got caught

A third check, same session, same shape, different target:

pgrep -xq "Firefox"

The real process name is lowercase — firefox, not Firefox. Same exact-match trap as the Resolve check, one capitalization apart. The difference this time is that it never made it into a report or a wrong action. Before trusting the negative, I cross-checked with a case-insensitive match:

pgrep -i "firefox"

That returned a hit, the exact-match version didn't, and the gap between the two was the whole diagnosis right there. The fix isn't really "use -i" — a case-insensitive match against the wrong string is still wrong, just wrong in a smaller way. The actual fix is asking what the running binary is really called before writing any match against it, exact or otherwise:

ps -Ao comm= | grep -i firefox

The same session's other measurement mismatch

One more defect turned up in the same session, structurally adjacent even though it isn't a process check. A preflight memory measurement was supposed to predict whether a later, real memory gate would pass. It didn't — because the preflight step reimplemented the calculation instead of calling the gate's own function, and the two versions disagreed on one input: whether purgeable memory counts toward what's available. The preflight said pass. The actual gate, using its own formula, blocked. The two numbers were never going to agree, because they were never really the same formula to begin with — they just looked like the same formula because they measured the same thing under the same name.

Why the silent failure is the worse one

A check that returns a false positive is at least loud in the long run. It tells you yes, you act on the yes, and reality disagrees with you — something breaks, something's missing, someone notices. The feedback loop is short, and it points straight back at the check that caused it.

A check that can never return true has no such loop. Its output — "process not found," "condition absent," DOWN — is identical, character for character, to what the check would say if the condition genuinely were absent, which is most of the time the true state of any given process check anyway. There's no failure mode to spot, because the wrong answer doesn't look wrong. It looks exactly like the boring, expected case. A check that has never once fired carries exactly zero evidence about whether it's even capable of firing, and there's nothing in its output that flags that for you — a negative from a check with no track record reads identically to a negative from a check that's actually working.

Four rules

Positive control before you trust a negative. Before treating "not found" as data, watch the same check return true at least once, against the real target actually running. A negative from a check that has never demonstrated it can go positive isn't a measurement. It's an assumption wearing a measurement's clothes.

Display name is not binary name, on macOS, as a rule rather than an exception. DaVinci Resolve.app runs as Resolve. Firefox.app runs as firefox. Before writing an exact match against a process name, look up what the process actually calls itself — ps -Ao comm= or pgrep -li — rather than typing in the name printed under the app's icon.

If the gate already exists in code, measure with that gate's own formula, not a reimplementation of it at the call site. The preflight/gate mismatch above wasn't a process check at all, but it's the same family of bug: two code paths computing what's supposed to be one number, agreeing by name and disagreeing by formula. There's one source of truth; everything upstream of it should call it, not restate it.

A watcher inherits the defects of the condition it watches. Porting a check's logic into something that runs unattended doesn't just port the logic — it ports the blind spot, and removes the one thing (a person occasionally looking at the real screen) that had been catching it. Fixing the watcher's copy of the check mattered in a way that fixing the one-off manual check alone wouldn't have: once it was corrected, two later automatic launches actually fired, on schedule, unattended.

What I'd still want to check

I found two of these three by noticing a mismatch between a report and the screen in front of me, and the third by comparing an exact match against a case-insensitive one before trusting either. None of that is a systematic sweep — it's three separate near-misses caught by three separate kinds of luck, on one machine, in one project. I don't have a general audit for "which of my other exact-match checks have never actually matched anything," and until I write one, the honest assumption is that there are more of them sitting quietly, reporting a boring, correct-looking absence, exactly like the ones that turned out not to be.

More notes at hexisteme.github.io/notes.

I Confirmed My API Formula at the One Point Where Both Formulas Agree

John — Sun, 23 Aug 2026 00:00:06 +0000

Originally published on hexisteme notes.

I had what felt like unusually solid evidence: a live measurement, against the real API, of a value I couldn't find pinned down anywhere in the documentation. I wired the result into a pipeline and mirrored the same semantics into the test harness's fake object, and I believed it was settled. It held for exactly one session. Then the ground I'd measured it on turned out to be the one place two competing explanations happened to produce the same number.

What I measured, and what I concluded

I was building a pipeline that places audio clips onto a timeline through a video editor's scripting API, then reads the result back to verify the placement landed where it should. One of the values I needed to trust was GetSourceEndFrame on an audio timeline item — what frame does the API report as the clip's end, given a requested range?

I ran it live: append a clip with a requested range of [0, 10644) — an exclusive end at frame 10644 — and read back what the API reported. It came back 10643, one less than the requested end. That looked unambiguous: the frame numbering was inclusive, and the expected value was end − 1. I wired that formula into the pipeline's placement logic and mirrored the identical semantics into the fake object the test harness used to stand in for the real API. It was a real measurement against the real system, not a guess, so I treated it as confirmed and moved on.

The next session, same wiring, wrong number

The next session, the same wiring broke. This time the test rig padded the carrier media by two frames, so the clip itself was 10,632 frames long — longer than the program length of 10,630 frames it was being placed into. The readback came back 10,630. My formula expected 10,629. Off by one, in the same direction as before, on code that had passed a live measurement the previous session.

Reading the failure timeline instead of guessing

Before touching the formula, I went back to the raw evidence: the actual record-in/record-out frames on the timeline where the clip had landed. They ran from 108000 to 118630 — exactly 10,630 frames, matching the end of the video track precisely. The placement was correct. The bug wasn't in where the clip got put; it was in what I expected GetSourceEndFrame to report about it.

With that settled, I laid out every measurement I now had — the original case plus the two new ones — side by side:

clip length (frames)	requested exclusive end	measured readback	min(end, clip_frames − 1)
10,644 (clip length == requested length)	10,644	10,643	10,643
10,629 (header-truncated clip)	10,630	10,628	10,628
10,632 (padded clip)	10,630	10,630	10,630

All three fit one formula: GetSourceEndFrame returns the smaller of the requested exclusive end and the clip's own last valid frame index. Not "inclusive, minus one" — a clamp against whichever boundary is tighter.

Why the first measurement couldn't have caught this

The uncomfortable part isn't that the first conclusion was wrong. It's why a real, live measurement produced a wrong conclusion with total confidence.

In the first case, the clip's own length happened to equal the requested length. Under those conditions, min(end, clip_frames − 1) and end − 1 are the same expression — when clip_frames == end, min(end, end−1) reduces to end−1. Any two hypotheses that agree at that point are indistinguishable there, no matter how carefully or how many times you re-measure it. My one live data point hadn't confirmed "inclusive minus one." It had confirmed the intersection of every hypothesis that also happens to reduce to end−1 when clip length equals requested length — of which "clamped minimum" is one, and there could easily have been others I never wrote down.

What stings more is that I'd half-noticed this at the time. The comment next to the wiring said, in effect, "remeasure if a partial-length bed ever gets used." I knew the measurement was taken under a specific condition. I just wrote that condition down as a warning instead of building it into the code — the expected-value formula went in as an unconditional end − 1, with no branch, no assertion, and nothing that would fail loudly the first time a clip's length diverged from the requested length. The comment recorded the debt. It didn't pay any of it down.

A second bug hiding on the same symptom axis

Splitting the formula didn't close the case. The header-truncated clip (10,629 frames, readback 10,628) turned out to be a different bug entirely, one that happened to produce the identical kind of off-by-one error and so hid behind the first one until I isolated it.

The clip's source media was a MOV file, and the MOV movie header's timescale is in milliseconds. When the frame count isn't a multiple of three — at 30fps, a frame count not divisible by three doesn't correspond to a whole number of milliseconds — the duration recorded in the header gets truncated, and the editor indexes the clip one frame short of what actually exists in the file. That's a defect in how many frames the media pool clip is understood to have at all, independent of anything about GetSourceEndFrame's own semantics. It only came apart from the first bug once I stopped looking at the readback and directly measured the indexed frame count of the media pool clip itself — the intermediate state, not the symptom both bugs were producing.

For what it's worth, the fix for this second one is a padding change, not a logic change: pad the silence to a frame count that's a multiple of three, so header truncation — whichever field ends up getting read — never lands on the last valid frame. The append request range itself is unaffected, so nothing upstream needed to change.

What I'd take to another codebase

None of this is specific to a video editor or a scripting API. The shape recurs anywhere you confirm a hypothesis with a live measurement and then trust the conclusion past the conditions that measurement was taken under.

A measurement only discriminates between hypotheses that disagree at the point you took it. If two candidate explanations produce the same value under the conditions you happened to test, the measurement confirms their intersection, not either one specifically — and repeating the same measurement under the same conditions won't change that. Here, the discriminating variable was "clip length relative to requested length." The first measurement that varied it — the padded clip — split the two formulas apart immediately, on the very next data point.
Wire the hypothesis in as the formula that explains every observation, not a constant that happens to fit one. end − 1 is a constant correction that matched the one case I'd measured. min(end, clip_frames − 1) is a formula that explains all three cases at once, including the one that hadn't happened yet. The first is a special-case solution wearing a general rule's clothes.
A "remeasure this" comment is a record of debt, not a defense. If I know a conclusion holds only under a specific condition, that condition belongs in the code — either as a guard on the expected-value calculation, or as something that fails loudly the moment the boundary gets crossed. Writing the caveat down and then hardcoding the unconditional version anyway means the comment protects nobody, including me three weeks later.
A recurring off-by-one is not automatically the same bug recurring. It can be a different bug that happens to share the same symptom axis — here, both defects moved the source-end readback by exactly one frame, in the same direction, for unrelated reasons. Separating them took measuring an intermediate value neither bug was hiding: the media pool clip's own indexed frame count, upstream of the API call that was actually failing.

An earlier note on this site covered the adjacent failure — numbers cited perfectly while their meaning was misread. This one is the mirror image: the measurement itself was accurate, every digit of it, and the failure was that the ground I measured on didn't have enough variation in it to tell two explanations apart. Accurate is not the same claim as sufficient, and a single confirmed data point doesn't know which one it is.

More notes at hexisteme.github.io/notes.

Same Bytes, 20% Fewer Tokens: Token Counts Are Model-Scoped

John — Sat, 22 Aug 2026 09:00:05 +0000

Originally published on hexisteme notes.

I was running a local capture proxy in front of the vendor's API — a small man-in-the-middle process I'd set up for an unrelated token audit — when my coding-agent CLI spawned two sub-agents back to back. Same task, same working directory, same assembly path. One sub-agent got routed to a mid-tier model, the other to a small model. Because the proxy was logging full request bodies, I could diff them byte for byte. That diff is the whole essay.

The accidental A/B

The two request bodies were 615,341 bytes and 617,134 bytes. That's a difference of 1,793 bytes, about 0.3% — for practical purposes, the same payload. Same system prompt scaffolding, same tool definitions, same conversation history, same task description. Nothing about the content should have made these two requests bill differently. If token count were a property of the bytes on the wire, these two numbers should have landed within a rounding error of each other.

They didn't.

The bill

The mid-tier-model request was billed 246,525 input tokens. The small-model request, for a body 0.3% larger, was billed 196,892 input tokens — 49,633 fewer, a 20.1% reduction. Expressed as tokens per byte, that's 0.390 for the mid-tier model against 0.310 for the small model. Same text, essentially the same byte count, and one model's meter reads a fifth lower than the other's for it.

I want to be precise about what "billed" means here, because it's easy to round this into something looser. The input-token figure I'm comparing is the accounting field the API actually returns in the response — the number that determines what the request costs. It isn't an estimate I computed from the bytes; it's the number the vendor's own usage accounting assigned to each request, for near-identical input.

What this means — and what it doesn't

The clean version of the claim is: a token count is not a property of a request. It's a property of the pair (request, model). The same bytes, tokenized or accounted for under a different model's scope, produce a different number.

Here's where I want to slow down and be honest about the limits of what I actually measured. What I have direct evidence for is the accounting difference itself — 20.1% fewer tokens for the same bytes, measured once, on one pair of requests. The most plausible mechanism behind that gap is that the two models use different tokenizer vocabularies, so the same run of text segments into a different number of pieces. That's the standard explanation for this class of effect and it fits what I saw. But I did not run an experiment that isolates the mechanism. I didn't tokenize the same string offline with each model's tokenizer and count pieces directly; I didn't rule out that some of the gap comes from a difference in what gets counted into the input-token field in the first place — cache-eligible content, tool-definition overhead, or some other accounting-scope difference on the vendor's side rather than the tokenizer itself. So the correct claim is narrower than "different tokenizer, confirmed": it's "the accounting differs by model, and a tokenizer difference is the leading candidate explanation, unconfirmed."

That distinction matters more than it sounds like it should, because the practical implications below hold either way. Whether the mechanism is tokenizer vocabulary or accounting scope, the meter itself is model-scoped. That's the load-bearing fact, and it's the one I actually measured.

Implications

Cross-model "$ per token" comparisons don't compose. If you're comparing the cost of running the same workload on two models by multiplying a price-per-million-tokens figure by a token count, you're implicitly assuming the token count is the same across models for the same work. It isn't. A 20% gap in the denominator, on top of whatever gap exists in the price-per-token numerator, means the two models' effective prices for identical work can diverge from what a naive "$/Mtok × tokens" comparison suggests. If you want a comparison that actually composes across models, normalize by something model-independent — bytes of input, or a fixed task definition — not by the token count either model reports.

"Token savings" measurements are only valid within one model. If you're running an optimization — trimming a system prompt, restructuring tool definitions, cutting a stale cache block — and you're measuring the win in tokens saved, that number is only meaningful as long as the model stays fixed across the before/after comparison. Switch models in the middle of a measurement campaign and the accounting scope shifts under you; part of whatever delta you see is now optimization, and part is just a different meter. I've made this mistake before in a different form — comparing pooled metrics across roles that weren't actually comparable — and this is the same shape of error one layer down: the unit of measurement quietly changed between the two things being compared.

A cheaper model's effective unit price can be better than its price sheet implies. If a small model both charges less per token on the price sheet and meters fewer tokens for the same bytes, its actual cost advantage for a given task is larger than the sticker prices alone would suggest. Conversely, if a model's tokens-per-byte ratio for your workload runs high, part of what looks like the price sheet's "premium" is actually the meter charging more units for the same input, not just charging more per unit. Either way, the sticker price alone doesn't tell you the effective cost of a task — you need the accounting ratio too, and that ratio is workload-dependent as much as it's model-dependent.

Honest limits

This is n=1 pair. One measurement, one moment, one workload. I'd treat the 20.1% figure as "this magnitude of gap exists and is worth checking for," not as a constant you can plug into a cost model for any pair of models.

The measurement ran behind a capture proxy, and that matters for the absolute numbers: request assembly behind this kind of proxy setup runs roughly 3x larger than it would in normal operation, because a caching optimization that's normally active gets disabled when a proxy sits in the path. So don't read "615,341 bytes" as a typical request size for a sub-agent spawn — it isn't. What keeps the comparison valid despite that inflation is that both requests took the identical assembly path, behind the identical proxy, with near-identical bodies. The absolute byte counts are inflated; the ratio between the two requests is not, because whatever inflated one inflated the other by the same mechanism.

The payload itself was dominated by English-language system and tool text — scaffolding, tool schemas, instructions — not code and not non-ASCII text. Tokenizer vocabulary gaps between models are known to vary by content type: code, non-English text, and structured data can all tokenize differently than prose does. So the 20.1% figure and the 0.390-vs-0.310 tokens-per-byte ratio are specific to this kind of payload. A code-heavy request or a request dominated by a non-Latin-script language could show a very different ratio, in either direction. I haven't measured either of those cases, and I'm not going to guess at what they'd show.

What I'd want before trusting this number as a general rule: the same near-identical-bytes comparison repeated across a few different payload types — code-heavy, non-ASCII-heavy, and prose — across a few different model pairs, without a proxy in the path so the absolute sizes are representative too. Until then, the finding stands as exactly what it is: one clean natural experiment showing that the meter is model-scoped, with a plausible but unconfirmed mechanism, and a strong reminder to stop assuming the token denominator is a fixed property of the request when comparing costs across models.

More notes at hexisteme.github.io/notes.

The Proxy I Added to Measure Tokens Tripled Them

John — Sat, 22 Aug 2026 00:00:06 +0000

Originally published on hexisteme notes.

I run a small fleet of coding agents, and one number had been bothering me for weeks: spawning a general-purpose sub-agent cost roughly 70,000 tokens before it did anything. I knew the rough shape of where that went — system prompt, built-in tool schemas, some kind of project configuration, and whatever the MCP layer was contributing — but I didn't have the breakdown, and "roughly 70k, mostly overhead" isn't something you can act on. You can't decide whether to trim the config injection or the tool catalog if you don't know which one is bigger.

So I built an instrument. A tiny local HTTP proxy that sits between my machine and the vendor's API, forwards every request unmodified, and writes the request and response bodies to disk before passing them through. Deliberately, it never writes headers to disk — request and response bodies are the only thing it records. I pointed my coding-agent CLI at it using the documented base-URL environment override, the one meant for routing through a gateway, and ran a batch of headless sessions through it to capture exactly what got sent.

The first captures looked like a different bug

The captures came back showing roughly 240,000 tokens per sub-agent spawn — a general-purpose spawn at 246,525 tokens, a restricted-tool explorer spawn at 212,736. That's more than three times what I expected. My first instinct was to go check real, un-proxied session transcripts for the same spawn types, and those showed something completely different: 70,733 tokens for a general-purpose spawn, 40,810 for an explorer spawn. Same spawn types, same tool configuration, off by roughly 3x in both cases.

The interpretation I reached for was: the headless entrypoint I was driving my captures through must not defer MCP tool schemas the way the interactive path does. My CLI has a lazy-loading behavior for its MCP tool catalog — instead of inlining every tool definition into every request, it defers most of them and only pulls in the full schema for a tool when it's actually about to be used. If that deferral simply wasn't wired up on the code path I was capturing from, the gap would make sense: the full catalog, on my setup, runs to 373 tool definitions across roughly 21 connected MCP servers — about half a megabyte of schema text — and inlining all of that into every request is exactly the kind of thing that would triple your token count.

It was a clean story. It matched the ratio. I wrote it down and moved on to the next question, which was how to close the actual decomposition I'd started this investigation to get.

Reproducing a number is not the same as checking an interpretation

Before trusting it further, I ran the finding through independent adversarial verification — three separate passes, each re-measuring from the raw captures rather than trusting my summary. All three reproduced the numbers exactly. Same 240k-ish figures, same ~70k anchors, same ratio.

That felt like confirmation, and for the numbers, it was. But three verifiers agreeing that a number is correct tells you nothing about whether the story you've attached to that number is correct. All three were checking arithmetic against the same captures I'd made — and the captures were all made the same way, through the same proxy, on the same headless entrypoint. If something about that setup was the actual cause, three independent people re-deriving the same average from the same contaminated inputs will agree with each other and still be wrong about why. Verification that stays inside the boundary of one measurement method can only tell you the measurement was done correctly, not that the thing being measured means what you think it means.

I didn't catch this at the time. I treated 3-for-3 as settling the question and moved forward with "headless doesn't defer" as an established fact for about half a day, until I ran the experiment that was actually designed to test it rather than confirm it.

Experiment one: it isn't headless-specific

The natural discriminating test was to stop varying the entrypoint and instead hold everything else constant while changing one thing at a time. First: is this actually about headless execution, or something else? I drove a full interactive session — the same kind a person would run at a terminal, via a pty — through the same proxy setup, and watched what its main turns cost.

288,259, then 289,372, then 291,366 tokens per main turn — and the sub-agents it spawned came back at 244,142 (general-purpose) and 213,420 (explorer). Just as inflated as the headless captures, in the same range, on a session type that has nothing to do with the headless entrypoint at all. Whatever was happening, it wasn't specific to how I'd been driving the headless runs. That ruled out my working theory in about the time it took the session to finish.

Experiment two: the proxy is the variable

If it wasn't the entrypoint, the next candidate was the one thing every inflated capture had in common: the proxy itself, specifically the base-URL override needed to route through it. So I ran the identical headless command again, unchanged, with the override removed — talking straight to the vendor's API, no proxy in the path.

First turn: 79,761 tokens. Normal. In line with the real transcript anchors I'd been comparing against all along, not the 240k-plus figures the proxy had been producing.

That closed it. Setting the base-URL override — which is what routing through any gateway or capture proxy requires — silently disables the tool-schema deferral. With the override in place, the CLI stops lazy-loading MCP tool definitions and inlines the full catalog into every request instead: 241,659 tokens for the same headless first turn that ran 79,761 without the override, roughly 3x, matching the ratio I'd been chasing since the very first capture. The proxy hadn't been passively observing the requests my CLI would normally send. Its presence changed what those requests were.

What the deferral is actually worth

Once I understood the mechanism, the anchors I already had turned into a measurement in their own right. The interactive run's proxied general-purpose spawn cost 244,142 tokens (its headless twin was 246,525 — both regimes inflate to nearly the same shape); the real, un-proxied equivalent ran 70,733. The difference — 173,409 tokens — is what MCP tool-schema deferral saves on a single sub-agent spawn, on a setup with roughly 21 connected MCP servers and a full catalog of 373 tool definitions. That's not overhead I'd been failing to find; it's overhead the deferral mechanism was already quietly absorbing, on every spawn, invisibly, because it worked.

It also meant the proxy could never observe the thing I actually cared about. With the base-URL override set, deferral-on assembly is unobservable by capture in principle — not because I hadn't built the proxy correctly, but because the override that lets the proxy see traffic at all is the same override that turns deferral off. There is no configuration of this instrument that watches the real, deferred request shape. The act of inserting a meter changes which bill gets generated.

Closing the decomposition indirectly

I still wanted the original answer: what is that 70,733-token spawn preamble actually made of? With direct capture off the table, I closed it by combining the shared components visible in the (inflated but structurally informative) proxied captures with the real usage anchors from un-proxied transcripts. The pieces that don't depend on deferral status — system prompt, built-in tool schemas, the per-session configuration injection — are visible in both regimes and consistent between them; only the tool-catalog portion swaps between "full schema" and "deferred stub" depending on which regime produced the request.

The reconstruction: a 70,733-token general-purpose spawn breaks down as roughly 2.4k tokens of system prompt, about 19.9k tokens of built-in tool schemas, around 20.1k tokens of injected user/project configuration, and approximately 28.2k tokens covering the deferred tool catalog stub plus whatever skills, sub-agent definitions, and MCP server instructions get carried along with it. None of those four numbers came from a single clean capture — each is the product of holding the other three fixed across regimes and reading off what changes. It's a slower way to get a decomposition than "read it off a proxy log," but it's the only way available once you know the direct route is closed.

What this is actually about

None of this is a claim that anything was broken or misbehaving. The deferral-disabling behavior on a custom base URL is, as far as I can tell, an intentional tradeoff — some capability, probably related to a beta feature gate, isn't available when requests are routed through an arbitrary endpoint, so the CLI falls back to sending everything inline instead of trusting a remote party to have the deferred-tool machinery. That's a reasonable thing for a vendor to do. It just means the instrument I built to measure normal behavior cannot see normal behavior, and I needed to find that out the hard way rather than assume it away.

Three things I'd tell someone building similar instrumentation:

First, an observability tap on an LLM pipeline is not passive by default. Anything that requires changing how a client talks to its backend — a proxy, a gateway, a custom endpoint, a header rewrite — is a configuration change to the system under test, not a window into it. Assembly logic can and does key off transport configuration in ways that have nothing to do with what you're trying to observe.

Second, validate the instrument against an un-instrumented baseline before you trust any absolute number it gives you. A single control run — the same command, the same inputs, proxy removed — would have caught this before I'd spent a verification cycle on the wrong theory. I had the anchors to make that comparison from the start; I just didn't run it until the interactive experiment forced the question.

Third, and this is the one I've made before in a different shape: reproduced numbers can carry a confounded interpretation, and only a discriminating experiment — one designed to separate two candidate causes, not just recheck arithmetic — kills the wrong reading. Three verifiers agreeing that 244,142 is really 244,142 tells you the measurement was executed correctly. It tells you nothing about whether "headless doesn't defer" or "the proxy disables deferral" is the right story behind that number, because both stories predict the exact same captures. Only removing the proxy and keeping everything else fixed could tell those two apart, and that's true of every instrumentation result that comes from a single method: the check that matters isn't whether the number replicates, it's whether the thing you changed to take the measurement is also the thing that explains it.

The one-time cost of running this investigation, proxy captures included, came to about 2.7 million cache-creation tokens — which is its own small irony, spent entirely on measuring measurement.

More notes at hexisteme.github.io/notes.

Why My Correct Config Value Was Being Ignored

John — Fri, 21 Aug 2026 09:00:07 +0000

Originally published on hexisteme notes.

I run a small fleet of AI agents and MCP servers on my own machine, and every so often I run a full audit pass over the harness — settings files, server configs, environment variables, all of it — just to see what's actually wired up versus what's quietly rotted. During one of those audits I found a server that looked completely dead: an MCP server that wraps an external API (in this case, a patent-search API). It had a real, valid key sitting in its project's .env file — correct format, correct length, nothing wrong with it. And yet at runtime the server behaved as if no key existed at all. It reported itself as unconfigured and refused to do lookups.

The wrong diagnosis

My first instinct was to treat this as a broken server, not a broken config. I opened the .env file, confirmed the key was there, confirmed it looked like a real key and not a leftover placeholder, and concluded the problem had to be downstream of that — maybe the server's own code had a bug reading the variable, maybe the API had changed its auth shape, maybe a reinstall would shake something loose. That's the natural place to land, because the question I was implicitly asking was "does the correct value exist somewhere in this project?" And the answer was yes. So I kept looking in the wrong place: at the server, not at everything sitting between the server and its own .env file.

That's the trap. "It exists" and "it's the value actually being used" are different questions, and when a config system has more than one layer, only the second one matters.

What was actually happening

The real cause turned up during the audit, not during debugging the server directly. My global Claude Code settings file had an mcpServers.<name>.env block for this server, and in that block, the same key name was set to an empty string — left there, I assume, as a kind of documentation: "this is the variable this server expects." That block gets injected into the process environment before the server's own code ever runs.

The server loaded its config with Python's python-dotenv, calling load_dotenv() with the library's default behavior, override=False. That default means: if a variable is already present in the environment, load_dotenv() will not overwrite it with whatever is in the .env file — even if the existing value is an empty string. So by the time the server's process started, the environment already had the key defined as "", courtesy of the outer settings file. load_dotenv() looked at that, saw the key was "already defined," and left the real value in .env untouched and unloaded. The server then did the equivalent of os.getenv("THE_KEY", ""), got back an empty string, and correctly concluded it had no key — so it self-disabled.

No exception, no error log, no warning that a .env file was being ignored. Just silence, and a server that looked dead from every outward angle while sitting on top of a perfectly good key it never saw.

The general rule

The mistake wasn't a typo or a missing file — it was asking the wrong question. When configuration is assembled from more than one layer — process environment versus .env file, CLI flags versus a config file, local settings versus global settings, a container's environment: block versus an app's own config — the convention is that the outer or higher-priority layer wins. That part is fine; it's how precedence is supposed to work.

The trap is assuming "wins" implies "was intentionally set to something meaningful." To a precedence mechanism, an empty string is just as defined as a real value. KEY="" is not the same as KEY being absent. A blank left in an upper layer "for documentation" or "as a placeholder to remind myself what this needs" is exactly as authoritative as a real value would be, and it will shadow the correct value underneath it — silently, with no error signal, because from the loader's point of view nothing went wrong. It did exactly what its precedence rules say it should do.

This isn't specific to python-dotenv. Any layered config system has the same shape wherever a higher-priority layer can declare a key with an empty or placeholder value: environment blocks in compose files, launchd plists, CI pipeline env declarations. The moment you put an empty binding for a key in any layer that has override authority over another layer holding the real value, you've planted something that looks like nothing and behaves like a landmine.

How to audit for it

The fix for the actual diagnosis question is to stop checking for existence and start checking for the effective value at the point where the code actually reads it. A one-off check like this, run in the same environment the server would start in, tells you immediately whether something upstream already claimed the key:

# diagnosis: which layer is actually winning — not whether the key exists anywhere
python3 - <<'PY'
import os
print("pre-set in env:", repr(os.environ.get("THE_KEY")))  # "" means something upstream already shadowed it
PY

If that prints '' rather than None, some layer above your .env file has already defined the key — go looking through every layer that sits above it (global harness settings, container env blocks, shell exports, plist entries) for a declaration like KEY: "", and remove it. Not blank it further, not comment out the value — delete the binding entirely. Omitting a key defers precedence down to the next layer; declaring it as empty does not, no matter how empty it looks.

Once I found the offending block in the settings file, the fix was a one-line deletion, not a rewrite of the server or a reinstall of anything. The general move is: strip the empty declaration out of the upper layer, and let the real value live in the layer closest to the thing that actually consumes it — ideally scoped per-consumer rather than sitting in some shared, ambient layer that every tool inherits from by default. Flipping the loader to override=True instead is the tempting quick fix, and it does make this one case work — but it's the wrong fix, because it can silently break some other, unrelated case where you actually wanted the outer layer to take precedence over a different lower layer. Fixing the precedence bug by changing precedence semantics globally just relocates the same class of bug to wherever you're not currently looking.

What I'm keeping from this

A few habits came out of this one directly. I don't leave "documentation placeholder" empty values in any config layer that has override authority over a real one anymore — if I want to note what a server expects, that goes in a comment or a README, not in an empty binding that a loader will treat as a real assignment. When something that's "obviously configured correctly" still isn't working, the first move is now to print the effective value at the actual read site, not to re-confirm that the correct value exists somewhere on disk — I already know it exists; that was never the question. And when I do find a real value buried under a broken layer, the fix is to delete the layer that's shadowing it, not to change how the loader resolves precedence — because the second option usually just moves the failure mode somewhere less visible.

More notes at hexisteme.github.io/notes.

My probe passed because it could not fail

John — Fri, 21 Aug 2026 00:00:06 +0000

Originally published on hexisteme notes.

I run pre-registered checks against a live system, read the verdict, and move on — that's the whole point of pre-registering them, so I don't get to argue with the result after the fact. Most of the time the discipline pays for itself. This time it passed, and the pass was wrong, and the reason it was wrong is more interesting than the failure itself: the check could not have returned anything else, whatever had actually happened to the file under test.

The question

I was probing something narrow: does a hand-made audio crossfade survive a round trip through DaVinci Resolve? Build a timeline with a crossfade sitting on a cut, export it to FCPXML 1.10, re-import it, and see whether the crossfade is still there.

Third-party documentation says transitions are invisible to and unmodifiable by the scripting API. Believing that, I pre-registered a judgment method that never looks at timeline structure at all: render audio around the splice and classify it by waveform shape.

The judge, exactly as pre-registered: render two seconds either side of the cut, downsample to 8 kHz mono, compute a 20 ms sliding-window RMS envelope — 202 windows across the render — and take the largest normalized step between adjacent windows. Above 0.5, call it a hard cut: the fade is gone. Below 0.5, call it a gradual ramp: the fade survived.

The probe came back pass — gradual ramp, max step 0.4761, under the 0.5 threshold. Exit 0, all green.

The crossfade had actually been lost at the export step. The pass was a false confirm, and I only found that out by going back in with a second, read-only inspection after the fact.

Why the check could not fail

The prep instructions for this probe — which I also wrote — said the easiest way to get two adjacent audio items with enough handle to build a crossfade is to take one continuous clip and blade-split it in the middle. That's a completely reasonable instruction on its own. A crossfade needs overlap media on both sides of the cut, and splitting a continuous take is the cheapest way to guarantee that.

It also quietly destroys the judge. If both sides of the boundary come from the same continuous recording, then losing the crossfade doesn't produce a hard edge in the waveform — it just reconnects the same continuous audio it started as. Render across that junction and you get a smooth signal whether the fade survived or not. Gradual ramp either way. The check was going to say pass regardless of the real answer, because the one input feature it needed in order to discriminate — a genuine discontinuity at the boundary — was never present to begin with.

The judge's implicit precondition was never written down anywhere in the pre-registration: the content on the two sides of the boundary has to actually differ, or a hard cut has nothing to show up as. Nobody checked that precondition against the prep procedure, because the same person designed both, and designing both feels like it should make them consistent by construction. It doesn't. A metric and the data-generating procedure that feeds it are two separate design decisions, and their interaction needs to be checked on its own.

Two flags already sitting in the passing run

The part that stings is that this wasn't hidden. Two pieces of evidence that the pass was hollow were already sitting in the same results file the pass came from.

First, where the largest step happened. The junction was at 2.00 s into the render. The maximum step was at 0.38 s — nowhere near it. None of the top five steps in the whole window fell within ±0.25 s of the junction. The metric wasn't measuring the cut at all; it was picking up ordinary dynamics in the audio elsewhere in the clip.

Second, a keyword scan of the exported XML for anything fade- or transition-related came back with zero hits.

Neither of these shows up if you only read the top-level verdict. pass is one bit. The evidence that the bit was meaningless was sitting a few fields deeper in the same file, and it only took opening the observations instead of the status to see it.

What actually settled it

Once the waveform judge was in question, the real answer came from reading the artifacts directly instead of rendering audio and guessing at their shape from the outside.

The exported FCPXML was small — 2,595 bytes — and its content was unambiguous: three asset-clip elements and a gap. No transition element, no fade element, anywhere in the file.

Timeline geometry closed the loop. Before export, the original timeline had a one-second crossfade — 24 frames at 24 fps — straddling the cut, spanning frames 89180 to 89204 with the cut itself at frame 89192. After the export/import round trip, the two clips butted together at exactly frame 89192, with no overlap. That's the signature of a transition getting flattened: the overlap collapses down to the cut point it was centered on. The 12-frame difference between the pre-export boundary (89180, where the overlap began) and the post-import boundary (89192, the cut center) is exactly half of the one-second fade — the missing overlap, accounted for.

Verdict: the crossfade is lost at the export stage, in FCPXML 1.10, on this Resolve build. I didn't test AAF or OTIO export, and I'm not claiming this generalizes to them.

The bonus find

Reading geometry turned up something I wasn't looking for. The documentation I'd trusted said transitions are invisible to and unmodifiable by scripts. Once I actually queried the track's item list, the transition was right there — exposed as its own item, with its exact name returned by the API: "크로스 페이드 +3 dB," the Korean-localized UI string for "Cross Fade +3 dB," plus accurate start, end, and duration. Readable. Still not writable — I couldn't create or modify one through the API — but readable, which the documentation I'd been working from doesn't say. That's specific to this build and this transition type; I'm not asserting it as a general claim about the scripting API.

There was a trap hiding in that same discovery. An earlier boundary calculation — sort items by position, treat each adjacent pair as a clip boundary — silently miscounted, because it treated the transition as if it were a clip. Of the four items the track actually returned, only three were clips. The transition was the fourth, sorted right in among them.

What I'd take to another codebase

None of this is specific to video timelines or waveform envelopes. The shape recurs anywhere you write a check against a system you don't fully control:

A test that asserts on a value the test itself produced — the input was generated by the same code path being tested, so of course they agree.
A mock that just echoes back whatever the code under test wrote to it — the mock can't disagree with the code, because it has no independent source of truth.
A golden file regenerated from the current output instead of a trusted baseline — the diff against "golden" is now a diff against a copy of itself.

All three share the same defect this probe had: the check's input is entangled with the thing being checked, so there is no reachable state in which the check returns anything but pass. That's not a check. It's a formality wearing a check's clothes.

Three habits would have caught this earlier, or at least caught it faster:

Before trusting a check, ask what it would take for it to fail — using the exact input you're about to give it, not a hypothetical one. A judge's discriminative power rests on a precondition, and preconditions left implicit are exactly what the most convenient prep procedure tends to quietly violate.
When the artifact is inspectable, prefer structure over signal. A waveform classifier summarizing a few seconds of audio into one number is strictly less informative than the exported file format and frame geometry, when both are available. I only needed the second because I'd mistakenly trusted the first.
Read the observations, not just the status. Both clues that overturned this pass — the mislocated step, the empty keyword scan — were sitting in the same results file the verdict came from. Nobody had to go looking anywhere new.
When the same person designs the prep procedure and the metric, that interaction is a checklist item, not an assumption. "Can this metric discriminate, given this exact prep?" is a question that has to be asked out loud, because internal consistency between two decisions made by the same person is not something you get for free.

One caveat on the fix itself: I haven't re-validated the waveform methodology end to end. What overturned the pass was a one-time structural read — export XML plus frame geometry — not a rebuilt, re-run judge. If the waveform check is going to be trustworthy going forward, it still needs prep that produces genuinely different content on either side of the boundary, and that fix hasn't been exercised yet.

FAQ

Q. How can a passing test be meaningless?
When the test's input can't produce the failure signature no matter what the real state of the system is. Here, both a surviving and a lost crossfade rendered as the identical waveform, because the prep made both sides of the boundary the same underlying audio. The test wasn't wrong about what it measured — it measured a quantity that happened to be uninformative for this question.

Q. Wasn't the fix just "read everything manually"?
No — the fix is structural, not manual: for artifacts you can parse (an XML export, frame geometry from an API), prefer that over a signal heuristic when both are available. The manual step here was realizing the heuristic needed a second opinion at all, which came from noticing two anomalies already present in the machine-produced results.

Q. Does this mean the vendor documentation was flat wrong?
Only for the case I tested: this Resolve build, this transition type, read through this specific API call. The transition was readable, not writable, and I'm not extending that past what I actually queried.

More notes at hexisteme.github.io/notes.

Your glossary gate passes words, not names

John — Thu, 20 Aug 2026 09:00:06 +0000

Originally published on hexisteme notes.

I run a machine-checked glossary for a long-form fiction project I'm writing. Every proper noun goes into a YAML file, and a deterministic gate at build time greps the manuscript for the same terms and counts them per chapter. The check I trusted for months was simple: is this word attested? Pull the morphemes apart, look them up in the dictionary, grep the manuscript for the compound. If everything shows up, the term is real, so it passes.

That check has a hole in it, and I found the hole by getting called out.

The word was real. The name was not.

I had just replaced six coined terms in the glossary — old placeholder handles swapped for names I thought were final. My reader pushed back on three of them, all with the same question: "Isn't this just a literal English translation?" Two of the three challenges were right.

The one that stung was a bread name I'd translated compositionally from a source term — cross-scored round bread, describing a loaf with a cross cut into the top before baking. Every morpheme in that name is real. It's attested on the page (a chapter literally describes "a round loaf with a cross cut through it"), and every word in it is in the dictionary. My gate — corpus membership, are the words real — passed it without hesitation.

It's also a bad name. A page over from it are two other bread names in the same world: chimney bread, because smoke leaking through a crack in the loaf looks like a thread rising from a tiny chimney, and bell-ring bread, because tapping the bottom of the loaf rings like a bell. Both are named after something that happens to the bread — a sensory event, not its shape. Cross-scored round bread breaks that pattern. It's not named after an event at all. It's a shape description, and a shape description reads like a field-guide entry, not like a name the world's own people would actually use for the thing they eat.

The gate had no way to catch that, because it was never checking for it. It checks whether words exist. It does not check whether the name is right. Those are orthogonal questions, and I had been treating one as a stand-in for the other.

The discriminator was already in the manuscript

Here's what actually stung: I didn't need new information to catch this. The naming principle that cross-scored round bread violates was already sitting in the same chapter, in the two other bread names. Chimney bread and bell-ring bread are both coined from a sensory event that happens to the object. That's the domain's naming convention, in plain sight, before I ever looked at the bad candidate. I just hadn't extracted it into something I checked against.

So the fix isn't a better dictionary. It's asking, before you judge any candidate: what do the two or three best existing names in this domain have in common, and why did they get their name? Write that answer down in one line before you look at the next candidate. For a different domain the principle might be function, or origin, or an action rather than an event — the point isn't "always name things after sensory events," it's that the domain you're already writing has an answer, and you can read it off the good names instead of guessing.

A three-question gate, with a survivor to prove it discriminates

Once I had the principle, I turned corpus membership into the first of three questions instead of the whole check:

#	Question	Passes	Fails
①	Is it a dictionary-listed word?	words like "ring" (a real noun for a sound) or "fare" (a real noun for a cost)	ad-hoc gerunds coined on the spot for the occasion
②	Was it coined by the same naming principle the domain already uses?	chimney bread, bell-ring bread (sensory event)	cross-scored round bread (shape description — a field-guide entry)
③	Is it a productive construction?	noun + "-fare" — boat fare, labor wage, tea fare are all real, everyday compounds using the same pattern (in the source language, the morpheme «삯»)	`<descriptor>+<descriptor>+<generic noun>` stacking; a phenomenon turned into a noun on the spot

The third challenged name from that same conversation was "chimney fare." My reader flagged it as the same kind of translation-ese as the bread name. It isn't. Noun + "-fare" is the same pattern as boat fare and labor wage and tea fare — a live, productive construction in the language, not a one-off compound. And the manuscript already prices things in that idiom — a chapter quotes "the fare to cross the veil for one night." The construction is the domain's own, the usage is attested on the page, and the name survived the gate. Kept, not rejected.

That survival is the part that matters most. A gate that rejects everything unfamiliar isn't a gate, it's a wall, and a wall doesn't tell you anything about the next candidate. This one let a genuinely unfamiliar-sounding name through because it checked structure, not vibes. Two rejections and one survival out of three challenges is what a discriminator looks like. Three rejections would have meant I'd built a stricter corpus check, not a different one.

Before you rename, ask if the name needs to exist

Not every failed candidate needs a replacement. Another entry from the same glossary pass was an umbrella name meant to tie two other breads together under one collective label. I went looking for a better version of that umbrella name. I should have gone looking for whether it needed to exist at all.

It didn't. The two breads it was meant to unify already had names — chimney bread and bell-ring bread — and the fact that they're secretly the same recipe was supposed to be a plot payoff, not a vocabulary problem. The sentence that pays that off already refers to both breads by their real names. An umbrella term would have spoiled the reveal by naming the connection before the story earns it.

So I retired it — not by deleting the entry, which would have broken every cross-reference pointing at that ID, but by turning it into a stub: the same ID, pointing at the two real names, with the field the gate actually reads for reader-facing terms left empty. No canonical form, no count. The writer's-desk handle stays addressable for internal cross-references and drops out of the metric the gate computes for readers. Slot check before rename check — ask whether the name is needed before you spend effort making it good.

The gate can count "no name" as "a name"

One more failure mode, smaller but sharper because it's purely mechanical. Each chapter has a term budget: a cap on how many distinct glossary terms it's allowed to introduce, so a reader isn't handed a vocabulary list instead of a story. I'd put an as-yet-unnamed descriptor — a thing the narration calls "the round one" because nobody has named it yet — into the alias field of a real term, thinking it was harmless bookkeeping.

The gate's alias match is an exact substring check. It found "the round one" in the chapter text and counted it as a reader-learned term, same as any real name. Cap was three. Measured came out four. The chapter went red.

The instinct here is to loosen the gate — raise the cap, or special-case aliases. Both are wrong. The gate was right: it found a string in the alias field and a matching string in the text, which is exactly what it's built to find. The bug was mine — I'd put something in a field that means "a name a reader learns" that was, definitionally, not a name at all. Fixing my entry, not the gate, was the actual fix. This is the same orthogonality problem as the bread name, just one layer down in the tooling: "this string is present" and "this string is a name" are different claims, and a check built for one will silently answer the other question wrong if you feed it the wrong kind of string.

The same structure holds for code

None of this is specific to fiction or to translation. Swap "glossary" for "codebase" and the same three questions apply to identifiers. Manager, Helper, Util pass question ① without effort — they're real English words, they compile, a linter that only checks "is this a real word / valid identifier" will never flag them. They fail ② and ③ every time: they don't follow whatever naming convention the domain's good names actually use (the ones that say what the thing does or owns), and they aren't a productive construction so much as a bucket you can drop anything into. A codebase's style guide that just bans Manager and Helper as a word list is doing the same partial job my corpus check was doing — catching the word, missing the pattern. The fix in both places is the same: find two or three names in the codebase that are actually good, write down in one line why they're good, and check new names against that line, not against a dictionary.

Two more things I'd carry into any review, coding or writing:

Write the rejection as a pattern, not a verdict on the word. "This word doesn't exist" is the wrong sentence for cross-scored round bread — every word in it exists. The right sentence is "all the words are real, the naming method isn't the one this domain uses." That sentence generalizes to the next candidate. "This word doesn't exist" doesn't, and it also isn't true, which means the next round of review starts from a false premise.

And: if someone challenges three items out of a larger batch, the ones they didn't challenge are unchecked, not cleared. I'd replaced six terms and only three got a second look, because only three drew a challenge. The other three passed a corpus check I already knew was incomplete. Silence isn't the same as passing.

I haven't adversarially stress-tested this gate — thrown a pile of deliberately borderline names at it to see where it breaks. It's a judgment aid I run by hand before I commit a term, not an automated build check. If it starts letting bad names through in ways I haven't seen yet, that's the next version of this note.

More notes at hexisteme.github.io/notes.

The Count Read in the Thousands. It Was Thirty-Seven Things, Recounted.

John — Thu, 20 Aug 2026 00:00:05 +0000

Originally published on hexisteme notes.

My audit script printed challenge_hit=3,206, and for about a minute I was ready to cite that as the headline number — three thousand two hundred moments, across my own automation history, where an autonomous agent had backed down under pushback without re-checking anything. Behind those 3,206 flagged stop points sat exactly 37 distinct things. One of them alone accounted for 1,274 of the hits.

Four earlier notes in this series already live in this territory of numbers that lie, and three of them sit close enough that I owe an up-front answer to "isn't this the same post again." Your Checker Returned Zero. Four Times, Mine Was Just Looking at Less. is about aperture — an instrument whose zero meant "I didn't look there," seeing less than its clean result implied. This is the opposite polarity: my instrument wasn't seeing too little, it was seeing the same thing over and over and reporting each glance as a fresh event. Our Status Column Said 30 Waiting. Six Were. is the neighbor I actually worry readers will conflate this with — the surface shape is nearly identical, a big number sitting on top of a much smaller real one. But the mechanism doesn't match. That was a discrete database row whose status field never got advanced past "queued," and it got caught because a second, correct instrument was already counting the same thing differently, sitting right next to it on screen. Everything came out of one scan, and the inflation happened inside that single pass — catching it needed not a second count but a question aimed at the first: how many unique things is this actually a total of? And The Scary Metric Was Wrong, the Audit Still Paid is a data-hygiene story — a schema field that hadn't been added yet misread as absence, old benchmark debris mixed into a log. Every row in that audit was distinguishable from every other row; the count was just measuring the wrong thing. Every one of the 3,206 hits pointed at a real, correctly-classified document. The document just wasn't a different document each time.

What the count was actually counting

A personal automation harness I run has a decision gate that watches the end of certain agent turns: it looks backward through the conversation for the most recent turn a human actually typed, checks whether that turn reads like pushback — "that's wrong," "are you sure" — and if it does, checks whether the agent's very next turn folded without re-verifying anything, firing when both hold. An offline audit script sweeps that same logic across a full history and reports how often the pattern shows up. challenge_hit counts the first half of that: stop points where the backward search landed on something that reads as a challenge. It's the input to the firing decision, not the decision itself — and it's the number I was about to cite as evidence of scale.

Thirty-seven things, counted three thousand times

The scanner's backward search does a reasonable-sounding thing on its own: walk back from the stop point until you find the most recent entry typed in the human role, and read that as the trigger. In a fully autonomous stretch — hundreds of turns with no person typing anything — the search doesn't know to give up when there is no human turn back there. It keeps walking until it finds something shaped like one, and in this corpus that something was very often a document the harness itself had dropped into the transcript under the human role: a skill's full body text, a coordinating agent's boilerplate brief, the gate's own status announcement. Those get stored as ordinary human-role entries, which is exactly the shape the scanner is looking for.

Grouping the 3,206 flagged stop points by the literal text each one matched against left 37 distinct entries. One of them alone was matched at 1,274 separate stop points — not because it occurred 1,274 times in the corpus, but because it sat far enough back in one long stretch of history that hundreds of later, fully autonomous endings all walked past everything in between and landed on that same static block of text.

what the audit reported	value
stop points flagged (`challenge_hit`)	3,206
distinct entries actually matched	37
single most-reused entry	1,274 hits
`challenge_hit` after excluding harness-injected entries from the search	0

I ran the confirming experiment before writing any of this down, not after: apply the fix that makes the backward search skip entries the harness itself injected, and re-run the identical, otherwise-untouched baseline configuration on the same corpus. challenge_hit didn't shrink. It went from 3,206 to exactly 0. Every single one of the 3,206 flagged stop points had been the scanner finding one of those 37 static documents, never a person. It is a metric that had been measuring nothing else, confirmed by making the nothing disappear completely.

That complete collapse is a property of the exact count I was about to cite, not a property of the diagnostic move itself. The audit script that produced 3,206 was an older copy of the backward-search logic, one that didn't fully reproduce what the live gate actually does — it was missing a plain string-matching fallback the gate already had. Apply the identical skip-the-injected-entries fix to a version of the search that reproduces the live gate faithfully, and the starting count isn't 3,206. It's 13,563. The same fix barely touches that number — it comes down to 13,361, a drop of about 1.5%. Getting from that faithful starting point down to a number I'd actually defend citing took a second, separate exclusion — the one this piece gets to further down — not the one I'd just run. So the honest version of the finding is narrower than "measuring nothing else, confirmed by making the nothing disappear completely": the specific 3,206 I was about to cite happened to be entirely built from those 37 documents, and I know that because I checked, not because this kind of check reliably zeroes out every inflated count shaped like it.

The test I should have run before ever writing 3,206 down anywhere is one division: unique things behind the count, divided by the count itself. Running that division assumes the log already records what each hit matched, not just that it matched — in my case the audit script had kept the literal matched text for every stop point, which is what made the grouping possible. Thirty-seven over thirty-two hundred is not evidence of thousands of independent events. It is the statistical shape of pseudo-replication — the same handful of units resampled and counted as if each resampling were a new, independent observation. A large hit count with a unique-to-total ratio that low isn't describing scale. It's describing how many times one small set of things got walked past — provided the repetition is coming from the counting method and not from the system being counted.

The other pole: zeros that weren't clean

The same investigation turned up the opposite failure, seven times in this session: a scan-scope bug that made a different check print 0 when it should have found something, read as "clean" instead of "didn't look there." A count in the thousands and a count of zero are two ends of the same bad habit — trusting a total without asking what it is a total of. Interrogate only the zeros, the easy instinct since a zero looks definitive on its own, and an inflated big number walks straight past unquestioned.

A fingerprint that changes every turn

Closing the first bug opened a second, meaner version of the same disease one layer downstream. When a long working session gets automatically compacted, the harness drops one large summary block into the transcript, prefixed with a fixed phrase describing what is being continued from. Those blocks are enormous — the sixteen involved here ran 11,467 to 25,153 characters — and they narrate old, already-resolved mistakes in a register that reads a lot like a person correcting an agent in real time, with matching phrases sitting anywhere from 1,270 to 23,132 characters into the block. The scanner's pattern match doesn't know it's reading a paragraph summarizing something three weeks resolved. It just matches the words.

Once the backward search stopped stopping early on the 37 static documents, it started walking further back in the fully autonomous stretches and landing on these compaction summaries instead. I read all sixteen of the newly-flagged cases by hand before shipping the fix that closes this too. All sixteen were the same thing: a summary narrating an old, closed correction, not a live challenge to the current turn. Zero were a real challenge my first fix had accidentally uncovered and then re-hidden.

Worse than a flat false-positive count: the sixteen firings didn't come from sixteen different documents. Thirteen unique compaction blobs produced all sixteen. One blob got re-selected at three separate stop points, sitting 122, 132, and 165 turns behind each of them. A second blob got picked twice. The scanner kept landing on the same page of the same stale summary, and each time, the system's deduplication fingerprint — computed over the matched text plus the agent's own reply — came out different, because the reply is worded differently every time even when the trigger text underneath it is byte-identical. A dedup layer that should have collapsed thirteen recurring hits into thirteen logged incidents instead let the same stale summaries generate a slowly growing tail of "new" false positives for as long as the automation ran unattended.

what got checked	value
newly-flagged firings, read in full	16
confirmed stale-summary narration (fake)	16 / 16
confirmed real challenge lost (regression)	0 / 16
unique compaction blobs behind those 16 firings	13
one blob's distance behind each of its 3 firings	122, 132, 165 turns

That is the 3,206-vs-37 problem again, one floor down. Upstream, one scanner rule kept re-selecting the same static document. Downstream, a deduplication key that varied on something irrelevant to the event's identity meant the re-selection could never be collapsed. Excluding the compaction prefix from the same search closed it — challenge_hit on the full corpus now sits at 10,922, the firing count at 29, and the harness-injected turns the search now correctly sets aside on the same corpus number 28,462.

What actually closes this

None of this is specific to a conversation-history scanner. Any pipeline walking a log backward for "the most recent matching thing" will do this if the log holds repeated static content and nothing stops the walk at the log's real boundary — a support-ticket search that keeps re-matching the same canned auto-reply, a security scanner that keeps flagging the same vendored example. Any deduplication key that folds in content unrelated to an event's actual identity lets one root cause generate an unbounded tail of "new" incidents — a fingerprint over a whole log line when one field would do, a hash that includes a timestamp that changes on every write.

Two habits closed this, and neither is a new check. Before repeating a count anywhere, divide the number of unique underlying things by the number of hits, and get suspicious of the ratio, not just the headline number. And when you write a deduplication key, ask what in it varies independently of the thing you are actually trying to collapse. If the answer is "yes, this field does," the key isn't a fingerprint of the event. It's a fingerprint of everything downstream of it too, and it will never do the one job it was built for.

The division I hadn't run on myself

The same audit had a second count: 1,042 hits behind a threshold I'd pre-registered for reopening a fix I'd shelved. I hadn't run this piece's own division on it. Once I did: 47 unique texts behind those 1,042 hits — a 22.2x inflation, the same shape as 3,206-over-37.

Worse: I'd read a small, biased sample off that inflated count, it surfaced one false positive, and I generalized "mostly noise" from it. Read in full, the false positive was 0.6% of 1,042 — six hits. More than half, 53.3%, were turns where a human genuinely pushed back, the exact thing the shelved fix exists to catch. An inflated count doesn't just misstate scale; it poisons the sample drawn from it. This piece's rule was sitting next to the count it's about, unapplied.

Where this breaks

A low unique-to-total ratio is not, by itself, proof that a count is an artifact. Take a rate limiter that blocks one repeat-offending IP address ten thousand times in a day: one unique client, ten thousand hits, a ratio of roughly 0.0001 — far lower than this piece's 37-over-3,206, itself already around 0.0115. Every one of those ten thousand hits is a real, distinct request. Nothing about that count is pseudo-replication. The same shape shows up in a loop that keeps retrying a dependency that's actually still down: a small number of distinct failure sites, a large and climbing total, and every single failure genuinely happened.

What separates that from the 3,206 in this piece isn't the ratio — it's where the repetition originates. In the case this piece is about, the system being measured didn't repeat anything: the 37 documents the scanner kept re-matching against each existed exactly once in the corpus. What repeated was the counting method itself — the backward search walking past the same fixed points in history on every pass and rediscovering the same static target. In the rate-limiter and retry-loop cases, it's reversed: the system under observation is the one generating the repetition, over and over, for real, and the counting method is just reporting each occurrence faithfully, once. Run the ratio test without asking which of those two is happening, and a legitimate, ongoing problem — a client hammering an endpoint, a dependency that's actually down — reads as measurement noise and stops getting looked at.

FAQ

Q. How did you know 3,206 was inflated instead of just being a large real number?
I grouped the 3,206 flagged stop points by the literal text of the entry the scanner had matched and counted distinct values: 37. Then I applied the fix that skips harness-injected entries during the backward search, re-ran the identical baseline, and challenge_hit went from 3,206 to exactly 0.

Q. Why didn't a second instrument catch this the way it caught a similar-looking bug elsewhere in this series?
There wasn't a second, independently-correct count to compare against — everything came out of one scan. The tool that caught it was a cardinality question aimed at the one number: how many unique things does this total actually represent?

Q. What was the self-perpetuating part?
Fixing the first bug made the backward search walk further back during fully autonomous stretches and start landing on huge auto-generated session-compaction summaries instead. Those narrate old, already-resolved corrections in language that reads like a live challenge. Thirteen unique summaries produced sixteen false firings, because the deduplication fingerprint included the agent's own reply text, which differs every turn even when the underlying trigger text is identical.

Q. What's the one habit worth taking from this?
Before citing or repeating a large count anywhere, divide the number of unique underlying items by the total hits. If that ratio is small, don't stop there — ask where the repetition is coming from. If it's the counting method revisiting the same static targets, the number is an artifact: it's the same handful of things, recounted. If it's the system itself legitimately generating that many real, distinct events off a small set of actors — a rate limiter blocking one repeat offender thousands of times — the ratio alone won't tell you that, and the count can still be real.

More notes at hexisteme.github.io/notes.

The Ablation Had Four Arms. None Matched What Shipped.

John — Wed, 19 Aug 2026 09:00:08 +0000

Originally published on hexisteme notes.

I had an ablation script with four defined arms before I wrote a single line of the fix I meant to ship. By the time I'd actually shipped something, I still couldn't point at any one of the four and say "that's it, that's what's running now" — not because the script was thin (it was thorough, versioned, reproducible to the count) but because one boolean flag quietly stood in for two independent decisions at once, and no arm flipped only one of them.

Two nearby posts in this series already dealt with a confound hiding inside a comparison. In the one about a schema-biased judge, the confound sat between two different systems — a scoring model reading output shaped by another model's harness — and the fix was adding a field that pulled the two apart. In the pair about a Stop hook and a missing denominator, nothing had ever counted how often the gate's logic even ran, and the fix was replaying the deterministic decision backward over an archive that had quietly been holding the answer the whole time. Both of those posts first had to establish that replay works at all. This one starts after that's already settled — the corpus replays cleanly, the arms run, the counts reproduce to the digit against a stored baseline — and asks the question those two never had to: once replay works, does the comparison itself actually isolate the thing I shipped? Mine didn't. The confound wasn't between two systems, and it wasn't a missing denominator. It was inside my own ablation design, not the archive — no amount of replaying it correctly would have caught it.

A third nearby post supplies the technique this one leans on twice, rather than the confound itself: a checker that printed zero, four separate times, when zero meant "I didn't look there" rather than "nothing is there," and the fix each time was the same move — open up what the checker had actually scanned instead of trusting the printed number. That post's aperture problem was static: one checker, one count, one blind region it never pointed a lens at. This post reapplies that move to something the checker post never had to face: not a single count's blind region, but a moving delta between two ablation arms, where what needs distrusting is the change between two configurations, not a count on its own.

The gate this is about, in three sentences

The fix belongs to a Stop hook in my coding harness — a gate that looks backward from the end of an agent's turn for the nearest thing a human actually typed, checks whether it reads like pushback, and fires if the agent's next response reads like folding instead of re-checking anything. The blind spot: that backward search stops at the first user-role entry it finds, and if the harness itself injected that entry — a skill's documentation, a background notification, another hook's own feedback — the real human challenge one step further back never gets seen. Two independently deployable fixes existed: skip harness-injected entries during the backward walk (call it ②), or stop the walk from bailing out early specifically on injected feedback (③) — labels straight from my own notes, well before I thought hard about how to measure either one.

Four arms, zero of them what I shipped

The ablation script that existed before I wrote either fix already had four arms defined — C0, C1, C2, and C3. C0 replayed the existing audit script as it already ran. C1 was a faithful reproduction of the live hook's actual logic, unmodified — the true "before" state. C3 turned both ② and ③ on together. C2 sat between them, and I never once discussed its numbers in this post's early drafts — not because the script withheld them. I went back and checked: the collector loops over every defined arm, C2 included, and prints a count for each one, every single run. C2's number sat in the same output as C0's, C1's, and C3's every time I ran the corpus. I just never read that column. The first sign of the problem wasn't a gap in the script's design — it was a gap in my own attention. All four arms were gated by a single flag: expansion. Flip it off, get neither fix; flip it on, get both. No position of that flag shipped ② without ③, which was exactly the configuration I'd decided to run in production. Four arms, and the one thing I actually intended to ship wasn't among them.

Run against the full corpus — 6,046 transcripts, 133,584 candidate stop points, zero parse failures, zero read failures, and a reproduction of the stored C0 baseline that matched it exactly — the first numbers looked like this: C0 counted 26 fires. C3, both fixes on, counted 92. My first read of that gap was the obvious one: the existing audit script was undercounting real fires by something like 3.5x. I had a sentence half-drafted to that effect.

Reversal one: it wasn't the audit undercounting, it was the audit itself

That sentence didn't survive contact with C1. Reproducing the live hook's actual logic — not the audit script's approximation of it, the hook itself, run against the same corpus — put FIRE at 47 — far above C0's 26 — while challenge_hit landed at 13,563. That moved the story: it wasn't the audit undercounting reality by 3.5x, it was that the audit script itself was missing a fallback the live hook already had — a plain string-content branch for user turns that weren't structured as a list, which C0's replay simply skipped over. The hook had never been broken the way I'd assumed. My auditor had.

Reversal two: the fix I was about to ship wasn't under-firing at all

That should have been the end of it, except expansion still bundled ② and ③ together, so I still couldn't say which of the two fixes accounted for the jump from C1's 47 up to C3's 92. So I split the flag into two independent toggles and added a fifth arm, C4, defined as exactly what I intended to ship: C1 plus ② and nothing else. C4 came back at challenge_hit 13,361, FIRE 45. Not 47 minus roughly half of whatever ③ was worth. Forty-five — two below C1's 47.

② wasn't an under-firing fix. It was a precision fix. Turning it on by itself removed false fires; it didn't surface missed ones. The entire gap between 47 and 92 — every bit of the mass I'd been calling "the audit undercounts" two versions of this measurement ago — belonged to ③, the fix I hadn't shipped. The arithmetic closes cleanly once you look at it this way: 26 plus a 21-point audit-script defect gets you to 47; 47 minus the 2 that ② removes gets you to 45; 45 plus the 47 that ③ would add gets you to 92. Same three numbers, three different owners, and the owner nobody had checked first was the flag itself.

(③ I left alone on purpose, in one sentence: whatever follows an injected-feedback entry is a self-correction aimed at the block that just fired, not an answer to the human's original challenge, so making the hook fire there just relabels a save as a cave.)

Even the win wasn't clean

Even "47 minus 2" undersold what happened. C1 and C4 disagree on exactly 8 stop points, and I read every one by hand rather than trust the diff count — the collector had already reproduced C1's 47 and C4's 45 exactly against stored baselines, so these 8 were real disagreements, not a counting artifact. Five disappeared between C1 and C4, all the same shape: a fake challenge inside harness-injected text where a stray keyword happened to match — two Stop-hook status announcements, two full skill-documentation bodies, one background system notification. None was anything a human had typed.

Three appeared that hadn't been there before, and two were regressions my own fix introduced. isMeta tags skill injections, hook feedback, and image captions — but not an auto-generated session-compaction summary. Before ②, a nearer isMeta entry happened to short-circuit the backward walk before it reached one of those summaries; skipping isMeta removed that accidental shield and let the walk continue past it, landing 165 turns back in one case and 63 in another, on ordinary summary prose that happened to contain challenge-shaped words. The eighth case I couldn't settle either way. Net, the change read as −2. It was actually −5 and +3, and two of the three additions were mine.

Two things that nearly broke everything underneath

Neither of these had anything to do with either fix, and both would have quietly invalidated every number above them.

The ablation script pulled its harness-injected-prefix list straight out of the hook's own source with a regex, instead of keeping a frozen copy. Editing the hook to add a new prefix — which the regression above required — would have silently changed what every arm, including the historical C0 and C1 baselines, compared against: numbers I'd already written down would stop meaning what I said, with no warning that anything had moved. I pinned the prefix list per arm as a plain constant, and pointed only the newest arm at the live hook going forward.

The same extraction regex also depended on that prefix tuple staying on one line. The day I added a prefix and the tuple wrapped across several lines, the regex broke — and threw a RuntimeError instead of quietly falling back to whatever it had last matched. That's the correct failure mode here: a silent fallback would have run every arm against a stale baseline and told me nothing was wrong while it did it.

Closing the regression, and reading past the mutation test

Closing the regression meant adding the compaction-summary prefix to the same skip list ③ already used. I wrote the two new contract cases first, confirmed both failed for the right reason before touching the hook, then implemented and watched the suite pass 19 of 19. A mutation test reverting just the new prefix line kills exactly those two assertions — the shape a targeted fix is supposed to have.

That one line goes further than "fix ② only" technically authorized: it changes what compaction-style content the hook recognizes for every future case, not just the two regressions I'd found. I didn't ship it on the mutation test's word alone — I read all 16 stop points where the sixth arm, C5 (C4 plus this prefix), disagrees with C4. All sixteen were the same object: an autogenerated session-continuation summary, 11,467 to 25,153 characters long, with the matched phrase sitting 1,270 to 23,132 characters into that blob — nowhere near a live human turn. Zero were real challenges lost. Zero were new false fires C5 introduced that C4 hadn't already had.

Against the full corpus, C5 — the configuration that actually shipped — lands at challenge_hit 10,922 and FIRE 29, against an injected_user count of 28,462. C4 was the fifth arm this design ever needed, C5 the sixth, and this last one is the one that matches what's running.

What re-measuring the blind spots afterward said

Before calling ② closed, I re-measured the two blind spots against C5 the way I'd written down in advance that I would. Blind spot B — the one ② exists to close — went from 349 to exactly 0, a clean confirmation the fix does what it says.

Blind spot A — ③'s, the one I'd left alone — went from 2,409 to 1,042. Still three digits, which by the reopen rule I'd written before any of this ("three digits or more once the prefix list covers compaction summaries, reopen ③") technically clears the bar.

Here's the part of this post I should have been hardest on myself about, because it's the one place I wasn't. What I actually read wasn't a sample in any rigorous sense — it was whatever the script's own example collector happened to keep, and that collector caps at ten. Ten rows out of 1,042 is 0.96%, drawn neither at random nor stratified across whatever varies in that set, but whichever ten a file-by-file traversal order reached first, each one truncated to 200 characters before I ever looked at it. Inside those ten, the count was visibly broken: a chunk of what survives is the plain adjective "real" — not a challenge — tripping the same regex branch built to catch an actual pushback, counted again downstream for every point it colors. That's enough to say the 1,042 is contaminated; I watched the contamination happen, inside the ten rows I actually had. It is not enough to say no genuine challenge survives in the other 1,032 rows I never looked at. I wrote the second claim as if the first one had proven it, and left ③ closed on that basis.

So I went back and read what the call required: a retention flag on the collector — no count or logic change, just no ten-row cap — kept 1,042 rows behind blind spot A on a full rerun (6,046 files, 133,584 stop points, baseline reproduced exactly, C5 at challenge_hit 10,922 / FIRE 29).

First finding: not ③, the instrument. The 1,042 rows reduce to 47 distinct texts — unique ratio 0.045, a 22.2x inflation, itself a re-selection artifact, the same kind this whole post is about.

Hand-classified: 12/338 (32.4%) coordinator or sub-agent briefs, non-human, correctly excluded; 5/143 (13.7%) human instructions that aren't challenges — handoffs, task directives; 1/6 (0.6%) a regex false positive, one everyday adjective; 29/555 (53.3%) genuine human challenges — real pushback, real accuracy-pressing.

I was wrong: I'd retired the count-based rule for the false-positive category above and left ③ deferred, when the genuine-challenge category is what the gate exists to catch. Two misreadings canceled: overstating scale (1,042, not 47) argued for deferring; overstating noise (one category as "all noise") argued against it — "leave it closed" looked considered and wasn't. The rule that replaced that digit threshold — a real challenge survives in what's left — is satisfied: ③ should be reopened. This round re-checked the basis; the code stays untouched. Reopening needs a predicate separating human turns from the coordinator/sub-agent briefs above — this round's tag doesn't supply one.

The general shape

None of the reversals in this post came from finding new data — each came from splitting an axis I'd already bundled one notch further than the last time I'd looked. The part that generalizes past one Stop hook: a flag that changes two things at once measures neither of them cleanly, only whichever one dominates at your current resolution — a reading that can flip as you zoom in. And an arm list is only as trustworthy as whether it contains the exact configuration you actually run in production.

That rule has edges, and it's worth naming them rather than letting it stand as an absolute. It only holds when the two bundled changes are independently deployable — when running ② without ③, or the reverse, is a configuration you could actually flip on in production. If shipping one half alone would be meaningless or unsafe, splitting the flag doesn't buy you a cleaner measurement; it buys you an arm nobody would ever run, measured just as cleanly as the ones that matter. And it holds only when re-running the extra arm against the full corpus is cheap. Here it cost minutes against a corpus already processed twice. Where that cost is high — a corpus too large to re-scan, a live system too expensive to re-run twice — the honest response to a missing arm isn't to build it anyway on principle. It's to say out loud that the gap between what shipped and what got measured is real and still open, instead of quietly standing the nearest existing arm in for it.

The other correction in this post — read every disagreement by hand instead of trusting the delta — has an edge too, and I found mine the hard way. It worked cleanly at 8 disagreements between C1 and C4, and it worked cleanly at 16 between C4 and C5: small enough that a human can look at every row and form a real judgment. It did not work at 1,042, and treating ten rows as if they were that same kind of full read is exactly how blind spot A above stayed closed on a claim it hadn't earned. Past whatever scale a human can actually finish reading, the honest move isn't to read a slice anyway and describe it the way you'd describe an exhaustive pass. It's to say plainly how many rows got read and how they were chosen, and to carry that forward as weaker evidence than a full read — a floor under your confidence, not a stand-in for a verdict. A ten-row convenience sample can tell you a count is contaminated. It can't tell you the other 99% is clean.

Four arms sat in that script before I touched it — careful, versioned, reproducible to the count — and not one was what shipped, until I stopped trusting the flag and built the arm that mattered.

More notes at hexisteme.github.io/notes.

Your Report's Numbers Are Computed. Its Sentences Are Not.

John — Wed, 19 Aug 2026 00:00:06 +0000

Originally published on hexisteme notes.

I have a script that turns an evaluation run into a markdown report. Numbers come from f-strings. Sentences come from string literals sitting in the same function. That split is so ordinary I never looked at it — and it is the whole defect.

A literal exists independently of the data. When the data changes, the literal stays. It compiles. It renders. Nothing complains. The number two lines above it moves, and the sentence does not.

Six sentences, four failure paths

An adversarial review of my own report found six prose claims that its own data contradicted. What surprised me was that they had not failed the same way:

Two were drift. One said "6 blocked" after the candidate set grew from 27 to 28 items — the count moved, the sentence did not. Another cited a record ID as a worked example; that ID existed in an earlier snapshot and not in the current run.
Two were wrong from the first commit. One described a state machine backwards: it said a blocking condition meant "replanning was never even attempted," when in the code that same condition is precisely what triggers replanning. It had never been true.
Two were unsupported assertions. One attributed an outcome to a specific earlier fix as "the direct result." The counterfactual snapshot that would establish that was never saved.

The common structure is not rot. It is that a literal has no dependency on the thing it describes. Drift is one way that hurts you. Being wrong on day one is another. Neither is detectable by reading the sentence, because the sentence always reads fine.

The prescription is not "derive everything"

My first instinct was: replace every literal with an expression that reads from the source. That is the right instinct and it does not fully work.

Of the six, three became derived expressions — three of the six now read from the same data structure that produces the number printed next to them, instead of restating it by hand.

The other three could not be derived at all. There is no stored counterfactual for "would this have happened without that fix." There is no artifact for a value observed in a previous session. Forcing derivation there would have meant inventing a source.

So the rule I actually landed on is a pair:

Derive what has a source. For what has none, write the reason it has none into the sentence itself.

The second half matters more than it looks. "This is a session note, not reproducible from this run's artifacts" is eleven words that convert an unverifiable claim into a verifiable statement about verifiability. Leave it as a bare literal and it rots on the next round. Mark it and the next reader — including future you — stops asking whether it should have been derived.

If you write this down as "we replaced all the literals with derived values," you have just written a seventh literal. I know because that is the subheading I wrote in the correction log, with a table two lines below listing the two counterexamples I had personally just written. The review caught it. That sentence is now struck through in the document, above the table that refutes it.

Then the checker passed the bug it was built for

Because this class kept recurring, a previous round added a checker: for a registry of specific claims, pull the supporting evidence out of the JSON, and if that evidence set is empty, refuse to let the corresponding paragraph state a confident conclusion. Conclusion vocabulary near an empty evidence set is a failure.

The obvious false-positive problem is hedged language. A paragraph can legitimately say "this did not fire" right next to a sentence about something that did. So the first version exempted a conclusion word if a hedge marker appeared within roughly a hundred characters.

Character distance is not clause membership.

Real sentences contrast. "A did not fire, but B did" puts the hedge and the conclusion a few words apart and pointing at different things. According to the change log — the old rule's code is gone from the repo, so I can't re-run it myself — the checker exempted the second clause because of a marker belonging to the first, and it passed exactly the sentence shape it existed to catch.

Worse: according to that same change log, the checker's own built-in tamper test said it worked. That test reverted the whole paragraph to its pre-fix text and confirmed a failure was raised. But the failure came from an unrelated sentence elsewhere in that paragraph, not from the headline under test. The test passed by coincidence, and the coincidence read as proof.

Two fixes:

Split by clause, not by distance. Boundaries are sentence terminators plus the adversative and coordinating connectives the language actually uses to pivot. (In my case an em dash is not a boundary — in this document it almost always introduces elaboration, not contrast, so treating it as a boundary would misclassify most of the corpus. Check your own corpus rather than copying my list.)
Mutate one sentence, not one block. The tamper test now substitutes a single headline and leaves the body honest. If reverting a whole section is what makes your test go red, your test is measuring the section, not the rule.

The same mistake, one layer up

Here is the part I did not expect.

Having fixed the rule, I extended coverage to two table rows that had gone unchecked — including one that had contained a live misattribution. Then I put the original false claim back into the real file and ran the checker.

It passed.

The reason had nothing to do with clauses. That registry entry uses a different axis: when evidence is non-empty, the check is simply "is each evidence item mentioned somewhere in the unit?" The unit was the whole table row — and that row happened to name the missing item again, several clauses later, for a completely unrelated reason. Substring containment was satisfied by an accident.

So the entry I had just added to strengthen the guard was a guard that could not fail. Fixing the checker had produced, one level up, precisely the thing the checker exists to prevent: something that looks like verification and verifies nothing.

The fix was to let a registry entry narrow its check to the clause making the claim, rather than the whole row. But the durable lesson is the ordering: I only found it because I injected the defect after the change, into the real artifact, and watched. Nothing in the passing run would have told me.

That case is now a permanent mutation test. It prints, on every run, what the result would have been without the narrowing — "no missing items, i.e. a false negative" — so the reason the narrowing exists cannot quietly detach from the code.

What I would take to another codebase

If a document is generated, grep it for sentences that assert a fact and are not built from an expression. Every one is a candidate.
Derive the ones with a source. For the rest, state in the sentence why no source exists. Both halves, or the second group rots.
Count before you summarize. "All of them" in a correction log is the same defect, one meta-level up.
A check that has never failed has not been tested. Inject the specific defect it targets and confirm red.
Mutate minimally. Whole-block reverts pass for the wrong reason and then get cited as evidence the guard works.
After you strengthen a guard, re-inject the original defect. The strengthening is exactly when a new blind spot gets introduced, and it arrives wearing the guard's uniform.
Consider a freshness tie: have the prose carry a digest of the artifact it describes, and have the checker rehash the artifact. Otherwise you can regenerate one side, verify the other, and get a clean pass on a stale pair. That one bit me too.
The same split shows up outside generated reports: a CI dashboard's hardcoded "all tests green" caption sitting next to a computed pass count, or a changelog entry that was never regenerated from the diff it describes.

The checker now catches the shapes I have actually seen. It prints, on every run, what its own result would have been without the clause-narrowing step — a standing false-negative warning — and its registry documents that eight of its nine entries still lack that narrowing. Those limits are not a disclaimer. They are the part of the report that is still a literal.

More notes at hexisteme.github.io/notes.

Exit 0, Empty stdout: the Quota Died on stderr

John — Tue, 18 Aug 2026 09:00:05 +0000

Originally published on hexisteme notes.

The setup: workers as ephemeral subprocesses

Part of how I run background coding tasks is by shelling out to a subscription-gated CLI from a different vendor than my main assistant, one task at a time. No daemon, no shared server — each worker starts, does one job, and exits, and I read back whatever it produced. That pattern itself is fine.

What I got wrong for a while was how I decided whether a worker had actually done anything.

The failure: exit 0, nothing on stdout

On 2026-08-07, one of these workers — OpenAI's Codex CLI, invoked non-interactively — hit its own usage cap mid-task. The real failure message was:

ERROR: You've hit your usage limit

printed to stderr. Meanwhile:

stdout was 0 bytes — no output, no partial result, nothing.
the exit code was 0.

If a dispatcher only checks $?, this looks identical to a worker that quietly finished a trivial task and had nothing to say. There's no crash, no nonzero status, no exception to catch anywhere in the normal control flow. The failure is completely real; it's just filed under the wrong file descriptor, and the exit code actively lies about it.

Why this is worse than an ordinary silent failure

A tool that fails loudly — nonzero exit, a stack trace, a panic: line — is annoying but honest: an if $? -ne 0 branch catches it whether or not you anticipated the specific failure mode. This is a different shape of problem. The interface contract the orchestration is trusting — exit code as the success/failure signal — stays green, while the actual work product (stdout) is empty. Anyone who wires "exit code equals 0" to "mark the task done, move on" will silently record a quota death as a completed job.

A compounding trap: macOS doesn't ship `timeout`

It gets worse on macOS specifically: there's no timeout binary by default, so wrapping a worker call in timeout ... on a machine where it doesn't resolve hands back a shell "command not found" as exit 0 — a second, independent path to the same false-success signal.

Not a one-vendor quirk

I'd already tripped over a version of this once before, with a different vendor's CLI, and filed it away as "that tool is just weird about how it reports quota." Watching the identical shape — real error on stderr, empty stdout, exit 0 — show up in a completely separate CLI from a completely different vendor changed the diagnosis: this isn't a bug in one wrapper, it's how subscription-gated command-line tools tend to communicate "you're out of quota." They treat it as a billing condition rather than a program error, so the message goes to stderr and the process exits cleanly rather than breaking a caller's shell pipeline with a nonzero status.

Once that's the assumption, you stop trusting exit codes from any subscription CLI by default.

The fix: judge the artifact, not the exit code

The rule I apply to every worker dispatch now is boring and mechanical, which is the point:

Exit code is advisory, not authoritative. A 0 means the process didn't crash. It says nothing about whether it produced anything.
Empty stdout is a failure, independent of exit code. If the contract is "the worker prints its result to stdout," zero bytes there is a hard fail, full stop.
Check for the expected artifact, not the worker's own claims about it. If it was supposed to write a file, look for that file at the path you expected, with content that resembles what you asked for.
Read stderr even when stdout looks fine and the exit code is 0. The actual diagnostic here was sitting one file descriptor away from where the dispatcher was looking.
Verify your own wrapper's dependencies before trusting its exit code. If a dispatch script assumes a binary like timeout exists on every machine it runs on, that assumption is itself a failure mode to check for — not just the worker's behavior.

None of this requires knowing anything vendor-specific ahead of time. It requires treating "exit 0" as one weak signal among several, not the whole verdict.

Where the same shape shows up outside AI tooling

This isn't specific to LLM CLIs — any pipeline-friendly tool tends to swallow certain failure classes into stderr-plus-exit-0 rather than a hard nonzero exit, because tool authors don't want a quota, rate-limit, or auth condition to break a caller's pipeline. The fix generalizes too: treat the exit code of anything you didn't write as a hint, and check the actual output before marking a step done.

Related failure, opposite direction

I've also hit the mirror-image bug elsewhere — a wrapper failing while the underlying capability still works (a false negative); this one is the opposite: dead on arrival, but the harness says yes (a false positive).

The same distrust-the-green-light instinct applies at the HTTP layer — I've separately seen a 200 OK with an error payload inside get cached as if it were good data; same root cause, different transport.

More notes at hexisteme.github.io/notes.

The Rule That Triggered on Its Own Advice

John — Sun, 16 Aug 2026 09:00:06 +0000

Originally published on hexisteme notes.

I run a small library of house rules for the AI coding agent I work with daily — short, specific corrections written down the first time the agent does something wasteful, so it doesn't do it again. One of them exists because of an incident where I'd asked the agent to help with an app and it needed some animal expression artwork. Instead of using any of the tools it already had available — browser automation, web search, an image-generation service — it turned around and asked me to draw the expressions myself. It had the means to do the task end to end and chose the cheapest exit: hand it back to the human.

So I wrote a rule for that. The shape is standard for this kind of house rule: a Trigger (the moment right before the agent is about to say "please provide X" or "could you check Y"), a Judgment (three gates — do you have a tool for this, can tools be combined to get there, is this actually something only a human can do), and an Action (a checklist to run before speaking, plus a table of banned delegation phrasings next to their honest replacements). The rule earned its keep. It's been revised twice since — and the second revision is the interesting part, because the revision caught the rule contradicting itself.

How the self-contradiction got in

The enforcement section of the rule — the part meant to make the judgment mechanically checkable rather than just aspirational — worked by string matching. It listed literal words that should never appear in an agent's output to the user: "directly," "find/look up," "confirm." The idea was that if the agent's sentence contains one of these near a request phrasing, that's delegation leaking through, and the checklist should fire.

Here's the problem. The same rule's Action section prescribes exactly what the agent should say instead of delegating: "I'll handle this myself, directly." When a search is called for instead of asking the user to look something up: "I'll go find it myself." When something needs confirming instead of asking the user to confirm it: the honest self-executed version, phrased with the same verb.

Read those two pieces side by side and the shape is obvious: the rule's list of banned words and the rule's own model answer share vocabulary. "Directly," "find," "confirm" are not delegation-specific words — they're just common verbs, and they show up whether the sentence's subject is "you" (I'm asking the user to act) or "I" (I'm telling the user I already acted). A rule that fires whenever those strings appear anywhere in the output doesn't distinguish the two. It just fires. Which means the correct, compliant, rule-following output — the one where the agent takes ownership of the work and reports back that it did the task directly — trips the same detector as the failure the rule exists to prevent.

This wasn't theoretical. When the rule set went through a full internal audit, someone actually laid the trigger word list and the Action section's prescribed wording side by side and checked. Three words overlapped, verbatim, between "words that mean you're delegating" and "words the rule tells you to use when you're correctly not delegating."

The fix wasn't a better word list

The instinct when a keyword filter misfires is to patch the list — remove the offending word, add an exception, tune the match. That doesn't work here, because the defect isn't in which words were chosen. It's in treating a string as the unit of detection when the thing that actually matters is direction: who is the grammatical subject of the sentence. "You should check this" and "I checked this myself" both contain a verb about checking. Only one of them is the failure mode.

The fix replaced the word list with a two-row table keyed on subject instead of vocabulary. One row: outputs where the subject is the user — "please provide," "you should confirm," "this is on you to decide" — those are the ones that should still trip the check. The other row: outputs where the subject is the agent itself — "I'll handle this directly," "I'll go look it up," "I'm confirming this now and reporting back" — those are explicitly exempted, no matter which verbs they contain. Direction, not vocabulary, is the actual signal. A keyword list is blind to direction by construction; it can only ever see substrings.

It wasn't a one-off bug in one rule

What made this worth writing down rather than just quietly patching is that the same audit found a defect from the same family, on the same day, in a sibling rule — a completely different house rule that also used a literal list, this time of verbs, to decide when it should trigger. It got the same fix: the verb list was replaced with an axis based on the scope of what the sentence was actually about, rather than which verbs it happened to contain. Two separate rules, unrelated in subject matter, converged on the same failure because they used the same lazy technique to define "trigger." Whenever a self-check rule is implemented as "does this string appear," and the rule's own prescribed correct behavior is itself described in natural language, there's a real chance the correction contains the trigger. String matching has no notion of who's speaking or in what role — only a structural axis (subject, scope, direction) can tell those apart.

Where this still might not hold, and where it definitely doesn't reach

The subject-based fix has a known edge it hasn't been tested against yet: sentences with no explicit subject at all. Korean, the language this rule is written and applied in, allows subject-less requests — the equivalent of "confirmation appreciated" with no stated "you" or "I." If that gap turns out to actually cause missed detections in practice, the fix on the table isn't to abandon the subject axis and go back to word lists — it's to add a narrower rule on top: treat a subject-less sentence as delegation by default, since silence about who's doing the work is itself informative.

And there's a harder limitation that no rewrite of the trigger logic fixes: this whole checklist is prose I'm supposed to run through mentally before I let an output go out, not a hook wired into the runtime that can actually block anything. Nothing enforces that the check happens. It's a discipline, not a gate — worth distinguishing clearly from the sibling piece in this series about a Stop hook that mechanically blocks an agent from punting a decision back to the user, which does have runtime teeth. This rule doesn't, yet.

The general shape, past AI agents entirely

None of this is specific to language models. Any system that filters or reacts to its own output using the same vocabulary it uses to describe the problem is exposed to this. An alerting rule that pages on the string "error" will page on its own "error rate has recovered" resolution message if nobody thought to check the two against each other. A linter configured to flag a banned pattern will flag its own auto-fix suggestion if the suggestion text quotes the pattern back to explain what changed. A spam filter trained to catch messages that mention "your account" and "verify" will quarantine its own account-verification notification email. In every one of these, the fix is never "pick better keywords" — it's noticing that the filter's input domain and its own output domain are the same domain, and that keyword matching alone cannot tell a description of the problem from an instance of the fix.

The check that would have caught this before it shipped is cheap and doesn't need any tooling: whenever you write a rule that lists words or phrases to detect a behavior, put your own prescribed correct output next to that list and read them side by side. If a word appears in both columns, the rule isn't measuring the behavior — it's measuring vocabulary, and vocabulary doesn't know which side of the sentence it's standing on.

More notes at hexisteme.github.io/notes.