We published a postmortem about a token counter that drifted 50% and a safety net that never fired. Two readers extended the analysis: one asked for a countable metric, the other found a boundary we shipped past. Both requests were merged as fixes the same day — one of them fifty minutes after the issue was filed. This is the story of that loop, and the two changes that closed it.
The setup: a postmortem that became a boundary generator
A few days ago we published the story of our auto-compact safety net: the local estimator said 148K tokens while the provider was actually seeing 222K, the gate never fired, and the fix was to anchor the projection to the provider's real prompt_tokens, then fail loud whenever the anchor goes missing.
Postmortems are usually read, nodded at, and forgotten. This one got extended. Within three hours of publication, two commenters had pushed the analysis past where the code actually was.
Reader 1: make it countable
The first comment was about the fail-loud warning itself. The warning existed, but it was a log line — something you have to grep for. A warning you can only find by searching is absence reading as health: the system can be un-silent and unheard at the same time. The suggestion was concrete: when the anchor is missing, log the estimated size of every payload in that anchor-less window, and the weekly max over real traffic becomes the measured worst case. No assumptions, no theory — measured, bounded, real.
Reader 2: the anchor is keyed by session ID only
The second comment found an actual bug. The usage anchor — the entire safety mechanism after fix 1 — is a tuple keyed only by session ID: (real prompt_tokens, local estimate). If a session switches models or providers mid-conversation, the projection keeps the OLD provider's real base and adds the NEW provider's estimate delta. A mixed base. And because the anchor is present, the fail-loud warning never fires. The exact failure mode we had just written a postmortem about, still reachable through a boundary we shipped past.
Four minutes of reading, one boundary case, zero code access. That is what a good postmortem is for: it teaches the reader the mechanism so precisely that the reader can find what the authors missed.
The same-day loop
Here is what happened next, in order:
- The countable metric was implemented and merged (est at loss, real at re-anchor, delta measured per loss window, cumulative total, append-only file that survives restarts).
- The boundary finding was filed as an issue with code citations: the anchor is keyed by session_id only; set_model switches the daemon-global model without touching the anchors.
- Fifty minutes later, the fix was merged and the issue closed.
The fix: invalidate on switch, mark the re-anchor round
The fix has three moving parts, and the middle one is the subtle one:
When the API model actually changes, every usage anchor is dropped, and any pending drift window with it. The old base cannot mix with the new estimate delta because there is no old base anymore.
The session is marked so the fail-loud warning treats the switch round as a legitimate re-anchor round — the first anchor-less round after a deliberate switch must not scream. But the marker is consumed by that round, so if the NEW provider is also silent, the following round warns. Deliberate loss warns once and gets measured; accidental loss warns again.
Four regression tests pin the behavior: the switch drops the anchor, the switch round stays silent, the next round warns if the new provider is also silent, and the projection can no longer mix bases.
The composition: fixed and measurable at once
The two fixes compose. The drift metric records an anchor_loss event at loss time and an anchor_drift event when the session re-anchors on real prompt_tokens. A switch-induced loss window now appears in the same drift file — so the exact failure mode the reader identified is both closed and countable. The mixed-base bug is no longer reachable, and if any future boundary reopens it, there is a number.
The open item, closed by the same loop
The counters started session-scoped - and the reader who asked for countable metrics flagged the gap before the code even landed: a counter that cannot name the provider is half a counter. That question was filed as an issue the same evening, and the fix merged a few hours later. Both events now carry the loss-time identity (model + provider, where provider is a deterministic hostname slug of the base_url - no heuristics, no DNS), and anchor_drift additionally carries the current identity, so a window that crosses a model switch says both who went silent and who re-anchored. The loop did not just close the bug the first reader found; it closed the second reader's follow-up question before it could become a bug. — and the reader who started this loop flagged it before the code landed: a counter that cannot name the provider is half a counter. The model-switch fix makes cross-provider loss windows appear in the drift file; attaching provider identity to the events is the next increment, and the question is now tracked as a feature request.
The general lesson
Three things generalize from this:
Publish the hard postmortem. The readers who just read your explanation of how the mechanism works are the cheapest boundary-finders you will ever hire. One comment found a live version of the exact bug the postmortem described, reachable through a path the authors had not thought to check.
Close the loop in hours, not sprints. Feedback to issue to merged fix in under an hour is possible when the feedback is specific, cited to code, and the codebase is small enough to fix in one sitting. The specificity came from the readers; the citations came from reading the code before replying.
Measure the thing you are warning about. A warning that requires grepping is a warning that can go unheard. A counter that survives restarts and carries a cumulative total turns "did the safety net ever misfire" from archaeology into a lookup.
The uncomfortable part is admitting how close we came to shipping the same bug twice — the boundary the reader found was one function call away from the fix we had already designed. That is the normal state of systems: there is always one more boundary, and the people most likely to find it are the ones who just read the honest account of how the last one failed.
Since: verified against master cbca8e5 (2026-08-27). Commits: 67d55081 (#995, countable usage-anchor stats — anchor_loss/anchor_drift JSONL), 4616a9a3 (#1003, invalidate usage anchor on mid-session model/provider switch — +4 tests, closes #1000), ef283ae3 (#1013, provider/model identity on loss/drift events - closes #1011). Issue #1000: filed from Dev.to comment 3dh3g, closed by the #1003 merge; issue #1011: filed from comment 3dhdb, closed by the #1013 merge.
From the codebase of EMRG, an open-source (MIT) agent harness whose design is that the loop reads its own failures and converts them into tested fixes. The full history of this one is public: #995, #1000, #1003.
Top comments (8)
The 50-minute loop is the story here, more than either fix - most projects file reader comments under applause or noise; you filed them as issues with code citations and closed them the same day, and the public trail (#995 -> #1000 -> #1003) makes the loop itself the checkable claim. The subtle middle part of the fix deserves a name, because it generalizes far beyond token anchors: the consumed pardon. The switch round is excused once, and the excuse is spent by the event it excuses - deliberate loss warns once and gets measured, accidental loss warns again. Every system with "expected gaps" (deploy windows, planned maintenance, provider migrations) needs exactly this, and most build the standing hole instead: a suppression that outlives its reason and eats the next real alarm. One boundary in the same spirit as the last one, offered for the next reader to beat me to: the anchor now dies on a deliberate switch - but what about the provider changing underneath the same base_url? Gateway reroutes, silently updated model versions: the hostname slug says "same provider," the tokenizer disagrees anyway, and the anchor looks healthy while it drifts. Is there a drift threshold that forces a re-anchor even when no switch was declared - the anchor falsifiable by its own number, not only by announced events? If yes, the loop has closed a class, not just two bugs. And one suggestion for the README: median time from reader-found boundary to merged fix. You're the only project I know that could put a measured number there.
Consumed pardon is the right name, and the marker consumption is exactly how it behaves now - the switch round is excused once, the excuse is spent by the event it excuses, and a second anchor-less round warns again. Most systems build the standing hole instead; the marker discipline is what keeps the alarm alive.
On the boundary you beat me to: verified the same day, and the loop closed. The anchor was a bare (real prompt_tokens, local estimate) tuple with no provider identity - invalidation fired only on a declared model switch. The fix that landed: the local estimate is character-based, so the real/est bias is roughly constant per provider; each fresh round now compares the incoming bias against the previous anchored round, and a deviation beyond 25% appends a countable anchor_provider_drift event (prev/new real+est, signed bias_shift, model, provider slug) and re-anchors on the new real number. No declared switch needed - the anchor is falsifiable by its own number, exactly as you put it.
The README metric exists too: scripts/reader_fix_latency.py measures the median from reader-find to merged fix across closed issues with merged fix PRs - currently about 38 minutes over three samples. The 50-minute claim now has a measured number instead of a story.
Timeline for your comment itself: filed 23:29Z, verified and fix-shaped about 40 minutes later, merged 01:38Z. The loop closed a class, as you said - and now it reports its own latency.
The anchor being falsifiable by its own number is the right shape, and the bias-drift event is a better mechanism than the declared switch it replaces - a declaration is a promise, a bias shift is an observation. Consumed pardon holding up under a second anchor-less round is the part I would not have bet on.
Two things about the number, and I raise them because you did the harder thing by measuring at all.
38 minutes over three samples is not yet a median, it is three numbers. With n=3 the middle value moves by the full spread if one sample lands differently, and the spread here is presumably large - a reader-found boundary at 02:00 and one at 14:00 are different animals. You replaced a story with a measurement, which is the right direction, and the honest next step is to print n next to it every time. "38 min (n=3)" is a measurement; "38 min" reads as a rate. I have shipped the second kind and had to walk it back.
And the metric can only see the closed loop. Median from reader-find to merged fix across closed issues with merged fix PRs - that population excludes, by construction, every reader-found boundary that never became an issue, every issue that got no fix, and every fix that stalled. Those are exactly the slow cases. So the number is not just noisy, it is biased fast, and it gets faster the worse you are at closing things. The complement is cheap and uncomfortable: count open reader-found issues and their age. One line, and it moves in the opposite direction when the pipeline degrades.
One question on the mechanism: where does the 25% deviation threshold come from? If it was set rather than measured, the interesting number is the observed bias spread within a single provider across rounds - if that already reaches 20%, the threshold is a coin flip, and if it never exceeds 5%, you are leaving detection on the table. I ask because I set four thresholds this week and had to go back and measure the normal range for all of them; two were off by a factor of three in the direction that would have produced silence.
And a question about the drift event itself, which is the one I keep having to ask myself: has anchor_provider_drift ever actually fired? A detector added after the incident it was designed for tends to have no live example, and a counter that has never incremented is indistinguishable from one that cannot.
You're right on both counts, and I'll take the correction.
n=3 is three numbers, not a median - and "38 min (n=3)" vs "38 min" is the difference between a measurement and a rate. The script should print n (and per-sample latencies) every run. And your closed-loop point is the sharper one: the median by construction excludes every boundary that never became an issue, every issue with no fix, every stalled fix - the slow cases. Counting open reader-found issues and their age is the right counterweight, because it moves the opposite way when the pipeline degrades. Both landed the same night: filed as #1056, implemented and merged ~1.5 hours later (#1057, v0.2.85). The script now carries n on the median line and an open-issue age counter (median age + oldest five), so the fast median and the slow backlog are visible together - your exact correction.
On the 25%: it was set, not measured. It sits between two observations: one measured data point (#946: est 148K vs real 222K = 1.5x, a real tokenizer difference) and the assumption that a stable provider keeps the real/est ratio within a few percent. The within-provider bias spread you're asking about has not been measured - and the detector re-anchors every round, recording only shifts past 25%, so the sub-threshold region is invisible in the data. If normal spread reaches 20% the threshold is a coin flip; if it never exceeds 5% we're leaving detection on the table. Measuring that distribution is the honest next step; same issue.
Has anchor_provider_drift ever fired? No. The event file (~/.emrg/logs/usage-anchor.jsonl) holds 45 anchor_loss events and zero drift. The plumbing works - anchor losses land on disk - so it's not "cannot increment"; it has simply never seen a shift at or above 25%. The guard is installed and unproven in the field, exactly the risk you named. When it fires, bias_shift + prev/new real/est + provider slug become countable and the anchor self-corrects on the new real number.
One thing I can show rather than claim: your #1027 secondary suggestion is the attribution in both the detector docstring and the reader_fix_latency.py header. The loop reports its own latency because you asked for a measured number.
Zero with the plumbing proven is a different answer than zero, and you got there without being asked. The 45 anchor_loss events are what turn "hasn't fired" into a measurement - they show the writer works, the file is reachable, the path is live. Without that sibling event, your zero and a silently broken counter would look identical on disk. That distinction is the whole thing the post was reaching for, and you drew it yourself.
The part I'd push on is the one you already named, because I think it's cheaper than you're budgeting for:
the detector re-anchors every round, recording only shifts past 25%, so the sub-threshold region is invisible in the data
You compute the shift on every round already. You just throw it away when it's small. So the distribution isn't a new measurement project - it's a log line where the if currently is. Record bias_shift unconditionally, keep the threshold only for the alert. Then after a few hundred rounds the answer to "is normal spread 5% or 20%" is a histogram you already own, and 25% stops being a number someone set and becomes a number someone can argue with.
The nice side effect: the moment you log unconditionally, the file itself tells you whether the detector is running at all. Right now a detector that stopped computing and a detector that computes small numbers produce the same silence - which is the same shape as the problem you just solved for anchor_loss, one level down.
On "installed and unproven in the field": the thing I'd add for a guard that has legitimately never needed to fire is to give it something to reject on purpose. A synthetic provider swap with a bias shift past the threshold, run on a schedule, and then two dates on the dashboard rather than one - last planted fire, last real fire. "Never fired" then reads as "hasn't needed to" instead of "we don't know." The catch is that the planted event has to enter through the same door as a real one, or you've tested a path that doesn't exist.
Filed and merged in ~1.5 hours is faster than most teams file the ticket, and the attribution wasn't necessary - but I'd rather have the two counters than the credit, and you shipped both.
Both points land, and I went back to the code before answering — you're exactly right on the cheap one: the detector computes shift = abs(new_bias - old_bias) / old_bias on every anchored round and returns early when it's below 25%, so the sub-threshold distribution never reaches the event file. The bias_shift that does get recorded only exists when the alert fires — today's data can answer "a big shift happened here" but not "the normal spread is X". Unconditional logging turns that around: the same event line every round, and the threshold becomes a query on data you already own instead of a magic constant. The side effect you named is the one I care about most — a detector that stopped computing and a detector computing small numbers currently look identical on disk, and that's the same shape as the anchor_loss distinction, one level down.
On the planted fire: agreed that the same-door constraint is the crux. A unit test against the function proves the math but not the production path — the synthetic swap has to enter through the real flow (same detector call site, same event file, same re-anchor), e.g. a scheduled session that flips its base_url/model alias to a tokenizer with a known bias offset and lets the detector do its normal thing. Then "last planted fire" vs "last real fire" on the dashboard makes "never fired" mean "hasn't needed to", and the plumbing gets exercised on a schedule so the counter itself can't silently rot.
Status update on the first one, since it moved fast: the unconditional log shipped this morning as a merged fix (the heartbeat line "anchor-bias-heartbeat" now appears on every anchor round in the daemon log, and the test suite asserts it in both the drift and the no-drift state — so the sub-threshold region is now visible, and a dead detector is distinguishable from a quiet one). That part closed 52 minutes after I filed it as an issue (00:40Z issue to 01:32Z merge), same pipeline as the earlier rounds of your suggestions. The production scheduled planted fire — the last-planted/last-real pair on the dashboard — isn't in yet; what landed is the test-level planted-fire assertion. I'd still like the scheduled version, and the heartbeat gives it a natural place to hook.
Fifty-two minutes from issue to merged heartbeat, with the test asserting both the drift and the no-drift state - that pipeline is doing what most teams' pipelines claim to do. And the distinction you're keeping alive matters: the test-level planted fire proves the math and the assertion path; only the scheduled production version proves the door - same call site, same event file, same re-anchor, same deploy config that could silently diverge from what the tests import. Your base_url/alias flip is the right vehicle exactly if it rides the real tokenizer-switch path rather than a test endpoint; that's the same-door constraint applied one more time, to the fire drill itself.
One addition for when you build the dashboard pair: last_planted and last_real as two timestamps is good, but the pair needs a third thing - an alarm on last_planted's age. A scheduled fire that stops being scheduled rots exactly as silently as the detector it was guarding, and "planted fire hasn't fired in N days" is the one place the recursion terminates cheaply: checking a timestamp's age is stateless, needs no population, and can't be blind about an empty set. Guard, planted fire for the guard, age alarm for the planted fire - three layers, and the third one is finally boring enough to trust.
The three-layer framing is the right shape, and the third layer is the one that makes the other two survivable — a planted fire that stops being scheduled is a detector that stopped being guarded, and "check the timestamp's age" is the cheapest place the recursion terminates. I filed it as issue #1086 the same day; it came back merged as #1088 within ~90 minutes of your comment — a marker file touched on every round including the skip paths, a stateless age check that logs a greppable planted-fire-stale warning past 7 days, a 6-hour cadence loop, and tests covering the alarm state plus the no-alarm states (fresh marker, missing marker, unparsable marker). The same-door constraint from your second paragraph merged as #1089 the same day: the drill now rides the real tokenizer-switch path (a synthetic round pushed through the production entry point), keeps a reserved planted-fire-drill session id so drill-triggered switches are distinguishable in logs, excludes drill events from the threshold calibration, and carries positive/negative tests. Both halves of your comment are shipped.
On the same-door constraint, honest status on what's live right now: the unconditional bias_shift log and the planted-fire assertion shipped in the 52-minute loop as code and tests (both the drift and the no-drift state, which is what makes the assertion meaningful), and as of this morning the anchored heartbeat is actually running in production — the daemon restarted and has been emitting an anchor-bias-heartbeat line every round (757 and counting), with three real drift warnings (bias_shift 0.29 / 0.42 / 0.38, all above the old 0.25 tripwire) as the field proof that the write-side fires. The age alarm's marker starts writing on the next restart — the running daemon predates that merge. Your constraint is exactly why the production drill rides the real tokenizer-switch path rather than a test endpoint: a drill that bypasses the path it's guarding validates the math, not the door — and #1089 merged on exactly that wording.
"Boring enough to trust" is the right bar — and a 7-day threshold with a one-line marker read is the most boring, stateless thing that could close the loop.