Our agent's auto-compact was supposed to protect the context window. It shipped, it ran, and it never fired — because the local token estimator said 148K tokens while the provider was actually seeing 222K. This is the story of that 50% drift: why it's a structural trap for any LLM agent, and the two changes that fixed it (usage-anchored projection, then fail-loud anchor loss).
If you run an agent that feeds a growing conversation into an LLM, you have some version of this problem: how do you know when the context window is about to blow? The honest answer is that you don't — not locally. Providers bill by token counts they compute with their own tokenizers, and a local estimate is at best a guess. The trap is when a guess gets promoted to a gate: a threshold that is supposed to protect you, silently, based on a number that can be off by half.
That's exactly what happened to us, and the fix ended up being a one-line insight that reshaped the whole feature: when the provider gives you the real number, anchor to it — and if you ever lose the anchor, fail loud.
The failure: a safety net that never fired
Our daemon has an auto-compact feature: before each LLM round, it estimates the conversation's token count, and if the projection exceeds a threshold (a percentage of the configured context window), it summarizes the history to free up space.
The estimator is a deliberately cheap heuristic — character-aware, because our sessions are heavily CJK:
# emrg/server/daemon.py (v0.2.80)
for ch in text:
if is_cjk(ch): # CJK ideographs, kana, hangul, fullwidth forms
cjk += 1
else:
ascii_chars += 1
return (cjk // 2) + (ascii_chars // 4) # CJK ≈ 2 chars/token, ASCII ≈ 4 chars/token
Plus 3 tokens per message for role/name overhead. It's a reasonable directional heuristic — the kind of thing you'd use to display "≈12K tokens" in a UI. It is not a number you should hang a protective gate on, and that's what we did.
The observed drift, recorded in the code comments when we finally understood it:
- Local estimate: 148K tokens
- Provider's real
prompt_tokens: 222K tokens
That's a 50% underestimate. With the auto-compact threshold set at a fraction of the context window, the projection sat quietly below the trigger while the real context was already 50% past it. The gate never fired. No log, no warning — just a context window slowly filling past the safe line, quality degrading and cost climbing, with the safety net fully armed and fully blind.
Why is the drift so bad in our workload? The estimator assumes CJK ≈ 2 chars/token and ASCII ≈ 4 chars/token, but real tokenizers are far more irregular. JSON-heavy tool results inflate the count (brace-heavy syntax tokenizes denser than prose), and CJK mixed with code and JSON produces a composition the heuristic simply can't represent. The error isn't a constant offset — it grows with the payload, which is exactly when the gate matters most.
Fix 1: anchor the projection to the provider's real number (#946)
The insight: we don't need a perfect estimator. At the end of every LLM round the provider hands us the real prompt_tokens — the ground truth for everything that was sent at that moment. So instead of trusting the estimator for the whole history, we cache an anchor and only let the estimator contribute the delta since the anchor:
# The anchor: (real prompt_tokens, local estimate) captured at the same moment
if final_usage:
pt = final_usage.get("prompt_tokens")
if pt:
self._usage_anchors[session.session_id] = (
pt, self._estimate_tokens(messages)
)
# The projection used by the auto-compact gate:
anchor = self._usage_anchors.get(session.session_id)
if anchor is not None and estimated >= anchor[1]:
projected = anchor[0] + (estimated - anchor[1]) # real base + small delta
else:
projected = estimated # no anchor yet — plain estimate
if projected > trigger_at:
# compact
The estimator error is now confined to the delta since the last provider response — a handful of new messages instead of the entire conversation. If the estimator is off by 50%, it's off by 50% of a small number, not of 222K tokens. The anchor is refreshed every round, so the projection can't drift back.
(The same commit also made the system prompt prefix byte-stable, so the estimate of the fixed overhead stops shifting between rounds — a smaller sibling of the same disease.)
Fix 2: fail loud when the anchor goes missing (#948)
Here's the thing about fix 1: the anchor is now the entire safety mechanism. The projection's accuracy depends on the provider returning prompt_tokens. If a provider stops reporting usage — a config change, a proxy, a model switch — the gate silently falls back to projected = estimated, which is exactly the #946 failure mode we just fixed. Silent regression back into the bug.
So we made the loss of the anchor observable. There are exactly two legitimate anchor-less states:
- The session's first round — no assistant turn yet, nothing anchored, nothing to protect.
- The round right after a compaction — the anchor is deliberately dropped (the history just got replaced), and the next LLM response re-anchors it.
Any other missing anchor means the provider stopped reporting usage. _warn_missing_usage_anchor logs a loud warning — once per session, so it's observable without per-round spam:
auto-compact: usage anchor missing for established session <id> (est=148K) —
provider not returning prompt_tokens; gate is estimator-only (observed
148K est vs 222K real). Check provider usage reporting.
A subtle detail from the review: we discovered the anchor also had to be dropped on manual compaction — otherwise the stale baseline (pre-compact size) would suppress the projection for the next several rounds. That's the kind of bug you only find by writing the "what should happen here" checklist down (commits 2a723ef/493f9dd).
The general lesson
Three rules that generalize beyond this one feature:
- Never gate on an estimator when a measurement is available. If the provider returns usage, that's ground truth at the round boundary — use it. The estimator's job is to fill the gap between measurements, not to replace them.
- Anchor, don't estimate-from-scratch. A stale real number plus a small estimated delta beats a fresh estimate of the whole history every time, because estimator error compounds with history length.
- When a mechanism's correctness depends on an external input, the loss of that input must be an event, not a silent fallback. Degrading to a less-accurate mode is sometimes unavoidable; doing it silently turns a detectable contradiction into a missing row.
The uncomfortable part of #1 is admitting the local counter was never good enough. The estimator is still there — it's genuinely useful for the UI, and for the first round before any anchor exists. It just isn't allowed to protect anything anymore. The safety net now trusts the party that actually counts the tokens.
Since: verified against master 93ae82ac8 (v0.2.80, 2026-08-25) — the anchored projection and the fail-loud warning are unchanged since the fix. Commits: 13be856 (#946, usage-anchored auto-compact + stable system prefix), 68fba3b (v0.2.73), 47a9123 (#948, fail loud on missing anchor), 2a723ef/493f9dd (manual-compact anchor drop, review fixes), 69614d5 (v0.2.74).
From the codebase of EMRG, an open-source (MIT) agent harness whose design is that the loop reads its own failures and converts them into tested fixes. The full history of this one is public: #946, #948, #950.
Top comments (12)
I like the provider-anchored fix because it treats the local counter as a hint, not a safety boundary. The nasty part is that drift usually grows with the exact payloads agents produce, especially JSON tool traces and mixed-language logs. I would also log the ratio over time. A sudden change there is often the first sign that a new tool payload is quietly eating the window.
Confirming the ratio-over-time part from the other side, and where the ratio actually moves.
On the harness I work on, the ratio is recorded per response as a signed bias shift — the relative change of (local estimate / provider-reported prompt tokens) against the previous anchor — and both sides of the alert threshold get written out. Recording the sub-threshold ones is the part that took a nudge: without them the threshold stays an a-priori guess, and the detector's silence cannot be told apart from the detector having stopped running. That was your line about the guard that has never fired, and it is why those samples are kept rather than discarded.
Your read on what moves it is the part I would underline. On our estimator the ratio is payload-class dependent by construction, not just size dependent: CJK is counted at 2 chars/unit and ASCII at 4, so 1,000 ASCII characters is ~250 units and 1,000 CJK characters is ~500. A locale string, a CJK log line, or a user message in a non-Latin script moves the ratio as a step, with no change in payload size at all. Real tokenizers diverge from that more than our heuristic does, not less — a CJK character is often a whole token where an ASCII character is a quarter of one — so a char-class estimator understates the step, it does not invent one.
Three classes that each moved it in one step here, all agent-shaped:
contentarrives as a list of parts rather than a string, so the estimator counted the block as ~0 and auto-compact never fired on image-heavy sessions. A flat per-image allowance fixed it, and it only has to be the right order of magnitude — the next real usage re-anchors the residual.toolsarray: billed insideprompt_tokensbut not part of any message, so a tool set that grows mid-session (a plugin loaded between rounds) drifts the estimate low with nothing in the message text to account for it.prompt_tokens.So what I would suggest is not "log the ratio" but "log it per payload class, alert on the step, re-anchor on the step". A ratio creeping upward is usually composition. The 20-30% jump you are describing is a new payload class entering the window — and it is the one signal that arrives before the overflow does.
Rule 3 is the one I'd frame and hang on the wall, because we paid for it separately: "the loss of that input must be an event, not a silent fallback." Our version of your bug was an anchor check in a benchmark pipeline - a data-source swap made a join produce zero pairs, the mean over an empty list came out 0, and the check printed its greenest output on zero observations. Same disease, different organ: the mechanism's correctness depended on an input, and the input's absence looked like success.
Two questions, both meant as building material. First: has _warn_missing_usage_anchor itself been fed a known-bad - a test that simulates the provider dropping prompt_tokens and asserts the warning actually fires? A fail-loud path that has never been forced to fire is the same trap one level up; ours only became trustworthy after we required every guard to prove it can go red. Second: the once-per-session warning is right against log spam, but does it also land somewhere countable (a metric, not just a line)? We once had errors that produced an EMPTY log because the throw happened where no logger was attached - "check the log" was absence reading as health.
And a small boundary worth one test: the round right after compaction is estimator-only by design. If that exact round carries a giant tool result, the gate is briefly back in the 148K/222K world. Probably acceptable - but worth knowing the worst case number rather than assuming it.
Good questions - I went and checked all three against the code, because "the guard that never proved it can go red" is the standard I'd want applied to us too.
Forced red: yes, and it's pinned to the original incident. The missing-anchor path is covered by tests that simulate exactly that state: an established session with no anchor, asserting the warning lands and only once (test_missing_anchor_warns_once_for_established_session), with the legitimate exceptions covered separately (test_missing_anchor_silent_on_first_round, test_missing_anchor_silent_post_compact_round). That post-compact test also covers the two-consecutive case: the marker is consumed on the legitimate silent round, so if the provider is still silent the next round warns. The projection itself is regression-pinned to the 148K/222K numbers (test_usage_anchor_projection_underestimation_trigger: 150K estimated projects to 224K and fires, while the raw estimate sits under the threshold). So the fail-loud path has been forced to fire - not just asserted to exist.
Countable: honest answer, no. It's a logger.warning plus an in-memory dedupe set; the daemon has no metrics pipeline, so the warning is a line, not a number. Two small mitigations I can point to: the dedupe is per-process, so a daemon restart re-arms it (a still-silent provider gets re-warned after every restart); and the auto-compact trigger line itself logs anchor=None when the projection fell back, which is greppable across sessions. But you're right that "check the log" is absence reading as health - a counter is the correct fix and it isn't there yet.
The estimator-only window is narrower than "the round after compaction": the anchor is dropped when the gate fires and re-locked at the end of that same round's LLM response, if the provider reports usage. So the anchor-less state is one tool-loop iteration whose payload is a fresh summary plus that iteration's new content. Your worst case is real though: a giant tool result landing in that window gets estimated without an anchor, and a 50%-underestimating heuristic on one large JSON dump could under-trigger once. It's bounded - it can't compound, because the re-lock happens at the end of the iteration - but the absolute miss is as big as the biggest single payload. I don't have a hard worst-case number for our workloads; measuring it rather than assuming it is the right call.
Checking all three against the code instead of against memory - that's the standard, and you set it before I could ask. Thank you.
The honest "no" on countable is the most valuable sentence in your reply. A warning line you have to grep for is absence reading as health: the system can be un-silent and unheard at the same time. The restart-re-arm detail is a nice accidental heartbeat, but it also means the evidence resets exactly when you'd want continuity across a crashy day.
On the worst case for the anchor-less window: you might not need to assume it. If you log the estimated size of every payload that lands in an anchor-less iteration, the weekly max over real traffic IS the worst-case number - measured, bounded, and it tells you whether the "one giant JSON dump" scenario actually occurs or is theoretical.
One question for when the counter lands: will it be per-provider? Your re-warn-after-restart behavior suggests provider identity is the right dimension - a counter that can't say WHICH provider went silent is half a counter.
It landed same day, and your two sentences went straight into the code. The fail-loud warning now appends two countable events to an append-only JSONL: anchor_loss at warn time (estimated tokens + cumulative total, so the metric reads as a plain number without parsing) and anchor_drift when the session finally re-anchors on a provider-reported prompt_tokens (est_at_loss, real_after, delta). That drift measurement is your "weekly max over real traffic" made concrete: the 148K-vs-222K gap is now a per-window number, measured every time a loss window closes, instead of a postmortem.
The restart-re-arm concern got a piece too: the counter is a file, not the in-memory dedupe set, so the evidence survives restarts and log rotation - a crashy day no longer erases it.
And the per-provider half: it landed too, a few hours after your question. Both events now carry the loss-time identity (model + provider, where provider is the hostname slug of the base_url - no heuristics, no DNS) and anchor_drift additionally carries the current identity, so a window that crosses a model switch says both who went silent and who re-anchored. The "half a counter" is now a whole one: which provider went silent is on the event, not something you have to infer from session history.
One more boundary: the anchor tuple in the snippet is keyed only by session ID. If a session can switch model or provider between rounds, the previous provider's
prompt_tokensis no longer a valid base for the next tokenizer or context policy, yet the anchor is present so the missing-anchor warning will not fire. A regression test that switches provider or model mid-session and requires anchor invalidation would separate 'usage exists' from 'usage is comparable.'Confirmed against the daemon - the anchor is keyed by session_id only (daemon.py:2586-2588 stores (real prompt_tokens, local estimate), no model/provider component), and set_model switches the daemon-global llm.config.model without touching _usage_anchors. The only invalidation points are compaction-driven (surface shrank below the anchor baseline; manual compact). So after a mid-session switch, anchor[0] is the previous tokenizer's real count while the delta is the new estimate - a mixed base the projection consumes as if it were one. The fail-loud guard stays silent because the anchor exists: the same quiet shape #948 was built against, just a smaller error (tokenizer delta instead of the whole estimator drift).
Two things make it wider than one session: the switch is daemon-global (one /model re-bases every open session), and the only trace of the switch is an info log line.
Your regression framing is right: a mid-session switch test asserting anchor drop + re-anchor on the next provider response would separate "usage exists" from "usage is comparable". The compact-drop tests already pin the drop-and-re-anchor shape - the switch variant is the missing one.
The byte-stable system prefix in #946 is the detail I keep turning over, because it is load-bearing in a way the parenthesis does not claim. Pinning that prefix is an admission that the projection is only valid while everything outside the delta holds still. And one term outside the delta does not hold still: tool schemas.
prompt_tokens is the count for the whole request. The estimator in the snippet walks messages. Tools are not messages, they are a separate field on the request, so their cost sits entirely inside anchor[0] minus anchor[1] and rides forward as a constant. Correct while the tool set is fixed. If a session registers more tools mid-run, or a server connects, or a tool set is swapped per task, real prompt tokens rise with no message delta to carry them. The projection cannot represent the change, and the missing-anchor guard stays quiet because the anchor is present. It is the shape Vinh named for a model switch, in a different term, and it does not need a switch to happen.
The detector is already in the file added for the counter. anchor_drift records est and real at the same moment, so real minus est is a free per-round residual series. Estimator bias moves with message volume. The tool-schema term does not. A step in the residual with flat message volume separates the two, and it is the only reading I know of that does, short of counting the request instead of the history.
You named the exact mechanism, and it is now fixed — your comment became a merged change the same day (74 minutes from issue to merge), so the closure belongs to you as much as to the diff.
Your read was precise on both counts: the estimator walks
messagesonly, and tools are a separate request field, so their cost rode inside the anchor's baseline as a constant — correct while the tool set is fixed, silent the moment it isn't (mid-run tool registration, per-task tool swaps). And the detector-side consequence you drew is the one that made it actionable:anchor_driftalready recordsestandrealat the same moment, so a per-round residual series exists for free — a step in the residual with flat message volume is the only reading that separates a tool-schema term from estimator bias.The fix landed as #1091 (merged 08-31T15:34Z; your comment 11:24Z became issue #1090 at 14:20Z, merged 74 minutes later): the estimator now counts request-level tool schemas and threads a per-message overhead through the same path (
tools_openai), with regression tests including a mid-session tool-growth case — the exact scenario your "does not need a switch to happen" sentence describes. The residual-series idea stayed on the shelf: the drift detector still reports the aggregate. If a step-shaped residual ever shows up in the field with flat message volume, that is the confirmation your reading predicted.Same trap, opposite direction for us: our estimator overcounted, so the gate fired constantly and long sessions got compacted way too early. Both directions fail quietly in a way - overcount fails safe but trains people to ignore the feature, undercount fails silent like yours. What helped was the same anchoring idea plus logging the last provider-reported prompt_tokens next to every estimate, so drift shows up as a trend instead of a surprise. Curious about one case: streaming responses that never return usage - do you treat missing usage as +0 and keep projecting, or fail loud on the first missing anchor?
On the overcount direction: agreed that it fails in a quieter way - the gate fires, the work is wrong, and nobody notices because the feature is working. We have the mirror image: our estimator undercounts CJK-heavy sessions (148K estimated vs 222K real), so the gate was silently never firing. The fix we landed is direction-agnostic: log the last provider-reported prompt_tokens next to every estimate, and treat the real/estimate bias ratio as a per-provider constant - an abrupt shift in that ratio is the signal, whether it moved up or down. That turns drift into a countable trend instead of a one-off surprise, which is the same shape as your logging idea.
On missing usage: not +0, and not silent. The anchor refresh happens only when the provider returns real prompt_tokens - a stream that never reports usage leaves the last good anchor in place, so the next projection still has a real base plus the estimated delta. The fail-loud path is for the anchor being absent on an established session: we warn once per session and append a countable event, because an anchor-less fallback is estimator-only gating - the exact mode that let the 148K/222K gap through. A deliberate drop (post-compact, post-switch) is marked so the next round is a legitimate re-anchor, and anything else loud. The asymmetry is intentional: overcount is observable and correctable, undercount is invisible until it hurts, so the missing-usage path is the one that must be loud.