Our agent's auto-compact was supposed to protect the context window. It shipped, it ran, and it never fired — because the local token estimator said 148K tokens while the provider was actually seeing 222K. This is the story of that 50% drift: why it's a structural trap for any LLM agent, and the two changes that fixed it (usage-anchored projection, then fail-loud anchor loss).
If you run an agent that feeds a growing conversation into an LLM, you have some version of this problem: how do you know when the context window is about to blow? The honest answer is that you don't — not locally. Providers bill by token counts they compute with their own tokenizers, and a local estimate is at best a guess. The trap is when a guess gets promoted to a gate: a threshold that is supposed to protect you, silently, based on a number that can be off by half.
That's exactly what happened to us, and the fix ended up being a one-line insight that reshaped the whole feature: when the provider gives you the real number, anchor to it — and if you ever lose the anchor, fail loud.
The failure: a safety net that never fired
Our daemon has an auto-compact feature: before each LLM round, it estimates the conversation's token count, and if the projection exceeds a threshold (a percentage of the configured context window), it summarizes the history to free up space.
The estimator is a deliberately cheap heuristic — character-aware, because our sessions are heavily CJK:
# emrg/server/daemon.py (v0.2.80)
for ch in text:
if is_cjk(ch): # CJK ideographs, kana, hangul, fullwidth forms
cjk += 1
else:
ascii_chars += 1
return (cjk // 2) + (ascii_chars // 4) # CJK ≈ 2 chars/token, ASCII ≈ 4 chars/token
Plus 3 tokens per message for role/name overhead. It's a reasonable directional heuristic — the kind of thing you'd use to display "≈12K tokens" in a UI. It is not a number you should hang a protective gate on, and that's what we did.
The observed drift, recorded in the code comments when we finally understood it:
- Local estimate: 148K tokens
- Provider's real
prompt_tokens: 222K tokens
That's a 50% underestimate. With the auto-compact threshold set at a fraction of the context window, the projection sat quietly below the trigger while the real context was already 50% past it. The gate never fired. No log, no warning — just a context window slowly filling past the safe line, quality degrading and cost climbing, with the safety net fully armed and fully blind.
Why is the drift so bad in our workload? The estimator assumes CJK ≈ 2 chars/token and ASCII ≈ 4 chars/token, but real tokenizers are far more irregular. JSON-heavy tool results inflate the count (brace-heavy syntax tokenizes denser than prose), and CJK mixed with code and JSON produces a composition the heuristic simply can't represent. The error isn't a constant offset — it grows with the payload, which is exactly when the gate matters most.
Fix 1: anchor the projection to the provider's real number (#946)
The insight: we don't need a perfect estimator. At the end of every LLM round the provider hands us the real prompt_tokens — the ground truth for everything that was sent at that moment. So instead of trusting the estimator for the whole history, we cache an anchor and only let the estimator contribute the delta since the anchor:
# The anchor: (real prompt_tokens, local estimate) captured at the same moment
if final_usage:
pt = final_usage.get("prompt_tokens")
if pt:
self._usage_anchors[session.session_id] = (
pt, self._estimate_tokens(messages)
)
# The projection used by the auto-compact gate:
anchor = self._usage_anchors.get(session.session_id)
if anchor is not None and estimated >= anchor[1]:
projected = anchor[0] + (estimated - anchor[1]) # real base + small delta
else:
projected = estimated # no anchor yet — plain estimate
if projected > trigger_at:
# compact
The estimator error is now confined to the delta since the last provider response — a handful of new messages instead of the entire conversation. If the estimator is off by 50%, it's off by 50% of a small number, not of 222K tokens. The anchor is refreshed every round, so the projection can't drift back.
(The same commit also made the system prompt prefix byte-stable, so the estimate of the fixed overhead stops shifting between rounds — a smaller sibling of the same disease.)
Fix 2: fail loud when the anchor goes missing (#948)
Here's the thing about fix 1: the anchor is now the entire safety mechanism. The projection's accuracy depends on the provider returning prompt_tokens. If a provider stops reporting usage — a config change, a proxy, a model switch — the gate silently falls back to projected = estimated, which is exactly the #946 failure mode we just fixed. Silent regression back into the bug.
So we made the loss of the anchor observable. There are exactly two legitimate anchor-less states:
- The session's first round — no assistant turn yet, nothing anchored, nothing to protect.
- The round right after a compaction — the anchor is deliberately dropped (the history just got replaced), and the next LLM response re-anchors it.
Any other missing anchor means the provider stopped reporting usage. _warn_missing_usage_anchor logs a loud warning — once per session, so it's observable without per-round spam:
auto-compact: usage anchor missing for established session <id> (est=148K) —
provider not returning prompt_tokens; gate is estimator-only (observed
148K est vs 222K real). Check provider usage reporting.
A subtle detail from the review: we discovered the anchor also had to be dropped on manual compaction — otherwise the stale baseline (pre-compact size) would suppress the projection for the next several rounds. That's the kind of bug you only find by writing the "what should happen here" checklist down (commits 2a723ef/493f9dd).
The general lesson
Three rules that generalize beyond this one feature:
- Never gate on an estimator when a measurement is available. If the provider returns usage, that's ground truth at the round boundary — use it. The estimator's job is to fill the gap between measurements, not to replace them.
- Anchor, don't estimate-from-scratch. A stale real number plus a small estimated delta beats a fresh estimate of the whole history every time, because estimator error compounds with history length.
- When a mechanism's correctness depends on an external input, the loss of that input must be an event, not a silent fallback. Degrading to a less-accurate mode is sometimes unavoidable; doing it silently turns a detectable contradiction into a missing row.
The uncomfortable part of #1 is admitting the local counter was never good enough. The estimator is still there — it's genuinely useful for the UI, and for the first round before any anchor exists. It just isn't allowed to protect anything anymore. The safety net now trusts the party that actually counts the tokens.
Since: verified against master 93ae82ac8 (v0.2.80, 2026-08-25) — the anchored projection and the fail-loud warning are unchanged since the fix. Commits: 13be856 (#946, usage-anchored auto-compact + stable system prefix), 68fba3b (v0.2.73), 47a9123 (#948, fail loud on missing anchor), 2a723ef/493f9dd (manual-compact anchor drop, review fixes), 69614d5 (v0.2.74).
From the codebase of EMRG, an open-source (MIT) agent harness whose design is that the loop reads its own failures and converts them into tested fixes. The full history of this one is public: #946, #948, #950.
Top comments (1)
Rule 3 is the one I'd frame and hang on the wall, because we paid for it separately: "the loss of that input must be an event, not a silent fallback." Our version of your bug was an anchor check in a benchmark pipeline - a data-source swap made a join produce zero pairs, the mean over an empty list came out 0, and the check printed its greenest output on zero observations. Same disease, different organ: the mechanism's correctness depended on an input, and the input's absence looked like success.
Two questions, both meant as building material. First: has _warn_missing_usage_anchor itself been fed a known-bad - a test that simulates the provider dropping prompt_tokens and asserts the warning actually fires? A fail-loud path that has never been forced to fire is the same trap one level up; ours only became trustworthy after we required every guard to prove it can go red. Second: the once-per-session warning is right against log spam, but does it also land somewhere countable (a metric, not just a line)? We once had errors that produced an EMPTY log because the throw happened where no logger was attached - "check the log" was absence reading as health.
And a small boundary worth one test: the round right after compaction is estimator-only by design. If that exact round carries a giant tool result, the gate is briefly back in the 148K/222K world. Probably acceptable - but worth knowing the worst case number rather than assuming it.