DEV Community

Discussion on: 148K estimated, 222K real: when the token counter drifts, the safety net goes silent

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen

One more boundary: the anchor tuple in the snippet is keyed only by session ID. If a session can switch model or provider between rounds, the previous provider's prompt_tokens is no longer a valid base for the next tokenizer or context policy, yet the anchor is present so the missing-anchor warning will not fire. A regression test that switches provider or model mid-session and requires anchor invalidation would separate 'usage exists' from 'usage is comparable.'

Collapse
 
pm25coder profile image
pm25coder

Confirmed against the daemon - the anchor is keyed by session_id only (daemon.py:2586-2588 stores (real prompt_tokens, local estimate), no model/provider component), and set_model switches the daemon-global llm.config.model without touching _usage_anchors. The only invalidation points are compaction-driven (surface shrank below the anchor baseline; manual compact). So after a mid-session switch, anchor[0] is the previous tokenizer's real count while the delta is the new estimate - a mixed base the projection consumes as if it were one. The fail-loud guard stays silent because the anchor exists: the same quiet shape #948 was built against, just a smaller error (tokenizer delta instead of the whole estimator drift).

Two things make it wider than one session: the switch is daemon-global (one /model re-bases every open session), and the only trace of the switch is an info log line.

Your regression framing is right: a mid-session switch test asserting anchor drop + re-anchor on the next provider response would separate "usage exists" from "usage is comparable". The compact-drop tests already pin the drop-and-re-anchor shape - the switch variant is the missing one.