DEV Community: Li Zhuojun

I audited six token usage trackers. They disagree with each other by 2x to 8x.

Li Zhuojun — Tue, 18 Aug 2026 13:39:25 +0000

Over the past five months I took apart the folding logic in six token usage trackers and recomputed each one with an independent implementation.

They don't agree. The gaps run from 2.00× to 8.09×, in both directions. Ten fixes have landed so far across five repositories.

The second finding matters more: none of these tools knows whether its own number is right. There is nothing for them to check against.

Here is each gap, why it happens, and how to check the one you use.

Disclosure first

Two of the issues below are my own submissions. Saying so up front, rather than letting you find out by clicking the link:

tokscale issue #1011 (filed 2026-08-03). The parser fix landed three days later. It's still open for a different reason — see mechanism 4.
claude-code-templates PR #754 (filed 2026-07-25, unmerged as of Aug 18). Checks green, no conflicts, no maintainer response yet; the thread is me and the bots.

For the second one, "unmerged" means "not picked up yet," not maintainer negligence. That's a 30k-star repo with a long queue.

How this started

I wasn't looking for bugs. I was auditing my own session logs and happened to read the same files with two tools. The numbers didn't match.

A small gap I would have written off as rounding. This was close to double, so I looked properly.

There's precedent for this, and it went well. In tokscale issue #76, johnpyp reported tokscale and ccusage differing by close to 2×. Maintainer junhoyeo labelled it a bug within twenty minutes, closed it within five hours, and traced the root cause to Claude Code upstream duplicating session history under stream-json (anthropics/claude-code#5034). Not a tokscale bug.

That case makes the point I want to make: when a discrepancy is large enough to see, it gets fixed, and fixed fast. The dangerous ones are the ones nobody can see.

They can't be seen because there is no third number. Subscription users especially. You pay a flat monthly fee, so Anthropic never sends you a per-token invoice. Claude Code's own dollar figure comes with a plain warning in the docs: usage for Max and Pro subscribers is included in the subscription, so that figure "isn't relevant for billing purposes."

So this whole category is a competition with no referee.

Five mechanisms

Every discrepancy I found falls into one of five buckets. Each one below has a measured number and the repo it came from.

1. The same usage is written to the log twice

The most common one, and the easiest to reproduce.

A streaming response emits a usage chunk, then assembles a complete message when the request finishes. Both carry the same usage. Sum everything you see and you get exactly 2×.

On the deepseek-harness corpus, naive folding came out at 2.000000× of the official projection. Two provider routes, computed separately, agreeing to six decimal places. Not approximately two. Two.

The same mechanism wears different clothes elsewhere:

claude-code-templates (30,276 stars) counts once per content block. Measured 2.36×. My fix PR #754 is still unmerged as of Aug 18.
Clawdmeter had the same per-content-block duplication. Measured 2.34–2.37×. The author fixed it in v3.0.1 on 2026-08-08, titled the release "token counts corrected (~2.5x lower)," and noted that the percentage progress bar reads Anthropic's rate-limit header and was never affected. That's candid handling and it deserves saying out loud.

Three projects, three implementations, one hole.

2. A forked child session physically contains its parent

This one hides better.

Sessions can fork, and the child's log file physically contains the parent's full prefix. If your aggregation walks every session file and sums, that prefix gets counted twice.

The two forks I caught were inflated 5.18× and 23.05×.

23× is not a typo. The later the fork point, the longer the inherited prefix and the shorter the child's own contribution, so the ratio gets worse. At the limit you are reporting an entire parent session as the cost of a child that did almost nothing.

The test has to be seq >= seedLength. Checking origin === 'subagent' alone is not enough, because an ordinary user-created fork inherits a prefix too.

3. Some model calls aren't in scope at all

Context compaction is a real model call and the provider reports usage for it. But it isn't a loop step, so it produces neither an assistant chunk nor an assistant message. Most folding logic matches only those two event types, so this cost was never counted at all.

Measured: 3 compaction events, 48,895 tokens, none of them counted. The largest summarize call reported 44,444 tokens (41,472 of them cache reads) to replace a history range of 19,962 tokens.

This omission has an unpleasant property. Compaction fires most often on long sessions, which are exactly the sessions where you most need the cost to be right.

4. Estimates and measurements share one field

In tokscale, the input field adds a character-count estimate of tool_result content (chars.div_ceil(4)) on top of the API-reported input_tokens. Claude Code never writes token metadata on tool_result blocks, so the fallback fires on every one of them. Measured before the fix: 87.6% of input values were the estimate, 8.09× inflation.

This is the best-handled case in the set, and it's worth telling in full.

I filed issue #1011 on 2026-08-03. Three days later junhoyeo merged the parser fix (#1037) and shipped v4.11.0, writing in the thread that "your measurement is what drove this." He deliberately left the issue open, because the correction isn't retroactive — already-cached totals stay inflated, and he laid out why bumping parser_version would be worse: it would discard retained assistant turns that compacted transcripts no longer contain.

What happened next is the part I'd point at. He designed a cache migration, then overturned his own design: an adversarial review of his plan showed the migration wasn't needed at all, because the provenance rebuild already clears stale estimates as a side effect. The blocker wasn't the design, it was that the commit hadn't been cut into a release. His words: "the remedy is a release, not a migration." He also apologised, unprompted, for a nine-day gap — "That's on me."

I re-measured on main (f169a1f0): input went from 10.10× to 1.00×, clearing 17,109,737 tokens, while output, cacheRead, cacheWrite and messageCount held identical. That second half is the load-bearing part — it says the estimate was recomputed away rather than records being retired.

So #1011 is still open, but not because nobody acted. Two things remain: the fix hasn't been cut into a release tag, and server-side stored submissions stay inflated because the merge guard refuses reductions.

Character estimation is fine on its own. The problem was that the estimate and the measurement were merged into one field and looked identical in the UI. A user had no way to tell whether the number in front of them was counted or guessed. That isn't an accuracy problem, it's a labelling problem. If a number contains an estimate, the interface should say so.

5. History gets rewritten silently

After resume and compact, historical totals drift. Nothing errors. The numbers just change.

I'm not putting a multiplier on this one, because it isn't a fixed multiplier — it depends on how many times you resumed. The mechanism is real; the direction varies.

Why nobody catches this

Because there is no ground truth.

Every tool compares its output to its own expectations. The person writing the tests and the person writing the implementation are the same person with the same mental model, so a passing test proves the code matches the author's understanding. It proves nothing about whether the author's understanding matches the log.

viberank's trust model shows the shape of the situation. It labels submissions verified, and that label is about identity — you are who your GitHub account says you are. The numbers themselves go through a separate set of server-side heuristics: token totals, cost-per-token ratio, date plausibility, and a review flag for outliers. Both things are done. They're just easy to read as one thing in the UI. It never claimed to verify the data, and I don't think this is deceptive — it's what happens when a category has no third-party baseline.

tokscale's README has a line that describes the situation precisely. Writing about subscription quotas, it says those numbers are vendor-reported and "are not independently verified against tokscale's own usage tracking." That sentence is about quotas rather than about its own token counting, but the structure it describes applies at every layer: each layer relays the layer above it, and no layer checks.

This is not a failing of these authors. The category is missing something to check against.

The method

Four steps. None of them needs a vendor account.

Build a synthetic corpus with hand-computed ground truth. Not "run it once and save the output as expected" — that records current behaviour, it doesn't test it. Work out on paper what the answer should be, then make the code match.

Make the probe check itself first. Before touching real data, run --self-test against the hand-computed fixture. If a fold disagrees with the fixture, that's a bug in the probe, and no number leaves the machine.

Import the subject's own entry point. Don't reimplement from its README — then you're testing the docs, not the code. For deepseek-harness I imported usage-projection.ts directly and ran it over a real log.

Get two implementations that share no code to produce the same number. This is the only thing that turns "I think this is wrong" into "this is wrong." In that case an independent fold and the official patch came out with zero difference across four buckets and two scopes. Past that point it stops being a matter of opinion.

The probe is stdlib-only Python, no dependencies, one command. Session logs stay out of the repository; only measurements get published.

What to check in your own tool

These are format-independent. They apply to any metering tool:

Can one usage event be written to two records? If so, what's your dedup key?
Does a child session's log contain its parent's content? Is your aggregation by file or by ownership?
Are there model calls outside the main loop? Compaction, summarization, title generation, search — those cost money too.
For every number in the UI: measured or estimated? Mixing them is lying to yourself.
Missing is not zero. No pricing model is an error, not $0. No sample is unknown, not free.
Don't stack your own estimate on top of a number the API returned. Either trust it or label it.

5 and 6 are the ones I've gotten wrong most often.

Closing

The point isn't the bugs. Most of these authors fixed things quickly: Clawdmeter shipped a fix in one release, tokscale merged a parser fix in three days and then overturned its own follow-up design. The two projects I wrote about at greatest length are the two that responded most seriously — that isn't a coincidence. People who engage leave a process worth writing down.

The point is that a number everyone uses and nobody can verify is a bad number, even when it happens to be correct.

These tools are being used right now for team cost allocation, subscription tier decisions, and public leaderboards. Until there's a third-party baseline, all of that rests on a number that vouches for itself.

Probe, protocol, and the full invariant catalog, Apache-2.0:
https://github.com/lizhuojunx86/traceguard/tree/main/usage-tracker-audit

Happy to answer questions. If you maintain one of these tools and want me to run the probe against it, open an issue.

DeepSeek Harness got append-only right. Its token projection still misses what compaction costs.

Li Zhuojun — Mon, 17 Aug 2026 01:47:47 +0000

Four numbers from a nine-day-old codebase, measured this week across two providers:

Summing every usage record in a DeepSeek Harness session log gives 2.000000× the correct total. Not roughly two. Six digits, no remainder, reproduced independently on both routes.
One forked session reported 263,790 tokens for 11,442 tokens of its own work, a 23.05× overstatement, because its log physically contains a copy of its parent's history. A second fork did the same thing at 5.18×. Nothing bounds the ratio.
48,895 tokens across three compaction events were counted by nothing, including the official projection that most plugins read from.
A stream that died and retried left three usage samples under one step, the first of them all zeros. Keep-first reports that step as free.

The first and last are traps for people writing plugins. The middle two are gaps in DSH's own code, and the compaction one is what I would fix first.

Why I went looking

I spend a lot of time reading agent transcripts and adding up tokens. Over the past few months that turned into ten upstream fixes across four usage trackers: splitrail (216 stars, three issues), tokscale (4.6k), Clawdmeter, viberank, plus an open PR against claude-code-templates (30k). The pattern was always the same. Claude Code rewrites session files in place on resume and compact, so anything recomputing totals from live files inherits the drift. Streaming leaves partial snapshots that get summed as if they were separate calls. Subagent transcripts sit one directory deeper than a flat glob reaches, and on one corpus 54% of messages never entered any total.

Eleven of those lessons are written up as invariants in a catalog. DeepSeek released Harness on August 13, and the ecosystem produced thousands of plugin repositories within days. A curated registry snapshot on August 15, covering 457 of them, listed 27 that count tokens. I wanted to know whether the same class of bug had been reproduced at scale.

It hadn't. That surprised me, and it is worth saying before the criticism.

What they designed out

DSH's session log is append-only, and that is a written contract rather than an observation. The JSONL backend's README says "Flushed events are never rewritten." Every event carries a dense contiguous sequence number, checked on append. Compaction shadows old events in the surface projection and leaves the bytes alone: "The shadowed events remain in the raw log, so replay is deterministic."

That one design decision removes the failure that cost viberank 11% of a month-to-date total between two submissions sixteen hours apart.

Two more. Adapters emit one terminal usage value per step, never a growing snapshot, so there is nothing to mis-sum. Child sessions are siblings in the same directory rather than nested underneath, so a depth-limited walk cannot lose a third of the spend the way it did in splitrail.

Four of my eleven invariants are structurally satisfied here. I put them in the DSH catalog anyway, marked as satisfied, because a catalog that only adds rules is not a catalog. It is a list of fears.

Then I measured, and found four new ones.

Every usage sample is written twice

One model call reports its usage on two different events. Once as a stream chunk with chunk.type === 'usage', once again on the assembled assistant/message. Same numbers both times.

Fold naively and you get exactly double. The ratio printed as 2.000000 on the full corpus, on the local qwen route alone, and on the MiniMax route alone, each computed independently.

An approximate factor is arguable. An exact one is not, and that is the whole reason to report it this way. It also tells you the mechanism is universal rather than occasional, which a ratio of 1.9 would not.

The cache buckets do it too. 249,728 cache-read tokens on the MiniMax route, 81% of that route's corrected total, doubling exactly like input and output. That is the number I most wanted, because cache is usually the biggest bucket and the cheapest per token, so a doubling there moves a cost report further than a doubling of output does. Cache writes are still untested: neither provider populated the field.

DSH's own token-meter handles this correctly. It keeps one slot for the last (turn, step) and subtracts the previous buckets before adding the new ones. Its README spells it out. The official implementation going to that trouble is the evidence that the hazard is real, not theoretical.

One caveat that keeps the number honest, and it turned into the fourth finding. A step is not always one request, and when it isn't, the two-samples-per-step assumption breaks. More on that below.

The fork trap, and the field you must not filter on

A forked child's log contains a copy of the parent's completed prefix. The header carries seedLength, and every event below it belongs to the parent. Add sessions together without checking and you count that prefix twice.

I expected this to be a subagent problem. I was wrong, and being wrong is the interesting part.

A subagent child is stamped origin: 'subagent' and its delegation depth increments. But ctx.sessions.fork() is an ordinary user-facing action available on any session, and the child it produces has parentSession and seedLength set, delegationDepth: 0, and no origin key at all. Here is the header I actually got, from an ordinary fork that showed up in the course of using the web UI. I wasn't trying to make one:

{"type":"session","version":0,"id":"session-e61d64ec-…",
 "parentSession":"session-001e8887-…","seedLength":1008,
 "delegationDepth":0,"agentPreset":"standard"}

That one inherited 1,008 of its parent's 1,012 events and reports 11,418 tokens against 2,204 of its own work. The second fork I caught inherited all 3,171 events of a longer parent, asked one question, and reported 263,790 tokens for 11,442 tokens of work.

That is the shape of it. The error is not a factor, it is the ratio of inherited work to own work, and nothing bounds it. Fork a long session, ask one question, and the child reports the entire parent as its own.

So the obvious defence is the wrong one. DSH's own lineage index opens with "Ordinary forks terminate propagation" and starts with if (descendant.origin !== 'subagent') continue. That filter is correct for counting subagent descendants. Reused for token accounting it admits every ordinary fork silently. The only sound discriminator is seedLength.

Official telemetry gets this right by the right key, emitting session.parent_id and session.seed_length and expecting receivers to stitch on the pair. Nothing does that for you if you read files.

The one that is DSH's own gap

Compaction summarizes older history by making a model call, and that call costs real tokens. They land on compaction/summary.usage.

The official tokenUsage projection cannot see them. Its usageOf() matches assistant/chunk and assistant/message and nothing else, and the summarize call is not a loop step, so it produces neither.

This is worse than a plugin bug, because the plugins doing the right thing inherit it. Reading sessionProjections.tokenUsage instead of folding the log yourself is the correct, recommended approach. It is also how you miss this.

On my corpus that was 48,895 tokens across three compaction events. The largest single one is worth quoting in full: a MiniMax-M3 summarize call reporting 44,444 tokens (536 input, 2,436 output, 41,472 cache reads) to remove a range whose own shadowedTokenCount was 19,962.

Whether spending 44,444 tokens to shed 19,962 is a good trade depends entirely on what you pay for cache reads. I'm not going to tell you it's bad. I am going to point out that nothing in the official projection tells you it happened.

The usage? field is optional, so a provider reporting nothing on the summarize call produces no gap. That is a condition on the finding, not an escape from it. Both of my providers populated it.

A retried step is not one request

The last one I found by accident, chasing why one step had three usage samples instead of two.

seq 3203  chunk/usage    {"inputTokens":0,"outputTokens":0}
seq 3204  chunk/finish   {kind:'error', failure:{code:'TRANSPORT'}}
seq 3205  llm/retry      retryId=afabacf1 provider=minimax-cn
seq 3206  llm/retry-started
…                        the whole response streams again
seq 3279  chunk/usage    {"inputTokens":32,"outputTokens":1042,"cacheReadTokens":10368}
seq 3281  message usage  {"inputTokens":32,"outputTokens":1042,"cacheReadTokens":10368}

The stream died, the harness retried under the same (turn, step), and the dead attempt left a usage chunk of zeros behind. Three samples, and one more retry would make four.

Keep-first reports that step as costing nothing. Not an approximation, the whole step, including 10,368 cache reads. If that sounds familiar, it is the DSH form of a Claude Code bug where keep-first lost 46.2% of output tokens on an agent-heavy tree.

It also means my 2.000000× survived that group by luck rather than structure. The dead attempt reported zeros, so summing three samples still gave twice the truth. That's why the probe prints the group-size distribution next to the ratio instead of the ratio alone.

There is a second consequence I can describe but not measure. The official fold replaces on a repeated (turn, step) rather than adding, and it never sees llm/retry. So a failed attempt that reported real tokens before dying would have its cost silently dropped. Every failed attempt in my corpus reported zeros, so I have no number for this and I'm not going to invent one. The discriminator is sitting right there in the log if you want to handle it: llm/retry carries a retryId between the attempts.

Where the two upstream ones stand

I filed the compaction gap and the retry replacement upstream on August 15, as discussion #1886. Same day, yha9806 posted a working fix on a fork (63688b0): it folds compaction/summary.usage into the totals, treats a finish chunk with an error or aborted reason as an attempt boundary so retried attempts add instead of replace, and bumps tokenUsage.stateVersion from 1 to 2. I re-folded my corpus against it and the corrected totals matched independently, which is a better outcome than agreement on prose. The attempt-boundary approach is also better than what I proposed, because it needs nothing outside the package.

One note I left on it, in case it helps whoever lands this: keying the boundary on an allowlist of two failure kinds means a future adapter with a third failure kind silently reverts to the old behaviour. Testing for 'failure' in event.data.chunk.reason fails safe instead.

None of it is upstream yet, and the version moved without the file moving. I measured all of this at 47f9438, v0.1.0-rc.6. Today the root package.json and packages/llm/token-meter/package.json both read 0.1.0-rc.7, while usageOf() still matches assistant/chunk and assistant/message and nothing else, apply() still never sees llm/retry, tokenUsage.stateVersion is still 1, and SESSION_FORMAT_VERSION is still 0. A defect that survives a version bump is worth more than one pinned to a date, so that is the claim: measured on rc.6, still there on rc.7. GitHub Releases is empty and there are no open pull requests — PR creation is restricted, which is why fixes here accumulate in fork branches and discussion threads instead. I'll update this section when it stops being true.

What I am not claiming

Four sessions, 8,650 events, 78 usage samples, two providers. The 2.000000× ratio is exact and does not need a large sample to mean what it says, and it reproduced independently on both routes. The fork trap has two observations, the compaction gap three, the retry one.

Cache writes are still untested. Reads are covered now, 249,728 of them, but neither provider populated cacheWriteTokens at all. MiniMax omits the key.

The compaction gap came to 15% of the corpus total, and I am deliberately keeping that percentage out of the summary. Three compactions over two short sessions inflates it. The citable facts are 48,895 tokens and three events.

Both routes are openai-completions-family. Whether an Anthropic-protocol or Responses-protocol route behaves the same, I don't know.

I got my own first pass wrong twice, which is why the probe prints four folds instead of two. My first attribution credited the compaction gap to the double-write line, because naive summing picks up compaction and the official projection doesn't. And I assumed the fork was a subagent until I read the header. Both corrections are in the probe now, along with a warning line that fires when a seed-bearing session has no origin.

Reproducing it

The probe is stdlib-only Python, about 400 lines, and runs against a session root:

python3 dsh_usage_probe.py --self-test        # four folds vs a hand-computed fixture
python3 dsh_usage_probe.py --root <sessions>  # the same four folds over your corpus

--self-test builds a parent/child pair exercising all three findings with constructed ground truth and asserts every fold against it. It takes under a second, touches no real data, and needs no vendor account. I ran it before every number in this post, and I would not quote one it hadn't preceded.

Write your probe corpus with compression: 'none' and packChunks: false and the log is line-readable by anything. One warning that cost me twenty minutes: a custom provider in DSH requires a credential even when the endpoint doesn't, because the provider id doubles as the credential name. An empty key fails the request with MISSING_CREDENTIAL rather than sending an unauthenticated one.

What would settle it

Three things, in the order I would do them.

Run it on a route that reports cache writes. Reads are settled; writes are the remaining hole, and it is an afternoon.

If you maintain a DSH plugin that counts tokens: fold per (turn, step), filter on seedLength, never keep-first, and add compaction/summary.usage. The first three you can fix today. The fourth needs the projection to change, or every consumer to fold the summary event themselves.

And if your corpus contradicts any of this, I want the numbers. A step whose two usage samples disagree, a seedLength that doesn't bound the inherited prefix, a provider that populates the summary usage into the projection, or a failed attempt that reported real tokens before it died. The Claude Code catalog was built entirely out of people sending me counterexamples, and six of its entries exist because someone did.

Probe, protocol, and the full invariant catalog are in the repository. Happy to answer questions about any of it.

— Li Zhuojun

The easiest numbers to fake are the hardest to catch. That is one property, not two.

Li Zhuojun — Sun, 16 Aug 2026 14:56:54 +0000

Six batches of a pharmaceutical extraction process. Yields sit at σ ≈ 0.02 percentage points, every one of them hugging the specification lower bound. Charging masses are recorded to 0.01 kg.

Nobody on a factory floor weighs to 0.01 kg. The operator said so plainly when I asked: there is always drift, nobody weighs precisely and nobody can, the records are written to match what the process spec requires, and the numbers are computed.

That is the same shape as earnings benchmark-beating, one industry over. What interests me is not the fraud angle. It is a measurement property that fell out of the data and that I think travels past this domain.

How hard a number is to fake backwards scales as 1 / (target × tolerance)

A main component specified at ≥30% with a 5% tolerance gives you a wide band of plausible values. A fabricated number is cheap to produce there, and nearly impossible to tell apart from a measured one. A trace component specified at ≤0.5 ppm has almost no room, so fabricate it and the distribution gives you away. In one incoming-inspection lot, the range ratio across three sub-channels came out around 55×, strictly decreasing with magnitude.

So the numbers easiest to manipulate are the ones hardest to detect. Those read like two separate facts. They are one fact said twice, because both follow from the width of the plausible band.

If that sounds familiar, it should. It is the same reason I have spent this year filing token-accounting bugs in usage trackers. A per-message token count with a loose upper bound drifts quietly for weeks. A hard-capped counter announces its own breakage the first time it is wrong.

The part I want stress-tested

Does 1 / (target × tolerance) hold in your domain?

Clinical trial records, emissions reporting, financial close, safety incident logs, SLA reports, anything with a compliance number and a stated tolerance. If it fails somewhere, that failure is worth more to me than agreement.

I have opened a comment window until September 30 on the three papers behind this: Open comment window on GitHub. They go to v1.0 in October, labelled not peer reviewed, with every comment logged and dispositioned in public. Zero comments will be recorded as zero.

The data is de-identified. Product names, batch numbers, lots, equipment and people are all pseudonyms. The σ, range ratios and multipliers keep their original values, so the results recompute. CC BY 4.0.

"Your cache hit rate is low" — true, and worth $0.16

Li Zhuojun — Sun, 16 Aug 2026 03:45:04 +0000

Anthropic emailed our org last week: prompt cache hit rate is low, and caching repeated content "could save up to 64% of direct API spend." The email is careful with scope — direct API only, Claude Code excluded because it manages caching itself. I went to check what that 64% was made of.

Our direct API traffic over the period: 16 calls, $0.25 total, average prompt 90 tokens. They come from traceguard's rerun harness, which replays self-contained consults as fresh single-turn calls — every prompt unique by construction. A workload like that cannot hit cache: there is no repeated prefix across calls, and 90 tokens is far below the minimum Anthropic will cache at all (1,024 tokens on Opus 4.8, the model those calls ran on; 512 on Opus 5 / Fable 5). Hit rate 0%, structurally. 64% of $0.25 is $0.16.

The metric was right. The money wasn't there.

The side the email excluded

The interesting numbers sit in the traffic the email correctly excluded. We ingest our own Claude Code session logs into traceguard's trace store (Claude Code transcripts carry no cost field, so we compute list price from usage). 71 days, 158 sessions, 58,753 API messages. This is section 1 of the command output, unedited:

model	messages	prompt tok	hit rate	input cost	no-cache	saved	saved %
claude-opus-4-8	23,759	5,535,807,891	96.1%	$4,616.39	$27,691.44	$23,075.05	83.3%
claude-fable-5	15,291	3,583,420,473	96.2%	$5,847.08	$35,834.20	$29,987.13	83.7%
claude-opus-5	6,316	1,750,760,343	97.7%	$1,205.08	$8,753.80	$7,548.72	86.2%
claude-sonnet-5	11,139	1,326,223,701	95.1%	$422.96	$2,652.45	$2,229.49	84.1%
claude-haiku-4-5-20251001	1,573	57,312,161	93.9%	$9.76	$57.31	$47.56	83.0%
claude-opus-4-7	64	11,045,966	96.6%	$9.04	$55.23	$46.19	83.6%
claude-sonnet-4-5-20250929	2	42,410	0.0%	n/a	n/a	n/a	n/a
(none)	609	0	n/a	n/a	n/a	n/a	n/a
TOTAL	58,753	12,264,612,945	96.2%	$12,110.31	$75,044.44	$62,934.13	83.9%

Hit rate is token-weighted; costs are the input side only, at list price. The two n/a rows are deliberate — no published price for a model means no guessed money, tokens counted anyway.

Caching nobody at our org configured cuts the input side by 83.9%. With output included, the whole bill lands at 1/5.5 of the no-cache counterfactual. There is nothing left to optimize here; the remaining spend is mostly first-write premiums on genuinely new content.

The keep-alive ping idea

There's a folk optimization going around: ping your session on a timer so the cache stays warm and the next turn hits. The mechanics say it could work — the cache TTL is 5 minutes by default, every hit refreshes it for free, and a 1-hour TTL costs 2× on writes. Instead of arguing, we ran the counterfactual on our own gap data.

97.3% of our in-session gaps are under 5 minutes; they need nothing. The addressable prize is 422 gaps longer than an hour: the context rewrites right after them cost at most $1,913 over 71 days (an overestimate, since it counts genuinely new content as rewrite). Bridging those same gaps with a ping every 55 minutes would have taken 6,765 pings costing $2,009 in cache reads alone, before counting the pings' own output tokens. The strategy loses money on our data, under assumptions tilted in its favor.

The reason is boring: the 1-hour TTL removed the ping's niche. Our logs show Claude Code already routes nearly all cache writes into the 1h bucket (opus-4-8: 155.9M tokens written at 1h TTL vs 56.4M at 5m). Pinging is a hack from before extended TTL existed. If a tool sells you auto-ping in 2026, ask it for the counterfactual on your logs first.

(Superseded — the Aug 18 note below has the current numbers.)

Update, Aug 16. Reid Marlow asked in the comments for cost broken out by idle gap rather than in aggregate. I ran it, and it overturns the paragraph above. Split by bucket: the 183 gaps of 1–4h would have cost $81 in pings against at most $887 of rewrite, an 11:1 win, while the 239 gaps over 4h cost $1,928 against at most $1,026. The refusal I published is a win and a loss added together. A policy that can actually be run (ping every 55 minutes, give up after 4 hours idle) costs $316 including every ping burned on the 239 gaps that outlive the cap, against at most $887 avoided. It pays.

So the correct claim is narrower than the one this section makes: unbounded pinging loses, capped pinging probably pays. Probably, because both sides of that comparison lean pro-ping. The $887 is an upper bound, since cache_creation mixes the re-established prefix with whatever the turn genuinely added, and the $316 charges pings as pure 0.1x reads of a frozen prompt. Under that tilt a refusal is safe and an endorsement is only as wide as its margin, and this one is 2.8x rather than 11x.

I had the aggregate and stopped there, which is the same mistake the email made: a number averaged over cases that behave differently is not a decision. The per-bucket money and both verdicts are in cache_audit in the repo now and ship in the next release; 1.2.0 prints the aggregate only.

(Numbers superseded by the Aug 18 note below. The reasoning stands.)

Update, Aug 17. The numbers in the note above are superseded. Not because the direction was wrong, but because 4 hours was picked to line up with a table boundary rather than solved for. Sweeping every cap from 1h to 12h in 15-minute steps, the best single value is 10h. It beats 9h45m by $7.63. Then I measured something this section had only assumed, and that one measurement moved the same cap by $18.76. A correction larger than the distance between first and second place can reorder them, so a single number is the wrong output. The answer is a band: ping every 55 minutes, give up somewhere between 9 and 12 hours idle. Anywhere in that band costs under 10% of the optimum.

Net over the window, after the ping bill: $663.69 to $811.30. That is two runs rather than an error bar. The low end treats every gap whose model I cannot determine as a cache-invalidating switch, the high end deducts only the ones I can prove. 54 of 432 gaps have no model recorded, and nothing in the data says which way they go.

What I measured is whether the model changed across the gap. A keep-alive only holds a cache for the model it was written on, so a switch makes every ping before it worthless. This section listed that as an assumption. It is 18 of the 378 gaps where both sides are known, and the rate climbs with idle time: zero in the shortest decile, 23.1% past 24 hours. All 18 were opus-4-8 and fable-5, in one direction or the other. Not one was a switch down to something cheaper. Leaving that out was not adding noise to the cap. It was pushing the cap long.

Every number here comes from 2026-05-30 to 2026-08-16, pinned behind a --benchmark flag so the run reproduces. That flag and the sweep are on main, not in 1.2.0, and land with the rest of this in the next release. The Aug 16 note was overconfident in the same shape as the paragraph it corrected: I had a number and quoted it before asking how wide it was. Checking that note turned up three more errors, one of them a denominator error of my own. That one is a separate post.

Update, Aug 18. The window did not do what I said it would. I pinned 2026-05-30 to 2026-08-16 so the run would reproduce, and it does not. The same window over the same store went from 432 expired gaps to 439, and from 168 sessions to 174, in under a day. Ingest reads a directory of transcripts, so a file that turns up late still carries messages from inside the window. Closing the dates closes the timestamps, not the corpus. The figures above moved with it: the two ends are now $650.53 and $806.82, not $663.69 and $811.30 (argmax_reference_only.net_usd in that file — the field named recommended_cap_band is a different interval).

So I stopped pointing at a command. The run those numbers come from is committed in the repo, fingerprinted over the traces it analysed, and the tool now refuses to overwrite an entry that already exists. A file can be cited. A rerun cannot. Everything else above stands: 9 to 12 hours at a 55-minute cadence, and the two ends are two runs rather than an error bar. Shipped in 1.3.0.

Reading is not measuring, again

While writing this I checked our own SDK and found the same class of bug I keep filing against other trackers. traceguard's wrap_anthropic recorded tokens_in = usage.input_tokens. On Anthropic's API, input_tokens excludes cache reads and writes — and on our traffic, cache reads run about 1,000× larger than raw input, so the wrapper under-counted by three orders of magnitude. Every field read correctly, the quantity still wrong (the splitrail #220 pattern). Fixed in PR #39, and the fix is what makes the next part possible.

A rate without a denominator is not a finding. Percentages travel well in emails; dollars make decisions. Before acting on a metric email, multiply it by the money it applies to.

Check yours

This analysis is now one command, merged in PR #40: python -m traceguard.routing_audit.cache_audit. Point it at your ingested Claude Code transcripts or a traceguard trace store, and get the table above for your own data — per-model hit rate, actual vs no-cache cost at list price, gap distribution, and a yes/no on whether keep-alive pings would pay for you (ours printed NOT WORTH IT — see the update above for what that verdict hides, and what the next release prints instead). Ships in traceguard 1.2.0 — pip install -U traceguard. The last line of ours came out like this:

Claude Code caching already saves us 84% ($12,110.31 vs $75,044.44 list). Checked with: python -m traceguard.routing_audit.cache_audit

If you run it, send me your numbers. Especially if pings win somewhere — I'd like to see the traffic shape that makes them rational.

Li Zhuojun

My checker scored one component compliant and another deviant. Neither had a rule behind it.

Li Zhuojun — Sat, 15 Aug 2026 10:59:00 +0000

This is a follow-up to part four, not a new part. Part four measured my stated routing policy against what actually ran: 425 decisions, 96 deviations, $1,248.13, none of them on the main thread.

Then a reader asked what happens when no rule applies. I went to measure it, and the answer was that my checker had been inventing verdicts. In both directions.

The question

Brian Jin, in the comments:

Have you considered making "no applicable rule" a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps.

I had not. My policy file has a default_tier, and any decision that matched no rule quietly took it. That felt safe when I wrote it. Every decision gets a verdict, no gaps in the table, nothing to explain.

What the default was doing

Frozen corpus, the same 425 decisions from part four: 424 matched an explicit rule. One did not.

That one is a product-manager component. No rule in my file mentions it. It took the default frontier expectation, ran Opus, and scored compliant. $0.91.

Replaying the same policy on the live window, 784 decisions, the fall-through population has doubled. A claude-code-guide component first appeared on July 30, matched nothing, took the same default frontier expectation, ran Haiku, and got flagged as a deviation. $0.08.

Two rows, two verdicts, opposite signs, and neither one has a rule behind it. The default fabricated compliance in the first case and fabricated a deviation in the second.

The fabricated deviation is the worse half

A fake compliant row is a missed catch. Annoying, but the audit was never going to catch what it does not know about.

A fake deviation is worse, because it puts noise into the one table the whole exercise asks you to trust. And in this case the routing was right: Haiku for a docs-lookup subagent is exactly what I would choose. What is wrong there is not the model. It is my policy's silence.

If I had acted on that row I would have gone and "fixed" a correct routing decision.

An unresolved state catches both rows for the same reason. Neither verdict has a rule behind it, so the honest output is not a verdict at all.

The dual: rules no decision ever reached

Measuring the fall-throughs turned up their mirror image, which I had not gone looking for.

My file has seven rules. One of them, research-explore outside the main thread to the cheap tier, has matched zero decisions in both windows.

Not because the case is rare. The frozen corpus alone has 83 research-explore decisions. Every component that does research has its own component-level rule sitting earlier in the file, and the specificity tie-break shadows the task rule every time. The rule is dead and the file gives no sign of it.

So coverage has two halves, and they fail differently:

direction	what it means
decision → no rule	uncovered behavior
rule → no decision	dead, shadowed, or unexercised policy

Those two labels are Brian's, not mine. I had the two halves and called them a coverage report; he named them after the change shipped, and the names are better than what I was using.

A policy file that grows for six months accumulates both, and reading it will not tell you which rules are load-bearing.

The same hole on the actual side

The expected side of the comparison is not the only one that can fail to resolve.

When claude-opus-5 entered my corpus with no entry in the tier table, 5,992 traces sat unscoreable for two weeks. Not compliant, not deviant, silently absent from the table. That is part four's unpriced-model incident wearing a different field.

Expected side or actual side, whenever a verdict cannot be resolved the honest output is a third state, and the size of that state is the coverage metric.

What shipped

Policy.match now returns no match instead of falling through, and a verdict column carries four values: compliant, deviation, unresolved:no_rule, unresolved:unknown_model. The default_tier key still exists, but it only applies to a rule that matched and omitted its expected tier. Nine new tests, 437 pass.

The summary carries the two coverage counts next to the verdict tallies: decisions out of coverage, and rules that never matched.

Current corpus, 789 decisions:

verdict	count
compliant	627
deviation	160
unresolved:no_rule	2
unresolved:unknown_model	0

That zero turned out to be the interesting entry. unknown_model is empty because I added the claude-opus-5 tier two days before the verdict column went in, and those 5,992 traces would all have landed there.

So the two unresolved classes decay differently, which is why they are counted apart. unknown_model is an operational gap: fill in the tier table and the count returns to zero by itself. no_rule does not. Those two rows sit until somebody writes a rule or decides the case belongs outside policy, and the count stays honest precisely because nothing clears it automatically.

Brian's reply when I showed him the result put it better than I had:

That makes unresolved much more than an error bucket. It becomes an observable property of the policy surface itself.

What I still get wrong

Tae Kim, in the same comment thread, described his orchestrator defaulting to Opus on every retry. He had only specified the model for the main agent, and it ran about three weeks before he caught it by accident.

My method would miss that. The grain is (unit, component), and a retry is not a separate component. It inherits inside the same one, so the deviation and the compliant call collapse into a single verdict. Same defect as mine, invisible to my checker.

I have not fixed it, because I have exactly one corpus and it is my own, which is a poor basis for a schema change. If your orchestrator writes the model per attempt rather than per step, I would like to see a sample.

If you want to run this

The policy file is 30 lines of YAML and the checker reads Claude Code's own session transcripts. Nothing leaves your machine.

python -m traceguard.routing_audit.agent_lint

That is the one-minute version: it reads .claude/agents/**/*.md frontmatter and tells you which agents never pinned a model. The full per-decision scoring needs the trace store, and it is in traceguard.

The part worth stealing even if you never run any of it: if your compliance checker has a fallback default, it is producing verdicts nobody wrote a rule for, and you cannot tell those apart from the real ones by looking at the output. Count them separately. Count the dead rules too.

Questions welcome, and if you have found a way to make the retry case visible I would rather hear it than work it out myself.

I measured his app with his own code. He measured my claim with his own corpus.

Li Zhuojun — Sun, 09 Aug 2026 01:34:47 +0000

This is part five of a series about pointing an append-only audit log at things that count tokens. Parts one through four found accounting defects in other people's trackers and one expensive hole in my own routing. This one is about what happens after the report lands, because this time the audit ran in both directions: the maintainer shipped the fix in three days, and his closing comment contained two findings aimed back at my report. One of them caught a claim I had published without its scope. My re-measurement then caught the mechanism he had guessed for it.

The target is Clawdmeter, a desktop app that shows your live Claude Code usage with a pixel mascot. weltern announced the cross-platform release on r/ClaudeAI; I pointed my harness at its token path the same day and filed #21: every transcript-derived number in the app read about 2.34× high.

The bug, briefly

Same class as part three, so one paragraph and no re-litigation. Claude Code writes one assistant message as several JSONL records, one per content block, and each record repeats the same message.id with the same usage object. src/transcript.py summed per record in three places: the token counts beside the 5h/7d bars, everything the Stats page prices, and the per-session totals. Driving those three functions over a synthetic corpus with a known-exact manifest: 2.34×, 2.34×, 2.37×. Both record-level paths emitted 1,249 events for 540 messages. The percentage bars were never affected, since those come from rate-limit headers, and the overage figure comes from the OAuth endpoint. The report said so, because a fair report names what is not broken.

Running his code instead of mine

The part of the method worth stealing. Nothing in the report reproduces Clawdmeter's logic, models it, or reimplements it. The harness imports the vendor's own transcript.py and calls his functions. The app is a Qt desktop program, and the token path doesn't need Qt, so the whole trick is a twelve-line stub:

pkg = types.ModuleType("PySide6")
qtcore = types.ModuleType("PySide6.QtCore")
qtcore.QObject = ...   # three no-op classes
sys.modules["PySide6"] = pkg
sys.modules["PySide6.QtCore"] = qtcore

sys.path.insert(0, str(clawdmeter_clone / "src"))
import transcript as T          # his module, unmodified
T._file_token_events(path)      # his functions, his numbers

Why bother, when reading the code already showed the defect? Because of the dispute it removes. A reimplementation invites "your model of my code is wrong", and that argument can run for days. Numbers that came out of the vendor's own functions leave exactly one question open, whether the input corpus is fair, and a synthetic corpus with a by-construction manifest answers that in the same breath. The report gets to say "your functions produce 2,592,168 where the manifest says 1,108,697" instead of "I believe your code over-counts".

It also makes reproduction nearly free: git clone, one script, about a minute, a fake $HOME, no PySide6 install, nothing real touched.

Three days

Filed August 5. Confirmed August 6: "Thank you again for this report, I've confirmed the bug and will be patching it in an upcoming hotfix release." Shipped August 8 as v3.0.1, with a release note that opens the way I suggested it should, because a correct fix that halves every visible number looks exactly like a regression: "Your token and value numbers will drop by roughly 2.5×. That is the fix, not a regression."

One detail I want to be flat about: he never ran my harness. He reproduced independently, on his own corpus, with his own tooling. That is not the fixture failing at its job. The fixture's job is to make the claim cheap to check and expensive to dispute; his choice to verify it his own way instead is a stronger confirmation than an exit code, and the diagnosis still arrived pre-named, which is what made three days possible. A reproduction script is an offer, not a toll booth.

The fix report audited me back

His closing comment is the reason this post exists. It opens with "Your report was accurate and the diagnosis was right", and then does something I had not seen a maintainer do in five repos: it treats the bug report itself as a measurable claim, measures it, and reports back where it breaks. "Two things turned up that are worth passing back, since you audit other trackers."

His corpus came out worse than mine: 44,196 usage-bearing records across 18,585 distinct message ids, 73.0% on more than one record, work tokens 2.575× high. And then the two findings.

First: the repeated usage objects are not always identical. My report said that on my corpus, 100.0% of duplicated messages carried byte-identical usage on every record. On his corpus, 3,799 groups differ, and in every one output_tokens is a running total: early records carry a partial count, the last carries the final figure. His example message reads 5, 5, 5, 328 across four records. That makes the choice of which record to keep load-bearing. A per-bucket max is correct; keep-first would have reported that message as 5 output tokens instead of 328.

Second: collapsing per file is not enough. 1,094 of his message ids appear in more than one transcript file, because resuming a session replays its records verbatim into a new file. Deduping within each file still left 1.095× inflation on his tree. The collapse has to live in the account-wide aggregator, and a replayed copy can carry an all-zero usage block where the original carries the real numbers, so a record-level dedup placed in front of the message collapse can keep the wrong copy. Folding every record into its message under a per-bucket max sidesteps the whole trap.

He also turned my one-line invariant, usage events should equal distinct message ids, into a regression test, added six more, and ran nine mutations of the fix against them. Nine of nine caught. That last number is a stronger claim than any test count, and it is the part I would not have thought to ask for.

The split is a field, not a version

Finding one corrects something I wrote, so I owed it a measurement rather than a thank-you. My "100.0% byte-identical" was true and under-scoped: it was true of the main conversation path, and I did not say so. On my corpus, regrouped:

grouped per (file, message.id)	groups	byte-identical
main path	10,645	10,645 (100.00%)
sidechain (subagent)	16,051	3,342 (20.82%)

Not one main-path group differs, across 63 days and 1,521 transcripts. Every differing group has a sidechain record in it. The discriminator is isSidechain, not the Claude Code version he suspected — and the reason I can say that with some force is that the version hypothesis makes a testable prediction and the field hypothesis makes a different one. If the shape tracked the writer version, main-path groups would differ on some machines. They never do here. His Linux and macOS corpora "still showing the byte-identical shape" is what a main-path-heavy tree looks like; an agent-heavy tree shows his 3,799.

His running-total observation, meanwhile, holds without exception on my side: 12,709 differing groups, 100% non-decreasing, 100% with the last record carrying the max. He was right about the shape and wrong about the cause. I was right about my corpus and wrong to publish the number without its denominator. Neither error survived a second corpus, which is the whole argument for sending measurements instead of opinions.

The artifact I nearly handed him

One warning went into my reply because it is the check anyone would write next, and it lies. Group repeated records by message.id alone and my corpus reports 560 non-monotonic groups, output counts that go down mid-message. Every one is fake: records from unrelated files spliced into one sequence by the sort. Group per (file, message.id) and all 560 vanish. I know the trap is real because I fell into it on a different tracker and reported the bad number before catching it.

Two more numbers from the re-measurement, since they size his findings on an agent-heavy tree. Keep-first against per-bucket max at corpus scale: 20,740,608 output tokens against 38,533,663. Keep-first loses 46.2% of output, so max is not a tiebreak, it is most of the number. And his all-zero replay trap does not reproduce here at all, zero groups in 1,521 transcripts, which means you cannot harvest a test fixture for it from a tree like mine. His synthetic fixture for that case is load-bearing. The script is duplicate_usage_shape.py, read-only, stdlib only, and it prints the artifact count next to the real one so the next person doesn't rediscover the splice.

Red, green, and an API change that proves the point

Yesterday I re-ran the harness against both tags. On v3.0.0 it is red the way the report said: 2.338×, 2.338×, 2.373× on the three call sites, exit 1. On v3.0.1 all three land exact, 1,108,697 and 732,191 to the digit, 540 rows for 540 messages, exit 0.

The check needed one adaptation, and the adaptation is itself the verification. _file_token_events no longer returns something you can sum: it now emits per-record events carrying both a record key and a message key, uncollapsed, with a docstring that says the collapse deliberately belongs to the account-wide caller. The fix is not a patch over the three sites; it is the architecture his second finding argued for, folded under the per-bucket max his first finding argued for. The code now contains the whole conversation.

Counterweights I owe you

My 100.0% shipped without its scope, and the fixture didn't stop it. The harness was green, the totals were exact, and the prose still over-claimed. A fixture pins code; nothing pins your sentences except someone else's corpus.

He never ran the harness, so this series' fastest fix owes nothing to the reproduction being runnable. What earned the three days was the diagnosis being named and the numbers being exact. I keep shipping the fixture anyway, but I should be honest that its value here was rhetorical, not operational.

The isSidechain split is established on one corpus: mine. His 3,799 differing groups were not regrouped under the split, so "every differing group has a sidechain record in it" is a claim about my tree that his data could still complicate.

I do not know why Claude Code writes sidechain usage as a running total and main-path usage finished. The field predicts the shape; the mechanism behind the field is unmeasured.

Part three argued response time tracks queue depth, and a solo maintainer answering in a day fits that line suspiciously well. One data point cannot separate "small queue" from "this particular maintainer", and the 30k-star PR from part three is still open, which proves the thesis only if you already believe it.

His version hypothesis was reasonable on his data. It took a second corpus to kill it, exactly as it took his corpus to kill my 100.0%. One corpus per claimant is how both of us over-generalized, and there is no reason to believe I have stopped.

What I would generalize

Run their code, not your model of it. A GUI is not a reason to reimplement a parser; a twelve-line stub deletes the entire "you misread my code" branch of the argument, and what is left is arithmetic.

A fix report is a dataset. The closing comment on a bug you filed can contain more measurement than the bug did. Read it the way you read a transcript, and answer it with numbers, because the person who just fixed your finding is the best-calibrated reviewer your method will ever get.

Publish invariants with their denominator. "100.0%" was true of a population I forgot to name. The correction cost a table; leaving it unscoped would have cost every future reader who audits an agent-heavy tree.

When two corpora disagree, look for a field before you blame a version. A version split and a field split predict different shapes. Ask what each hypothesis forbids, then check for the forbidden thing. Zero differing main-path groups is the kind of zero that settles arguments.

Collapse under per-bucket max. It is correct when duplicates are identical, correct when they are running totals, and immune to the all-zero replay copy. Keep-first was quietly losing 46.2% of output on my tree.

Group per (file, id), or the sort will manufacture your finding for you. My 560 phantom groups came from splicing files together before grouping. Print the artifact count next to the real one so the check that lies gets caught by the check that doesn't.

Timeline

date (2026)	event
Aug 5	v3.0.0 announced on r/ClaudeAI; harness pointed at the token path the same day; #21 filed: 2.34× at three call sites
Aug 6	weltern confirms, promises a hotfix
Aug 8	v3.0.1 ships. Release note leads with "That is the fix, not a regression". His closing comment: 2.575× on his corpus, two findings passed back
Aug 8	Re-measurement posted: the split is `isSidechain`, not the writer version; running total confirmed 100%; the 560-group splice artifact named
Aug 9	Harness re-run: red on v3.0.0 as reported, exact on v3.0.1 through the new aggregator API

The layer underneath

All five parts run on TraceGuard's routing_audit, an append-only, message.id-keyed ingest of Claude Code transcripts into SQLite. Apache-2.0, pip install traceguard. The Clawdmeter harness and the shape-measurement script live in usage-tracker-audit/clawdmeter-dedup; the corpus generator builds a fake $HOME and touches nothing real.

The series scoreboard, since part four's count is stale: seven shipped fixes across four repos — three in splitrail, two in tokscale (v4.9.0 and v4.11.0), Clawdmeter v3.0.1, and viberank's high-water-mark fix, deployed. The eighth is still an open PR behind a 135-deep queue, which part three already explained.

The thing this part adds to the series is not a seventh notch. It is that the method survived contact with a maintainer who measured back, and got better for it: per-bucket max and per-(file, id) grouping are now in my collapse rules because his corpus broke my claim, and the version hypothesis died because mine broke his. Audit is not a thing you do to a codebase. Done right, it is a thing two corpora do to each other.

Li Zhuojun

My routing policy and my traces disagreed 96 times. Never once on the main thread.

Li Zhuojun — Sat, 08 Aug 2026 07:13:08 +0000

This is part four of a series about pointing an append-only audit log at things that count tokens. Part one found that a missing model line was half my agent overspend. Part two found two accounting bugs in splitrail. Part three found a 2.35× over-count in a 30k-star repo that the vendor's own documentation had warned about.

I promised this post at the end of part two and pushed it at the start of part three. Here it is.

The first three parts asked whether tools counted correctly. This one assumes the counting is right and asks the next question: given trustworthy numbers, do they say what I think they say about my own behaviour?

I wrote my routing policy down as a file. Then I measured which model actually ran, per decision, and priced the difference.

425 decisions. 96 deviations, 22.6%, $1,248.13.

Zero of them on the main thread.

What "stated versus revealed" means here

The stated half is a 30-line YAML file. The revealed half is an append-only trace store keyed on message.id. A deviation is a disagreement between them.

Three decisions in that file are doing most of the work.

It scores tiers, not models. frontier is Opus 4.8, Opus 4.7, Fable 5. mid is Sonnet 5. cheap is Haiku 4.5. Swapping Opus 4.8 for Fable 5 is not a deviation; both are frontier. expected_model exists in the file but never triggers a flag, only appears in the report. An audit that flags every model substitution reports noise, and an audit that reports noise gets switched off.

The grain is (unit, component). A unit is a stretch of my own prompts segmented by idle gap. One unit contains a main-thread stretch plus every subagent it spawned, so each (unit, component) pair gets one verdict and the dominant model wins.

One rule in that file exists only because the first draft was wrong. The draft said research work belongs on the cheap tier. Running it produced a pile of main-thread deviations, and they were not deviations. Interactive research with me in the loop on a frontier model is what I actually want; the draft policy was describing a different intent than the one I hold. So main × research-explore → frontier went in, with the reasoning written into the file's own header.

That correction is the part I would keep if I had to throw the rest away. A policy you never revise is not a policy, it is a wish. And the revision has to happen before you measure, or every measurement afterwards is scoring your behaviour against a straw man.

Why believe the revealed half

Before any of the numbers below mean anything, the trace log has to be recording what happened rather than what I meant. Here is the case, and I did not construct it. I found it while looking for something else.

Claude Fable 5 became available to me on 2026-06-10. My store's first Fable trace is timestamped 2026-06-10 00:00:38.

Daily Fable traces from there:

date	traces
2026-06-10	1,599
2026-06-11	1,541
2026-06-12	769
2026-06-13	31
2026-06-14 → 2026-07-01	0
2026-07-02	2,334

Eighteen days of nothing, then it comes back.

The 31 are the interesting part. All of them fall between 00:00:12 and 01:39:07, the first hundred minutes of that day. They belong to three sessions, and every one of those sessions had started earlier: 2026-06-12 14:57, 2026-06-10 00:00:38, 2026-06-10 11:28.

Not one new Fable session began on 2026-06-13.

That shape is hard to produce by choosing. If I had decided to stop using Fable, the last day would contain new sessions that ended early, because deciding to stop is something I do between tasks. What the log holds instead is drainage: work already in flight, finishing. A decision stops new work. An availability change stops new work and lets running work drain.

I never told the trace store about any of this. It has no concept of a model being available. It records model_id per message because that is the field that arrives, and by recording it faithfully for a boring reason it captured an external event nobody instrumented.

That is the whole argument for the revealed half, and it is the strongest form the argument can take: the log is credible about my intentions precisely because it does not know what my intentions were.

The result

Where the deviations are.

component	decisions	deviations	deviation cost
`main`	314	0	$0.00
`Explore`	14	0	$0.00
`workflow-subagent`	73	73	$1,146.23
`general-purpose`	23	23	$101.90
`product-manager`	1	0	$0.00
total	425	96 (22.6%)	$1,248.13

95 of the 96 are mid → frontier, $1,248.10. The last one is mid → cheap and costs three cents. So this is not scatter in both directions. It is one direction, in one place.

Those numbers are a snapshot of one window, 2026-05-30 to 2026-07-04, and I am going to hold myself to the standard I apply to everyone else in this series: they reproduce from a frozen copy of the database, not from the live one. I have since backfilled tags over the following five weeks and regenerated, which gives 784 decisions and 161 deviations at 20.5%. That is not a corrected version of the table above and I am not going to present it as one. The denominator changed, and every unit added carries an automatically assigned task type that no human has reviewed. Two numbers computed over different populations are two numbers.

Every deviation is a subagent. Every main-thread decision is compliant. The main thread is where I sit, where I pick a model on purpose, where I notice what I am spending. It never once got it wrong.

What the subagents were running. For each deviation, I asked what the parent session's main thread was running inside that unit's own time window.

verdict	n	cost
main ran exclusively the subagent's model	95	$1,248.10
main never on the subagent's model	1	$0.03
main mixed, or untestable	0	$0.00

Before that table means anything: 312 of 314 unit windows have a single-model main thread anyway. "Main was exclusively on one model" is nearly free. The comparison that carries the weight is against the subagents that did not deviate.

subagent decisions	ran the parent's model	ran something else
deviating (96)	95	1
compliant (15)	1	14

95 of 96 against 1 of 15. The 14 compliant non-matches are all Explore on Haiku, and the reason is not the one I gave when this post went up. I have no agent definition files at all, none at project level and none at user level. Explore ran Haiku because until Claude Code v2.1.198 the built-in Explore agent was hardcoded to Haiku regardless of what the parent was running.

My own archive brackets that change without having been built to look for it. Every Explore trace on v2.1.195 and below ran Haiku, 1,565 of them. Every Explore trace from v2.1.199 up ran whatever the main thread was running. The documented cutover is v2.1.198, and I have no versions in between.

So the contrast is not pinned versus unpinned. It is: a subagent with any model determination, from anywhere, does not inherit. A subagent with none does. Same conclusion, and it no longer rests on a claim about files I do not have.

The mechanism is documented upstream: a subagent whose definition omits model takes the main thread's. That it produced these 95 is inference, and I want to keep those two sentences next to each other. What the contrast rules out is deliberate per-subagent selection. Choosing on purpose 95 times would leave mixed windows, and there are none.

The one compliant match is the most useful row in the table. It is product-manager on Opus 4.8, $0.91. No rule in my policy covers that component, so it fell through to default_tier: frontier and scored compliant. It inherited exactly like the other 95. The policy's own default hid it.

A component with no rule cannot deviate. Which means gaps in a stated policy read as compliance, and the more incomplete your policy is, the better your compliance looks. That is a warning about this method, produced by the method, and it is worth more to me than another confirming number.

I got that table wrong the first time. The first run said 75 exclusive matches, 20 untestable, 1 non-match. traces.invoked_at is declared DATETIME, which gives the column NUMERIC affinity in SQLite; my open-ended upper bound '9999' converted to the integer 9999, and every integer sorts before every text value, so the comparison was false for all twenty rows with an open window. Both runs summed to 96 rows and $1,248.13. The cross-foot reconciled and the classification was still wrong.

What it cost, and what the fix is worth

$1,248.10 of $1,248.13 sits in the match bucket. The config-field hole is not most of the deviation cost, it is all of it but three cents.

Part one already called a missing model line half my overspend. That was an anecdote with a number attached. This is the same claim with a control group behind it, and the interesting part is not the size. It is the location.

Deviation clustered perfectly on the components where no human is choosing. I built this expecting to find undisciplined routing, and discipline turned out not to be the failing variable. It is a way of finding the places in a system where no decision is being made at all, and pricing them.

The 50% that is arithmetic, not a finding

79 Fable units, re-priced against Opus 4.8 with the token mix held fixed, saves $1,383.14, exactly 50.0%.

That number is a corollary of the price sheet and nothing else. Same tokenizer, same token counts, and Opus is half the price of Fable, so the saving is identically the price ratio. It could not have come out any other way, and quoting it without this paragraph attached would be quoting a division problem as a result.

The two questions that would make it a finding are whether quality held, and whether the token count itself changes on a different model. Neither is answered by arithmetic. One of them is answered badly below.

The blind eval, and three ways it is compromised

12 pairs, judged blind, original answer against a cold replay on the cheaper model.

The main result uses only the 8 pairs where the reviewer did not recognise the original: original wins 7, replay 1, tie 0.

Read that asymmetrically.

Replay wins-or-ties 1 of 8. That is the direction that would support downgrading, and it is weak. This is not evidence for downgrading.

Original wins 7 of 8. That cannot be read as the Fable premium being justified. The replay is a cold start without the original conversation's context, so its disadvantage may be missing context rather than model quality. This design does not separate them.

Three flaws, all mine:

The reviewer recognised 4 of 12. The reviewer was me, judging my own recent conversations, a 33% leak. The main result drops the recognised pairs; the fix for next time is a longer interval or somebody else.

Position was unbalanced. 9 of 12 originals landed in slot B through a deterministic hash that was not as balanced as I assumed, and 8 of the b_better verdicts picked the slot-B original. Winner is collinear with "slot B" and with "original", and this round cannot separate them.

I estimated the replay cost at $22.81. It came to $0.21. A factor of 110. I had sized the replay by the original consultation's full context footprint, when a self-contained replay is cold, uncached, prompt body only. estimate_costs now sizes by the replay payload.

That last one is the same mistake this series is about, one level up. I read the shape of the work and inferred a number instead of running it and measuring one. It cost me nothing because I ran it anyway. It is in here because the version of me that trusted the estimate would have concluded the replay experiment was too expensive to do.

Three things the source did while I was not looking

The window grew after I published it. 3,098 traces dated before 2026-07-05 were first ingested after 2026-07-05, carrying $917.33. Move the cut later and it grows: 3,658 traces and $1,194.06 at 2026-07-06, 4,796 and $1,573.23 at 2026-07-10.

What I measured is the arrival lag, not the cause. The likely mechanism is the one this series has been documenting from the outside since part one: resume and compact rewrite session files in place, so a session still open on 4 July grows records afterwards that carry their original timestamps. I have not proved that is what happened to these 3,098. An append-only ingest log is the only reason that sentence contains a number at all. A store that overwrites on re-read shows a window that quietly changed size with nothing recording that it did.

A model arrived that my own price sheet had never heard of. claude-opus-5, first seen 2026-07-25, 5,992 traces, 10.3% of the corpus. cost_usd was NULL on every one, because the pricing function refuses to guess when a model has no entry.

That refusal is correct and I am not going to change it. The consequence is not. $1,213.91 of list-price spend was missing from every total silently, as a zero rather than as an error, for two weeks. The model was also absent from the tier list, so those traces could not be scored compliant or deviant.

I want to be exact about how I found it, because the honest version is worse than the flattering one. There was a check. It had a passing test. It had already fired: the phrase "no price entry" appears ten times in my scheduler's stdout log. And the machine-readable file, the one an alert would actually read, records eighteen ingest runs over the same period with an empty warnings list on every single one, because the counter that incremented and the list that got serialised were two different things.

The alarm existed, worked, was tested, and was wired to a channel nobody reads.

Two separate failures, found in two different ways, and I want to keep them apart. I found the missing money by hand, while fact-checking this post. Why it had gone missing without a sound only came out during the fix, when somebody went looking for the check I assumed did not exist and found it sitting there, firing into the dark. My diagnosis at the time was "nobody built the alarm." That was wrong and it was the flattering version.

And it happened again while I was writing this. Between the first read for this section and the last, the scheduled ingest ran and added 678 traces. My archive's cost total moved by $0.91, because 645 of the 678 were on the unpriced model.

A tracker's dangerous state is not "wrong". A wrong number invites an argument. A NULL typed as zero invites nothing.

Does the rate hold on more data?

The corpus has since more than doubled. The 22.6% is at decision grain and needs manually corrected task tags, which the newer traces do not have, so this is a different and weaker measurement at cost grain. Four components have an expected tier fixed by the policy regardless of task type, which is what makes it computable at all.

window	scored cost	cross-tier cost share
before 2026-07-05	$7,198.14	20.0%
from 2026-07-05	$5,388.85	12.3%
same, with the unpriced model classified and priced	$6,602.70	13.7%

Report the sensitivity row, not just the flattering one. 20.0% to 13.7% is roughly a 30% relative reduction.

I am not claiming the audit caused it. The two windows differ in project mix, in workflow shape, and in which models existed. This is a before-and-after on populations that are not comparable, and it is offered as "the number did not blow up", nothing more.

Counterweights I owe you

The one-line fix was part one's finding. This post measured it. It did not discover it.

Co-occurrence is not causation. 95 exclusive matches and zero mixed windows against 1 of 15 among compliant subagents is consistency. I have not proved the inheritance mechanism produced them.

I was wrong twice while writing this, in the same shape both times. I claimed a cache-pricing defect had understated my whole store by $1,393.49 across 27,130 traces. The real figure is $88.75 across 3,918 rows, and $86.13 of that landed on rows my own earlier fix had created. I had measured what a broken function would produce and reported it as what the system had produced, because there were two costing paths and I had read one. Later I called a projected figure "void, it assumes a refresh that cannot occur" — and then the refresh occurred, one change later. Both times I turned "this is currently true" into "this is how it is."

Make that three times, and this one is after publication. I wrote that the compliant subagents were compliant because their definitions pin model. I have no definitions. The behaviour came from a built-in that changed in v2.1.198, which my own archive dates to within three patch versions without having been asked to. Same shape as the other two: I explained a measurement with a mechanism I had not checked.

22.6% is not a benchmark. One operator, one policy file, 36 days. If your subagents are pinned you should get zero, and that is the point rather than a caveat.

The blind eval is the weakest evidence here and its main result rests on 8 pairs.

The 50% is arithmetic.

My policy's coverage gaps read as compliance, and I do not have a good answer for that yet beyond noticing it.

What I would generalize

Audit where nobody is choosing. Compliance on the main thread was never in doubt and measuring it produced nothing. Every dollar was in the components with no human in the loop.

Write the policy down before you measure. The act of writing mine exposed a rule that was wrong, before a single deviation was computed. An unwritten policy cannot be violated, which is exactly why it feels like it never is.

Score tiers, not models. Otherwise the audit reports substitutions you meant and you stop reading it.

A coverage gap looks like compliance. Anything a stated policy fails to mention scores clean. Count the rules that never matched.

Absent is worse than wrong, and an alarm delivered to a channel nobody reads is absent.

Test the alternative, not the hypothesis. The inheritance finding got strong not from 95 matches but from zero mixed windows, because deliberate selection would have produced them.

Timeline

date (2026)	event
Jun 10	Fable 5 becomes available; 1,599 traces that day
Jun 13	31 Fable traces, all tails of earlier sessions, then eighteen days of none
Jul 04	routing policy v1 ratified at 16:04; decisions generated at 18:59
Jul 25	an unpriced model enters the corpus. The alarm fires to stdout, ten times
Aug 08	fact-checking this post finds it by hand. Two of my own claims collapse on the way

The layer underneath

All four parts run on TraceGuard's routing_audit module, an append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store. Apache-2.0, pip install traceguard.

If you want the cheapest possible version of this post's finding, you do not need any of that. Every deviation I found was in a component with no model determination of its own. If you write agent definitions, this lists the ones that will inherit:

ls .claude/agents/*.md >/dev/null 2>&1 \
  && grep -L '^model:' .claude/agents/*.md \
  || echo "no agent definitions - every subagent inherits the main thread"

The else branch is the case I was in and did not notice. I have no definitions at all, so the first version of this command told me nothing on my own machine. Nothing leaves your machine either way. It does not tell you what the gap cost you, which is the part that needed the trace store. It does tell you whether you have one.

Li Zhuojun

The vendor documents this bug. A 30k-star repo shipped it anyway.

Li Zhuojun — Tue, 04 Aug 2026 13:03:18 +0000

This is part three of a series about pointing an append-only audit log at things that count tokens. Part one found that a single missing model line was half my AI agent overspend. Part two found two accounting bugs in splitrail, both of which undercounted — one hid 54% of my messages.

This one points the other way. The target is claude-code-templates — 30.1k stars, 3.3k forks, the largest Claude Code template repo — and its analytics over-counted my tokens by 2.35×.

The part that changed how I wanted to write this up: while drafting the report I went looking for whether the behaviour was documented anywhere. It is. Anthropic's own SDK documentation has a warning box about it, and ships a reference implementation of the fix. The bug was not hiding. Four separate mechanisms surrounded this code and none of them closed the loop, and the reason they didn't is the actual subject of this post.

The fix has now been open for ten days. Two AI reviewers approved it. No human has replied. More on that near the end, because the silence turned out to carry information too.

(Part four is the stated-vs-revealed routing analysis I promised at the end of part two. It got bumped. This was worth the detour.)

The bug

calculateRealTokenUsage() summed message.usage over every parsed record.

Claude Code writes a single assistant message as several JSONL records — one per content block (thinking / text / tool_use) — and every one of those records repeats the same message.id and a byte-identical usage object. So each message's tokens were counted once per block, and everything downstream inflated by the average block count: conversation.tokens, summary.totalTokens, the per-project rollups, and the Total Tokens figure on the analytics dashboard.

Measured against my frozen ~50-day corpus:

property	value
distinct assistant `message.id`s	8,123
ids appearing on exactly one record	2,533
ids appearing on more than one record	5,590 (68.8%)
of those, ids where every record's `usage` is byte-identical	5,590 (100.0%)
records-per-id histogram	`{1: 2533, 2: 1351, 3: 3553, 4: 415, 5: 58, 6: 138, 7: 24, 8: 33}`
records ÷ messages	≈2.36×

Three records per message is the mode — thinking, then text, then a tool call.
Two-thirds of my assistant messages were being counted more than once, and on
every single one of them the repeated usage was byte-identical, which is the
detail that makes the over-count a multiplication rather than a rounding error.

Then I drove the analyzer over a synthetic corpus with known-exact totals:

field	before	after	ground truth
input_tokens	27,678	11,447	11,447
output_tokens	2,632,674	1,115,321	1,115,321
cache_creation	19,609,374	8,130,148	8,130,148
cache_read	59,626,489	24,447,923	24,447,923
messagesWithUsage	1,308	540	540 (distinct ids)

The line that turns this from a hunch into a diagnosis: before the fix, every field equalled the per-record sum exactly. Not approximately. Exactness is what pins the cause to record-level summation rather than to anything in parsing, and it is why the report could name the defect instead of reporting a discrepancy.

There is also a check that needs no corpus, no fixture and no golden file, which is the part I'd want you to take away: messagesWithUsage should equal the number of distinct message.ids. It equalled the number of records. One assertion, available from day one, would have failed loudly.

It was in the docs the whole time

Here is what I found when I went to check whether this was known. From Anthropic's Agent SDK cost-tracking guide, in a callout box:

Parallel tool calls produce multiple assistant messages whose nested BetaMessage shares the same id and identical usage. Always deduplicate by ID to get accurate per-step token counts.

And in the prose above it:

When Claude uses multiple tools in one turn, all messages in that turn share the same ID, so deduplicate by ID to avoid double-counting.

The page ships a working seenIds implementation. Its explanatory diagram is captioned, in part, "Step 1 has four assistant messages sharing the same ID and usage (count once)." There is a public issue on the Claude Code repo with the symptom in its title — #6805, "Token Usage Statistics Duplicated in stream-json Mode Causing Massive Cost Inflation". And codeburn, another tracker in this space, documents its own global seenMsgIds set doing precisely what the SDK page recommends.

So this is not an obscure edge case. The SDK that produces the data tells consumers, in a warning box, to deduplicate by ID. A widely used consumer of that data summed per record instead, for as long as the function existed.

I want to be exact about what my audit log did and did not contribute here, because the distinction is the whole point and I would rather make it myself than have it made at me: an append-only log was not necessary to discover this. Reading the documentation would have done it. What the log contributed was three things the documentation cannot give you — whether this particular codebase had the defect, what it cost (2.36×, on 8,123 real messages), and proof that the fix was exact rather than merely different.

Documentation describes hazards. Measurement finds instances. Those are different verbs and the gap between them is where this bug lived.

What the two AI reviewers said

The PR went through two commercial AI code reviewers. Both cleared it. Neither mentioned the over-count.

Greptile posted a summary and a verdict:

Confidence Score: 5/5 — The PR appears safe to merge. No blocking failure remains.

It reviewed both commits, and left the only 👍 the PR has received.

cubic ran twice. Its first pass reported "AI review completed with 2 reviews. Found 2 issues across 3 files" — a maintainability note about the function being duplicated across two files, and this:

P3: Distinct records can still be merged when an id equals a generated positional key or another record's uuid, causing token undercounting for that conversation. Namespace persisted identifiers and use a non-colliding generated key for id-less records.

That is a fair catch and it deserves credit. It is a real hazard: ConversationAnalyzer.js:297 sets id: item.message.id || item.uuid || null, so a uuid can already arrive in the id field, and my fallback chain put id, uuid and a positional key in a single namespace. On real Claude Code data the probability is approximately zero. The cost to eliminate it was four lines — namespace each source (id: / uuid: / idx:) — plus a test that goes red before the change and green after. cubic's second pass dropped to "1 issue found across 3 files."

Now the observation, which I want to state as flatly as I can.

Both tools read this function closely enough to reason about a hypothetical key collision in a line I had written minutes earlier. Neither remarked that the function's output had been wrong by a factor of two since the day it was written.

That is not a miss. By the standard each was applying, both answers were correct. A diff-scoped reviewer answers "is this change safe and self-consistent?" The pre-existing error was not in the diff, and nothing in a review context contains an independent measurement of what the right number is.

The asymmetry is worth naming precisely, because it generalizes past these two products: cubic's finding was a hypothesis about the shape of code. The bug was a discrepancy between a number and reality. Only the second kind of claim requires evidence from outside the repository. A reviewer holding the entire codebase in context still would not catch this one, because the code is internally consistent — it does exactly what it says it does, and what it says is wrong. You cannot read your way to that conclusion. Somebody has to measure.

The hole nobody is looking at

While I was in there: this repo runs no test suite on pull requests. The only check on #754 is the AI reviewer. "Eight unit tests pass" is a sentence I had to type into a comment, because nothing in the pipeline can demonstrate it.

With 135 open pull requests and 86 open issues, that is capacity, not negligence. But it is exactly the condition under which a silent numerical defect lives indefinitely, and it reorders the priorities: two AI reviewers and no test CI is a weaker correctness story than no AI reviewers and one test that asserts a total against a known input. Reviewers are additive. They are not a ground truth.

Ten days later

As of publication, #754 has been open ten days. Every check is green, the branch merges cleanly, and the two AI-reviewer verdicts haven't moved since day one. Human words on the thread: zero. No label, no assignee, no milestone. The one open review question — which of the two duplicated copies of the function should survive — can only be answered by a maintainer, and hasn't been. In those same ten days the repo gained about 200 stars and its open-PR queue grew from 129 to 135.

The same ten days, elsewhere in this series:

splitrail, 216 stars. I filed #220, a streaming-snapshot over-count, with the inflation ratios predicted from my corpus before any fix existed. A fix matching the prediction on every field was merged within two hours. Third shipped fix in that repo for this series.
tokscale, 4.6k stars. My drift report #994 was fixed and released in v4.9.0. From the maintainer's closing note: "The measurement in this report is what made it actionable — the before/after table landing exactly on the drifted prediction pinned it to persistence rather than parsing, and saved us looking in the wrong place."
viberank. In an issue I filed about leaderboard drift, yoo-minho (he maintains the submission tool for a rival leaderboard, clauderank) confirmed the same class at a scale I can't generate: a month of roughly $15.5k list-price usage whose month-to-date total fell 11% between two submissions 16 hours apart. His words: "A cumulative month-to-date total went down, which shouldn't be possible." He then wrote the countermeasure up as usage-drift-log, a six-field append-only record, and traceguard now implements it verbatim. One page of spec, two independent implementations.

So the method is not slow, and maintainers are not indifferent. Response time tracked queue depth, nothing else: the 216-star repo answered in hours, the 4.6k-star repo in days, and the 30.1k-star repo has 135 pull requests ahead of mine.

Which corrected something I had wrong when I started this series. I treated the merge as the finish line. It isn't — a merge is a statement about maintainer bandwidth, and only the fixture is a statement about the code. It runs in a minute on anyone's machine, red on upstream main, green on the branch, and it means the same thing whether or not anyone at the repo ever reads it. If the Total Tokens figure on that dashboard matters to you, you don't need to trust me and you don't need to wait for the queue: ./run_check.sh settles it.

Counterweights I owe you

The documentation already said it. Worth repeating, because it is the strongest argument against the framing I originally wanted for this post.
Other trackers get this right. ccusage, opcode and ccseva key on messageId:requestId; codeburn keys on a global seenMsgIds set. On my corpus the two strategies are equivalent — distinct (message.id, requestId) pairs came out equal to distinct message.ids, so message.id alone was sufficient here. There is no "trackers are broken" story to sell.
Both signs occur. splitrail undercounted. This one over-counted. No directional rule.
A hypothesis of mine was false. After part two I expected the subagent-directory blind spot to be an industry-wide pattern. I read thirteen trackers from source. It wasn't — only one unmaintained project had it, and a third-party had already reported the same class of gap in tokscale eighteen days before I filed mine. Saying so is more useful than the story I was hoping for.
In tokscale, my proposed fix lost the argument, and should have. I suggested the append-only store this series runs on. The maintainer turned it down: retention that never deletes would resurrect sessions a user deliberately removed, so he scoped retention to dedup keys that stay stable across files instead. I tested his rule — deleting a transcript dropped exactly that file's contents, 40 messages, 78,770 output tokens, nothing resurrected — and conceded in the thread. The audit finds the discrepancy. It has no special authority over the fix.
cubic found something my method wouldn't have. A latent hazard in code that had not yet met hostile data leaves no trace in any measurement, because it hasn't happened yet. The two approaches cover different failure classes, and I'd run both.

What I'd generalize

Reading is not measuring. Documentation, stars, code review, and more code review are all mechanisms for reading. Stacking more of them does not asymptotically produce a measurement. If nothing in your pipeline compares an output to an independently derived number, that comparison is not happening, however many eyes are on the diff.

Assert an invariant, not a total. Totals need fixtures, corpora and maintenance. messagesWithUsage == distinct message.ids needs none of those and would have failed on the first commit. Nearly every accounting path has an invariant like this hiding in it; the work is noticing which one.

Exactness is the evidence. "Roughly double" is a shrug that invites a debate about whose numbers are right. "Exactly the per-record sum, to the digit, on every field" is a diagnosis that ends the debate before it starts.

Ship the mechanism with the report. Four upstream fixes across two projects so far, three in splitrail and one in tokscale, the fastest merged two hours after the report — all because a runnable red-green fixture arrived attached to the claim rather than after someone asked for one.

When the same hazard hits several tools, converge on a contract, not more reports. The rewrite-drift class now has a one-page spec with two independent implementations behind two different leaderboards. That does more than a third bug report would have.

When the vendor documents a hazard, treat it as a test case, not as trivia. Every warning box in an SDK's docs is a bug someone has already shipped. That is a free list of things to go measure.

Timeline

date (2026)	event
Jul 24	corpus reconciliation flags the ratio: 8,123 distinct message ids spread across ~2.36× as many records
Jul 25	PR #754 filed with fixture, fix and measurements
Jul 25	Greptile: 5/5, safe to merge. cubic: 2 issues found
Jul 25	`7793bba8` — namespaced dedup key, two added tests, answering cubic's P3
Jul 25	cubic re-review: 1 issue. All checks pass, no conflicts
Jul 31	splitrail merges the series' third fix there (#220 → #222) two hours after the report; yoo-minho confirms rewrite drift on a $15.5k month, −11% silent
Aug 1	his six-field usage-drift-log spec adopted verbatim into traceguard (PR #30) — second independent implementation
Aug 3	tokscale ships v4.9.0, closing #994 with credit to the measurement
Aug 4	this post. #754: checks green, two bot approvals, zero human replies, 135 open PRs in the queue

The layer underneath

All three parts of this series run on TraceGuard's routing_audit module — an append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store. It is a few hundred lines and a database file, and everything above is downstream of one design decision: never let the source rewrite history.

Since part two, that layer grew a public edge: after every scheduled ingest it appends the six-field usage-drift-log record (frozen at first sight, never recomputed), so the drift class yoo-minho measured shows up as a warning line instead of a silent change.

Stable totals are the substrate, not the product. The thing worth building on top is stated-vs-revealed routing analysis, priced per decision — which model you said you'd route to, which one actually ran, and what the difference cost. That's part four, and this time I mean it.

pip install traceguard — Apache-2.0. The reproduction harness for this post is self-contained, runs in about a minute, and touches no real ~/.claude data — it builds a synthetic $HOME and hands the upstream analyzer nothing but a directory: usage-tracker-audit/cct-dedup-check.

An append-only audit log caught two accounting bugs in a 216-star usage tracker

Li Zhuojun — Sat, 25 Jul 2026 02:17:08 +0000

Last month I wrote about auditing 34 days of multi-model Claude Code usage and finding that a single missing model line was half my overspend. The core of that piece was a boring engineering decision: ingest every assistant message from Claude Code's session files into an append-only log keyed by message.id, and never let the source rewrite history.

This is the follow-up I didn't plan to write. The same log just caught two accounting bugs in splitrail, a 216-star Rust usage tracker for agentic CLIs — and within twelve days of the first report, both were fixed in an official release: one through my PR, one through a maintainer-authored patch that adopted the report's analysis. Here's the play-by-play, with the numbers.

Bug 1: your history quietly rewrites itself

Claude Code rewrites session JSONL files in place when you resume or compact a conversation. Between two scans of the same file, five of my assistant messages vanished — usage that was already incurred, gone from the record. Any tracker that recomputes totals from live files inherits this drift: yesterday's numbers change while you sleep.

I filed it as #200. The maintainer confirmed the mechanism the same day — splitrail re-read live files, so messages removed by resume/compaction disappeared from historical totals — and shipped a fix in 3.6.0: a local SQLite history store that persists normalized usage and merges it with current session data before deduplication. Fast, clean work.

One wrinkle: he suggested validating by comparing 3.5.9 vs 3.6.0 totals, expecting "3.6.0 should be higher." That's subtly wrong, and the distinction matters for anyone testing this class of fix: on a cold start the two versions are equal — the history store has nothing to restore yet. The divergence only appears across drift events. So I built the test to create one.

The validation protocol

Everything ran against a frozen snapshot (APFS clone) of ~/.claude/projects, so both versions scanned identical bytes, with every invocation under an isolated $HOME — own config, own history store, no upload path. Then:

#	assertion	result
A	cold-start parity: fresh 3.5.9 == fresh 3.6.0 on identical input	✅ identical
B	3.5.9 across a simulated resume/compact rewrite	✅ drops by exactly the removed usage
C	3.6.0 across the same rewrite	✅ totals unchanged — history store restores
D	3.6.0 restart stability, 3 consecutive runs	✅ byte-identical output

The simulated rewrite removes the last five assistant message-groups from the largest transcript — the same shape as the original drift event. The whole thing is now a self-contained regression fixture (PR #208, since merged): it fails red on 3.5.9 and passes green on 3.6.0, which is exactly what you want a regression test to do.

The part I care most about: on the transcripts splitrail scans, 3.6.0 agreed with my append-only log token-exact — 18,548,947 output tokens on both sides across 13.5k messages. Two independent implementations, different languages, different dedup strategies, same number to the digit. (The tiny message-count delta was zero-usage api-error records splitrail intentionally skips.) When two systems reconcile exactly, every remaining discrepancy is a finding, not noise.

Bug 2: the discrepancy that was a finding

Because the log records where every message came from, I could classify all of them against the live tree:

class	files	messages	output tokens
live, main transcript (what splitrail scans)	76	13,704	18,548,947
live, `subagents/**` transcript	1,423	16,160	13,738,324
vanished (file exists, message.id gone)	0	0	0
deleted (session file gone)	874	11,821	14,238,006

Row two is the second bug. Claude Code writes subagent transcripts (Task tool: Explore, general-purpose, custom agents) under projects/<slug>/<sessionId>/subagents/, at directory depth ≥ 4. Splitrail's discovery hard-caps at depth 2 in three places (WalkDir…min_depth(2).max_depth(2), a components() == 2 path check, and the glob pattern). Those files are structurally invisible — 54% of my live messages, and roughly a third of the dollars, never entered any total.

The model mix makes the blind spot vivid: splitrail saw 22 of my 5,059 Sonnet messages and 0 of 1,566 Haiku messages. Those models run almost exclusively inside subagents. If you delegate heavily to cheap models — which is exactly what cost-conscious agent users do — your tracker undercounts the most, on the workflows you optimized hardest.

Filed as #207 with the layout, a two-line repro, and a suggested fix. Two days later it was closed — and not by my patch. The maintainer wrote the fix himself: #209 "Include Claude Code subagent transcripts" (+428/−100 across seven files), whose description restates the issue's analysis and adopts its central caution — simply lifting the depth cap isn't enough; subagent records have to flow through the same local_hash dedup semantics as main transcripts, or you trade an undercount for a double-count. Both fixes shipped together in splitrail 3.6.1, which hit Homebrew within a day. On my corpus, the newly visible class is row two of the table above: 16,160 messages and 13.7M output tokens finally entering the totals. A fresh 3.6.1-vs-log reconciliation of the live tree is the obvious next check — part 3 starts there.

What I'd generalize

Live mutable files are not an audit trail. If your source can rewrite history, recomputation is not accounting. Append-only ingest with a stable per-record identity (message.id, last-write-wins on partials) is cheap insurance — mine is a few hundred lines and a SQLite file.

Reconcile token-exact or you know nothing. "Close enough" totals hide entire bug classes. It was exact agreement on the scanned subset that turned the remaining gap into two nameable, fixable defects instead of a shrug.

Decompose the gap before you blame anyone. "Your numbers are lower than mine" is an accusation; "the gap is exactly deleted-files + an unscanned directory class, here's the table" is a bug report a maintainer can act on in hours. Both turnarounds here — #200→#204 in eight days, #207→#209 in two — happened because the mechanism arrived with the report.

Frozen snapshot + isolated $HOME is the whole trick for tracker A/B. Clone the data, pin HOME (and XDG_STATE_HOME/XDG_DATA_HOME — on Linux, state dirs honor XDG and will escape your sandbox; CodeRabbit's review caught that one in my own fixture, fair is fair), and two binaries scanning identical bytes become a controlled experiment.

Timeline

date (2026)	event
Jul 11	#200 filed — resume/compact rewrites session JSONL in place
Jul 19	#204 merged; 3.6.0 ships the SQLite history store
Jul 20	4-assertion A/B validation on a frozen corpus; token-exact reconciliation vs the audit log
Jul 21	#207 filed (subagent blind spot); PR #208 opened; CodeRabbit review round addressed same day
Jul 22	PR #208 merged
Jul 22–23	Maintainer authors and merges #209; #207 closed; 3.6.1 released, Homebrew bumped

Twelve days, two bugs, zero arguments about whose numbers were right — the log settled every question before it became a debate.

The layer underneath

All of this ran on TraceGuard's routing_audit module (v1.1.0, "audit evidence layer") — the append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store that part 1 was built on. Stable totals are the substrate; the point of the layer above is stated-vs-revealed routing analysis, priced per decision — which model you said you'd route to, which one actually ran, and what the difference cost. That's part 3, and my totals finally sit still long enough to write it. (Apparently not a niche itch, either — the splitrail maintainer starred TraceGuard during this exchange and said he'd be interested in the audit part.)

pip install traceguard — Apache-2.0. The regression fixture is ~600 lines of stdlib Python + bash, merged in PR #208; the subagent fix shipped in splitrail 3.6.1.

A missing `model:` line was half my AI agent overspend — auditing 34 days of multi-model Claude Code

Li Zhuojun — Sun, 05 Jul 2026 16:41:59 +0000

I'm a solo developer running 10+ projects with a multi-model agent setup: Opus 4.8 as the main thread, Fable 5 (2× Opus price) as a decision advisor, Sonnet/Haiku for mechanical subagent work. The question I couldn't answer: which tasks actually deserve the expensive tier, and where was I overpaying?

So I built a routing audit as an opt-in module of my tracing SDK, backfilled ~26,000 traces (34 days) from Claude Code's local session logs, priced them at official list prices, and diffed my stated routing policy against my revealed routing behavior. Everything below is list-price equivalent (I'm on a subscription; API-paying teams see these numbers for real).

Finding 0: your usage history is quietly rewriting itself

Claude Code's resume/compact rewrites session JSONL files. Between two ingest runs, 5 messages vanished from a source file. If your usage numbers come from re-reading live session files (as most usage dashboards do), they drift. An append-only ingest log fixed it — and gave an unexpected validation: my Fable traces reproduce June's export-control suspension window to the hour (1,599 traces on Jun 10, exactly zero from Jun 14–30, resuming Jul 1).

Takeaway: snapshot your telemetry into an immutable store, or you're auditing sand.

Finding 1: one config line was most of the waste

I wrote my routing policy down as a declarative YAML (frontier/mid/cheap tiers × agent role × task type), then diffed it against 425 observed routing decisions.

22.6% deviated from my own policy — $1,248 of list-price spend.
Root cause of the single biggest cluster ($657, 53% of deviation cost): subagents inherit the parent thread's model unless the agent definition pins one. When my main thread ran the advisor model, mechanical subagents silently ran on it too — 2× Opus price for grep-tier work. Verified 19/19 against parent sessions, zero counterexamples.
The fix is one model: line per agent file.

Bonus honesty: my first audit pass "found" 28.2% deviation. A third of that was my policy file being wrong, not my behavior (I'd written rules I never actually intended). The first thing a routing audit fixes is your policy statement.

Finding 2: counterfactuals cost pennies; protocols are the hard part

"Was the 2× advisor model worth it?" I replayed 12 real advisor consults on the cheaper tier. Predicted cost: $22.81. Actual: $0.21 — my estimator was 110× off, because replays are cold-start prompts without the original ~660k-token cached context. Quality counterfactuals are essentially free; nobody skips them because of cost.

Then I blind-judged the 12 pairs myself, and the protocol failed in three instructive ways: I recognized my own original conversations in 4/12 pairs (33% leak); my deterministic position assignment put 9/12 originals in slot B (so "B wins" was confounded with "original wins"); and originals had full context while replays didn't. Net result: the advisor-premium question is still open. Protocol v2: third-party judge, position-balanced, longer delay. I'm publishing the failure because an audit tool's dev log should look like this.

If you run agent fleets

Pin model: in every subagent definition. Inheritance is silent and expensive.
Snapshot usage logs immutably. Live session files rewrite themselves.
Write your routing policy down. The diff between stated policy and revealed routing is where the money hides — and half of what you'll find is that your policy was never real.

The module is contract-external, local-first (nothing leaves your machine), built on traceguard — a point-in-time-correct LLM instrumentation SDK originally built for quant pipelines, where "which model knew what, when" is an audit requirement, not a nice-to-have.

Ask: I'm doing a handful of free routing audits for teams running multi-model agent fleets (especially where quality is measurable — trading, data pipelines, eval-gated workflows), in exchange for anonymized learnings. DM or open an issue.

The `epsActual` That Wasn't: 15% of an LLM Backtest's Trades Were Decided on Data That Didn't Exist Yet

Li Zhuojun — Thu, 18 Jun 2026 11:16:59 +0000

We were backtesting an LLM-driven earnings signal against a field called epsActual — the kind of field everyone treats as ground truth. It isn't.

About 41.4% of those "actual" values were different from what the vendor had first reported. About 15.3% differed enough to flip a tradeable decision. When we re-ran the backtest using only the values that actually existed at each decision date, the strategy kept ~73% of its returns and ~82% of its Sharpe. The rest was look-ahead bias — and it rode in through a field whose name promised it was final.

This is a writeup of how we found it, how we measured it honestly, and the one-line invariant that turns it from a silent inflation into a loud test failure.

The setup

The signal is a post-earnings drift play: at each earnings print, an LLM scores the release and we take a position. To backtest it you replay history — for every past print, reconstruct what the model would have decided, then check what happened next.

That reconstruction needs one obviously-trustworthy input: what the earnings number actually was. Our vendor exposes exactly that, in a field named epsActual. "Actual." Final. Settled. You query a print from two years ago and get a number back. What could go wrong?

The invisible killer

Vendor "actuals" are not frozen at print time. They get backfilled, corrected, and restated — sometimes the next day, sometimes months later. Restatements, late filings, parser fixes, standardization passes: all of them quietly rewrite history. The value you query today for a 2023 print is not, in general, the value that was available the day after that print.

This is textbook look-ahead bias, and it's especially dangerous here because it doesn't look like leakage. Nobody fed the model future data on purpose. It rode in on a field everyone trusts — and "actual" is about the most trustworthy-sounding name a field can have. A backtest built on today's epsActual is quietly asking the model to react to numbers that, on the decision date, did not yet exist.

How we measured it honestly

You can't detect this from a single snapshot of the database — by definition the revision has already overwritten the original. So we built a forward-polling harness: poll the vendor on a schedule, snapshot every value we care about, and watch for changes over time. It had accumulated ~1,400 snapshots in the first day of polling.

The decision that mattered most:

Detect revisions by the value itself, not by the vendor's lastUpdated timestamp.

lastUpdated is unreliable — it doesn't reliably fire on silent backfills, and trusting it would have hidden exactly the revisions we were hunting. So change detection keys on the value-tuple: if any tracked field changes between two snapshots, that's a revision, regardless of what the metadata claims.

# Revision = the tracked value-tuple changed between snapshots,
# NOT "the vendor bumped lastUpdated".
def is_revision(prev_snapshot, curr_snapshot, tracked_fields):
    prev = tuple(prev_snapshot[f] for f in tracked_fields)
    curr = tuple(curr_snapshot[f] for f in tracked_fields)
    return prev != curr

To quantify the trading impact, we compared two backtests over a four-month point-in-time window: a naive one using today's revised epsActual, and an as-of one using only each value as first seen on (or before) the decision date.

What we found

41.4% of epsActual values (896/2163) differed between first-seen and final.
15.3% of cases (332/2163) differed enough to flip a tradeable decision — a sign change or a threshold crossing in the signal.
Over the four-month window, the as-of backtest retained ~73% of the naive backtest's returns and ~82% of its Sharpe. (The FINAL leg keeps drifting as the vendor keeps revising, so treat the ratio as more stable than the levels.)
Read inversely: roughly a quarter of the headline returns, and a fifth of the Sharpe, were look-ahead artifacts.

The encouraging half: most of the strategy survives honest data. The sobering half: a naive backtest overstated it by a wide margin, and a meaningful fraction of "winning" trades were decided on numbers that did not exist at decision time. A 15% decision-flip rate is not noise you can wave away.

Why this is structural, not a one-off

The natural reaction is "okay, we'll be careful with that field." That doesn't hold. The risk is reintroduced by every new feature, every new vendor, every rerun, every teammate who reaches for "the actual value." Carefulness is a property of a person on a good day; as-of correctness has to be a property of the pipeline.

So treat the question "could this value have been known at the decision time we're simulating?" as an invariant the code enforces and CI checks. A vendor "actual" is time-versioned reference data: it only becomes valid at the instant you first observed it. Use it to decide before that instant and you're using a value from the future.

That's exactly what the look-ahead invariant below checks — it requires valid_from <= feature_as_of:

from traceguard.validators.lookahead import validate_reference_timing

# The eps "actual" is time-versioned reference data: valid_from is when this
# specific value first existed (first-seen in our snapshots), feature_as_of is
# the decision moment we are simulating.
validate_reference_timing(
    valid_from=eps_first_seen,    # when this value actually existed
    feature_as_of=decision_date,  # the moment we're simulating
    kind="vendor_eps_actual",
)  # raises InvariantViolation if eps_first_seen > decision_date

When a value is used before its availability timestamp, the run fails loudly rather than silently inflating a Sharpe ratio.

Two kinds of look-ahead — don't conflate them

It's worth being precise about scope. There are two distinct kinds of look-ahead bias in LLM pipelines:

Training contamination — the model itself was pre-trained on the future you're predicting, so it "recalls" rather than reasons. That's a separate research problem (membership-inference tests, point-in-time LLMs, claim-level temporal verification), and it needs different tooling.
Harness / pipeline leakage — your code uses a value, prompt, or model that didn't exist at the simulated time. This story is entirely about this kind, and it's the kind a pipeline can be made to refuse structurally.

Both matter. They are not the same problem, and conflating them is how teams "fix" one and ship the other.

A checklist you can apply today

Treat every actual / final / reported vendor field as a moving target until you've proven otherwise with your own snapshots.
Detect revisions by value, not by the vendor's update timestamp.
Backtest on as-of (first-seen) data, and explicitly measure the gap against revised data. That gap is your look-ahead tax — quantify it instead of assuming it's zero.
Encode "known at decision time?" as a CI invariant, so the failure mode is a red test, not a flattering backtest.

Limitations

One vendor, one field, a four-month window. The exact percentages are dataset-specific and should not be read as universal constants — your numbers will differ. And again: this addresses harness leakage only, not whether the model itself has seen the future.

The validators and point-in-time instrumentation here are part of traceguard — an open-source Python library for point-in-time-correct LLM instrumentation: a model registry that refuses anachronistic picks, a git-tracked prompt registry, canonical input hashing, and look-ahead invariants you call in CI. It's not a dashboard — it exports OpenTelemetry spans into Langfuse / Phoenix, so it sits underneath your observability stack and keeps the timeline honest.

pip install traceguard

If you've been burned by a backtest that looked great and meant nothing, I'd genuinely like to hear how it happened — that's the failure mode this is built to catch.