This is part four of a series about pointing an append-only audit log at things that count tokens. Part one found that a missing model line was half my agent overspend. Part two found two accounting bugs in splitrail. Part three found a 2.35× over-count in a 30k-star repo that the vendor's own documentation had warned about.
I promised this post at the end of part two and pushed it at the start of part three. Here it is.
The first three parts asked whether tools counted correctly. This one assumes the counting is right and asks the next question: given trustworthy numbers, do they say what I think they say about my own behaviour?
I wrote my routing policy down as a file. Then I measured which model actually ran, per decision, and priced the difference.
425 decisions. 96 deviations, 22.6%, $1,248.13.
Zero of them on the main thread.
What "stated versus revealed" means here
The stated half is a 30-line YAML file. The revealed half is an append-only trace store keyed on message.id. A deviation is a disagreement between them.
Three decisions in that file are doing most of the work.
It scores tiers, not models. frontier is Opus 4.8, Opus 4.7, Fable 5. mid is Sonnet 5. cheap is Haiku 4.5. Swapping Opus 4.8 for Fable 5 is not a deviation; both are frontier. expected_model exists in the file but never triggers a flag, only appears in the report. An audit that flags every model substitution reports noise, and an audit that reports noise gets switched off.
The grain is (unit, component). A unit is a stretch of my own prompts segmented by idle gap. One unit contains a main-thread stretch plus every subagent it spawned, so each (unit, component) pair gets one verdict and the dominant model wins.
One rule in that file exists only because the first draft was wrong. The draft said research work belongs on the cheap tier. Running it produced a pile of main-thread deviations, and they were not deviations. Interactive research with me in the loop on a frontier model is what I actually want; the draft policy was describing a different intent than the one I hold. So main × research-explore → frontier went in, with the reasoning written into the file's own header.
That correction is the part I would keep if I had to throw the rest away. A policy you never revise is not a policy, it is a wish. And the revision has to happen before you measure, or every measurement afterwards is scoring your behaviour against a straw man.
Why believe the revealed half
Before any of the numbers below mean anything, the trace log has to be recording what happened rather than what I meant. Here is the case, and I did not construct it. I found it while looking for something else.
Claude Fable 5 became available to me on 2026-06-10. My store's first Fable trace is timestamped 2026-06-10 00:00:38.
Daily Fable traces from there:
| date | traces |
|---|---|
| 2026-06-10 | 1,599 |
| 2026-06-11 | 1,541 |
| 2026-06-12 | 769 |
| 2026-06-13 | 31 |
| 2026-06-14 → 2026-07-01 | 0 |
| 2026-07-02 | 2,334 |
Eighteen days of nothing, then it comes back.
The 31 are the interesting part. All of them fall between 00:00:12 and 01:39:07, the first hundred minutes of that day. They belong to three sessions, and every one of those sessions had started earlier: 2026-06-12 14:57, 2026-06-10 00:00:38, 2026-06-10 11:28.
Not one new Fable session began on 2026-06-13.
That shape is hard to produce by choosing. If I had decided to stop using Fable, the last day would contain new sessions that ended early, because deciding to stop is something I do between tasks. What the log holds instead is drainage: work already in flight, finishing. A decision stops new work. An availability change stops new work and lets running work drain.
I never told the trace store about any of this. It has no concept of a model being available. It records model_id per message because that is the field that arrives, and by recording it faithfully for a boring reason it captured an external event nobody instrumented.
That is the whole argument for the revealed half, and it is the strongest form the argument can take: the log is credible about my intentions precisely because it does not know what my intentions were.
The result
Where the deviations are.
| component | decisions | deviations | deviation cost |
|---|---|---|---|
main |
314 | 0 | $0.00 |
Explore |
14 | 0 | $0.00 |
workflow-subagent |
73 | 73 | $1,146.23 |
general-purpose |
23 | 23 | $101.90 |
product-manager |
1 | 0 | $0.00 |
| total | 425 | 96 (22.6%) | $1,248.13 |
95 of the 96 are mid → frontier, $1,248.10. The last one is mid → cheap and costs three cents. So this is not scatter in both directions. It is one direction, in one place.
Those numbers are a snapshot of one window, 2026-05-30 to 2026-07-04, and I am going to hold myself to the standard I apply to everyone else in this series: they reproduce from a frozen copy of the database, not from the live one. I have since backfilled tags over the following five weeks and regenerated, which gives 784 decisions and 161 deviations at 20.5%. That is not a corrected version of the table above and I am not going to present it as one. The denominator changed, and every unit added carries an automatically assigned task type that no human has reviewed. Two numbers computed over different populations are two numbers.
Every deviation is a subagent. Every main-thread decision is compliant. The main thread is where I sit, where I pick a model on purpose, where I notice what I am spending. It never once got it wrong.
What the subagents were running. For each deviation, I asked what the parent session's main thread was running inside that unit's own time window.
| verdict | n | cost |
|---|---|---|
| main ran exclusively the subagent's model | 95 | $1,248.10 |
| main never on the subagent's model | 1 | $0.03 |
| main mixed, or untestable | 0 | $0.00 |
Before that table means anything: 312 of 314 unit windows have a single-model main thread anyway. "Main was exclusively on one model" is nearly free. The comparison that carries the weight is against the subagents that did not deviate.
| subagent decisions | ran the parent's model | ran something else |
|---|---|---|
| deviating (96) | 95 | 1 |
| compliant (15) | 1 | 14 |
95 of 96 against 1 of 15. And the 14 compliant non-matches are all Explore on Haiku, which is the component whose agent definition pins model. Pinned subagents ignore the main thread. Unpinned ones don't.
The mechanism is documented upstream: a subagent whose definition omits model takes the main thread's. That it produced these 95 is inference, and I want to keep those two sentences next to each other. What the contrast rules out is deliberate per-subagent selection. Choosing on purpose 95 times would leave mixed windows, and there are none.
The one compliant match is the most useful row in the table. It is product-manager on Opus 4.8, $0.91. No rule in my policy covers that component, so it fell through to default_tier: frontier and scored compliant. It inherited exactly like the other 95. The policy's own default hid it.
A component with no rule cannot deviate. Which means gaps in a stated policy read as compliance, and the more incomplete your policy is, the better your compliance looks. That is a warning about this method, produced by the method, and it is worth more to me than another confirming number.
I got that table wrong the first time. The first run said 75 exclusive matches, 20 untestable, 1 non-match. traces.invoked_at is declared DATETIME, which gives the column NUMERIC affinity in SQLite; my open-ended upper bound '9999' converted to the integer 9999, and every integer sorts before every text value, so the comparison was false for all twenty rows with an open window. Both runs summed to 96 rows and $1,248.13. The cross-foot reconciled and the classification was still wrong.
What it cost, and what the fix is worth
$1,248.10 of $1,248.13 sits in the match bucket. The config-field hole is not most of the deviation cost, it is all of it but three cents.
Part one already called a missing model line half my overspend. That was an anecdote with a number attached. This is the same claim with a control group behind it, and the interesting part is not the size. It is the location.
Deviation clustered perfectly on the components where no human is choosing. I built this expecting to find undisciplined routing, and discipline turned out not to be the failing variable. It is a way of finding the places in a system where no decision is being made at all, and pricing them.
The 50% that is arithmetic, not a finding
79 Fable units, re-priced against Opus 4.8 with the token mix held fixed, saves $1,383.14, exactly 50.0%.
That number is a corollary of the price sheet and nothing else. Same tokenizer, same token counts, and Opus is half the price of Fable, so the saving is identically the price ratio. It could not have come out any other way, and quoting it without this paragraph attached would be quoting a division problem as a result.
The two questions that would make it a finding are whether quality held, and whether the token count itself changes on a different model. Neither is answered by arithmetic. One of them is answered badly below.
The blind eval, and three ways it is compromised
12 pairs, judged blind, original answer against a cold replay on the cheaper model.
The main result uses only the 8 pairs where the reviewer did not recognise the original: original wins 7, replay 1, tie 0.
Read that asymmetrically.
Replay wins-or-ties 1 of 8. That is the direction that would support downgrading, and it is weak. This is not evidence for downgrading.
Original wins 7 of 8. That cannot be read as the Fable premium being justified. The replay is a cold start without the original conversation's context, so its disadvantage may be missing context rather than model quality. This design does not separate them.
Three flaws, all mine:
The reviewer recognised 4 of 12. The reviewer was me, judging my own recent conversations, a 33% leak. The main result drops the recognised pairs; the fix for next time is a longer interval or somebody else.
Position was unbalanced. 9 of 12 originals landed in slot B through a deterministic hash that was not as balanced as I assumed, and 8 of the b_better verdicts picked the slot-B original. Winner is collinear with "slot B" and with "original", and this round cannot separate them.
I estimated the replay cost at $22.81. It came to $0.21. A factor of 110. I had sized the replay by the original consultation's full context footprint, when a self-contained replay is cold, uncached, prompt body only. estimate_costs now sizes by the replay payload.
That last one is the same mistake this series is about, one level up. I read the shape of the work and inferred a number instead of running it and measuring one. It cost me nothing because I ran it anyway. It is in here because the version of me that trusted the estimate would have concluded the replay experiment was too expensive to do.
Three things the source did while I was not looking
The window grew after I published it. 3,098 traces dated before 2026-07-05 were first ingested after 2026-07-05, carrying $917.33. Move the cut later and it grows: 3,658 traces and $1,194.06 at 2026-07-06, 4,796 and $1,573.23 at 2026-07-10.
What I measured is the arrival lag, not the cause. The likely mechanism is the one this series has been documenting from the outside since part one: resume and compact rewrite session files in place, so a session still open on 4 July grows records afterwards that carry their original timestamps. I have not proved that is what happened to these 3,098. An append-only ingest log is the only reason that sentence contains a number at all. A store that overwrites on re-read shows a window that quietly changed size with nothing recording that it did.
A model arrived that my own price sheet had never heard of. claude-opus-5, first seen 2026-07-25, 5,992 traces, 10.3% of the corpus. cost_usd was NULL on every one, because the pricing function refuses to guess when a model has no entry.
That refusal is correct and I am not going to change it. The consequence is not. $1,213.91 of list-price spend was missing from every total silently, as a zero rather than as an error, for two weeks. The model was also absent from the tier list, so those traces could not be scored compliant or deviant.
I want to be exact about how I found it, because the honest version is worse than the flattering one. There was a check. It had a passing test. It had already fired: the phrase "no price entry" appears ten times in my scheduler's stdout log. And the machine-readable file, the one an alert would actually read, records eighteen ingest runs over the same period with an empty warnings list on every single one, because the counter that incremented and the list that got serialised were two different things.
The alarm existed, worked, was tested, and was wired to a channel nobody reads.
Two separate failures, found in two different ways, and I want to keep them apart. I found the missing money by hand, while fact-checking this post. Why it had gone missing without a sound only came out during the fix, when somebody went looking for the check I assumed did not exist and found it sitting there, firing into the dark. My diagnosis at the time was "nobody built the alarm." That was wrong and it was the flattering version.
And it happened again while I was writing this. Between the first read for this section and the last, the scheduled ingest ran and added 678 traces. My archive's cost total moved by $0.91, because 645 of the 678 were on the unpriced model.
A tracker's dangerous state is not "wrong". A wrong number invites an argument. A NULL typed as zero invites nothing.
Does the rate hold on more data?
The corpus has since more than doubled. The 22.6% is at decision grain and needs manually corrected task tags, which the newer traces do not have, so this is a different and weaker measurement at cost grain. Four components have an expected tier fixed by the policy regardless of task type, which is what makes it computable at all.
| window | scored cost | cross-tier cost share |
|---|---|---|
| before 2026-07-05 | $7,198.14 | 20.0% |
| from 2026-07-05 | $5,388.85 | 12.3% |
| same, with the unpriced model classified and priced | $6,602.70 | 13.7% |
Report the sensitivity row, not just the flattering one. 20.0% to 13.7% is roughly a 30% relative reduction.
I am not claiming the audit caused it. The two windows differ in project mix, in workflow shape, and in which models existed. This is a before-and-after on populations that are not comparable, and it is offered as "the number did not blow up", nothing more.
Counterweights I owe you
The one-line fix was part one's finding. This post measured it. It did not discover it.
Co-occurrence is not causation. 95 exclusive matches and zero mixed windows against 1 of 15 among compliant subagents is consistency. I have not proved the inheritance mechanism produced them.
I was wrong twice while writing this, in the same shape both times. I claimed a cache-pricing defect had understated my whole store by $1,393.49 across 27,130 traces. The real figure is $88.75 across 3,918 rows, and $86.13 of that landed on rows my own earlier fix had created. I had measured what a broken function would produce and reported it as what the system had produced, because there were two costing paths and I had read one. Later I called a projected figure "void, it assumes a refresh that cannot occur" — and then the refresh occurred, one change later. Both times I turned "this is currently true" into "this is how it is."
22.6% is not a benchmark. One operator, one policy file, 36 days. If your subagents are pinned you should get zero, and that is the point rather than a caveat.
The blind eval is the weakest evidence here and its main result rests on 8 pairs.
The 50% is arithmetic.
My policy's coverage gaps read as compliance, and I do not have a good answer for that yet beyond noticing it.
What I would generalize
Audit where nobody is choosing. Compliance on the main thread was never in doubt and measuring it produced nothing. Every dollar was in the components with no human in the loop.
Write the policy down before you measure. The act of writing mine exposed a rule that was wrong, before a single deviation was computed. An unwritten policy cannot be violated, which is exactly why it feels like it never is.
Score tiers, not models. Otherwise the audit reports substitutions you meant and you stop reading it.
A coverage gap looks like compliance. Anything a stated policy fails to mention scores clean. Count the rules that never matched.
Absent is worse than wrong, and an alarm delivered to a channel nobody reads is absent.
Test the alternative, not the hypothesis. The inheritance finding got strong not from 95 matches but from zero mixed windows, because deliberate selection would have produced them.
Timeline
| date (2026) | event |
|---|---|
| Jun 10 | Fable 5 becomes available; 1,599 traces that day |
| Jun 13 | 31 Fable traces, all tails of earlier sessions, then eighteen days of none |
| Jul 04 | routing policy v1 ratified at 16:04; decisions generated at 18:59 |
| Jul 25 | an unpriced model enters the corpus. The alarm fires to stdout, ten times |
| Aug 08 | fact-checking this post finds it by hand. Two of my own claims collapse on the way |
The layer underneath
All four parts run on TraceGuard's routing_audit module, an append-only, message.id-keyed ingest of Claude Code transcripts into a SQLite trace store. Apache-2.0, pip install traceguard.
If you want the cheapest possible version of this post's finding, you do not need any of that. Every deviation I found was in a component whose agent definition has no model: field, and you can list yours with one command:
grep -L '^model:' .claude/agents/*.md
Nothing leaves your machine. It does not tell you what the gap cost you, which is the part that needed the trace store. It does tell you whether you have one.
Li Zhuojun
Top comments (0)