"Your cache hit rate is low" — true, and worth $0.16

Anthropic emailed our org last week: prompt cache hit rate is low, and caching repeated content "could save up to 64% of direct API spend." The email is careful with scope — direct API only, Claude Code excluded because it manages caching itself. I went to check what that 64% was made of.

Our direct API traffic over the period: 16 calls, $0.25 total, average prompt 90 tokens. They come from traceguard's rerun harness, which replays self-contained consults as fresh single-turn calls — every prompt unique by construction. A workload like that cannot hit cache: there is no repeated prefix across calls, and 90 tokens is far below the minimum Anthropic will cache at all (1,024 tokens on Opus 4.8, the model those calls ran on; 512 on Opus 5 / Fable 5). Hit rate 0%, structurally. 64% of $0.25 is $0.16.

The metric was right. The money wasn't there.

The side the email excluded

The interesting numbers sit in the traffic the email correctly excluded. We ingest our own Claude Code session logs into traceguard's trace store (Claude Code transcripts carry no cost field, so we compute list price from usage). 71 days, 158 sessions, 58,753 API messages. This is section 1 of the command output, unedited:

model	messages	prompt tok	hit rate	input cost	no-cache	saved	saved %
claude-opus-4-8	23,759	5,535,807,891	96.1%	$4,616.39	$27,691.44	$23,075.05	83.3%
claude-fable-5	15,291	3,583,420,473	96.2%	$5,847.08	$35,834.20	$29,987.13	83.7%
claude-opus-5	6,316	1,750,760,343	97.7%	$1,205.08	$8,753.80	$7,548.72	86.2%
claude-sonnet-5	11,139	1,326,223,701	95.1%	$422.96	$2,652.45	$2,229.49	84.1%
claude-haiku-4-5-20251001	1,573	57,312,161	93.9%	$9.76	$57.31	$47.56	83.0%
claude-opus-4-7	64	11,045,966	96.6%	$9.04	$55.23	$46.19	83.6%
claude-sonnet-4-5-20250929	2	42,410	0.0%	n/a	n/a	n/a	n/a
(none)	609	0	n/a	n/a	n/a	n/a	n/a
TOTAL	58,753	12,264,612,945	96.2%	$12,110.31	$75,044.44	$62,934.13	83.9%

Hit rate is token-weighted; costs are the input side only, at list price. The two n/a rows are deliberate — no published price for a model means no guessed money, tokens counted anyway.

Caching nobody at our org configured cuts the input side by 83.9%. With output included, the whole bill lands at 1/5.5 of the no-cache counterfactual. There is nothing left to optimize here; the remaining spend is mostly first-write premiums on genuinely new content.

The keep-alive ping idea

There's a folk optimization going around: ping your session on a timer so the cache stays warm and the next turn hits. The mechanics say it could work — the cache TTL is 5 minutes by default, every hit refreshes it for free, and a 1-hour TTL costs 2× on writes. Instead of arguing, we ran the counterfactual on our own gap data.

97.3% of our in-session gaps are under 5 minutes; they need nothing. The addressable prize is 422 gaps longer than an hour: the context rewrites right after them cost at most $1,913 over 71 days (an overestimate, since it counts genuinely new content as rewrite). Bridging those same gaps with a ping every 55 minutes would have taken 6,765 pings costing $2,009 in cache reads alone, before counting the pings' own output tokens. The strategy loses money on our data, under assumptions tilted in its favor.

The reason is boring: the 1-hour TTL removed the ping's niche. Our logs show Claude Code already routes nearly all cache writes into the 1h bucket (opus-4-8: 155.9M tokens written at 1h TTL vs 56.4M at 5m). Pinging is a hack from before extended TTL existed. If a tool sells you auto-ping in 2026, ask it for the counterfactual on your logs first.

(Superseded — the Aug 18 note below has the current numbers.)

Update, Aug 16. Reid Marlow asked in the comments for cost broken out by idle gap rather than in aggregate. I ran it, and it overturns the paragraph above. Split by bucket: the 183 gaps of 1–4h would have cost $81 in pings against at most $887 of rewrite, an 11:1 win, while the 239 gaps over 4h cost $1,928 against at most $1,026. The refusal I published is a win and a loss added together. A policy that can actually be run (ping every 55 minutes, give up after 4 hours idle) costs $316 including every ping burned on the 239 gaps that outlive the cap, against at most $887 avoided. It pays.

So the correct claim is narrower than the one this section makes: unbounded pinging loses, capped pinging probably pays. Probably, because both sides of that comparison lean pro-ping. The $887 is an upper bound, since cache_creation mixes the re-established prefix with whatever the turn genuinely added, and the $316 charges pings as pure 0.1x reads of a frozen prompt. Under that tilt a refusal is safe and an endorsement is only as wide as its margin, and this one is 2.8x rather than 11x.

I had the aggregate and stopped there, which is the same mistake the email made: a number averaged over cases that behave differently is not a decision. The per-bucket money and both verdicts are in cache_audit in the repo now and ship in the next release; 1.2.0 prints the aggregate only.

(Numbers superseded by the Aug 18 note below. The reasoning stands.)

Update, Aug 17. The numbers in the note above are superseded. Not because the direction was wrong, but because 4 hours was picked to line up with a table boundary rather than solved for. Sweeping every cap from 1h to 12h in 15-minute steps, the best single value is 10h. It beats 9h45m by $7.63. Then I measured something this section had only assumed, and that one measurement moved the same cap by $18.76. A correction larger than the distance between first and second place can reorder them, so a single number is the wrong output. The answer is a band: ping every 55 minutes, give up somewhere between 9 and 12 hours idle. Anywhere in that band costs under 10% of the optimum.

Net over the window, after the ping bill: $663.69 to $811.30. That is two runs rather than an error bar. The low end treats every gap whose model I cannot determine as a cache-invalidating switch, the high end deducts only the ones I can prove. 54 of 432 gaps have no model recorded, and nothing in the data says which way they go.

What I measured is whether the model changed across the gap. A keep-alive only holds a cache for the model it was written on, so a switch makes every ping before it worthless. This section listed that as an assumption. It is 18 of the 378 gaps where both sides are known, and the rate climbs with idle time: zero in the shortest decile, 23.1% past 24 hours. All 18 were opus-4-8 and fable-5, in one direction or the other. Not one was a switch down to something cheaper. Leaving that out was not adding noise to the cap. It was pushing the cap long.

Every number here comes from 2026-05-30 to 2026-08-16, pinned behind a --benchmark flag so the run reproduces. That flag and the sweep are on main, not in 1.2.0, and land with the rest of this in the next release. The Aug 16 note was overconfident in the same shape as the paragraph it corrected: I had a number and quoted it before asking how wide it was. Checking that note turned up three more errors, one of them a denominator error of my own. That one is a separate post.

Update, Aug 18. The window did not do what I said it would. I pinned 2026-05-30 to 2026-08-16 so the run would reproduce, and it does not. The same window over the same store went from 432 expired gaps to 439, and from 168 sessions to 174, in under a day. Ingest reads a directory of transcripts, so a file that turns up late still carries messages from inside the window. Closing the dates closes the timestamps, not the corpus. The figures above moved with it: the two ends are now $650.53 and $806.82, not $663.69 and $811.30 (argmax_reference_only.net_usd in that file — the field named recommended_cap_band is a different interval).

So I stopped pointing at a command. The run those numbers come from is committed in the repo, fingerprinted over the traces it analysed, and the tool now refuses to overwrite an entry that already exists. A file can be cited. A rerun cannot. Everything else above stands: 9 to 12 hours at a 55-minute cadence, and the two ends are two runs rather than an error bar. Shipped in 1.3.0.

Reading is not measuring, again

While writing this I checked our own SDK and found the same class of bug I keep filing against other trackers. traceguard's wrap_anthropic recorded tokens_in = usage.input_tokens. On Anthropic's API, input_tokens excludes cache reads and writes — and on our traffic, cache reads run about 1,000× larger than raw input, so the wrapper under-counted by three orders of magnitude. Every field read correctly, the quantity still wrong (the splitrail #220 pattern). Fixed in PR #39, and the fix is what makes the next part possible.

A rate without a denominator is not a finding. Percentages travel well in emails; dollars make decisions. Before acting on a metric email, multiply it by the money it applies to.

Check yours

This analysis is now one command, merged in PR #40: python -m traceguard.routing_audit.cache_audit. Point it at your ingested Claude Code transcripts or a traceguard trace store, and get the table above for your own data — per-model hit rate, actual vs no-cache cost at list price, gap distribution, and a yes/no on whether keep-alive pings would pay for you (ours printed NOT WORTH IT — see the update above for what that verdict hides, and what the next release prints instead). Ships in traceguard 1.2.0 — pip install -U traceguard. The last line of ours came out like this:

Claude Code caching already saves us 84% ($12,110.31 vs $75,044.44 list). Checked with: python -m traceguard.routing_audit.cache_audit

If you run it, send me your numbers. Especially if pings win somewhere — I'd like to see the traffic shape that makes them rational.

Li Zhuojun

---

A routing audit for your team, fixed price. The method in this series runs on any Claude Code trace store. Send me 30 days of your team's usage records (model, tokens, timestamps, agent and session ids; I'll send a one-line jq filter that drops prompt and answer text before anything leaves your machine) and within ten working days you get one number and a five-page report: what share of your spend ran on a tier your own routing policy would not have chosen, what that cost at list price, and which components caused it. On my own 26,131 traces the number was 22.6% and $1,248.13, all of it on subagents and none on the main thread. US$1,500, flat. Write to [e-mail] with the subject "routing audit".

Comments

[Reid Marlow]

The keep-alive ping section is the useful bit. Cache advice often stops at the hit rate, but the gap distribution decides whether the trick has any economic shape. I'd probably graph cost by idle gap next to the token table, because that's where the 1h TTL quietly changes the decision.

[Li Zhuojun]

Update on this. I solved the cap instead of picking it, and the answer stopped being a number.

Sweeping 1h to 12h in 15-minute steps: the best single cap is 10h, but it leads 9h45m by $7.63, and measuring one thing I had previously only assumed moved that same cap by $18.76. When a correction is larger than the distance between first and second place, the argmax is not the output. The band is 9 to 12 hours at a 55-minute cadence, net $663.69 to $811.30 over the window. The two ends are two runs, not a confidence interval: the low one treats all 54 gaps with no recorded model as cache-invalidating, the high one deducts only the 18 I can prove.

Those 18 are your question one level down. A keep-alive only holds the cache for the model it was written on, so if the session comes back on a different model the pings bought nothing. The rate climbs with idle length: 0% in the shortest decile of gaps, 23.1% past 24 hours. So leaving it unmeasured was not adding noise, it was pushing the cap long, and every version of that section before this one was answering with the bias in it.

One detail you might find as odd as I did: all 18 switches were opus-4-8 and fable-5, both directions. Not one switch down to a cheaper model. I had assumed mid-session model switching was mostly cost-driven. On this traffic it is not.

The window is pinned now, 2026-05-30 to 2026-08-16 behind a --benchmark flag, so the numbers reproduce. That flag and the sweep are on main rather than in 1.2.0, next release. The post is updated.

[Li Zhuojun]

Correction to my last comment: the window does not make it reproduce. Same window, same store, 432 expired gaps became 439 and 168 sessions became 174 inside a day — ingest picks up transcript files late, so closing the dates closes the timestamps and not the corpus. Net band is $650.53 to $806.82 now, and the shape of the answer is unchanged.

What I cite instead is a committed file, packages/traceguard/benchmark/data/001-traceguard-self-b07cc061.json, fingerprinted over the traces behind it. The tool refuses to overwrite an existing entry, so a re-run over grown traffic becomes a second file rather than a quiet edit to the first. Those two side by side are their own measurement: how far this drifts when nobody touches anything.

[Li Zhuojun]

Ran it, and the split changes the article's answer. Thanks for pushing on that.

Cost by bucket, same 71 days. 1–4h gaps: 183 of them, pings would have cost $81 against at most $887 of rewrite avoided. >4h gaps: 239 of them, pings cost $1,928 against at most $1,026.

So the aggregate verdict I published ($2,010 vs $1,913, NOT WORTH IT) is a bucket where pinging wins about 11:1 summed with a bucket where it loses about 2:1. The average hid both.

A policy you could actually run prospectively (ping every 55 minutes, give up after 4 hours idle) costs $316 including the pings wasted on the 239 gaps that outlive the cap, against at most $887 avoided. The honest claim is narrower than the one I made: unbounded pinging loses, capped pinging probably pays. Probably, because both sides of that comparison lean pro-ping — $887 is an upper bound, since cache_creation mixes the re-established prefix with whatever the turn genuinely added, and the $316 charges pings as pure 0.1x reads of a frozen prompt. That tilt is why the original refusal was safe and why this flip is only 2.8x of headroom, not 11x.

Your column is in the repo and ships in the next release: section 2 now carries rewrite and ping cost per bucket, and section 3 runs the capped policy next to the unbounded one. A verdict averaged over buckets that behave differently is not a decision, which is what the article was about, one level down. I've put an update note on the post too. Thanks for the nudge.

[Boris Dzhingarov]

64% of $0.25 is a brutal punchline. I'm from the SEO side and our tools pull the same trick, site health drops 8 points and the email never says what that means in actual traffic or money. Also had no idea Claude Code was already routing most writes to the 1h TTL. I had the keep-alive ping saved in my notes from some Twitter thread and apparently never needed it. Going to run cache_audit on my own logs, curious what a mostly vibe coded workload looks like.

[Li Zhuojun]

Quick update, since I handed you a cutoff number in that last reply.

The 4-hour cutoff was picked to match a table boundary, not solved for. Solving it gives a band instead: ping every 55 minutes, give up somewhere between 9 and 12 hours idle. Net $663.69 to $811.30 over a pinned window, and those two ends are two different runs rather than a confidence interval.

The SEO version of what I got wrong: I reported the peak of a curve without checking how flat the top was. Same shape as reporting a rank without the traffic behind it.

The two commands I gave you are unchanged and still work on 1.2.0, which prints the aggregate verdict only. The cap sweep and the pinned reporting window are on main and land in the next release. Once they ship, --benchmark is what makes your numbers and mine comparable rather than just similar.

[Li Zhuojun]

One correction to what I told you: I said --benchmark would make your numbers and mine comparable. It narrows the gap, it does not close it. The same window over my own store drifted inside a day, because late-arriving transcripts still land inside a closed date range.

Comparable now means matching the corpus fingerprint in the committed file, not just the dates. The SEO version: same date range is not the same crawl.

[Boris Dzhingarov]

Your SEO translation is more literal than you might know. Search Console does exactly what your late transcripts do: the last two or three days of any date range are provisional and quietly restate themselves after you've exported. Every SEO who builds weekly reports learns that the expensive way, usually when a client asks why last week's numbers changed since the last email. Our fix ended up the same shape as your fingerprint: freeze the export only after the data stops moving, and record when you pulled it, not just what you pulled.

Holding off on running mine until --benchmark ships, since an aggregate verdict I can't compare against yours is just a number that makes me feel something. Will send the table when the release lands.

[Li Zhuojun]

1.3.0 is out, so --benchmark and the per-bucket table are a pip install -U traceguard away. The share export is not in it. That landed on main after the release and goes out in the next one. Use your own dates rather than mine, though. Two orgs can never share a window, and that was never the thing that made numbers comparable. What the fingerprint buys is that one file names exactly one corpus, so nobody lines up two different pulls and calls it a trend.

Recording when you pulled it, not just what you pulled, is going into the schema. I had the what and not the when, which means a reader can see two of my entries disagree without being able to see why. Search Console's provisional tail is the same failure with a name on it: my window closes on Aug 16 and the corpus behind it is still filling in, so an export pulled the day it closed is a different object from one pulled a month later even when the dates match. The file will carry the pull time and the gap between it and the end of the window. You are credited in the benchmark README for it.

Client calls rather than build times is the run I most want to see. Every entry so far is one laptop doing one kind of work, and the gap distribution is the only part of this that is about the person rather than the tool. If a cache audit turns out to describe a marketer's calendar, that is your post and I would rather read it than write it.

[Li Zhuojun]

The SEO parallel is exact. Site health down 8 points and 64% of $0.25 are the same email: a rate with the denominator left out.

Two commands if you want your own numbers:

pip install -U traceguard
python -m traceguard.routing_audit.ingest --write --db sqlite:///traces.db
python -m traceguard.routing_audit.cache_audit --db sqlite:///traces.db

First one reads ~/.claude/projects and is a dry run without --write. Second is read-only and prints the table. Checking those commands for you turned up a bug in my own README, which documented an ingest command that imported a module, ran nothing and exited 0. Fixed, along with the module that let it happen quietly.

On a vibe-coded workload I'd expect the hit rate to land close to ours, because Claude Code configures the caching and you don't. The section that will actually differ is the gap distribution, which is about how you work rather than what the tool does. Send it over if you run it.

One correction to my own post, since you had the ping trick in your notes: I split the counterfactual by gap length after Reid's comment, and pinging does pay on our 1–4h gaps ($81 to avoid up to $887). It only loses once you keep pinging across the very long idle stretches. So the note was not worthless, it just needs a cutoff.

[Boris Dzhingarov]

Replying here since this one had the homework in it. The ping note goes back into the notes file, this time with a cutoff stapled to it. And a comment thread that ends with someone's README getting fixed is about the most dev.to outcome possible. Your gap distribution prediction is the part I'm now curious about, because my gaps aren't build times, they're client calls. A cache audit that accidentally describes a marketer's calendar might be worth a post on its own.

[Li Zhuojun]

1.4.0 is up, so the export is installable. pip install -U traceguard, then the same two commands I gave you before, with --emit-share yours.json added to the second one and your own --since and --until rather than my window.

Run --show-share first. It prints the whole payload to your terminal without writing anything, so you can read every byte before deciding whether to send it. That is the half of this feature I would want if someone handed me their tool.

One number to look at when you get it: settling_days. Mine is 2.2, and the entry in the repo has the same fingerprint as the pull before it, which is the first evidence my window has stopped filling in. Yours will tell you whether you pulled too early.

[Kartik N V J K]

The catch that the flagged 64% was structurally unreachable, every prompt unique and under the 1,024-token floor, is a great reminder that a true metric can still point at no money. I have been burned the same way by cost alerts that ignore prompt structure, so now I gate them on repeated-prefix ratio before they can fire. How are you computing list price for the Claude Code transcripts given they carry no cost field?

[Li Zhuojun]

There is no cost field to read, so it gets computed. I checked every local session file across CC 2.1.150 to 2.1.198 and there is no costUSD anywhere, so the audit prices message.usage against a hand-maintained table: input and output per MTok, cache reads at 0.1x, cache writes at 1.25x for the 5-minute TTL and 2x for the 1-hour one. Every entry in that table carries a source and the date it was verified against anthropic.com, sitting in the module docstring, because a price nobody can check is the same problem as a rate nobody can check.

Two things it does that a flat lookup would not. Prices are looked up by model and timestamp, because Sonnet 5 had an introductory price before a standard one and the same model id has to bill differently either side of that date. And a record with usage.speed == "fast" bills at the model's published fast multiplier, 2x on Opus 4.8 and Opus 5. Models with no published fast price return nothing rather than a guess.

That last part is where the n/a rows in the table come from. If a model has no published price I have verified, the audit counts its tokens and refuses to put a dollar figure on them. Two rows in the article print that way. It also carries the count of unpriced messages next to every money figure, 14 post-gap and 43 pre-gap in the current entry, so you can see how much of the bill is being left out rather than quietly folded in. A cost number resting on a guessed price is the same email I was complaining about.

Your repeated-prefix gate is better than anything I have on that side, and I do not have it. One thing worth wiring in if you build it out: the minimum cacheable prompt is model-dependent, 1,024 tokens on Opus 4.8 and 512 on Opus 5 and Fable 5, so a prefix ratio that ignores which model served the call will pass workloads that structurally cannot cache. That is the exact shape of the thing that produced my $0.16.