On August 21, a16z's Charts of the Week ran OpenRouter data with a line that should bother anyone who puts a token count on a dashboard: "More than 85% of agentic token burn comes from the cached prompt." The same note says agents use nearly 5x as many tokens as humans on that network, and that agent usage has grown about 14x since February. Four days earlier, LeadDev reported from its AI Impact Report 2026 that 58% of respondents measure AI impact by token usage while 57% say it fails to gauge real value. Meta shut down its "Claudeonomics" leaderboard. Amazon retired KiroRank.
We ran into the same effect from the other side. shell.online 0.16.1 shipped a token counter on September 17, and the source records that an earlier version of it was 99% cache reads.
Why the counting happens on your machine
shell.online puts a running terminal process behind a browser link, and the session is encrypted end to end. 0.16.1 added Shell Keep, an optional game skin over the session list. One piece of it, the elixir vial, shows tokens your local coding agents have spent, next to commits, lines changed and open pull requests.
The service cannot compute any of that. It derives no key and holds no session password, so terminal output is opaque to it. The package comment in internal/stats/stats.go puts it this way: anything richer than counting rows "exists only where the plaintext already is, which is here. So the reading happens here and what leaves is counts."
Cache reads were 99% of the total
Tokens come from .jsonl files under ~/.claude/projects and ~/.codex/sessions that were modified inside the window (seven days by default). Each line is decoded into a struct that declares four int64 usage fields, at both usage and message.usage, and nothing else. The prompt, the reply and any file contents on that line are dropped by the JSON decoder because the type has no field for them. There is no later filtering step that someone could forget.
SumTokens adds input, output and cache-creation tokens. It decodes cache_read_input_tokens and deliberately leaves it out of the sum. The comment explains why: counted at full weight, cache reads "were ninety-nine per cent of the total, so three days of ordinary work reported four and a half billion tokens," which pinned the vial at its maximum on the first run. TestSumTokensLeavesCacheReadsOut holds that behaviour.
a16z's 85% is a network-wide average. One developer machine running long coding-agent sessions sits further along the same curve. A leaderboard that ranks raw totals is mostly ranking how many times a long context got re-read.
The server side carries a related scar. Every reported figure is floored and clamped between zero and ten billion in app/server/routes/gathering.ts. The comment explains why the cap is ten billion and not a hundred million: a real machine reported eighty-three million tokens for three days of ordinary work, and "a cap that clips honest reports is worse than no cap at all, because the number it produces is wrong and looks reasonable."
A payload with nowhere to put a string
The Run struct has five int64 fields (tokens, pull_requests, commits, insertions, deletions) and one error string. TestRunCarriesOnlyNumbers uses reflection to assert the struct has exactly six fields, so adding a seventh fails the test suite until someone argues for it. Our own docs call this "six integers", which is one integer too generous. The sixth field is text, built from three fixed phrases (git: unavailable, gh: unavailable, agents: unavailable) and cut at 200 characters. TestCollectNeverReportsLocalPathsFromErrors feeds in an error containing a home-directory path and checks that it does not come out the other end.
The git numbers come from git log --since=... --no-merges --pretty=tformat:%H --numstat. The path column is split off and never assigned, and binary files (which print - for both counts) are skipped. Pull requests come from gh pr list --author @me --state open --limit 100 --json number, decoded into []struct{}, so not even the PR numbers survive. Only stdout is read from either tool, because stderr is prose.
Checking it without sending it
shell stats [--json] [--days N] [--dir PATH] runs the same Collect function and prints the result. It makes no network call. The reasoning is in cmd/shell/stats.go: "a command that both gathered and sent would make this a way to find out what it reads only by doing it."
There is one difference from the wire format. The real report adds an id of run_ plus the id of the command that asked for it, so a retry after a lost reply is not recorded as a second run. The server accepts that id only if it matches ^run_[A-Za-z0-9_-]{1,64}$.
Gathering is off until the account turns it on. POST /api/game/gather returns 403 unless the stored profile has consented, then queues a probe command for each machine whose agent polled within the last 15 seconds. The probe carries no arguments, and the comment in agent_loop.go says that is the point: a command that could name a directory "would be a way to ask somebody's machine to go and look somewhere on a browser's behalf." The report goes to POST /api/agent/stats, authenticated with the CLI's token instead of the browser session, and consent is checked a second time there. Apart from that id, the handler reads the five counts and the error sentence out of the body and ignores anything else. The stored run is tagged with the machine's label, looked up from the account's existing device list.
The same constraint one layer down
Pilot Protocol has this shape too. Tunnels use an Ed25519-signed X25519 exchange and AES-256-GCM, and when hole-punching fails the beacon relays traffic that stays end-to-end encrypted, as the README in the source repo describes. A relay can count packets. It cannot see what an agent did. Any usage metric richer than bytes has to be computed at an endpoint and shipped in a shape narrow enough to audit by reading one struct.
Dropping cache reads does not turn a token count into a productivity metric. It is still a measure of spend, and a total that is 99% cache reads misstates even that.
Top comments (0)