I audited 110 AI usage/cost tools and filed 45+ verified billing bugs. Here's what they get wrong.

Roy Tong — Thu, 10 Sep 2026 06:47:09 +0000

I audited 110 AI usage/cost tools and filed 45+ verified billing bugs

TL;DR: If a tool tells you what your AI coding agents cost, there is a real chance it is wrong in a systematic direction. I audited ~110 repositories across the Claude Code / Codex / gateway / eval / observability ecosystems and filed 65+ evidence-backed findings; 10 have already been merged or accepted by maintainers, several with public credit. The majority of audited tools passed clean — including some of the largest (promptfoo, Portkey's model data, TokenTracker) — so this is not "everything is broken"; it is five specific, findable bug classes.

Full report with every finding linked

The five classes that keep appearing

1. One assistant message ≠ one line (10 confirmed tools)

Claude Code writes one JSONL line per content block of an assistant message, and every line carries the same message.id and the same message.usage. Sum per line and a message with text + two tool calls is billed 3×. Real overcounts we measured: 2×–5×.

tokendash #37 (49★, merged): 3-block message → 3× tokens and cost
ccem #12: same shape, first-wins variant undercounts instead
samewrite #1: a research project's headline numbers rested on it

2. Re-emitted events (5 tools)

Codex re-emits token_count events with identical totals after compaction, settings changes, and rate-limit refreshes. Naive per-event accumulation double-counts. Measured on real corpora: +0.5% to +15% per session file.

token-vision #1: one real rollout, +14.97%
tokenscope #6 (merged), coding-agent-usage-tracker #1 (merged)

3. Cache-pricing semantics (9 tools)

Anthropic's input_tokens excludes cache tokens; OpenAI's includes them. 5-minute cache writes cost 1.25× input, 1-hour writes 2×, reads 0.1×. Tools that apply one convention to both providers clamp input to $0, double-charge cache, or price 1h writes as 1.25×.

claude-hud #758 (27.9k★): 1h writes at 1.25× — 37.5% understated on that component
one-api-pro #13: streaming += double-counts input (+50.7% with no cache) and the billing formula re-subtracts cache

4. Price-table drift (7 tools, including the source)

Embedded price snapshots diverge from published rates by 1.2×–5× — including in vendored copies of LiteLLM's price file, the industry's source of record (litellm #40360).

5. Resume/fork history loss (4 tools)

Codex resume writes a new rollout file whose session_meta.id is the original session id. Stores that replace-per-session wipe everything before the resume; fork re-plays the parent prefix and double-counts it.

ai-usage-inspector #1: 86% of a session's tokens vanish (real pair measured)
swarm #145 → fixed in #146 with maintainer credit: "reported and diagnosed by @roy-tong, who also supplied the patch this follows"

Why you should care even if you don't use these tools

Your billing dashboards, your FinOps exports, and every "how much did this agent cost" number downstream inherit these errors. The bugs are findable with synthetic fixtures in minutes — most tools' own tests never model these shapes.

Audit your own tool in 10 minutes

The fixture corpus is open data — plain JSONL + expected totals, no runtime:

git clone --depth 1 https://github.com/roy-tong/AgentMeasure
python3 conformance/pack/agentmeasure selftest   # see the expected-totals style

Full method and per-class pointers: tool-authors.md.
Full case list: measurement-casebook.md.

Honesty notes

Counts are of filings with pinned evidence, as of 2026-09-10; merges happen as maintainers get to them (10 accepted so far).
Most audited repos passed. We name the clean ones because that matters.
All reproductions are synthetic; no private logs anywhere.
This audit was performed with the methods and fixtures of AgentMeasure — that is the demo. No audited project endorses it.

DEV Community: Roy Tong