DEV Community: Roy Tong

We audited 110 AI usage tools. Here is where the numbers go wrong.

Roy Tong — Tue, 22 Sep 2026 23:12:53 +0000

AI products are moving from flat subscriptions to usage-based and outcome-based billing: per-resolution support agents, premium request quotas, per-token gateway billing. Every one of those numbers is produced by a meter, and the meter is usually the vendor's own. We spent the last month auditing the open-source side of that world — 110 tools that count tokens, track costs, or enforce budgets — and this post is the field report.

The short version: 45+ verified bugs, five recurring families, 23 fixes already merged upstream (including langfuse at 34k stars and codeburn at 11k). An independent auditor reproduced our entire conformance suite and confirmed one cache-accounting path that under-reports usage by roughly 99%. Every claim below links to a reproducible check.

The five bug families

1. Stale pricing tables. The most common real bug, and the most boring: pricing tables that lack the current generation of models. In one batch of seven tools, five shipped outdated or missing rate rows. The tool computes confidently from numbers that stopped being true months ago — and every downstream total inherits the error.

2. Provider-wrong cache multipliers. Cache reads and writes are billed at different multipliers per provider. A common shortcut hardcodes one provider's ratios and applies them to everyone. We found a tool applying Anthropic's cache-read discount to OpenAI models, underestimating cache reads by 5x. The billing math is wrong in a way that looks plausible on every individual line.

3. Retry double-counting in stream aggregation. When a streaming request is retried byte-identical, some aggregators count both attempts. In one public corpus of 604 re-emitted events — 46% of the corpus — the bytes were identical, and naive aggregation doubled them. Conversely, tools that deduplicate too aggressively miss genuine retried work. Both directions lose money for someone.

4. Absent treated as zero. When a usage field is missing, or 0 turns "unknown" into "free". It reads like defensive coding and behaves like a discount nobody authorized. The fix is semantic, not syntactic: absent must stay absent, and rollups must be allowed to say UNPROVABLE instead of zero.

5. Window boundary errors. Quota windows anchored to wall-clock time with fixtures pinned to absolute dates — the tool passes tests for 30 days and then fails everywhere at once. We know this one intimately because our own CI did it to us.

An independent reproduction

The part we are proudest of is not our numbers — it is that an external auditor, working separately, reproduced our full conformance suite (236 checks) and then contributed his own quantified audit of a commercial provider's cache accounting: four code paths failing, usage under-reported by 98.9% and 95.1% on the affected paths, pinned to an exact commit. That report is now in our evidence tree with named credit. A third party spending real effort to check your work is worth more than any testimonial.

What merged upstream

23 fixes across the ecosystem carry the audit's fingerprints — pricing table updates, cache multipliers corrected per provider, retry-collapse guards with regression fixtures, absent-vs-zero semantics made explicit. None of these were adversarial finds. Every one started as a thread where the maintainer was already discussing the problem, which is the only place we work.

What vendors should publish

We checked 20 commercial vendors for one thing: a published process for what happens when the meter is wrong — a dispute path, a correction policy, anything. None had one. When the vendor runs the meter, grades its own homework, and publishes no correction process, "trust us" is the whole control framework. Two minimum standards would change that: a named dispute path, and machine-checkable billing disclosures.

Verify your own bill in ten minutes

The conformance pack is open source (MIT). Export your usage data, run the checker locally — nothing leaves your machine — and it separates logical operations from physical attempts, cache reads from cache writes, and absent from zero, with every verdict traceable to a named rule. If a number on your invoice survives an independent recount, you can defend it. If it doesn't, now you have the receipt.

The full audit report with per-tool findings: campaigns/audit-report-2026-09.md. The settlement spec behind the checker: AMS-1.

I audited 110 AI usage/cost tools and filed 45+ verified billing bugs. Here's what they get wrong.

Roy Tong — Thu, 10 Sep 2026 06:47:09 +0000

I audited 110 AI usage/cost tools and filed 45+ verified billing bugs

TL;DR: If a tool tells you what your AI coding agents cost, there is a real chance it is wrong in a systematic direction. I audited ~110 repositories across the Claude Code / Codex / gateway / eval / observability ecosystems and filed 65+ evidence-backed findings; 10 have already been merged or accepted by maintainers, several with public credit. The majority of audited tools passed clean — including some of the largest (promptfoo, Portkey's model data, TokenTracker) — so this is not "everything is broken"; it is five specific, findable bug classes.

Full report with every finding linked

The five classes that keep appearing

1. One assistant message ≠ one line (10 confirmed tools)

Claude Code writes one JSONL line per content block of an assistant message, and every line carries the same message.id and the same message.usage. Sum per line and a message with text + two tool calls is billed 3×. Real overcounts we measured: 2×–5×.

tokendash #37 (49★, merged): 3-block message → 3× tokens and cost
ccem #12: same shape, first-wins variant undercounts instead
samewrite #1: a research project's headline numbers rested on it

2. Re-emitted events (5 tools)

Codex re-emits token_count events with identical totals after compaction, settings changes, and rate-limit refreshes. Naive per-event accumulation double-counts. Measured on real corpora: +0.5% to +15% per session file.

token-vision #1: one real rollout, +14.97%
tokenscope #6 (merged), coding-agent-usage-tracker #1 (merged)

3. Cache-pricing semantics (9 tools)

Anthropic's input_tokens excludes cache tokens; OpenAI's includes them. 5-minute cache writes cost 1.25× input, 1-hour writes 2×, reads 0.1×. Tools that apply one convention to both providers clamp input to $0, double-charge cache, or price 1h writes as 1.25×.

claude-hud #758 (27.9k★): 1h writes at 1.25× — 37.5% understated on that component
one-api-pro #13: streaming += double-counts input (+50.7% with no cache) and the billing formula re-subtracts cache

4. Price-table drift (7 tools, including the source)

Embedded price snapshots diverge from published rates by 1.2×–5× — including in vendored copies of LiteLLM's price file, the industry's source of record (litellm #40360).

5. Resume/fork history loss (4 tools)

Codex resume writes a new rollout file whose session_meta.id is the original session id. Stores that replace-per-session wipe everything before the resume; fork re-plays the parent prefix and double-counts it.

ai-usage-inspector #1: 86% of a session's tokens vanish (real pair measured)
swarm #145 → fixed in #146 with maintainer credit: "reported and diagnosed by @roy-tong, who also supplied the patch this follows"

Why you should care even if you don't use these tools

Your billing dashboards, your FinOps exports, and every "how much did this agent cost" number downstream inherit these errors. The bugs are findable with synthetic fixtures in minutes — most tools' own tests never model these shapes.

Audit your own tool in 10 minutes

The fixture corpus is open data — plain JSONL + expected totals, no runtime:

git clone --depth 1 https://github.com/roy-tong/AgentMeasure
python3 conformance/pack/agentmeasure selftest   # see the expected-totals style

Full method and per-class pointers: tool-authors.md.
Full case list: measurement-casebook.md.

Honesty notes

Counts are of filings with pinned evidence, as of 2026-09-10; merges happen as maintainers get to them (10 accepted so far).
Most audited repos passed. We name the clean ones because that matters.
All reproductions are synthetic; no private logs anywhere.
This audit was performed with the methods and fixtures of AgentMeasure — that is the demo. No audited project endorses it.