DEV Community: Roy Tong

We audited 110 AI usage tools for metering bugs — 45+ reproduced, every finding linkable

Roy Tong — Wed, 14 Oct 2026 03:43:29 +0000

Outcome-based pricing won the pricing page. Per-resolution AI support, usage-billed agents, API metering by the call — the industry has decided that software is billed by what it does. One question got a lot quieter than it deserves: who verifies the billed number?

We kept running into the same class of silent accounting errors while metering AI agent usage — retries billed as new operations, duplicate records after crashes, cache hits priced as fresh tokens. So we did the boring thing: we audited. 110 AI usage tools so far, each with the tool's own parser on synthetic fixtures, every bug reproduced against a pinned commit, every fix re-verified after it landed upstream (25 merged so far). The full index is public — every entry has exactly one status, one date, one link:

→ https://roy-tong.github.io/AgentMeasure/claims.html

The five shapes that keep showing up

1. Retries billed as new operations. A runtime retry chain — 429, timeout, then success — is one logical operation with three executions. If the meter counts executions, the customer pays 3×. We reproduced this shape across runtimes; it is the single most common finding, and dashboards stay green the whole time because every individual record is "correct."

2. Duplicate records after a crash or resume. Re-emitted lines after a crash look like new usage. Byte-identical duplicates inside session files showed up in more tools than we expected. The fix is boring — dedupe on record id before counting anything — which is exactly why it doesn't get done.

3. Cache hits priced as fresh tokens. Where a tool reports cache accounting, cached-read tokens sometimes ride the same field as fresh tokens, priced identically. At cache-hit rates typical for agent workloads, that's not a rounding error.

4. Attempt grain reported as operation grain. "Median steps per operation: 2.5" computed over successful attempts reads very different from 4.0 computed over total consumption including retries. Both numbers are defensible; only one matches what the bill should say. The metric's grain is the claim.

5. Absent zero. A boolean "consumed" field reused for "not set" turns 9,107 referenced observations into a confident zero. Missing evidence formatted as data is the nastiest one, because it looks like a result.

UNPROVABLE is a result, not an error

The audit's vocabulary has five states and two of them are "we can't say": Verified Mismatch / Fixed · Re-verified / Conformance Pass / UNPROVABLE / Not-a-bug. If the export lacks the field that decides a metric, the finding is UNPROVABLE — stated out loud, not filled with a zero. Most metering disputes we've seen are not dishonesty; they're someone's UNPROVABLE rendered as a confident number.

That discipline is also why the audit is reproducible: every claim binds to a fixture you can run. pip install agentmeasure gives you the checker (agentmeasure check) which scans your own Codex / Claude Code session logs locally — duplicate records, retry chains, token anomalies, thirty seconds, nothing uploaded, no account. New in v0.7.0 (2026-10-14): Windows CI coverage and machine-checkable sanitized summaries (agentmeasure share exports aggregate counts only — previews first, exports after you look).

What we do when a finding needs a human

The tools are and stay free. When your own check flags something and you want to know what it means in your setup — bug or intended semantics, why, what to do next — that's a human diagnosis, and that's the paid part: a $10 anomaly diagnosis and a $20 metering QA check, email application, fixed scope, refund terms on the page (https://roy-tong.github.io/AgentMeasure/services.html). "No finding" is a valid deliverable and is priced in.

The community part matters more than the bugs

This cycle merged three external PRs — an SSE conformance vector, a Freshdesk field mapping, an Intercom walkthrough — plus the first availability-certification vector contributed via PR. agentmeasure conformance --new-vector <name> scaffolds a CI-valid external vector in one command: if you meter an MCP server or an agent API, you can pin your own semantics and let CI catch regressions when you change retry logic or pricing.

Numbers as of 2026-10-14: 110 tools audited · 45+ reproduced metering bugs · 25 upstream fixes merged · 477 tests green. Every one of those claims links from the index above — that's the whole point.

If you run usage-billed agents, run the check before your next invoice. If you build the tools, file a vector. If you find a bug, the index has room.

We audited 110 AI usage tools. Here is where the numbers go wrong.

Roy Tong — Tue, 22 Sep 2026 23:12:53 +0000

AI products are moving from flat subscriptions to usage-based and outcome-based billing: per-resolution support agents, premium request quotas, per-token gateway billing. Every one of those numbers is produced by a meter, and the meter is usually the vendor's own. We spent the last month auditing the open-source side of that world — 110 tools that count tokens, track costs, or enforce budgets — and this post is the field report.

The short version: 45+ verified bugs, five recurring families, 23 fixes already merged upstream (including langfuse at 34k stars and codeburn at 11k). An independent auditor reproduced our entire conformance suite and confirmed one cache-accounting path that under-reports usage by roughly 99%. Every claim below links to a reproducible check.

The five bug families

1. Stale pricing tables. The most common real bug, and the most boring: pricing tables that lack the current generation of models. In one batch of seven tools, five shipped outdated or missing rate rows. The tool computes confidently from numbers that stopped being true months ago — and every downstream total inherits the error.

2. Provider-wrong cache multipliers. Cache reads and writes are billed at different multipliers per provider. A common shortcut hardcodes one provider's ratios and applies them to everyone. We found a tool applying Anthropic's cache-read discount to OpenAI models, underestimating cache reads by 5x. The billing math is wrong in a way that looks plausible on every individual line.

3. Retry double-counting in stream aggregation. When a streaming request is retried byte-identical, some aggregators count both attempts. In one public corpus of 604 re-emitted events — 46% of the corpus — the bytes were identical, and naive aggregation doubled them. Conversely, tools that deduplicate too aggressively miss genuine retried work. Both directions lose money for someone.

4. Absent treated as zero. When a usage field is missing, or 0 turns "unknown" into "free". It reads like defensive coding and behaves like a discount nobody authorized. The fix is semantic, not syntactic: absent must stay absent, and rollups must be allowed to say UNPROVABLE instead of zero.

5. Window boundary errors. Quota windows anchored to wall-clock time with fixtures pinned to absolute dates — the tool passes tests for 30 days and then fails everywhere at once. We know this one intimately because our own CI did it to us.

An independent reproduction

The part we are proudest of is not our numbers — it is that an external auditor, working separately, reproduced our full conformance suite (236 checks) and then contributed his own quantified audit of a commercial provider's cache accounting: four code paths failing, usage under-reported by 98.9% and 95.1% on the affected paths, pinned to an exact commit. That report is now in our evidence tree with named credit. A third party spending real effort to check your work is worth more than any testimonial.

What merged upstream

23 fixes across the ecosystem carry the audit's fingerprints — pricing table updates, cache multipliers corrected per provider, retry-collapse guards with regression fixtures, absent-vs-zero semantics made explicit. None of these were adversarial finds. Every one started as a thread where the maintainer was already discussing the problem, which is the only place we work.

What vendors should publish

We checked 20 commercial vendors for one thing: a published process for what happens when the meter is wrong — a dispute path, a correction policy, anything. None had one. When the vendor runs the meter, grades its own homework, and publishes no correction process, "trust us" is the whole control framework. Two minimum standards would change that: a named dispute path, and machine-checkable billing disclosures.

Verify your own bill in ten minutes

The conformance pack is open source (MIT). Export your usage data, run the checker locally — nothing leaves your machine — and it separates logical operations from physical attempts, cache reads from cache writes, and absent from zero, with every verdict traceable to a named rule. If a number on your invoice survives an independent recount, you can defend it. If it doesn't, now you have the receipt.

The full audit report with per-tool findings: campaigns/audit-report-2026-09.md. The settlement spec behind the checker: AMS-1.

I audited 110 AI usage/cost tools and filed 45+ verified billing bugs. Here's what they get wrong.

Roy Tong — Thu, 10 Sep 2026 06:47:09 +0000

I audited 110 AI usage/cost tools and filed 45+ verified billing bugs

TL;DR: If a tool tells you what your AI coding agents cost, there is a real chance it is wrong in a systematic direction. I audited ~110 repositories across the Claude Code / Codex / gateway / eval / observability ecosystems and filed 65+ evidence-backed findings; 10 have already been merged or accepted by maintainers, several with public credit. The majority of audited tools passed clean — including some of the largest (promptfoo, Portkey's model data, TokenTracker) — so this is not "everything is broken"; it is five specific, findable bug classes.

Full report with every finding linked

The five classes that keep appearing

1. One assistant message ≠ one line (10 confirmed tools)

Claude Code writes one JSONL line per content block of an assistant message, and every line carries the same message.id and the same message.usage. Sum per line and a message with text + two tool calls is billed 3×. Real overcounts we measured: 2×–5×.

tokendash #37 (49★, merged): 3-block message → 3× tokens and cost
ccem #12: same shape, first-wins variant undercounts instead
samewrite #1: a research project's headline numbers rested on it

2. Re-emitted events (5 tools)

Codex re-emits token_count events with identical totals after compaction, settings changes, and rate-limit refreshes. Naive per-event accumulation double-counts. Measured on real corpora: +0.5% to +15% per session file.

token-vision #1: one real rollout, +14.97%
tokenscope #6 (merged), coding-agent-usage-tracker #1 (merged)

3. Cache-pricing semantics (9 tools)

Anthropic's input_tokens excludes cache tokens; OpenAI's includes them. 5-minute cache writes cost 1.25× input, 1-hour writes 2×, reads 0.1×. Tools that apply one convention to both providers clamp input to $0, double-charge cache, or price 1h writes as 1.25×.

claude-hud #758 (27.9k★): 1h writes at 1.25× — 37.5% understated on that component
one-api-pro #13: streaming += double-counts input (+50.7% with no cache) and the billing formula re-subtracts cache

4. Price-table drift (7 tools, including the source)

Embedded price snapshots diverge from published rates by 1.2×–5× — including in vendored copies of LiteLLM's price file, the industry's source of record (litellm #40360).

5. Resume/fork history loss (4 tools)

Codex resume writes a new rollout file whose session_meta.id is the original session id. Stores that replace-per-session wipe everything before the resume; fork re-plays the parent prefix and double-counts it.

ai-usage-inspector #1: 86% of a session's tokens vanish (real pair measured)
swarm #145 → fixed in #146 with maintainer credit: "reported and diagnosed by @roy-tong, who also supplied the patch this follows"

Why you should care even if you don't use these tools

Your billing dashboards, your FinOps exports, and every "how much did this agent cost" number downstream inherit these errors. The bugs are findable with synthetic fixtures in minutes — most tools' own tests never model these shapes.

Audit your own tool in 10 minutes

The fixture corpus is open data — plain JSONL + expected totals, no runtime:

git clone --depth 1 https://github.com/roy-tong/AgentMeasure
python3 conformance/pack/agentmeasure selftest   # see the expected-totals style

Full method and per-class pointers: tool-authors.md.
Full case list: measurement-casebook.md.

Honesty notes

Counts are of filings with pinned evidence, as of 2026-09-10; merges happen as maintainers get to them (10 accepted so far).
Most audited repos passed. We name the clean ones because that matters.
All reproductions are synthetic; no private logs anywhere.
This audit was performed with the methods and fixtures of AgentMeasure — that is the demo. No audited project endorses it.