Outcome-based pricing won the pricing page. Per-resolution AI support, usage-billed agents, API metering by the call — the industry has decided that software is billed by what it does. One question got a lot quieter than it deserves: who verifies the billed number?
We kept running into the same class of silent accounting errors while metering AI agent usage — retries billed as new operations, duplicate records after crashes, cache hits priced as fresh tokens. So we did the boring thing: we audited. 110 AI usage tools so far, each with the tool's own parser on synthetic fixtures, every bug reproduced against a pinned commit, every fix re-verified after it landed upstream (25 merged so far). The full index is public — every entry has exactly one status, one date, one link:
→ https://roy-tong.github.io/AgentMeasure/claims.html
The five shapes that keep showing up
1. Retries billed as new operations. A runtime retry chain — 429, timeout, then success — is one logical operation with three executions. If the meter counts executions, the customer pays 3×. We reproduced this shape across runtimes; it is the single most common finding, and dashboards stay green the whole time because every individual record is "correct."
2. Duplicate records after a crash or resume. Re-emitted lines after a crash look like new usage. Byte-identical duplicates inside session files showed up in more tools than we expected. The fix is boring — dedupe on record id before counting anything — which is exactly why it doesn't get done.
3. Cache hits priced as fresh tokens. Where a tool reports cache accounting, cached-read tokens sometimes ride the same field as fresh tokens, priced identically. At cache-hit rates typical for agent workloads, that's not a rounding error.
4. Attempt grain reported as operation grain. "Median steps per operation: 2.5" computed over successful attempts reads very different from 4.0 computed over total consumption including retries. Both numbers are defensible; only one matches what the bill should say. The metric's grain is the claim.
5. Absent zero. A boolean "consumed" field reused for "not set" turns 9,107 referenced observations into a confident zero. Missing evidence formatted as data is the nastiest one, because it looks like a result.
UNPROVABLE is a result, not an error
The audit's vocabulary has five states and two of them are "we can't say": Verified Mismatch / Fixed · Re-verified / Conformance Pass / UNPROVABLE / Not-a-bug. If the export lacks the field that decides a metric, the finding is UNPROVABLE — stated out loud, not filled with a zero. Most metering disputes we've seen are not dishonesty; they're someone's UNPROVABLE rendered as a confident number.
That discipline is also why the audit is reproducible: every claim binds to a fixture you can run. pip install agentmeasure gives you the checker (agentmeasure check) which scans your own Codex / Claude Code session logs locally — duplicate records, retry chains, token anomalies, thirty seconds, nothing uploaded, no account. New in v0.7.0 (2026-10-14): Windows CI coverage and machine-checkable sanitized summaries (agentmeasure share exports aggregate counts only — previews first, exports after you look).
What we do when a finding needs a human
The tools are and stay free. When your own check flags something and you want to know what it means in your setup — bug or intended semantics, why, what to do next — that's a human diagnosis, and that's the paid part: a $10 anomaly diagnosis and a $20 metering QA check, email application, fixed scope, refund terms on the page (https://roy-tong.github.io/AgentMeasure/services.html). "No finding" is a valid deliverable and is priced in.
The community part matters more than the bugs
This cycle merged three external PRs — an SSE conformance vector, a Freshdesk field mapping, an Intercom walkthrough — plus the first availability-certification vector contributed via PR. agentmeasure conformance --new-vector <name> scaffolds a CI-valid external vector in one command: if you meter an MCP server or an agent API, you can pin your own semantics and let CI catch regressions when you change retry logic or pricing.
Numbers as of 2026-10-14: 110 tools audited · 45+ reproduced metering bugs · 25 upstream fixes merged · 477 tests green. Every one of those claims links from the index above — that's the whole point.
If you run usage-billed agents, run the check before your next invoice. If you build the tools, file a vector. If you find a bug, the index has room.
Top comments (0)