An electricity meter is sealed. It is calibrated by a body that does not work for the utility, it can be read by the person paying, and a disputed bill has a physical artifact to point at. Metered billing works for one reason. The meter is trustworthy independently of the seller.
An AI bill has no such meter. The counter sits inside the provider's serving stack. It reports how many tokens were used, the invoice is paid on that number and most of what it counts is never returned. On a frontier reasoning model the bulk of the spend is reasoning and cache tokens, billed but never shown.
There is no sealed meter. There is a number and a request to trust it.
The pattern is universal. Every major model meters by the token, from OpenAI and Anthropic to Google, Meta, Mistral, DeepSeek, xAI and Alibaba's Qwen.
The hyperscalers reselling them, AWS, Microsoft Azure and Google Cloud bill the same way and gateways like OpenRouter and Hugging Face pass the meter straight
through. All of them keep the meter on their own side of the glass.
I spent a few weeks building the independent meter reader. This is what the build found including the part that proved the premise wrong, which turned out to be the most useful finding of all.
The research said the problem was real
Before writing code I ran the same adversarial research method I use for app store intelligence. Fan out across sources, extract falsifiable claims, then verify each with a three-vote pass where two refutations kill the claim. One hundred and three agents, ninety seven claims, three killed under scrutiny. The kill list is the point. The goal is to be as hard on my own conclusions as the tool is meant to be on token counts. Hold that thought, because the same discipline later saved the project.
The verdict was a real, academically validated white space.
- CoIn (arXiv 2505.13778): users are billed for invisible reasoning tokens, which often account for the majority of the cost, yet have no way to verify their authenticity.
- Invisible Tokens, Visible Bills (arXiv 2505.18471): users are billed for operations they cannot observe, verify or contest.
- PALACE (arXiv 2508.00912): commercial services conceal internal reasoning traces while still charging for every generated token.
A preprint on token inflation put a number on the exposure. Hidden reasoning usage is inflatable by roughly 1,469% on average without detection. A hundred-dollar honest bill becomes about fifteen hundred. Treat the magnitudes as directional rather than settled, but three independent groups agree on the shape. The spend is largely for work that cannot be observed.
Buried in that research was one sentence I read, nodded at, and did not actually absorb.
The trust paradox:
Every audit must trust some artifact, but current frameworks trust exactly the ones a provider has > the strongest reason to manipulate.
Remember that one.
The build, and the finding that stopped me
The leading academic approach, CoIn, is cooperative. It needs the provider to build a Merkle tree of token fingerprints, commit the root, and serve proofs on audit. Elegant and commercially dead on
arrival. No provider volunteers to make its own meter auditable.
So the build went the other way. Passive and outside in. Retokenise the delivered output locally, with the model's own tokenizer and reconcile it against the reported number. No provider cooperation,
nothing leaves the machine. Label every figure by confidence. EXACT when re counted with the real tokenizer, BOUNDED when estimated within a band, UNVERIFIABLE for reasoning and cache, which are billed but never returned.
Pointed at BytePlus, the finding that stopped me was video. A five second clip, billed 246,840 tokens. Video is metered by a published formula, width times height divided by 1024 times frames, so the bill is re derivable from the delivered file with ffprobe. It matched to the token. Gap zero. A
second clip at a different resolution, 108,900 tokens, gap zero. Four live text completions, gap zero on all four.
A wall of gap-zero results from live, paid calls. It looked like proof. A launch started forming around it.
The question that broke it
Someone asked one sentence. Are these not the same number from two sides?
And the trust paradox came back to collect.
Here is the uncomfortable part. Re-counting the delivered output and matching the reported number is a consistency check. It binds the bill to the artifact that was handed over. It is not an independent measure of true cost. The provider counts the tokens the model generated. The tool counts the canonical encoding of the text the model chose to return. Two computation paths, so a match means generation was canonical and nothing was dropped in transit.
A real check, the class of a checksum.
Against an honest provider that check is a near-guaranteed pass. Against a rational one set on overcharging it is toothless, because nobody inflates the one bucket that can be recomputed. The inflation lives in the buckets that cannot be, reasoning, cache, the rate. The gap-zero wall was demonstrating the one thing that was never the risk.
Worse was the reflex. On a small text gap, the instinct was to swap tokenizers until it vanished. That is the trust paradox in miniature, tuning the audit until it agrees with the bill. An audit that can be adjusted until it passes is not an audit. A gap, it turns out, has three causes that the number alone cannot separate. Over billing, the wrong tokenizer, or legitimate non canonical generation. So a gap is a flag to investigate, never a verdict, exactly as a match is never proof.
The research had said this in one sentence. Building the wrong thing was the cost of understanding it.
What actually survives
Killing the headline claim left the real one standing, and it is stronger.
An AI bill cannot be independently verified. The verifiable part verifies itself, and the rest is structurally out of reach. What can be done is to measure how much of the bill has any ground truth at all. That is the honest, novel signal.
The delivered part can be bound to the artifact, which catches a 1080p-billed-but-480p-delivered swap and catches metering bugs, including the prompt-cache failures that have over-billed real users by ten to twenty times. The undelivered part, reasoning and cache, most of a modern bill, is unverifiable by anyone, and the right move is to say so, loudly, with a number.
The product is not "we check the bill". It is a measurement of how much of the bill nobody can check and how small the sliver that can be. For a five second video that sliver is a six figure token count
that at least ties to the file. For a reasoning call the sliver is almost nothing and the honest output is the size of the dark.
The discipline is admitting where verification ends
The thing that saved this project is the thing that built it. An adversarial pass that kills the claims
which do not survive. It killed three of ninety-seven claims in the literature. It should have been run
on the headline before anyone got attached to gap zero. When a one-sentence question can dismantle the
strongest demo, the demo was the problem.
It is also the house style. The app-store work is outside-in. Read anyone's public reviews, cooperate with no one, and treat the only honest trend source as the only honest trend source. This is the same philosophy aimed at billing. Read the artifact that was handed over, cooperate with no one and be ruthless about the line between what can be known and what is being asked on trust. The discipline is not the verification. The discipline is admitting where verification ends.
The AI meter is unsigned and a large part of it reads in the dark. The useful move is not to pretend the dark can be read. It is to measure exactly how much of it there is.
TokenLedger is open source under Apache-2.0: pip install retoken. The known limitations, including the rule that a gap is a flag and never a verdict, are written up in the repository, because a tool about trust should hold itself to its own standard.
Top comments (0)