Originally published on hexisteme notes.
I was running a local capture proxy in front of the vendor's API — a small man-in-the-middle process I'd set up for an unrelated token audit — when my coding-agent CLI spawned two sub-agents back to back. Same task, same working directory, same assembly path. One sub-agent got routed to a mid-tier model, the other to a small model. Because the proxy was logging full request bodies, I could diff them byte for byte. That diff is the whole essay.
The accidental A/B
The two request bodies were 615,341 bytes and 617,134 bytes. That's a difference of 1,793 bytes, about 0.3% — for practical purposes, the same payload. Same system prompt scaffolding, same tool definitions, same conversation history, same task description. Nothing about the content should have made these two requests bill differently. If token count were a property of the bytes on the wire, these two numbers should have landed within a rounding error of each other.
They didn't.
The bill
The mid-tier-model request was billed 246,525 input tokens. The small-model request, for a body 0.3% larger, was billed 196,892 input tokens — 49,633 fewer, a 20.1% reduction. Expressed as tokens per byte, that's 0.390 for the mid-tier model against 0.310 for the small model. Same text, essentially the same byte count, and one model's meter reads a fifth lower than the other's for it.
I want to be precise about what "billed" means here, because it's easy to round this into something looser. The input-token figure I'm comparing is the accounting field the API actually returns in the response — the number that determines what the request costs. It isn't an estimate I computed from the bytes; it's the number the vendor's own usage accounting assigned to each request, for near-identical input.
What this means — and what it doesn't
The clean version of the claim is: a token count is not a property of a request. It's a property of the pair (request, model). The same bytes, tokenized or accounted for under a different model's scope, produce a different number.
Here's where I want to slow down and be honest about the limits of what I actually measured. What I have direct evidence for is the accounting difference itself — 20.1% fewer tokens for the same bytes, measured once, on one pair of requests. The most plausible mechanism behind that gap is that the two models use different tokenizer vocabularies, so the same run of text segments into a different number of pieces. That's the standard explanation for this class of effect and it fits what I saw. But I did not run an experiment that isolates the mechanism. I didn't tokenize the same string offline with each model's tokenizer and count pieces directly; I didn't rule out that some of the gap comes from a difference in what gets counted into the input-token field in the first place — cache-eligible content, tool-definition overhead, or some other accounting-scope difference on the vendor's side rather than the tokenizer itself. So the correct claim is narrower than "different tokenizer, confirmed": it's "the accounting differs by model, and a tokenizer difference is the leading candidate explanation, unconfirmed."
That distinction matters more than it sounds like it should, because the practical implications below hold either way. Whether the mechanism is tokenizer vocabulary or accounting scope, the meter itself is model-scoped. That's the load-bearing fact, and it's the one I actually measured.
Implications
Cross-model "$ per token" comparisons don't compose. If you're comparing the cost of running the same workload on two models by multiplying a price-per-million-tokens figure by a token count, you're implicitly assuming the token count is the same across models for the same work. It isn't. A 20% gap in the denominator, on top of whatever gap exists in the price-per-token numerator, means the two models' effective prices for identical work can diverge from what a naive "$/Mtok × tokens" comparison suggests. If you want a comparison that actually composes across models, normalize by something model-independent — bytes of input, or a fixed task definition — not by the token count either model reports.
"Token savings" measurements are only valid within one model. If you're running an optimization — trimming a system prompt, restructuring tool definitions, cutting a stale cache block — and you're measuring the win in tokens saved, that number is only meaningful as long as the model stays fixed across the before/after comparison. Switch models in the middle of a measurement campaign and the accounting scope shifts under you; part of whatever delta you see is now optimization, and part is just a different meter. I've made this mistake before in a different form — comparing pooled metrics across roles that weren't actually comparable — and this is the same shape of error one layer down: the unit of measurement quietly changed between the two things being compared.
A cheaper model's effective unit price can be better than its price sheet implies. If a small model both charges less per token on the price sheet and meters fewer tokens for the same bytes, its actual cost advantage for a given task is larger than the sticker prices alone would suggest. Conversely, if a model's tokens-per-byte ratio for your workload runs high, part of what looks like the price sheet's "premium" is actually the meter charging more units for the same input, not just charging more per unit. Either way, the sticker price alone doesn't tell you the effective cost of a task — you need the accounting ratio too, and that ratio is workload-dependent as much as it's model-dependent.
Honest limits
This is n=1 pair. One measurement, one moment, one workload. I'd treat the 20.1% figure as "this magnitude of gap exists and is worth checking for," not as a constant you can plug into a cost model for any pair of models.
The measurement ran behind a capture proxy, and that matters for the absolute numbers: request assembly behind this kind of proxy setup runs roughly 3x larger than it would in normal operation, because a caching optimization that's normally active gets disabled when a proxy sits in the path. So don't read "615,341 bytes" as a typical request size for a sub-agent spawn — it isn't. What keeps the comparison valid despite that inflation is that both requests took the identical assembly path, behind the identical proxy, with near-identical bodies. The absolute byte counts are inflated; the ratio between the two requests is not, because whatever inflated one inflated the other by the same mechanism.
The payload itself was dominated by English-language system and tool text — scaffolding, tool schemas, instructions — not code and not non-ASCII text. Tokenizer vocabulary gaps between models are known to vary by content type: code, non-English text, and structured data can all tokenize differently than prose does. So the 20.1% figure and the 0.390-vs-0.310 tokens-per-byte ratio are specific to this kind of payload. A code-heavy request or a request dominated by a non-Latin-script language could show a very different ratio, in either direction. I haven't measured either of those cases, and I'm not going to guess at what they'd show.
What I'd want before trusting this number as a general rule: the same near-identical-bytes comparison repeated across a few different payload types — code-heavy, non-ASCII-heavy, and prose — across a few different model pairs, without a proxy in the path so the absolute sizes are representative too. Until then, the finding stands as exactly what it is: one clean natural experiment showing that the meter is model-scoped, with a plausible but unconfirmed mechanism, and a strong reminder to stop assuming the token denominator is a fixed property of the request when comparing costs across models.
More notes at hexisteme.github.io/notes.
Top comments (0)