I recently saw an experiment I can't stop thinking about: someone gave four identical tasks to two different frontier models, each in a clean session, and compared three things — the quality of the results, the tokens spent, and the time taken. Not benchmarks. Not price lists. His tasks, their consumption.
I'm convinced this is the only model comparison that actually means anything for people who build on LLM APIs. And then, at the end of June, our own pipeline ran that experiment for us — by accident.
The setup
We run a content pipeline for bestaiweb.ai, an educational site about AI. The pipeline generates articles in batches we call entities: several parallel agents share one brief and one research pass, and produce a bundle of ~6 articles (four personas, four article types — explainer, guide, news, opinion), plus the glossary entries that explain the concepts involved, plus a hub page. Every run logs every call, and we convert the token counts to API list prices to track cost.
The pipeline's model was configured as an alias — "sonnet", meaning whatever the current Sonnet is. Between June 28 and 30, that alias quietly started resolving to the new Sonnet 5 instead of Sonnet 4.6. Nobody decided this. Nothing broke. I had even done my due diligence: I generated a test article with the new model from the same brief, it passed all our checks, and the content was very close to the old model's. So I let it in.
Then the cost per entity jumped from a typical $13–20 to $36–60. Overnight. A 2.8× multiplier.
First reflex: check the price list. The price list said Sonnet 5 costs exactly what Sonnet 4.6 costs — $3 per million input tokens, $15 per million output. Until the end of August it's actually cheaper, thanks to introductory pricing.
So the model didn't get more expensive. Its behavior did.
Where the money went
Because the pipeline logs everything, we had a clean A/B test: same pipeline, same tasks, same price list, different model. The difference decomposed into three numbers:
1. A new tokenizer: ~30% more tokens for the same text. Same brief, same article length — a third more billable tokens. A token is not a physical unit. Every model slices text its own way, and when the slicer changes, every number on your bill changes with it. (This is the same mechanism that makes non-English speakers pay more per token for the same content — tokenizer economics are sneaky.)
2. A more thorough working style: 35 rounds per agent instead of 18–26. The new model verifies more, reads more, takes more intermediate steps. Fine in itself — except every round re-reads the entire accumulated context of the conversation.
3. The effects multiply. More tokens per text × more rounds × bigger context in every round. Measured: the average context loaded per round went from ~45k to ~99k tokens. More than double.
None of these three things is a bug. They're documented properties of the new model — it's more diligent, it "thinks" more. That diligence just doesn't appear anywhere on the price list.
The aha moment
Price per token and price per task are two different quantities. Price lists state the first one. Your invoice is the second one.
What stuck with us
- Pin your model version. An alias like "sonnet" means "a surprise at some point in the future." A model upgrade should be a decision, not an accident.
- Log which model actually ran. Our run reports didn't record it — we reconstructed it forensically from CLI transcripts. One field in the log would have saved the whole investigation.
- Measure cost per task, not per token. When a new model arrives, run the same tasks on old and new, compare tokens + time + result quality. An hour of work that reveals multipliers.
- New behavior ≠ worse model. Sonnet 5 does more work for those tokens. The question isn't "why is it more expensive" — it's "is that thoroughness worth it for my task?" For some it will be. For others it won't.
A footnote on subscriptions vs. API
We run on a subscription, so the numbers above are API-equivalents computed from logged tokens — in real money, this cost us nothing extra; we "only" burn through our quota windows faster. But the token counts themselves are not estimates: they're the exact values the API returns with every call and bills from, including thinking tokens and the overhead of each call (system prompt, tool definitions).
What our conversion doesn't capture would mostly push a real API bill higher: cache writes with a 1-hour TTL are billed at double the input rate (97% of ours turned out to be 1-hour — measuring that corrected my own first estimate upward), calls killed mid-generation still get billed for what the server produced, and web search is billed per use on top of tokens. Pushing the other way: Sonnet 5's introductory pricing, and the Batch API, which halves everything if your workload can wait.
What we don't know yet
Honesty section. We haven't yet measured whether Sonnet 5's extra diligence produces better articles for our specific pipeline — the test article passed the same checks, but "passes checks" and "is worth 2.8× the spend" are different claims. A blind quality comparison is on the list. We also don't know how much of the behavior settles as the model matures, or exactly how the 2.8× splits between tokenizer and working style on other workloads than ours. And our web-search costs remain unmeasured entirely — the subscription absorbs them silently.
How to read AI price lists
This won't be the last time. Every few months a headline claim arrives — "just as good, but faster and cheaper" — and the only way to know whether that's marketing or reality is to run your own four tasks, in clean sessions, and count. Tokens, time, quality. Price per task, not price per token.
The price list is the beginning of the conversation, not the answer.
I'm an IT analyst who works with Claude Code daily on bestaiweb.ai. Not a pricing expert or an ML engineer. Someone who's fascinated by how AI responds — and a little envious of the polymath breadth it has at its fingertips: it knew the tokenizer mechanics, the cache billing tiers, and where my own cost estimate was wrong before I finished asking. So sometimes I stop building things and let the numbers themselves become the experiment. This is what I found. It might be wrong in places. But I love experimenting with AI about AI — and the best experiments are the ones you can't keep to yourself.
Top comments (2)
This is a great reminder that API pricing and workload pricing are very different things. One point I’d add is that cost per task should never be evaluated in isolation. If a new model consumes 2.8× more tokens but reduces hallucinations, eliminates a second review pass, or increases first-pass success rates, the total system cost may actually decrease despite the higher API bill.
I also liked your recommendation to pin model versions. I’d extend that further by treating model upgrades like dependency upgrades: run representative workloads, compare quality, latency, cost, and operational metrics, then promote the new version only after it passes predefined acceptance criteria. LLMs are becoming infrastructure components, and they deserve the same change-management discipline as databases or application frameworks.
The "price per token vs. price per task" framing matches exactly what we see running content pipelines on Claude — and I'd argue your scariest number isn't the 2.8× headline, it's the context per round doubling from ~45k to ~99k. The rounds factor compounds itself: every extra agent round re-reads the entire accumulated conversation, so going from ~20 to ~35 rounds doesn't cost +75% — the added rounds are also the most expensive ones, because context is biggest at the end.
Curious: have you experimented with effort/verbosity controls on Sonnet 5? On our template-driven content they cut round counts noticeably without measurable quality loss — which would attack the multiplier at its root rather than at the invoice.