Every AI provider publishes a price in dollars per million tokens. Every comparison table ranks by it. Every build-versus-buy spreadsheet runs on it.
That number is misleading, and not in a small way. It can be wrong by 10x, and wrong in the direction that makes the expensive option look cheap.
This piece explains seven things about the real cost of AI work. Each one is a point where I have watched smart people, including me, get it backwards.
1. The sticker price is not the price
THE POINT: you are billed for tokens the model produces while thinking, even though you never see them and cannot use them.
Modern models emit "reasoning tokens." The model works through the problem, and that working-out is generated text. You are charged for it. On many APIs you never even receive it.
Here is a real evaluation we ran. We were considering swapping our low-cost tier for a cheaper model:
Current model: $0.07 in / $0.27 out per million tokens
Candidate model: $0.05 in / $0.20 out per million tokens
On paper: about 30% cheaper on both sides.
Obvious swap. So we sent it one real request. It answered correctly. Then we read the billing detail:
274 prompt tokens
132 completion tokens
of which 123 were REASONING tokens
of which 9 were the actual answer
You are billed for all 132. You can only use 9.
Do that arithmetic and the effective price of a useful output token was $2.93 per million, which is 14.7 times the advertised rate. The "30% cheaper" model was roughly ten times more expensive than the one it appeared to undercut.
We verified this against the provider's own reported cost for that call and the two agreed exactly, so this is not a units error on our side.
One honest limit: that was a single call. The ratio will move with how hard the question is. Treat the mechanism as the finding, not the specific 14.7. The direction does not move.
There is a second trap in the same family. On some models the reasoning goes into a separate field and the content field comes back empty. Under a tight output limit, the reasoning eats the entire budget before any answer is produced. We saw exactly this: at a 100 token limit, empty response. At 200, a correct one. An empty response looked like the model was incapable. It was actually a budget symptom. If we had trusted the first reading we would have discarded a model that works fine.
WHAT TO DO ABOUT IT: before you believe any price comparison, send one real request and read completion_tokens_details.reasoning_tokens in the response. Then compute dollars per useful output token. If your provider does not expose that field, you cannot actually price the model, and you should say so out loud in the meeting.
2. Cheap models are not uniformly worse. They fail differently, and the difference is the whole story.
THE POINT: the standard mental model, that models sit on one line from dumb to smart, will get you hurt. Two models with nearly identical scores can behave completely differently when it matters.
Most people picture a single quality axis. Expensive is smart, cheap is dumb, pick a point that fits the budget. If that were true, choosing a model would be a budget exercise.
It is not true. Here is a field of models on 800 tool-calling tasks. First, plain accuracy:
frontier model A 95.9
frontier model B 95.1
ours ~94
strong open model 93.6
cheap model C 93.0
cheap model D 91.0
Read that column alone and the cheap models look like a steal. Four percentage points for a fraction of the price.
Now here is a second column. Inside those 800 tasks are deliberate traps: requests where the correct behavior is to decline, because the right tool is not available or the request is malformed. This column is the percentage of traps handled without grabbing the wrong tool:
frontier model A 100%
ours 100%
strong open model 100%
cheap model C 90%
cheap model D 52% <-- and this model scored 91 on accuracy
Stop on that last line. A model that looks four points behind on accuracy will pick up the wrong tool half the time it is baited.
Think about what that means in a system with real permissions. That is not "slightly less accurate." That is a model that will confidently call delete_records when it should have said "I do not have a tool for that." The accuracy average washed the single most important behavior completely out of view.
WHAT TO DO ABOUT IT: build a small set of tasks where the right answer is to refuse, and measure the refusal rate separately. Never let it be averaged into an accuracy score. If a vendor cannot tell you what their model does when it should do nothing, you do not have the number that matters.
3. Because they fail differently, sorting beats upgrading
THE POINT: if cheap models were uniformly worse, your only lever would be paying more. Because they fail in specific, predictable ways, you have a much better lever: send each task to the weakest thing that can actually do it.
This is routing, and the important thing about routing is that it is a sorting problem, not an intelligence problem. Sorting is cheap. Intelligence is expensive. Any time you can convert the second into the first, you win.
A concrete picture over thirty days of our real production traffic, 68,369 requests:
Priced at frontier rates, this exact traffic: $166.25
What it actually cost us: ~$46 to $51
Gross margin: about 70%
But the composition is the part worth internalizing:
84% of our total cost was the escalations to the expensive model. Everything else, all the cheap serving, all the infrastructure, was rounding error next to it.
That means the cost dial is not which model you picked, and not the price you negotiated. It is how often you have to escalate. A 10% reduction in escalation rate does more for your bill than a 10% discount from any vendor.
WHAT TO DO ABOUT IT: instrument your escalation rate before you optimize anything else. If you do not know what fraction of your requests need the expensive model, you do not know what your system costs or why.
4. Verification is what makes cheap safe
THE POINT: routing on its own is a gamble. What turns it into engineering is being able to cheaply check whether the cheap answer is right.
Here is the asymmetry the whole approach rests on:
Producing a correct answer is expensive. Checking one is often very cheap.
You already know this from normal software. Writing the function is the hard part. Running the tests is the easy part. That asymmetry does not disappear when a model writes the function, and it is the thing you should be exploiting.
Cheap checks available to you: run the tests. Check the types. See if it compiles. And one more that people underuse: ask two independent models and see if they agree.
We measured that last one. On questions with no tests to run, using two independent endpoints:
When the two agreed, probability the answer was wrong: 0.00 (n=160)
How often they agreed: 76%
Those 160 cases spanned four task families including deliberately hard traps.
Read what that actually buys you. It is not "cheap models are good enough," which is a hope. It is: 76% of the time you can use the cheap answer AND KNOW that you can. The other 24% escalates to the expensive model.
That is the difference between gambling and sorting. You are not hoping the cheap model was right. You have a test that tells you when it was.
WHAT TO DO ABOUT IT: for every class of work you send to a model, write down how you would check the answer cheaply. If you cannot answer that, that class of work is not a routing candidate yet, and that is useful to know before you build.
5. The benchmarks actively punish the behavior you want
THE POINT: the leaderboards score a correct refusal as a failure. If you pick models by leaderboard, you are selecting against safety.
This one is worth being very explicit about, because it is counterintuitive and it is expensive.
The most valuable behavior in a production agent is declining to act when the request is ambiguous, malformed, or outside its remit.
The major agent benchmarks score task success. A refusal is a failed task. They award exactly nothing for "correctly declined to do the dangerous thing."
So a system tuned for production safety scores WORSE on the headline number than a system that always attempts and is occasionally catastrophically wrong.
Sit with that. The public number that everyone compares is, in this specific and important respect, pointing the wrong way.
WHAT TO DO ABOUT IT: run the benchmarks anyway, because your customers and your competitors will. But report the wrong-action rate right next to the task-success rate, every time. And know both numbers privately before anyone runs them at you publicly.
6. Your workload shape decides your economics, not your architecture
THE POINT: a single blended cost-per-request number hides the variable that actually determines whether this is profitable for you.
Two workloads through our identical system:
Tool-calling work: almost never needs the expensive model
Coding work: about 57% escalated on fresh problems
Same code. Same models. Same prices. One of those is enormously profitable and the other is thin.
Our healthy margin exists partly because our traffic happens to be tool-calling heavy. A customer whose work is mostly fresh coding would see materially worse economics, and it would be dishonest of us to quote them our number.
I am saying that plainly because the whole industry quotes blended numbers, and a blended number is a hidden assumption about your mix.
WHAT TO DO ABOUT IT: when anyone shows you a cost-per-request for an AI system, your first question is "on what mix of work?" If they do not have an answer, the number describes their traffic, not yours. And measure your own mix before you forecast anything.
7. Publish the ceiling honestly, because someone else will find it
THE POINT: we are at parity on code, not ahead, and saying so is the only version that survives contact with a skeptic.
On a clean, cache-free run of a standard coding benchmark:
Our cascade: 92.1
Frontier model A: 92.7
Frontier model B: 93.3
The bare cheap model alone: 81.1
Same harness for all four. We are slightly under the frontier. The architecture adds about 11 points over the cheap model by itself.
We were tempted by a "beats the frontier" line. The measurement did not support it, so we do not use it. Parity is the honest word, and parity at a fraction of the cost is the actual product.
WHAT TO DO ABOUT IT: the number that does not flatter you is the only one worth publishing, because it is the only one that holds up when a customer reruns it. A claim you cannot survive being checked on is a liability with a delayed fuse.
8. The hardest part is not building the system. It is trusting your own measurements.
THE POINT: a wrong measurement is more dangerous than no measurement, because it comes with confidence attached.
This is the one I would most want a reader to take away, because it applies whether or not you ever build any of the above.
In a single working session, we chased seven separate alarms. All seven were broken instruments, not real problems. A parser reading five rows of a seventy-nine row file and reporting a catastrophe. A checker matching error strings against its own console output and finding "errors" it had printed itself. A meter reporting 135% of a hard limit when the true figure was 27%.
Three more from production, all instructive:
We were dropping two thirds of every traffic burst and could not see it. Our server had a careful queue that answered overload with a polite "busy, try again," which is exactly what an aggregator wants. But the operating system's own accept queue underneath it was at its default of five connections, so any burst deeper than five was refused by the kernel before a single line of our code ran. Every load test we had ever written sent exactly as much traffic as the server was willing to admit, which made those tests structurally incapable of finding this. The test design guaranteed the blind spot.
Then we fixed it wrong, in a way that looked right. A configuration flag said a feature was off. The deployed code was an older version that could not express "off" and instead disabled the feature completely. Latency improved. Of course it did, because doing nothing is fast. Every dashboard was green while a feature was 100% dead.
And we measured a worst case at 17 seconds using 10 samples. Later we measured the identical component with 828 samples. The real worst case was 126 seconds. Ten samples gave us an accurate median and a completely wrong tail.
WHAT TO DO ABOUT IT, and these are the four habits that would have caught every case above:
- Change one variable at a time. If your fix changes three things and it works, you have learned that the bundle works. You have learned nothing about why, and you will keep the two useless changes forever.
- Turn your fix off and confirm the problem comes back. A test that only ever passes has told you nothing.
- Match your sample size to the statistic. A median settles in tens of samples. A worst case needs hundreds. A tail measured with 10 samples is not a cautious estimate, it is a wrong one that reads as cautious.
- Chase the gap you cannot explain. When a number is slightly off and you invent a plausible reason to dismiss it, that reason is usually the bug. Twice in one week the explanation I reached for ("network overhead") was covering a real defect.
The one page version
If you remember nothing else:
- Price in dollars per successful result, never per million tokens.
- Read the reasoning token count before believing any quoted price.
- Measure the wrong-action rate separately. Averages hide the failure that matters most.
- Track your escalation rate. It is the cost dial, not your model choice.
- Verify cheaply so you can generate cheaply. Tests, types and independent agreement all cost less than intelligence.
- Quote economics per workload shape. A blended number is a hidden assumption about someone else's traffic.
- Verify the instrument before acting on its number, and be most suspicious when the number is good.
None of this requires owning a frontier model. It requires taking measurement seriously, which is rarer, and considerably cheaper.
Top comments (0)