Every AI provider publishes a price in dollars per million tokens. Every comparison table ranks by it. Every build-versus-buy spreadsheet runs on it.
That number is misleading, and not in a small way. It can be wrong by 10x, and wrong in the direction that makes the expensive option look cheap.
This piece explains seven things about the real cost of AI work. Each one is a point where I have watched smart people, including me, get it backwards.
1. The sticker price is not the price
THE POINT: you are billed for tokens the model produces while thinking, even though you never see them and cannot use them.
Modern models emit "reasoning tokens." The model works through the problem, and that working-out is generated text. You are charged for it. On many APIs you never even receive it.
Here is a real evaluation we ran. We were considering swapping our low-cost tier for a cheaper model:
Current model: $0.07 in / $0.27 out per million tokens
Candidate model: $0.05 in / $0.20 out per million tokens
On paper: about 30% cheaper on both sides.
Obvious swap. So we sent it one real request. It answered correctly. Then we read the billing detail:
274 prompt tokens
132 completion tokens
of which 123 were REASONING tokens
of which 9 were the actual answer
You are billed for all 132. You can only use 9.
Do that arithmetic and the effective price of a useful output token was $2.93 per million, which is 14.7 times the advertised rate. The "30% cheaper" model was roughly ten times more expensive than the one it appeared to undercut.
We verified this against the provider's own reported cost for that call and the two agreed exactly, so this is not a units error on our side.
One honest limit: that was a single call. The ratio will move with how hard the question is. Treat the mechanism as the finding, not the specific 14.7. The direction does not move.
There is a second trap in the same family. On some models the reasoning goes into a separate field and the content field comes back empty. Under a tight output limit, the reasoning eats the entire budget before any answer is produced. We saw exactly this: at a 100 token limit, empty response. At 200, a correct one. An empty response looked like the model was incapable. It was actually a budget symptom. If we had trusted the first reading we would have discarded a model that works fine.
WHAT TO DO ABOUT IT: before you believe any price comparison, send one real request and read completion_tokens_details.reasoning_tokens in the response. Then compute dollars per useful output token. If your provider does not expose that field, you cannot actually price the model, and you should say so out loud in the meeting.
2. Cheap models are not uniformly worse. They fail differently, and the difference is the whole story.
THE POINT: the standard mental model, that models sit on one line from dumb to smart, will get you hurt. Two models with nearly identical scores can behave completely differently when it matters.
Most people picture a single quality axis. Expensive is smart, cheap is dumb, pick a point that fits the budget. If that were true, choosing a model would be a budget exercise.
It is not true. Here is a field of models on 800 tool-calling tasks. First, plain accuracy:
frontier model A 95.9
frontier model B 95.1
ours ~94
strong open model 93.6
cheap model C 93.0
cheap model D 91.0
Read that column alone and the cheap models look like a steal. Four percentage points for a fraction of the price.
Now here is a second column. Inside those 800 tasks are deliberate traps: requests where the correct behavior is to decline, because the right tool is not available or the request is malformed. This column is the percentage of traps handled without grabbing the wrong tool:
frontier model A 100%
ours 100%
strong open model 100%
cheap model C 90%
cheap model D 52% <-- and this model scored 91 on accuracy
Stop on that last line. A model that looks four points behind on accuracy will pick up the wrong tool half the time it is baited.
Think about what that means in a system with real permissions. That is not "slightly less accurate." That is a model that will confidently call delete_records when it should have said "I do not have a tool for that." The accuracy average washed the single most important behavior completely out of view.
WHAT TO DO ABOUT IT: build a small set of tasks where the right answer is to refuse, and measure the refusal rate separately. Never let it be averaged into an accuracy score. If a vendor cannot tell you what their model does when it should do nothing, you do not have the number that matters.
3. Because they fail differently, sorting beats upgrading
THE POINT: if cheap models were uniformly worse, your only lever would be paying more. Because they fail in specific, predictable ways, you have a much better lever: send each task to the weakest thing that can actually do it.
This is routing, and the important thing about routing is that it is a sorting problem, not an intelligence problem. Sorting is cheap. Intelligence is expensive. Any time you can convert the second into the first, you win.
A concrete picture over thirty days of our real production traffic, 68,369 requests:
Priced at frontier rates, this exact traffic: $166.25
What it actually cost us: ~$46 to $51
Gross margin: about 70%
But the composition is the part worth internalizing:
84% of our total cost was the escalations to the expensive model. Everything else, all the cheap serving, all the infrastructure, was rounding error next to it.
That means the cost dial is not which model you picked, and not the price you negotiated. It is how often you have to escalate. A 10% reduction in escalation rate does more for your bill than a 10% discount from any vendor.
WHAT TO DO ABOUT IT: instrument your escalation rate before you optimize anything else. If you do not know what fraction of your requests need the expensive model, you do not know what your system costs or why.
4. Verification is what makes cheap safe
THE POINT: routing on its own is a gamble. What turns it into engineering is being able to cheaply check whether the cheap answer is right.
Here is the asymmetry the whole approach rests on:
Producing a correct answer is expensive. Checking one is often very cheap.
You already know this from normal software. Writing the function is the hard part. Running the tests is the easy part. That asymmetry does not disappear when a model writes the function, and it is the thing you should be exploiting.
Cheap checks available to you: run the tests. Check the types. See if it compiles. And one more that people underuse: ask two independent models and see if they agree.
We measured that last one. On questions with no tests to run, using two endpoints we believed were independent:
WITHDRAWN, see the correction below:
When the two agreed, probability the answer was wrong: 0.00 (n=160)
How often they agreed: 76%
What re-measurement found, once the gate was tested per workload shape:
short-answer arithmetic (what the original probe used) ~0
code, with executable tests as ground truth 1.7% to 3.5%
faithfulness judgement, real off-trap data 27.5% (95% CI 16.1 to 42.8, n=70)
how often the two agreed on that data 61%
CORRECTION, added 2026-07-27. Do not use the 0.00 above. A reader (@anp2network) took the number apart in the comments, and our own follow up measurements agree with them on every point. Leaving it uncorrected would be the exact failure this piece complains about elsewhere.
One. Zero wrong in 160 does not bound the error rate at zero. For a zero numerator the 95% upper bound runs about 3/n, roughly 1.9%. With the gate passing 76% of traffic that permits something like 1.4% of everything served to be wrong and never checked. The point estimate was 0.00. The bound never was.
Two. That probe used short answer arithmetic, where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail, and every row showing the primary correct should have been the tell. Re-run on code, using the same agreement check and the benchmark's own tests as ground truth, the gate leaks low single digit percent, not zero.
Three. Agreement substitutes for correctness only if the two endpoints fail independently, and independence is a property of what the provider is serving, not of your code. Ours quietly stopped being independent. The witness was selected by model name, and all three of our cheap backends turned out to be the same underlying model under three different spellings. A same model pair agrees about 88% of the time against 70 to 75% for a genuinely different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.
Those 160 cases spanned four task families including deliberately hard traps. We originally read the traps as reassurance. That was backwards. A trap works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate thins out on the cases that cost the most to get wrong.
What the gate actually buys you, stated honestly: it lowers the rate at which you serve a wrong cheap answer. It does not drive it to zero, and how far it lowers it depends on the two endpoints really being different. Verify that they still are, and keep verifying, because the answer can change without you deploying anything.
The asymmetry this section opens with still holds. Checking is cheaper than producing, and routing by a check still beats routing by hope. What does not hold is treating a single agreement number, measured once, on the wrong shape of task, as a guarantee.
WHAT TO DO ABOUT IT: for every class of work you send to a model, write down how you would check the answer cheaply. If you cannot answer that, that class of work is not a routing candidate yet, and that is useful to know before you build.
5. The benchmarks actively punish the behavior you want
THE POINT: the leaderboards score a correct refusal as a failure. If you pick models by leaderboard, you are selecting against safety.
This one is worth being very explicit about, because it is counterintuitive and it is expensive.
The most valuable behavior in a production agent is declining to act when the request is ambiguous, malformed, or outside its remit.
The major agent benchmarks score task success. A refusal is a failed task. They award exactly nothing for "correctly declined to do the dangerous thing."
So a system tuned for production safety scores WORSE on the headline number than a system that always attempts and is occasionally catastrophically wrong.
Sit with that. The public number that everyone compares is, in this specific and important respect, pointing the wrong way.
WHAT TO DO ABOUT IT: run the benchmarks anyway, because your customers and your competitors will. But report the wrong-action rate right next to the task-success rate, every time. And know both numbers privately before anyone runs them at you publicly.
6. Your workload shape decides your economics, not your architecture
THE POINT: a single blended cost-per-request number hides the variable that actually determines whether this is profitable for you.
Two workloads through our identical system:
Tool-calling work: almost never needs the expensive model
Coding work: about 57% escalated on fresh problems
Same code. Same models. Same prices. One of those is enormously profitable and the other is thin.
Our healthy margin exists partly because our traffic happens to be tool-calling heavy. A customer whose work is mostly fresh coding would see materially worse economics, and it would be dishonest of us to quote them our number.
I am saying that plainly because the whole industry quotes blended numbers, and a blended number is a hidden assumption about your mix.
WHAT TO DO ABOUT IT: when anyone shows you a cost-per-request for an AI system, your first question is "on what mix of work?" If they do not have an answer, the number describes their traffic, not yours. And measure your own mix before you forecast anything.
7. Publish the ceiling honestly, because someone else will find it
THE POINT: we are at parity on code, not ahead, and saying so is the only version that survives contact with a skeptic.
On a clean, cache-free run of a standard coding benchmark:
Our cascade: 92.1
Frontier model A: 92.7
Frontier model B: 93.3
The bare cheap model alone: 81.1
Same harness for all four. We are slightly under the frontier. The architecture adds about 11 points over the cheap model by itself.
We were tempted by a "beats the frontier" line. The measurement did not support it, so we do not use it. Parity is the honest word, and parity at a fraction of the cost is the actual product.
WHAT TO DO ABOUT IT: the number that does not flatter you is the only one worth publishing, because it is the only one that holds up when a customer reruns it. A claim you cannot survive being checked on is a liability with a delayed fuse.
8. The hardest part is not building the system. It is trusting your own measurements.
THE POINT: a wrong measurement is more dangerous than no measurement, because it comes with confidence attached.
This is the one I would most want a reader to take away, because it applies whether or not you ever build any of the above.
In a single working session, we chased seven separate alarms. All seven were broken instruments, not real problems. A parser reading five rows of a seventy-nine row file and reporting a catastrophe. A checker matching error strings against its own console output and finding "errors" it had printed itself. A meter reporting 135% of a hard limit when the true figure was 27%.
Three more from production, all instructive:
We were dropping two thirds of every traffic burst and could not see it. Our server had a careful queue that answered overload with a polite "busy, try again," which is exactly what an aggregator wants. But the operating system's own accept queue underneath it was at its default of five connections, so any burst deeper than five was refused by the kernel before a single line of our code ran. Every load test we had ever written sent exactly as much traffic as the server was willing to admit, which made those tests structurally incapable of finding this. The test design guaranteed the blind spot.
Then we fixed it wrong, in a way that looked right. A configuration flag said a feature was off. The deployed code was an older version that could not express "off" and instead disabled the feature completely. Latency improved. Of course it did, because doing nothing is fast. Every dashboard was green while a feature was 100% dead.
And we measured a worst case at 17 seconds using 10 samples. Later we measured the identical component with 828 samples. The real worst case was 126 seconds. Ten samples gave us an accurate median and a completely wrong tail.
WHAT TO DO ABOUT IT, and these are the four habits that would have caught every case above:
- Change one variable at a time. If your fix changes three things and it works, you have learned that the bundle works. You have learned nothing about why, and you will keep the two useless changes forever.
- Turn your fix off and confirm the problem comes back. A test that only ever passes has told you nothing.
- Match your sample size to the statistic. A median settles in tens of samples. A worst case needs hundreds. A tail measured with 10 samples is not a cautious estimate, it is a wrong one that reads as cautious.
- Chase the gap you cannot explain. When a number is slightly off and you invent a plausible reason to dismiss it, that reason is usually the bug. Twice in one week the explanation I reached for ("network overhead") was covering a real defect.
The one page version
If you remember nothing else:
- Price in dollars per successful result, never per million tokens.
- Read the reasoning token count before believing any quoted price.
- Measure the wrong-action rate separately. Averages hide the failure that matters most.
- Track your escalation rate. It is the cost dial, not your model choice.
- Verify cheaply so you can generate cheaply. Tests, types and independent agreement all cost less than intelligence.
- Quote economics per workload shape. A blended number is a hidden assumption about someone else's traffic.
- Verify the instrument before acting on its number, and be most suspicious when the number is good.
None of this requires owning a frontier model. It requires taking measurement seriously, which is rarer, and considerably cheaper.
Edits to this article
2026-07-27. The agreement figure this piece originally published, a 0.00 probability of a wrong
answer when two endpoints agree at n=160, was withdrawn. A reader, @anp2network, took it apart in the
comments and was right on every point: a zero numerator bounds the error near 3/n rather than at zero,
the probe used short-answer arithmetic where the cheap tier is essentially never wrong so the gate was
measured where the primary does not fail, and independence between two endpoints is a property of what
a provider is serving rather than of your code, so it can decay with nothing in your system changing.
Ours had decayed exactly that way.
2026-07-28. Replaced the withdrawn 0.00 inside the table itself instead of leaving it standing
above a correction, and added the re-measured numbers by workload shape. The honest summary is that
P(wrong when they agree) is not one number. It is close to zero on arithmetic, 1.7 to 3.5 percent on
code checked by executable tests, and 27.5 percent on faithfulness judgement. Quoting any single one of
those as the property of the gate is the mistake this article made the first time.
Also worth stating plainly, since the same commenter suggested it: we now escalate a sample of the agreed cases to the expensive model anyway and score whether it concurs, which turns a one-time n into a number that moves when a route changes behind either endpoint. That is running in production as of this edit. We will publish what it says, including if it is unflattering.
Top comments (25)
The 0.00 measures agreement, and agreement substitutes for correctness only when the two endpoints' errors are independent. Independence is a property of where those endpoints come from, and section 1 already establishes that what sits behind a model name is under provider control. So the correlation between them can rise without anything in your system changing, and the gate would not report it.
Zero wrong in 160 also does not bound the error rate at zero. For a zero-numerator binomial the 95% upper bound runs about 3/n, so roughly 1.9%. With the gate passing 76% of traffic, that permits something like 1.4% of everything you serve to be wrong and never checked. The point estimate is 0.00. The bound is not.
The trap families are where independence is least plausible rather than most, which cuts against reading their inclusion as reassurance. A bait works by shared structure in the input, and shared structure is exactly a correlation source. Two endpoints taken in by the same bait agree, and agreement is your signal to skip the expensive read, so the gate thins out on the wrong-action class section 2 identifies as the one that matters.
Habit two from section 8 has an obvious target here. Escalate a random sample of the agreed cases anyway and score whether the expensive model concurs. At a low sample rate that costs a small slice of an escalation budget already being paid, and it converts a one-time n=160 into a number that moves when a route changes behind either endpoint. The agreement gate is the instrument the piece never turns off to confirm the problem comes back.
You are right, and I have the measurements that say so. I have corrected the article rather than quietly dropping the number.
On the bound: zero wrong in 160 gives a 95% upper bound near 3/n, about 1.9%. We re-ran that gate on code instead of short answer arithmetic, using the production agreement check with the benchmark's own tests as ground truth. It leaks low single digit percent, not zero. Your bound bracketed our own follow up measurement, which is the part of your comment I find most useful.
The original 0.00 came from a probe of short answer arithmetic where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail. Every row of that probe showed the primary correct, which should have been the tell and was not.
On independence, it is not a risk for us, it already happened. Our witness was selected by model name, and all three of our cheap backends turned out to be serving the same underlying model under three different spellings. The genuinely different witness had silently stopped being chosen weeks earlier. So the pair went from independent to identical with nothing in our code changing, exactly as you describe. Measured afterwards, a same model pair agrees about 88% of the time against 70 to 75% for a different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.
Your third point is the one I had not made, and I think it is the sharpest. I had been reading the trap families as evidence the gate was safe. They are the opposite. A bait works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate thins out precisely on the class where being wrong costs the most.
We are taking habit two. Escalating a random sample of agreed cases and scoring whether the expensive model concurs turns a one time n into a number that moves when a route changes behind either endpoint, and it costs a slice of an escalation budget we already pay. You are right that the agreement gate was the one instrument the piece never turned off.
The re-run on code matters more than the correction notice, since a gate measured where the primary never fails was never measured at all.
The 88% against 70 to 75% reads as a postmortem statistic. It also works as a live detector, and it is one you already compute. Agreement rate falls out of every request that reaches the gate, across the whole population, at no added cost. Sampled escalation spends escalation budget and sees a sample. A witness collapsing back into the primary shows up as agreement drifting up toward the same-model band, so that signal was sitting in your traffic for however many weeks the genuinely different witness had stopped being chosen.
The defect is workload. Easier traffic raises agreement too, and a raw rate alarm moves the same way for both causes while saying nothing about which one moved.
Freezing a small canary set and replaying it on a schedule takes the confound out. The input distribution is held constant, so the only thing left that can move agreement on that set is the endpoints. No ground-truth labels are needed, because the measurement is agreement drift and correctness never enters it.
Then choose the canary items deliberately, from cases where the two endpoints historically disagreed. Those carry the discriminating power. If items that used to split stop splitting, that is attribution rather than an alarm. Your sampled escalation will tell you the leak rate got worse. The discriminating canary tells you why.
Under all of it, a model name is a pointer. Nothing in the request or the response bound the provider to what was actually served, which is how three spellings resolved to one model with no error surfacing anywhere. Everything downstream, including the sampled escalation you are adding, is inference about what you were served. The canary does not repair that. It notices when the unverifiable assumption has already broken, which is the weaker position and the one available.
You are right that we already compute it, and that is the part that stings. Agreement falls out of every request that reaches the gate, across the whole population, at no cost. The sampled escalation I was adding spends budget to see a slice of the same thing. A witness collapsing into the primary would have shown up as agreement drifting toward the same model band, and that drift was sitting in our traffic for the entire period the genuinely different witness had stopped being chosen. Nobody looked, because we were treating agreement as a gate to act on rather than a time series to watch.
On the confound, I accept it, and the canary is the better instrument for the why. A raw rate cannot separate an easier workload from a collapsed pair. Freezing the input distribution leaves the endpoints as the only thing that can move the number. Choosing the frozen items from cases where the two endpoints historically disagreed is the part I had not considered, and it is the part that converts an alarm into an attribution. Items that used to split and stop splitting says which side moved. A rate that moved does not. So the two instruments answer different questions and we want both.
Your last point is the one I cannot argue with. A model name is a pointer, nothing in the exchange binds the provider to what was served, and every measurement downstream of that, including the audit I am adding, is inference about what we were given. I am not going to pretend the canary repairs it.
One thing I think it can do that is stronger than noticing drift. At temperature zero on a frozen input, two endpoints serving the same underlying model do not merely agree on the answer, they emit close to the same text. Agreement is a semantic comparison and it gave us 88 percent against 70 to 75. Byte level similarity on a fixed prompt should separate those two cases much harder, because two different models rarely phrase an identical answer identically, while one model asked twice usually does. If that holds, the canary stops being a drift alarm and becomes a same model detector, and it would have fired on the first scheduled replay instead of after enough traffic to move a trend.
That is a claim and not a result. I have not measured it yet. It is cheap to falsify, so I will, and I will post the number either way.
Byte similarity is the right next probe, but as a detector it will not work yet, because it has no reference line under it. Temperature zero does not buy determinism. Greedy decoding still moves at the serving layer: batch composition changes the order of floating point reductions, and a mixture-of-experts router will pick differently depending on which other requests happen to share the batch. Kernel selection and speculative decoding paths shift with load too. So "one model asked twice usually emits the same text" is a measurement about your provider's serving stack. Temperature does not guarantee it.
The first run should be one endpoint against itself, same frozen prompt, repeated across a day at different load. That gives you the self-similarity distribution including its ugly tail, and every cross-endpoint number has to be read against it. If self-similarity sits at 0.93 with a long low tail, a cross-endpoint 0.90 identifies nothing. It might just mean the platform was busy that hour.
There is a second trap at the other end. Separation depends on output entropy. For short constrained answers, an arithmetic result or a single label, genuinely different models return byte-identical text routinely. Both distributions pile up near 1.00, so the byte detector is weakest on exactly the shape of the original 0.00 probe. The canary items need entropy in them. Prose and code leave phrasing room to diverge.
The bigger gain is that the self-series is probably the stronger instrument of the two. Cross-endpoint similarity says two names look like one model. Self-similarity over time on a single endpoint, frozen prompt bytes, says one name became a different model. A step change in that trace is a silent swap under a stable name. Same class of failure you already hit, running the other way, and agreement-rate watching cannot see it when both endpoints move together.
The honest limit stays where it was. Both readings are still your inference about your provider, and nothing here binds the provider to anything. What changes is who is able to check. Publish the replay record as prompt bytes, timestamp, endpoint identifier, returned bytes, with a hash over the whole thing, and the 88-versus-70 number and whatever the byte-similarity number turns out to be stop being private figures. Someone with no relationship to you can reissue the same prompt and see whether it reproduces. The provider is still unbound. Your claim about the provider becomes falsifiable by a stranger.
I measured it before your reply landed, and your reply predicts the result. Posting the number as promised, because it goes against me.
Live setup rather than a fixture. Our three cheap slots today are slot 1 gpt-oss-120b at one provider, slot 3 the same gpt-oss-120b at a different provider under a different name, and slot 2 llama-3.3-70b. So the failure I wanted to detect, one model wearing two endpoint names, is in production right now, and slot 2 is a true negative. Twelve free-form prompts, temperature zero, four arms. The fourth arm is the reference line you are describing: the same endpoint asked the identical question twice.
Byte similarity, difflib on normalized text:
self against itself 0.305, same model different provider 0.182, different model 0.072.
The reference line is the whole result. One endpoint asked the same question twice scores 0.305 against itself, so my premise failed a level below my conclusion. Two providers were never going to emit near-identical text when one provider does not emit near-identical text to itself. Without that arm I would have read 0.182 as "different models" and concluded the exact inverse of the truth.
It also does not beat agreement. Separation was +0.110 for byte similarity against +0.145 for a semantic word-overlap proxy, and no metric separates per item. Every distribution overlaps. Aggregate or nothing.
On your load point, I have accidental evidence for it. An earlier partial run of the identical script put self-similarity at 0.208 and same-model at 0.196. The full run an hour later gave 0.305 and 0.182. Same prompts, same endpoints, same temperature. That is your batch composition and kernel selection moving under me, and it means my single number is a point estimate on a distribution I have not characterised. A run across a day at varying load is the right first experiment and I had it in the wrong order.
Your entropy trap is the one I would have walked into next. My prompts were deliberately several sentences of prose, so I never saw the ceiling effect, and you are right that it bites hardest on exactly the short arithmetic shape the original 0.00 probe used. The detector is weakest where that claim was made.
The self-series being the stronger instrument is the part I did not have and it is better than either of ours. Cross-endpoint similarity answers "are these two names one model". Self-similarity over time on one endpoint answers "did this one name quietly become a different model", and a step change in that trace is a silent swap. Agreement watching is blind to it when both endpoints move together, which is the version of our failure we have not been looking for at all.
One fragment of mine that might survive into that design. I also scored similarity on the first 120 characters alone, and it separated best of anything I tried: self 0.810, same model 0.702, different model 0.481. Divergence accumulates token by token, so identity lives at the start of an answer and is gone by the end. Comparing whole answers throws that away. For a self-series trace the prefix is probably the sharper signal and it is cheaper to store.
On publishing the replay record, I have no argument against it and it is the part that actually changes anything. Prompt bytes, timestamp, endpoint identifier, returned bytes, hashed. I will not pretend it binds the provider. It moves the claim from private to falsifiable by a stranger, which is the only honest position available and better than the one we have.
Caveats so you have them rather than find them: n is 12, no significance test, the agreement column is a proxy and not our production comparator, and one prompt family.
Three for three. You have now improved this system more than our last two internal reviews did.
The 120 char result is the interesting part, but it has a nasty confound. The opening of an answer is also the most conventional region: framing sentence, list setup, "Here's..." style, formatting habits. That is exactly the layer a provider wrapper can impose, and also the layer a fine tune can flatten. Two different base models behind the same serving prompt could score high on prefix similarity for reasons that have little to do with shared weights.
The next run should attack that directly. Use prompts that force the answer to begin with payload only, no preamble, no explanation before the value. Or strip the leading sentence before scoring. If prefix separation survives, then identity really may be front loaded. If it falls back toward the full text numbers, the metric mostly found convention.
And given 0.208 becoming 0.305 with everything fixed, self cannot be a scalar threshold. It has to be a time measured band per endpoint. A swap then shows up as the band's location moving, while load mostly changes width.
You are right, and the confound is worse for me than for the metric, because the prefix number is the one I was most pleased with. Self 0.810, same model 0.702, different model 0.481. That separation sits exactly where a serving wrapper and a flattened fine tune would live, so I cannot distinguish front loaded identity from front loaded convention with what I ran. I have not done the payload only re run yet.
One thing I can add, from a different measurement the same day, that gives a way to test your confound without having to remove it. I was probing a semantic cache, embedding cosine rather than byte similarity, on paraphrase pairs. That distribution came out sharply bimodal: real matches at 0.93 and above, non matches between 0.1 and 0.5, and almost nothing in between. Moving the acceptance threshold from 0.70 to 0.90 changed the verdict on 3 of 27 probes. That is what a metric with genuine discriminating power looks like on this kind of task.
My byte similarity numbers look nothing like that. They are a smooth band with means about 0.1 apart on full text and about 0.3 apart on the prefix, with every distribution overlapping and no item level separation at all. So the shape itself is diagnostic. If prefix similarity were tracking shared weights I would expect it to sharpen toward bimodal, because two endpoints either are the same model or are not. A smooth band is what a graded stylistic property produces. So the test I would run alongside yours is to report the distribution shape rather than the means, on both the full text and the prefix. If stripping the preamble sharpens the prefix distribution, identity really is front loaded. If it stays a smooth band, convention is what the metric found, and no amount of threshold tuning will fix it.
On the band rather than a scalar, I accept it completely and my own accident is the evidence: 0.208 self similarity in a partial run, 0.305 an hour later, same script, same endpoints, same temperature. Nothing changed except load. A scalar cannot survive that.
And there is a consequence of the band framing that I think is a real limitation of the whole approach, not just of my run. If the detector needs a per endpoint baseline measured over time before it can say anything, then it cannot catch the swap that happened before you established the band. You would be calibrating against an endpoint that may already be the wrong model, and a swap only becomes visible as movement away from whatever it was serving when you started watching. In our case the pair had already collapsed to one model weeks before anyone looked, so a band established today would have quietly enshrined the broken state as normal. Which argues for the canary being seeded from cases where the two endpoints historically disagreed, exactly as you said, because those items carry a prior about what disagreement used to look like. That is the only part of the setup that is not purely relative to the moment you started measuring.
The bimodality test needs one more constraint: the same-model mode cannot be narrower than the self arm. Serving variance sets the minimum width. Your own self measurement moving from 0.208 to 0.305 with the endpoint and prompt fixed is exactly the evidence that byte similarity is preserving variance that the serving stack introduces.
That is why the cache probe is a dangerous analogy. Embedding cosine is deliberately insensitive to many surface-form changes, so it collapses a lot of the load, routing, wrapper, and decoding variation that byte comparison exposes. Also, paraphrase matching has a real binary label underneath it. Byte similarity between two generated answers does not get that clean target. So if stripping the preamble fails to sharpen the prefix distribution, that does not separate "the metric only found convention" from "same weights, smeared by serving nondeterminism." The shape test can give you positive evidence when it sharpens in the right place. It cannot carry the negative case by itself.
I'd make the self arm the reference distribution for identity. The null is: two endpoints serving the same weights should look statistically indistinguishable from one endpoint asked twice, measured under the same replay protocol and over comparable wall-clock windows. Then the test becomes distribution equality between cross-provider and self, plus separation of known-different from the self band. A two-sample KS test is the crude version, though the operational detail matters more than the named test: the cross-provider pair has to look like self, not like an abstract bimodal ideal.
On the pre-baseline swap problem, yes. The disagreement-seeded canary is the only piece that reaches backward, but it only reaches as far as the disagreement record, and that record has a half-life. Old disagreements stop being clean priors because models get legitimately updated.
The structural escape is overlapping baselines. A collapse that predates your first measurement may still postdate somebody else's frozen record. That is what the replay artifact buys you: prompt bytes, timestamp, returned bytes, hash. You can rebuild bands from records whose operators you do not need to trust, because the claims are re-checkable at the byte level. One baseline has a fixed blind spot. A population of verifiable baselines only shares the blind spot before the oldest usable record.
Conceded on the analogy, and you have named why it was doing work it had not earned. Embedding cosine is built to be insensitive to surface form, which is precisely the load, routing, wrapper and decoding variance that byte comparison exposes, and paraphrase matching has a real binary label underneath it that "are these two generations from the same weights" does not. So the sharpness of the cache distribution was borrowed from an easier problem. My shape argument can give positive evidence if the prefix sharpens in the right place, and it cannot carry the negative case. I will not use it that way.
Your reformulation is better than what I had, and I am adopting it. Self is the reference distribution, the null is that two endpoints on the same weights are statistically indistinguishable from one endpoint asked twice under the same replay protocol, and the test is distribution equality against self plus separation of known-different from the self band. That is a real null. Mine was a vibe about shape.
The operational detail you flagged is the one I already have evidence for, and it is worse than a caveat. My 0.208 and my 0.305 were the same script, same prompts, same endpoints, an hour apart. So the self band moves on wall-clock, and any protocol that runs arm-by-arm is comparing arms measured under different serving conditions. The fix is cheap and I had not been doing it: interleave the arms inside each replay round and randomise their order, so load is a shared nuisance rather than a per-arm confound. Sequential arms are how I got a self number lower than a cross number in the partial run.
One thing I can add, from a different measurement today, and I think it sidesteps the smearing problem entirely rather than modelling it.
Text similarity is the wrong observable for endpoint identity because generation is high-entropy and the serving stack adds variance you then have to subtract back out. A capability probe has neither problem. Today I was measuring parallel tool-calling and got this, over eight cases, counting the number of tool calls returned against the number required:
The observable is an integer, not a string. There is no phrasing room, so serving nondeterminism has almost nothing to act on, and the separation is total rather than a band with overlapping means. It also survives the entropy trap you raised at the other end: this is not a short constrained answer where different models coincide, because the structure of the response is what differs, not its content.
So I would put a capability fingerprint next to the self-series rather than under it. A fixed set of items chosen because they split on structural behaviour, replayed on a schedule, scored as a discrete count. If an endpoint's fingerprint changes, the weights behind the name changed, and you do not need a reference distribution to say so because the answer is 8 or it is 0.
The honest limits: it detects a change of model, not a change of quantisation or a wrapper, and it needs items where models genuinely differ structurally, which is a harder set to build than a paraphrase pair. And it is only a pointer test, same as everything else downstream of a model name, which is your standing point and remains the one I cannot argue with.
A discrete structural observable is the right class of instrument for this particular nuisance. If the measurement is "how many tool calls were emitted" rather than "how similar was the text," most of the serving variance that forced the self-band apparatus has very little surface area to act on. That is a real improvement for change detection.
The stronger reading still does not follow from your own table. The same named model under a reasoning setting scores 0/8, while the same named model without that setting scores 1/8. Same weights, different serving configuration, different fingerprint. So the observable is reading the served system: wrapper, reasoning toggle, tool-call planning policy, serializer, and whatever constraints sit between logits and the returned API object. The "exactly one call, every time" case especially looks like a tool-call serialization or format constraint rather than a weight-level behavioral boundary.
That does not weaken the instrument. It just locates it correctly. Anything observable downstream of a model name measures served behavior. A fingerprint shift is strong evidence that something behind the name changed, and for that claim it can be a low-false-positive detector that does not need a self reference band. The layer attribution remains an inference. Weights are one candidate. A wrapper policy or a tool serialization path would produce the same shift. The test says the served object changed.
The other limit is cardinality. Eight items scored as small integers carry only a few bits. Your table already shows collisions: two systems at 8/8, two systems at 0/8. A replacement inside one collision class is invisible to this fingerprint, even if the replacement would be obvious under another probe family. So the item set has to be chosen against the specific substitutions you care about. That prior belongs in the claim, otherwise "different model" silently means "different along the axes this small probe set separates."
The clean shape is an ensemble. The structural fingerprint is the cheap no-baseline tripwire for served-behavior changes, the self-similarity band is the graded instrument for pairs the fingerprint cannot split, and the published replay artifact is what lets anyone with no relationship to either of us re-run the claim against the same items and scoring rules.
Taken straight, and the correction is the right one. My own table refutes my stronger reading: same named model, reasoning on scores 0/8, reasoning off scores 1/8. Same weights, different serving configuration, different fingerprint. So the instrument reads the served object, not the weights, and the layer attribution was an inference I had not earned.
I will keep the instrument and move the claim. "Something behind this name changed" is what it can support, and that is worth having on its own, because it needs no self reference band and it is cheap to run.
Your framing landed somewhere useful this week. We had a cost meter keyed on our own slot name rather than the endpoint that actually served, so tool calls served by one provider were recorded under another, and every per provider cost number derived from it was wrong. The fix was to stop trusting the name and read what the provider returns: the cost it reports, and the field naming the upstream that served it. Anything observable downstream of a model name measures served behavior, as you put it, which is exactly why the name was never a safe key.
On cardinality you are right and I will not defend eight. It sizes whether the observable separates at all, which is a capability question. It cannot carry a rate.
We actually have to deal with this issue, we monitor how much we use AI by counting the raw tokens and set a budget for each month, with warnings when we're getting close to the limit. But if our model does a lot of work behind the scenes before giving an answer, like it did here, the real cost of each feature might be a lot higher than we think. It's worth looking into: compare the number of tokens we're charged for to how many of those are actually useful and shown to users. If there's a big difference, we're probably underestimating the true cost and should start tracking "useful tokens" instead of just the total. This way, we can get a better idea of how much we're really spending.
You are describing exactly the gap the piece is about, and I have numbers on it from this week that will probably surprise you in the direction that matters for your budget.
We compared a cheap reasoning tier against a frontier model on identical prompts. The cheap one emitted 244 output tokens on average against the frontier's 83, and it was longer on 10 of 10, with both answers correct. So the model you are billed least for per token can be the one consuming the most of them, and a monthly budget built on the assumption that cheap equals fewer tokens will drift in the wrong direction quietly.
The part that matters for your "useful tokens" idea is where the invisible tokens actually come from, because it decides whether you can do anything about them.
We tried the obvious lever first and it failed. A terseness instruction in the prompt recovered 3 percent, and 4 of the 10 responses got LONGER. That is because these are reasoning tokens, not verbosity: the model is thinking before answering, and asking it to be brief does not persuade it to think less. Unpromptable by construction.
The lever that worked is a provider parameter. Setting reasoning effort to low on one of our slots took it from 52.0 output tokens to 16.4, a 68 percent reduction, non-overlapping ranges across 8 samples per arm. So the ratio you want to track is not really per-feature, it is per-slot-configuration, because a single config field moved two thirds of the spend without touching a line of application code.
Two traps in doing this, both of which cost us real money before we noticed:
One. The parameter shape is not portable. One provider takes a flat
reasoning_effort, another takes it nested under areasoningobject. Send the wrong shape and it is accepted with a 200 and silently ignored, so your config reads as done and you are billed at full price. We had an empty config field sitting on a reasoning model for days for exactly this reason. Verify by measuring output tokens before and after, never by reading the config back.Two. Do not estimate this in characters. We measured characters falling 36 percent while tokens fell 17 percent on the same change, so a character-based proxy overstated the saving by about 2x.
On your actual proposal, tracking useful against total: I would do it, and I would define useful as tokens present in the response you show the user, with everything else in one bucket rather than trying to attribute it. The absolute ratio is less informative than its movement, because it changes when a provider swaps what sits behind a model name, and you find out from that ratio or you do not find out at all.
That "accepted with a 200 but silently ignored" trap is the scariest part, you'd never catch that from logs alone, only by actually measuring output tokens. We've been eyeballing character counts as a rough proxy, so that 36% vs 17% mismatch is a good wake-up call to stop doing that.
Going to start tracking output tokens per slot before/after any config change instead of trusting the response code. Appreciate you sharing real numbers, way more useful than the theory I was working from.
Character counts as a proxy will hold until they suddenly do not, and the failure is quiet, so tracking real output tokens per slot is the right move.
One trap worth knowing before you build it, because we shipped it ourselves last week. Our cost meter went in, and a provider that returns no cost field left the meter at zero, so those requests recorded as $0.0000. Free and not measured looked identical in the table, and zero reads as the best number in any comparison, so nothing flagged it.
So: unknown should be NULL, never 0, and record the reason beside it. Three states rather than two. No call made is genuinely zero. Every call priced is a real sum. Any call unpriced makes the whole request unknown, because a partial total is missing money, not a cheaper request.
We are two days into having that meter running properly and it has already corrected two numbers I had said out loud.
That's a sneaky one, zero looking better than every real number is exactly the kind of bug that hides in plain sight. Good instinct making unknown its own state instead of quietly rounding it to free.
Appreciate you sharing what broke on your end too, saves everyone else from finding it the hard way.
Since that exchange we walked into the next trap on exactly the path you described, so it is worth passing on before you build it.
Tracking output tokens per slot before and after a config change is the right instinct. What bit us is that a before/after on a small sample is mostly noise, and it does not look like noise at the time.
The flag was never real, and the rollback was itself an unverified change to production, which is the part that still bothers me.
The cheap fix is one extra run before you trust any comparison: run the identical thing twice with nothing changed at all, and count how many individual items give a different answer. That percentage is your noise floor. On a fixed 800 item benchmark ours came out at 2.0 to 2.9 percent, so any gap smaller than roughly twice that is not a finding, it is the same measurement twice.
Worth doing on your per-slot token counts before the first config change rather than after, because the floor is what tells you which deltas mean anything.
Good call, would've absolutely mistaken that kind of noise for a real regression. Going to run the same-twice check first before trusting any before/after numbers now. Thanks for walking through all of this, saved me a bunch of mistakes I would've made blind.
Eye-opening breakdown on reasoning tokens! When benchmarking LLM costs across providers, do you have a standard script or test set you run to estimate 'useful output' vs hidden reasoning bloat before committing to an API?
Thanks. Honest answer: no polished script I can hand you, but the method is four lines of arithmetic and it is the part that matters, so here it is.
Do not benchmark. Run your OWN workload shape against the candidate, and for each call read both the total completion tokens and the reasoning tokens. Most OpenAI compatible providers expose the second as usage.completion_tokens_details.reasoning_tokens. Then compute the only number that means anything:
effective output price = sticker output price times (total completion tokens divided by useful completion tokens)
That is it. The useful count is total minus reasoning.
Here is the measurement from the post, which is the one that stopped us swapping our cheap tier. Candidate advertised 0.05 dollars in and 0.20 out per million against the model we were already running at 0.07 and 0.27, so on paper about a 30 percent saving on both sides. On a real tool call it returned 274 prompt and 132 completion tokens, and 123 of those 132 were reasoning. Nine were useful. Effective output price 2.93 dollars per million, which is 14.7 times sticker and roughly 10 times more expensive than the tier it appeared to undercut.
Three traps I would flag, because the arithmetic is easy and the setup is where it goes wrong.
The ratio is a property of your WORKLOAD, not of the model. A reasoning model on a short tool call is almost all overhead, because the thinking does not shrink to match a nine token answer. Give the same model a long analytical prose task and the ratio gets far more reasonable. So measuring on a benchmark and deploying on tool calls will mislead you in the expensive direction.
Some providers do not return reasoning tokens in usage at all. You then cannot compute this, and I would treat that as information rather than as an inconvenience. If you cannot see what you are billed for, price the risk in.
And that figure above is n equals 1. It was decisive for us because the gap was an order of magnitude and not a few percent, but do not take a single call as a rate. Run your own spread and look at the distribution, because the tail is where a per request budget actually breaks.
The general version, which is the thing I would put on a wall: tokens are the unit you are billed in, not the unit you receive. Price per useful token is the only comparison that survives contact with a reasoning model.
"Tokens are the unit you are billed in, not the unit you receive"—truly a quote for the ages!
Thanks for laying out the math so clearly, Tom. Looking at price per useful token completely shifts how we need to structure our evaluation runs for tool-calling workloads.
Thank you, that is kind.
One thing I would add for tool calling specifically, since that is the shape you are about to measure. Put a trap subset in the run and score it as its own column, never averaged into accuracy. By trap I mean requests where the correct behaviour is to decline, because the right tool is not available or the request is malformed. In the 800 task field in the post, the model that scored 91 on plain accuracy handled 52 percent of those, and a model that scored 93 handled 90 percent. That gap is completely invisible in an accuracy average, and it is the number that decides whether you can hand the thing real permissions.
And on the cost side, budget at p90 rather than the mean. The reasoning ratio has a tail, and a per request budget breaks at the tail. The mean is how you go broke, p90 is how you stay honest.
That trap-subset idea is brilliant—averaging refusal accuracy into overall accuracy definitely hides model hallucination risks when permissions are on the line. And "budget at p90" is getting printed and stuck to my monitor. Thanks for the extra insights, Tom!
One more for the monitor, and it is new since I wrote that, because it caught us out yesterday.
When you run those provider comparisons, the reasoning bloat may not be something you can prompt away. We asked our cheap tier and a frontier model the same ten short questions. The cheap one used three times the output tokens while both got every answer right, and it was not padding: it answers "Lisbon", bare, and still spends 17 tokens. An explicit "bare value only, no preamble" system prompt recovered 3 percent, and made four of the ten longer, because the instruction is itself input and reasoning tokens are generated regardless.
The lever is a provider parameter rather than a prompt. Setting a reasoning cap on one tier that had none took its output from a mean of 52 tokens to 16, a 68 percent cut, measured over eight runs per arm.
Two traps in doing that, both of which cost us time. The parameter name is not portable: Together honours a flat reasoning_effort, and the nested shape another provider uses is accepted and then silently ignored, so it reads as configured and still bills full price. And temperature zero is not deterministic on these endpoints. The uncapped arm returned 36, 50 or 60 tokens for the same prompt, so a single before-and-after call is noise. I nearly changed production config on one draw before repeating it.
So the practical version of price per useful token: check whether your candidate exposes a reasoning cap at all, confirm it is actually honoured by watching the token count move rather than by reading the docs, and repeat the measurement enough times to see the spread.