Every AI provider publishes a price in dollars per million tokens. Every comparison table ranks by it. Every build-versus-buy spreadsheet runs on i...
For further actions, you may consider blocking this person and/or reporting abuse
The 0.00 measures agreement, and agreement substitutes for correctness only when the two endpoints' errors are independent. Independence is a property of where those endpoints come from, and section 1 already establishes that what sits behind a model name is under provider control. So the correlation between them can rise without anything in your system changing, and the gate would not report it.
Zero wrong in 160 also does not bound the error rate at zero. For a zero-numerator binomial the 95% upper bound runs about 3/n, so roughly 1.9%. With the gate passing 76% of traffic, that permits something like 1.4% of everything you serve to be wrong and never checked. The point estimate is 0.00. The bound is not.
The trap families are where independence is least plausible rather than most, which cuts against reading their inclusion as reassurance. A bait works by shared structure in the input, and shared structure is exactly a correlation source. Two endpoints taken in by the same bait agree, and agreement is your signal to skip the expensive read, so the gate thins out on the wrong-action class section 2 identifies as the one that matters.
Habit two from section 8 has an obvious target here. Escalate a random sample of the agreed cases anyway and score whether the expensive model concurs. At a low sample rate that costs a small slice of an escalation budget already being paid, and it converts a one-time n=160 into a number that moves when a route changes behind either endpoint. The agreement gate is the instrument the piece never turns off to confirm the problem comes back.
You are right, and I have the measurements that say so. I have corrected the article rather than quietly dropping the number.
On the bound: zero wrong in 160 gives a 95% upper bound near 3/n, about 1.9%. We re-ran that gate on code instead of short answer arithmetic, using the production agreement check with the benchmark's own tests as ground truth. It leaks low single digit percent, not zero. Your bound bracketed our own follow up measurement, which is the part of your comment I find most useful.
The original 0.00 came from a probe of short answer arithmetic where the cheap tier is essentially never wrong. A gate cannot be measured where the primary does not fail. Every row of that probe showed the primary correct, which should have been the tell and was not.
On independence, it is not a risk for us, it already happened. Our witness was selected by model name, and all three of our cheap backends turned out to be serving the same underlying model under three different spellings. The genuinely different witness had silently stopped being chosen weeks earlier. So the pair went from independent to identical with nothing in our code changing, exactly as you describe. Measured afterwards, a same model pair agrees about 88% of the time against 70 to 75% for a different one. That is close to a rubber stamp, and a rubber stamp also reports 0.00.
Your third point is the one I had not made, and I think it is the sharpest. I had been reading the trap families as evidence the gate was safe. They are the opposite. A bait works by shared structure in the input, shared structure is exactly what correlates two endpoints, so the gate thins out precisely on the class where being wrong costs the most.
We are taking habit two. Escalating a random sample of agreed cases and scoring whether the expensive model concurs turns a one time n into a number that moves when a route changes behind either endpoint, and it costs a slice of an escalation budget we already pay. You are right that the agreement gate was the one instrument the piece never turned off.
The re-run on code matters more than the correction notice, since a gate measured where the primary never fails was never measured at all.
The 88% against 70 to 75% reads as a postmortem statistic. It also works as a live detector, and it is one you already compute. Agreement rate falls out of every request that reaches the gate, across the whole population, at no added cost. Sampled escalation spends escalation budget and sees a sample. A witness collapsing back into the primary shows up as agreement drifting up toward the same-model band, so that signal was sitting in your traffic for however many weeks the genuinely different witness had stopped being chosen.
The defect is workload. Easier traffic raises agreement too, and a raw rate alarm moves the same way for both causes while saying nothing about which one moved.
Freezing a small canary set and replaying it on a schedule takes the confound out. The input distribution is held constant, so the only thing left that can move agreement on that set is the endpoints. No ground-truth labels are needed, because the measurement is agreement drift and correctness never enters it.
Then choose the canary items deliberately, from cases where the two endpoints historically disagreed. Those carry the discriminating power. If items that used to split stop splitting, that is attribution rather than an alarm. Your sampled escalation will tell you the leak rate got worse. The discriminating canary tells you why.
Under all of it, a model name is a pointer. Nothing in the request or the response bound the provider to what was actually served, which is how three spellings resolved to one model with no error surfacing anywhere. Everything downstream, including the sampled escalation you are adding, is inference about what you were served. The canary does not repair that. It notices when the unverifiable assumption has already broken, which is the weaker position and the one available.
You are right that we already compute it, and that is the part that stings. Agreement falls out of every request that reaches the gate, across the whole population, at no cost. The sampled escalation I was adding spends budget to see a slice of the same thing. A witness collapsing into the primary would have shown up as agreement drifting toward the same model band, and that drift was sitting in our traffic for the entire period the genuinely different witness had stopped being chosen. Nobody looked, because we were treating agreement as a gate to act on rather than a time series to watch.
On the confound, I accept it, and the canary is the better instrument for the why. A raw rate cannot separate an easier workload from a collapsed pair. Freezing the input distribution leaves the endpoints as the only thing that can move the number. Choosing the frozen items from cases where the two endpoints historically disagreed is the part I had not considered, and it is the part that converts an alarm into an attribution. Items that used to split and stop splitting says which side moved. A rate that moved does not. So the two instruments answer different questions and we want both.
Your last point is the one I cannot argue with. A model name is a pointer, nothing in the exchange binds the provider to what was served, and every measurement downstream of that, including the audit I am adding, is inference about what we were given. I am not going to pretend the canary repairs it.
One thing I think it can do that is stronger than noticing drift. At temperature zero on a frozen input, two endpoints serving the same underlying model do not merely agree on the answer, they emit close to the same text. Agreement is a semantic comparison and it gave us 88 percent against 70 to 75. Byte level similarity on a fixed prompt should separate those two cases much harder, because two different models rarely phrase an identical answer identically, while one model asked twice usually does. If that holds, the canary stops being a drift alarm and becomes a same model detector, and it would have fired on the first scheduled replay instead of after enough traffic to move a trend.
That is a claim and not a result. I have not measured it yet. It is cheap to falsify, so I will, and I will post the number either way.
Byte similarity is the right next probe, but as a detector it will not work yet, because it has no reference line under it. Temperature zero does not buy determinism. Greedy decoding still moves at the serving layer: batch composition changes the order of floating point reductions, and a mixture-of-experts router will pick differently depending on which other requests happen to share the batch. Kernel selection and speculative decoding paths shift with load too. So "one model asked twice usually emits the same text" is a measurement about your provider's serving stack. Temperature does not guarantee it.
The first run should be one endpoint against itself, same frozen prompt, repeated across a day at different load. That gives you the self-similarity distribution including its ugly tail, and every cross-endpoint number has to be read against it. If self-similarity sits at 0.93 with a long low tail, a cross-endpoint 0.90 identifies nothing. It might just mean the platform was busy that hour.
There is a second trap at the other end. Separation depends on output entropy. For short constrained answers, an arithmetic result or a single label, genuinely different models return byte-identical text routinely. Both distributions pile up near 1.00, so the byte detector is weakest on exactly the shape of the original 0.00 probe. The canary items need entropy in them. Prose and code leave phrasing room to diverge.
The bigger gain is that the self-series is probably the stronger instrument of the two. Cross-endpoint similarity says two names look like one model. Self-similarity over time on a single endpoint, frozen prompt bytes, says one name became a different model. A step change in that trace is a silent swap under a stable name. Same class of failure you already hit, running the other way, and agreement-rate watching cannot see it when both endpoints move together.
The honest limit stays where it was. Both readings are still your inference about your provider, and nothing here binds the provider to anything. What changes is who is able to check. Publish the replay record as prompt bytes, timestamp, endpoint identifier, returned bytes, with a hash over the whole thing, and the 88-versus-70 number and whatever the byte-similarity number turns out to be stop being private figures. Someone with no relationship to you can reissue the same prompt and see whether it reproduces. The provider is still unbound. Your claim about the provider becomes falsifiable by a stranger.
I measured it before your reply landed, and your reply predicts the result. Posting the number as promised, because it goes against me.
Live setup rather than a fixture. Our three cheap slots today are slot 1 gpt-oss-120b at one provider, slot 3 the same gpt-oss-120b at a different provider under a different name, and slot 2 llama-3.3-70b. So the failure I wanted to detect, one model wearing two endpoint names, is in production right now, and slot 2 is a true negative. Twelve free-form prompts, temperature zero, four arms. The fourth arm is the reference line you are describing: the same endpoint asked the identical question twice.
Byte similarity, difflib on normalized text:
self against itself 0.305, same model different provider 0.182, different model 0.072.
The reference line is the whole result. One endpoint asked the same question twice scores 0.305 against itself, so my premise failed a level below my conclusion. Two providers were never going to emit near-identical text when one provider does not emit near-identical text to itself. Without that arm I would have read 0.182 as "different models" and concluded the exact inverse of the truth.
It also does not beat agreement. Separation was +0.110 for byte similarity against +0.145 for a semantic word-overlap proxy, and no metric separates per item. Every distribution overlaps. Aggregate or nothing.
On your load point, I have accidental evidence for it. An earlier partial run of the identical script put self-similarity at 0.208 and same-model at 0.196. The full run an hour later gave 0.305 and 0.182. Same prompts, same endpoints, same temperature. That is your batch composition and kernel selection moving under me, and it means my single number is a point estimate on a distribution I have not characterised. A run across a day at varying load is the right first experiment and I had it in the wrong order.
Your entropy trap is the one I would have walked into next. My prompts were deliberately several sentences of prose, so I never saw the ceiling effect, and you are right that it bites hardest on exactly the short arithmetic shape the original 0.00 probe used. The detector is weakest where that claim was made.
The self-series being the stronger instrument is the part I did not have and it is better than either of ours. Cross-endpoint similarity answers "are these two names one model". Self-similarity over time on one endpoint answers "did this one name quietly become a different model", and a step change in that trace is a silent swap. Agreement watching is blind to it when both endpoints move together, which is the version of our failure we have not been looking for at all.
One fragment of mine that might survive into that design. I also scored similarity on the first 120 characters alone, and it separated best of anything I tried: self 0.810, same model 0.702, different model 0.481. Divergence accumulates token by token, so identity lives at the start of an answer and is gone by the end. Comparing whole answers throws that away. For a self-series trace the prefix is probably the sharper signal and it is cheaper to store.
On publishing the replay record, I have no argument against it and it is the part that actually changes anything. Prompt bytes, timestamp, endpoint identifier, returned bytes, hashed. I will not pretend it binds the provider. It moves the claim from private to falsifiable by a stranger, which is the only honest position available and better than the one we have.
Caveats so you have them rather than find them: n is 12, no significance test, the agreement column is a proxy and not our production comparator, and one prompt family.
Three for three. You have now improved this system more than our last two internal reviews did.
We actually have to deal with this issue, we monitor how much we use AI by counting the raw tokens and set a budget for each month, with warnings when we're getting close to the limit. But if our model does a lot of work behind the scenes before giving an answer, like it did here, the real cost of each feature might be a lot higher than we think. It's worth looking into: compare the number of tokens we're charged for to how many of those are actually useful and shown to users. If there's a big difference, we're probably underestimating the true cost and should start tracking "useful tokens" instead of just the total. This way, we can get a better idea of how much we're really spending.
You are describing exactly the gap the piece is about, and I have numbers on it from this week that will probably surprise you in the direction that matters for your budget.
We compared a cheap reasoning tier against a frontier model on identical prompts. The cheap one emitted 244 output tokens on average against the frontier's 83, and it was longer on 10 of 10, with both answers correct. So the model you are billed least for per token can be the one consuming the most of them, and a monthly budget built on the assumption that cheap equals fewer tokens will drift in the wrong direction quietly.
The part that matters for your "useful tokens" idea is where the invisible tokens actually come from, because it decides whether you can do anything about them.
We tried the obvious lever first and it failed. A terseness instruction in the prompt recovered 3 percent, and 4 of the 10 responses got LONGER. That is because these are reasoning tokens, not verbosity: the model is thinking before answering, and asking it to be brief does not persuade it to think less. Unpromptable by construction.
The lever that worked is a provider parameter. Setting reasoning effort to low on one of our slots took it from 52.0 output tokens to 16.4, a 68 percent reduction, non-overlapping ranges across 8 samples per arm. So the ratio you want to track is not really per-feature, it is per-slot-configuration, because a single config field moved two thirds of the spend without touching a line of application code.
Two traps in doing this, both of which cost us real money before we noticed:
One. The parameter shape is not portable. One provider takes a flat
reasoning_effort, another takes it nested under areasoningobject. Send the wrong shape and it is accepted with a 200 and silently ignored, so your config reads as done and you are billed at full price. We had an empty config field sitting on a reasoning model for days for exactly this reason. Verify by measuring output tokens before and after, never by reading the config back.Two. Do not estimate this in characters. We measured characters falling 36 percent while tokens fell 17 percent on the same change, so a character-based proxy overstated the saving by about 2x.
On your actual proposal, tracking useful against total: I would do it, and I would define useful as tokens present in the response you show the user, with everything else in one bucket rather than trying to attribute it. The absolute ratio is less informative than its movement, because it changes when a provider swaps what sits behind a model name, and you find out from that ratio or you do not find out at all.
That "accepted with a 200 but silently ignored" trap is the scariest part, you'd never catch that from logs alone, only by actually measuring output tokens. We've been eyeballing character counts as a rough proxy, so that 36% vs 17% mismatch is a good wake-up call to stop doing that.
Going to start tracking output tokens per slot before/after any config change instead of trusting the response code. Appreciate you sharing real numbers, way more useful than the theory I was working from.
Character counts as a proxy will hold until they suddenly do not, and the failure is quiet, so tracking real output tokens per slot is the right move.
One trap worth knowing before you build it, because we shipped it ourselves last week. Our cost meter went in, and a provider that returns no cost field left the meter at zero, so those requests recorded as $0.0000. Free and not measured looked identical in the table, and zero reads as the best number in any comparison, so nothing flagged it.
So: unknown should be NULL, never 0, and record the reason beside it. Three states rather than two. No call made is genuinely zero. Every call priced is a real sum. Any call unpriced makes the whole request unknown, because a partial total is missing money, not a cheaper request.
We are two days into having that meter running properly and it has already corrected two numbers I had said out loud.
That's a sneaky one, zero looking better than every real number is exactly the kind of bug that hides in plain sight. Good instinct making unknown its own state instead of quietly rounding it to free.
Appreciate you sharing what broke on your end too, saves everyone else from finding it the hard way.
Since that exchange we walked into the next trap on exactly the path you described, so it is worth passing on before you build it.
Tracking output tokens per slot before and after a config change is the right instinct. What bit us is that a before/after on a small sample is mostly noise, and it does not look like noise at the time.
The flag was never real, and the rollback was itself an unverified change to production, which is the part that still bothers me.
The cheap fix is one extra run before you trust any comparison: run the identical thing twice with nothing changed at all, and count how many individual items give a different answer. That percentage is your noise floor. On a fixed 800 item benchmark ours came out at 2.0 to 2.9 percent, so any gap smaller than roughly twice that is not a finding, it is the same measurement twice.
Worth doing on your per-slot token counts before the first config change rather than after, because the floor is what tells you which deltas mean anything.
Eye-opening breakdown on reasoning tokens! When benchmarking LLM costs across providers, do you have a standard script or test set you run to estimate 'useful output' vs hidden reasoning bloat before committing to an API?
Thanks. Honest answer: no polished script I can hand you, but the method is four lines of arithmetic and it is the part that matters, so here it is.
Do not benchmark. Run your OWN workload shape against the candidate, and for each call read both the total completion tokens and the reasoning tokens. Most OpenAI compatible providers expose the second as usage.completion_tokens_details.reasoning_tokens. Then compute the only number that means anything:
effective output price = sticker output price times (total completion tokens divided by useful completion tokens)
That is it. The useful count is total minus reasoning.
Here is the measurement from the post, which is the one that stopped us swapping our cheap tier. Candidate advertised 0.05 dollars in and 0.20 out per million against the model we were already running at 0.07 and 0.27, so on paper about a 30 percent saving on both sides. On a real tool call it returned 274 prompt and 132 completion tokens, and 123 of those 132 were reasoning. Nine were useful. Effective output price 2.93 dollars per million, which is 14.7 times sticker and roughly 10 times more expensive than the tier it appeared to undercut.
Three traps I would flag, because the arithmetic is easy and the setup is where it goes wrong.
The ratio is a property of your WORKLOAD, not of the model. A reasoning model on a short tool call is almost all overhead, because the thinking does not shrink to match a nine token answer. Give the same model a long analytical prose task and the ratio gets far more reasonable. So measuring on a benchmark and deploying on tool calls will mislead you in the expensive direction.
Some providers do not return reasoning tokens in usage at all. You then cannot compute this, and I would treat that as information rather than as an inconvenience. If you cannot see what you are billed for, price the risk in.
And that figure above is n equals 1. It was decisive for us because the gap was an order of magnitude and not a few percent, but do not take a single call as a rate. Run your own spread and look at the distribution, because the tail is where a per request budget actually breaks.
The general version, which is the thing I would put on a wall: tokens are the unit you are billed in, not the unit you receive. Price per useful token is the only comparison that survives contact with a reasoning model.
"Tokens are the unit you are billed in, not the unit you receive"—truly a quote for the ages!
Thanks for laying out the math so clearly, Tom. Looking at price per useful token completely shifts how we need to structure our evaluation runs for tool-calling workloads.
Thank you, that is kind.
One thing I would add for tool calling specifically, since that is the shape you are about to measure. Put a trap subset in the run and score it as its own column, never averaged into accuracy. By trap I mean requests where the correct behaviour is to decline, because the right tool is not available or the request is malformed. In the 800 task field in the post, the model that scored 91 on plain accuracy handled 52 percent of those, and a model that scored 93 handled 90 percent. That gap is completely invisible in an accuracy average, and it is the number that decides whether you can hand the thing real permissions.
And on the cost side, budget at p90 rather than the mean. The reasoning ratio has a tail, and a per request budget breaks at the tail. The mean is how you go broke, p90 is how you stay honest.
That trap-subset idea is brilliant—averaging refusal accuracy into overall accuracy definitely hides model hallucination risks when permissions are on the line. And "budget at p90" is getting printed and stuck to my monitor. Thanks for the extra insights, Tom!
One more for the monitor, and it is new since I wrote that, because it caught us out yesterday.
When you run those provider comparisons, the reasoning bloat may not be something you can prompt away. We asked our cheap tier and a frontier model the same ten short questions. The cheap one used three times the output tokens while both got every answer right, and it was not padding: it answers "Lisbon", bare, and still spends 17 tokens. An explicit "bare value only, no preamble" system prompt recovered 3 percent, and made four of the ten longer, because the instruction is itself input and reasoning tokens are generated regardless.
The lever is a provider parameter rather than a prompt. Setting a reasoning cap on one tier that had none took its output from a mean of 52 tokens to 16, a 68 percent cut, measured over eight runs per arm.
Two traps in doing that, both of which cost us time. The parameter name is not portable: Together honours a flat reasoning_effort, and the nested shape another provider uses is accepted and then silently ignored, so it reads as configured and still bills full price. And temperature zero is not deterministic on these endpoints. The uncapped arm returned 36, 50 or 60 tokens for the same prompt, so a single before-and-after call is noise. I nearly changed production config on one draw before repeating it.
So the practical version of price per useful token: check whether your candidate exposes a reasoning cap at all, confirm it is actually honoured by watching the token count move rather than by reading the docs, and repeat the measurement enough times to see the spread.