The default way to use an LLM is to pick one and send it everything. That is what the integration
looks like, it is what a model marketplace is shaped around, and it is what most agents I read about
do. Pick the good one. Send it the work.
We run a routing and verification gateway, so we have had to measure a lot of models against the same
tasks with money attached. I want to put down what our own data keeps saying, including the parts
where it makes us look bad, because the conclusion is not "we found the better model."
We ran the experiment everyone runs to pick a model. We ran it twice. We got a different winner.
Three models, 164 HumanEval+ problems, temperature 0, one completion each. Then the identical thing
again an hour later. Nothing changed between the two runs except time.
| model | run 1 | run 2 | rank |
|---|---|---|---|
| qwen3-235b | 89.6% | 92.1% | 3 -> 1 |
| gpt-oss-120b | 91.5% | 90.2% | 1 -> 3 |
| qwen3.8-max | 91.2% | 91.6% | 2 -> 2 |
Each model is scored over the problems it actually answered. Two of them answered all 164 in both
runs. qwen3.8-max timed out on 4 and then on 9, so its denominator is 160 and then 155. The partition
table further down uses the smaller set where all three answered, so its percentages are not directly
comparable to these.
First and last swapped. The model we would have picked on Tuesday is the one we would have rejected on
Wednesday, and the new frontier model that lost to our cheap tier by 1.3 points beat it by 1.4 the
second time.
Temperature 0 is not determinism, and 164 problems is not enough to separate models sitting two points
apart. We can put a number on how much of this is flicker: run the same model over the same problem
twice and it changes its answer 3.7 to 5.2 percent of the time. Any ranking gap smaller than that is
not a result. Ours were all smaller than that.
But something did survive both runs, and it is the more useful thing.
Stop asking how many each model got and ask which ones.
| run 1 | run 2 | |
|---|---|---|
| all three correct | 130 | 130 |
| all three wrong | 2 | 2 |
| they DISAGREE | 28 (17.5%) | 23 (14.8%) |
Roughly 15 to 18 percent of the time, the model you picked decided whether you got a right answer.
Note what the same table says about the other 82 percent: all three models agree, 130 of them right
and 2 of them wrong together, and the choice is irrelevant. This is a claim about a fifth of the work, not about all of it.
The reason to believe the fifth is real rather than flicker is not the size of the number, it is that
it lands on the same problems both times. Twenty four problems were contested in the first run and
twenty three in the second, and seventeen of them are the same problems. If the disagreement were
noise you would expect about four to coincide. Ten of them are contested in both runs without a single
model ever changing its own answer.
The ceiling is the part I would think hardest about. The best single model scores about 92 percent.
If you could always take whichever of the three happened to be right, you would score 98.8 percent,
and that number was stable across both runs too. Picking one model, even correctly, leaves about six
points on the table.
A leaderboard cannot show you any of this. A leaderboard is column totals, and all of it lives in the
rows.
Capability is shaped like the task, not like a ranking
Our verification witness is llama-3.3-70b, pinned to one provider. On 400 standard tool-calling tasks
it scores 382, which is 95.5 percent, and matches our primary path. On an adversarial set built to
tempt it into calling a tool when it should decline, it scores 30 of 40.
I have to be careful with that 40, and I only learned why today. It is four distinct trap shapes,
each parameterised ten ways. The replicates swap a city or a number; they do not add a fifth kind of
trap. So the honest denominator is four, not forty, and I am not going to put a confidence interval on
it. What the instrument legitimately supports is: across four adversarial tool-selection shapes,
parameterised ten ways each, it failed ten times.
The interesting part is not the 75 percent, which I now think is close to meaningless. It is that
all ten misses are the same verdict. Every one is a false call: it invoked a tool where the correct
behaviour was to abstain. So the model matches the primary at doing the thing and fails it at knowing
when not to. Those are different
abilities, they are not correlated the way a single score implies, and only one of them was on the
benchmark.
If you are building on a model because it scored well at a task, check whether the thing you actually
need is the task or the restraint.
The price on the page is not the price
We screened a cheaper candidate for our cheap tier. Advertised at 0.05 and 0.20 per million against
our shipped tier at 0.07 and 0.27, roughly 30 percent cheaper on both sides.
On a real tool call it emitted 132 completion tokens, of which 123 were reasoning tokens that we paid
for and never saw. Its effective output price came out around fifteen times its sticker, making it
roughly ten times more expensive than the tier it appeared to undercut. The tool call itself was
correct. The capability was real. Only the economics were inverted.
This is one call, so treat it as a mechanism rather than a rate. The mechanism is enough: compare cost
per successful result, never cost per million tokens, and read the reasoning token count before you
believe any price comparison.
You may not be choosing a model at all
Same model, same 400 prompts, same scorer. Pinned to a single serving provider it scored 95.5 percent.
Unpinned, letting the marketplace route it, 78.2 percent. The model was held constant and the score
moved 17.3 points.
For scale, adjacent models in our own sweep differ by 0.5 to 6 points, so the routing spread was
larger than the entire model spread we were trying to measure.
And then we checked whether that was a law, because it would be a convenient one for us. We ran the
same pinned-versus-unpinned experiment on gpt-oss-120b. It moved 0.6 points, 65.2 percent pinned
against 65.8 unpinned. So this is not "marketplaces degrade models." It is a property of the specific
model and provider pair, and the actionable version is that you have to check yours rather than assume
either way.
The newest one is not automatically the answer either
Qwen 3.8 came out yesterday. We benchmarked it the same night against the two models it might replace.
And this is where I have to be careful, because the first version of this section said "it did not
beat our cheap tier" and the second run says it did. On accuracy it is indistinguishable, in both
directions, and I am not going to pretend otherwise in either.
What did not move between runs is the part that actually decides it. It cost 64 to 76 times more
per solved problem, depending on the run, priced at the marketplace list rate rather than at what
the vendor's own API would have charged us, which we did not read. It was slower at the median in both. And its timeouts
more than doubled, 4 then 9 out of 164, with a worst case over 200 seconds. The cost multiple moves
because it is a ratio of two noisy things, so treat it as a band and not a constant, which is the same
rule this piece applies to everything else.
So the reason not to adopt it is not that it is worse at the task. It is that it is indistinguishable
at the task, dramatically more expensive, and less reliable, and those three are all stable across
runs while the score is not. The score was the only thing I originally judged it on, and the score
was the only thing that turned out to be noise.
So what do you do instead
If models differ by task rather than rank, then "which model is best" is optimising the wrong thing.
The unit we route on is the task shape plus a check: send the request to whatever suits its shape,
serve the cheap answer when something independent agrees it is right, and escalate only when the check
disagrees. The model becomes a swappable component, and being wrong about one becomes survivable.
And the uncomfortable part, because leaving it out would make this an advertisement. Our own
escalation target never cleared the cheap tier it exists to rescue on code, in either run. On the
first run it scored below it, 90.0 against 92.5. On the second the two tied at 92.9. By this piece's
own rule I cannot report either of those as a ranking, so the honest version is the weaker and worse
one: we have no evidence that the model we escalate to is better at code than the model it is
rescuing. That is not what an escalation target is for. A gate can only be as good as the thing it
routes to, so the thing we are fixing is not the gate, it is the ladder, and last night's benchmark
was us checking whether the new frontier model was the fix. It was not.
That is scoped to code, and I should say so rather than let it sound general. This benchmark did not
test the tool path at all, and our own data has the ordering of these tiers inverting by task shape.
Which is the whole argument again, made against us this time: the rung that looks wrong on one task
shape is not automatically the wrong rung.
Caveats, in full
The 98.8 percent ceiling assumes an oracle that always picks the model that happened to be right. No
router has that, ours included, and we have not measured what share of those six points our own
routing actually captures. Read it as the size of the prize, not as a result.
Single run per arm on the 160 problem comparison, temperature 0, one completion each. Temperature 0
was not deterministic for us: two runs of the same 8 problems gave different results, so treat
small gaps as noise, including ours. The 95.5 versus 78.2 pair is 400 per arm, single run, no
confidence intervals. The trap set is four shapes replicated ten times, not forty independent cases, so treat it as four
observations and ignore any interval anyone computes on n=40, including one of ours that is now
struck. The pricing finding is
one call. Our benchmarks are code and tool calling and they are not your workload.
None of this says frontier models are overrated. We escalate to one and would not ship without it. It
says that after measuring a lot of them against the same tasks, the most useful thing we learned was
not which one wins. It was that they fail in different places, and the difference between them is
bigger than the gap in their scores.
Top comments (0)