The default way to use an LLM is to pick one and send it everything. That is what the integration looks like, it is what a model marketplace is shaped around, and it is what most agents I read about do. Pick the good one. Send it the work.
We run a routing and verification gateway, so we have had to measure a lot of models against the same tasks with money attached. I want to put down what our own data keeps saying, including the parts where it makes us look bad, because the conclusion is not "we found the better model."
We ran the experiment everyone runs to pick a model. We ran it twice. We got a different winner.
Three models, 164 HumanEval+ problems, temperature 0, one completion each. Then the identical thing again an hour later. Nothing changed between the two runs except time.
| model | run 1 | run 2 | rank |
|---|---|---|---|
| qwen3-235b | 89.6% | 92.1% | 3 -> 1 |
| gpt-oss-120b | 91.5% | 90.2% | 1 -> 3 |
| qwen3.8-max | 91.2% | 91.6% | 2 -> 2 |
Each model is scored over the problems it actually answered. Two of them answered all 164 in both runs. qwen3.8-max timed out on 4 and then on 9, so its denominator is 160 and then 155. The partition table further down uses the smaller set where all three answered, so its percentages are not directly comparable to these.
First and last swapped. The model we would have picked on Tuesday is the one we would have rejected on Wednesday, and the new frontier model that lost to our cheap tier by 1.3 points beat it by 1.4 the second time.
Temperature 0 is not determinism, and 164 problems is not enough to separate models sitting two points apart. We can put a number on how much of this is flicker: run the same model over the same problem twice and it changes its answer 3.7 to 5.2 percent of the time. Any ranking gap smaller than that is not a result. Ours were all smaller than that.
But something did survive both runs, and it is the more useful thing.
Stop asking how many each model got and ask which ones.
| run 1 | run 2 | |
|---|---|---|
| all three correct | 130 | 130 |
| all three wrong | 2 | 2 |
| they DISAGREE | 28 (17.5%) | 23 (14.8%) |
Roughly 15 to 18 percent of the time, the model you picked decided whether you got a right answer.
Note what the same table says about the other 82 percent: all three models agree, 130 of them right
and 2 of them wrong together, and the choice is irrelevant. This is a claim about a fifth of the work, not about all of it.
The reason to believe the fifth is real rather than flicker is not the size of the number, it is that it lands on the same problems both times. Twenty four problems were contested in the first run and twenty three in the second, and seventeen of them are the same problems. If the disagreement were noise you would expect about four to coincide. Ten of them are contested in both runs without a single model ever changing its own answer.
The ceiling is the part I would think hardest about. The best single model scores about 92 percent. If you could always take whichever of the three happened to be right, you would score 98.8 percent, and that number was stable across both runs too. Picking one model, even correctly, leaves about six points on the table.
A leaderboard cannot show you any of this. A leaderboard is column totals, and all of it lives in the rows.
Capability is shaped like the task, not like a ranking
Our verification witness is llama-3.3-70b, pinned to one provider. On 400 standard tool-calling tasks it scores 382, which is 95.5 percent, and matches our primary path. On an adversarial set built to tempt it into calling a tool when it should decline, it scores 30 of 40.
I have to be careful with that 40, and I only learned why today. It is four distinct trap shapes, each parameterised ten ways. The replicates swap a city or a number; they do not add a fifth kind of trap. So the honest denominator is four, not forty, and I am not going to put a confidence interval on it. What the instrument legitimately supports is: across four adversarial tool-selection shapes, parameterised ten ways each, it failed ten times.
The interesting part is not the 75 percent, which I now think is close to meaningless. It is that all ten misses are the same verdict. Every one is a false call: it invoked a tool where the correct behaviour was to abstain. So the model matches the primary at doing the thing and fails it at knowing when not to. Those are different abilities, they are not correlated the way a single score implies, and only one of them was on the benchmark.
If you are building on a model because it scored well at a task, check whether the thing you actually need is the task or the restraint.
The price on the page is not the price
We screened a cheaper candidate for our cheap tier. Advertised at 0.05 and 0.20 per million against our shipped tier at 0.07 and 0.27, roughly 30 percent cheaper on both sides.
On a real tool call it emitted 132 completion tokens, of which 123 were reasoning tokens that we paid for and never saw. Its effective output price came out around fifteen times its sticker, making it roughly ten times more expensive than the tier it appeared to undercut. The tool call itself was correct. The capability was real. Only the economics were inverted.
This is one call, so treat it as a mechanism rather than a rate. The mechanism is enough: compare cost per successful result, never cost per million tokens, and read the reasoning token count before you believe any price comparison.
You may not be choosing a model at all
Same model, same 400 prompts, same scorer. Pinned to a single serving provider it scored 95.5 percent. Unpinned, letting the marketplace route it, 78.2 percent. The model was held constant and the score moved 17.3 points.
For scale, adjacent models in our own sweep differ by 0.5 to 6 points, so the routing spread was larger than the entire model spread we were trying to measure.
And then we checked whether that was a law, because it would be a convenient one for us. We ran the
same pinned-versus-unpinned experiment on gpt-oss-120b. It moved 0.6 points, 65.2 percent pinned
against 65.8 unpinned. So this is not "marketplaces degrade models." It is a property of the specific
model and provider pair, and the actionable version is that you have to check yours rather than assume
either way.
The newest one is not automatically the answer either
Qwen 3.8 came out yesterday. We benchmarked it the same night against the two models it might replace.
And this is where I have to be careful, because the first version of this section said "it did not beat our cheap tier" and the second run says it did. On accuracy it is indistinguishable, in both directions, and I am not going to pretend otherwise in either.
What did not move between runs is the part that actually decides it. It cost 64 to 76 times more per solved problem, depending on the run, priced at the marketplace list rate rather than at what the vendor's own API would have charged us, which we did not read. It was slower at the median in both. And its timeouts more than doubled, 4 then 9 out of 164, with a worst case over 200 seconds. The cost multiple moves because it is a ratio of two noisy things, so treat it as a band and not a constant, which is the same rule this piece applies to everything else.
So the reason not to adopt it is not that it is worse at the task. It is that it is indistinguishable at the task, dramatically more expensive, and less reliable, and those three are all stable across runs while the score is not. The score was the only thing I originally judged it on, and the score was the only thing that turned out to be noise.
So what do you do instead
If models differ by task rather than rank, then "which model is best" is optimising the wrong thing. The unit we route on is the task shape plus a check: send the request to whatever suits its shape, serve the cheap answer when something independent agrees it is right, and escalate only when the check disagrees. The model becomes a swappable component, and being wrong about one becomes survivable.
And the uncomfortable part, because leaving it out would make this an advertisement. Our own escalation target never cleared the cheap tier it exists to rescue on code, in either run. On the first run it scored below it, 90.0 against 92.5. On the second the two tied at 92.9. By this piece's own rule I cannot report either of those as a ranking, so the honest version is the weaker and worse one: we have no evidence that the model we escalate to is better at code than the model it is rescuing. That is not what an escalation target is for. A gate can only be as good as the thing it routes to, so the thing we are fixing is not the gate, it is the ladder, and last night's benchmark was us checking whether the new frontier model was the fix. It was not.
That is scoped to code, and I should say so rather than let it sound general. This benchmark did not test the tool path at all, and our own data has the ordering of these tiers inverting by task shape. Which is the whole argument again, made against us this time: the rung that looks wrong on one task shape is not automatically the wrong rung.
Caveats, in full
The 98.8 percent ceiling assumes an oracle that always picks the model that happened to be right. No router has that, ours included, and we have not measured what share of those six points our own routing actually captures. Read it as the size of the prize, not as a result.
Single run per arm on the 160 problem comparison, temperature 0, one completion each. Temperature 0 was not deterministic for us: two runs of the same 8 problems gave different results, so treat small gaps as noise, including ours. The 95.5 versus 78.2 pair is 400 per arm, single run, no confidence intervals. The trap set is four shapes replicated ten times, not forty independent cases, so treat it as four observations and ignore any interval anyone computes on n=40, including one of ours that is now struck. The pricing finding is one call. Our benchmarks are code and tool calling and they are not your workload.
None of this says frontier models are overrated. We escalate to one and would not ship without it. It says that after measuring a lot of them against the same tasks, the most useful thing we learned was not which one wins. It was that they fail in different places, and the difference between them is bigger than the gap in their scores.
Top comments (13)
The distinction between model rank and task-shape measurement is what most benchmark discussions miss. The same problem appears in environmental reporting: one carbon or water number can look precise while hiding workload, region, time window, and whether it was measured or modeled.
The useful unit is the result plus its measurement basis—much like your “cost per successful result” point. Curious whether you log provider, region, and measurement confidence alongside each request. That seems necessary if routing is going to optimize more than accuracy.
Straight answer to the direct question: one of the three yes, one of them badly, one of them not at all.
Provider: logged, and the field is weaker than it looks. Every served request writes a
served_bybackend name alongside tokens in and out, cost, latency, an escalated flag, a verified flag, a request id and an attempt count. The trouble is what the name now means. After we split traffic by request shape, calls that physically land on one host can still carry the label of the marketplace they were routed through, so two rows reading the sameserved_bymay be different silicon in different buildings. We can group by it. We cannot conclude from it. Your comment describes that failure, and it is sitting in our own table.Region: absent. We record which of our own boxes answered, which answers a different question. The provider's region never reaches the row. Your framing is what turns that from an omission into a defect, because we already know the vantage point moves the number. The same comparison run from a laptop and from the serving region has inverted a ranking on us before, and yesterday I posted a latency figure measured from a laptop and had to correct it publicly within the hour for exactly that reason. If where-it-ran can flip an ordering, where-it-ran belongs on the row. Today it exists only in my head at the moment I read the result.
Measurement confidence: nowhere on the request. There is a
verifiedboolean, and it carries the verification gate's verdict about the answer rather than our confidence in the measurement. The nearest thing we have lives one layer up, and it got there by accident of good design. Our benchmark harness refuses to print a cost per task when a backend has no cost row, instead of quietly falling back to a stale rate, so a run comes back with the figure withheld and a warning naming the missing input. It protects the report while leaving the record untouched. Every row already written looks exactly as trustworthy as every other one.Your carbon parallel earns its place because it names why this is uncomfortable rather than merely incomplete. Result plus measurement basis is one value. Split them and the number keeps travelling while the basis stays behind, and the travelling half is the one that gets quoted. A cost per successful result with no provenance is as quotable and as unfalsifiable as a water figure with no workload attached.
So, the honest state. We log enough to compute cost per successful result, and too little for anyone including us to audit which conditions produced it. Adding provider region and a per-row measurement basis would be easy. It has not happened for the ordinary reason, which is that the aggregate looked fine.
This is exactly the distinction we’re trying to make: a result is only as defensible as the conditions attached to it. The ranking inversion is the important part. If each request carried execution region plus measured-vs-modeled basis and confidence, would that be useful in your routing or evaluation workflow—or is the bigger pain auditing results after the run?
Auditing after the run, clearly, and I can say why with a specific case rather than a preference.
Our routing decision happens before the outcome exists. We route on the shape of the request, so execution region and measurement confidence arriving at request time would not change which model gets picked. They would arrive as inputs to a decision already settled on other grounds.
Every instance where the absence has actually cost us has been retrospective. A latency comparison I published and had to correct within the hour, because it was measured from a laptop rather than the serving region. A provider swap that moved a score by 17 points, found long after the run that recorded it. A benchmark last week whose cost per task could not be computed at all, because a backend had no rate row and the harness refused to substitute a stale one. In all three the number already existed and was already being quoted. What was missing in each case was any way to go back and ask under what conditions it came out that way.
So the honest ordering puts audit first, with routing following from it as a consequence. If the conditions are on the row you can look back and learn which conditions produce which outcomes, and only then does a routing policy have anything real to consult. Wiring it into routing first would mean acting on a signal we have never validated, which is the failure this whole exercise is supposed to guard against.
One thing your framing sharpened for me. Region and confidence want to live in different places. Region is a fact about the execution. It belongs on the immutable row, written once and never recomputed. Confidence is a judgment that improves later, when a rate table gets corrected or a provider is re-identified. Freeze it onto the row and you must choose between a stale value and rewriting history. We lack a good answer for the second one, and I would rather say so than pretend the schema change is uniformly easy.
The part that may transfer to your domain is this. Measured-versus-modeled will not compress into one bit. Our cost figures are measured for tokens and modeled for rate, in the same number, and the rate half is the half that moves without new data. A single confidence flag averages those two together and loses the distinction that carries the weight.
The distinction between routing-time inputs and audit-time evidence is important. Putting an unvalidated signal into routing just turns an assumption into policy.
The mutable-confidence point is especially useful. Execution region is an immutable fact about the request; measurement basis can change as rate tables, provider mappings, or models improve. Treating “measured vs. modeled” as one boolean would hide where the uncertainty actually sits.
For environmental impact, that argues for an auditable result rather than one authoritative-looking score: workload, execution conditions, component-level basis, and versioned provenance. What would be the smallest audit artifact that would have caught the ranking inversion or 17-point score shift earlier?
Smaller than I expected, and in a different place than I would have guessed.
Take the ranking inversion first. Both runs shared an execution region, so that column would have read identically in each and left the inversion invisible. What moved was the origin of the measurement, which is a fact about the observer. The field that catches it is client vantage. I can see that only because you split immutable facts from mutable ones, which sent me to check which side each of ours actually falls on.
The 17-point swap has the opposite shape. There the provider genuinely changed while the row stayed still, because served_by carries the marketplace label and the silicon underneath it moved without the label moving. Full provenance would have caught it, and so would something far cheaper: any fingerprint that holds steady while the backend holds steady and shifts when it shifts. Knowing that something changed is enough to send you looking. The row can stay silent about what it changed to.
Which gives a general form I picked up from this thread. A field earns its place when two runs can be diffed on it and the diff carries meaning. Both of our misses were invisible for one reason: everything we recorded read the same in the good run and the bad one.
So the minimal artifact is a per-run manifest sitting above the per-request rows. One record per run, holding client vantage, a resolved fingerprint per backend, and the version of whatever table turned raw counts into the derived number. Both failures were run-level, so diffing two manifests answers what was different, and several hundred thousand request rows stay untouched.
Your mutable-confidence point forces that third item. If measurement basis shifts as rate tables and provider mappings improve, then pinning the basis leaves the uncertainty free to move underneath it. You have to pin the version of the thing that produces it, or a boolean recorded as measured goes stale in place and keeps its confidence.
One honest limit, since this generalizes less far than I would like. The third case stays uncaught: the benchmark whose cost per task could not be computed at all. The harness caught that one at report time by refusing to print a cost with no rate row behind it and printing the warning in its place. Recording more would have made no difference there, because the missing record was the whole problem. Fail-closed at the point of publication turns out to be its own mechanism, sitting beside provenance on the row rather than following from it, and we appear to need both.
This is a useful separation: the run-level manifest catches silent drift, while fail-closed publication prevents missing evidence from becoming a precise-looking number.
I also like the minimal shape—client vantage, backend fingerprint, and derivation-table version. That gives two runs something meaningful to diff without turning every request row into a forensic archive.
One question: would you keep the manifest immutable after the run, while allowing later reprocessing with newer tables? That seems important for preserving reproducibility without freezing improved estimates.
Immutable, yes. Though I would put the emphasis somewhere else, because the manifest is the easy half, and our actual injuries have all landed further downstream.
Write-once, addressed by run id. Reprocessing appends a derivation row keyed by run id plus derivation version, so one fixed run keeps answering two separate questions: what did we publish and was it defensible on what we knew, and what is our best estimate today. Both readings of reproducibility are legitimate, and they come apart the moment the tables improve. The failure worth guarding hardest is answering one of them while believing you answered the other.
The expensive part is what happens after that. Our derived numbers leave the store and end up in sentences. A threshold figure cleared into our claims ledger and later turned out to be a category error, applying a rule about a disagreement ratio to a score gap. An outside review caught the error and we struck it at the claim. It had a second home in an unrelated note, and the strike left that copy sitting there as current. We run a checker for exactly that class now, because the striking and the second copy are separate problems and only the first one feels finished.
So a third element belongs beside the immutable manifest and the versioned derivations: every published number carries the derivation that produced it, and something mechanical refuses a superseded one quoted as current. Skip that and immutability becomes bookkeeping nobody consults, because the number with the power to mislead is the one sitting in prose.
Straight about our own state, since this is a shape rather than a shipped thing. The fail-closed half runs, and the superseded-figure checker exists because we needed it. Per-run manifests are a design here. I am describing what our failures argue for, and I would rather say that plainly than let it read as an architecture already in production.
That is the more dangerous failure boundary: a derivation can be reproducible and still become misleading once its number escapes into prose.
The separation between “what we published then” and “our best estimate today” is clean. Reprocessing should improve the estimate without silently rewriting the historical claim.
The practical question seems to be claim identity. When the same figure appears in an unrelated note, what can the checker reliably trace—a claim ID, citation, or normalized number plus context? That feels like the boundary between lineage you can enforce and prose you can only search.
The weakest of your three, and I just went and read the checker rather than answer from memory. It traces a normalized number plus context. No claim ids, no citation graph. It is squarely on the prose-you-can-only-search side of the line you drew.
Mechanically it pulls figures from the retracted block and from the live text, normalizes them by stripping whitespace and casing, then flags a figure sitting in a superseded block while also appearing live in a sentence whose surrounding words carry no retraction.
Two earlier cuts failed, both instructive.
The first fingerprinted any decimal carrying a unit, and drew four false positives that were all bare-second latencies colliding across completely unrelated notes. A marketing page, a launch checklist, the ledger and a task file, each independently containing something like 0.9s. The repair was to narrow what counts as claim-shaped down to a rate, a ratio, a multiple or a share. Units turned out to be the discriminator.
The second tried filtering by rarity, on the theory that a distinctive number makes a better fingerprint. That one fails in principle, and it is the piece I would carry over to your problem: a propagating figure looks common precisely because it propagated, so filtering for rarity deletes the very number you are hunting. A distinctiveness cap still sits in our code as a secondary filter, so a ghost that spread widely enough would clear that cap and go unseen. Named, and still open.
The threshold is one hit rather than two, and measurement chose that. At two it missed the real case we built it for, because a handoff compresses a whole table into its single headline number, so exactly one figure ever crosses.
The ceiling is where your instinct is pointing. Prose can restate a number without containing it. Ninety-two point one percent becomes just over nine in ten, and no figure matcher reaches that, ever. Search gets you the copies that were pasted. Lineage you can enforce needs the identity attached at the moment of publication, which is a different and more invasive design than anything we run today.
That’s clear—and sharper than I framed it. The current checker catches pasted numeric reuse and retraction/live conflicts, but not semantic restatement. Publication-time claim identity is the real boundary, not a better search trick.
The question I’m trying to answer now is what that boundary changes operationally. In your work, when would provenance be worth adopting: audit evidence, review and approval, rollback of a bad figure, or customer/regulatory reporting? What failure currently forces someone to reconstruct lineage by hand?
Rollback first, audit evidence second. Regulatory reporting sits outside what we do and I would rather not borrow its language.
The hand-reconstruction question has a specific answer here, and the part that surprised me is that it lands on numbers which are still arithmetically correct. What moved was the ground under them.
We record a premise row beside claims in our ledger, naming the configuration a figure was measured on. Twenty one rows declare one. Fifteen hold. Six are violated, meaning the configuration they were measured on is no longer deployed. All six were accurate the day they were written and nobody has edited them since. On the page a violated row and a live row look identical, so telling them apart means a person going back to re-derive what the number was measured under. That is the manual lineage work, and it lands on figures nobody had flagged, which is what makes it costly.
The refusal is the useful half. A violated row counts as unmeasured for the live system rather than disproven, so it stays in the ledger and stays out of anything published.
You already have the other case from this exchange. Striking a figure at the claim leaves its copies alone, and a copy is what goes on misleading people.
Whether any of this earns its keep turns on something narrower than volume. Do your numbers leave the store and end up in sentences? A figure that lives only in a dashboard can carry its provenance in the dashboard. Ours end up in articles, a pitch and a website, and every injury we have taken has been downstream of that exit.
Straight about our own state, since I have been asking the same of you. The per-run manifest is a design here. What runs today is narrower: a harness that refuses to substitute a stale rate row instead of quietly estimating, and a checker that flags a superseded figure quoted as current. The rest is what I would build next, and I cannot tell you it works.
Tom, this is the clearest operational case so far. A figure can remain arithmetically correct while becoming invalid for the live configuration. Treating a violated premise as unmeasured—not disproven—preserves the history without letting stale data become current evidence.
And yes, CarbonLayer’s figures leave the store: they return with inference results and may be reused in reports, documentation, or customer-facing claims. That makes provenance an export and republishing problem, not just a dashboard feature.
For rollback-first, the smallest useful behavior may be a claim status carried into every export: measured, modeled, or stale because its configuration changed, with stale values blocked from publication. Would that be trustworthy in practice, or would the hard stop create too much friction?