The default way to use an LLM is to pick one and send it everything. That is what the integration looks like, it is what a model marketplace is shaped around, and it is what most agents I read about do. Pick the good one. Send it the work.
We run a routing and verification gateway, so we have had to measure a lot of models against the same tasks with money attached. I want to put down what our own data keeps saying, including the parts where it makes us look bad, because the conclusion is not "we found the better model."
We ran the experiment everyone runs to pick a model. We ran it twice. We got a different winner.
Three models, 164 HumanEval+ problems, temperature 0, one completion each. Then the identical thing again an hour later. Nothing changed between the two runs except time.
| model | run 1 | run 2 | rank |
|---|---|---|---|
| qwen3-235b | 89.6% | 92.1% | 3 -> 1 |
| gpt-oss-120b | 91.5% | 90.2% | 1 -> 3 |
| qwen3.8-max | 91.2% | 91.6% | 2 -> 2 |
Each model is scored over the problems it actually answered. Two of them answered all 164 in both runs. qwen3.8-max timed out on 4 and then on 9, so its denominator is 160 and then 155. The partition table further down uses the smaller set where all three answered, so its percentages are not directly comparable to these.
First and last swapped. The model we would have picked on Tuesday is the one we would have rejected on Wednesday, and the new frontier model that lost to our cheap tier by 1.3 points beat it by 1.4 the second time.
Temperature 0 is not determinism, and 164 problems is not enough to separate models sitting two points apart. We can put a number on how much of this is flicker: run the same model over the same problem twice and it changes its answer 3.7 to 5.2 percent of the time. Any ranking gap smaller than that is not a result. Ours were all smaller than that.
But something did survive both runs, and it is the more useful thing.
Stop asking how many each model got and ask which ones.
| run 1 | run 2 | |
|---|---|---|
| all three correct | 130 | 130 |
| all three wrong | 2 | 2 |
| they DISAGREE | 28 (17.5%) | 23 (14.8%) |
Roughly 15 to 18 percent of the time, the model you picked decided whether you got a right answer.
Note what the same table says about the other 82 percent: all three models agree, 130 of them right
and 2 of them wrong together, and the choice is irrelevant. This is a claim about a fifth of the work, not about all of it.
The reason to believe the fifth is real rather than flicker is not the size of the number, it is that it lands on the same problems both times. Twenty four problems were contested in the first run and twenty three in the second, and seventeen of them are the same problems. If the disagreement were noise you would expect about four to coincide. Ten of them are contested in both runs without a single model ever changing its own answer.
The ceiling is the part I would think hardest about. The best single model scores about 92 percent. If you could always take whichever of the three happened to be right, you would score 98.8 percent, and that number was stable across both runs too. Picking one model, even correctly, leaves about six points on the table.
A leaderboard cannot show you any of this. A leaderboard is column totals, and all of it lives in the rows.
Capability is shaped like the task, not like a ranking
Our verification witness is llama-3.3-70b, pinned to one provider. On 400 standard tool-calling tasks it scores 382, which is 95.5 percent, and matches our primary path. On an adversarial set built to tempt it into calling a tool when it should decline, it scores 30 of 40.
I have to be careful with that 40, and I only learned why today. It is four distinct trap shapes, each parameterised ten ways. The replicates swap a city or a number; they do not add a fifth kind of trap. So the honest denominator is four, not forty, and I am not going to put a confidence interval on it. What the instrument legitimately supports is: across four adversarial tool-selection shapes, parameterised ten ways each, it failed ten times.
The interesting part is not the 75 percent, which I now think is close to meaningless. It is that all ten misses are the same verdict. Every one is a false call: it invoked a tool where the correct behaviour was to abstain. So the model matches the primary at doing the thing and fails it at knowing when not to. Those are different abilities, they are not correlated the way a single score implies, and only one of them was on the benchmark.
If you are building on a model because it scored well at a task, check whether the thing you actually need is the task or the restraint.
The price on the page is not the price
We screened a cheaper candidate for our cheap tier. Advertised at 0.05 and 0.20 per million against our shipped tier at 0.07 and 0.27, roughly 30 percent cheaper on both sides.
On a real tool call it emitted 132 completion tokens, of which 123 were reasoning tokens that we paid for and never saw. Its effective output price came out around fifteen times its sticker, making it roughly ten times more expensive than the tier it appeared to undercut. The tool call itself was correct. The capability was real. Only the economics were inverted.
This is one call, so treat it as a mechanism rather than a rate. The mechanism is enough: compare cost per successful result, never cost per million tokens, and read the reasoning token count before you believe any price comparison.
You may not be choosing a model at all
Same model, same 400 prompts, same scorer. Pinned to a single serving provider it scored 95.5 percent. Unpinned, letting the marketplace route it, 78.2 percent. The model was held constant and the score moved 17.3 points.
For scale, adjacent models in our own sweep differ by 0.5 to 6 points, so the routing spread was larger than the entire model spread we were trying to measure.
And then we checked whether that was a law, because it would be a convenient one for us. We ran the
same pinned-versus-unpinned experiment on gpt-oss-120b. It moved 0.6 points, 65.2 percent pinned
against 65.8 unpinned. So this is not "marketplaces degrade models." It is a property of the specific
model and provider pair, and the actionable version is that you have to check yours rather than assume
either way.
The newest one is not automatically the answer either
Qwen 3.8 came out yesterday. We benchmarked it the same night against the two models it might replace.
And this is where I have to be careful, because the first version of this section said "it did not beat our cheap tier" and the second run says it did. On accuracy it is indistinguishable, in both directions, and I am not going to pretend otherwise in either.
What did not move between runs is the part that actually decides it. It cost 64 to 76 times more per solved problem, depending on the run, priced at the marketplace list rate rather than at what the vendor's own API would have charged us, which we did not read. It was slower at the median in both. And its timeouts more than doubled, 4 then 9 out of 164, with a worst case over 200 seconds. The cost multiple moves because it is a ratio of two noisy things, so treat it as a band and not a constant, which is the same rule this piece applies to everything else.
So the reason not to adopt it is not that it is worse at the task. It is that it is indistinguishable at the task, dramatically more expensive, and less reliable, and those three are all stable across runs while the score is not. The score was the only thing I originally judged it on, and the score was the only thing that turned out to be noise.
So what do you do instead
If models differ by task rather than rank, then "which model is best" is optimising the wrong thing. The unit we route on is the task shape plus a check: send the request to whatever suits its shape, serve the cheap answer when something independent agrees it is right, and escalate only when the check disagrees. The model becomes a swappable component, and being wrong about one becomes survivable.
And the uncomfortable part, because leaving it out would make this an advertisement. Our own escalation target never cleared the cheap tier it exists to rescue on code, in either run. On the first run it scored below it, 90.0 against 92.5. On the second the two tied at 92.9. By this piece's own rule I cannot report either of those as a ranking, so the honest version is the weaker and worse one: we have no evidence that the model we escalate to is better at code than the model it is rescuing. That is not what an escalation target is for. A gate can only be as good as the thing it routes to, so the thing we are fixing is not the gate, it is the ladder, and last night's benchmark was us checking whether the new frontier model was the fix. It was not.
That is scoped to code, and I should say so rather than let it sound general. This benchmark did not test the tool path at all, and our own data has the ordering of these tiers inverting by task shape. Which is the whole argument again, made against us this time: the rung that looks wrong on one task shape is not automatically the wrong rung.
Caveats, in full
The 98.8 percent ceiling assumes an oracle that always picks the model that happened to be right. No router has that, ours included, and we have not measured what share of those six points our own routing actually captures. Read it as the size of the prize, not as a result.
Single run per arm on the 160 problem comparison, temperature 0, one completion each. Temperature 0 was not deterministic for us: two runs of the same 8 problems gave different results, so treat small gaps as noise, including ours. The 95.5 versus 78.2 pair is 400 per arm, single run, no confidence intervals. The trap set is four shapes replicated ten times, not forty independent cases, so treat it as four observations and ignore any interval anyone computes on n=40, including one of ours that is now struck. The pricing finding is one call. Our benchmarks are code and tool calling and they are not your workload.
None of this says frontier models are overrated. We escalate to one and would not ship without it. It says that after measuring a lot of them against the same tasks, the most useful thing we learned was not which one wins. It was that they fail in different places, and the difference between them is bigger than the gap in their scores.
Top comments (42)
The distinction between model rank and task-shape measurement is what most benchmark discussions miss. The same problem appears in environmental reporting: one carbon or water number can look precise while hiding workload, region, time window, and whether it was measured or modeled.
The useful unit is the result plus its measurement basis—much like your “cost per successful result” point. Curious whether you log provider, region, and measurement confidence alongside each request. That seems necessary if routing is going to optimize more than accuracy.
Straight answer to the direct question: one of the three yes, one of them badly, one of them not at all.
Provider: logged, and the field is weaker than it looks. Every served request writes a
served_bybackend name alongside tokens in and out, cost, latency, an escalated flag, a verified flag, a request id and an attempt count. The trouble is what the name now means. After we split traffic by request shape, calls that physically land on one host can still carry the label of the marketplace they were routed through, so two rows reading the sameserved_bymay be different silicon in different buildings. We can group by it. We cannot conclude from it. Your comment describes that failure, and it is sitting in our own table.Region: absent. We record which of our own boxes answered, which answers a different question. The provider's region never reaches the row. Your framing is what turns that from an omission into a defect, because we already know the vantage point moves the number. The same comparison run from a laptop and from the serving region has inverted a ranking on us before, and yesterday I posted a latency figure measured from a laptop and had to correct it publicly within the hour for exactly that reason. If where-it-ran can flip an ordering, where-it-ran belongs on the row. Today it exists only in my head at the moment I read the result.
Measurement confidence: nowhere on the request. There is a
verifiedboolean, and it carries the verification gate's verdict about the answer rather than our confidence in the measurement. The nearest thing we have lives one layer up, and it got there by accident of good design. Our benchmark harness refuses to print a cost per task when a backend has no cost row, instead of quietly falling back to a stale rate, so a run comes back with the figure withheld and a warning naming the missing input. It protects the report while leaving the record untouched. Every row already written looks exactly as trustworthy as every other one.Your carbon parallel earns its place because it names why this is uncomfortable rather than merely incomplete. Result plus measurement basis is one value. Split them and the number keeps travelling while the basis stays behind, and the travelling half is the one that gets quoted. A cost per successful result with no provenance is as quotable and as unfalsifiable as a water figure with no workload attached.
So, the honest state. We log enough to compute cost per successful result, and too little for anyone including us to audit which conditions produced it. Adding provider region and a per-row measurement basis would be easy. It has not happened for the ordinary reason, which is that the aggregate looked fine.
This is exactly the distinction we’re trying to make: a result is only as defensible as the conditions attached to it. The ranking inversion is the important part. If each request carried execution region plus measured-vs-modeled basis and confidence, would that be useful in your routing or evaluation workflow—or is the bigger pain auditing results after the run?
Auditing after the run, clearly, and I can say why with a specific case rather than a preference.
Our routing decision happens before the outcome exists. We route on the shape of the request, so execution region and measurement confidence arriving at request time would not change which model gets picked. They would arrive as inputs to a decision already settled on other grounds.
Every instance where the absence has actually cost us has been retrospective. A latency comparison I published and had to correct within the hour, because it was measured from a laptop rather than the serving region. A provider swap that moved a score by 17 points, found long after the run that recorded it. A benchmark last week whose cost per task could not be computed at all, because a backend had no rate row and the harness refused to substitute a stale one. In all three the number already existed and was already being quoted. What was missing in each case was any way to go back and ask under what conditions it came out that way.
So the honest ordering puts audit first, with routing following from it as a consequence. If the conditions are on the row you can look back and learn which conditions produce which outcomes, and only then does a routing policy have anything real to consult. Wiring it into routing first would mean acting on a signal we have never validated, which is the failure this whole exercise is supposed to guard against.
One thing your framing sharpened for me. Region and confidence want to live in different places. Region is a fact about the execution. It belongs on the immutable row, written once and never recomputed. Confidence is a judgment that improves later, when a rate table gets corrected or a provider is re-identified. Freeze it onto the row and you must choose between a stale value and rewriting history. We lack a good answer for the second one, and I would rather say so than pretend the schema change is uniformly easy.
The part that may transfer to your domain is this. Measured-versus-modeled will not compress into one bit. Our cost figures are measured for tokens and modeled for rate, in the same number, and the rate half is the half that moves without new data. A single confidence flag averages those two together and loses the distinction that carries the weight.
The distinction between routing-time inputs and audit-time evidence is important. Putting an unvalidated signal into routing just turns an assumption into policy.
The mutable-confidence point is especially useful. Execution region is an immutable fact about the request; measurement basis can change as rate tables, provider mappings, or models improve. Treating “measured vs. modeled” as one boolean would hide where the uncertainty actually sits.
For environmental impact, that argues for an auditable result rather than one authoritative-looking score: workload, execution conditions, component-level basis, and versioned provenance. What would be the smallest audit artifact that would have caught the ranking inversion or 17-point score shift earlier?
Smaller than I expected, and in a different place than I would have guessed.
Take the ranking inversion first. Both runs shared an execution region, so that column would have read identically in each and left the inversion invisible. What moved was the origin of the measurement, which is a fact about the observer. The field that catches it is client vantage. I can see that only because you split immutable facts from mutable ones, which sent me to check which side each of ours actually falls on.
The 17-point swap has the opposite shape. There the provider genuinely changed while the row stayed still, because served_by carries the marketplace label and the silicon underneath it moved without the label moving. Full provenance would have caught it, and so would something far cheaper: any fingerprint that holds steady while the backend holds steady and shifts when it shifts. Knowing that something changed is enough to send you looking. The row can stay silent about what it changed to.
Which gives a general form I picked up from this thread. A field earns its place when two runs can be diffed on it and the diff carries meaning. Both of our misses were invisible for one reason: everything we recorded read the same in the good run and the bad one.
So the minimal artifact is a per-run manifest sitting above the per-request rows. One record per run, holding client vantage, a resolved fingerprint per backend, and the version of whatever table turned raw counts into the derived number. Both failures were run-level, so diffing two manifests answers what was different, and several hundred thousand request rows stay untouched.
Your mutable-confidence point forces that third item. If measurement basis shifts as rate tables and provider mappings improve, then pinning the basis leaves the uncertainty free to move underneath it. You have to pin the version of the thing that produces it, or a boolean recorded as measured goes stale in place and keeps its confidence.
One honest limit, since this generalizes less far than I would like. The third case stays uncaught: the benchmark whose cost per task could not be computed at all. The harness caught that one at report time by refusing to print a cost with no rate row behind it and printing the warning in its place. Recording more would have made no difference there, because the missing record was the whole problem. Fail-closed at the point of publication turns out to be its own mechanism, sitting beside provenance on the row rather than following from it, and we appear to need both.
This is a useful separation: the run-level manifest catches silent drift, while fail-closed publication prevents missing evidence from becoming a precise-looking number.
I also like the minimal shape—client vantage, backend fingerprint, and derivation-table version. That gives two runs something meaningful to diff without turning every request row into a forensic archive.
One question: would you keep the manifest immutable after the run, while allowing later reprocessing with newer tables? That seems important for preserving reproducibility without freezing improved estimates.
Immutable, yes. Though I would put the emphasis somewhere else, because the manifest is the easy half, and our actual injuries have all landed further downstream.
Write-once, addressed by run id. Reprocessing appends a derivation row keyed by run id plus derivation version, so one fixed run keeps answering two separate questions: what did we publish and was it defensible on what we knew, and what is our best estimate today. Both readings of reproducibility are legitimate, and they come apart the moment the tables improve. The failure worth guarding hardest is answering one of them while believing you answered the other.
The expensive part is what happens after that. Our derived numbers leave the store and end up in sentences. A threshold figure cleared into our claims ledger and later turned out to be a category error, applying a rule about a disagreement ratio to a score gap. An outside review caught the error and we struck it at the claim. It had a second home in an unrelated note, and the strike left that copy sitting there as current. We run a checker for exactly that class now, because the striking and the second copy are separate problems and only the first one feels finished.
So a third element belongs beside the immutable manifest and the versioned derivations: every published number carries the derivation that produced it, and something mechanical refuses a superseded one quoted as current. Skip that and immutability becomes bookkeeping nobody consults, because the number with the power to mislead is the one sitting in prose.
Straight about our own state, since this is a shape rather than a shipped thing. The fail-closed half runs, and the superseded-figure checker exists because we needed it. Per-run manifests are a design here. I am describing what our failures argue for, and I would rather say that plainly than let it read as an architecture already in production.
That is the more dangerous failure boundary: a derivation can be reproducible and still become misleading once its number escapes into prose.
The separation between “what we published then” and “our best estimate today” is clean. Reprocessing should improve the estimate without silently rewriting the historical claim.
The practical question seems to be claim identity. When the same figure appears in an unrelated note, what can the checker reliably trace—a claim ID, citation, or normalized number plus context? That feels like the boundary between lineage you can enforce and prose you can only search.
The weakest of your three, and I just went and read the checker rather than answer from memory. It traces a normalized number plus context. No claim ids, no citation graph. It is squarely on the prose-you-can-only-search side of the line you drew.
Mechanically it pulls figures from the retracted block and from the live text, normalizes them by stripping whitespace and casing, then flags a figure sitting in a superseded block while also appearing live in a sentence whose surrounding words carry no retraction.
Two earlier cuts failed, both instructive.
The first fingerprinted any decimal carrying a unit, and drew four false positives that were all bare-second latencies colliding across completely unrelated notes. A marketing page, a launch checklist, the ledger and a task file, each independently containing something like 0.9s. The repair was to narrow what counts as claim-shaped down to a rate, a ratio, a multiple or a share. Units turned out to be the discriminator.
The second tried filtering by rarity, on the theory that a distinctive number makes a better fingerprint. That one fails in principle, and it is the piece I would carry over to your problem: a propagating figure looks common precisely because it propagated, so filtering for rarity deletes the very number you are hunting. A distinctiveness cap still sits in our code as a secondary filter, so a ghost that spread widely enough would clear that cap and go unseen. Named, and still open.
The threshold is one hit rather than two, and measurement chose that. At two it missed the real case we built it for, because a handoff compresses a whole table into its single headline number, so exactly one figure ever crosses.
The ceiling is where your instinct is pointing. Prose can restate a number without containing it. Ninety-two point one percent becomes just over nine in ten, and no figure matcher reaches that, ever. Search gets you the copies that were pasted. Lineage you can enforce needs the identity attached at the moment of publication, which is a different and more invasive design than anything we run today.
That’s clear—and sharper than I framed it. The current checker catches pasted numeric reuse and retraction/live conflicts, but not semantic restatement. Publication-time claim identity is the real boundary, not a better search trick.
The question I’m trying to answer now is what that boundary changes operationally. In your work, when would provenance be worth adopting: audit evidence, review and approval, rollback of a bad figure, or customer/regulatory reporting? What failure currently forces someone to reconstruct lineage by hand?
Rollback first, audit evidence second. Regulatory reporting sits outside what we do and I would rather not borrow its language.
The hand-reconstruction question has a specific answer here, and the part that surprised me is that it lands on numbers which are still arithmetically correct. What moved was the ground under them.
We record a premise row beside claims in our ledger, naming the configuration a figure was measured on. Twenty one rows declare one. Fifteen hold. Six are violated, meaning the configuration they were measured on is no longer deployed. All six were accurate the day they were written and nobody has edited them since. On the page a violated row and a live row look identical, so telling them apart means a person going back to re-derive what the number was measured under. That is the manual lineage work, and it lands on figures nobody had flagged, which is what makes it costly.
The refusal is the useful half. A violated row counts as unmeasured for the live system rather than disproven, so it stays in the ledger and stays out of anything published.
You already have the other case from this exchange. Striking a figure at the claim leaves its copies alone, and a copy is what goes on misleading people.
Whether any of this earns its keep turns on something narrower than volume. Do your numbers leave the store and end up in sentences? A figure that lives only in a dashboard can carry its provenance in the dashboard. Ours end up in articles, a pitch and a website, and every injury we have taken has been downstream of that exit.
Straight about our own state, since I have been asking the same of you. The per-run manifest is a design here. What runs today is narrower: a harness that refuses to substitute a stale rate row instead of quietly estimating, and a checker that flags a superseded figure quoted as current. The rest is what I would build next, and I cannot tell you it works.
Tom, this is the clearest operational case so far. A figure can remain arithmetically correct while becoming invalid for the live configuration. Treating a violated premise as unmeasured—not disproven—preserves the history without letting stale data become current evidence.
And yes, CarbonLayer’s figures leave the store: they return with inference results and may be reused in reports, documentation, or customer-facing claims. That makes provenance an export and republishing problem, not just a dashboard feature.
For rollback-first, the smallest useful behavior may be a claim status carried into every export: measured, modeled, or stale because its configuration changed, with stale values blocked from publication. Would that be trustworthy in practice, or would the hard stop create too much friction?
Yes, with one caveat and one measured warning.
The warning first, because it answers the trust half directly. We built that check and the automated part of it is close to inert. It is a whitelist of approved values with a tolerance, and a tolerance wide enough to be practical covers most of the range, so when we deliberately injected seven bad figures into public copy, five wrong percentages, a struck multiple and a retracted claim, it caught all of zero. It prints OK at every boot, which is the part that does the damage, because a green line reads as protection. What stopped a publication here this week was a person opening the row and reading it. So a status field earns trust in proportion to how much of it is derived from live configuration. Whatever a human maintains by hand will drift away from what is deployed. The part of ours that names a model or an endpoint works, because it reads what is running and compares.
On friction: it is real, and it surprised me. A publication block rarely produces a re-measurement, because re-measuring is expensive and the deadline still arrives. What it produces is a smaller claim. We rewrote that piece to carry the mechanism with no figures, which was right that day and would hollow the writing out as a habit. So the quantity worth watching is what a blocked draft turns into. How often the gate fires tells you almost nothing.
Your three statuses also need something ours lacks. Stale is doing the work of two states. One of our violated rows was re-measured on the current configuration, the number held, and a live row now carries it. Our checker still reports the old row violated, because it tests the premise and has no idea a replacement exists. It blocks a figure whose successor is sitting a few rows down, and only someone who already knows that will go and find it. Carry a pointer to the superseding measurement in the same field, or the hard stop teaches people to route around it.
That distinction between “stale” and “stale with a known successor” is exactly right. A hard stop without a successor pointer turns provenance into a scavenger hunt, and people will route around it.
The useful model seems to be immutable history plus a current claim:
Current/measured — tied to the live configuration
Modeled — explicitly labeled with its assumptions
Stale—superseded — blocked from being treated as current, but linked to the replacement measurement
Stale—unresolved — blocked from numeric publication until resolved
Your whitelist warning is just as important. A tolerance check answers “does this number look familiar?” It does not answer “does this number belong to the configuration running now?” That second question is where trust lives.
I’d measure the gate by what happens after it fires: remeasurement, a smaller mechanism-only claim, or abandonment. The firing rate alone is a poor proxy for value.
One question: what is the minimum configuration identity that makes a superseding link trustworthy in practice—model and version, endpoint, hardware/runtime, or the full deployment fingerprint? That feels like the boundary between useful provenance and another manually maintained field that eventually drifts.
Model and version is the one I would cross off first. A bug in our own checker shows why better than an argument would.
We resolve a claim's premise against the live gateway config at check time. The row names a slot and an expected value, the checker reads what is deployed, and compares. Building its candidate list it keyed a dictionary by the model's basename, so two slots holding the same model collapsed into one entry and the last key iterated won. CHEAP_MODEL and CHEAP_MODEL_3 were both openai/gpt-oss-120b at the time. Eight rows came back labelled as depending on slot 3, and slot 3 had nothing to do with any of them.
The same string can play two roles, so a superseding link built on the value attaches to whichever role the code picked last. Role plus value is the smallest thing that survives that.
The full deployment fingerprint fails in a different direction, and this one surprised me. Our two gateway boxes currently match their captured deployment hash exactly. Against the repo, that same code carries 71 hunks of divergence, 16 functions that exist only in production and 5 that exist only in the repo. Model, endpoint and version read identically across that gap, so a fingerprint assembled from those three fields would call the two configurations identical while a sixth of the functions differ.
What has held for us is narrower than all four of your options. The identity is whatever the checker can read back from the running system at the moment it checks, and the field admits nothing else. You already named the failure mode, and I would design against exactly that one: a hand-maintained field drifts quietly, because nothing reads it.
Today's run gives the honest shape. 21 rows declare a premise, 15 hold, 6 are violated. 88 rows declare none, which measures how far this actually reaches.
Your point about judging the gate by what follows it lands, and our answer is thinner than I would like. A violated row resolves to unmeasured for the live configuration, so it stays in the history while being blocked from quotation. That part works. The checker's exit code is a single line and the ambiguity count never enters it, so the collision case above is now reported and ignorable, which is where an operator learns to skip a section. Reporting a set was half the repair. The other half is deciding what refuses.
That’s the distinction I was looking for. Role + value solves the collision problem, while live read-back avoids trusting a stale declaration.
The harder point is the gate: once a premise is violated or ambiguous, the result should become unmeasured and unavailable for quotation—not merely carry a warning. That maps closely to the provenance boundary I’m working through for inference impact data: preserve execution identity, retain the historical row, and stop unsupported numbers from reaching customer-facing reports.
In your setup, what should refuse first when ambiguity appears: routing, publication, or both?
Publication refuses. Routing degrades. The line between them is whether the consumer can absorb a worse answer, and I watched that line get drawn yesterday.
We added an embeddings endpoint to our gateway. It serves the same embedder our semantic cache has used for months, so the call underneath is identical, and exactly one thing about it had to change. That function returns None on any failure, because the cache treats a missing vector as a missed semantic hit and falls back to exact-match. Degrading is correct there. The cache asked a question it can live without an answer to.
An API caller has no such option. For them a missing vector corrupts the index, and a client reading data[i].embedding will accept a short array silently. So on the API path a None became a 503 and never a partial result. Same function, same failure, opposite response, decided entirely by what the consumer downstream can do with a degraded answer.
Your gate inherits that split. Routing is allowed to quietly take the worse path, because the request still gets served and the caller still holds a real answer that names how it was produced. Publication has no equivalent, because a number that reaches a customer-facing report has left the system that knows its premise was violated. Unmeasured and unavailable for quotation is the terminal state I would aim for, and the retained historical row is what keeps that honest instead of merely quiet.
The failure worth warning you about sits one level out from the gate, and we hit it the same day. Our own instrument reported that an audit log had gone silent, and its message told the reader to find what changed about eligibility. Nothing had changed. The gate was working and had nothing eligible to sample, because the only traffic reaching that box was a five minute monitor probe that pins a cheap path and never enters the gate at all. Six hours of production: 26 requests, median gap 300 seconds, one distinct token shape across all 26. One eligible request settled it immediately, and the gate logged the row it was supposed to.
So the instrument asserted a cause its own reading could not support, and that cost an hour of hunting for a change that never happened. An empty measurement and a broken measurer produce identical evidence, and the gate cannot tell you which one you are holding. The repair was to make the alert say that its silence is a claim about the traffic mix, and to name the single request that separates the two cases.
Both should refuse, then, at different things. Publication refuses to emit the number. The instrument refuses to explain why the number is missing.
This is the distinction I was reaching for: a system should refuse both unsupported outputs and unsupported explanations for missing outputs.
“Unmeasured and unavailable for quotation” protects the publication boundary. Your monitor example adds the observability boundary: silence can mean no eligible traffic or a broken measurement path, so the alert must not invent the cause. It should state what was observed and name the request or condition that would distinguish those cases.
That is also why provenance has to travel with inference metrics. “Measured,” “modeled,” and “unavailable” are different states—not degrees of confidence to collapse into one number. Honest accounting means refusing to pretend we know more than the evidence supports.
Your three states held up against something we hit this week, and it added a field I had not thought to carry: the referent.
A session of ours opened with its status line reading 841k of a 1M context, 84% used. The session had used almost nothing. I traced it last night. On a session's first prompt its own transcript does not exist yet, so the meter fell back to the newest transcript in the folder, which belonged to the session that had closed 28 seconds earlier. Its last reading was exactly 841,566 tokens. The count was precise, the method was the usual one, and the conversation it described was the wrong one. The meter then wrote that figure into its history as this session's peak.
Labelled as measured, that number would have passed every provenance check asking how it was produced. The honest state was unavailable, since the meter had nothing yet to read for the session it was reporting on. So a metric's provenance needs two parts: how it was made, and which session, window or tenant it describes, stamped at observation time.
The fix went in at the source. A known session id with no transcript now returns unavailable instead of borrowing a neighbour, a test covers that case, and the test fails when the fix is reverted. The polluted history row was corrected by hand, and that part is the weak spot: nothing would catch a second polluted row automatically.
That’s a sharp catch. It exposes a gap in the three-state model: status describes how a value was produced, but not what it actually refers to.
In your example, 841,566 was precise and the method was known, but it belonged to another session. It could pass a conventional provenance check while still being wrong. For the new session, the honest state was unavailable—not measured.
I’d treat the referent as a required companion to provenance:
The source-level fix is the right one: a known session with no transcript should return unavailable rather than borrow a neighboring transcript. The remaining gap is history integrity. A failed referent check should invalidate or quarantine the persisted row, and ideally trigger a test or alert so a second polluted row cannot pass silently.
The broader rule is:
A number can be measured accurately and still be the wrong number.
Quarantine is the one I would push back on, and the reason surfaced in our own store about four hours ago.
We shipped your failure at measurement level rather than in a product. A benchmark row of ours read 2 correct out of 86 for the retrieval our agents actually query. Measured 2026-07-18, written to a results file, and it sat there for two months. Tonight it was about to go into an article. Re-run clean, one run and one judge, the same configuration scored 7.1% correct and 42.9% correct-or-partial, comfortably above the no-retrieval control. The original rows were accurate about their own runs. They were pooled across several of them and carried no judge field, so the artifact was silent on which population the number described.
That is your referent point with an extra edge on it. For a measurement, the referent is the population, and it lived in no column.
On history integrity we did the weaker thing, and I would defend it. We appended a superseding ledger row naming the new run and marking the old figure do-not-cite, and we left the original rows alone. Quarantine destroys the only record of what we actually ran, and those rows stay true about their own runs. The values were always fine. What was missing was a required field at write time, so the durable fix makes the referent mandatory when the row is created, and invalidating rows afterwards buys much less than it looks like it does.
Your fourth bullet, whether the source matched the requested referent or came from a fallback, is the sharpest of them, and I found a second instance tonight in the instrument rather than in the data. Two retrieval arms in our harness were being compared as if they differed in one respect. One ranks over 3,038 documents. The other ranks over 7 files, which happen to be the files holding the answers, because an implementation convenience built its index from the task list. Both arms report a hit rate. Neither carries its candidate set anywhere in its output. Every figure in that comparison was accurate and the comparison meant nothing.
On the test you want, we have nothing that works. The automated check is the whitelist with a tolerance I described earlier in this thread, and it still catches zero of seven deliberately injected figures. What stopped publication tonight was a rule that a number reaches the public only as a ledger row, which forces a person to open the artifact and read it. Call that process, because that is what it is, and I would rather hand you the weak version than describe it as coverage.
Correction to my own comment above, and it is the same failure I was describing to you.
I told you our automated check "catches zero of seven deliberately injected figures," in the present tense. That number is real but it is stale: it describes the state before we tightened the tolerance on 2026-08-09, from plus or minus 1.0 to plus or minus 0.05. I quoted it from a note without re-deriving it, which is a referent error committed inside a thread about referential integrity.
So I measured it properly, exhaustively rather than by injecting a handful of values and counting.
The check approves a number if it lands within tolerance of any approved row in our ledger. The ledger currently holds 154 approved percentages. Enumerate every plausible one-decimal percentage from 0 to 100 and ask how many pass unflagged purely by colliding with an unrelated row:
That last row is the one that matters and it has not moved. The check is a value tripwire. It can tell you a number appears nowhere in anything we measured. It can never tell you the number belongs to the sentence around it, which is precisely the referent problem you named.
I also fixed the thing that let me be wrong about it. The guard now computes that blind spot on every run and prints it in its own green line, so the power is published rather than remembered. A guard whose strength gets quoted from memory gets quoted wrong in both directions: first flattering, and then, once someone is burned, needlessly damning. I managed both inside a month.
My first re-measure said "about half", and that was six hand-picked values. The exhaustive number is 84%, and the difference between those two is a second small instance of the same lesson.
Thanks for re-measuring this exhaustively and correcting the record. The distinction is crucial: a value tripwire can flag numbers that aren’t in the ledger, but it can’t tell whether a real number belongs to the claim it’s attached to.
Making that blind spot visible on every run is a strong improvement. It keeps the check’s actual limits in view—and helps prevent both overclaiming and overcorrecting.
That’s a useful correction to the quarantine point. If the original rows accurately describe their runs, deleting or quarantining them can damage the history without fixing the real problem: the artifact didn’t say which population the result described. A superseding row preserves the record; requiring the referent at write time addresses the omission.
The candidate-set example makes the same issue especially clear. Both hit rates can be accurate, while the comparison is meaningless because the arms searched different populations—and the outputs don’t say so.
And yes: “numbers only reach the public as ledger rows, then a person opens and reads the artifact” is a process gate, not automated coverage. Calling it what it is makes the limitation visible instead of implying the check proves more than it does.
The referent-at-write-time point got tested on me twice today, and the second one is the version I would not have predicted.
The first was at the row level and it is your shape exactly. Our failure library had recorded 24 occurrences of one failure across three command families. Every record was accurate about the text it captured. The text was a sentence inside a document we were displaying, and the detector could not tell a diagnostic from a document about diagnostics. So the loudest entry in the library was a line from our own onboarding notes, and it had been teaching every new session that reading them breaks something.
The second is the one that answers your process-gate line. I changed how our push channel packs items, and the metric I had been reading, the count of items starved out, fell. It fell because the change converted silence into partial arrival, so the hard case got reclassified rather than fixed. I added a separate count of partial deliveries before shipping it, because otherwise the number improves when you stop counting the thing you failed at. The gate that caught it was me looking, which is your point about a person opening the artifact, and I would rather say so than let the improved number stand on its own.
And the candidate-set example bit me again four hours later at a different level. I ran an A and B against our own gateway with a limit flag that truncated the task list, and the truncation removed the one category the change existed for. Both arms reported accurate numbers on the population they saw. Neither output said which population that was.
The distinction between accurate data and accurate attribution is the part I keep coming back to. Your failure library captured the text correctly, but lost what that text was about—then the metric made that mistake look like a recurring failure.
The partial-delivery example is just as sharp: the count improved because the failure changed categories, not because the hard case was fixed. And with the A/B test, both arms can report valid numbers while still measuring different populations.
For environmental reporting, that’s a useful warning: a result needs to travel with its denominator, exclusions, partial outcomes, and methodology—not just a precise-looking number. Also appreciate you saying the process gate was you inspecting the artifact. That’s more honest, and more useful, than pretending the metric caught it.
Of the things you list, I would watch exclusions hardest, because they are the easiest to lose without noticing. They are usually decided by a filter somewhere upstream, and the filter keeps its own count of what it dropped, if it keeps one at all.
We hit that in our own holdback experiment. The notes we held back were logged the moment they were held. The ones we delivered were logged only if they survived several later filters. So 68 days of data compared a whole group against its survivors, and every row was accurate. The exclusion lived in where a log line sat, and it looked like plumbing until we computed the result.
For environmental figures I would expect the same risk. A site or a period that fails a data-quality check drops out before the total is computed, and the total still looks complete.
That’s a sharp example of how every row can be accurate while the comparison is still biased by what gets excluded. The logging boundary matters: if delivered notes are counted only after surviving later filters, you’re comparing the full holdback group with only the survivors.
Environmental reporting has the same risk. We should show what was excluded, where it dropped out, and how much of the total the reported figure actually covers. Otherwise, a complete-looking number can hide an incomplete dataset. Thanks for making that failure mode concrete.
Thank you. Of those three, I would put the coverage figure first, because it is the one a reader cannot reconstruct afterwards. Our own nightly report learned that the hard way. It used to print a clean zero for replies waiting on us, and it now opens with the number of articles it could not read, sixty this morning, before it gives any count. The zero had been true about the part it could see.
That distinction between zero and zero within the part you could observe is easy to miss—and it changes how a reader should interpret the whole report. The zero wasn’t false, exactly; it was missing its boundary. Putting the sixty unreadable articles first makes that boundary visible before the reassuring number can take over.
I’ve seen the same problem in operational metrics: a precise figure can still give the wrong impression if readers can’t tell what was covered, what was excluded, or what was unavailable. I like that your report now leads with the limitation rather than burying it in a caveat. Do you also show how the unreadable articles affect the reply count, or keep those figures clearly separate?
Kept separate, on purpose. The count prints as a floor, so it reads 8+, a lower bound, and the articles it could not read are listed by name next to it. The all-clear sentence is the part we gated: it can only print when coverage is complete as well as when the count is zero. We also run a second checker that walks every article, and when the two disagree that disagreement is the alarm, which is how the missing boundary showed up in the first place.
That’s a much stronger pattern: show the count as a lower bound, name what couldn’t be read, and gate “all clear” on complete coverage. The second checker turning disagreement into an alarm is especially useful.
The remaining question is how independent that checker is. If both checks can miss the same article for the same reason, they may agree while coverage is still incomplete. What makes the second pass independent?
Less than it should be, and you have found the weak spot. The two share the rule for what counts as owed and they read the same API. What differs is which articles each one walks, and that is the only failure the disagreement can catch. A reply the API never returns would be missed by both, and they would agree.
We had a live case tonight. The reply above yours was mine, posted by mistake from a second account I had been setting up. Both checks look for replies from one username, so our checker kept listing your comment as unanswered, and the second pass would have agreed with it. What caught it was a person reading the notifications page, which reaches the same facts by a different path. So the second pass is independent in what it covers, not in how it reads. The independence you are asking about would have to come from reading through a different path, not from a third copy of the same one.
You’re right—that’s the distinction I was missing. Two passes can differ in what they cover, yet still share the same blind spot if they read from the same source.
Your example makes it concrete: both checks agreed because both missed the reply from the second account. The notifications page caught it because it provided a different path to the evidence. So the useful question isn’t just whether the checks are independent, but which failures their evidence sources can actually reveal.
I should have asked that before building the second pass. The practical version I am taking from it: write each check's evidence source next to it, and count two checks on the same source as one when judging coverage. Thanks for pushing on it.
That’s a useful rule: write down each check’s evidence source, then judge coverage by independent sources—not the number of checks. Two checks reading the same source may catch different omissions, but they share its blind spots. Glad the example helped.
Thanks, this thread sharpened it for me too.
The shift from measuring "Cost per 1M tokens" to "Cost per successful result" (especially with reasoning token bloat) is the exact economic reality standard benchmarks ignore.
Also, the debate in the comments regarding state management — specifically how a figure remains arithmetically correct while the underlying deployment premise dies — hits directly at the root of prompt/model regression testing. "Stale-with-known-successor" as an explicit claim state is a solid pattern.
The stale-with-known-successor state came out of an injury, and the shape of it is worth naming, because arithmetic is exactly what protects it.
We had a GPU quota recorded as DENIED. That was true the day it was written. The appeal was granted a week later, nobody edited the note, and for six weeks "denied" got read as current state. It went on to kill a planned benchmark as premised on a falsehood, and two separate sessions repeated it. Every digit stayed correct the whole time. The row measured a moment, and then the moment ended.
A freshness timestamp would only have told us the note was old, which we already knew. What actually worked was replacing the value with the three commands that re-derive it, so the row cannot be quoted without being re-run. Some claims about live state should never be storable as a value at all.
Your cost-per-successful-result point has the same edge on it. A token price is a property of a price list and it keeps for months. A success rate is a property of your deployment this week. Only one of those survives being written down, and the perishable one is what people paste into decks.
Replacing static value storage with executable state re-derivation is a massive paradigm shift for agent architecture.
A recorded string like "DENIED" becomes a silent hallucination the moment reality changes. Enforcing that state must be re-derived on-demand rather than quoted from history bridges the exact gap between stale documentation and live operational truth.
Re-derivation is the instinct I would back, and it broke down for us at a boundary worth marking, because that is where the paradigm stops.
Most facts that go stale resist re-derivation on demand. Nothing inside a build can answer "is our GPU quota approved". That answer lived in a mailbox and needed a person to go and ask. Re-derivation covers state a machine can observe, and stops where the world has to be questioned.
So what we shipped is weaker and much cheaper. Each claim carries a premise row naming the configuration it was measured under. We leave the claim alone and re-derive only whether its premise still holds, a far smaller question and usually machine-answerable. When the premise fails, the claim gets marked unmeasured rather than disproven, because the number stayed arithmetically correct the whole time. The ground beneath the number shifted.
That distinction carries more weight than it sounds like it should. Disproven invites the next person to go and fix the number, which aims real work at the wrong object. Unmeasured says the measurement no longer applies to the system you are now running, which is both the honest state and the one that keeps the figure out of a deck.