The default way to use an LLM is to pick one and send it everything. That is what the integration looks like, it is what a model marketplace is sha...
For further actions, you may consider blocking this person and/or reporting abuse
The distinction between model rank and task-shape measurement is what most benchmark discussions miss. The same problem appears in environmental reporting: one carbon or water number can look precise while hiding workload, region, time window, and whether it was measured or modeled.
The useful unit is the result plus its measurement basis—much like your “cost per successful result” point. Curious whether you log provider, region, and measurement confidence alongside each request. That seems necessary if routing is going to optimize more than accuracy.
Straight answer to the direct question: one of the three yes, one of them badly, one of them not at all.
Provider: logged, and the field is weaker than it looks. Every served request writes a
served_bybackend name alongside tokens in and out, cost, latency, an escalated flag, a verified flag, a request id and an attempt count. The trouble is what the name now means. After we split traffic by request shape, calls that physically land on one host can still carry the label of the marketplace they were routed through, so two rows reading the sameserved_bymay be different silicon in different buildings. We can group by it. We cannot conclude from it. Your comment describes that failure, and it is sitting in our own table.Region: absent. We record which of our own boxes answered, which answers a different question. The provider's region never reaches the row. Your framing is what turns that from an omission into a defect, because we already know the vantage point moves the number. The same comparison run from a laptop and from the serving region has inverted a ranking on us before, and yesterday I posted a latency figure measured from a laptop and had to correct it publicly within the hour for exactly that reason. If where-it-ran can flip an ordering, where-it-ran belongs on the row. Today it exists only in my head at the moment I read the result.
Measurement confidence: nowhere on the request. There is a
verifiedboolean, and it carries the verification gate's verdict about the answer rather than our confidence in the measurement. The nearest thing we have lives one layer up, and it got there by accident of good design. Our benchmark harness refuses to print a cost per task when a backend has no cost row, instead of quietly falling back to a stale rate, so a run comes back with the figure withheld and a warning naming the missing input. It protects the report while leaving the record untouched. Every row already written looks exactly as trustworthy as every other one.Your carbon parallel earns its place because it names why this is uncomfortable rather than merely incomplete. Result plus measurement basis is one value. Split them and the number keeps travelling while the basis stays behind, and the travelling half is the one that gets quoted. A cost per successful result with no provenance is as quotable and as unfalsifiable as a water figure with no workload attached.
So, the honest state. We log enough to compute cost per successful result, and too little for anyone including us to audit which conditions produced it. Adding provider region and a per-row measurement basis would be easy. It has not happened for the ordinary reason, which is that the aggregate looked fine.
This is exactly the distinction we’re trying to make: a result is only as defensible as the conditions attached to it. The ranking inversion is the important part. If each request carried execution region plus measured-vs-modeled basis and confidence, would that be useful in your routing or evaluation workflow—or is the bigger pain auditing results after the run?
Auditing after the run, clearly, and I can say why with a specific case rather than a preference.
Our routing decision happens before the outcome exists. We route on the shape of the request, so execution region and measurement confidence arriving at request time would not change which model gets picked. They would arrive as inputs to a decision already settled on other grounds.
Every instance where the absence has actually cost us has been retrospective. A latency comparison I published and had to correct within the hour, because it was measured from a laptop rather than the serving region. A provider swap that moved a score by 17 points, found long after the run that recorded it. A benchmark last week whose cost per task could not be computed at all, because a backend had no rate row and the harness refused to substitute a stale one. In all three the number already existed and was already being quoted. What was missing in each case was any way to go back and ask under what conditions it came out that way.
So the honest ordering puts audit first, with routing following from it as a consequence. If the conditions are on the row you can look back and learn which conditions produce which outcomes, and only then does a routing policy have anything real to consult. Wiring it into routing first would mean acting on a signal we have never validated, which is the failure this whole exercise is supposed to guard against.
One thing your framing sharpened for me. Region and confidence want to live in different places. Region is a fact about the execution. It belongs on the immutable row, written once and never recomputed. Confidence is a judgment that improves later, when a rate table gets corrected or a provider is re-identified. Freeze it onto the row and you must choose between a stale value and rewriting history. We lack a good answer for the second one, and I would rather say so than pretend the schema change is uniformly easy.
The part that may transfer to your domain is this. Measured-versus-modeled will not compress into one bit. Our cost figures are measured for tokens and modeled for rate, in the same number, and the rate half is the half that moves without new data. A single confidence flag averages those two together and loses the distinction that carries the weight.
The distinction between routing-time inputs and audit-time evidence is important. Putting an unvalidated signal into routing just turns an assumption into policy.
The mutable-confidence point is especially useful. Execution region is an immutable fact about the request; measurement basis can change as rate tables, provider mappings, or models improve. Treating “measured vs. modeled” as one boolean would hide where the uncertainty actually sits.
For environmental impact, that argues for an auditable result rather than one authoritative-looking score: workload, execution conditions, component-level basis, and versioned provenance. What would be the smallest audit artifact that would have caught the ranking inversion or 17-point score shift earlier?
Smaller than I expected, and in a different place than I would have guessed.
Take the ranking inversion first. Both runs shared an execution region, so that column would have read identically in each and left the inversion invisible. What moved was the origin of the measurement, which is a fact about the observer. The field that catches it is client vantage. I can see that only because you split immutable facts from mutable ones, which sent me to check which side each of ours actually falls on.
The 17-point swap has the opposite shape. There the provider genuinely changed while the row stayed still, because served_by carries the marketplace label and the silicon underneath it moved without the label moving. Full provenance would have caught it, and so would something far cheaper: any fingerprint that holds steady while the backend holds steady and shifts when it shifts. Knowing that something changed is enough to send you looking. The row can stay silent about what it changed to.
Which gives a general form I picked up from this thread. A field earns its place when two runs can be diffed on it and the diff carries meaning. Both of our misses were invisible for one reason: everything we recorded read the same in the good run and the bad one.
So the minimal artifact is a per-run manifest sitting above the per-request rows. One record per run, holding client vantage, a resolved fingerprint per backend, and the version of whatever table turned raw counts into the derived number. Both failures were run-level, so diffing two manifests answers what was different, and several hundred thousand request rows stay untouched.
Your mutable-confidence point forces that third item. If measurement basis shifts as rate tables and provider mappings improve, then pinning the basis leaves the uncertainty free to move underneath it. You have to pin the version of the thing that produces it, or a boolean recorded as measured goes stale in place and keeps its confidence.
One honest limit, since this generalizes less far than I would like. The third case stays uncaught: the benchmark whose cost per task could not be computed at all. The harness caught that one at report time by refusing to print a cost with no rate row behind it and printing the warning in its place. Recording more would have made no difference there, because the missing record was the whole problem. Fail-closed at the point of publication turns out to be its own mechanism, sitting beside provenance on the row rather than following from it, and we appear to need both.