The first LLM router metric I check is the missing-row count

A router can choose a model for every prompt and still leave you with a very incomplete evaluation. Those are different achievements. I want to see both before I trust the quality chart. 🔎

AutoBench's new router audit makes that distinction unusually easy to inspect: it publishes the decisions as well as the scores. The author reports that none of the commercial settings in its headline comparison beats a particular static model on quality and cost together. That is the author's result, not a fresh model test from me. [1]

My contribution here is smaller and reproducible: read the released decision ledger, count which decisions have scores, and check whether a supposedly adaptive policy mostly picks the same model.

Original chart computed from the pinned AutoBench decision export. Unscored means unavailable for scoring in this ledger; it does not establish a failed request.

A missing score needs its own denominator

In the exported azure_cost pass, all 252 decisions name a model, but only 35 have scored=True. Its top model receives 149 decisions. In orca_constrained_quality, the corresponding counts are 245 scored and 252 top-model selections, again out of 252 decisions. [2]

The first pass raises a coverage question. The second raises a routing question. A single quality percentage hides that distinction.

Imagine printing a restaurant review after tasting only the starters. The score may describe the starters perfectly. It still needs a label. 🍽️

For a benchmark, I would keep a decision count, a scored count and a missing-cell explanation next to every quality number. A score calculated over available cells is a conditional result. It should not silently become a statement about the entire workload.

That does not mean every unavailable cell is a bad answer. The provider may have selected a model outside the stored score matrix. Penalizing every such cell as a miss is another evaluation choice, not an automatic repair.

Print the ledger before the leaderboard 🧪

Save the following as audit.py and run python3 audit.py with Python 3.10 or newer and internet access. It uses the standard library, pins the dataset revision and checks the downloaded file's digest before parsing it.

import csv
import hashlib
import io
from collections import Counter, defaultdict
from urllib.request import urlopen
revision = '43755180be7d8097068982d7a3f37fc04cd15379'
url = f'https://huggingface.co/datasets/AutoBench/router-benchmark-2026-10/resolve/{revision}/decisions.csv'
raw = urlopen(url).read()
assert hashlib.sha256(raw).hexdigest() == 'd000e5310744ae15f9c9d572ee1f88f84858fc0ba01fb54eef1e6a73a505d540'
passes = defaultdict(list)
for row in csv.DictReader(io.StringIO(raw.decode())):
    passes[row['pass']].append(row)
assert sum(map(len, passes.values())) == 7046
assert len(passes) == 28
collapsed = 0
for rows in passes.values():
    assert len({r['prompt_id'] for r in rows}) == len(rows)
    top = Counter(r['selected_model'] for r in rows).most_common(1)[0][1]
    collapsed += 10 * top >= 9 * len(rows)
print('exported passes:', len(passes), '| collapsed:', collapsed)
for name in ['orca_constrained_quality', 'azure_cost', 'nd_subset_k0', 'or_auto']:
    rows = passes[name]
    top = Counter(r['selected_model'] for r in rows).most_common(1)[0][1]
    scored = sum(r['scored'] == 'True' for r in rows)
    print(f'{name}: scored {scored}/{len(rows)}, top-model picks {top}/{len(rows)}')

The output is:

exported passes: 28 | collapsed: 9
orca_constrained_quality: scored 245/252, top-model picks 252/252
azure_cost: scored 35/252, top-model picks 149/252
nd_subset_k0: scored 248/252, top-model picks 252/252
or_auto: scored 245/252, top-model picks 248/252

✅ Here, nine of the 28 exported passes send at least 90% of decisions to one model. The threshold uses integer arithmetic, so a floating-point boundary cannot change the classification.

There is an easy trap here: the export's 28 passes are not the blog's 17 headline settings. The dump also contains secondary and reference passes. My count describes the whole exported ledger; it does not correct or contradict the author's narrower count. [2]

A constant can be a good policy

A concentrated policy is not automatically useless. If one model is suitable for the workload, consistently choosing it may be sensible. Reliability requirements, contractual constraints and availability can also narrow a usable pool.

What concentration changes is the comparison I want. If a policy nearly always selects model A, compare it with always selecting A. Then ask what the remaining decisions contribute. Merely showing that it beats random selection does not isolate the value of adapting to the prompt. 💡

I would also compare routers on the same scored prompt set before interpreting a quality difference. Otherwise a router evaluated on difficult questions and another evaluated on an easier subset can trade places without either improving. Keep the missing rows visible, and report the full workload separately from that matched subset.

What this check does—and doesn't—establish

This is an offline audit of published decisions, not another round of router or model calls. I did not reproduce AutoBench's judges, bootstrap procedure or cost ranking. The source says its router calls predate this release; the ledger is not a guarantee about current vendor behavior. [2]

The useful habit survives those limits: print coverage and concentration before celebrating a quality score. A routing graph gets much more interesting once you know which decisions it actually describes.

References

[1] Peter Kruger, Do LLM Routers Actually Route?.

[2] AutoBench, router benchmark dataset and methodology notes. Counts and original chart derived from its CC BY 4.0 decision table. The script pins the exact revision.

AI disclosure

AI was used to draft the article.