Every few weeks the same post does the rounds. Someone's routing layer cut their LLM bill by 60%, "with no drop in quality", and here's a lovely chart to prove it.
Except it doesn't prove anything. The eval set is private. The grading is private. The actual model responses that produced that score are never published. You're being asked to trust a screenshot, and we've been in I.T. long enough to know roughly what a screenshot is worth.
So we built the thing that was missing. It's called OmnisBench, it's open source (Apache 2.0), and its whole job is to measure how good LLM routing actually is, in a way you can check for yourself.
What routing is, and why it should save money
If you're not knee deep in this yet: an LLM router looks at each request and sends it to the cheapest model that can still do the job, rather than firing everything at one expensive frontier model. Easy prompts go to something small. Hard ones go to the big model. In theory you keep most of the quality and pay a fraction of the bill.
That's the theory. OmnisBench measures the practice.
The number we care about most is what we call the oracle. For every task, the oracle picks the cheapest model that actually got it right. We know which ones did, because we ran them all. That is not a router you can ship. It's the ceiling. It's the best any router could possibly do on a given set of tasks, and it tells you how much money is genuinely on the table before anyone starts making claims.
The first run
We ran it over 364 tasks: HumanEval (164 coding problems, graded by their own unit tests) and GSM8K (200 maths problems, graded on the answer). The model pool was Claude Opus 5, GPT-5, Claude Haiku 4.5 and GPT-5-nano.
Here is what came out.
| Policy | What it does | Task success | Cost per 1,000 requests |
|---|---|---|---|
| oracle | cheapest model that actually solved each task | 99.7% | $0.62 |
| always-frontier | everything to Claude Opus 5 | 99.2% | $6.25 |
| random | pick a model at random | 96.2% | $4.01 |
| always-cheapest | everything to GPT-5-nano | 94.5% | $0.43 |
So the headline writes itself. Ideal routing hit 99.7% success for roughly 90% less than sending everything to the top model. Very nice.
Now let me talk you out of it a little.
The honest bit
Two things you will not see on most of those 60% charts.
First, the oracle is a ceiling, not a product. It cheats. It knows, after the fact, which model got each task right, and no live router has that luxury. The real score for a router you can actually deploy is how close it gets to that oracle line, and the gap is the interesting part. Anyone quoting an oracle-style number as their shipping product is selling you the ceiling.
Second, and this is the one that surprised us: the cheapest model on its own already scored 94.5%. On this particular set of tasks GPT-5-nano is just quietly good. So routing is not buying you a magic 60%. It's buying you the last few points of quality without paying frontier prices for all of it. That is still very much worth having. It just isn't the fairy tale.
We would rather tell you that up front than have you find out after you have quoted the shiny number to your boss.
The part that actually matters: you can check it
This is the whole reason OmnisBench exists. It does not ask you to trust the table above. It publishes the actual model response for every task, and ships a command that re-grades the lot offline.
pip install -e .
python scripts/prepare_datasets.py # HumanEval + GSM8K
python -m omnisbench.cli run --config configs/v0.yaml --run runs/mine
python -m omnisbench.cli report --run runs/mine
python -m omnisbench.cli verify --run runs/2026-08-19 # re-grades the published run, no API calls
verify re-runs the graders against the published responses and rebuilds the entire leaderboard, cost included, without calling a single API or needing a key. Tamper with one stored answer and it fails. That is the point. If our numbers are wrong, you can prove it in about a minute, and we would genuinely like you to try.
Have at it
The results, the method and the honest caveats are all on the site, and the code is on GitHub:
- Results: https://omnisbench.fortitude-omnis.group/
- Code: https://github.com/Fortitude-Group/OmnisBench
We also build an actual open router, OmnisRouter, which this benchmark grades on every release, so it can't quietly regress and hope nobody notices. It is live now at https://omnisrouter.fortitude-omnis.group/.
If you can break the numbers, poke a hole in the method, or you just reckon we've measured the wrong thing, tell us. That is what open and verifiable is for.
OmnisBench is built by Fortitude Omnis. We make small, sharp tools, and we try not to lie in our own marketing.
Top comments (9)
Oracle-as-ceiling instead of oracle-as-product is the honest move most 60% posts skip. One more check worth running: HumanEval and GSM8K are old enough that some models in the pool may have trained on graded examples from them. A router benchmarked on contaminated tasks can still overstate the real ceiling even when the methodology publishing every response is completely clean.
Great call out. We'll come up with a plan for this. That was in the back of my mind but you've bumped it to the top if the list :) THANKS!
Also, just discussed this with a colleague , we can split the suite into likely-contaminated (HumanEval, GSM8K) and fresh, using date-filtered LiveCodeBench and postcutoff maths problems, then report the two side by side.If the routing story only holds on the old sets, that shows up straight away :) . Second, a contamination probe on the pool before each run, so the fresh split is doing real work. If the oracle gap moves on the fresh split, that is the honest number and we will lead with it. LiveCodeBench was already on the roadmap, so this mostly moves it up the queue. Thanks again for the nudge. More pressure 😂
That splits cleanly. One addition worth considering: publish the contamination probe's actual value next to the split, not just a pass or fail on it. A binary contaminated versus clean call hides how close to the threshold the pool sits, and readers can't independently judge how much to trust the fresh-split number without seeing that.
Right, we did the thing you both pushed for and built the fresh split (LiveCodeBench, dated after the training cutoff), then ran it.
@jugeni i, you were right about contamination. On the old benchmarks the cheap model looks near-perfect, but on the fresh set it drops from 90% to 60%. Those numbers were flattering it.
@deanlee , your escape-rate point held up too. On fresh tasks ideal routing has to escalate to the frontier a fifth of the time instead of never, and routing's gain over the cheap model goes from ten points to thirty-three. It earns its keep exactly where the data's clean.
The twist was our own fault. The first fresh run looked catastrophic until I read the responses and found the frontier model had emitted nothing at all on the hard problems. Our output budget was truncating the reasoning models mid-thought, so we'd built a benchmark that measured how good models are at running out of room. Fixed the budget, re-ran, and those are the corrected numbers.
Full writeup, with the corrected split and the whole embarrassing detour: dev.to/fortitudeomnis/we-built-a-b... and the run's re-gradable with omnisbench verify. Thanks for the nudge, it genuinely made the thing better :)
This is the right caveat to put next to routing numbers. The impressive part is not that an oracle is cheap. It is whether the policy survives when task labels are messy and the cheap model fails in ways the router cannot cheaply observe. Publishing the stored answers makes that argument much easier to audit.
exactly - visibility is the premium, honest answer here :)
The useful bit here is treating the oracle as a ceiling rather than a product metric. A lot of routing claims hide exactly that gap. I would also want to see how the frontier escape rate changes when the task mix moves from benchmarks to messy internal prompts, because that is where the savings usually leak.
Also, that is exactly the number we care about, and it is the one benchmarks flatter. On HumanEval and GSM8K the labels are clean and the cheap model fails loudly, so the oracle-to-router gap looks tame. On mssy internal prompts the cheap model fails quietly, in ways a router with no grader at inference time cannot see, and that is where the escalation rate and the realised savings drift.
So the next suite is deliberately messier: longer, more agentic, less tidily labelled than a benchmark. We will report the frontier escape rate and the realised-router-versus-oracle gap on it, alongside the headline ceiling rather than in place of it. The oracle tells you the money on the table. That suite tells you how much a shippable policy actually collects once it has to decide blind. And because every stored answer is published, you can argue with our grading on the messy set too, which is the point. This is turning out to be trickier than first though!
Hope that makes sense, we'll probably get round to this later this week.