Disclosure: I maintain Lynkr, the router being benchmarked, so read everything here with that in mind. The mitigation: every number in this post comes from RouterArena's automated evaluation pipeline, run by their CI on their infrastructure, not by me — including the numbers that make us look mediocre.
Every LLM router's README — ours included — makes the same claim: it sends easy queries to cheap models and hard queries to good ones, and saves you money without hurting quality. Almost none of them attach evidence. The numbers that do exist are self-reported, measured on datasets the vendor picked, with baselines the vendor chose.
So when RouterArena showed up — an open, standardized benchmark for LLM routers out of the RouteWorks group (paper, leaderboard) — we submitted Lynkr to it. This post is what we learned, including the metrics where we did badly, because a benchmark result you only quote selectively is just marketing with extra steps.
What RouterArena actually measures
RouterArena evaluates a router over 8,400 queries spanning 9 domains and 44 categories at three difficulty levels, and scores it on five axes:
- Arena Score — the headline number, a combined accuracy-vs-cost tradeoff.
- Accuracy — did the model your router picked answer correctly?
- Cost — average $ per 1K queries, at real provider prices.
- Optimality — three sub-metrics against the oracle: how often you picked the cheapest correct model (Opt.Sel), how close your spend was to optimal (Opt.Cost), and how close your accuracy was to the best achievable with your pool (Opt.Acc).
- Robustness — do trivial rephrasings of the same query flip your routing decision?
One rule matters more than the metrics: RouterArena is evaluation-only. Any router component trained, fitted, or tuned on their data or labels gets rejected, and violations found later get withdrawn. Keep that rule in mind — it comes back at the end of this post.
What we submitted
We submitted Lynkr's untuned production default — the same complexity scorer and the same tier boundaries that ship in the repo. The adapter is ~70 lines of Python that sends each query to a live Lynkr instance's /routing/analyze endpoint and maps the returned tier to a model. No benchmark-special code path: the endpoint runs the exact intent scorer the live proxy uses (local embeddings, no LLM call in the routing decision itself).
The model pool, all via OpenRouter:
| Lynkr tier | Model |
|---|---|
| SIMPLE | gpt-oss-120b |
| MEDIUM | Qwen3-235B-A22B |
| COMPLEX / REASONING | GLM-4.7 |
Running all 8,400 queries cost $2.46 total. That's the entire eval bill, which says something on its own about where inference prices are in mid-2026.
The results
From RouterArena's automated evaluation on our submission PR:
| Metric | Lynkr |
|---|---|
| Arena Score | 67.65 |
| Accuracy | 68.41% |
| Cost per 1K queries | $0.29 |
| Robustness | 92.38 |
| Opt.Acc (accuracy vs. optimal) | 84.48 |
| Opt.Cost (cost efficiency vs. optimal) | 16.08 |
| Opt.Sel (optimal model selection) | 10.97 |
On the official leaderboard (our submission merged 2026-07-23), that lands mid-table — 15th of 27 routers.
The parts we'll be quoting
We beat GPT-5 used as a router — at 34× lower cost. The leaderboard includes GPT-5 itself as a routing baseline: Arena score 64.32 at $10.02 per 1K queries. Lynkr scores 67.65 at $0.29 per 1K. A local intent scorer making the decision without any LLM call outperforms asking a frontier model to route — which is the entire thesis Lynkr is built on, so we're relieved the benchmark agrees. Several well-known systems (NotDiamond at 57.29, RouteLLM at 48.07, both RouterBench baselines) also land below us.
Robustness 92.38 is top-five territory. Most leaderboard routers sit between 22 and 72 — meaning a rephrased query frequently flips their model choice. Lynkr's routing decision survives rephrasing 92% of the time. For an interactive tool where users iterate on prompts, decision stability is arguably worth more than a couple of accuracy points: a router that sends your reworded follow-up to a different model mid-conversation is a router you turn off.
The parts we'd rather hide (but won't)
Opt.Sel is 10.97. When multiple models in our pool could answer a query correctly, we picked the cheapest correct one about 11% of the time. Lynkr routes conservatively — when in doubt, it escalates a tier. That's a deliberate live-serving bias (a wrong cheap answer costs more user trust than an unnecessarily good one costs dollars), and $0.29/1K shows the absolute spend stays low. But the oracle comparison is unambiguous: there was money on the table we didn't pick up.
The accuracy gap to the top is real. Cross-Router leads at 78.14% accuracy; we're at 68.41%. Part of that is the pool (three self-hostable open-weight models — no frontier closed models to escalate to), but part is genuinely the router. Mid-table is where we are, not where we'd spin it.
What the benchmark taught us — and why we won't act on all of it
Reading our failure cases was the most valuable part of the exercise. The clearest pattern: our SIMPLE→MEDIUM boundary is conservative — a chunk of queries scored just past the boundary and got a mid-tier model when the cheap tier would have answered correctly.
So we prototyped a shifted boundary and ran it on a ~10% subsample locally: Arena score 69.93, accuracy 71.07%, Opt.Sel jumping from ~11 to ~66. A two-point arena gain from moving one threshold.
That config will never appear on the leaderboard. It was diagnosed from RouterArena's own failure cases, which makes it exactly what their evaluation-only rule exists to prevent — a router fitted to the eval. The leaderboard entry is the untuned default, and stays that way. The honest generalization claim is much narrower: the benchmark showed us which knob matters, and we'll validate new defaults on our own traffic, not on theirs.
If you maintain a router: this is the trap. The moment a public benchmark exists, the incentive is to tune against it, and every point you gain that way is a point of overfitting you ship to your actual users.
Caveats, so you can weigh this properly
- RouterArena is single-turn Q&A-style evaluation. Lynkr's primary workload is multi-turn coding agents, which RouterArena doesn't measure — agentic routing (tool-call density, context accumulation) is a different problem than classifying one prompt.
- Our score depends on our chosen pool. A pool with a frontier model on top would score differently in both directions (higher ceiling, higher cost).
- Leaderboard positions move; check the live leaderboard rather than trusting this snapshot.
If you're evaluating routers — including ours — ask every vendor for their RouterArena number. It costs a few dollars and a PR to get one. "We haven't submitted" is also an answer.
Links: RouterArena paper (arXiv:2510.00202) · leaderboard · our submission PR · Lynkr
Top comments (0)