Stop Optimizing for the Cheapest Token. Optimize Quality-per-Dollar.

#llm #ai #api #machinelearning

Originally published on the TierUp blog. The 2026 evidence on LLM routing: why both "always the flagship" and "always the cheapest" leave money on the table.

For the first couple of years of the LLM API era, teams picked a model the way they picked a database: once, emotionally, and then defended the choice in perpetuity. Some hardcoded the frontier model because "quality matters." Others hardcoded the cheapest model because "it's mostly good enough." Both camps are leaving money — or capability — on the table, and in 2026 the third-party evidence for that has gotten hard to ignore.

The price spread makes single-model choices indefensible

The gap between tiers is not 2x. Digital Applied's June 2026 routing guide puts current input pricing at roughly $0.44/M tokens for DeepSeek V4, $1/M for Claude Haiku 4.5, $3/M for Sonnet 4.6, $5/M for GPT-5.5, and $25/M for Opus 4.8 — with the full spread from cheapest input to priciest frontier output running around 100x.

A 100x spread means the routing decision is worth more than almost any other optimization you can make. Prompt caching might save you 50–90% on repeated prefixes; batching might save 50%. Sending a "reformat this JSON" request to a model priced 100x below the frontier saves 99%.

And the spread is a moving target. Epoch AI's analysis found that the price to reach a fixed capability level has been falling between 9x and 900x per year depending on the benchmark, with a median around 50x annually. Concretely: the capability you're paying frontier prices for today will be available at mid-tier prices in months. A hardcoded model choice is a depreciating asset. Gartner now projects that inference on a trillion-parameter model will cost providers over 90% less by 2030 than in 2025.

The research: most queries don't need the frontier

This isn't just a pricing observation — it's an empirical one about workloads. The peer-reviewed RouteLLM work (cited in Digital Applied's guide) showed a trained router achieving 85% cost savings on MT Bench while retaining 95% of GPT-4 quality, with its matrix-factorization router needing the frontier model on only about 14% of queries. The authors' principle is worth framing: all queries that can be handled by weaker models should be routed to those models.

Production numbers line up with the lab. Eden AI's 2026 router comparison reports routing reduces LLM costs by 30–85% depending on workload and quality requirements, and Digital Applied cites teams seeing 40–85% bill reductions, with even a crude 70/30 cheap-to-frontier split yielding roughly 67% savings.

Notice what "quality-per-dollar" is not: it is not "use the cheapest model." On the hard 14–30% of your traffic, the cheap model fails, you retry, you burn user trust, and your effective cost per successful outcome exceeds what the frontier model would have charged. Cheapest-token optimization and best-benchmark optimization are the same mistake in opposite directions — both evaluate the model in isolation instead of evaluating cost per solved task.

The honest caveats

Routing is not free lunch, and it's worth stating the failure modes plainly:

Silent quality regression is the real risk. Digital Applied's guide describes degraded answers surfacing in customer tickets days later rather than on a dashboard. The mitigation is unglamorous: an eval suite of a few hundred representative cases that gates any routing-policy change, exactly like a test suite gates a deploy.
Router overhead matters, but less than you'd think — rule-based routing adds under 1ms and even ML classifiers add 50–100ms against typical 500–2,000ms inference times.
Some workloads shouldn't be routed. If 95% of your traffic genuinely needs frontier reasoning, a router is complexity without payoff. Measure first.

Where TierUp fits

This thesis is why TierUp exists. Instead of hardcoding a model ID, you pick a performance tier and we route each request to the model currently offering the best quality-per-dollar at that tier — repriced as the market moves, so Epoch AI's 50x-per-year deflation shows up on your bill instead of your provider's margin. Same API shape, below-retail pricing, and no vendor archaeology every time a new model ships.