For two years the reflex was simple: reach for the biggest model you can afford and call it a day. In 2026 that reflex quietly became a bug in your cost model.
The clearest signal came this summer, when a smaller, cheaper "flash"-tier model started edging out its own flagship sibling on the workload developers care about most — multi-step agentic coding — at a fraction of the price. When the fast tier wins the hard benchmark, "always use the flagship" stops being a safe default and starts being waste.
Here's how to fix it without turning your stack into a science project.
Why the reflex is expensive
Agent workloads are not one big call. A single task fans out into dozens of small ones: planning, tool selection, argument formatting, summarizing a file, deciding whether to continue. Most of those steps are easy. Routing every one of them through a frontier model is like taking a helicopter to the corner store — it works, but you are paying helicopter prices for a walk.
The trap is that the cost is invisible per call and enormous in aggregate. You never see the moment you overpaid; you just see the invoice.
The three-tier ladder
Think in tiers, not models:
- Cheap/fast tier — classification, extraction, short rewrites, routing decisions, "is this done?" checks. Most steps live here.
- Mid tier — normal reasoning, code edits, tool use with moderate context.
- Flagship tier — genuinely hard reasoning, long-context synthesis, the step where a wrong answer poisons everything downstream.
The goal is to keep the flagship tier for the 5–15% of steps that actually need it, and let the cheap tier carry the volume.
How to decide the tier per request
Two mechanisms, used together:
Static heuristics for the obvious cases. Short prompt + structured output + low stakes → cheap tier. Anything touching a large context window or a irreversible action → escalate.
Eval-gated escalation for everything else. Start at the cheap tier, and only promote to a bigger model when your evals prove the cheap tier fails on that class of input. This is the key discipline: escalation is earned by evidence, not assumed. If you can't point to an eval where the small model loses, you don't get to pay for the big one.
A simple confidence signal helps too — if the cheap model hedges, returns malformed output, or low log-probs, retry one tier up. One retry at a higher tier is still cheaper than sending everything to the flagship.
Measure it or you're guessing
You cannot route what you don't measure. Log, per step: which tier ran, token counts, latency, and a success signal. Then compute the boring but decisive number — cost per completed task, not cost per token. Teams that optimize per-token often make per-task cost worse by adding retries; the task-level metric keeps you honest.
Re-run this monthly. Model prices and capabilities move fast enough that last quarter's optimal routing table is this quarter's mistake.
Pitfalls
- Cheap-tier false economy. If a weak model fails a planning step, every downstream step inherits the error. Put your best model at the top of the plan, cheap models on the leaves.
- Silent capability drift. A model point-release can flip your routing assumptions overnight. Pin versions in evals and re-test before upgrading.
- Over-engineering the router. A 50-line heuristic plus one escalation rule beats a fancy learned router for most teams. Add complexity only when the data demands it.
The takeaway
The winning move in 2026 is not "use the best model" or "use the cheapest model." It's building a thin routing layer and an eval harness that lets you place each step on the right tier — and swap the underlying models in an afternoon when the market moves again.
Default-to-flagship felt safe because it was simple. It's still simple. It's just no longer cheap.
How are you routing model tiers in production right now — heuristics, a learned router, or still one model for everything? Curious what's working.
Top comments (0)