Route most requests to the cheapest model that passes your evals, and send a request to the expensive model only when a cheap, checkable signal says the first answer is doubtful. That one design decision usually cuts the blended inference bill by more than half without a measurable quality drop. Routing "by vibes" (long prompt goes to the big model, short prompt goes to the small one) gives you the worst of both.
Why does everything end up on the flagship model?
Because the demo was built on it and nobody went back. During the pilot the team was optimizing for "does it work," the flagship worked, and the model id got hard-coded. Six months later the bill is a line item the CFO asks about, and switching feels risky because there's no eval set to prove the cheaper model is fine.
The math is worth writing down. Say the small model costs a tenth of the flagship per token. Blended cost is then whatever share the cheap tier can actually carry:
| Share the cheap tier handles | Blended cost vs flagship-only | Escalation rate |
|---|---|---|
| 0 percent | 1.00 | none |
| 70 percent | 0.37 | 30 percent |
| 85 percent | 0.235 | 15 percent |
Illustrative, at a tenth the price per token and before caching. Those are the numbers that make an automation case hold up once you look at the real bill for running it instead of the build quote.
What signal decides escalation?
An escalation signal is any cheap, checkable fact about the first answer that predicts it is wrong: a failed schema validation, a disagreeing checker, a weak retrieval score, or a task class known to be hard. The one signal I don't trust is the model rating its own confidence.
It's poorly calibrated and it drifts between versions. Signals that have held up for us:
- Structured-output validation failing (schema errors, an invalid enum, a missing field).
- A cheap checker disagreeing: a rules pass, or a second small-model call asked a narrow yes/no question about the first answer.
- Retrieval quality below a threshold, when the task depends on retrieved context.
- An explicit task class from a lightweight classifier: "refund policy question" stays cheap, "multi-step account change" escalates immediately.
- The user retrying or giving a thumbs-down, which triggers a re-run on the bigger model.
Each of those is checkable in code and testable on its own, and each one gets logged with the request so you can see later why a call escalated.
Where does routing go wrong?
Routing goes wrong when the rule measures something other than difficulty. Input length and user tier both correlate with cost, and neither says how hard a request is. Aggregate quality metrics then look fine while one specific slice quietly degrades, and nobody spots it until somebody reads the complaints.
Our own version of that mistake, on an internal support tool: we routed by input length. Long tickets went to the flagship, short ones went cheap. Then we read the complaints and saw they clustered on short tickets, because short tickets were the ambiguous ones ("still not working"), and the cheap model was confidently guessing. Routing by task class plus validation outcome fixed it in a day.
Order matters as much as the rule. The path a request takes now:
- Check the cache, exact match first, then a normalized version of the input.
- Classify the task with a lightweight classifier and pick a tier.
- Call the cheap model and validate the output against its schema.
- Escalate to the flagship only when one of the signals above fires, and log which one.
- Record the route, the outcome and the cost on the request.
There is no cheaper model than the one you don't call, which is why the cache sits at step one.
How do you keep the router honest?
Four habits keep a router from rotting: an eval set per route, the route decision logged on every request, an escalation rate somebody watches, and a re-run of the whole comparison whenever a provider ships a new model. Skip them and the cheap tier drifts without anyone noticing.
The escalation rate is the one people skip. If 60 percent of requests escalate, your cheap tier is theatre and you're paying for two calls per request. Re-run the comparison on every provider release, because this year's cheap tier often beats last year's flagship and your thresholds were tuned for the old pair.
We ship a lot of AI features at Shanti Infosoft for mid-sized companies where the monthly inference bill matters as much as the accuracy number, and routing is the first lever we reach for. It's also the first thing we look at when auditing an AI integration somebody else built. Rarely glamorous, but it tends to decide whether a feature survives the budget review. If you want a second opinion on your own tier split, a short call is usually enough to find the obvious win.
What percentage of your requests truly need the most expensive model you're paying for?
Sagar Jain, technical co-founder at Shanti Infosoft, a CMMI Level 5 firm, spends most of his week on the cost side of AI features rather than the demo side.
Top comments (0)