Last week I made the case for measuring whether your routing actually works — comparing output distributions per route, computing cost per successful task instead of per token, versioning the harness. If you did that, you've already beaten most teams.
But even teams that route correctly are usually routing at the wrong granularity. The decision they make is "is this request hard?" — and then the whole request goes to frontier or to cheap. The request is the wrong unit.
The expensive part was never the model
A recent write-up described cutting token use roughly 42x on a code-editing workload. The tempting read is "they found a smaller model." They didn't. The saving came from deleting the agent's explore-reread-redecide loop — the tokens spent figuring out what to do rather than doing it. The model barely changed. What changed was which job the tokens were performing.
That's the tell. When a workload gets dramatically cheaper without changing models, the lever wasn't model choice. It was that someone stopped spending tokens on a job that didn't need a frontier model at all.
Most "hard" requests are mostly mechanical
Look at almost any production request. A single task that feels hard — summarize this thread, draft a reply, extract structured fields from a doc — is usually a thin slice of genuine reasoning wrapped in a lot of mechanical work: pull the right context, classify it, format the output, validate the schema, retry on parse failure.
Route the whole request to a frontier model and you pay frontier prices for the 80% that was extraction and formatting. Route the whole request to a cheap model and you gamble the 20% that actually needed the frontier.
Per-request routing forces you to answer a question you can't reliably answer up front: "how hard is this?" Difficulty is a property you discover, not one you know at the door.
Route by token-purpose, not by difficulty
The more useful axis is what the token is for:
- Instruction and system tokens. Stable, repeated, cacheable. They barely benefit from a frontier model — and they're the easiest to keep on the cheap route.
- Structured extraction, classification, formatting. Routine by definition. These are where most of your volume lives, and where cheaper models — typically 70%+ cheaper, often up to 90%+ on China models — are indistinguishable in output.
- First pass vs. escalation. You can't know a request is hard before you generate. A gate that decides after a cheap first pass fails — and only escalates the part that failed — spends frontier tokens only where they earn it.
- The genuinely ambiguous slice. Keep frontier here. But make it the slice, not the request.
Why you can't do this by hand
You can't easily split a prompt into "tokens that are formatting" and "tokens that are reasoning" before you send it. That's the job of the call boundary. A routing gateway sits there, sees each request, and assigns the job — so the 80% mechanical portion never reaches a frontier model, and the thin hard portion does.
And you can only trust it if you can see it. This is where last week's point lands: route by job, then measure cost per successful task including retries and escalations, with request-level tracing. A gateway that hides the blend is just a black box you're told is cheaper.
A realistic shape
Take a support pipeline: classify the ticket, draft a reply, validate against a template. At frontier prices the extraction and formatting alone might run six figures a year. Split by job — cheap model for classify and format, frontier only for the draft's hard cases, escalation after a failed first pass — and the bill typically drops 70%+. On China models the same split can reach 90%+ cheaper. The saving only counts, of course, if you measured cost per successful task and the retries didn't eat it back.
Where this matters most
For teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules that US-default tooling tends to ignore, the unit of routing is also a unit of control. A gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned, exposing one OpenAI-compatible endpoint over 25+ models, lets a KL-based team route by job without shipping data out of the region or stitching five providers together.
The habit I'd steal
Next time you reach for "is this task hard?", stop. Ask "what job is this token doing?" Route the mechanical jobs to cheap models, keep frontier for the slice that earns it, escalate after the cheap pass fails — and trace every request so you can prove the blend worked. Routing by difficulty guesses at the door. Routing by token-purpose decides after it reads the work.
If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you route is the blend you can actually prove.
Top comments (0)