Your agent's token bill is probably 5x higher than it needs to be — not because the models are expensive, but because of routing discipline. Most teams wire every call to one frontier model and call it a day. This post is the practical fix: how to send the easy 80% of agent calls to cheap models, keep frontier for the hard 20%, and not lose a point of quality doing it.
Everything below uses real numbers from a gateway that exposes 22 models across 8 providers behind one OpenAI-compatible API.
The default that's costing you
Here's the shape of a typical agent loop:
classify the request
extract structured fields
pick a tool
call the tool, parse the result
summarize what happened
decide the next step
draft the reply
Of those, maybe one or two steps actually need frontier-level reasoning. The rest are classification, extraction, and formatting — work that a fast, cheap model does indistinguishably from a flagship. Yet most setups send all of it to gpt-5.5 (or whatever their default is) because it's the path of least resistance.
That's the tax. You're paying flagship prices for steps that don't need it.
The 80/20 split, with the math
Route by task difficulty, not by model loyalty:
Easy 80% — classification, extraction, summarization, short tool-formatting, routing decisions → a fast China-model tier
Medium 15% — coding, planning, multi-step reasoning → a strong reasoning tier
Hard 5% — the genuinely gnarly frontier cases → flagship, reserved
A blended-cost estimate (input tokens, per 1M):
80% × ¤160 (cheap tier input) = ¤128
15% × ¤560 (reasoning tier) = ¤84
5% × ¤7000 (flagship input) = ¤350
blended input cost ≈ ¤562 / 1M
vs. all-flagship = ¤7000 / 1M
savings ≈ 70%+ in this example
Even with a more conservative split, 70%+ cheaper is the floor — and quality doesn't move, because the hard calls still go to frontier.
The trap: why "just use the cheapest model" backfires
Naive cost-cutting picks one cheap model for everything. That fails for two reasons:
Quality cliffs. A ¤160/1M model is great at extraction and terrible at planning a 12-file refactor. Push hard tasks down and users notice.
Cache invalidation (the silent tax). This is the one almost nobody budgets for. Long agent conversations re-send the system prompt + tool definitions on every turn. If you switch providers mid-conversation, you reset prompt-cache affinity and re-pay the full prefix each turn. A conversation that should be cheap suddenly isn't.
The win isn't "use a cheap model" — it's routing discipline with cache affinity intact: pin the easy 80% to a stable cheap model so the cache stays warm, and only escalate to frontier for the calls that earn it.
What a real routing layer looks like
With a unified gateway, this is roughly ten lines. One API, models swapped by model: — no per-provider SDK juggling:
def route(task):
if task.complexity == "easy": # classify, extract, summarize
return "TokenLat-deepseek-v4-Flash" # ¤160/1M input
if task.needs_reasoning: # code, plan, multi-step
return "TokenLat-deepseek-v4-Pro" # ¤560/1M input, ¤10/1M cache read
return "chatgpt-5.5" # frontier, only the hard 5%
resp = client.responses.create(
model=route(task),
input=task.prompt,
)
Or skip the router entirely and let the gateway decide per call:
resp = client.responses.create(
model="auto", # gateway picks the right tier per request
input=task.prompt,
)
model: "auto" is the zero-config version of the same idea — the gateway applies routing policy so you don't hand-roll it.
Real numbers from a 22-model gateway
A unified gateway should show you its pricing up front. Per 1M tokens, in credits (¤), here's a slice (input / cache-read):
TokenLat-deepseek-v4-Flash — cheap / fast tier → ¤160 / ¤40
TokenLat-qwen3.5-plus — general tier → ¤130 / ¤20
TokenLat-deepseek-v4-Pro — reasoning tier → ¤560 / ¤10 (lowest cache-read)
TokenLat-glm-5.1 — Chinese-stable → ¤950 / ¤160
TokenLat-kimi-k3 — flagship (China) → ¤3000 / ¤300
gemini-3.1-pro — frontier → ¤2800 / ¤300
chatgpt-5.5 — frontier → ¤7000 / ¤700
The headline gap: cheap-tier input is ~44x cheaper than flagship input. That's the entire 80/20 argument in one line.
Cache-hit pricing — the agentic loophole
For agentic systems, the input price is almost a distraction. What matters is the cache-read price, because the static prefix (system prompt, tool schemas, retrieved context) gets re-sent every turn.
TokenLat-deepseek-v4-Pro caches reads at ¤10/1M
TokenLat-qwen3.5-plus at ¤20/1M
So a long-running agent loop that pins a stable model and keeps its prefix cached can serve most turns at the cache-read rate — a fraction of the input rate. The providers that reset your cache on every switch quietly erase this saving. Cache affinity is a routing decision, not an afterthought.
Make it visible
You can't optimize what you can't see. Every request through a proper gateway should leave a trace:
REQUEST → AUTH → ROUTE(auto→text-pro) → RESPONSE → METER
region: SEA | status: 200 OK | latency: 842ms | tokens: 1,284 | cost: ¤0.0048
Five stages, one line, per call. When a bill spikes, you see which model, which route, and which request — instead of staring at a monthly total and guessing.
Don't guess — measure quality
The one thing this playbook assumes: you actually check that the cheap tier is good enough. Route a sample of real tasks through both tiers, score the outputs on your success criteria, and only then commit the split. Most teams find the quality cliff is far smaller than they feared — and the ones where it isn't, they've already reserved frontier for.
The takeaway: the savings aren't in hunting for a cheaper model. They're in routing the easy 80% to a stable cheap tier, keeping cache affinity, and escalating only the hard calls to frontier. Do that and 70%+ cheaper is routine — with quality intact.
If you want to try this without wiring up eight providers yourself: TokenLat is a unified AI gateway for Malaysia & Southeast Asia — one OpenAI-compatible API across 22 models, with auto routing and per-request cost traces. The model list above is live at tokenlat.com.
What's your current cost per agent-task — single digits or scaling faster than you'd like? Curious how the 80/20 split would land on your workload.
Top comments (0)