DEV Community

Cover image for The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality
TokenLat
TokenLat

Posted on

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

Your agent's token bill is probably 5x higher than it needs to be — not because the models are expensive, but because of routing discipline. Most teams wire every call to one frontier model and call it a day. This post is the practical fix: how to send the easy 80% of agent calls to cheap models, keep frontier for the hard 20%, and not lose a point of quality doing it.

Everything below uses real numbers from a gateway that exposes 22 models across 8 providers behind one OpenAI-compatible API.

The default that's costing you

Here's the shape of a typical agent loop:

  • classify the request
  • extract structured fields
  • pick a tool
  • call the tool, parse the result
  • summarize what happened
  • decide the next step
  • draft the reply

Of those, maybe one or two steps actually need frontier-level reasoning. The rest are classification, extraction, and formatting — work that a fast, cheap model does indistinguishably from a flagship. Yet most setups send all of it to gpt-5.5 (or whatever their default is) because it's the path of least resistance.

That's the tax. You're paying flagship prices for steps that don't need it.

The 80/20 split, with the math

Route by task difficulty, not by model loyalty:

  • Easy 80% — classification, extraction, summarization, short tool-formatting, routing decisions → a fast China-model tier
  • Medium 15% — coding, planning, multi-step reasoning → a strong reasoning tier
  • Hard 5% — the genuinely gnarly frontier cases → flagship, reserved

A blended-cost estimate (input tokens, per 1M):

80% × ¤160   (cheap tier input)   = ¤128
15% × ¤560   (reasoning tier)     = ¤84
 5% × ¤7000  (flagship input)     = ¤350
---------------------------------------
blended input cost               ≈ ¤562 / 1M
vs. all-flagship                 = ¤7000 / 1M
savings                          ≈ 70%+ in this example
Enter fullscreen mode Exit fullscreen mode

Even with a more conservative split, 70%+ cheaper is the floor — and quality doesn't move, because the hard calls still go to frontier.

The trap: why "just use the cheapest model" backfires

Naive cost-cutting picks one cheap model for everything. That fails for two reasons:

  1. Quality cliffs. A ¤160/1M model is great at extraction and terrible at planning a 12-file refactor. Push hard tasks down and users notice.
  2. Cache invalidation (the silent tax). This is the one almost nobody budgets for. Long agent conversations re-send the system prompt + tool definitions on every turn. If you switch providers mid-conversation, you reset prompt-cache affinity and re-pay the full prefix each turn. A conversation that should be cheap suddenly isn't.

The win isn't "use a cheap model" — it's routing discipline with cache affinity intact: pin the easy 80% to a stable cheap model so the cache stays warm, and only escalate to frontier for the calls that earn it.

What a real routing layer looks like

With a unified gateway, this is roughly ten lines. One API, models swapped by model: — no per-provider SDK juggling:

def route(task):
    if task.complexity == "easy":        # classify, extract, summarize
        return "TokenLat-deepseek-v4-Flash"   # ¤160/1M input
    if task.needs_reasoning:             # code, plan, multi-step
        return "TokenLat-deepseek-v4-Pro"     # ¤560/1M input, ¤10/1M cache read
    return "chatgpt-5.5"                 # frontier, only the hard 5%

resp = client.responses.create(
    model=route(task),
    input=task.prompt,
)
Enter fullscreen mode Exit fullscreen mode

Or skip the router entirely and let the gateway decide per call:

resp = client.responses.create(
    model="auto",          # gateway picks the right tier per request
    input=task.prompt,
)
Enter fullscreen mode Exit fullscreen mode

model: "auto" is the zero-config version of the same idea — the gateway applies routing policy so you don't hand-roll it.

Real numbers from a 22-model gateway

A unified gateway should show you its pricing up front. Per 1M tokens, in credits (¤), here's a slice (input / cache-read):

  • TokenLat-deepseek-v4-Flash — cheap / fast tier → ¤160 / ¤40
  • TokenLat-qwen3.5-plus — general tier → ¤130 / ¤20
  • TokenLat-deepseek-v4-Pro — reasoning tier → ¤560 / ¤10 (lowest cache-read)
  • TokenLat-glm-5.1 — Chinese-stable → ¤950 / ¤160
  • TokenLat-kimi-k3 — flagship (China) → ¤3000 / ¤300
  • gemini-3.1-pro — frontier → ¤2800 / ¤300
  • chatgpt-5.5 — frontier → ¤7000 / ¤700

The headline gap: cheap-tier input is ~44x cheaper than flagship input. That's the entire 80/20 argument in one line.

Cache-hit pricing — the agentic loophole

For agentic systems, the input price is almost a distraction. What matters is the cache-read price, because the static prefix (system prompt, tool schemas, retrieved context) gets re-sent every turn.

  • TokenLat-deepseek-v4-Pro caches reads at ¤10/1M
  • TokenLat-qwen3.5-plus at ¤20/1M

So a long-running agent loop that pins a stable model and keeps its prefix cached can serve most turns at the cache-read rate — a fraction of the input rate. The providers that reset your cache on every switch quietly erase this saving. Cache affinity is a routing decision, not an afterthought.

Make it visible

You can't optimize what you can't see. Every request through a proper gateway should leave a trace:

REQUEST → AUTH → ROUTE(auto→text-pro) → RESPONSE → METER
region: SEA | status: 200 OK | latency: 842ms | tokens: 1,284 | cost: ¤0.0048
Enter fullscreen mode Exit fullscreen mode

Five stages, one line, per call. When a bill spikes, you see which model, which route, and which request — instead of staring at a monthly total and guessing.

Don't guess — measure quality

The one thing this playbook assumes: you actually check that the cheap tier is good enough. Route a sample of real tasks through both tiers, score the outputs on your success criteria, and only then commit the split. Most teams find the quality cliff is far smaller than they feared — and the ones where it isn't, they've already reserved frontier for.

The takeaway: the savings aren't in hunting for a cheaper model. They're in routing the easy 80% to a stable cheap tier, keeping cache affinity, and escalating only the hard calls to frontier. Do that and 70%+ cheaper is routine — with quality intact.

If you want to try this without wiring up eight providers yourself: TokenLat is a unified AI gateway for Malaysia & Southeast Asia — one OpenAI-compatible API across 22 models, with auto routing and per-request cost traces. The model list above is live at tokenlat.com.

What's your current cost per agent-task — single digits or scaling faster than you'd like? Curious how the 80/20 split would land on your workload.

Top comments (1)

Collapse
 
hiper2d profile image
Aliaksei Zelianouski •

I track cost on a system with no tools and no code generation, plain text only, and I still can't predict a turn before it runs. So I estimate from past data. A model update and it's all wiped out.