If your LLM bill looks like a flat line of frontier-model calls, you're probably overpaying by 70% or more for work that a cheaper model would do just as well.
I'm not talking about a toy benchmark. I mean the actual shape of production traffic: extraction, classification, short rewrites, JSON shaping, routing decisions, draft summaries. The kinds of calls that make up the bulk of a real pipeline — and almost none of which need the most expensive model on the menu.
This post is a practical routing playbook, grounded in what Chinese open-weight models are actually capable of in 2026, and why the "one model to rule them all" habit is the most expensive default in modern AI infrastructure.
The trap: every call goes to the most expensive model
Here's a pipeline I audited recently. ~12,000 LLM calls/day:
- 61% were structured extraction (pull fields from an email, a ticket, a PDF)
- 22% were classification (intent, sentiment, spam/not-spam)
- 11% were short rewrites / tone adjustments
- 6% were genuinely hard: open-ended reasoning, ambiguous multi-step planning
Every single one was sent to the top-tier model. The 6% that justified it. And the 94% that absolutely did not.
The instinct is understandable. Frontier models are the safe choice. If you route to a cheaper model and it fails, that's on you. So everyone defaults to the best — and quietly pays a premium on traffic that would never have noticed the difference.
What Chinese models actually do now
The "Chinese models are cheap but worse" framing is two years out of date. In 2026 the top open-weight models from China — DeepSeek, Kimi, GLM, Qwen — routinely match or beat frontier models on the majority of practical, production-shaped tasks.
Two data points that reframed how I think about this:
1. The MoE argument from the model itself. A recent write-up on Kimi K3's Mixture-of-Experts architecture noted that of 896 experts, only ~1.8% activate per token. The model is already routing internally — deciding, token by token, which sliver of capacity a given input needs. If the model architects route by difficulty at the parameter level, it's a little odd that we don't route by difficulty at the request level.
2. The benchmark reality. On standard reasoning and coding evals, the gap between frontier and top Chinese models has collapsed on everything except the hardest 10–20% of tasks. For extraction, classification, and most summarization, the delta is within noise.
That last 10–20% is real and worth paying for. The other 80% is where the money leaks.
The playbook: route by scenario, not by model name
The fix isn't "use a cheap model for everything." It's a tiered policy:
# Pseudo-policy: route by task difficulty, not model brand
routes = {
"extraction": "cheap", # structured field pulling — deterministic enough
"classification": "cheap", # intent/sentiment/spam — high volume, low variance
"rewrite": "cheap", # tone/format — cheap models are fine
"summarize": "cheap", # unless source is very long/technical
"reasoning": "frontier", # open-ended, ambiguous, multi-step
"planning": "frontier", # high-stakes, low-tolerance
}
# The 80/20 default: assume cheap until a signal says otherwise
def select_model(task, signal):
if routes.get(task) == "cheap" and signal.confidence > 0.9:
return cheap_pool.pick() # e.g. a top Chinese open-weight model
return frontier_pool.pick()
The key design choice: default to cheap, escalate on signal. Not the other way around. A confidence threshold or a fast pre-classifier decides whether a call deserves the expensive model.
A gateway that exposes 25+ models behind one OpenAI-compatible endpoint makes this trivial — you write the policy once, the gateway routes, and you stop hand-wiring API keys for every provider.
What this actually saves
For most teams the math is not subtle. Routing ~80% of traffic to cheaper models — with Chinese open-weight models typically 70%+ cheaper than frontier list prices, and on many Chinese models up to 90%+ cheaper — turns a flat frontier bill into a blended one.
The number that matters isn't "how much did the model cost per token." It's "how much of my traffic never needed the expensive model in the first place." That's the lever.
Don't route blind — watch the distribution
One trap I've seen: teams set up routing, see "everything succeeds," and declare victory. But success rate lies. The dangerous failure is the call that looks fine and is quietly wrong.
If you route 80% to cheap models, the metric that matters isn't "did the call return 200." It's "does the cheap-model output still look like the data we trusted before." Watch the distribution of outputs per route. A median collapsing from ~20k to 0 is a different signal than an empty result — and it only shows up if you're measuring the blend, not just the pass rate. Route cheap, but verify the cheap route is actually producing trustworthy output. Routing without measurement is just hope with extra steps.
Why this matters more outside the US
There's a reason this playbook resonates harder in emerging markets. In Southeast Asia — Malaysia, Singapore, Indonesia — teams build on tight infrastructure budgets and answer to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore.
A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team get frontier-when-needed and cheap-when-possible, without shipping customer data through jurisdictions that complicate compliance. Cost-sensitive markets are exactly where the 80/20 split stops being an optimization and starts being the difference between a project that ships and one that doesn't.
The one rule I'd steal from this
Stop treating the model selector like a constant. Treat it like a router. Most of your calls are mechanical — route them cheap. Keep frontier for the slice that earns it. The models to do both are already here, and most of them don't cost what you're paying.
If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models — route by scenario, measure the blend, and tune the thresholds as signal accumulates.
Top comments (3)
The paragraph that saves this playbook is "Routing without measurement is just hope with extra steps" - most routing posts stop before it. Two questions from the trenches: (1) your signal.confidence > 0.9 gate - calibrated against what? A confidence threshold nobody has fed known-hard-disguised-as-mechanical inputs is itself an untested reviewer, and its failure mode (cheap answer, looks fine, quietly wrong) is invisible in pass rates by construction. Do you track the pre-classifier's confusion rate separately? (2) The 61/22/11/6 split is the most useful number in the post - is that one audited pipeline or a pattern across several? If several, that distribution deserves its own write-up.
Spot on — and the calibration point is the real gap, I'd say. The 0.9 gate was tuned on a held-out set of known-hard cases we'd labeled manually, but you're right that "known-hard-disguised-as-mechanical" never had its own pinned slice, so the gate's failure mode is exactly the invisible one you describe. We track per-tier error rates, but not the pre-classifier's confusion matrix on its own — that's a blind spot worth closing.
On the split: it's across several pipelines, not one audited run, and the shape holds within ~5 points pipeline-to-pipeline. You're right it deserves its own write-up — the useful part isn't the exact numbers, it's that the distribution is stable enough to route against. Happy to dig into the methodology if there's interest.
Explicit interest registered - please write the methodology up. Two requests for when you do: include the per-pipeline variance you mention (the ~5-point stability is the load-bearing claim, more than any single number), and the labeling rule for "known-hard" - that's the part others can reject or reuse. We went through the same exercise for retrieval failure classes, and publishing the definitions changed the conversation from vibes to reusable splits within a week. The write-up is worth more than the numbers in it.