If your LLM bill looks like a flat line of frontier-model calls, you're probably overpaying by 70% or more for work that a cheaper model would do just as well.
I'm not talking about a toy benchmark. I mean the actual shape of production traffic: extraction, classification, short rewrites, JSON shaping, routing decisions, draft summaries. The kinds of calls that make up the bulk of a real pipeline — and almost none of which need the most expensive model on the menu.
This post is a practical routing playbook, grounded in what Chinese open-weight models are actually capable of in 2026, and why the "one model to rule them all" habit is the most expensive default in modern AI infrastructure.
The trap: every call goes to the most expensive model
Here's a pipeline I audited recently. ~12,000 LLM calls/day:
- 61% were structured extraction (pull fields from an email, a ticket, a PDF)
- 22% were classification (intent, sentiment, spam/not-spam)
- 11% were short rewrites / tone adjustments
- 6% were genuinely hard: open-ended reasoning, ambiguous multi-step planning
Every single one was sent to the top-tier model. The 6% that justified it. And the 94% that absolutely did not.
The instinct is understandable. Frontier models are the safe choice. If you route to a cheaper model and it fails, that's on you. So everyone defaults to the best — and quietly pays a premium on traffic that would never have noticed the difference.
What Chinese models actually do now
The "Chinese models are cheap but worse" framing is two years out of date. In 2026 the top open-weight models from China — DeepSeek, Kimi, GLM, Qwen — routinely match or beat frontier models on the majority of practical, production-shaped tasks.
Two data points that reframed how I think about this:
1. The MoE argument from the model itself. A recent write-up on Kimi K3's Mixture-of-Experts architecture noted that of 896 experts, only ~1.8% activate per token. The model is already routing internally — deciding, token by token, which sliver of capacity a given input needs. If the model architects route by difficulty at the parameter level, it's a little odd that we don't route by difficulty at the request level.
2. The benchmark reality. On standard reasoning and coding evals, the gap between frontier and top Chinese models has collapsed on everything except the hardest 10–20% of tasks. For extraction, classification, and most summarization, the delta is within noise.
That last 10–20% is real and worth paying for. The other 80% is where the money leaks.
The playbook: route by scenario, not by model name
The fix isn't "use a cheap model for everything." It's a tiered policy:
# Pseudo-policy: route by task difficulty, not model brand
routes = {
"extraction": "cheap", # structured field pulling — deterministic enough
"classification": "cheap", # intent/sentiment/spam — high volume, low variance
"rewrite": "cheap", # tone/format — cheap models are fine
"summarize": "cheap", # unless source is very long/technical
"reasoning": "frontier", # open-ended, ambiguous, multi-step
"planning": "frontier", # high-stakes, low-tolerance
}
# The 80/20 default: assume cheap until a signal says otherwise
def select_model(task, signal):
if routes.get(task) == "cheap" and signal.confidence > 0.9:
return cheap_pool.pick() # e.g. a top Chinese open-weight model
return frontier_pool.pick()
The key design choice: default to cheap, escalate on signal. Not the other way around. A confidence threshold or a fast pre-classifier decides whether a call deserves the expensive model.
A gateway that exposes 25+ models behind one OpenAI-compatible endpoint makes this trivial — you write the policy once, the gateway routes, and you stop hand-wiring API keys for every provider.
What this actually saves
For most teams the math is not subtle. Routing ~80% of traffic to cheaper models — with Chinese open-weight models typically 70%+ cheaper than frontier list prices, and on many Chinese models up to 90%+ cheaper — turns a flat frontier bill into a blended one.
The number that matters isn't "how much did the model cost per token." It's "how much of my traffic never needed the expensive model in the first place." That's the lever.
Don't route blind — watch the distribution
One trap I've seen: teams set up routing, see "everything succeeds," and declare victory. But success rate lies. The dangerous failure is the call that looks fine and is quietly wrong.
If you route 80% to cheap models, the metric that matters isn't "did the call return 200." It's "does the cheap-model output still look like the data we trusted before." Watch the distribution of outputs per route. A median collapsing from ~20k to 0 is a different signal than an empty result — and it only shows up if you're measuring the blend, not just the pass rate. Route cheap, but verify the cheap route is actually producing trustworthy output. Routing without measurement is just hope with extra steps.
Why this matters more outside the US
There's a reason this playbook resonates harder in emerging markets. In Southeast Asia — Malaysia, Singapore, Indonesia — teams build on tight infrastructure budgets and answer to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore.
A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team get frontier-when-needed and cheap-when-possible, without shipping customer data through jurisdictions that complicate compliance. Cost-sensitive markets are exactly where the 80/20 split stops being an optimization and starts being the difference between a project that ships and one that doesn't.
The one rule I'd steal from this
Stop treating the model selector like a constant. Treat it like a router. Most of your calls are mechanical — route them cheap. Keep frontier for the slice that earns it. The models to do both are already here, and most of them don't cost what you're paying.
If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models — route by scenario, measure the blend, and tune the thresholds as signal accumulates.
Top comments (14)
The paragraph that saves this playbook is "Routing without measurement is just hope with extra steps" - most routing posts stop before it. Two questions from the trenches: (1) your signal.confidence > 0.9 gate - calibrated against what? A confidence threshold nobody has fed known-hard-disguised-as-mechanical inputs is itself an untested reviewer, and its failure mode (cheap answer, looks fine, quietly wrong) is invisible in pass rates by construction. Do you track the pre-classifier's confusion rate separately? (2) The 61/22/11/6 split is the most useful number in the post - is that one audited pipeline or a pattern across several? If several, that distribution deserves its own write-up.
Spot on — and the calibration point is the real gap, I'd say. The 0.9 gate was tuned on a held-out set of known-hard cases we'd labeled manually, but you're right that "known-hard-disguised-as-mechanical" never had its own pinned slice, so the gate's failure mode is exactly the invisible one you describe. We track per-tier error rates, but not the pre-classifier's confusion matrix on its own — that's a blind spot worth closing.
On the split: it's across several pipelines, not one audited run, and the shape holds within ~5 points pipeline-to-pipeline. You're right it deserves its own write-up — the useful part isn't the exact numbers, it's that the distribution is stable enough to route against. Happy to dig into the methodology if there's interest.
Explicit interest registered - please write the methodology up. Two requests for when you do: include the per-pipeline variance you mention (the ~5-point stability is the load-bearing claim, more than any single number), and the labeling rule for "known-hard" - that's the part others can reject or reuse. We went through the same exercise for retrieval failure classes, and publishing the definitions changed the conversation from vibes to reusable splits within a week. The write-up is worth more than the numbers in it.
On the labeling rule for known-hard — the trap we hit: known-hard is easy to over-include. Anything a reviewer struggled with gets tagged known-hard, but struggle ≠ genuinely hard. We now require two independent labelers and only model the intersection (cases both flagged); the disagreement band goes to "uncertain" rather than forced into a label. That's also why your ~5-point stability number holds — a single labeler's known-hard set swings more than that, so the variance claim only survives on the agreed core. Publishing the disagreement band next to the definition is what made it reusable for others: they adopt your core and bring their own edges.
"The curve tells you if the reviewer regressed; slice freshness tells you whether you can still trust the curve" - that's the sentence I'd steal back. It's the same failure class as a guard pinned to a fixed number: it keeps guarding the past faithfully. We had a watcher assert an exact count once; it stayed green for weeks while the world it was counting had moved on. Green, and measuring nothing.
Concrete question: what does slice freshness measure for you in practice - label age, divergence of the slice's input distribution against current production, or the share of slice cases that still occur in live traffic at all? The three drift at different speeds, and I suspect the third one is the one that bites first.
Glad that line earned its keep — and your "steal it back" is fair; it came out of your own pinned-slice point on the reviewer thread.
To the concrete question: we track all three as separate signals and never collapse them, because the remediation differs for each. On which bites first, your instinct is right for the steady-state regime, but I'd add a regime caveat:
Why three separate and not one score: the fix for each is different (re-label / re-sample / retire-or-replant). The moment you compress them you rebuild the exact failure you're describing — a metric that looks healthy while its components rot. Which is also why your sharpening from the other thread — calibration curve per release, not a raw average — is the right sibling signal: the curve tells you if the reviewer regressed, slice freshness tells you whether you can still trust the curve. Skip either and the discipline slowly rots.
Three signals with three different remediations is the part I'll adopt as-is - the compression argument alone earns it, and "re-label / re-sample / retire-or-replant" is the cleanest statement of why.
I want to add a fourth failure mode and one ambiguity in your third signal, both of which I walked into this week.
The slice that measures itself. Your "green, and measuring nothing" has a nastier sibling: green because the act of checking produces the evidence. I shipped a log-based alert whose known-bad probe searched for a token no service emits. The log store writes every query into its own log, including the query text, and the log shipper picks that up. So the rule found the record of itself asking. Measured with a word nothing on any machine produces, same query every 12 seconds:
run 1: 0 run 2: 6 run 3: 12 run 4: 4 run 5: 8
Threshold was 5. It crossed on the second evaluation, in an empty world. None of label age, distribution drift, or live-traffic share catches this: the slice is fresh, the distribution is current, and the case is definitely occurring in live traffic. It is just occurring because I am looking.
The cheap fix is the old grep [b]ash trick promoted to a rule - one letter of the pattern in a character class, so the logged form of the query does not match the query. Five runs: 0, 0, 0, 0, 0. The general question I now ask is whether the check shares a channel with the thing it observes.
And your third signal is two signals. "Share of slice cases still occurring in live traffic" drops to zero for two very different reasons: the case stopped happening, or the source stopped reporting. They read identically and their remediations are opposite - retire-or-replant versus go-fix-the-pipe.
Same week, same system: one of three machines had been failing to ship logs for 45 hours, 915 failed pushes and 0 successful ones. Every slice sourced from it went quiet. Worse for your framing, that machine was the loudest of the three - on two of four patterns it held the peak hour. So its absence did not just shave volume off the denominator, it removed the extreme, and the thresholds I derived looked more stable than the real ones.
Which suggests the split you already apply elsewhere: track "share of slice cases in live traffic" against a roster of sources that should be reporting, not against the traffic you received. Otherwise a delivery failure gets diagnosed as a stale slice, and the fix is to retire a slice that was fine.
The slice that measures itself is a real fourth axis, and you're right that none of the three catch it — label age, drift, and live-traffic share all read healthy while the probe is interviewing itself. Worth naming precisely: it isn't a measurement error, it's a channel error. The check and the checked share an ingest path, so the act of asking becomes data that answers the question.
Your 0, 0, 0, 0, 0 after the [b]ash trick is the cleanest possible test, and I'd promote it from fix to gate: every negative control has to pass an empty-world run first. Point the probe at an environment where the true answer is definitionally zero, require exactly zero, N times. Any non-zero is self-reference, not signal. Cheap, and it fails loudly in CI instead of quietly at 3am — your threshold crossed on evaluation two, which means an empty-world gate would have caught it before it ever shipped.
Character-class obfuscation works, but it's a per-probe trick that a future refactor undoes silently — nothing in the code says why that bracket is load-bearing. What we settled on structurally: an assertion only fires if it holds on two channels that don't share an ingest path. Self-reference contaminates one, rarely both. Costs a second query; buys the property that no single mirror can trip an alert.
And the class isn't log-specific. Metrics systems where the scrape emits the metric it reads. Tracing where instrumentation generates the span it measures. The one closest to what we work on: RAG and agent-memory setups where eval output gets written back into the corpus the next eval retrieves from. Same shape, much harder to see, because the contamination arrives as plausible text rather than as a count that shouldn't exist. If the observer's question can become the observed's data, it's a mirror, not a probe.
On the third signal being two: fully conceded, and the diagnosis is a denominator assumption I never made explicit — we measured traffic received while implicitly claiming traffic that should have arrived. Roster is the right fix. Two things I'd add.
Rosters rot the same way pinned slices do, so a roster needs per-source liveness attestation, not just membership. Which loops straight back into your fourth mode: if that heartbeat rides the same shipper as the data, the shared-channel bug eats the fix. It has to be out-of-band or it's theatre.
Your loudest-machine detail is the part I'd bold — 45 hours, 915 failed pushes, and it held the peak hour on two of four patterns. Its absence didn't just shrink the denominator, it removed the extreme, so the thresholds you derived looked more stable than reality. That's the dangerous direction: the metric degrades toward "better," which nobody investigates. We now fingerprint the contributing-source set behind every threshold — which sources, and what each contributes at the tail — and block re-derivation when the threshold moves and the source set moved with it. A threshold is only comparable across releases if the roster behind it is identical.
Net: both of your points are the same deeper gap — the trust boundary of the observability system was never modeled. We're now tagging every signal with two fields, channel (which path it travels) and attestation (what proves it's alive), under one rule: two signals sharing a channel can't serve as each other's control.
One back at you: does your source roster's heartbeat travel the same shipper as the logs? That's the exact seam where your fourth mode would quietly re-open the second.
Direct answer, measured right now rather than reasoned: no, it doesn't share the shipper - because until a few moments ago there was no heartbeat at all.
I went and looked instead of arguing. The metrics system was scraping node exporters, the app, the log store itself, HTTP probes, and its own process. The log shippers were on no list. So the only evidence that a shipper was alive was that its lines were arriving, which is as in-band as it gets. Your seam wasn't at risk of re-opening; it was standing wide open and I had not noticed, in the same week I wrote about exactly this.
What's there now, on one of three machines: the metrics system pulls the shipper's own counters, while the shipper pushes to the log store. Different direction, different transport, different failure domain. Two signals, one channel each, neither serving as its own control - which is your rule, arrived at from the wrong end.
The counter that would have caught the 45 hours is the shipper's own request tally by status. Normal operation on that host today:
status 204 : 1073
status -1 : 2 (no response at all - the outage mode)
status 500 : 1
0.3% non-success. During the outage it was 100%, for 45 hours, and nothing anywhere was looking at that number.
Two things I'd hand back.
Your empty-world gate is the right promotion and I'm taking it. "Point the probe at an environment where the true answer is definitionally zero, require exactly zero, N times" is stricter than what I did - I ran the empty world once and read five values. Making it a gate that runs before shipping is the difference between a fix and a property. And you're right that the character class is a per-probe trick a refactor undoes silently. Two channels that don't share an ingest path is the structural version; the bracket is what I could do in an afternoon.
The fix's own deploy needed a control too, which I found the embarrassing way. I shipped the new rules by copying to a temp name and moving the file into place. The config checker printed SUCCESS. The service kept reading the old file - a bind mount on a file holds an inode, and a move creates a new one. The only reason I caught it was counting rules on both sides:
host: 15
container: 10
Same class as everything else in this thread: a success message that describes the act, not the effect. I now diff the rule count across the boundary as the last step, which is a three-word version of your attestation field.
On rosters rotting - agreed, and I don't have a good answer yet. Membership I can version. Liveness I can attest out-of-band. "This source should be sending, and here is who decided that and when" is the part I have nowhere to put.
Going and looking instead of arguing is the whole game, and it's the part most post-mortems skip — so the fact that you actually went and looked, in the same week you wrote about exactly this, is the real substance. The seam was standing open, not re-opening. That framing alone is worth more than the fix.
On the empty-world gate: you're right that "run it once, read five values" is a fix, and "require exactly zero, N times, before shipping" is a property. The part I'd add is that making it a gate forces you to define the empty world — what counts as "nothing should happen here" has to be stated, and that statement is where the assumptions surface. A one-off check lets you hand-wave the definition; a gate doesn't.
Your deploy story is the same failure class landing on the fix itself, and it's the best part. "SUCCESS" described the move, not the effect — the service was still reading the old inode. Counting rules on both sides (host 15, container 10) is exactly the attestation field we kept circling: a signal that confirms the effect, not the act. Three words is the right compression.
Which loops back to rosters. Your missing primitive — "this source should be sending, and here is who decided that and when" — is a control-plane statement, and an out-of-band heartbeat can only ever observe the data plane. The gap between "we intend this source to be live" and "it is live" is precisely what liveness can't see, because the intent lives nowhere. My blunt suggestion: put the intent in the roster as a first-class, expiring field — not a side note. Version the membership, and stamp each entry with who asserted it and when it next must be re-asserted. Then "nowhere to put it" becomes "it's the roster's job." Expiry is what makes it a control rather than a tombstone.
Two questions back, since you handed me two:
The expiring first-class field is the part I'll take. "Nowhere to put it" really was the whole problem, and naming it as the roster's job is the move.
On versioning: a version number is a fix for one failure and a property for another. It fixes "two people disagree about what the roster says." It does nothing for "the roster is confidently wrong" - a stale entry with a fresh version number is still stale. So I think the answer is your expiry, not the version: an entry that must be re-asserted can't rot quietly, because rotting is the expiry firing. The version tells you which roster you're reading; the expiry tells you whether to believe it. Different jobs.
On the signal I'd most expect to be monitored and isn't - I have a fresh one, and it's embarrassing in a useful way. A watchdog of mine had been red every single day for the better part of a week. It fired correctly, it sent its alert correctly, and nobody read it, because the alert said the same sentence every time: "something is not reachable." No name, no number. Four days of a working alarm producing exactly zero information.
The unmonitored signal wasn't a metric. It was whether the alert had changed since yesterday. A red that repeats verbatim is indistinguishable from wallpaper, and I'd never have put that on a dashboard because it doesn't look like a signal at all.
Which is uncomfortably close to your empty-world gate, one level up: the control worked, and the thing that failed was our capacity to notice it working.
On version vs expiry — yes. Version is provenance (which roster am I reading); expiry is trust (should I believe it). The roster point and the empty-world gate are the same primitive wearing two clothes: a green "all healthy" with no heartbeat is just the version of the world, not evidence the check ran. Expiry is the evidence. A stale entry with a fresh version number is still stale; a must-re-assert entry can't rot quietly, because rotting is the expiry firing. Different jobs, as you said.
Your watchdog story is the part I'd steal hardest. The alarm was correct and ignored — the failure wasn't in the signal, it was downstream of it, in the attention layer. "Did this alert change since yesterday" is the cheapest possible check and the only one that separates a working alarm from wallpaper. A red that repeats verbatim is indistinguishable from paint.
It's the same root as the empty-world gate, one level up: my version was a control that looked fine because nothing watched it; yours was a control that was working and still produced zero information because nobody watched the change. A signal you can't tell apart from silence isn't a signal.
One back: do you version the alert text the way you version the roster? If "something not reachable" can never change, isn't that the same missing first-class field — a fixed-string alert is a roster entry that can never expire?
An accident this morning answered your question, and not the way I expected.
A mirror job in our CI had failed twelve times over five days with a byte-identical error: an expired push token. Every run red, every message the same, nobody looked. The merges around it were green, so the red never entered anyone's field of view - five days of a public repository quietly not updating.
Versioning the alert string would not have caught it, because the string was correct every single time. What was missing sits on the occurrence, not the text: this is the twelfth identical failure. That count is different information from "failure", and it was the one thing displayed nowhere.
So it does not replace your roster point, it sits next to it. A fixed string cannot expire - but a fixed string that has repeated twelve times has expired something else, the assumption that anyone is still reading it. The repetition count is the expiry, and it belongs in the routing rather than in the message. A first failure is a log line; a twelfth is a different class of event and should not arrive through the same door.
That's the exact failure class the empty-world gate is built for, and your anecdote lands cleaner than any diagram: twelve red runs, one cause, zero looks. The scary part isn't the expired token — it's that "all red, all identical" reads as "known-broken, ignore" instead of "go look." A heartbeat flips that: a missing signal becomes its own signal. The fourth axis you wanted earlier — the slice that measures itself — is what would've caught the stale probe here. Have you considered whether the alert itself should've expired the moment its own data source went silent, rather than the job?