DEV Community: TokenLat

Deterministic delivery is an architecture decision, not a model setting

TokenLat — Fri, 06 Nov 2026 05:40:25 +0000

1. The argument is being filed under the wrong heading

Most debates about agent reliability get filed as model problems. The proposed fix is a better model, a lower temperature, a longer system prompt, a different framework, or more evals. All of those can help. None of them change where the decision actually lives.

Our position is that reliability in an agent system is not a property of the model. It is a property of the architecture around it: which decisions the model is allowed to make, which ones are made by ordinary code, and whether the system keeps enough evidence to tell the two apart afterwards.

That is a technical position, not a product claim. The phrase we use for it is deterministic delivery, and it is worth stating precisely, because it is easy to oversell.

2. What the position actually says

Deterministic delivery is controllability, auditability and fallbacks at the architecture level. It is explicitly not a claim that the model will always be right. A model reading an ambiguous email can still get it wrong inside a perfectly built backbone.

Three conditions have to hold before we'd use the phrase:

Every decision node leaves a replayable, traceable record: inputs, outputs, latency, and the acceptance criterion it was supposed to meet.
Rule logic is deterministic by construction — same input, same state transition, same audit entry — while semantic nodes stay probabilistic and are measured rather than assumed.
A failure can be located to a node, not just observed as a wrong final answer.

If any of those is missing, adding tests or tuning prompts treats a symptom. The gap is architectural.

This also fixes a common misreading. "Deterministic" here does not describe the model's output. It describes the parts of the system that are not allowed to improvise.

3. The split, and where the other side is right

The alternative is a probabilistic black box: each step decides what happens next, state lives inside the prose, a retry is a model call that sounds like a retry, authorization is a line in a prompt asking nicely. When the output is wrong, you get a paragraph, not a location.

We'd argue that's the wrong default for production. But it is not wrong everywhere, and a position that cannot say when its opposite is fine is a preference, not a position. For a one-off script, a throwaway prototype, or internal exploration where a wrong answer costs nothing and nobody has to explain it later, chaining model calls is the cheaper and more honest choice. A deterministic backbone earns its keep when failures must be attributed, outputs reach external users, an audit trail matters, or more than one person has to maintain the system past the demo.

So the real disagreement is not about whether to use workflows, or how autonomous an agent should be. It is about a boundary: which decisions require judgement, and which decisions require a diff? Framed that way, the boundary is usually much narrower than the initial design assumed.

4. Three constraints, one property

The position above is not implemented by a single component. Three engineering constraints produce it, and deterministic delivery is the property that emerges when all three hold. They are not a feature list; each has a criterion you can check and a characteristic way of failing.

C1 — hybrid orchestration. A deterministic backbone owns state, retries, authorization, audit trails and rollback; a model is invoked only at nodes requiring semantic judgement.
Criterion: count the nodes that could be written as an if. If the answer is "almost none," there is no orchestration, only a chain of prompts with extra latency.
Failure mode: state hidden in generated text, so a retry silently multiplies the same call and nobody can reconstruct what ran.

C2 — structured prompt contract. A versionable spec of role, hard constraints, I/O contract and fallback rules. Not a persona — persona in English means role-play, and role-play is the part that does not survive contact with production. What matters is the constraint block: missing_context → ask for clarification, never infer; output must be a typed object; if a constraint cannot be satisfied, emit fallback_reason and stop.
Criterion: you can diff the constraint set between two deploys and assert on it without running the whole agent.
Failure mode: the rule exists in a system prompt, but nothing records whether it was ever consulted on this input.

C3 — white-box evaluation. Capture per-node inputs, outputs, latency and acceptance criteria into replayable test sets, instead of judging the final answer by feel. Three sets, kept separate: positive (the happy path with grounded context), boundary (empty retrieval, contradictory sources, out-of-scope questions, malformed tool responses), and follow-up (multi-turn drift and corrections).
Criterion: a failure from last week can be replayed as a single node, with its original input and criterion.
Failure mode: evaluation that scores only the final answer, which tells you something is wrong and nothing about where.

Note the ordering. C1 defines where decisions live, C2 makes the rules at those boundaries explicit and testable, C3 proves they fire. Skipping C1 and going straight to C3 is the most common mistake: you end up with a large test suite over a system whose nodes only exist in the model's output text.

5. What this looks like in one trace

Consider a support agent that tells a customer refunds apply to a product category no policy document covers. Nobody notices until a customer complains, because the only thing the system kept was the final answer. An illustrative trace of that node would look like this:

{
  "step": "answer_policy_question",
  "input": { "question": "...", "retrieved_chunks": 0 },
  "constraint": "missing_context -> ask_for_clarification",
  "constraint_evaluated": false,
  "fallback_reason": null,
  "output": { "answer": "Refunds apply to this category" },
  "latency_ms": 1840
}

Read against the three constraints: C2's rule exists but was never evaluated; C1 has no record of the state transition that should have stopped the call; C3 has no boundary case asserting on retrieved_chunks = 0. All three failed at once, and the trace is the only reason we can say so. The model produced a plausible continuation — that is what a language model does with no grounded context. Blaming hallucination ends the investigation; the trace starts it.

One detail worth keeping: log the evaluation, not just the spec. Plenty of systems record the constraint and assume it ran. The gap between "we wrote the rule" and "the rule fired on this input" is where most of these incidents live.

6. Scope and limits

A position is only useful if it says what it does not cover.

It does not fix semantic errors on genuinely ambiguous input, missing knowledge in the retrieval corpus, or requirements that were wrong to begin with. A perfect backbone around a bad question still produces a bad answer.
It does not mean fewer model calls, cheaper inference, or no hallucinations. There is a cost consequence — explicit state makes retries visible and spend attributable to a step instead of to "the agent ran long" — but that is downstream of the architecture, not the goal.
It does not remove the need for model quality. It changes what you can do when model quality is insufficient: add a boundary case, tighten a fallback, move a decision into code, and prove the change with a replay.
It is not worth the overhead when failures are cheap, invisible and unattributable — the cases in §3 where the other side is right.

What it does buy is a short, boring postmortem. Failures become addressable, "add a boundary case" becomes a normal engineering response instead of "tune the prompt and hope," and the model can be changed later without re-litigating the whole architecture. That is what keeps an agent maintainable past the demo — not a cleverer chain of prompts.

7. What follows

If this position holds, the first thing to build is not an eval suite. It is the definition of where the nodes are and who owns them, because every artefact above — the contract, the trace, the test sets — is indexed by that boundary. Get the boundary wrong and the rest becomes bookkeeping over a system you still cannot inspect.

Two questions follow, and we'd ask them before discussing models at all: where is the line between rule logic and model judgement written down so it can be diffed, and can a wrong answer be replayed to the exact node with its input, its criterion and the fallback that did not fire? In your stack today, does that boundary live in the orchestrator's config — or in whatever the model happens to emit at runtime?

Your AI bill grew 4x and nobody can tell you which step did it

TokenLat — Wed, 04 Nov 2026 02:24:22 +0000

A team I talked to last month had a 4x jump in inference spend over one quarter. Not a crisis — they had shipped an agent, usage grew, revenue grew. The problem was smaller and more annoying: when finance asked which part of the product had gotten more expensive, nobody could answer.

Three engineers, three different guesses. Every dashboard was green. Every invoice was correct. And still, no one could name the step.

That is the failure mode I keep running into. Not "the bill is too high." The bill is correct and unreadable.

I want to argue this isn't a finance problem or a pricing problem. It's an instrumentation problem, and it has a specific cause: the unit of work changed, and the unit of attribution didn't.

Per-token prices fell. The bill still went up.

Every model you depend on has gotten cheaper per token over the last two years. And yet the bill went up. Both things are true, which should tell you that price is not the variable that matters.

What changed is the number of calls behind a single unit of user-visible work.

In the chat era, "summarize this customer's last quarter" was one request: a prompt, a completion, done. In the agent era, the same sentence is a plan call, a handful of retrieval or embedding calls, two or three tool calls against internal systems, a compression step so the tool output fits, a context rebuild, a generation, a validation pass, and possibly a second generation because validation failed.

Twenty to forty calls for one thing the user perceives as a single action.

And here's the part that makes the arithmetic non-linear: context is a multiplier, not an addend. Call #30 doesn't carry the cost of call #30. It carries the accumulated results of calls #1 through #29. As a loop gets deeper, input tokens grow roughly with the square of the step count while useful output stays flat.

So the shape of the bill changed. It used to be a small number of expensive things. Now it's a large number of individually cheap things, where cost concentrates in the plumbing between the things everyone is looking at.

The attribution problem

Most teams instrument cost along axes that made sense for a chat product:

by API key
by endpoint
by team or environment

These answer who. They do not answer why.

The conversation I described above usually goes like this. Someone pulls up spend by API key and says: platform team is 60% of the bill. Okay — but the platform team runs eleven features. Which one? And inside that feature, was it the retrieval step, the tool calls, or the retries? The attribution stops one layer above where the money is actually being spent.

When the unit of work becomes a task chain, attribution has to move with it. In practice that means you need to answer three different questions at three different altitudes:

Total. How much this month, split by model and by outcome (200 vs. 429 vs. timeout — a status code is a cost signal, not just a health signal).
Attribution. Which feature, which tenant, which task type.
Single step. Inside this one trace, which of the 34 calls ate 60% of the cost.

Almost every team I've talked to has level 1. Some have level 2. Very few have level 3 — and level 3 is the only one that changes behavior, because it's the only one that tells you what to fix.

There's a name for this in the vendor world now: an observability center with global, attribution, and per-step views. It's being sold as an enterprise feature. It shouldn't be a feature. It's the minimum for operating an agent at all.

Why the application layer can't answer this alone

Here's the part that's genuinely structural, and it's the reason this doesn't get fixed by "just add better logging."

Your application knows business semantics. It knows this is task tsk_8f21, for tenant acme, in the support_draft feature, at step retrieve_compress. What it does not know is the cross-model reality of that step: how many tokens the provider actually billed, whether the prompt cache hit, whether a silent retry tripled the input, or whether the same step got routed to a different model than last week.

The call layer knows all of that and knows none of the semantics. It sees POST /v1/chat/completions, 18,432 prompt tokens, 412 completion tokens, cache miss, 340ms. It has no idea this is a support draft for a customer named Acme.

So each side holds half the picture, and the split is exactly along the boundary where cost decisions get made. That's why "we'll join the logs later" fails: by the time you join them, the fields you need were never emitted.

The fix is unglamorous. Put the business coordinates on the request as it goes out, and let the layer that sees the tokens record them alongside the usage row.

# Business coordinates travel with the call, not in a side table.
COORDS = {
    "X-Task-Id": "tsk_8f21",
    "X-Task-Step": "retrieve_compress",
    "X-Tenant": "acme",
    "X-Feature": "support_draft",
}

resp = client.chat.completions.create(
    model=model_for_this_step,
    messages=messages,
    extra_headers=COORDS,
)

# What comes back is not just a completion. It's a cost line item
# that already knows where it belongs.

That's it. One dict on the way out. Everything downstream — the usage table, the aggregation, the per-step breakdown — becomes possible; without it, nothing downstream can be reconstructed later.

Then the query you actually want is boring SQL instead of an archaeology project:

SELECT task_step,
       SUM(prompt_tokens + completion_tokens) AS tokens,
       SUM(cost_usd)                         AS cost,
       COUNT(*)                              AS calls,
       SUM(CASE WHEN status <> 200 THEN 1 ELSE 0 END) AS failures
FROM   model_usage
WHERE  task_id = 'tsk_8f21'
GROUP  BY task_step
ORDER  BY cost DESC;

Six lines. If your current stack can't answer that query, you don't have a cost problem yet — you have a cost problem you can't see, which is worse.

Three blind spots that show up in almost every audit

When teams do get per-step visibility, the same three things light up. It's predictable enough that I now look for them first.

1. Context rebuild. The loop re-sends conversation history, tool schemas, and retrieved documents on every iteration. Nobody budgeted for this because nobody counted it as a feature. A single page snapshot can be 20k+ tokens; do that inside a five-step loop and it dominates your bill while your completion tokens look healthy.

2. Silent retry. A retry that re-runs the whole step doesn't just cost 2x — it usually re-sends a larger context (the failed attempt got appended) and quietly resets whatever spend cap you set at the task level. Your guardrail was on the task; the cost was incurred on the step.

3. Tool output inflation. A tool returns 400KB of JSON. Two lines of it matter. It enters the context raw, gets re-sent on every subsequent call, and is paid for on each one. Trimming at the tool boundary is usually the single highest-leverage change available — higher than switching models.

None of these are model-quality problems. All three are invisible without step-level attribution and obvious with it.

Put the cap where the cost is incurred

One more thing that only becomes visible once you have step-level data: most teams define their spending limits in the wrong place.

A task-level cap is the intuitive choice — "this job may cost at most X." But it answers the wrong question. It tells you a task blew its budget; it doesn't tell you which step did it, and by the time it fires the task has usually already failed in a way the user noticed. Meanwhile the pathological call keeps happening: the context rebuilt for the twelfth time, the tool output that inflated 40x, the retry chain that never terminated.

A step-level cap fires on the specific call that went wrong, while the task can still degrade gracefully instead of dying.

There's a second, quieter problem. Caps implemented in application code are per instance. Three workers, three counters, three times the budget — and the counter resets on every deploy. A cap enforced in the layer that every call already passes through is global by construction, and it survives a rolling restart.

That's the same lesson as the attribution one, just from the other direction: if the layer that sees the tokens isn't the layer that enforces the policy, your guardrail is an estimate.

The self-test

I'll leave you with the question I now ask every team, because it separates the ones who can operate an agent from the ones who are still guessing:

Take the most expensive task your product ran last week. Can you name its three most expensive steps, and the reason each one cost what it did?

If yes, you have the instrumentation and the rest is tuning. If no, the bill isn't the problem — the missing step-level attribution is. And no amount of model shopping fixes a number you can't decompose.

We instrumented 11 production agents with real token accounting. In 9 of them, the most expensive step wasn't the model.

TokenLat — Wed, 28 Oct 2026 01:06:28 +0000

Last quarter we finally stopped trusting the LLM bill. We wired real per-step token attribution into 11 production agents — not the default "total tokens" the provider hands you, but a tracer that tags every call, every tool, every cache lookup with the trajectory it belonged to.

The number that broke our mental model:

Across those 11 agents, the single most expensive step — measured per resolved task, not per API call — was the model in exactly 2 of them. In the other 9, something we'd filed under "negligible overhead" was quietly eating more than the LLM.

Here is the trap, in one picture. Same agent, same steps — two ways of ranking them.

WHAT THE DASHBOARD SHOWS (per-call mean)        WHAT THE BILL ACTUALLY IS (per-trajectory total)
Step                  $/call     rank            Step                  $/task      rank
LLM completion        $0.0028     #1              PII / guardrail       $0.0042     #1
PII / guardrail       $0.00035    #4              Embedding loop        $0.0036     #2
Embedding (in loop)   $0.00002    #5              LLM completion        $0.0028     #3
Semantic-cache chk   $0.00090    #3              Semantic-cache chk   $0.0009     #4
Tool call             $0.00010    #2              Tool call             $0.0008     #5

Left column: what every dashboard reports. Right column: what one task actually costs. The model drops from #1 to #3. A step you pay two-hundredths of a cent per call becomes the #2 cost driver. The ranking is not the same list.

And across all 11 agents, the #1 cost step by trajectory-total:

#1 cost driver (per trajectory), across 11 production agents
Embedding / retrieval loop   ██████████  5 / 11
PII / guardrail model        ██████      3 / 11
Model (LLM)                  ████        2 / 11
Cache-verify model           ██          1 / 11

Nine of eleven. The model isn't even in the lead most of the time.

The hidden axis: mean-vs-trajectory

The reason dashboards mislead is structural. They report mean cost per call, because that's what one request easily exposes. But an agent step isn't called once — it's called inside loops, per chunk, per re-plan.

The real cost of a step is:

step_cost_per_task  =  calls_per_task  ×  cost_per_call

A step at $0.00002/call looks free on the mean chart. Call it 180 times in a retrieval loop and it's $0.0036/task — more than the LLM, which runs once. The "free" step wins by 28%, on exactly the hard tasks you care about.

The invisible cost axis is trajectory-total vs call-mean. Every dashboard I've seen only draws one of them.

Three cases we actually shipped

These aren't hypotheticals — they're line items from the 11 agents.

Case 1 — The "free" embedding step that wasn't.
We had a retrieval step the finance team literally labeled free in the cost model (embeddings at $0.00002/call). It lives inside the agent's re-planning loop, which on hard tasks re-retrieves up to 180 times. 180 × $0.00002 = $0.0036/task. The LLM completion, called once, is $0.0028. So the step we scored as zero out-costs the model by 28% — on every hard task, which is the only tasks that matter.

Case 2 — The cache hit that cost more than the miss.
Our semantic cache doesn't just return the cached answer on a hit. It runs a reranker plus a 70B "is this still relevant?" check, to avoid serving stale advice. That verification costs $0.0009. A genuine miss regenerates from the 8B model for $0.0006. So a hit — the thing caches exist to save you — costs 50% more than a miss, across the 40% of traffic that hits. We were paying a "savings" tax and calling it a win.

Case 3 — The guardrail that 4×'d the bill.
A PII-scrubber runs a 7B model on every message chunk before the LLM sees it: 12 chunks × $0.00035 = $0.0042/task. We added it for a compliance ask and never costed it per-trajectory. It became the #1 line item on 3 of the 11 agents. Removing it wasn't on the table — but routing it to a 0.5B model dropped it to $0.0003, a 14× cut, with no detectable quality change on our eval set.

How you should actually measure

The fix is almost never "pick a cheaper model." It's accounting per trajectory so the loop multiplier can't hide, then routing each step to the right-sized model. We sit in front of 25+ models behind one OpenAI-compatible gateway; the interesting savings came from moving the scrubber and the cache-verify down, not from swapping the completion.

Three changes that took an afternoon:

Attribute every token to a trajectory_id, not a request. If you can't say what one task cost, you don't know your cost.
Rank by total, sample real tasks. Pull N resolved tasks, sum per step, sort by total. The mean chart will lie to you; the total won't.
Put a cost_per_resolved_task SLO in place. Alert on the trajectory, not the call.

A minimal version of what we run:

cost_attribution:
  unit: trajectory          # not 'request' and not 'call'
  trace_id: trajectory_id   # tag every step, tool, cache lookup
  rank_by: total_cost       # never mean_cost
  alert:
    metric: cost_per_resolved_task
    threshold_usd: 0.02
  exclude_steps: []         # never exclude the 'cheap' steps

And the five lines that surface the buried step:

from collections import defaultdict

def rank_steps_by_task(events):
    per_step = defaultdict(float)
    for e in events:
        per_step[e.step] += e.cost_usd          # already trajectory-attributed
    return sorted(per_step.items(), key=lambda kv: -kv[1])

# prints the REAL ranking — usually not what your dashboard shows
for step, cost in rank_steps_by_task(events):
    print(f"{step:24} ${cost:.5f}")

The question I'll leave you with

The dashboards aren't lying. They're reporting the axis that's easiest to measure. We spent a year optimizing model choice while a loop multiplier and a "free" safety step ran away from us — invisible the whole time, because nothing on the screen was drawing the trajectory-total axis.

So: when you last looked at your agent's cost, were you ranking by what one call costs — or by what one task actually costs? Because those are not the same list.

What's the step you've been treating as free?

Your retry loop is your most expensive feature

TokenLat — Thu, 22 Oct 2026 06:16:29 +0000

A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the model client and never put it in the budget.

That retry path only fires when the model is slow or down — i.e. at peak load. And every retry replays the entire context. So the one part of the system nobody accounts for is the part that costs the most, exactly when you can afford it least.

The number that surprises people

Take a support agent for an ASEAN e-commerce shop. A hard ticket pulls a 50k-token context (order history, policy docs, prior turns) and emits ~1k tokens.

One attempt ≈ 51k tokens.
With retries: 3 and the task still failing, you pay for 4 attempts: 51k × 4 ≈ 204k tokens for a task you budgeted at 51k.

That's a 4× tax — and it isn't spread evenly. The tasks that fail are the expensive ones: long context, heavy tool chains, messy inputs. Retries concentrate exactly where cost is highest.

retry_policy:
  naive:
    attempts: 3
    on_failure: "replay full context to same model"
    hidden_cost_multiplier: 4        # 1 attempt + 3 full-context replays
    shows_up_in_dashboard: false
  cost_aware:
    attempts: 3
    on_failure: "route to next equivalent model"
    carries_context: true
    hidden_cost_multiplier: 1        # context travels, never replayed
    shows_up_in_dashboard: true

Why the bill stays invisible

Most cost dashboards report average cost per successful request. A task that fails three times and finally succeeds doesn't appear as "4× cost" — it appears as one success with a latency blip. The retry is a hidden fourth axis of spend:

Latency triples → someone notices.
Cost triples → almost nobody measures it, because it's buried in "failed then retried" traffic.

So teams tune the happy path and ignore the one code path that runs under stress and multiplies the bill.

The fix is routing, not retrying

You don't make retries cheaper by retrying harder. You stop replaying context: when a model is unavailable, advance the task to the next equivalent model that can take the same handoff context. The context travels with the task; you don't re-pay for it as a "retry."

With 25+ models behind one OpenAI-compatible endpoint, "equivalent" stops being theoretical — there's a tier you can actually route to without rewriting prompts or adapters. For teams under data-residency rules (PDPA-aligned, SG-hosted), this also matters: re-running context on a different provider just to "retry" raises compliance questions, not just cost.

flowchart LR
  subgraph Naive[Naive retry: context replayed]
    N1[Task · 50k ctx] --> N2[Attempt 1: 50k in]
    N2 -->|fail| N3[Replay 50k in]
    N3 -->|fail| N4[Replay 50k in]
    N4 -->|fail| N5[Replay 50k in]
    N1 -.paid 4x.-.-> N5
  end
  subgraph Smart[Cost-aware: context travels]
    S1[Task · 50k ctx] --> S2[Model A call]
    S2 -->|unavailable| S3[Model B · same ctx]
    S3 -->|unavailable| S4[Model C · same ctx]
    S1 -.ctx paid once.-.-> S4
  end

def step(task, model_registry):
    # cost-aware: context is paid ONCE, then travels with the task
    for model in model_registry.ordered_for(task.capability):
        try:
            return model.call(task.handoff_context)   # no replay
        except Unavailable:
            continue                                   # next equivalent, same context
    raise NoCapableModel

# for contrast — the naive version re-sends the FULL context every attempt
def naive_retry(task, model, attempts=3):
    for _ in range(attempts):
        try:
            return model.call(task.full_context)        # full context RE-SENT each time
        except (Timeout, Unavailable):
            continue
    raise RetryExhausted

The router doesn't retry the model. It advances the task to the next model that can take its context. Same uptime, a fraction of the bill.

The question I'll leave you with

Before you ship retries: 3: for the tasks that fail, how many tokens does the retry actually replay — and is that number in any dashboard you look at?

No tool to sell, no link to click. Just this: most LLM systems that survive a model outage are just retry loops with better logging — and the loop is the most expensive thing they do.

Your failover is a retry loop pretending to be resilience

TokenLat — Tue, 13 Oct 2026 05:32:26 +0000

A team I worked with had a clean SLA: agent responses in under 8 seconds, 99.5% of the time. Then the frontier model they routed everything through started rate-limiting during peak hours. Their "failover" was three retries with exponential backoff.

The SLA held on paper. In practice, users got a 24-second response that sometimes failed anyway. The retries weren't failover — they were the same request, replayed, hoping the wall had moved.

The counterintuitive part

Retry and failover look identical in a dashboard ("request failed → tried again"). They are not the same operation.

A retry assumes the system was momentarily unavailable and the task is intact. That's true for a database connection. It's false for an agent mid-task: the moment you re-send the whole conversation, you've thrown away the in-flight state — the half-formed tool call, the retrieved doc, the branch the agent had already chosen. You pay for the first attempt's tokens and the replay's tokens. Latency triples. And if the model is down (not just slow), every retry hits the same wall.

So the failure mode isn't "generation failed." It's state continuity failed. The task had accumulated context, and retry discards it.

This is a contract problem, not a wrapper problem

You don't fix this by adding retries=3 to your client. You fix it by defining, per sub-task, what "equivalent" means — and making the fallback consume the same handoff context rather than "start over."

That requires three things most single-model setups don't have:

An abstraction so the fallback speaks the same interface — if the backup model is OpenAI-compatible, swapping it in doesn't mean rewriting prompts or adapters. The task never learns a model went down.
A ranked fallback registry, not a single backup — one standby model is a single point of failure with a different name. A registry orders equivalents by capability and cost.
The routing layer carries state — the switch happens inside the task, not above it. The user sees one continuous attempt.

failover_policy:
  primary: frontier-model
  fallback_tier:
    - model: mid-tier-instruct
      equivalence: "handles same tool-calling schema"
      carries_state: true
    - model: small-instruct
      equivalence: "degrades to summary, preserves context"
      carries_state: true
  on_primary_unavailable: route_to_first_capable   # not: retry_primary

With a catalog of 25+ models behind one interface, "equivalent" stops being theoretical — there's a梯队 you can actually fail over to without dropping the task.

sequenceDiagram
    participant T as Task (in-flight)
    participant R as Router
    participant P as Primary (frontier)
    participant F as Fallback (mid-tier)
    T->>R: needs next step
    R->>P: call (carries handoff context)
    P-->>R: 429 / timeout
    R->>F: call (same handoff context)
    F-->>T: continuation (state preserved)
    Note over T,F: user sees ONE attempt, not a replay

The SLA only means something if the task survives

"99.9% availability" is meaningless if failover loses the task. A retry that returns after 3x the latency and a fresh start isn't a success — the user waited, paid, and got a do-over. The SLA number is green; the experience is red.

Real failover measures task completion, not request return. The fallback has to be able to pick up where the primary left off — which is exactly why the interface abstraction matters. Without it, you're not failing over, you're restarting.

def step(task, model_registry):
    for model in model_registry.ordered_for(task.capability):
        try:
            return model.call(task.handoff_context)   # state travels with the task
        except Unavailable:
            continue                                   # next equivalent, same context
    raise NoCapableModel

The router doesn't retry the model. It advances the task to the next model that can take its context.

The question I'll leave you with

Before you add a retry_on_failure wrapper: for the step you're protecting, what does "equivalent" mean — and does your fallback carry its state, or just rerun it?

No tool to sell, no link to click. Just this: most "resilient" LLM systems are retry loops with better logging. Resilience is a contract the fallback has to honor.

200 agents, 2,011,438 tool calls: who's paying for your AI?

TokenLat — Sat, 10 Oct 2026 07:16:07 +0000

A team I worked with shipped 200 internal agents last quarter. The monthly bill landed at 2,011,438 tool calls. Finance asked the usual thing: "which model is eating our budget?"

The intuitive answer — "the frontier model, obviously" — was wrong. And why it was wrong says something useful about how agentic systems actually spend money.

The counterintuitive part

Most token spend in an agent loop isn't in the hard reasoning step you're picturing. It's in the boring middle:

the 60-line retrieval that reformats a doc the model already saw,
the JSON reshaping between two tools,
the "which tool next?" classifier that fires before every action,
the summary pass that condenses a transcript nobody reads.

None of those need a frontier model. They need a model — a cheap one, routed to that specific sub-task. But the agent was wired with one "default" model for the whole loop, so every sub-task paid frontier prices.

We measured it: roughly 80% of the bill was sub-tasks that cleared the quality bar at ~1/20th the cost on a smaller model.

This is a routing problem, not a prompting problem

You don't fix this with a cleverer system prompt. You fix it by making the sub-task boundary visible to a router.

Instead of agent.llm = frontier_model, each sub-task declares what it needs:

subtasks:
  retrieve_context:
    min_capability: embedding-lookup
    route_to: small-retriever      # cheap, no reasoning needed
  classify_next_tool:
    min_capability: light-classification
    route_to: mini-classifier      # tiny, fast
  synthesize_answer:
    min_capability: frontier-reasoning
    route_to: frontier-model       # only here do we pay up

The router picks the cheapest model that clears min_capability. The agent's behavior doesn't change. The bill does.

flowchart LR
    A[Sub-task emitted] --> R{Router}
    R -->|embedding-lookup| S[small-retriever]
    R -->|light-classification| M[mini-classifier]
    R -->|frontier-reasoning| F[frontier-model]
    S --> L[per-step ledger]
    M --> L
    F --> L
    L --> C{PDPA-aligned?}
    C -->|yes| OK[serve in SG-hosted region]
    C -->|no| B[route to compliant region]

Per-step cost attribution is the missing half

Routing only helps if you can see it. Most teams have one line item: "LLM." That hides the problem — finance sees a number, not a decision.

Attach cost to every sub-task, not every request:

def route(subtask):
    model = cheapest_that_clears(subtask.min_capability)
    cost = price_per_1k[model] * estimate_tokens(subtask)
    ledger.log(subtask.name, model, cost)   # per-step, not per-request
    return model

Now the bill reads by capability tier: "retrieval $X, classification $Y, synthesis $Z." When synthesis balloons, you know which sub-task to attack — not which model to blame.

Cost and compliance converge on the same layer

Here's what surprised the team: the router was also where data-residency rules had to live. Some sub-tasks touched regulated customer data, so they had to stay in a PDPA-aligned, SG-hosted region even when a cheaper model sat elsewhere.

So the same routing layer that cut the bill also enforced jurisdiction. Cost optimization and compliance stopped being separate meetings — they became one config block.

That's the real unlock: a routing layer isn't infrastructure you tolerate. It's where your cost and your constraints get expressed as code.

The question I'll leave you with

If you mapped every sub-task in your agent loop to the cheapest model that clears its quality bar, how much of your current bill would survive?

No tool to sell, no link to click. Just this: most agent bills are really routing bills in disguise.

Retrieval is a routing problem. Your RAG stack just hides it.

TokenLat — Tue, 29 Sep 2026 05:46:44 +0000

Most RAG failures aren't model failures. They're routing failures wearing a retrieval costume.

We spend enormous effort choosing the right model for a RAG pipeline — the right context window, the right reasoning strength, the right price tier — and then we feed every query through the exact same retrieval path. Same vector index. Same top_k. Same chunking. Same reranker. A yes/no factual lookup and a ten-document synthesis get treated as the same shape of problem.

That isn't a retrieval step. That's a routing decision we forgot to make.

The retrieval step decides your model for you

Retrieval is upstream of model selection. The chunks you pull determine what context the model sees, and the shape of that context determines which model is even appropriate. If retrieval drowns a simple lookup in noise, no model choice saves it. If retrieval starves a synthesis of breadth, the strongest model just hallucinates confidently.

Yet we treat retrieval as fixed preprocessing — a hardcoded preamble the model has to swallow whole. The leverage is right there, and we've bolted it shut.

Three intents, three retrieval strategies

The same top_k=5 vector search serves three very different jobs badly:

Lookup — "What's our refund policy for cross-border orders?" Needs precision. Narrow, high-recall-on-target, maybe keyword + metadata filter, low tolerance for irrelevant context. Top-1 to top-3.
Synthesis — "Summarize the Q3 incident reports." Needs breadth. Multi-source, high-recall, dedup, section-aware assembly. Starved by a small k.
Comparison — "How does our p99 latency compare to the regional average?" Often isn't a vector problem at all. Structured extraction, maybe a SQL or semi-structured path, not free-text chunks the model has to reverse-engineer into numbers.

Feed all three through one undifferentiated retrieval call and you get the worst of each: lookups drowned, syntheses starved, comparisons given prose it can't reason over.

Concretely, a retrieval plan keyed by intent — not hardcoded — looks like this:

# retrieval plan, selected by classified intent
strategies:
  lookup:
    index: policy_dense
    k: 3
    filter: "doc_type:policy AND region:{user_region}"
    reranker: none
  synthesis:
    index: [incidents_dense, incidents_faq]
    k: 12
    filter: "date >= now() - 90d"
    reranker: cross_encoder
    fusion: rrf
  comparison:
    backend: sql_extract        # not a vector path at all
    source: metrics_db
    output: structured_rows

The defensible lever: make retrieval a routable step

The fix is to pull retrieval up into the routing layer. Before the query hits the index, classify intent and select a retrieval strategy — which index, what k, what filter expression, which reranker, what fusion method. This is the same shape as model routing: you're deciding which backend serves this step.

And here's the constraint that shapes the whole architecture: context is a budget, not a free resource. More retrieved tokens are not better — irrelevant context dilutes attention (the same dilution problem that makes an unmeasured step expensive). So retrieval routing is simultaneously a quality router and a cost router. With 25+ models behind an OpenAI-compatible interface, the router can pair "cheap retrieval + cheap model for lookup" against "broad retrieval + stronger model for synthesis" — but only if retrieval is itself a routable step, not a hardcoded preamble.

Compliance adds a second axis. For PDPA-aligned workloads, the source index and the model that reads it may need to live in the same region — SG-hosted, not shipped offshore. Retrieval routing then has a compliance gate before a quality gate: you can't route a strategy that would move regulated data out of jurisdiction, no matter how good its recall looks.

The pattern, concretely

A retrieval router takes (query, metadata) and emits a retrieval plan: { index, k, filter, reranker, fusion }. The model router then consumes the projected context shape as an input — not just the query text. Routing becomes two-stage: retrieve-shape → model-shape.

And the failover lesson carries straight over from model routing: if the primary retrieval backend degrades, the router should have a fallback retrieval path — keyword, SQL, a secondary index — not just a blind retry of the same call. A retried vector search that's already degraded returns the same degraded result, just billed twice.

The two-stage flow, side by side with the naive path:

flowchart LR
  Q[Query] --> C{Intent classifier}
  C -->|lookup| L[policy index · k=3 · filtered]
  C -->|synthesis| S[incidents · k=12 · RRF + rerank]
  C -->|comparison| P[SQL extract · structured rows]
  L --> M[Model router]
  S --> M
  P --> M
  M --> A[Answer]
  subgraph Naive[Naive RAG]
    NQ[Query] --> NV[one vector index · top_k=5]
    NV --> NM[Model]
  end

And the function that produces the plan — note the compliance gate sits before the quality gate, and the failover is a different path, not a retry:

def route_retrieval(query, metadata):
    intent = classify_intent(query)      # lookup | synthesis | comparison
    plan = STRATEGY_TABLE[intent]        # {index, k, filter, reranker, ...}

    # compliance gate BEFORE quality gate
    if not in_jurisdiction(plan.index, metadata.region):
        plan = fall_back_to(region_local_index)

    context = retrieve(plan, query)
    if context.degraded:                 # primary backend soft-failed
        context = retrieve(FAILOVER[plan], query)  # different path, not a retry
    return context   # model router consumes context.shape, not just the query

The question I keep landing on

Do you route retrieval by query intent, or does the same top_k serve every question in your stack? If you split retrieval strategies, where does that decision live — before the query reaches the index, or after the model has already committed to an answer?

Your agent's cost problem isn't the model. It's the steps you never measured.

TokenLat — Tue, 22 Sep 2026 02:47:08 +0000

A few weeks back I watched an agentic pipeline burn a month of budget in three days. The instinct everyone reached for was the obvious one: "the frontier model is too expensive, switch to a smaller one." That's the wrong diagnosis, and it's worth writing down why.

When we finally tagged every token to the (step, tool, model) tuple that spent it, the picture was uncomfortable. The expensive model wasn't the problem. The problem was that every step — the trivial classifier, the summary, the hard reasoning call — silently defaulted to it, and nobody could see which step was responsible. We'd been arguing about which model to buy when we should have been arguing about which step was allowed to call it.

Cost in an agent isn't a number. It's a tree.

In a single LLM call, cost is one line item. In an agentic loop it's a tree: each step calls a model, maybe a tool, maybe a retry, maybe a sub-step that calls again. The bill at the end is the sum over that whole tree.

If you only ever see the total, you are optimizing blind. "Make it cheaper" with only a total in front of you means guessing. And the guess is almost always "use a smaller model everywhere," which throws away quality on the steps that actually needed the frontier model while leaving the real waste untouched.

The silent default that inflates everything

Most agent frameworks let you set "the model" once, at the top. So a routing decision that should have been per-step gets frozen into a single global choice. A classification step that a small model handles fine? Frontier. A summarization of text the model just produced? Frontier. The one genuinely hard reasoning call? Also frontier — but now it's lost in the noise.

The waste here isn't that frontier models are pricey. It's that ~80% of those calls never needed them, and without per-step attribution you can't even prove which 80%. You're paying a premium on steps whose output quality wouldn't have moved.

Per-step attribution is boring infrastructure, not a feature

There's no clever trick. You log, for every call: step id, tool, model, input/output tokens, latency, retry count. That's it. Once that data exists, the waste shows up in minutes — usually a handful of steps dominate the bill, and they're rarely the ones you'd have guessed.

I've lost count of how many times the "expensive" step turned out to be a retry loop quietly re-running a hard call five times because the tool returned a shape the parser rejected, not because the model was wrong. That's not a model-cost problem. That's an un-instrumented step.

Failover is where the blind spots explode

Here's the part that actually blows budgets. A model errors, or times out, or returns garbage. The naive fix re-runs the whole trajectory — or at best the whole step — against the same model. Cost spikes exactly when you can least afford it, on the run that was already failing.

A planned failover routes that single step to a fallback at the step boundary: same tool contract, different model, the rest of the run continues. The gap between "retry the universe" and "swap the model for this one call" is most of the cost story nobody talks about, because you only see it when things break — and by then you're not measuring.

The lever lives at the routing layer, not in prompts

The point that took me too long to internalize: you cannot prompt your way to per-step cost governance. The layer that decides which model serves each step is the only place that can both measure the spend and act on it. Put the attribution and the failover decision anywhere else and you're asking a component that doesn't see the calls to fix calls it can't see.

This is also where the compliance constraint and the cost lever happen to land on the same component. In Southeast Asia, PDPA-aligned handling means prompts and outputs can't wander out of region — so the routing layer has to be SG-hosted (Tencent Cloud) and keep data resident. That same in-region gateway is exactly where per-step attribution naturally lives: the router is already on the path of every token, so measuring them costs nothing extra. A gateway that exposes 25+ models behind one OpenAI-compatible endpoint isn't just a convenience for failover — it's the only place the measurement and the swap can share state.

The question I keep coming back to

When your agent's bill doubles, can you point to the three steps that caused it — or do you just know the total went up? If you can't attribute per step, the fix was never a cheaper model. It was instrumenting the routing you already had.

Language is a routing problem, not a translation problem

TokenLat — Tue, 15 Sep 2026 08:54:48 +0000

Most teams approach a multilingual support bot as a translation problem. Wire up a translation layer, send everything to one big model, done. We did roughly that the first time, and it mostly worked — until we looked at the bill and at which customers were actually being served well.

This is the field-note version of what we learned building support bots for businesses serving Southeast Asia: the "multilingual" part is not one problem, it's three, and conflating them is why these deployments quietly cost 3x and quietly fail the Tamil- or Malay-speaking customer.

Three problems hiding under "multilingual"

1. Language detection under code-switching.
A real SEA customer doesn't pick a language. They write "boleh tolong check my order ah?" — Malay and English in one breath. A monolingual pipeline treats that as an error condition. The first routing decision isn't "which model," it's "is this even one language," and most systems answer wrong. Miss it and you've already degraded the interaction before the model sees it.

2. Language is not model size.
This is the one nobody warns you about. English is relatively simple — SVO, minimal inflection — and a small model holds quality fine. Arabic, Malay, and Tamil carry morphological complexity that needs a larger model to stay accurate. Routing every language to the same frontier model leaves money and latency on the table for English while still under-serving the harder languages. We saw a 3.5B-class model hold English quality while a 7B-class tier was needed for the others. One published pipeline routes low-resource languages through translate-to-English before classification and lifts low-tier accuracy from ~0.46 to ~0.68 Macro-F1 — same idea: not every language deserves the same path.

3. Per-step routing inside a single turn.
"Check my order" is classify-and-lookup. "Explain why my transfer failed and draft a reply" is synthesis. Same customer, same language, two completely different model tiers — and most bots send both to the expensive one because the router only sees "it's a support message."

Then PDPA decides your topology

Singapore's PDPA (and Malaysia's, moving alongside it) wants clarity on where data is processed and stored, PII redaction, and no-train terms with the underlying providers. For a support bot handling customer PII, that means inference runs in-region.

Here's the part we didn't expect: the residency requirement and the latency requirement turn out to be the same decision. Keep inference in-region and you've satisfied the residency half of PDPA while staying under the ~300ms threshold for natural conversation. Push it overseas and you're violating one and degrading the other. "Where does the model run" stops being a cost footnote and becomes the architecture. We run inference in-region (SG-hosted, on Tencent Cloud, PDPA-aligned) partly because the regulation demands it and partly because it's the cleanest way the latency math works — the compliance team and the performance team ended up arguing for the identical topology.

Why this only works if the models are a dial

None of the above is a single-model story. The language-tier decision — 3.5B for English, 7B for the harder languages, a frontier model for the synthesis step — only pays off if switching tiers is a config change, not a migration. We sit 25+ models behind one OpenAI-compatible endpoint, so "route Malay to a bigger tier" is a line in a routing table, not a procurement cycle. That unglamorous fact is the reason routing works at all: the models are a dial, not a commitment. And per-language routing is also where most of the cost saving actually comes from — you stop paying frontier rates for English classification.

The part we're still getting wrong

We measure routing quality with one global score, but the failure modes are per-tier. English can be perfect while a low-resource language drifts for a week and nobody notices until a customer complains in Tamil. We're moving to Macro-F1 per language tier as the monitoring signal. Curious whether others running multilingual routing monitor drift per-language or still trust a single aggregate — that's the open question we haven't settled.

The whole point: a multilingual bot isn't a translation layer bolted onto one model. It's a router whose first dimension is language, and language is messier, more tiered, and more regulated than the translation mental model admits.

We're writing these up as we run them — more at tokenlat.com.

Stop asking "is this task hard?" Route by what the token is for.

TokenLat — Fri, 11 Sep 2026 08:03:38 +0000

Last week I made the case for measuring whether your routing actually works — comparing output distributions per route, computing cost per successful task instead of per token, versioning the harness. If you did that, you've already beaten most teams.

But even teams that route correctly are usually routing at the wrong granularity. The decision they make is "is this request hard?" — and then the whole request goes to frontier or to cheap. The request is the wrong unit.

The expensive part was never the model

A recent write-up described cutting token use roughly 42x on a code-editing workload. The tempting read is "they found a smaller model." They didn't. The saving came from deleting the agent's explore-reread-redecide loop — the tokens spent figuring out what to do rather than doing it. The model barely changed. What changed was which job the tokens were performing.

That's the tell. When a workload gets dramatically cheaper without changing models, the lever wasn't model choice. It was that someone stopped spending tokens on a job that didn't need a frontier model at all.

Most "hard" requests are mostly mechanical

Look at almost any production request. A single task that feels hard — summarize this thread, draft a reply, extract structured fields from a doc — is usually a thin slice of genuine reasoning wrapped in a lot of mechanical work: pull the right context, classify it, format the output, validate the schema, retry on parse failure.

Route the whole request to a frontier model and you pay frontier prices for the 80% that was extraction and formatting. Route the whole request to a cheap model and you gamble the 20% that actually needed the frontier.

Per-request routing forces you to answer a question you can't reliably answer up front: "how hard is this?" Difficulty is a property you discover, not one you know at the door.

Route by token-purpose, not by difficulty

The more useful axis is what the token is for:

Instruction and system tokens. Stable, repeated, cacheable. They barely benefit from a frontier model — and they're the easiest to keep on the cheap route.
Structured extraction, classification, formatting. Routine by definition. These are where most of your volume lives, and where cheaper models — typically 70%+ cheaper, often up to 90%+ on China models — are indistinguishable in output.
First pass vs. escalation. You can't know a request is hard before you generate. A gate that decides after a cheap first pass fails — and only escalates the part that failed — spends frontier tokens only where they earn it.
The genuinely ambiguous slice. Keep frontier here. But make it the slice, not the request.

Why you can't do this by hand

You can't easily split a prompt into "tokens that are formatting" and "tokens that are reasoning" before you send it. That's the job of the call boundary. A routing gateway sits there, sees each request, and assigns the job — so the 80% mechanical portion never reaches a frontier model, and the thin hard portion does.

And you can only trust it if you can see it. This is where last week's point lands: route by job, then measure cost per successful task including retries and escalations, with request-level tracing. A gateway that hides the blend is just a black box you're told is cheaper.

A realistic shape

Take a support pipeline: classify the ticket, draft a reply, validate against a template. At frontier prices the extraction and formatting alone might run six figures a year. Split by job — cheap model for classify and format, frontier only for the draft's hard cases, escalation after a failed first pass — and the bill typically drops 70%+. On China models the same split can reach 90%+ cheaper. The saving only counts, of course, if you measured cost per successful task and the retries didn't eat it back.

Where this matters most

For teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules that US-default tooling tends to ignore, the unit of routing is also a unit of control. A gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned, exposing one OpenAI-compatible endpoint over 25+ models, lets a KL-based team route by job without shipping data out of the region or stitching five providers together.

The habit I'd steal

Next time you reach for "is this task hard?", stop. Ask "what job is this token doing?" Route the mechanical jobs to cheap models, keep frontier for the slice that earns it, escalate after the cheap pass fails — and trace every request so you can prove the blend worked. Routing by difficulty guesses at the door. Routing by token-purpose decides after it reads the work.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you route is the blend you can actually prove.

You routed 80% to cheaper models. Now measure whether it worked.

TokenLat — Thu, 03 Sep 2026 09:11:43 +0000

Last week I argued the obvious part: most production LLM traffic — extraction, classification, short rewrites — rarely needs the frontier model, and routing it to cheaper models (Chinese open-weight models are typically 70%+ cheaper, often up to 90%+ on China models) turns a flat bill into a blended one.

The easy part is done. The hard part is the question nobody answers: did the routing actually work, or did you just make your bills cheaper and your outputs silently worse?

Success rate is a half-truth

The first metric teams reach for is "did the call succeed." 200 OK. JSON parsed. Task completed. Green checkmark.

That metric can't answer the question. A cheap-model call can return 200, parse cleanly, and still be wrong in a way that only surfaces three steps later. The call that looks fine and is quietly wrong is the dangerous failure — and it's exactly the failure routing introduces, because you changed which model handles the work.

Measure distributions, not pass rates

For a sample of real traffic on each route, compare the distribution of outputs against a frontier baseline — not a single "did it pass" flag.

A median output length collapsing from ~20k tokens to 0 is a different signal than an empty result. A confidence score shifting by a point is noise; a whole cluster of outputs landing in a range the frontier never produced is a behavior change. You only see this if you measure the blend, not the pass rate. Route cheap, but verify the cheap route is still producing output you'd have trusted before.

Cost per successful task, not cost per token

The second trap is counting tokens. "I saved 70% per call" is true and irrelevant if the cheap call fails 15% of the time and each failure triggers a retry or an escalation to frontier anyway.

The number that matters is cost per successful task: total spend across the retry and escalation path divided by tasks that actually completed correctly. A cheap-first policy with a clean fallback can still win — but you have to measure the whole path, not the first hop. If your "savings" vanish once you include the retries, you haven't optimized, you've deferred the cost.

The router is part of the benchmark

Here's the one people miss: when you change the routing policy and re-run your eval, you're not measuring the model. You're measuring the router plus the model. Version the harness — the prompts, the retry logic, the memory, the tool schemas — the same way you version the model. A benchmark number with an unversioned harness is a number you can't reproduce and can't trust.

And don't trust a single run. "All 20 passed" tells you the mean passed; it tells you nothing about the tail. Variance in latency, in failure modes, in output quality lives in the tail, and the tail is where production breaks. One trial is a point estimate, not a confidence interval.

Why this bites harder on a tight budget

This is where it lands hardest for teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore. When every dollar of inference is scrutinized, "we saved 70%" on a slide means nothing if nobody measured whether the work still got done. A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team route cheap-when-possible and frontier-when-needed — but the saving only counts if you can prove the cheap route is trustworthy.

The habit I'd steal

Stop celebrating "everything succeeded." Start measuring the blend. Pick a sample, compare distributions per route, compute cost per successful task including retries, and version the harness so the number means something next month. Routing without measurement is just hope with extra steps.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you measure is the blend you actually shipped.

Most of Your LLM Spend Is Wasted on Calls That Don't Need a Frontier Model

TokenLat — Tue, 25 Aug 2026 07:31:22 +0000

If your LLM bill looks like a flat line of frontier-model calls, you're probably overpaying by 70% or more for work that a cheaper model would do just as well.

I'm not talking about a toy benchmark. I mean the actual shape of production traffic: extraction, classification, short rewrites, JSON shaping, routing decisions, draft summaries. The kinds of calls that make up the bulk of a real pipeline — and almost none of which need the most expensive model on the menu.

This post is a practical routing playbook, grounded in what Chinese open-weight models are actually capable of in 2026, and why the "one model to rule them all" habit is the most expensive default in modern AI infrastructure.

The trap: every call goes to the most expensive model

Here's a pipeline I audited recently. ~12,000 LLM calls/day:

61% were structured extraction (pull fields from an email, a ticket, a PDF)
22% were classification (intent, sentiment, spam/not-spam)
11% were short rewrites / tone adjustments
6% were genuinely hard: open-ended reasoning, ambiguous multi-step planning

Every single one was sent to the top-tier model. The 6% that justified it. And the 94% that absolutely did not.

The instinct is understandable. Frontier models are the safe choice. If you route to a cheaper model and it fails, that's on you. So everyone defaults to the best — and quietly pays a premium on traffic that would never have noticed the difference.

What Chinese models actually do now

The "Chinese models are cheap but worse" framing is two years out of date. In 2026 the top open-weight models from China — DeepSeek, Kimi, GLM, Qwen — routinely match or beat frontier models on the majority of practical, production-shaped tasks.

Two data points that reframed how I think about this:

1. The MoE argument from the model itself. A recent write-up on Kimi K3's Mixture-of-Experts architecture noted that of 896 experts, only ~1.8% activate per token. The model is already routing internally — deciding, token by token, which sliver of capacity a given input needs. If the model architects route by difficulty at the parameter level, it's a little odd that we don't route by difficulty at the request level.

2. The benchmark reality. On standard reasoning and coding evals, the gap between frontier and top Chinese models has collapsed on everything except the hardest 10–20% of tasks. For extraction, classification, and most summarization, the delta is within noise.

That last 10–20% is real and worth paying for. The other 80% is where the money leaks.

The playbook: route by scenario, not by model name

The fix isn't "use a cheap model for everything." It's a tiered policy:

# Pseudo-policy: route by task difficulty, not model brand
routes = {
    "extraction":      "cheap",    # structured field pulling — deterministic enough
    "classification":  "cheap",    # intent/sentiment/spam — high volume, low variance
    "rewrite":         "cheap",    # tone/format — cheap models are fine
    "summarize":       "cheap",    # unless source is very long/technical
    "reasoning":       "frontier", # open-ended, ambiguous, multi-step
    "planning":        "frontier", # high-stakes, low-tolerance
}

# The 80/20 default: assume cheap until a signal says otherwise
def select_model(task, signal):
    if routes.get(task) == "cheap" and signal.confidence > 0.9:
        return cheap_pool.pick()    # e.g. a top Chinese open-weight model
    return frontier_pool.pick()

The key design choice: default to cheap, escalate on signal. Not the other way around. A confidence threshold or a fast pre-classifier decides whether a call deserves the expensive model.

A gateway that exposes 25+ models behind one OpenAI-compatible endpoint makes this trivial — you write the policy once, the gateway routes, and you stop hand-wiring API keys for every provider.

What this actually saves

For most teams the math is not subtle. Routing ~80% of traffic to cheaper models — with Chinese open-weight models typically 70%+ cheaper than frontier list prices, and on many Chinese models up to 90%+ cheaper — turns a flat frontier bill into a blended one.

The number that matters isn't "how much did the model cost per token." It's "how much of my traffic never needed the expensive model in the first place." That's the lever.

Don't route blind — watch the distribution

One trap I've seen: teams set up routing, see "everything succeeds," and declare victory. But success rate lies. The dangerous failure is the call that looks fine and is quietly wrong.

If you route 80% to cheap models, the metric that matters isn't "did the call return 200." It's "does the cheap-model output still look like the data we trusted before." Watch the distribution of outputs per route. A median collapsing from ~20k to 0 is a different signal than an empty result — and it only shows up if you're measuring the blend, not just the pass rate. Route cheap, but verify the cheap route is actually producing trustworthy output. Routing without measurement is just hope with extra steps.

Why this matters more outside the US

There's a reason this playbook resonates harder in emerging markets. In Southeast Asia — Malaysia, Singapore, Indonesia — teams build on tight infrastructure budgets and answer to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore.

A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team get frontier-when-needed and cheap-when-possible, without shipping customer data through jurisdictions that complicate compliance. Cost-sensitive markets are exactly where the 80/20 split stops being an optimization and starts being the difference between a project that ships and one that doesn't.

The one rule I'd steal from this

Stop treating the model selector like a constant. Treat it like a router. Most of your calls are mechanical — route them cheap. Keep frontier for the slice that earns it. The models to do both are already here, and most of them don't cost what you're paying.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models — route by scenario, measure the blend, and tune the thresholds as signal accumulates.