DEV Community: TokenLat

You routed 80% to cheaper models. Now measure whether it worked.

TokenLat — Thu, 03 Sep 2026 09:11:43 +0000

Last week I argued the obvious part: most production LLM traffic — extraction, classification, short rewrites — rarely needs the frontier model, and routing it to cheaper models (Chinese open-weight models are typically 70%+ cheaper, often up to 90%+ on China models) turns a flat bill into a blended one.

The easy part is done. The hard part is the question nobody answers: did the routing actually work, or did you just make your bills cheaper and your outputs silently worse?

Success rate is a half-truth

The first metric teams reach for is "did the call succeed." 200 OK. JSON parsed. Task completed. Green checkmark.

That metric can't answer the question. A cheap-model call can return 200, parse cleanly, and still be wrong in a way that only surfaces three steps later. The call that looks fine and is quietly wrong is the dangerous failure — and it's exactly the failure routing introduces, because you changed which model handles the work.

Measure distributions, not pass rates

For a sample of real traffic on each route, compare the distribution of outputs against a frontier baseline — not a single "did it pass" flag.

A median output length collapsing from ~20k tokens to 0 is a different signal than an empty result. A confidence score shifting by a point is noise; a whole cluster of outputs landing in a range the frontier never produced is a behavior change. You only see this if you measure the blend, not the pass rate. Route cheap, but verify the cheap route is still producing output you'd have trusted before.

Cost per successful task, not cost per token

The second trap is counting tokens. "I saved 70% per call" is true and irrelevant if the cheap call fails 15% of the time and each failure triggers a retry or an escalation to frontier anyway.

The number that matters is cost per successful task: total spend across the retry and escalation path divided by tasks that actually completed correctly. A cheap-first policy with a clean fallback can still win — but you have to measure the whole path, not the first hop. If your "savings" vanish once you include the retries, you haven't optimized, you've deferred the cost.

The router is part of the benchmark

Here's the one people miss: when you change the routing policy and re-run your eval, you're not measuring the model. You're measuring the router plus the model. Version the harness — the prompts, the retry logic, the memory, the tool schemas — the same way you version the model. A benchmark number with an unversioned harness is a number you can't reproduce and can't trust.

And don't trust a single run. "All 20 passed" tells you the mean passed; it tells you nothing about the tail. Variance in latency, in failure modes, in output quality lives in the tail, and the tail is where production breaks. One trial is a point estimate, not a confidence interval.

Why this bites harder on a tight budget

This is where it lands hardest for teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore. When every dollar of inference is scrutinized, "we saved 70%" on a slide means nothing if nobody measured whether the work still got done. A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team route cheap-when-possible and frontier-when-needed — but the saving only counts if you can prove the cheap route is trustworthy.

The habit I'd steal

Stop celebrating "everything succeeded." Start measuring the blend. Pick a sample, compare distributions per route, compute cost per successful task including retries, and version the harness so the number means something next month. Routing without measurement is just hope with extra steps.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you measure is the blend you actually shipped.

Most of Your LLM Spend Is Wasted on Calls That Don't Need a Frontier Model

TokenLat — Tue, 25 Aug 2026 07:31:22 +0000

If your LLM bill looks like a flat line of frontier-model calls, you're probably overpaying by 70% or more for work that a cheaper model would do just as well.

I'm not talking about a toy benchmark. I mean the actual shape of production traffic: extraction, classification, short rewrites, JSON shaping, routing decisions, draft summaries. The kinds of calls that make up the bulk of a real pipeline — and almost none of which need the most expensive model on the menu.

This post is a practical routing playbook, grounded in what Chinese open-weight models are actually capable of in 2026, and why the "one model to rule them all" habit is the most expensive default in modern AI infrastructure.

The trap: every call goes to the most expensive model

Here's a pipeline I audited recently. ~12,000 LLM calls/day:

61% were structured extraction (pull fields from an email, a ticket, a PDF)
22% were classification (intent, sentiment, spam/not-spam)
11% were short rewrites / tone adjustments
6% were genuinely hard: open-ended reasoning, ambiguous multi-step planning

Every single one was sent to the top-tier model. The 6% that justified it. And the 94% that absolutely did not.

The instinct is understandable. Frontier models are the safe choice. If you route to a cheaper model and it fails, that's on you. So everyone defaults to the best — and quietly pays a premium on traffic that would never have noticed the difference.

What Chinese models actually do now

The "Chinese models are cheap but worse" framing is two years out of date. In 2026 the top open-weight models from China — DeepSeek, Kimi, GLM, Qwen — routinely match or beat frontier models on the majority of practical, production-shaped tasks.

Two data points that reframed how I think about this:

1. The MoE argument from the model itself. A recent write-up on Kimi K3's Mixture-of-Experts architecture noted that of 896 experts, only ~1.8% activate per token. The model is already routing internally — deciding, token by token, which sliver of capacity a given input needs. If the model architects route by difficulty at the parameter level, it's a little odd that we don't route by difficulty at the request level.

2. The benchmark reality. On standard reasoning and coding evals, the gap between frontier and top Chinese models has collapsed on everything except the hardest 10–20% of tasks. For extraction, classification, and most summarization, the delta is within noise.

That last 10–20% is real and worth paying for. The other 80% is where the money leaks.

The playbook: route by scenario, not by model name

The fix isn't "use a cheap model for everything." It's a tiered policy:

# Pseudo-policy: route by task difficulty, not model brand
routes = {
    "extraction":      "cheap",    # structured field pulling — deterministic enough
    "classification":  "cheap",    # intent/sentiment/spam — high volume, low variance
    "rewrite":         "cheap",    # tone/format — cheap models are fine
    "summarize":       "cheap",    # unless source is very long/technical
    "reasoning":       "frontier", # open-ended, ambiguous, multi-step
    "planning":        "frontier", # high-stakes, low-tolerance
}

# The 80/20 default: assume cheap until a signal says otherwise
def select_model(task, signal):
    if routes.get(task) == "cheap" and signal.confidence > 0.9:
        return cheap_pool.pick()    # e.g. a top Chinese open-weight model
    return frontier_pool.pick()

The key design choice: default to cheap, escalate on signal. Not the other way around. A confidence threshold or a fast pre-classifier decides whether a call deserves the expensive model.

A gateway that exposes 25+ models behind one OpenAI-compatible endpoint makes this trivial — you write the policy once, the gateway routes, and you stop hand-wiring API keys for every provider.

What this actually saves

For most teams the math is not subtle. Routing ~80% of traffic to cheaper models — with Chinese open-weight models typically 70%+ cheaper than frontier list prices, and on many Chinese models up to 90%+ cheaper — turns a flat frontier bill into a blended one.

The number that matters isn't "how much did the model cost per token." It's "how much of my traffic never needed the expensive model in the first place." That's the lever.

Don't route blind — watch the distribution

One trap I've seen: teams set up routing, see "everything succeeds," and declare victory. But success rate lies. The dangerous failure is the call that looks fine and is quietly wrong.

If you route 80% to cheap models, the metric that matters isn't "did the call return 200." It's "does the cheap-model output still look like the data we trusted before." Watch the distribution of outputs per route. A median collapsing from ~20k to 0 is a different signal than an empty result — and it only shows up if you're measuring the blend, not just the pass rate. Route cheap, but verify the cheap route is actually producing trustworthy output. Routing without measurement is just hope with extra steps.

Why this matters more outside the US

There's a reason this playbook resonates harder in emerging markets. In Southeast Asia — Malaysia, Singapore, Indonesia — teams build on tight infrastructure budgets and answer to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore.

A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team get frontier-when-needed and cheap-when-possible, without shipping customer data through jurisdictions that complicate compliance. Cost-sensitive markets are exactly where the 80/20 split stops being an optimization and starts being the difference between a project that ships and one that doesn't.

The one rule I'd steal from this

Stop treating the model selector like a constant. Treat it like a router. Most of your calls are mechanical — route them cheap. Keep frontier for the slice that earns it. The models to do both are already here, and most of them don't cost what you're paying.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models — route by scenario, measure the blend, and tune the thresholds as signal accumulates.

Your coding agent bills per task, not per token

TokenLat — Tue, 18 Aug 2026 07:15:01 +0000

If you price a coding agent the way you price a chatbot, you will misread the bill every time.

A normal code chat turns one prompt into one completion. A coding agent runs a loop: it reads files, calls tools, reads the output, edits, re-runs the tests, and self-corrects — often a dozen turns. 2026 research (Stanford, MIT, and others) puts a single agentic-coding task at roughly 1–3.5 million tokens, with ~76% of them reads. One study found agentic coding burns up to 1000× more tokens than a normal code chat.

The takeaway is not "agents are expensive." It is that the unit is wrong. You do not pay per million tokens. You pay per task — and two levers move that number 5–10× more than the model name does.

Lever 1: caching (the one that beats model-switching)

A coding agent re-sends the same context — system prompt, project files, tool definitions — on nearly every turn. Cache reads bill a fraction of base input. On a gateway with transparent cache pricing, the gap is stark:

Model	Input (¤/1M)	Cache read (¤/1M)	Cache advantage
deepseek-v4-Pro	560	10	~56×
qwen3.5-plus	130	20	~6.5×
hy3-preview	180	60	3×
minimax-m3	390	80	~4.9×

Structure prompts so the stable context is cacheable and new content appends at the end, and that discount compounds across a long session. One change, bigger impact than most model swaps.

Lever 2: route by difficulty, not by habit

The same studies show a ~40× per-task cost spread across models for the same task. The pattern that holds: run the cheapest model that passes your evals by default, and escalate only the hard tasks. A unified gateway with 25+ models across 8 providers makes that a one-line switch instead of a refactor:

REQUEST -> AUTH -> ROUTE(model:"auto") -> RESPONSE -> METER
                                    ^ picks by cost / quality / speed

The catch most teams miss: you cannot route by difficulty if you cannot see the cost. That is why per-request traceability matters — every call should show model choice, latency, tokens, and cost on one line, not buried in a dashboard.

The move

Stop benchmarking models on $/M. Benchmark them on $/task for 15–30 real tasks from your own repo. Cache the stable context. Route the routine work to cheaper models. And make the meter visible, because an agent that hides its own cost is the most expensive agent you will run.

Building agentic systems on a unified, OpenAI-compatible gateway (SG-hosted, PDPA-aligned, up to 90%+ cheaper on China models): tokenlat.com

Cache affinity in practice: 5 patterns that keep your agent's prompt cache warm

TokenLat — Tue, 11 Aug 2026 07:46:05 +0000

You shipped an agent. Tokens/month exploded. You switched to a "cheaper" model and the bill barely moved.

Here's the thing nobody prints on the pricing page: in a loop, the model price is often a rounding error next to the repeat tax — and the repeat tax is, at its core, a cache-affinity problem.

This is the practical follow-up to why agentic systems should care about cache-hit pricing. That post argued the cost lives in cache behavior, not in raw model price. This one is about what you can actually do about it.

What prefix caching actually rewards

Most providers (OpenAI, Gemini, Anthropic, and the OpenAI-compatible gateways on top of them) offer automatic prefix caching: if the start of your prompt is byte-for-byte identical to a previous request, the cached tokens cost a fraction of a fresh input token.

The keyword is byte-for-byte identical, and it only helps if the identical part sits at the front of the prompt. Reorder one line, inject a timestamp above the system prompt, or re-serialize history with a fresh UUID, and you've evicted your own cache. The model never sees the hit — you just pay full input, every step.

So "cheaper model" optimizes the wrong number. The lever is cache affinity: how stable is your prefix across the loop?

5 patterns that keep the cache warm

Stable system prompt + fixed tool schemas at the very top. Tools definitions rarely change between steps. Put them first, verbatim, and stop touching them.
Append-only history. Don't re-serialize the whole conversation each step. Keep a canonical transcript and append; let the unchanged prefix stay cached.
Inject volatile context after the stable prefix. Scratchpads, retrieved docs, and tool results are fine — as long as they sit below the system prompt + history, not above it.
Route the easy 80% to a small model, but preserve the shared prefix. Routing by scenario is smart. Just don't let the small model re-format the prefix; a different tokenizer can silently break the cache.
One gateway, one canonical formatter. When three sub-agents each format "the context" their own way, you get three incompatible prefixes and zero cache reuse. Centralize prompt assembly.

5 patterns that evict your cache (the O(n²) traps)

Re-sending the entire conversation every step. Frameworks that replay full memory each turn pay O(n²) tokens over a session. The prefix can never stabilize.
Re-serializing history with a timestamp or UUID in the prefix. A new updated_at each call = a new prefix = no cache, forever.
Putting volatile content above the stable prefix. Current time, request id, trace id — if it's before the system prompt, it poisons every cache hit downstream.
Mixing tokenizers across models without a stable canonical form. Switching models mid-loop without canonicalizing the prefix resets the cache and doubles your input cost.
No observability. If you can't see cache hit vs miss per request, you can't tell which of the above you're doing. You're flying blind on the single biggest cost lever.

The trace you're missing

The fix that makes everything else measurable is boring but decisive: a per-request trace that reports, for every call, whether the prefix hit cache and how many tokens were charged vs cached.

Once you have that, the O(n²) traps show up as a line item. You stop guessing and start watching the cache-hit ratio the way you watch p99 latency. That's the difference between "we cut model cost" and "we cut agent cost."

Landing AI means making this the default

"AI 落地" (getting AI into production) is sold as a model-access problem. It isn't. Access is solved. The hard part is the boring operational layer: warm caches, visible traces, sane routing — by default, not as a heroic refactor after the bill arrives.

If you want a gateway that surfaces cache-hit/miss per request and keeps one canonical prefix across models, that's the whole point of what we're building at TokenLat: https://tokenlat.com

Why agentic systems should care about cache-hit pricing

TokenLat — Fri, 07 Aug 2026 08:32:46 +0000

The metric that quietly decides your agent bill isn't the input price of your model. It's how much you pay to read what you already sent.

If you run multi-step agents, you've probably had this moment: a task you expected to cost pennies comes back as a small surprise on the invoice. You didn't change models. You didn't prompt more. So where did the tokens go?

Most of the time, they went to paying for the same context, again and again.

A 2-minute task can fire 40+ billable calls

Here's a shape I keep seeing. An agent does a "2-minute" job:

reads a file
drafts a plan
calls a tool
reflects on the result
retries
summarizes

Each step is an LLM call. And each call re-sends the same scaffolding: the system prompt, the task description, and — critically — the growing conversation history. A step that adds 200 new tokens of thinking can still carry 4,000 tokens of context it already paid for once.

Multiply that across 40 steps and the math stops being about "model price." It's about how many times you re-pay for context you already have.

What cache-hit pricing actually is

Normal input pricing charges you per token you send, every time. Cache-hit pricing changes the unit: if the provider already has your prefix cached (because you sent it recently and it hasn't expired), the read of that cached prefix is billed at a deep discount instead of full input price.

On a unified gateway this is visible per model. Two concrete examples from the model catalog:

DeepSeek-v4-Pro: ¤10 / 1M tokens for a cached read (¤ is the platform's billing unit)
Qwen3.5-plus: ¤20 / 1M tokens for a cached read

Those aren't the full input rates — they're the cached-read rates, and that's the number that matters for agentic workloads, because agentic workloads are mostly repeats.

The takeaway isn't "this model is cheaper." It's: for any loop that re-sends context, cached-read price is the real marginal cost, and most teams optimize for the wrong number.

The math that actually moves the bill

Skip the exact figures and look at the shape. Say a loop runs 40 calls, and each carries ~4k tokens of repeated context plus ~200 tokens of new content.

Pay full input on every repeated token: you're billed for 4k × 40 = 160k "new" tokens that were actually old.
Pay cached-read on the repeated prefix: that 160k drops to a fraction — the cached-read rate instead of full input.

On the two models above, cached reads land at ¤10–¤20 / 1M versus full input that's multiple times higher. The loop's cost doesn't go to zero, but the repeat tax collapses. In agentic systems, the repeat tax is most of the bill — so this is where the 70%+ savings actually live, not in "pick a cheaper model."

You can't optimize what you can't see

There's a trap here. Cache-hit pricing only helps if you can see hits and misses. A black-box API that just returns text hides the one number you need: was this token a cache hit or a fresh charge?

A request trace makes it observable. Each call should expose its stages:

REQUEST — what came in
AUTH — who/what called
ROUTE — which model served it
RESPONSE — what came back
METER — what it cost, including cache hit/miss

When every call shows its cache hit/miss and per-stage cost, the loop stops being a mystery. You can see which step blows the budget and whether your prefix is actually staying warm.

Routing + cache affinity is the real lever

Cache-hit pricing is necessary but not sufficient. The other half is keeping the cache warm, and that's a routing problem.

The 80/20 pattern holds: route the easy 80% of calls to a small/fast model, keep frontier for the hard 20%. But the part people miss is cache affinity — if you keep the same system prompt and stable prefix across the loop, the cache stays warm and the cheap reads keep hitting. Change the prefix on every step (reformat the history, rewrite the system prompt, shuffle the order) and you silently evict your own cache. You pay full input forever.

With model: "auto" style routing behind one OpenAI-compatible endpoint, the loop doesn't have to think about which model serves which step — but it still has to respect cache affinity, because that's what turns "cheap model" into "cheap and cached."

A practical cache-friendly checklist

Keep the system prompt and stable prefix byte-identical across loop steps
Append new content; don't rebuild the whole context each time
Prefer providers/models that expose cache-hit pricing and a request trace
Watch cache hit rate per step, not just total spend
Route by task, but preserve prefix stability so the cache survives

The point

Agentic cost isn't a model-selection problem. It's a repeat-tax problem: how many times you pay to read context you already sent, and whether you can see it happening.

Cache-hit pricing + request trace + cache-friendly routing is the combination that turns a scary agent bill into a boring one. Most teams optimize the first and ignore the other two — which is why their invoices still surprise them.

If you want the routing playbook that pairs with this (the 80/20 split and how to keep the cache warm), it's at https://tokenlat.com — but the idea stands on its own: stop counting input price. Start counting cache hits.

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

TokenLat — Fri, 31 Jul 2026 04:43:52 +0000

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

Your agent's token bill is probably 5x higher than it needs to be — not because the models are expensive, but because of routing discipline. Most teams wire every call to one frontier model and call it a day. This post is the practical fix: how to send the easy 80% of agent calls to cheap models, keep frontier for the hard 20%, and not lose a point of quality doing it.

Everything below uses real numbers from a gateway that exposes 25 models across 8 providers behind one OpenAI-compatible API.

The default that's costing you

Here's the shape of a typical agent loop:

classify the request
extract structured fields
pick a tool
call the tool, parse the result
summarize what happened
decide the next step
draft the reply

Of those, maybe one or two steps actually need frontier-level reasoning. The rest are classification, extraction, and formatting — work that a fast, cheap model does indistinguishably from a flagship. Yet most setups send all of it to gpt-5.5 (or whatever their default is) because it's the path of least resistance.

That's the tax. You're paying flagship prices for steps that don't need it.

The 80/20 split, with the math

Route by task difficulty, not by model loyalty:

Easy 80% — classification, extraction, summarization, short tool-formatting, routing decisions → a fast China-model tier
Medium 15% — coding, planning, multi-step reasoning → a strong reasoning tier
Hard 5% — the genuinely gnarly frontier cases → flagship, reserved

A blended-cost estimate (input tokens, per 1M):

80% × ¤160   (cheap tier input)   = ¤128
15% × ¤560   (reasoning tier)     = ¤84
 5% × ¤7000  (flagship input)     = ¤350
---------------------------------------
blended input cost               ≈ ¤562 / 1M
vs. all-flagship                 = ¤7000 / 1M
savings                          ≈ 70%+ in this example

Even with a more conservative split, 70%+ cheaper is the floor — and quality doesn't move, because the hard calls still go to frontier.

The trap: why "just use the cheapest model" backfires

Naive cost-cutting picks one cheap model for everything. That fails for two reasons:

Quality cliffs. A ¤160/1M model is great at extraction and terrible at planning a 12-file refactor. Push hard tasks down and users notice.
Cache invalidation (the silent tax). This is the one almost nobody budgets for. Long agent conversations re-send the system prompt + tool definitions on every turn. If you switch providers mid-conversation, you reset prompt-cache affinity and re-pay the full prefix each turn. A conversation that should be cheap suddenly isn't.

The win isn't "use a cheap model" — it's routing discipline with cache affinity intact: pin the easy 80% to a stable cheap model so the cache stays warm, and only escalate to frontier for the calls that earn it.

What a real routing layer looks like

With a unified gateway, this is roughly ten lines. One API, models swapped by model: — no per-provider SDK juggling:

def route(task):
    if task.complexity == "easy":        # classify, extract, summarize
        return "TokenLat-deepseek-v4-Flash"   # ¤160/1M input
    if task.needs_reasoning:             # code, plan, multi-step
        return "TokenLat-deepseek-v4-Pro"     # ¤560/1M input, ¤10/1M cache read
    return "chatgpt-5.5"                 # frontier, only the hard 5%

resp = client.responses.create(
    model=route(task),
    input=task.prompt,
)

Or skip the router entirely and let the gateway decide per call:

resp = client.responses.create(
    model="auto",          # gateway picks the right tier per request
    input=task.prompt,
)

model: "auto" is the zero-config version of the same idea — the gateway applies routing policy so you don't hand-roll it.

Real numbers from a 25-model gateway

A unified gateway should show you its pricing up front. Per 1M tokens, in credits (¤), here's a slice (input / cache-read):

TokenLat-deepseek-v4-Flash — cheap / fast tier → ¤160 / ¤40
TokenLat-qwen3.5-plus — general tier → ¤130 / ¤20
TokenLat-deepseek-v4-Pro — reasoning tier → ¤560 / ¤10 (lowest cache-read)
TokenLat-glm-5.1 — Chinese-stable → ¤950 / ¤160
TokenLat-kimi-k3 — flagship (China) → ¤3000 / ¤300
gemini-3.1-pro — frontier → ¤2800 / ¤300
chatgpt-5.5 — frontier → ¤7000 / ¤700

The headline gap: cheap-tier input is ~44x cheaper than flagship input. That's the entire 80/20 argument in one line.

Cache-hit pricing — the agentic loophole

For agentic systems, the input price is almost a distraction. What matters is the cache-read price, because the static prefix (system prompt, tool schemas, retrieved context) gets re-sent every turn.

TokenLat-deepseek-v4-Pro caches reads at ¤10/1M
TokenLat-qwen3.5-plus at ¤20/1M

So a long-running agent loop that pins a stable model and keeps its prefix cached can serve most turns at the cache-read rate — a fraction of the input rate. The providers that reset your cache on every switch quietly erase this saving. Cache affinity is a routing decision, not an afterthought.

Make it visible

You can't optimize what you can't see. Every request through a proper gateway should leave a trace:

REQUEST → AUTH → ROUTE(auto→text-pro) → RESPONSE → METER
region: SEA | status: 200 OK | latency: 842ms | tokens: 1,284 | cost: ¤0.0048

Five stages, one line, per call. When a bill spikes, you see which model, which route, and which request — instead of staring at a monthly total and guessing.

Don't guess — measure quality

The one thing this playbook assumes: you actually check that the cheap tier is good enough. Route a sample of real tasks through both tiers, score the outputs on your success criteria, and only then commit the split. Most teams find the quality cliff is far smaller than they feared — and the ones where it isn't, they've already reserved frontier for.

The takeaway: the savings aren't in hunting for a cheaper model. They're in routing the easy 80% to a stable cheap tier, keeping cache affinity, and escalating only the hard calls to frontier. Do that and 70%+ cheaper is routine — with quality intact.

If you want to try this without wiring up eight providers yourself: TokenLat is a unified AI gateway for Malaysia & Southeast Asia — one OpenAI-compatible API across 25 models, with auto routing and per-request cost traces. The model list above is live at tokenlat.com.

What's your current cost per agent-task — single digits or scaling faster than you'd like? Curious how the 80/20 split would land on your workload.

Your agent's token bill is 5x too high — and it's not the model price

TokenLat — Mon, 27 Jul 2026 09:45:27 +0000

Most teams blame their model provider when the inference bill spikes. They're looking at the wrong line item.

The real leak is architecture — and it's the difference between a token bill that scales with value and one that scales with chaos. Here's what we see shipping agentic systems in production.

The hidden multiplier: agent loops

A "2-minute task" is never one call. An agent fires 30–60 tool calls per run, and most frameworks stuff the entire conversation history into every prompt. So a job you'd estimate at ~4K tokens becomes 40 calls × 8K context = 320K tokens — billed at frontier rates.

Frontier pricing per call looks cheap. Multiplied by agent-loop iterations, it quietly becomes the largest line in your cloud bill.

The 80/20 of inference

Not every call needs a frontier model.

~80% of agent traffic is routing, extraction, formatting, classification, summarization. Trivial. Leading efficient models — including top China models — handle these at near-parity.
~20% is genuine reasoning, open-ended generation, ambiguous planning. That's where frontier earns its price.

Route the 80% to efficient models and reserve frontier for the 20%. Same output quality. A fraction of the bill.

A unified gateway beats a drawer of API keys

The trap most teams hit: they wire 4 providers with 4 clients, then let a naïve router "roam" between them. On failover it loses cache affinity, re-embeds context, and your 1.5x cost target drifts back toward ~1x — or worse.

A single OpenAI-compatible endpoint across OpenAI + Gemini + leading China models fixes this:

One client, one code path.
Provider pinning holds cache locality; it only fails over on hard error, not price drift.
Your application code never changes when you swap a model.

In SEA, "PDPA-aligned" is the baseline, not a premium

For Malaysia and SEA teams, inference isn't just a cost question — it's a compliance one.

PDPA requires 72-hour breach notification and a designated DPO.
In-region data residency (SG-hosted) is now the default expectation, not a paid add-on.
The adoption gap is real: ~93% of the workforce uses GenAI, but ~73% are stuck at L1 (experimenting, not in production). The blocker is almost always cost + compliance friction — not the models.

Compliance stops being a tax and becomes table stakes when your gateway is PDPA-aligned by design.

The numbers

In our pricing analysis, frontier output runs on the order of 100x the cost of efficient China-model output per million tokens. Route the easy 80% there and blend:

Up to 90%+ cheaper on the routine 80%
70%+ lower blended bill overall
Quality on the 20% that matters: unchanged

The fix is architecture, not negotiation

You don't fix a 5x token bill by begging your provider for a discount. You fix it by routing, caching, and unifying — then letting compliance be the default instead of the exception.

We break down agent cost architecture weekly. If you're shipping agents and your bill is climbing faster than your usage, come find us:

Discord: discord.gg/rxEtWG897V
Site: tokenlat.com