DEV Community: TokenLat

Your agent's cost problem isn't the model. It's the steps you never measured.

TokenLat — Tue, 22 Sep 2026 02:47:08 +0000

A few weeks back I watched an agentic pipeline burn a month of budget in three days. The instinct everyone reached for was the obvious one: "the frontier model is too expensive, switch to a smaller one." That's the wrong diagnosis, and it's worth writing down why.

When we finally tagged every token to the (step, tool, model) tuple that spent it, the picture was uncomfortable. The expensive model wasn't the problem. The problem was that every step — the trivial classifier, the summary, the hard reasoning call — silently defaulted to it, and nobody could see which step was responsible. We'd been arguing about which model to buy when we should have been arguing about which step was allowed to call it.

Cost in an agent isn't a number. It's a tree.

In a single LLM call, cost is one line item. In an agentic loop it's a tree: each step calls a model, maybe a tool, maybe a retry, maybe a sub-step that calls again. The bill at the end is the sum over that whole tree.

If you only ever see the total, you are optimizing blind. "Make it cheaper" with only a total in front of you means guessing. And the guess is almost always "use a smaller model everywhere," which throws away quality on the steps that actually needed the frontier model while leaving the real waste untouched.

The silent default that inflates everything

Most agent frameworks let you set "the model" once, at the top. So a routing decision that should have been per-step gets frozen into a single global choice. A classification step that a small model handles fine? Frontier. A summarization of text the model just produced? Frontier. The one genuinely hard reasoning call? Also frontier — but now it's lost in the noise.

The waste here isn't that frontier models are pricey. It's that ~80% of those calls never needed them, and without per-step attribution you can't even prove which 80%. You're paying a premium on steps whose output quality wouldn't have moved.

Per-step attribution is boring infrastructure, not a feature

There's no clever trick. You log, for every call: step id, tool, model, input/output tokens, latency, retry count. That's it. Once that data exists, the waste shows up in minutes — usually a handful of steps dominate the bill, and they're rarely the ones you'd have guessed.

I've lost count of how many times the "expensive" step turned out to be a retry loop quietly re-running a hard call five times because the tool returned a shape the parser rejected, not because the model was wrong. That's not a model-cost problem. That's an un-instrumented step.

Failover is where the blind spots explode

Here's the part that actually blows budgets. A model errors, or times out, or returns garbage. The naive fix re-runs the whole trajectory — or at best the whole step — against the same model. Cost spikes exactly when you can least afford it, on the run that was already failing.

A planned failover routes that single step to a fallback at the step boundary: same tool contract, different model, the rest of the run continues. The gap between "retry the universe" and "swap the model for this one call" is most of the cost story nobody talks about, because you only see it when things break — and by then you're not measuring.

The lever lives at the routing layer, not in prompts

The point that took me too long to internalize: you cannot prompt your way to per-step cost governance. The layer that decides which model serves each step is the only place that can both measure the spend and act on it. Put the attribution and the failover decision anywhere else and you're asking a component that doesn't see the calls to fix calls it can't see.

This is also where the compliance constraint and the cost lever happen to land on the same component. In Southeast Asia, PDPA-aligned handling means prompts and outputs can't wander out of region — so the routing layer has to be SG-hosted (Tencent Cloud) and keep data resident. That same in-region gateway is exactly where per-step attribution naturally lives: the router is already on the path of every token, so measuring them costs nothing extra. A gateway that exposes 25+ models behind one OpenAI-compatible endpoint isn't just a convenience for failover — it's the only place the measurement and the swap can share state.

The question I keep coming back to

When your agent's bill doubles, can you point to the three steps that caused it — or do you just know the total went up? If you can't attribute per step, the fix was never a cheaper model. It was instrumenting the routing you already had.

Language is a routing problem, not a translation problem

TokenLat — Tue, 15 Sep 2026 08:54:48 +0000

Most teams approach a multilingual support bot as a translation problem. Wire up a translation layer, send everything to one big model, done. We did roughly that the first time, and it mostly worked — until we looked at the bill and at which customers were actually being served well.

This is the field-note version of what we learned building support bots for businesses serving Southeast Asia: the "multilingual" part is not one problem, it's three, and conflating them is why these deployments quietly cost 3x and quietly fail the Tamil- or Malay-speaking customer.

Three problems hiding under "multilingual"

1. Language detection under code-switching.
A real SEA customer doesn't pick a language. They write "boleh tolong check my order ah?" — Malay and English in one breath. A monolingual pipeline treats that as an error condition. The first routing decision isn't "which model," it's "is this even one language," and most systems answer wrong. Miss it and you've already degraded the interaction before the model sees it.

2. Language is not model size.
This is the one nobody warns you about. English is relatively simple — SVO, minimal inflection — and a small model holds quality fine. Arabic, Malay, and Tamil carry morphological complexity that needs a larger model to stay accurate. Routing every language to the same frontier model leaves money and latency on the table for English while still under-serving the harder languages. We saw a 3.5B-class model hold English quality while a 7B-class tier was needed for the others. One published pipeline routes low-resource languages through translate-to-English before classification and lifts low-tier accuracy from ~0.46 to ~0.68 Macro-F1 — same idea: not every language deserves the same path.

3. Per-step routing inside a single turn.
"Check my order" is classify-and-lookup. "Explain why my transfer failed and draft a reply" is synthesis. Same customer, same language, two completely different model tiers — and most bots send both to the expensive one because the router only sees "it's a support message."

Then PDPA decides your topology

Singapore's PDPA (and Malaysia's, moving alongside it) wants clarity on where data is processed and stored, PII redaction, and no-train terms with the underlying providers. For a support bot handling customer PII, that means inference runs in-region.

Here's the part we didn't expect: the residency requirement and the latency requirement turn out to be the same decision. Keep inference in-region and you've satisfied the residency half of PDPA while staying under the ~300ms threshold for natural conversation. Push it overseas and you're violating one and degrading the other. "Where does the model run" stops being a cost footnote and becomes the architecture. We run inference in-region (SG-hosted, on Tencent Cloud, PDPA-aligned) partly because the regulation demands it and partly because it's the cleanest way the latency math works — the compliance team and the performance team ended up arguing for the identical topology.

Why this only works if the models are a dial

None of the above is a single-model story. The language-tier decision — 3.5B for English, 7B for the harder languages, a frontier model for the synthesis step — only pays off if switching tiers is a config change, not a migration. We sit 25+ models behind one OpenAI-compatible endpoint, so "route Malay to a bigger tier" is a line in a routing table, not a procurement cycle. That unglamorous fact is the reason routing works at all: the models are a dial, not a commitment. And per-language routing is also where most of the cost saving actually comes from — you stop paying frontier rates for English classification.

The part we're still getting wrong

We measure routing quality with one global score, but the failure modes are per-tier. English can be perfect while a low-resource language drifts for a week and nobody notices until a customer complains in Tamil. We're moving to Macro-F1 per language tier as the monitoring signal. Curious whether others running multilingual routing monitor drift per-language or still trust a single aggregate — that's the open question we haven't settled.

The whole point: a multilingual bot isn't a translation layer bolted onto one model. It's a router whose first dimension is language, and language is messier, more tiered, and more regulated than the translation mental model admits.

We're writing these up as we run them — more at tokenlat.com.

Stop asking "is this task hard?" Route by what the token is for.

TokenLat — Fri, 11 Sep 2026 08:03:38 +0000

Last week I made the case for measuring whether your routing actually works — comparing output distributions per route, computing cost per successful task instead of per token, versioning the harness. If you did that, you've already beaten most teams.

But even teams that route correctly are usually routing at the wrong granularity. The decision they make is "is this request hard?" — and then the whole request goes to frontier or to cheap. The request is the wrong unit.

The expensive part was never the model

A recent write-up described cutting token use roughly 42x on a code-editing workload. The tempting read is "they found a smaller model." They didn't. The saving came from deleting the agent's explore-reread-redecide loop — the tokens spent figuring out what to do rather than doing it. The model barely changed. What changed was which job the tokens were performing.

That's the tell. When a workload gets dramatically cheaper without changing models, the lever wasn't model choice. It was that someone stopped spending tokens on a job that didn't need a frontier model at all.

Most "hard" requests are mostly mechanical

Look at almost any production request. A single task that feels hard — summarize this thread, draft a reply, extract structured fields from a doc — is usually a thin slice of genuine reasoning wrapped in a lot of mechanical work: pull the right context, classify it, format the output, validate the schema, retry on parse failure.

Route the whole request to a frontier model and you pay frontier prices for the 80% that was extraction and formatting. Route the whole request to a cheap model and you gamble the 20% that actually needed the frontier.

Per-request routing forces you to answer a question you can't reliably answer up front: "how hard is this?" Difficulty is a property you discover, not one you know at the door.

Route by token-purpose, not by difficulty

The more useful axis is what the token is for:

Instruction and system tokens. Stable, repeated, cacheable. They barely benefit from a frontier model — and they're the easiest to keep on the cheap route.
Structured extraction, classification, formatting. Routine by definition. These are where most of your volume lives, and where cheaper models — typically 70%+ cheaper, often up to 90%+ on China models — are indistinguishable in output.
First pass vs. escalation. You can't know a request is hard before you generate. A gate that decides after a cheap first pass fails — and only escalates the part that failed — spends frontier tokens only where they earn it.
The genuinely ambiguous slice. Keep frontier here. But make it the slice, not the request.

Why you can't do this by hand

You can't easily split a prompt into "tokens that are formatting" and "tokens that are reasoning" before you send it. That's the job of the call boundary. A routing gateway sits there, sees each request, and assigns the job — so the 80% mechanical portion never reaches a frontier model, and the thin hard portion does.

And you can only trust it if you can see it. This is where last week's point lands: route by job, then measure cost per successful task including retries and escalations, with request-level tracing. A gateway that hides the blend is just a black box you're told is cheaper.

A realistic shape

Take a support pipeline: classify the ticket, draft a reply, validate against a template. At frontier prices the extraction and formatting alone might run six figures a year. Split by job — cheap model for classify and format, frontier only for the draft's hard cases, escalation after a failed first pass — and the bill typically drops 70%+. On China models the same split can reach 90%+ cheaper. The saving only counts, of course, if you measured cost per successful task and the retries didn't eat it back.

Where this matters most

For teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules that US-default tooling tends to ignore, the unit of routing is also a unit of control. A gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned, exposing one OpenAI-compatible endpoint over 25+ models, lets a KL-based team route by job without shipping data out of the region or stitching five providers together.

The habit I'd steal

Next time you reach for "is this task hard?", stop. Ask "what job is this token doing?" Route the mechanical jobs to cheap models, keep frontier for the slice that earns it, escalate after the cheap pass fails — and trace every request so you can prove the blend worked. Routing by difficulty guesses at the door. Routing by token-purpose decides after it reads the work.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you route is the blend you can actually prove.

You routed 80% to cheaper models. Now measure whether it worked.

TokenLat — Thu, 03 Sep 2026 09:11:43 +0000

Last week I argued the obvious part: most production LLM traffic — extraction, classification, short rewrites — rarely needs the frontier model, and routing it to cheaper models (Chinese open-weight models are typically 70%+ cheaper, often up to 90%+ on China models) turns a flat bill into a blended one.

The easy part is done. The hard part is the question nobody answers: did the routing actually work, or did you just make your bills cheaper and your outputs silently worse?

Success rate is a half-truth

The first metric teams reach for is "did the call succeed." 200 OK. JSON parsed. Task completed. Green checkmark.

That metric can't answer the question. A cheap-model call can return 200, parse cleanly, and still be wrong in a way that only surfaces three steps later. The call that looks fine and is quietly wrong is the dangerous failure — and it's exactly the failure routing introduces, because you changed which model handles the work.

Measure distributions, not pass rates

For a sample of real traffic on each route, compare the distribution of outputs against a frontier baseline — not a single "did it pass" flag.

A median output length collapsing from ~20k tokens to 0 is a different signal than an empty result. A confidence score shifting by a point is noise; a whole cluster of outputs landing in a range the frontier never produced is a behavior change. You only see this if you measure the blend, not the pass rate. Route cheap, but verify the cheap route is still producing output you'd have trusted before.

Cost per successful task, not cost per token

The second trap is counting tokens. "I saved 70% per call" is true and irrelevant if the cheap call fails 15% of the time and each failure triggers a retry or an escalation to frontier anyway.

The number that matters is cost per successful task: total spend across the retry and escalation path divided by tasks that actually completed correctly. A cheap-first policy with a clean fallback can still win — but you have to measure the whole path, not the first hop. If your "savings" vanish once you include the retries, you haven't optimized, you've deferred the cost.

The router is part of the benchmark

Here's the one people miss: when you change the routing policy and re-run your eval, you're not measuring the model. You're measuring the router plus the model. Version the harness — the prompts, the retry logic, the memory, the tool schemas — the same way you version the model. A benchmark number with an unversioned harness is a number you can't reproduce and can't trust.

And don't trust a single run. "All 20 passed" tells you the mean passed; it tells you nothing about the tail. Variance in latency, in failure modes, in output quality lives in the tail, and the tail is where production breaks. One trial is a point estimate, not a confidence interval.

Why this bites harder on a tight budget

This is where it lands hardest for teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore. When every dollar of inference is scrutinized, "we saved 70%" on a slide means nothing if nobody measured whether the work still got done. A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team route cheap-when-possible and frontier-when-needed — but the saving only counts if you can prove the cheap route is trustworthy.

The habit I'd steal

Stop celebrating "everything succeeded." Start measuring the blend. Pick a sample, compare distributions per route, compute cost per successful task including retries, and version the harness so the number means something next month. Routing without measurement is just hope with extra steps.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you measure is the blend you actually shipped.

Most of Your LLM Spend Is Wasted on Calls That Don't Need a Frontier Model

TokenLat — Tue, 25 Aug 2026 07:31:22 +0000

If your LLM bill looks like a flat line of frontier-model calls, you're probably overpaying by 70% or more for work that a cheaper model would do just as well.

I'm not talking about a toy benchmark. I mean the actual shape of production traffic: extraction, classification, short rewrites, JSON shaping, routing decisions, draft summaries. The kinds of calls that make up the bulk of a real pipeline — and almost none of which need the most expensive model on the menu.

This post is a practical routing playbook, grounded in what Chinese open-weight models are actually capable of in 2026, and why the "one model to rule them all" habit is the most expensive default in modern AI infrastructure.

The trap: every call goes to the most expensive model

Here's a pipeline I audited recently. ~12,000 LLM calls/day:

61% were structured extraction (pull fields from an email, a ticket, a PDF)
22% were classification (intent, sentiment, spam/not-spam)
11% were short rewrites / tone adjustments
6% were genuinely hard: open-ended reasoning, ambiguous multi-step planning

Every single one was sent to the top-tier model. The 6% that justified it. And the 94% that absolutely did not.

The instinct is understandable. Frontier models are the safe choice. If you route to a cheaper model and it fails, that's on you. So everyone defaults to the best — and quietly pays a premium on traffic that would never have noticed the difference.

What Chinese models actually do now

The "Chinese models are cheap but worse" framing is two years out of date. In 2026 the top open-weight models from China — DeepSeek, Kimi, GLM, Qwen — routinely match or beat frontier models on the majority of practical, production-shaped tasks.

Two data points that reframed how I think about this:

1. The MoE argument from the model itself. A recent write-up on Kimi K3's Mixture-of-Experts architecture noted that of 896 experts, only ~1.8% activate per token. The model is already routing internally — deciding, token by token, which sliver of capacity a given input needs. If the model architects route by difficulty at the parameter level, it's a little odd that we don't route by difficulty at the request level.

2. The benchmark reality. On standard reasoning and coding evals, the gap between frontier and top Chinese models has collapsed on everything except the hardest 10–20% of tasks. For extraction, classification, and most summarization, the delta is within noise.

That last 10–20% is real and worth paying for. The other 80% is where the money leaks.

The playbook: route by scenario, not by model name

The fix isn't "use a cheap model for everything." It's a tiered policy:

# Pseudo-policy: route by task difficulty, not model brand
routes = {
    "extraction":      "cheap",    # structured field pulling — deterministic enough
    "classification":  "cheap",    # intent/sentiment/spam — high volume, low variance
    "rewrite":         "cheap",    # tone/format — cheap models are fine
    "summarize":       "cheap",    # unless source is very long/technical
    "reasoning":       "frontier", # open-ended, ambiguous, multi-step
    "planning":        "frontier", # high-stakes, low-tolerance
}

# The 80/20 default: assume cheap until a signal says otherwise
def select_model(task, signal):
    if routes.get(task) == "cheap" and signal.confidence > 0.9:
        return cheap_pool.pick()    # e.g. a top Chinese open-weight model
    return frontier_pool.pick()

The key design choice: default to cheap, escalate on signal. Not the other way around. A confidence threshold or a fast pre-classifier decides whether a call deserves the expensive model.

A gateway that exposes 25+ models behind one OpenAI-compatible endpoint makes this trivial — you write the policy once, the gateway routes, and you stop hand-wiring API keys for every provider.

What this actually saves

For most teams the math is not subtle. Routing ~80% of traffic to cheaper models — with Chinese open-weight models typically 70%+ cheaper than frontier list prices, and on many Chinese models up to 90%+ cheaper — turns a flat frontier bill into a blended one.

The number that matters isn't "how much did the model cost per token." It's "how much of my traffic never needed the expensive model in the first place." That's the lever.

Don't route blind — watch the distribution

One trap I've seen: teams set up routing, see "everything succeeds," and declare victory. But success rate lies. The dangerous failure is the call that looks fine and is quietly wrong.

If you route 80% to cheap models, the metric that matters isn't "did the call return 200." It's "does the cheap-model output still look like the data we trusted before." Watch the distribution of outputs per route. A median collapsing from ~20k to 0 is a different signal than an empty result — and it only shows up if you're measuring the blend, not just the pass rate. Route cheap, but verify the cheap route is actually producing trustworthy output. Routing without measurement is just hope with extra steps.

Why this matters more outside the US

There's a reason this playbook resonates harder in emerging markets. In Southeast Asia — Malaysia, Singapore, Indonesia — teams build on tight infrastructure budgets and answer to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore.

A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team get frontier-when-needed and cheap-when-possible, without shipping customer data through jurisdictions that complicate compliance. Cost-sensitive markets are exactly where the 80/20 split stops being an optimization and starts being the difference between a project that ships and one that doesn't.

The one rule I'd steal from this

Stop treating the model selector like a constant. Treat it like a router. Most of your calls are mechanical — route them cheap. Keep frontier for the slice that earns it. The models to do both are already here, and most of them don't cost what you're paying.

If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models — route by scenario, measure the blend, and tune the thresholds as signal accumulates.

Your coding agent bills per task, not per token

TokenLat — Tue, 18 Aug 2026 07:15:01 +0000

If you price a coding agent the way you price a chatbot, you will misread the bill every time.

A normal code chat turns one prompt into one completion. A coding agent runs a loop: it reads files, calls tools, reads the output, edits, re-runs the tests, and self-corrects — often a dozen turns. 2026 research (Stanford, MIT, and others) puts a single agentic-coding task at roughly 1–3.5 million tokens, with ~76% of them reads. One study found agentic coding burns up to 1000× more tokens than a normal code chat.

The takeaway is not "agents are expensive." It is that the unit is wrong. You do not pay per million tokens. You pay per task — and two levers move that number 5–10× more than the model name does.

Lever 1: caching (the one that beats model-switching)

A coding agent re-sends the same context — system prompt, project files, tool definitions — on nearly every turn. Cache reads bill a fraction of base input. On a gateway with transparent cache pricing, the gap is stark:

Model	Input (¤/1M)	Cache read (¤/1M)	Cache advantage
deepseek-v4-Pro	560	10	~56×
qwen3.5-plus	130	20	~6.5×
hy3-preview	180	60	3×
minimax-m3	390	80	~4.9×

Structure prompts so the stable context is cacheable and new content appends at the end, and that discount compounds across a long session. One change, bigger impact than most model swaps.

Lever 2: route by difficulty, not by habit

The same studies show a ~40× per-task cost spread across models for the same task. The pattern that holds: run the cheapest model that passes your evals by default, and escalate only the hard tasks. A unified gateway with 25+ models across 8 providers makes that a one-line switch instead of a refactor:

REQUEST -> AUTH -> ROUTE(model:"auto") -> RESPONSE -> METER
                                    ^ picks by cost / quality / speed

The catch most teams miss: you cannot route by difficulty if you cannot see the cost. That is why per-request traceability matters — every call should show model choice, latency, tokens, and cost on one line, not buried in a dashboard.

The move

Stop benchmarking models on $/M. Benchmark them on $/task for 15–30 real tasks from your own repo. Cache the stable context. Route the routine work to cheaper models. And make the meter visible, because an agent that hides its own cost is the most expensive agent you will run.

Building agentic systems on a unified, OpenAI-compatible gateway (SG-hosted, PDPA-aligned, up to 90%+ cheaper on China models): tokenlat.com

Cache affinity in practice: 5 patterns that keep your agent's prompt cache warm

TokenLat — Tue, 11 Aug 2026 07:46:05 +0000

You shipped an agent. Tokens/month exploded. You switched to a "cheaper" model and the bill barely moved.

Here's the thing nobody prints on the pricing page: in a loop, the model price is often a rounding error next to the repeat tax — and the repeat tax is, at its core, a cache-affinity problem.

This is the practical follow-up to why agentic systems should care about cache-hit pricing. That post argued the cost lives in cache behavior, not in raw model price. This one is about what you can actually do about it.

What prefix caching actually rewards

Most providers (OpenAI, Gemini, Anthropic, and the OpenAI-compatible gateways on top of them) offer automatic prefix caching: if the start of your prompt is byte-for-byte identical to a previous request, the cached tokens cost a fraction of a fresh input token.

The keyword is byte-for-byte identical, and it only helps if the identical part sits at the front of the prompt. Reorder one line, inject a timestamp above the system prompt, or re-serialize history with a fresh UUID, and you've evicted your own cache. The model never sees the hit — you just pay full input, every step.

So "cheaper model" optimizes the wrong number. The lever is cache affinity: how stable is your prefix across the loop?

5 patterns that keep the cache warm

Stable system prompt + fixed tool schemas at the very top. Tools definitions rarely change between steps. Put them first, verbatim, and stop touching them.
Append-only history. Don't re-serialize the whole conversation each step. Keep a canonical transcript and append; let the unchanged prefix stay cached.
Inject volatile context after the stable prefix. Scratchpads, retrieved docs, and tool results are fine — as long as they sit below the system prompt + history, not above it.
Route the easy 80% to a small model, but preserve the shared prefix. Routing by scenario is smart. Just don't let the small model re-format the prefix; a different tokenizer can silently break the cache.
One gateway, one canonical formatter. When three sub-agents each format "the context" their own way, you get three incompatible prefixes and zero cache reuse. Centralize prompt assembly.

5 patterns that evict your cache (the O(n²) traps)

Re-sending the entire conversation every step. Frameworks that replay full memory each turn pay O(n²) tokens over a session. The prefix can never stabilize.
Re-serializing history with a timestamp or UUID in the prefix. A new updated_at each call = a new prefix = no cache, forever.
Putting volatile content above the stable prefix. Current time, request id, trace id — if it's before the system prompt, it poisons every cache hit downstream.
Mixing tokenizers across models without a stable canonical form. Switching models mid-loop without canonicalizing the prefix resets the cache and doubles your input cost.
No observability. If you can't see cache hit vs miss per request, you can't tell which of the above you're doing. You're flying blind on the single biggest cost lever.

The trace you're missing

The fix that makes everything else measurable is boring but decisive: a per-request trace that reports, for every call, whether the prefix hit cache and how many tokens were charged vs cached.

Once you have that, the O(n²) traps show up as a line item. You stop guessing and start watching the cache-hit ratio the way you watch p99 latency. That's the difference between "we cut model cost" and "we cut agent cost."

Landing AI means making this the default

"AI 落地" (getting AI into production) is sold as a model-access problem. It isn't. Access is solved. The hard part is the boring operational layer: warm caches, visible traces, sane routing — by default, not as a heroic refactor after the bill arrives.

If you want a gateway that surfaces cache-hit/miss per request and keeps one canonical prefix across models, that's the whole point of what we're building at TokenLat: https://tokenlat.com

Why agentic systems should care about cache-hit pricing

TokenLat — Fri, 07 Aug 2026 08:32:46 +0000

The metric that quietly decides your agent bill isn't the input price of your model. It's how much you pay to read what you already sent.

If you run multi-step agents, you've probably had this moment: a task you expected to cost pennies comes back as a small surprise on the invoice. You didn't change models. You didn't prompt more. So where did the tokens go?

Most of the time, they went to paying for the same context, again and again.

A 2-minute task can fire 40+ billable calls

Here's a shape I keep seeing. An agent does a "2-minute" job:

reads a file
drafts a plan
calls a tool
reflects on the result
retries
summarizes

Each step is an LLM call. And each call re-sends the same scaffolding: the system prompt, the task description, and — critically — the growing conversation history. A step that adds 200 new tokens of thinking can still carry 4,000 tokens of context it already paid for once.

Multiply that across 40 steps and the math stops being about "model price." It's about how many times you re-pay for context you already have.

What cache-hit pricing actually is

Normal input pricing charges you per token you send, every time. Cache-hit pricing changes the unit: if the provider already has your prefix cached (because you sent it recently and it hasn't expired), the read of that cached prefix is billed at a deep discount instead of full input price.

On a unified gateway this is visible per model. Two concrete examples from the model catalog:

DeepSeek-v4-Pro: ¤10 / 1M tokens for a cached read (¤ is the platform's billing unit)
Qwen3.5-plus: ¤20 / 1M tokens for a cached read

Those aren't the full input rates — they're the cached-read rates, and that's the number that matters for agentic workloads, because agentic workloads are mostly repeats.

The takeaway isn't "this model is cheaper." It's: for any loop that re-sends context, cached-read price is the real marginal cost, and most teams optimize for the wrong number.

The math that actually moves the bill

Skip the exact figures and look at the shape. Say a loop runs 40 calls, and each carries ~4k tokens of repeated context plus ~200 tokens of new content.

Pay full input on every repeated token: you're billed for 4k × 40 = 160k "new" tokens that were actually old.
Pay cached-read on the repeated prefix: that 160k drops to a fraction — the cached-read rate instead of full input.

On the two models above, cached reads land at ¤10–¤20 / 1M versus full input that's multiple times higher. The loop's cost doesn't go to zero, but the repeat tax collapses. In agentic systems, the repeat tax is most of the bill — so this is where the 70%+ savings actually live, not in "pick a cheaper model."

You can't optimize what you can't see

There's a trap here. Cache-hit pricing only helps if you can see hits and misses. A black-box API that just returns text hides the one number you need: was this token a cache hit or a fresh charge?

A request trace makes it observable. Each call should expose its stages:

REQUEST — what came in
AUTH — who/what called
ROUTE — which model served it
RESPONSE — what came back
METER — what it cost, including cache hit/miss

When every call shows its cache hit/miss and per-stage cost, the loop stops being a mystery. You can see which step blows the budget and whether your prefix is actually staying warm.

Routing + cache affinity is the real lever

Cache-hit pricing is necessary but not sufficient. The other half is keeping the cache warm, and that's a routing problem.

The 80/20 pattern holds: route the easy 80% of calls to a small/fast model, keep frontier for the hard 20%. But the part people miss is cache affinity — if you keep the same system prompt and stable prefix across the loop, the cache stays warm and the cheap reads keep hitting. Change the prefix on every step (reformat the history, rewrite the system prompt, shuffle the order) and you silently evict your own cache. You pay full input forever.

With model: "auto" style routing behind one OpenAI-compatible endpoint, the loop doesn't have to think about which model serves which step — but it still has to respect cache affinity, because that's what turns "cheap model" into "cheap and cached."

A practical cache-friendly checklist

Keep the system prompt and stable prefix byte-identical across loop steps
Append new content; don't rebuild the whole context each time
Prefer providers/models that expose cache-hit pricing and a request trace
Watch cache hit rate per step, not just total spend
Route by task, but preserve prefix stability so the cache survives

The point

Agentic cost isn't a model-selection problem. It's a repeat-tax problem: how many times you pay to read context you already sent, and whether you can see it happening.

Cache-hit pricing + request trace + cache-friendly routing is the combination that turns a scary agent bill into a boring one. Most teams optimize the first and ignore the other two — which is why their invoices still surprise them.

If you want the routing playbook that pairs with this (the 80/20 split and how to keep the cache warm), it's at https://tokenlat.com — but the idea stands on its own: stop counting input price. Start counting cache hits.

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

TokenLat — Fri, 31 Jul 2026 04:43:52 +0000

The 80/20 routing playbook: cut your AI agent bill 70%+ without touching quality

Your agent's token bill is probably 5x higher than it needs to be — not because the models are expensive, but because of routing discipline. Most teams wire every call to one frontier model and call it a day. This post is the practical fix: how to send the easy 80% of agent calls to cheap models, keep frontier for the hard 20%, and not lose a point of quality doing it.

Everything below uses real numbers from a gateway that exposes 25 models across 8 providers behind one OpenAI-compatible API.

The default that's costing you

Here's the shape of a typical agent loop:

classify the request
extract structured fields
pick a tool
call the tool, parse the result
summarize what happened
decide the next step
draft the reply

Of those, maybe one or two steps actually need frontier-level reasoning. The rest are classification, extraction, and formatting — work that a fast, cheap model does indistinguishably from a flagship. Yet most setups send all of it to gpt-5.5 (or whatever their default is) because it's the path of least resistance.

That's the tax. You're paying flagship prices for steps that don't need it.

The 80/20 split, with the math

Route by task difficulty, not by model loyalty:

Easy 80% — classification, extraction, summarization, short tool-formatting, routing decisions → a fast China-model tier
Medium 15% — coding, planning, multi-step reasoning → a strong reasoning tier
Hard 5% — the genuinely gnarly frontier cases → flagship, reserved

A blended-cost estimate (input tokens, per 1M):

80% × ¤160   (cheap tier input)   = ¤128
15% × ¤560   (reasoning tier)     = ¤84
 5% × ¤7000  (flagship input)     = ¤350
---------------------------------------
blended input cost               ≈ ¤562 / 1M
vs. all-flagship                 = ¤7000 / 1M
savings                          ≈ 70%+ in this example

Even with a more conservative split, 70%+ cheaper is the floor — and quality doesn't move, because the hard calls still go to frontier.

The trap: why "just use the cheapest model" backfires

Naive cost-cutting picks one cheap model for everything. That fails for two reasons:

Quality cliffs. A ¤160/1M model is great at extraction and terrible at planning a 12-file refactor. Push hard tasks down and users notice.
Cache invalidation (the silent tax). This is the one almost nobody budgets for. Long agent conversations re-send the system prompt + tool definitions on every turn. If you switch providers mid-conversation, you reset prompt-cache affinity and re-pay the full prefix each turn. A conversation that should be cheap suddenly isn't.

The win isn't "use a cheap model" — it's routing discipline with cache affinity intact: pin the easy 80% to a stable cheap model so the cache stays warm, and only escalate to frontier for the calls that earn it.

What a real routing layer looks like

With a unified gateway, this is roughly ten lines. One API, models swapped by model: — no per-provider SDK juggling:

def route(task):
    if task.complexity == "easy":        # classify, extract, summarize
        return "TokenLat-deepseek-v4-Flash"   # ¤160/1M input
    if task.needs_reasoning:             # code, plan, multi-step
        return "TokenLat-deepseek-v4-Pro"     # ¤560/1M input, ¤10/1M cache read
    return "chatgpt-5.5"                 # frontier, only the hard 5%

resp = client.responses.create(
    model=route(task),
    input=task.prompt,
)

Or skip the router entirely and let the gateway decide per call:

resp = client.responses.create(
    model="auto",          # gateway picks the right tier per request
    input=task.prompt,
)

model: "auto" is the zero-config version of the same idea — the gateway applies routing policy so you don't hand-roll it.

Real numbers from a 25-model gateway

A unified gateway should show you its pricing up front. Per 1M tokens, in credits (¤), here's a slice (input / cache-read):

TokenLat-deepseek-v4-Flash — cheap / fast tier → ¤160 / ¤40
TokenLat-qwen3.5-plus — general tier → ¤130 / ¤20
TokenLat-deepseek-v4-Pro — reasoning tier → ¤560 / ¤10 (lowest cache-read)
TokenLat-glm-5.1 — Chinese-stable → ¤950 / ¤160
TokenLat-kimi-k3 — flagship (China) → ¤3000 / ¤300
gemini-3.1-pro — frontier → ¤2800 / ¤300
chatgpt-5.5 — frontier → ¤7000 / ¤700

The headline gap: cheap-tier input is ~44x cheaper than flagship input. That's the entire 80/20 argument in one line.

Cache-hit pricing — the agentic loophole

For agentic systems, the input price is almost a distraction. What matters is the cache-read price, because the static prefix (system prompt, tool schemas, retrieved context) gets re-sent every turn.

TokenLat-deepseek-v4-Pro caches reads at ¤10/1M
TokenLat-qwen3.5-plus at ¤20/1M

So a long-running agent loop that pins a stable model and keeps its prefix cached can serve most turns at the cache-read rate — a fraction of the input rate. The providers that reset your cache on every switch quietly erase this saving. Cache affinity is a routing decision, not an afterthought.

Make it visible

You can't optimize what you can't see. Every request through a proper gateway should leave a trace:

REQUEST → AUTH → ROUTE(auto→text-pro) → RESPONSE → METER
region: SEA | status: 200 OK | latency: 842ms | tokens: 1,284 | cost: ¤0.0048

Five stages, one line, per call. When a bill spikes, you see which model, which route, and which request — instead of staring at a monthly total and guessing.

Don't guess — measure quality

The one thing this playbook assumes: you actually check that the cheap tier is good enough. Route a sample of real tasks through both tiers, score the outputs on your success criteria, and only then commit the split. Most teams find the quality cliff is far smaller than they feared — and the ones where it isn't, they've already reserved frontier for.

The takeaway: the savings aren't in hunting for a cheaper model. They're in routing the easy 80% to a stable cheap tier, keeping cache affinity, and escalating only the hard calls to frontier. Do that and 70%+ cheaper is routine — with quality intact.

If you want to try this without wiring up eight providers yourself: TokenLat is a unified AI gateway for Malaysia & Southeast Asia — one OpenAI-compatible API across 25 models, with auto routing and per-request cost traces. The model list above is live at tokenlat.com.

What's your current cost per agent-task — single digits or scaling faster than you'd like? Curious how the 80/20 split would land on your workload.

Your agent's token bill is 5x too high — and it's not the model price

TokenLat — Mon, 27 Jul 2026 09:45:27 +0000

Most teams blame their model provider when the inference bill spikes. They're looking at the wrong line item.

The real leak is architecture — and it's the difference between a token bill that scales with value and one that scales with chaos. Here's what we see shipping agentic systems in production.

The hidden multiplier: agent loops

A "2-minute task" is never one call. An agent fires 30–60 tool calls per run, and most frameworks stuff the entire conversation history into every prompt. So a job you'd estimate at ~4K tokens becomes 40 calls × 8K context = 320K tokens — billed at frontier rates.

Frontier pricing per call looks cheap. Multiplied by agent-loop iterations, it quietly becomes the largest line in your cloud bill.

The 80/20 of inference

Not every call needs a frontier model.

~80% of agent traffic is routing, extraction, formatting, classification, summarization. Trivial. Leading efficient models — including top China models — handle these at near-parity.
~20% is genuine reasoning, open-ended generation, ambiguous planning. That's where frontier earns its price.

Route the 80% to efficient models and reserve frontier for the 20%. Same output quality. A fraction of the bill.

A unified gateway beats a drawer of API keys

The trap most teams hit: they wire 4 providers with 4 clients, then let a naïve router "roam" between them. On failover it loses cache affinity, re-embeds context, and your 1.5x cost target drifts back toward ~1x — or worse.

A single OpenAI-compatible endpoint across OpenAI + Gemini + leading China models fixes this:

One client, one code path.
Provider pinning holds cache locality; it only fails over on hard error, not price drift.
Your application code never changes when you swap a model.

In SEA, "PDPA-aligned" is the baseline, not a premium

For Malaysia and SEA teams, inference isn't just a cost question — it's a compliance one.

PDPA requires 72-hour breach notification and a designated DPO.
In-region data residency (SG-hosted) is now the default expectation, not a paid add-on.
The adoption gap is real: ~93% of the workforce uses GenAI, but ~73% are stuck at L1 (experimenting, not in production). The blocker is almost always cost + compliance friction — not the models.

Compliance stops being a tax and becomes table stakes when your gateway is PDPA-aligned by design.

The numbers

In our pricing analysis, frontier output runs on the order of 100x the cost of efficient China-model output per million tokens. Route the easy 80% there and blend:

Up to 90%+ cheaper on the routine 80%
70%+ lower blended bill overall
Quality on the 20% that matters: unchanged

The fix is architecture, not negotiation

You don't fix a 5x token bill by begging your provider for a discount. You fix it by routing, caching, and unifying — then letting compliance be the default instead of the exception.

We break down agent cost architecture weekly. If you're shipping agents and your bill is climbing faster than your usage, come find us:

Discord: discord.gg/rxEtWG897V
Site: tokenlat.com