Originally published on LLM Gateway. Disclosure: I work on LLM Gateway; this republication was prepared with AI assistance.
If you spend $100,000 a month on inference, the largest line item is usually
not a model that is too expensive. It is a model that is too expensive for
part of your traffic. The same endpoint answers "reword this paragraph" and
"find the race condition in this scheduler", and if you pinned a frontier model
to be safe, you are paying frontier prices for the first one several million
times a month.
This post is a worked model of LLM cost optimization through difficulty-based
routing at that scale. Every price below is the current list price from the
LLM Gateway catalogue; every assumption is
stated so you can substitute your own.
The assumptions, stated up front
This is a model, not a measurement. It rests on four things, and if any of them
are wrong for you the answer changes:
- A representative request is 2,000 input tokens and 500 output tokens. Long context shifts the arithmetic toward input price; heavy generation shifts it toward output.
- Prices are Anthropic first-party list prices, as served through the gateway.
- Traffic mixes are illustrative. The whole result hinges on what fraction of your requests are genuinely easy, and that is the number you should measure rather than borrow from a blog post.
- Quality is held constant by assumption. In reality, routing trades some quality risk on the traffic you push down. More on that at the end.
At those token counts the three models cost:
| Model | Input / output per 1M | Cost per request |
|---|---|---|
| Claude Haiku 4.5 | $1.00 / $5.00 | $0.0045 |
| Claude Sonnet 4.6 | $3.00 / $15.00 | $0.0135 |
| Claude Opus 4.6 | $5.00 / $25.00 | $0.0225 |
Case 1: you pinned the frontier model
$100,000 a month at $0.0225 a request is about 4.4 million requests. Now
route them by difficulty instead, into Low / Medium / High bands backed by those
three models:
| Traffic mix (Haiku / Sonnet / Opus) | Inference | Classifier | Net saving |
|---|---|---|---|
| Conservative — 20 / 50 / 30 | $64,000 | $467 | $35,500 (36%) |
| Typical — 40 / 40 / 20 | $52,000 | $467 | $47,500 (48%) |
| Aggressive — 60 / 30 / 10 | $40,000 | $467 | $59,500 (60%) |
Even the conservative mix — where 30% of traffic still goes to Opus and only a
fifth drops to Haiku — takes a third off the bill. The classifier is under half
a percent of the saving in every row.
Case 2: you pinned the mid-tier model
Pinning Sonnet is the more common starting point, and the honest one to model,
because the headline number is smaller. $100,000 at $0.0135 is about 7.4
million requests, and now routing cuts both ways: some traffic moves down to
Haiku, but some moves up to Opus.
| Traffic mix (Haiku / Sonnet / Opus) | Inference | Classifier | Net saving |
|---|---|---|---|
| Conservative — 30 / 55 / 15 | $90,000 | $778 | $9,200 (9%) |
| Typical — 45 / 40 / 15 | $80,000 | $778 | $19,200 (19%) |
| Aggressive — 60 / 30 / 10 | $66,700 | $778 | $32,600 (33%) |
Smaller, but note what you are buying alongside it: 15% of your traffic now
reaches a frontier model it previously did not. That is a quality gain you were
not getting before, paid for out of the savings on the easy half.
What the classifier costs
TypeSafe's Jev decision model is
$0.042 per million input tokens, with output priced at zero. One
classification is the rubric plus a bounded slice of the request:
| Scenario | Input tokens | Cost per call |
|---|---|---|
| Short prompt, 3 configured models | ~1,100 | $0.000046 |
| Mid prompt, ~10 models | ~2,500 | $0.000105 |
| Capped state, 30 models | ~4,000 | $0.000168 |
About $0.0001 a call, or 10,000 classifications per dollar. Each call is
billed to your account as its own log entry, against the same project and API
key as the request that triggered it — so the line above is something you can
read off your own usage rather than a figure you have to model. The rubric grows
with your model list — each configured model contributes its name, description
and price band to the question — so a 30-model list costs roughly twice a
3-model list per call.
And a request carrying a session id is classified once for the whole
conversation, not once per turn. For agent or chat traffic averaging ten turns,
every classifier figure above divides by ten.
The break-even is about 1%
The cleanest way to think about whether this is worth switching on:
Moving one request from Sonnet to Haiku saves $0.009. A classification costs
$0.000105. The classifier pays for itself if more than about 1.2% of your
requests move down a band.
Below that threshold your traffic is uniformly hard and you should pin a model
and skip the classifier. Above it — and almost all mixed production traffic is
well above it — the classifier is noise against the saving. That asymmetry is
the actual argument for difficulty routing: the downside is bounded at a tenth
of a percent of spend, and the upside is tens of percent.
Where this model is optimistic
Three honest caveats, because a savings estimate with no failure modes is a
sales pitch:
Quality is not free. Every request you push to a cheaper model is a bet that
the cheaper model handles it. A classifier makes that bet better than a coin
flip, but it does not make it free. Measure output quality on the traffic that
moves down, not just the invoice.
Your mix is probably not ours. The 40/40/20 row is an illustration. Run your
own traffic through a classifier in observe-only mode — point the Low and Medium
bands at the model you already use — and read the recorded difficulty
distribution off your logs before you change any routing.
Cheap traffic changes the ratio. These numbers assume a $0.0135–$0.0225
average request. If your average request costs $0.002, the classifier is around
5% of spend rather than 0.5%, and the case is weaker unless sessions amortize it.
Running it
On LLM Gateway this is configuration, not code. Choose the models smart may
resolve to and the classifier, and the gateway sorts them into price bands and
routes each request to the matching one. Every decision — the difficulty, the
task type, the candidates, the model chosen — lands on the request's log entry,
so you can audit the mix you actually got against the mix you assumed here.
Smart routing itself carries no platform fee: you pay for the models it selects
and for each classifier call, at the rates above. A verdict reused from a sticky
session and a classifier call that fails are both charged nothing. It is
available to every organization including pay-as-you-go while it is in beta; it
is not available on DevPass yet. If you want the routing logic explicit instead, a
dynamic route can branch on
the same verdict.
Top comments (2)
I appreciate that you've explicitly labeled these numbers as a model rather than measured production savings. That distinction matters, especially when a 40/40/20 traffic mix can make the projected economics look very compelling.
The observe-only recommendation is particularly interesting. Recording difficulty classifications before changing the serving model seems like a sensible first step, but I'd be curious about the next stage of validation.
For example, knowing that 40% of requests were classified as easy doesn't necessarily establish that a cheaper model would handle those requests successfully. You'd still need some way to compare task outcomes, retry rates, and latency against the existing model.
I also like that your request logs preserve the difficulty, candidates, and selected model. One useful extension might be recording whether a routing decision was simulated, evaluated against an alternative, or actually executed, alongside the evidence supporting any claimed savings.
How would you approach validating the classifier's decisions before moving from observe-only mode to live routing? Would you start with a representative offline evaluation set, shadow comparisons, or a small controlled rollout?
Thanks, this is the right question, and the honest answer is that the gateway only gives you the first half of it today.
What I'd do, in order:
Offline set first, but small and task-specific. A few hundred real requests from the traffic you'd push down, each with a pass/fail judgement that comes from the task itself: did the rewrite keep the facts, did the extraction match the schema, did the user accept the answer. Run them through the cheaper model and count failures. This tells you whether "easy" according to the classifier is "easy" for Haiku, which, as you say, is a different claim. It won't tell you the distribution, only the risk per band.
Observe-only for the distribution. Point every band at the model you already use, let it run for a week, and read the difficulty mix off the logs. Then join that against step 1: the only band whose failure rate matters is the one that would actually move.
Controlled rollout, split by session. A dynamic route with a percentage node: 90% to the current model, 10% to the routed set. The draw is deterministic per session id, so a conversation stays in one arm, and you compare arms on the signals you listed: retries and regenerations, follow-up turns in the same session, latency, and whatever outcome signal the product has (accept, copy, thumbs). Widen the slice as the gap stays flat.
Shadow comparisons I'd skip for most teams. Running every request twice doubles the bill you're trying to cut, and you still need a judge, so it collapses into step 1 with worse economics.
On your extension: the log entry today carries the difficulty, task type, candidates and the chosen model, but not whether that decision was executed or only simulated, and nothing about the outcome. The workaround right now is request metadata: send the arm as an
X-LLMGateway-Arm: control|routedheader so it lands on the log entry and can be filtered on, and join the outcome from your side on the request id. A first-class "routing mode" field and an outcome hook are a good idea, I'm taking that back to the team.