On July 30, OpenAI cut GPT-5.6 Luna to $0.20 per million input tokens and $1.20 per million output — down from $1 and $6. An 80% cut. Azure matched it on August 1.
We did what most teams did with that news: nothing. Our gateway sent every request to the strong model, because it always had, and because "just use the cheap model" sounds like a decision that comes back as a support ticket three weeks later.
Then someone put the bill next to the traffic mix, and the awkward part was obvious. The overwhelming majority of our requests were title this thread, summarise this diff, extract the fields from this form, name this file. We were paying frontier prices to generate three-word document titles.
Here's what we shipped instead. It's about forty lines of Go, it moved 81% of requests off the expensive model, and it cut the bill by 71%.
It also broke four things, and those are the interesting part.
The approach that doesn't work: classify the prompt
The obvious design — the one everyone writes first — is a classifier in front of the router:
Look at the incoming request. Decide if it's easy or hard. Send it to the model that matches.
We built this. It's worse than it looks, for three reasons.
You need a model to run the classifier. Now every request pays an extra call before any work happens. A cheap classifier is another Luna call and another 300ms; an accurate classifier is a strong-model call, which is the cost you were trying to avoid.
Short is not easy. "Fix the timezone bug" is eleven characters and needs everything you've got. "Summarise the following 4,000-word RFC" is long and trivial. Length, token count, keyword lists — every cheap heuristic we tried correlated with the shape of the request and not with the difficulty of it.
You're predicting the answer before you've seen it. That's the actual problem. Difficulty is a property of the work, and the only honest way to learn it is to do the work.
The rule: don't classify the prompt. Run the cheap model and judge the output.
Cheap-first with an escalation gate
The design that works is embarrassingly simple:
- Send it to the cheap model.
- Look at what came back.
- If it fails a gate, run it again on the strong model and return that instead.
The thing that makes this viable is arithmetic that surprised us. At our average request shape — roughly 3,000 input tokens and 700 output — a Luna call costs about $0.0014 and a strong-model call about $0.0145. Ten to one.
So a request that gets escalated costs 0.0014 + 0.0145 = $0.0159 instead of $0.0145. About 10% more.
Work out the break-even escalation rate:
cheap + (p × strong) < strong
p < (strong − cheap) / strong
p < (0.0145 − 0.0014) / 0.0145
p < 0.90
You would have to escalate nine times out of ten before the cheap attempt costs you money. We escalate 15% of the time. Cost is not the constraint here, and if you're arguing about whether cheap-first is worth the double billing, you're arguing about the wrong resource.
Latency is the constraint. We'll come back to that.
The router
type Result struct {
Text string
Model string
Tokens Usage
}
// A gate returns the reason this answer can't be trusted, or "" to accept it.
type Gate func(req Request, out string, fin FinishReason) string
func (r *Router) Do(ctx context.Context, req Request) (Result, error) {
if req.ForceStrong || r.alwaysStrong[req.Kind] {
return r.call(ctx, r.strong, req)
}
cheap, err := r.call(ctx, r.cheap, req)
if err != nil {
// Availability, not quality. Fall through, don't fail the request.
return r.call(ctx, r.strong, req)
}
for _, gate := range r.gates {
reason := gate(req, cheap.Text, cheap.Finish)
if reason == "" {
continue
}
r.metrics.Escalate(req.Kind, reason)
strong, err := r.call(ctx, r.strong, req) // NB: req, not req+cheap.Text
if err != nil {
return cheap, nil // degraded, not failed
}
return strong, nil
}
r.metrics.Accept(req.Kind)
return cheap, nil
}
Two lines in there are load-bearing and neither is obvious.
return r.call(ctx, r.strong, req) when the cheap call errors — a 429 or a timeout is an availability problem, and availability problems should not surface to the user as a failed request when you have a second provider sitting right there.
return cheap, nil when the strong call errors — you already have an answer. It may be a worse answer. Shipping a worse answer beats shipping a spinner.
And the comment on the escalation call is the one people get wrong, which is failure mode #4 below.
The four gates
The gates are the whole product. The router is plumbing.
var defaultGates = []Gate{
SchemaInvalid, // had to be JSON matching a schema, and wasn't
ToolArgsMissing, // called a tool, omitted a required argument
Truncated, // finish reason != stop
EmptyOrHedged, // under 24 chars, or matches the hedge set
}
They are ordered by how cheap they are to evaluate, and every one of them is structural. None of them asks a model to grade another model.
SchemaInvalid does the most work by a distance. Anything with a defined output shape — field extraction, classification, structured summaries — gets validated against the schema you already have. If it doesn't parse or doesn't conform, escalate. This gate alone catches about 60% of our escalations.
ToolArgsMissing is the same idea for function calls. The cheap model picks the right tool far more reliably than it fills in the right arguments, and a missing required argument is a free, exact signal.
Truncated is one field comparison and people skip it constantly. A finish_reason of length means you have a sentence that stops mid-
EmptyOrHedged is the weakest one, and I want to be specific about how weak, because it's the one everybody wants to build first.
We started with the intuitive version: ask the cheap model to say when it isn't confident, then escalate on that. It fired on 0.4% of responses. Our measured error rate on the same traffic was around 15%. The model's self-reported uncertainty was not a signal, it was decoration.
What actually works in that slot is a small, boring list: empty, under 24 characters, or an exact-ish match against a hedge set you build by reading two hundred real failures ("I don't have enough information", "As an AI", "Could you clarify"). Not confidence. Refusal.
The rule: gate on structure you can check, not on the model's opinion of itself.
The four things that broke
1. p50 improved. p95 got worse.
This was immediate and it's the real cost of the design.
| All-strong | Cheap-first + gate | |
|---|---|---|
| p50 | 2.9s | 1.4s |
| p95 | 7.8s | 9.6s |
| p99 | 11.2s | 16.4s |
Luna is fast, so the 85% that get accepted got much faster. The 15% that escalate pay for both calls, serially, and they land in your tail.
If you have a latency SLO, that tail is where the design either survives or doesn't. Two things helped: run the gates on the streamed head rather than the finished response where you can, and put a hard escalateBudget on the clock — if the cheap call already burned 4 seconds, return it and log the miss rather than starting a second call you can't afford.
2. You can't stream an answer you might throw away
The whole architecture assumes you get to look at the output before deciding. Streaming assumes you've already committed.
There is no clever fix, only a choice:
- Buffer, then decide. Correct, and you've given up time-to-first-token — which for chat is the number users actually feel.
- Stream, and never escalate. Fine for anything conversational.
- Split your traffic by shape. This is what we do. Structured, non-streamed work — extraction, titling, classification, summarisation, tool selection — goes through the router. Free-form chat streams straight from whichever model that surface is pinned to.
Most of the money was in the first bucket anyway. Structured background work is high volume and nobody is watching a cursor blink at it.
3. The cheap model fails the correctness test long before it fails the eye test
The output is fluent. It's well-formatted. It uses your headings, it hits your tone, it's the right length. It's just wrong.
That's why "does this look like a good answer" gates — including LLM-as-judge in the hot path — did badly for us. Fluency is exactly the axis where the price gap has closed most. Judgement, multi-step reasoning, and knowing what it doesn't know are where it hasn't closed at all.
Structural gates work because they don't have an opinion. Valid JSON is valid JSON.
4. Don't show the strong model the cheap model's answer
Our first version passed the failed attempt along as context — here's a draft, improve it. It seemed obviously more efficient.
It anchors, badly. The strong model inherits the cheap one's framing, keeps its structure, and corrects wording rather than reasoning. On the escalations we hand-checked, the "improve this draft" path was worse than a clean run about a third of the time — and it was worse in the specific way that matters, because it repeated the mistake that triggered the escalation while polishing the prose around it.
Escalation is not a retry. It's a fresh attempt by someone better. Send the original request.
What it did
Per 1,000 requests, at our mix:
| All-strong | Cheap-first + gate | |
|---|---|---|
| Requests attempted on cheap | 0 | 950 |
| Escalated | — | 143 |
| Requests touching the strong model | 1,000 | 193 |
| Cost | $14.50 | $4.16 |
71% off. 81% of requests are served entirely by a model that costs a tenth as much, and the 5% we force to the strong model never enter the router at all.
The forced list is short and it is a policy decision, not a measurement: anything a user is going to send to a customer, anything that writes to production, and anything in the app builder's codegen path. Those never touch the cheap model regardless of what a gate would have said.
When not to do this
- Low volume. Under a few hundred thousand requests a month, 71% of your bill is not worth a new component in the hot path. Go negotiate your seat pricing instead.
- Long tool-calling chains. Errors compound across steps and the gates only see one step at a time. A cheap model that's right 85% of the time per call is right 44% of the time across five calls.
- Anything with a user-visible retry. If your surface already shows a spinner, doubling the tail is worse than paying the bill.
-
Regulated or audited output. "Which model produced this" becomes a question you have to answer per request. Log
Result.Modelfrom day one if there's any chance you're in this bucket — retrofitting it is miserable.
The version you can ship this week
You don't need the whole thing to get most of the money.
- Pick one high-volume, non-streamed, structured job. Titling. Field extraction. Tagging. One.
- Add one gate: validate against the schema you already have.
- Shadow it for a day. Run both models, serve the strong one, log where the gate would have fired. This is the step people skip and it's the one that tells you whether your escalation rate is 15% or 60%.
- Flip it, with
Result.Modelin your logs and a kill switch on the config.
Then measure the escalation rate per request kind, because that's the number the whole design lives on — and it's the number that tells you which job to move next.
The pitch for cheap models in 2026 isn't that they got good enough to replace the frontier. It's that they got cheap enough that checking whether they were good enough is now free.
Top comments (6)
Shadow mode is the only window where both answers exist for the same request, and step 3 spends it measuring the fire rate. The number you cannot recover afterwards is the miss rate: cheap output that cleared all four structural gates and still differs materially from the strong one, which is the class your fluency point says the gates have no opinion about. Once you flip, the strong answer stops being produced for accepted requests, so that residual goes unobservable permanently and the 81% served cheap becomes a statement about gate coverage rather than about answers. Diffing the two outputs during the same shadow day costs nothing extra, since you are already paying for both calls that day.
This is a fair hit, and the fix is cheap.
You're right that shadow mode is the only window where both answers exist for the same input, and that fire rate is the less interesting of the two numbers available in it. Gate fire rate tells me what the gates catch. It says nothing about what they wave through. The 81% is a coverage number, and reading it as a quality number is exactly the mistake the fluency section should have warned against.
The residual you're describing — cheap output that clears schema, tool args, truncation and hedge checks and still differs materially from the strong answer — is the one class the design has no instrument for. And after the flip it can't be reconstructed from production logs, because the second answer simply stops being produced for accepted requests. Agreed that it goes dark permanently.
Concretely, on the same shadow day: log both and split the diff by output shape. For structured responses, field-level exact match gives a per-field disagreement rate for free. For prose, exact match is useless, so it needs either a semantic distance threshold or a judge over the pair — which I resisted, since the whole premise was avoiding a model grading a model. But grading a pair offline is a different risk profile than grading a single answer inline: it never gates a request, it just produces the residual.
One extension to your framing: the number doesn't have to go dark at the flip. Keeping 1–2% of accepted traffic dual-run in production preserves the same measurement at roughly 1% of the savings, and turns a one-off shadow reading into drift detection — which probably matters more than the initial number, since the cheap model is the one that gets silently updated underneath you.
Adding this as step 3b, with the diff harness. Thanks for the push.
The drift number will end up dominated by the kinds you least need it for. Your metrics already stratify by
req.Kind, but a uniform 1-2% sample of accepted traffic spends most of its budget on the high-volume kinds, so a disagreement that is rare and specific to one kind yields a handful of pairs per window and no alarm can separate that from noise. The place to over-sample is the kinds with the fewest gate fires, since zero fires reads as "the cheap model handles this one" and is indistinguishable from "no gate has an opinion about this kind", which is the same absence the fluency point is about.Agreed — uniform sampling buys precision on the kinds I already have the most signal about, and leaves the rare-and-specific case with too few pairs per window to alarm on.
One thing I'd add: the ambiguity you name is separable without spending sample budget. "Zero fires because the cheap model handles this kind" and "zero fires because no gate has an opinion" are indistinguishable if you read fire counts, but not if you read gate applicability — which is static. A schema gate has nothing to say about free prose; a tool-args gate has nothing to say about a kind that calls no tools. Counting how many of the four gates can structurally fire, per kind, splits the zero-fire set into "plausibly handled, cheap to confirm" and "uninstrumented by construction". The second pile is the one you're pointing at, and it should be small enough to enumerate by hand.
So the rule becomes a floor of pairs per kind per window, sized from the disagreement rate I want to detect rather than from traffic share — raised for low-applicability kinds, near zero for kinds that are high-volume and fully gated. Probably cheaper than a uniform 1–2%, since that's where the percentage is expensive and the pairs are least needed.
The part that doesn't resolve: prose kinds have low applicability almost by definition, so the pile that most needs dual-run is the same one where the diff needs a judge rather than field-level match. That puts the weight on the offline pair-grader, which is the piece I'm least confident in.
Making this 3b's allocation rule. Thanks.
the sharp edge in this pattern is the fallback. the classifier routes to the cheap model, the cheap model returns confident nonsense for an edge case, and the user sees a bad answer without knowing why.
we had to build a second pass — a lightweight output scorer that flagged low confidence responses and promoted them to the strong model after the fact. that added a call but still net positive on the bill because the promotion rate was low.
what does your fallback look like when the cheap model misfires? do you catch it in flight or let the user report it?
Good catch — that's exactly the failure mode that pushed us away from an upfront classifier.
We don't route before the call, so there's no "classifier picked wrong" state to recover from. Every non-critical request hits the cheap model first, and the gates run on the response we already have. If a gate fires, we escalate in-flight — the user never sees the cheap output. So it's close to your second pass, except it's the only pass, and it's structural rather than a scored confidence: schema invalid, tool args missing, finish_reason != "stop", empty/hedged. About 60% of escalations come from the schema gate alone. Escalation rate sits at 15% against a 90% break-even, so the promotion cost stays noise — same economics you saw.
The reason we went structural instead of a confidence scorer is that a scorer is another model call whose judgment you now have to trust, and cheap models are cheerfully confident about their nonsense. Structure can't be bluffed — invalid JSON is invalid JSON.
Where you're still right: the gates catch malformed, not wrong. A well-formed, fluent, factually incorrect answer sails straight through. That's the fluency-masking problem in the post and it's genuinely unsolved on our side — that class we catch late, from user reports, and the fix has been moving those request kinds into alwaysStrong rather than trying to detect them after the fact.
One thing worth flagging if you try the in-flight version: don't pass the failed cheap output to the strong model as a draft. We did, and it anchored to the bad answer. Resend the original request.