40 Lines of Go That Cut Our LLM Bill by 71%

On July 30, OpenAI cut GPT-5.6 Luna to $0.20 per million input tokens and $1.20 per million output — down from $1 and $6. An 80% cut. Azure matched it on August 1.

We did what most teams did with that news: nothing. Our gateway sent every request to the strong model, because it always had, and because "just use the cheap model" sounds like a decision that comes back as a support ticket three weeks later.

Then someone put the bill next to the traffic mix, and the awkward part was obvious. The overwhelming majority of our requests were title this thread, summarise this diff, extract the fields from this form, name this file. We were paying frontier prices to generate three-word document titles.

Here's what we shipped instead. It's about forty lines of Go, it moved 81% of requests off the expensive model, and it cut the bill by 71%.

It also broke four things, and those are the interesting part.

---

The approach that doesn't work: classify the prompt

The obvious design — the one everyone writes first — is a classifier in front of the router:

Look at the incoming request. Decide if it's easy or hard. Send it to the model that matches.

We built this. It's worse than it looks, for three reasons.

You need a model to run the classifier. Now every request pays an extra call before any work happens. A cheap classifier is another Luna call and another 300ms; an accurate classifier is a strong-model call, which is the cost you were trying to avoid.

Short is not easy. "Fix the timezone bug" is eleven characters and needs everything you've got. "Summarise the following 4,000-word RFC" is long and trivial. Length, token count, keyword lists — every cheap heuristic we tried correlated with the shape of the request and not with the difficulty of it.

You're predicting the answer before you've seen it. That's the actual problem. Difficulty is a property of the work, and the only honest way to learn it is to do the work.

The rule: don't classify the prompt. Run the cheap model and judge the output.

---

Cheap-first with an escalation gate

The design that works is embarrassingly simple:

1. Send it to the cheap model.
2. Look at what came back.
3. If it fails a gate, run it again on the strong model and return that instead.

The thing that makes this viable is arithmetic that surprised us. At our average request shape — roughly 3,000 input tokens and 700 output — a Luna call costs about $0.0014 and a strong-model call about $0.0145. Ten to one.

So a request that gets escalated costs 0.0014 + 0.0145 = $0.0159 instead of $0.0145. About 10% more.

Work out the break-even escalation rate:

cheap + (p × strong) < strong
p < (strong − cheap) / strong
p < (0.0145 − 0.0014) / 0.0145
p < 0.90

You would have to escalate nine times out of ten before the cheap attempt costs you money. We escalate 15% of the time. Cost is not the constraint here, and if you're arguing about whether cheap-first is worth the double billing, you're arguing about the wrong resource.

Latency is the constraint. We'll come back to that.

---

The router

type Result struct {
    Text   string
    Model  string
    Tokens Usage
}

// A gate returns the reason this answer can't be trusted, or "" to accept it.
type Gate func(req Request, out string, fin FinishReason) string

func (r *Router) Do(ctx context.Context, req Request) (Result, error) {
    if req.ForceStrong || r.alwaysStrong[req.Kind] {
        return r.call(ctx, r.strong, req)
    }

    cheap, err := r.call(ctx, r.cheap, req)
    if err != nil {
        // Availability, not quality. Fall through, don't fail the request.
        return r.call(ctx, r.strong, req)
    }

    for _, gate := range r.gates {
        reason := gate(req, cheap.Text, cheap.Finish)
        if reason == "" {
            continue
        }
        r.metrics.Escalate(req.Kind, reason)

        strong, err := r.call(ctx, r.strong, req) // NB: req, not req+cheap.Text
        if err != nil {
            return cheap, nil // degraded, not failed
        }
        return strong, nil
    }

    r.metrics.Accept(req.Kind)
    return cheap, nil
}

Two lines in there are load-bearing and neither is obvious.

return r.call(ctx, r.strong, req) when the cheap call errors — a 429 or a timeout is an availability problem, and availability problems should not surface to the user as a failed request when you have a second provider sitting right there.

return cheap, nil when the strong call errors — you already have an answer. It may be a worse answer. Shipping a worse answer beats shipping a spinner.

And the comment on the escalation call is the one people get wrong, which is failure mode #4 below.

---

The four gates

The gates are the whole product. The router is plumbing.

var defaultGates = []Gate{
    SchemaInvalid,   // had to be JSON matching a schema, and wasn't
    ToolArgsMissing, // called a tool, omitted a required argument
    Truncated,       // finish reason != stop
    EmptyOrHedged,   // under 24 chars, or matches the hedge set
}

They are ordered by how cheap they are to evaluate, and every one of them is structural. None of them asks a model to grade another model.

SchemaInvalid does the most work by a distance. Anything with a defined output shape — field extraction, classification, structured summaries — gets validated against the schema you already have. If it doesn't parse or doesn't conform, escalate. This gate alone catches about 60% of our escalations.

ToolArgsMissing is the same idea for function calls. The cheap model picks the right tool far more reliably than it fills in the right arguments, and a missing required argument is a free, exact signal.

Truncated is one field comparison and people skip it constantly. A finish_reason of length means you have a sentence that stops mid-

EmptyOrHedged is the weakest one, and I want to be specific about how weak, because it's the one everybody wants to build first.

We started with the intuitive version: ask the cheap model to say when it isn't confident, then escalate on that. It fired on 0.4% of responses. Our measured error rate on the same traffic was around 15%. The model's self-reported uncertainty was not a signal, it was decoration.

What actually works in that slot is a small, boring list: empty, under 24 characters, or an exact-ish match against a hedge set you build by reading two hundred real failures ("I don't have enough information", "As an AI", "Could you clarify"). Not confidence. Refusal.

The rule: gate on structure you can check, not on the model's opinion of itself.

---

The four things that broke

1. p50 improved. p95 got worse.

This was immediate and it's the real cost of the design.

	All-strong	Cheap-first + gate
p50	2.9s	1.4s
p95	7.8s	9.6s
p99	11.2s	16.4s

Luna is fast, so the 85% that get accepted got much faster. The 15% that escalate pay for both calls, serially, and they land in your tail.

If you have a latency SLO, that tail is where the design either survives or doesn't. Two things helped: run the gates on the streamed head rather than the finished response where you can, and put a hard escalateBudget on the clock — if the cheap call already burned 4 seconds, return it and log the miss rather than starting a second call you can't afford.

2. You can't stream an answer you might throw away

The whole architecture assumes you get to look at the output before deciding. Streaming assumes you've already committed.

There is no clever fix, only a choice:

• Buffer, then decide. Correct, and you've given up time-to-first-token — which for chat is the number users actually feel.
• Stream, and never escalate. Fine for anything conversational.
• Split your traffic by shape. This is what we do. Structured, non-streamed work — extraction, titling, classification, summarisation, tool selection — goes through the router. Free-form chat streams straight from whichever model that surface is pinned to.

Most of the money was in the first bucket anyway. Structured background work is high volume and nobody is watching a cursor blink at it.

3. The cheap model fails the correctness test long before it fails the eye test

The output is fluent. It's well-formatted. It uses your headings, it hits your tone, it's the right length. It's just wrong.

That's why "does this look like a good answer" gates — including LLM-as-judge in the hot path — did badly for us. Fluency is exactly the axis where the price gap has closed most. Judgement, multi-step reasoning, and knowing what it doesn't know are where it hasn't closed at all.

Structural gates work because they don't have an opinion. Valid JSON is valid JSON.

4. Don't show the strong model the cheap model's answer

Our first version passed the failed attempt along as context — here's a draft, improve it. It seemed obviously more efficient.

It anchors, badly. The strong model inherits the cheap one's framing, keeps its structure, and corrects wording rather than reasoning. On the escalations we hand-checked, the "improve this draft" path was worse than a clean run about a third of the time — and it was worse in the specific way that matters, because it repeated the mistake that triggered the escalation while polishing the prose around it.

Escalation is not a retry. It's a fresh attempt by someone better. Send the original request.

---

What it did

Per 1,000 requests, at our mix:

	All-strong	Cheap-first + gate
Requests attempted on cheap	0	950
Escalated	—	143
Requests touching the strong model	1,000	193
Cost	$14.50	$4.16

71% off. 81% of requests are served entirely by a model that costs a tenth as much, and the 5% we force to the strong model never enter the router at all.

The forced list is short and it is a policy decision, not a measurement: anything a user is going to send to a customer, anything that writes to production, and anything in the app builder's codegen path. Those never touch the cheap model regardless of what a gate would have said.

---

When not to do this

• Low volume. Under a few hundred thousand requests a month, 71% of your bill is not worth a new component in the hot path. Go negotiate your seat pricing instead.
• Long tool-calling chains. Errors compound across steps and the gates only see one step at a time. A cheap model that's right 85% of the time per call is right 44% of the time across five calls.
• Anything with a user-visible retry. If your surface already shows a spinner, doubling the tail is worse than paying the bill.
• Regulated or audited output. "Which model produced this" becomes a question you have to answer per request. Log Result.Model from day one if there's any chance you're in this bucket — retrofitting it is miserable.

---

The version you can ship this week

You don't need the whole thing to get most of the money.

1. Pick one high-volume, non-streamed, structured job. Titling. Field extraction. Tagging. One.
2. Add one gate: validate against the schema you already have.
3. Shadow it for a day. Run both models, serve the strong one, log where the gate would have fired. This is the step people skip and it's the one that tells you whether your escalation rate is 15% or 60%.
4. Flip it, with Result.Model in your logs and a kill switch on the config.

Then measure the escalation rate per request kind, because that's the number the whole design lives on — and it's the number that tells you which job to move next.

The pitch for cheap models in 2026 isn't that they got good enough to replace the frontier. It's that they got cheap enough that checking whether they were good enough is now free.