A 70-line model router: try the cheap LLM first, escalate only when the answer fails validation

Most LLM features I see in production use one model for everything. It is usually the strong, expensive one, because that is what the prototype used and nobody wanted to find out what breaks with a cheaper model.

A lot of real traffic is boring, though: classify a support ticket, pull three fields out of an email, decide whether a message is spam. Small models handle most of it. The trick is knowing which requests they got wrong, without asking a human.

This post shows a small pattern I like for that: cheap model first, cheap validation in code, strong model only on failure. It is about 70 lines of Python and works with any OpenAI-compatible endpoint.

The task

Our example: support tickets come in as free text. We want back a JSON object:

{"category": "billing", "priority": "high", "order_id": "A-10293", "confidence": 0.86}

category must be one of a fixed set, priority one of three values, order_id either a string matching a pattern or null, and confidence a number the model gives about its own answer.

This is a good fit for routing because the output is checkable. If you cannot write a validator for your output, routing gets much harder. More on that at the end.

The code

import json, os, re, time
from openai import OpenAI

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])

CHEAP = os.getenv("CHEAP_MODEL", "deepseek/deepseek-v4-flash")
STRONG = os.getenv("STRONG_MODEL", "anthropic/claude-sonnet-5")

CATEGORIES = {"billing", "delivery", "refund", "technical", "other"}
PRIORITIES = {"low", "normal", "high"}
ORDER_RE = re.compile(r"^[A-Z]-\d{5}$")

SYSTEM = (
    "You label customer support tickets. Reply with JSON only: "
    '{"category": one of ' + json.dumps(sorted(CATEGORIES)) + ", "
    '"priority": "low"|"normal"|"high", '
    '"order_id": string like "A-12345" or null, '
    '"confidence": number from 0 to 1}'
)

def call(model: str, ticket: str):
    t0 = time.perf_counter()
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": ticket}],
        temperature=0,
        max_tokens=200,
        response_format={"type": "json_object"},
    )
    return r.choices[0].message.content, r.usage, time.perf_counter() - t0

def validate(raw: str):
    try:
        d = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return None, "not json"
    if d.get("category") not in CATEGORIES:
        return None, "bad category"
    if d.get("priority") not in PRIORITIES:
        return None, "bad priority"
    oid = d.get("order_id")
    if oid is not None and not (isinstance(oid, str) and ORDER_RE.match(oid)):
        return None, "bad order_id"
    conf = d.get("confidence")
    if not isinstance(conf, (int, float)) or not 0 <= conf <= 1:
        return None, "bad confidence"
    return d, None

def label(ticket: str, min_conf: float = 0.7):
    log = []
    for model in (CHEAP, STRONG):
        raw, usage, secs = call(model, ticket)
        data, err = validate(raw)
        if err is None and model == CHEAP and data["confidence"] < min_conf:
            err = f"low confidence {data['confidence']}"
        log.append({"model": model, "error": err, "secs": round(secs, 2),
                    "in": usage.prompt_tokens, "out": usage.completion_tokens})
        if err is None:
            return data, log
    return None, log  # both failed: send to a human queue

A few details that matter more than they look:

• temperature=0 and a small max_tokens. We want a label, not an essay. Capping output also caps the cost of a model that decides to "explain its reasoning".
• response_format={"type": "json_object"} helps, but support for it varies between models and providers. The validator is what actually protects you, so never skip it.
• The confidence threshold applies only to the cheap model. Self-reported confidence is a weak signal, but in practice it does catch some cases where the small model is guessing. For the strong model we accept any valid answer.
• The log keeps every attempt. You need it to answer the only question that matters: how often do we escalate?

Does routing actually save money?

Here is the part people skip. Routing adds a call on every escalated request: you pay for the cheap attempt and the strong one. So it only pays off if escalations are rare enough.

Let c be the average cost of one cheap call, s the cost of one strong call, and p the share of requests that escalate. With routing, the expected cost per request is:

c + p * s

Without routing it is just s. Routing wins when c + p*s < s, which gives:

p < 1 - c / s

When the cheap model costs a few percent of the strong one, almost any escalation rate below 90% saves money. That sounds great, but the real constraint is latency and quality, not price: every escalated request takes two round trips, and every cheap answer that passes validation but is wrong goes straight to your users.

For a sense of scale, here are list prices in one Russian gateway's catalog on October 23, 2026, in rubles per million tokens (input / output): DeepSeek V4 Flash 11.5 / 22.5, Gemini 3.7 Flash 51.5 / 257, Claude Sonnet 5 274.5 / 1,370.5. For a short classification call the gap between the cheapest and the strongest is well over 20x, so the break-even escalation rate is above 95%. Check your own provider's numbers; the ratio is what matters, not the currency.

How to tune it

1. Start with logging only. Run both models on a sample of a few hundred real tickets and store both answers. Do not route yet.
2. Measure agreement. Where the cheap model passed validation, how often did it agree with the strong one? Those disagreements are your silent errors. If they are rare on the categories you care about, you are in good shape.
3. Tighten the validator, not the prompt. Every new disagreement pattern is a candidate for a rule: an order ID that does not exist in your database, a "refund" ticket with no order ID, a priority that contradicts keywords like "urgent". Code is cheaper and more predictable than another paragraph in the system prompt.
4. Pick min_conf from data. Plot accuracy against self-reported confidence on your sample. If there is no relationship, drop the threshold and rely on the validator.

When not to do this

Routing works when the output is structured and checkable: labels, extracted fields, yes/no decisions, SQL that you can dry-run, code that you can compile. It works badly for open-ended text, such as a reply to a customer or a summary, because there is no cheap validator for "good". For those, a better approach is to pick one model per task type up front and evaluate it offline, instead of deciding per request.

Wrapping up

The pattern is boring on purpose: a cheap call, a strict validator, a fallback, and a log. The log is the real product. After a week you will know your escalation rate, your silent error rate, and exactly which kinds of tickets the small model cannot handle, which is far more useful than any benchmark.

---

Disclosure: I work on Heli (getheli.ru), an OpenAI-compatible LLM gateway for companies in Russia; the model names above use its provider/model format. The code works with any OpenAI-compatible endpoint.

This post was drafted with the help of an AI assistant and edited and checked by me.

Comments

[fedasiaz]

This is a brilliant write-up. The "cheap model first + strict validator + fallback" pattern is exactly how you make LLM features production-ready without burning a hole in your budget.

I've applied a very similar architecture to an automated dependency upgrade bot. Instead of extracting JSON, the agent patches code that broke after an npm update. The "validator" in my case is the actual test suite.

The agent (LLM) tries to fix the ESM import error, and then we run the tests. If the tests fail, that's our validation failure, and we feed the error back to the LLM for a retry loop. If the repo didn't have tests, we wouldn't be able to trust the agent at all.

I really resonate with your point: "Tighten the validator, not the prompt". Code doesn't hallucinate. In my experience, adding a simple type-checker or linter as a validation step catches 80% of LLM hallucinations before you even need to spend tokens on a retry.

Have you noticed any specific edge cases where a strict schema validator passes, but the data is still semantically wrong (e.g., a valid order ID that doesn't exist in the database)? Do you handle those with a secondary API check, or just accept the error rate?

[Reid Marlow]

The break-even math holds up, but the silent failure mode in production is usually downstream: a cheap model outputs a syntactically valid ID that does not exist in your database or belongs to another tenant. Adding a quick existence check against a key-value cache or bloom filter inside the Python validator catches those before you ever commit the record or fall back to Sonnet.