DEV Community

x3-team
x3-team

Posted on

A 70-line model router: try the cheap LLM first, escalate only when the answer fails validation

Most LLM features I see in production use one model for everything. It is usually the strong, expensive one, because that is what the prototype used and nobody wanted to find out what breaks with a cheaper model.

A lot of real traffic is boring, though: classify a support ticket, pull three fields out of an email, decide whether a message is spam. Small models handle most of it. The trick is knowing which requests they got wrong, without asking a human.

This post shows a small pattern I like for that: cheap model first, cheap validation in code, strong model only on failure. It is about 70 lines of Python and works with any OpenAI-compatible endpoint.

The task

Our example: support tickets come in as free text. We want back a JSON object:

{"category": "billing", "priority": "high", "order_id": "A-10293", "confidence": 0.86}
Enter fullscreen mode Exit fullscreen mode

category must be one of a fixed set, priority one of three values, order_id either a string matching a pattern or null, and confidence a number the model gives about its own answer.

This is a good fit for routing because the output is checkable. If you cannot write a validator for your output, routing gets much harder. More on that at the end.

The code

import json, os, re, time
from openai import OpenAI

client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])

CHEAP = os.getenv("CHEAP_MODEL", "deepseek/deepseek-v4-flash")
STRONG = os.getenv("STRONG_MODEL", "anthropic/claude-sonnet-5")

CATEGORIES = {"billing", "delivery", "refund", "technical", "other"}
PRIORITIES = {"low", "normal", "high"}
ORDER_RE = re.compile(r"^[A-Z]-\d{5}$")

SYSTEM = (
    "You label customer support tickets. Reply with JSON only: "
    '{"category": one of ' + json.dumps(sorted(CATEGORIES)) + ", "
    '"priority": "low"|"normal"|"high", '
    '"order_id": string like "A-12345" or null, '
    '"confidence": number from 0 to 1}'
)

def call(model: str, ticket: str):
    t0 = time.perf_counter()
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": ticket}],
        temperature=0,
        max_tokens=200,
        response_format={"type": "json_object"},
    )
    return r.choices[0].message.content, r.usage, time.perf_counter() - t0

def validate(raw: str):
    try:
        d = json.loads(raw)
    except (json.JSONDecodeError, TypeError):
        return None, "not json"
    if d.get("category") not in CATEGORIES:
        return None, "bad category"
    if d.get("priority") not in PRIORITIES:
        return None, "bad priority"
    oid = d.get("order_id")
    if oid is not None and not (isinstance(oid, str) and ORDER_RE.match(oid)):
        return None, "bad order_id"
    conf = d.get("confidence")
    if not isinstance(conf, (int, float)) or not 0 <= conf <= 1:
        return None, "bad confidence"
    return d, None

def label(ticket: str, min_conf: float = 0.7):
    log = []
    for model in (CHEAP, STRONG):
        raw, usage, secs = call(model, ticket)
        data, err = validate(raw)
        if err is None and model == CHEAP and data["confidence"] < min_conf:
            err = f"low confidence {data['confidence']}"
        log.append({"model": model, "error": err, "secs": round(secs, 2),
                    "in": usage.prompt_tokens, "out": usage.completion_tokens})
        if err is None:
            return data, log
    return None, log  # both failed: send to a human queue
Enter fullscreen mode Exit fullscreen mode

A few details that matter more than they look:

  • temperature=0 and a small max_tokens. We want a label, not an essay. Capping output also caps the cost of a model that decides to "explain its reasoning".
  • response_format={"type": "json_object"} helps, but support for it varies between models and providers. The validator is what actually protects you, so never skip it.
  • The confidence threshold applies only to the cheap model. Self-reported confidence is a weak signal, but in practice it does catch some cases where the small model is guessing. For the strong model we accept any valid answer.
  • The log keeps every attempt. You need it to answer the only question that matters: how often do we escalate?

Does routing actually save money?

Here is the part people skip. Routing adds a call on every escalated request: you pay for the cheap attempt and the strong one. So it only pays off if escalations are rare enough.

Let c be the average cost of one cheap call, s the cost of one strong call, and p the share of requests that escalate. With routing, the expected cost per request is:

c + p * s
Enter fullscreen mode Exit fullscreen mode

Without routing it is just s. Routing wins when c + p*s < s, which gives:

p < 1 - c / s
Enter fullscreen mode Exit fullscreen mode

When the cheap model costs a few percent of the strong one, almost any escalation rate below 90% saves money. That sounds great, but the real constraint is latency and quality, not price: every escalated request takes two round trips, and every cheap answer that passes validation but is wrong goes straight to your users.

For a sense of scale, here are list prices in one Russian gateway's catalog on October 23, 2026, in rubles per million tokens (input / output): DeepSeek V4 Flash 11.5 / 22.5, Gemini 3.7 Flash 51.5 / 257, Claude Sonnet 5 274.5 / 1,370.5. For a short classification call the gap between the cheapest and the strongest is well over 20x, so the break-even escalation rate is above 95%. Check your own provider's numbers; the ratio is what matters, not the currency.

How to tune it

  1. Start with logging only. Run both models on a sample of a few hundred real tickets and store both answers. Do not route yet.
  2. Measure agreement. Where the cheap model passed validation, how often did it agree with the strong one? Those disagreements are your silent errors. If they are rare on the categories you care about, you are in good shape.
  3. Tighten the validator, not the prompt. Every new disagreement pattern is a candidate for a rule: an order ID that does not exist in your database, a "refund" ticket with no order ID, a priority that contradicts keywords like "urgent". Code is cheaper and more predictable than another paragraph in the system prompt.
  4. Pick min_conf from data. Plot accuracy against self-reported confidence on your sample. If there is no relationship, drop the threshold and rely on the validator.

When not to do this

Routing works when the output is structured and checkable: labels, extracted fields, yes/no decisions, SQL that you can dry-run, code that you can compile. It works badly for open-ended text, such as a reply to a customer or a summary, because there is no cheap validator for "good". For those, a better approach is to pick one model per task type up front and evaluate it offline, instead of deciding per request.

Wrapping up

The pattern is boring on purpose: a cheap call, a strict validator, a fallback, and a log. The log is the real product. After a week you will know your escalation rate, your silent error rate, and exactly which kinds of tickets the small model cannot handle, which is far more useful than any benchmark.


Disclosure: I work on Heli (getheli.ru), an OpenAI-compatible LLM gateway for companies in Russia; the model names above use its provider/model format. The code works with any OpenAI-compatible endpoint.

This post was drafted with the help of an AI assistant and edited and checked by me.

Top comments (5)

Collapse
 
fedasiaz profile image
fedasiaz •

This is a brilliant write-up. The "cheap model first + strict validator + fallback" pattern is exactly how you make LLM features production-ready without burning a hole in your budget.

I've applied a very similar architecture to an automated dependency upgrade bot. Instead of extracting JSON, the agent patches code that broke after an npm update. The "validator" in my case is the actual test suite.

The agent (LLM) tries to fix the ESM import error, and then we run the tests. If the tests fail, that's our validation failure, and we feed the error back to the LLM for a retry loop. If the repo didn't have tests, we wouldn't be able to trust the agent at all.

I really resonate with your point: "Tighten the validator, not the prompt". Code doesn't hallucinate. In my experience, adding a simple type-checker or linter as a validation step catches 80% of LLM hallucinations before you even need to spend tokens on a retry.

Have you noticed any specific edge cases where a strict schema validator passes, but the data is still semantically wrong (e.g., a valid order ID that doesn't exist in the database)? Do you handle those with a secondary API check, or just accept the error rate?

Collapse
 
efe_genc profile image
Efe Genc •

Step 2 uses the strong model as the reference: where the cheap model passed validation, how often did it agree? Disagreements count as silent errors, but cases where both models are wrong in the same way are invisible, and ambiguous tickets are the likeliest to do that. Step 3 then tunes the validator on those same disagreements, so one sample builds the rule and also estimates its error rate. I would hand-label a few hundred tickets once, keep them out of the tuning loop, and report the silent error rate from that set only. The cost formula c + p*s also has no term for a wrong answer that passes validation. Have you measured that rate on real traffic?

Collapse
 
reidmarlow profile image
Reid Marlow •

The break-even math holds up, but the silent failure mode in production is usually downstream: a cheap model outputs a syntactically valid ID that does not exist in your database or belongs to another tenant. Adding a quick existence check against a key-value cache or bloom filter inside the Python validator catches those before you ever commit the record or fall back to Sonnet.

Collapse
 
iteracompute profile image
IteraCompute •

Nice write-up, and the break-even formula is the right way to frame it.

One thing to watch with deepseek-v4-flash as CHEAP: DeepSeek's docs say it runs in thinking mode by default. Depending on the provider, reasoning tokens can count against max_tokens, so a 200-token cap can end with finish_reason="length" and empty or cut-off content. Your validator then logs "not json" and escalates. That looks like the cheap model failing on quality, but it's really a token budget problem, and it quietly pushes p up.

Two easy fixes: log finish_reason and usage.completion_tokens_details.reasoning_tokens next to the error in your attempt log, and for a pure labeling task either turn thinking off (the parameter name differs by provider, so check its docs) or give the cheap call a larger max_tokens.

Since every request shares the same SYSTEM prompt, it's also worth logging cached prompt tokens. The cheap call's real cost per request is often lower than the list input price suggests, which moves the break-even point even further in your favor.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Cheap first only saves when the cheap answer passes. A failed validation sends the same context to the strong model, so both calls get paid. Did most of your calls stop at the cheap model?