DEV Community

Cover image for Cheap Model First, Escalate When Needed: LLM Routing by Task
SandBase AI
SandBase AI

Posted on Originally published at blog.sandbase.ai

Cheap Model First, Escalate When Needed: LLM Routing by Task

Originally published on the SandBase Blog.

If your support bot sends every message to a frontier model, most of the bill pays for questions like "how do I reset my password?". The fix is a two-step call: a cheap model answers first and flags the messages it should not handle, and only those go to a stronger model. This tutorial builds that in about 30 lines of Python against one OpenAI-compatible endpoint, then reports what a 10-message run actually cost. The routing logic is yours; SandBase supplies one key and one base URL for both models.

Key takeaway

  • Pattern: google/gemini-3.5-flash-lite answers first and returns JSON with an escalate flag. If the flag is true, the same message is re-asked to anthropic/claude-sonnet-5.5.
  • In a 10-message run on 2026-11-22 (6 easy, 4 hard), routing cost $0.0205 against $0.0287 for sending everything to Sonnet 5.5, 29% less. The six easy messages cost 87% less. The four hard ones cost 3% more, because they pay for the cheap attempt and then the strong answer.
  • Escalated messages were slower (about 8.3 s against 6.0 s). Easy ones were faster on average (about 3.0 s against 4.7 s), though two of the six were slower. Single runs, so treat timings as rough.
  • The savings depend on your mix. On this data the cost advantage fades only when about 90% of traffic is hard (an extrapolation from 10 messages), but average latency gets worse once more than about 40% is hard.
  • The escalate flag is the model's own judgment. Log it, sample the non-escalated answers, and keep a rule-based override for anything high-risk.

The pattern

Three pieces:

  1. A cheap first pass. The cheap model gets the same context as the strong one and must answer in a fixed JSON shape: answer, escalate, reason.
  2. Escalation rules in the prompt. Tell it when to escalate: calculations, several issues in one message, legal threats or policy exceptions, or any case where the context does not fully answer the question.
  3. A strong fallback. If escalate is true, or the JSON does not parse, call the strong model with the original message and discard the cheap answer.

Both calls use the same base_url and key, so switching models is a string change. That is the part a unified endpoint saves you: no second SDK, no second billing account.

The two models and their prices

The run uses two models from the SandBase catalog. Prices below are the per-million-token list prices returned by SandBase's inspect endpoint on 2026-11-22. The model pages showed a 10% and 15% discount that day, which the cost figures in this article do not apply, so your bill can be lower.

Role Model Input / 1M tokens Output / 1M tokens
Cheap first pass google/gemini-3.5-flash-lite $0.30 $2.50
Strong fallback anthropic/claude-sonnet-5.5 $2.00 $10.00

SandBase model page for Gemini 3.5 Flash Lite: google/gemini-3.5-flash-lite, input price $0.27 (list $0.30) and output price $2.25 (list $2.50) per million tokens, 1M context window, capabilities chat, vision, reasoning, structured output, function calling
Gemini 3.5 Flash-Lite on SandBase, captured 2026-11-22 (UTC). The page showed a 10% discount over the list price.

SandBase model page for Claude Sonnet 5.5: anthropic/claude-sonnet-5.5, input price $1.70 (list $2.00) and output price $8.50 (list $10.00) per million tokens, 1M context window, 128K max output
Claude Sonnet 5.5 on SandBase, captured 2026-11-22 (UTC). The page showed a 15% discount over the list price.

Any pair works if the cheap model supports JSON output and the strong one is clearly better at your hard cases. Check both on your own data before you pick.

The code

Install the OpenAI SDK, export your key as SANDBASE_API_KEY, and run:

import json, os, re
from openai import OpenAI

client = OpenAI(base_url="https://api.sandbase.ai/v1", api_key=os.environ["SANDBASE_API_KEY"])
CHEAP, STRONG = "google/gemini-3.5-flash-lite", "anthropic/claude-sonnet-5.5"

POLICY = "..."  # your support policy or knowledge-base excerpt
ANSWER = "You are Acme Cloud support. Answer only from the policy below. Be concise.\n\n" + POLICY
ROUTE = ANSWER + """
Reply with JSON only: {"answer": "...", "escalate": true|false, "reason": "..."}.
Set escalate=true if the question needs calculation, combines several issues, involves a legal
threat or an exception to policy, or you are not certain the policy fully answers it.
Otherwise escalate=false."""

def ask(model, system, message, **kw):
    r = client.chat.completions.create(
        model=model, max_tokens=800,
        messages=[{"role": "system", "content": system}, {"role": "user", "content": message}], **kw)
    return r.choices[0].message.content, r.usage

def route(message):
    raw, usage = ask(CHEAP, ROUTE, message, response_format={"type": "json_object"})
    try:
        out = json.loads(re.search(r"\{.*\}", raw, re.S).group(0))
        if not out["escalate"]:
            return out["answer"], "cheap", usage
    except (AttributeError, KeyError, ValueError):
        pass  # unparseable JSON counts as "escalate"
    answer, usage2 = ask(STRONG, ANSWER, message)
    return answer, "strong", usage2
Enter fullscreen mode Exit fullscreen mode

Three details that matter:

  • The strong model gets the plain prompt, not the JSON prompt. It should answer, not self-assess again.
  • Parse failures escalate. A cheap model that returns broken JSON is telling you the message was not easy.
  • Cost accounting. Each response has usage.prompt_tokens and usage.completion_tokens. Multiply by the prices above, and add the cheap call's usage when you escalate, since you paid for it. The route function above returns only the last call's usage for brevity; sum both in production.

What the run showed

The test set was ten customer messages against a short fictional policy (plans, refunds, password reset, SSO, outage credits). Six were lookups, four needed arithmetic, several issues at once, or a policy exception. The baseline sent all ten to Sonnet 5.5. The routed version used the same logic as the code above, in a slightly longer script.

Table of the 10-message routing run on 2026-11-22: six easy messages answered by Flash-Lite at about $0.0002 each, four hard messages escalated to Sonnet 5.5, total $0.02868 for all-Sonnet against $0.02048 routed
Per-message route, cost and time. Costs are computed from the token usage in each response and the list prices above.

All Sonnet 5.5 Routed Change
6 easy messages, cost $0.01008 $0.00126 -87%
4 hard messages, cost $0.01859 $0.01922 +3%
All 10, cost $0.02868 $0.02048 -29%
Easy, average time 4.7 s 3.0 s faster on 4 of 6
Hard, average time 6.0 s 8.3 s slower

What the numbers do and do not say:

  • All four hard messages were escalated, and none of the six easy ones were. On two of the four hard ones, the cheap model's own draft was clearly incomplete: for the $1,000 annual-plan refund it restated the policy without computing a figure, and for the legal-threat message it quoted the policy without addressing the threat. The Sonnet answer worked the refund out step by step ($495.83 under one stated reading of the proration rule) and flagged where the policy was silent.
  • The hard messages cost slightly more when routed. That is the price of the cheap attempt. Each easy message saved about $0.0015 and each hard one cost about $0.0002 extra, so on this data the savings hold until roughly 90% of traffic is hard. That break-even is extrapolated from ten messages. Latency turns against you sooner: the average gets worse once more than about 40% of messages escalate.
  • Ten messages is a demo, not a benchmark. The set was written for this article, the answers were checked by looking for the expected policy facts and by reading them, and each timing comes from one call. Rebuild the test with your own tickets before you rely on the savings number.

Making it safe in production

  • Sample the cheap answers. The risk is a confident wrong answer with escalate: false. Send a random 2-5% of non-escalated answers to a human or to the strong model and compare.
  • Add hard rules before the model. Refunds above a threshold, legal words, account deletion and anything regulated should skip the cheap model entirely. A keyword check costs nothing.
  • Log reason. Escalation reasons show you which kinds of questions the cheap model hands off, and which prompts to tighten.
  • Cap retries. One escalation is enough. If the strong model also fails, hand off to a person.
  • Track the escalation rate. A rate near zero means your rules are too loose; near 100% means routing only adds cost.

When not to route

  • Most of your traffic is hard. The savings shrink and every escalated message is slower.
  • Latency matters more than cost on the hard path, such as live voice. The two-step path adds seconds.
  • You have fewer than a few thousand requests a month. The engineering time costs more than the savings.

Try it

Get a key and test the pair on your own messages with Gemini 3.5 Flash-Lite and Claude Sonnet 5.5. For how this compares with gateway products that route for you, see LiteLLM vs OpenRouter and OpenRouter alternatives. For which frontier model is worth paying for on your hard cases, see frontier AI models worth paying for.

FAQ

Do I need a router product to do this?

No. The router here is a prompt and an if. A gateway helps when you want routing rules, budgets and logging managed outside your code.

Why not let the strong model decide too?

Then every message pays for the strong model, which defeats the point. The cheap model only has to know when it is out of its depth.

Does SandBase route or fall back between models for me?

No. In this tutorial you decide which model gets each message. SandBase gives you one key and one endpoint for both models.

Which cheap model should I use?

Pick the cheapest one that returns valid JSON reliably and passes your easy cases. Run your own messages through two or three candidates and compare cost, accuracy and escalation rate.

Top comments (0)