Your model router is a classifier, and you probably aren't evaluating it like one

Most LLM cost discussions end with the same advice: don't send everything to the biggest model. Put a small, fast model in front, escalate when you need to, and watch the bill drop.

That advice is fine. The problem is what it leaves out. The moment you add a router, you've added a second model to your system: one whose job is to make a decision, whose mistakes are mostly invisible, and which almost nobody evaluates with the same care as the models it routes between.

The tutorial version vs. the production version

The tutorial version of a cascade looks like this:

def answer(request):
    draft = small_model(request)
    if looks_ok(draft):
        return draft
    return large_model(request)

Three lines of logic, and every one of them hides a decision. What does looks_ok actually check? What happens when it's wrong in either direction? How long did the user wait when the first call was thrown away?

In production, those questions are the system. Here's where it tends to go wrong. (The scenarios below are illustrative, not drawn from a specific deployment.)

Failure mode 1: confident wrong answers never escalate

Escalation usually triggers on things you can detect: malformed JSON, a failed schema check, a refusal, a low self-reported confidence score. What it rarely catches is a well-formed, fluent, wrong answer.

Picture a support assistant that routes "simple" billing questions to the small model. A customer asks about a proration edge case. The small model returns a clean, polite, perfectly structured answer that quotes the wrong number. Every validator passes. The cascade reports a successful cheap resolution. Your dashboard says routing is working.

The failures your router can't see are the ones that cost the most.

Failure mode 2: "difficulty" isn't a feature you can compute

Many routers decide based on a proxy for difficulty: prompt length, keyword lists, a classifier trained on "easy vs. hard" labels someone made up in an afternoon.

But difficulty is relative to the model. A short prompt asking for a regex that handles a nasty edge case is hard. A long prompt asking to summarize a meeting transcript might be easy. If your routing signal is a vague human intuition about difficulty, you're encoding that intuition as a production decision without ever measuring it.

Failure mode 3: the cheap path can be the slow path

A cascade that escalates often pays twice: one call to the small model, then a full call to the large one. If your escalation rate on some task type is high, the "fast first pass" is adding latency, not removing it.

Average latency hides this. Look at end-to-end latency per task type, including retries and validation time, and the picture can flip: for some categories, going straight to the larger model is both faster and simpler.

Failure mode 4: treating all errors as equally expensive

A wrong answer in an internal draft generator costs someone thirty seconds. A wrong answer in a compliance summary or a code change that ships can cost far more. A router optimized purely on token spend will happily save cents on requests where an error costs real money.

What to actually measure

Evaluate the router as its own component, against real traffic, broken down by task type:

per task type:
  - quality of the small-model path vs. the large-model path
  - escalation rate
  - false "ok" rate (validator passed, answer was wrong)
  - p50 / p95 end-to-end latency, including retries
  - cost per request, and estimated cost of an error

The false "ok" rate is the one most teams don't have, because it requires sampling routed answers and grading them. It's also the one that tells you whether your cheap path is actually cheap.

Where to draw the line

A practical starting point:

• Route to the small model where the output is verifiable (structured extraction, classification against a fixed label set) and errors are cheap or caught downstream.
• Go straight to the large model where errors are expensive, hard to detect automatically, or where the measured escalation rate is high enough that the cascade costs latency.
• Re-measure whenever the models, the prompts, or the traffic mix change. The boundary moves.

Model routing isn't a cost optimization with a classifier attached. It's a decision system, and it deserves the same evaluation discipline as any model you'd put in front of users.

How are you deciding where your cheap path ends? Measured boundaries, or gut feel that hasn't been revisited?