DEV Community

Cover image for Your default model is too big: start small and escalate only the hard requests
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Your default model is too big: start small and escalate only the hard requests

Wednesday AI Opinion. Here's my hot take: sending every request to your biggest model is not a quality strategy. It's a cost bug you haven't measured. Most production traffic is easy. Start with a small model, check the answer, and escalate only the requests that actually need the big one.

This morning I wrote about pinning your model. This afternoon is the other half: once it's pinned, ask whether it should be your only model.

The price gap is not small

Look at one provider's current price list. On Anthropic's pricing page, Claude Haiku 5.5 (for prompts up to 100,000 tokens) is $0.10 / MTok input and $0.50 / MTok output. Claude Opus 5.5 is $4 / MTok input and $20 / MTok output. That's 40x on both input and output.

This isn't new. In 2023, the FrugalGPT paper compared 12 commercial LLM APIs and found fees that "can differ by two orders of magnitude". Their example: prompt cost for 10M tokens was $30 for GPT-4 but $0.2 for GPT-J hosted by Textsynth.

If the small model gets most of your traffic right, paying the big-model price on all of it is waste.

The evidence: cascades and routers

FrugalGPT (Chen, Zaharia, Zou, Stanford, 2023) proposed an LLM cascade: call models in order, cheapest first, and use a scoring function to decide whether to accept the answer or move on. The scoring function was a small DistilBERT regression model that predicts whether an answer is correct from the query plus the answer.

On HEADLINES (a financial news dataset), the learned cascade was GPT-J → J1-L → GPT-4. Accept GPT-J's answer if its score is above 0.96; otherwise ask J1-L and accept above 0.37; otherwise call GPT-4. With a budget of one fifth of GPT-4's cost, they report cost reduced by 80% with accuracy 1.5% higher than GPT-4.

Their headline result: FrugalGPT "can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost." Cost savings to match the best single model were 98.3% on HEADLINES, 73.3% on OVERRULING and 59.2% on COQA.

Why can a cheap model ever beat GPT-4? Because models make different mistakes. On HEADLINES, for about 6% of the data GPT-4 was wrong while a cheaper model was right.

RouteLLM (LMSYS / UC Berkeley / Anyscale, 2024) takes a different approach: a router looks at the query before any generation and sends it to either a strong or a weak model, so each query makes only one LLM call. Routers were trained on Chatbot Arena preference data. Routing between GPT-4 and Mixtral 8x7B, the LMSYS blog reports cost reductions of over 85% on MT Bench, 45% on MMLU and 35% on GSM8K compared to using only GPT-4, "while still achieving 95% of GPT-4's performance." On MT Bench, the best router hit 95% of GPT-4 performance with only 14% of calls going to GPT-4 (after augmenting training data with an LLM judge).

Router overhead was small: their most expensive router added "no more than 0.4%" compared to GPT-4 generation cost.

Two shapes, pick deliberately

  • Cascade (small first, verify, escalate): best when you can score an answer cheaply (a classifier, a schema check, a test, a verifier). Downside: escalated requests pay for two calls and wait for both.
  • Router (decide up front): one call per request, so latency stays flat. Downside: it has to guess difficulty from the query alone. The RouteLLM paper notes that multi-call methods like FrugalGPT "can increase latency", which is why they route to a single model.

The playbook I'd use

  1. Log first. For every request: model ID, input/output tokens, cost, latency, and outcome (accepted, edited, retried, thumbs down). You can't route traffic you haven't characterized.
  2. Build an eval set per tier. Real (redacted) production inputs with labels or rubrics. Run the small model and the big model on it, and find out where the small one actually fails.
  3. Pick the escalation signal. Schema validation failed, a verifier or test rejected it, a scorer is below threshold, the model said it didn't know, or the request type is on a known-hard list.
  4. Tune the threshold on held-out data, like FrugalGPT's 0.96 / 0.37, not by feel. One knob, two curves: quality and cost.
  5. Track cost and quality per tier, plus escalation rate and p95 latency for escalated requests. If escalation creeps up, your traffic changed.
  6. Re-check when models or traffic change. New small model, new big model, new product feature: rerun the tier evals before trusting old thresholds.

Honest limits: when routing doesn't help

  • Your traffic is mostly hard. If nearly everything escalates, a cascade just adds a call and latency.
  • You can't tell good from bad cheaply. No scorer, no verifier, no labels means no safe escalation rule.
  • Distribution shift. FrugalGPT needs labeled examples "from the same or similar distribution as the test examples." RouteLLM's routers trained only on Arena data performed "at a near-random level" on MMLU, which the authors attribute to most MMLU questions being out-of-distribution. Adding about 1,500 golden-label samples from the MMLU validation split led to "significant performance improvements."
  • Upfront cost. Training a cascade or router takes data and time. FrugalGPT calls it a one-time cost that pays off when your query volume is larger than the training data.
  • Low volume. If your monthly bill is small, a second model and a router may cost more engineering time than they save.
  • Risk-critical outputs. For some answers, the extra latency or a missed escalation isn't worth any savings.

The take

The biggest model is a great escalation target and a lazy default. Measure your traffic, build an eval set per tier, and let a scorer or a router decide when the expensive model is worth it.

What share of your production traffic do you think actually needs your biggest model?

References

Top comments (0)