Originally published on robatdasorvi.com
I spent the better part of a year defending a decision I now think was wrong: routing every single request to the biggest model available because I was scared of the alternative looking cheap. Semantic routing for LLM cost reduction isn't a research curiosity anymore. It's the difference between an agent stack that survives its own success and one that gets killed in a budget review the first time usage 10x's. I found that out by almost getting killed in a budget review.
Here's the part nobody wants to say in public: most of us aren't routing based on task difficulty. We're routing based on vibes, and then writing blog posts that make the vibes sound like architecture.
What Is Semantic Routing, Actually?
Semantic routing is a classifier, usually a small embedding model or a lightweight LLM, that sits in front of your model calls and decides which model should handle a given request based on the meaning and difficulty of the query, not just its length or source. The direct answer is: it's a traffic cop, not a genius.
The reason this gets dressed up in fancier language is that "traffic cop" doesn't sound fundable. But that's what it is. You embed the incoming prompt, compare it against clusters of past queries labeled by which model handled them well, and route accordingly. Stanford's FrugalGPT paper from 2023 formalized the idea of cascading through cheaper models before escalating to expensive ones, and reported cost reductions of up to 98% while matching or exceeding GPT-4-level accuracy on benchmark tasks by only escalating the queries that actually needed it (arXiv:2305.05176, Chen et al., 2023). That number sounds absurd until you sit with your own logs and realize how many of your "important" queries are actually "what's your refund policy" restated eleven different ways.
The uncomfortable truth is that the router isn't discovering hidden complexity in your traffic. It's discovering that you never needed the expensive model for most of it in the first place. You just didn't know that until something measured it.
How Much Money Does Semantic Routing Actually Save?
Somewhere between 40% and 90%, depending on how skewed your traffic already is toward simple queries, and for most production agent stacks, it's skewed a lot more than the team building it wants to admit. My own number, across three months of a Twitter reply-and-outreach agent handling a mix of intent classification, short replies, and longer strategic drafting, landed at 70%.
The academic numbers back this up and then some. RouteLLM, out of LMSYS and UC Berkeley in 2024, trained routers on human preference data and reported cost reductions of over 85% on the MT-Bench and MMLU benchmarks while retaining 95% of GPT-4's response quality, simply by sending the queries a cheaper model could handle to that cheaper model (arXiv:2406.18665, Ong et al., 2024). Microsoft's hybrid LLM routing work, published the same year, found similar patterns: a well-trained router pushing the "easy" 60-70% of traffic to a small model achieved near-parity quality scores against always-use-the-large-model baselines, at a fraction of the inference cost (arXiv:2404.14618, Ding et al., 2024).
None of these numbers are about the model getting smarter. They're about admitting the large model was doing a lot of unnecessary work. That's a hard thing to admit when the large model's invoice is the thing justifying your budget request to leadership.
Why Do Experienced Teams Still Route Everything to the Expensive Model?
Because nobody gets blamed for the bill when the eval scores look perfect, and everybody gets blamed for the one embarrassing output that a cheap model produces in front of a client. That asymmetry, not technical ignorance, is the actual reason so many teams with real routing infrastructure sitting unused still send 100% of traffic to Opus or GPT-4-class models by default.
I've sat in the planning meeting. Someone proposes routing simple intent-classification calls to a cheaper model. Someone else says "what if it gets one wrong in front of the CEO," and the conversation ends there, because nobody in the room can quantify the actual risk on the spot, and everybody can vividly imagine the specific bad outcome. Fear with a face beats a percentage on a slide every time. So the safe move, expensive-model-for-everything, survives quarter after quarter, dressed up as "quality first," when it's really just "I don't want to be the one explaining a mistake."
This is the unsaid part. Most senior engineers I know privately think their production traffic is over-provisioned by 2-3x on model tier. Almost none of them will say that in a public design doc, because the design doc becomes the artifact someone points to later.
Does Semantic Routing Actually Hurt Output Quality?
For well-designed routers, no. The quality delta is close to zero on the queries that get routed correctly, and the failures cluster in predictable, fixable places rather than spreading randomly across your traffic. That predictability is the whole point; it's the thing that turns "quality risk" from a vague fear into a measurable, patchable surface.
The honest counterargument is that routers misclassify edge cases, and when they do, a query that needed the expensive model gets stuck with a weak one and the output is visibly worse. This happens. RouteLLM's own evaluation shows quality dips at the margins, specifically on queries near the difficulty boundary where the classifier is least confident (arXiv:2406.18665, 2024). But the fix isn't to abandon routing. It's to route on confidence, not just classification. When the router's confidence score falls below a threshold, you escalate to the expensive model automatically. That single guardrail closes most of the gap, because it converts "the router might be wrong" into "the router knows when it might be wrong," which is a solvable problem instead of an existential one.
What actually degrades quality in the field isn't the router being wrong occasionally. It's teams building a router once, shipping it, and never re-training it as their traffic mix shifts. A router trained on last year's query distribution silently drifts into misclassifying this year's traffic, and because nobody's watching the classifier's confidence scores in production, the degradation looks like "the model got worse" instead of "our router is stale." I've watched this happen to my own system after adding a new content pillar. The classifier had never seen that shape of query and quietly misrouted it for two weeks before anyone noticed the reply quality had dropped.
What Actually Breaks When You Turn a Router On?
Your evals, first, because most eval suites are built assuming a single model handled every response, and a router breaks that assumption on day one. The second thing that breaks is cost attribution. Someone in finance asks "why did the average cost per request drop 70% but the total token count barely moved," and you realize you now need per-tier cost tracking you never built.
The third thing, and the one nobody warns you about, is caching. If you're using Anthropic's prompt caching, which cuts costs up to 90% and latency up to 85% on repeated long-context prompts by reusing previously processed prompt prefixes (Anthropic, "Prompt caching with Claude," 2024), a naive router that shuffles requests between models on every call destroys your cache hit rate, because the cache is keyed to a specific model and prompt structure. I lost most of a week's savings this way before realizing my router was alternating models on near-identical consecutive requests instead of holding a sticky classification per conversation thread. The fix was embarrassingly simple: cache the routing decision alongside the conversation, not just the prompt. But I only found the bug because the savings number stopped moving and I went looking for why, not because anything alerted me.
Is Semantic Routing Just Load Balancing With a New Name?
No, load balancing distributes identical requests across identical or near-identical capacity to manage throughput, while semantic routing sends different requests to genuinely different models based on what the request needs. The distinction matters because load balancing never risks quality; semantic routing always does, in exchange for the cost savings, which is exactly why it needs the confidence-threshold safety net and not just a classifier running unsupervised in production.
The two get conflated because both sit in the same place in your stack, in front of the model call, making a routing decision before the request goes out. But conflating them is how teams end up shipping a router with load-balancer-level monitoring, which means nobody notices quality drift until a customer does.
The Part That Should Worry You More Than It Currently Does
Every team I've talked to that adopted semantic routing treated it as a cost project. Almost none of them treated it as a reliability project, and that's backward. The moment you have a classifier deciding which model handles a request, you have introduced a new failure mode that didn't exist before, misclassification, and most teams ship it with less monitoring than they'd put on a database migration. The savings are real and the papers back it up, but the actual engineering discipline required to run this safely in production is closer to what you'd apply to a fraud-detection model than a routing table. Almost nobody is building it that way yet, which means the first wave of embarrassing "the AI said something dumb" incidents that get publicly traced back to a root cause is going to be traced back to a stale, unmonitored router, not a bad base model.
My prediction, and it's a narrow one: within the next two years, the routing layer itself becomes the thing vendors charge for and audit, the same way API gateways became a paid, monitored layer instead of a bit of glue code teams wrote themselves. The teams still running an unmonitored classifier they built in an afternoon are the ones who'll be explaining an incident report to someone who asks "wait, what actually decided to use the cheap model here?", and won't have a good answer.
Top comments (0)