I've been building an agent system for a while now, and one question keeps coming back the deeper I get into it: why does everything have to go through the LLM?
Obviously there are plenty of cases where the model belongs in the middle. Human input is messy. People are vague, change their mind halfway through a sentence, leave out context, or ask for something where there isn't one objectively correct answer. That's exactly where a model is useful.
But a lot of work inside an agent isn't like that. Sometimes the system already knows what kind of operation it's dealing with, and regular software already knows how to handle it reliably. In those cases, sending the request through a model just so the model can tell the system to do something it already knew how to do feels unnecessary.
A lot of agent systems can be simplified down to something like this:
request → model → tool → model → result
I've started questioning whether the model really needs to be the default path.
Let software be software
The direction I've been experimenting with is pretty simple in theory. If a request actually needs interpretation, reasoning, planning, or some understanding of what the person means, use the model. If it can be handled predictably by software, let software handle it.
Something closer to this:
┌→ deterministic handling
request → route ─┤
└→ model reasoning
The point isn't to avoid AI or minimize model calls just for the sake of it. It's more about not using probabilistic reasoning for a problem that doesn't actually require probabilistic reasoning.
Of course, drawing that diagram is the easy part. The routing problem shows up almost immediately.
You can start with rules, regex, intent classifiers, and special cases, but that can become its own mess pretty quickly. Keep adding enough exceptions and eventually you've built a giant state machine that somebody has to babysit forever. At that point, maybe you've just moved the complexity somewhere else.
A smaller model acting as a classifier is another option. It doesn't need to solve the task; it could simply decide whether the request needs semantic reasoning or whether an existing piece of software is better suited for it.
I like that idea for the ambiguous cases, but I don't think I'd want it to become a mandatory first hop either. If something is already obvious to the runtime, calling a model just to classify what the system already knows puts probabilistic inference right back in front of every request.
What makes more sense to me right now is a layered approach. Handle the things you already know how to handle. Use semantic classification where the answer genuinely isn't obvious. Fall back to deeper reasoning when you actually need it.
I'm still figuring out where that line should sit.
Reasoning isn't authority
The other thing this has made me think about is how much responsibility we put on the model once it does enter the picture.
Understanding what should happen and having the authority to make it happen are two different problems.
I keep coming back to a separation that looks roughly like this:
reason → authorize → execute → verify
The model may be involved in reasoning about an action, but that doesn't mean the same component should automatically decide the action is allowed, perform it, and then declare that it worked.
That distinction matters more as agents get more capable. A model confidently saying something was completed isn't the same thing as the system actually knowing it happened.
I don't think this requires taking all decision-making away from the model. Models are useful precisely because they can deal with situations that don't fit nicely into a pile of hard-coded rules. I just don't think that usefulness automatically makes them the right place to put every other responsibility too.
The part I'm not convinced we've solved
There's an obvious criticism of all of this: eventually the router itself can become the problem.
If you keep expanding the deterministic side, how long before you're maintaining a second system that's more complicated than the model call you were trying to avoid? If you use a model to route to another model, when does the extra layer stop being worth it? And what happens when something gets routed down the wrong path?
Those are the questions I find more interesting than simply asking whether agents should use more or less AI.
I don't think the answer is "use less AI." I think the question is where AI actually earns its place in the system.
Use the model for the parts that need meaning. Let deterministic software handle the things it already does well. Keep reasoning separate from authority. Then figure out what to do with everything that falls somewhere in between.
I'm curious how people building real agent systems are approaching this. Are you letting the model handle most of the routing and putting deterministic controls around what happens afterward? Are you keeping some work away from the model entirely? And if you've tried splitting the two, at what point did the routing layer start becoming harder than the problem it was supposed to solve?
Top comments (8)
The routing instinct here is right, and the part most pieces miss is the cost side. Every
request → modelhop that could have been deterministic pays full model price, full latency, and the risk of a nondeterministic answer for no reason. The hard part isn't deciding whether to route — it's keeping the router itself cheap enough that routing a trivial call doesn't cost more than just sending it to the model. A router that needs its own model call to decide defeats the purpose. So the practical bar is: route only when the saving (skipping a frontier call) exceeds the router's own cost. How do you keep the router cheap — static rules, a tiny model, or do you just accept the overhead on the bet that most calls are worth routing?Yeah, that’s the part I’ve been watching pretty closely too. If the router needs a meaningful model call just to decide whether to avoid another model call, you can end up saving nothing. What’s worked best for me so far is not treating routing as one universal mechanism. If the runtime already knows what kind of operation it’s looking at, there’s no reason to pay for classification at all. Those cases can stay deterministic and finish without touching the model. The model only becomes useful once the request actually needs interpretation or lands in the ambiguous middle. So I’m less interested in building one “smart router” that sees everything, and more interested in making sure the obvious cases stay cheap while the uncertain ones are allowed to cost more. I think that’s probably the real tradeoff: the router has to be cheaper than the model call it’s avoiding, otherwise you’re just moving the bill around. ** (Before this gets ai generated tested yes it is, however it's based off MY work, and english is not my first language and i need to ensure i'm not spilling secrets, also well spoken for you guys to understand)**
Exactly — the router has to be cheaper than the call it's avoiding, otherwise you're just moving the bill. "Make the obvious cases skip classification entirely" is the part most router designs get wrong: they pay for a model call to decide whether to avoid a model call.
Keeping the deterministic path free of any model involvement, and only escalating the ambiguous middle, is what makes the economics actually work. The tradeoff you named — obvious cases cheap, uncertain ones allowed to cost more — is the whole game.
The layered router makes sense if the deterministic lane is explicit about when it no longer applies. I would have each handler return
handled,rejected, orunknown, rather than forcing every request into a route.unknowncan escalate to semantic classification without turning a regex miss into a confident decision. And, as you say, authorization should remain a separate gate after routing.Yeah, I like that a lot. The
unknownstate is probably the important part, because it gives the deterministic side permission to say “this isn’t mine” without pretending it understands the request. I’ve already found the same idea useful on the verification side, where a result isn’t just treated as success or failure based on what the model says, the runtime has explicit states for whether the work actually proved out or not. Bringing that same mindset into routing makes sense.handledmeans the deterministic path owns it,rejectedmeans it knows the request doesn’t fit, andunknownmeans stop guessing and escalate. And yeah, I’d still keep authorization completely separate after that. Routing should decide where something belongs, not whether it’s allowed to happen.I’d decide where the model earns its place empirically: run a proposed deterministic route in shadow beside the current model path, then compare accepted outcomes, disagreement, latency and cost by consequence class. Promote only stable classes to the fast path and keep an immediate rollback when drift changes the disagreement rate. That limits the state-machine problem because rules graduate on evidence rather than accumulating as exceptions.
Great question — and in practice NOT everything should go through the LLM. When we built our routing layer (32 models behind one OpenAI-compatible key at heypico.ai), the reliable agents turned out to be the ones that use the LLM for judgment calls and deterministic code for everything else: retries, schema validation, failover between providers. The LLM decides WHAT to do; plain code decides HOW it executes. Once we moved model-selection and fallback logic out of the prompt and into routing config, agent flakiness dropped noticeably — and swapping models became a config change instead of a rewrite.
Moving fallback and model selection out of the prompt is a big one. Once that stuff becomes runtime config instead of prompt behavior, the whole system gets easier to swap, test, and reason about. That separation is where things started feeling much less fragile for me too.