Why does everything in an agent go through the LLM?

I've been building an agent system for a while now, and one question keeps coming back the deeper I get into it: why does everything have to go through the LLM?

Obviously there are plenty of cases where the model belongs in the middle. Human input is messy. People are vague, change their mind halfway through a sentence, leave out context, or ask for something where there isn't one objectively correct answer. That's exactly where a model is useful.

But a lot of work inside an agent isn't like that. Sometimes the system already knows what kind of operation it's dealing with, and regular software already knows how to handle it reliably. In those cases, sending the request through a model just so the model can tell the system to do something it already knew how to do feels unnecessary.

A lot of agent systems can be simplified down to something like this:

request → model → tool → model → result

I've started questioning whether the model really needs to be the default path.

Let software be software

The direction I've been experimenting with is pretty simple in theory. If a request actually needs interpretation, reasoning, planning, or some understanding of what the person means, use the model. If it can be handled predictably by software, let software handle it.

Something closer to this:

                 ┌→ deterministic handling
request → route ─┤
                 └→ model reasoning

The point isn't to avoid AI or minimize model calls just for the sake of it. It's more about not using probabilistic reasoning for a problem that doesn't actually require probabilistic reasoning.

Of course, drawing that diagram is the easy part. The routing problem shows up almost immediately.

You can start with rules, regex, intent classifiers, and special cases, but that can become its own mess pretty quickly. Keep adding enough exceptions and eventually you've built a giant state machine that somebody has to babysit forever. At that point, maybe you've just moved the complexity somewhere else.

A smaller model acting as a classifier is another option. It doesn't need to solve the task; it could simply decide whether the request needs semantic reasoning or whether an existing piece of software is better suited for it.

I like that idea for the ambiguous cases, but I don't think I'd want it to become a mandatory first hop either. If something is already obvious to the runtime, calling a model just to classify what the system already knows puts probabilistic inference right back in front of every request.

What makes more sense to me right now is a layered approach. Handle the things you already know how to handle. Use semantic classification where the answer genuinely isn't obvious. Fall back to deeper reasoning when you actually need it.

I'm still figuring out where that line should sit.

Reasoning isn't authority

The other thing this has made me think about is how much responsibility we put on the model once it does enter the picture.

Understanding what should happen and having the authority to make it happen are two different problems.

I keep coming back to a separation that looks roughly like this:

reason → authorize → execute → verify

The model may be involved in reasoning about an action, but that doesn't mean the same component should automatically decide the action is allowed, perform it, and then declare that it worked.

That distinction matters more as agents get more capable. A model confidently saying something was completed isn't the same thing as the system actually knowing it happened.

I don't think this requires taking all decision-making away from the model. Models are useful precisely because they can deal with situations that don't fit nicely into a pile of hard-coded rules. I just don't think that usefulness automatically makes them the right place to put every other responsibility too.

The part I'm not convinced we've solved

There's an obvious criticism of all of this: eventually the router itself can become the problem.

If you keep expanding the deterministic side, how long before you're maintaining a second system that's more complicated than the model call you were trying to avoid? If you use a model to route to another model, when does the extra layer stop being worth it? And what happens when something gets routed down the wrong path?

Those are the questions I find more interesting than simply asking whether agents should use more or less AI.

I don't think the answer is "use less AI." I think the question is where AI actually earns its place in the system.

Use the model for the parts that need meaning. Let deterministic software handle the things it already does well. Keep reasoning separate from authority. Then figure out what to do with everything that falls somewhere in between.

I'm curious how people building real agent systems are approaching this. Are you letting the model handle most of the routing and putting deterministic controls around what happens afterward? Are you keeping some work away from the model entirely? And if you've tried splitting the two, at what point did the routing layer start becoming harder than the problem it was supposed to solve?

Comments

[TokenLat]

The routing instinct here is right, and the part most pieces miss is the cost side. Every request → model hop that could have been deterministic pays full model price, full latency, and the risk of a nondeterministic answer for no reason. The hard part isn't deciding whether to route — it's keeping the router itself cheap enough that routing a trivial call doesn't cost more than just sending it to the model. A router that needs its own model call to decide defeats the purpose. So the practical bar is: route only when the saving (skipping a frontier call) exceeds the router's own cost. How do you keep the router cheap — static rules, a tiny model, or do you just accept the overhead on the bet that most calls are worth routing?

[Jo Do]

The layered router makes sense if the deterministic lane is explicit about when it no longer applies. I would have each handler return handled, rejected, or unknown, rather than forcing every request into a route. unknown can escalate to semantic classification without turning a regex miss into a confident decision. And, as you say, authorization should remain a separate gate after routing.