There is a pattern I keep seeing when people build AI agents.
The first version is usually simple. You give the model access to a search tool. It works. Then someone asks, “Can it also search our internal documentation?” So another tool gets added. Then it needs customer information, so you connect the CRM. Then someone wants it to send emails. Then update records. Then query a database. Then search the web.
A few iterations later, you have an agent with ten, twenty, or even more tools. On paper, the system looks significantly more capable. In practice, it can become significantly less predictable.
The problem isn't that any individual tool is bad. The problem is that every new tool increases the number of decisions the model has to make before it can actually solve the user's request.
The agent now has to determine which tool to use, what arguments to provide, whether it needs one tool or several, what order those tools should be called in, and whether it should perform an action at all.
Tool selection has quietly become another reasoning problem. And that reasoning problem gets harder as the action space grows.
The Hidden Cost of Adding Tools
Consider an agent with the following capabilities: web search, internal documentation search, database search, customer lookup, account lookup, customer updates, account updates, and email.
A user asks: “What is our refund policy?”
The model needs to understand the intent first. Is this a question about internal policy? If so, should it search documentation or query a database? Does the database contain policy information? Should it search the public web?
Now change the question: “Has this customer requested a refund before?”
The correct data source is probably different. Now change it again: “Issue a refund and email the customer.”
The agent needs to perform an action involving multiple systems, potentially in a specific order, while also respecting authorization rules.
What looks like a simple tool-calling problem is actually a sequence of decisions:
User request → intent → data source → tool → arguments → execution → result
Every additional tool expands that decision space. This is why adding tools doesn't simply increase capability. It also increases the system's decision surface.
A useful agent isn't the one that has access to every system in your company. It's the one that has access to the systems it needs, and little else.
Retrieval and Actions Are Different Problems
One of the first architectural decisions I make with an agent is separating knowledge retrieval from actions. A question such as: “How does our refund policy work?” should not necessarily have access to the same tools as: “Refund this customer's subscription.”
The first is a knowledge problem. The second is an operational problem. Combining both into one unrestricted agent creates unnecessary complexity.
A better architecture separates them. The initial agent determines what type of task the user is trying to perform and routes it to a more specialized component. Conceptually:
The knowledge agent might have access to internal documentation and approved search systems. The data agent might have access to customer or account information. The action agent might have access to transactional APIs.
This means the model is no longer choosing between every tool available in the organization. It is reasoning within a much smaller action space. That reduction matters.
Tool Descriptions Are Part of the Model's Decision Boundary
Tool definitions are often treated as developer-facing documentation. For an LLM, they are much more than that. The model uses the tool name, description, parameters, and schema to determine whether and how the tool should be called. Compare these two definitions.
The first tells the model almost nothing:
- Search what data?
- Internal documents?
- Customer records?
- The public internet?
- Billing information?
Now consider a more explicit definition:
The second tool has a much clearer boundary. That distinction becomes increasingly important when several tools have similar names or overlapping functionality.
If an agent has search_docs, search_web, and search_db, the descriptions need to clearly establish when each tool should and should not be used. The same applies to parameters.
A parameter called id is ambiguous. An argument called customer_id tells the model considerably more about what it expects. Good schemas reduce uncertainty before the model has to reason about the action. This is why I consider tool schemas part of the agent's architecture, not just API documentation. The tool definition influences the model's decision boundary.
Overlapping Tools Are Particularly Dangerous
A large number of tools isn't always the biggest problem. Overlapping tools are. Suppose an agent has:
Those tools have distinct names, but depending on their descriptions, there may still be significant overlap. Now imagine the tools are:
The model has even less information to distinguish them. When several tools can plausibly handle the same request, the model has to infer which data source the developer intended it to use.
That creates unnecessary routing ambiguity. If two tools perform similar operations but against different data sources, the distinction should be explicit in both the tool name and schema. For example:
The names themselves now communicate something about the architecture. This may sound like a small implementation detail. In a tool-heavy agent, it isn't. Clear tool boundaries reduce incorrect calls, unnecessary retries, and routing uncertainty.
Not Every Decision Needs an LLM
Another mistake is assuming that every decision in an AI system should be made by an LLM. It shouldn't.
LLMs are useful when the system needs to interpret ambiguous language, reason about information, or decide between possibilities that aren't easily expressed as deterministic rules.
But many routing decisions are predictable. If your application already knows that a request is a customer lookup, there is little value in asking another LLM which agent should handle customer lookup. A simple application-level route can do it:
This makes the system easier to test and easier to reason about. It also removes unnecessary model calls, which can reduce latency and cost.
The principle is straightforward: Use probabilistic reasoning where uncertainty exists. Use deterministic logic where the decision is predictable.
An AI architecture doesn't become more intelligent by putting an LLM between every two components. Sometimes the best engineering decision is to remove the model from the loop.
The LLM Should Never Be Your Security Boundary
Tool selection becomes even more important when tools can modify data or perform privileged operations.
Imagine an agent has access to operations such as deleting users, updating accounts, issuing refunds, or sending sensitive information. It is not enough to tell the model: “Only call this tool when the user is authorized.”
The model should never be responsible for enforcing authorization. An LLM can misunderstand context. It can interpret a request incorrectly. It can be manipulated through prompt injection. It can select the wrong tool. Even a highly capable model remains a probabilistic component.
Authorization needs to exist outside of it. The architecture should look more like:
The model requests an operation. The application decides whether that operation is permitted.
For example, an authorization service might evaluate the authenticated user, requested action, target resource, and applicable policies before allowing execution.
This distinction is critical: The LLM can request an action. It should not be the security boundary. The same principle applies to sensitive data access. Don't rely on the model to decide whether it is allowed to retrieve a customer's private information simply because the model has been given access to a customer lookup tool. Enforce that boundary at the application or service layer.
Tool Permissions Should Follow the Architecture
Once you start separating agents, permissions become much easier to manage. A knowledge agent doesn't need permission to modify customer records. A reporting agent doesn't need permission to issue refunds. A communication agent might be allowed to draft emails but not send them without an additional approval step.
This gives you a useful security property: least privilege at the agent level. Instead of asking one model to safely navigate every capability in the organization, each component gets access only to the tools necessary for its job. This also makes failures easier to contain.
If the knowledge agent makes a mistake, it doesn't automatically have access to transactional operations. If a customer lookup fails, it doesn't have the ability to modify the customer's account as a fallback. The architecture itself becomes part of the safety mechanism.
Measure Tool Selection, Not Just the Final Answer
Agent evaluation often focuses on the final response. That's necessary, but it isn't enough.
Suppose an agent eventually gives the correct answer after making four unnecessary tool calls, querying the wrong system first, retrying a failed request twice, and spending several seconds navigating tools it didn't need.
The final answer is correct. The system is still poorly designed. For agents, intermediate decisions matter. You should measure things such as tool-selection accuracy, argument accuracy, routing latency, tool failure rate, retry rate, and unauthorized attempts.
The evaluation question shouldn't only be: “Did the agent answer correctly?” It should also be: “Did the agent choose the correct action?” And: “Did it use the minimum number of actions necessary?”
This distinction becomes particularly important as agents move from read-only tasks into real-world operations.
A wrong tool selection can mean more than a bad answer. It can mean unnecessary API costs, increased latency, stale data, incorrect writes, or a security incident.
More Capability Does Not Mean More Reliability
There is a natural tendency to measure an agent by the number of things it can do. An agent with access to 30 tools sounds more capable than an agent with five. But if the five-tool agent solves the target workflow more reliably, which system is actually better?
In production, capability is only useful when it can be exercised reliably. Every tool introduces another possible selection error. Every write operation introduces another potential failure. Every overlapping tool introduces another routing ambiguity. Every privileged tool introduces another security boundary that needs to be enforced.
This is why I prefer thinking about agent design in terms of action-space reduction. Start with the task. Determine what actions are genuinely required. Remove unnecessary tools. Separate tools that belong to different domains. Make tool descriptions explicit. Use deterministic routing when possible. Give each agent the minimum permissions required. Then measure whether the resulting system actually performs better.
Build the Smallest Action Space That Solves the Problem
The best agent architecture isn't the one with the most integrations. It's the one where every integration has a reason to exist.
If an agent is designed to answer internal product questions, it probably doesn't need access to payments, account deletion, or customer modification. If an agent is responsible for customer operations, it may not need unrestricted web access. If an agent is sending communications, it shouldn't automatically have administrative access to every backend system. This creates a much cleaner architecture:
User
↓
Task Classification
↓
Explicit Route
├── Knowledge Agent
├── Data Agent
└── Action Agent
↓
Authorized Tools
↓
Tool Result
↓
Response
The result is not necessarily a less capable system. It's a more constrained system. And constraints are valuable in agentic systems.
A model that has fewer possible actions has fewer opportunities to choose the wrong one. That's the counterintuitive part of building reliable AI agents. The instinct is usually to add capabilities whenever the agent encounters a limitation. But sometimes the correct engineering response is the opposite.
Remove the tools it doesn't need. Give it clearer schemas. Reduce overlapping capabilities. Route predictable tasks deterministically. Keep authorization outside the model. And measure the decisions happening inside the agent, not just the final response.
The goal isn't to build an agent that can do everything. The goal is to build an agent that can do the right things, reliably. Fewer tools + clearer boundaries + explicit routing + hard permission controls = a much more predictable agent.
Top comments (0)