When I tried to automate supply chain integrations with AI agents in a manufacturing ERP, everything seemed perfect at first. The agent was supposed to analyze incoming orders, check stock levels, call the supplier API for new raw material orders if necessary, and update the production plan. On paper, this flow would reduce manual intervention by 70%.
However, when it came to the tool-use part, meaning when the agent needed to interact with the outside world, the expected "magic" automation was replaced by complex architectural choices and their accompanying operational costs. I experienced firsthand that providing a tool to an agent was much more than just exposing a function signature; it could have serious consequences in terms of both performance and security. In this post, I will share the costs incurred by these architectural choices and what I learned in the process.
The Basic Logic and Expectations of Tool-Use in AI Agents
The tool-use capability of AI agents refers to the ability of large language models (LLMs) not only to generate text but also to use external tools to perform specific actions. This can mean an agent querying a database, making a request to an API, or even executing specific code. In my experience, this capability was indispensable for automating critical workflows, such as a manufacturing planning agent fetching data from PostgreSQL to check stock levels or sending orders to a supplier portal.
Our general expectation from this capability is that the agent can think like a human and solve a problem by using the right tool at the right time. However, in reality, the quality of the tools provided to the agent, their descriptions, and how the agent interprets these tools form the basis of success. If a tool description is incomplete or lacks sufficient information for the agent to understand the context correctly, the agent might "hallucinate" and call the wrong tool with the wrong parameters. Once, in the financial calculators of a side project, the agent called the get_euro_exchange_rate tool instead of get_dollar_exchange_rate and made an incorrect calculation because the tool description was not specific enough.
[
{
"name": "get_stock_level",
"description": "Fetch the current stock level for the specified product from the database. The 'product_id' parameter is mandatory.",
"parameters": {
"type": "object",
"properties": {
"product_id": {
"type": "string",
"description": "The unique ID of the product to query."
}
},
"required": ["product_id"]
}
},
{
"name": "place_supplier_order",
"description": "Place an order for the specified product with the supplier. 'product_id', 'quantity', and 'supplier_id' parameters are mandatory. Stock check must be performed before ordering.",
"parameters": {
"type": "object",
"properties": {
"product_id": { "type": "string" },
"quantity": { "type": "integer" },
"supplier_id": { "type": "string" }
},
"required": ["product_id", "quantity", "supplier_id"]
}
}
]
Clear tool definitions like the ones above make it easier for the agent to select the correct tool. Otherwise, when the agent detects a "place order" command, it might try to place an order directly without checking stock, which can lead to serious errors in a manufacturing environment. Therefore, it's crucial that tools are defined correctly not only technically but also from a workflow perspective, and that every edge case is considered.
Architectural Choice 1: Monolithic vs. Microservice Tool Architectures
The first major architectural decision I faced when designing the tools provided to AI agents was how to organize these tools: should I group them under a monolithic structure, or should each tool run as its own microservice? This decision directly impacts operational costs, especially in large and complex enterprise systems.
Initially, in a manufacturing ERP, I considered offering the agent "one big tool" by consolidating all internal APIs and database operations under a single RESTful endpoint. This seemed to increase development speed because all tools were defined and managed within the same codebase. However, I quickly realized it would become a bottleneck for scalability and maintenance. For example, the get_stock_level tool, which was called 10,000 times a day, lived within the same monolithic service as the rarely used "add new supplier" tool. A slowdown in the stock inquiry service (e.g., due to an N+1 query error) affected all tool-use capabilities.
ℹ️ Experience
In a client project, the monolithic tool service responded in approximately 1.5 seconds. After moving the stock inquiry service to a separate FastAPI microservice and optimizing indexes in PostgreSQL, we reduced the response time to 150 ms. This enabled the agent to make faster decisions and completed the overall workflow 25% faster.
With the microservice approach, each tool or related group of tools runs in its own independent service. This allows each service to have its own resources, scale independently, and isolate errors. For instance, a StockManagementService and a SupplierOrderService run separately. This separation prevents one service crashing or slowing down from affecting others. In the backend of my own side project, the tools managing user data and the financial calculation tools run in separate services. This way, when there's a heavy load on financial calculations, basic user operations are not affected. However, this approach brings more operational management and monitoring overhead; one has to deal with the complexity of distributed systems. Each service has its own CI/CD pipeline, its own logging and metrics system.
Architectural Choice 2: Tool Orchestration and State Management
How AI agents call tools in sequence and manage the state in between is a critical architectural decision for the agent to successfully complete complex tasks. This decision directly impacts error management, debugging costs, and the agent's overall "reasoning" ability. There are two main approaches: the LLM itself handles orchestration, or an external orchestrator is used.
LLM-driven orchestration means the agent directly reflects its "thinking" ability in tool calls. Through prompt engineering, the agent is given a "planning" capability, and it decides which tool to call and when. For example, when creating a production plan, it's expected to call the get_stock_level, then get_supplier_prices, and then calculate_production_plan tools sequentially. This approach offers flexibility but also carries the risk of the agent "forgetting" or "hallucinating." Once, in a client project, the agent skipped a tool it was supposed to call in step 4 and tried to produce a result directly because it had lost context from the previous steps. This can occur due to LLM token limits or complexity in the prompt.
Using an external orchestrator, on the other hand, means expecting the agent to only suggest "which tool to call and with which parameters," leaving the actual call sequence and state management to a separate service or workflow engine. For example, a ProductionWorkflowService receives suggestions from the agent, sequences them according to its business rules, calls the tools, and returns the results to the agent. This approach allows us to build more robust and fault-tolerant systems. Using the transaction outbox pattern, we can ensure that tool calls are idempotent; meaning, even if the same tool is called multiple times, the system state does not change.
# Simple external orchestrator pseudo-code
def orchestrate_production_plan(agent_suggestion):
if agent_suggestion.tool_name == "get_stock_level":
stock_data = call_tool(agent_suggestion.tool_name, agent_suggestion.parameters)
# Save state to a database or pass to the next step
return {"status": "success", "data": stock_data, "next_step_hint": "get_supplier_prices"}
elif agent_suggestion.tool_name == "get_supplier_prices":
# Use state from the previous step
supplier_data = call_tool(agent_suggestion.tool_name, agent_suggestion.parameters)
return {"status": "success", "data": supplier_data, "next_step_hint": "calculate_production_plan"}
# ... other tools
else:
return {"status": "error", "message": "Unknown tool or invalid state"}
This external orchestrator approach reduces the likelihood of the agent making errors in complex workflows, but it also introduces the cost of the orchestration logic itself becoming a separate codebase that needs maintenance and testing. However, in my experience, especially for critical flows like financial transactions, this additional cost was much lower than the potential cost of the agent's unpredictable behavior.
Security and Isolation: The Dark Side of Tool-Use
The tool-use capabilities of AI agents undoubtedly offer great power, but this power also brings significant security risks. The agent's interaction with external systems can potentially open doors to malicious or erroneous tool calls. Therefore, securely managing agent tool calls is vital in production environments.
Even in the simplest scenario, an agent might accidentally, or due to a prompt injection, attempt to access an unauthorized API. I observed an attempt by an analytics agent in a bank's internal platform to call an update_user_data tool, when it should have had only read permissions. To prevent such situations, we must strictly enforce authorization mechanisms (OAuth2/JWT) for every tool call. Before an agent can call any tool, checking if it has the necessary permissions is a fundamental security layer.
⚠️ Important Note
Due to the inherent "hallucination" risk in LLMs, AI agent tool-use capabilities can lead to unexpected and potentially harmful tool calls. Therefore, every tool call should be treated as if it came from a malicious user and undergo strict input validation and authorization checks.
Furthermore, there are risks of DDoS or resource exhaustion. An agent might try to bring down a service by making excessive requests to an API with incorrect parameters. In the backend of one of my side projects, during a development phase, an AI agent brought down a service by making requests to an internal service at a rate of 150/second, causing it to crash in about 28 seconds. I resolved this with fail-safe mechanisms; implementing a rule to block for 1 hour after 5 failed attempts from a specific IP or agent ID within 5 minutes. This protected the service while also reducing the cost of the agent making errors.
Deeper security measures, such as kernel module blacklisting, should also be considered. For example, if the agent has code execution capabilities, blacklisting certain modules like algif_aead can provide an additional layer of protection against potential CVEs. Additionally, isolating the environment where the agent runs using Docker containers and setting cgroup limits protects other system components by limiting the agent's resource consumption. When I set the soft memory limit of a container to 2GB using memory.high, it prevented the agent from attempting to consume 4GB of memory in a faulty operation, causing it to be OOM-killed, which helped keep the main application stable.
Cost Optimization: LLM Provider Selection and Fallback Strategies
The choice of LLMs used for AI agents to interpret tool calls and make decisions is critical for both performance and cost. There are many different LLM providers on the market, such as Gemini Flash, Groq, Cerebras, and OpenRouter, each with its unique advantages and disadvantages. In my experience, instead of relying on a single provider, strategically combining these providers has both reduced costs and increased system resilience.
For example, in an AI-powered financial calculator, I use LLMs to translate natural language queries from users (e.g., "Calculate the inflation-adjusted return on 100,000 TL for a 5-year term") into tool calls. In this scenario, latency was critical because the user expected an instant response. Groq's high processing speeds and low costs ($0.0002/1k tokens) were ideal for such scenarios; I generally received responses within 300 ms. However, for more complex and less frequent production planning optimizations, I preferred Gemini Pro ($0.002/1k tokens), a more expensive but more comprehensive model, because accuracy and complex reasoning were more important here, while latency was secondary.
💡 Multi-Provider Strategy
Allocating different LLM providers based on specific workloads optimizes costs and reduces dependence on a single provider. If one provider becomes unavailable, automatic fallback mechanisms engage to ensure uninterrupted service.
Another significant advantage of the multi-provider strategy is resilience. If one provider becomes unavailable, I have implemented a fallback mechanism that automatically switches to another provider. Platforms like OpenRouter facilitate this kind of orchestration. In my own system, if a response is not received from the primary provider within 500ms or an API error is returned, a logic is in place to automatically switch to the secondary provider. This way, I've been able to meet the 99.99% uptime target for my financial calculator. Thanks to this fallback mechanism, even during a brief 2-hour outage of Groq, my users experienced no disruption, only an average increase of 500ms in response times.
While this architectural choice required a bit more integration effort initially, it significantly improved both operational costs (I saw an approximate 70% reduction in token costs) and the overall reliability of the system in the long run.
Operational Challenges and Observability
Running AI agents in production environments presents operational challenges different from traditional software systems. Especially for agents with tool-use capabilities, their behavior can be complex and unpredictable, making observability a vital role in this process. Understanding what the agent is thinking, which tool it called when and with which parameters, and whether these calls were successful, simplifies debugging and troubleshooting.
The agents I used in the manufacturing ERP performed complex tasks by calling multiple tools sequentially. In a production order creation workflow, there could be 5-6 different tool calls, such as get_raw_material_stock, check_machine_availability, calculate_production_time. An error or delay in any of these calls could halt the entire workflow or lead to an incorrect result. For example, on May 28th at 03:14 AM, a production planning agent failed to complete a task because the check_machine_availability tool timed out, responding in 3500ms.
To detect such issues, I had to establish a comprehensive observability infrastructure. Logging, metrics collection, and tracing (with OpenTelemetry) were the cornerstones of this infrastructure. I kept detailed logs for each tool call: the agent's intent, the name of the tool called, its parameters, the API response, the elapsed time, and error codes. These logs were collected via journald and forwarded to a central logging system. journald's rate limits prevented the agent's erroneous log generation from crashing my system.
On the metrics side, I monitored the latency, success rate, and error rate of each tool call with Prometheus. On a Grafana dashboard, I could see these metrics for each tool in real-time.
I monitored the average latency of a specific tool with a PromQL query like this:
rate(tool_call_duration_seconds_sum{tool_name="get_stock_level"}[5m]) / rate(tool_call_duration_seconds_count{tool_name="get_stock_level"}[5m])
This allowed me to instantly notice that the 95th percentile latency of the get_stock_level tool increased from 200ms to 1200ms. Investigating the cause of this increase, I discovered that a recently added index in PostgreSQL had broken the query plan and led to an N+1 problem. Such observations enabled us to intervene quickly and prevent service outages.
With tracing, I visualized the agent's decision-making process and how multiple tool calls formed a chain. An OpenTelemetry trace showed the entire process in a single flow, from the agent's call to the LLM, to the LLM's tool selection, to the tool's HTTP request to the API, and finally to the API's response. This was invaluable for pinpointing exactly where and when an error occurred, especially in complex, multi-step tasks.
Conclusion: Where Are We Heading?
The tool-use capabilities of AI agents hold the potential to transform our business processes, that's clear. However, as I've seen in my 20 years of field experience, bringing this technology to production is more than just writing code. Architectural choices, especially how we design, manage, and secure tools, have a direct impact on operational costs and the overall resilience of the system.
Choosing between monolithic or microservice tool architectures, leaving orchestration to the LLM or using an external service, combining different LLM providers, and implementing robust security measures are all critical decisions we cannot simply dismiss as "it is what it is." A wrong choice, while it might offer speed in the short term, can manifest as high maintenance costs, security vulnerabilities, and system outages in the long run.
The biggest lesson I've learned in this process is that we shouldn't view AI agents as black boxes. They have limitations, and they make mistakes. Building a solid engineering infrastructure to anticipate, minimize, and quickly resolve these mistakes is essential. In the coming period, I expect agent patterns to mature further and, especially when combined with RAG (Retrieval-Augmented Generation), to produce more reliable and accurate results.
The experiences in this field have opened many different doors in my career. In particular, applying my knowledge of [related: complex enterprise software architectures] and [related: observability in distributed systems] to AI agent architectures has allowed me to approach problems with a more holistic perspective. As AI agents become more autonomous in the future, the importance of these architectural decisions will only increase exponentially.
Top comments (0)