Design schemas, handle errors, and secure API connections for production agentic systems.
Function calling is the mechanism by which a language model requests execution of an external function with specific parameters, parsed into a structured format that your application can validate and execute. It is the operational core of agentic systems: the bridge between model reasoning and real-world action. Without function calling, an LLM is a text predictor. With it, an LLM becomes a decision engine that can inspect data, modify state, and solve multi-step problems by orchestrating APIs, databases, and business logic.
Why this matters now
For most of 2023 and 2024, function calling was a novel feature that distinguished capable models from others. By 2026, it is table stakes. Every major model provider (OpenAI, Anthropic, Google, together.ai) ships native function calling with reasonable quality. The question is no longer "does our model support this?" but "how do we design, validate, and operate tool-calling agents safely and efficiently?" Agentic applications are moving from prototype to production, and that transition demands rigorous schema design, error handling, and authorization boundaries. Builders who skip these steps ship agents that fail in unexpected ways, expose unintended capabilities to users, or waste tokens on malformed requests.
The competitive surface has also shifted. Smaller, cheaper models now rival larger ones at function calling. GPT-4o-mini and Claude 3.5 Haiku reliably handle 10 to 15 tool calls in a single conversation turn with accurate argument parsing. This means cost and latency, not capability, often drive model selection. Builders must understand schema design, error recovery, and parallel execution to extract maximum value from these smaller, faster models without sacrificing reliability.
How function calling works: the loop
Photo by www.kaboompics.com on Pexels.
Function calling operates on a simple loop: the model sees a list of available tools, generates a response that may include a structured request to invoke one or more of them, and your application executes those requests and returns the results as new context for the model. The model then continues reasoning and may call more tools, generate a final response, or request clarification.
The flow is:
User sends a query or task.
Your system prepares a list of available tools with their schemas (name, description, parameters).
You send the user message and tool list to the LLM API.
The model responds with either text, a tool call, or both.
If a tool call is present, you validate the arguments against the schema, execute the function, and capture the result.
You append the tool result to the conversation history as a new message.
You send the updated conversation back to the model (no user input required).
Repeat from step 4 until the model generates a final response or an error condition is met.
This loop is stateless from the model's perspective. The entire conversation history, including all tool calls and results, must be sent with every request. Stateless design simplifies deployment, debugging, and recovery, but it also means your conversation memory grows with each tool invocation. Most production systems implement a sliding window or summarization strategy after 10 to 15 turns to manage token count.
Designing tool schemas that models understand
The quality of your tool definitions directly affects the model's ability to call them correctly. A vague or incomplete schema leads to malformed arguments, incorrect tool selection, and wasted tokens on error recovery. A well-designed schema is explicit, concise, and self-documenting.
Each tool should include:
Name: A single snake-case identifier, e.g., "search_knowledge_base", not "kbSearch" or "search_kb". Consistency matters.
Description: A one-sentence summary of what the tool does. Not why it exists, but what it accomplishes. "Retrieves documents matching a search query from the company knowledge base" is better than "Searches the KB".
Parameters: A JSON schema defining the input structure. Every parameter should have a type, description, and (if required) an enum of valid values or constraints.
Return type: Document what the tool returns, especially if it can return an error. Models benefit from knowing the shape of success and failure.
A concrete example. Instead of this vague definition:
"get_user_data": "Gets user info"
Define it like this:
"get_user_data": { "description": "Fetch user account information by user ID. Returns account status, email, and subscription tier. Returns 404 error if user does not exist.", "parameters": { "type": "object", "properties": { "user_id": { "type": "string", "description": "The unique user ID (UUID format)" } }, "required": ["user_id"] } }
The second definition tells the model the expected input format, what it will receive in response, and what to expect if something goes wrong. This reduces hallucinated arguments and improves error recovery.
Enums are especially powerful. If a tool accepts a status filter, enumerate the valid values:
"status": { "type": "string", "enum": ["active", "paused", "cancelled"], "description": "Filter by subscription status" }
Without the enum, models may try "archived" or "inactive" and trigger an error. With it, they have hard boundaries. This is not a suggestion; it is a correctness constraint.
Avoid tools that do too much. A single "execute_query" tool that accepts arbitrary SQL is dangerous and often fails because models struggle with correct syntax. Break it into specific, named operations: "query_users_by_email", "count_orders_by_date", "list_recent_transactions". Specificity reduces error rates and makes auditing easier.
Error handling and retry logic
Photo by Mustafa Alper alper on Pexels.
Tools fail. Networks time out, APIs return 429 errors, user permissions change mid-conversation. The difference between a robust agent and a fragile one is how it handles these failures.
When a tool call fails, return the error to the model as a tool result, not as an exception. The model needs to see the error in context to adjust its reasoning. Include an error field with a human-readable message:
{ "tool_use_id": "call_12345", "content": [{"type": "text", "text": "Error: User does not have permission to delete this resource. Only the owner or an admin can delete it."}] }
The model will then recognize the constraint and either ask for escalation, try a different operation, or inform the user. If you catch the error and terminate the agent, the user gets no explanation and no option to recover.
For transient failures (timeouts, rate limits), implement exponential backoff at the application level before returning an error to the model. If a tool call fails due to rate limiting, wait 2 seconds, try again up to 3 times, then return a "rate_limit_exceeded" message to the model. This absorbs temporary hiccups without surfacing them to the model.
For validation errors (malformed arguments), do not re-call the model with the same tool. Instead, return the validation error and let the model adjust. For example, if the model passes user_id as an integer but the schema requires a string, respond with: "Error: user_id must be a UUID string, e.g., '550e8400-e29b-41d4-a716-446655440000'". The model will self-correct on the next attempt.
Set a maximum tool call depth (e.g., 15 turns) to prevent infinite loops. If the model reaches that limit without generating a final response, return a message like "I've reached my thinking limit for this query. Here's what I found so far" and summarize the results. This is rare with modern models, but it prevents runaway costs.
Parallel tool calls and efficiency
Modern LLMs can invoke multiple tools in a single response. Instead of waiting for one tool to complete before requesting the next, you execute all of them concurrently and return all results in one batch. This reduces latency and token overhead.
For example, an agent planning a business trip might call three tools in parallel: "check_flight_prices", "check_hotel_availability", "check_rental_car_rates". Executing serially would require three separate model calls. In parallel, it is one call to the model, three concurrent executions, and one call to synthesize the results.
Parallel execution is straightforward in code (thread pools, asyncio), but sequencing the return is critical. Return all results in a single message, grouped by tool call ID, so the model sees them together:
[ { "tool_use_id": "call_abc", "content": [{"type": "text", "text": "Flights: 3 options found, prices from $450 to $620"}] }, { "tool_use_id": "call_def", "content": [{"type": "text", "text": "Hotels: 8 options, rates from $80 to $250 per night"}] }, { "tool_use_id": "call_ghi", "content": [{"type": "text", "text": "Rental cars: 5 options, rates from $35 to $75 per day"}] } ]
The model now has all three results in context and can synthesize a recommendation. If you returned them in separate messages, the model would lose some reasoning cohesion and waste tokens reconstructing the full picture.
One caveat: if one tool call depends on the result of another, do not parallelize. If you need to "fetch_customer" before you can "fetch_orders_for_customer", execute them serially. The model will not hallucinate a customer ID, but it may retry a failing tool call if it sees the dependencies in the schema. Use tool descriptions to hint at dependencies: "Requires customer_id from fetch_customer".
Authorization and security boundaries
A powerful agent is a dangerous one if you do not constrain what it can do. The model itself is not a security boundary. It will follow instructions in a user prompt to perform actions outside its intended scope. You must enforce permissions at the application layer.
Implement three levels of control:
Tool availability per user: Not every user should see every tool. An admin agent might have 20 tools; a customer support agent might have 5. Load the tool list based on the user's role and permissions.
Argument validation and filtering: Even if a user has access to a tool, they should not be able to operate on all data. If a customer support agent calls "update_customer_note", validate that the customer_id belongs to a ticket assigned to that agent. This is application logic, not LLM logic.
Execution audit and rate limiting: Log every tool call with the user ID, timestamp, tool name, and arguments. This is your evidence trail. Implement rate limits per user per tool to detect abuse: a single user calling "delete_resource" 100 times in a minute is anomalous.
A concrete pattern: before executing a tool, pass the arguments through a policy engine. Does the authenticated user have permission to perform this action on this data? Only then execute.
if not authorize_action(user_id, tool_name, arguments): return "Error: You do not have permission to perform this action." tool_result = execute_tool(tool_name, arguments)
Never embed secrets (API keys, database passwords) in tool definitions. If a tool needs credentials, fetch them server-side using the user's identity, then execute the tool with those credentials. The model never sees them.
Tool use expands the attack surface of your application. A prompt injection attack that was merely annoying (model generates rude text) becomes serious if the attacker can manipulate tool calls. Sanitize user input, limit tool scope, and assume the model will be tricked occasionally. Design your tools so that even if they are called unexpectedly, the damage is contained.
When function calling fails
Despite best efforts, function calling agents break in production. Models hallucinate tool names that do not exist. They misunderstand parameter types and pass strings where numbers are required. They get stuck in loops trying the same failing tool repeatedly. They succeed at the technical task but misinterpret what the user actually wanted.
Hallucinated tool calls are common with smaller models or when tool definitions are vague. If the model invents a tool name like "get_customer_details" when only "fetch_customer" exists, return a "tool_not_found" error and list the available tools again. The model will almost always self-correct. If it does not after two attempts, escalate to a human or degrade gracefully.
Type mismatches (passing a list where a string is expected) indicate schema confusion. Validate arguments strictly and return clear errors. Over time, you will see patterns in these errors. If 10% of calls to "search_orders" pass a non-string query, the schema description is ambiguous. Revise it.
Reasoning failures are harder to debug. The model selects the correct tool but with wrong arguments, or it calls tools in a nonsensical order, or it gives up too early. These are usually due to unclear task framing or missing context. Provide concrete examples in the system prompt of how to use tools. Instead of "You can search for customers", say "When a user asks for customer info, first search_customer_by_name to find their ID, then fetch_customer with that ID to get details".
Finally, acknowledge that at some scale, human-in-the-loop is necessary. An agent that handles 95% of queries correctly but occasionally tries to delete the entire customer database without confirmation is unacceptable. Implement guardrails: require approval for destructive operations, cap the total impact per query (e.g., update no more than 100 records), and have a circuit breaker that disables the agent if error rates spike.
Choosing and evaluating models for tool use
Not all models are equally good at function calling. The major divider is instruction-following capability and reasoning. GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash parse complex schemas reliably and generate correct arguments consistently. Smaller models like GPT-4o-mini and Claude 3.5 Haiku are 70% to 85% as capable at a fraction of the cost and latency. Open-source models (Llama, Mistral) vary widely. Some fine-tuned versions handle tools well, others struggle with schema validation.
Evaluate on your actual tool set. Build a test suite of realistic queries, define the ground-truth tool calls, and measure accuracy (correct tool selected, all required arguments present and valid type). Run the test suite on your candidate models. A 95% accuracy model might seem good in aggregate, but if the 5% failures are all on your most critical tools, that is unacceptable.
Cost and latency also matter. GPT-4o is more capable but costs more and is slightly slower than GPT-4o-mini. If you have a latency-sensitive application (customer support chat), GPT-4o-mini with tight error handling might be better than GPT-4o with sloppy retries. If you have a batch job (nightly report generation), cost per query dominates and a smaller model is preferable.
As of early 2026, the recommendation for most new projects is: start with Claude 3.5 Haiku or GPT-4o-mini, measure accuracy on your tool set, and upgrade to a larger model only if error rates are unacceptable. The cost and speed advantages of smaller models often outweigh the minimal accuracy gains of larger ones, especially once you implement proper error handling and retries.
Building a production agentic system is not a one-time engineering task. It requires ongoing monitoring, schema refinement, error analysis, and model evaluation. Invest in observability from day one. Log every tool call, every error, every retry. This data is your feedback loop for improving agent reliability. Function calling will remain a core capability of LLMs for years. The builders who treat it as a serious engineering problem, not a novelty feature, will ship the most reliable and cost-effective agents.
This article was originally published on AI Glimpse.
Top comments (0)