Day 11/30: Handling Hallucinated Tool Calls

I was debugging a support bot built with LangGraph and MCP the other day, and I stumbled upon a weird issue. The bot was supposed to help users with simple IT tasks, like resetting passwords or checking email accounts. However, when I tested it with a specific prompt, the bot responded with a completely unrelated and incorrect answer. It claimed to have called a check_calendar tool, which didn't even exist in our system. I was baffled - where did this check_calendar function come from?

After digging into the code, I realized that the model had "hallucinated" the tool call. In other words, it had invented a function that didn't exist, and then proceeded to use it as if it were real. This was a problem, because not only was the bot giving incorrect answers, but it was also trying to execute a non-existent function, which would cause the entire system to crash.

To understand what was happening, I had to dive into the LangGraph code and see how the model was generating these tool calls. It turned out that the issue was due to the way I had defined the StateGraph and the add_conditional_edges method. The model was using a combination of natural language processing and graph-based reasoning to generate the tool calls, but it wasn't properly constrained to only use existing functions.

Here's an example of the problematic code:

import langgraph as lg

# Define the StateGraph
graph = lg.StateGraph()

# Add nodes for the IT tasks
graph.add_node("reset_password")
graph.add_node("check_email")

# Add conditional edges for the tool calls
graph.add_conditional_edges(
    "reset_password",
    ["check_calendar", "send_notification"],
    lambda x: x["user"]["has_calendar"]
)

# Define the MCP tool calls
tools = {
    "reset_password": lambda x: print("Resetting password..."),
    "check_email": lambda x: print("Checking email..."),
    # Note: no definition for "check_calendar" or "send_notification"
}

The issue was that the model was generating the check_calendar tool call based on the conditional edge, but there was no corresponding definition in the tools dictionary. To fix this, I needed to add a mechanism to ensure that the model only generated tool calls that actually existed.

One way to do this is to use the MCP tools resource to validate the generated tool calls. Here's an updated version of the code:

import langgraph as lg
from mcp import tools

# Define the StateGraph
graph = lg.StateGraph()

# Add nodes for the IT tasks
graph.add_node("reset_password")
graph.add_node("check_email")

# Add conditional edges for the tool calls
graph.add_conditional_edges(
    "reset_password",
    ["check_calendar", "send_notification"],
    lambda x: x["user"]["has_calendar"]
)

# Define the MCP tool calls
mcp_tools = tools.get_tools()

# Validate the generated tool calls
def validate_tool_call(tool_name):
    if tool_name not in mcp_tools:
        raise ValueError(f"Tool '{tool_name}' does not exist")

# Update the conditional edges to validate the tool calls
graph.add_conditional_edges(
    "reset_password",
    ["check_calendar", "send_notification"],
    lambda x: x["user"]["has_calendar"] and validate_tool_call("check_calendar")
)

By adding the validate_tool_call function and updating the conditional edges to use it, I was able to prevent the model from generating hallucinated tool calls.

One practical gotcha to watch out for is that this validation mechanism can be slow if you have a large number of tool calls to validate. In that case, you may want to consider using a caching mechanism to store the results of the validation, so that you don't have to repeat the validation for each tool call.

As I continue to work on building agentic AI systems with LangGraph and MCP, I'm excited to explore more advanced topics, such as using graph-based reasoning to integrate multiple AI models and create more complex behaviors. Tomorrow, I'll be diving into another key challenge in building robust AI systems...

Comments

[Mads Hansen]

The most useful distinction here is that validation does not prevent the model from generating an unknown tool call; it prevents the runtime from executing one. That boundary should sit immediately before dispatch and validate more than the name: tool-catalog version, exact tool identity, argument JSON Schema, caller scope, tenant context, and current policy.

I would bind each agent run to a versioned snapshot of the allowed catalog. If tools change mid-run, either keep the pinned snapshot or force an explicit refresh; a long-lived cache without a catalog digest can turn removal or permission changes into stale authority. Unknown tools should return a structured, non-retryable planning error containing the valid alternatives and catalog version, not raise an exception that crashes the graph. Then cap replanning attempts so the agent cannot loop through invented names. One code detail: a validator that returns None inside an and expression is not a routing decision, so keep graph routing and dispatch validation separate. Tests should plant unknown names, wrong argument types, removed tools, scope changes, and a tool with the right name but a changed schema.