Building Reliable AI Agents: How I Fixed JSON Hallucinations in Tool Calling

#ai #devops

If you've built an AI Agent that uses external tools, you know the pain. You define a strict JSON schema for your tool, send the prompt, and the LLM returns something like this:

Here is the data you requested:

json
{
"action": "search",
"query": "OpenAI API limits"
}

I hope this helps!

markdown

And your Python json.loads() function just crashed.

LLMs are text predictors, not schema validators. When you ask them to output JSON, they love to add Markdown syntax, conversational filler, or even hallucinate extra parameters.

In this post, I’ll share the 3-layer approach I used to bring the Tool Calling accuracy of my Agent from ~70% to near 99%.

The Test: OpenAI vs. Claude vs. Llama 3

I ran a simple stress test with 50 complex nested parameters using raw prompts. Here's the failure rate of raw JSON output:

GPT-4o: 12% failure (mostly added Markdown backticks)
Claude 3.5 Sonnet: 15% failure (tended to add conversational text)
Llama 3 70B (via Groq): 45% failure (hallucinated keys, missing quotes)

To fix this, relying on a simple prompt like "Please output in JSON" is no longer enough. We need a structured engineering approach.

Layer 1: The System Prompt & Schema Validation

First, stop parsing with standard json.loads(). Use a library that can repair broken JSON or enforce schemas, like json_repair and pydantic.

Always pair your model with a strict Pydantic model:

from pydantic import BaseModel, Field
from typing import Literal

class AgentAction(BaseModel):
    tool_name: Literal["web_search", "database_query"] = Field(description="The name of the tool to use")
    parameters: dict = Field(description="The exact arguments for the tool")

# This ensures we strictly validate the output before feeding it to the next node.

Layer 2: Native Function Calling (Stop Parsing Text)

If you are still prompting models to "output JSON", you are doing it wrong. Use the native tool calling APIs provided by OpenAI, Anthropic, or Google.

Here is how it works with the OpenAI Python SDK:

import openai

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["c", "f"]}
            },
            "required": ["location"]
        }
    }
}]

response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools,
    tool_choice="auto" 
)
# The API guarantees valid JSON structure in response.choices[0].message.tool_calls

Note: Even with native tool calling, models occasionally generate invalid enum values. Always catch the exception and pass it back to the model in a retry loop.

Layer 3: The Retry & "Self-Correction" Loop

When parsing fails, don't crash. Feed the error back to the LLM. This is the most powerful pattern in Agent engineering.

def execute_tool_with_retry(llm_response, max_retries=3):
    for attempt in range(max_retries):
        try:
            # Attempt to parse and validate
            parsed_action = AgentAction.model_validate_json(llm_response)
            return run_tool(parsed_action.tool_name, parsed_action.parameters)

        except Exception as e:
            error_msg = f"Your previous output caused an error: {str(e)}. Please fix the JSON format and try again."
            # Send the error back to the model
            llm_response = get_new_completion(error_msg) 

    raise Exception("Agent failed to generate valid tool calls after multiple attempts.")

The Result

By combining strict Pydantic validation, native Tool Calling, and a feedback retry loop, the Agent's execution reliability jumped from a frustrating 70% to virtually 99%. The key is to never trust the LLM blindly.

Building a reliable AI Agent requires treating the LLM not as a magic brain, but as an unreliable API that needs strict input/output sanitization.

Have you encountered JSON parsing errors in your Agent workflows? How do you handle model hallucinations when calling external tools? Let me know in the comments!

P.S. I'm currently documenting real-world performance and stress tests of various AI Agents on my personal blog. If you're building with LLMs, I'd love to connect and hear what metrics you care about most.

DEV Community