DEV Community

MUHAMMAD MUSADIQ
MUHAMMAD MUSADIQ

Posted on

Designing Robust Schemas and System Prompts to Prevent Agent Function Hallucinations

When building autonomous AI agents, few issues are as frustrating as function hallucination. You ask an agent to look up a user account, and instead of passing a standard UUID, it invents a nonexistent parameter like user_search_string or passes the literal string "null" for a required integer ID. These subtle failures break downstream API contracts, trigger unexpected runtime exceptions, and undermine production reliability. To build dependable AI systems, developers must move past naive prompt engineering and implement strict schema enforcement alongside defensive system prompt design.

In this tutorial, you will learn how to combine strict typing, detailed JSON schemas, and structured system prompts to minimize tool hallucinations in your LLM agents.

Function hallucinations typically happen for three reasons:

  1. Ambiguous Tool Schemas: The schema lacks explicit field descriptions, valid ranges, or enum constraints, forcing the LLM to guess valid parameter formats.
  2. Over-eager System Instructions: The system prompt instructs the model to "always try to use a tool," causing it to make up arguments when required information is missing.
  3. Unenforced Output Formats: The API call relies on standard text generation instead of native structured outputs or strict JSON schema bounds.

By combining rigid Pydantic models with defensive prompt engineering, you can create a dual-layer strategy that forces the LLM to either provide valid parameters or fail gracefully.

Step 1: Define Precise Tool Calling Schemas with Pydantic

A weak tool calling schema relies on basic types like string or dict without boundary conditions. A robust schema uses Pydantic to define clear descriptions, narrow value types using Python Enums, and enforce regex patterns or strict numeric ranges.

Here is a comparison between a weak tool schema and a robust, hallucination-resistant Pydantic schema:

from enum import Enum
from typing import Optional
from pydantic import BaseModel, Field, ConfigDict

# --- Weak Schema (Prone to Hallucinations) ---
class BadTransferFunds(BaseModel):
    account_id: str
    amount: float
    currency: str


# --- Robust Schema (Hallucination-Resistant) ---
class CurrencyEnum(str, Enum):
    USD = "USD"
    EUR = "EUR"
    GBP = "GBP"

class RobustTransferFunds(BaseModel):
    model_config = ConfigDict(extra="forbid") # Prevents LLM from inventing new parameters

    source_account_id: str = Field(
        ...,
        pattern=r"^ACC-\d{6}$",
        description="The unique source account identifier. MUST strictly match format 'ACC-XXXXXX' where X is a digit.",
        examples=["ACC-123456"]
    )
    destination_account_id: str = Field(
        ...,
        pattern=r"^ACC-\d{6}$",
        description="The unique destination account identifier. MUST strictly match format 'ACC-XXXXXX' where X is a digit.",
        examples=["ACC-654321"]
    )
    amount: float = Field(
        ...,
        gt=0.0,
        description="The monetary value to transfer. MUST be a positive number greater than 0. Do not guess."
    )
    currency: CurrencyEnum = Field(
        ...,
        description="The 3-letter ISO code for the transaction currency. MUST be one of the explicitly listed enum options."
    )
    memo: Optional[str] = Field(
        default=None,
        max_length=100,
        description="Optional brief note for the transfer. Pass null if not explicitly provided by the user."
    )
Enter fullscreen mode Exit fullscreen mode

Key Elements Implemented:

  • extra="forbid": Disallows any fields invented by the model that are not explicitly defined in the class.
  • pattern: Uses standard regex rules to validate string formats (e.g., account IDs).
  • Enum: Restricts arbitrary string inputs to strict, approved options.
  • description: Gives the LLM explicit instructions on parameter formatting directly inside the schema context.

Step 2: Convert Pydantic Models to JSON Schema for the LLM

LLM providers expect tool definitions formatted as JSON Schema. When using OpenAI or compatible APIs, extracting the JSON schema LLM configuration from your Pydantic models enables native structured outputs.

import json
from typing import Dict, Any

def get_tool_definition(pydantic_model: type[BaseModel], name: str, description: str) -> Dict[str, Any]:
    """Generates a strict tool definition compatible with OpenAI function calling API."""
    schema = pydantic_model.model_json_schema()

    return {
        "type": "function",
        "function": {
            "name": name,
            "description": description,
            "parameters": {
                "type": "object",
                "properties": schema.get("properties", {}),
                "required": schema.get("required", []),
                "additionalProperties": False  # Enforces strict schema matching
            },
            "strict": True  # Enables native structured outputs enforcement where supported
        }
    }

# Example Usage
transfer_tool = get_tool_definition(
    RobustTransferFunds, 
    name="transfer_funds", 
    description="Transfer funds securely between two internal bank accounts."
)

print(json.dumps(transfer_tool, indent=2))
Enter fullscreen mode Exit fullscreen mode

Setting additionalProperties: False combined with strict: True forces the API to apply exact JSON schema parsing rules, preventing the generation of arbitrary key-value pairs during function execution.

Step 3: Write Anti-Hallucination System Prompts

Schemas define what parameters are allowed, but system prompts define when and how tools should be invoked. An effective system prompt must explicitly handle missing parameters and prohibit assumptions.

Here is a system prompt template designed specifically to curb function calling hallucinations:

You are an AI financial assistant with access to system tools.

CORE OPERATIONAL RULES:
1. TOOL USE MANDATE: Only invoke a tool when you have extracted ALL required parameters directly from the user's explicit input.
2. NO GUESSING OR INFERENCE: Never invent, estimate, or extrapolate required parameters (e.g., account numbers, amounts, currency types). 
3. MISSING PARAMETERS PROTOCOL: If a user request is missing required parameters for a tool call:
   - DO NOT call the tool.
   - Reply directly to the user asking specifically for the missing details.
4. STRICT SCHEMA ADHERENCE: All parameters passed to tools MUST conform to the format constraints (regex patterns, allowed enum values) provided in the tool schema.
5. FALLBACK: If you are uncertain about any tool argument, ask the user for clarification instead of attempting the call.
Enter fullscreen mode Exit fullscreen mode

Step 4: Execute and Validate (End-to-End Implementation)

Now, let's wire up the Pydantic schema, system prompt, and LLM call into a cohesive Python workflow. This example uses Python's native error handling to catch validation failures before executing backend functions.

import os
from openai import OpenAI
from pydantic import ValidationError

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "your-key-here"))

# Define actual execution function
def execute_transfer(data: RobustTransferFunds) -> str:
    # Safe to execute because data is guaranteed valid by Pydantic
    return f"Successfully transferred ${data.amount} {data.currency.value} from {data.source_account_id} to {data.destination_account_id}."

def process_user_request(user_input: str):
    system_prompt = """
    You are an AI assistant. You have access to the 'transfer_funds' tool.
    Do NOT call the tool if mandatory information (source, destination, amount, currency) is missing.
    Ask the user for clarification instead.
    """

    tools = [
        get_tool_definition(
            RobustTransferFunds, 
            name="transfer_funds", 
            description="Transfer funds strictly between two validated accounts."
        )
    ]

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_input}
        ],
        tools=tools,
        tool_choice="auto"
    )

    message = response.choices[0].message

    # Check if model wants to call a tool
    if message.tool_calls:
        for tool_call in message.tool_calls:
            if tool_call.function.name == "transfer_funds":
                raw_args = tool_call.function.arguments
                print(f"[LOG] Raw Tool Arguments Received: {raw_args}")

                try:
                    # Defensive parsing through Pydantic
                    validated_args = RobustTransferFunds.model_validate_json(raw_args)
                    result = execute_transfer(validated_args)
                    print(f"[SUCCESS] {result}")
                except ValidationError as e:
                    print(f"[ERROR] Tool call rejected due to schema mismatch:\n{e}")
                    # Handle graceful retry or pass error context back to model
    else:
        print(f"[ASSISTANT]: {message.content}")

# --- Test Case 1: Incomplete Input (Triggers Direct Response, No Tool Call) ---
print("--- Test Case 1: Incomplete Input ---")
process_user_request("Please send $500 to account ACC-654321.")

# --- Test Case 2: Complete Input (Triggers Tool Call) ---
print("\n--- Test Case 2: Valid Complete Input ---")
process_user_request("Transfer 500 USD from ACC-123456 to ACC-654321 for consulting fees.")
Enter fullscreen mode Exit fullscreen mode

Best Practices Summary

To maintain zero-hallucination agent architectures, apply these core rules across your application layer:

Mechanism Purpose Implementation Strategy
Strict Pydantic Constraints Validates structural and logical type invariants Use Field(pattern=...), Enum, and set extra="forbid".
JSON Schema LLM Bindings Restricts payload parameters at the model API level Set additionalProperties: False and strict: True in OpenAI function definitions.
Defensive Prompting Prevents speculative tool execution when data is lacking Add negative rules like "DO NOT invoke tool X if argument Y is missing."
Runtime Model Validation Acts as a circuit breaker before backend API calls Re-parse raw LLM tool arguments using .model_validate_json() in your application logic.

By combining rigid tool calling schemas, system-level operational bounds, and client-side Pydantic validation, you convert unpredictable LLM tool execution into a deterministic API layer built for production workloads.

Looking for a production-ready solution? Check out Gaper deploy AI agents that integrate with your real workflows, from support to finance to sales automation.

Top comments (0)