Schema Drift Is the Real Agent Bug: A Free-Model Contract Test

Every agent bug I've chased lately looked like a reasoning failure. It wasn't. The model made a valid decision, then emitted an argument with quantity: "5" when the function expected quantity: 5. A type error killed the call. The root cause was schema drift: the tool contract changed, but the prompt and test suite didn't.

Tool calls are external input. Treat them like you treat any API payload: validate against a schema before touching business logic. That validation has to run continuously, on prompts you didn't hand-craft. This is where free models and a free server become practical infrastructure, not just a demo toy.

MonkeyCode offers free model access and a free server option, which fits a nightly contract-test pipeline without stealing your GPU or wallet. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Contract: One JSON Schema Per Tool

The first step is defining what "valid" means. Here is a minimal schema for a product-search tool:

{
  "type": "object",
  "properties": {
    "query": { "type": "string", "minLength": 1 },
    "max_price": { "type": "number", "minimum": 0 },
    "in_stock_only": { "type": "boolean" }
  },
  "required": ["query"],
  "additionalProperties": false
}

This schema is the contract. The model may be great at reasoning; if the arguments fail this schema, the tool call is invalid. No fallback, no silent coercion. Fail loudly.

Generate Adversarial Prompts with a Free Model

Hand-writing test prompts gives you confirmation bias. You write prompts that match the schema because you already know the schema. The trick is to use a free model to generate a wide variety of prompts that should trigger this tool, including ambiguous, terse, or oddly phrased ones.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["FREE_MODEL_KEY"],
    base_url=os.environ["FREE_MODEL_BASE"],
)

schema_excerpt = """
Tool: search_products
Parameters:
- query: string, required
- max_price: number, optional, must be >= 0
- in_stock_only: boolean, optional
"""

prompt = f"""
You are a test-data generator. Generate 20 user queries that might invoke the tool below.
Vary intent: some queries should be short, some have price limits, some are questions.

{schema_excerpt}

Output each query on its own line, no numbering.
"""

resp = client.chat.completions.create(
    model=os.environ["FREE_MODEL_NAME"],
    messages=[{"role": "user", "content": prompt}],
    temperature=0.9,
)

queries = resp.choices[0].message.content.strip().splitlines()
print(f"generated {len(queries)} prompts")
with open("generated_prompts.txt", "w") as f:
    f.write("\n".join(queries))

The free model does not need to be perfect. It just needs to produce enough linguistic variety to catch schema edge cases.

Validate Tool Calls Against the Schema

Now you have two roles in the pipeline:

1. A generator model that invents the user prompts.
2. An agent model that must turn each prompt into a tool call.

Both can be free models. The generator shakes the tree; the agent tries to stay on the contract.

import json
import jsonschema
from openai import OpenAI

agent = OpenAI(
    api_key=os.environ["FREE_MODEL_KEY"],
    base_url=os.environ["FREE_MODEL_BASE"],
)

SCHEMA = json.load(open("tool_schema.json"))

def validate_tool_call(query: str):
    resp = agent.chat.completions.create(
        model=os.environ["FREE_MODEL_NAME"],
        messages=[
            {"role": "system", "content": "You call tools with strict JSON arguments."},
            {"role": "user", "content": f"search_products for: {query}"},
        ],
        temperature=0,
    )
    raw = resp.choices[0].message.content
    try:
        args = json.loads(raw)
    except json.JSONDecodeError:
        return False, "malformed-json"
    try:
        jsonschema.validate(args, SCHEMA)
        return True, "ok"
    except jsonschema.ValidationError as e:
        return False, e.message

Run this across every generated query. Collect the failures.

Schedule It on a Free Server

The pipeline should run nightly, not when you remember. A free server gives you a neutral execution environment: no laptop battery, no shared GPU, no accidental local state. Use a cron workflow in CI.

name: tool-contract-nightly
on:
  schedule:
    - cron: "0 2 * * *"
  workflow_dispatch:

jobs:
  contract:
    runs-on: ubuntu-latest
    env:
      FREE_MODEL_KEY: ${{ secrets.FREE_MODEL_KEY }}
      FREE_MODEL_BASE: ${{ secrets.FREE_MODEL_BASE }}
      FREE_MODEL_NAME: ${{ secrets.FREE_MODEL_NAME }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install openai jsonschema
      - run: python generate_prompts.py
      - run: python validate_tool_calls.py
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: contract-failures
          path: failure_report.json

When a run fails, read the report. Common failures are:

• String where a number is required
• Extra property from a leftover prompt instruction
• Missing required field because the system prompt drifted

Where to Run Each Stage: A Decision Table

Stage	Local machine	Free server	Why
Prompt generation	Maybe	Recommended	Cheap to run, high volume
Agent tool-call validation	Maybe	Recommended	No GPU contention, reproducible image
Private/prompt source code	Required	Avoid	Confidentiality risk
Debugging a specific failure	Recommended	Optional	Needs interactive repl speed

Limitations and Who Should Skip This

This pipeline does not prove your agent will behave in production. It only proves the tool-call arguments respect the schema on generated prompts. Free models have rate limits and their output may be cached or filtered; do not rely on them for strict private-data workloads.

If your tool surface is two functions and your prompts are static, you likely do not need this setup. A single validator in the service boundary is enough. If you change schemas weekly and debug agent failures in production, start the nightly contract test.

Schema drift is boring. It will not make a dramatic demo. But it is the bug that eats your weekend, and a few lines of validation plus a scheduled run can push it back to where it belongs. Try the free model and the free server for this exact workflow; the only cost is the time it takes to read one failure report.

Comments

[Luis Cruz]

Your approach to treating tool calls as API payloads is a crucial insight that can significantly enhance the reliability of AI interactions. Implementing continuous schema validation, especially using generated prompts, is a smart way to uncover edge cases that hand-crafted tests might miss. One idea could be to integrate logging mechanisms to track common schema violations, which might help in iterating and refining your model further. If you're exploring enhancements to the validation process or need an extra hand with the implementation, I’d be glad to discuss a paid collaboration. How do you envision scaling this validation framework as your toolset evolves?