I spent forty-eight hours celebrating a green pytest run that never touched a real model. The mock returned a two-key payload every time, and my handler treated that shape as law. Then I flipped one environment variable and watched a live completion invent a third argument. Have you ever trusted a patch so much that production had to teach you the schema?
What I thought I had covered
I was wiring a tiny weather tool into an agent that already logged retries, timeouts, and HTTP status codes. The client used an OpenAI-compatible tools array, and the unit tests patched the chat completion call. That patch always returned get_weather with city and units, so the parser never saw extra keys. Why would I spend money on a live model during a refactor that already felt complete?
The fixture looked boring on the first pass, which is exactly how fixtures hide design bugs. I copied the payload from my own happy-path notes and never asked a model to argue. I also asserted the absence of a key the mock could not invent, which is a special kind of theater.
# tests/test_weather_agent.py — example mock, not a live run
from types import SimpleNamespace
from unittest.mock import patch
def make_response(arguments: str):
return SimpleNamespace(
choices=[
SimpleNamespace(
message=SimpleNamespace(
tool_calls=[
SimpleNamespace(
id="call_test_1",
function=SimpleNamespace(
name="get_weather",
arguments=arguments,
),
)
]
)
)
]
)
@patch("weather_agent.client.chat.completions.create")
def test_agent_calls_weather_tool(mock_create):
mock_create.return_value = make_response(
'{"city": "Oslo", "units": "metric"}'
)
result = run_agent("Weather in Oslo?")
assert result["city"] == "Oslo"
assert "lang" not in result
That last assertion never ran against a model; it only ran against my own optimism. The suite stayed green through a retry refactor, a logging cleanup, and a tiny JSON helper. I shipped confidence, not coverage, and the difference showed up only after the patch was gone.
Hour 8: the live call looked successful
I finally pointed OPENAI_BASE_URL at a live chat endpoint and kept the same tool schema. The HTTP layer returned 200, and the first tool name still said get_weather without complaint. json.loads on arguments did not raise, so the tiny dashboard I hacked together printed ok. Then I logged sorted argument keys and stared at city, lang, and units in one list.
Who asked for lang in a schema that only documented city and units for the handler? Nobody in my code did. The model did, because the description said short summary and helpful models fill in missing context. I had treated HTTP success as contract success, which is a habit I need to break.
Here is the schema I thought was already a contract.
WEATHER_TOOL = {
"type": "function",
"function": {
"name": "get_weather",
"description": "Return a short weather summary for a city.",
"parameters": {
"type": "object",
"additionalProperties": False,
"properties": {
"city": {"type": "string"},
"units": {"type": "string", "enum": ["metric", "imperial"]},
},
"required": ["city", "units"],
},
},
}
I had set additionalProperties to false in my head, not in the first request I actually sent. The live model did what models do when a description says short summary: it tried to help. Helpful, in this case, meant inventing lang as if my backend had a translation switch. Does your client reject unknown keys, or does it pass star-star args into a function?
Hour 19: the arguments were a string, then a dict, then markdown
The next surprise was not a new key sitting in the object; it was a new wrapper around it. One completion returned arguments as a JSON object that the Python SDK had already parsed. The following completion returned a string that began with a markdown json fence marker. I wrote a decoder that called json.loads twice, felt clever, then hit a nested input object.
How many one-off adapters were you planning to maintain across SDKs, servers, and prompt edits? I started a scratch log on disk so I could diff shapes without rerunning the whole agent. I wanted a counter of key tuples, not another opinion about whether the model was being creative.
Commands I used to dump shapes
I installed the two libraries the example harness imports, then refused to add a third helper.
python -m pip install openai jsonschema
export OPENAI_BASE_URL="${OPENAI_BASE_URL:?set me}"
export OPENAI_API_KEY="${OPENAI_API_KEY:?set me}"
# Optional: export OPENAI_MODEL to whatever alias your endpoint expects.
python contract_tool_calls.py --prompt "Weather in Oslo?" --dump /tmp/tool-shapes.jsonl
python - <<'PY'
import json
from collections import Counter
keys = Counter()
with open("/tmp/tool-shapes.jsonl", encoding="utf-8") as handle:
for line in handle:
rec = json.loads(line)
keys[tuple(rec.get("argument_keys") or [])] += 1
print(keys)
PY
The counter printed two tuples I expected and one I did not recognize from the mock. That is the whole incident in miniature: the mock had a cardinality of exactly one. Live traffic had a long tail, and my tests had never sampled it.
The contract harness I wish I had at hour one
I needed a check that failed when a live model drifted from the schema, even with green pytest. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I ran the harness on MonkeyCode's free model access and used the free server option as a scratch box. I did that so I was not burning a paid production key on schema experiments. I am not naming models, quotas, or hardware here, and I did not collect benchmarks.
The idea is deliberately small and boring enough to rerun after every prompt change I make. Send one prompt, record the tool call, and assert the JSON Schema, the name, and missing fences. If the live payload cannot pass this, my agent should not splat it into Python.
# contract_tool_calls.py
"""Example harness: live tool-call contract check. Not a benchmark."""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
from typing import Any
import jsonschema
from openai import OpenAI
FENCE_RE = re.compile(r"^```
(?:json)?\s*|\s*
```$", re.I)
WEATHER_TOOL = {
"type": "function",
"function": {
"name": "get_weather",
"description": "Return a short weather summary for a city.",
"parameters": {
"type": "object",
"additionalProperties": False,
"properties": {
"city": {"type": "string", "minLength": 1},
"units": {"type": "string", "enum": ["metric", "imperial"]},
},
"required": ["city", "units"],
},
},
}
def strip_fences(raw: str) -> str:
return FENCE_RE.sub("", raw.strip())
def decode_arguments(raw: Any) -> dict[str, Any]:
if isinstance(raw, dict):
return raw
if not isinstance(raw, str):
raise TypeError(f"arguments must be str or dict, got {type(raw)!r}")
payload = json.loads(strip_fences(raw))
if (
isinstance(payload, dict)
and set(payload) == {"input"}
and isinstance(payload["input"], dict)
):
payload = payload["input"]
if not isinstance(payload, dict):
raise TypeError("decoded arguments were not an object")
return payload
def check_call(message: Any) -> dict[str, Any]:
calls = message.tool_calls or []
if len(calls) != 1:
raise AssertionError(f"expected exactly one tool call, got {len(calls)}")
call = calls[0]
name = call.function.name
if name != "get_weather":
raise AssertionError(f"unexpected tool name: {name!r}")
args = decode_arguments(call.function.arguments)
jsonschema.validate(args, WEATHER_TOOL["function"]["parameters"])
return {"name": name, "argument_keys": sorted(args), "args": args}
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument(
"--prompt",
default="What is the weather in Oslo? Use metric units.",
)
parser.add_argument("--dump")
args = parser.parse_args()
client = OpenAI(
base_url=os.environ.get("OPENAI_BASE_URL"),
api_key=os.environ["OPENAI_API_KEY"],
)
response = client.chat.completions.create(
model=os.environ.get("OPENAI_MODEL", "local-model"),
messages=[{"role": "user", "content": args.prompt}],
tools=[WEATHER_TOOL],
tool_choice={
"type": "function",
"function": {"name": "get_weather"},
},
)
record = check_call(response.choices[0].message)
line = json.dumps(record, sort_keys=True)
print(line)
if args.dump:
with open(args.dump, "a", encoding="utf-8") as handle:
handle.write(line + "\n")
return 0
if __name__ == "__main__":
sys.exit(main())
I am labeling this as an example harness, not as a production evaluation suite for agents. You still need retries, timeouts, and a stub backend that matches your own tool side effects. A green contract check means the shape held for this prompt, not that the answer is true.
The failures I actually hit
I kept a running list on a sticky note, then copied each failure into the JSONL log. The names were ordinary. The combinations were not.
- Extra keys such as
lang,country, orformatappeared even when the prompt was short. -
argumentsarrived as a dict from one SDK path and as a string from another path. - Markdown fences wrapped valid JSON, so a naive
json.loadsfailed on the first character. - A nested
{"input": {...}}object passedjson.loadsand then blew up my function signature. -
tool_choicereduced invented names, but it did not stop invented properties inside the chosen tool.
Would a stricter JSON Schema have saved hour nineteen, or would it only have made the failure louder? Loud is what I wanted. Silent **args is how a translation key becomes a TypeError in a worker you are not watching.
Decision table I now keep next to the agent
I needed a single page I could glance at before calling a mock sufficient. The table is not clever. It is a refusal to confuse transport success with schema success.
| Check | Mocked pytest | Live completion | What I do on fail |
|---|---|---|---|
| HTTP 200 | skipped | required | stop, dump headers |
| Tool name equals schema | hardcoded | asserted | treat as model drift |
arguments decodes to object |
hardcoded string | asserted | run strip_fences once |
| Extra keys | never seen | additionalProperties: false |
do not splat **args
|
| Required keys present | hardcoded | asserted | retry once, then fail |
Nested input wrapper |
never seen | unwrap only if documented | log and fail closed |
If a row only passes under a mock, I no longer call that row covered in the contract. That sounds obvious this morning after the incident, but it was not obvious at hour six. Green tests had been answering a narrower question than the one I was asking out loud.
What I would repeat
I would pin the schema in git and run the live harness on every prompt edit, even when green. I would refuse to splat decoded arguments into a Python function without an explicit allowlist. I would dump JSONL shapes before I dump opinions in Slack about the model being broken. I would keep paid production keys off the scratch loop so experiments stay cheap and contained.
Would I still write mocks for control flow, retries, and timeout math in the agent loop? Yes, I would keep those mocks, but I would not let them define the tool contract anymore. Mocks are good at branches. They are bad at imagination.
Limitations, and who should skip this
This harness does not measure quality, latency, or cost, and it will not verify weather answers. Free model access can differ from the model you ship, so a green live check is not certification. additionalProperties: false is ignored by some runtimes unless the server actually enforces JSON Schema. Streaming completions can split arguments across chunks, and this script does not assemble those deltas.
Skip this approach if you handle regulated data that must not leave your own network boundary. Skip it if you need a contractual SLA, a named model pin, or production-identical networking. Skip it if your tools have side effects; a live contract check should hit a stub backend. Do not point this at a tool that creates tickets, sends mail, or charges cards.
I also would not use a single prompt as an evaluation of the long tail of tool calling. One invented lang key taught me the shape problem, and it did not teach me rare failures. If you need ranking, build a real eval set. This field note is only about refusing silent extra keys.
What broke, and what I am keeping
pytest was never wrong about the code I had written around a fake payload from a patch. It was wrong about the world the agent would meet once a live model chose the arguments. The live model was not malicious; it completed a helpful sentence that my schema failed to forbid. After forty-eight hours I have a decoder, a schema assert, and a JSONL dump I can diff.
That is enough to start the next prompt change without lying to myself about coverage. I am not asking you to throw away mocks; I am asking you to stop treating them as the schema. If you already have an OpenAI-compatible endpoint, run the harness twice and compare the key tuples. I would love to hear which extra argument showed up first in your own JSONL log.
Top comments (0)