Learn Tool-Call Type Drift by Building a Tiny Contract Guard

The first fixture I trusted looked like this:

{"name": "get_hours", "arguments": {"building": "Killam", "weekday": true}}

It parsed. It even survived isinstance(weekday, int), because in Python a bool is a subclass of int. Then my hours table treated True as 1, and the club bot told a study group that Killam was on a Monday schedule. The learning question is small and mean: can a tool call be valid JSON, pass a casual type check, and still be the wrong call?

I was in the Killam basement on a wet Halifax Tuesday, trying to answer one campus question. When does this building close? I did not want a framework. I wanted a Python function and a model that promised to "call" it. That promise is where this case study starts.

Background

Tool calling, in the beginner writeups that keep circulating, looks like a handshake. You publish a schema. The model returns a name plus arguments. You json.loads the blob and splat it into a function. What could go wrong? Plenty, if you treat JSON types as Python types, and Python types as the types you actually meant.

I had already burned other weeks on answer drift and weak oracles. This was a different failure. The model was not inventing library hours. It was inventing the shape of the call. String "2" instead of integer 2. Boolean true instead of a weekday index. An extra key open that my function never declared. Each one looks harmless in a chat log. Each one is a different bug.

The analogy I kept coming back to is a vending machine that accepts any coin-shaped object. A washer will fit the slot. It will not buy you a drink. JSON is the slot. Your function is the drink. Are you checking the metal, or only the diameter?

Goal

I wanted a lab, not a product. One tool. Five fixtures. A guard that prints a table. One place where isinstance quietly betrays you. If the guard could not explain a failure in one line, it was too clever. If it needed pip packages, it was the wrong lab for a student laptop.

Prerequisites were boring on purpose: CPython 3.11 or newer, the standard library, and a terminal. No API key for the checker itself. The checker is local. A model, if you use one, is optional fuel for generating more fixtures. I am labeling the table below as the expected output of this script, not as a benchmark of any hosted model. Do not turn it into a leaderboard. That is how this lesson dies.

Implementation

Here is the whole guard. Copy it to contract_guard.py. Read it once before you run it. Notice that I refuse isinstance for the weekday field on purpose. Python will lie to you if you let it.

#!/usr/bin/env python3
"""Tiny tool-call contract guard. Stdlib only. Python 3.11+."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Any

TOOLS = {
    "get_hours": {
        "required": ["building", "weekday"],
        "types": {"building": str, "weekday": int},
        "constraints": {
            "weekday": lambda v: type(v) is int and 0 <= v <= 6,
            "building": lambda v: type(v) is str and len(v.strip()) > 0,
        },
    }
}

@dataclass
class Verdict:
    ok: bool
    reason: str
    fixture_id: str


def parse_call(raw: str) -> dict[str, Any]:
    data = json.loads(raw)
    if not isinstance(data, dict):
        raise ValueError("root must be an object")
    return data


def check_call(fixture_id: str, raw: str) -> Verdict:
    try:
        data = parse_call(raw)
    except json.JSONDecodeError as exc:
        return Verdict(False, f"not json: {exc}", fixture_id)

    name = data.get("name")
    args = data.get("arguments")
    if name not in TOOLS:
        return Verdict(False, f"unknown tool: {name!r}", fixture_id)
    if not isinstance(args, dict):
        return Verdict(False, "arguments must be an object", fixture_id)

    spec = TOOLS[name]
    missing = [k for k in spec["required"] if k not in args]
    if missing:
        return Verdict(False, f"missing keys: {missing}", fixture_id)

    extra = [k for k in args if k not in spec["types"]]
    if extra:
        return Verdict(False, f"extra keys: {extra}", fixture_id)

    for key, expected in spec["types"].items():
        if key not in args:
            continue
        value = args[key]
        # Exact type. isinstance(True, int) is True in Python; that is the bug.
        if type(value) is not expected:
            return Verdict(
                False,
                f"type drift on {key}: wanted {expected.__name__}, "
                f"got {type(value).__name__}={value!r}",
                fixture_id,
            )
        if not spec["constraints"][key](value):
            return Verdict(False, f"constraint failed on {key}: {value!r}", fixture_id)

    return Verdict(True, "contract ok", fixture_id)


FIXTURES = [
    ("ok-killam-tuesday",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":2}}'),
    ("string-weekday",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":"2"}}'),
    ("bool-weekday",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":true}}'),
    ("extra-open",
     '{"name":"get_hours","arguments":{"building":"Killam","weekday":2,"open":true}}'),
    ("unknown-tool",
     '{"name":"get_open_now","arguments":{"building":"Killam","weekday":2}}'),
]


def main() -> int:
    print("fixture_id\tok\treason")
    failed = 0
    for fid, raw in FIXTURES:
        v = check_call(fid, raw)
        print(f"{v.fixture_id}\t{v.ok}\t{v.reason}")
        if not v.ok:
            failed += 1
    print(f"\n{failed}/{len(FIXTURES)} fixtures failed the contract")
    # Four of five should fail. If they do not, the guard itself drifted.
    return 0 if failed == 4 else 1


if __name__ == "__main__":
    sys.exit(main())

Run it like this:

python3 contract_guard.py

The script is the experiment. It does not call a network. That is the point of a contract guard. You can shame a tool call without spending a token. You can also shame yourself, which is more useful.

When I later wanted fresh fixtures from a real model, I needed a retry lane that was not another paid round trip. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and the free server option as that lane, so the checker stayed the source of truth and the model stayed a fixture factory. If you already have another endpoint, skip the lane. The lesson does not live in the vendor.

Results

The script is written to print this table:

fixture_id            ok     reason
ok-killam-tuesday     True   contract ok
string-weekday        False  type drift on weekday: wanted int, got str='2'
bool-weekday          False  type drift on weekday: wanted int, got bool=True
extra-open            False  extra keys: ['open']
unknown-tool          False  unknown tool: 'get_open_now'

4/5 fixtures failed the contract

Four of five fixtures fail. The interesting row is bool-weekday. JSON true becomes Python True. And True is an int subclass, equal to 1. A Monday-shaped lie. If you do not believe me, run this in a REPL before you argue with the guard:

>>> isinstance(True, int)
True
>>> True == 1
True
>>> {1: "Monday"}[True]
'Monday'

Ask yourself before you keep reading: would your first draft have caught that row? Mine did not. My first draft used isinstance(value, int) because every tutorial does. It printed contract ok for a boolean. The hours function then did SCHEDULE[weekday], and SCHEDULE[True] is SCHEDULE[1]. The bot sounded confident. The calendar was wrong.

That is type drift. Not a hallucination about the library. A hallucination about the type lattice. I also kept a second, worse helper while I was debugging: a "helpful" coercer that ran int(weekday) on everything. int(True) is 1. int("2") is 2. Coercion hides the crime. The guard's job is to refuse the crime.

One more error input, because string-versus-int is the row everyone expects and float-versus-int is the row that sneaks in later:

python3 - <<'PY'
from contract_guard import check_call
print(check_call(
    "float-weekday",
    '{"name":"get_hours","arguments":{"building":"Killam","weekday":2.0}}',
))
PY

Expected verdict: type drift on weekday, wanted int, got float=2.0. JSON numbers are not Python ints just because they look whole. 2.0 is still a float. Would you have coerced it? That is the trap.

What broke, and why

The extra-key fixture is the social version of the same bug. Models love to be helpful. They add "open": true because the user asked if the building is open. Your function never asked for that. If you **kwargs the dict, Python either crashes on an unexpected keyword or, worse, a wrapper swallows it. I want the crash. A silent extra key is how a bot starts answering a question you did not instrument.

The unknown-tool fixture is the identity version. get_open_now is not get_hours. A fuzzy matcher will "fix" this for you. Do not let it. If the model invented a tool, that is a failed call, not a close one. Close is how eval gaming starts, and I already lost time to that habit in another lab.

String weekday is the one everyone predicts. It is also the one JSON makes easy, because models emit numbers as strings constantly. json.loads('{"weekday":"2"}') gives you a str. Your schema said integer. Believe the schema, not the vibe of the transcript.

Common mistakes I made

I treated the chat transcript as evidence the call worked. A pretty JSON blob is not a passing test. I also treated bool as "basically a flag" and forgot that in CPython it is a full citizen of the integer line. I almost added a default: if weekday is missing, use today. Defaults turn missing arguments into plausible lies. The guard should fail closed.

Another mistake: validating only at the JSON Schema dialect I copied from a vendor doc, then assuming CPython would agree. JSON Schema integer does not include booleans. Python isinstance(True, int) does. Those two worlds do not share a court. Your guard has to pick a court and stay there. I picked type(value) is expected. It is rude. Rude is the point.

What you should understand after this

A tool call has three layers, and they fail independently. Layer one is bytes that parse as JSON. Layer two is a name that exists in your tool table. Layer three is argument types and constraints that match the function you will actually invoke. Tutorials stop at layer one. Quiet outages live at layer three. You do not need a framework to see this. You need five fixtures and a mean checker.

You should also understand that free generation does not make free correctness. A model that costs nothing can still emit weekday: true. The cheap part is the retry. The expensive part is believing the retry. Keep the checker on your laptop. Let the model be a factory for bad inputs. That inversion is the whole workflow.

Limitations, and who should skip this

This guard does not talk to a real calendar. It does not stream. It does not implement JSON Schema. It does not handle union types, optional nested objects, or tool results flowing back into a second model turn. If you are shipping an agent with money or email or room bookings on the line, this is a classroom wedge, not a runtime. Side effects need idempotency and auth. This file refuses to pretend it has either.

Do not use this as evidence that one host is more accurate than another. I did not publish accuracy numbers. I published a checker. If your fixtures all pass, you have not tried a boolean yet. Students who need a production agent framework should not start here and then declare victory. Start here, watch one fixture fail, then decide whether you even want a framework.

Extension

Add a sixth fixture that your current guard gets wrong on purpose. Nested arguments. A float weekday 2.0. An empty building string. Predict the verdict. Then run it. If you want a nastier lab, generate twenty model fixtures and keep only the ones the guard rejects. The rejected set is the curriculum. The accepted set is just a mood.

Which of the five rows would your code have rubber-stamped? Tell me the fixture id and the one-line reason. A counterexample is more useful than a compliment.