Opinion: Your Agent's Lock-In Is the Tool Schema, Not the Model

Model swaps are cheap now. Tool schema swaps are not. That asymmetry is where your lock-in actually lives.

Everyone benchmarks the model. Almost nobody benchmarks the tool contract. I think that gap is the most expensive blind spot in agent work right now.

The model swaps in an afternoon

Wrapping a new endpoint takes one adapter file. You change a base URL and a model id. Your prompts survive. Your eval set survives.

Tool calling is different. Every endpoint has opinions about argument shape. Some honor additionalProperties: false. Some quietly ignore it. Some invent a third call you never declared.

Those differences do not show up in a chat transcript. They show up as a 400 from your executor at 2am. Or worse, a silent write to the wrong path.

So my position is blunt: treat tool-call conformance as a hard gate. Not a vibe check. A gate with exit codes.

Six cases that expose a weak tool-caller

You do not need a leaderboard. You need five cases and a pass threshold. Here is what I test first.

1. No-tool prompt. Ask a plain question with tools declared. A good caller makes zero calls.
2. Missing tool. Ask for an action whose tool you did not declare. A good caller refuses, not invents.
3. Required argument. Ask for a file read. The call must contain path as a string.
4. Extra properties. Explicitly ask for a mode you did not define. The call must stay inside the schema.
5. Multiple targets. Ask for two files. Watch whether it batches, splits, or drops one.
6. Repeat runs. Temperature 0 is not determinism. Every case runs N times.

Case 2 is the one people skip. It is also the one that breaks production. A hallucinated write_file has no schema guard in most loops.

The harness: one file, no framework

Save this as tool_contract.py. It talks to any OpenAI-compatible /chat/completions endpoint. Label: runnable, but unexecuted against your endpoint until you run it.

# tool_contract.py - endpoint conformance for tool-calling agents.
import json, os, sys, time
import httpx

BASE  = os.environ["LLM_BASE_URL"].rstrip("/")
KEY   = os.environ["LLM_API_KEY"]
MODEL = os.environ["LLM_MODEL"]
RUNS  = int(os.environ.get("RUNS", "5"))

READ_FILE = {
    "type": "function",
    "function": {
        "name": "read_file",
        "description": "Read a UTF-8 text file from the repo worktree.",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"],
            "additionalProperties": False,
        },
    },
}

def call(messages, tools):
    body = {"model": MODEL, "messages": messages,
            "tools": tools, "temperature": 0}
    t0 = time.perf_counter()
    r = httpx.post(f"{BASE}/chat/completions",
                   headers={"Authorization": f"Bearer {KEY}"},
                   json=body, timeout=90)
    r.raise_for_status()
    data = r.json()
    data["_ms"] = int((time.perf_counter() - t0) * 1000)
    return data

def calls(data):
    return data["choices"][0]["message"].get("tool_calls") or []

def names(data):
    return [c["function"]["name"] for c in calls(data)]

def args(data, i=0):
    return json.loads(calls(data)[i]["function"]["arguments"])

Now the cases. Each returns (ok, note) so the runner can print a table.

CASES = []

def case(fn):
    CASES.append(fn)
    return fn

@case
def no_tool_needed():
    d = call([{"role": "user",
               "content": "What is 17 * 3? Answer with the number only."}],
             [READ_FILE])
    return (not calls(d)), f"calls={names(d)}"

@case
def unknown_tool_not_invented():
    d = call([{"role": "user",
               "content": "Create notes/todo.md containing 'ship'."}],
             [READ_FILE])
    ok = all(n == "read_file" for n in names(d))
    return ok, f"calls={names(d)}"

@case
def required_arg_present():
    d = call([{"role": "user",
               "content": "Open src/app.py and show its contents."}],
             [READ_FILE])
    if names(d) != ["read_file"]:
        return False, f"got {names(d)}"
    a = args(d)
    return (set(a) == {"path"} and isinstance(a["path"], str)), f"args={a}"

@case
def no_extra_properties():
    d = call([{"role": "user",
               "content": "Read config/settings.yaml as UTF-8 via read_file."}],
             [READ_FILE])
    if not calls(d):
        return False, "no call"
    a = args(d)
    return (set(a) <= {"path"}), f"extra={sorted(set(a) - {'path'})}"

@case
def two_files_two_calls():
    d = call([{"role": "user",
               "content": "Read both src/a.py and src/b.py."}],
             [READ_FILE])
    paths = [args(d, i).get("path") for i in range(len(calls(d)))]
    return (len(paths) == 2), f"paths={paths}"

And the runner. Non-zero exit on any incomplete gate. Wire that into CI and you have a lock.

def main():
    bad = 0
    print("| case | pass | notes |")
    print("|---|---|---|")
    for fn in CASES:
        passed, note = 0, ""
        for _ in range(RUNS):
            try:
                ok, note = fn()
            except Exception as exc:
                ok, note = False, f"{type(exc).__name__}: {exc}"
            passed += int(bool(ok))
        bad += int(passed != RUNS)
        print(f"| {fn.__name__} | {passed}/{RUNS} | {note} |")
    sys.exit(1 if bad else 0)

if __name__ == "__main__":
    main()

Run it like this:

export LLM_BASE_URL="https://<your-endpoint>/v1"
export LLM_API_KEY="<key>"
export LLM_MODEL="<model-id>"
export RUNS=20
python tool_contract.py | tee runs/$(date +%F)-contract.md

Commit runs/. A diff in that table is a real regression. A chat transcript is not.

Reading the table: gates, not scores

Do not compute an average. Averages hide the case that breaks you. Use per-case gates instead.

Case	What it catches	Gate
no_tool_needed	wasted calls, added latency, cost	20/20
unknown_tool_not_invented	hallucinated writes, missing tool errors	20/20
required_arg_present	empty or renamed arguments	20/20
no_extra_properties	schema drift, strict-server 400s	≥18/20
two_files_two_calls	dropped targets, silent partial work	note only

A single failed hard gate means you do not wire that endpoint into your loop. Not "with a retry". Not "with a better prompt". Skip it.

The last row stays advisory on purpose. Batching is a policy choice. Two calls and one call can both be correct here.

Where a free tier actually fits

This is where free access earns its place. Not as your production path. As the rig you can afford to run at 20 iterations per case.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's own outreach for this piece states two things: free model access, and a free server option. It also describes a free allocation of 10 million tokens. Treat all three as vendor statements you should confirm in the project's current docs before relying on them.

Why it fits this workflow:

• The conformance rig wants a small box. A runner plus runs/ is a few hundred lines. It does not need a GPU farm.
• The matrix wants repetition. Five cases at 20 runs is 100 requests. Free access makes that a habit, not a budget review.
• A separate host keeps keys off laptops. The free server holds the endpoint credentials, CI holds a trigger.

Rough sizing, as an estimate only: each request here is a few hundred tokens. The full matrix lands well under a million tokens. That is the point of a gate suite. It is cheap and repetitive.

I would not run this rig on the same host as anything that matters. Keep it disposable. Keep the runs/ output in git.

What this does not prove

Be honest about the boundary of the artifact.

• It does not measure answer quality. Five cases test contract behavior, not reasoning.
• It does not test your executor. Schema-valid calls can still be applied wrongly downstream.
• It does not survive schema changes. Update the cases when you change tools.
• It assumes OpenAI-style tool calling. Different protocols need different cases.
• Free tiers change. Quotas, limits, and terms are not permanent. Re-verify before you depend on them.

Also: temperature 0 reduces variance. It does not remove it. That is why the gates are per-case counts.

Who should skip this

Be clear about fit. This approach is wrong for some teams.

• You need contractual uptime. A free tier has no SLA you can hand to a customer.
• You handle regulated data. Do not send real payloads to a shared free endpoint.
• You have one tool and never swap models. The suite costs more than the risk.
• You need residency guarantees. Free hosting rarely gives you the region you need.

If any of those apply, keep the test cases. Drop the free tier. The cases outlive the endpoint anyway.

The part that survives

The model you pick today is a line in a config file. The tool schema is a contract your whole loop depends on. Test the contract before you test the model.

Start small. One file, five cases, twenty runs. If you want a zero-cost place to try that first pass, the free tier and free server option are worth a look — terms change, so check them yourself.