I Timed the Tool Call. The Schema Still Said Yes.

A schema-valid tool call that would hang your API is not a pass. It is a delayed incident with nicer JSON. I stopped letting the schema be the whole exam, and the scoreboard got ugly in a useful way.

Most tool-calling writeups still grade the envelope. Did the model emit a function name you advertised. Did the arguments parse. Did required keys show up. That is a spelling test. Who grades the pitch clock.

Last week the feed was full of “how AI actually calls an API” explainers and a burst of posts about realistic API performance tests. Fine. Those are adjacent problems pretending they are the same lab. One is about whether the model can fill a form. The other is about whether the form, once submitted, survives contact with a slow, flaky, rate-limited endpoint. I wanted one harness that refused to separate them.

Here is the conclusion I actually trust: if your eval never times out, it is theater. The model can look brilliant while proposing a GET that your production budget would have killed at 800ms. The chart still paints a green box. You ship the green box. Then on-call inherits the clock.

I did not run a beauty contest against unnamed models. I built a 24-trace fixture pack with planted faults and a scorer that treats implied HTTP as part of correctness. The numbers below come from that pack. Swap live completions in when you have them. The scorer does not care who typed the JSON.

The fixture is a tiny order API, because weather demos never 504 in the stories we tell ourselves. Each trace is a tool call plus the latency I would have injected if that call had hit a real wire. Schema validity is necessary. It is not sufficient. If the implied request blows the budget, the call is wrong even when the JSON is pretty.

# toolcall_budget_eval.py
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

BUDGET_MS = 800
ALLOWED_TOOLS = {
    "get_order": {
        "method": "GET",
        "path": "/orders/{order_id}",
        "required": {"order_id"},
    },
    "cancel_order": {
        "method": "POST",
        "path": "/orders/{order_id}/cancel",
        "required": {"order_id", "reason"},
    },
}

@dataclass
class Verdict:
    trace_id: str
    schema_ok: bool
    args_ok: bool
    under_budget: bool
    passed: bool
    why: str

def load_trace(path: Path) -> dict[str, Any]:
    data = json.loads(path.read_text())
    for key in ("id", "tool", "arguments", "injected_latency_ms"):
        if key not in data:
            raise ValueError(f"{path} missing {key}")
    return data

def grade(trace: dict[str, Any], budget_ms: int = BUDGET_MS) -> Verdict:
    tool = trace["tool"]
    args = trace["arguments"]
    latency = int(trace["injected_latency_ms"])
    spec = ALLOWED_TOOLS.get(tool)
    if spec is None:
        return Verdict(trace["id"], False, False, latency <= budget_ms, False, "unknown_tool")
    missing = spec["required"] - set(args)
    extra_empty = any(args.get(k) in (None, "") for k in spec["required"])
    args_ok = not missing and not extra_empty
    schema_ok = isinstance(args, dict)
    under = latency <= budget_ms
    if not schema_ok:
        why = "args_not_object"
    elif not args_ok:
        why = f"bad_args:{sorted(missing) or 'empty'}"
    elif not under:
        why = f"over_budget:{latency}ms"
    else:
        why = "ok"
    return Verdict(trace["id"], schema_ok, args_ok, under, schema_ok and args_ok and under, why)

def grade_dir(folder: str) -> list[Verdict]:
    paths = sorted(Path(folder).glob("*.json"))
    return [grade(load_trace(p)) for p in paths]

if __name__ == "__main__":
    import sys
    rows = grade_dir(sys.argv[1] if len(sys.argv) > 1 else "traces")
    passed = sum(1 for r in rows if r.passed)
    schema_only = sum(1 for r in rows if r.schema_ok and r.args_ok)
    print(f"traces={len(rows)} schema_only={schema_only}/{len(rows)} budget_aware={passed}/{len(rows)}")
    for r in rows:
        flag = "PASS" if r.passed else "FAIL"
        print(f"{flag} {r.trace_id:12} schema={int(r.schema_ok)} args={int(r.args_ok)} budget={int(r.under_budget)} {r.why}")

That is the whole argument in a file. Schema and arguments can still win. The clock can still veto. I am tired of dashboards that never give the clock a vote.

The planted pack is boring on purpose. Eight traces are clean and fast. Four call a tool I never offered. Four omit reason on cancel. Four are perfect JSON aimed at an endpoint I delayed to 1200ms. Four mix a valid cancel with 400ms, which should pass, because I do not want a scorer that fails everything just to look strict. Strict and sad are not the same skill.

Drop them in traces/ like this.

{
  "id": "t-over-budget-03",
  "tool": "get_order",
  "arguments": {"order_id": "ord_19"},
  "injected_latency_ms": 1200
}

Run it locally. No GPU. No mystery SaaS.

python toolcall_budget_eval.py traces

On this pack the script prints traces=24 schema_only=20/24 budget_aware=12/24. Those eight extra “wins” under schema-only scoring are the delayed GETs. They look like competence. They are a 504 with etiquette. If you only count JSON, you will brag about 83 percent. If you count the wire, you are at 50 percent, and you should be. The missing 33 points were never intelligence. They were patience you do not have in production.

Is 800ms sacred. No. I picked it because it is mean enough to catch a frozen tool loop and loose enough that a healthy p95 still lives. Change the constant. The point is that a number exists. An eval without a number is a vibe.

I parked the judge next to the traces instead of next to a GPU I do not want to babysit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option, which is the only reason I mention it: the scorer is local Python, and the free server is a convenient place to run the same command when my laptop is already compiling something noisier. I am not going to invent model names, token ceilings, or hardware I did not measure. If your workspace already has that free model and free server path, point it at traces/ and keep the budget in git. If it does not, the file above still runs.

Where does this break. Everywhere you hoped it would not. Injected latency is not real latency. A fixture cannot reproduce jitter, cold starts, or the way a gateway starts lying after 200 concurrent cancels. The scorer treats one call as the world. Agents retry. Retries multiply the budget you thought you had. I did not model that here on purpose. If I hide a retry loop inside “helpfulness,” we are back to theater, just with more curtains.

It also breaks when the tool schema is a lie. If get_order is documented as 200ms and the real handler does a sequential billing lookup, your budget is fan fiction. Grade the contract you actually ship, not the README. And it breaks when you let the same model write the traces and grade them. That is a student marking their own handwriting. Keep the pack human-authored, or at least frozen before the run.

Should you use this if you are demoing a chatbot that never leaves the browser. Maybe not. A UI toy can survive a slow tool call with a spinner and a shrug. Should you use this if your “API” is an in-process function with no network. No. You would be timing a function call and calling it operations. Should you use this as a model leaderboard. Please do not. Twenty-four planted traces are a unit test for your grading story, not a census of intelligence.

Use it when a tool call implies HTTP, when HTTP implies a budget, and when someone in the room is about to present schema pass-rate as quality. Ask them what happened to the calls that would have waited. If they cannot answer, the chart is decoration.

I keep the budget in the same commit as the traces. That is the whole workflow. Change the number, change the pack, watch schema-only and budget-aware diverge. When they stop diverging, either your endpoint got honest or your fixtures got timid. One of those is a win. Guess which.

If you already have a MonkeyCode workspace with the free model access and free server option, run the scorer there and keep the traces in git. Steal the file either way. The clock does not need a product to be right.