Your Agent Called the Tool. The Contract Still Failed

A tool call is not a passing test. I refuse that substitution on every trace. Did the payload match a contract you actually own?

Most demos celebrate the call itself. I watch the next turn swallow raw JSON. Then someone ships the session as evidence.

This catalog is about the tool boundary. Not the prompt pack. Not the CI oracle.

The rule I use

If a tool result is unchecked, the run is theater. I want a fixture, an envelope, and a replay. Cheap runs still need that triangle.

Ask a harder question after every call. What field would falsify this turn?

Anti-pattern 1: Status-code theater

Transport success is not semantic success. Agents still treat them as twins. Why do we keep buying that story?

Symptoms

• The trace shows 200 and a victory line.
• The agent claims the file was written.
• Nobody parsed the body before the next tool.
• Exit code 0 becomes a patch summary.

Root cause

HTTP status and process exit are transport. Meaning lives in the envelope. The loop never named that envelope.

Replacement

Assert a tiny contract before the next thought. Status, error, and typed result. Fail closed on unknown shapes.

# labeled example: envelope check, not a live service
REQUIRED = ("ok", "error", "result")

def accept_tool_body(body: dict) -> dict:
    missing = [k for k in REQUIRED if k not in body]
    if missing:
        raise ValueError(f"missing {missing}")
    if body["ok"] is True and body["error"] is not None:
        raise ValueError("ok/error conflict")
    if body["ok"] is False and body["result"] is not None:
        raise ValueError("failed call still carried result")
    return body["result"]

I run that check in the harness. The model never sees a raw 200.

Anti-pattern 2: Raw dump into the next turn

This is not the app schema bug. This is the tool envelope bug. Huge difference. Did your last article already cover database fields? This one covers the wire.

Symptoms

• The next prompt is a wall of stdout.
• The model summarizes noise as facts.
• Later tools receive invented ids.
• Logs look rich and still prove nothing.

Root cause

The loop concatenates because concatenation is easy. Easy context is not true context. Unknown keys become future arguments.

Replacement

Normalize to a struct. Drop unknown keys. Hash what remains. Pass only fields the next tool may legally consume.

# labeled example: allowlist, then hash
import hashlib, json

ALLOW = {"path", "sha256", "bytes"}

def project_result(result: dict) -> dict:
    slim = {k: result[k] for k in ALLOW if k in result}
    if set(slim) != ALLOW:
        raise ValueError(f"incomplete result {slim.keys()}")
    blob = json.dumps(slim, sort_keys=True).encode()
    slim["digest"] = hashlib.sha256(blob).hexdigest()[:12]
    return slim

I paste digest into the next turn. Not the novel. The model can still reason. It cannot launder junk keys.

Anti-pattern 3: Same-args retry theater

Identical calls look like persistence. They are usually a stuck loop. Does four search_repo hits mean the model thought harder?

Symptoms

• Four identical tool names and payloads.
• Latency looks like careful thinking.
• The writeup says the agent persisted.
• The working tree never changed.

Root cause

The loop has no call fingerprint. Retry policy is missing. Skill gets the credit for a spinner.

Replacement

Fingerprint name + canonical args. A second identical call is a bug. Allow it only when the contract says the tool is volatile.

# labeled example: fingerprint gate
import json

class CallGate:
    def __init__(self):
        self.seen = {}

    def check(self, name, args, volatile=False):
        key = name + json.dumps(args, sort_keys=True)
        n = self.seen.get(key, 0) + 1
        self.seen[key] = n
        if n > 1 and not volatile:
            raise RuntimeError(f"duplicate call {name} x{n}")
        return n

I log duplicate call as a harness fail. I do not log it as extra effort.

Anti-pattern 4: Side-effect amnesia

Tools mutate the world. Context is a cache. Did anyone invalidate the named artifact?

Symptoms

• A tool wrote a file.
• A later read uses an old snapshot.
• The agent "fixes" the stale text again.
• Diffs oscillate across two versions.

Root cause

The prompt still holds the pre-mutation copy. The model reconciles against a ghost. Your eyes follow the ghost too.

Replacement

After every mutating tool, refresh the named artifact. Version it. Feed the new digest, not the memory.

# labeled example: refresh after a mutating tool
set -euo pipefail
TARGET="src/bill.py"
BEFORE=$(sha256sum "$TARGET" | awk '{print $1}')
# agent tool would edit here
AFTER=$(sha256sum "$TARGET" | awk '{print $1}')
if [ "$BEFORE" = "$AFTER" ]; then
  echo "mutation claimed, digest unchanged" >&2
  exit 1
fi
echo "artifact $TARGET digest $AFTER"

I treat an unchanged digest as a failed mutation. The model does not get a second speech about it.

Anti-pattern 5: Single-session folklore

One chat is a story. An eval is a replay. Did you freeze the tool server first?

Symptoms

• One model session becomes the report.
• No fixture server exists.
• "It worked on my run" is the metric.
• Tomorrow the same prompt drifts.

Root cause

People confuse availability with evidence. A free lane is useful. It is not a regression suite by itself.

Replacement

Replay against a frozen tool server. Same args. Same expected envelope. Store the transcript as a fixture, not as folklore.

# labeled example: replay one frozen turn
import json
from pathlib import Path

def replay_turn(path: str, handler):
    rec = json.loads(Path(path).read_text())
    got = handler(rec["name"], rec["args"])
    if got != rec["expected"]:
        raise AssertionError(f"{path}: {got} != {rec['expected']}")
    return got

I keep those JSON files next to the tests. A green chat window does not land in that folder.

Artifact: a contract fixture you can run

Here is a tiny fake tool server. It is local. It is boring. Boring is the point.

# labeled example: fixture_server.py
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

TOOLS = {
    "write_note": {
        "ok": True,
        "error": None,
        "result": {"path": "notes/a.md", "sha256": "abc", "bytes": 12},
    }
}

class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        n = int(self.headers.get("Content-Length", "0"))
        payload = json.loads(self.rfile.read(n) or b"{}")
        name = payload.get("name")
        body = TOOLS.get(name)
        code = 200 if body else 404
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(body or {"ok": False, "error": "unknown", "result": None}).encode())

if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run the server. Then hit it with a contract client. Do not skip the accept function.

python fixture_server.py &
curl -sS -X POST http://127.0.0.1:8765 \
  -H 'Content-Type: application/json' \
  -d '{"name":"write_note","args":{"path":"notes/a.md"}}'

I also keep a decision table beside the fixture. Use it before you trust a run.

Signal in the trace	What I still do not know	Gate that must pass
HTTP 200	Envelope validity	accept_tool_body
Long stdout	Which keys are legal	project_result
Repeated tool name	Whether args changed	CallGate.check
"I wrote the file"	Whether bytes moved	digest before/after
One pretty session	Whether it replays	frozen replay_turn

Where a free lane actually helps

I use a scratch lane for the fixture, not for folklore. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are a place to exercise that triangle when I do not want a production box in the loop.

The product does not replace the contract. It only hosts a cheap run. If the envelope check is missing, a free server still lies politely.

Limitations

This catalog will not save a bad problem statement. Contract tests need an owner. They rot when tools change and fixtures do not.

Volatile tools break the duplicate-call gate. You must mark them volatile on purpose. Networked third-party APIs need recorded cassettes, not live hope.

I am not claiming model rankings here. I am not citing quotas, hardware, or duration. Those numbers go stale overnight. The envelope does not.

Who should not use this

Skip this if you only generate one-off snippets. Skip it if no tool can mutate state. Skip it if you already replay frozen envelopes in CI.

Do not use a shared scratch server for secrets. Do not point a fixture at production data. Do not treat my labeled examples as a security boundary.

What I do on the next trace

I ask five questions in order. Did transport get confused with meaning? Did raw stdout enter the next turn? Did identical args retry? Did a mutation keep a stale snapshot? Did one session become the report?

If any answer is yes, I stop blaming the model. I fix the tool boundary first. Then I replay. Then I talk about the patch.