If It Can Write, Pin the Contract: A Fail-Closed Agent Tool Checklist

If an agent can call a tool that writes, deletes, pays, or talks to a network, you do not merge on a demo. You merge on a pinned tool contract, a budget, and evidence that the run stayed inside both. Everything else is a live experiment wearing a pull request.

Tool calling is the part of agent work that actually changes production. Prompts do not. Completions do not. A JSON blob that becomes rm, transfer, or deploy does. Treat that blob as an API, not as a suggestion.

This checklist is for teams shipping agents that invoke tools—local functions, HTTP actions, or MCP-style servers. Copy it. Fail closed. Do not soften a red gate because the trace “looked reasonable.”

What you are actually shipping

You are not shipping a chat loop. You are shipping a remote-control surface with a probabilistic driver.

That surface needs the same things any other production API needs: a schema, a version, an owner, a timeout, a retry policy, and a recorded decision about side effects. If you cannot point to those artifacts in the repo, you do not have a product. You have a script that sometimes works.

Write the contract first. Then let the model talk to the contract. Never the other way around.

Gate 0 — Inventory before intelligence

List every tool the agent may invoke in this change. If the list is generated at runtime from a directory scrape, a plugin folder, or “whatever the MCP server advertised today,” the gate fails.

Required evidence:

• A committed tools/inventory.yaml with a frozen name set
• A content hash of each tool descriptor
• An owner and on-call path per tool
• A side-effect class: read, write, delete, network, payment, or deploy

Fail closed when:

• A runtime tool name is missing from inventory
• A descriptor hash drifted without a contract bump
• Any tool has side-effect class unknown

Unknown is not a category. Unknown is a merge blocker.

Gate 1 — Pin the contract, not the vibe

Every inventoried tool gets a contract file. One file per tool. No “see README.” No “the model knows the shape.”

# tools/contracts/create_invoice.contract.yaml
name: create_invoice
version: 3
owner: billing-platform
side_effects: [write, network]
allow_write: true
timeout_ms: 4000
max_calls_per_run: 2
retry:
  max: 1
  backoff_ms: 250
  retry_on: [timeout]
idempotency_key_required: true
input_schema:
  type: object
  additionalProperties: false
  required: [account_id, amount_cents, currency, idempotency_key]
  properties:
    account_id: { type: string, pattern: "^acc_[a-z0-9]{8,}$" }
    amount_cents: { type: integer, minimum: 1, maximum: 50000000 }
    currency: { type: string, enum: [USD, EUR] }
    idempotency_key: { type: string, minLength: 16, maxLength: 64 }
output_schema:
  type: object
  additionalProperties: false
  required: [status, invoice_id]
  properties:
    status: { type: string, enum: [created, duplicate, rejected] }
    invoice_id: { type: ["string", "null"] }
    error: { type: ["string", "null"] }
untrusted_output: true
forbidden_args:
  - raw_sql
  - shell
  - file_path

Fail closed when the contract is missing, when additionalProperties is true on a write tool, or when a write tool has no idempotency key. Soft schemas are how “create invoice” becomes “create invoice and also whatever field the model invented.”

Gate 2 — Arguments are data, not instructions

Validate arguments before the tool runs. Validate outputs before they re-enter the model context. Both checks are mechanical. Neither is a prompt.

You want three failures, not one vague exception:

1. Schema miss: wrong types, extra keys, broken enums
2. Policy miss: amount over limit, cross-tenant id, missing idempotency key
3. Shape miss on the way back: tool returned prose, a stack trace, or a field you never allowed

If output is untrusted—and tool output is untrusted—strip it down to the schema. Do not concatenate raw tool text into the next prompt. That is how a search result becomes an instruction.

Gate 3 — Budgets are merge gates, not dashboards

A loop without a budget is an unbounded job. Unbounded jobs do not belong in CI or in prod.

Pin all four limits in tools/budget.yaml:

• Wall clock for the whole run
• Model-call count
• Tool-call count per tool and in total
• Max payload bytes in and out of each tool

The run dies when any limit trips. Dying is success for the gate. Continuing “just to finish the task” is the failure mode.

Record the counters in the evidence packet. A green CI job with no counters is not green. It is unmeasured.

Gate 4 — Writes need a dry-run path you actually execute

Read tools can run against fixtures. Write tools need a dry-run that cannot mutate the real system, plus one recorded mutation against a disposable fixture.

Fail closed when:

• allow_write: true and there is no dry-run implementation
• Dry-run and real path share no schema
• The only proof of the write is a screenshot or a chat log
• The fixture is the staging database you share with humans

If you cannot replay the write against a throwaway store, you cannot review it. If you cannot review it, you do not merge it.

Gate 5 — Trace or it did not happen

Every tool call in the PR needs a trace row you can grep. Not a marketing “observability story.” A row.

Minimum columns:

• run_id, step, tool, contract_version
• argument digest (not raw secrets)
• output digest and schema-validation result
• latency_ms, retry_count, budget remaining
• decision: allowed, blocked_schema, blocked_policy, blocked_budget

Fail closed when a write has no row, when the argument digest does not match the validated payload, or when the model saw a blocked output anyway. Hidden retries are still retries. Hidden writes are incidents.

Reproducible artifact: fail the build if the contract is theater

Save this as scripts/check_tool_contracts.py. It is a proposal you can run locally. It does not call a model. It only checks that the repo can tell the truth about tools.

#!/usr/bin/env python3
"""Fail closed if agent tool contracts are missing, writable, or unpinned."""
from __future__ import annotations

import hashlib, json, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("Install pyyaml before running this check.", file=sys.stderr)
    sys.exit = 2

ROOT = Path("tools")
REQUIRED = {
    "name", "version", "owner", "side_effects", "allow_write",
    "timeout_ms", "max_calls_per_run", "input_schema", "output_schema",
    "untrusted_output",
}
WRITE_CLASSES = {"write", "delete", "payment", "deploy"}


def load(path: Path):
    return yaml.safe_load(path.read_text()) or {}


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def fail(msg: str, errors: list[str]) -> None:
    errors.append(msg)


def main() -> int:
    errors: list[str] = []
    inv_path = ROOT / "inventory.yaml"
    if not inv_path.exists():
        print("missing tools/inventory.yaml", file=sys.stderr)
        return 1
    inventory = load(inv_path)
    tools = inventory.get("tools") or []
    if not tools:
        print("inventory tools[] is empty", file=sys.stderr)
        return 1

    budget_path = ROOT / "budget.yaml"
    if not budget_path.exists():
        errors.append("missing tools/budget.yaml")
    else:
        budget = load(budget_path)
        for key in ("wall_clock_ms", "max_model_calls", "max_tool_calls", "max_payload_bytes"):
            if not isinstance(budget.get(key), int) or budget[key] <= 0:
                fail(f"budget.{key} must be a positive int", errors)

    seen = set()
    for item in tools:
        name = item.get("name")
        expected_hash = item.get("descriptor_sha256")
        contract_path = ROOT / "contracts" / f"{name}.contract.yaml"
        if not name:
            fail("inventory entry missing name", errors)
            continue
        seen.add(name)
        if not contract_path.exists():
            fail(f"{name}: missing {contract_path}", errors)
            continue
        digest = sha256(contract_path)
        if expected_hash != digest:
            fail(f"{name}: hash mismatch (inventory={expected_hash} file={digest})", errors)
        spec = load(contract_path)
        missing = REQUIRED - set(spec)
        if missing:
            fail(f"{name}: missing fields {sorted(missing)}", errors)
        effects = set(spec.get("side_effects") or [])
        if effects & WRITE_CLASSES:
            if not spec.get("allow_write"):
                fail(f"{name}: write-class tool must set allow_write true", errors)
            if not spec.get("idempotency_key_required"):
                fail(f"{name}: write-class tool needs idempotency_key_required", errors)
            if spec.get("input_schema", {}).get("additionalProperties") is not False:
                fail(f"{name}: write tools must set additionalProperties false", errors)
        if spec.get("untrusted_output") is not True:
            fail(f"{name}: untrusted_output must be true", errors)
        if int(spec.get("timeout_ms") or 0) <= 0:
            fail(f"{name}: timeout_ms must be positive", errors)
        if int(spec.get("max_calls_per_run") or 0) <= 0:
            fail(f"{name}: max_calls_per_run must be positive", errors)

    extra = sorted(p.stem.replace(".contract", "") for p in (ROOT / "contracts").glob("*.contract.yaml") if p.stem.split(".")[0] not in seen)
    # filenames are name.contract.yaml so stem is name.contract
    extra = []
    for p in (ROOT / "contracts").glob("*.contract.yaml"):
        n = p.name.removesuffix(".contract.yaml")
        if n not in seen:
            extra.append(n)
    if extra:
        fail(f"contracts not in inventory: {extra}", errors)

    if errors:
        print("TOOL CONTRACT CHECK FAILED", file=sys.stderr)
        for e in errors:
            print(f"- {e}", file=sys.stderr)
        return 1
    print(json.dumps({"ok": True, "tools": sorted(seen)}))
    return 0


if __name__ == "__main__":
    sys.exit(main())

Wire it so a missing hash is a red build, not a warning:

pip install pyyaml
python3 scripts/check_tool_contracts.py

Add a tiny inventory so the check has something honest to chew on:

# tools/inventory.yaml
tools:
  - name: create_invoice
    descriptor_sha256: REPLACE_WITH_SHA256_OF_THE_CONTRACT_FILE
    owner: billing-platform

Compute the hash the same way CI will:

python3 - <<'PY'
from pathlib import Path
import hashlib
p = Path("tools/contracts/create_invoice.contract.yaml")
print(hashlib.sha256(p.read_bytes()).hexdigest())
PY

Paste that digest into inventory. If someone “just tweaks” an enum, the hash moves, and merge stops. That is the point.

Decision table you can paste into the PR template

Question	Evidence	Fail closed if
Which tools can this agent call?	tools/inventory.yaml	Runtime discovery or globbing
Is the descriptor pinned?	SHA-256 in inventory matches file	Hash missing or drifted
Can it write?	side_effects + allow_write	Write with no idempotency key
Are args constrained?	additionalProperties: false + enums	Open objects on write tools
Is output hostile?	untrusted_output: true + schema strip	Raw tool text in next prompt
What stops the loop?	tools/budget.yaml counters	Any counter absent in the trace
Can you replay the write?	Dry-run + fixture mutation log	Staging-as-fixture or screenshot
Did CI see the same contracts?	checker exit code 0	Warning-only lint

If any row is “we’ll add it later,” the answer is no merge. Later is how contracts rot.

Where a free eval box helps—and where it does not

You still need the checker in your CI. Contracts live in your repo. A hosted loop does not replace Gate 0 through Gate 5.

It can help when you want a throwaway process that hammers fixture tools without burning a paid API on every contract tweak. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option; those are the only product claims here. Use them, if they fit your constraints, to run the checker and a fixture tool server in a box you can destroy. Do not use that box as production. Do not treat a successful playground run as evidence. Evidence is the inventory hash, the budget counters, and the trace rows in the PR.

If you try it, keep the same fail-closed rule: no pinned contract, no write.

Limitations

This checklist does not prove the model chose the right tool. It proves the tool could not silently change shape, retry forever, or write without a key. Those are different problems. Do not pretend a schema gate is an eval suite.

It also does not replace IAM. A contract that says account_id must match a pattern does not bind the credential the tool process holds. If the process can reach every tenant, the model only needs one bad id. Put real authz in the tool implementation. Put the contract in front of it anyway.

Hashes pin files, not behavior. A server behind an MCP endpoint can change while the local descriptor stays frozen. If you call remote tools, pin the server identity and fail on advertisement drift. Local YAML is necessary. It is not sufficient for a moving endpoint.

Who should not use this

Skip this if your agent is read-only over public docs and cannot call anything with side effects. A search loop with no tools is not this problem.

Skip it if you cannot fail the build. A checklist that reviewers “keep in mind” is not a gate. It is a blog post.

Skip it if your tools are one-off shell aliases on a laptop. Pinning contracts for a personal scratchpad is ceremony. The moment that scratchpad can charge a card or deploy, it stops being a scratchpad.

Merge rule

You merge when inventory, hashes, schemas, budgets, dry-run proof, and traces are in the packet, and the checker exits 0. You do not merge because the agent completed the happy path once.

Pin the contract. Treat output as hostile. Kill the loop on budget. If it can write, those are not style preferences. They are the difference between a tool and an incident.