The Patch Cannot Widen the Tool Surface

A payment worker sits in the Tuesday merge queue. An agent rewrote the retry helper during the night. Local unit tests remain green and very quiet. Later production traces show a new outbound tool name.

The helper did more than retry failed charges. It also called a refund endpoint without review. No test watched the outbound call surface closely. Green meant the function returned. It did not mean the wire stayed closed.

Agent patches fail in this quiet pattern. They add a helpful tool under a refactor. Assertions on local return values miss extra calls. The club gained a side door overnight. The bouncer still watched the front rope.

Treat the tool surface as a printed guest list. Names on that list may enter the room. A new name fails at the door without debate. An extra JSON field counts as a new name. Humans own the list in review. The agent does not edit that file.

This piece proposes a merge gate, not a memoir. No production timings or pass rates are claimed. The code is a labeled, runnable sketch. Execute it on a laptop before trusting CI.

Commit a JSON contract beside the service code. The file names every allowed tool in play. It pins argument shapes with closed schemas. It also pins the error classes callers may raise.

{
  "tools": {
    "charges.create": {
      "type": "object",
      "required": ["amount_cents", "currency", "idempotency_key"],
      "additionalProperties": false,
      "properties": {
        "amount_cents": {"type": "integer", "minimum": 1},
        "currency": {"enum": ["usd", "eur"]},
        "idempotency_key": {"type": "string", "minLength": 16}
      }
    },
    "charges.get": {
      "type": "object",
      "required": ["charge_id"],
      "additionalProperties": false,
      "properties": {
        "charge_id": {"type": "string", "pattern": "^ch_"}
      }
    }
  },
  "errors": ["Timeout", "Conflict", "Unavailable"]
}

The agent may patch Python under src/. It may not patch tool_surface.json in that lane. Reviewers treat a surface edit as a product change. That split keeps the contract with humans.

Wrap the HTTP or tool client before tests run. Log tool name, arguments, and error class. Write one JSON line for every call. Do this inside characterization tests you already own. Do not wait for production traffic to notice drift.

# trace_client.py
import json
from pathlib import Path

class TracingClient:
    def __init__(self, inner, sink: Path):
        self.inner = inner
        self.sink = sink

    def call(self, name: str, args: dict):
        record = {"name": name, "args": args, "error": None}
        try:
            return self.inner.call(name, args)
        except Exception as exc:
            record["error"] = type(exc).__name__
            raise
        finally:
            with self.sink.open("a") as handle:
                handle.write(json.dumps(record, sort_keys=True) + "\n")

Point the characterization suite at TracingClient. Keep the sink on a temp path. Copy the sink after a green characterization run. Diff every row against the committed surface file. A new tool name fails the gate. A new argument key fails the gate. An unknown error class fails the gate.

The gate test should look almost boring. Boring checks survive agent charm in review. A poetic assertion gets rewritten to stay green. A schema check does not bargain.

# test_tool_surface.py
import json
from pathlib import Path
import jsonschema

SURFACE = json.loads(Path("tool_surface.json").read_text())

def load_trace(path: Path):
    rows = []
    for line in path.read_text().splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows

def test_trace_stays_inside_surface(tmp_path):
    sink = tmp_path / "trace.jsonl"
    # Proposal: call your real characterization suite here.
    run_characterization(sink)
    rows = load_trace(sink)
    assert len(rows) >= 100, "empty traces cannot pass the surface gate"
    for row in rows:
        assert row["name"] in SURFACE["tools"], row
        jsonschema.validate(row["args"], SURFACE["tools"][row["name"]])
        if row["error"] is not None:
            assert row["error"] in SURFACE["errors"], row

A closed list is still not a correct list. Valid fields can move the wrong money. After the surface holds, throw junk at known tools. The client must reject junk before the wire. It must not open a new tool to help.

# test_tool_properties.py
import json
from pathlib import Path
from hypothesis import given, strategies as st

SURFACE = json.loads(Path("tool_surface.json").read_text())
NAMES = list(SURFACE["tools"])

class StrictClient:
    def __init__(self, inner, surface):
        self.inner = inner
        self.surface = surface

    def call(self, name, args):
        if name not in self.surface["tools"]:
            raise ValueError("unknown tool")
        schema = self.surface["tools"][name]
        extra = set(args) - set(schema.get("properties", {}))
        if extra:
            raise ValueError(f"extra fields: {extra}")
        return self.inner(name, args)

@given(
    name=st.sampled_from(NAMES),
    extra=st.dictionaries(st.text(min_size=1, max_size=8), st.integers(), min_size=1),
)
def test_unknown_fields_never_hit_the_wire(name, extra):
    sent = []

    def fake_call(tool_name, args):
        sent.append((tool_name, args))
        return {"ok": True}

    client = StrictClient(inner=fake_call, surface=SURFACE)
    try:
        client.call(name, {"_probe": True, **extra})
    except ValueError:
        assert sent == []
        return
    raise AssertionError("strict client must reject extra fields")

Hypothesis use here is a labeled proposal. Tune strategies to the committed schemas. Do not let the agent own the strategy module. Apply the same review split used for tool_surface.json.

Flaky tests still haunt agent merges on busy nights. A flake that fails the surface looks like a widen. A flake that passes can hide a new tool. Those tests must leave the merge vote. Move their nodeids into a quarantine file. CI reads that file on every agent patch. Quarantined names cannot fail the job. They also cannot pass the job.

# conftest.py
from pathlib import Path
import pytest

FROZEN = {
    line.strip()
    for line in Path("flake_quarantine.txt").read_text().splitlines()
    if line.strip() and not line.startswith("#")
}

def pytest_collection_modifyitems(config, items):
    for item in items:
        if item.nodeid in FROZEN:
            item.add_marker(pytest.mark.skip(reason="flake quarantined; no merge vote"))

# flake_quarantine.txt
# nodeids only. Agent patches may not edit this file.
tests/test_retry.py::test_eventual_success

The quarantine is a process freeze, not a wish. A human reproduces the flake on a quiet branch. Then the test returns to voting or dies. Agents do not bargain with that list. The rule is the control.

Run characterization first, then the surface gate. Run property checks after those two stay green. Keep quarantined nodeids skipped the whole time.

python -m pytest tests/characterization -q --trace-sink=/tmp/trace.jsonl
python -m pytest tests/test_tool_surface.py -q
python -m pytest tests/test_tool_properties.py -q
python -m pytest tests --strict-markers -q

Fail the build when tool_surface.json moves with agent code. A small git check enforces the human split. Reviewers may still change both files together. They do that in a separate human commit.

changed=$(git diff --name-only origin/main)
echo "$changed" | grep -q 'tool_surface.json'
code_moved=$?
echo "$changed" | grep -qE '^(src/|app/)'
src_moved=$?
if [ "$code_moved" -eq 0 ] && [ "$src_moved" -eq 0 ]; then
  echo "surface and service code moved together" >&2
  exit 1
fi

Some teams lack idle CI minutes on merge day. The harness is pytest, JSON, and a schema library. It can run on a laptop beside the diff. It can also run on a free remote server when that laptop is busy. MonkeyCode offers free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Use the free model only to mint extra invalid payloads for the property file. Never use it to edit tool_surface.json. Never use it to edit flake_quarantine.txt. The server should run the same commands you run locally. If those two files change in the job, reject the job.

A closed tool surface is not semantic safety. The agent can still pass a wrong amount. The schema allows any integer above zero. Idempotency keys can match shape and still collide. Property checks catch extra fields at the door. They do not catch a business lie in a valid field.

High-churn public APIs will trip this gate often. That pain is a signal, not a defect. Update the surface in a human-owned change. Do not set additionalProperties true to silence the agent.

Skip this approach for research agents on purpose. Those agents must explore tools that are not listed. A closed guest list would block that work. Do not use it as the only control in payments. Add ledgers, limits, and a human review. Do not install the gate when traces cannot be collected. A gate without traces is only theater.

Empty characterization still passes if no tool is called. Pair the schema diff with a minimum trace count. One hundred calls is a starting proposal only. Measure against your own suite before copying it. Do not treat that number as a law.

The patch can stay clever in the helper. The door list stays dull on purpose. Dull lists are what merge queues can actually enforce.