Don't Merge Agent Patches That Invent I/O

The refund job posted two credits on Tuesday night. The customer support queue filled before anyone noticed. The unit suite had stayed green for several hours.

The agent patch looked small on the GitHub diff. It renamed one helper and claimed to fix retries. It also imported httpx into a database-only module.

That extra client never appeared in the test names. The mock layer still returned the old fixture body. Staging paid the cost in duplicated refunds.

Treat this as a review problem, not a model problem. Agent patches often add sockets that existing tests cannot see. Your suite still knocks on the front door.

The side gate is new. The contractor copied a key you never issued. That is the analogy that holds under load.

A passing test file is not a permit. Green assertions can ignore a fresh network call. The agent tells a story about retries and robustness.

Review the socket, not the story. Freeze new I/O the same way you freeze schema. Do that before the pull request reaches shared CI.

What counts as invented I/O

Invented I/O is any new path off the process. Network clients, subprocess shells, and extra file writes qualify. Cloud SDKs belong on the same list.

The baseline is the module before the agent touched it. If the old code talked only to Postgres, HTTP is a new privilege. If the old code read config, a write is a new privilege.

This is not a style nit. Extra I/O changes failure domains and billing. It also changes what your mocks actually prove.

Label the next snippets as a proposed gate, not production lore. Run them on a throwaway branch first. Adjust the allowlists to your tree.

A permit list over the diff

Start from git diff against the merge base. Parse added lines, not the whole file. Score only production paths, not tests.

#!/usr/bin/env python3
"""permit_io.py — fail if an agent patch invents I/O."""
from __future__ import annotations

import re
import subprocess
import sys
from pathlib import Path

DENIED = re.compile(
    r"^\+\s*(?:from\s+(httpx|requests|aiohttp|urllib|socket|subprocess|boto3|paramiko)"
    r"|import\s+(httpx|requests|aiohttp|socket|subprocess|boto3|paramiko)"
    r"|os\.system\(|pathlib\.Path\([^)]*\)\.write_text\()"
)
SKIP_DIRS = {"tests", "test", "fixtures", "scripts"}

def merge_base() -> str:
    return subprocess.check_output(
        ["git", "merge-base", "HEAD", "origin/main"],
        text=True,
    ).strip()

def added_lines() -> list[tuple[str, str]]:
    raw = subprocess.check_output(
        ["git", "diff", "-U0", merge_base(), "--", "*.py"],
        text=True,
    )
    file = ""
    out: list[tuple[str, str]] = []
    for line in raw.splitlines():
        if line.startswith("+++ b/"):
            file = line[6:]
            continue
        if line.startswith("+") and not line.startswith("+++"):
            out.append((file, line))
    return out

def production(path: str) -> bool:
    parts = set(Path(path).parts)
    return SKIP_DIRS.isdisjoint(parts)

def main() -> int:
    hits = [
        f"{path}: {line}"
        for path, line in added_lines()
        if production(path) and DENIED.search(line)
    ]
    if not hits:
        print("permit_io: no new I/O privileges in production paths")
        return 0
    print("permit_io: agent patch invents I/O")
    print("\n".join(hits))
    print("Add a review note or move I/O behind an existing port.")
    return 1

if __name__ == "__main__":
    sys.exit(main())

Wire it as a required check, not a chat comment. Developers can still add HTTP on purpose. They must do it in a port module with a human note.

git fetch origin main
python3 permit_io.py
# exit 1 means the patch grew a new socket, shell, or write

The script is deliberately dumb. Dumb gates survive model chatter better than clever ones. Dynamic imports will slip through, and that is a documented gap.

Replay the tools, leave the model out

CI should not call a live model to judge a patch. Live calls drift, throttle, and invent extra tools. Record the tool transcript once, then replay it.

Keep the recording next to the failing production module. Name it after the job, not after the prompt. The prompt is not the contract.

# tests/replay/test_refund_tools.py
import json
from pathlib import Path

from refund_job import apply_refund, ToolBus

TRACE = Path(__file__).with_name("refund_ok.tools.json")

class ReplayBus(ToolBus):
    def __init__(self, events):
        self.events = list(events)
        self.seen = []

    def call(self, name, payload):
        expected = self.events.pop(0)
        assert name == expected["name"]
        assert payload == expected["payload"]
        self.seen.append((name, payload))
        return expected["result"]

def test_refund_replays_recorded_tools_only():
    events = json.loads(TRACE.read_text())
    bus = ReplayBus(events)
    apply_refund(order_id="ord_9", bus=bus)
    assert bus.events == []
    assert [name for name, _ in bus.seen] == ["ledger.get", "ledger.credit"]

A sample trace stays tiny and boring on purpose. Boring traces make silent extra calls obvious. An extra notify.slack entry fails the pop assertion.

[
  {
    "name": "ledger.get",
    "payload": {"order_id": "ord_9"},
    "result": {"status": "paid", "cents": 4200}
  },
  {
    "name": "ledger.credit",
    "payload": {"order_id": "ord_9", "cents": 4200},
    "result": {"ok": true, "id": "cr_1"}
  }
]

Record traces from a staging replay, not from chat. Hash the file in the pull request body. If the agent needs a new tool, the hash changes and review starts.

pytest tests/replay/test_refund_tools.py -q
sha256sum tests/replay/refund_ok.tools.json

This is not a golden snapshot of printed text. It is a permit for named tools. The difference matters when formatters churn and behavior stays still.

Where a free coding server actually fits

Generating the candidate patch can happen off your laptop. Cheap generation is useful when the gate is strict. The gate must still run on your tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for that generation step. It does not replace the permit script or the replay suite.

Keep the split boring and explicit. Let the server propose a diff. Let local permit_io.py and pytest accept or reject it.

# proposed local flow after a generated diff lands in a branch
git checkout -b agent/refund-retry
python3 permit_io.py || exit 1
pytest tests/replay tests/unit -q --maxfail=1

If the generator adds httpx beside the ledger port, the permit fails first. You never spend CI minutes on a philosophical retry debate. The socket was the bug.

Do not point CI at a free remote model as an oracle. Oracles that chat will bless extra tools with extra stories. Replay files do not tell stories.

A small decision table in prose

When the diff only renames locals, skip the drama. Run unit tests and merge. No new privilege appeared.

When the diff touches a port module on purpose, demand a human note. Update the replay trace in the same commit. That is a licensed I/O change.

When the diff sneaks I/O into a domain file, reject it. Move the call behind the existing port. Record a new tool only after that move.

When the generator times out or returns partial files, do not partial-merge. Incomplete patches hide the second import. Rerun generation, then rerun the gate.

Limitations, stated without theater

The permit regex will miss __import__("httpx") and late binding. It will also miss shell-outs through asyncio.create_subprocess_exec. Teams with heavy metaprogramming need AST checks instead.

Replay tests freeze payload equality. That hurts if you add harmless timestamps inside tools. Pin clocks in the port, or strip time keys in the bus.

This approach assumes a clear port boundary already exists. Greenfield scripts that print to the network everywhere will drown in denials. Do not adopt the gate there.

Do not use this flow as proof that an agent understood payments. It only proves the patch did not grow a new socket. Correctness of credit math still needs domain tests.

Skip the method if your runtime already sandboxes outbound calls. A platform proxy that denies unknown hosts overlaps this gate. Two overlapping gates rot, and one will be ignored.

What to keep in the review comment

Write the review in three short facts. Name the new privilege if any exists. Name the replay file that must change.

Name the port module that should own the call. Ask for a one-line license in the commit. Then stop talking about the model's confidence.

Confidence is not a socket. A green suite is not a permit. The refund queue already taught that lesson once.

If you generate patches on a free server, run this gate before the PR. The cheap step is generation. The expensive step is an unreviewed call in production.