Replay the CLI Contract Before You Accept an Agent Diff

The billing export script printed invoices to stdout, mixed logs into stderr, and encoded tax rules as nested if statements. Nobody on the current team could still name the last person who understood the rounding path. A coding agent could rewrite the module in minutes, which is exactly why the rewrite was the wrong first move.

This walkthrough uses a constructed brownfield example rather than an unverified production incident from a private repo. The working goal is a strict replay gate that treats process output as the missing specification. After the pin file is locked, one internal edit is allowed only if replay still matches. Internals may move; the process boundary may not.

Cheap patches do not make behavior cheap to trust

Faster generation does not retire implicit contracts with cron, downstream parsers, and operators who grep log lines. An agent that cleans names can still change a trailing newline, a money format, or an exit code that a wrapper treats as success. Those are behavior changes, even when the pull request reads like a refactor.

Technical debt does not shrink just because patches arrive faster than reviews. Debt grows when unverified edits accumulate around undocumented edges that only production still exercises. The practical response is not a larger prompt and a broader cleanup. The practical response is a frozen I/O record that the next patch must satisfy line for line.

What a pin file actually records

A pin is one complete process execution, not a unit assertion about private helpers. Each line in the pin log is a JSON object with a closed field list. Humans should be able to read a pin and explain the scenario in one sentence.

• id: stable name for the scenario, such as tax-exempt-q3
• argv: argument vector after the interpreter name
• env: only the keys the script actually reads
• stdin: raw text used for the recorded run
• exit_code: integer status observed from the process
• stdout and stderr: exact Unicode text with a documented decode mode
• cwd_rel: working directory relative to the repository root

The pin file is the specification the original authors never wrote down. If a field is not in that list, it is not part of the merge contract yet.

A messy module worth pinning

The example below is intentionally awkward on purpose. It mutates globals, prints progress on stderr, and formats money with ad hoc rounding. Treat it as labeled sample code, not as a measured production extract.

# export_invoices.py — constructed example, not production code
from __future__ import annotations

import json
import os
import sys
from decimal import Decimal, ROUND_HALF_EVEN

TAX = Decimal(os.environ.get("TAX_RATE", "0.0875"))
SEEN = []


def money(n: str) -> str:
    d = Decimal(n).quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
    if os.environ.get("CURRENCY") == "USD":
        return f"${d}"
    return str(d)


def main(argv: list[str]) -> int:
    path = argv[1] if len(argv) > 1 else "-"
    raw = sys.stdin.read() if path == "-" else open(path, encoding="utf-8").read()
    rows = json.loads(raw) if raw.strip() else []
    exempt = os.environ.get("TAX_EXEMPT", "0") == "1"
    out = []
    for row in rows:
        SEEN.append(row["id"])
        base = Decimal(str(row["amount"]))
        tax = Decimal("0.00") if exempt or row.get("kind") == "internal" else (base * TAX)
        tax = tax.quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
        line = f"{row['id']},{money(str(base))},{money(str(tax))},{money(str(base + tax))}"
        out.append(line)
        print(f"wrote {row['id']}", file=sys.stderr)
    print("\n".join(out))
    print(f"count={len(SEEN)}", file=sys.stderr)
    return 0 if out else 2


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

That script is small enough to read and still unsafe to improve without a gate. Rounding, exemption flags, empty-input exit codes, and stderr chatter are all load-bearing. A rename-only patch can still break a downstream cut or a cron mail filter.

Record pins before any production edit

The recorder runs the current script as the oracle and does not interpret business rules. Label this harness as a proposed workflow, not as a benchmark with timings or pass rates. Commit the pins while the code is still ugly.

# record_pins.py — proposed recorder
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
PIN_PATH = ROOT / "pins.jsonl"


def run_case(case: dict) -> dict:
    env = os.environ.copy()
    for key in ("TAX_RATE", "TAX_EXEMPT", "CURRENCY"):
        env.pop(key, None)
    env.update(case.get("env") or {})
    stdin = case.get("stdin", "")
    proc = subprocess.run(
        [sys.executable, str(ROOT / "export_invoices.py"), *case.get("argv", ["-"])],
        input=stdin,
        text=True,
        capture_output=True,
        env=env,
        cwd=ROOT,
        check=False,
    )
    return {
        "id": case["id"],
        "argv": case.get("argv", ["-"]),
        "env": case.get("env") or {},
        "stdin_sha256": hashlib.sha256(stdin.encode()).hexdigest(),
        "stdin": stdin,
        "exit_code": proc.returncode,
        "stdout": proc.stdout,
        "stderr": proc.stderr,
        "cwd_rel": ".",
    }


CASES = [
    {
        "id": "empty-stdin",
        "argv": ["-"],
        "env": {"TAX_RATE": "0.0875", "CURRENCY": "USD"},
        "stdin": "",
    },
    {
        "id": "mixed-kinds",
        "argv": ["-"],
        "env": {"TAX_RATE": "0.0875", "CURRENCY": "USD", "TAX_EXEMPT": "0"},
        "stdin": json.dumps(
            [
                {"id": "A-1", "amount": "10.00", "kind": "external"},
                {"id": "A-2", "amount": "10.005", "kind": "internal"},
                {"id": "A-3", "amount": "0.015", "kind": "external"},
            ]
        ),
    },
    {
        "id": "org-exempt",
        "argv": ["-"],
        "env": {"TAX_RATE": "0.0875", "CURRENCY": "USD", "TAX_EXEMPT": "1"},
        "stdin": json.dumps([{"id": "B-9", "amount": "99.99", "kind": "external"}]),
    },
]


def main() -> None:
    lines = [json.dumps(run_case(c), ensure_ascii=False) for c in CASES]
    PIN_PATH.write_text("\n".join(lines) + "\n", encoding="utf-8")
    print(f"wrote {len(lines)} pins to {PIN_PATH}")


if __name__ == "__main__":
    main()

Run the recorder once against the unedited script, then commit pins.jsonl in a change that contains no refactor. After that commit, the agent is not allowed to regenerate pins to make a patch pass.

python record_pins.py
git add export_invoices.py record_pins.py pins.jsonl
git commit -m "Pin invoice exporter I/O before any refactor"

Replay is the only merge gate

The test below rehydrates each pin and compares exit code, stdout, and stderr as strings. It is deliberately strict on whitespace and log wording. If you later need normalization, document that function in the test file instead of quietly editing pins.

# test_replay_pins.py
from __future__ import annotations

import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent


def load_pins():
    text = (ROOT / "pins.jsonl").read_text(encoding="utf-8")
    return [json.loads(line) for line in text.splitlines() if line.strip()]


def replay(pin: dict) -> subprocess.CompletedProcess:
    env = os.environ.copy()
    for key in ("TAX_RATE", "TAX_EXEMPT", "CURRENCY"):
        env.pop(key, None)
    env.update(pin["env"])
    return subprocess.run(
        [sys.executable, str(ROOT / "export_invoices.py"), *pin["argv"]],
        input=pin["stdin"],
        text=True,
        capture_output=True,
        env=env,
        cwd=ROOT / pin["cwd_rel"],
        check=False,
    )


def test_every_pin_replays():
    failures = []
    for pin in load_pins():
        proc = replay(pin)
        if proc.returncode != pin["exit_code"]:
            failures.append(f"{pin['id']}: exit {proc.returncode} != {pin['exit_code']}")
        if proc.stdout != pin["stdout"]:
            failures.append(
                f"{pin['id']}: stdout mismatch\n---\n{proc.stdout!r}\n===\n{pin['stdout']!r}"
            )
        if proc.stderr != pin["stderr"]:
            failures.append(
                f"{pin['id']}: stderr mismatch\n---\n{proc.stderr!r}\n===\n{pin['stderr']!r}"
            )
    assert not failures, "\n\n".join(failures)

python -m pytest test_replay_pins.py -q

A tiny hook keeps the agent honest when it tries to refresh the oracle. The hook is a proposal you can drop into .git/hooks/pre-commit after reviewing it.

#!/bin/sh
# proposed pre-commit: fail if pins.jsonl changes beside production code
if git diff --cached --name-only | grep -q '^pins\.jsonl$'; then
  if git diff --cached --name-only | grep -q '^export_invoices\.py$'; then
    echo "refuse: pins.jsonl and export_invoices.py in the same commit"
    exit 1
  fi
fi

The first internal edit after green replay

Only after replay is green may the next patch touch internals, and only for one concern. A reasonable first edit in this example extracts rounding without altering prints. Keep main as the only process entry so argv and env stay in one place.

Proposed rules, labeled as unexecuted guidance rather than a completed refactor:

1. Move rounding into a pure function that main already calls.
2. Do not silence stderr lines that operators may grep in cron mail.
3. Do not change the empty-input exit code from 2 to 0.
4. Reject any patch that updates pins.jsonl in the same commit as production code.

If replay fails, revert the patch instead of editing the pin. The pin is older than the patch, which is the entire point of the gate. Green replay with a logging rewrite is still a contract break.

Decision table: keep the diff or stop the session

The table below is the workflow. Prompts around it are optional commentary and never override a red replay.

Observation after the patch	Action
Replay green, diff limited to one function	Keep the patch and end the session
Replay green, but the diff rewrites logging	Revert; logging is part of the CLI contract
Replay red on stdout only	Revert; treat it as a behavior change
Replay red on stderr only	Revert unless an operator signs off in review
Agent rewrites pins.jsonl	Reject the commit in hook and in CI
Agent adds features while cleaning	Reject; new behavior needs human-authored pins

Coverage holes remain possible. A hidden branch that no pin exercises will replay green and still be wrong. Add pins when you discover those branches in production logs, not when an agent asks for more freedom.

Where a disposable agent session belongs

An agent is useful after pins exist, because failed attempts then show up as red replay instead of plausible prose. It is not useful as a substitute for recording the current oracle on a dirty tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option fit this loop as a disposable place to attempt that single internal edit. Record pins on your own checkout, commit them, and copy only the repository snapshot onto that server. The merge decision stays with local replay, not with the session transcript. If you remove the product from the method, the pin-and-replay gate still works on a laptop.

Limitations, and who should skip this

Process-level pins are weak when the script talks to clocks, networks, or unordered maps. They also overfit to log wording, which is intended here and harmful if you already planned a logging redesign. Empty coverage of a hidden branch will look like success.

Skip this approach when any of the following are true.

• The program is not a batch boundary, so pins cannot see GUI or long-lived server state.
• Output includes timestamps, random identifiers, or unstable iteration order.
• You need to specify new behavior rather than preserve old behavior.
• The agent would receive production credentials or write access to real data.
• A human cannot read pins.jsonl and explain each case in one sentence.

Clock and network seams should be injected before recording, not after the agent starts editing. If you cannot freeze time, you do not yet have a pin worth merging against. Do not treat green replay as proof that the tax rules are correct; it only proves they did not change.

A pull-request checklist instead of a prompt dump

Use this list as review notes. It is slower than letting the agent sweep the file, and that slowness is the control.

1. A human records pins against the current default branch with no code edits in that commit.
2. CI runs pytest test_replay_pins.py and fails the build if pins.jsonl is missing.
3. The agent receives one sentence of intent, the pin path, and a diff budget of one function.
4. A human reruns replay locally and inspects git diff --stat before review.
5. If the diff crosses logging, environment parsing, or exit codes, the patch is discarded.

Generation is no longer the scarce resource on a messy script. Agreement with the last known process boundary is. Lock that boundary in JSONL, then accept an agent diff only when replay still prints the same contract.