Case Study: Reject Agent Patches That Touch Paths You Never Declared

You should reject a coding agent patch the moment it touches a path, env var, or command you never declared. This case study walks you through a small, reproducible ledger that fails closed before any model explanation is invited. You will keep the checker deterministic, keep the model optional, and keep the repo layout from drifting under cheap generated edits. The method is deliberately boring, because boring contracts survive weekend experiments better than prompt optimism.

Background: cheap patches, silent assumptions

Agent write-ups this week keep circling a practical failure, not a glossary problem. The model assumes a helper module exists, invents an env var, then emits a unified diff that looks locally coherent. You review the prose in the commit message and miss the undeclared path, because the patch is small and the story is confident. Cheap generation makes that miss expensive, because the next session treats the invented file as load-bearing architecture.

You do not need a new agent framework to interrupt that loop. You need a freeze point that a script can enforce without interpreting intent. The freeze point in this project is a JSON ledger you write first, then a checker that reads only the diff and the ledger. If the diff names something absent from the ledger, the patch never reaches your working tree.

Goal of this small project

You will build a tiny assumption-ledger toolbox you can copy into any repo in under an hour. The goal is not to make the model smarter. The goal is to make undeclared edits unmergeable, even when the generated rationale sounds careful.

Success looks like four concrete behaviors you can test:

1. A patch that only edits declared paths exits zero.
2. A patch that adds src/utils/helpers.py without a ledger entry exits non-zero.
3. A patch that exports DATABASE_URL when the ledger listed only PORT exits non-zero.
4. A patch that shells out to curl when the ledger allowed only pytest exits non-zero.

You should treat every number below as a fixture result from this sample project, not as a production benchmark. The checker is a gate, not a quality score.

The ledger contract

Create a directory that holds the contract, the checker, and the tests. Keep the surface small so you can read every line before you trust it.

assumption-ledger/
  ledger.schema.json
  ledger.json
  check_patch.py
  test_check_patch.py
  fixtures/
    ok.patch
    undeclared_path.patch
    undeclared_env.patch
    undeclared_cmd.patch

The ledger answers three questions before any model runs. Which paths may appear in a diff? Which environment names may appear in added lines? Which executable tokens may appear after a shell prompt or in subprocess calls?

{
  "version": 1,
  "allowed_paths": [
    "src/app.py",
    "src/config.py",
    "tests/test_app.py"
  ],
  "allowed_env": ["PORT", "LOG_LEVEL"],
  "allowed_commands": ["pytest", "python"]
}

You update that file in a human commit, not in the same generated patch you are about to check. If the agent needs a new path, you extend the ledger first, then regenerate. That ordering is the entire product of this case study.

Implementation

Parse only what a diff actually claims

The checker should not import your application, because application imports hide the assumptions you are trying to catch. Parse a unified diff, collect file headers, then scan added lines for env-like tokens and command-like tokens. Label the patterns as heuristics. They are strict enough for a weekend service and too coarse for a kernel tree.

#!/usr/bin/env python3
"""Fail closed when a unified diff violates assumption-ledger.json."""

from __future__ import annotations

import json
import re
import sys
from pathlib import Path

ENV_RE = re.compile(r"\b([A-Z][A-Z0-9_]{2,})\b")
CMD_RE = re.compile(r"\b(?:subprocess\.run\(|os\.system\(|popen\(|`)([a-zA-Z0-9._-]+)")
DIFF_FILE_RE = re.compile(r"^\+\+\+ b/(.+)$")


def load_ledger(path: Path) -> dict:
    data = json.loads(path.read_text(encoding="utf-8"))
    for key in ("allowed_paths", "allowed_env", "allowed_commands"):
        if key not in data or not isinstance(data[key], list):
            raise ValueError(f"ledger missing list field: {key}")
    return data


def collect_claims(diff_text: str) -> tuple[set[str], set[str], set[str]]:
    paths, envs, cmds = set(), set(), set()
    for line in diff_text.splitlines():
        header = DIFF_FILE_RE.match(line)
        if header:
            paths.add(header.group(1).strip())
            continue
        if not line.startswith("+") or line.startswith("+++"):
            continue
        added = line[1:]
        envs.update(ENV_RE.findall(added))
        cmds.update(m.group(1) for m in CMD_RE.finditer(added))
    return paths, envs, cmds


def violations(ledger: dict, diff_text: str) -> list[str]:
    paths, envs, cmds = collect_claims(diff_text)
    problems: list[str] = []
    allowed_paths = set(ledger["allowed_paths"])
    allowed_env = set(ledger["allowed_env"])
    allowed_cmds = set(ledger["allowed_commands"])
    for path in sorted(paths):
        if path != "/dev/null" and path not in allowed_paths:
            problems.append(f"undeclared path: {path}")
    for env in sorted(envs):
        if env not in allowed_env and env not in {"True", "False", "None"}:
            problems.append(f"undeclared env token: {env}")
    for cmd in sorted(cmds):
        if cmd not in allowed_cmds:
            problems.append(f"undeclared command: {cmd}")
    return problems


def main(argv: list[str]) -> int:
    if len(argv) != 3:
        print("usage: check_patch.py LEDGER.json PATCH.diff", file=sys.stderr)
        return 2
    ledger = load_ledger(Path(argv[1]))
    diff_text = Path(argv[2]).read_text(encoding="utf-8")
    problems = violations(ledger, diff_text)
    if problems:
        print("assumption ledger rejected this patch:")
        for item in problems:
            print(f"- {item}")
        return 1
    print("assumption ledger accepted this patch")
    return 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

You should notice what this script refuses to do. It does not call a model. It does not pretty-print architecture advice. It prints a list of contract breaks and a process status the rest of your pipeline can branch on.

Pin the four behaviors with fixtures

Keep the tests boring and file-based so a future agent cannot “fix” them by stubbing a network call. The following example is a proposed suite. Run it yourself before you treat the output as evidence.

from pathlib import Path

from check_patch import load_ledger, violations

ROOT = Path(__file__).parent
LEDGER = load_ledger(ROOT / "ledger.json")


def _diff(name: str) -> str:
    return (ROOT / "fixtures" / name).read_text(encoding="utf-8")


def test_declared_edit_is_clean():
    assert violations(LEDGER, _diff("ok.patch")) == []


def test_new_helper_module_is_rejected():
    problems = violations(LEDGER, _diff("undeclared_path.patch"))
    assert "undeclared path: src/utils/helpers.py" in problems


def test_new_env_token_is_rejected():
    problems = violations(LEDGER, _diff("undeclared_env.patch"))
    assert "undeclared env token: DATABASE_URL" in problems


def test_curl_is_rejected():
    problems = violations(LEDGER, _diff("undeclared_cmd.patch"))
    assert "undeclared command: curl" in problems

A minimal failing fixture looks like this. Save it as fixtures/undeclared_path.patch and keep the prose out of the checker.

diff --git a/src/utils/helpers.py b/src/utils/helpers.py
new file mode 100644
index 0000000..1111111
--- /dev/null
+++ b/src/utils/helpers.py
@@ -0,0 +1,3 @@
+def load_db():
+    import os
+    return os.environ["DATABASE_URL"]

A worked session you can reproduce

Label this session as a walkthrough, not as a live traffic report. You start from an empty feature branch, write the ledger, then ask any coding agent for a patch that “adds a health endpoint and retries.” The useful part is what you do with the diff, not which vendor produced it.

python3 -m venv .venv
. .venv/bin/activate
pip install pytest
python check_patch.py ledger.json fixtures/ok.patch
# expected sample: assumption ledger accepted this patch
python check_patch.py ledger.json fixtures/undeclared_path.patch
# expected sample: non-zero exit, undeclared path listed
pytest -q

If the agent returns a patch that creates src/utils/helpers.py, you do not debate naming in the chat transcript. You paste the diff into the checker, paste the violation list back, and require a ledger amendment before a second generation. That feedback is cheaper than a review comment written after the file already exists.

Where a model still helps

The model is useful after the checker fails, and only for the why. You can paste the violation list and ask for a three-bullet explanation of which assumption leaked, which ledger field would legalize it, and which test you should add. You should not ask the model to invent the next path set, because that recreates the original failure inside the contract file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you want the checker off your laptop, MonkeyCode’s free model access and free server option can host the same script and draft those failure notes after the process already exited non-zero. Remove that hosting choice and the ledger still works, which is the point of keeping the gate in plain Python.

Results you should expect

You should expect a binary outcome per patch, not a quality dashboard. In this sample project, declared edits pass, undeclared helpers fail, undeclared env tokens fail, and undeclared shell commands fail. That is the whole result table.

Diff fixture	Ledger state	Checker result
ok.patch	paths, env, commands already listed	accept
undeclared_path.patch	src/utils/helpers.py absent	reject
undeclared_env.patch	DATABASE_URL absent	reject
undeclared_cmd.patch	curl absent	reject

You will not get a measure of whether the accepted patch is elegant. You will get a measure of whether the patch stayed inside a surface you froze on purpose. If you need design quality, you still read the diff.

Limitations

This checker reads unified diffs and regular expressions, so it will miss assumptions that never appear as added text. Runtime imports, generated files, and string-built commands can slip through. Env detection will also false-positive on constants that look like MAX_RETRIES, which is why the allowlist must stay explicit and short.

The ledger can rot if you update it in the same commit as a sprawling generated change. You should require a separate human commit for ledger edits, or the contract becomes a diary of whatever the model already did. The tool also says nothing about license, security, or product correctness. It only answers whether the patch stayed inside a declared envelope.

Who should not use this

You should not use this as a substitute for code review on payment, identity, or safety-critical paths. You should not use it if your team cannot freeze a path list for even one service. You should not use it if you want the agent to discover architecture by creating files until tests pass. This workflow assumes you already know the files that may change during a tightly scoped task.

Skip it for throwaway spikes where undeclared files are the experiment. Run it when a cheap agent is about to touch a brownfield tree you must still explain next quarter.

Lessons learned

You learned that the expensive part of agent output is not token volume. It is the silent promotion of invented files into the next session’s context. A JSON allowlist and a thirty-line checker interrupt that promotion before review fatigue sets in.

Keep the sequence stable when you reuse the pattern. Freeze the ledger, generate the patch, run the checker, then optionally ask a model to explain a failure. If you invert that sequence, you are back to hoping the prose is honest. Hope is not a merge gate, and this case study exists so you do not have to pretend it is.