Grade AI-Generated Patches by the File System They Leave Behind, Not by Diff Size

#ai #cicd #security #testing

Why this is worth reading. A generated patch can pass every unit test and still leave a measurable mess: a new .env.example with your local path, a symlink that points outside the checkout, or a mode change that makes a deploy script executable. Diff size and test output do not show any of that. This article gives you a reproducible state gate that treats the file system after a candidate run as a first-class test artifact.

A model wins trust when it can explain its diff. The file system is where an unexplained side effect shows up first. If you only grade patches by diff size and test status, you are grading the part the model wanted you to see. The harness below grades the part it left behind.

Snapshot, replay, compare to expected state

Run the candidate in a disposable checkout, fingerprint every file and symlink before and after, then compare the changed set against a list of paths the patch is allowed to change. Anything outside that list fails the gate.

The script records content hashes, symlink targets, and permission modes. It skips .git, caches, virtualenvs, and node_modules so normal tooling noise does not drown the signal. You can adjust SKIP_DIRS if your project stores code in a directory with one of those names.

#!/usr/bin/env python3
import hashlib
import os
import shlex
import subprocess
import sys
from pathlib import Path

SKIP_DIRS = {'.git', '__pycache__', '.venv', 'node_modules'}


def fingerprint(root: Path) -> dict[Path, tuple[str, str]]:
    state = {}
    for path in root.rglob('*'):
        relative = path.relative_to(root)
        if any(part in SKIP_DIRS for part in relative.parts):
            continue
        if path.is_symlink():
            digest = hashlib.sha256()
            digest.update(b'symlink:')
            digest.update(os.readlink(path).encode('utf-8'))
            state[relative] = (digest.hexdigest(), 'symlink')
        elif path.is_file():
            digest = hashlib.sha256()
            with path.open('rb') as handle:
                for block in iter(lambda: handle.read(65536), b''):
                    digest.update(block)
            mode = oct(path.stat().st_mode & 0o777)
            state[relative] = (digest.hexdigest(), mode)
    return state


def main() -> int:
    if len(sys.argv) < 3:
        print('usage: filesystem_gate.py ROOT COMMAND [EXPECTED_PATH ...]', file=sys.stderr)
        return 2
    root = Path(sys.argv[1]).resolve()
    command = shlex.split(sys.argv[2])
    expected = {Path(item).relative_to(root) for item in sys.argv[3:]}

    before = fingerprint(root)
    subprocess.run(command, cwd=root)
    after = fingerprint(root)

    changed: set[Path] = set()
    changed.update(p for p in after if after[p] != before.get(p))
    changed.update(p for p in before if p not in after)
    unexpected = sorted(changed - expected)

    if unexpected:
        print('Unexpected filesystem changes:')
        for path in unexpected:
            print(f'- {path}')
        return 1
    print('Filesystem state gate passed.')
    return 0


if __name__ == '__main__':
    raise SystemExit(main())

Pass the command as one string and the allowed paths as the remaining arguments. To review a staged patch, derive the expected paths from the diff itself:

CHANGED_FILES=$(git diff --cached --name-only | paste -sd ' ' -)
python3 filesystem_gate.py . 'bash ./apply_patch.sh' $CHANGED_FILES

When the patch edits only a tracked source file, the gate stays green. When the same patch also writes an unexpected file, changes a permission bit, or creates a symlink, the gate prints the changed path and exits non-zero.

Make the gate useful, not noisy

Define expected paths from the patch, not from observation after the fact. If you add every file the candidate created to the allowlist without an explanation, this becomes an echo chamber.

Use a decision table when triaging a failure:

For generated files that are legitimate, commit an allowlist file instead of passing a moving list on the command line. Keep the allowlist small enough to review in a pull request.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. According to the operator, MonkeyCode provides free model access and a free server option. That capacity can be useful here because you can generate a candidate patch from the free model and run the gate on the free server, without spending paid CI time on exploratory revisions. Verify current limits before depending on the free tier; this article does not assume any particular model name, quota, or retention policy.

The workflow stays the same whether the candidate comes from a free model or a paid one: snapshot, replay, compare, then review the unexpected paths.

Test the gate before you trust it

Run these five checks against a scratch repo:

Commit a known-good patch. The gate should pass.
Add a candidate script that runs a harmless command and writes unexpected.log. The gate should fail on unexpected.log.
Add a symlink to /etc/hosts from inside the workspace. The gate should fail on that path.
Delete a tracked file without putting it in the expected list. The gate should fail on that path.
Change a source file and add it to the expected list. The gate should pass.

If check 2 passes instead of failing, your skip list is hiding the file. If check 5 fails, your expected path format does not match relative_to(root).

Limitations and who should skip this

This is a second signal, not a sandbox. It sees only changes under the workspace root. It cannot catch network calls, memory-resident behavior, daemons that keep running, or writes to absolute paths outside the workspace. Pair it with a no-network sandbox or syscall trace when the command is allowed to touch external services.

It also skips directories named .git, __pycache__, .venv, and node_modules. If your candidate legitimately changes files in those directory names, edit SKIP_DIRS or use a dedicated export without those names. The gate can be noisy for commands that create timestamps, caches, or random output; disable those features before capture.

Do not use this gate if you cannot define an expected path set. For a broad refactor that intentionally rewrites hundreds of files, the expected list from git diff --cached --name-only still works, but a patch that writes to arbitrary external paths will only produce false positives. If you already have a read-only worktree gate, add this as a reporting layer rather than a replacement.

Run one candidate through the gate this week. When it flags an unexpected file, add that path to your allowlist only after you can explain why it exists.

DEV Community