DEV Community

Dakota Liu
Dakota Liu

Posted on

Name Every File the Agent May Touch. Fail the Job If git Disagrees.

#ai #git #testing #python

I don't start with the model's explanation. I start with git status --porcelain. If a path shows up that I did not name, the job is already a no.

That is the whole method. Prompts are not reviews. Diff stories are not reviews. A file allowlist plus a scorer that can fail closed is a review you can run at 1 a.m. without lying to yourself.

Sound harsh? Good. The useful question is not whether a model already codes better than you. Can your loop prove the model stayed inside the job you actually opened?

What we are building

A from-zero workflow. Six stages. Each stage has a command and a verification. When we are done you have a tiny repo, a JSON manifest, a scorer, and a rule: nothing merges unless the scorer prints PASS.

I keep the agent off to the side until the gate exists. Then a free remote lane may propose a patch. The scorer still runs on my machine. The model does not get a vote.

Stage 1 — Make a repo that can cheat

Create an empty git repo with one real module and one test. Keep it boring. Boring is easier to score.

mkdir pricing-gate && cd pricing-gate
git init -q
python -m venv .venv
source .venv/bin/activate
pip install pytest -q
mkdir -p src tests
Enter fullscreen mode Exit fullscreen mode

Drop two files. Notice the bug. discount ignores percent on purpose.

# src/pricing.py
def discount(price: float, percent: float) -> float:
    if percent < 0 or percent > 100:
        raise ValueError("percent out of range")
    return price
Enter fullscreen mode Exit fullscreen mode 
# tests/test_pricing.py
from src.pricing import discount

def test_ten_percent_off():
    assert discount(100, 10) == 90
Enter fullscreen mode Exit fullscreen mode 
git add src/pricing.py tests/test_pricing.py
git commit -m "red: discount ignores percent"
python -m pytest tests/test_pricing.py -q
Enter fullscreen mode Exit fullscreen mode

Verification: pytest must fail. If it passes, you copied a fixed function by accident. Stop and look at return price. We need a red baseline so a wandering agent has a "helpful" excuse to touch extra files.

Also add a decoy the agent will love.

# src/legacy_pricing.py
# leftover. do not edit.
def discount(price, percent):
    return price * 0.5
Enter fullscreen mode Exit fullscreen mode

Temptation belongs in the tree. I commit it on purpose.

git add src/legacy_pricing.py
git commit -m "decoy: leftover module"
git ls-files
Enter fullscreen mode Exit fullscreen mode

Verification: you should see exactly three tracked source files: src/legacy_pricing.py, src/pricing.py, tests/test_pricing.py. Only two of those are in scope. Remember that.

Stage 2 — Write the manifest before the prompt

I refuse to prompt first. What would I even ask? "Please only edit the right files"? That is a wish, not a control.

{
  "job_id": "fix-discount-2026-09-17",
  "allowed_paths": [
    "src/pricing.py",
    "tests/test_pricing.py"
  ],
  "forbidden_globs": [
    "**/.env",
    "**/*.pem",
    "**/secrets.*",
    "src/legacy_pricing.py"
  ],
  "max_files_changed": 2,
  "max_hunk_lines": 40,
  "required_test": ["python", "-m", "pytest", "tests/test_pricing.py", "-q"],
  "forbidden_substrings": [
    "eval(",
    "exec(",
    "subprocess",
    "os.system",
    "pickle"
  ]
}
Enter fullscreen mode Exit fullscreen mode

Save it as job_manifest.json. Commit it. The manifest is product code. If it lives in a chat window, it will drift.

python -c "import json; json.load(open('job_manifest.json')); print('manifest ok')"
git add job_manifest.json && git commit -m "pin the job boundary"
Enter fullscreen mode Exit fullscreen mode

Verification: the one-liner prints manifest ok, and git log -1 --oneline shows the manifest commit. No model has been invited yet. That is the point.

Stage 3 — The scorer (this is the artifact)

Do not parse the model's markdown. Parse git. The script below is a worked example, not a published benchmark. I am not attaching timings, pass rates, or hardware claims, because I do not have numbers that would survive a week.

#!/usr/bin/env python3
"""score_diff.py — fail closed if the working tree wandered."""
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path


def run(cmd: list[str]) -> subprocess.CompletedProcess:
    return subprocess.run(cmd, text=True, capture_output=True)


def changed_paths() -> list[str]:
    out = run(["git", "diff", "--name-only", "HEAD"])
    untracked = run(["git", "ls-files", "--others", "--exclude-standard"])
    names = (out.stdout + untracked.stdout).splitlines()
    return sorted({n for n in names if n})


def hunk_line_count() -> int:
    diff = run(["git", "diff", "HEAD"]).stdout
    return sum(
        1
        for line in diff.splitlines()
        if line.startswith("+") or line.startswith("-")
    )


def main() -> int:
    manifest = json.loads(Path("job_manifest.json").read_text())
    allowed = set(manifest["allowed_paths"])
    forbidden_exact = {g for g in manifest["forbidden_globs"] if "*" not in g}

    changed = changed_paths()
    errors: list[str] = []

    if not changed:
        errors.append("no diff. the job did no work.")

    for path in changed:
        if path not in allowed:
            errors.append(f"unlisted path: {path}")
        if path in forbidden_exact:
            errors.append(f"forbidden path: {path}")

    if len(changed) > manifest["max_files_changed"]:
        errors.append(f"too many files: {len(changed)}")

    lines = hunk_line_count()
    if lines > manifest["max_hunk_lines"]:
        errors.append(f"diff too large: {lines} hunk lines")

    tree = ""
    for path in changed:
        p = Path(path)
        if p.exists() and p.is_file():
            tree += p.read_text(errors="replace")
    for token in manifest["forbidden_substrings"]:
        if token in tree:
            errors.append(f"forbidden token: {token}")

    test = run(manifest["required_test"])
    if test.returncode != 0:
        errors.append("required test failed")
        sys.stderr.write(test.stdout + test.stderr)

    print("changed:", ", ".join(changed) or "(none)")
    print("hunk_lines:", lines)
    if errors:
        print("RESULT: FAIL")
        for e in errors:
            print(" -", e)
        return 1
    print("RESULT: PASS")
    return 0


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode 
chmod +x score_diff.py
python score_diff.py; echo exit:$?
Enter fullscreen mode Exit fullscreen mode

Verification: on a clean tree after the last commit, you should see RESULT: FAIL and no diff. That is correct. A scorer that passes an empty tree is a toy.

Stage 4 — Prove the scorer can say no

Hand-edit the decoy. Pretend you are a helpful model.

echo '# cleanup' >> src/legacy_pricing.py
python score_diff.py; echo exit:$?
Enter fullscreen mode Exit fullscreen mode

Verification: FAIL, and unlisted path: src/legacy_pricing.py. Restore it.

git checkout -- src/legacy_pricing.py
Enter fullscreen mode Exit fullscreen mode

Now hand-fix the real function and only that file:

def discount(price: float, percent: float) -> float:
    if percent < 0 or percent > 100:
        raise ValueError("percent out of range")
    return price * (1 - percent / 100)
Enter fullscreen mode Exit fullscreen mode 
python score_diff.py; echo exit:$?
Enter fullscreen mode Exit fullscreen mode

Verification: RESULT: PASS. Pytest green. The two-path allowlist still holds because we did not need to edit the test. If you also tweak the test, that is fine — it is listed. If you "improve" legacy_pricing.py, it is not.

Would you have noticed the decoy edit in a 400-line chat log? I wouldn't. Not at speed.

Use this table when you are too tired to reread the script:

Event Scorer result Human next step
Empty diff FAIL Do not merge. The job did no work.
Unlisted or forbidden path FAIL Do not read the patch. Restore the tree.
Forbidden token (eval(, pickle, ...) FAIL Treat it as untrusted input.
Required test still red FAIL The allowlist is not a flashlight.
Listed paths, small hunk, tests green PASS Now you may read the diff.

Stage 5 — Only now invite a model

The gate exists. Now a model may propose a patch. I still do not paste my working tree into a random box. I keep secrets out. I keep .env out. I apply the patch on a throwaway branch and I run score_diff.py locally.

I use MonkeyCode here as the optional remote lane, not as the reviewer.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding-agent project. The operator-supplied facts I am willing to repeat are narrow: there is free model access, and there is a free server option. I am not going to invent model names, token quotas, hardware, or how long that free lane stays free. Those numbers go stale. A stale number in a tutorial is worse than no number.

What I actually do:

  1. Branch from main.
  2. Let the remote lane propose a patch against src/pricing.py and tests/test_pricing.py only.
  3. Apply that patch on my machine.
  4. Run python score_diff.py.
  5. Delete the branch if the scorer prints FAIL.

If you already run that free server for the proposal step, point this scorer at the resulting tree. The value is not "a model wrote code." The value is that a wandering edit dies before you read it.

git checkout -b agent-fix-discount
# apply the remote patch, then:
python score_diff.py
git diff --stat
Enter fullscreen mode Exit fullscreen mode

Verification: expect either PASS plus a tiny --stat, or FAIL plus a path you did not list. There is no third outcome you should merge.

Stage 6 — Make the gate the only entry

A scorer you remember to run is a hobby. A scorer git runs for you is a workflow.

I do not install a global hook on day one. Why? Because a teammate fixing a docs typo should not fight your agent allowlist. Scope it to agent branches.

cat > .git/hooks/pre-commit <<'EOF'
#!/bin/sh
case "$(git rev-parse --abbrev-ref HEAD)" in
  agent-*) python score_diff.py || exit 1 ;;
esac
EOF
chmod +x .git/hooks/pre-commit
Enter fullscreen mode Exit fullscreen mode

Verification:

git checkout agent-fix-discount
echo '# cleanup' >> src/legacy_pricing.py
git add src/legacy_pricing.py
git commit -m "should fail"
# expect the hook to block, then:
git checkout -- src/legacy_pricing.py
Enter fullscreen mode Exit fullscreen mode

If the commit goes through, the hook never ran. Check ls -l .git/hooks/pre-commit before you blame the scorer.

Limitations

This does not read intent. A model can still write the wrong formula inside src/pricing.py if the test is weak. The scorer will PASS a confident, listed, tiny, wrong patch. That is not a scorer bug. That is your test bug.

Globs in the example are underpowered. Exact forbidden files are treated as exact paths. If you copy this, replace that branch with fnmatch or pathlib.PurePath.match. Label that as homework.

Lockfiles, generated code, and formatter noise will blow max_hunk_lines. Either list those files or stop auto-formatting inside the agent job. Pick one. Do not argue with the scorer in Slack.

A free remote server is still a remote server. Free does not mean "upload the company wiki." An allowlist on my laptop does not protect a tarball I should never have packed.

Who should not use this

Skip it if you are in a true spike and you want the model to rummage. Skip it if the job is "find the bug," because the bug may not live in the two files you named. Skip it if you cannot write a failing test. The allowlist is a fence, not a flashlight.

If your review process already diffs against a ticket's listed paths, you already have this idea. You might only need the scorer.

What I want you to copy

Not a prompt. The order.

  1. Manifest.
  2. Scorer that fails closed.
  3. Proof that the scorer says no.
  4. Then a model, maybe on a free remote lane.
  5. Merge only on PASS.

I still read the passing diff. I just refuse to read the failing ones. That is the whole time save. Everything else is commentary.

Top comments (0)