Route the Diff Before You Read It: A File-Level Triage Script for Agent-Generated PRs

Most agent-generated PRs are not reviewed badly because reviewers are lazy. They are reviewed badly because reviewers read in the wrong order: the summary first, then the hunks top to bottom, and the manifest or CI change at the bottom of the diff where attention is already gone.

Fix the order and the review gets cheaper. The file list is a smaller, more truthful artifact than the summary the agent wrote about itself, so route the review from the file list.

The failure surface is in the file list

In my experience triaging agent PRs, the risky changes cluster in a handful of paths, and they are almost never the ones the agent highlights:

• Lockfiles and build output edited directly, because the agent "fixed" a resolution error by hand.
• CI workflow files, where a permission, trigger, or secret reference changed by one line.
• Config and env defaults, where a new key silently changes runtime behavior.
• The tests themselves, where the oracle moved to meet the code.
• Ordinary source, which is usually the safest part and the part reviewers spend 80% of their time on.

That last bullet is the whole problem. Review effort is inversely correlated with risk.

A triage script you can run in one command

The script below reads git diff --name-status between two refs, matches each path against an ordered rule list, and prints a Markdown table with a bucket and the evidence the PR must carry. First match wins, so specific patterns go above general ones.

#!/usr/bin/env python3
"""triage.py - bucket every file in a PR diff before anyone reads a hunk.

    python triage.py origin/main HEAD
    python triage.py origin/main HEAD --fail-on REVERT
"""
import argparse
import fnmatch
import subprocess
import sys

# glob, bucket, reason, evidence the PR must carry
RULES = [
    ("package-lock.json",     "REVERT", "lockfile edited by hand",      "matching package.json diff"),
    ("*.lock",                "REVERT", "lockfile edited by hand",      "matching manifest diff"),
    ("*.min.js",              "REVERT", "build output committed",       "source file + build command"),
    ("dist/*",                "REVERT", "build output committed",       "source file + build command"),
    (".github/workflows/*",   "TEST",   "permissions/triggers/secrets", "before and after permissions block"),
    ("Dockerfile",            "TEST",   "base image and entrypoint",    "built image + start command"),
    ("*.env*",                "TEST",   "config surface",               "every key added, with default"),
    ("*migrations/*",         "TEST",   "irreversible state change",    "forward run and rollback run"),
    ("test_*.py",             "TEST",   "the oracle itself changed",    "test fails on the parent commit"),
    ("tests/*",               "TEST",   "the oracle itself changed",    "test fails on the parent commit"),
    ("*.py",                  "TRUST",  "ordinary source",              "existing unit tests still green"),
    ("*.ts",                  "TRUST",  "ordinary source",              "existing unit tests still green"),
]

LARGE_DIFF_LINES = 400


def changed(base, head):
    out = subprocess.run(
        ["git", "diff", "--name-status", "-M", f"{base}...{head}"],
        capture_output=True, text=True, check=True,
    ).stdout
    rows = []
    for line in out.splitlines():
        parts = line.split("\t")
        if len(parts) >= 2:
            rows.append((parts[0], parts[-1]))
    return rows


def numstat(base, head):
    out = subprocess.run(
        ["git", "diff", "--numstat", f"{base}...{head}"],
        capture_output=True, text=True, check=True,
    ).stdout
    counts = {}
    for line in out.splitlines():
        added, removed, path = line.split("\t")
        if added.isdigit() and removed.isdigit():
            counts[path] = int(added) + int(removed)
    return counts


def classify(path, size):
    for pattern, bucket, reason, evidence in RULES:
        if fnmatch.fnmatch(path, pattern) or fnmatch.fnmatch(path.rsplit("/", 1)[-1], pattern):
            if size > LARGE_DIFF_LINES and bucket == "TRUST":
                return "TEST", f"{reason}; {size} changed lines", "read the hunks in order"
            return bucket, reason, evidence
    return "TRUST", "no rule matched", "existing tests still green"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("base")
    ap.add_argument("head")
    ap.add_argument("--fail-on", choices=["REVERT", "TEST"], default=None)
    args = ap.parse_args()

    sizes = numstat(args.base, args.head)
    order = {"REVERT": 0, "TEST": 1, "TRUST": 2}
    rows = []
    for status, path in changed(args.base, args.head):
        bucket, reason, evidence = classify(path, sizes.get(path, 0))
        rows.append((order[bucket], bucket, status, path, reason, evidence))
    rows.sort()

    print("| bucket | status | path | why | evidence required |")
    print("|---|---|---|---|---|")
    for _, bucket, status, path, reason, evidence in rows:
        print(f"| {bucket} | {status} | `{path}` | {reason} | {evidence} |")

    counts = {b: sum(1 for r in rows if r[1] == b) for b in order}
    print(f"\nREVERT={counts['REVERT']} TEST={counts['TEST']} TRUST={counts['TRUST']}")

    if args.fail_on and counts[args.fail_on]:
        sys.exit(1)


if __name__ == "__main__":
    main()

Run it against the PR head:

git fetch origin pull/4123/head:pr-4123
python triage.py origin/main pr-4123 --fail-on REVERT

Example output shape (illustrative, not a measurement from a specific repo):

| bucket | status | path                       | why                        | evidence required              |
| REVERT | M      | package-lock.json          | lockfile edited by hand    | matching package.json diff     |
| TEST   | M      | .github/workflows/test.yml | permissions/triggers       | before and after permissions   |
| TEST   | M      | tests/test_billing.py      | the oracle itself changed  | test fails on the parent commit|
| TRUST  | M      | src/billing/retry.py       | ordinary source            | existing unit tests green      |

REVERT=1 TEST=2 TRUST=1

The decision table

Bucket	What to trust	What to test	Revert trigger
REVERT	Nothing. Treat the file as unreviewable until a generator produced it.	Regenerate from the manifest (npm ci, poetry lock --no-update) and diff the result against the PR.	Any hand-edited lockfile, committed build output, or vendored blob with no source.
TEST	The intent, not the effect.	Run the thing the file controls: CI on a throwaway branch, the container locally, the migration forward and back.	A permission widened, a trigger added on pull_request_target, a new env key with a hardcoded default, a test weakened in the same commit as the fix.
TRUST	The shape of the change, subject to green tests.	Existing suite plus one new assertion that would fail on the parent commit.	Logic that only passes because a TEST-bucket file changed.

Reading order: REVERT, then TEST, then TRUST

1. Clear REVERT first. These are five-minute decisions with binary outcomes. A hand-edited lockfile is not a discussion; ask for a regenerate commit.
2. Spend the real time on TEST. This is where agent PRs actually break: one line in a workflow file, one default in a config loader.
3. Skim TRUST. If the tests are green and the file is ordinary application code, the marginal value of a line-by-line read is low. Read the function signatures and the deleted lines, not every hunk.

This inverts the default behavior, where reviewers start at the top of a diff that is usually sorted alphabetically and exhaust themselves on src/ before reaching .github/.

Three checks before you read any hunk text

BASE=origin/main; HEAD=pr-4123

# 1. Lockfile moved without the manifest, or the reverse
 git diff --name-only $BASE...$HEAD -- 'package.json' 'package-lock.json' 'pyproject.toml' 'poetry.lock'

# 2. CI permission and trigger delta
git diff $BASE...$HEAD -- .github/workflows/ | grep -E '^[+-].*(permissions|pull_request_target|secrets\.|runs-on)'

# 3. Config and credential surface
git diff $BASE...$HEAD -- '*.env*' '*.yaml' '*.yml' | grep -E '^\+.*(KEY|TOKEN|SECRET|PASSWORD)'

If check 2 prints a line starting with +, stop and read that hunk before anything else. A single added permissions: write-all outranks every code improvement in the PR.

Where free model access and a free server actually fit

Triage is deterministic work, so it is a poor place to spend a paid model call. It is a reasonable place to spend a free one.

Two narrow uses, both of which keep the model out of the approval path:

• Drafting review comments from the bucket table only. Pass the script's output — paths, buckets, reasons, no source code — and ask for one comment per REVERT entry. The model never sees the diff, so it cannot hallucinate an approval.
• Running the script itself. Triage is a cron job or a webhook handler, not an interactive tool. That is exactly the workload a small always-on box handles well.

The MonkeyCode open-source project advertises free model access and a free server option. At the time of writing the operator states the free allotment is 10M tokens plus a free server; those terms are time-sensitive, so confirm them in the repository before you design a pipeline around them. For this workflow the free model is enough because the model is only formatting a table that triage.py already produced. It is not asked to judge the PR.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A minimal webhook handler that runs the script and posts the table as a PR comment is about 30 lines; the interesting part is the shell call, not the transport:

python triage.py "$BASE" "$HEAD" --fail-on REVERT > triage.md || echo "BLOCKED" >> triage.md
gh pr comment "$PR" --body-file triage.md

Fail the check on REVERT and the PR cannot merge until a human regenerates the lockfile. That is a review gate, not a model judgment.

Limitations and who should skip this

• Generated-code-heavy repos. If half the tree is codegen output, the rule list fills with noise and every PR trips REVERT. Match on the generator config instead of the output paths.
• Monorepos without ownership boundaries. The script has no idea which team owns which path. Add CODEOWNERS parsing before you trust the TRUST bucket.
• Teams without a green test suite. The TRUST bucket assumes tests are a usable oracle. If they are not, everything is TEST and the triage buys you nothing.
• PRs that are genuinely architectural. A 40-file refactor does not need bucketing; it needs a design review with a human in the room.

If you want to try the free-model and free-server path, start by running triage.py on your last ten merged agent PRs and see whether the REVERT bucket would have caught anything. That answer is more useful than any review checklist, including this one.