Route an Incoming OSS Patch by Blast Radius Before a Model Drafts the Review

A review queue that looks calm and is not

On a Monday morning, the maintainer of a small Python library opens the pull request list and finds eleven items waiting. Four are dependency bumps, three fix docstring typos, two add small features, one only rewrites tests, and one rewrites the argument parser from scratch. The natural instinct is to open the parser rewrite first, because it is the largest and most intimidating diff in the queue.

That instinct is expensive and frequently wrong. A three-line change to a lockfile can break every downstream install, while a two-hundred-line refactor inside a well-covered module may be exactly what the existing tests already describe. Review attention is finite, and it should follow risk rather than diff size.

The workflow below is a composite illustration rather than a report from one named repository. It defines a small routing procedure that assigns each incoming patch a tier, then decides which tiers may receive model-drafted review notes and which must not. Every command in it is ordinary git, so it runs against any clone without extra services.

Three signals that predict review cost

Diffs carry a few cheap, language-agnostic signals that correlate with how much human attention a patch will consume. None of them is a correctness signal, and none of them replaces reading the code.

Signal	Why it raises review cost	Cheap check
Build or dependency files	Changes install resolution, CI images, and lockfile integrity for everyone	Basename match against a known build-file set
Public surface paths	Alters behaviour callers depend on, including undocumented callers	Path prefix match (src/, lib/, pkg/)
No test delta	Removes the fastest way to falsify the claimed behaviour	Marker match (tests/, spec/, test_)
Wide file spread	Suggests a cross-cutting change with more integration edges	Count of files in --numstat
Large net change	Longer to read, more places for a silent regression to hide	Sum of added and removed lines

The first three signals are properties of what the patch touches, and the last two are properties of how much it touches. Combining them produces a coarse but useful ordering, and the ordering is what the routing script encodes.

A routing script for a fresh clone

The script below reads a diff between a base ref and a head ref, scores it, and prints a tier. It has no dependencies outside the standard library, and it is written as a starting point rather than a finished product, so pinning your own path globs and build-file set is expected.

#!/usr/bin/env python3
"""Route an incoming OSS pull request by blast radius.

Usage:
    python review_route.py --base origin/main --head pr-4821
"""
from __future__ import annotations

import argparse
import subprocess

PUBLIC_PREFIXES = ("src/", "lib/", "pkg/", "internal/api/")
BUILD_FILES = {
    "pyproject.toml", "setup.py", "setup.cfg", "requirements.txt",
    "package.json", "package-lock.json", "Cargo.toml", "Cargo.lock",
    "go.mod", "go.sum", "Makefile", "Dockerfile",
}
TEST_MARKERS = ("/tests/", "/test/", "spec/", "test_", "_test.")


def git(*args: str) -> str:
    proc = subprocess.run(["git", *args], check=True, capture_output=True, text=True)
    return proc.stdout


def numstat(base: str, head: str) -> list[tuple[int, int, str]]:
    rows = []
    for line in git("diff", "--numstat", f"{base}...{head}").splitlines():
        added, removed, path = line.split("\t", 2)
        rows.append((int(added) if added.isdigit() else 0,
                     int(removed) if removed.isdigit() else 0,
                     path))
    return rows


def classify(rows):
    added = sum(r[0] for r in rows)
    removed = sum(r[1] for r in rows)
    paths = [r[2] for r in rows]
    public = any(p.startswith(PUBLIC_PREFIXES) for p in paths)
    build = any(p.split("/")[-1] in BUILD_FILES for p in paths)
    tests = any(any(m in p for m in TEST_MARKERS) for p in paths)
    return added, removed, public, build, tests, len(paths)


def tier_for(files, added, removed, public, build, tests) -> tuple[int, str]:
    score = 0
    score += 3 if build else 0
    score += 2 if public else 0
    score -= 2 if tests else 0
    score += 1 if files > 5 else 0
    score += 1 if (added + removed) > 200 else 0
    if score >= 4:
        return score, "T0 human-only review"
    if score >= 2:
        return score, "T1 human review; model may draft notes"
    return score, "T2 model draft first; human sign-off"


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--head", default="HEAD")
    args = ap.parse_args()

    rows = numstat(args.base, args.head)
    added, removed, public, build, tests, files = classify(rows)
    score, tier = tier_for(files, added, removed, public, build, tests)
    print(f"base={args.base} head={args.head}")
    print(f"files={files} +{added} -{removed}")
    print(f"public_surface={public} build_files={build} tests_touched={tests}")
    print(f"score={score} tier={tier}")


if __name__ == "__main__":
    main()

Sample output

$ python review_route.py --base origin/main --head pr-4821
base=origin/main head=pr-4821
files=7 +412 -96
public_surface=True build_files=False tests_touched=True
score=2 tier=T1 human review; model may draft notes

A one-line lockfile bump from the same queue scores differently, because build=True adds three points and tests=False adds two more. That inversion is the point of the exercise: the smallest diff in the queue is not the cheapest one to review.

How to read the tiers in practice

• T0 — human-only review. Dependency and lockfile edits, authentication or authorization paths, CI workflow files, and any patch that deletes or weakens tests. No model-drafted note should precede a human read here.
• T1 — human review, model notes allowed. Public surface changes that ship with tests. A model may summarize the diff or list untested branches, but the review comment itself is written by a person.
• T2 — model draft first. Test-only changes, internal helpers, docstring fixes, and small additions behind private names. A draft is useful here because the acceptance criteria are usually mechanical.

Two rules keep the tier list honest. First, a maintainer's knowledge of their own codebase always overrides the score, because the score has never read the code. Second, the tier decides who drafts the first comment, never whether the patch gets merged.

Where free model access fits in the queue

Drafting review notes for a T2 patch is a low-stakes use of a hosted model, since the reviewer still reads the diff before responding. MonkeyCode is an open-source project whose operator states it offers free model access and a free server option, which matters mainly for contributors who do not want a paid API key attached to routine review work.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two boundaries are worth stating plainly. Model access is used only for T1 and T2 drafting, and T0 patches are read by a human before anything else happens. Availability, model selection, and any usage limits are operator-stated and change over time, so confirm the current terms on the project page rather than trusting a figure quoted in an older post.

Two runs, then mechanical verification

The weakest part of model-drafted review notes is that they cite symbols and files that may not exist. A cheap protocol removes most of that noise, and it has three steps.

1. Run the drafting prompt twice with different phrasing or sampling settings, and collect both answer sets as JSONL.
2. Keep only claims that appear in both runs with the same file and the same symbol, which discards most one-off hallucinations.
3. Verify every surviving claim mechanically against the patch before it reaches the contributor.

The drafting prompt should require one JSON object per line, with a file, a symbol, and a short note, and it should forbid claims that cannot name a symbol. Claims without a symbol are opinions, and opinions belong in the human review, not in an automated pass.

The verification script

The script below keeps only claims that survive two mechanical checks: the file appears in the patch, and the symbol resolves at the head revision. It is deliberately narrow, and it proves existence rather than correctness.

#!/usr/bin/env python3
"""Keep only model review claims that resolve mechanically against the patch.

Input JSONL, one claim per line:
    {"file": "src/parser.py", "symbol": "parse_args", "note": "..."}
"""
from __future__ import annotations

import argparse
import json
import subprocess
import sys


def git(*args: str):
    proc = subprocess.run(["git", *args], capture_output=True, text=True)
    return proc.returncode, proc.stdout


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("claims")
    ap.add_argument("--base", default="origin/main")
    ap.add_argument("--head", default="HEAD")
    args = ap.parse_args()

    _, changed_raw = git("diff", "--name-only", f"{args.base}...{args.head}")
    changed = set(changed_raw.split())

    kept, dropped = [], []
    with open(args.claims, encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if not line:
                continue
            claim = json.loads(line)
            path, symbol = claim["file"], claim["symbol"]
            reasons = []
            if path not in changed:
                reasons.append("file not in patch")
            rc, hits = git("grep", "-n", "-F", symbol, args.head, "--", path)
            if rc != 0 or not hits.strip():
                reasons.append("symbol not found at head")
            record = {**claim, "rejected_because": reasons}
            (dropped if reasons else kept).append(record)

    print(json.dumps({"kept": kept, "dropped": dropped}, indent=2))
    print(f"kept={len(kept)} dropped={len(dropped)}", file=sys.stderr)


if __name__ == "__main__":
    main()

What the filter does not catch

A symbol that exists can still be described incorrectly, and the filter says nothing about whether a suggested change is right. Claims that survive both checks are candidates for human reading, not conclusions, and the rejection list is often the more informative output because it shows which class of claims the model tends to fabricate.

Limitations and who should not adopt this

• Very small repositories. The script overhead exceeds the review time saved, and a maintainer who already knows every file gains nothing from a tier column.
• Repositories without tests. The tests=False penalty has no meaning when no test suite exists, so the score degenerates into a diff-size rank.
• Monorepos with sparse paths. Public prefix matching needs per-package configuration, and a single global glob set will misclassify most of the tree.
• Security-sensitive patches. A tier score is the wrong instrument for vulnerability fixes, and those should bypass the routing pipeline entirely.
• Projects that forbid hosted tooling. If repository policy excludes third-party model services, run the routing script alone and stop before the drafting step.

The scripts above are a starting point rather than a benchmarked tool, and they have not been validated across every language layout. Treat the build-file set and prefix list as configuration to maintain, not as constants.

A twenty-minute checklist

1. Add review_route.py to a scripts directory and adjust PUBLIC_PREFIXES and BUILD_FILES for your language.
2. Run it against the three oldest open pull requests and compare the tiers with your own intuition.
3. Fix the scoring only where your intuition is consistently right and the script is consistently wrong.
4. Draft notes for one T2 patch twice, and diff the two answer sets before verifying anything.
5. Run verify_claims.py and read the rejected list first, because it reveals the failure mode fastest.
6. Record which tiers produced useful notes, and drop model drafting from any tier that did not earn its place.

The procedure is deliberately unglamorous, and its value comes from being repeatable on a Monday morning when eleven patches are waiting. If you want to try the drafting step without wiring up local inference, the free model access and free server option described by MonkeyCode are a reasonable place to start, beginning with the T2 queue and a small rejected-claims list.