Reproduce, Minimize, Then Review: A Three-Pass Loop for Open-Source Patches

A pull request arrives with a confident description, three changed files, and a passing CI badge that turns green only on the author's branch. The failing test that motivated the change is nowhere in the diff, and the reviewer is asked to judge correctness from prose alone. This loop is a composite scenario, not a report of one specific repository, but it is common enough that the review order matters more than the review tools.

The loop below reverses the usual habit. Reproduce first, minimize second, and only then ask a model for review, with the model's output treated as a list of claims that must be verified against files and logs. Free model access helps at the last pass, not the first, and that ordering is the whole point.

Why the review order beats the review tool

Most review failures are not reasoning failures; they are context failures. A reviewer who has not reproduced the bug cannot tell which hunks are load-bearing and which are decorative refactors. A model reading the same diff inherits that ambiguity and amplifies it with fluent guesses.

Reproduction produces two artifacts that constrain everything after it: a deterministic command with one exit code, and a captured log. Minimization produces a third: a patch whose hunks can each be tied to a reason. A review request built from those artifacts is short, checkable, and hard to bluff.

Pass 0: pin the reproduction before reading the diff

The goal is a command that fails on the base commit for the stated reason, on a clean clone, with a pinned toolchain. Versions inherited from a workstation make failures unrepeatable, so the script records rather than assumes them.

#!/usr/bin/env bash
# repro.sh - deterministic reproduction of the reported failure
set -euo pipefail

REPO="${REPO:-https://github.com/example/project}"
BASE_SHA="${BASE_SHA:?set BASE_SHA to the commit under test}"
WORK="$(mktemp -d)"

git clone --quiet --no-checkout "$REPO" "$WORK/src"
cd "$WORK/src"
git checkout --quiet "$BASE_SHA"

# Pin everything that can drift between machines.
export PYTHONHASHSEED=0
export SOURCE_DATE_EPOCH=0
python3 -m pip install --quiet --require-hashes -r requirements-test.txt

set +e
python3 -m pytest -x -q tests/test_reported_case.py 2>&1 | tee "$WORK/test.log"
STATUS=${PIPESTATUS[0]}
set -e

echo "exit=${STATUS} base=${BASE_SHA}" | tee "$WORK/repro.result"
exit "$STATUS"

The --require-hashes flag matters because it turns a floating dependency into a recorded one. If the project cannot produce hashed requirements, pin exact versions in a lockfile and commit it alongside the script. The captured test.log becomes the only acceptable evidence for claims made later in the review.

A reproduction is accepted only when all four conditions hold:

1. It runs from a fresh clone with no local state.
2. It fails on BASE_SHA with the message quoted in the issue.
3. It passes on the patched commit, using the same command.
4. Reverting the patch restores the failure, byte-identical in the assertion line.

Pass 1: minimize the patch and prove each hunk earns its place

A large patch hides its own bugs. Two cheap gates catch most of them before any model sees the diff: a diffstat ceiling agreed with the maintainer, and a revert test.

BASE_SHA=origin/main
git fetch --quiet origin "$BASE_SHA"

git diff --numstat "$BASE_SHA"...HEAD \
  | awk '{add+=$1; del+=$2; printf "%s +%s -%s\n", $3, $1, $2} END {print "total +", add, " -", del}'

# Revert test: the patch must be the reason the test passes.
git diff --binary "$BASE_SHA"...HEAD > /tmp/patch.diff
git checkout --quiet "$BASE_SHA"
git apply -R /tmp/patch.diff
BASE_SHA="$BASE_SHA" ./repro.sh || echo "revert restored the failure, as expected"

When the revert does not restore the failure, one of three things is true: the test never exercised the changed path, the fix lives in an untracked file, or the environment drifted. Each of those is a blocker, and none of them is visible from reading the diff.

Refactors mixed into a fix belong in a separate commit, because they change what the reviewer must hold in mind. Splitting them is not style policing; it is a way to make the revert test meaningful.

Pass 2: build a scoped review brief instead of pasting a diff

A raw diff invites a raw summary. A brief that pairs the diff with the failure lines and the changed-file inventory invites a bounded answer. The script below emits that brief as JSON, which keeps the prompt small and the claims enumerable.

#!/usr/bin/env python3
"""build_brief.py - emit a review brief from patch stats and the test log."""
import json, subprocess, sys
from pathlib import Path

base, log_path = sys.argv[1], Path(sys.argv[2])
test_log = log_path.read_text(errors="replace")

numstat = subprocess.run(
    ["git", "diff", "--numstat", f"{base}...HEAD"],
    capture_output=True, text=True, check=True,
).stdout

files = []
for row in numstat.splitlines():
    added, deleted, path = row.split("\t")
    files.append({"path": path, "added": int(added), "deleted": int(deleted)})

failure_lines = [
    line for line in test_log.splitlines()
    if "FAILED" in line or line.startswith("E ")
][:20]

print(json.dumps({
    "base": base,
    "files_touched": files,
    "failure_lines": failure_lines,
    "questions": [
        "Which hunks are required for the failing assertion to pass?",
        "Which hunks are behaviour-neutral and belong in another commit?",
        "Does any hunk change behaviour outside the failing test?",
    ],
}, indent=2))

The accompanying prompt should impose a citation format, because an answer without locations cannot be audited. A workable instruction set uses four rules: cite path:line for every claim, quote the failing assertion the hunk addresses, mark unsupported statements as UNVERIFIED, and output one row per hunk rather than one paragraph per file.

Auditing the claims that come back

Model output is a hypothesis list. Each row is copied into a claim table and closed only with evidence from the repository or the log.

Claim	Cited location	Evidence	Verdict
Guard clause prevents the null deref	src/parse.py:212	test.log assertion line matches	accepted
Extra branch preserves old behaviour	src/parse.py:240	no test covers the branch	needs test
Helper rename is safe	src/util.py:8	grep finds third-party import	rejected

Rows marked UNVERIFIED are deleted rather than debated. A confident sentence with no location costs more reviewer time than the original bug report.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that offers free model access and a free server option, per the operator. In this loop, the free server is the natural host for repro.sh and the review job, because a disposable machine removes the local-state variable that makes reproductions unrepeatable. The free models serve the Pass 2 role: bounded questions over a small brief, with citations required.

Confirm the current terms, limits, and availability before depending on either offer, since free-tier conditions change and this article makes no claim about quotas, hardware, or duration. The workflow above stays valid if the free options disappear tomorrow, which is the test any tooling recommendation should pass. Anyone working through the same loop may find the free model access and free server worth trying against their own repository's reproduction script.

When model review is adequate, and when it is not

Patch type	Free-model review	Human gate
Mechanical rename, formatting	useful	diffstat skim
Test-only change	useful	confirm failure on base
Bug fix with a new assertion	useful with brief	revert test plus one maintainer read
Dependency bump	limited	changelog and lockfile diff
Concurrency or scheduling	not reliable	stress test under load
Auth, crypto, licence, CLA	not reliable	specialist and legal review

The pattern is consistent: the more a patch depends on behaviour that no test captures, the less a text-only reviewer can contribute.

Limitations and who should skip this

Text-only review never executes the patch, so a model cannot confirm that the suite passes or that a race is absent. Long diffs get truncated, and truncation silently removes the hunks most likely to be wrong. Line numbers can drift between the model's reading and the current branch, so citations need re-checking after any rebase.

Teams working on security-sensitive code, unreviewed production changes, or repositories whose contribution rules restrict AI-assisted review should use the reproduction and revert gates while skipping the model pass entirely. Some projects require disclosure when AI assistance is used in preparing a contribution, and that rule overrides convenience.

A checklist to run before the pull request opens

1. repro.sh fails on the base commit and passes on the patch, same command.
2. Reverting the patch restores the exact failing assertion.
3. The diffstat matches the scope agreed in the issue, with refactors split out.
4. The review brief contains failure lines, file inventory, and three questions.
5. Every model claim was accepted or rejected through the claim table.
6. No UNVERIFIED row survives into the final description.

The order is the contribution: a patch that has been reproduced and minimized is already most of the way to reviewable, and a model then works on evidence instead of adjectives.