DEV Community: Taylor Wang

Shrink an OSS Bug to One Fixture Before Patching

Taylor Wang — Wed, 23 Sep 2026 17:22:31 +0000

A patch without a tiny fixture wastes maintainer time. Upstream reviews stall when failure still spans five modules. Shrink the reported bug to one fixture first.

The fixture then becomes the only allowed proof. Production code then changes against that proof alone. Model comments arrive later as review notes only.

Capture the report, not the guess

Issue text often mixes symptoms, theories, and stack noise. A contributor copies the raw command and output. Dates, host names, and secrets get stripped first.

mkdir -p .repro
printf '%s\n' "$ISSUE_URL" > .repro/source.txt
# Replace make test-one with the upstream single-test command.
script -q -c 'make test-one TEST=failing_case' .repro/raw.log
grep -E 'FAIL|Error|Exception' .repro/raw.log > .repro/errors.txt

That log is evidence, not a patch plan. No production file is edited in this step. Theories from the issue stay out of .repro.

Shrink until one fixture fails

A full suite failure hides the real unit. The next job is strict input reduction. Keep deleting inputs until one fixture still fails.

Copy the failing input into tests/fixtures/shrink/case.json.
Run only the project's documented single-test command.
Drop unused fields while the same assertion still fails.
Stop when one more deletion changes the error class.
Record the surviving command in .repro/repro.lock.

# Example only: swap in the repository's real runner.
npm test -- tests/shrink.test.js
# or
pytest -k test_shrink_case -vv

Label every runner as project-specific before sharing it. Do not invent a framework the repo lacks. A fixture that still needs three services is not small.

Reduce network, clock, and filesystem seams next. Prefer in-process fakes the project already ships. Live cloud calls do not belong in the shrink loop.

Stop shrinking with a decision table

Reduction without a stop rule deletes the bug. Use the table below as the exit gate. Each row is a hard stop, not a preference.

Observation	Action
Next deletion changes the error class	Keep the previous fixture and stop
Fixture still needs live network	Replace bytes with a recorded stub
Two unrelated assertions fail	Split fixtures; patch only one
Result depends on test order	Stop; isolate the test first
Clock skew changes the assertion	Freeze time with the project's helper
Fixture spans two packages	Cut imports until one package remains

Copy the chosen row into .repro/stop.txt. Reviewers then see why shrinking ended. A missing stop row means the fixture is still too large.

# .repro/repro.lock  (example layout, not a real hash)
test_cmd: pytest -k test_shrink_case -vv
fixture: tests/fixtures/shrink/case.json
stop_row: Next deletion changes the error class
delta: .repro/delta.md

Pin the command text, not a remembered alias. Aliases rot across contributor shells. The lock file must replay on a clean clone.

Record a behavior delta table

Maintainers read tables faster than long essays. The fixture needs a before row and after row. Expected values come from public docs or the failing assertion.

Probe	Command or call	Before (failing)	After (required)
Status	`cli parse case.json`	exit 2	exit 0
Message	stderr line 1	`invalid token`	(empty)
Return	`parse()`	throws `ParseError`	object with `ok: true`

This table is the only in-scope contract. Any extra behavior change needs a new row. Silent extras stay out of the pull request.

# .repro/delta.md
probe: parse(case.json)
before: ParseError("invalid token")
after: { ok: true, value: 42 }
out_of_scope: logging format, metrics counters

Keep the out-of-scope list short and explicit. Reviewers then ignore unrelated diffs with cause. Changelog noise is not a fixture win.

Patch only the path the fixture touches

Open the stack from the fixture's failing assertion. Change the smallest function that makes the after column true. Leave nearby refactors for a later pull request.

// Example fixture driver. Adapt names to the upstream tree.
const { parse } = require('../src/parse')
const caseJson = require('./fixtures/shrink/case.json')

test('shrunk fixture reaches documented success', () => {
  const result = parse(caseJson)
  expect(result).toEqual({ ok: true, value: 42 })
})

Run the single test until it passes. Then run the nearest package suite. A green fixture with a red neighbor package is incomplete.

git diff --stat
git diff -- src tests/fixtures/shrink tests/shrink.test.js
python - <<'PY'
# Example: fail the workflow if extra paths appear.
import subprocess, sys
allowed = {"src/parse.py", "tests/fixtures/shrink/case.json", "tests/shrink.test.js"}
out = subprocess.check_output(["git", "diff", "--name-only"], text=True)
files = {line for line in out.splitlines() if line}
extra = sorted(files - allowed)
if extra:
    print("extra paths:", extra)
    sys.exit(1)
print("diff stays inside the fixture graph")
PY

If the stat list grows past the fixture path, stop. Split the extra files out before review. Import-graph creep is a common silent expansion.

# Example import-graph check for a Python package.
python -c "import ast, pathlib, sys
root = pathlib.Path('src')
# Proposal only: replace with the project's module walker.
print('review import edges from the fixture next')"

Treat that walker as unlabeled proposal code. Real graphs come from the upstream package layout. Do not paste a generic graph into a foreign repo.

Re-read the diff without writing new code

A second pass should hunt extra behavior, not new features. Check license headers, changelog rules, and the commit template. Match the repository's existing style, not a personal one.

git log -5 --oneline -- src
git diff --check
# Example: reuse the project's own lint entry if it exists.
[ -f Makefile ] && grep -n '^lint' Makefile || true

Free model access can help that second pass. A contributor pastes the fixture, the delta table, and the diff. MonkeyCode provides free model access and a free server option for that review chat.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model does not own the fixture. Review notes that lack a table row get discarded. Suggested rewrites stay out of the tree until a human replays the test.

Review prompt (example, not executed):
1. Read tests/fixtures/shrink/case.json
2. Read .repro/delta.md and .repro/stop.txt
3. Read the git diff only
4. List claims that lack a delta row
5. List files outside the fixture import graph

Keep the prompt numbered and boring on purpose. Creative refactors from a model are noise here. The optional review chat can use that free server.

Publish the reproduction with the pull request

Maintainers should replay the fixture without a meeting. Put the shrink command in the pull request body. Link the issue URL already stored in .repro/source.txt.

## Reproduction
1. `git checkout <this-branch>`
2. Run the locked single-test command in `.repro/repro.lock`
3. Confirm the after-column in `.repro/delta.md`

## Out of scope
- logging format
- metrics counters

Do not attach secrets or machine-specific paths. The fixture must run on a clean clone. Host-local caches are not reproduction.

git status --porcelain
test -z "$(git status --porcelain)" && echo 'tree clean before push'

A dirty tree hides fixture files from reviewers. Commit the lock, delta, stop row, and test together. Skip generated caches and editor swap files.

Limitations

This workflow assumes a test runner already exists. Greenfield repos without assertions need tests first. Flaky fixtures also fail the method, because shrink cannot split bugs from races.

Large performance issues rarely collapse to one JSON file. Multi-service outages need a different harness. Generated code and vendored trees also resist tiny fixtures.

Model review is optional and advisory only. It does not replace CI, owners, or maintainer judgment. Free access and a free server do not imply named models, quotas, hardware, or uptime here.

Who should skip this

Do not use fixture shrink for speculative cleanups. Do not use it when the issue already names a one-line typo. Embargoed security reports need the project's private channel, not a public fixture.

Contributors who cannot run the project tests locally should stop. A model chat is not a substitute for that run. The fixture and the delta table remain the patch.

Run a Fail-Then-Pass Replay Before an OSS Fix

Taylor Wang — Tue, 22 Sep 2026 15:38:27 +0000

An OSS fix without a replayable failure remains a guess.
Reviewers cannot verify a story that never failed locally.

This workflow pins one issue to one command.
That command must fail before any production edit.
The same command must pass after the patch.

The artifact is a fail-then-pass replay gate.
The gate stores needles from real test logs.
It refuses a patch when either needle is missing.

Replay beats a narrative

Issue threads often describe symptoms more than commands.
A contributor then ships a purely speculative diff.
The maintainer reruns folklore instead of a gate.

A fail-then-pass replay removes most of that folklore.
The failing log becomes the local contract.
The passing log remains the only proof.

The method fits libraries with already deterministic tests.
It also fits command-line tools with golden output.
It does not fit flaky network-only incidents.

Declare the gate

Keep the gate file beside the patch notes.
Use boring machine-readable fields for every run.
Treat this file as an example, not policy.

# example only: review/issue-replay.toml
issue_url = "https://github.com/example/lib/issues/1842"
upstream_ref = "v2.4.1"
test_cmd = "python -m pytest tests/test_parse.py::test_trailing_comma_issue_1842 -q"
fail_needle = "AssertionError: trailing comma must not emit Token.COMMA"
pass_needle = "1 passed"
log_dir = "review/logs"

Needles are literal substrings from local runs.
They are not clever regular expressions at all.
The contributor copies them from the first failure.

1. Isolate a worktree

Do not patch the daily working clone.
Add a detached worktree on the issue tag.
Record the resolved SHA beside the captured logs.

# example commands, not a live upstream
git fetch origin tag v2.4.1
git worktree add --detach ../lib-issue-1842 v2.4.1
mkdir -p ../lib-issue-1842/review/logs
git -C ../lib-issue-1842 rev-parse HEAD > ../lib-issue-1842/review/logs/HEAD.sha

The worktree keeps the main tree clean.
The SHA file stops silent tag movement.
Later logs must cite that exact SHA.

Use the version named in the issue, not folklore.
A tag is a stated reproduction target.
It is not a search across random history.

2. Write one regression assertion

The first edit belongs in the test file.
The test name should carry the issue number.
The assertion should encode the requested behavior.

# example sketch, not production code
def test_trailing_comma_issue_1842():
    tokens = parse("item,")
    kinds = [token.kind for token in tokens]
    assert kinds[-1] != "COMMA", (
        "trailing comma must not emit Token.COMMA"
    )

Run that test before touching library code.
It should fail with the same fail needle.
A pass here means the reproduction is wrong.

3. Prove the failure first

Install test extras only in that worktree.
Run the command stored in the gate file.
Capture stdout and stderr as one log.

# example
cd ../lib-issue-1842
python -m venv .venv
. .venv/bin/activate
pip install -e ".[test]"
python -m pytest tests/test_parse.py::test_trailing_comma_issue_1842 -q \
  > review/logs/fail.log 2>&1 || true
grep -F "trailing comma must not emit Token.COMMA" review/logs/fail.log

The captured log must contain the fail needle.
Absence of that needle aborts the whole workflow.
The contributor then tightens the failing reproduction.

A missing failure remains a blocked local state.
No production file should change before failure.
The test selection is the first bug to fix.

4. Patch the production surface

Change the smallest function that feeds the test.
Leave formatting and comments for a later commit.
Keep the patch readable in ordinary diff output.

# example sketch, not production code
def parse(source: str) -> list[Token]:
    raw = tokenize(source)
    if raw and raw[-1].kind == "COMMA":
        raw = raw[:-1]
    return raw

This parser sketch stays deliberately tiny and local.
Real upstream code will look quite different.
The rule is still one behavior per patch.

5. Require the pass log

Re-run the identical command after the production edit.
Save a sibling pass log next to the SHA.
Require the pass needle and forbid the fail needle.

# example
python -m pytest tests/test_parse.py::test_trailing_comma_issue_1842 -q \
  > review/logs/pass.log 2>&1
grep -F "1 passed" review/logs/pass.log
! grep -F "trailing comma must not emit Token.COMMA" review/logs/pass.log

A pass without the old needle is necessary.
It is not sufficient for an upstream merge.
Maintainers still own design and compatibility choices.

6. Automate both gates

Manual log reading still drifts across long sessions.
A small script should enforce both log needles.
The script below is a labeled example harness.

# example: tools/replay_gate.py
from __future__ import annotations

import argparse
import subprocess
import sys
import tomllib
from pathlib import Path


def load_gate(path: Path) -> dict:
    data = tomllib.loads(path.read_text(encoding="utf-8"))
    required = (
        "test_cmd",
        "fail_needle",
        "pass_needle",
        "log_dir",
    )
    missing = [key for key in required if key not in data]
    if missing:
        raise SystemExit(f"missing gate keys: {missing}")
    return data


def run_cmd(cmd: str, log_path: Path) -> str:
    log_path.parent.mkdir(parents=True, exist_ok=True)
    proc = subprocess.run(
        cmd,
        shell=True,
        text=True,
        capture_output=True,
    )
    body = proc.stdout + proc.stderr
    log_path.write_text(body, encoding="utf-8")
    return body


def require(body: str, needle: str, label: str, present: bool) -> None:
    found = needle in body
    if present and not found:
        raise SystemExit(f"{label}: missing needle: {needle!r}")
    if not present and found:
        raise SystemExit(f"{label}: unexpected needle: {needle!r}")


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--gate", default="review/issue-replay.toml")
    parser.add_argument("--phase", choices=("fail", "pass"), required=True)
    args = parser.parse_args()

    gate = load_gate(Path(args.gate))
    log_dir = Path(gate["log_dir"])
    log_path = log_dir / f"{args.phase}.log"
    body = run_cmd(gate["test_cmd"], log_path)

    if args.phase == "fail":
        require(body, gate["fail_needle"], "fail", True)
        require(body, gate["pass_needle"], "fail", False)
    else:
        require(body, gate["pass_needle"], "pass", True)
        require(body, gate["fail_needle"], "pass", False)

    sha = Path("review/logs/HEAD.sha")
    if not sha.exists():
        raise SystemExit("missing review/logs/HEAD.sha")
    print(f"{args.phase} gate ok sha={sha.read_text().strip()}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The example uses shell mode only for local brevity.
A stricter harness should pass an argument list.
The harness ignores process exit codes on purpose.

Needles decide the gate, not pytest status.
A green pytest run can still miss the pass needle.
A red pytest run can still miss the fail needle.

# example sequence, run inside the worktree
python tools/replay_gate.py --phase fail
# edit tests, then library code
python tools/replay_gate.py --phase pass
git add tests/test_parse.py src/lib/parse.py review/issue-replay.toml

Run the fail phase before any production edits.
Run the pass phase after those production edits.
Commit the gate file with the patch notes.

The harness does not upload any local files.
It does not call a language model.
It only compares local logs to declared needles.

7. Second reader on the log delta

A local replay can still miss weak assertions.
A second reader can inspect the log delta only.
The source tree itself should stay off that path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option.
Those availability notes come from the account operator.
They are not measured quotas, hardware claims, or benchmarks.

The prompt should receive three artifacts only.
Send the gate file, the fail log, and the pass log.
Do not send secrets, credentials, or vendor dumps.

# example review prompt, unexecuted
Read issue-replay.toml, fail.log, and pass.log.
List assertions that still look too weak.
List log lines that changed for the wrong reason.
Do not rewrite the patch.
Do not invent files that are not in the logs.

The model output is a checklist, not a merge vote.
The contributor still runs the local replay harness.
The maintainer still owns the final merge.

Contributors who already hold both logs can continue.
They can try that free model path on the delta alone.

Limitations

The gate trusts deterministic tests above all else.
Flaky suites will flap both declared needles.
Pin time, locale, and working directory first.

The gate cannot replay GUI clicks at all.
It cannot replay production-only race conditions.
Those bugs need other capture and tracing tools.

Needle matching is literal and somewhat brittle.
A wording change can fake a passing gate.
Re-copy needles after each test message edit.

Model review of logs stays inherently shallow.
It cannot see unlogged process state at all.
It cannot replace a human maintainer on the merge.

Free model access may change without prior notice.
A free server option is not a lab promise.
Do not build release process around either one.

Who should skip this workflow

Skip it when the project has no automated tests.
Skip it when the bug is a leaked secret.
Skip it when the patch is a docs typo.

Binary-only vendors cannot run this local harness.
Emergency hotfixes may not wait for extra worktrees.
Those cases need a different local discipline.

The core rule stays small and local.
Fail first on the issue's stated tag.
Pass later on the same recorded command.

A patch that cannot replay both states remains a guess.

Build an OSS Patch Allowlist From One Failing Test

Taylor Wang — Mon, 21 Sep 2026 13:58:50 +0000

An OSS patch is ready only after one named test flips. That named failing test is the merge oracle. The issue thread is only background noise here.

This workflow turns one test into a file allowlist. Coverage names every path the test actually executes. Files outside that map stay out of the commit.

Why issue threads fail as oracles

Bug reports mix symptoms, guesses, and local environment detail. Contributors then edit the modules that look related. The resulting pull request often touches idle code.

CI can still pass after that extra contact. Distant callers then break on the next release. A coverage allowlist stops that drift before review starts.

Artifact overview

The artifact is a small local review gate. It records one failing test and writes a coverage allowlist. It then rejects extra git paths before commit.

The Python snippets below are an unexecuted proposal. Projects should swap in their real test runner. Do not add a new framework for a single bug.

Preconditions

Match the issue against the default remote branch. The tree must be clean before coverage runs. A dirty worktree will poison the coverage allowlist.

git fetch origin
git switch --detach origin/main
test -z "$(git status --porcelain)"

The last command must print nothing at all. Stash or commit leftover files before continuing.

The project must already have a test command. This method does not install a new stack.

Step 1. Encode the bug as one test

Name the test after the issue number. Keep exactly one hard assertion in the test. Avoid snapshots of an entire production module.

# tests/test_issue_8841_retry_budget.py
from pkg.client import RetryBudget


def test_issue_8841_retry_budget_stops_after_two_failures():
    budget = RetryBudget(max_attempts=2)
    budget.record_failure()
    budget.record_failure()
    assert budget.can_retry() is False

Run only that test file for now. Confirm the failure matches the original report.

pytest -q tests/test_issue_8841_retry_budget.py -vv --tb=short

A passing result means the oracle is wrong. Rewrite the assertion before any production code edit.

Step 2. Collect coverage from that test only

Full-suite coverage is too wide for a patch allowlist. Collect data from the named test alone.

pytest tests/test_issue_8841_retry_budget.py \
  --cov=pkg \
  --cov-report=json:issue_8841_cov.json \
  --cov-report=term-missing

Turn covered files into a plain allowlist. Print that list before any edit starts.

# tools/cov_allowlist.py  (proposal)
import json
from pathlib import Path

data = json.loads(Path("issue_8841_cov.json").read_text())
files = sorted(
    path
    for path, row in data.get("files", {}).items()
    if row.get("summary", {}).get("covered_lines", 0) > 0
)
out = Path("issue_8841_allow.txt")
out.write_text("\n".join(files) + "\n")
print("\n".join(files))

python tools/cov_allowlist.py

Add the new test path to the allowlist by hand. Production files must come only from coverage output.

Step 3. Reject git paths outside the allowlist

Do not stage any changed files yet. Diff the working tree against HEAD first. Fail the gate when extra paths appear.

# tools/enforce_allowlist.py  (proposal)
from pathlib import Path
import subprocess
import sys

allow = {
    line.strip()
    for line in Path("issue_8841_allow.txt").read_text().splitlines()
    if line.strip()
}
allow.add("tests/test_issue_8841_retry_budget.py")
raw = subprocess.check_output(
    ["git", "diff", "--name-only", "HEAD"],
    text=True,
)
changed = {line for line in raw.splitlines() if line}
extra = sorted(changed - allow)
if extra:
    sys.stderr.write("paths outside coverage allowlist:\n")
    sys.stderr.write("\n".join(extra) + "\n")
    raise SystemExit(1)
print("allowlist ok:", ", ".join(sorted(changed)))

python tools/enforce_allowlist.py

A nonzero exit means the patch is too wide. Shrink the diff until the gate passes. Do not grow the allowlist to match it.

Step 4. Edit only executed files

Change production code after the gate exists. Stay strictly inside the coverage allowlist paths. Re-run the named test after every edit.

pytest -q tests/test_issue_8841_retry_budget.py --tb=short

The test must keep failing until the real fix lands. A stub that forces a pass is a false oracle.

Then run tests that import the same module. Package regressions hide beside a green single test.

pytest -q pkg/client tests/test_issue_8841_retry_budget.py

Public function signatures stay frozen unless a new test says otherwise. Extra helpers belong in a later separate change.

Step 5. Write a replay log for reviewers

Reviewers should not reconstruct the local ritual. Dump the test name, HEAD, allowlist, and output. Attach that file to the pull request.

{
  echo "TEST=tests/test_issue_8841_retry_budget.py"
  echo "HEAD=$(git rev-parse HEAD)"
  echo "--- ALLOWLIST ---"
  cat issue_8841_allow.txt
  echo "--- DIFFSTAT ---"
  git diff --stat
  echo "--- TEST ---"
  pytest -q tests/test_issue_8841_retry_budget.py --tb=short
} > issue_8841_replay.txt

The replay log is the primary review surface. The raw diff is supporting evidence only here.

Maintainer checks on the pull request

The maintainer re-runs one command from the replay log. The named test must fail on origin/main. It must pass on the proposed contributor branch.

git switch --detach origin/main
pytest -q tests/test_issue_8841_retry_budget.py --tb=short
# expect fail
git switch --detach FETCH_HEAD
pytest -q tests/test_issue_8841_retry_budget.py --tb=short
# expect pass
python tools/enforce_allowlist.py

A test that never failed on main is not an oracle. Reject that pull request without any further review.

Decision table

Use this table before the pull request opens. Each missed row should block the merge.

Signal	Action
Named test never failed	Rewrite the test. Do not patch.
Coverage JSON lists no files	The test never imported production code.
Diff includes files off the map	Split the work or add another named test.
Single test passes, package tests fail	Keep the patch. Fix the regression first.
Public API shape changed	Add a dedicated compatibility test.
Only comments or lockfiles changed	Stop. The oracle did not move.
Allowlist contains generated files	Exclude those paths. Do not hand-edit them.

Each table row is a hard stop condition. Green CI without a named oracle is not enough.

Optional second pass on the replay log

A language model can read the replay log and the allowlisted diff. It still cannot replace the failing named test. Treat those notes as extra maintainer questions only.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two availability facts are the only product claims in this article.

A contributor without a local GPU can run the replay-log review there. Keep the model prompt inside the allowlist. Send the log, the diff, and the test output only.

Review only files listed in issue_8841_allow.txt.
Ignore every path outside that list.
The oracle is tests/test_issue_8841_retry_budget.py.
List contradictions between the diff and the replay log.
Do not suggest new modules.
Do not delete or weaken the failing test.

Discard any note that widens the change set. Discard any note that removes the oracle. Humans still own the final merge decision.

Limits

Coverage is not a proof of full correctness. It only records lines the named test executed. Untested branches in the same file can still fail.

The coverage allowlist remains a tool-specific artifact. coverage.py maps Python poorly onto C extension code. Go, Rust, and Java need their own cover commands.

Cross-cutting refactors do not fit this gate. Wide migrations need a different review contract. Do not force one test onto a wide rename.

Flaky tests also break this merge oracle. Pin time, network, and random seeds first. A flake is not a bug reproduction.

Who should not use this

Authors of large public API redesigns should skip this method. Contributors without any runnable harness should skip it. People editing generated or vendored trees should skip it.

Security fixes that must touch many files need another process. A coverage allowlist can hide required extra paths. Follow the project security advisory flow instead.

Closing

The named failing test is the merge oracle. Coverage from that test is the file allowlist. The replay log is what reviewers actually read.

A model may comment on that replay log. That model does not get a merge vote. The test still has to fail, then pass, on the contributor machine.

Require a Local Review Contract Before an OSS Pull Request

Taylor Wang — Sun, 20 Sep 2026 11:54:03 +0000

An OSS patch should not leave a fork until four artifacts exist. Those artifacts are reproduction notes, a bounded patch, tests, and a review contract.

A model may fill fields in that contract. A human maintainer still owns the merge decision.

This workflow keeps model output out of git history. It also keeps contributor judgment in one signed file. The contract is the only place a model is allowed to speak.

Why the contract exists

Maintainers do not merge confidence scores or vibes. They merge commands, paths, and observed test output. A raw chat transcript is not review evidence.

The contract records answers maintainers already expect in review. It does not invent extra narrative or scope. It stores commands, paths, and residual risk.

The next sections propose a local convention. These files are not a published industry standard. Teams should adapt field names to their project.

Current AI review chatter often hides missing tests. A filled chat window is easy to fake. A signed contract with rerun commands is harder to fake.

Artifact layout

Keep review files off the upstream default branch. Use a local directory that never ships by default.

.review/
  reproduce.md
  test-record.md
  review-contract.yml
  model-fill.yml
  series.diff
  SIGNED_BY.txt

The patch itself lives only in git commits. The review contract lives beside those git commits. Do not add .review/ unless the project asks.

Step 1: Reproduce the bug on a frozen tree

Clone the project at the reporter commit SHA. Do not patch the tree before reproduction. Write the failing command into reproduce.md first.

git clone https://github.com/example/project.git
cd project
git checkout --detach "$REPORTER_SHA"
git switch -c review/issue-1234

Record the environment in two short lines. Record the exact command that fails. Record the observed output, truncated if noisy.

# .review/reproduce.md
- sha: 9f3c1aa
- os: linux
- command: cargo test queue::drop_on_cancel -- --exact
- observed: assertion failed: ticket.cancelled
- notes: fails with RUST_TEST_THREADS=1 only

Stop if the failure does not reproduce locally. A remote model cannot review a ghost bug. Leave a comment on the issue instead.

This example is a proposed layout, not a run log. Replace the cargo command with the project runner. Do not invent timings or pass rates.

Step 2: Bound the patch to one behavior change

Create commits that a maintainer can read in order. One commit should add or fix tests. One commit should change production code.

Optional third commit updates nearby docs only. Reject mixed commits at this stage. A later model reviews each commit alone.

git add src/queue.rs tests/queue.rs
# split the index if both files landed together
git reset HEAD
git add tests/queue.rs
git commit -m "test(queue): cover cancel during drop"

git add src/queue.rs
git commit -m "fix(queue): honor cancel on drop"

git add README.md
git commit -m "docs(queue): note cancel on drop"

The oneline log from main must stay short. List two or three commits after the split. Keep the subject lines boring and specific.

git log --oneline origin/main..HEAD
git diff --stat origin/main...HEAD

If the stat output spans unrelated packages, stop. Split the work before any model pass. Do not ask a model to justify a mixed diff.

A stacked series also helps revert. Maintainers can drop the docs commit. They can keep the test commit during bisect later.

Step 3: Record a test result a stranger can rerun

Write the test record by hand. Paste command output without paraphrase. Keep the original failing command in the same file.

mkdir -p .review
{
  echo "date: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "sha: $(git rev-parse HEAD)"
  echo "fail_cmd: cargo test queue::drop_on_cancel -- --exact"
  echo "pass_cmd: cargo test queue::drop_on_cancel -- --exact"
  RUST_TEST_THREADS=1 cargo test queue::drop_on_cancel -- --exact
} | tee .review/test-record.md

The record must include the failing command from step one. It must include the passing command after the fix. Both commands should share the same runner flags.

Label this block as unexecuted on a public repo. Replace cargo with the real project runner. Do not invent pass rates or machine specs.

If CI is required, capture one job log excerpt. Store only the failing test names. Do not dump secrets from the log.

gh run view --log --job test | tail -n 80 > .review/ci-excerpt.txt

That gh command is optional and proposed. Skip it when the issue is local only. Never paste tokens into the excerpt file.

Step 4: Freeze a review contract schema

The contract is YAML with empty string defaults. Empty fields block the pull request locally. A small script enforces the required keys.

# .review/review-contract.yml
issue: "https://github.com/example/project/issues/1234"
reporter_sha: "9f3c1aa"
patch_shas: []
scope:
  allowed_paths:
    - "src/queue.rs"
    - "tests/queue.rs"
    - "README.md"
  forbidden_paths:
    - "vendor/"
    - ".github/workflows/"
repro_command: ""
fail_output_excerpt: ""
pass_command: ""
pass_output_excerpt: ""
behavior_change: ""
public_api_change: "none"
residual_risk: ""
maintainer_answers:
  why_this_fix: ""
  why_not_larger_refactor: ""
  how_to_revert: ""
model_role: "fill_blanks_only"
contributor_attestation: ""

This schema is the original artifact for this workflow. It is intentionally boring and explicit. Boring fields are easier for humans to verify.

Copy the reporter SHA from step one. Leave patch_shas empty until commits exist. Fill patch_shas with git rev-parse after the split.

git rev-parse origin/main..HEAD > /tmp/patch_shas.txt

Step 5: Let a model fill blanks, not commits

Copy the contract to model-fill.yml before any model run. Send only that file plus the commit diffs. Do not paste the entire repository into the prompt.

cp .review/review-contract.yml .review/model-fill.yml
git diff origin/main...HEAD > .review/series.diff
git log -p origin/main..HEAD > .review/series-commits.diff

A proposed prompt, not a production prompt, follows. It forbids new files and invented commands. It also forbids commit rewrites.

Fill empty string fields in model-fill.yml.
Use only series.diff, reproduce.md, and test-record.md.
Do not invent commands that were not run.
Leave unknown fields as UNKNOWN.
Do not add paths outside allowed_paths.
Do not rewrite commit messages.
Return YAML only.

The contributor then diffs the fill against the original. Keep every human-run command in place. Replace model-invented commands with UNKNOWN.

diff -u .review/review-contract.yml .review/model-fill.yml

Reject any fill that adds new paths. Reject any fill that claims the model ran tests. The model did not run those tests.

Merge accepted fields by hand. Do not overwrite human command strings. The signed file later attests to that edit.

Where a hosted model fits

Local review still needs a place to run the fill step. Some contributors lack a spare local GPU. Some forks keep diffs and prompts on a small server.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. That option can host this YAML fill step.

The contract, tests, and commits stay under contributor control. The model still does not own the merge.

Use a hosted fill only after steps one through four exist. Do not start the issue by chatting with a model.

Step 6: Validate the contract before opening the PR

The validator is a proposed shell script. It does not contact the network. It fails closed on empty required fields.

#!/usr/bin/env bash
# validate-review-contract.sh — proposed local gate, not production YAML parsing
set -euo pipefail
file=".review/review-contract.yml"
need=(
  issue reporter_sha repro_command fail_output_excerpt
  pass_command pass_output_excerpt behavior_change
  residual_risk contributor_attestation
)
for key in "${need[@]}"; do
  val="$(awk -F': ' -v k="$key" '$1==k{print $2; exit}' "$file" | tr -d '"')"
  if [[ -z "$val" || "$val" == "UNKNOWN" ]]; then
    echo "missing $key" >&2
    exit 1
  fi
done
while IFS= read -r p; do
  case "$p" in
    src/queue.rs|tests/queue.rs|README.md) ;;
    *) echo "path outside scope: $p" >&2; exit 1 ;;
  esac
done < <(git diff --name-only origin/main...HEAD)
echo "review contract ok"

Mark the script as a proposal only. Real projects should parse YAML with a real parser. The awk shortcut is for teaching.

Run the script on the filled contract. Sign the attestation after it passes. Only then open the pull request.

chmod +x validate-review-contract.sh
./validate-review-contract.sh
printf '%s\n' \
  'I ran the commands in test-record.md.' \
  'I read the model fill and discarded inventions.' \
  > .review/SIGNED_BY.txt

The PR body can quote three contract fields. It should not paste the model YAML. Maintainers should not have to read the fill file.

Step 7: Write the maintainer-facing PR body from the contract

Copy facts, not generated style. Keep the body shorter than the contract. Use the same commands as the test record.

Fixes #1234

Reproduction: `cargo test queue::drop_on_cancel -- --exact`
Change: honor cancel on drop in src/queue.rs
Tests: same command passes on the patch SHA
API: none
Revert: revert the fix commit; keep the test commit

This body is boring on purpose. Maintainers can rerun the listed command. They do not need a model summary.

If the tracker wants a checklist, quote contract keys. Do not paste residual model prose. Residual prose belongs in .review/model-fill.yml only.

Decision table

Situation	Action
Bug does not reproduce	Stop. Comment on the issue.
Diff crosses forbidden paths	Split or abandon the patch.
Model fill invents a command	Replace with UNKNOWN and rerun.
Tests pass only on one laptop	Do not open the PR.
Maintainer asks for a squash	Squash after review, not before.
Embargoed security patch	Skip hosted models entirely.

Limitations

This workflow does not prove correctness. It only proves that required fields were filled. UNKNOWN handling still depends on contributor honesty.

YAML parsed by awk is brittle. Nested lists need a real parser. The path check is a demo for three files.

A model can still hallucinate causal language. The contract reduces that risk. It does not delete it.

Hosted models also see the supplied diff. Secret patches do not belong in that prompt. Embargoed security issues stay offline.

Availability of a hosted fill is not a quality benchmark. Duration and hardware are out of scope. Do not treat a free fill host as proof.

The stacked commit rule can annoy squash-only projects. Follow the project guide after review. Do not fight a stated squash policy.

Who should not use this

Do not use this flow for security embargoes. Do not upload embargo diffs to any hosted model. Do not use this for drive-by refactors.

Skip it when the project already has a review schema. Do not run two parallel contracts. Skip it when the contributor cannot reproduce the bug.

New contributors who cannot run the test command should not sign. Signing without a local test run is the failure mode. The attestation is a statement of fact.

Skip it for generated code dumps with no tests. A contract cannot rescue an untested tree. Write the failing test first.

Closing

The useful order is reproduce, patch, test, then contract. Models enter at the contract step. They do not enter at clone time.

A pull request that cannot cite commands is not ready. A model comment that cannot cite those commands is noise.

Keep residual model noise in the YAML file. Keep the commit history only in git.

Treat Model Comments as Named Hypotheses in an OSS Review

Taylor Wang — Sat, 19 Sep 2026 09:59:18 +0000

An OSS patch stays reviewable when a human owns every edit. Model output belongs in a hypothesis file, not the tree. Maintainers merge tested evidence instead of untested model suggestions.

Chat-authored diffs hide invented helpers and missing tests. Reviewers then spend cycles proving the patch is even real. The project loses time on scope that nobody requested.

This workflow freezes a review contract before any model sees the diff. The contributor authors the production change by hand. A model may only emit named, testable hypotheses.

Why chat-authored OSS patches stall

A model does not share the project's history of rejected APIs. It also does not feel the cost of a noisy pull request. It often optimizes for a complete-looking but unbounded diff.

OSS maintainers review ownership, scope, and proof together. They do not review autocomplete confidence as evidence. Ungrounded comments therefore fail at first human pass.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those options can host the later review pass only.

The local contract still governs every accepted line. The method remains useful without any hosted model. Remove the product and the gate still holds.

Core rule

Treat every model comment as a named hypothesis. Require a frozen path and a testable claim. Discard any comment that invents a symbol.

The model never writes files in the working tree. The model never receives tokens or private mail. The model never proposes files outside the allowlist.

Artifact: the review contract

Keep one small contract beside the clone. Pin the public issue, commands, and allowlisted paths. The review bundle is generated from this file.

# example only: .review/review-contract.yaml
issue_url: "https://github.com/example/libparse/issues/842"
base_sha_file: ".review/base.sha"
repro_cmd: "python -m pytest tests/test_parse_empty.py -q"
smoke_cmd: "python -m pytest tests/test_public_api.py -q"
allow_paths:
  - "src/libparse/scanner.py"
  - "tests/test_parse_empty.py"
forbid_substrings:
  - "API_KEY"
  - "BEGIN PRIVATE KEY"
  - "xoxb-"
hypotheses_md: ".review/hypotheses.md"

This YAML is a proposed example, not a production manifest. Replace the issue URL with a real public ticket. Keep embargoed security issues out of this file.

Numbered workflow

1. Detach at a recorded upstream SHA

Fetch the default branch and detach HEAD there. Record the SHA in .review before any local edit. Later review comments must name that recorded SHA.

git fetch origin
git checkout --detach origin/main
mkdir -p .review
git rev-parse HEAD > .review/base.sha
git status --porcelain

An empty status is required at this point. Unrelated dirty files leak into the later bundle. Clean the tree before writing the review contract.

2. Pin the public issue and the allowlist

Copy only the public issue URL into the contract. List the few paths that may still change. Add the test paths that prove the reported bug.

Do not add the entire src tree to the allowlist. Broad allowlists recreate the same noisy patch problem. Two or three paths usually bound a single bug.

3. Write the failing test on the frozen SHA

Add one test that follows the public report. Run the recorded repro command on the frozen SHA. Store the non-zero exit code beside the contract.

python -m pytest tests/test_parse_empty.py -q
echo $? | tee .review/repro.exit
test "$(cat .review/repro.exit)" != "0"

Stop when the test already passes on base. The report then lacks a local failing reproducer. Do not ask a model to invent the missing proof.

4. Author a minimal production edit by hand

Change only allowlisted production files after the red test exists. Keep public signatures stable unless the issue requires a break. Avoid new exported helpers that expand public surface.

git diff --stat -- src/libparse/scanner.py tests/test_parse_empty.py
git diff -- src/libparse/scanner.py tests/test_parse_empty.py > .review/allowlisted.diff

Inspect the stat output before any model run. Extra files mean the allowlist was ignored. Restore those files instead of explaining them later.

5. Re-run the same commands after the edit

Run the original repro command until it passes. Run the smoke command against public API tests. Record both exit codes under the .review directory.

python -m pytest tests/test_parse_empty.py -q
echo $? | tee .review/fix.exit
python -m pytest tests/test_public_api.py -q
echo $? | tee .review/smoke.exit

Both recorded files must contain a zero exit code. Non-zero smoke results block the review pass. The model must not debug a still-failing tree.

6. Build a secret-stripped bundle

Concatenate the contract, the SHA, and the allowlisted diff. Strip known credential markers before the model sees the bundle. Refuse to continue if a marker remains.

# example helper: build_review_bundle.sh
set -euo pipefail
BUNDLE=".review/bundle.md"
{
  echo "# Review bundle"
  echo "base_sha: $(cat .review/base.sha)"
  echo
  echo "## contract"
  cat .review/review-contract.yaml
  echo
  echo "## allowlisted diff"
  cat .review/allowlisted.diff
} > "$BUNDLE"

if grep -E "API_KEY|BEGIN PRIVATE KEY|xoxb-|AKIA" "$BUNDLE"; then
  echo "secret marker found; refuse model review" >&2
  exit 1
fi

This script is an unexecuted example helper only. Projects should extend the marker list for their stack. Never paste .env files into the bundle.

7. Ask the model for hypotheses, not patches

Use a strict review prompt against the bundle only. Demand hypothesis IDs, frozen paths, and testable claims. Ban unified diffs in the model reply.

The example prompt text follows in the next block. Label that block as a proposed prompt only. Do not treat it as a vendor template.

You are a second reviewer, not a patch author.
Comment only on files listed in allow_paths.
Each finding must use this shape:
H-001 | path | claim | test idea
If you cannot test a claim, write DISCARD.
Do not invent files, APIs, or commit messages.
Do not return a patch.

MonkeyCode free model access can run that prompt. A free server option can keep the pass off the laptop. The contributor still types every accepted edit locally.

Write the model output into .review/hypotheses.md for audit. Keep the raw reply next to the contract. Reviewers then see what the model actually said.

8. Accept, convert, or discard each hypothesis

Read the hypothesis file from top to bottom. Accept a claim only when the path is allowlisted. Convert that claim into a test or a tighter assertion.

# example: .review/hypotheses.md
H-001 | src/libparse/scanner.py | empty input still returns None | assert scanner.parse("") == []
H-002 | src/libparse/scanner.py | invented helper skip_bom exists | DISCARD
H-003 | src/libparse/compat.py | missing py2 branch | DISCARD

H-002 invents a helper the patch never added. H-003 names a file outside the allowlist. Both of those rows are discarded without further discussion.

Add a test for H-001 if the claim is true. Re-run repro and smoke commands after that test. Drop the hypothesis if the new test cannot fail on the base SHA.

9. Open the pull request with the hypothesis log

Commit only allowlisted production paths plus the new tests. Attach the contract and hypothesis log in the PR body. State the base SHA in the first paragraph.

BASE=$(cat .review/base.sha)
git add src/libparse/scanner.py tests/test_parse_empty.py
git commit -m "Fix empty input handling in scanner.parse

Base SHA: ${BASE}
Hypotheses accepted: H-001
Hypotheses discarded: H-002 H-003"

Do not commit .review bundles that contain issue text copies. Some projects treat those copies as review noise. Paste a short summary instead of the raw bundle.

Decision table

Each row maps a hypothesis shape to a required human action. The reason column blocks extra debate during triage. Apply the table before any extra model round.

Hypothesis shape	Human action	Reason
Path outside the allowlist	Discard	Scope is unbounded
Symbol not present in the diff	Discard	Model invented an API
Claim with no test idea	Discard	Untestable review noise
Test idea fails on the base SHA	Add the test	Claim is real
Test idea passes on the base SHA	Discard	Claim is not a regression
Suggestion of a full rewrite	Discard	Authorship left the human
Mention of secrets or keys	Stop the review	Bundle leaked

Use the table during the hypothesis triage pass. Do not negotiate rows that say Discard. The contract is cheaper than a long maintainer thread.

Limitations

This workflow does not replace a maintainer review. It only reduces ungrounded model text in the PR. Hidden behavioral bugs can still pass the smoke file.

The path allowlist cannot express semantic code ownership. A one-line change may still break release notes. Public API policy still needs a human decision.

Free model access and a free server option do not define correctness. They only host the optional hypothesis review pass. Latency, context limits, and model quality vary and are not measured here.

The example commands assume a Python pytest tree. Other languages need their own equivalent frozen test commands. The contract format stays the same across languages.

Who should not use this approach

Do not use this flow on embargoed vulnerability reports. Those reports must not enter a third-party model. Follow the project's private security list instead.

Do not use this flow when the license forbids AI review. Some maintainer communities ban model-assisted patches as policy. Read CONTRIBUTING.md before sending any hosted prompt.

Do not use this flow to generate the production patch. Authorship of the diff must stay with the contributor. A model-authored tree still fails this method.

Do not use this flow on huge refactors. Allowlists of two paths cannot bound a rewrite. Split the work or skip the model pass.

Closing

Keep the human as the author of every OSS edit. Freeze a contract, prove the bug, then request hypotheses. Convert only testable claims and discard the rest.

The hypothesis file is the durable review artifact. It survives after any hosted model session ends. Maintainers can audit it without sharing a chat log.

Bound an OSS Patch With Pathspecs and CODEOWNERS

Taylor Wang — Fri, 18 Sep 2026 08:20:40 +0000

An OSS patch should touch only owned files. Unbounded diffs hide regressions and invite drive-by refactors. Freeze pathspecs and CODEOWNERS before any model review.

The failure mode

Assistants expand scope when the whole tree is visible. Neighbor modules receive drive-by cleanups during one-line fixes. Maintainers then review style noise instead of the defect.

Issue comments often describe a single failing function. The opened pull request then rewrites three packages. Review time then shifts to ownership instead of correctness.

This is not a taste argument about generated code. It is a git-radius problem with a checkable gate. The issue must name files before editors open.

Core rule

The issue defines a path allowlist on disk. Git enforces that allowlist on every staging step. A model may comment only after the allowlist holds.

The allowlist is not a chat preference or a remembered glob. Store it beside the worktree as review/paths.allow. Treat a missing file as a hard stop.

Artifact: an ownership map and pathspec gate

The script below is an example workflow, not a live benchmark. Contributors should rename branches and owners for the target repository. Run it from a throwaway clone, never from a secrets-bearing tree.

#!/usr/bin/env bash
# pathspec-gate.sh — example workflow for a bounded OSS patch
set -euo pipefail

ROOT="$(git rev-parse --show-toplevel)"
ALLOW="$ROOT/review/paths.allow"
OWNERS="$ROOT/CODEOWNERS"
MAP="$ROOT/review/ownership-map.txt"
REPORT="$ROOT/review/blast-radius.txt"

mkdir -p "$ROOT/review"

if [[ ! -f "$ALLOW" ]]; then
  echo "missing $ALLOW" >&2
  exit 2
fi

if [[ ! -f "$OWNERS" ]]; then
  echo "missing $OWNERS" >&2
  exit 2
fi

BASE="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main)"
mapfile -t CHANGED < <(git diff --name-only "$BASE"...HEAD)

fail=0
{
  echo "base $BASE"
  echo "changed_count ${#CHANGED[@]}"
} > "$REPORT"

: > "$MAP"
for f in "${CHANGED[@]}"; do
  allowed=0
  while IFS= read -r spec; do
    [[ -z "$spec" || "$spec" =~ ^# ]] && continue
    case "$f" in
      $spec) allowed=1; break ;;
    esac
  done < "$ALLOW"

  owner_hit=0
  while IFS= read -r line; do
    [[ -z "$line" || "$line" =~ ^# ]] && continue
    own_path="${line%% *}"
    case "$f" in
      ${own_path}*) owner_hit=1; echo "$f :: $line" >> "$MAP"; break ;;
    esac
  done < "$OWNERS"

  if [[ "$allowed" -eq 0 ]]; then
    echo "OUT_OF_SCOPE $f" >> "$REPORT"
    fail=1
  elif [[ "$owner_hit" -eq 0 ]]; then
    echo "UNOWNED $f" >> "$REPORT"
    fail=1
  else
    echo "IN_SCOPE $f" >> "$REPORT"
  fi
done

exit "$fail"

The script exits non-zero on extra or unowned files. Local CI can invoke it before compiling tests. Reviewers read review/blast-radius.txt and review/ownership-map.txt together.

Numbered workflow

1. Open a linked worktree for the issue

An assistant should not sit on the only local clone. Create a linked worktree pinned to upstream main. Failed experiments then die with that directory.

git fetch origin
git worktree add ../issue-8123 origin/main
cd ../issue-8123
git switch -c fix/8123-null-guard
mkdir -p review tools

Keep production credentials out of this worktree. Copy only the files the build system requires.

2. Build the allowlist from named paths

Read the issue title, labels, and linked file list. Write the smallest glob set that can hold a legal fix. Put those globs in review/paths.allow.

# review/paths.allow — example for a parser crash
src/parser/*.c
src/parser/*.h
tests/parser/**
docs/parser.md

Do not add src/** for staging convenience. Wide globs recreate the original scope failure. A later extra file must reopen the issue thread.

3. Overlay CODEOWNERS on the same paths

Open CODEOWNERS in the same sitting as the allowlist. Every allowlist path needs a named owner line. A path without an owner is already a social conflict.

# CODEOWNERS excerpt — example only
src/parser/    @parser-maintainers
tests/parser/  @parser-maintainers
docs/parser.md @parser-maintainers @docs-team

An allowlist path with no owner stops the workflow. The tracker must name an owner before coding starts. Guessed teams do not belong in the pull request.

4. Limit the editor and the index

Edit files under the allowlist only. Stage with explicit pathspecs on every commit. Refuse git add -A for the entire working tree.

git add -- src/parser/scan.c src/parser/scan.h
git add -- tests/parser/null_guard_test.cc
git diff --cached --stat
git diff --cached --check
git diff --cached --name-only

--check catches whitespace errors before review. --stat shows radius in one screen. --name-only becomes the later model input.

5. Run the gate before any test binary

A green suite on an unbounded diff is still a bad patch. The ownership map must pass first. Only then should unit tests run on the named component.

chmod +x tools/pathspec-gate.sh
./tools/pathspec-gate.sh
cmake --build build --target parser_unittests
./build/parser_unittests --gtest_filter=NullGuard.* \
  | tee review/test.log

If tests require a directory outside the allowlist, the allowlist is wrong. Widen it with tracker evidence, not with local instinct. Record the extra path in the issue before restaging.

6. Shape the commit to the same radius

Bound patches still need a bound history. One logical commit should match one allowlist. Docs-only files stay out unless the issue lists them.

git commit -m "parser: guard NULL tokens in scan_ident"
git log -1 --format='%s%n%n%b'
git diff origin/main...HEAD --shortstat

A useful subject names the owned component first. The body should cite the issue number and review/paths.allow. Changelog files join the allowlist before they join the index.

# extra allowlist line only when the project requires it
CHANGELOG.md

7. Hand a model the radius, not the repository

Export names, owners, and the cached diff. Do not paste the whole tree into a prompt. Strip tokens from logs before any upload.

{
  echo "## allowlist"
  cat review/paths.allow
  echo
  echo "## ownership-map"
  cat review/ownership-map.txt
  echo
  echo "## names"
  git diff --cached --name-only
  echo
  echo "## stat"
  git diff --cached --stat
} > review/radius.md

sed -i 's/ghp_[A-Za-z0-9]*/REDACTED/g' review/radius.md
sed -i 's/Bearer [^ ]*/Bearer REDACTED/g' review/radius.md
git diff --cached > review/cached.diff

The model input is review/radius.md plus review/cached.diff. Anything else is out of scope by construction.

Where a free coding assistant fits

A model is useful after the radius is frozen. It is not a substitute for CODEOWNERS or for git pathspecs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. That pair can host a bounded diff review when the laptop is already compiling. The assistant should flag missing tests, unsafe API use, and lines that contradict the allowlist.

Do not ask the model to improve surrounding code. That prompt destroys the gate in one sentence. Ask it to list changed functions without tests. Ask it to quote the CODEOWNERS line for each path.

A sample prompt follows as an unexecuted template.

Review one bounded OSS patch and nothing else.
Discuss only files listed under ## names.
If a fix needs another path, answer OUT_OF_SCOPE.
Check tests, NULL handling, and log noise.
Do not propose refactors, renames, or dependency bumps.

The free server option is for redacted radius files, not for embargoed security diffs. Upload review/radius.md and review/cached.diff only after the sed redaction step.

Decision table

Signal	Action	Forbidden shortcut
File outside allowlist	Stop and reopen the issue	Quiet `git add` of the extra path
Allowlist path with no CODEOWNER	Ping the tracker for an owner	Invent a team in the pull request
Tests need an extra module	Widen the allowlist with evidence	Keep the old file and ship
Model suggests a rename	Record OUT_OF_SCOPE	Apply the rename to look complete
Tests pass, gate fails	Reject the patch	Ship because the suite is green
Radius file contains tokens	Redact and rotate credentials	Upload the log anyway

Limitations

This gate does not prove functional correctness. It only proves the diff stayed inside declared paths. A wrong allowlist will certify the wrong radius with great confidence.

CODEOWNERS files are often stale in long-lived trees. A listed team may no longer merge that directory. The example script does not check GitHub or GitLab membership.

Pathspec matching here uses bash case globbing. That is not identical to gitignore semantics. Leading slashes and nested globs need per-repo tests.

The merge-base line assumes origin/main or main. Repositories on master or trunk must edit that lookup. Monorepos usually need one allowlist per package, not one file at the root.

Model output remains advisory text. It cannot accept a CLA or DCO. It cannot judge whether a public API change needs a major version.

Free model access and a free server option are availability claims only. This article does not state quotas, model names, uptime, or hardware. Those product details change and must be read from current primary docs.

Who should not use this

Skip this gate for a one-word documentation typo. A single README fix does not need a linked worktree ritual.

Skip it when the issue is an explicit module refactor. Broad scope work needs a design review, not a glob file. The allowlist would only hide a missing architecture thread.

Do not point any assistant at a clone that holds production secrets. The radius files are for redacted logs and cached hunks.

Embargoed security patches need private review channels. A free server is the wrong place for those diffs.

Close

Keep the allowlist smaller than the first instinct. Run the ownership map before the test binary. Let a model read the radius files only after both gates are green.

If a bounded review loop would help a current issue, try free model access on a throwaway worktree and keep the production clone offline.

Freeze a Bisect SHA and Public API Before an OSS Patch Review

Taylor Wang — Thu, 17 Sep 2026 06:36:18 +0000

An OSS patch needs two frozen artifacts before model review. The first artifact is a git bisect SHA. The second is a snapshot of public names.

Models that skip those bounds invent extra files. They also invent extra public APIs without cause. Review then drifts away from the real bug.

This workflow is for maintainers who already reproduced the bug. It does not replace a real failing test. It only stops a model from widening scope.

The usual failure

A volunteer pastes an issue into a coding model. The model returns a large and confident diff. The diff often touches helpers the crash never reached.

It also adds a new public function for clarity. Reviewers then argue about style instead of behavior. The merge still ships a silent API expansion.

Downstream packages break on the next minor tag. The original crash may even stay present. The repository now owns two problems, not one.

Two artifacts that bound the work

Keep both artifacts in the issue thread. Treat both files as hard merge gates. A model may edit only inside those gates.

The first gate is the FIRST_BAD commit file. That file stores the first bad commit. The second gate file is api.snap.bad with public names.

The recipe below uses a small Python package layout. Commands are a local proposal, not a published benchmark. They belong on a throwaway clone only.

1. Pin the tree without a model

The maintainer clones the project at the reporter's tag. A chat session does not start yet. The exact ref is recorded in the issue.

git clone https://github.com/example/lib.git
cd lib
git switch --detach v2.4.1
python -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
pytest tests/test_parse.py::test_empty_header -q

The selected test must fail on that detached ref. A passing test means the bug is elsewhere. Work stops there, with no model session.

2. Bisect until one SHA remains

The failing test becomes the bisect run command. Logs are not pasted into a model here. The SHA becomes the later review window.

git bisect start
git bisect bad HEAD
git bisect good v2.3.0
git bisect run pytest tests/test_parse.py::test_empty_header -q
git bisect log > bisect.log
BAD=$(git rev-parse refs/bisect/bad)
echo "$BAD" > FIRST_BAD
git bisect reset
git switch --detach "$(cat FIRST_BAD)"

FIRST_BAD now names the commit that introduced failure. Files from that commit and its parent are listed. That list is the only allowed edit set.

git diff --name-only "$(cat FIRST_BAD)^" "$(cat FIRST_BAD)" > allowed_files.txt
cat allowed_files.txt

A flaky bisect run stops the workflow. Flaky tests will poison every later bound. The test harness is fixed before any patch.

3. Snapshot public names on the bad tree

Public names are the hidden merge risk. A model often exports a helper as the fix. The surface is captured before any human edit.

The script below is an unexecuted recipe. It walks AST nodes for top-level definitions. The glob is adjusted to match the real package.

# tools/api_snapshot.py
"""Write sorted top-level public names. Recipe only."""
from pathlib import Path
import ast
import sys

root = Path(sys.argv[1])
names = []
for path in sorted(root.rglob("*.py")):
    if "test" in path.parts:
        continue
    tree = ast.parse(path.read_text(encoding="utf-8"))
    for node in tree.body:
        if isinstance(node, (ast.FunctionDef, ast.ClassDef, ast.AsyncFunctionDef)):
            if not node.name.startswith("_"):
                names.append(f"{path.as_posix()}:{node.name}")
        if isinstance(node, ast.Assign):
            for t in node.targets:
                if isinstance(t, ast.Name) and t.id.isupper():
                    names.append(f"{path.as_posix()}:{t.id}")
Path("api.snap").write_text("\n".join(names) + "\n", encoding="utf-8")
print(f"wrote {len(names)} public names")

python tools/api_snapshot.py src/lib
cp api.snap api.snap.bad
wc -l api.snap.bad

api.snap.bad is stored beside the FIRST_BAD file. Those two files form the review contract. Later diffs must keep the snapshot byte identical.

4. Write a patch contract the model must obey

The bounds go into a small YAML file. That file is attached to the review prompt. The contract is the prompt, not the issue text.

# patch_contract.yaml
first_bad: "REPLACE_WITH_SHA"
allowed_files:
  - src/lib/parse.py
  - tests/test_parse.py
forbidden:
  - new public names
  - new modules
  - dependency bumps
  - formatter-only rewrites
required_tests:
  - pytest tests/test_parse.py::test_empty_header
  - pytest tests/test_parse.py
api_snapshot: api.snap.bad

allowed_files comes from the bisect file list. A model does not extend that list. New tests may import existing public names only.

A tiny checker can fail the review early. It is also a recipe, not a product. It runs before and after the patch.

# tools/check_contract.sh
set -euo pipefail
test -f FIRST_BAD
test -f api.snap.bad
test -f allowed_files.txt
git diff --name-only | while read -r f; do
  grep -Fxq "$f" allowed_files.txt || {
    echo "out of bounds: $f"
    exit 1
  }
done
python tools/api_snapshot.py src/lib
diff -u api.snap.bad api.snap

5. Patch only inside the window

Listed files are edited and nothing else. New helpers stay private with a leading underscore. The public snapshot is re-run after each edit.

python tools/api_snapshot.py src/lib
diff -u api.snap.bad api.snap

An empty snapshot diff is the surface pass. A new public name fails review immediately. Tests that pass do not excuse an API expansion.

The original test and its nearest file are run next.

pytest tests/test_parse.py::test_empty_header -q
pytest tests/test_parse.py -q

The first command must pass after the patch. The second command must remain fully green. A dead sibling test means leaked state.

Both test commands are recorded in the issue. They are not treated as the only merge rule. They only prove the window still holds.

6. Call a model only after the contract exists

A coding model can read the allowed files. It cannot choose the git bisect window. The SHA, file list, snapshot, and failing test are fed in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those options can host this review loop without a local GPU.

They still cannot replace bisect or a frozen API snapshot. A short prompt is used for the review. The model must return a unified diff only.

You may edit only these files:
{allowed_files}

Do not add public functions, classes, or constants.
Public names must match api.snap.bad exactly.

Failing test:
pytest tests/test_parse.py::test_empty_header

Return a unified diff and nothing else.

The diff is applied on a clean worktree. The check_contract script is run at once. The patch is dropped if the checker prints any path.

7. Gate license headers and version policy

OSS merges fail on paperwork as often as logic. Every touched file is scanned for a license header. The changelog must match the stated version policy.

while read -r f; do
  grep -q "SPDX-License-Identifier" "$f" || echo "missing SPDX: $f"
done < allowed_files.txt

SemVer treats a new public name as a minor bump. This workflow forbids that bump on a bugfix branch. Intentional API work moves to a separate issue.

A model does not rewrite CHANGELOG tone. One factual bullet is inserted for the crash. The rest of the file stays untouched.

Limitations

Bisect lies when tests are order dependent. It also lies when fixtures need production data. The AST snapshot also misses all re-exports.

It also misses C extensions and generated protobuf stubs. Private names can still break subclasses and plugins. Comment-only edits can still destroy git blame.

Models still invent imports and dead branches. The contract bounds files and public names only. It does not prove races or performance.

Free model access and a free server are availability claims. This article does not state quotas, model names, or hardware. Verify current terms on the product site before use.

Who should skip this method

Do not use this method under a security embargo. Do not use it for private CVE patches. Do not use it on generated vendor trees.

New contributors who cannot run bisect should wait. Header-only libraries need a different surface snapshot. Projects without a failing test still need reproduction first.

Monorepos with mixed languages need per-language snapshots. One Python AST walk will not cover Go exports. Extend the snapshot tool before claiming the gate holds.

What the two artifacts buy

The bisect SHA stops a model from rewriting history. The API snapshot stops a model from shipping new surface. Together they keep a bugfix inside one commit's blast radius.

Reviewers then read a small and bounded diff. They compare every changed path to allowed_files.txt. They also require an empty api.snap diff.

That is the merge bar for this class of patch. It is mechanical and cheap to re-run. It leaves taste debates out of the bugfix.

Maintainers with both artifacts can run this review loop. A free server can host that loop if local GPUs are missing. The contract stays in the issue, and the model stays inside it.

Review OSS Patches Against an Evidence Packet, Not the Diff Alone

Taylor Wang — Wed, 16 Sep 2026 04:38:20 +0000

OSS review fails when a patch arrives without portable evidence. Maintainers then rebuild the bug, the test, and the scope. An evidence packet restores that missing contract before review starts.

This workflow treats the packet as the review object. The diff is only an attachment. Local proof still happens before any remote scoring.

Why bare diffs stall maintainers

A pull request with only code hides the reproduction path. Reviewers hunt comments for commands that may not exist. Hours then go into rebuilding a one-line failure.

Issue threads mix measured facts with speculative refactors. Contributors often patch the speculation instead. Maintainers reject the extra files and the extra story.

Model-authored diffs make that stall more common. The generated change can look tidy while omitting proof. Reviewers still need a fail command, an oracle, and a file budget.

A typed packet stops that drift. It stores those three facts in one file. Reviewers read the file before they read the hunks.

Packet fields that actually matter

A useful packet stays small and machine-checkable. It does not dump the repository. It binds one issue to one oracle.

Keep these fields and drop decorative prose:

Canonical issue id and URL.
Environment hash from a lockfile.
Fail command that is red on main.
Oracle command that is green after the patch.
File allowlist with a numeric cap.
Explicit non-goals for extra refactors.
Model role locked to completeness scoring.

The packet file lives at the repository root. Name it EVIDENCE.yml for this proposal. Continuous integration can fail when the file is invalid.

Artifact: YAML packet for a CSV BOM bug

The next file is a proposal. It is not a published standard. Replace the hash after a local run.

# EVIDENCE.yml — proposal for one OSS bug
schema_version: 1
issue:
  id: "GH-1842"
  url: "https://github.com/example/libparse/issues/1842"
  summary: "CSV header row is dropped when the file has a BOM"
env:
  kind: "lockfile"
  hash_command: "sha256sum poetry.lock"
  recorded_hash: "REPLACE_WITH_LOCAL_SHA256"
repro:
  setup: "poetry install --sync"
  fail_cmd: "poetry run pytest tests/test_csv_bom.py::test_header_kept -q"
  fail_pattern: "AssertionError|FAILED"
oracle:
  pass_cmd: "poetry run pytest tests/test_csv_bom.py -q"
  pass_pattern: "passed"
patch_budget:
  max_files: 3
  max_hunks: 8
  allowlist:
    - "src/libparse/csv.py"
    - "tests/test_csv_bom.py"
    - "docs/csv.md"
non_goals:
  - "rewrite the parser"
  - "reformat unrelated modules"
model_review:
  role: "score_packet_only"
  forbidden: "propose_unrelated_files"

The packet travels with the pull request. Reviewers open it before the diff. Later scoring tools open it before they comment.

Artifact: a local validator

Incomplete packets must not reach reviewers. The script below is a proposal. Run it on a local checkout before push.

# validate_evidence.py — proposal, unexecuted in this article
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("Install pyyaml before running this validator.\n")
    sys.exit(2)

REQUIRED = (
    "schema_version",
    "issue",
    "env",
    "repro",
    "oracle",
    "patch_budget",
    "non_goals",
    "model_review",
)

KNOWN_RUNNERS = ("pytest", "cargo test", "go test", "npm test", "mvn test")


def load_packet(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict):
        raise ValueError("packet must be a mapping")
    return data


def check_required(data: dict) -> list[str]:
    errors = []
    for key in REQUIRED:
        if key not in data:
            errors.append(f"missing key: {key}")
    issue = data.get("issue") or {}
    for key in ("id", "url", "summary"):
        if not issue.get(key):
            errors.append(f"issue.{key} is empty")
    return errors


def check_budget(data: dict) -> list[str]:
    errors = []
    budget = data.get("patch_budget") or {}
    allow = budget.get("allowlist") or []
    max_files = budget.get("max_files")
    if not allow:
        errors.append("allowlist is empty")
    if isinstance(max_files, int) and len(allow) > max_files:
        errors.append("allowlist exceeds max_files")
    return errors


def check_commands(data: dict) -> list[str]:
    errors = []
    for section, key in (("repro", "fail_cmd"), ("oracle", "pass_cmd")):
        cmd = (data.get(section) or {}).get(key, "")
        if not isinstance(cmd, str) or not any(r in cmd for r in KNOWN_RUNNERS):
            errors.append(f"{section}.{key} lacks a known test runner")
    return errors


def main() -> int:
    path = Path(sys.argv[1] if len(sys.argv) > 1 else "EVIDENCE.yml")
    data = load_packet(path)
    errors = check_required(data) + check_budget(data) + check_commands(data)
    if errors:
        print("INVALID PACKET")
        for item in errors:
            print(f"- {item}")
        return 1
    print("PACKET OK")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Install the YAML parser, then validate the packet:

python3 -m pip install pyyaml
python3 validate_evidence.py EVIDENCE.yml

A failed validator means the pull request is not ready. Do not paste that packet into a model. Fix the fields first.

Numbered contributor workflow

Follow the gates in order. Do not skip the oracle. Do not start with a model-authored patch.

Gate 1. Mine facts from the issue thread

Copy claims that name inputs and outputs. Ignore calls to rewrite the module. Store the issue URL in the packet.

gh issue view 1842 --comments > /tmp/issue-1842.txt
rg -n "reproduce|expected|actual|fail" /tmp/issue-1842.txt

Write a summary under forty words. Put it in issue.summary. Drop every sentence that lacks an input or output.

Gate 2. Record the environment hash

Reviewers need the reporter's lockfile, not a similar one. Hash the lockfile only.

sha256sum poetry.lock | tee /tmp/env.hash
# paste the digest into env.recorded_hash

Paste the digest into env.recorded_hash. If the project uses images, hash the digest instead. Do not hash local caches or editor files.

Gate 3. Prove the fail command on main

Checkout main and install from the lockfile. Run only the recorded fail command. The exit code must be non-zero.

git fetch origin
git checkout origin/main
poetry install --sync
poetry run pytest tests/test_csv_bom.py::test_header_kept -q
echo $?

A green fail command invalidates the packet. Rewrite the test until main is red. Then freeze repro.fail_cmd and keep it copy-pasteable.

Gate 4. Write the smallest oracle

The oracle is the command that must turn green. Prefer one test file. Do not use the full suite as the only gate.

# proposal: keep the oracle equal to one file
poetry run pytest tests/test_csv_bom.py -q

Store the command and a pass pattern. Later log scoring depends on that pattern. Keep the pattern boring and stable.

Gate 5. Freeze the file budget

Search the tree for symbols named in the issue. Cap the resulting file list. Extra paths become non-goals.

git grep -n "BOM\|header" -- src tests

Write the allowlist into patch_budget. Three files is a reasonable starting cap. Raise the cap only with a written reason in non_goals.

Gate 6. Patch inside the allowlist

Create a branch named after the issue id. Edit only listed paths. Keep the commit message factual.

git checkout -b fix/1842-csv-bom-header
# edit src/libparse/csv.py and tests/test_csv_bom.py only
git diff --stat
git add src/libparse/csv.py tests/test_csv_bom.py EVIDENCE.yml
git commit -m "Fix CSV BOM header drop (GH-1842)"
git diff origin/main...HEAD --name-only

Confirm the name-only diff is a subset. Include EVIDENCE.yml as an extra allowed path. Reject surprise files before push.

Gate 7. Replay fail and oracle on the branch

Run the original fail test on the patched tree. Then run the oracle command. Save both logs beside the packet.

mkdir -p evidence
poetry run pytest tests/test_csv_bom.py::test_header_kept -q | tee evidence/fail-on-branch.log
poetry run pytest tests/test_csv_bom.py -q | tee evidence/oracle.log

Attach evidence/oracle.log in the pull request body. Do not use screenshots as the only proof. Text logs remain searchable in review tools.

Gate 8. Score completeness before human review

Score each gate zero or one. Require six points before asking a maintainer. The rubric lives in the next section.

A model may apply the same rubric later. It must not expand the allowlist. It must not become the author of record.

Completeness rubric

This table is a proposal. Projects may tighten the pass bar. Humans still merge.

Gate	Pass condition	Score
Fail command recorded	`repro.fail_cmd` is copy-pasteable	0/1
Fail proven on main	Log shows the fail pattern	0/1
Oracle recorded	`oracle.pass_cmd` is copy-pasteable	0/1
Oracle proven on branch	Log shows the pass pattern	0/1
Env hash matches	Lockfile digest equals `env.recorded_hash`	0/1
Allowlist respected	`git diff --name-only` stays inside allowlist	0/1
Non-goals honored	Diff has no drive-by refactors	0/1
Model stayed in role	Review comments do not add files	0/1

A tiny scorer can read logs. The next script is not production code. Treat it as a proposal.

# score_packet.py — proposal, unexecuted in this article
from __future__ import annotations

import re
from pathlib import Path


def score_log(path: Path, pattern: str, expect_match: bool) -> int:
    text = path.read_text(errors="replace")
    found = re.search(pattern, text) is not None
    return int(found is expect_match)


def score_allowlist(changed: list[str], allow: list[str], extras: list[str]) -> int:
    allowed = set(allow) | set(extras)
    return int(all(path in allowed for path in changed))


# Example after CI writes logs:
# fail_score = score_log(Path("evidence/fail-main.log"), r"FAILED|AssertionError", True)
# pass_score = score_log(Path("evidence/oracle.log"), r"passed", True)
# files_score = score_allowlist(changed, allow, extras=["EVIDENCE.yml"])

CI should fail below six points. Maintainers then review a complete packet. Incomplete packets bounce without a long debate.

Optional remote completeness scoring

Local proof stays first. A remote model is a second reader. It comments on missing fields and allowlist drift.

Keep the prompt narrow and mechanical:

Role: packet completeness reviewer.
Inputs: EVIDENCE.yml, git diff, oracle.log.
Task: score the eight gates. Quote missing fields.
Forbidden: new files, new features, full-repo rewrites.
Output: table of 0/1 scores and one paragraph of risks.

Upload the packet, the bounded diff, and the oracle log. Do not upload the full repository. Do not upload secrets or production dumps.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. This article asserts only those two availability facts. No model names, quotas, hardware, or duration are claimed. After validate_evidence.py prints PACKET OK, that optional worker can score the eight gates as a checklist, never as a merge vote.

The workflow still holds if that worker is absent. The validator, the logs, and the rubric already gate the packet. Maintainers can replay the same commands without any model.

Maintainer replay in a few commands

Maintainers can check a stranger's packet quickly. They replay commands instead of rereading essays. The packet is the checklist.

git fetch origin pull/221/head:pr-221
git checkout pr-221
python3 validate_evidence.py EVIDENCE.yml
sha256sum poetry.lock
# compare the digest to env.recorded_hash
git diff origin/main...HEAD --name-only
poetry install --sync
poetry run pytest tests/test_csv_bom.py -q

Reject the pull request when any gate fails. Request an updated packet. Do not request a longer cover letter.

Compare the name-only diff with patch_budget.allowlist. Extra files are a hard fail. Missing oracle logs are a hard fail as well.

What this method does not prove

This method assumes a deterministic test runner. Flaky tests break the fail and pass contract. Intermittent bugs need a different harness.

The validator does not prove security properties. It does not prove performance budgets. It does not prove API stability beyond the oracle command.

Model scores can be wrong. A high score can hide a logic error. A low score can flag a valid one-line fix.

Free model access and a free server option can change. This article does not rank tools. This article does not promise capacity or permanence.

The YAML schema is a proposal. Field names may change per repository. Do not treat the file as an ecosystem standard.

Who should skip this workflow

Do not publish packets for embargoed security issues. Public fail logs can leak exploit details. Use private maintainer channels instead.

Do not invent an oracle for a project without tests. The method has nothing honest to score then. Add tests first, then open the packet.

Do not send proprietary code to a remote worker. Keep the packet local in that case. License review comes before any upload.

Skip the eight gates for a one-word docs fix. The overhead would exceed the change. Use judgment on tiny typo pulls.

Do not list a model as the patch author. The contributor owns the diff. The maintainer owns the merge.

Practical close

OSS review gets faster when evidence is typed. The packet carries the issue, the oracle, and the file budget. Models may score that packet after local proof exists. They should not replace the oracle or the maintainer.

Reproduce the Reporter's Container Before You Patch an OSS Bug

Taylor Wang — Mon, 14 Sep 2026 21:57:25 +0000

Most rejected open-source patches fail for a boring reason. The bug was never reproduced on the reporter's environment. The diff looks reasonable, the tests pass locally, and the maintainer closes the pull request with one line: "cannot reproduce."

This post describes a five-step loop that turns an issue report into a patch a maintainer can verify. The loop is: capture an environment ledger, bake a reproducer container, require a failing exit code, patch the smallest diff, then re-run the same script. A hosted model can help review the evidence bundle at step five. It should never be asked to guess at the bug.

Nothing here depends on the project's language. The examples use Bash, Docker, and Python.

1. Capture the environment ledger before reading the diff

Issue reports contain the environment in prose. Contributors skim that prose and start editing. That is the first failure.

Extract the environment into a machine-readable ledger first. The ledger records only facts the reporter supplied.

# tools/env-ledger.sh
set -euo pipefail

issue="${1:?usage: env-ledger.sh <issue.md>}"
out="env-ledger.json"

jq -n \
  --arg os    "$(grep -iPo '(?<=^OS:\s).*' "$issue" | head -1)" \
  --arg runtime "$(grep -iPo '(?<=^Runtime:\s).*' "$issue" | head -1)" \
  --arg version "$(grep -iPo '(?<=^Version:\s).*' "$issue" | head -1)" \
  --arg repro "$(grep -iPo '(?<=^Steps:\s).*' "$issue" | head -1)" \
  '{os:$os, runtime:$runtime, runtime_version:$version, repro_steps:$repro, captured_from:"issue"}' \
  > "$out"

grep -q '"os":""' "$out" && { echo "ledger incomplete: ask reporter"; exit 2; }
echo "ledger written to $out"

A missing field is a request for information, not an invitation to guess. Send the ledger back in the issue thread and ask the reporter to fill the gaps.

2. Bake a reproducer container from the ledger

The ledger is the only input to the container definition. No version comes from the contributor's laptop.

# repro/Dockerfile
FROM python:3.11-slim

ARG RUNTIME_VERSION
RUN pip install --no-cache-dir "acme-lib==${RUNTIME_VERSION}"

COPY repro/ /repro/
WORKDIR /repro
ENTRYPOINT ["bash", "run.sh"]

Pin one thing at a time. If the ledger says 3.9.7, the build fails loudly on 3.9.8. That failure is useful evidence, not an inconvenience.

Record the image digest, not just the tag. Tags move; digests do not.

docker build \
  --build-arg RUNTIME_VERSION="$(jq -r .runtime_version env-ledger.json)" \
  -t acme-repro:local repro/ | tee build.log

docker image inspect acme-repro:local --format '{{index .RepoDigests 0}}' \
  | tee image-digest.txt

The digest line belongs in the pull request body. A maintainer can rebuild the exact same image from it.

3. Require a failing exit code before touching source

A reproducer that exits zero proves nothing. The script must assert the bug is present.

# repro/run.sh
set -uo pipefail

python - <<'PY' > output.txt 2>&1
try:
    import acme
    acme.parse(b'{"ok": true, "retries": 0}')
    print("NO_CRASH")
except Exception as exc:
    print(type(exc).__name__, exc)
PY

cat output.txt

grep -q 'NO_CRASH' output.txt && { echo "BUG NOT PRESENT"; exit 1; }
echo "BUG PRESENT"; exit 0

The exit codes are inverted on purpose. exit 0 means the bug reproduced. exit 1 means the environment is wrong and the contributor must stop.

Run this before writing any patch. Save the output as before.log.

4. Patch the smallest diff, then re-run the identical script

Edit source only. Do not edit the reproducer to make it pass. The reproducer is the contract.

After the edit, the same script must print NO_CRASH and exit 1.

docker run --rm acme-repro:local > after.log; echo "exit=$?"
# before.log -> Exception ... exit=0
# after.log  -> NO_CRASH     exit=1

Two log files and two exit codes are the whole argument. The table below covers the drift cases that appear most often.

Drift source	Symptom	Evidence to add
Unpinned dependency	Passes locally, fails for reporter	`pip freeze` diff in the ledger
OS-specific path handling	Bug vanishes in the container	`before.log` with the original traceback
Locale or timezone	Intermittent output	Container `TZ` and `LANG` values
Reporter already patched	`BUG NOT PRESENT` on first run	Version string from the issue
Non-deterministic seed	Flaky reproducer	Seed value plus ten consecutive runs

If BUG NOT PRESENT appears, the ledger is wrong. Go back to step one. Do not weaken the assertion.

5. Give the reviewer an evidence bundle, not a repository

Step five is where hosted models help, and where they mislead. A model reading a whole repository invents context. A model reading six artifacts stays grounded.

The bundle is small and fixed:

env-ledger.json
image-digest.txt
before.log and after.log
the unified diff
the reproducer script, unchanged

A hashing step keeps the bundle honest and detects accidental edits during review.

find bundle -type f -print0 | sort -z \
  | xargs -0 sha256sum > bundle/SHA256SUMS
sha256sum -c bundle/SHA256SUMS

This is where MonkeyCode's free model access is relevant. The operator offers free model access and a free server option, and I use the free model access to read bundles like this one. The free server option matters when a reproducer build is slow and the contributor does not want to occupy a laptop for an hour. The bundle stays the same either way; only the machine running the build changes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Ask the model narrow questions. "Does after.log show the same code path as before.log?" is answerable. "Is this patch correct?" is not, and the answer will be confident and unreliable.

Limitations

This loop assumes the reporter can describe their environment. Vague reports need a round trip before any code changes.

It also assumes a deterministic bug. Race conditions and memory-pressure failures need repeated runs and a different evidence format.

Digest pinning adds minutes to every build. That cost is acceptable for a bug that blocked someone in production, and wasteful for a documentation fix.

The reviewer budget is real. Sending a large bundle to any hosted model costs time. Keeping the bundle small is part of the method, not a preference.

Who should not use this loop

Do not use it for trivial patches. Typos and doc edits do not need a container.

Do not use it when the project forbids new CI artifacts. Some maintainers reject added build files, and their contribution guide wins over this workflow.

Do not use it as a substitute for reading the code. The reproducer proves the bug exists. It says nothing about whether the fix is correct.

Closing

Environment drift is boring, and boring problems have checkable answers. Capture the ledger, bake the container, require the failing exit code, patch one diff, then re-run the identical script. The evidence bundle then goes to a reviewer with a question it can actually answer.

Readers who want to try this loop without provisioning anything can start from the free model access and free server option that MonkeyCode provides. The workflow above works without them; they only remove a local build step.

Lock OSS Fixes to a Fail-Pass Command Pair

Taylor Wang — Sun, 13 Sep 2026 20:50:04 +0000

Open-source patches fail when models invent extra scope. A fail-pass command pair freezes the bug before review. Maintainers then judge a contract, not a chat thread.

Issue threads leak patch scope

Public issues mix traces, guesses, and leftover diffs. A review model reads that mix as license. Neighbor files then change without a failing command.

Chat history cannot be replayed on a clean clone. Exit codes can be replayed on a clean clone. A contract stores those codes before any model runs.

The contract is the review surface

The YAML card lives beside the local clone. It names one fail command and one pass command. It also names an invariant command and a hard budget.

The card does not propose a patch. Review models may annotate the signed card only. Those models may not add files or raise limits.

# issue-contract.yml
id: "ISSUE-4419"
origin: "https://github.com/example/libparse/issues/4419"
base_ref: "main"
fail:
  workdir: "."
  cmd: ["python", "-m", "pytest", "tests/test_header.py::test_empty_header", "-q"]
  expect_exit: 1
pass:
  workdir: "."
  cmd: ["python", "-m", "pytest", "tests/test_header.py::test_empty_header", "-q"]
  expect_exit: 0
invariant:
  workdir: "."
  cmd: ["python", "-m", "pytest", "tests/test_header.py", "-q"]
  expect_exit: 0
budget:
  max_files: 3
  max_changed_lines: 80
  deny_globs:
    - "pyproject.toml"
    - "setup.cfg"
    - ".github/**"
    - "docs/**"

This block is a template, not production evidence. Maintainers replace the path with the issue command. Deny globs stay tight against build and docs trees.

Why two expected exits matter

The same test is the cheapest honest oracle. The base tree must still fail that test. The patched tree must pass that same test.

A new test that exists only in the patch is weaker. It never failed on the base hash. The contract then cannot prove the original bug.

Prefer the reporter command when it is deterministic. Write a new failing test only when none exists. That new test must fail on the unpatched hash first.

Sign the card in eight steps

The maintainer follows the eight steps in strict order. Speed is not a reason to skip a step. The model enters only at step five.

1. Pin the reported revision

The contributor clones the project at the issue ref. The clone must avoid a dirty local worktree. The commit hash is recorded beside the card.

git clone https://github.com/example/libparse.git
cd libparse
git checkout --detach origin/main
git rev-parse HEAD > .contract-base.sha

The hash belongs in the review packet. Later diffs must start from that hash. Detached HEAD keeps accidental commits off main.

2. Record the fail command on a clean tree

The contributor runs the command the issue claims fails. Only the exit code is written into the card. Logs stay on disk until the card is signed.

python -m pytest tests/test_header.py::test_empty_header -q
echo $?

The fail block is valid only on a matching exit. A passing main tree means a stale issue. Signing stops when the fail command already passes.

3. Record the invariant command

The contributor picks a nearby suite that must keep passing. The same file without the failing test is enough. The full project suite waits until later sign-off.

python -m pytest tests/test_header.py -q
echo $?

The invariant must pass on the base hash. A failing invariant means a dirty tree. The contract cannot be signed in that state.

4. Freeze the budget before any edit

The contributor counts files the stack actually names. max_files becomes that count plus one. max_changed_lines comes from the smallest plausible fix.

A review model does not pick these numbers. A human maintainer rereads the original issue first. Budget limits rise only after that reread.

5. Draft a patch against the card

A model may see the contract and a short stack. It may not receive a full tree listing. The only accepted output is a unified diff.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those options can host the runner and a bounded diff review. They do not replace the fail-pass pair or set the budget.

The prompt stays mechanical and short. The YAML and git diff --stat are the only inputs. Prose that proposes files outside the card is rejected.

6. Apply the diff on a second worktree

A second worktree keeps the base tree clean. git apply --check runs before a real apply. A check failure ends the run at once.

git worktree add ../libparse-fix "$(cat .contract-base.sha)"
cd ../libparse-fix
git apply --check /tmp/issue-4419.diff
git apply /tmp/issue-4419.diff

Maintainers do not hand-edit a rejected diff. A new diff must match the signed card. The extra worktree must stay disposable after review.

7. Run fail, pass, and invariant together

The Python runner below executes the signed card. It also enforces the budget against git diff. The command runs only inside the disposable worktree.

#!/usr/bin/env python3
"""Run an OSS issue contract against a unified diff.

This script is a template. It is not production CI.
"""
from __future__ import annotations

import argparse
import fnmatch
import subprocess
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit(2)


def run_cmd(cmd, workdir):
    proc = subprocess.run(
        cmd,
        cwd=workdir,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
    )
    return proc.returncode, proc.stdout


def git(args, workdir):
    code, out = run_cmd(["git", *args], workdir)
    if code != 0:
        raise SystemExit(f"git {' '.join(args)} failed:\n{out}")
    return out.strip()


def load_card(path):
    data = yaml.safe_load(Path(path).read_text())
    for key in ("fail", "pass", "invariant", "budget"):
        if key not in data:
            raise SystemExit(f"card missing {key}")
    return data


def numstat(workdir):
    out = git(["diff", "--numstat", "--", "."], workdir)
    files = []
    total = 0
    for line in out.splitlines():
        if not line.strip():
            continue
        added, deleted, name = line.split("\t", 2)
        if added == "-" or deleted == "-":
            raise SystemExit(f"binary file in diff: {name}")
        total += int(added) + int(deleted)
        files.append(name)
    return files, total


def glob_hit(path, patterns):
    return any(fnmatch.fnmatch(path, pat) for pat in patterns)


def check_step(label, code, expected):
    if code != expected:
        raise SystemExit(f"{label}: exit {code}, expected {expected}")
    print(f"{label}: exit {code} (ok)")


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--card", required=True)
    parser.add_argument("--diff", required=True)
    parser.add_argument("--workdir", default=".")
    args = parser.parse_args()
    card = load_card(args.card)
    root = args.workdir

    base = git(["rev-parse", "HEAD"], root)
    print(f"base: {base}")

    code, _ = run_cmd(card["fail"]["cmd"], card["fail"].get("workdir", root))
    check_step("fail on base", code, card["fail"]["expect_exit"])

    code, _ = run_cmd(
        card["invariant"]["cmd"],
        card["invariant"].get("workdir", root),
    )
    check_step("invariant on base", code, card["invariant"]["expect_exit"])

    diff_path = str(Path(args.diff).resolve())
    git(["apply", "--check", diff_path], root)
    git(["apply", diff_path], root)
    print("apply: ok")

    files, lines = numstat(root)
    budget = card["budget"]
    if len(files) > budget["max_files"]:
        raise SystemExit(
            f"files changed: {len(files)} > {budget['max_files']}"
        )
    print(f"files changed: {len(files)} (ok)")
    if lines > budget["max_changed_lines"]:
        raise SystemExit(
            f"changed lines: {lines} > {budget['max_changed_lines']}"
        )
    print(f"changed lines: {lines} (ok)")

    deny = budget.get("deny_globs", [])
    hits = [name for name in files if glob_hit(name, deny)]
    if hits:
        raise SystemExit(f"deny glob hits: {hits}")
    print("deny glob hits: 0 (ok)")

    code, _ = run_cmd(card["pass"]["cmd"], card["pass"].get("workdir", root))
    check_step("pass on patch", code, card["pass"]["expect_exit"])

    code, _ = run_cmd(
        card["invariant"]["cmd"],
        card["invariant"].get("workdir", root),
    )
    check_step("invariant on patch", code, card["invariant"]["expect_exit"])
    print("GATE: PASS")


if __name__ == "__main__":
    main()

Install pyyaml as the extra dependency first. Then run the gate from the disposable worktree.

pip install pyyaml
python contract_gate.py --card issue-contract.yml --diff /tmp/issue-4419.diff

8. Accept a green gate plus a human note

A green gate is necessary and still not sufficient. A maintainer still reads the unified diff. The merge note must cite the contract id.

Merge proceeds only when all three commands match. The issue closes with the hash and card path. The disposable worktree is dropped after the merge.

What the runner treats as truth

The runner treats exit codes as the only oracle. Stdout is stored for humans only. Stdout never scores the gate.

Budget math uses git diff --numstat on the worktree. Binary files count as an immediate miss. Denied globs fail the gate for a single line.

The fail command runs on the base hash. The pass command runs on the patched tree. The invariant command runs on both trees.

That last rule catches bad cards early. An invariant that fails on main is invalid. An invariant that fails after the patch is a regression.

Decision table before signing

Observation on the base hash	Action
fail matches, invariant passes	Sign the card and continue
fail command already passes	Stop, the issue is stale
invariant already fails	Stop, the tree is dirty
fail command is not deterministic	Stop, do not sign

Sign only the first row. The other rows need human triage. Models do not override this table.

Sample session

$ python contract_gate.py --card issue-contract.yml --diff /tmp/issue-4419.diff
base: 9f3c1a2
fail on base: exit 1 (ok)
invariant on base: exit 0 (ok)
apply: ok
files changed: 2 (ok)
changed lines: 37 (ok)
deny glob hits: 0 (ok)
pass on patch: exit 0 (ok)
invariant on patch: exit 0 (ok)
GATE: PASS

A failed session prints the first broken rule. Later rules can stay unrun. The contributor then edits the diff, not the card.

Limitations

The contract cannot encode timing bugs well. Flaky tests poison the fail block. Snapshot tests can pass for the wrong reason.

Multi-package monorepos need one card per package. A single budget across packages is too coarse. Generated files will blow the line cap fast.

The runner trusts every command in the YAML. A hostile card can run arbitrary processes. Maintainers must read the card before execution.

Free model access does not prove behavioral correctness. A free server does not replace project CI. Environment-specific failures can still slip through.

Who should skip this gate

This gate does not serve design debates. It does not serve refactors with no failing command. It does not serve issues without a local reproduction.

Security patches that must not run locally need another path. Docs-only changes need a docs checklist instead. Brand new clones without tests gain little here.

Keep the card with the pull request

Paste the YAML into the pull request body. Link the runner command in the same note. Reviewers then replay the gate without the original chat.

Scope drift is a process bug, not a model bug. A fail-pass pair is a small process fix. Sign the card first and let models comment later.

A free model review on a throwaway clone can try the gate. Keep the card in the pull request so reviewers can replay it. Do not paste the full issue thread into the prompt.

Derive OSS Review Scope From the Crash Transcript

Taylor Wang — Sat, 12 Sep 2026 18:47:50 +0000

OSS patch review should start from a crash transcript. The transcript defines files, claims, and comments. A model that cannot cite that transcript should not review the diff.

Whole-tree prompts invite drive-by refactors. Review threads then drift into style. The original failure leaves the discussion. A frozen worktree plus a captured transcript keeps the bug in view.

This article proposes a local gate. Stack frames become the file scope. Review notes must cite transcript line numbers. The scripts below are a workflow, not a production study.

Why a transcript beats a repo dump

A crash log names the path that actually failed. Maintainers already trust that path. A model does not need the rest of the tree.

Issue text is often incomplete. Stack frames are harder to fake. The patch should touch those frames first. Comments that ignore the log are noise.

This gate does not replace tests. It only binds review scope to one reproduction. Human judgment still owns the merge.

What the gate records

The gate writes four local artifacts. Each file is small and replayable.

worktree.lock stores the frozen commit SHA.
repro.cmd stores the exact failing command.
transcript.txt stores stdout, stderr, and exit code.
scope.paths stores files parsed from stack frames.

The patch may change only files in scope.paths, plus tests. Review comments must cite transcript.txt line numbers. Anything else fails the gate.

Step 1: Freeze a disposable worktree

Do not review on the contributor clone. Create a worktree from the issue SHA. Keep main untouched.

#!/usr/bin/env bash
set -euo pipefail
# Proposed local workflow. Run from the maintainer clone.
ISSUE_SHA="${1:?need issue sha}"
BUG_ID="${2:?need bug id}"
ROOT="$(git rev-parse --show-toplevel)"
WT="${ROOT}/.review-worktrees/${BUG_ID}"

git fetch --quiet origin "${ISSUE_SHA}"
git worktree add --detach "${WT}" "${ISSUE_SHA}"
printf '%s\n' "${ISSUE_SHA}" > "${WT}/worktree.lock"
echo "frozen ${ISSUE_SHA} at ${WT}"

The worktree dies after the review. No model process should write it. Use a second clone if the first tool is untrusted.

Step 2: Capture one reproduction transcript

Run the command the issue claims. Capture both streams. Store the exit code. Do not edit the log by hand.

#!/usr/bin/env bash
set -euo pipefail
WT="${1:?need worktree}"
shift
CMD=( "$@" )
LOG="${WT}/transcript.txt"
printf 'cmd: %q ' "${CMD[@]}" > "${LOG}"
printf '\n---\n' >> "${LOG}"
set +e
( cd "${WT}" && "${CMD[@]}" ) >> "${LOG}" 2>&1
STATUS=$?
set -e
printf '\n---\nexit: %s\n' "${STATUS}" >> "${LOG}"
printf '%s\n' "${CMD[*]}" > "${WT}/repro.cmd"
if [[ "${STATUS}" -eq 0 ]]; then
  echo "reproduction did not fail; stop the review" >&2
  exit 2
fi

A green command is not a review input. The bug is not proven. Stop before any model sees a diff.

Step 3: Derive file scope from stack frames

Parse the transcript for source paths. Keep paths that exist in the worktree. Drop site-packages and generated noise.

#!/usr/bin/env python3
"""Proposed parser for Python and similar 'File "path", line N' frames."""
from pathlib import Path
import re
import sys

FRAME = re.compile(r'File "([^"]+)", line (\d+)')
SKIP_PARTS = (".venv", "site-packages", "dist-packages", "__pycache__")


def existing_repo_file(worktree: Path, raw: str) -> Path | None:
    candidate = Path(raw)
    if not candidate.is_absolute():
        candidate = worktree / candidate
    try:
        resolved = candidate.resolve()
        resolved.relative_to(worktree.resolve())
    except (OSError, ValueError):
        return None
    if not resolved.is_file():
        return None
    if any(part in SKIP_PARTS for part in resolved.parts):
        return None
    return resolved.relative_to(worktree.resolve())


def main() -> int:
    worktree = Path(sys.argv[1])
    text = (worktree / "transcript.txt").read_text(encoding="utf-8", errors="replace")
    found: list[str] = []
    for match in FRAME.finditer(text):
        rel = existing_repo_file(worktree, match.group(1))
        if rel is None:
            continue
        key = rel.as_posix()
        if key not in found:
            found.append(key)
    if not found:
        print("no in-repo stack frames; refuse model review", file=sys.stderr)
        return 2
    out = worktree / "scope.paths"
    out.write_text("\n".join(found) + "\n", encoding="utf-8")
    print(f"wrote {out} ({len(found)} paths)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The parser is language-shaped. Java, Go, and Rust need different frame regexes. Empty scope is a hard stop, not a prompt to guess.

Step 4: Bound the incoming patch

Apply the patch inside the worktree. Then list changed files. Reject extras before review text exists.

#!/usr/bin/env bash
set -euo pipefail
WT="${1:?need worktree}"
PATCH="${2:?need patch file}"
cd "${WT}"
git apply --check "${PATCH}"
git apply "${PATCH}"
mapfile -t CHANGED < <(git diff --name-only)
mapfile -t SCOPE < <(grep -v '^$' scope.paths)
# tests may change even when they are not in the crash frames
allow() {
  local f="$1"
  [[ "${f}" == tests/* || "${f}" == test/* || "${f}" == *_test.py ]] && return 0
  printf '%s\n' "${SCOPE[@]}" | grep -Fxq "${f}"
}
for f in "${CHANGED[@]}"; do
  if ! allow "${f}"; then
    echo "out-of-scope path: ${f}" >&2
    exit 3
  fi
done
echo "patch stays inside transcript scope"

Drive-by cleanup fails here. Format-only files fail here. A later model cannot excuse them.

Step 5: Demand transcript citations in the review

Review notes live in review.md. Each claim needs a transcript: cite. The checker reads the log length. Invalid lines fail the packet.

#!/usr/bin/env python3
"""Fail review notes that do not cite transcript.txt line numbers."""
from pathlib import Path
import re
import sys

CITE = re.compile(r"transcript:(\d+)(?:-(\d+))?")
CLAIM = re.compile(r"(?m)^-\s+")


def main() -> int:
    worktree = Path(sys.argv[1])
    notes = (worktree / "review.md").read_text(encoding="utf-8")
    log_lines = (worktree / "transcript.txt").read_text(encoding="utf-8").splitlines()
    n = len(log_lines)
    claims = CLAIM.findall(notes)
    cites = CITE.findall(notes)
    if not claims:
        print("review.md has no claim bullets", file=sys.stderr)
        return 2
    if len(cites) < len(claims):
        print("every claim needs a transcript:N cite", file=sys.stderr)
        return 3
    for start, end in cites:
        a = int(start)
        b = int(end) if end else a
        if a < 1 or b < a or b > n:
            print(f"cite transcript:{a}-{b} is outside 1-{n}", file=sys.stderr)
            return 4
    print(f"accepted {len(claims)} claims against {n} transcript lines")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

A valid review.md looks like this. Short claims. Concrete cites. No repo tour.

# Review packet for BUG-1842

- The parser still raises on empty tokens. transcript:14-21
- The patch guards that branch in `pkg/scan.py`. transcript:18
- The new unit test replays the same command. transcript:2

A rejected note looks like this. It may be true. It still fails the gate.

- Consider renaming helpers for clarity.
- Also reformat imports in adjacent modules.

Those sentences do not point at the crash. They do not belong in this review.

Step 6: Replay after the patch

The same command must change outcome. Capture a second transcript. Diff the two logs. Keep both files.

#!/usr/bin/env bash
set -euo pipefail
WT="${1:?need worktree}"
CMD_FILE="${WT}/repro.cmd"
read -r -a CMD < "${CMD_FILE}"
AFTER="${WT}/transcript.after.txt"
set +e
( cd "${WT}" && "${CMD[@]}" ) > "${AFTER}" 2>&1
STATUS=$?
set -e
if [[ "${STATUS}" -ne 0 ]]; then
  echo "command still fails after patch" >&2
  exit 4
fi
diff -u "${WT}/transcript.txt" "${AFTER}" > "${WT}/transcript.delta" || true
echo "wrote transcript.delta"

A silent success is not enough. The delta shows what vanished. Reviewers can read the delta without opening the tree.

Where a free model fits

The model reads three files only. Those files are transcript.txt, transcript.delta, and the patch. It writes review.md. The citation checker still runs locally.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option that can host this narrow review loop. The scripts do not require that product. They fail closed without any model at all.

Do not paste secrets into the prompt. Do not upload .env files. Do not send the full worktree. The transcript is the budget.

A compact decision table

Use this table before inviting a model. It is a process check, not a score.

Signal	Action
Reproduction exits zero	Stop. No review.
No in-repo stack frames	Stop. Fix the parser or the bug report.
Patch touches extra files	Reject. Ask for a scoped revision.
`review.md` lacks cites	Reject the comments, not necessarily the patch.
After-patch command still fails	Reject the patch.
Delta is empty after a claimed fix	Reject. The log did not change.
Cite points at setup noise	Ask for a tighter command.

Humans still read security-sensitive diffs. This table does not bless cryptography changes. It only blocks unscoped chatter.

Limitations

The Python frame regex misses many languages. A Go panic needs another parser. A C abort may print only addresses.

Flaky tests poison the transcript. One green rerun can fake a fix. Pin the seed when the suite allows it.

Generated sources confuse scope. Vendor trees and protobuf outputs look like project files. Extend SKIP_PARTS before trusting the list.

The citation checker is syntactic. A model can cite the right lines and still be wrong. Maintainers read the claims.

Networked reproduction is out of scope. The gate assumes a local command. External fixtures need a recorded snapshot, not a live API.

Who should not use this

New contributors who need mentoring should not face this gate first. Docs-only patches have no crash log. Do not invent a transcript for them.

Security embargo patches should stay off shared model hosts. The frozen worktree still helps. The model step should not.

Huge monorepos with weak traces will reject most patches. Fix tracing first. Do not loosen cites to keep a bot busy.

Teams without a failing command should pick another process. This workflow has nothing to cite.

Close the loop

Delete the worktree after the decision. Keep the four artifacts beside the issue. The next reviewer can replay without a chat history.

git worktree remove --force "${WT}"

The crash transcript remains the contract. Models may draft review.md. They do not choose the scope. A spare free server can run the same commands. Keep the worktree disposable either way.

Prove the OSS Bug Locally Before a Model Reviews the Diff

Taylor Wang — Fri, 11 Sep 2026 16:21:44 +0000

OSS patches should meet a frozen reproduction first. A model may critique that evidence later. It must not author the change.

This split keeps review honest. Merge authority stays with humans.

The inverted PR pattern

Many AI-assisted PRs invert the order. A model drafts a patch from issue prose. Tests arrive as decoration. Reviewers then argue about wording, not proof.

Issue text is not a reproduction. Green CI on a new test can still miss the bug. A model suggestion is not a merge vote.

The safer order is boring and strict. Freeze a failing command. Patch by hand in a tight scope. Only then request a read-only model pass.

Three gates, never reordered

The workflow uses three sequential gates. Gate one freezes a failing local command. Gate two applies a human-scoped patch. Gate three runs a shadow review on evidence only.

Skip a gate and later gates lie. The model must not see the diff before gate one succeeds.

Gate 1 — Freeze the reproduction

The contributor starts on a clean checkout. No patch exists yet. One command must fail in a recorded way.

Check out the default branch at a known commit.
Install only the project's documented dependencies.
Run one command that should fail today.
Store the command, exit code, and output hash.
Stop immediately if that command passes.

The receipt is the source of truth. Chat summaries are not. A passing command means the bug is not frozen.

The script below is a proposed local harness. It is not a published benchmark and was not timed in this article.

#!/usr/bin/env bash
# repro-receipt.sh — record a failing command before any patch.
set -euo pipefail

RECEIPT="${RECEIPT:-repro-receipt.env}"
CMD="${1:?usage: repro-receipt.sh <shell-command>}"

if [[ -n "${PATCH_APPLIED:-}" ]]; then
  echo "refusing: patch flag set before the first receipt" >&2
  exit 2
fi

tmp="$(mktemp)"
set +e
bash -lc "$CMD" >"$tmp" 2>&1
code=$?
set -e

if [[ "$code" -eq 0 ]]; then
  echo "refusing: command passed; no bug is frozen" >&2
  rm -f "$tmp"
  exit 3
fi

hash="$(sha256sum "$tmp" | awk '{print $1}')"
{
  printf 'RECEIPT_CMD=%q\n' "$CMD"
  echo "RECEIPT_EXIT=$code"
  echo "RECEIPT_SHA256=$hash"
  echo "RECEIPT_COMMIT=$(git rev-parse HEAD)"
  echo "RECEIPT_STATUS=failed"
} >"$RECEIPT"

echo "wrote $RECEIPT (exit=$code sha=$hash)"
rm -f "$tmp"

Example first run on a parser bug:

chmod +x repro-receipt.sh
./repro-receipt.sh 'python -m pytest tests/test_parse.py::test_empty_header -q'
cat repro-receipt.env

A useful receipt names one test, not the whole suite. Broad suites hide the original failure. Maintainers cannot replay what they cannot name.

Gate 2 — Scope the human patch

After the receipt exists, a human edits files. The patch must change the failing behavior. Drive-by refactors wait for a later PR.

Read RECEIPT_CMD again before touching code.
Change the smallest set of files that can fix it.
Re-run the same command with no flag edits.
Require a zero exit and a new output hash.
Record paths with git diff --name-only.

Unchanged output hashes are a red flag. The command may have passed for the wrong reason. Silence is not proof.

#!/usr/bin/env bash
# prove-patch.sh — same command, opposite exit, after the patch.
set -euo pipefail
# Proposed local harness. Treat as unexecuted sample code.

# shellcheck disable=SC1091
source ./repro-receipt.env
tmp="$(mktemp)"
set +e
bash -lc "$RECEIPT_CMD" >"$tmp" 2>&1
code=$?
set -e
hash="$(sha256sum "$tmp" | awk '{print $1}')"

if [[ "$code" -ne 0 ]]; then
  echo "patch failed the frozen command (exit=$code)" >&2
  rm -f "$tmp"
  exit 4
fi

if [[ "$hash" == "$RECEIPT_SHA256" ]]; then
  echo "output hash unchanged; proof is weak" >&2
  rm -f "$tmp"
  exit 5
fi

{
  echo "PROOF_EXIT=$code"
  echo "PROOF_SHA256=$hash"
  echo "PROOF_FILES=$(git diff --name-only "$RECEIPT_COMMIT" | tr '\n' ' ')"
  echo "RECEIPT_STATUS=passed"
} >> repro-receipt.env

echo "proof recorded"
rm -f "$tmp"

Keep the patch packet small on purpose. Receipt, unified diff, and proof lines are enough. Extra design essays belong in the issue, not in the merge path.

git diff "$RECEIPT_COMMIT" -- > patch.diff
{
  echo '--- receipt ---'
  cat repro-receipt.env
  echo '--- diff ---'
  cat patch.diff
} > shadow-packet.txt
wc -l shadow-packet.txt

If PROOF_FILES lists docs, lockfiles, and unrelated packages, stop. The patch escaped its bug. Split the work before any model reads it.

Gate 3 — Shadow review, never authorship

Only now may a model read the packet. The packet is the receipt, the diff, and the proof hashes. The model returns findings. It cannot approve the merge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A free model tier and a free server option can host this read-only pass when the maintainer wants a second reader without buying a review seat. Those two availability facts are the only product claims used here. No model names, quotas, or hardware details are stated.

The review prompt must fail closed. Missing evidence means no finding. New files outside the diff are rejected. approve_merge stays false in every response.

You are a shadow reviewer, not a patch author.
Read only the attached receipt and unified diff.
Return JSON that matches this schema exactly:
{
  "verdict": "comment_only",
  "findings": [
    {
      "file": "path",
      "line": 0,
      "claim": "one sentence",
      "evidence": "quote from diff or receipt",
      "blocking": false
    }
  ],
  "approve_merge": false
}
Rules:
- approve_merge must be false.
- Every finding needs a file path from the diff.
- Do not propose new files.
- Do not rewrite the patch.
- If evidence is missing, return findings=[].

A thin wrapper keeps the model on the packet. It also keeps secrets off the prompt. Do not paste .env files, tokens, or private fixtures.

#!/usr/bin/env bash
# shadow-review.sh — package evidence; do not send secrets.
set -euo pipefail
# Proposed packaging step. Wire your own model host later.

test -f repro-receipt.env
test -f patch.diff
grep -q 'RECEIPT_STATUS=passed' repro-receipt.env

if git diff --name-only | grep -E '(^|/)(\.env|id_rsa|credentials)' ; then
  echo "refusing: secret-like path in the diff" >&2
  exit 6
fi

# Replace the next line with the host you actually run.
# The model must receive shadow-packet.txt and the JSON schema only.
echo "packet ready at shadow-packet.txt"

Human maintainers still decide. Each finding is accepted, rejected, or deferred. Blocking claims need a follow-up command, not a vibe.

Decision table for the model pass

Use the table before spending a review call. The model is optional. The receipt is not.

Situation	Frozen repro	Human patch	Model shadow review
New functional bug with a named test	Required	Required	Optional second reader
Flaky test, no stable command	Do not proceed	Do not proceed	Do not call a model
Docs-only typo	Skip receipt	Tiny patch	Skip model
Public API change	Required	Required plus changelog	Allowed, still non-binding
Security embargo or private report	Local only	Trusted humans only	Do not send off-machine
Diff includes secrets or dumps	Stop	Stop	Forbidden

The table is a policy artifact. It is not a score. Maintainers who need a numeric rank should pick a different method.

What the model is for

Shadow review is good at cheap, local questions. Missed null paths. Tests that assert the wrong string. Comments that contradict the diff. Duplicate helpers already in tree.

It is weak at product intent. It cannot feel a maintainer's release nerve. It cannot hold an embargo. It should not invent missing product context.

Treat every finding as a hypothesis. Re-run a command when the claim is blocking. Discard claims that cannot point at the packet.

Limitations

This method assumes a deterministic failing command. Heisenbugs, race conditions, and GUI-only defects will not freeze cleanly. Multi-service reproductions need a documented compose file, which this receipt format does not encode.

Hash comparison is crude. Log timestamps can change a hash without changing behavior. Redirect logs through a filter if the project prints clocks or PIDs.

The scripts do not talk to GitHub, GitLab, or any host API. They do not prove CI equivalence. They do not replace maintainers, CODEOWNERS, or license review.

Free model access and a free server option can disappear or change. Do not build a release process that requires a specific vendor seat. Keep the receipt and the human patch even if the model host is down.

Who should not use this

Security-sensitive patches should not leave the trusted machine. Embargoed issues need a private channel, not a shared model host. Binary-only bugs with no command line should use a different lab notes format.

First-time contributors who need mentoring should not receive a JSON critique as their first review. Humans teach project norms. Models do not.

Teams that already require formal audit trails may find the receipt too light. Add signed tags and review tickets instead of stretching this packet.

Close the loop on the PR

Paste the receipt block into the PR body. Link the issue. Attach the command that failed, then passed. State that any model notes are comments only.

## Reproduction
- commit: 1a2b3c4
- command: python -m pytest tests/test_parse.py::test_empty_header -q
- before: exit 1, sha256 …
- after: exit 0, sha256 …

## Shadow review
- non-binding comments only
- approve_merge remains false

Reviewers can replay the same two commands. They do not need the model transcript to merge. That is the point of the sequence.

A maintainer who already isolates authorship from review can run the same split on a free model host. The frozen command still leads. The model still does not vote.