Ship an OSS Patch With a Reviewer Question Bank, Not Just a Diff

A maintainer opened the crash-fix pull request after a long triage shift and found a three-line diff with no reproduction notes. The commit message claimed the parser no longer panicked, yet the issue still lacked a failing command for a clean checkout. Review time vanished into guesswork about callers, error paths, and whether the public API still returned the same sentinel values. The tab closed, and the contributor waited three days for a review that never started.

That stall is rarely about missing brilliance in the patch, and it is almost never about git hygiene alone. Maintainers ration attention across many repositories, and an unexplained diff is expensive even when the change is correct. Contributors who used a coding model to draft the hunks often ship fluent code that still fails the social contract of review. The missing artifact is not more commentary in the PR template; it is a question bank the contributor already answered.

Unexplained diffs stall because review is interrogation

Open-source review is a time-boxed interrogation, not a rubber stamp on a green CI badge. A careful reviewer still needs the reproduction, the moved public symbols, and the error paths that must keep failing. Those demands appear whether or not the contributor writes them down in advance of the request. Writing them first turns the pull request into evidence instead of a riddle wrapped in a unified diff.

Most contribution templates ask for a summary, a test plan, and a checklist of boxes. Those boxes age into ritual, and they do not change when the diff touches an exported error type. A question bank is generated from this branch's paths, headers, and exit codes, so the questions track the patch. The template in CONTRIBUTING.md can stay generic while the bank stays specific to the branch.

The artifact: REVIEWER_QUESTIONS.md

The proposed artifact is a single Markdown file named REVIEWER_QUESTIONS.md that lives beside the patch branch. Each entry is a maintainer question, a required evidence pointer, and a short answer that cites a file and line. Unanswered items remain marked TODO so a local gate can refuse to open the pull request. The file is not a blog post and should stay short enough to read in about twelve minutes.

# Reviewer Question Bank

Project: example-parser
Issue: https://github.com/example/parser/issues/1842
Branch: fix/issue-1842
Base: origin/main

## Q1. Reproduction
- Question: What single command fails on a clean checkout of main and passes on this branch?
- Evidence: TODO
- Answer: TODO

## Q2. Public surface
- Question: Which exported symbols changed name, type, or error contract?
- Evidence: TODO
- Answer: TODO

## Q3. Negative path
- Question: Which input must still fail, and what exit code is preserved?
- Evidence: TODO
- Answer: TODO

## Q4. Regression test
- Question: Where is the test that would have failed before the patch?
- Evidence: TODO
- Answer: TODO

## Q5. Commit shape
- Question: Does each commit encode one logical change a revert could isolate?
- Evidence: TODO
- Answer: TODO

## Q6. Docs and changelog
- Question: What user-visible behavior needs a changelog note, or why is none required?
- Evidence: TODO
- Answer: TODO

Six steps from a failing command to a fail-closed gate

The following workflow is a proposed local checklist, and the commands are illustrative rather than a recorded run against a public crate. Contributors should swap the binary names, fixture paths, and test runners for the project under patch. The point of the checklist is the review packet rather than any particular language or CI toolchain.

1. Freeze the failing command

Before any edit, record one command that fails on a clean checkout of the upstream default branch. Store that command in the question bank under reproduction so later answers cannot drift into a private setup. If the crash needs a fixture, check the fixture into the branch rather than describing it in prose. Reviewers should be able to paste the command without guessing at environment variables or hidden data files.

git fetch origin
git switch -c fix/issue-1842 origin/main
# Proposed: capture the failing invocation, not a narrative.
./target/debug/tool parse tests/fixtures/issue-1842.json
echo "expected non-zero on main"

2. Keep every hunk load-bearing

Edit only the symbols required to make that command pass, and resist drive-by cleanups that enlarge the review surface. A question bank cannot rescue a diff that also reformats a neighboring module or renames an unrelated helper. Split formatting and refactors into separate pull requests if the project will accept them at all. The bank should list every touched path so a maintainer can see the blast without running extra git archaeology.

git diff --name-only origin/main...HEAD
git diff --stat origin/main...HEAD
git log --oneline origin/main..HEAD

3. Collect exit codes instead of vibe screenshots

Run the original failing command, the nearest unit tests, and one negative path that should still error after the fix. Capture the exit codes and a one-line summary rather than pasting entire logs into the pull request body. The question bank needs those exit codes as answers, not as decorative terminal screenshots that hide the command. Label any unexecuted matrix as proposed so a maintainer does not treat a wish as a result.

# Proposed local evidence, not a claimed CI run.
set +e
./target/debug/tool parse tests/fixtures/issue-1842.json
echo "repro_exit=$?"
cargo test parse_issue_1842 -- --nocapture
echo "unit_exit=$?"
./target/debug/tool parse tests/fixtures/truncated.json
echo "negative_exit=$?"

4. Pack the diff and draft questions

A coding model can draft questions from the unified diff, but it cannot invent evidence the tests never produced. Feed the model the diff, the test commands, and the project's CONTRIBUTING snippet, then keep only questions a human can answer with citations.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft the bank from a unified diff, and the free server option can run the recorded test commands.

# Proposed helper: pack a small context bundle for a coding model.
mkdir -p .review-packet
git diff origin/main...HEAD > .review-packet/patch.diff
git log --oneline origin/main..HEAD > .review-packet/commits.txt
git diff --name-only origin/main...HEAD > .review-packet/paths.txt
python3 scripts/list_fn_headers.py < .review-packet/patch.diff \
  > .review-packet/headers.md

#!/usr/bin/env python3
"""Proposed helper: list added and removed fn headers from a unified diff."""
import sys


def main() -> None:
    added, removed = [], []
    for raw in sys.stdin:
        if raw.startswith("+++") or raw.startswith("---"):
            continue
        if raw.startswith("+") and "fn " in raw:
            added.append(raw[1:].rstrip())
        elif raw.startswith("-") and "fn " in raw:
            removed.append(raw[1:].rstrip())
    print("## Removed headers")
    for row in removed:
        print(f"- `{row}`")
    print("## Added headers")
    for row in added:
        print(f"- `{row}`")


if __name__ == "__main__":
    main()

Proposed prompt for the drafting pass, to be edited by a human before any answer is trusted:

You are drafting reviewer questions for an OSS patch, not writing the patch.
Read patch.diff, headers.md, and commits.txt.
Propose at most eight questions a maintainer would ask before merging.
Reject style nits. Prefer reproduction, public surface, negative paths,
tests, commit shape, and changelog impact.
Do not invent test results. Leave every Answer and Evidence as TODO.

5. Fill answers with file:line citations

Each answer should be one or two sentences and must point at a path a reviewer can open. Saying tests pass is not an answer, because it hides which command and which assertion moved. If the model drafted a question the tests cannot answer, delete the question instead of fabricating a citation. A short honest bank beats a long bank that launders missing evidence behind confident, fluent prose.

## Q1. Reproduction
- Question: What single command fails on a clean checkout of main and passes on this branch?
- Evidence: tests/fixtures/issue-1842.json, src/parse.rs:214
- Answer: `./target/debug/tool parse tests/fixtures/issue-1842.json` exited 101 on main and 0 on HEAD.

## Q2. Public surface
- Question: Which exported symbols changed name, type, or error contract?
- Evidence: src/lib.rs:88 (still `pub fn parse`)
- Answer: No exported name changed; `parse` still returns `Result<Doc, ParseError>`.

## Q3. Negative path
- Question: Which input must still fail, and what exit code is preserved?
- Evidence: tests/fixtures/truncated.json, src/parse.rs:240
- Answer: Truncated input still exits 1 with `ParseError::UnexpectedEof`.

6. Refuse to open the pull request while TODO remains

A local gate keeps the social contract from depending on memory when the pull request is finally opened. The script below is proposed, and it only searches for the TODO markers the template uses. Contributors can wire it as a pre-push hook or as a make target named review-packet. Opening the pull request then becomes a copy of the bank into the PR body, not a second writing task.

#!/usr/bin/env bash
# Proposed gate: refuse to open a PR while the bank still has TODO answers.
set -euo pipefail
BANK="REVIEWER_QUESTIONS.md"
if [[ ! -f "$BANK" ]]; then
  echo "missing $BANK" >&2
  exit 1
fi
if grep -E '^- (Answer|Evidence): TODO' "$BANK"; then
  echo "question bank is incomplete" >&2
  exit 1
fi
echo "question bank is complete"
# Proposed: only run after the gate passes.
# gh pr create --fill --body-file REVIEWER_QUESTIONS.md

Pass rules for each required question

Question	Required evidence	Pass rule
Reproduction	command plus exit codes	Fails on origin/main, passes on HEAD
Public surface	exported names from the diff	No undocumented rename or signature change
Negative path	one input that must still fail	Non-zero exit and original error type preserved
Regression test	test path and assertion line	Test failed before the patch and passes after
Commit shape	git log --oneline	One logical change per commit, revertible
Docs	changelog, rustdoc, or skip reason	Skip only for internal helpers with no user behavior

Limitations

The bank does not replace a maintainer who knows the crate's history, and it will not catch semantic bugs the tests never expressed. Models often over-generate questions about style and under-generate questions about concurrency, ABI, and platform quirks. A free server run is only as good as the command the contributor recorded in the first step. Teams that already require a formal design review should treat this file as a packet, not as a substitute for that review.

Generated questions also drift when the diff includes vendored code, snapshots, or generated protobuf bindings. In those cases the header lister will emit noise, and a human should delete those rows before the drafting pass. The gate only proves that answers exist as text; it does not prove that the cited lines implement the claim. Lying in the bank is still possible, so maintainers should sample at least one citation against the working tree.

Who should skip this packet

Tiny typo pull requests do not need a twelve-minute packet, and the process would look like ceremony. Security issues that cannot discuss the exploit path in public should stay in the project's private disclosure channel. Contributors who cannot run the failing command at all should not invent passing answers for a server they never invoked. Projects that forbid model-assisted patches in CONTRIBUTING.md should skip the drafting step and fill the bank by hand, or skip the bank.

• Typo-only or comment-only pull requests
• Private security disclosures that cannot describe the failure mode
• Patches the contributor cannot execute on a clean checkout
• Repositories that ban model-assisted contributions

The next time a crash fix looks ready, the contributor can spend twenty minutes answering the questions a tired maintainer will ask anyway. The resulting packet does not make the patch correct, but it makes incorrectness cheaper to see. A filled bank also gives a coding model a narrower job than rewriting a module until continuous integration turns green. That narrower job is the difference between a reviewable patch and a fluent diff that still stalls for days.