Prove the OSS Bug Locally Before a Model Reviews the Diff

OSS patches should meet a frozen reproduction first. A model may critique that evidence later. It must not author the change.

This split keeps review honest. Merge authority stays with humans.

The inverted PR pattern

Many AI-assisted PRs invert the order. A model drafts a patch from issue prose. Tests arrive as decoration. Reviewers then argue about wording, not proof.

Issue text is not a reproduction. Green CI on a new test can still miss the bug. A model suggestion is not a merge vote.

The safer order is boring and strict. Freeze a failing command. Patch by hand in a tight scope. Only then request a read-only model pass.

Three gates, never reordered

The workflow uses three sequential gates. Gate one freezes a failing local command. Gate two applies a human-scoped patch. Gate three runs a shadow review on evidence only.

Skip a gate and later gates lie. The model must not see the diff before gate one succeeds.

Gate 1 — Freeze the reproduction

The contributor starts on a clean checkout. No patch exists yet. One command must fail in a recorded way.

1. Check out the default branch at a known commit.
2. Install only the project's documented dependencies.
3. Run one command that should fail today.
4. Store the command, exit code, and output hash.
5. Stop immediately if that command passes.

The receipt is the source of truth. Chat summaries are not. A passing command means the bug is not frozen.

The script below is a proposed local harness. It is not a published benchmark and was not timed in this article.

#!/usr/bin/env bash
# repro-receipt.sh — record a failing command before any patch.
set -euo pipefail

RECEIPT="${RECEIPT:-repro-receipt.env}"
CMD="${1:?usage: repro-receipt.sh <shell-command>}"

if [[ -n "${PATCH_APPLIED:-}" ]]; then
  echo "refusing: patch flag set before the first receipt" >&2
  exit 2
fi

tmp="$(mktemp)"
set +e
bash -lc "$CMD" >"$tmp" 2>&1
code=$?
set -e

if [[ "$code" -eq 0 ]]; then
  echo "refusing: command passed; no bug is frozen" >&2
  rm -f "$tmp"
  exit 3
fi

hash="$(sha256sum "$tmp" | awk '{print $1}')"
{
  printf 'RECEIPT_CMD=%q\n' "$CMD"
  echo "RECEIPT_EXIT=$code"
  echo "RECEIPT_SHA256=$hash"
  echo "RECEIPT_COMMIT=$(git rev-parse HEAD)"
  echo "RECEIPT_STATUS=failed"
} >"$RECEIPT"

echo "wrote $RECEIPT (exit=$code sha=$hash)"
rm -f "$tmp"

Example first run on a parser bug:

chmod +x repro-receipt.sh
./repro-receipt.sh 'python -m pytest tests/test_parse.py::test_empty_header -q'
cat repro-receipt.env

A useful receipt names one test, not the whole suite. Broad suites hide the original failure. Maintainers cannot replay what they cannot name.

Gate 2 — Scope the human patch

After the receipt exists, a human edits files. The patch must change the failing behavior. Drive-by refactors wait for a later PR.

1. Read RECEIPT_CMD again before touching code.
2. Change the smallest set of files that can fix it.
3. Re-run the same command with no flag edits.
4. Require a zero exit and a new output hash.
5. Record paths with git diff --name-only.

Unchanged output hashes are a red flag. The command may have passed for the wrong reason. Silence is not proof.

#!/usr/bin/env bash
# prove-patch.sh — same command, opposite exit, after the patch.
set -euo pipefail
# Proposed local harness. Treat as unexecuted sample code.

# shellcheck disable=SC1091
source ./repro-receipt.env
tmp="$(mktemp)"
set +e
bash -lc "$RECEIPT_CMD" >"$tmp" 2>&1
code=$?
set -e
hash="$(sha256sum "$tmp" | awk '{print $1}')"

if [[ "$code" -ne 0 ]]; then
  echo "patch failed the frozen command (exit=$code)" >&2
  rm -f "$tmp"
  exit 4
fi

if [[ "$hash" == "$RECEIPT_SHA256" ]]; then
  echo "output hash unchanged; proof is weak" >&2
  rm -f "$tmp"
  exit 5
fi

{
  echo "PROOF_EXIT=$code"
  echo "PROOF_SHA256=$hash"
  echo "PROOF_FILES=$(git diff --name-only "$RECEIPT_COMMIT" | tr '\n' ' ')"
  echo "RECEIPT_STATUS=passed"
} >> repro-receipt.env

echo "proof recorded"
rm -f "$tmp"

Keep the patch packet small on purpose. Receipt, unified diff, and proof lines are enough. Extra design essays belong in the issue, not in the merge path.

git diff "$RECEIPT_COMMIT" -- > patch.diff
{
  echo '--- receipt ---'
  cat repro-receipt.env
  echo '--- diff ---'
  cat patch.diff
} > shadow-packet.txt
wc -l shadow-packet.txt

If PROOF_FILES lists docs, lockfiles, and unrelated packages, stop. The patch escaped its bug. Split the work before any model reads it.

Gate 3 — Shadow review, never authorship

Only now may a model read the packet. The packet is the receipt, the diff, and the proof hashes. The model returns findings. It cannot approve the merge.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A free model tier and a free server option can host this read-only pass when the maintainer wants a second reader without buying a review seat. Those two availability facts are the only product claims used here. No model names, quotas, or hardware details are stated.

The review prompt must fail closed. Missing evidence means no finding. New files outside the diff are rejected. approve_merge stays false in every response.

You are a shadow reviewer, not a patch author.
Read only the attached receipt and unified diff.
Return JSON that matches this schema exactly:
{
  "verdict": "comment_only",
  "findings": [
    {
      "file": "path",
      "line": 0,
      "claim": "one sentence",
      "evidence": "quote from diff or receipt",
      "blocking": false
    }
  ],
  "approve_merge": false
}
Rules:
- approve_merge must be false.
- Every finding needs a file path from the diff.
- Do not propose new files.
- Do not rewrite the patch.
- If evidence is missing, return findings=[].

A thin wrapper keeps the model on the packet. It also keeps secrets off the prompt. Do not paste .env files, tokens, or private fixtures.

#!/usr/bin/env bash
# shadow-review.sh — package evidence; do not send secrets.
set -euo pipefail
# Proposed packaging step. Wire your own model host later.

test -f repro-receipt.env
test -f patch.diff
grep -q 'RECEIPT_STATUS=passed' repro-receipt.env

if git diff --name-only | grep -E '(^|/)(\.env|id_rsa|credentials)' ; then
  echo "refusing: secret-like path in the diff" >&2
  exit 6
fi

# Replace the next line with the host you actually run.
# The model must receive shadow-packet.txt and the JSON schema only.
echo "packet ready at shadow-packet.txt"

Human maintainers still decide. Each finding is accepted, rejected, or deferred. Blocking claims need a follow-up command, not a vibe.

Decision table for the model pass

Use the table before spending a review call. The model is optional. The receipt is not.

Situation	Frozen repro	Human patch	Model shadow review
New functional bug with a named test	Required	Required	Optional second reader
Flaky test, no stable command	Do not proceed	Do not proceed	Do not call a model
Docs-only typo	Skip receipt	Tiny patch	Skip model
Public API change	Required	Required plus changelog	Allowed, still non-binding
Security embargo or private report	Local only	Trusted humans only	Do not send off-machine
Diff includes secrets or dumps	Stop	Stop	Forbidden

The table is a policy artifact. It is not a score. Maintainers who need a numeric rank should pick a different method.

What the model is for

Shadow review is good at cheap, local questions. Missed null paths. Tests that assert the wrong string. Comments that contradict the diff. Duplicate helpers already in tree.

It is weak at product intent. It cannot feel a maintainer's release nerve. It cannot hold an embargo. It should not invent missing product context.

Treat every finding as a hypothesis. Re-run a command when the claim is blocking. Discard claims that cannot point at the packet.

Limitations

This method assumes a deterministic failing command. Heisenbugs, race conditions, and GUI-only defects will not freeze cleanly. Multi-service reproductions need a documented compose file, which this receipt format does not encode.

Hash comparison is crude. Log timestamps can change a hash without changing behavior. Redirect logs through a filter if the project prints clocks or PIDs.

The scripts do not talk to GitHub, GitLab, or any host API. They do not prove CI equivalence. They do not replace maintainers, CODEOWNERS, or license review.

Free model access and a free server option can disappear or change. Do not build a release process that requires a specific vendor seat. Keep the receipt and the human patch even if the model host is down.

Who should not use this

Security-sensitive patches should not leave the trusted machine. Embargoed issues need a private channel, not a shared model host. Binary-only bugs with no command line should use a different lab notes format.

First-time contributors who need mentoring should not receive a JSON critique as their first review. Humans teach project norms. Models do not.

Teams that already require formal audit trails may find the receipt too light. Add signed tags and review tickets instead of stretching this packet.

Close the loop on the PR

Paste the receipt block into the PR body. Link the issue. Attach the command that failed, then passed. State that any model notes are comments only.

## Reproduction
- commit: 1a2b3c4
- command: python -m pytest tests/test_parse.py::test_empty_header -q
- before: exit 1, sha256 …
- after: exit 0, sha256 …

## Shadow review
- non-binding comments only
- approve_merge remains false

Reviewers can replay the same two commands. They do not need the model transcript to merge. That is the point of the sequence.

A maintainer who already isolates authorship from review can run the same split on a free model host. The frozen command still leads. The model still does not vote.