Your First Reproduction Is Usually Wrong: A Receipt-Before-Patch Workflow for OSS Fixes

A contributor opens a pull request against a parser library with a failing test, a two-line fix, and a confident description of the bug. The test goes red on their machine, green after the patch, and the maintainer merges it after a quick skim. Six weeks later the same crash reappears in a downstream project, because the original test never actually depended on the code that was changed. Patches drafted with an assistant are common enough now that maintainers triage them like any other contribution, which makes the reproduction step more important than the fix itself.

This article describes a small workflow that biases the whole contribution toward evidence: pin the failure, emit a machine-readable receipt, and prove the assertion is tied to the fix by removing the fix again. Free-tier models can help with parts of that loop, and a free server option can host the reproducer so a maintainer does not need to rebuild the environment on their laptop.

Three ways a reproduction lies

A reproduction is a claim about causality, and most bad reproductions fail in one of three predictable ways.

• Hidden local state. The failure depends on a virtualenv, a cached build directory, a .env file, or a fixture that exists only on the reporter's disk.
• Wrong revision. The report was filed against a tag, but the script runs against a moving branch, so the observed behavior is from different code.
• Assertion drift. The test was written against the patched output, so it cannot fail before the patch and proves nothing about the fix.

The first two are environment problems and are usually solved by a clean clone at an explicit commit. The third is a logic problem, and it survives clean environments, green CI runs, and polite review comments, which is why it needs a mechanical check.

Step 1: pin the failure on a clean checkout

Start from a throwaway clone at the exact commit the report describes, and record the exit code of the command that is supposed to fail. The following script assumes only git, bash, jq, and sha256sum, and it treats every unexpected outcome as an error rather than a warning.

#!/usr/bin/env bash
# repro-receipt.sh — prove a bug fails before the patch and passes after it.
set -euo pipefail

REPO_URL="${REPO_URL:?set REPO_URL}"
BASE_SHA="${BASE_SHA:?set BASE_SHA}"        # commit the bug was reported against
TEST_CMD="${TEST_CMD:?set TEST_CMD}"        # command that must fail on BASE_SHA
PATCH="${PATCH:?set PATCH}"                 # path to the proposed .patch
WORKDIR="$(mktemp -d)"

git clone --quiet "$REPO_URL" "$WORKDIR/repo"
cd "$WORKDIR/repo"
git checkout --quiet "$BASE_SHA"

run () {  # prints: <exit code> <digest of combined output>
  local out code digest
  set +e
  out="$($TEST_CMD 2>&1)"; code=$?
  set -e
  digest="$(printf '%s' "$out" | sha256sum | cut -c1-16)"
  printf '%s %s\n' "$code" "$digest"
}

read -r BEFORE_CODE BEFORE_DIGEST <<<"$(run)"
if [ "$BEFORE_CODE" -eq 0 ]; then
  echo "FAIL: test passes before the patch — this is not a reproduction" >&2
  exit 2
fi

git apply --check "$PATCH"
git apply "$PATCH"
read -r AFTER_CODE AFTER_DIGEST <<<"$(run)"
if [ "$AFTER_CODE" -ne 0 ]; then
  echo "FAIL: patch does not turn the test green" >&2
  exit 3
fi

git apply -R "$PATCH"
read -r REVERT_CODE REVERT_DIGEST <<<"$(run)"
if [ "$REVERT_CODE" -eq 0 ]; then
  echo "FAIL: test stays green with the patch removed — assertion is not tied to the fix" >&2
  exit 4
fi

git apply "$PATCH"
jq -n \
  --arg base "$BASE_SHA" --arg cmd "$TEST_CMD" --arg url "$REPO_URL" \
  --argjson b "$BEFORE_CODE" --arg bd "$BEFORE_DIGEST" \
  --argjson a "$AFTER_CODE"  --arg ad "$AFTER_DIGEST" \
  --argjson r "$REVERT_CODE" --arg rd "$REVERT_DIGEST" \
  '{repo:$url, base_sha:$base, test_cmd:$cmd,
    before:{exit:$b, digest:$bd},
    after:{exit:$a, digest:$ad},
    reverted:{exit:$r, digest:$rd}}' \
  > "$WORKDIR/receipt.json"

cat "$WORKDIR/receipt.json"

A plausible invocation for a Python project looks like this, and it takes about as long as reading the contributor's description carefully.

REPO_URL=https://github.com/example/parser \
BASE_SHA=3f9c1ab \
TEST_CMD='pytest -q tests/test_parser.py::test_empty_input' \
PATCH=./fix-1394.patch \
./repro-receipt.sh

The exit-code contract matters more than the output itself. Codes 2, 3, and 4 name the three failure modes above, so a reviewer can tell from one number whether the problem is the report, the patch, or the test.

Step 2: emit a receipt the reviewer can check in seconds

The JSON block produced at the end is the artifact worth attaching to the pull request. It records the repository URL, the pinned commit, the exact command, and the three exit codes with short digests of the captured output.

{
  "repo": "https://github.com/example/parser",
  "base_sha": "3f9c1ab",
  "test_cmd": "pytest -q tests/test_parser.py::test_empty_input",
  "before":   { "exit": 1, "digest": "9c1f4a0e77b2d315" },
  "after":    { "exit": 0, "digest": "0a77c2b1e5d90f43" },
  "reverted": { "exit": 1, "digest": "9c1f4a0e77b2d315" }
}

Two properties make this useful rather than decorative. First, before.digest and reverted.digest should match, because the same failure should recur when the fix is removed; a mismatch usually means the environment is not deterministic. Second, the receipt is small enough to paste into a PR comment, which keeps the evidence next to the discussion instead of in a linked log that expires.

Step 3: let a free model review the receipt, not the patch

This is where a free model tier earns its place in the loop, and where it should be fenced in. Summarizing a diff, listing branches the test does not exercise, and checking a receipt for internal contradictions are narrow, checkable tasks. Deciding whether a parser change breaks a public contract is not.

Review task	Reasonable for a free model tier	Reason	Human still required
Summarize the diff and the reported symptom	Yes	Text-to-text comparison against the issue	Confirm the summary matches intent
Enumerate untested input branches	Yes	Static reading of conditionals and fixtures	Decide which gaps block the merge
Validate receipt consistency (exit codes, digest equality)	Yes	Mechanical checks with a fixed schema	Nothing, if the script produced it
Judge API or ABI compatibility	No	Requires knowledge of downstream consumers	Always
Security and input-sanitization review	No	Failure modes are adversarial, not textual	Always
License, CLA, and provenance checks	No	Policy, not code	Always

MonkeyCode, an open-source coding assistant project, is one place to run those first three passes: the operator describes free model access and a free server option, and the harness above runs unchanged in either setting. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Availability, model choices, and any usage limits can change, so read the project's own pages before wiring the workflow into a team process instead of relying on a blog post.

A practical guardrail is to make the model's output an input to the review, never the gate. Ask for a numbered list of missing test cases, then add the ones that matter to the contributor's own test file and re-run the script. The receipt ends up in the PR, the model's prose does not.

Step 4: hand over a reproducer, not a description

Maintainers triage on small screens and short windows, and rebuilding a contributor's environment is the most expensive part of the job. A free server option changes the handoff: run the receipt script there and publish only the receipt, so the reviewer fetches evidence without installing anything.

# on the server instance, with the same variables as before
./repro-receipt.sh > receipt.json
python3 -m http.server 8080 --directory .   # reviewer fetches /receipt.json

Two cautions apply. Do not expose an unpatched checkout on a public interface, because the bug may be exploitable, and do not treat a hosted run as proof for other platforms, since a Linux container says nothing about a Windows-only failure. Publish the receipt, keep the working tree private, and delete the instance when the discussion closes.

Limits and who should skip this

The workflow assumes deterministic failures, and it degrades badly elsewhere. Flaky tests, timing-sensitive races, and bugs that need proprietary services or specific hardware will produce digests that never match, which turns the receipt into noise.

• Nondeterministic bugs: use repetition counts and report a failure rate instead of a single exit code.
• Security patches: keep the reproduction private until a coordinated disclosure date.
• Cross-platform claims: one hosted run proves one platform; say so in the PR.
• Very large builds: a fresh clone per run can cost more than the review saves, so cache the dependency layer and pin it.
• Projects with strict provenance rules: if the contribution policy requires signed artifacts or a specific build toolchain, a free server run is supplementary evidence rather than a substitute.

Teams that already require a failing test in CI will find most of this redundant, and that is fine; the value here is for external contributors who cannot see the maintainer's pipeline. The point is not to add ceremony to healthy projects, but to make an unfamiliar fix checkable by someone who has never seen the codebase.

A checklist to paste into the pull request

1. Clean clone at an explicit commit hash, not a branch name.
2. Command that fails before the patch, with its exit code recorded.
3. Patch applied with git apply --check first, then the same command passing.
4. Patch removed, and the same failure returning with a matching digest.
5. Receipt pasted into the PR body, with a note on the platform it ran on.
6. Test gaps listed, with the blocking ones added to the contributor's test file.

If you want to run the model-assisted passes on a free tier, the harness above is a reasonable first script to try, and it is worth checking the project's current documentation for what the free options actually include today.