Exit Codes First: A Reproduce-Patch-Verify Loop for OSS Bug Triage

Most open-source issues do not die in code. They die in triage. A reporter says it crashes on their machine. Nobody reproduces it. The thread goes quiet for six weeks.

Three states fix most of that. They are R (reproduce), P (patch), and V (verify). Each state writes an exit code into a ledger. Nothing advances on opinion.

The loop takes about twenty minutes per issue. It produces artifacts a reviewer can argue with. It also survives maintainer turnover.

1. Why issue queues stall

Unreproducible reports dominate most backlogs. Volume is not the only problem. A recent DEV trend post counted thousands of open-source AI projects landing in a single weekend (https://dev.to/dailycontext/one-weekend-3000-open-source-ai-projects-3g65). Supply grows faster than triage capacity.

Reviewers then guess. A guessed fix becomes a guessed test. The patch merges, and the same bug returns three releases later.

The loop replaces guessing with three checkable facts. The bug fails at commit X. The patch is minimal. The same check passes after the patch.

2. Step 1: Pin the reporter's commit

Never triage on main. Check out the commit named in the report. Branch drift hides half of all reproduction failures.

git clone --filter=blob:none https://github.com/<org>/<repo>.git work
cd work
git checkout <reporter-sha>
git rev-parse --short HEAD   # record this value

Build and install dependencies at that commit. If the build fails, the issue is a build report, not a bug report. Say so in the thread.

3. Step 2: Record states, not opinions

One script drives the loop. R expects a non-zero exit. V expects zero. P is a diff, so it has no exit code.

#!/usr/bin/env bash
# triage.sh <repo-dir> <check-command>
set -uo pipefail

REPO="${1:?usage: triage.sh <repo-dir> <check-command>}"
CHECK="${2:?usage: triage.sh <repo-dir> <check-command>}"
LEDGER="triage-ledger.ndjson"

cd "$REPO" || exit 2
commit=$(git rev-parse --short HEAD)
: > "$LEDGER"

record () {
  printf '{"commit":"%s","state":"%s","want":%s,"exit":%s,"seconds":%s}\n' \
    "$commit" "$1" "$2" "$3" "$4" >> "$LEDGER"
  echo "state=$1 exit=$3 want=$2"
}

run_state () {
  local state="$1" want="$2" t0 t1 code
  t0=$(date +%s)
  bash -c "$CHECK" >"/tmp/triage-$state.log" 2>&1
  code=$?
  t1=$(date +%s)
  record "$state" "$want" "$code" "$((t1 - t0))"
  [ "$code" -eq "$want" ]
}

run_state R 1 || { echo "R did not fail; not reproducible here"; exit 1; }

git apply --check patch.diff && git apply patch.diff

run_state V 0 || { echo "V did not pass; patch is incomplete"; exit 1; }

Run it with a narrow check first, never the full suite.

bash triage.sh ~/work/parser \
  "python -m pytest tests/test_offset.py::test_zero_width -q"

Output is two lines, plus one NDJSON row per state. Paste both into the issue.

4. Step 3: Gate the ledger before review

A ledger nobody reads is a diary. Add a verifier to CI and to the local script chain. Missing states must fail the build.

# verify_ledger.py
import json
import sys

WANT = {"R": 1, "V": 0}
rows = [json.loads(l) for l in open("triage-ledger.ndjson") if l.strip()]
seen = {row["state"]: row for row in rows}

for state, want in WANT.items():
    row = seen.get(state)
    if row is None:
        sys.exit(f"missing state {state}")
    if row["exit"] != want:
        sys.exit(f"{state}: exit {row['exit']}, want {want}")

print("ledger ok:", {s: seen[s]["exit"] for s in WANT})

One useful second check: reverse the patch and rerun R. If the failure does not return, the test never covered the bug.

5. Step 4: Split review work by cost

Not every review question deserves a human. Sort them by cost of a wrong answer.

Review question	Cheapest reliable signal	Draft model review	Maintainer sign-off
Does it reproduce?	exit code R	no	no
Is the diff minimal?	git diff --stat line count	yes	only if contested
Does the test cover the fix?	revert-and-rerun R	yes	no
Is the API change acceptable?	none	draft only	yes
Security, licensing, data handling	none	no	yes

The rule is simple. Machine checks own the left column. Humans own the right one.

6. Where free model access fits

Rows two and three are cheap model work. A wrong answer there wastes a comment, not a release.

MonkeyCode offers free model access and a free server option, per the project operator. That helps contributors without a paid API budget run the reviewer step. Point the model at the diff and the ledger, not at the whole repository.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance and server terms can change. Check the project page before planning a workflow around them. Nothing in this article was benchmarked against a paid tier.

7. Limitations and who should skip this

The ledger proves environment, not correctness. A green V on a weak test is still a weak test.

Flaky checks poison the loop. Fix nondeterminism before trusting R or V. Network-dependent tests break the exit-code contract by design.

Model reviewers tend to rubber-stamp a diff that compiles. Keep the exit-code gate in front of them. Treat their output as a draft, never as approval.

Skip this loop if failures need private data or specific hardware. Skip it if CI minutes are the bottleneck and two extra runs per issue are unaffordable. Skip it for one-off scripts nobody depends on.

8. Apply it to one stalled issue

Pick a single issue this week. Pin the commit. Write the narrowest failing check you can. Record R, apply a small patch, record V.

If the reviewer step is the bottleneck, MonkeyCode's free model access and free server option are a reasonable place to start. Keep the exit codes as the source of truth.

Two exit codes beat a forty-comment thread of opinions.