DEV Community: Taylor Wang

Turn an OSS Issue Into a Repro Contract Before You Write a Patch

Taylor Wang — Fri, 06 Nov 2026 17:24:21 +0000

Most failed open-source contributions fail at triage, not at coding. A patch is only as good as the reproduction standing behind it. Write the contract first, then the diff.

That order matters more this month. Post-event contribution waves bring volume, and coding agents now produce plausible patches at almost no cost. Maintainers receive diffs that look reasonable and cannot be reproduced. The bottleneck is no longer writing code. It is proving the bug exists and that the change removes it.

This article describes a triage loop: reproduce, contract, patch, verify, then ask a model to review. It assumes one shell, one repository clone, and no special tooling.

The core rule

An issue becomes work only when a command fails for a stated reason. Everything before that is a conversation.

So the loop starts with a repro contract. That is a small JSON file answering five things: which commit, which environment, which command, which expected output, which actual output. If any field is empty, the issue is not ready for a patch.

Step 1: Capture the environment fingerprint

Environments drift. Record the toolchain before touching code. The script below is a starting point, not a universal solution; extend the version checks for your stack.

#!/usr/bin/env bash
# fingerprint.sh — unexecuted example, adapt to your stack
set -euo pipefail
repo="${1:?usage: fingerprint.sh <repo-dir>}"
cd "$repo"

commit=$(git rev-parse HEAD)
dirty=$(git status --porcelain=v1 | wc -l | tr -d ' ')
os=$(uname -srm)

printf '{\n'
printf '  "commit": "%s",\n' "$commit"
printf '  "dirty_files": %s,\n' "$dirty"
printf '  "os": "%s",\n' "$os"
printf '  "node": "%s",\n' "$(node -v 2>/dev/null || echo none)"
printf '  "python": "%s",\n' "$(python3 -V 2>&1 | awk '{print $2}' || echo none)"
printf '  "captured_at": "%s"\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
printf '}\n'

Run it, save the output as triage/fingerprint.json, and commit nothing. Triage artifacts stay local.

Step 2: Write the contract file

Keep the contract boring and machine-readable. Reviewers and scripts both read it.

{
  "issue": 4127,
  "base_commit": "9f3c2ab",
  "command": "npm test -- --run tests/parser.spec.ts",
  "expected": "parser accepts trailing comma",
  "actual": "SyntaxError at line 88",
  "exit_code_base": 1,
  "repro_time_seconds": 42,
  "status": "confirmed"
}

Allowed status values are unconfirmed, confirmed, flaky, and needs-info. Only confirmed unlocks patching. That restriction is the whole point of the file.

Step 3: Make reproduction one command

A contract that needs six manual steps is not reproducible. Wrap it in a runner that always starts from a clean state.

#!/usr/bin/env bash
# verify.sh — unexecuted example
set -euo pipefail
repo="${1:?usage: verify.sh <repo-dir>}"
contract="$repo/triage/contract.json"

cmd=$(jq -r .command "$contract")
base=$(jq -r .base_commit "$contract")
wt="$(mktemp -d)/repro"

git -C "$repo" worktree add --detach "$wt" "$base" >/dev/null
set +e
( cd "$wt" && eval "$cmd" ) 2>&1 | tee "$repo/triage/base.log"
code=${PIPESTATUS[0]}
set -e

echo "base exit code: $code"
git -C "$repo" worktree remove --force "$wt"

If the base exit code matches the contract, the report is trustworthy. If it is zero, the reporter's environment differs and the issue goes back to needs-info.

Step 4: Decide with a budget, not with enthusiasm

Time-box triage. Pick a limit you can defend, such as 45 minutes, and let the result decide the next move.

Signal after triage	Action
Base fails, patch passes, log stable	Open the pull request
Base fails intermittently	Isolate the flake first; do not patch
No failing command after the time box	Ask for a minimal case; park the issue
Fix needs API or design changes	Open a discussion, not a pull request
Model suggests a cause, no command confirms it	Keep it as a hypothesis

The fourth row prevents the most common bad outcome. A correct fix in the wrong shape costs maintainers more than an unfixed bug.

Step 5: Put free model access where it is safe

Models are good at reading long threads and proposing candidate causes. They are bad at knowing whether your repository actually fails. Place them after the contract, never before.

A workable handoff sends four things only: the contract JSON, the last 80 lines of the base log, the list of files touched by the failing code path, and the project test command. Whole-repository dumps waste context and leak unrelated code.

Ask for three candidate causes, one minimal diagnostic command per cause, and likely file locations. Then record the answers in a plain ledger with explicit status.

## Hypotheses for issue 4127
- [verified] Trailing comma parser stops at token `,` before `)` — confirmed by `node -e` probe
- [rejected] Node version mismatch — base fails on both 20 and 22
- [open] Loader caches the old grammar — needs a cache-clearing run

Only [verified] lines may appear in the pull request body. Everything else stays in the ledger. This single rule keeps generated text out of public claims.

Free model access fits this step well, because triage consumes many small reads, not one large one. A clean remote machine also helps, since the contract runner needs a toolchain you may not have locally.

This is where the MonkeyCode project enters the loop. Its free model access covers the hypothesis step above, and its free server option gives the contract runner an isolated place to execute. According to the operator, the offering includes 10 million free tokens and a free server; verify the current limits and terms on the project's own documentation before depending on them, since free tiers change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 6: Verify the patch against the same contract

Reuse verify.sh with the patch branch and compare both logs. Same command, same environment, same expected output. A patch that only passes on your machine is an anecdote.

If the base log is flaky, stop. A green run on a moving baseline proves nothing, and reviewers will rightly reject it.

Limitations and who should skip this

Security-sensitive reports need private handling. Do not paste logs or code into a third-party model.
Projects without a test harness get less value. The contract still helps, but reproduction cost rises sharply.
The time box misses slow-burn bugs. Deep performance issues do not fit a 45-minute gate.
Model output is a hint. It never replaces a command with a recorded exit code.
Free-tier terms, quotas, and server capacity are external facts. They can change without notice.

Skip this loop if you cannot run local commands, if your organization forbids sending code to external services, or if the repository has no stable entry point to execute.

What changes after a week

Triage becomes cheaper and pull requests become smaller. Maintainers argue about design, not about whether the bug exists. Contributors stop guessing which issue is worth an evening.

The contract file is the artifact that survives. It travels into the pull request, the review, and the eventual regression test. If you want to try the loop, start with one issue and one contract.json, using free model access only for the hypothesis step, and compare the result against your usual process.

Triage OSS Issues Before Cloning: A Deterministic Gate That Cuts Model Calls

Taylor Wang — Thu, 05 Nov 2026 16:01:28 +0000

Most OSS triage burns model calls on issues a regex can reject in 40 milliseconds. A cheap gate should run first. The model should see only what survives it.

This article describes a pre-clone gate: six text signals, fixed weights, one routing decision. The deterministic stage runs anywhere, including a small remote box. The model stage handles one narrow job.

1. The costly default is clone-first

The default workflow is clone, install, guess, then ask a model. That order spends the expensive step before the cheap one. Half the queue never deserved a worktree.

A triage gate inverts the order. Score the report text first. Clone only what scores high enough to justify it.

2. Six signals, fixed weights

The gate looks for evidence a maintainer would demand anyway. Each signal is a regex over title plus body. Weights are integers, not learned values, so the output is auditable.

#!/usr/bin/env python3
"""triage.py - deterministic pre-clone gate for OSS issues."""
import json
import re
import sys

SIGNALS = {
    "version":    r"\bv?\d+\.\d+(\.\d+)?([-\w.]*)?\b",
    "command":    r"(?m)^\s*(\$|>)\s+\S+",
    "expected":   r"(?i)expected\s*(behaviou?r|result|output)?\s*[:\-]",
    "actual":     r"(?i)(actual|got|instead)\s*(behaviou?r|result|output)?\s*[:\-]",
    "repro_link": r"https?://(gist\.|pastebin|github\.com/\S+/(issues|pull)/|replit|codesandbox)",
    "traceback":  r"(?i)(traceback \(most recent call last\)|error:|exception|panic:)",
}

WEIGHTS = {
    "version": 2,
    "command": 3,
    "expected": 2,
    "actual": 2,
    "repro_link": 4,
    "traceback": 2,
}

SECURITY_LABELS = {"security", "cve", "vulnerability"}


def score(issue: dict) -> tuple[int, dict[str, bool]]:
    text = f"{issue.get('title', '')}\n{issue.get('body', '')}"
    hits = {name: bool(re.search(pat, text)) for name, pat in SIGNALS.items()}
    total = sum(WEIGHTS[name] for name, ok in hits.items() if ok)
    return total, hits


def decide(total: int, issue: dict) -> str:
    labels = {str(l).lower() for l in issue.get("labels", [])}
    if labels & SECURITY_LABELS:
        return "human-first"
    if total >= 9:
        return "reproduce-locally"
    if total >= 5:
        return "ask-two-questions"
    return "model-hypothesis"


def main(path: str) -> None:
    with open(path, encoding="utf-8") as fh:
        issue = json.load(fh)
    total, hits = score(issue)
    out = {
        "issue": issue.get("number"),
        "score": total,
        "max_score": sum(WEIGHTS.values()),
        "signals": hits,
        "route": decide(total, issue),
        "model_calls_allowed": 0 if total >= 9 else 1,
    }
    json.dump(out, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main(sys.argv[1])

The maximum score is 15. The model_calls_allowed field is the point of the whole file. It is a hard budget written next to the evidence.

3. Run the gate and keep the receipt

Fetch the issue as JSON, then run the gate locally. Store the output beside the patch notes.

$ gh issue view 4821 --repo some-org/some-project \
    --json number,title,body,labels > issues/4821.json
$ python3 triage.py issues/4821.json | tee issues/4821.triage.json
{
  "issue": 4821,
  "score": 11,
  "max_score": 15,
  "signals": {
    "version": true,
    "command": true,
    "expected": true,
    "actual": true,
    "repro_link": false,
    "traceback": true
  },
  "route": "reproduce-locally",
  "model_calls_allowed": 0
}

A score of 11 with no repro link still routes to local reproduction. The traceback plus command is enough to try. The model is not invited.

4. Split the work across three places

The gate creates a natural split. Deterministic steps need a shell. Ambiguous steps need a model. Long runs need somewhere other than a laptop.

Stage	Job	Where it runs	Model involved
Gate	Score signals, choose route	Any shell, 40 ms	No
Reproduce	Clone, install, run one command	Local or remote box	No
Verify	Base run plus patched run	Same box, same commit pair	No
Frame	Draft two clarifying questions	Model	Yes, one call
Review	Check findings cite diff lines	Deterministic checker	No

Two of these stages are pure compute. They do not need a model at all. They do need a machine that stays up during a 20-minute install.

This is where MonkeyCode's free model access and free server option fit the workflow described here. Both are operator-supplied availability claims; check the current limits in the product before you plan around them. The useful part for this gate is the split, not the branding: a remote box runs the clone-and-test stages while the model handles only the framing step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

5. Write the handoff contract

The model stage gets a small payload. Sending the whole issue thread invites summary theatre. Send the score, the missing signals, and the failure text only.

{
  "issue": 4821,
  "score": 11,
  "missing_signals": ["repro_link"],
  "route": "reproduce-locally",
  "task": "ask-two-questions",
  "failure_text": "<trimmed traceback, 1200 bytes max>",
  "forbidden": ["summary", "root cause", "patch"]
}

The forbidden list matters. Without it, free-tier models happily produce a confident root cause from a stack trace. That output is worse than nothing, because reviewers read it.

Cap the payload before sending. Strip tokens, paths, and environment dumps first. A secret gate belongs upstream of this step, not inside it.

6. Where the gate breaks

Regex scoring is brittle. Three known failure modes are worth stating plainly.

Non-English reports miss most signals and fall to model-hypothesis by default.
Issues that describe behaviour in prose without labels score low despite being real bugs.
Teams that paste logs into attachments score zero on traceback and command.

The fix is not a smarter regex. The fix is a manual override flag, recorded in the same JSON, so the routing decision stays visible:

$ python3 triage.py issues/4821.json --override reproduce-locally

Security-labelled issues should always route to a human first. No score justifies automating that queue.

7. Who should skip this

Small projects with fewer than ten open issues do not need a gate. Read the queue directly. Projects with a triage rota already have the human equivalent.

Skip this if the team cannot store triage receipts. The value comes from the audit trail, not the score. A number with no recorded evidence is just a vibe with a decimal point.

The gate is roughly 60 lines. It costs one afternoon and saves the expensive step on every low-evidence report. Start with the receipts, then decide whether the model stage earns its place.

If you want to test the remote-box half without provisioning anything, the free server option is the cheapest way to try the split described above.

Gate Flaky Fixes: Run the Test on Base and Patched Worktrees Before Anyone Calls It Fixed

Taylor Wang — Tue, 27 Oct 2026 13:32:06 +0000

A patch that turns a red test green is not automatically a fix. On a flaky test, it may be luck. The cheapest defence is repetition on two checkouts: the base ref and the patched ref.

This post gives a small worktree-based harness. It repeats one target test N times on both refs and writes a JSON receipt. A model review is only allowed after that receipt exists.

The failure mode this catches

A contributor reports a bug. They add a test, it fails once locally, they patch the code, it passes once. Everyone approves. The test was non-deterministic from the start, and the patch changed nothing relevant.

Three months later the test flakes in CI. The blame lands on the wrong commit.

Repetition separates the two cases. Base fails 4 of 10. Patched fails 0 of 10. That is evidence. Base fails 1 of 10 and patched fails 0 of 10 is not.

Step 1: Fix the target test and the refs

Scope to a single test node, not the whole suite. Ten runs of one test is cheap. Ten runs of a large suite is not.

# pick the narrowest test that still reproduces the report
BASE_REF=origin/main \
PATCH_REF=HEAD \
RUNS=10 \
TEST_CMD='pytest -q tests/test_parser.py::test_nested_alias' \
  ./flakegate.sh

The env vars are the whole interface. Nothing in the repo needs patching.

Step 2: Run the harness

The script creates two detached worktrees from the same clone. Both share one object store, so the setup is fast.

#!/usr/bin/env bash
# flakegate.sh - repeat one test on base and patched worktrees.
set -euo pipefail

REPO=${REPO:-$PWD}
BASE_REF=${BASE_REF:-origin/main}
PATCH_REF=${PATCH_REF:-HEAD}
RUNS=${RUNS:-10}
TEST_CMD=${TEST_CMD:?'set TEST_CMD'}

ROOT=$(mktemp -d)
BASE_WT="$ROOT/base"
PATCH_WT="$ROOT/patched"

git -C "$REPO" fetch --quiet origin || true
git -C "$REPO" worktree add --detach "$BASE_WT"  "$BASE_REF"  >/dev/null
git -C "$REPO" worktree add --detach "$PATCH_WT" "$PATCH_REF" >/dev/null

run_n() {
  local dir=$1 fails=0 i
  for i in $(seq 1 "$RUNS"); do
    if ! (cd "$dir" && eval "$TEST_CMD") >/dev/null 2>&1; then
      fails=$((fails + 1))
    fi
  done
  echo "$fails"
}

BASE_FAILS=$(run_n "$BASE_WT")
PATCH_FAILS=$(run_n "$PATCH_WT")

cat <<JSON
{"base_ref":"$BASE_REF","patch_ref":"$PATCH_REF",
 "base_sha":"$(git -C "$BASE_WT" rev-parse HEAD)",
 "patch_sha":"$(git -C "$PATCH_WT" rev-parse HEAD)",
 "runs":$RUNS,"test":"$TEST_CMD",
 "base_failures":$BASE_FAILS,"patched_failures":$PATCH_FAILS}
JSON

git -C "$REPO" worktree remove --force "$BASE_WT"
git -C "$REPO" worktree remove --force "$PATCH_WT"

Both worktrees point at the same interpreter and site-packages. Build once, not twice. Pin the dependency state before the run, or the two refs get different environments.

Step 3: Read the receipt as a gate

The numbers are a decision, not a report. Use the table below.

base failures / N	patched failures / N	Verdict
0	0	No reproduction. Do not claim a fix.
0	>0	The patch broke something. Stop.
>0	0	Gate passes. Evidence is monotonic.
>0	>0, lower	Partial. Publish both rates, never "fixed".
>0	>= base	No evidence of improvement. Reframe or revert.

The middle row is the only one that unlocks review. Everything else goes back to the contributor with the raw counts.

Step 4: Let a free model review, but only the receipt and the diff

After the gate passes, model review is worth it. Before it passes, model review just adds confident noise.

The review input is small and bounded: the JSON receipt, the diff, and the source file the test covers. No repo, no issue thread, no chat history.

You receive: a flake gate receipt, a unified diff, one source file.
Answer exactly these three questions, one line each.
1. Does the diff modify the function exercised by the named test?
2. Does the diff introduce a new code path the test does not cover?
3. Name one line of the diff you cannot justify from the source file.
If you cannot answer from the provided text, write UNKNOWN.

The third question is the useful one. It forces a concrete citation instead of a summary. UNKNOWN is a valid and frequent answer.

This step is cheap enough to run on free model access, which is how I exercised it. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project advertises free model access and a free server option; treat current allowances and limits as changing, and read the project page for the terms in force.

Step 5: Move the loop onto a throwaway runner

A local machine accumulates state. Sockets, caches, and stale fixtures all shift flake rates. Running the same harness on a clean machine makes the receipt reproducible by someone else.

A free server option is enough for this. The harness needs no GPU. It needs a clean checkout and ten minutes of wall clock.

The receipt should record the runner identity and the commit SHAs. Without that, two contributors get two different answers and neither can be checked.

Limitations worth stating up front

N=10 is a smoke test. A 2% flake survives ten runs most of the time. Raise N when the counts are close.
Environment-dependent flakes need the failing environment. A green run elsewhere proves nothing.
Timing, network, and thread-order bugs are the worst fit. Repetition on a fixed machine may never trigger them.
Shared state between tests pollutes the measurement. Isolate the test before trusting any count.
Long suites make N×cost painful. Always narrow to one node, then widen later if needed.

Who should not use this

The gate is wrong for purely deterministic test failures. Run the test once and move on. It is also wrong for reviewer-time emergencies: a one-line revert does not need a ten-run harness.

Skip it when the bug report contains no trigger at all. Repetition without a trigger measures nothing.

What you keep

One script, one JSON boundary, one decision table. The receipt is small enough to paste into a pull request and stable enough to compare across weeks.

That is the real gain. The conversation shifts from "does this look right" to "what did the counts say". Most disagreements end there.

If you want to try the review step without paying for inference or a runner, MonkeyCode publishes free model access and a free server option for experimentation; start with the harness above and see whether the numbers change your review habits.

Anchored Review Findings: Reject Model Comments That Cite No Diff Line

Taylor Wang — Mon, 26 Oct 2026 12:07:57 +0000

A model review earns its place only when every finding points at a real changed line. Unanchored comments cost maintainers more time than they save.

The workflow below keeps the normal open-source loop: reproduce the bug, patch it, test it. It adds one deterministic check to the review step.

That check is a script. It verifies file path, line number, and quoted text against the diff. Findings that fail are dropped before any human reads them.

Three predictable failure modes

Model reviews fail in boring ways. The first is a path that never appears in the diff. The second is a line number that shifted after a rebase. The third is a quote that paraphrases code nobody wrote.

All three are cheap to detect. Paths, line numbers, and quotes are facts about the patch. No semantic understanding is required to reject them.

A review with ten findings and four invented anchors is worse than no review. Maintainers spend the first ten minutes disproving fiction.

The review contract

Give the model a strict output schema. Ambiguity in the schema becomes noise in the review.

Rules handed to the model:

file must be a path from the diff, without the b/ prefix.
line must be an added line number in the new file.
quote must be copied verbatim and stay at 120 characters or fewer.
claim must name a defect, not a style preference.
Output is JSON only. No prose, no code fences, no severity inflation.

severity takes high, medium, or low. Unknown values are treated as low by the gate.

Step 1: export a zero-context diff

Zero context keeps the anchor map small and unambiguous.

git fetch origin
git diff -U0 origin/main...HEAD -- '*.py' > /tmp/patch.diff
wc -l /tmp/patch.diff

Limit the diff to source files. Lockfiles and generated assets add noise and no reviewable logic.

Split large diffs by directory. The gate runs per diff file, so splitting is safe.

Step 2: run the review

Send prompt and diff as two separate inputs. Separation stops the model from mixing instructions with patch text.

The shell below assumes an OpenAI-compatible chat endpoint. Substitute the base URL and key from the provider's current docs.

jq -n --rawfile p review_prompt.txt --rawfile d /tmp/patch.diff \
  '{model: env.MODEL, messages: [{role: "user", content: ($p + "\n\n" + $d)}]}' \
  > /tmp/request.json

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $MODEL_KEY" \
  -H "Content-Type: application/json" \
  -d @/tmp/request.json > /tmp/review_raw.json

Free model access is why this step is cheap to repeat. The same diff can go through several models, and the gate decides which output survives. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option keeps the review off a sleeping laptop. Both availability claims are operator-supplied. Check current limits and terms before relying on them in a recurring job.

Strip code fences before gating. Models add them despite instructions.

jq -r '.choices[0].message.content' /tmp/review_raw.json \
  | sed 's/^```

json//; s/^

```$//' > /tmp/review.json

Step 3: gate the findings

Save this as anchor_gate.py. It reads only the diff and the model output.

#!/usr/bin/env python3
"""anchor_gate.py: drop review findings that cite no real diff line."""
import json
import re
import sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")


def added_lines(diff_path):
    added = {}
    current = None
    new_no = None
    with open(diff_path, encoding="utf-8", errors="replace") as fh:
        for raw in fh:
            line = raw.rstrip("\n")
            if line.startswith("+++ b/"):
                current = line[6:]
                continue
            if line.startswith("---") or line.startswith("+++"):
                continue
            match = HUNK.match(line)
            if match:
                new_no = int(match.group(1))
                continue
            if current is None or new_no is None:
                continue
            if line.startswith("+"):
                added.setdefault(current, {})[new_no] = line[1:]
                new_no += 1
            elif line.startswith("-"):
                continue
            elif line.startswith(" "):
                new_no += 1
    return added


def verdict(finding, added):
    path = finding.get("file", "")
    line = finding.get("line")
    quote = (finding.get("quote") or "").strip()
    if path not in added:
        return "file_not_in_diff"
    if not isinstance(line, int) or line not in added[path]:
        return "line_not_added"
    if quote and quote not in added[path][line]:
        return "quote_mismatch"
    return "ok"


def main(diff_path, review_path):
    added = added_lines(diff_path)
    with open(review_path, encoding="utf-8") as fh:
        review = json.load(fh)
    kept, dropped = [], []
    for finding in review.get("findings", []):
        tag = verdict(finding, added)
        record = dict(finding, anchor=tag)
        if tag == "ok":
            kept.append(record)
        else:
            dropped.append(record)
    total = len(kept) + len(dropped)
    report = {
        "kept": kept,
        "dropped": dropped,
        "drop_ratio": round(len(dropped) / total, 3) if total else 0.0,
    }
    print(json.dumps(report, indent=2))
    return 0 if not dropped else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

python3 anchor_gate.py /tmp/patch.diff /tmp/review.json; echo "exit=$?"

Exit code 0 means every finding is anchored. Exit code 1 means at least one was dropped.

Step 4: escalate by verdict

Anchor status drives the next action. Nothing else should.

Anchor result	Severity	Action
`ok`	`high`	Read the diff line yourself, then patch or reply
`ok`	`medium`	Batch with the next commit, do not interrupt the queue
`ok`	`low`	Keep in the report, never forward to the maintainer
`line_not_added`	any	Discard. Log the count per model
`quote_mismatch`	any	Re-ask once with the exact diff excerpt attached
`file_not_in_diff`	any	Discard and drop that model from the run

The drop_ratio value is a property of the model, not of the patch. Track it across runs per repository.

A rising ratio on one model usually means the diff format changed. Check the +++ b/ prefix before blaming the model.

Step 5: a test plan you can run once

Build four findings against a known two-line diff. Expected results are fixed, so the gate itself is testable.

{"findings": [
  {"file": "src/parse.py", "line": 42, "quote": "return None", "severity": "high", "claim": "Swallows the parse error"},
  {"file": "src/parse.py", "line": 99, "quote": "return None", "severity": "high", "claim": "Line does not exist"},
  {"file": "src/missing.py", "line": 10, "quote": "x = 1", "severity": "medium", "claim": "File not in diff"},
  {"file": "src/parse.py", "line": 42, "quote": "return retry()", "severity": "low", "claim": "Quote never written"}
]}

Expected output: one kept finding, three dropped, drop_ratio of 0.75, exit code 1.

The check is reproducible. Create a branch with two added lines, edit the line numbers, and rerun.

Limitations

The gate checks position, not truth. A neatly anchored wrong claim still passes.
Zero-context diffs cannot anchor unchanged lines. Use -U3 plus extra bookkeeping if context findings matter.
A patch that adds a line starting with +++ b/ confuses the parser. Skip such patches.
Renames, binary files, and mode changes need separate handling.
Vendored and generated files inflate the diff and produce anchor noise. Exclude them early.
Quote matching is exact substring. Reformatted quotes fail and should be re-requested, not trusted.

Who should not use this

The gate adds a step, so it needs a reason. Skip it when the diff is under ten lines and you will read it anyway.

Skip it for repositories whose maintainers require human-only review. A gate plus model output does not satisfy that policy.

Never wire the exit code to auto-merge. Anchored findings are candidates for attention, not approvals.

If the team cannot store model output for later audit, the drop log has no value. Fix retention first.

What to measure before scaling

Run the gate on five real patches. Record kept findings, dropped findings, and the reason for each drop.

A useful run keeps at least one finding per patch and drops under half. A run that drops most findings is a prompt problem, not a patch problem.

When the numbers look stable, extend the loop. A small anchored review run is the cheapest way to find out whether free-model review fits your contribution queue.

A Claim Ledger for AI-Assisted OSS Reviews

Taylor Wang — Tue, 20 Oct 2026 10:12:23 +0000

AI review is cheap. Verified review is not.

The AI conversation has moved from generation to verification. Many OSS patches now pass through a model before a maintainer reads them. The output often sounds correct. Some claims are correct. Others cite lines that do not exist.

This article describes a claim ledger for AI-assisted OSS review. The ledger checks each model claim against a diff and a test log. It rejects claims without local evidence. The model stays in a reviewer role.

1. Reproduce the bug before any model sees it

Start with a failing test. A reproducible failure gives the review a ground truth.

git switch -c fix/parser-timeout
python -m pytest tests/test_parser.py::test_timeout -q

Record the exit code and the full log. The failure must be stable across two runs.

set -o pipefail
python -m pytest tests/test_parser.py::test_timeout -q 2>&1 | tee /tmp/repro.log

A model cannot reproduce a bug from a vague description. It can review a patch against a known failure. Keep the reproduction step human-owned.

2. Patch small and capture the evidence

Make the smallest change that fixes the failure. Avoid unrelated cleanup in the same patch. Review quality drops when the diff mixes concerns.

git diff --binary > /tmp/patch.diff
python -m pytest tests/test_parser.py -q 2>&1 | tee /tmp/test.log

Run the narrow test first. Then run the package suite. Save both logs. The model will see a capsule, not the whole repository.

The capsule has three files: the patch, the test log, and a short note about the expected behavior. This boundary matters. It limits context and reduces accidental disclosure.

3. Ask a free model for claims, not prose

MonkeyCode is an open-source project with free model access. Its free access fits this step as a reviewer, not an author. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes the current offer as 10 million free tokens and a free server option. Verify current terms before relying on either. Free access can change. The workflow must still work when the free tier changes.

Use a narrow prompt. Require JSON. Require file and line evidence.

You are reviewing one OSS patch. Use only the diff and test log. Return a JSON array. Each item needs claim, file, line, evidence, severity. Use an added line number. Do not give general advice. Mark uncertainty as low confidence.

The model should return claims like this.

[{"claim": "The new timeout path can return None.", "file": "src/parser.py", "line": 88, "evidence": "return None", "severity": "bug"}]

Only added lines are valid. The script ignores removed lines. A claim about an unchanged line is not a review of this patch.

4. Validate every claim with a ledger script

The script below reads the patch, the review, and the test log. It accepts a claim only when the file and line are in the diff. It also requires the evidence string to appear in the patch or the log.

import argparse
import json
from pathlib import Path


def changed_lines(patch_text):
    current = None
    new_line = 0
    out = {}
    for raw in patch_text.splitlines():
        if raw.startswith('+++ b/'):
            current = raw[len('+++ b/'):]
            out.setdefault(current, set())
        elif raw.startswith('--- a/'):
            current = None
        elif raw.startswith('@@'):
            part = raw.split('+', 1)[1]
            new_line = int(part.split(',', 1)[0])
        elif current and raw.startswith('+') and not raw.startswith('+++'):
            out[current].add(new_line)
            new_line += 1
        elif current and raw.startswith('-') and not raw.startswith('---'):
            continue
        elif current and not raw.startswith(chr(92)):
            new_line += 1
    return out


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--patch', required=True)
    ap.add_argument('--review', required=True)
    ap.add_argument('--tests', required=True)
    args = ap.parse_args()

    patch = Path(args.patch).read_text()
    tests = Path(args.tests).read_text().lower()
    claims = json.loads(Path(args.review).read_text())
    changed = changed_lines(patch)
    bad = 0

    for c in claims:
        f = c.get('file', '')
        line = c.get('line', -1)
        ev = c.get('evidence', '').lower()
        ok_file = f in changed
        ok_line = ok_file and line in changed[f]
        ok_ev = len(ev) >= 6 and (ev in patch.lower() or ev in tests)
        if ok_file and ok_line and ok_ev:
            print('ACCEPT', f + ':' + str(line), c.get('claim', ''))
        else:
            bad += 1
            print('REJECT', f + ':' + str(line), c.get('claim', ''))

    raise SystemExit(2 if bad else 0)


if __name__ == '__main__':
    main()

Run the ledger after the model returns its review.

python review_ledger.py --patch /tmp/patch.diff --review /tmp/review.json --tests /tmp/test.log
echo $?

Exit code 0 means every claim has local evidence. Exit code 2 means at least one claim failed the check. The ledger does not prove the patch is correct. It proves that the review is checkable.

5. Use a decision table before merging

Ledger output	Human action
All claims ACCEPT	Read each claim, then review design.
One claim REJECT	Discard that claim and ask for a narrower review.
Many claims REJECT	Stop the model step and inspect the capsule.
Evidence matches but risk is high	Add a test before merge.

A clean ledger is not an approval. It is a short list of claims that deserve human attention. The maintainer still owns the merge decision.

6. Know the limits and the wrong fits

The ledger checks local evidence, not semantics. A claim can cite a real line and still be wrong. Line numbers drift after a rebase. Test logs can miss race conditions and integration failures.

Do not send secrets, private code, or customer data to a free model. Do not treat the ledger as a security audit. Do not use this workflow for large refactors without a stronger review process.

Some teams should not use this approach. Skip it when the project lacks reproducible tests. Skip it when the patch changes authentication or payment code. Skip it when the reviewer cannot inspect the model output directly.

7. Place the free server carefully

The free server option can run the ledger and the test runner. Keep repository credentials off that machine. The model should see the capsule, not the server.

A free server is useful for a repeatable check. It is not a trust boundary. Run the same script locally when the result matters. Compare both outputs before merge.

A small ledger turns model review into a checklist. That is a better use of free access than another broad prompt. Try it on a documentation patch first, then on a narrow bug fix.

Turn a Patch Session Into a PR Body: A JSONL Contribution Journal

Taylor Wang — Mon, 19 Oct 2026 08:54:22 +0000

A patch is only as reviewable as its evidence trail. Most pull request bodies are written from memory, hours after the last command. The distance between what ran and what the description claims is where reviewer time disappears.

This article defines a small journal format for open-source patches. Each reproduction, patch inspection, and test run becomes one JSON line. A standard-library Python script renders the PR body from those lines.

The artifact below is complete and runnable. It fits any patch session where a model, free or paid, sits somewhere in the loop.

The bookkeeping failure

Three problems repeat in model-assisted contribution work.

The failing command is never captured.
A model answer gets promoted to a test result.
The PR body describes intent, not observed exits.

None of these are model problems. They are record-keeping problems. One append-only file solves all three.

Two record kinds

A run record stores a command, exit code, duration, and the last output lines. An ask record stores a question, its source, the answer, and a verified flag.

The verified flag carries the weight. It stays false until a human checks the answer against code or output. An unverified answer is a to-do item, not evidence.

Save the next script as journal.py.

#!/usr/bin/env python3
"""journal.py - append contribution events to JSONL and render a PR body."""
import argparse
import json
import pathlib
import shlex
import subprocess
import sys
import time

JOURNAL = pathlib.Path("contribution.jsonl")


def append(record):
    with JOURNAL.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(record, ensure_ascii=False) + "\n")


def run(step, argv, note=""):
    start = time.monotonic()
    proc = subprocess.run(argv, capture_output=True, text=True)
    append({
        "kind": "run",
        "step": step,
        "cmd": " ".join(shlex.quote(a) for a in argv),
        "exit": proc.returncode,
        "seconds": round(time.monotonic() - start, 2),
        "note": note,
        "tail": (proc.stdout + proc.stderr).strip().splitlines()[-6:],
    })
    return proc.returncode


def ask(question, source, answered="", verified=False):
    append({
        "kind": "ask",
        "step": "ask",
        "cmd": "",
        "exit": 0,
        "seconds": 0,
        "note": question,
        "source": source,
        "answered": answered,
        "verified": verified,
        "tail": [],
    })


def report():
    text = JOURNAL.read_text(encoding="utf-8") if JOURNAL.exists() else ""
    rows = [json.loads(line) for line in text.splitlines() if line.strip()]
    runs = [r for r in rows if r["kind"] == "run"]
    asks = [r for r in rows if r["kind"] == "ask"]
    latest = {a["note"]: a for a in asks}
    asks = list(latest.values())

    print("## What was run\n")
    print("| step | command | exit | seconds |")
    print("| --- | --- | --- | --- |")
    for r in runs:
        print(f"| {r['step']} | `{r['cmd']}` | {r['exit']} | {r['seconds']} |")

    failed = [r for r in runs if r["exit"] != 0]
    print("\n## Non-zero exits\n")
    print("\n".join(f"- `{r['step']}`: exit {r['exit']}" for r in failed) or "- none recorded")

    print("\n## Questions asked of a model\n")
    print("| question | source | answered | verified |")
    print("| --- | --- | --- | --- |")
    for a in asks:
        mark = "yes" if a["verified"] else "no"
        print(f"| {a['note']} | {a['source']} | {a['answered'] or '-'} | {mark} |")

    open_items = [a for a in asks if not a["verified"]]
    print("\n## Unverified claims\n")
    print("\n".join(f"- {a['note']}" for a in open_items) or "- none")


def main():
    parser = argparse.ArgumentParser()
    subs = parser.add_subparsers(dest="command", required=True)
    run_p = subs.add_parser("run")
    run_p.add_argument("step", choices=["reproduce", "patch", "test"])
    run_p.add_argument("--note", default="")
    run_p.add_argument("argv", nargs=argparse.REMAINDER)
    ask_p = subs.add_parser("ask")
    ask_p.add_argument("question")
    ask_p.add_argument("--source", default="model")
    ask_p.add_argument("--answered", default="")
    ask_p.add_argument("--verified", action="store_true")
    subs.add_parser("report")
    ns = parser.parse_args()

    if ns.command == "report":
        report()
    elif ns.command == "ask":
        ask(ns.question, ns.source, ns.answered, ns.verified)
    else:
        argv = ns.argv[1:] if ns.argv[:1] == ["--"] else ns.argv
        sys.exit(run(ns.step, argv, ns.note))


if __name__ == "__main__":
    main()

The script has no third-party dependencies. It writes to contribution.jsonl in the current directory.

Step 1: Record the reproduction

Run the failing case through the journal. Do not paste it into notes afterward.

python journal.py run reproduce --note "deadline case" \
  -- python -m pytest tests/test_retry.py::test_deadline -x || true

The trailing || true matters. The script exits with the test code, which stops shells running under set -e. A non-zero exit is recorded either way.

Step 2: Record the patch shape

Cheap evidence still counts as evidence. Capture the diff summary before the test rerun.

python journal.py run patch --note "one file, retry.go" -- git diff --stat
python journal.py run patch --note "full diff" -- git diff -- src/retry.go

Two records beat one. The summary shows blast radius, the full diff shows intent.

Step 3: Record the same test again

Use the identical command from step 1. Same node id, same flags, same environment.

python journal.py run test --note "after patch" \
  -- python -m pytest tests/test_retry.py::test_deadline

The journal now holds a before and an after with exit codes. That pair is the strongest sentence in any PR body.

Step 4: Record model questions

Free model access lowers the cost of asking. It does not lower the cost of being wrong. Record the question, then record the verification separately.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

python journal.py ask "Does the retry loop reset the deadline?" \
  --source "free model review" \
  --answered "no, it reuses the first deadline"

The answer sits in the report as unverified. Read src/retry.go, find the assignment, then re-record the same question with --verified. The renderer keeps the latest entry per question, so the flag flips cleanly.

python journal.py ask "Does the retry loop reset the deadline?" \
  --source "free model review" \
  --answered "no, it reuses the first deadline, see retry.go:88" --verified

Teams already testing free model access, or the free server option, can run these commands unchanged. The journal format stays provider-agnostic, which matters when the tool of the month changes.

Step 5: Render the body

python journal.py report > pr_body.md

The output contains four sections. Reviewers get the command table, the non-zero exits, the questions with sources, and the still-unverified claims.

Any entry in the last section is a blocker. Ship the PR only when that list is empty or explained.

When a model question earns its place

Situation	Ask a model?	Check first
Assertion message names the mismatch	No	Read the assertion
Behavior spans three or more files	Yes, ask for call sites	`grep -rn` the symbol
Suspected race or timing bug	Rarely	Build a flaky reproduction loop
Error string is unparseable	Yes, ask what raises it	Search the string in the repo
"Should this be a config flag?"	Weak yes	Read existing config conventions
Roadmap or maintainer intent	No	Read `CONTRIBUTING.md` and open issues

The left column is a filter. Most patch questions fail it. That is the point.

Limits worth stating

The journal proves what ran. It does not prove the patch is correct. A green exit code can still hide a wrong fix.

The tail field captures raw output. Tokens, internal hostnames, and customer paths can land in the file. Scrub it before it leaves the machine.

Free model answers vary between calls. That variability is why the verified flag exists. The record must not inherit the model's confidence.

The file grows with every session. One journal per branch keeps it readable. Do not commit it to the repo by default.

Shared capacity on any free tier means long test suites may need local runs. Cheap review and cheap execution are separate bets.

Who should skip this

Repositories that require signed audit logs should use CI artifacts instead. A local JSONL file is not an audit trail.

One-line typo fixes do not need a journal. The bookkeeping costs more than the patch.

Teams looking for a model to approve merges should stop here. This workflow keeps the human as the verification step, permanently.

Anyone unable to store the journal outside version control should not start. Output tails leak context.

Pre-publish checklist

contribution.jsonl contains a reproduce run with a non-zero exit.
The test run repeats the same command after the patch.
Every ask entry has a source and a verification state.
pr_body.md shows no unexplained unverified claims.
The journal file is excluded via .gitignore.

The last check is the one people forget. Add contribution.jsonl to .gitignore before the first run, not after the first leak.

A Byte Cap and Secret Gate Before You Send an OSS Patch to a Free Model

Taylor Wang — Sun, 18 Oct 2026 07:32:10 +0000

A free model review is only as good as the context you send it. Most OSS triage bundles are too large, too leaky, or too vague. The fix is a local gate, not a better prompt.

This is not a post about a new model release. It is about the context you send one. AI fatigue is a rational response to unbounded model use. A bounded bundle makes model help auditable and cheap to stop.

1. The failure mode: an unbounded paste

A maintainer sees a stack trace, a full diff, and a vague request. The model sees the same noise. Neither gets a testable claim.

Three failures repeat:

The bundle includes .env values or CI tokens.
The diff is 4,000 lines, so the failing frame disappears.
The test log is missing, so the model guesses.

A byte cap and a secret gate catch all three before any network call. Both run locally. Both fail closed.

Reproduce first. A failing command with an exit code is the anchor. Without it, the bundle is a story.

2. Build the bundle with a hard byte cap

ctxpack.py collects repo state, a unified diff, and the tail of a test log. It redacts known secret shapes. It refuses to write a bundle if a high-confidence secret pattern survives. It then truncates to an exact byte limit.

#!/usr/bin/env python3
'''Build a bounded, redacted context bundle for model review.'''
import argparse, hashlib, io, json, re, subprocess, sys
from pathlib import Path

PATTERNS = [
    ('private_key_block',
     re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----', re.S),
     '[REDACTED_PRIVATE_KEY_BLOCK]'),
    ('assignment',
     re.compile(r'(?i)([A-Z0-9_]*(?:TOKEN|SECRET|PASSWORD|PASSWD|API_KEY)[A-Z0-9_]*[ ]*=[ ]*)(\S+)'),
     r'\1[REDACTED]'),
    ('github_token', re.compile(r'gh[pousr]_[A-Za-z0-9]{20,}'), '[REDACTED_GITHUB_TOKEN]'),
    ('api_key', re.compile(r'sk-[A-Za-z0-9_-]{16,}'), '[REDACTED_API_KEY]'),
    ('aws_key', re.compile(r'AKIA[0-9A-Z]{16}'), '[REDACTED_AWS_KEY]'),
]
LEFTOVER = [
    re.compile(r'gh[pousr]_[A-Za-z0-9]{20,}'),
    re.compile(r'sk-[A-Za-z0-9_-]{16,}'),
    re.compile(r'AKIA[0-9A-Z]{16}'),
    re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----'),
]

def run(cmd):
    return subprocess.run(cmd, text=True, capture_output=True, check=False).stdout

def main():
    p = argparse.ArgumentParser()
    p.add_argument('repo')
    p.add_argument('out')
    p.add_argument('max_bytes', type=int)
    p.add_argument('test_log')
    a = p.parse_args()
    if a.max_bytes < 1024:
        sys.exit('max_bytes must be at least 1024')

    log = Path(a.test_log).read_text(errors='replace')
    buf = io.StringIO()
    print('## Repo state', file=buf)
    print('', file=buf)
    print('```

', file=buf)
    print(run(['git', '-C', a.repo, 'status', '--porcelain=v1']), file=buf)
    print('

```', file=buf)
    print('', file=buf)
    print('## Diff', file=buf)
    print('', file=buf)
    print('```

diff', file=buf)
    print(run(['git', '-C', a.repo, 'diff', '--unified=3', '--no-color']), file=buf)
    print('

```', file=buf)
    print('', file=buf)
    print('## Test log tail', file=buf)
    print('', file=buf)
    print('```

', file=buf)
    print(chr(10).join(log.splitlines()[-400:]), file=buf)
    print('

```', file=buf)

    text = buf.getvalue()
    counts = {}
    for name, pat, repl in PATTERNS:
        text, n = pat.subn(repl, text)
        if n:
            counts[name] = n

    for pat in LEFTOVER:
        if pat.search(text):
            print('secret gate failed: unredacted pattern remains', file=sys.stderr)
            sys.exit(3)

    data = text.encode('utf-8')[:a.max_bytes]
    Path(a.out).write_text(data.decode('utf-8', errors='ignore'))
    manifest = {
        'bytes': len(data),
        'sha256': hashlib.sha256(data).hexdigest(),
        'redactions': counts,
        'commit': run(['git', '-C', a.repo, 'rev-parse', '--short', 'HEAD']).strip(),
    }
    Path(str(a.out) + '.manifest.json').write_text(json.dumps(manifest, indent=2))
    print(json.dumps(manifest), file=sys.stderr)

if __name__ == '__main__':
    main()

Run it from the repo root:

python3 ctxpack.py . /tmp/oss-bundle.md 12000 /tmp/failing-test.log

The manifest is the audit record. Keep it with the patch. Do not send the bundle until the gate returns 0.

3. Test the gate before trusting it

A redaction tool that is never tested is a decoration. Use three fixtures and assert exit codes.

Create a repo with a fake GitHub token in the test log.
Assert that the output contains token=[REDACTED] and not the fake value.
Assert that a truncated private key block exits with code 3.
Assert that a 200,000-line log is cut to the byte cap.

set -eu
work=$(mktemp -d)
git -C "$work" init -q
printf 'x\n' > "$work/a.txt"
git -C "$work" add a.txt
git -C "$work" -c user.email=a@b -c user.name=t commit -qm init

fake="ghp_$(printf 'a%.0s' $(seq 1 36))"
printf 'token=%s\n' "$fake" > "$work/test.log"
python3 ctxpack.py "$work" "$work/out.md" 20000 "$work/test.log"
grep -q 'token=\[REDACTED\]' "$work/out.md"
if grep -q "$fake" "$work/out.md"; then echo 'leak'; exit 1; fi

printf -- '-----BEGIN RSA PRIVATE KEY-----\nMIIE\n' > "$work/bad.log"
set +e
python3 ctxpack.py "$work" "$work/bad.md" 20000 "$work/bad.log"
code=$?
set -e
test "$code" -eq 3

awk 'BEGIN { for (i=0; i<200000; i++) print "frame " i ": assertion failed" }' > "$work/big.log"
python3 ctxpack.py "$work" "$work/big.md" 8000 "$work/big.log"
test "$(wc -c < "$work/big.md")" -le 8000

Run these checks in CI. A gate without tests is a suggestion.

4. Use a free model for a bounded second opinion

The bundle is now small and auditable. That is the right time to ask for review. A free model can list failure modes, propose missing tests, and challenge the diff summary. It should not be asked to declare the patch correct.

MonkeyCode is an open-source project. It states free model access with a 10-million-token allowance and a free server option, as of this writing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free server option matters when the build and test log are large. Run the harness there and keep the same byte cap. Confirm current quotas, limits, and terms before a team depends on them.

Use a narrow prompt. Ask for gaps, not approval.

You are reviewing a patch, not writing one.
The bundle is complete for this task. Do not assume missing files.
Return exactly three sections:
1. Three failure modes that the tests do not cover.
2. One test per failure mode.
3. One question a maintainer will ask before merging.
Do not claim the patch is correct.

The token allowance is not a reason to send more context. The cap is the control. A small bundle with a clear failing test beats a full repository dump.

5. Decision table

Situation	Send bundle to a free model?	Required first
Deterministic failing test exists	Yes, for failure-mode review	Redaction gate returns 0
Patch touches auth, crypto, or payments	No	Human security review
Test is flaky or missing	No	A stable repro
Diff changes more than 400 lines	Yes, for a summary only	Split the patch
Project bans AI-assisted review	No	Read CONTRIBUTING
Log contains customer data	No	Data removal and legal review

6. Limitations and who should not use this

Regex redaction is best effort. It misses base64 payloads, split secrets, and novel token formats. It cannot prove that a bundle is safe.

The byte cap can cut the one stack frame that matters. Keep the test log tail and the diff. Raise the cap only with a reason.

Do not use this workflow for security-sensitive patches. Do not use it when a project forbids model-assisted review. Do not use it as a substitute for running the tests.

The free model access and free server option are availability claims, not guarantees. Quotas, limits, and terms change. Verify them before a team depends on them.

7. Start with the gate, not the model

The sequence is fixed: reproduce, cap, redact, test, then review. The model is the last step, not the first.

A reproducible bundle takes minutes to build. It gives a maintainer something to inspect. It gives the model a bounded job.

If the workflow fits, start with the free model access and free server option. Record the current limits in the manifest, then keep the gate.

A Review Packet for OSS Patches: Reproduce, Patch, Test, Then Ask a Model

Taylor Wang — Sat, 17 Oct 2026 06:31:10 +0000

The review half of an OSS patch stalls more contributions than the code does. A reviewer cannot merge what they cannot rerun. A patch that arrives with a captured failing transcript, a size-bounded diff, and a short claim ledger clears triage faster than a larger patch with a better story.

Model review can shorten the loop, but only inside a contract. Free model access is now common enough to be part of a normal workflow. The useful question is not "which model" but "what gets sent, and what is allowed back in".

This article builds a five-step loop around one artifact: the review packet. Nothing below assumes a specific language, and every command is plain shell and git.

What a reviewer actually needs

Three artifacts cover most of the ground:

A transcript that reproduces the bug and its exit status.
A diff small enough to read in one pass.
A claim ledger mapping each changed line to evidence.

Everything else is commentary. A model can help produce commentary, and can flag gaps. It cannot substitute for the transcript.

Step 1: Capture the failing transcript first

Run the failing case before touching the source. Record the command, the output, and the exit code. The exit code matters as much as the message.

mkdir -p transcripts

record() {
  local name=$1; shift
  local log="transcripts/${name}.log"
  printf '$ %s\n' "$*" >"$log"
  "$@" >>"$log" 2>&1
  local code=$?
  printf 'exit=%d\n' "$code" >>"$log"
  return "$code"
}

# Expected to fail before the patch.
record before npx vitest run test/parse.spec.ts || true

sha256sum transcripts/*.log > transcripts/SHA256SUMS
cat transcripts/before.log

Two rules keep this honest. The transcript must come from a clean checkout of the base commit, not a modified working tree. And the failing assertion must name the surface behavior, not an internal detail.

Step 2: Build a size-bounded packet

Large diffs get skimmed. Skimming is where review quality dies. Cap the packet and let the stat summary carry the rest.

#!/usr/bin/env bash
# packet.sh BASE [LINE_BUDGET]
set -euo pipefail

BASE="${1:-origin/main}"
BUDGET="${2:-400}"

if ! git rev-parse --verify "$BASE" >/dev/null 2>&1; then
  echo "unknown base: $BASE" >&2
  exit 2
fi

echo "# diffstat"
git diff --stat "$BASE"...HEAD

echo
echo "# commits"
git log --oneline --no-decorate "$BASE"..HEAD

echo
echo "# patch (first ${BUDGET} lines)"
git diff "$BASE"...HEAD | head -n "$BUDGET"

echo
echo "# transcript hashes"
sha256sum transcripts/*.log 2>/dev/null || echo "none"

Run ./packet.sh origin/main 400 > review-packet.md. If the patch exceeds the budget, split the change. A patch that cannot be capped is usually two patches.

Order files by review risk, not by directory. Public API and schema changes go first. Generated files and lockfiles go last, summarized by diffstat only.

Step 3: Write the claim ledger

Every behavioral claim in the pull request description needs a pointer. This is the artifact that catches the most errors, because it makes vague claims expensive.

Claim	Evidence	Command	Status
Parser accepts trailing commas	`transcripts/before.log`, `transcripts/after.log`	`npx vitest run test/parse.spec.ts`	verified
No change to public exports	`git diff --stat`, `api-extractor` run	`npm run api:check`	verified
Faster on large inputs	not measured	—	removed

That last row is the important one. A claim that cannot be evidenced gets deleted, not softened. Softened claims still waste reviewer time.

Step 4: Add a model as a second reader, inside a contract

A model pass is useful on one narrow task: find places where the diff and the ledger disagree. It is poor at judging intent, and it has no access to the failing case unless the packet includes it.

This is where MonkeyCode fits the workflow. The operator advertises free model access, including a free token allowance (currently stated as ten million tokens) and a free server option. Treat both numbers as time-sensitive and confirm them on the project page before planning around them. The relevant property for this loop is availability: a second reader that costs nothing per pass can be run on every patch instead of only the risky ones.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The contract below is a prompt template. It is not executed output.

You are reviewing a patch packet. Do not rewrite code.

Inputs: diffstat, commit list, capped patch, claim ledger,
transcript hashes (contents omitted).

Return at most 12 items. Each item must have:
- file and hunk
- type: missing-test | ledger-mismatch | unhandled-input | scope-creep
- one sentence on what evidence would settle it

Do not propose refactors. Do not comment on style.
If the packet is insufficient to judge, say so and stop.

Strip transcripts before sending. Send hashes and the failing assertion line instead. Bug reports can contain user data, and a hosted model is a third party.

Step 5: Triage every suggestion the same way

Model output is a list of hypotheses, not findings. Each one gets one of three verdicts, and only one verdict allows a code change.

Verdict	Condition	Action
Accept	A test fails before the change and passes after	Patch, add transcript
Defer	Plausible, but no reproducible input yet	Open an issue, link the packet
Reject	Contradicts the transcript or the ledger	Note it in the PR thread

A suggestion is never applied because it sounds right. It is applied because a new failing test exists for it. That single rule prevents most of the damage a confident model can do to a small patch.

Two failure modes worth naming

Ledger drift. The patch changes during review, the ledger does not. Fix: regenerate the packet and hashes after every rebase, and diff the ledger against the patch in CI.

Scope creep by suggestion. A model flags an adjacent problem, and the patch grows to fix it. Fix: hard-cap the diff budget, and move every accepted out-of-scope item into a follow-up issue in the same session.

Both failures are structural, not model-specific. They appear with human reviewers too, just slower.

Limitations and who should skip this

The free tier terms can change without notice. Pin the workflow to self-hosted or paid capacity before depending on it in a schedule.
Model review does not read the whole repository. Cross-file invariants and generated code are out of scope.
Repositories with an AI-assistance policy, CLA restrictions, or embargoed security fixes need a human-only path. Do not paste embargoed reports into a hosted model.
The loop costs setup time. For a one-line typo fix, the packet is overhead; ship the patch.
Contributors who cannot run the test suite locally get no value from steps 1 and 5. Fix the environment first.

The packet itself is the durable part. Models change, and free allowances change with them. A capped diff with a hash-verified transcript and a ledger that admits what was not measured stays reviewable either way.

If the loop is useful, run it once end to end on a small patch before adopting it. The first packet usually reveals which claims were never evidenced to begin with.

Test-First OSS Patches: Write the Failing Fixture Before You Touch the Code

Taylor Wang — Fri, 16 Oct 2026 05:33:58 +0000

Test-First OSS Patches: Write the Failing Fixture Before You Touch the Code

A bug report is not reproducible until a test fails on your machine. Before that moment it is just a story. This workflow converts the story into a fixture, a captured failure signature, and a two-commit patch. The first commit holds only the test. The second holds only the fix.

The loop takes about twenty minutes on a small parser bug. It saves maintainer attention, which is the scarcest resource in open source. It also makes review faster, because the reviewer sees the broken behavior encoded as code.

Why "it does not work" is not a bug report

Three patterns consume most open-source issue threads. The reporter describes an outcome without an input. The maintainer reproduces on a different version and sees nothing. The patch lands without a test, so the bug returns in six months.

A failing test removes all three. It pins the input, the version, and the expected output in one file. The test then guards the fix forever, at zero recurring cost.

The fixture contract

A usable fixture satisfies four properties. It is deterministic, so repeated runs give the same result. It is offline, so it needs no network or credentials. It is ordered, so it never depends on filesystem enumeration. It is minimal, so one assertion explains the whole failure.

If a candidate fixture breaks any of these properties, it is a flaky test in disguise. Fix the fixture before touching product code.

Step 1: freeze the reporter's input

Create a branch and a fixture directory. Copy the smallest input that still triggers the bug.

git switch -c repro/4821-trailing-comma
mkdir -p tests/regression/fixtures/issue_4821

Store the raw input exactly as the reporter supplied it. Strip secrets, absolute paths, and hostnames. Add the expected output beside it, taken from documented behavior rather than from current output.

The expected value matters. Copying current output encodes the bug as correct behavior. Derive the expectation from the spec, the docs, or a sibling parser that behaves properly.

Step 2: assert the wrong behavior on purpose

The test module below is illustrative. Adjust imports and names to your package. It was not executed in this article.

# tests/regression/test_issue_4821_trailing_comma.py
from pathlib import Path

from yourpackage.config import parse_record  # adjust import

FIXTURES = Path(__file__).parent / "fixtures" / "issue_4821"


def read(name: str) -> str:
    return (FIXTURES / name).read_text(encoding="utf-8")


def test_trailing_comma_keeps_previous_field():
    record = parse_record(read("input.conf"))
    assert record.as_text() == read("expected.txt")

One test, one assertion, one fixture pair. Name the file after the issue number so the link between code and thread stays obvious.

Run it and expect a failure:

python -m pytest -q tests/regression/test_issue_4821_trailing_comma.py \
  2>&1 | tee /tmp/4821.before.txt

If this test passes, you have not reproduced the bug. Stop. Do not write a patch against an unproven theory.

Step 3: capture the failure signature

Extract the assertion lines into a small file. This becomes the before-image of the fix.

grep -E '^(E |FAILED|AssertionError)' /tmp/4821.before.txt | head -20

The signature should mention the exact field and value that differ. A signature of "assert False" tells you the fixture is too coarse. Split it until the diff names one behavior.

Step 4: patch the smallest surface

Change the least code that makes the assertion true. Resist refactoring nearby functions in the same branch. Reviewers approve small diffs faster, and small diffs revert cleanly.

Most parser bugs need a guard clause or an index fix. If the patch grows past roughly fifteen lines, the fixture may be testing two bugs. Split it into two issues and two branches.

Step 5: verify with two commands, then commit in two parts

Run the targeted test first, then the full suite.

python -m pytest -q tests/regression/test_issue_4821_trailing_comma.py \
  2>&1 | tee /tmp/4821.after.txt
python -m pytest -q 2>&1 | tail -5

Diff the failure signatures to prove the assertion lines disappeared:

diff <(sed -n '/^E /p' /tmp/4821.before.txt) \
     <(sed -n '/^E /p' /tmp/4821.after.txt)

Then commit the test and the fix separately:

git add tests/regression
git commit -m "test: reproduce #4821 with trailing comma fixture"
git add src/
git commit -m "fix: ignore trailing comma in config parser"

Two commits let a reviewer verify the test fails before the fix and passes after it. That is the whole argument, expressed as git history.

Step 6: use model assistance where it cannot lie

Drafting fixtures is repetitive work, which is where model assistance helps most. Useful prompts ask for edge cases the reporter did not mention: empty values, duplicate keys, unicode boundaries, very long lines. A model can also summarize a long issue thread into candidate inputs.

Verification stays human. Every proposed fixture is confirmed by running the pytest command above and reading the output. A model cannot observe your failure signature, so it cannot confirm reproduction.

MonkeyCode provides free model access and a free server option, per the operator. That matters for this loop because the workflow needs only a shell, git, and Python. Free model access lowers the cost of generating candidate edge-case fixtures. The free server option lets the reproduce-patch-verify cycle run on a remote machine, keeping a local checkout clean while the fixture set grows. Free-tier limits and server terms change over time, so check the current product page instead of trusting a number in a blog post. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Decision table: patch, ask, or close

Report state	Action	Reason
No version string or no input	Ask for both	Nothing is reproducible yet
Fixture fails on main branch	Write the patch	Reproduction is proven
Fails only on the reporter's machine	Request environment dump	Likely platform-specific
Test passes after the patch, suite green	Open the pull request	Evidence is complete
Cannot reproduce, suite green	Comment with the fixture, keep the issue open	Guessing produces bad fixes

Post the fixture in the thread even when reproduction fails. A maintainer on another platform can run it in one command.

Limitations and who should skip this

Fixtures do not fit every bug class. Flaky concurrency bugs, hardware-dependent crashes, and performance regressions need repeated runs and benchmark harnesses instead. Private data cannot be committed as a fixture, so a redacted or synthetic input is required.

Security fixes need extra care. A public failing test can describe a vulnerability before a patched release exists. In that case keep the reproduction private and coordinate disclosure with the maintainers first.

Teams with a strict contribution guide should follow it. Some projects require tests in a specific directory or forbid new fixture folders. Read CONTRIBUTING.md before creating structure.

Closing

Write the fixture, watch it fail, fix the smallest surface, watch it pass. That order is the entire method, and it scales from a two-line parser bug to a protocol change. If a reader wants a second opinion on a candidate fixture, the free model access in MonkeyCode is one reasonable place to look for edge cases before running them locally.

Exit Codes First: A Reproduce-Patch-Verify Loop for OSS Bug Triage

Taylor Wang — Thu, 15 Oct 2026 11:34:33 +0000

Exit Codes First: A Reproduce-Patch-Verify Loop for OSS Bug Triage

Most open-source issues do not die in code. They die in triage. A reporter says it crashes on their machine. Nobody reproduces it. The thread goes quiet for six weeks.

Three states fix most of that. They are R (reproduce), P (patch), and V (verify). Each state writes an exit code into a ledger. Nothing advances on opinion.

The loop takes about twenty minutes per issue. It produces artifacts a reviewer can argue with. It also survives maintainer turnover.

1. Why issue queues stall

Unreproducible reports dominate most backlogs. Volume is not the only problem. A recent DEV trend post counted thousands of open-source AI projects landing in a single weekend (https://dev.to/dailycontext/one-weekend-3000-open-source-ai-projects-3g65). Supply grows faster than triage capacity.

Reviewers then guess. A guessed fix becomes a guessed test. The patch merges, and the same bug returns three releases later.

The loop replaces guessing with three checkable facts. The bug fails at commit X. The patch is minimal. The same check passes after the patch.

2. Step 1: Pin the reporter's commit

Never triage on main. Check out the commit named in the report. Branch drift hides half of all reproduction failures.

git clone --filter=blob:none https://github.com/<org>/<repo>.git work
cd work
git checkout <reporter-sha>
git rev-parse --short HEAD   # record this value

Build and install dependencies at that commit. If the build fails, the issue is a build report, not a bug report. Say so in the thread.

3. Step 2: Record states, not opinions

One script drives the loop. R expects a non-zero exit. V expects zero. P is a diff, so it has no exit code.

#!/usr/bin/env bash
# triage.sh <repo-dir> <check-command>
set -uo pipefail

REPO="${1:?usage: triage.sh <repo-dir> <check-command>}"
CHECK="${2:?usage: triage.sh <repo-dir> <check-command>}"
LEDGER="triage-ledger.ndjson"

cd "$REPO" || exit 2
commit=$(git rev-parse --short HEAD)
: > "$LEDGER"

record () {
  printf '{"commit":"%s","state":"%s","want":%s,"exit":%s,"seconds":%s}\n' \
    "$commit" "$1" "$2" "$3" "$4" >> "$LEDGER"
  echo "state=$1 exit=$3 want=$2"
}

run_state () {
  local state="$1" want="$2" t0 t1 code
  t0=$(date +%s)
  bash -c "$CHECK" >"/tmp/triage-$state.log" 2>&1
  code=$?
  t1=$(date +%s)
  record "$state" "$want" "$code" "$((t1 - t0))"
  [ "$code" -eq "$want" ]
}

run_state R 1 || { echo "R did not fail; not reproducible here"; exit 1; }

git apply --check patch.diff && git apply patch.diff

run_state V 0 || { echo "V did not pass; patch is incomplete"; exit 1; }

Run it with a narrow check first, never the full suite.

bash triage.sh ~/work/parser \
  "python -m pytest tests/test_offset.py::test_zero_width -q"

Output is two lines, plus one NDJSON row per state. Paste both into the issue.

4. Step 3: Gate the ledger before review

A ledger nobody reads is a diary. Add a verifier to CI and to the local script chain. Missing states must fail the build.

# verify_ledger.py
import json
import sys

WANT = {"R": 1, "V": 0}
rows = [json.loads(l) for l in open("triage-ledger.ndjson") if l.strip()]
seen = {row["state"]: row for row in rows}

for state, want in WANT.items():
    row = seen.get(state)
    if row is None:
        sys.exit(f"missing state {state}")
    if row["exit"] != want:
        sys.exit(f"{state}: exit {row['exit']}, want {want}")

print("ledger ok:", {s: seen[s]["exit"] for s in WANT})

One useful second check: reverse the patch and rerun R. If the failure does not return, the test never covered the bug.

5. Step 4: Split review work by cost

Not every review question deserves a human. Sort them by cost of a wrong answer.

Review question	Cheapest reliable signal	Draft model review	Maintainer sign-off
Does it reproduce?	exit code R	no	no
Is the diff minimal?	`git diff --stat` line count	yes	only if contested
Does the test cover the fix?	revert-and-rerun R	yes	no
Is the API change acceptable?	none	draft only	yes
Security, licensing, data handling	none	no	yes

The rule is simple. Machine checks own the left column. Humans own the right one.

6. Where free model access fits

Rows two and three are cheap model work. A wrong answer there wastes a comment, not a release.

MonkeyCode offers free model access and a free server option, per the project operator. That helps contributors without a paid API budget run the reviewer step. Point the model at the diff and the ledger, not at the whole repository.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The allowance and server terms can change. Check the project page before planning a workflow around them. Nothing in this article was benchmarked against a paid tier.

7. Limitations and who should skip this

The ledger proves environment, not correctness. A green V on a weak test is still a weak test.

Flaky checks poison the loop. Fix nondeterminism before trusting R or V. Network-dependent tests break the exit-code contract by design.

Model reviewers tend to rubber-stamp a diff that compiles. Keep the exit-code gate in front of them. Treat their output as a draft, never as approval.

Skip this loop if failures need private data or specific hardware. Skip it if CI minutes are the bottleneck and two extra runs per issue are unaffordable. Skip it for one-off scripts nobody depends on.

8. Apply it to one stalled issue

Pick a single issue this week. Pin the commit. Write the narrowest failing check you can. Record R, apply a small patch, record V.

If the reviewer step is the bottleneck, MonkeyCode's free model access and free server option are a reasonable place to start. Keep the exit codes as the source of truth.

Two exit codes beat a forty-comment thread of opinions.

Set an Eighty-Point Gate Before an OSS Model Review

Taylor Wang — Sun, 11 Oct 2026 22:16:10 +0000

An OSS contribution should clear a local score before model review. The score freezes reproduction, patch scope, tests, and rollback. A model may comment later, but it must not set the bar.

This note gives a proposed gate for ordinary bug-fix pull requests. The checker below is a template, not a recorded benchmark. No runtime metric in this draft comes from a live maintainer host.

The failure this gate stops

A loose patch invite produces comments about tone and naming. The maintainer still lacks a failing command and a bounded diff. Review time goes to prose, not to the defect.

A model amplifies that failure when the prompt is vague. It will happily discuss style while the bug stays open. The gate blocks that prompt until local evidence exists.

What the scorecard stores

The packet lives in one directory beside the clone. Five notes carry fixed weights that sum to one hundred. A missing note scores zero for that weight.

repro.md holds the failing command and the observed result. patch.diff holds the candidate change and nothing beyond it. tests.md holds the fail command and the later pass command.

scope.txt holds git diff stat output plus a path allowlist. rollback.md holds the branch name and the drop command. The weights are twenty, forty, twenty, ten, and ten.

Forty points sit on the diff because scope creep is common. Twenty points sit on reproduction because hearsay is common. The remaining forty points cover tests, scope, and rollback.

Step 1. Freeze the reproduction

The contributor creates a clean directory and a shallow clone. The branch name carries the issue id and the word repro. Work does not start on the default branch.

The clone URL below is a placeholder, not a real project. The contributor replaces it with the target repository. A depth of one is enough for a small bug note.

mkdir -p oss-scorecard
cd oss-scorecard
git clone --depth 1 https://example.com/org/project.git src
cd src
git switch -c issue-1842-repro

The contributor runs the smallest command that shows the bug. The note stores the command, the exit code, and key lines. Tokens, emails, and internal host names are redacted first.

set -o pipefail
./scripts/repro.sh > ../repro.md 2>&1
printf 'exit:%s\n' $? >> ../repro.md

A needed fixture is committed on the throwaway branch. A private database dump never enters the note. A maintainer should rerun the command on a normal laptop.

Step 2. Bound the patch

The fix starts only after the failing command is saved. The diff stays inside the path allowlist in scope.txt. Extra refactors move to a separate pull request.

The contributor writes scope.txt from git diff stat. An allowlist of paths is appended under that stat. The full diff is saved as patch.diff in the packet.

git diff --stat > ../scope.txt
printf '\nallow:\n' >> ../scope.txt
git diff --name-only >> ../scope.txt
git diff > ../patch.diff

A lockfile edit without a stated reason fails review. A hand edit of a generated file also fails review. Those two checks stay human after the score passes.

Step 3. Attach the test evidence

tests.md lists the fail command and the pass command. The same script name should appear on both lines. A renamed test with no behavior change does not count.

The note records the expected exit code for each run. One line names the assertion that holds after the patch. Timing claims stay out unless a local clock was recorded.

cat > ../tests.md << 'EOF'
fail: ./scripts/repro.sh
pass: ./scripts/repro.sh
expect_fail_exit: 1
expect_pass_exit: 0
assertion: the parser rejects an empty name field
EOF

Step 4. Write the rollback note

rollback.md tells a reviewer how to abandon the branch. It includes the branch name and any files the patch adds. It omits credentials for any shared staging host.

cat > ../rollback.md << 'EOF'
branch: issue-1842-repro
drop: git switch main && git branch -D issue-1842-repro
added_files: none
data_loss: none expected
EOF

Step 5. Run the local checker

The checker looks at presence, weight, and rough length. It does not execute the project tests by itself. A passing score only permits a later comment request.

The script below is proposed code for a local laptop. It has not been certified against a specific language runtime. The contributor should read it before trusting the exit code.

#!/usr/bin/env python3
'''Proposed local gate for an unexecuted template.'''

import pathlib
import sys

REQUIRED = {
    'repro.md': 20,
    'patch.diff': 40,
    'tests.md': 20,
    'scope.txt': 10,
    'rollback.md': 10,
}

MIN_CHARS = 40
HOLD_BELOW = 80

def score_packet(root: pathlib.Path) -> int:
    total = 0
    for name, weight in REQUIRED.items():
        path = root / name
        if not path.is_file():
            print('missing ' + name)
            continue
        text = path.read_text(encoding='utf-8').strip()
        if len(text) < MIN_CHARS:
            print('thin ' + name)
            continue
        total += weight
    return total

def main() -> int:
    root = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else '.')
    total = score_packet(root)
    print('score ' + str(total))
    if total < HOLD_BELOW:
        print('hold the model prompt')
        return 1
    print('scorecard is ready for a model pass')
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

python3 gate_check.py ..

A score under eighty stops the workflow at once. The contributor repairs the thin note and reruns the checker. The threshold is not lowered merely to ship faster.

A missing ten-point file can still clear eighty. The illustrative output below shows that scoring hole. The contributor should not treat ninety as full evidence.

missing rollback.md
score 90
scorecard is ready for a model pass

Step 6. Send the card to a model

MonkeyCode enters only after the local gate has passed. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The product is a review aid here, not the acceptance authority.

The operator states that free model access is available. The operator also states that a free server option exists. This draft does not treat either offer as permanent.

Current limits belong on the product page, not in this note. No token quota, model name, or machine size is assumed. Those details change, and stale numbers would mislead a team.

The free server fits when the maintainer host must stay clean. The upload should contain the five notes, not the whole clone. A smaller packet is easier to redact and easier to audit.

The model task should stay narrow on purpose. It scores each file against the written weights. It lists gaps, contradictions, and risky file paths.

It does not rewrite the patch as the first action. The prompt below is a template for that narrow task. The contributor pastes the five notes under that prompt.

Review the five scorecard files only.
Score each file from 0 to its weight.
Cite a line for every deducted point.
List contradictions between repro.md and tests.md.
Do not propose a new feature.
Stop if a secret or credential appears.

Step 7. Sort every model comment

A fixed decision table keeps the human owner visible. Each comment maps to exactly one local action. The model does not merge, rebase, or close the issue.

Comment type	Local action	Model role
Missing command	Return to reproduction	Point at the empty field
Path outside allowlist	Split the diff	Name the extra path
Test rename only	Demand a behavior check	Flag the mismatch
Speculative refactor	Defer to another issue	Must not expand scope
Possible secret	Stop and redact	Must not repeat the secret
Logic doubt with a citation	Reproduce on the laptop	May suggest a check command

The contributor applies the table before editing the commit text. A comment is accepted only after a local command confirms it. Comments that cite files outside the packet are discarded.

Limits of the gate

File length is only a weak proxy for truth. A long incorrect repro.md can still earn twenty points. The reviewer reruns the command instead of trusting the score.

The checker ignores comment quality and project license terms. Some projects forbid sending code to an external host. That policy is read before any free server upload.

Free model access can rate-limit, queue, or change without notice. A sprint plan should not depend on an unchecked offer. The same packet keeps a fully local review path.

This gate is a poor fit for urgent security patches. Those diffs may contain exploit detail that should stay private. A trusted maintainer channel replaces a general model host.

Generated files and binary assets need extra local rules. The sample checker does not parse diff headers. The team extends it before use on a release branch.

Who should skip this workflow

Teams skip it when the bug is not reproduced locally. Teams skip it when the license blocks external model review. Teams skip it when they need measured model accuracy.

Auth changes and credential grants also sit outside this gate. A scorecard cannot prove that an auth change is safe. That proof belongs in a dedicated security review.

New contributors may still use the file list as a writing aid. They ask a human maintainer which files may be shared. The model pass is optional, not a requirement to contribute.

Keep the merge with the maintainer

The useful order is reproduce, patch, test, then model review. The eighty-point checker enforces that order on disk. The human maintainer still owns the merge decision.

A redacted scorecard is the only packet that should leave the laptop. A contributor checks MonkeyCode's current free model and server offer.

Hold OSS Model Notes Behind a Local Review Contract

Taylor Wang — Sat, 10 Oct 2026 18:53:40 +0000

An OSS model note stays non-binding until a local contract passes. The contract locks one command, one diff scope, and two exit codes. A free model may comment only after that gate.

Why unbound notes fail

A long model reply can sound like a finished review. It still lacks a rerun of the failing command. Maintainers then merge model confidence instead of local evidence.

This workflow splits the work into three lanes. Local reproduction owns the claim about the bug. A disposable rerun and a model read stay secondary.

Contract fields that matter

Keep the contract in review-contract.json next to the patch. Record only facts that a stranger can check. Leave coding style opinions out of that file.

{
  "repo": "example/widget",
  "base": "origin/main",
  "command": "python -m unittest tests.test_widget",
  "expect_before": 1,
  "expect_after": 0,
  "allowed_paths": ["widget/parse.py", "tests/test_widget.py"],
  "model_role": "non_binding_reader"
}

The expect_before field stores the exit before the patch. The expect_after field stores the exit after the patch. Both numbers must come from commands, not from chat.

The widget paths and example.invalid host are placeholders, not a real project. Swap in the public repository before a real review run.

Three lanes, one owner

Lane	Where it runs	What it proves	Can it approve the patch
Reproduce	Local worktree	Command exit and log	No, it only supplies evidence
Rerun	Disposable free server	Same command, clean tree	No, it only confirms the exit
Read	Free model session	Risks outside the command	No, comments stay non-binding

Use the free server as a clean rerun host. Use free model access as a narrow second reader.

The operator presents MonkeyCode as an open-source project. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those claims cover the rerun lane and the read lane only.

This draft does not name models, quotas, hardware, or duration. Cite a current primary source before publishing any of those figures.

Step 1. Freeze one failing command

Choose a command that fails on the base commit today. Write it into the contract before any edit. Ban prompts, watches, and commands that never exit.

git fetch origin
git switch --detach origin/main
python -m unittest tests.test_widget
echo "before=$?"

Copy the printed exit code into the expect_before field. A zero means this command does not show the bug. Replace that command before writing any code patch.

Run the snippet without errexit, or the shell stops early. A stopped shell never prints the before marker. The missing marker is a failed reproduction, not a pass.

Step 2. Limit the patch to named paths

Open a detached worktree for the patch experiment. Check the diff, then apply it only if the path list matches. A wider diff makes the written contract false.

git worktree add --detach ../review-wt origin/main
git -C ../review-wt apply --check patch.diff
git -C ../review-wt apply patch.diff
git -C ../review-wt diff --name-only > changed.txt

Compare changed.txt with allowed_paths before the next lane. Extra paths mean the contract is already stale. Update the list or remove the extra file.

Step 3. Capture the local after log

Run the locked command inside the review worktree. Save the combined log and the exit marker. Never trim a failure just to save space.

cd ../review-wt
python -m unittest tests.test_widget > after.log 2>&1
echo "after=$?" | tee -a after.log

Require the after marker when expect_after equals zero. A different marker blocks every later review lane. The local log stays the primary review evidence.

Step 4. Confirm on a disposable server

Push nothing private into the disposable server session. Clone the public base, then apply the same patch file. Run the same command and record the server exit beside the local one.

Reuse the same patch.diff on the server lane. Do not rebuild the patch from memory on that host.

# Proposal only. Replace the placeholder before any real run.
git clone --depth 1 https://example.invalid/widget.git review-srv
cd review-srv
git apply --check ../patch.diff
git apply ../patch.diff
python -m unittest tests.test_widget
echo "server=$?"

Label the server block as an unexecuted proposal. A free server may lack compilers, tokens, or long job time. Record that gap and keep the local log as authority.

Step 5. Build a redacted read packet

Copy four items into a folder named packet. Include the contract, the path list, both exit lines, and a scoped diff. Strip secrets, env files, and absolute home paths first.

mkdir -p packet
cp review-contract.json changed.txt packet/
git -C ../review-wt diff -- widget/parse.py tests/test_widget.py > packet/scope.diff
python3 -c 'from pathlib import Path; text = Path("after.log").read_text(encoding="utf-8", errors="replace"); lines = [ln for ln in text.splitlines() if ln.startswith("after=")]; Path("packet/exits.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")'

That packet is the only model input. A full repository upload adds risk without review value. Delete the packet after the note is written.

Step 6. Request a non-binding read

Ask the free model to list uncovered risks. Require each bullet to cite a path or a contract key. Forbid a verdict of correct, safe, or ready.

Role: non_binding_reader
Files: review-contract.json, changed.txt, exits.txt, scope.diff
Task: list risks the locked command does not cover
Rule: cite a path or a contract key in every bullet
Rule: do not approve the patch

A free model session can serve this narrow read. The model can still miss hidden fixtures and local conventions. A human maintainer still decides whether the patch lands.

Step 7. Check the contract before the note is posted

Run the checker on the contract, the path list, and the local log. Post model bullets only when the checker prints success. Prefix each bullet with the non-binding label.

python3 check_review_contract.py review-contract.json changed.txt after.log

The script below is a local unexecuted proposal. It checks fields, role, path scope, and the exit marker. It never contacts a network and it never opens a pull request.

#!/usr/bin/env python3
"""Proposal: validate an OSS review contract before model notes."""
import json
import sys

def main() -> int:
    if len(sys.argv) != 4:
        print("usage: check_review_contract.py CONTRACT CHANGED LOG")
        return 2
    contract_path, changed_path, log_path = sys.argv[1:4]
    contract = json.loads(open(contract_path, encoding="utf-8").read())
    required = [
        "command",
        "expect_before",
        "expect_after",
        "allowed_paths",
        "model_role",
    ]
    missing = [key for key in required if key not in contract]
    if missing:
        print("missing fields:", ", ".join(missing))
        return 2
    if contract["model_role"] != "non_binding_reader":
        print("model_role must stay non_binding_reader")
        return 2
    changed = [
        line.strip()
        for line in open(changed_path, encoding="utf-8")
        if line.strip()
    ]
    extra = sorted(set(changed) - set(contract["allowed_paths"]))
    if extra:
        print("paths outside contract:", ", ".join(extra))
        return 3
    log = open(log_path, encoding="utf-8").read()
    marker = "after=%s" % contract["expect_after"]
    if marker not in log:
        print("local log lacks", marker)
        return 4
    print("contract ok; model notes remain non-binding")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Step 8. File the maintainer note

Use a fixed note shape so reviewers can scan it. Put local evidence first and model text last. Keep the model section short enough to scan.

Command: python -m unittest tests.test_widget
Local before: 1
Local after: 0
Server after: not run
Checker: contract ok
Model role: non-binding
Model notes:
- non-binding: tests/test_widget.py covers only the named parser path

A missing server exit line is still acceptable. A missing local exit line is not acceptable. Readers should rerun the command before they trust the note.

How failed checks should end

Exit 2 means required fields or the model role are wrong. Exit 3 means the diff left the allowed path list. Exit 4 means the local log lacks the expected marker.

Repair the failed lane before another model call. Do not ask a second model to overwrite a bad log. Edit the contract only when the command itself changed.

Limits that stay in force

One command can pass while a neighbor behavior fails. A clean server can still miss private fixtures or system packages. Free model wording can be fluent and still incomplete.

A log line that merely contains the marker can fool this check. Treat the checker as a gate, not as full test proof. Read the unittest summary before accepting the marker.

Normalize relative paths before running the set comparison. A leading prefix can cause a false extra-path failure.

This article states no quota, latency, or uptime figure. Those values were not verified against a primary source here. A free tier is optional assistance, not a promised production host.

Who should not use it

Skip the flow when the patch holds secrets or customer data. Skip it when project rules ban external model reads. Skip it when the bug depends on devices the server cannot attach.

Skip it for broad refactors that no single command represents. Skip it when the review must be a formal security audit. A non-binding model reader cannot replace that audit.

Practice to keep

Store the contract, the log marker, and the checker result with the patch. Mark every model sentence as non-binding in the review thread. Ask one maintainer to rerun the locked command locally.

Optional reruns and model reads can support that second look. They do not replace the local exit code. Operators who keep evidence separate from commentary can try the optional lanes in MonkeyCode.