DEV Community: Blake Yang

Score the Issue Before You Claim It: A Reproducibility Ledger for OSS Work

Blake Yang — Thu, 15 Oct 2026 11:11:56 +0000

Liquid syntax error: Unknown tag 'endraw'

A Citation Check for OSS Patches Starts With a Minimized Public Fixture

Blake Yang — Mon, 12 Oct 2026 23:58:48 +0000

A maintainer on a small library closed a pull request after the patch looked tidy and the model review sounded confident. The contributor had pasted a long log into chat, received praise, and opened the request without a public failing fixture. On a clean clone of the default branch, the reported command exited zero, so the patch had nothing to prove. That scene is a composite of common review failures, not a claim about one named project.

The same pattern appears when a model reviews a diff before anyone proves the bug on the upstream default branch. Agreement then becomes a style comment about the patch, rather than evidence that the failure existed at all. A useful workflow freezes a public bundle first, then invites a model to comment only on fields inside that bundle. The bundle, not the model's tone, decides whether the patch is ready for a human maintainer.

Freeze a public bundle the model is allowed to see

A review bundle is a directory committed on a scratch branch, or attached to the issue, before any model sees the patch. It holds the failing command, the expected exit status, a fixture file, and a short note about the default-branch commit. It also holds a license check result and a test command copied from the repository's own documentation. Secrets, tokens, and private hostnames stay out of that directory so the packet can be shared with a model.

Fields that make the packet checkable

The file base_sha.txt records the upstream commit where the failure was observed, and later reruns must detach at that commit.
The file exit_status.txt stores the status of the failing command, and a clean default-branch run must reproduce that status.
The file fixture.sha256 names the public input, so a review cannot swap in a quieter file after the model has spoken.
The file license_gate.txt records whether an SPDX identifier search succeeded, which is a gate and not a full legal review.

Build the bundle with a script that can be rerun

The following shell script is an unexecuted proposal, not a recorded run from this account, and paths must be adapted. It writes the base commit, copies one fixture, captures status streams, probes for an SPDX line, and hashes the packet. A contributor should replace the sample CLI invocation with the command already documented by the upstream project. The hash file is the handle a later review uses to prove the packet did not drift between machines.

#!/usr/bin/env bash
# Proposal: unexecuted example. Adapt paths before use.
set -euo pipefail

ROOT="$(git rev-parse --show-toplevel)"
SHA="$(git rev-parse HEAD)"
OUT="${ROOT}/.review-bundle"
mkdir -p "${OUT}/fixture"

echo "${SHA}" > "${OUT}/base_sha.txt"
cp "${ROOT}/repro/input.txt" "${OUT}/fixture/input.txt"
sha256sum "${OUT}/fixture/input.txt" > "${OUT}/fixture.sha256"

set +e
python3 -m project.cli --input "${OUT}/fixture/input.txt" \
  > "${OUT}/stdout.txt" 2> "${OUT}/stderr.txt"
STATUS=$?
set -e
echo "${STATUS}" > "${OUT}/exit_status.txt"

if git grep -n "SPDX-License-Identifier" -- '*.py' > "${OUT}/spdx.txt"; then
  echo "spdx-present" > "${OUT}/license_gate.txt"
else
  echo "spdx-missing" > "${OUT}/license_gate.txt"
fi

find "${OUT}" -type f ! -name 'packet.sha256' -print0 \
  | sort -z | xargs -0 sha256sum > "${OUT}/packet.sha256"

Running the script on a dirty worktree will mix unrelated edits into the story the packet tells the reviewer. A clean detached checkout of the recorded base commit is the honest place to capture the failing status. If that status is already zero, the contributor stops and does not ask a model to invent a bug. That stop is the point of the bundle, and it saves the maintainer from reviewing a speculative patch.

Climb three rungs before any model comment

Rung one runs the failing command on a clean checkout of the recorded base commit and expects the frozen exit status. Rung two applies the patch on that same commit and expects the command to exit zero. Rung three runs the repository test command, not a new test invented for the patch alone. A model should not be asked to invent those commands after the fact, because that invites a moving target.

Repeat the rungs on a second checkout

git fetch origin
git switch --detach "$(cat .review-bundle/base_sha.txt)"
bash scripts/oss-review-bundle.sh
test "$(cat .review-bundle/exit_status.txt)" = "1"
git apply --check patches/fix.diff
git apply patches/fix.diff
# Proposal: after the patch, rerun the documented command and expect status 0.

The first test line fails the workflow when the recorded status is not reproduced, which is the desired alarm. The apply check catches a patch that does not fit the recorded base commit, before anyone discusses coding style. After the patch applies, the same command should exit zero, and the repository test command should still pass. A model review that arrives before these three results exist is commentary, not evidence a maintainer can trust.

Minimize the fixture without deleting the failure

A long log invites a model to summarize noise and then miss the line that actually fails the command. The contributor should cut the fixture in halves, rerun the failing command, and keep the half that still fails. Each cut is recorded by appending the new fixture hash to a short minimization log stored inside the bundle. When a cut makes the status become zero, that cut is discarded and the previous fixture remains the public input.

Steps that keep the failure alive

Copy the original fixture into the bundle and record its hash, its line count, and the failing exit status.
Split that fixture into two halves and rerun the documented failing command against each half on the recorded commit.
Keep the smallest half that still returns the frozen status, and append that hash to the minimization log.
Stop when neither half fails, restore the previous fixture, and treat that file as the public input for review.

The log format below is a proposal with placeholder hashes, not measured output from a repository. A real run replaces those tokens with sha256sum output and the exit status just observed. The discarded row stays in the log so a reviewer can see which cut destroyed the failure. That trail matters more than a shorter file that no longer fails.

step=0 lines=48 sha256=placeholder-original status=1 kept
step=1 lines=24 sha256=placeholder-half status=1 kept
step=2 lines=12 sha256=placeholder-quarter status=0 discarded

Score review comments by citation

A model review earns trust only when each remark names a file that already sits inside the frozen bundle. A comment that calls the patch correct without naming exit_status.txt or fixture.sha256 scores zero for evidence. A comment that cites a packet file and quotes a line can be checked by a human in one pass. This scoring habit resists fluent agreement that replaces a failed reproduction on the recorded base commit itself.

The table below is a proposal for sorting comments, not a measured study from this account's history. Strong rows still require a human to open the cited file before the patch is allowed to move. Weak rows are corrections to the note, not reasons to merge the patch into the default branch. Empty rows are discarded even when the wording sounds confident, polite, and ready for a maintainer.

Model comment pattern	Packet citation	Human action
Fluent praise with no filename	None	Discard the comment
Cites `exit_status.txt` but quotes the wrong digit	Weak	Open the file and correct the note
Cites `fixture.sha256` and the minimization log	Strong	Compare hashes, then read the diff
Suggests a test command absent from the repo docs	Out of packet	Ignore until a human adds that command

# Proposal: unexecuted. Scores whether review lines cite packet files.
from pathlib import Path

PACKET = {
    "base_sha.txt",
    "exit_status.txt",
    "fixture.sha256",
    "license_gate.txt",
    "packet.sha256",
}
text = Path("model-review.md").read_text(encoding="utf-8")
cited = {name for name in PACKET if name in text}
missing = PACKET - cited
print(f"cited={sorted(cited)}")
print(f"missing={sorted(missing)}")
if not cited:
    raise SystemExit("review cited no packet file")

The checker is also an unexecuted proposal, and it only tests whether filenames appear in the review text. It cannot tell whether the model understood the exit status, so a human still opens every cited file. If the checker exits because no packet file was named, the review is rejected without a second pleading prompt. A second prompt that begs for agreement would only recreate the original failure mode this bundle is meant to stop.

Use free model access only after the hash is frozen

MonkeyCode is relevant here only as an optional place to run the citation pass and to repeat the bundle script. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator-supplied claim used here is that free model access and a free server option currently exist. The operator also describes the project as open source, which readers should confirm in the repository before they depend on it.

This article does not state model names, token quotas, hardware sizes, or how long that access lasts. Readers should confirm current terms on the project page before relying on either option for a real review. A missing or changed offer does not break the ladder, because the packet can be reviewed by a person alone. Hosted help is a convenience after the hash exists, not a requirement for the reproduction or the patch.

A free model pass is useful after the packet hash is frozen, because the model then has a closed set of files to cite. A free server rerun is useful when the contributor's laptop has extra packages that the upstream image does not install. Both steps stay optional, and the ladder still works with a local model or with a second container on the same desk. The packet hash, not the model's tone, remains the acceptance criterion for moving the patch toward a human reviewer.

The prompt should ask for missing packet fields, mismatched hashes, and a license gate that printed spdx-missing. It should not ask for a verdict that the patch is ready to merge, because that verdict is a human decision. Output belongs in model-review.md beside the packet, so the filename checker can read it without extra parsing. The contributor then keeps only sentences that name a packet file before the maintainer sees the note.

The contributor deletes any sentence that does not name a packet file before the maintainer sees the note. That deletion is a human edit, and the pull request should say the model text was filtered by the checker. A maintainer can then review the patch, the packet, and a short list of cited gaps without rereading praise. The model does not appear as an author of the commit, because authorship stays with the person who can answer questions.

Limitations that should stay visible

This workflow does not prove that a patch is correct, and it does not replace the maintainer's review or the project's legal process. A minimized fixture can drop the real bug if the contributor cuts the input until the failure disappears. License header searches miss files that use a different SPDX line, or a license that lives only in a top-level file. A model can still cite the right filename and misread the exit status, so a human must open the cited file.

The free server option, where it is available, does not guarantee the same packages as the project's continuous integration image. A rerun that passes on a generic machine can still fail inside the upstream workflow file on the next push. Contributors should treat a green second machine as a portability hint, not as a substitute for the project's own checks. Quotas, model names, and how long free access lasts sit outside this article and must be read from current terms.

Who should skip the bundle

Skip this bundle when the repository forbids attached fixtures, or when the bug depends on private data that cannot be reduced. Skip it when the change is a documentation typo with no failing command, because a fake exit status would only add noise. Skip it when the contributor cannot read the test output well enough to reject a confident but wrong model comment. In those cases a short issue comment and a human review remain the honest path, and a model pass would not help.

What the maintainer should be able to replay

A maintainer should be able to detach at the recorded commit, rerun the bundle script, and see the same fixture hash. The patch should flip the frozen status, and the citation checker should show at least one packet file in the review note. If either hash differs, the conversation stops until the contributor publishes a new packet rather than explaining the drift in chat. That replay is the whole method, and it remains useful even if no hosted model or free server is used.

Readers who still want a clean second machine for that replay can check whether MonkeyCode's free server option is open. They should require the same packet hash on that machine before they trust the rerun at all. The free model access, if currently offered, fits only the citation pass after the hash file already exists. Neither option is a merge approval, and the project page is the place to confirm what is available that day.

Separate the Evidence Commit From the OSS Fix Before Model Output Runs

Blake Yang — Sat, 10 Oct 2026 17:06:52 +0000

A maintainer declined a parser pull request after the diff rewrote three files and never preserved the original failure. The contributor had pasted a repository link into a coding model, accepted the generated patch, and pushed one squashed commit. Local checks passed on that laptop, yet reviewers could not replay the reported bug from any commit on the branch. The project wanted a failing case stored in history before anyone discussed the shape of the repair.

The scene above is a composite illustration, not a report from this account or a named maintainer. The rejection pattern still points to a missing contribution gate rather than a simple score of model quality. A careful open-source change records the failure in its own commit, then limits what a model may read and where its patch may run. The workflow below uses a red commit, a narrow packet, and a disposable runner, and it still works with no hosted model.

What reviewers can replay

Maintainers usually gain confidence when a failure replays on a clean tree that does not depend on the author's shell profile. The first commit stores the command, the fixture, and a captured non-zero result before production code changes. The second commit may repair the defect, but it should not rewrite, weaken, or delete that recorded evidence. A reviewer can move between those commits and watch the same command change from failing to passing.

This pattern fits a public issue, a license that permits sharing the selected files, and a command that fails for a known reason. It does not require write access on the upstream project, and it does not treat a model suggestion as already correct. Patch text stays untrusted until a clean runner executes it and a human compares the result with the red commit.

Land the evidence commit first

The contributor clones the upstream project, opens a topic branch, and adds only reproduction assets. Those assets may be a focused test, a tiny fixture, or a shell script kept under a clearly named directory. Formatting sweeps, lockfile refreshes, and drive-by refactors remain outside the evidence commit on purpose. If the command cannot fail on a clean upstream checkout, work stops until the fixture is honest.

The shell below is a proposed example, not a transcript from a real repository run:

git clone --depth 1 https://example.invalid/org/cli.git
cd cli
git checkout -b issue-1842-quoted-flag
mkdir -p repro
cat > repro/quoted-flag.sh << 'EOF'
#!/usr/bin/env bash
set -eu
# Proposed stand-in. Point this at the real project binary and fixture.
./bin/cli --config repro/quoted.toml run
EOF
chmod +x repro/quoted-flag.sh
set +e
./repro/quoted-flag.sh > repro/actual.txt 2> repro/stderr.txt
status=$?
set -e
printf 'exit=%s\nstderr_bytes=%s\n' "$status" "$(wc -c < repro/stderr.txt)" > repro/status.txt
git add repro
git commit -m "test: reproduce quoted flag drop from issue 1842"

The status note should record the exit code and the stderr size, not a dump of the local environment. Home-directory paths, access tokens, and internal hostnames do not belong in files that may later leave the laptop. A case that fails only inside one customized shell profile is not ready for a model or a maintainer.

Choose a packet, then choose a runner

A free model can read the reproduction and one suspected function, then propose an edit that leaves the evidence files untouched. That help is optional, and the boundary around execution matters more than the vendor behind the suggestion. Generated patch text should run first on a disposable runner, not beside developer credentials or private clones. A full-tree upload is unnecessary when the evidence commit already names the command and the fixture.

The reproduction script and fixture may be sent when the license allows those files to be redistributed.
A short status note may be sent after paths, tokens, and hostnames have been removed from the captured text.
One suspected function may be sent, while an archive of the whole working tree should stay on the laptop.
Environment files, keys, cookies, private logs, and unpublished vulnerability detail stay out of the packet.
Unseen patch text should not run on a laptop that holds SSH keys, release tokens, or private clones.

Decision	Allowed default	Stop if this is false
Evidence is its own commit	Yes	The failure exists only in chat history
Model sees a narrow packet	Yes	The only prompt is a full-tree paste
First execution is disposable	Yes	The only runner holds personal credentials
Reproduction files stay unchanged	Yes	The patch edits the evidence to force green
Human edits the pull-request text	Yes	A model summary would be posted unchanged

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant only as one way to reach free model access and a free server for this bounded review step. The same checks remain valid in another editor, with another model, or with no hosted service in the loop. No model names, token quotas, hardware sizes, time limits, or permanent terms are stated here, because those details were not verified for this draft.

The operator-supplied claim is limited to free model access for the review note and a free server option for the first patch execution. The contributor still owns redaction, the evidence commit, and the later decision to open a pull request. Readers who skip the product name still have the commit split, the runner sequence, and the rejection rules.

Apply the suggestion away from the laptop

After the evidence commit exists, the contributor may ask for a minimal production change that makes the reproduction return zero. The reply is saved as patch text and is not executed on the laptop that holds SSH keys or browser sessions. A disposable free server, or any other clean virtual machine, receives a fresh clone and that patch file. The runner has no maintainer credentials, no personal tokens, and no extra repositories beyond the public topic branch.

# Proposed sequence for a clean runner. Do not target a production host.
git clone --depth 1 --branch issue-1842-quoted-flag https://example.invalid/you/cli.git
cd cli
git apply --check /tmp/model-fix.patch
git apply /tmp/model-fix.patch
set +e
./repro/quoted-flag.sh
echo "post-patch:$?"
set -e
git diff --exit-code -- repro

Stop conditions

Three observable results are enough to stop the trial before a pull request is opened at all. Each result is a reason to keep the evidence commit and to discard the current suggestion entirely. None of these results should be patched over by sending a larger private context to the model.

The patch fails git apply --check, so the next request stays narrow instead of adding private files for context.
The reproduction command still fails, so the suggestion is discarded and the evidence commit remains authoritative.
A non-empty diff under the reproduction path means the patch mixed repair with evidence and should be rejected.

When the same command passes and the reproduction diff is empty, a second commit may hold only the production change. The branch history then shows the miss first and the attempted repair second, which is easier to review than a squash. A model may draft the pull-request summary from those two diffs, but a human must edit every factual claim. The maintainer should be able to replay both commits without trusting the tool that wrote the second one.

Check the branch shape before upload

Proposed checker

The script below is a proposal for the fix commit, and it has not been executed against a real upstream project here. Path names, branch names, and the reproduction directory must be adapted before anyone relies on the result. A passing shape check still does not replace the reproduction command or the project's own test target.

#!/usr/bin/env bash
# Proposed shape check. Run while HEAD is the fix commit.
set -eu
base="${1:-HEAD~1}"
git diff --exit-code "$base" -- repro
if git diff --quiet "$base" -- . ':!repro'; then
  echo "fix commit has no production change" >&2
  exit 1
fi
if git diff --name-only "$base" | grep -E '(^|/)\.env$|id_rsa|credentials\.json'; then
  echo "sensitive-looking path in the fix diff" >&2
  exit 1
fi
echo "commit shape accepted; still rerun the reproduction command"

Manual replay

The numbered checks below are the human gate between a green runner and a public pull request. These steps remain proposed checks for the contributor, not measured results from an executed upstream trial. A skipped check leaves the branch unready, even when the shape script prints an acceptance line.

The contributor checks out the evidence commit on a clean tree and confirms a non-zero status.
The contributor checks out the fix commit and confirms a zero status from the same command.
The contributor confirms that the reproduction path did not change between the evidence commit and the fix.
The contributor runs the repository's own targeted test and records that exact command in the pull request.
The contributor searches the diff for home paths, tokens, and private hostnames before any upload.
If a model remains in the loop, it may only hint at callers outside the suspected function, and project tests must confirm that hint.

Limits, and who should skip this

This gate suits small, replayable defects in projects that accept external pull requests and can tolerate a failing test beside a fix. It is a poor fit for races that vanish on a clean runner and for failures that require private production data. Security reports that belong in a private disclosure channel should not be turned into a public reproduction commit. A free server is also the wrong place for customer logs, licensed corpora, or any tree the contributor may not upload.

Model assistance stays optional here and is easy to overrate once the reproduction already looks clean. A free model can suggest a narrow edit, yet it cannot certify license fit, maintainer preference, or timing stability. If the failing command cannot be published, sending a richer private log to a hosted model is not an acceptable workaround. The honest alternatives are a purely local fix, a question to the maintainers, or no patch until the fixture is shareable.

Teams that already require a paired regression test can adopt the commit split without subscribing to any hosted service. The disposable runner matters most when the patch text came from a model and has not been read line by line. Contributors who cannot isolate a public fixture should pause rather than widen the packet to force progress.

A single open issue is a better trial than a batch of generated pull requests across unrelated repositories. The contributor can land the evidence commit locally, then use free model access only on the redacted packet and a free server only for the first apply. The resulting pull request should stay small enough that replay, not the drafting tool, carries the review.

Freeze a Failing Oracle Before Any Model Touches an OSS Patch

Blake Yang — Fri, 09 Oct 2026 13:17:09 +0000

A weekend contributor clones a small library, reproduces a parser bug, and asks a coding model to repair it before dinner. The model returns a tidy diff, a confident summary, and a test that already passes on the patched tree. The maintainer later discovers that the same test also passed on the unpatched tag, so the pull request never proved a fix. That pattern wastes review time, and a frozen oracle stops it before the branch leaves the fork.

Why a passing test is not evidence

An open-source review needs a failure that exists before the patch and disappears after the patch, under the same command. A generated test written after the edit can encode the new behavior without ever having failed. Reviewers then argue about intended behavior instead of comparing two captured runs from the same command. The workflow below treats those two runs as the contract that every later comment must respect.

Run the loop in order

Write the manifest with an argument list, both expected statuses, and two markers before any production edit.
Capture the before side on the untouched checkout and stop if the status is already the success code.
Apply the smallest patch that can clear the failure marker, whether a person or a model drafted the diff.
Capture the after side, run the checker, and refuse the pull request text when the checker returns an error.
Ask the model to compare the diff with those files, and leave the merge decision with a human reviewer.

Lay out the evidence before editing

The contributor creates an evidence directory before touching production code, then records the failing command in a small manifest. That manifest names the command, the expected nonzero status, and a short marker that must appear in the failure output. A later checker refuses to bless a patch when either capture is missing or when the command string has changed. The point is narrow: prove this bug, on this command, before any model rewrites the story.

Directory contract

evidence/
  manifest.json
  before/stdout.txt
  before/stderr.txt
  before/status.txt
  after/stdout.txt
  after/stderr.txt
  after/status.txt

Proposed manifest

The sample below is a proposal for one pytest node, and it is not a result from a named upstream repository. Replace the argument list with the project's real test command before capturing either side of the oracle. Keep markers short and specific so a later reviewer can see them in the saved output without guessing. Do not treat the sample statuses as defaults for every language or every test runner on the project.

{
  "command": ["python", "-m", "pytest", "-q", "tests/test_parser.py::test_trailing_comma"],
  "before_status": 1,
  "after_status": 0,
  "failure_marker": "trailing comma",
  "success_marker": "1 passed"
}

Capture both sides with one script

The capture script reads the manifest, runs the listed command, and writes status plus both streams into the requested side directory. It does not interpret the bug, and it does not accept a free-form shell string, because argument lists avoid accidental quoting surprises. The contributor runs it once on the untouched checkout, applies the human or model patch, then runs it again with the after side. If the before status already matches the after status, the oracle is invalid and the patch should not be opened.

#!/usr/bin/env python3
"""Proposed local helper. Not a recorded run against an upstream repository."""
import json
import subprocess
import sys
from pathlib import Path

def capture(side: str) -> int:
    manifest = json.loads(Path("evidence/manifest.json").read_text())
    out = Path("evidence") / side
    out.mkdir(parents=True, exist_ok=True)
    proc = subprocess.run(
        manifest["command"],
        text=True,
        capture_output=True,
        check=False,
    )
    (out / "stdout.txt").write_text(proc.stdout)
    (out / "stderr.txt").write_text(proc.stderr)
    (out / "status.txt").write_text(str(proc.returncode))
    return proc.returncode

if __name__ == "__main__":
    raise SystemExit(capture(sys.argv[1]))

python capture_oracle.py before
python capture_oracle.py after
python check_oracle.py

Check the oracle before review comments exist

The checker loads the manifest and both status files, then confirms the failure marker and the success marker landed on the correct side. It also rejects a success marker that already appears in the before output, which catches tests that never failed. A nonzero checker result means the pull request description must not claim that the captured bug is fixed. The script below is a proposal for a fork-local gate, and it should be adapted to the project's real test entry point.

#!/usr/bin/env python3
"""Proposed checker. Sample markers are not universal pass criteria."""
import json
import sys
from pathlib import Path

def main() -> int:
    root = Path("evidence")
    manifest = json.loads((root / "manifest.json").read_text())
    before_status = int((root / "before" / "status.txt").read_text().strip())
    after_status = int((root / "after" / "status.txt").read_text().strip())
    before_text = (
        (root / "before" / "stdout.txt").read_text()
        + (root / "before" / "stderr.txt").read_text()
    )
    after_text = (
        (root / "after" / "stdout.txt").read_text()
        + (root / "after" / "stderr.txt").read_text()
    )
    errors = []
    if before_status != manifest["before_status"]:
        errors.append("before status does not match the manifest")
    if after_status != manifest["after_status"]:
        errors.append("after status does not match the manifest")
    if manifest["failure_marker"] not in before_text:
        errors.append("failure marker missing from before output")
    if manifest["success_marker"] not in after_text:
        errors.append("success marker missing from after output")
    if manifest["success_marker"] in before_text:
        errors.append("success marker already present before the patch")
    if errors:
        print("\n".join(errors))
        return 1
    print("oracle holds: failure captured, then cleared, under one command")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Let a free model review the contract, not invent it

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access can sit in the review lane after the oracle already exists on disk. The useful task is a bounded reading of the diff against the evidence files, not a fresh verdict on the bug. A careful prompt asks the model to list mismatches, untested branches, and claims that the captured files do not support. The free server option helps when a clean remote run is needed, provided the resulting evidence files still pass the local checker.

Review only evidence/manifest.json, the before and after captures, and the diff.
Do not claim a bug is fixed unless the checker would pass on those files.
List pull request claims that the captured output does not support.
Do not widen the patch, and do not suggest new dependencies.

Split model comments from maintainer decisions

Question	Model may draft	Human must decide
Did the before status match the manifest?	Quote the status file	Reject the request if it does not
Did the marker move from failure to success?	Point at the two captures	Confirm the marker is specific enough
Does the diff touch paths outside the fix and the test?	List the extra paths	Accept the scope or split the series
Are license, API, or release notes correct?	Flag uncertainty only	Own the final answer
Should this merge?	No draft of a merge vote	Yes, after reading the files

The table is the review boundary for this workflow, and it stays useful even if the model name changes next month. Current quotas, model lists, and server limits are product facts that expire, so this article does not quote them. Contributors should read the current MonkeyCode documentation before relying on a free run for a time-boxed contribution. A changed limit is a reason to run the same scripts locally, not a reason to skip the oracle.

Keep the patch inside the oracle

A frozen oracle does not prove that every edited line was necessary, so the contributor still reads the diff before opening the request. Extra refactors, renamed helpers, and drive-by formatting changes should move to a separate commit or leave the branch. The model may list those extra paths, but it should not be asked to clean up the module after the evidence is green. A small patch plus a failing-then-passing command is easier to revert than a broad rewrite with a flattering summary.

Commands a reviewer can rerun

The pull request body should contain the manifest command and the checker result, copied from the terminal rather than paraphrased. A reviewer on another machine can then run the same argument list, compare status files, and stop if the marker is too vague. A documented setup target belongs in the manifest notes, and this article does not invent a universal container name. The evidence files stay in the fork until the maintainer asks for them, because some projects prefer not to commit bulky logs.

When the marker lies

A marker that appears in ordinary passing output will make the checker reject a real fix, and that failure is useful. The contributor should choose a string that shows up only in the broken run, such as a specific exception text or a fixture name. If no such string exists, the test itself is the missing artifact, and editing production code first would hide that gap. Flaky commands are out of scope, because a status that changes between two before captures means the oracle is not ready for model review.

Who should skip this gate

This approach is a poor fit for patches that cannot be reduced to one command with a stable marker. Documentation-only edits, visual changes, and rare race conditions need a different proof, because a single captured status can look green while the bug remains. Projects that already require a recorded replay file should follow that project rule instead of adding a second evidence format. The checker also does not replace secret scanning, license review, or a blast-radius look at shared helpers.

Close the branch only after the files agree

A contributor who wants a second reading can paste the checker output and the diff into a free model session. The only requested output should be unsupported claims, checked against files already stored in the evidence directory. Those scripts remain unexecuted proposals, so each fork should adjust markers to its own test runner before trusting a green line. When the two captures and the checker agree, the branch is ready for a human reviewer rather than for another generated summary.

Convert CONTRIBUTING.md Into an Executable Preflight Gate Before an OSS Patch Leaves Your Fork

Blake Yang — Thu, 08 Oct 2026 09:26:47 +0000

A contributor spends a weekend on a null-dereference fix, opens the pull request on Monday morning, and watches CI fail on a rule that CONTRIBUTING.md stated in paragraph nine. The patch logic is correct, but the branch name lacks the expected prefix and the changelog entry is missing. Three review rounds later, the maintainer merges a change whose packaging cost more than its implementation.

That failure mode is rarely about code quality, and it is almost always about packaging. Contribution guidelines are written as prose for humans and enforced as machines somewhere else, so the gap gets paid for in review latency. The workflow below closes that gap with an executable preflight gate, a reproducible test step, and a narrow review role for free-tier models.

Why prose rules and CI checks drift apart

Most repositories accumulate contribution rules in three separate places, and none of them is authoritative. CONTRIBUTING.md explains intent, CI enforces a subset, and maintainers enforce the remainder from memory during review.

The document drifts after a tooling change, because nobody re-reads paragraph nine.
CI cannot check intent, so reviewers absorb the difference as unpaid coordination work.
Contributors learn the real rules only after a failed pull request, which is the most expensive classroom.

The practical fix is not stricter CI. It is a local gate that runs the rules the repository already claims to have, before a patch becomes other people's problem.

The snippets below are illustrative and were not run against a specific public repository; adapt paths and tool names to your target project before relying on them.

Step 1: Extract the rules into a reviewed YAML file

Convert prose into a small machine-readable file that a human approves once and CI can diff later. Keep every rule traceable to the sentence it came from, so a disagreement can be settled by reading the source document.

A free model tier handles the mechanical extraction well, because the task is reading and structuring, not deciding. The prompt below deliberately forbids inference, which is the failure mode that turns an extraction pass into invented policy.

You are extracting contribution rules from a repository document.
Return YAML only, with this schema:
  version: 1
  branch_pattern: <regex or null>
  required_when:
    - path: <file that must change>
      when_changed: [<glob>, ...]
Return a separate key `unresolved` for any sentence that is advisory rather than checkable.
For every rule, include `source_line`, the line number in the document.
Do not infer rules that are not written. Mark ambiguous rules with `needs_human: true`.

The resulting file stays small on purpose. A gate with forty checks gets disabled within a week, while one with five honest checks survives contact with a real release cycle.

# .contrib-rules.yml — extracted from CONTRIBUTING.md, human-reviewed
version: 1
branch_pattern: '^(fix|feat|docs|chore)/[a-z0-9._-]+$'
required_when:
  - path: CHANGELOG.md
    when_changed: ['src/**/*.py', 'lib/**/*.js']
  - path: docs/
    when_changed: ['**/public_api.py']
unresolved:
  - "Prefer small pull requests."

Step 2: Build the preflight gate

The gate has two jobs: reject packaging mistakes locally, and run the project's own test dialect exactly as the repository defines it. A shell wrapper handles ordering, while a small Python checker evaluates the file-level rules.

#!/usr/bin/env bash
# scripts/preflight.sh — run the repo's stated rules before pushing a patch
set -euo pipefail

BASE="${1:-origin/main}"
RULES=".contrib-rules.yml"

branch="$(git rev-parse --abbrev-ref HEAD)"
pattern="$(python3 -c "import yaml;print(yaml.safe_load(open('$RULES'))['branch_pattern'])")"

if [[ ! "$branch" =~ $pattern ]]; then
  echo "preflight: branch '$branch' does not match $pattern" >&2
  exit 1
fi

python3 scripts/preflight_rules.py "$BASE"
make fmt
make lint
make test

#!/usr/bin/env python3
"""preflight_rules.py — verify file-level contribution rules locally."""
from __future__ import annotations

import fnmatch
import pathlib
import subprocess
import sys

import yaml

RULES = yaml.safe_load(pathlib.Path(".contrib-rules.yml").read_text())


def changed_files(base: str) -> list[str]:
    out = subprocess.run(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        capture_output=True,
        text=True,
        check=True,
    )
    return [line for line in out.stdout.splitlines() if line.strip()]


def check_required_files(changed: list[str]) -> list[str]:
    problems: list[str] = []
    for rule in RULES.get("required_when", []):
        triggers = rule.get("when_changed", [])
        touched = any(fnmatch.fnmatch(f, p) for f in changed for p in triggers)
        if touched and rule["path"] not in changed:
            problems.append(f"required file not updated: {rule['path']}")
    return problems


if __name__ == "__main__":
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    issues = check_required_files(changed_files(base))
    for issue in issues:
        print(f"preflight: {issue}", file=sys.stderr)
    sys.exit(1 if issues else 0)

Reproducing the failure in a clean checkout

A gate that only passes on a dirty working tree teaches nothing, so reproduce the failure once in a throwaway worktree. This isolates the patch from editor state, local build artifacts, and half-installed dependencies.

git worktree add ../oss-repro -b preflight-check origin/main
cd ../oss-repro
git cherry-pick <your-commit-sha>
bash scripts/preflight.sh

If the gate passes in the clean worktree and fails in CI, the difference is environmental rather than logical, and the useful artifact becomes a pinned environment file instead of another code change.

Step 3: Use a free model tier as a second reviewer

Preflight catches mechanical rules, but it cannot notice that a diff touches a code path the rule file never anticipated. That is where a free model tier adds value, provided the review is constrained to the same rule file that the gate uses.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, and the workflow below uses the free model access tier for the two reading-heavy steps.

Input A: RULES (the .contrib-rules.yml content)
Input B: DIFF (unified diff, no unrelated files)

List every rule from RULES that the diff may violate.
For each candidate, output: rule path, evidence line from the diff, confidence (low/medium/high),
and the exact question a human must answer.
Do not suggest preferences that are absent from RULES.
If no rule applies, return the single line: no rule-based findings.

The prompt is intentionally boring, because an unconstrained reviewer produces stylistic noise that costs more to triage than it saves. Every finding is a question for a human, never an automatic change request.

Where the free server option fits

Some projects cannot run their test suite on a contributor laptop, because the suite needs containers, services, or a long wall-clock budget. In those cases the free server option can host the preflight and test run, so the gate executes in a consistent environment instead of a laptop that happens to have the right toolchain.

Availability limits, provisioning details, and whether the free tier covers a specific workload should be confirmed against the project's current documentation before a team depends on it. Treat the free path as a convenience for review and reproduction steps, not as a substitute for the project's own CI.

Decision table: what to automate, what to keep human

Task	Delegate to a model?	Human check required
Extract rules from CONTRIBUTING.md	Yes, free tier is sufficient	Every rule and its source line
Branch and file-naming checks	No, run deterministically in preflight	None beyond approving the rules
Test failure triage	Yes, as a summarizer	Reproduce the failure in a clean worktree
Writing the patch itself	Optional, with a scoped prompt	Full diff review plus test evidence
Changelog and docs wording	Yes, as a first draft	Accuracy of version and behavior claims
Security-sensitive changes	No	Maintainer review and threat modeling
Final merge decision	Never	Maintainer, always

Limitations and failure modes

Rule extraction can misread a conditional sentence, so needs_human markers matter more than tidy output.
Model review produces confident false positives, and each one consumes maintainer attention.
The rules file drifts from CONTRIBUTING.md unless a scheduled CI job re-extracts and diffs it.
A preflight gate hides failures only if contributors remember to run it, so wire it into a pre-push hook rather than a wiki page.
Free-tier availability is not a service guarantee, and long-running suites still need a fallback path.

Who should skip this workflow

Repositories that already expose one authoritative make check target do not need a parallel gate, because the rules are already executable. Projects with strict supply-chain requirements should keep model-assisted review out of the loop entirely, since prompts can leak unreleased code. Single-maintainer projects with low contribution volume usually gain more from a shorter CONTRIBUTING.md than from automation.

A short checklist before push

Confirm which sentence in CONTRIBUTING.md each rule in the YAML file came from.
Run the preflight script in a clean worktree, not in your editor's dirty tree.
Restrict model review to rules that already exist, and convert every finding into a question.
Re-extract the rules file when the underlying document changes, and review the diff.
Revert any change that the gate cannot explain in one sentence.

The goal is not to automate contribution judgment; it is to stop paying review latency for rules that were written down months ago. Teams that want to try the free model access and free server path described above can evaluate it against their own repository rules before adopting any of it.

Redact the Review Packet Before a Model Sees an OSS Failure Log

Blake Yang — Thu, 24 Sep 2026 19:24:40 +0000

A small open-source patch often dies in review for reasons that never appear in the diff stat. A contributor copies a failing CI log into a chat box and asks a model to propose a fix. The log still holds a short-lived fixture token and an absolute path from the runner home directory. The returned diff also deletes a comment that explained why a locale-sensitive comparison had to stay explicit.

The pull request then looks cleaner than the bug, and that kind of cleanliness is a poor form of progress. A maintainer can reject the change before anyone gives the new tests a careful and patient reading. The failure is a workflow problem, not a reason to ban careful model help on contributor patches. The durable order remains reproduce, isolate, patch, and test, using a packet the model is allowed to see.

Free model access can critique that packet once the current quota and server terms are confirmed in the docs. The method stays useful for the reader even if every product name is removed from the article. Unverified allowance numbers stay labeled as outreach claims and are not treated as measured product facts.

Reproduce before the model speaks

The first artifact is a local reproducer that a stranger could run again from a clean checkout. A contributor checks out the default branch, records the commit, and runs the narrowest command that fails. Failure output lands in a scratch directory that the repository already ignores through its gitignore rules. Only after that file exists should anyone ask whether a model is able to interpret the failure.

git fetch origin
git switch --detach origin/main
mkdir -p /tmp/oss-repro
git rev-parse HEAD > /tmp/oss-repro/base.txt
# Replace the test path with the failing target named in the issue.
pytest tests/test_locale_sort.py -q --tb=short > /tmp/oss-repro/fail.txt 2>&1
echo $? > /tmp/oss-repro/exit_code.txt

Those commands illustrate a pattern, and they do not claim that the target repository actually uses pytest. The contributor still matches the project's documented test entry point before trusting the saved failure log. A model should not guess the runner from a README fragment pasted without the surrounding project config. If the documented command needs a service the laptop lacks, that gap is written down instead of invented away.

Build a redacted review packet

The second artifact is a short redaction script that strips obvious secrets and machine-specific local paths. The script below is a proposal, and it has not been executed against a real CI log for this draft. It stays small so a human can audit every substitution rule before the first local run. It will miss some secret shapes, and it should not be treated as a complete security scanner.

#!/usr/bin/env python3
"""Proposal: redact a local failure log before model review. Not executed here."""
import re
import sys
from pathlib import Path

RULES = [
    (re.compile(r"(?i)(api[_-]?key|token|password|secret)\s*[:=]\s*\S+"), r"\1=<redacted>"),
    (re.compile(r"/home/[^/\s]+"), "/home/<user>"),
    (re.compile(r"/Users/[^/\s]+"), "/Users/<user>"),
    (re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b"), "<email>"),
]

def redact(text: str) -> str:
    for pattern, repl in RULES:
        text = pattern.sub(repl, text)
    return text

def main() -> int:
    source = Path(sys.argv[1])
    target = Path(sys.argv[2])
    target.write_text(redact(source.read_text(encoding="utf-8")), encoding="utf-8")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

A contributor reads the rules, runs the script against a copy, and then compares both files by eye. Lines that still show hostnames, customer identifiers, or signed URLs remain stored on the local disk. The packet allowed to leave holds the base commit, the exit code, the redacted traceback, and a human note.

python3 scripts/redact_log.py /tmp/oss-repro/fail.txt /tmp/oss-repro/fail.redacted.txt
diff -u /tmp/oss-repro/fail.txt /tmp/oss-repro/fail.redacted.txt | less

What stays out of the packet

Several items never belong in a hosted review session, even after a quick automated redaction pass. Raw environment dumps, private keys, and unpublished vulnerability notes all stay on the local machine. Screenshots of internal dashboards are also excluded, because extracted text can still hide live tokens. When the failure cannot be described without those materials, the contributor asks maintainers for a public fixture.

Lock comments that carry invariants

Clear code often depends on a comment that a tidy automated rewrite will happily delete without asking. Recent public developer discussions have stressed that comments were never the real enemy of practical clarity. A note about a locale rule, a protocol quirk, or a security boundary belongs to the module contract. A model that prefers shorter functions will treat that note as clutter unless the packet forbids deletion.

The contributor extracts those sentences into a lock file before anyone starts drafting the actual patch. Each anchor is a full sentence copied from the current source, not a paraphrase written for the model. Three or four anchors are enough for a focused bugfix that touches only a single small module. A change that needs dozens of locks is too wide for one session and should be split before review.

A lock file can start as plain text, with one verbatim source sentence stored on each line.

# Copied verbatim from current source. Sample anchors only, not from a real repository.
Keep the comparison explicit because locale-aware sort order is not stable across runner images.
Do not cache the token beyond this function, because the fixture clock resets between tests.

# Proposal check, run from the feature branch after the patch is applied locally.
while IFS= read -r anchor; do
  case "$anchor" in ''|\#*) continue ;; esac
  if ! git grep -F -- "$anchor" >/dev/null; then
    printf 'missing invariant comment: %s\n' "$anchor"
  fi
done < /tmp/oss-repro/comment-locks.txt

The check uses exact text search, so a harmless rewording looks like a failure, which is the intended bias. The contributor restores the original sentence or asks the maintainer before changing the locked wording. A passing lock check does not prove the behavior is correct, and it only shows the chosen sentences survived.

Ask for a review, not an unscoped rewrite

The prompt stays narrow once the redacted packet and the comment lock file both exist on disk. The human asks for risks, missing tests, and comment deletions, and does not request a whole-file replacement. A useful reply names a line range, a suspected behavior change, and a test that would expose that suspicion. A reply that rewrites unrelated functions is discarded, even when the surrounding prose sounds fully confident.

Local commands remain authoritative after any accepted hunk is applied by hand in the working tree. The contributor reruns the original failing command, then one neighboring test target, and records both exit codes. A green result on a free remote server does not replace project CI, because images may differ from the matrix. Remote runs stay a first-pass convenience, and they are not evidence that the contributor patch is finished.

Attach only the redacted log, the recorded base commit, and the comment-lock list to the model session.
Require a risk list that includes a test idea, and reject any reply that edits project files on its own.
Apply the accepted hunks manually, then rerun the original command together with one neighboring test target.
Record the new exit codes in the pull request, and keep every redacted log file out of git history.

Decision table for where the work should run

The following table routes work among the laptop, a free model review, and a free remote server option. It is a decision aid for contributors, and it is not a benchmark of speed, price, or model accuracy. Cells marked unsuitable mean the hosted step stays unused, not that a sharper prompt makes the upload safe. Contributors who want measurements should time their own repository rather than borrow numbers from this article.

Situation	Local machine	Free model review	Free remote server
Log may contain a token or a home path	Redact first, then decide	Do not upload the raw log	Do not upload the raw log
One-file logic bug with locked comments	Reproduce and patch here	Critique the diff and the locks	Optional smoke test only
Needs a maintainer-only secret or private fixture	Stop and ask maintainers	Unsuitable	Unsuitable
Touches public API, migrations, or security checks	Write the behavior note locally	Second opinion only	Not sufficient evidence
Suite needs services the laptop cannot start	Record the gap in the note	Explain the gap, invent nothing	Use only if the image is documented

Where a free assistant fits, without invented limits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode enters this method as an open-source coding assistant only after the packet is redacted and comments are locked. Operator notes supplied with this draft describe free model access and a free server option for that bounded review. Those notes also mention a free token allowance on the order of ten million tokens for outreach readers. A plan that assumes a permanent quota will break the moment the published limits change without notice.

This article does not treat that allowance, any model name, hardware shape, or server duration as a verified fact. No primary source dated 2026-09-25 was attached, so the figure stays an unverified operator claim. Readers should confirm the live offer in the current project documentation before planning a session around it. If the docs and the outreach notes disagree, the documentation wins and the notes should be ignored.

A free model pass can list risks against the redacted packet when the docs still describe that access. A free server can host a smoke run only when the current docs offer an image that matches the task. Neither option should ever receive raw CI logs, private keys, customer fixtures, or unpublished security reports. Readers who never open the project can still keep the script, the lock file, and the routing table.

One practical follow-up is to place this packet layout beside the current MonkeyCode docs and compare coverage. The comparison should ask whether the free options cover a single redacted review, not an entire migration. That check belongs to the contributor, and it should happen before any log leaves the laptop.

Limitations

The sample rules miss multiline secrets, encoded blobs, and tokens that lack a familiar key name nearby. An exact comment lock fails closed when a later commit has already reworded the invariant in place. A model can still invent a test path the repository does not contain, so exit codes must come from local commands. Free access can shrink, change models, or end without notice, which makes a fixed-quota plan fragile from the start.

This draft records no timings, no accuracy scores, and no claim that a hosted run matches maintainer CI images. The redaction script can also over-redact and hide the very path segment a reviewer needed to see. Human review of the diff between raw and redacted logs is mandatory, and it cannot be skipped for speed. A workflow this small will not catch a wrong bug theory that happens to leave every locked comment intact.

Who should skip this workflow

Teams handling production credentials, medical records, or unpublished vulnerability details should not upload those failure logs. Maintainers who require a contributor agreement before any external processing should enforce that policy before upload. Contributors who cannot yet run the documented tests locally should close that gap before asking a model for help. A wide refactor is a poor fit, because a short lock list cannot describe every behavior change on the branch.

The habit that survives a rejected suggestion is smaller and older than any particular assistant product. Reproduce the failure, redact the evidence, lock the comments that carry invariants, and let the project tests decide. A free review pass can sit beside that habit, and it never replaces the maintainer who must still read the diff.

Freeze User-Visible Strings Before an OSS Patch Hits Review

Blake Yang — Wed, 23 Sep 2026 16:17:23 +0000

A mid-size command-line library accepted a one-line error-message change on an otherwise quiet Thursday afternoon. The contributor had reproduced a confusing parser failure, corrected the branch, and then rewrote the printed message for newcomers. Local unit tests stayed green because they asserted exception classes rather than the exact printed text. Two days later a packaging script that grepped the old phrase failed inside a downstream nightly job.

Maintainers meet this pattern often enough that it no longer reads as a harmless wording nit. Scripts, runbooks, support macros, and tutorial snippets treat error text, log lines, and CLI help as shipped contracts. A coding model asked to improve clarity will rewrite those strings unless the contribution process freezes them first. The rest of this article records a ledger workflow that keeps that freeze explicit.

Why string drift survives green tests

Most OSS suites protect control flow rather than the exact bytes that leave the process. Assertions check status codes, exception types, and structured fields, while the human-readable remainder stays unpinned. Formatters and AI review passes then treat that remainder as commentary instead of interface. That is how a correct parser fix still breaks a downstream nightly job.

Treat three classes of text as user-visible until a maintainer writes otherwise in review notes.

Error messages and panic text that operators may grep during an incident
Log lines that on-call documents quote inside runbooks and playbooks
CLI help, flag names, and usage examples rendered by --help output

Documentation headings can move without a migration window in most projects. Previously shipped messages and wire-format names should not move without an explicit note.

Reproduce, inventory, patch, then review

The sequence below assumes a clean local checkout of the upstream default branch. It does not require one language, though the sample helper is Python. Each stage produces a file that later stages can check. The model pass, if used at all, only compares those files.

Reproduce on a throwaway worktree

Create an isolated worktree so the inventory never mixes with unrelated dirty files. Fetch the default branch before copying so the ledger compares against current upstream text. Run the original failing command and keep stderr outside the repository tree.

git fetch origin
git worktree add /tmp/oss-string-freeze origin/main
cd /tmp/oss-string-freeze
python -m pytest tests/test_parser.py -k bad_timestamp -vv
python -m app.cli parse broken.json 2> /tmp/before.err || true

The worktree is disposable after the patch lands. The stderr capture is evidence, not product code, and should stay uncommitted. If the project uses fixtures rather than a CLI, dump the raised message with the same redirect pattern.

Build a string ledger from the intended diff

After a candidate patch exists on a branch, extract added and removed quoted strings instead of reading the diff as prose. Exclude vendored trees and snapshot files so generated noise does not flood the ledger. The helper below is a starting point, not a parser for every language.

git diff origin/main...HEAD -- ':!vendor' ':!*.snap' ':!po/*.po' > /tmp/patch.diff
python3 extract_strings.py origin/main > STRING_LEDGER.md

#!/usr/bin/env python3
"""Extract added and removed quoted strings from git diff."""
import re
import subprocess
import sys

STRING_RE = re.compile(r'["\']([^"\']{8,})["\']')

def diff_lines(base):
    cmd = ["git", "diff", "-U0", f"{base}...HEAD", "--", ":!vendor", ":!*.snap"]
    out = subprocess.check_output(cmd, text=True)
    return out.splitlines()

def main():
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    added, removed = [], []
    for line in diff_lines(base):
        if line.startswith("+++") or line.startswith("---"):
            continue
        if line.startswith("+"):
            added.extend(STRING_RE.findall(line[1:]))
        elif line.startswith("-"):
            removed.extend(STRING_RE.findall(line[1:]))
    print("# String ledger")
    print("\n## Removed")
    for item in dict.fromkeys(removed):
        print(f"- {item}")
    print("\n## Added")
    for item in dict.fromkeys(added):
        print(f"- {item}")

if __name__ == "__main__":
    main()

Classify every removed line as freeze, migrate, or accident in the same file. Search the default branch for each removed string so the class is based on callers, not taste.

git grep -n -F "invalid timestamp in input" origin/main -- ':!tests' || true

# String ledger

| String | Direction | Class | Notes |
| --- | --- | --- | --- |
| invalid timestamp in input | removed | migrate | keep old text for one minor release |
| cannot parse timestamp: expected RFC3339 | added | new | document in CHANGELOG |
| debug dump written to /tmp | added | accident | drop before review |

A freeze class means the patch must restore the old bytes before review. A migrate class requires a changelog bullet and, when possible, a dual-message window. An accident class is text the author or an assistant introduced without a product reason.

Patch against the ledger, not against taste

Apply the functional fix first and only then decide whether any user-visible string must change. If a message is false because the behavior was false, keep the old wording when downstream parsers depend on it. Ship both sentences for one release when the project already has a deprecation policy.

# Proposal: dual-message window, not a silent rewrite.
# Leave unexecuted until the project agrees on deprecation policy.
raise ParserError(
    "invalid timestamp in input; expected RFC3339"
)

Label dual-message examples as proposals when the repository has no published deprecation policy. Do not let a model pick the friendlier sentence as a default. Restore any accident lines by deleting them from the working tree before the next test run.

Test the ledger, then the original behavior

Add a cheap presence check so frozen strings still appear in source. This is not a localization framework and does not parse format strings. It only stops an accidental rewrite from riding along with a logic fix.

# tests/test_string_ledger.py
from pathlib import Path

FROZEN = [
    "invalid timestamp in input",
]

def test_frozen_strings_still_present():
    paths = Path("src").rglob("*.py")
    joined = "\n".join(p.read_text(encoding="utf-8") for p in paths)
    missing = [item for item in FROZEN if item not in joined]
    assert missing == [], missing

Run the ledger test before the original reproducer so a wording regression fails first. Capture stderr again and compare it with the file from the worktree setup. If the wording diff includes a freeze class string, restore those bytes before any review pass.

python -m pytest tests/test_string_ledger.py tests/test_parser.py -vv
python -m app.cli parse broken.json 2> /tmp/after.err || true
diff -u /tmp/before.err /tmp/after.err || true

Compare the ledger with the diff on a free model

This is the only stage where a coding model should enter the loop. The task is comparison, not rewrite, and the prompt should forbid wording suggestions. Paste git diff origin/main...HEAD together with STRING_LEDGER.md and ask for omissions only.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Contributors who already build this inventory locally can send the ledger and the diff through MonkeyCode's free model access, and can use the free server option when that comparison should run off the laptop. The model exists to catch table omissions, not to invent clearer error text.

You are comparing STRING_LEDGER.md with the unified diff.
List: (1) added or removed quoted strings in the diff that lack a ledger row,
(2) ledger rows whose text does not appear in the diff,
(3) any freeze row whose old bytes are still missing from the tree.
Do not suggest wording improvements. Do not rewrite the patch.

If the model reports extra strings, update the ledger or revert the extra edits. If it reports none, the patch can go to a human maintainer, who still owns tone and deprecation policy.

Decision table for string classes

Class	When to use it	Required artifact	Ready for review
freeze	Downstream scripts or docs quote the exact bytes	ledger row plus presence test	old string still in the tree
migrate	Behavior change makes the old sentence false	ledger row plus changelog bullet	old and new text both documented
new	Entirely new failure mode or help flag	ledger row only	no removed twin hiding nearby
accident	Debug dumps, speculative help, assistant asides	none; delete the text	gone from `git diff`

Use the table as a gate, not as a style guide. A patch that still contains an accident row is not ready, even when tests are green. A migrate row without a changelog bullet is also not ready, because downstream authors will not learn the new bytes.

Limitations

Quoted-string extraction misses template literals, concatenated fragments, and gettext catalogs. Dynamic messages built from several pieces will under-count, so the ledger is a net rather than a proof. Generated code, vendored files, and snapshot fixtures can flood the report unless those paths are excluded from git diff.

Projects that localize every user-facing sentence should run this workflow against message catalogs rather than source literals. Security advisories sometimes must change an error to avoid leaking internals, and those patches should skip the freeze class. A hosted model will not know the deprecation policy unless that policy is pasted beside the ledger.

The free model pass does not replace the local worktree, the presence test, or maintainer judgment. No duration, quota, hardware size, or model name is implied by using a free server for the comparison. Teams that need a guarantee against string drift still have to keep the presence test in CI.

Who should skip this approach

Drive-by typo fixes in comments do not need a ledger or a model pass. Embargoed security patches should not paste diffs into any hosted model, even for a string comparison. Repositories without a shipped CLI, log contract, or scriptable error text will spend more time classifying strings than they will save in review.

Authors who cannot run the original reproducer locally should not certify that user-visible output is stable. The extra file exists to keep taste out of the first review pass, not to freeze language forever. Maintainers can still improve wording after the functional fix lands, and they should do that in a dedicated docs change.

Contributors who already keep a worktree and a ledger can run the comparison on MonkeyCode's free models when they want a second pass on the table.

Match the Repo Test Dialect Before an OSS Patch Leaves the Fork

Blake Yang — Tue, 22 Sep 2026 14:34:17 +0000

A first-time contributor cloned a popular HTTP client on a Sunday evening and reproduced a timeout bug. The crash appeared only when a redirect chain mixed HTTP/2 frames with a stale connection pool. The patch looked small in the editor, yet the project's CI matrix covered three language versions and two TLS backends. Maintainers bounced the pull request because the new test used a helper name that did not exist in the suite.

This composite scene shows up in public trackers even when the functional fix itself is technically correct. Reviewers then spend scarce maintainer time teaching house style instead of judging the intended behavior change. A merge packet that freezes the failing assertion, shadows the upstream job, and checks dialect cuts that extra round trip. The workflow treats a free coding model as a dialect reviewer rather than as the author of production code.

Why dialect mismatches stall otherwise valid patches

Open-source test suites accumulate local verbs, fixture factories, and assertion helpers that never appear in tutorials. A new file that imports unittest mock patterns into a pytest house, or the reverse, reads like a drive-by rewrite. Continuous integration then fails on collection errors long before it can exercise the regression the patch claims to close. Maintainers reasonably ask for a rewrite because they cannot tell a real fix from a second coding style landing in tree.

House dialect is not the same thing as clean code in the abstract. Neighboring tests already show how the project fakes time, sockets, clocks, and HTTP backends. Copying that local grammar keeps the review focused on behavior instead of on renaming helpers under discussion. The fork should learn that grammar before any model is invited to comment on the diff.

Build a merge packet on the fork

The packet is a small directory that travels beside the branch and survives a clean clone of the public fork. It stores a frozen assertion, a CI test selector, a dialect sample, and the rebased candidate diff. Maintainers can replay those files after checkout without reconstructing the original laptop or the container image. The same packet is the only context a free model should receive during a later dialect review pass.

Packet layout

Place the files under .merge-packet/ and keep them untracked or tracked, according to the project's ignore rules:

assertion.lock — failing node id, exact assertion text, and captured at SHA
selector.txt — the CI test command copied from the workflow or Makefile
base.sha — default-branch commit that the selector was last proven against
dialect_sample.md — excerpts from three nearby tests, with repository paths
patch.diff — behavior change plus matching test, rebased onto the default branch
review_checklist.md — allowed dialect findings, isolation notes, and rollback path

Step 1 — Freeze one assertion, not the entire CI log

Contributors should copy the failing node id and the exact assertion text into assertion.lock before editing production files. Full logs mix compiler warnings, retry noise, and unrelated suite failures that hide the contract under review. A frozen message becomes the oracle that later tests must fail before the patch and pass after the patch. If the suite cannot address that node in isolation, the contribution is not ready for a model-assisted review.

The following file is a template, not a capture from a live upstream run:

# .merge-packet/assertion.lock
# Template only — replace with values from the local failing node.
node_id: tests/client/test_redirects.py::test_timeout_on_stale_pool
assertion: >
  AssertionError: expected ReadTimeout after stale HTTP/2 pool reuse,
  got 200 from redirected origin
captured_at: 2026-09-22T00:00:00Z
notes:
  - Fail this node on the default branch before applying patch.diff.
  - Pass this node on the branch after patch.diff lands.

Step 2 — Copy the upstream selector from the workflow file

Most projects already declare the test command inside GitHub Actions, GitLab CI, or a Makefile target. The fork should run that same selector rather than a personal pytest invocation with extra flags and random markers. Shadowing the job catches missing extras, skipped markers, and environment variables that never appear in README snippets. The selector belongs in selector.txt so a reviewer can see the exact command without opening the workflow history.

#!/usr/bin/env bash
# .merge-packet/shadow_ci.sh
# Template only — not executed against a live upstream in this article.
set -euo pipefail

ROOT="$(git rev-parse --show-toplevel)"
PACKET="${ROOT}/.merge-packet"
cd "${ROOT}"

git rev-parse HEAD > "${PACKET}/head.sha"
BASE="$(tr -d '\r' < "${PACKET}/base.sha")"
echo "shadowing selector against recorded base ${BASE}"

SELECTOR="$(tr -d '\r' < "${PACKET}/selector.txt")"
# Example selector.txt line:
# PYTHONPATH=src pytest tests/client/test_redirects.py::test_timeout_on_stale_pool -q
eval "${SELECTOR}"

Record the default-branch SHA before the first shadow run so later rebases have a comparison point. A green run against an unknown base does not prove that upstream collection will accept the new module path. Contributors should refresh base.sha whenever they merge or rebase onto the project's default branch. The packet then shows reviewers which tree the selector last executed against.

Step 3 — Sample three nearby tests as the dialect source

Dialect is easier to copy from neighboring files than from a style guide that the repository never actually follows. The sample should include fixture names, assertion helpers, time control, and how the suite talks to the network. Three files are enough to show repeated patterns without dumping the entire tests directory into a prompt window. A short extractor can print imports and assertion calls so the sample stays mechanical instead of hand-wavy.

# .merge-packet/extract_dialect.py
# Template only — unexecuted example for pytest-like suites.
from __future__ import annotations

import ast
from pathlib import Path

MARKERS = ("assert", "pytest", "fixture", "monkeypatch", "responses", "respx", "freezegun", "httpx")

def summarize(path: Path) -> str:
    tree = ast.parse(path.read_text(encoding="utf-8"))
    imports: list[str] = []
    asserts: list[str] = []
    for node in ast.walk(tree):
        if isinstance(node, ast.ImportFrom) and node.module:
            imports.append(f"from {node.module} import ...")
        if isinstance(node, ast.Assert):
            asserts.append(ast.get_source_segment(path.read_text(encoding="utf-8"), node) or "assert ...")
    kept_imports = [line for line in imports if any(m in line.lower() for m in MARKERS)]
    lines = [f"## {path.as_posix()}", "### imports"] + kept_imports[:12]
    lines += ["### assertions"] + asserts[:12]
    return "\n".join(lines) + "\n"

if __name__ == "__main__":
    neighbors = [
        Path("tests/client/test_redirects.py"),
        Path("tests/client/test_pool.py"),
        Path("tests/client/test_timeouts.py"),
    ]
    Path(".merge-packet/dialect_sample.md").write_text(
        "\n".join(summarize(p) for p in neighbors if p.exists()),
        encoding="utf-8",
    )

Choose neighbors by directory, not by popularity of the helper names. A timeout bug in the HTTP client should sample other client tests rather than unrelated CLI snapshot tests. Generated golden files are a weak dialect source because they hide the assertions that humans actually review. If the suite is generated, freeze one generated output and say so inside dialect_sample.md.

Step 4 — Isolate the patch from drive-by refactors

Reviewers reject mixed diffs that reformat imports, rename locals, and fix the bug inside a single commit. The candidate patch.diff should contain the behavior change and the matching test, rebased onto the default branch. Formatting and comment cleanups belong in a follow-up issue if the project even wants them at all. Isolation makes the dialect review cheaper because the model is not asked to judge unrelated churn.

# Template only — produce a reviewable diff after tests match dialect.
git fetch origin
git rebase origin/main
git diff origin/main...HEAD -- . ':!.merge-packet' > .merge-packet/patch.diff

The test added for the frozen assertion should sit next to those three neighbor files. New support helpers need a name already used in the sample, or a short note explaining why a new helper is required. Private modules that neighboring tests never import should stay out of the new file. That rule prevents the patch from widening the public test surface by accident.

Step 5 — Run a dialect review on free model access

A contributor who already uses MonkeyCode can run the dialect review through free model access and the free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The model should receive the merge packet only, not a dump of unrelated files from the working tree. Human maintainers remain the merge authority, and the model output is a checklist of dialect mismatches.

The prompt below is a labeled template. It asks for findings, not for a rewritten production patch, and it forbids inventing helpers that the sample does not contain.

You are reviewing an OSS merge packet for test dialect only.
Read assertion.lock, selector.txt, dialect_sample.md, and patch.diff.
Do not invent fixtures, helpers, or modules that dialect_sample.md does not show.
Return markdown with these headings only:
1. Dialect mismatches (file:line, nearby pattern, required change)
2. Isolation issues (drive-by refactors, unrelated renames)
3. Oracle check (does the new test target assertion.lock?)
4. Rollback note (how to revert without leaving the new helper)
If evidence is missing, say "unknown" instead of guessing house style.

Allowed findings stay narrow so the human can apply them without a second design debate:

Unknown helper or fixture name that the sample never uses.
Wrong fixture scope, including function versus module setup that neighbors avoid.
Real network calls where neighboring tests fake HTTP, TLS, or DNS.
Test file location that does not match the package layout in the sample.
Assertion style drift, such as raw == where the house wraps pytest.raises.

Replay the packet after every rebase

Rebases rewrite context lines, so the frozen assertion and the shadowed selector must run again after each rebase. A green local run against an old default branch is not evidence that upstream CI will collect the new test. The packet should record the default-branch SHA beside the selector so reviewers can see the exact base. If the assertion text drifts after rebase, the oracle is stale and the contributor must freeze a new message.

Store a short replay log next to the selector rather than pasting terminal color codes into the pull request. One passing line and one failing line, taken from the isolated node, are enough evidence for most maintainers. Timestamp the replay against base.sha so later comments do not argue about an obsolete tree. The pull request body can link the packet path instead of embedding a screenshot of an entire CI log.

Decision table

Signal on the fork	Action	Stop the PR if
Frozen assertion missing	Capture node id before editing production files	The suite cannot isolate the node
Selector differs from CI	Copy the command from the workflow file	The job needs secrets or private images
Dialect sample empty	Extract three neighbor tests	Tests are generated and have no assertions
Diff includes rename or format-only hunks	Split commits and regenerate `patch.diff`	The bugfix cannot be rebased alone
Model flags a helper mismatch	Rewrite the test to match `dialect_sample.md`	Production code needs an API redesign
Oracle still fails after the patch	Do not open the pull request	The frozen message no longer matches reality

Limitations

The workflow does not replace a maintainer reading the production change or checking license headers and security notes. Free model access can miss project-specific macros, generated fixtures, or custom pytest plugins that a sample will under-represent. Shadowing CI still fails when the upstream job needs secrets, hardware, or proprietary containers the fork cannot reproduce. None of the commands above were executed against a live upstream repository for this article, so treat them as templates.

Model review also cannot certify performance, licensing, or ABI promises. A dialect match only says the new test looks like the house, not that the patch is the right architectural change. Contributors still need the project's contributing guide, code of conduct, and any required test matrix beyond the single frozen node. Treat model output as a pre-submit lint for tests, not as a substitute for maintainer judgment.

Who should skip this approach

One-line documentation or typo pull requests with no test change.
Embargoed security fixes that must not land on a shared public server.
Changes that require proprietary hardware, licensed protocol dumps, or private containers.
Repositories that already enforce dialect with formatters, custom linters, and coverage gates.

Contributors sending one-line documentation fixes do not need a merge packet or a model review loop. Embargoed security patches should stay off public free servers until the project's disclosure process is complete. Newcomers who cannot run the upstream selector locally should ask for a repro harness instead of guessing dialect from chat. Teams that already enforce dialect with linters and coverage gates can keep those tools and skip the model pass.

Keep the oracle and the selector even without a model

The Sunday bounce was not about intelligence or effort; it was about landing a foreign test dialect in a living suite. A frozen assertion, a shadowed CI selector, and a three-file dialect sample give maintainers something they can replay. The merge packet remains useful if every model mention is removed, because the oracle and the selector are the artifacts. Readers who already have free model access nearby can run the dialect checklist there and still keep humans as the merge gate.

Ship an OSS Patch With a Reviewer Question Bank, Not Just a Diff

Blake Yang — Mon, 21 Sep 2026 12:46:43 +0000

A maintainer opened the crash-fix pull request after a long triage shift and found a three-line diff with no reproduction notes. The commit message claimed the parser no longer panicked, yet the issue still lacked a failing command for a clean checkout. Review time vanished into guesswork about callers, error paths, and whether the public API still returned the same sentinel values. The tab closed, and the contributor waited three days for a review that never started.

That stall is rarely about missing brilliance in the patch, and it is almost never about git hygiene alone. Maintainers ration attention across many repositories, and an unexplained diff is expensive even when the change is correct. Contributors who used a coding model to draft the hunks often ship fluent code that still fails the social contract of review. The missing artifact is not more commentary in the PR template; it is a question bank the contributor already answered.

Unexplained diffs stall because review is interrogation

Open-source review is a time-boxed interrogation, not a rubber stamp on a green CI badge. A careful reviewer still needs the reproduction, the moved public symbols, and the error paths that must keep failing. Those demands appear whether or not the contributor writes them down in advance of the request. Writing them first turns the pull request into evidence instead of a riddle wrapped in a unified diff.

Most contribution templates ask for a summary, a test plan, and a checklist of boxes. Those boxes age into ritual, and they do not change when the diff touches an exported error type. A question bank is generated from this branch's paths, headers, and exit codes, so the questions track the patch. The template in CONTRIBUTING.md can stay generic while the bank stays specific to the branch.

The artifact: REVIEWER_QUESTIONS.md

The proposed artifact is a single Markdown file named REVIEWER_QUESTIONS.md that lives beside the patch branch. Each entry is a maintainer question, a required evidence pointer, and a short answer that cites a file and line. Unanswered items remain marked TODO so a local gate can refuse to open the pull request. The file is not a blog post and should stay short enough to read in about twelve minutes.

# Reviewer Question Bank

Project: example-parser
Issue: https://github.com/example/parser/issues/1842
Branch: fix/issue-1842
Base: origin/main

## Q1. Reproduction
- Question: What single command fails on a clean checkout of main and passes on this branch?
- Evidence: TODO
- Answer: TODO

## Q2. Public surface
- Question: Which exported symbols changed name, type, or error contract?
- Evidence: TODO
- Answer: TODO

## Q3. Negative path
- Question: Which input must still fail, and what exit code is preserved?
- Evidence: TODO
- Answer: TODO

## Q4. Regression test
- Question: Where is the test that would have failed before the patch?
- Evidence: TODO
- Answer: TODO

## Q5. Commit shape
- Question: Does each commit encode one logical change a revert could isolate?
- Evidence: TODO
- Answer: TODO

## Q6. Docs and changelog
- Question: What user-visible behavior needs a changelog note, or why is none required?
- Evidence: TODO
- Answer: TODO

Six steps from a failing command to a fail-closed gate

The following workflow is a proposed local checklist, and the commands are illustrative rather than a recorded run against a public crate. Contributors should swap the binary names, fixture paths, and test runners for the project under patch. The point of the checklist is the review packet rather than any particular language or CI toolchain.

1. Freeze the failing command

Before any edit, record one command that fails on a clean checkout of the upstream default branch. Store that command in the question bank under reproduction so later answers cannot drift into a private setup. If the crash needs a fixture, check the fixture into the branch rather than describing it in prose. Reviewers should be able to paste the command without guessing at environment variables or hidden data files.

git fetch origin
git switch -c fix/issue-1842 origin/main
# Proposed: capture the failing invocation, not a narrative.
./target/debug/tool parse tests/fixtures/issue-1842.json
echo "expected non-zero on main"

2. Keep every hunk load-bearing

Edit only the symbols required to make that command pass, and resist drive-by cleanups that enlarge the review surface. A question bank cannot rescue a diff that also reformats a neighboring module or renames an unrelated helper. Split formatting and refactors into separate pull requests if the project will accept them at all. The bank should list every touched path so a maintainer can see the blast without running extra git archaeology.

git diff --name-only origin/main...HEAD
git diff --stat origin/main...HEAD
git log --oneline origin/main..HEAD

3. Collect exit codes instead of vibe screenshots

Run the original failing command, the nearest unit tests, and one negative path that should still error after the fix. Capture the exit codes and a one-line summary rather than pasting entire logs into the pull request body. The question bank needs those exit codes as answers, not as decorative terminal screenshots that hide the command. Label any unexecuted matrix as proposed so a maintainer does not treat a wish as a result.

# Proposed local evidence, not a claimed CI run.
set +e
./target/debug/tool parse tests/fixtures/issue-1842.json
echo "repro_exit=$?"
cargo test parse_issue_1842 -- --nocapture
echo "unit_exit=$?"
./target/debug/tool parse tests/fixtures/truncated.json
echo "negative_exit=$?"

4. Pack the diff and draft questions

A coding model can draft questions from the unified diff, but it cannot invent evidence the tests never produced. Feed the model the diff, the test commands, and the project's CONTRIBUTING snippet, then keep only questions a human can answer with citations.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft the bank from a unified diff, and the free server option can run the recorded test commands.

# Proposed helper: pack a small context bundle for a coding model.
mkdir -p .review-packet
git diff origin/main...HEAD > .review-packet/patch.diff
git log --oneline origin/main..HEAD > .review-packet/commits.txt
git diff --name-only origin/main...HEAD > .review-packet/paths.txt
python3 scripts/list_fn_headers.py < .review-packet/patch.diff \
  > .review-packet/headers.md

#!/usr/bin/env python3
"""Proposed helper: list added and removed fn headers from a unified diff."""
import sys


def main() -> None:
    added, removed = [], []
    for raw in sys.stdin:
        if raw.startswith("+++") or raw.startswith("---"):
            continue
        if raw.startswith("+") and "fn " in raw:
            added.append(raw[1:].rstrip())
        elif raw.startswith("-") and "fn " in raw:
            removed.append(raw[1:].rstrip())
    print("## Removed headers")
    for row in removed:
        print(f"- `{row}`")
    print("## Added headers")
    for row in added:
        print(f"- `{row}`")


if __name__ == "__main__":
    main()

Proposed prompt for the drafting pass, to be edited by a human before any answer is trusted:

You are drafting reviewer questions for an OSS patch, not writing the patch.
Read patch.diff, headers.md, and commits.txt.
Propose at most eight questions a maintainer would ask before merging.
Reject style nits. Prefer reproduction, public surface, negative paths,
tests, commit shape, and changelog impact.
Do not invent test results. Leave every Answer and Evidence as TODO.

5. Fill answers with file:line citations

Each answer should be one or two sentences and must point at a path a reviewer can open. Saying tests pass is not an answer, because it hides which command and which assertion moved. If the model drafted a question the tests cannot answer, delete the question instead of fabricating a citation. A short honest bank beats a long bank that launders missing evidence behind confident, fluent prose.

## Q1. Reproduction
- Question: What single command fails on a clean checkout of main and passes on this branch?
- Evidence: tests/fixtures/issue-1842.json, src/parse.rs:214
- Answer: `./target/debug/tool parse tests/fixtures/issue-1842.json` exited 101 on main and 0 on HEAD.

## Q2. Public surface
- Question: Which exported symbols changed name, type, or error contract?
- Evidence: src/lib.rs:88 (still `pub fn parse`)
- Answer: No exported name changed; `parse` still returns `Result<Doc, ParseError>`.

## Q3. Negative path
- Question: Which input must still fail, and what exit code is preserved?
- Evidence: tests/fixtures/truncated.json, src/parse.rs:240
- Answer: Truncated input still exits 1 with `ParseError::UnexpectedEof`.

6. Refuse to open the pull request while TODO remains

A local gate keeps the social contract from depending on memory when the pull request is finally opened. The script below is proposed, and it only searches for the TODO markers the template uses. Contributors can wire it as a pre-push hook or as a make target named review-packet. Opening the pull request then becomes a copy of the bank into the PR body, not a second writing task.

#!/usr/bin/env bash
# Proposed gate: refuse to open a PR while the bank still has TODO answers.
set -euo pipefail
BANK="REVIEWER_QUESTIONS.md"
if [[ ! -f "$BANK" ]]; then
  echo "missing $BANK" >&2
  exit 1
fi
if grep -E '^- (Answer|Evidence): TODO' "$BANK"; then
  echo "question bank is incomplete" >&2
  exit 1
fi
echo "question bank is complete"
# Proposed: only run after the gate passes.
# gh pr create --fill --body-file REVIEWER_QUESTIONS.md

Pass rules for each required question

Question	Required evidence	Pass rule
Reproduction	command plus exit codes	Fails on `origin/main`, passes on `HEAD`
Public surface	exported names from the diff	No undocumented rename or signature change
Negative path	one input that must still fail	Non-zero exit and original error type preserved
Regression test	test path and assertion line	Test failed before the patch and passes after
Commit shape	`git log --oneline`	One logical change per commit, revertible
Docs	changelog, rustdoc, or skip reason	Skip only for internal helpers with no user behavior

Limitations

The bank does not replace a maintainer who knows the crate's history, and it will not catch semantic bugs the tests never expressed. Models often over-generate questions about style and under-generate questions about concurrency, ABI, and platform quirks. A free server run is only as good as the command the contributor recorded in the first step. Teams that already require a formal design review should treat this file as a packet, not as a substitute for that review.

Generated questions also drift when the diff includes vendored code, snapshots, or generated protobuf bindings. In those cases the header lister will emit noise, and a human should delete those rows before the drafting pass. The gate only proves that answers exist as text; it does not prove that the cited lines implement the claim. Lying in the bank is still possible, so maintainers should sample at least one citation against the working tree.

Who should skip this packet

Tiny typo pull requests do not need a twelve-minute packet, and the process would look like ceremony. Security issues that cannot discuss the exploit path in public should stay in the project's private disclosure channel. Contributors who cannot run the failing command at all should not invent passing answers for a server they never invoked. Projects that forbid model-assisted patches in CONTRIBUTING.md should skip the drafting step and fill the bank by hand, or skip the bank.

Typo-only or comment-only pull requests
Private security disclosures that cannot describe the failure mode
Patches the contributor cannot execute on a clean checkout
Repositories that ban model-assisted contributions

The next time a crash fix looks ready, the contributor can spend twenty minutes answering the questions a tired maintainer will ask anyway. The resulting packet does not make the patch correct, but it makes incorrectness cheaper to see. A filled bank also gives a coding model a narrower job than rewriting a module until continuous integration turns green. That narrower job is the difference between a reviewable patch and a fluent diff that still stalls for days.

A One-Command Reproducer Kit That Survives a Clean OSS Checkout

Blake Yang — Sun, 20 Sep 2026 10:57:53 +0000

A contributor opened a pull request against a popular HTTP client with a three-line timeout fix and a screenshot. Maintainers could not reproduce the hang on CI, so the branch sat idle across two review cycles. The hidden cause was a leftover HTTP_PROXY export in a shell profile plus a vendored snapshot that no longer matched main. The patch itself was reasonable; the missing artifact was a reproduction kit that survived a clean clone.

Industry chatter in mid-September 2026 keeps returning to the same failure mode: generated diffs look complete while the engineering work around them is skipped. Open-source review is where that gap becomes public. A maintainer cannot merge a story about a laptop. A maintainer can merge a command that fails the same way on a fresh tree.

Dirty trees hide the actual bug

Local developer machines accumulate exports, toolchains, cached wheels, and half-applied patches that never exist on a maintainer laptop. A failure that depends on that residue will vanish the moment someone clones the repository into an empty directory. Maintainers then spend review time debugging the reporter's environment instead of the library. A clean-clone gate makes that class of confusion expensive for the contributor and cheap for the project.

Screenshots, truncated logs, and "it works on my machine after I source .envrc" notes do not travel. They also do not bisect. The kit described here is the first deliverable of an OSS bugfix, not an afterthought pasted under the diff. The patch comes only after the kit fails for the same reason in two isolated trees.

What the kit must contain

A useful kit is not a paragraph of remembered steps inside the issue form. It is a script, a pinned revision, an expected failure, and a short environment dump that another person can run without extra narrative. The kit should fail for the same observable reason on two machines before any production patch is written. The following items are the minimum set for most application-level bugs in published libraries:

pinned upstream remote plus a commit SHA or release tag
one command that exits non-zero while the bug is present
captured tool versions for the language, package manager, compiler, and OS
explicit unset of proxy, credential, and vendor-cache variables
a recorded last-known-good tag when the reporter already has that information
a short expected-versus-actual block that a stranger can compare by eye

The kit is not a substitute for the project's own test suite. It is a passport that lets a stranger reach the failing assertion without inheriting the reporter's shell.

A one-command template

The script below is a labeled template, not a captured run from a specific repository. Operators should replace the remote, SHA, language toolchain, and failing invocation with values from the issue they are actually working. The important property is isolation: a temporary directory, a fresh clone, and an environment that starts empty of proxy and virtualenv hints.

#!/usr/bin/env bash
# repro_kit.sh — labeled template for a clean-clone OSS reproducer
set -euo pipefail

REMOTE="${REMOTE:-https://github.com/example/http-client.git}"
SHA="${SHA:-a1b2c3d4e5f60718293a4b5c6d7e8f9012345678}"
WORKDIR="$(mktemp -d /tmp/oss-repro.XXXXXX)"
trap 'rm -rf "$WORKDIR"' EXIT

# Strip common local residue that maintainers will not have.
unset HTTP_PROXY HTTPS_PROXY ALL_PROXY http_proxy https_proxy all_proxy || true
unset VIRTUAL_ENV PYTHONPATH NODE_PATH GOPATH GOFLAGS CARGO_HOME || true
export PATH="/usr/bin:/bin:/usr/local/bin"

echo "workdir=$WORKDIR"
echo "uname=$(uname -a)"
command -v python3 >/dev/null && python3 --version || true
command -v git >/dev/null && git --version

git clone --quiet "$REMOTE" "$WORKDIR/src"
cd "$WORKDIR/src"
git checkout --quiet "$SHA"
echo "HEAD=$(git rev-parse HEAD)"
echo "describe=$(git describe --always --dirty)"

# Project-specific install. Keep it deterministic; avoid user-site packages.
python3 -m venv .venv
# shellcheck disable=SC1091
source .venv/bin/activate
python -m pip install -e . -q

# The single failing command. Replace with the issue's minimal trigger.
set +e
python - <<'PY'
from client import Session

s = Session(timeout=0.05)
try:
    s.get("https://example.invalid/slow")
    raise SystemExit("expected timeout, got success")
except Exception as exc:
    name = type(exc).__name__
    if name != "Timeout":
        print(f"actual_exception={name}: {exc}")
        raise SystemExit(2)
    print(f"reproduced:{name}")
    raise SystemExit(1)
PY
status=$?
set -e
echo "repro_exit=$status"
exit "$status"

A passing mental check for this script is simple. A stranger with network access and the language toolchain should reach the same non-zero exit without reading the pull request. If the script needs a paragraph of preamble, the kit is still a blog post and not yet a kit.

Expected versus actual, in the repo

Issue threads lose formatting and lose files. Checking a tiny sidecar into the branch keeps the contract next to the script. The file below is also a template; the strings must come from a real run, not from memory of the original crash.

# repro_kit.expect
remote: https://github.com/example/http-client.git
sha: a1b2c3d4e5f60718293a4b5c6d7e8f9012345678
command: python repro_min.py
expected_exit: 1
expected_stderr_contains: reproduced:Timeout
forbidden_stderr: HTTP_PROXY
last_known_good_tag: v2.4.1

Store the script and the expect file in a directory that reviewers can delete after merge, such as repro/issue-8412/. Do not hide them inside a personal gist that will rot. The pull request body should link to the path and paste the last local run, including HEAD and repro_exit.

Decision table: when the kit is honest

Use the table before writing production code. Each row is a gate. A "no" in the remote column means the bug is still a local myth, even if the laptop demo looks perfect.

Observation after `repro_kit.sh`	Local dirty tree	Fresh `/tmp` clone	Remote clean machine	Action
Same non-zero exit and same exception name	yes	yes	yes	Write the patch; kit is load-bearing
Fails locally, passes on fresh clone	yes	no	no	Stop; inspect exports, caches, extra remotes
Fails on fresh clone, passes on remote	yes	yes	no	Record OS, CPU, and filesystem; do not send a Linux-only guess
Passes everywhere, including the reported SHA	no	no	no	The issue is closed or mis-filed; do not invent a fix
Fails with different exceptions across machines	yes	yes	yes	Narrow the trigger; the kit is still too wide
Requires secrets, production data, or a private registry	yes	maybe	maybe	Do not publish the kit; use a redacted fixture or a private maintainer channel

The interesting row is the second one. That is the screenshot pull request. The clean clone is the cheapest way to discover it before a maintainer does.

Run the kit where the laptop cannot help

A second machine is not a luxury for this workflow. It is the control group. Continuous integration on the fork is one control group, provided the workflow file does not reuse caches that hide the bug. A disposable remote workspace is another, provided it starts without the reporter's direnv, keychain, or Docker layer cache.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can host a throwaway clone for this control-group run. The models are useful later for reviewing the kit text; the server is useful now because it does not inherit the laptop's residue. No quota, model catalog, or hardware profile is claimed here, because those details change and do not belong in a frozen article.

A practical remote pass looks like the following sequence. Clone into a new directory, copy only repro_kit.sh and repro_kit.expect, and refuse to copy .env, venv, or node_modules. Run the script once, capture stdout and stderr to a file, and attach that file to the issue. If the remote exit code disagrees with the laptop, the patch waits.

# Labeled example: capture a remote control-group run.
chmod +x repro_kit.sh
./repro_kit.sh > repro_remote.log 2>&1
echo "remote_exit=$?" | tee -a repro_remote.log
grep -E '^(HEAD|describe|repro_exit|reproduced:|actual_exception=)' repro_remote.log

Free-model review of the kit, not the vibes

Generated patches often read as confident while the reproduction story remains sloppy. The safer use of a free coding model in this workflow is editorial, and it happens before the production diff exists. Paste the script, the expect file, and the draft issue comment. Ask for a maintainer-facing review of clarity, not for a speculative fix.

A prompt that stays inside that boundary looks like the block below. It is labeled, because it is a proposal rather than a log from a particular model run.

You are reviewing an OSS reproduction kit, not writing a patch.
Point out missing pins, leftover environment coupling, and claims
that a stranger cannot verify with one command. List concrete edits
to repro_kit.sh, repro_kit.expect, and the issue comment. Do not
propose production code. Do not invent versions I did not supply.

Treat the model output as a diff against the kit. Keep every edit that removes hidden state, vague verbs, or un-pinned installs. Discard advice that adds a rewrite of the library. If the model wants to "just catch Timeout and retry," the workflow has already failed, because the task was documentation of the bug, not product design.

After the kit is tightened, a second short pass can check the pull request body for maintainer time costs. Useful flags include missing SHA, missing exit code, commands that assume a particular absolute path, and logs that still contain home-directory fragments. Those are review comments the reporter can fix without waiting for a human round trip.

Only then write the patch

Once local fresh-clone and remote clean-machine runs agree, the patch has a job: make repro_kit.sh exit zero for the right reason. Keep the kit on the branch until reviewers say otherwise. A green kit plus a green project test suite is a stronger story than a green suite alone, because the suite may never have covered the reported path.

Commit shape still matters. One commit should add the kit and show the failure on the parent SHA if the project's CI will allow a known-failing commit. The next commit should contain the fix and the permanent regression test that will remain after repro/ is deleted. Do not squash away that history if the project likes bisectable series. The kit is evidence; the regression test is the long-term contract.

# Labeled local sequence after both environments agree on the failure.
git checkout -b issue-8412-timeout-repro
git add repro/issue-8412/repro_kit.sh repro/issue-8412/repro_kit.expect
git commit -m "repro: add clean-clone kit for issue 8412"

# ... implement the library fix and a permanent test ...
git add src tests
git commit -m "fix: surface Timeout when the deadline is already expired"

If the permanent test cannot be written without the kit's private fixture, the fixture needs to be reduced until it can live under tests/. A kit that cannot be turned into a project test is still valuable for triage, but it should not be the only safety net after merge.

Limitations

This workflow assumes the bug is deterministic enough to fail a script twice. Flaky timing bugs, GPU-only faults, and races that need hours of load will not honor a one-command kit. Hardware-specific failures need a documented machine profile that a free generic server will not provide. The clean-clone gate also assumes the project can be built by a stranger from a tag; monorepos that require internal Bazel seeds or private wheels need a reduced public fixture first.

The workflow is the wrong tool for embargoed security reports. A public reproducer kit that demonstrates a practical exploit does not belong on a fork. Those issues follow the project's security policy, which often means a private channel and a delayed disclosure, not a repro/ directory on a personal branch. License and CLA constraints are also out of scope for the script; they still block the pull request even when the kit is perfect.

Free model access will not know whether the exception name is stable across minor tags. Free remote machines will not magically match an affected user's older distro. Both are controls against local dirt, not oracles. When the table's remote column cannot be filled, say so in the issue instead of implying a complete matrix.

Who should skip this approach

Drive-by typo fixes, documentation-only pull requests, and issues already failing on upstream CI do not need a personal reproducer kit. Maintainers who already pasted a failing job URL have done the isolation work. Contributors without permission to clone the code, or without a legal right to redistribute a fixture, should stop at a redacted description and wait for maintainer instructions.

Teams that punish extra files on a branch may prefer a gist or a CI job, but the isolation rules stay the same. The method is for reporters who are about to change shared behavior and who need a stranger to believe the bug exists. Readers who want a disposable machine for the control-group step can use MonkeyCode's free server option with its free model access, then keep the kit in the pull request regardless of where it was run.

Classify the Public Surface Before an OSS Crash Fix Hits Review

Blake Yang — Sat, 19 Sep 2026 09:15:48 +0000

Consider a mid-size TypeScript CLI that received a three-line pull request stopping a null crash in flag parsing. The reporter attached a passing unit test, a screenshot of the local run, and a short stack note. Maintainers still closed the request after one pass because an exported helper changed return values for empty argv. Downstream packages had treated that helper as a stable contract, so the crash fix was a breaking change wearing patch-sized clothing.

Correctness is not the only gate on an upstream pull request for libraries that already have downstream dependents in late 2026. Maintainers also protect exported names, documented flags, config keys, and the promise implied by the next version tag. A contributor who only proves the bug is gone still leaves contract classification sitting on the reviewer. The packet below keeps a human in charge of reproduce, patch, and test, then records a Semver Impact Card before anyone files the diff.

Why small crash fixes still stall on versioning

Issue threads reward a green test and a short, readable diff against the reported file. Versioning rewards a boring, named claim about what downstream code may keep calling after the next tag. Those two rewards pull in opposite directions when a crash lives inside a helper that already ships in __all__, export, or a public header. The helper can start returning a documented empty object instead of throwing, which is kinder to interactive users and still a meaning change for callers that caught the exception.

Review time then migrates from the hunk to an argument about frozen intent. Maintainers compare README examples, type stubs, changelog footnotes, and the last tagged tarball. Contributors repeat that the crash was never written down as a feature. The Semver Impact Card moves that argument into a table before the pull request exists, so the first human review starts on a classified claim rather than on informal confidence.

Four stages before `gh pr create`

The working sequence is reproduce, patch, test, and classify. Each stage writes a file under .patch-packet/ so the pull request can link evidence instead of retelling the narrative. Model output appears only after the first three stages are green on a clean tag, because a model that reviews a still-failing tree invents compatibility stories.

Reproduce the reported failure on the issue tag, not on a dirty local main.
Patch the owning module without drive-by refactors or formatting sweeps.
Run the project's test runner plus one new assertion that binds the contract.
Fill the Semver Impact Card and invite a model to falsify the proposed bump.

Stage 1: Reproduce on the reported tag

Check out the tag named in the issue, not a local branch that already contains extra commits. A dirty tree hides whether upstream already shipped a fix, which makes any compatibility claim dishonest. Record the command, expected output, and observed output in repro.md beside a tiny shell wrapper that exits non-zero on the broken tag.

git fetch --tags origin
git switch --detach v4.2.1
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
mkdir -p .patch-packet

cat > .patch-packet/repro.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
# Documented contract: empty argv yields an empty object and exit 0.
# Observed on v4.2.1: TypeError from parse_argv([]).
python -m samplecli parse --argv '' --format json
EOF
chmod +x .patch-packet/repro.sh
./.patch-packet/repro.sh; echo "exit=$?"

The wrapper must fail on the tagged release and later pass on the patched tree. If it cannot fail on that tag, the issue is not isolated enough for a compatibility claim, and the card should not be filled yet.

Stage 2: Patch the owning module only

Keep the diff inside the file that owns the crash path. Cross-cutting cleanups belong in a follow-up issue, not in a crash fix that also claims a version bump. After the edit, store a unified diff and a name-only file list next to the card so later review does not depend on a working tree.

git diff --stat
git diff -U3 -- src/samplecli/parse.py > .patch-packet/patch.diff
git diff --name-only > .patch-packet/files.txt

A crash fix in this shape is usually a null guard plus a default that already appears in the docs. If the hunk set grows past the failing function, split the work before filling the card, because extra files inflate the public-surface table without extra evidence.

Stage 3: Bind the new behavior with the project runner

Call the same runner the CI workflow already uses. Add one assertion that encodes the old crash input and the intended contract, then keep neighboring tests unmodified unless they encoded the crash as expected behavior. Do not introduce a second framework for a single upstream patch, because reviewers will not install it.

# tests/test_parse_empty_argv.py
from samplecli.parse import parse_argv


def test_empty_argv_returns_empty_object_without_raising():
    result = parse_argv([])
    assert result == {"flags": {}, "positionals": []}
    assert "help" not in result["flags"]

pytest -q tests/test_parse_empty_argv.py tests/test_parse.py
git add tests/test_parse_empty_argv.py src/samplecli/parse.py

Green tests still do not prove that exported meaning stayed compatible for downstream imports. That proof is the card in the next stage, not another assertion about private helpers or log lines.

Stage 4: Classify public surface before the pull request

Copy the template into .patch-packet/semver-impact.md and complete every field on the same day as the diff. Blank cells are a stop for the contributor, not a reminder for the maintainer to finish the taxonomy. The proposed bump must cite the project's own versioning document, even when that document is only a short README section.

Artifact: the Semver Impact Card

# Semver Impact Card

- Project / issue: samplecli#1842
- Base tag: v4.2.1
- Proposed bump: PATCH | MINOR | MAJOR
- Versioning source: VERSIONING.md (commit abc1234) or README "Versioning"

## Public surface touched
| Kind | Name | Before | After | Visible to downstream? |
| ---- | ---- | ------ | ----- | ---------------------- |
| function | parse_argv | raises TypeError on [] | returns empty object | yes, listed in __all__ |
| CLI flag | --format | unchanged | unchanged | yes |
| config key | (none) | | | |
| file format | (none) | | | |

## Classification rules used
- MAJOR: remove or rename an export, change return meaning, or reject previously valid input
- MINOR: add an export or optional flag whose default preserves old calls
- PATCH: fix a crash or wrong result without changing documented meaning

## Claim
PATCH. `parse_argv([])` now matches the documented empty-object example in README.md.
The TypeError was never listed as a contract in VERSIONING.md or the type stubs.

## Explicit non-claims
- Does not add --strict
- Does not change JSON key order
- Does not backport to the 3.x line

## Tests that bind the claim
- tests/test_parse_empty_argv.py (new)
- tests/test_parse.py (unchanged, still green)

## Items for maintainer confirmation
- Confirm parse_argv is public because __all__ includes it
- Confirm the README example, not the TypeError, is the frozen contract

The table is the artifact a maintainer can scan without opening every hunk. The non-claims section matters as much as the bump, because it blocks extra behavior from sneaking into the pull request title. Store the filled card in git even if the project will not merge the .patch-packet/ directory, then paste a summary into the pull request body.

Extract candidate exports from the unified diff

Hand-filled tables miss aliases, re-exports, and definitions added only as +export. A small illustrative script greps added definition lines, then checks __all__ on the current file. The output is a candidate list for the card, not a legal API boundary and not a substitute for reading VERSIONING.md.

#!/usr/bin/env python3
"""semver_scan.py — list definition lines in a unified diff (illustrative)."""
from __future__ import annotations

import re
import sys
from pathlib import Path

DEF_RE = re.compile(
    r"^\+\s*(?:export\s+)?(?:async\s+)?(?:def|function|class|const|exports\.)\s*([A-Za-z_][\w.]*)"
)
ALL_RE = re.compile(r"__all__\s*=\s*\[([^\]]*)\]", re.S)


def definitions(diff: str) -> list[str]:
    found: list[str] = []
    for line in diff.splitlines():
        match = DEF_RE.match(line)
        if match:
            found.append(match.group(1))
    return found


def public_all(py_source: str) -> set[str]:
    match = ALL_RE.search(py_source)
    if not match:
        return set()
    return {tok.strip(" \"'") for tok in match.group(1).split(",") if tok.strip()}


def main() -> None:
    diff = Path(sys.argv[1]).read_text(encoding="utf-8")
    names = definitions(diff)
    print("definitions_added_or_modified:")
    for name in names:
        print(f"  - {name}")
    if len(sys.argv) > 2:
        src = Path(sys.argv[2]).read_text(encoding="utf-8")
        pub = public_all(src)
        print("in__all__:")
        for name in names:
            print(f"  - {name}: {name in pub}")


if __name__ == "__main__":
    main()

python .patch-packet/semver_scan.py .patch-packet/patch.diff src/samplecli/parse.py

Label the script illustrative, because languages disagree about the word public. Go uses capitalization, Rust uses pub and pub use, C uses installed headers, and TypeScript uses export. The card must accept or reject each candidate by name, including names the scanner never saw because they live only in documentation examples.

Falsify the bump with a free model after tests pass

A language model is a weak author of production hunks and a useful adversary for a versioning claim. The human still reproduces the crash, writes the change, and runs the project test runner. The model receives only the card, the unified diff, and VERSIONING.md, then tries to show that the bump is too low, too high, or unsupported by the files.

Contributors without a private GPU can run that second pass on a free model endpoint. MonkeyCode currently offers free model access and a free server option for this kind of review host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Teams that want a hosted loop for the review prompt can use the free server option as one of several possible hosts. The product is not required; any local endpoint that accepts a text prompt can execute the same checklist against the packet files.

Save the prompt as .patch-packet/review-prompt.md and paste the card plus git diff --name-only under a clear delimiter.

You are reviewing a Semver Impact Card for an upstream pull request.
Do not rewrite the patch. Do not invent symbols that are absent from the diff.

Inputs:
1) semver-impact.md
2) git diff --name-only and the unified diff
3) VERSIONING.md or the README versioning section

Tasks:
- List every added, removed, or retyped name that looks exported.
- State whether the proposed bump is too low, too high, or consistent.
- Quote the rule from VERSIONING.md that supports that judgment.
- Flag any line in Explicit non-claims that the diff already implements.
- Return at most 12 lines with sections: missed_exports, bump_judgment, rule_quote, overclaims.

If evidence is missing, write insufficient evidence instead of guessing.

Run the prompt only after the wrapper fails on the base tag and passes on HEAD. Store the output in .patch-packet/model-review.txt and treat disagreements as contributor work, not as automatic edits to parse.py. A model that rewrites the hunk at this stage undoes the whole point of classifying a human patch.

Decision table for the proposed bump

Observation on the tagged tree	Typical bump	Stop filling the card if
Crash on input the docs already allow	PATCH	docs and tests disagree about the input
New optional flag with a preserving default	MINOR	the default changes old output
Return type or error class changes for valid input	MAJOR	the issue asked only for a crash fix
Rename of an exported helper	MAJOR	no alias remains for one release
Internal function only, absent from docs and `__all__`	PATCH or no release	the name appears in a public example

The table is a starting filter for contributors who do not yet know the project's customs. The project's VERSIONING.md wins when it conflicts with this chart, and the card should quote the winning sentence. When VERSIONING.md is missing, the card should say so in the versioning-source field and propose PATCH only for undocumented crash behavior that existing tests already allowed.

Sample pull request body

## Summary
Crash on empty argv in parse_argv. Packet claims PATCH against v4.2.1.

## Evidence
- Repro: .patch-packet/repro.sh fails on v4.2.1, passes on this branch
- Test: tests/test_parse_empty_argv.py
- Surface: .patch-packet/semver-impact.md
- Adversarial pass: .patch-packet/model-review.txt (informational only)

## Maintainer confirmation requested
Please confirm parse_argv is public and that README empty-object example is the contract.

Keep the model report informational. Maintainers did not ask for a generated review persona, and a contributor should not present model prose as CI.

Limitations

The scanner does not understand re-exports, decorator-wrapped APIs, or documentation-only contracts. A README sample can freeze behavior that never appears in __all__, and plugin entry points can expose names that live only in packaging metadata. Generated bindings need a human pass over the installed artifact, not merely over the source diff. Model review can miss domain meaning, such as a numeric default that looks compatible but breaks a wire protocol or a sort order that tests never pinned.

Free model access and a free server option do not make the review authoritative. They do not replace maintainer judgment, signed tags, or the project's CI matrix. The card also fails for embargoed security fixes that must not describe public surface in a public packet. Those patches follow the project's security policy rather than this template, and they should not be sent to a third-party endpoint.

Time-sensitive product quotas, model catalogs, and hardware lists change often and are omitted here on purpose. Operators should confirm current availability on the surface they actually use, rather than copying capacity claims from a blog post.

Who should skip this packet

Skip the card for typo-only documentation edits and for private forks with no downstream consumers. Skip it for contributors who have not reproduced the failure on a clean tag, because there is no honest base for a bump. Skip model review when the repository forbids sending source to third-party endpoints, including patches that contain secrets, customer data, or undisclosed vulnerabilities. Skip the whole packet when the maintainer template already requires a changelog with the same fields, because duplicate paperwork helps nobody and slows the merge.

What the pull request should link

A pull request that links the repro wrapper, the new test, semver-impact.md, and model-review.txt gives reviewers a contract claim they can accept or correct in one pass. The patch can still be wrong on performance or style, but the argument is about a named export and a named bump. Engineering work stays in reproduce, patch, and test; the model only pressure-tests the written claim after those stages are already green.

Declare the Behavior Delta Before Maintainers Open an OSS Bugfix

Blake Yang — Fri, 18 Sep 2026 07:11:15 +0000

A contributor cloned a popular CLI, reproduced a parser crash, and typed a confident patch in one sitting. The pull request touched a helper, two tests, and a changelog line that promised a complete fix. Maintainers opened the diff and still could not tell which captured outputs were supposed to change. Screenshots arrived later, then a second patch, then a request for logs the contributor had already discarded.

That stall is common when a bugfix is treated as a narrative instead of a measurable delta. Reviewers need the command, the before output, the after output, and a short allowlist of intended change. Without those four pieces, a green local run is only a private memory of one machine. The workflow below turns that memory into files a distant maintainer can replay without extra archaeology.

Treat the patch as a declared behavior delta

An OSS bugfix is not finished when the process exits zero on one laptop. The finished unit is a declared delta: old behavior, new behavior, and the exact invocation that produces both. Code review then checks whether the git hunks are necessary for that delta, rather than guessing intent from renamed helpers. This framing also limits accidental drive-by refactors that hide inside a supposedly small fix.

A useful delta statement is boring, local, and specific enough to fail in public. It names the binary or test target, the fixture, and the lines that must differ after the patch. It also names the outputs that must stay byte-identical so collateral churn cannot hide. Maintainers can reject a patch that cannot produce that statement in a few quiet minutes.

The oracle packet

Keep the packet beside the worktree, not inside a chat log that will rot. A simple directory is enough for most CLI tools and library bugs that fail in user space.

oracle/
  README.md
  invoke.sh
  before.txt
  after.txt
  before.exit
  after.exit
  allowlist.txt
  observed.diff
  check_allowlist.py
  check_oracle.sh

Each file has one job, and mixing those jobs is how packets become essays. invoke.sh is the only entry point a stranger should need to execute. before.txt and after.txt are raw captured bytes, not edited summaries of what the contributor remembers. allowlist.txt lists the fragments that are allowed to differ, and check_oracle.sh fails when the live diff escapes that list.

README.md as the human index

# Oracle packet: parse crash on empty --config

- Upstream issue: #4182
- Failing commit used for the before capture: 9f3c1aa
- Invocation: ./oracle/invoke.sh
- Intended delta: stderr gains a one-line usage error; process status becomes 2
- Forbidden delta: stdout of `--help` and the JSON fixture under tests/data/

The README is an index so a maintainer can skip archaeology, not a second design document. Link the issue, pin the commit, and state the delta in one plain sentence. Leave motivation and design opinions in the pull request body, where discussion belongs.

Step 1: freeze the failing invocation

Do not start in the editor. Write the invocation that currently fails, including working directory and environment variables that the tool actually reads. If the bug needs a fixture, generate that fixture in the same script so the packet stays closed.

#!/usr/bin/env bash
# oracle/invoke.sh
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"

export PYTHONUNBUFFERED=1
# Status is recorded below; do not abort the script on the expected failure.
set +e
python -m tools.parser --config tests/data/empty.toml \
  >oracle/stdout.txt 2>oracle/stderr.txt
echo "$?" > oracle/exit_code.txt
set -e

Record status, stdout, and stderr as separate streams when those streams matter to the bug. Mixing them in a terminal screenshot is how reviews lose the only signal they can replay. If a test runner can target one case, prefer that invocation over a manual CLI session with hidden shell history.

Step 2: capture the before side without editing it

Run the frozen invocation on the parent commit that still contains the bug. Save the streams before any local patch exists in that tree. Resist the urge to trim timestamps or absolute paths by hand, because silent edits make the after file incomparable.

git switch --detach 9f3c1aa
./oracle/invoke.sh
cp oracle/stderr.txt oracle/before.txt
cp oracle/exit_code.txt oracle/before.exit

If logs embed clocks, hostnames, or random request ids, freeze those sources in invoke.sh before treating the files as oracles. A packet that cannot be replayed tomorrow is a diary entry, not evidence. Characterization of the failure belongs in the captured bytes, not in a paraphrase of a stack trace.

Step 3: apply the smallest patch that should move the delta

Copy before.txt and before.exit to a safe path if the working tree will rerun the same script. Then restore a writable branch and change only what the issue asked to change. Do not fold formatter noise, comment rewrites, or unrelated helper cleanups into the same commit, because those extra hunks make the allowlist dishonest even when the crash is gone.

git switch --create fix/empty-config 9f3c1aa
# edit tools/parser.py and the focused test only
git diff --stat

A short self-check before capture keeps the later allowlist honest:

Every edited path is named in the issue or required by the failing invocation.
No lockfiles, generated docs, or IDE metadata slipped into git status.
The changelog line, if required by the project, matches the declared delta and nothing else.

Step 4: capture the after side with the same invocation

Use the identical invoke.sh. If the command line changes, the packet is invalid and the README must be rewritten first. Copy the new streams into after.txt and after.exit, then produce a unified diff the checker can read.

./oracle/invoke.sh
cp oracle/stderr.txt oracle/after.txt
cp oracle/exit_code.txt oracle/after.exit
diff -u oracle/before.txt oracle/after.txt > oracle/observed.diff || true

Read observed.diff as a product artifact, not as debug scrap from a local terminal. If the diff is empty, the patch did not change the failing behavior and review is premature. If the diff is huge, the patch changed more than the issue asked for and the allowlist will not save it.

Step 5: publish an allowlist, not a vibe

The allowlist is the original technical artifact of this workflow. It states which observed changes are load-bearing for the issue. Everything else is a defect in the packet, the invocation, or the patch itself.

# oracle/allowlist.txt
# One fragment per line. Lines starting with # are comments.
# EXIT: lines record an intended status change.
+error: empty --config is not a valid document
-Traceback (most recent call last):
EXIT: 1 -> 2

The checker below is a proposal for local use, not a harvested suite from production CI. It treats uncovered unified-diff lines as a failed packet, which is stricter than a glance at git diff.

#!/usr/bin/env python3
"""Fail if oracle/observed.diff contains lines not covered by allowlist.txt."""
from pathlib import Path
import sys

root = Path(__file__).resolve().parent
observed = (root / "observed.diff").read_text(encoding="utf-8", errors="replace")
allow = []
exit_rule = None
for raw_line in (root / "allowlist.txt").read_text(encoding="utf-8").splitlines():
    line = raw_line.strip()
    if not line or line.startswith("#"):
        continue
    if line.startswith("EXIT:"):
        exit_rule = line
        continue
    allow.append(line)

uncovered = []
for raw in observed.splitlines():
    if not raw.startswith(("+", "-")) or raw.startswith(("+++", "---")):
        continue
    text = raw[1:]
    if not any(pat in text for pat in allow):
        uncovered.append(raw)

before_exit = (root / "before.exit").read_text(encoding="utf-8").strip()
after_exit = (root / "after.exit").read_text(encoding="utf-8").strip()
if exit_rule:
    expected = exit_rule.split(":", 1)[1].strip()  # e.g. "1 -> 2"
    actual = f"{before_exit} -> {after_exit}"
    if expected != actual:
        print(f"exit allowlist expected {expected!r}, got {actual!r}")
        sys.exit(1)
elif before_exit != after_exit:
    print("exit code changed but allowlist.txt has no EXIT: rule")
    sys.exit(1)

if uncovered:
    print("oracle allowlist missed these diff lines:")
    print("\n".join(uncovered))
    sys.exit(1)
print("oracle allowlist covered the observed diff")

Wrap the checker so a maintainer runs one script after cloning the branch.

#!/usr/bin/env bash
# oracle/check_oracle.sh
set -euo pipefail
cd "$(dirname "$0")"
test -s before.txt
test -s after.txt
python3 check_allowlist.py

Adjust the exit-code rule to the issue rather than copying the sample. Some fixes must keep the same status and only change a message, and the allowlist should say so in words the checker can enforce.

Step 6: review the packet against the git hunks

Once the files exist, a second reader can compare three artifacts that humans often keep in separate windows. The first is observed.diff. The second is allowlist.txt. The third is git diff for the actual source change. Contradictions among those three are the review, not whether the patch looks tidy in isolation.

A narrow prompt keeps that second reader on the packet. The block below is a template, not a transcript of a run performed for this article.

You are reviewing an OSS bugfix packet, not writing new code.
Inputs: issue excerpt, git diff, oracle/before.txt, oracle/after.txt,
oracle/allowlist.txt, oracle/observed.diff, oracle/invoke.sh.
Report only:
1. Allowlist claims that do not appear in observed.diff
2. Observed.diff lines that the allowlist does not mention
3. Git hunks that cannot be tied to the declared delta
4. Steps in invoke.sh that look non-deterministic (time, net, random)
Do not suggest extra features. Do not rewrite the patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access can run that constrained pass over the packet, and the free server option can replay invoke.sh plus check_oracle.sh on a machine that is not already loaded with the contributor's daemons. The packet remains the useful artifact if both of those options are skipped.

Local caches, language shims, and half-installed extras are a frequent source of unreproducible fixes that only exist on one laptop. A clean clone that reproduces before.txt and then after.txt is part of the proof. If a shared server is used for that clone, scrub host paths, tokens, and customer fixtures from the oracle files before they leave the machine.

Decision table for the packet

Signal in the packet	Meaning	Action before opening a PR
`observed.diff` is empty	The patch did not move the failing behavior	Keep debugging; do not request review
Diff is large, allowlist is short	Hidden refactors or noisy logs	Split the commit or quiet the invocation
Allowlist mentions lines absent from the diff	The declaration is aspirational	Rewrite the allowlist against real bytes
Exit code unchanged when the issue needs a status change	Incomplete fix	Add an `EXIT:` rule and recapture
Invocation embeds timestamps or absolute home paths	Replay will rot on another host	Fix `invoke.sh` before any model pass

Limitations

This workflow assumes a deterministic invocation. Flaky tests, live network calls, and clocks inside log lines will poison before.txt and after.txt on the second run. Binary fixtures and huge snapshot dumps also make the packet hard to review, so a focused assertion beats megabytes of pretty-printed JSON.

The allowlist checker is string oriented on purpose. It can miss semantic changes that keep the same text, and it can flag harmless path reordering that still confuses a reviewer. Contributors still need a human pass on the git hunks, especially around public API and error-code stability. A model will not see uncommitted files, private credentials in environment dumps, or project policy on breaking changes.

Do not paste secrets into a shared server or a model prompt. Oracle files often contain host paths, tokens from failed auth, or fixtures that should never be public. If the bug only reproduces with proprietary data, a public before-and-after packet is the wrong shape for the proof.

Who should skip this approach

Maintainers of one-line typo fixes do not need an oracle directory sitting in the review. Security patches that should not advertise exploit details in before and after logs should use a private disclosure channel instead of this packet. Contributors without a failing invocation, including design debates and API proposals, should write a decision record rather than invent a fake output pair.

Teams that already have a golden-test harness covering the same invocation can point the pull request at that harness and skip the extra files. The principle remains even then: declare the delta, prove it on both sides of the patch, and keep the proof next to the diff. Chat summaries disappear; replayable bytes do not.