Run a Fail-Then-Pass Replay Before an OSS Fix

An OSS fix without a replayable failure remains a guess.
Reviewers cannot verify a story that never failed locally.

This workflow pins one issue to one command.
That command must fail before any production edit.
The same command must pass after the patch.

The artifact is a fail-then-pass replay gate.
The gate stores needles from real test logs.
It refuses a patch when either needle is missing.

Replay beats a narrative

Issue threads often describe symptoms more than commands.
A contributor then ships a purely speculative diff.
The maintainer reruns folklore instead of a gate.

A fail-then-pass replay removes most of that folklore.
The failing log becomes the local contract.
The passing log remains the only proof.

The method fits libraries with already deterministic tests.
It also fits command-line tools with golden output.
It does not fit flaky network-only incidents.

Declare the gate

Keep the gate file beside the patch notes.
Use boring machine-readable fields for every run.
Treat this file as an example, not policy.

# example only: review/issue-replay.toml
issue_url = "https://github.com/example/lib/issues/1842"
upstream_ref = "v2.4.1"
test_cmd = "python -m pytest tests/test_parse.py::test_trailing_comma_issue_1842 -q"
fail_needle = "AssertionError: trailing comma must not emit Token.COMMA"
pass_needle = "1 passed"
log_dir = "review/logs"

Needles are literal substrings from local runs.
They are not clever regular expressions at all.
The contributor copies them from the first failure.

1. Isolate a worktree

Do not patch the daily working clone.
Add a detached worktree on the issue tag.
Record the resolved SHA beside the captured logs.

# example commands, not a live upstream
git fetch origin tag v2.4.1
git worktree add --detach ../lib-issue-1842 v2.4.1
mkdir -p ../lib-issue-1842/review/logs
git -C ../lib-issue-1842 rev-parse HEAD > ../lib-issue-1842/review/logs/HEAD.sha

The worktree keeps the main tree clean.
The SHA file stops silent tag movement.
Later logs must cite that exact SHA.

Use the version named in the issue, not folklore.
A tag is a stated reproduction target.
It is not a search across random history.

2. Write one regression assertion

The first edit belongs in the test file.
The test name should carry the issue number.
The assertion should encode the requested behavior.

# example sketch, not production code
def test_trailing_comma_issue_1842():
    tokens = parse("item,")
    kinds = [token.kind for token in tokens]
    assert kinds[-1] != "COMMA", (
        "trailing comma must not emit Token.COMMA"
    )

Run that test before touching library code.
It should fail with the same fail needle.
A pass here means the reproduction is wrong.

3. Prove the failure first

Install test extras only in that worktree.
Run the command stored in the gate file.
Capture stdout and stderr as one log.

# example
cd ../lib-issue-1842
python -m venv .venv
. .venv/bin/activate
pip install -e ".[test]"
python -m pytest tests/test_parse.py::test_trailing_comma_issue_1842 -q \
  > review/logs/fail.log 2>&1 || true
grep -F "trailing comma must not emit Token.COMMA" review/logs/fail.log

The captured log must contain the fail needle.
Absence of that needle aborts the whole workflow.
The contributor then tightens the failing reproduction.

A missing failure remains a blocked local state.
No production file should change before failure.
The test selection is the first bug to fix.

4. Patch the production surface

Change the smallest function that feeds the test.
Leave formatting and comments for a later commit.
Keep the patch readable in ordinary diff output.

# example sketch, not production code
def parse(source: str) -> list[Token]:
    raw = tokenize(source)
    if raw and raw[-1].kind == "COMMA":
        raw = raw[:-1]
    return raw

This parser sketch stays deliberately tiny and local.
Real upstream code will look quite different.
The rule is still one behavior per patch.

5. Require the pass log

Re-run the identical command after the production edit.
Save a sibling pass log next to the SHA.
Require the pass needle and forbid the fail needle.

# example
python -m pytest tests/test_parse.py::test_trailing_comma_issue_1842 -q \
  > review/logs/pass.log 2>&1
grep -F "1 passed" review/logs/pass.log
! grep -F "trailing comma must not emit Token.COMMA" review/logs/pass.log

A pass without the old needle is necessary.
It is not sufficient for an upstream merge.
Maintainers still own design and compatibility choices.

6. Automate both gates

Manual log reading still drifts across long sessions.
A small script should enforce both log needles.
The script below is a labeled example harness.

# example: tools/replay_gate.py
from __future__ import annotations

import argparse
import subprocess
import sys
import tomllib
from pathlib import Path


def load_gate(path: Path) -> dict:
    data = tomllib.loads(path.read_text(encoding="utf-8"))
    required = (
        "test_cmd",
        "fail_needle",
        "pass_needle",
        "log_dir",
    )
    missing = [key for key in required if key not in data]
    if missing:
        raise SystemExit(f"missing gate keys: {missing}")
    return data


def run_cmd(cmd: str, log_path: Path) -> str:
    log_path.parent.mkdir(parents=True, exist_ok=True)
    proc = subprocess.run(
        cmd,
        shell=True,
        text=True,
        capture_output=True,
    )
    body = proc.stdout + proc.stderr
    log_path.write_text(body, encoding="utf-8")
    return body


def require(body: str, needle: str, label: str, present: bool) -> None:
    found = needle in body
    if present and not found:
        raise SystemExit(f"{label}: missing needle: {needle!r}")
    if not present and found:
        raise SystemExit(f"{label}: unexpected needle: {needle!r}")


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--gate", default="review/issue-replay.toml")
    parser.add_argument("--phase", choices=("fail", "pass"), required=True)
    args = parser.parse_args()

    gate = load_gate(Path(args.gate))
    log_dir = Path(gate["log_dir"])
    log_path = log_dir / f"{args.phase}.log"
    body = run_cmd(gate["test_cmd"], log_path)

    if args.phase == "fail":
        require(body, gate["fail_needle"], "fail", True)
        require(body, gate["pass_needle"], "fail", False)
    else:
        require(body, gate["pass_needle"], "pass", True)
        require(body, gate["fail_needle"], "pass", False)

    sha = Path("review/logs/HEAD.sha")
    if not sha.exists():
        raise SystemExit("missing review/logs/HEAD.sha")
    print(f"{args.phase} gate ok sha={sha.read_text().strip()}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The example uses shell mode only for local brevity.
A stricter harness should pass an argument list.
The harness ignores process exit codes on purpose.

Needles decide the gate, not pytest status.
A green pytest run can still miss the pass needle.
A red pytest run can still miss the fail needle.

# example sequence, run inside the worktree
python tools/replay_gate.py --phase fail
# edit tests, then library code
python tools/replay_gate.py --phase pass
git add tests/test_parse.py src/lib/parse.py review/issue-replay.toml

Run the fail phase before any production edits.
Run the pass phase after those production edits.
Commit the gate file with the patch notes.

The harness does not upload any local files.
It does not call a language model.
It only compares local logs to declared needles.

7. Second reader on the log delta

A local replay can still miss weak assertions.
A second reader can inspect the log delta only.
The source tree itself should stay off that path.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode offers free model access and a free server option.
Those availability notes come from the account operator.
They are not measured quotas, hardware claims, or benchmarks.

The prompt should receive three artifacts only.
Send the gate file, the fail log, and the pass log.
Do not send secrets, credentials, or vendor dumps.

# example review prompt, unexecuted
Read issue-replay.toml, fail.log, and pass.log.
List assertions that still look too weak.
List log lines that changed for the wrong reason.
Do not rewrite the patch.
Do not invent files that are not in the logs.

The model output is a checklist, not a merge vote.
The contributor still runs the local replay harness.
The maintainer still owns the final merge.

Contributors who already hold both logs can continue.
They can try that free model path on the delta alone.

Limitations

The gate trusts deterministic tests above all else.
Flaky suites will flap both declared needles.
Pin time, locale, and working directory first.

The gate cannot replay GUI clicks at all.
It cannot replay production-only race conditions.
Those bugs need other capture and tracing tools.

Needle matching is literal and somewhat brittle.
A wording change can fake a passing gate.
Re-copy needles after each test message edit.

Model review of logs stays inherently shallow.
It cannot see unlogged process state at all.
It cannot replace a human maintainer on the merge.

Free model access may change without prior notice.
A free server option is not a lab promise.
Do not build release process around either one.

Who should skip this workflow

Skip it when the project has no automated tests.
Skip it when the bug is a leaked secret.
Skip it when the patch is a docs typo.

Binary-only vendors cannot run this local harness.
Emergency hotfixes may not wait for extra worktrees.
Those cases need a different local discipline.

The core rule stays small and local.
Fail first on the issue's stated tag.
Pass later on the same recorded command.

A patch that cannot replay both states remains a guess.