DEV Community

Taylor Wang
Taylor Wang

Posted on

Make the Failing Test the OSS Merge Token

#git #opensource #testing #ai

Open-source patches stall when nobody can replay the bug. A merge-ready change starts as a failing test, not a diff. Maintainers should reject patches that lack a hermetic reproduction.

The failure mode

Issue threads often describe failure symptoms in loose prose. Contributors then guess a fix from that prose. Reviewers cannot tell whether the bug still exists.

Language models make this worse through assumed context. They treat the issue body as ground truth. They emit a patch before anyone reruns the failure.

A recorded reproduction command closes that exact gap. The reproduction test must fail on current main. The same test must pass on the fix branch.

Scope of this loop

This article describes a reproduce-patch-test contributor loop for OSS. It keeps generated text in the review seat only. It does not let a model author the production patch.

The loop fits small OSS libraries and command-line tools. It expects a local test runner and git history. It does not replace maintainer judgment or project CI.

Artifact: the reproduction record

Store one machine-readable file beside the test suite. Name that file repro.json for this walkthrough. Keep every field tiny, explicit, and easy to grep.

{
  "issue": "https://github.com/example/libparse/issues/1847",
  "claim": "empty quoted field shifts remaining CSV columns",
  "setup": [
    "python3 -m venv .repro",
    ".repro/bin/pip install -e .[test]"
  ],
  "fail_cmd": ".repro/bin/python tests/test_repro_1847.py",
  "expect_fail_on": "main",
  "expect_pass_on": "fix/1847-empty-quoted-field"
}
Enter fullscreen mode Exit fullscreen mode

Treat this schema as a proposal, not a standard. Teams may rename keys to match their stack. The file exists only to bind a claim to a command.

Step 1: Freeze the observable claim

Copy the issue title into one concrete sentence. Strip speculation, stack traces, and any proposed fixes. Record only the observable failure and the public URL.

Write that claim into repro.json before any code change. Stop if the claim requires private customer data. Synthetic fixtures must replace any production dumps here.

Example claim language should stay measurable and narrow. "Parser drops the last column on empty quotes" works. "The parser feels wrong on some files" does not.

Step 2: Add one hermetic failing test

Create one new test file for the issue number. Do not edit existing tests in this step. The new file should fail on current main.

# tests/test_repro_1847.py
# Proposal: isolated reproduction for issue 1847.
from libparse.csv import parse_row


def test_empty_quoted_field_keeps_column_alignment():
    row = parse_row('a,"",c')
    assert row == ["a", "", "c"]
Enter fullscreen mode Exit fullscreen mode

Run the command stored in repro.json against main. Confirm a non-zero process exit code from that run. Append both stdout and stderr into repro.log.

git switch main
git pull --ff-only
python3 -m venv .repro
.repro/bin/pip install -e ".[test]"
.repro/bin/python tests/test_repro_1847.py
echo "exit:$?" | tee -a repro.log
Enter fullscreen mode Exit fullscreen mode

If the test passes on main, the issue is stale. Comment with the passing command and then stop. Do not invent a production patch for a stale issue.

Step 3: Bound the production patch

Create a branch named after the issue number. Change the smallest production file that can flip the test. Keep the new test in the same commit series.

git switch -c fix/1847-empty-quoted-field
# edit src/libparse/csv.py only
.repro/bin/python tests/test_repro_1847.py
git add src/libparse/csv.py tests/test_repro_1847.py repro.json
git diff --cached --stat
Enter fullscreen mode Exit fullscreen mode

Inspect the staged file list before every commit. Extra docs and lockfiles should wait for later. Formatting-only noise does not belong in this patch.

A minimal parser change might look like the next snippet. Treat the snippet as illustrative pseudocode only, not production. Do not drop it into an unrelated codebase.

# src/libparse/csv.py  (illustrative; not a drop-in patch)
def parse_row(line: str) -> list[str]:
    fields: list[str] = []
    buf: list[str] = []
    in_quotes = False
    i = 0
    while i < len(line):
        ch = line[i]
        if ch == '"':
            in_quotes = not in_quotes
            i += 1
            continue
        if ch == "," and not in_quotes:
            fields.append("".join(buf))
            buf = []
            i += 1
            continue
        buf.append(ch)
        i += 1
    fields.append("".join(buf))
    return fields
Enter fullscreen mode Exit fullscreen mode

Re-run the reproduction command on the fix branch. Require a zero exit before touching other tests. Then run the existing suite without extra flags.

.repro/bin/python tests/test_repro_1847.py
.repro/bin/python -m pytest -q
Enter fullscreen mode Exit fullscreen mode

If the suite regresses, revert the production file immediately. Do not widen the patch to hide breakage. A red suite means the claim is still unfinished.

Step 4: Automate the fail and pass gate

Automate the fail-on-main and pass-on-branch check. Place the script under tools/check_repro.sh. Treat the script as a proposal.

#!/usr/bin/env bash
# tools/check_repro.sh
# Proposal: verify fail-on-main and pass-on-branch.
set -euo pipefail
cmd=$(python3 -c 'import json; print(json.load(open("repro.json"))["fail_cmd"])')
git switch --detach main
set +e
$cmd
status=$?
set -e
if [ "$status" -eq 0 ]; then
  echo "unexpected pass on main" >&2
  exit 1
fi
git switch -
set +e
$cmd
status=$?
set -e
if [ "$status" -ne 0 ]; then
  echo "unexpected fail on branch" >&2
  exit 1
fi
echo "repro gate ok"
Enter fullscreen mode Exit fullscreen mode

Run that script only on a clean worktree. It detaches HEAD to compare two refs. Commit the production file before you invoke it.

chmod +x tools/check_repro.sh
./tools/check_repro.sh
Enter fullscreen mode Exit fullscreen mode

The script is not a sandbox for stranger patches. It executes fail_cmd from local JSON. Untrusted commands still need human reading first.

Step 5: Review claim, test, and diff together

Human review still owns the final merge decision. A free-tier model can check one alignment question. That question is whether the diff implements the frozen claim.

Keep the model away from the full working tree. Feed only the claim, the test, and the staged diff. Drop the packet if any of those three parts is missing.

git diff main...HEAD -- src/libparse/csv.py tests/test_repro_1847.py > review.patch
python3 tools/pack_review.py \
  --claim repro.json \
  --test tests/test_repro_1847.py \
  --patch review.patch \
  --out review_packet.md
Enter fullscreen mode Exit fullscreen mode

The packer script only concatenates three local files. It does not call a network API at all. It does not rewrite or tidy the patch.

# tools/pack_review.py
# Proposal: concatenate claim, test, and diff for review.
import argparse
from pathlib import Path


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--claim", required=True)
    parser.add_argument("--test", required=True)
    parser.add_argument("--patch", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    chunks = [
        "# Claim\n",
        Path(args.claim).read_text(encoding="utf-8"),
        "\n# Test\n",
        Path(args.test).read_text(encoding="utf-8"),
        "\n# Diff\n",
        Path(args.patch).read_text(encoding="utf-8"),
    ]
    Path(args.out).write_text("".join(chunks), encoding="utf-8")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Paste review_packet.md into a constrained review prompt. Demand a fixed machine-readable answer shape from it. Reject free-form essays from the model during review.

Review only the claim, test, and diff below.
Do not invent files outside the diff.
Answer with:
1. claim_covered: yes|no
2. test_asserts_claim: yes|no
3. extra_behavior: list
4. merge_advice: reject|ask|accept
Refuse if the packet is incomplete.
Enter fullscreen mode Exit fullscreen mode

Hosted review without expanding scope

Local GPUs are optional for this review step. Hosted free model access can read the bounded packet. A free server option can execute the reproduction command.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Use those only after repro.json and the failing test exist. Do not upload secrets, credentials, or private fixtures.

The workflow does not depend on that product. The same packet works with any review model. The same commands run on a laptop or CI runner.

Teams without spare GPUs can run packet review there. Contributors still must produce the failing test first. Hosted review never replaces the human merge vote.

Decision table

Observation Action
Test passes on main Close the issue or request new evidence
Test fails on main, patch still fails Keep editing the one production file
Test fails on main, patch passes, suite red Shrink the patch and rerun pytest
Packet contains extra unrelated files Reject the review request
Model says accept, human disagrees Human decision wins

Print the table during triage as a short checklist. Do not skip rows because a model sounded confident. Spoken confidence is not a reproduction of the bug.

What to record in the pull request

Paste three evidence blocks into the PR body. Include the frozen claim and the fail-on-main log. Include the pass-on-branch log after those two.

Claim: empty quoted field shifts remaining CSV columns
Fail on main: tests/test_repro_1847.py exit 1
Pass on branch: tests/test_repro_1847.py exit 0
Suite: pytest -q exit 0
Enter fullscreen mode Exit fullscreen mode

Reviewers should rerun fail_cmd before reading the prose. The command logs are evidence, not decoration. Missing logs mean the pull request is incomplete.

Limitations

This loop assumes a unit-testable library surface exists. GUI bugs need a different harness and operator setup. Flaky network tests do not belong in fail_cmd.

Free models miss domain nuance on purpose-built parsers. They rubber-stamp weak tests that assert nothing. They also over-reject safe and local refactors.

The repro.json file is not an access control. It does not sandbox untrusted patches from strangers. It does not prove performance, memory safety, or API stability.

Do not paste proprietary logs into a hosted model. Redact sample rows down to synthetic fixtures only. Prefer public issue links over private attachments always.

Project CI remains the source of merge truth. A laptop pass is necessary but not sufficient. Nightly jobs still catch platform drift after merge.

Who should not use this

Do not use this workflow for security embargoes. Do not use it for coordinated disclosure patches. Do not use it when a fixture would leak user data.

Skip it on drive-by typo pull requests entirely. Skip it when CI already encodes the same claim. Skip it if the project cannot run tests offline.

Large monorepos may find repro.json too local. They already have bisect jobs and target selectors. This loop is for small packages with thin CI.

Closing

The merge token is a flipped failing test. Prose-only issues do not ship by themselves. Model review stays downstream of that failing test.

Record the frozen claim in repro.json first. Fail it on main with one new file. Pass it on the branch, then ask a model to read the packet.

Top comments (0)