Lock OSS Fixes to a Fail-Pass Command Pair

Open-source patches fail when models invent extra scope. A fail-pass command pair freezes the bug before review. Maintainers then judge a contract, not a chat thread.

Issue threads leak patch scope

Public issues mix traces, guesses, and leftover diffs. A review model reads that mix as license. Neighbor files then change without a failing command.

Chat history cannot be replayed on a clean clone. Exit codes can be replayed on a clean clone. A contract stores those codes before any model runs.

The contract is the review surface

The YAML card lives beside the local clone. It names one fail command and one pass command. It also names an invariant command and a hard budget.

The card does not propose a patch. Review models may annotate the signed card only. Those models may not add files or raise limits.

# issue-contract.yml
id: "ISSUE-4419"
origin: "https://github.com/example/libparse/issues/4419"
base_ref: "main"
fail:
  workdir: "."
  cmd: ["python", "-m", "pytest", "tests/test_header.py::test_empty_header", "-q"]
  expect_exit: 1
pass:
  workdir: "."
  cmd: ["python", "-m", "pytest", "tests/test_header.py::test_empty_header", "-q"]
  expect_exit: 0
invariant:
  workdir: "."
  cmd: ["python", "-m", "pytest", "tests/test_header.py", "-q"]
  expect_exit: 0
budget:
  max_files: 3
  max_changed_lines: 80
  deny_globs:
    - "pyproject.toml"
    - "setup.cfg"
    - ".github/**"
    - "docs/**"

This block is a template, not production evidence. Maintainers replace the path with the issue command. Deny globs stay tight against build and docs trees.

Why two expected exits matter

The same test is the cheapest honest oracle. The base tree must still fail that test. The patched tree must pass that same test.

A new test that exists only in the patch is weaker. It never failed on the base hash. The contract then cannot prove the original bug.

Prefer the reporter command when it is deterministic. Write a new failing test only when none exists. That new test must fail on the unpatched hash first.

Sign the card in eight steps

The maintainer follows the eight steps in strict order. Speed is not a reason to skip a step. The model enters only at step five.

1. Pin the reported revision

The contributor clones the project at the issue ref. The clone must avoid a dirty local worktree. The commit hash is recorded beside the card.

git clone https://github.com/example/libparse.git
cd libparse
git checkout --detach origin/main
git rev-parse HEAD > .contract-base.sha

The hash belongs in the review packet. Later diffs must start from that hash. Detached HEAD keeps accidental commits off main.

2. Record the fail command on a clean tree

The contributor runs the command the issue claims fails. Only the exit code is written into the card. Logs stay on disk until the card is signed.

python -m pytest tests/test_header.py::test_empty_header -q
echo $?

The fail block is valid only on a matching exit. A passing main tree means a stale issue. Signing stops when the fail command already passes.

3. Record the invariant command

The contributor picks a nearby suite that must keep passing. The same file without the failing test is enough. The full project suite waits until later sign-off.

python -m pytest tests/test_header.py -q
echo $?

The invariant must pass on the base hash. A failing invariant means a dirty tree. The contract cannot be signed in that state.

4. Freeze the budget before any edit

The contributor counts files the stack actually names. max_files becomes that count plus one. max_changed_lines comes from the smallest plausible fix.

A review model does not pick these numbers. A human maintainer rereads the original issue first. Budget limits rise only after that reread.

5. Draft a patch against the card

A model may see the contract and a short stack. It may not receive a full tree listing. The only accepted output is a unified diff.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those options can host the runner and a bounded diff review. They do not replace the fail-pass pair or set the budget.

The prompt stays mechanical and short. The YAML and git diff --stat are the only inputs. Prose that proposes files outside the card is rejected.

6. Apply the diff on a second worktree

A second worktree keeps the base tree clean. git apply --check runs before a real apply. A check failure ends the run at once.

git worktree add ../libparse-fix "$(cat .contract-base.sha)"
cd ../libparse-fix
git apply --check /tmp/issue-4419.diff
git apply /tmp/issue-4419.diff

Maintainers do not hand-edit a rejected diff. A new diff must match the signed card. The extra worktree must stay disposable after review.

7. Run fail, pass, and invariant together

The Python runner below executes the signed card. It also enforces the budget against git diff. The command runs only inside the disposable worktree.

#!/usr/bin/env python3
"""Run an OSS issue contract against a unified diff.

This script is a template. It is not production CI.
"""
from __future__ import annotations

import argparse
import fnmatch
import subprocess
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml\n")
    sys.exit(2)


def run_cmd(cmd, workdir):
    proc = subprocess.run(
        cmd,
        cwd=workdir,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
    )
    return proc.returncode, proc.stdout


def git(args, workdir):
    code, out = run_cmd(["git", *args], workdir)
    if code != 0:
        raise SystemExit(f"git {' '.join(args)} failed:\n{out}")
    return out.strip()


def load_card(path):
    data = yaml.safe_load(Path(path).read_text())
    for key in ("fail", "pass", "invariant", "budget"):
        if key not in data:
            raise SystemExit(f"card missing {key}")
    return data


def numstat(workdir):
    out = git(["diff", "--numstat", "--", "."], workdir)
    files = []
    total = 0
    for line in out.splitlines():
        if not line.strip():
            continue
        added, deleted, name = line.split("\t", 2)
        if added == "-" or deleted == "-":
            raise SystemExit(f"binary file in diff: {name}")
        total += int(added) + int(deleted)
        files.append(name)
    return files, total


def glob_hit(path, patterns):
    return any(fnmatch.fnmatch(path, pat) for pat in patterns)


def check_step(label, code, expected):
    if code != expected:
        raise SystemExit(f"{label}: exit {code}, expected {expected}")
    print(f"{label}: exit {code} (ok)")


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--card", required=True)
    parser.add_argument("--diff", required=True)
    parser.add_argument("--workdir", default=".")
    args = parser.parse_args()
    card = load_card(args.card)
    root = args.workdir

    base = git(["rev-parse", "HEAD"], root)
    print(f"base: {base}")

    code, _ = run_cmd(card["fail"]["cmd"], card["fail"].get("workdir", root))
    check_step("fail on base", code, card["fail"]["expect_exit"])

    code, _ = run_cmd(
        card["invariant"]["cmd"],
        card["invariant"].get("workdir", root),
    )
    check_step("invariant on base", code, card["invariant"]["expect_exit"])

    diff_path = str(Path(args.diff).resolve())
    git(["apply", "--check", diff_path], root)
    git(["apply", diff_path], root)
    print("apply: ok")

    files, lines = numstat(root)
    budget = card["budget"]
    if len(files) > budget["max_files"]:
        raise SystemExit(
            f"files changed: {len(files)} > {budget['max_files']}"
        )
    print(f"files changed: {len(files)} (ok)")
    if lines > budget["max_changed_lines"]:
        raise SystemExit(
            f"changed lines: {lines} > {budget['max_changed_lines']}"
        )
    print(f"changed lines: {lines} (ok)")

    deny = budget.get("deny_globs", [])
    hits = [name for name in files if glob_hit(name, deny)]
    if hits:
        raise SystemExit(f"deny glob hits: {hits}")
    print("deny glob hits: 0 (ok)")

    code, _ = run_cmd(card["pass"]["cmd"], card["pass"].get("workdir", root))
    check_step("pass on patch", code, card["pass"]["expect_exit"])

    code, _ = run_cmd(
        card["invariant"]["cmd"],
        card["invariant"].get("workdir", root),
    )
    check_step("invariant on patch", code, card["invariant"]["expect_exit"])
    print("GATE: PASS")


if __name__ == "__main__":
    main()

Install pyyaml as the extra dependency first. Then run the gate from the disposable worktree.

pip install pyyaml
python contract_gate.py --card issue-contract.yml --diff /tmp/issue-4419.diff

8. Accept a green gate plus a human note

A green gate is necessary and still not sufficient. A maintainer still reads the unified diff. The merge note must cite the contract id.

Merge proceeds only when all three commands match. The issue closes with the hash and card path. The disposable worktree is dropped after the merge.

What the runner treats as truth

The runner treats exit codes as the only oracle. Stdout is stored for humans only. Stdout never scores the gate.

Budget math uses git diff --numstat on the worktree. Binary files count as an immediate miss. Denied globs fail the gate for a single line.

The fail command runs on the base hash. The pass command runs on the patched tree. The invariant command runs on both trees.

That last rule catches bad cards early. An invariant that fails on main is invalid. An invariant that fails after the patch is a regression.

Decision table before signing

Observation on the base hash	Action
fail matches, invariant passes	Sign the card and continue
fail command already passes	Stop, the issue is stale
invariant already fails	Stop, the tree is dirty
fail command is not deterministic	Stop, do not sign

Sign only the first row. The other rows need human triage. Models do not override this table.

Sample session

$ python contract_gate.py --card issue-contract.yml --diff /tmp/issue-4419.diff
base: 9f3c1a2
fail on base: exit 1 (ok)
invariant on base: exit 0 (ok)
apply: ok
files changed: 2 (ok)
changed lines: 37 (ok)
deny glob hits: 0 (ok)
pass on patch: exit 0 (ok)
invariant on patch: exit 0 (ok)
GATE: PASS

A failed session prints the first broken rule. Later rules can stay unrun. The contributor then edits the diff, not the card.

Limitations

The contract cannot encode timing bugs well. Flaky tests poison the fail block. Snapshot tests can pass for the wrong reason.

Multi-package monorepos need one card per package. A single budget across packages is too coarse. Generated files will blow the line cap fast.

The runner trusts every command in the YAML. A hostile card can run arbitrary processes. Maintainers must read the card before execution.

Free model access does not prove behavioral correctness. A free server does not replace project CI. Environment-specific failures can still slip through.

Who should skip this gate

This gate does not serve design debates. It does not serve refactors with no failing command. It does not serve issues without a local reproduction.

Security patches that must not run locally need another path. Docs-only changes need a docs checklist instead. Brand new clones without tests gain little here.

Keep the card with the pull request

Paste the YAML into the pull request body. Link the runner command in the same note. Reviewers then replay the gate without the original chat.

Scope drift is a process bug, not a model bug. A fail-pass pair is a small process fix. Sign the card first and let models comment later.

A free model review on a throwaway clone can try the gate. Keep the card in the pull request so reviewers can replay it. Do not paste the full issue thread into the prompt.