DEV Community

Taylor Wang
Taylor Wang

Posted on

Build an OSS Patch Allowlist From One Failing Test

#opensource #git #testing #python

An OSS patch is ready only after one named test flips. That named failing test is the merge oracle. The issue thread is only background noise here.

This workflow turns one test into a file allowlist. Coverage names every path the test actually executes. Files outside that map stay out of the commit.

Why issue threads fail as oracles

Bug reports mix symptoms, guesses, and local environment detail. Contributors then edit the modules that look related. The resulting pull request often touches idle code.

CI can still pass after that extra contact. Distant callers then break on the next release. A coverage allowlist stops that drift before review starts.

Artifact overview

The artifact is a small local review gate. It records one failing test and writes a coverage allowlist. It then rejects extra git paths before commit.

The Python snippets below are an unexecuted proposal. Projects should swap in their real test runner. Do not add a new framework for a single bug.

Preconditions

Match the issue against the default remote branch. The tree must be clean before coverage runs. A dirty worktree will poison the coverage allowlist.

git fetch origin
git switch --detach origin/main
test -z "$(git status --porcelain)"
Enter fullscreen mode Exit fullscreen mode

The last command must print nothing at all. Stash or commit leftover files before continuing.

The project must already have a test command. This method does not install a new stack.

Step 1. Encode the bug as one test

Name the test after the issue number. Keep exactly one hard assertion in the test. Avoid snapshots of an entire production module.

# tests/test_issue_8841_retry_budget.py
from pkg.client import RetryBudget


def test_issue_8841_retry_budget_stops_after_two_failures():
    budget = RetryBudget(max_attempts=2)
    budget.record_failure()
    budget.record_failure()
    assert budget.can_retry() is False
Enter fullscreen mode Exit fullscreen mode

Run only that test file for now. Confirm the failure matches the original report.

pytest -q tests/test_issue_8841_retry_budget.py -vv --tb=short
Enter fullscreen mode Exit fullscreen mode

A passing result means the oracle is wrong. Rewrite the assertion before any production code edit.

Step 2. Collect coverage from that test only

Full-suite coverage is too wide for a patch allowlist. Collect data from the named test alone.

pytest tests/test_issue_8841_retry_budget.py \
  --cov=pkg \
  --cov-report=json:issue_8841_cov.json \
  --cov-report=term-missing
Enter fullscreen mode Exit fullscreen mode

Turn covered files into a plain allowlist. Print that list before any edit starts.

# tools/cov_allowlist.py  (proposal)
import json
from pathlib import Path

data = json.loads(Path("issue_8841_cov.json").read_text())
files = sorted(
    path
    for path, row in data.get("files", {}).items()
    if row.get("summary", {}).get("covered_lines", 0) > 0
)
out = Path("issue_8841_allow.txt")
out.write_text("\n".join(files) + "\n")
print("\n".join(files))
Enter fullscreen mode Exit fullscreen mode 
python tools/cov_allowlist.py
Enter fullscreen mode Exit fullscreen mode

Add the new test path to the allowlist by hand. Production files must come only from coverage output.

Step 3. Reject git paths outside the allowlist

Do not stage any changed files yet. Diff the working tree against HEAD first. Fail the gate when extra paths appear.

# tools/enforce_allowlist.py  (proposal)
from pathlib import Path
import subprocess
import sys

allow = {
    line.strip()
    for line in Path("issue_8841_allow.txt").read_text().splitlines()
    if line.strip()
}
allow.add("tests/test_issue_8841_retry_budget.py")
raw = subprocess.check_output(
    ["git", "diff", "--name-only", "HEAD"],
    text=True,
)
changed = {line for line in raw.splitlines() if line}
extra = sorted(changed - allow)
if extra:
    sys.stderr.write("paths outside coverage allowlist:\n")
    sys.stderr.write("\n".join(extra) + "\n")
    raise SystemExit(1)
print("allowlist ok:", ", ".join(sorted(changed)))
Enter fullscreen mode Exit fullscreen mode 
python tools/enforce_allowlist.py
Enter fullscreen mode Exit fullscreen mode

A nonzero exit means the patch is too wide. Shrink the diff until the gate passes. Do not grow the allowlist to match it.

Step 4. Edit only executed files

Change production code after the gate exists. Stay strictly inside the coverage allowlist paths. Re-run the named test after every edit.

pytest -q tests/test_issue_8841_retry_budget.py --tb=short
Enter fullscreen mode Exit fullscreen mode

The test must keep failing until the real fix lands. A stub that forces a pass is a false oracle.

Then run tests that import the same module. Package regressions hide beside a green single test.

pytest -q pkg/client tests/test_issue_8841_retry_budget.py
Enter fullscreen mode Exit fullscreen mode

Public function signatures stay frozen unless a new test says otherwise. Extra helpers belong in a later separate change.

Step 5. Write a replay log for reviewers

Reviewers should not reconstruct the local ritual. Dump the test name, HEAD, allowlist, and output. Attach that file to the pull request.

{
  echo "TEST=tests/test_issue_8841_retry_budget.py"
  echo "HEAD=$(git rev-parse HEAD)"
  echo "--- ALLOWLIST ---"
  cat issue_8841_allow.txt
  echo "--- DIFFSTAT ---"
  git diff --stat
  echo "--- TEST ---"
  pytest -q tests/test_issue_8841_retry_budget.py --tb=short
} > issue_8841_replay.txt
Enter fullscreen mode Exit fullscreen mode

The replay log is the primary review surface. The raw diff is supporting evidence only here.

Maintainer checks on the pull request

The maintainer re-runs one command from the replay log. The named test must fail on origin/main. It must pass on the proposed contributor branch.

git switch --detach origin/main
pytest -q tests/test_issue_8841_retry_budget.py --tb=short
# expect fail
git switch --detach FETCH_HEAD
pytest -q tests/test_issue_8841_retry_budget.py --tb=short
# expect pass
python tools/enforce_allowlist.py
Enter fullscreen mode Exit fullscreen mode

A test that never failed on main is not an oracle. Reject that pull request without any further review.

Decision table

Use this table before the pull request opens. Each missed row should block the merge.

Signal Action
Named test never failed Rewrite the test. Do not patch.
Coverage JSON lists no files The test never imported production code.
Diff includes files off the map Split the work or add another named test.
Single test passes, package tests fail Keep the patch. Fix the regression first.
Public API shape changed Add a dedicated compatibility test.
Only comments or lockfiles changed Stop. The oracle did not move.
Allowlist contains generated files Exclude those paths. Do not hand-edit them.

Each table row is a hard stop condition. Green CI without a named oracle is not enough.

Optional second pass on the replay log

A language model can read the replay log and the allowlisted diff. It still cannot replace the failing named test. Treat those notes as extra maintainer questions only.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two availability facts are the only product claims in this article.

A contributor without a local GPU can run the replay-log review there. Keep the model prompt inside the allowlist. Send the log, the diff, and the test output only.

Review only files listed in issue_8841_allow.txt.
Ignore every path outside that list.
The oracle is tests/test_issue_8841_retry_budget.py.
List contradictions between the diff and the replay log.
Do not suggest new modules.
Do not delete or weaken the failing test.
Enter fullscreen mode Exit fullscreen mode

Discard any note that widens the change set. Discard any note that removes the oracle. Humans still own the final merge decision.

Limits

Coverage is not a proof of full correctness. It only records lines the named test executed. Untested branches in the same file can still fail.

The coverage allowlist remains a tool-specific artifact. coverage.py maps Python poorly onto C extension code. Go, Rust, and Java need their own cover commands.

Cross-cutting refactors do not fit this gate. Wide migrations need a different review contract. Do not force one test onto a wide rename.

Flaky tests also break this merge oracle. Pin time, network, and random seeds first. A flake is not a bug reproduction.

Who should not use this

Authors of large public API redesigns should skip this method. Contributors without any runnable harness should skip it. People editing generated or vendored trees should skip it.

Security fixes that must touch many files need another process. A coverage allowlist can hide required extra paths. Follow the project security advisory flow instead.

Closing

The named failing test is the merge oracle. Coverage from that test is the file allowlist. The replay log is what reviewers actually read.

A model may comment on that replay log. That model does not get a merge vote. The test still has to fail, then pass, on the contributor machine.

Top comments (0)