DEV Community

Taylor Wang
Taylor Wang

Posted on

Score Incoming OSS PRs Against the Issue Contract

#git #opensource #testing #ai

Incoming OSS patches fail when they ignore the issue. Maintainers then debate the story instead of the contract. Extract a review contract from the issue first.

This workflow treats the GitHub issue as the spec. The pull request must satisfy that written spec only. Extra files, extra refactors, and extra APIs fail review.

The artifact below is a small maintainer loop. It freezes the issue, bounds the diff, and scores patches. A free model can draft the first-pass notes.

Why the issue beats the PR body

Pull request descriptions usually sell intent, not failure. GitHub issues still record the original broken behavior. Reviewers need that failure more than the pitch.

Many patches add logging, rename helpers, and retouch docs. Those extra edits hide the actual one-line fix. A written contract makes that extra work visible.

The maintainer loop

Follow five numbered gates before any merge discussion. Stop at the first failing gate without exception. Do not merge a patch on narrative charm.

1. Freeze the issue text

Pin the issue number before reading any patch diff. Save the title, body, and current labels together. Ignore later comment drift until that contract exists.

ISSUE=1842
mkdir -p review/1842
gh issue view "$ISSUE" --json number,title,labels,body \
  > review/1842/issue.json
Enter fullscreen mode Exit fullscreen mode

Keep that JSON beside later review notes on disk. The frozen file is the only accepted spec.

2. Extract a review contract

Convert the frozen issue into a short YAML contract. Keep every field small, observable, and testable by command. Reject vague goals such as improved robustness claims.

issue: 1842
must_fail_before:
  - command: "python -m pytest tests/test_cache.py::test_stale_etag"
    exit_code: 1
must_pass_after:
  - command: "python -m pytest tests/test_cache.py"
    exit_code: 0
allowed_paths:
  - "src/cache.py"
  - "tests/test_cache.py"
forbidden_changes:
  - public API signatures
  - dependency pins
  - formatter-only diffs
out_of_scope:
  - performance rewrites
  - new CLI flags
Enter fullscreen mode Exit fullscreen mode

Label this YAML as a proposal without a repro. Do not invent failing tests for missing fixtures. Request a reporter fixture before any model sees code.

3. Bound the incoming diff

Fetch the pull request as a local review branch. Measure path scope against the YAML allowed list. Reject unbounded file lists before reading any hunks.

PR=991
gh pr checkout "$PR"
git fetch origin main
git diff --name-only origin/main...HEAD > review/1842/pr.files
git diff --stat origin/main...HEAD | tee review/1842/pr.stat
Enter fullscreen mode Exit fullscreen mode

Compare pr.files with allowed_paths from the contract. Extra paths mean scope creep against the issue. Scope creep needs a new issue, not a merge.

A tiny Python helper makes the check boring. Run it as a gate, not as taste.

# scope_check.py — labeled proposal, unexecuted example
from pathlib import Path
import sys
import yaml

contract = yaml.safe_load(Path(sys.argv[1]).read_text())
files = Path(sys.argv[2]).read_text().splitlines()
allowed = set(contract["allowed_paths"])
extra = [f for f in files if f not in allowed]

if extra:
    print("OUT_OF_SCOPE")
    for path in extra:
        print(f"  + {path}")
    raise SystemExit(2)

print("SCOPE_OK")
Enter fullscreen mode Exit fullscreen mode 
python scope_check.py review/1842/contract.yaml review/1842/pr.files
Enter fullscreen mode Exit fullscreen mode

Exit code two blocks review of the hunks. Maintainers then ask for a split pull request.

4. First-pass review with a free model

Human reviewers still own the final merge decision. A first-pass model only scores the YAML contract. Keep the prompt glued to those contract fields.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two availability facts fit this maintainer loop. The scoring job can leave the reviewer's laptop. No local GPU is required for text.

Do not treat the model output as a maintainer vote. Feed it the frozen issue, contract, and bounded diff. Ask only for contract violations with hunk evidence.

Score this pull request against the YAML contract.
Return only these items in order:
- PASS or FAIL for each contract field
- the supporting hunk evidence
- one merge word: merge, request-changes, needs-repro
Do not suggest extra features.
Do not rewrite the patch.
Enter fullscreen mode Exit fullscreen mode

Save the model output into review/1842/score.md right away. Maintainers edit that file before any public comment. The model draft is never the final verdict.

A pack script keeps the three inputs together.

# review_pack.sh — labeled proposal
set -euo pipefail
ROOT="review/${1}"
git diff origin/main...HEAD > "${ROOT}/pr.diff"
{
  echo "===== ISSUE ====="
  cat "${ROOT}/issue.json"
  echo "===== CONTRACT ====="
  cat "${ROOT}/contract.yaml"
  echo "===== DIFF ====="
  cat "${ROOT}/pr.diff"
} > "${ROOT}/pack.txt"
wc -l "${ROOT}/pack.txt"
Enter fullscreen mode Exit fullscreen mode

Send pack.txt to the free model on the free server. Strip tokens, emails, and customer dumps before packing. Secret leakage is a review failure, not a model failure.

5. Apply the decision table

Use this table and refuse a fourth outcome. Incomplete evidence means needs-repro, not a hopeful merge.

| Gate | Result | Action |
| freeze issue | missing repro command | needs-repro |
| scope_check | extra paths | request-changes |
| must_fail_before | already passing | needs-repro |
| must_pass_after | still failing | request-changes |
| forbidden_changes | FAIL | request-changes |
| all gates | PASS | human merge review |

The must_fail_before gate protects against theatrical green tests. A patch that never failed the reported test is theater. The must_pass_after gate is the only test green light.

# contract_run.py — labeled proposal, unexecuted example
import subprocess
import sys
import yaml
from pathlib import Path

def run(cmd: str) -> int:
    proc = subprocess.run(cmd, shell=True)
    return int(proc.returncode)

def main(path: str, phase: str) -> None:
    data = yaml.safe_load(Path(path).read_text())
    key = "must_fail_before" if phase == "before" else "must_pass_after"
    failed = False
    for item in data[key]:
        code = run(item["command"])
        expect = item["exit_code"]
        print(f"{phase} code={code} expect={expect} {item['command']}")
        if code != expect:
            failed = True
    raise SystemExit(1 if failed else 0)

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])
Enter fullscreen mode Exit fullscreen mode 
# labeled proposal; run inside a clean clone
python contract_run.py review/1842/contract.yaml before
git stash push -u -m "incoming-pr"
python contract_run.py review/1842/contract.yaml before
git stash pop
python contract_run.py review/1842/contract.yaml after
Enter fullscreen mode Exit fullscreen mode

Record both exit codes inside the review directory. Missing numbers mean the YAML contract is incomplete. Incomplete contracts block merge under this review loop.

Synthetic walkthrough

This walkthrough is synthetic and not a live benchmark. Issue 1842 claims stale ETag reuse after 304. The contract allows src/cache.py and one test file.

Pull request 991 also rewrites src/http.py today. It adds a retry flag outside the issue. The scope_check script therefore exits with code two.

The decision table returns request-changes without reading style. The model notes the extra flag against out_of_scope. The maintainer asks the author to split work.

That outcome is the entire point here. The contract stopped a drive-by rewrite early. The original issue stayed small enough to review.

Secondary checks after scope passes

Scope success does not mean the patch is honest. Read the remaining hunks against forbidden_changes with git.

git diff origin/main...HEAD -- src/cache.py
git log --oneline origin/main...HEAD
git diff origin/main...HEAD | git apply --check
Enter fullscreen mode Exit fullscreen mode

Watch for signature edits on public functions in the hunk. Watch for lockfile churn without an issue sentence. Watch for formatter noise across otherwise untouched modules.

Add a blame note when the patch touches recent lines. Recent lines often belong to another open issue. Cross-issue edits belong in a second pull request.

git blame -L 40,80 src/cache.py
Enter fullscreen mode Exit fullscreen mode

Public functions need a grep pass after the blame note. A renamed helper can still break downstream callers. Search the repo before trusting a local green suite.

rg -n "def fetch_with_etag" src tests
rg -n "retry_on_304" src tests
Enter fullscreen mode Exit fullscreen mode

Limitations

This loop fails on open design questions quickly. Public API shape debates still need human maintainers. Contracts cannot encode taste, tone, or project politics.

Generated tests can echo the incoming patch too closely. Prefer reporter fixtures over any model-authored test files. Stop the loop when the issue has no command.

Free models miss domain invariants and license problems. They also miss CLA status and export-control flags. Keep those checks outside the score file always.

Large diffs overwhelm short model context windows fast. Split the pull request instead of summarizing hunks away. Summaries hide the exact lines under review.

The free server option is for text scoring jobs. It is not a substitute for project CI runners. Always execute the real suite on project hardware.

Who should skip this approach

Do not use this loop for security embargo patches. Do not paste private diffs into a shared model. Do not score legal questions with a text model.

New maintainers on huge monorepos should start much smaller. Pick issues that already name one failing test. Grow the allowed_paths list only after repeats.

Teams with issue-linked acceptance tests may skip YAML. Keep their existing fixtures as the contract instead. Add only the path scope check in that case.

What this does not claim

No latency numbers appear anywhere in this article. No model names appear in this article either. No quota, hardware, or uptime figures appear here.

The value is the contract, not any vendor. Remove the model and the gates still function. The model only drafts the first review notes.

Maintainers can run the pack script on the next PR. Use a free model only after secrets are stripped.

Top comments (0)