DEV Community

Taylor Wang
Taylor Wang

Posted on

Review OSS Patches Against an Evidence Packet, Not the Diff Alone

#opensource #git #testing #ai

OSS review fails when a patch arrives without portable evidence. Maintainers then rebuild the bug, the test, and the scope. An evidence packet restores that missing contract before review starts.

This workflow treats the packet as the review object. The diff is only an attachment. Local proof still happens before any remote scoring.

Why bare diffs stall maintainers

A pull request with only code hides the reproduction path. Reviewers hunt comments for commands that may not exist. Hours then go into rebuilding a one-line failure.

Issue threads mix measured facts with speculative refactors. Contributors often patch the speculation instead. Maintainers reject the extra files and the extra story.

Model-authored diffs make that stall more common. The generated change can look tidy while omitting proof. Reviewers still need a fail command, an oracle, and a file budget.

A typed packet stops that drift. It stores those three facts in one file. Reviewers read the file before they read the hunks.

Packet fields that actually matter

A useful packet stays small and machine-checkable. It does not dump the repository. It binds one issue to one oracle.

Keep these fields and drop decorative prose:

  1. Canonical issue id and URL.
  2. Environment hash from a lockfile.
  3. Fail command that is red on main.
  4. Oracle command that is green after the patch.
  5. File allowlist with a numeric cap.
  6. Explicit non-goals for extra refactors.
  7. Model role locked to completeness scoring.

The packet file lives at the repository root. Name it EVIDENCE.yml for this proposal. Continuous integration can fail when the file is invalid.

Artifact: YAML packet for a CSV BOM bug

The next file is a proposal. It is not a published standard. Replace the hash after a local run.

# EVIDENCE.yml — proposal for one OSS bug
schema_version: 1
issue:
  id: "GH-1842"
  url: "https://github.com/example/libparse/issues/1842"
  summary: "CSV header row is dropped when the file has a BOM"
env:
  kind: "lockfile"
  hash_command: "sha256sum poetry.lock"
  recorded_hash: "REPLACE_WITH_LOCAL_SHA256"
repro:
  setup: "poetry install --sync"
  fail_cmd: "poetry run pytest tests/test_csv_bom.py::test_header_kept -q"
  fail_pattern: "AssertionError|FAILED"
oracle:
  pass_cmd: "poetry run pytest tests/test_csv_bom.py -q"
  pass_pattern: "passed"
patch_budget:
  max_files: 3
  max_hunks: 8
  allowlist:
    - "src/libparse/csv.py"
    - "tests/test_csv_bom.py"
    - "docs/csv.md"
non_goals:
  - "rewrite the parser"
  - "reformat unrelated modules"
model_review:
  role: "score_packet_only"
  forbidden: "propose_unrelated_files"
Enter fullscreen mode Exit fullscreen mode

The packet travels with the pull request. Reviewers open it before the diff. Later scoring tools open it before they comment.

Artifact: a local validator

Incomplete packets must not reach reviewers. The script below is a proposal. Run it on a local checkout before push.

# validate_evidence.py — proposal, unexecuted in this article
from __future__ import annotations

import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write("Install pyyaml before running this validator.\n")
    sys.exit(2)

REQUIRED = (
    "schema_version",
    "issue",
    "env",
    "repro",
    "oracle",
    "patch_budget",
    "non_goals",
    "model_review",
)

KNOWN_RUNNERS = ("pytest", "cargo test", "go test", "npm test", "mvn test")


def load_packet(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    if not isinstance(data, dict):
        raise ValueError("packet must be a mapping")
    return data


def check_required(data: dict) -> list[str]:
    errors = []
    for key in REQUIRED:
        if key not in data:
            errors.append(f"missing key: {key}")
    issue = data.get("issue") or {}
    for key in ("id", "url", "summary"):
        if not issue.get(key):
            errors.append(f"issue.{key} is empty")
    return errors


def check_budget(data: dict) -> list[str]:
    errors = []
    budget = data.get("patch_budget") or {}
    allow = budget.get("allowlist") or []
    max_files = budget.get("max_files")
    if not allow:
        errors.append("allowlist is empty")
    if isinstance(max_files, int) and len(allow) > max_files:
        errors.append("allowlist exceeds max_files")
    return errors


def check_commands(data: dict) -> list[str]:
    errors = []
    for section, key in (("repro", "fail_cmd"), ("oracle", "pass_cmd")):
        cmd = (data.get(section) or {}).get(key, "")
        if not isinstance(cmd, str) or not any(r in cmd for r in KNOWN_RUNNERS):
            errors.append(f"{section}.{key} lacks a known test runner")
    return errors


def main() -> int:
    path = Path(sys.argv[1] if len(sys.argv) > 1 else "EVIDENCE.yml")
    data = load_packet(path)
    errors = check_required(data) + check_budget(data) + check_commands(data)
    if errors:
        print("INVALID PACKET")
        for item in errors:
            print(f"- {item}")
        return 1
    print("PACKET OK")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Install the YAML parser, then validate the packet:

python3 -m pip install pyyaml
python3 validate_evidence.py EVIDENCE.yml
Enter fullscreen mode Exit fullscreen mode

A failed validator means the pull request is not ready. Do not paste that packet into a model. Fix the fields first.

Numbered contributor workflow

Follow the gates in order. Do not skip the oracle. Do not start with a model-authored patch.

Gate 1. Mine facts from the issue thread

Copy claims that name inputs and outputs. Ignore calls to rewrite the module. Store the issue URL in the packet.

gh issue view 1842 --comments > /tmp/issue-1842.txt
rg -n "reproduce|expected|actual|fail" /tmp/issue-1842.txt
Enter fullscreen mode Exit fullscreen mode

Write a summary under forty words. Put it in issue.summary. Drop every sentence that lacks an input or output.

Gate 2. Record the environment hash

Reviewers need the reporter's lockfile, not a similar one. Hash the lockfile only.

sha256sum poetry.lock | tee /tmp/env.hash
# paste the digest into env.recorded_hash
Enter fullscreen mode Exit fullscreen mode

Paste the digest into env.recorded_hash. If the project uses images, hash the digest instead. Do not hash local caches or editor files.

Gate 3. Prove the fail command on main

Checkout main and install from the lockfile. Run only the recorded fail command. The exit code must be non-zero.

git fetch origin
git checkout origin/main
poetry install --sync
poetry run pytest tests/test_csv_bom.py::test_header_kept -q
echo $?
Enter fullscreen mode Exit fullscreen mode

A green fail command invalidates the packet. Rewrite the test until main is red. Then freeze repro.fail_cmd and keep it copy-pasteable.

Gate 4. Write the smallest oracle

The oracle is the command that must turn green. Prefer one test file. Do not use the full suite as the only gate.

# proposal: keep the oracle equal to one file
poetry run pytest tests/test_csv_bom.py -q
Enter fullscreen mode Exit fullscreen mode

Store the command and a pass pattern. Later log scoring depends on that pattern. Keep the pattern boring and stable.

Gate 5. Freeze the file budget

Search the tree for symbols named in the issue. Cap the resulting file list. Extra paths become non-goals.

git grep -n "BOM\|header" -- src tests
Enter fullscreen mode Exit fullscreen mode

Write the allowlist into patch_budget. Three files is a reasonable starting cap. Raise the cap only with a written reason in non_goals.

Gate 6. Patch inside the allowlist

Create a branch named after the issue id. Edit only listed paths. Keep the commit message factual.

git checkout -b fix/1842-csv-bom-header
# edit src/libparse/csv.py and tests/test_csv_bom.py only
git diff --stat
git add src/libparse/csv.py tests/test_csv_bom.py EVIDENCE.yml
git commit -m "Fix CSV BOM header drop (GH-1842)"
git diff origin/main...HEAD --name-only
Enter fullscreen mode Exit fullscreen mode

Confirm the name-only diff is a subset. Include EVIDENCE.yml as an extra allowed path. Reject surprise files before push.

Gate 7. Replay fail and oracle on the branch

Run the original fail test on the patched tree. Then run the oracle command. Save both logs beside the packet.

mkdir -p evidence
poetry run pytest tests/test_csv_bom.py::test_header_kept -q | tee evidence/fail-on-branch.log
poetry run pytest tests/test_csv_bom.py -q | tee evidence/oracle.log
Enter fullscreen mode Exit fullscreen mode

Attach evidence/oracle.log in the pull request body. Do not use screenshots as the only proof. Text logs remain searchable in review tools.

Gate 8. Score completeness before human review

Score each gate zero or one. Require six points before asking a maintainer. The rubric lives in the next section.

A model may apply the same rubric later. It must not expand the allowlist. It must not become the author of record.

Completeness rubric

This table is a proposal. Projects may tighten the pass bar. Humans still merge.

Gate Pass condition Score
Fail command recorded repro.fail_cmd is copy-pasteable 0/1
Fail proven on main Log shows the fail pattern 0/1
Oracle recorded oracle.pass_cmd is copy-pasteable 0/1
Oracle proven on branch Log shows the pass pattern 0/1
Env hash matches Lockfile digest equals env.recorded_hash 0/1
Allowlist respected git diff --name-only stays inside allowlist 0/1
Non-goals honored Diff has no drive-by refactors 0/1
Model stayed in role Review comments do not add files 0/1

A tiny scorer can read logs. The next script is not production code. Treat it as a proposal.

# score_packet.py — proposal, unexecuted in this article
from __future__ import annotations

import re
from pathlib import Path


def score_log(path: Path, pattern: str, expect_match: bool) -> int:
    text = path.read_text(errors="replace")
    found = re.search(pattern, text) is not None
    return int(found is expect_match)


def score_allowlist(changed: list[str], allow: list[str], extras: list[str]) -> int:
    allowed = set(allow) | set(extras)
    return int(all(path in allowed for path in changed))


# Example after CI writes logs:
# fail_score = score_log(Path("evidence/fail-main.log"), r"FAILED|AssertionError", True)
# pass_score = score_log(Path("evidence/oracle.log"), r"passed", True)
# files_score = score_allowlist(changed, allow, extras=["EVIDENCE.yml"])
Enter fullscreen mode Exit fullscreen mode

CI should fail below six points. Maintainers then review a complete packet. Incomplete packets bounce without a long debate.

Optional remote completeness scoring

Local proof stays first. A remote model is a second reader. It comments on missing fields and allowlist drift.

Keep the prompt narrow and mechanical:

Role: packet completeness reviewer.
Inputs: EVIDENCE.yml, git diff, oracle.log.
Task: score the eight gates. Quote missing fields.
Forbidden: new files, new features, full-repo rewrites.
Output: table of 0/1 scores and one paragraph of risks.
Enter fullscreen mode Exit fullscreen mode

Upload the packet, the bounded diff, and the oracle log. Do not upload the full repository. Do not upload secrets or production dumps.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. This article asserts only those two availability facts. No model names, quotas, hardware, or duration are claimed. After validate_evidence.py prints PACKET OK, that optional worker can score the eight gates as a checklist, never as a merge vote.

The workflow still holds if that worker is absent. The validator, the logs, and the rubric already gate the packet. Maintainers can replay the same commands without any model.

Maintainer replay in a few commands

Maintainers can check a stranger's packet quickly. They replay commands instead of rereading essays. The packet is the checklist.

git fetch origin pull/221/head:pr-221
git checkout pr-221
python3 validate_evidence.py EVIDENCE.yml
sha256sum poetry.lock
# compare the digest to env.recorded_hash
git diff origin/main...HEAD --name-only
poetry install --sync
poetry run pytest tests/test_csv_bom.py -q
Enter fullscreen mode Exit fullscreen mode

Reject the pull request when any gate fails. Request an updated packet. Do not request a longer cover letter.

Compare the name-only diff with patch_budget.allowlist. Extra files are a hard fail. Missing oracle logs are a hard fail as well.

What this method does not prove

This method assumes a deterministic test runner. Flaky tests break the fail and pass contract. Intermittent bugs need a different harness.

The validator does not prove security properties. It does not prove performance budgets. It does not prove API stability beyond the oracle command.

Model scores can be wrong. A high score can hide a logic error. A low score can flag a valid one-line fix.

Free model access and a free server option can change. This article does not rank tools. This article does not promise capacity or permanence.

The YAML schema is a proposal. Field names may change per repository. Do not treat the file as an ecosystem standard.

Who should skip this workflow

Do not publish packets for embargoed security issues. Public fail logs can leak exploit details. Use private maintainer channels instead.

Do not invent an oracle for a project without tests. The method has nothing honest to score then. Add tests first, then open the packet.

Do not send proprietary code to a remote worker. Keep the packet local in that case. License review comes before any upload.

Skip the eight gates for a one-word docs fix. The overhead would exceed the change. Use judgment on tiny typo pulls.

Do not list a model as the patch author. The contributor owns the diff. The maintainer owns the merge.

Practical close

OSS review gets faster when evidence is typed. The packet carries the issue, the oracle, and the file budget. Models may score that packet after local proof exists. They should not replace the oracle or the maintainer.

Top comments (0)