DEV Community

Taylor Wang
Taylor Wang

Posted on

Keep OSS Reviews Inside a Four-File Packet

#git #opensource #testing #ai

Free-model reviews drift when they read an entire tree. Maintainers then chase defects that git never recorded. A four-file packet pins every comment to frozen evidence.

Core rule

The patch does not start as a chat thread. The contributor freezes SHA, paths, diff, and test output together. A model may read those four files only.

Why unbounded reviews fail on OSS work

Agent helpers assume architecture the repository never stated. The model invents a cause, then reviews the invention.

OSS pull requests cannot carry those invented causes. Maintainers need hunk citations plus the failing command. The packet below makes both items mandatory.

Packet layout

The contributor keeps a packet directory on the branch.

  1. FAIL.txt stores the command, SHA, and captured output.
  2. BOUND.txt stores allowed paths and forbidden paths.
  3. PATCH.diff stores the unified diff against the freeze.
  4. REVIEW.md stores model notes that cite hunk headers.

Chat logs stay out of the packet directory. Those logs are not valid review evidence.

Step 1: Freeze the SHA on a clean tree

The contributor refuses to patch a dirty worktree. Extra files leak into later review diffs.

git status --porcelain
git rev-parse HEAD
git symbolic-ref --short HEAD
Enter fullscreen mode Exit fullscreen mode

An empty porcelain listing is required here. Any status output means the freeze stops.

mkdir -p packet
{
  printf 'sha: %s\n' "$(git rev-parse HEAD)"
  printf 'branch: %s\n' "$(git symbolic-ref --short HEAD)"
} > packet/BOUND.txt
Enter fullscreen mode Exit fullscreen mode

The SHA in BOUND.txt never changes during the patch. A rebase restarts the packet from the first step.

Step 2: Capture the failing command

The contributor records one command, not a whole CI log. Extra noise hides the contract under review.

{
  printf 'sha: %s\n' "$(git rev-parse HEAD)"
  printf 'cmd: pytest -q tests/test_retry.py::test_timeout_on_4xx\n'
  pytest -q tests/test_retry.py::test_timeout_on_4xx
} > packet/FAIL.txt 2>&1 || true
tail -n 30 packet/FAIL.txt
Enter fullscreen mode Exit fullscreen mode

The labeled block above is a template. It is not a measured benchmark or timing study.

FAIL.txt must mention the same SHA as BOUND.txt. Mismatched SHAs invalidate the whole review packet.

Step 3: Write the smallest failing test

The test states the contract in code. Plain comments cannot replace that failing assertion.

# tests/test_retry.py
# Proposal: example test, not a claimed upstream file.

def test_timeout_on_4xx_does_not_retry(monkeypatch):
    calls = {"n": 0}

    def fake_get(_url, **_kwargs):
        calls["n"] += 1

        class Resp:
            status_code = 400
            text = "bad request"

        return Resp()

    monkeypatch.setattr("httpx.get", fake_get)
    from retrykit import fetch
    fetch("https://example.invalid/item", retries=3)
    assert calls["n"] == 1
Enter fullscreen mode Exit fullscreen mode

The contributor runs that test at the frozen SHA. It must fail before the patch exists.

Passing tests at freeze time mean missing reproduction. The packet then stops until a real failure exists.

Step 4: Fill BOUND.txt with paths

Review context dies at the allow list. Files outside it stay invisible to the model.

sha: 9f3c1e2ab21d
branch: fix/retry-4xx
allow:
  src/retrykit/fetch.py
  tests/test_retry.py
forbid:
  src/retrykit/cli.py
  docs/
  .github/
Enter fullscreen mode Exit fullscreen mode

The allow list should stay tiny on purpose. Two or three paths suffice for most bug fixes.

Step 5: Emit PATCH.diff from the index

The contributor stages only the allowed paths. Then the diff is written to disk.

git add src/retrykit/fetch.py tests/test_retry.py
git diff --cached --stat
git diff --cached > packet/PATCH.diff
wc -l packet/PATCH.diff
Enter fullscreen mode Exit fullscreen mode

A huge line count signals an accidental refactor. Refactors need a new packet and a new bound.

The reviewer must cite hunks in this form.

--- a/src/retrykit/fetch.py
+++ b/src/retrykit/fetch.py
@@ -41,8 +41,11 @@ def fetch(url, retries=3):
     for attempt in range(retries):
         resp = httpx.get(url, timeout=5)
-        if resp.status_code >= 400:
-            continue
+        if 400 <= resp.status_code < 500:
+            break
+        if resp.status_code >= 500:
+            continue
         return resp
     return resp
Enter fullscreen mode Exit fullscreen mode

This hunk is an unlabeled example only. It is not a claimed fix from a real project.

Step 6: Run a local matrix before any model

Local commands remain the source of truth. Models do not get a vote on pass or fail.

pytest -q tests/test_retry.py::test_timeout_on_4xx_does_not_retry
pytest -q tests/test_retry.py
python -m compileall src/retrykit
Enter fullscreen mode Exit fullscreen mode

The contributor appends a short matrix to FAIL.txt after the patch.

matrix:
  new_test: pass
  file_tests: pass
  compileall: pass
  full_suite: skipped_no_secrets
Enter fullscreen mode Exit fullscreen mode

Skipped rows need a written reason beside them. Silent skips read as green and mislead reviewers.

Step 7: Let a free model read the packet only

Free-tier models help as second readers on small contexts. They fail as authors of unbounded refactors.

The contributor writes a single PROMPT.txt file.

Role: second reader, not patch author.
Read FAIL.txt, BOUND.txt, and PATCH.diff only.
Ignore any file outside the allow list.
Ignore any cause FAIL.txt does not support.
Every finding must cite a PATCH.diff hunk header.
Classify findings as blocking, note, or unknown.
Do not rewrite code unless a blocking issue is proven.
Enter fullscreen mode Exit fullscreen mode

REVIEW.md must quote hunk headers from PATCH.diff. Findings without those headers get deleted immediately.

This review step can run on a free hosted model. Many laptops have no spare accelerator for local serving.

A free server can host that narrow review job. The test matrix still runs on the contributor machine.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. Those options fit the packet review step above. They do not replace pytest, git freeze, or maintainer judgment.

The contributor uploads the packet directory only. The rest of the clone never enters the prompt.

Artifact: packet_check.py

The script below is a local gate. It never calls a model over the network. It blocks review when evidence is incomplete.

#!/usr/bin/env python3
"""Validate an OSS review packet. Unexecuted until the contributor runs it."""
from pathlib import Path
import re
import sys

ROOT = Path("packet")
REQUIRED = ("FAIL.txt", "BOUND.txt", "PATCH.diff", "REVIEW.md")


def read(name: str) -> str:
    path = ROOT / name
    if not path.is_file():
        raise SystemExit(f"missing {path}")
    return path.read_text(encoding="utf-8")


def allowed_paths(bound: str) -> set[str]:
    paths: set[str] = set()
    capture = False
    for line in bound.splitlines():
        stripped = line.strip()
        if stripped == "allow:":
            capture = True
            continue
        if stripped.endswith(":") and capture:
            break
        if capture and stripped:
            paths.add(stripped)
    if not paths:
        raise SystemExit("BOUND.txt has no allow: paths")
    return paths


def diff_paths(patch: str) -> set[str]:
    found: set[str] = set()
    for line in patch.splitlines():
        if line.startswith("+++ b/"):
            found.add(line[6:])
    return found


def review_cites_hunks(review: str, patch: str) -> None:
    hunks = re.findall(r"^@@ .+ @@", patch, flags=re.M)
    if not hunks:
        raise SystemExit("PATCH.diff has no hunk headers")
    if "@@" not in review:
        raise SystemExit("REVIEW.md cites no hunk headers")


def main() -> int:
    fail, bound, patch, review = (read(n) for n in REQUIRED)
    blob = f"{fail}\n{bound}".lower()
    if "sha:" not in blob:
        print("packet_check: no sha: marker", file=sys.stderr)
        return 2
    extra = diff_paths(patch) - allowed_paths(bound)
    if extra:
        print(f"packet_check: paths outside bound: {sorted(extra)}")
        return 2
    review_cites_hunks(review, patch)
    print("packet_check: ok")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode 
python3 packet_check.py
echo $?
Enter fullscreen mode Exit fullscreen mode

Exit status 2 means the packet is incomplete. The contributor fixes files before any prompt. The model is not prompted in that state.

Drop invented comments

A leaked comment may name a forbidden path. That comment is deleted, not debated in thread.

Another leaked comment may blame DNS without evidence. FAIL.txt showed HTTP 400, so the comment goes.

Kept comments follow the cited shape below.

## blocking
`@@ -41,8 +41,11 @@ def fetch(url, retries=3):`
The 4xx branch breaks the loop but still returns the error body.
FAIL.txt only requires a single HTTP call.
Confirm the caller treats a 400 payload as a hard error.
Enter fullscreen mode Exit fullscreen mode

Unknowns stay labeled as unknown on purpose. They do not become blocking issues by default.

Decision table

The table below maps signals to concrete actions.

Signal Action
Blocking finding cites hunk and FAIL.txt Edit the diff and rerun the matrix
Finding names a path outside allow Delete the finding
Unknowns outnumber blocking items Ping a maintainer instead of another model
Tests fail after a model rewrite Reset to the frozen SHA and restart
packet_check.py exits 2 Do not open the pull request
Matrix shows skip without reason Fill the reason or run the row

Humans keep the merge decision on the request. Models never merge the pull request themselves.

When a model rewrite breaks tests

The contributor restores the freeze from BOUND.txt. Then the packet is rebuilt from the first step.

FREEZE=$(awk '/^sha:/{print $2; exit}' packet/BOUND.txt)
git reset --hard "$FREEZE"
Enter fullscreen mode Exit fullscreen mode

A hard reset is destructive for uncommitted work. The contributor confirms the SHA before running it.

Open the pull request with the packet

The pull request body should paste FAIL.txt and the matrix. Maintainers should not hunt through chat history.

git log -1 --format=%B
git push -u origin HEAD
Enter fullscreen mode Exit fullscreen mode

A short pull request template follows. It is a proposal, not a project rule.

## Freeze
SHA from packet/BOUND.txt

## Contract
Command and output from packet/FAIL.txt

## Bound paths
Allow list from packet/BOUND.txt

## Review
packet/REVIEW.md after invented comments were removed
Enter fullscreen mode Exit fullscreen mode

The four files can ride in the branch. They can also sit in the pull request body.

Limits

The workflow does not prove production safety. It only keeps review tied to evidence.

Free models still invent causes. The packet makes those inventions easy to delete. It does not grade API design.

The free server option is not a CI system. Secrets, private submodules, and live fixtures stay local.

packet_check.py does not understand every diff dialect. Renames can dodge +++ b/ parsing. Binary files do not belong in PATCH.diff.

No quota, hardware, model name, or uptime claim appears here. Those figures move and need a vendor dashboard.

Who should not use this

Incident responders should not build packets during an outage. They should follow the existing runbook.

Contributors without a failing command should stop. Reproduction is a prior workflow.

Projects that ban hosted models should keep REVIEW.md human-written. The same four files still apply.

Wide refactors will not fit a tiny allow list. Split the work before review starts.

Close

Maintainers trust files they can hash. They do not trust chat transcripts.

Keep FAIL.txt, BOUND.txt, PATCH.diff, and REVIEW.md together. Give a free model that set only. Then open the pull request with evidence.

Contributors who need a hosted second reader after the packet exists may use MonkeyCode free model access and the free server option for that single step.

Top comments (0)