Anchor-Checked Review: A Mechanical Gate for Model Findings on OSS Patches

A maintainer opens a pull request that replaces a hand-rolled retry loop with a small helper module, then asks a free model for a review before reading the diff closely. The model returns eleven findings: an off-by-one index, two race conditions, a missing null check, a leaked file handle, and a remark about a function that does not exist anywhere in the repository. Six of the findings point at lines the patch never touched, because the off-by-one is reported three hundred lines away from the only hunk in the file.

The uncomfortable part is not that the model was wrong. The uncomfortable part is that nothing in the loop could separate a claim about the patch from a claim about the file's general reputation. The maintainer spent forty minutes on findings that had no anchor, and the two real defects in the patch were found later by a human reviewer on a second pass.

Anchor drift is the actual cost of cheap review

Low-cost or free model access changes the economics of review, not the epistemics of it. When a review costs nothing, contributors run it on every patch, and the volume of unanchored prose grows faster than human attention. Three failure modes dominate in practice.

• Line drift: findings cite a file and a line that exist, but sit outside every changed hunk, so the comment is about pre-existing code presented as a patch defect.
• Phantom symbols: the model names a helper or config key that was never in the diff, usually reconstructed from the function name rather than the packet.
• Evidence-free severity: a finding arrives labelled high with no command output, no quoted line, and no reproduction, which makes it impossible to accept or reject on technical grounds.

A workable loop needs one mechanical rule: every finding must resolve to a real line inside the diff, and must carry evidence that a script can check without judgement. Everything else is commentary.

The artifact: a review packet with verifiable anchors

The artifact below is deliberately small. It consists of a contract file, a packet built from git diff, a JSON output schema, and a checker that rejects findings whose anchors do not resolve. The examples are illustrative and were not executed against a specific upstream repository.

Step 1 — Freeze what the patch must prove

Start with a contract, committed next to the patch or kept in the review branch. It states the claim, the command that demonstrates it, and the invariants the reviewer may reason about.

{
  "claim": "the retry helper preserves a 3-attempt cap and closes the body exactly once",
  "reproduce": ["python -m pytest tests/test_retry.py -q"],
  "invariants": ["at most 3 attempts", "body closed exactly once on failure"],
  "out_of_scope": ["logging format", "metrics naming"]
}

The out_of_scope field matters more than it looks. It gives the reviewer a licensed place to put style opinions, and it gives the checker a category for findings that are legitimate but not patch defects.

Step 2 — Build a packet small enough to stay anchored

set -eu
mkdir -p review
git fetch origin pull/1234/head:pr-1234
git diff --unified=6 origin/main...pr-1234 -- '*.py' > review/diff.patch
git log --format='%h %s' origin/main..pr-1234 > review/commits.txt
wc -l review/diff.patch

The unified context is set to six lines on purpose. That window is wide enough for a reviewer to see surrounding control flow, and narrow enough that anything outside it is visibly a separate concern. Sending the whole repository makes phantom findings more likely, because the model can mine unrelated files for plausible defects.

Step 3 — Contract the answer as JSON with mandatory anchors

You are reviewing a patch. Use only review/diff.patch and review/contract.json.
Return JSON only, no prose before or after:
{
  "findings": [
    {
      "file": "<path exactly as it appears in the diff>",
      "line": <integer, new-side line number inside a hunk>,
      "severity": "high" | "medium" | "low",
      "claim": "<one sentence about this patch only>",
      "evidence": "<quoted diff line or command output>",
      "confidence": "low" | "medium" | "high"
    }
  ]
}
Rules:
1. file must appear in the diff; line must be inside a hunk.
2. evidence must quote text that appears in the packet.
3. If a claim cannot be anchored, omit it and do not guess.
4. Do not restate the contract as a finding.

This is where free model access becomes useful rather than noisy. MonkeyCode is described by its maintainers as an open-source project offering free model access and a free server option, and the operator states that the free tier provides 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. No model names, rate limits, or hardware details are assumed here; verify current terms yourself before relying on them.

Step 4 — Reject findings that do not resolve to the diff

The checker is the whole point. It uses only the standard library, reads the packet diff, and rewrites the model's output into an accepted list and a rejected list with a reason per finding.

#!/usr/bin/env python3
"""anchor_check.py — reject model findings that do not resolve to the reviewed diff.

Usage: anchor_check.py findings.json [allow_context]
Reads: review/diff.patch (built in step 2)
"""
import json, re, sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")


def anchored_lines(diff_text):
    """Map path -> set of new-side line numbers visible in the diff (context + added)."""
    touched, path, new_line = {}, None, None
    for raw in diff_text.splitlines():
        if raw.startswith("diff --git "):
            path, new_line = None, None          # reset per file
            continue
        if new_line is None and raw.startswith("+++ b/"):
            path = raw[6:].split("\t")[0]
            touched.setdefault(path, set())
            continue
        m = HUNK.match(raw)
        if m:
            new_line = int(m.group(1))
            continue
        if path is None or new_line is None:
            continue
        if raw.startswith("+") or raw.startswith(" "):
            touched[path].add(new_line)
            new_line += 1
        elif raw.startswith("-"):
            pass                                 # deletion: no new-side line
    return touched


def verdict(items, touched, allow_context=0):
    accepted, rejected = [], []
    for item in items:
        path, line = item.get("file"), item.get("line")
        if path not in touched:
            reason = "file not in diff"
        elif not isinstance(line, int):
            reason = "line is not an integer"
        elif allow_context == 0 and line not in touched[path]:
            reason = "line outside visible hunk lines"
        elif allow_context and not any(abs(line - t) <= allow_context for t in touched[path]):
            reason = "line beyond allowed context window"
        elif not str(item.get("evidence", "")).strip():
            reason = "no evidence field"
        else:
            reason = None
        (accepted if reason is None else rejected).append(
            item if reason is None else {**item, "reject_reason": reason}
        )
    return accepted, rejected


def main():
    findings = json.load(open(sys.argv[1], encoding="utf-8"))["findings"]
    allow = int(sys.argv[2]) if len(sys.argv) > 2 else 0
    touched = anchored_lines(open("review/diff.patch", encoding="utf-8").read())
    accepted, rejected = verdict(findings, touched, allow)
    report = {
        "accepted": accepted,
        "rejected": rejected,
        "reject_ratio": round(len(rejected) / max(1, len(findings)), 3),
    }
    print(json.dumps(report, indent=2, ensure_ascii=False))
    sys.exit(0 if accepted else 1)               # non-zero when nothing survives


if __name__ == "__main__":
    main()

python review/anchor_check.py review/findings.json 0 | tee review/verdict.json
jq '.rejected[] | {file, line, reject_reason}' review/verdict.json

The exit code is the enforcement mechanism. A CI job that runs this after the model call turns review into a gate rather than a suggestion, and the jq line gives the maintainer a one-screen list of what the model tried to assert without support.

Read the reject ratio, not the finding count

One number is worth tracking per patch: the share of findings that survive the anchor check. A ratio near zero means the packet is too small to review meaningfully, or the prompt is inviting speculation. A ratio near one across many patches usually means the checker is too permissive and the context window has swallowed the whole file.

• Ratio below 0.3: shrink the question set, add the contract's out_of_scope list, and stop sending unrelated files.
• Ratio above 0.9 with high severity everywhere: enforce a cap on findings per file and require a quoted evidence line for each one.
• Accepted findings that a human later dismisses: record the dismissal reason next to the anchor, so the prompt contract learns the repository's actual conventions.

None of these metrics measure model quality in the abstract. They measure whether the review output can be acted on, which is the only property a maintainer can verify cheaply.

Where a free server option fits in the loop

Anchor checking, pytest, and lint runs are deterministic and cheap, which makes them reasonable candidates for a small always-on environment rather than a laptop. If the free server option can host the repository's toolchain and reach the git remote, the sequence fetch → diff → test → anchor_check can run there on a schedule, and the maintainer only reads verdict.json plus a green test log.

Verify the runtime, persistence, and network reachability yourself before depending on it, because a resolver failure inside that environment looks exactly like a legitimate anchor rejection from the outside. A one-line sanity check at the start of the job prevents that confusion.

python review/anchor_check.py review/selftest_findings.json 0 >/dev/null \
  && echo "checker ok" || { echo "checker misconfigured"; exit 2; }

Planning a 10-million-token allowance

The allowance is rarely the bottleneck once packets are small. The table below is a planning model, not a product fact: every token figure is an assumption to be replaced with measured numbers from your own runs.

Item	Assumption	Monthly total
Model calls per patch	3 (triage, review, re-check of rejected)	24 calls at 8 patches
Input tokens per call	~6,000 (contract + diff + rules)	~144,000
Output tokens per call	~1,500 (JSON findings)	~36,000
Total per month	—	~180,000

At that scale a 10-million-token allowance covers many months of eight-patch cadence, which suggests the real constraint is not tokens but the human time spent on accepted findings. Teams that burn through allowances quickly are usually sending whole files instead of packets, or asking for exploration instead of anchored review.

Limitations and who should skip this

The anchor check proves that a finding points at the patch. It proves nothing about whether the claim is correct, and a model can attach a real line number to a wrong diagnosis with no difficulty. The checker also rejects legitimate design-level comments about untouched code, which is why out_of_scope and the review comments should stay in a separate channel.

Renames, deletions, and rebases shift line numbers, so the packet must be rebuilt after any rebase and the verdict regenerated from the new diff. Security review is out of scope here: absence of anchored findings is not evidence of absence of vulnerabilities, and the loop should not be presented as an audit.

Skip this approach if the repository has no runnable reproduction command, if the patch is dominated by moves and deletions, or if policy forbids sending diff content to an external model. In those cases the same contract and anchor rules still help a human reviewer, but the automated gate adds little.

The practical takeaway is narrow: put the evidence requirement in the prompt, verify the anchors with a script, and let the reject ratio decide how much a model contributes to review. A team that wants to try the loop before committing budget can start on the free tier, keep anchor_check.py local, and treat the reject ratio as the deciding signal for whether model review earns a permanent slot in the patch path.