Anchored Review Findings: Reject Model Comments That Cite No Diff Line

A model review earns its place only when every finding points at a real changed line. Unanchored comments cost maintainers more time than they save.

The workflow below keeps the normal open-source loop: reproduce the bug, patch it, test it. It adds one deterministic check to the review step.

That check is a script. It verifies file path, line number, and quoted text against the diff. Findings that fail are dropped before any human reads them.

Three predictable failure modes

Model reviews fail in boring ways. The first is a path that never appears in the diff. The second is a line number that shifted after a rebase. The third is a quote that paraphrases code nobody wrote.

All three are cheap to detect. Paths, line numbers, and quotes are facts about the patch. No semantic understanding is required to reject them.

A review with ten findings and four invented anchors is worse than no review. Maintainers spend the first ten minutes disproving fiction.

The review contract

Give the model a strict output schema. Ambiguity in the schema becomes noise in the review.

Rules handed to the model:

1. file must be a path from the diff, without the b/ prefix.
2. line must be an added line number in the new file.
3. quote must be copied verbatim and stay at 120 characters or fewer.
4. claim must name a defect, not a style preference.
5. Output is JSON only. No prose, no code fences, no severity inflation.

severity takes high, medium, or low. Unknown values are treated as low by the gate.

Step 1: export a zero-context diff

Zero context keeps the anchor map small and unambiguous.

git fetch origin
git diff -U0 origin/main...HEAD -- '*.py' > /tmp/patch.diff
wc -l /tmp/patch.diff

Limit the diff to source files. Lockfiles and generated assets add noise and no reviewable logic.

Split large diffs by directory. The gate runs per diff file, so splitting is safe.

Step 2: run the review

Send prompt and diff as two separate inputs. Separation stops the model from mixing instructions with patch text.

The shell below assumes an OpenAI-compatible chat endpoint. Substitute the base URL and key from the provider's current docs.

jq -n --rawfile p review_prompt.txt --rawfile d /tmp/patch.diff \
  '{model: env.MODEL, messages: [{role: "user", content: ($p + "\n\n" + $d)}]}' \
  > /tmp/request.json

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $MODEL_KEY" \
  -H "Content-Type: application/json" \
  -d @/tmp/request.json > /tmp/review_raw.json

Free model access is why this step is cheap to repeat. The same diff can go through several models, and the gate decides which output survives. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option keeps the review off a sleeping laptop. Both availability claims are operator-supplied. Check current limits and terms before relying on them in a recurring job.

Strip code fences before gating. Models add them despite instructions.

jq -r '.choices[0].message.content' /tmp/review_raw.json \
  | sed 's/^```

json//; s/^

```$//' > /tmp/review.json

Step 3: gate the findings

Save this as anchor_gate.py. It reads only the diff and the model output.

#!/usr/bin/env python3
"""anchor_gate.py: drop review findings that cite no real diff line."""
import json
import re
import sys

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")


def added_lines(diff_path):
    added = {}
    current = None
    new_no = None
    with open(diff_path, encoding="utf-8", errors="replace") as fh:
        for raw in fh:
            line = raw.rstrip("\n")
            if line.startswith("+++ b/"):
                current = line[6:]
                continue
            if line.startswith("---") or line.startswith("+++"):
                continue
            match = HUNK.match(line)
            if match:
                new_no = int(match.group(1))
                continue
            if current is None or new_no is None:
                continue
            if line.startswith("+"):
                added.setdefault(current, {})[new_no] = line[1:]
                new_no += 1
            elif line.startswith("-"):
                continue
            elif line.startswith(" "):
                new_no += 1
    return added


def verdict(finding, added):
    path = finding.get("file", "")
    line = finding.get("line")
    quote = (finding.get("quote") or "").strip()
    if path not in added:
        return "file_not_in_diff"
    if not isinstance(line, int) or line not in added[path]:
        return "line_not_added"
    if quote and quote not in added[path][line]:
        return "quote_mismatch"
    return "ok"


def main(diff_path, review_path):
    added = added_lines(diff_path)
    with open(review_path, encoding="utf-8") as fh:
        review = json.load(fh)
    kept, dropped = [], []
    for finding in review.get("findings", []):
        tag = verdict(finding, added)
        record = dict(finding, anchor=tag)
        if tag == "ok":
            kept.append(record)
        else:
            dropped.append(record)
    total = len(kept) + len(dropped)
    report = {
        "kept": kept,
        "dropped": dropped,
        "drop_ratio": round(len(dropped) / total, 3) if total else 0.0,
    }
    print(json.dumps(report, indent=2))
    return 0 if not dropped else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

python3 anchor_gate.py /tmp/patch.diff /tmp/review.json; echo "exit=$?"

Exit code 0 means every finding is anchored. Exit code 1 means at least one was dropped.

Step 4: escalate by verdict

Anchor status drives the next action. Nothing else should.

Anchor result	Severity	Action
ok	high	Read the diff line yourself, then patch or reply
ok	medium	Batch with the next commit, do not interrupt the queue
ok	low	Keep in the report, never forward to the maintainer
line_not_added	any	Discard. Log the count per model
quote_mismatch	any	Re-ask once with the exact diff excerpt attached
file_not_in_diff	any	Discard and drop that model from the run

The drop_ratio value is a property of the model, not of the patch. Track it across runs per repository.

A rising ratio on one model usually means the diff format changed. Check the +++ b/ prefix before blaming the model.

Step 5: a test plan you can run once

Build four findings against a known two-line diff. Expected results are fixed, so the gate itself is testable.

{"findings": [
  {"file": "src/parse.py", "line": 42, "quote": "return None", "severity": "high", "claim": "Swallows the parse error"},
  {"file": "src/parse.py", "line": 99, "quote": "return None", "severity": "high", "claim": "Line does not exist"},
  {"file": "src/missing.py", "line": 10, "quote": "x = 1", "severity": "medium", "claim": "File not in diff"},
  {"file": "src/parse.py", "line": 42, "quote": "return retry()", "severity": "low", "claim": "Quote never written"}
]}

Expected output: one kept finding, three dropped, drop_ratio of 0.75, exit code 1.

The check is reproducible. Create a branch with two added lines, edit the line numbers, and rerun.

Limitations

• The gate checks position, not truth. A neatly anchored wrong claim still passes.
• Zero-context diffs cannot anchor unchanged lines. Use -U3 plus extra bookkeeping if context findings matter.
• A patch that adds a line starting with +++ b/ confuses the parser. Skip such patches.
• Renames, binary files, and mode changes need separate handling.
• Vendored and generated files inflate the diff and produce anchor noise. Exclude them early.
• Quote matching is exact substring. Reformatted quotes fail and should be re-requested, not trusted.

Who should not use this

The gate adds a step, so it needs a reason. Skip it when the diff is under ten lines and you will read it anyway.

Skip it for repositories whose maintainers require human-only review. A gate plus model output does not satisfy that policy.

Never wire the exit code to auto-merge. Anchored findings are candidates for attention, not approvals.

If the team cannot store model output for later audit, the drop log has no value. Fix retention first.

What to measure before scaling

Run the gate on five real patches. Record kept findings, dropped findings, and the reason for each drop.

A useful run keeps at least one finding per patch and drops under half. A run that drops most findings is a prompt problem, not a patch problem.

When the numbers look stable, extend the loop. A small anchored review run is the cheapest way to find out whether free-model review fits your contribution queue.