A Review Claim Without a Line Number Is a Rumor: Verifying Model Review Output Against the Post-Patch Tree

A patch lands, a reviewer writes 'this will raise KeyError around line 210', and the author spends twenty minutes hunting for a KeyError that does not exist. The line moved during a rebase, the reviewer read a stale tree, and the comment quietly became unfalsifiable. That scene is a composite of the failure mode rather than a specific incident, yet it now appears wherever a language model drafts the first review pass.

Model-assisted review has one weakness that is cheap to fix and rarely discussed. The prose is fluent, the severity labels look considered, and the line numbers are frequently invented. A comment that cannot be located in the tree is not a review comment; it is a rumor with a tag attached to it.

The rule: assertions carry coordinates, opinions do not

Every claim is sorted into one of three classes before it reaches the author.

• Assertion: a statement about what the code does at a specific place. It must carry a path, a post-patch line number, and a literal token copied from that line.
• Speculation: a plausible risk the reviewer could not confirm. It may name a file but can never be a blocker.
• Recommendation: a preference about structure or naming. It needs a rationale, not a citation.

Only assertions enter the blocking queue, and only after a script resolves their coordinates. Everything else stays visible without being able to stall a merge. That single split removes most of the friction, because authors stop arguing about claims that were never checkable in the first place.

Step 1: Freeze the tree the review will be judged against

Claims rot the moment the branch moves, so the tree has to be pinned before anything reads it.

git fetch origin
git switch -c review/patch-topic origin/main
git am 0001-patch-subject.patch
git rev-parse HEAD
git format-patch -1 --stdout > patch.diff

Record that HEAD hash in the claims file and in the pull request comment. Any later rebase invalidates the file and forces a regeneration, which is the intended cost rather than an inconvenience.

Step 2: Draft candidate claims with a model

The drafting step is mechanical: read the diff, propose claims, assign a class. A free model tier is sufficient here, and the cost matters because the pass runs per hunk instead of per patch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is the environment used for the drafting pass described below, and the operator describes its free tier as including a generous token allowance plus a free server option; treat any published figure as a snapshot and confirm the current limits on the site before planning around them.

A short and strict prompt produces usable output.

You are reviewing patch.diff against the tree at HEAD <hash>.
Emit JSON Lines only, one claim per line, with fields:
id, kind (assertion|speculation|recommendation), severity, claim.
For kind=assertion also emit path, line, token.
line is the post-patch number from the +++ side of the hunk header.
token must be a literal substring of that exact line, copied character for character.
If you cannot point at a line, change the kind to speculation.

Two constraints do most of the work here. Requiring a literal token makes fabricated coordinates fail loudly, and banning blocker-level speculation keeps the queue honest. A reviewer that cannot cite may still speak, but it speaks in a different lane.

Step 3: Verify every citation with a script

The verifier reads the claims file and the post-patch tree, then rejects anything it cannot resolve. It depends only on the standard library, which matters when the check runs inside a minimal CI image.

#!/usr/bin/env python3
"""review_cite.py - confirm that review assertions point at lines that exist.

Usage:
    python review_cite.py --repo . --claims claims.jsonl --diff patch.diff

Exit codes: 0 = clean, 1 = invalid claims or uncovered hunks.
"""
import argparse
import json
import sys
from pathlib import Path


def parse_hunk(raw):
    if not raw.startswith('@@'):
        return None
    try:
        plus = raw.split('+', 1)[1].split(' ', 1)[0]
        parts = plus.split(',')
        start = int(parts[0])
        count = int(parts[1]) if len(parts) > 1 else 1
    except (IndexError, ValueError):
        return None
    return start, count


def added_ranges(diff_text):
    ranges = {}
    path = None
    for raw in diff_text.splitlines():
        if raw.startswith('+++ '):
            rest = raw[4:].strip()
            path = rest[2:] if rest.startswith('b/') else None
            if path:
                ranges.setdefault(path, [])
            continue
        hunk = parse_hunk(raw)
        if not (hunk and path):
            continue
        start, count = hunk
        if count > 0:
            ranges[path].append((start, start + count - 1))
    return ranges


def load_claims(filename):
    claims = []
    text = Path(filename).read_text(encoding='utf-8')
    for number, line in enumerate(text.splitlines(), 1):
        line = line.strip()
        if not line:
            continue
        try:
            claims.append(json.loads(line))
        except json.JSONDecodeError as exc:
            raise SystemExit('claims line %d: %s' % (number, exc))
    return claims


def check_claims(repo, claims):
    problems = []
    for claim in claims:
        if claim.get('kind') != 'assertion':
            continue
        cid = claim.get('id', 'no-id')
        missing = [k for k in ('path', 'line', 'token') if not claim.get(k)]
        if missing:
            problems.append((cid, 'missing field(s): ' + ', '.join(missing)))
            continue
        target = repo / claim['path']
        if not target.is_file():
            problems.append((cid, 'file not found: ' + claim['path']))
            continue
        lines = target.read_text(encoding='utf-8', errors='replace').splitlines()
        index = int(claim['line'])
        if not 1 <= index <= len(lines):
            problems.append((cid, 'line %d outside 1..%d of %s' % (index, len(lines), claim['path'])))
            continue
        if claim['token'] not in lines[index - 1]:
            problems.append((cid, 'token %r absent from %s:%d' % (claim['token'], claim['path'], index)))
    return problems


def hunk_coverage(ranges, claims):
    cited = {}
    for claim in claims:
        if claim.get('kind') == 'assertion' and claim.get('path') and claim.get('line'):
            cited.setdefault(claim['path'], set()).add(int(claim['line']))
    report = []
    for path, spans in ranges.items():
        for start, end in spans:
            hits = [n for n in cited.get(path, ()) if start <= n <= end]
            report.append((path, start, end, end - start + 1, len(hits)))
    return report


def main(argv=None):
    parser = argparse.ArgumentParser(description='verify review citations against a post-patch tree')
    parser.add_argument('--repo', default='.', type=Path)
    parser.add_argument('--claims', required=True)
    parser.add_argument('--diff', required=True, type=Path)
    parser.add_argument('--allow-uncovered', type=int, default=0)
    args = parser.parse_args(argv)

    claims = load_claims(args.claims)
    ranges = added_ranges(args.diff.read_text(encoding='utf-8'))
    problems = check_claims(args.repo, claims)
    report = hunk_coverage(ranges, claims)

    print('%-38s %-14s %6s %6s' % ('path', 'hunk', 'added', 'cited'))
    uncovered = 0
    for path, start, end, size, hits in report:
        if hits == 0:
            uncovered += 1
        note = '' if hits else '  <-- uncovered'
        print('%-38s %-14s %6d %6d%s' % (path, '%d-%d' % (start, end), size, hits, note))

    for cid, message in problems:
        print('INVALID %s: %s' % (cid, message), file=sys.stderr)

    print('claims=%d invalid=%d uncovered_hunks=%d' % (len(claims), len(problems), uncovered))
    return 1 if problems or uncovered > args.allow_uncovered else 0


if __name__ == '__main__':
    sys.exit(main())

A claims file stays small because every line is one object.

{"id":"c1","kind":"assertion","severity":"blocker","path":"src/parser.py","line":142,"token":"options.pop(","claim":"Unknown keys raise KeyError before validation runs"}
{"id":"c2","kind":"assertion","severity":"nit","path":"src/parser.py","line":150,"token":"except KeyError","claim":"The fallback only catches KeyError, not TypeError"}
{"id":"c3","kind":"recommendation","severity":"nit","claim":"Move the schema check next to the parser factory"}
{"id":"c4","kind":"speculation","severity":"question","claim":"Concurrent callers may observe a partially filled dict"}

Run the check against the frozen tree.

python review_cite.py --repo . --claims claims.jsonl --diff patch.diff

Output lists each added range with its citation count, then the rejected claims.

path                                   hunk            added  cited
src/parser.py                          120-127             8      3
src/parser.py                          140-151            12      2
src/schema.py                           18-24             7      0  <-- uncovered

claims=4 invalid=0 uncovered_hunks=1

The exit code is the point of the exercise. A claim whose token is not on the cited line never becomes a blocker, which saves the author from chasing a fluent sentence into the wrong file.

Step 4: Confirm that changed hunks were actually reviewed

Citations prove that claims are locatable, while coverage proves that the patch was read at all. The same report pairs each added range with the number of citations landing inside it.

An uncovered hunk is not automatically a defect, but it is a known blind spot that deserves a second look. Most of them turn out to be generated files, lockfiles, or pure renames, so a path-prefix ignore list is cheaper than an argument.

A decision table for where each step runs

Situation	Local verifier	Model drafting pass	Free server option
One module, familiar code	Required	Optional	Not needed
Unfamiliar subsystem, 10+ hunks	Required	One pass per hunk	Not needed
No local toolchain available	Required	One pass per hunk	Run the verifier remotely after checking current limits
Code that cannot leave the network	Required	Local model only	Not permitted

The table is deliberately boring, because the useful decision is almost always the same one. Run the verifier locally, keep drafting cheap, and only move execution elsewhere when the local machine cannot do the job.

What this approach does not do

• Token matching is syntactic. A wrong assertion with a correct citation still passes, so a human reads the content.
• Coverage is a count rather than a judgement. A noisy reviewer can satisfy it with irrelevant citations, which is one more reason to read the claims file directly.
• Free tiers change without notice. Any quota quoted in a blog post expires, and the provider page is the only current source.
• The check does not replace maintainer taste. It removes unlocatable blockers and does nothing else.

Who should not use this

• Teams whose review value is mostly architectural discussion, where line coordinates add noise instead of signal.
• Projects where the patch is regenerated by a tool on every run, so line numbers never stabilize long enough to verify.
• Repositories that need automatic approval; this workflow filters claims and never approves a merge.
• Anyone who has not checked the provider's data handling terms before sending confidential code through a drafting pass.

The whole workflow costs one script and one prompt template. In return the review queue contains claims that can be confirmed or refuted in seconds, and the drafting step becomes cheap enough to run on every hunk instead of only on the ones somebody worried about.

If you want to test the drafting half without an API budget, MonkeyCode's free model access and free server option are the two pieces this workflow assumes; confirm the current terms on the site before depending on them.