Two Reviewers, One Patch: Treating Model Disagreement as a Triage Signal

A maintainer opens a pull request that touches eleven files under src/net/, then asks two free coding models for a review. Reviewer A reports a null dereference at line 214; reviewer B reports the same function, a different line, and a leaked file handle. The patch is small enough to read in twenty minutes, but the two reports share no anchors, and the maintainer now has to decide which report deserves the next hour.

That situation is common in open-source review, and the usual response is to trust whichever model sounds more confident. A better response is to measure where the two reviewers disagree, because disagreement is cheap to compute and it concentrates human attention on the parts of a patch that are genuinely ambiguous. This article describes a four-step loop that produces that measurement, along with two small scripts that do the mechanical work.

Why a single reviewer inflates confidence

One model producing twelve findings looks like thorough coverage, but the output rarely distinguishes between a verified defect and a restatement of the diff. A reviewer that walks a hunk and narrates it will always produce findings, and the count says nothing about correctness. The failure modes repeat across providers:

• Findings anchored to line numbers that do not exist in the post-image, which makes verification slower than reading the file.
• Style observations mixed with defect claims in one flat list, so a missing error check ranks beside a naming preference.
• Silent truncation on large diffs, where later files receive no attention at all but the output does not say so.
• Confident restatement, where the review describes what the patch does and calls that description a risk.

A second reviewer does not eliminate those failures, but the intersection of two independent runs filters a useful share of them. Phantom line numbers rarely agree by accident, and restatement noise rarely lands on the same hunk twice.

The four-step loop

1. Build a bounded, line-anchored packet from the patch so both reviewers see identical material.
2. Restrict findings to a small, fixed vocabulary so that matching is meaningful.
3. Bucket the two finding sets into consensus, contested, and singleton anchors.
4. Spend human reading time by bucket, with a time box attached to each.

Steps one and three are mechanical, and the two scripts below cover them. Step two is a convention that has to be agreed before the first run, and step four is the part that actually saves hours.

Step 1: a bounded, line-anchored packet

Feeding a raw diff to a reviewer invites anchor drift, because hunk headers are hard to map back to source. The packet builder below reads a unified diff, extracts the changed line ranges per file, and emits the full patch plus numbered excerpts around each range. A character budget keeps the packet small enough to stay inside whatever context the reviewer accepts.

#!/usr/bin/env python3
# review_packet.py - build a bounded, line-anchored review packet from a unified diff
import pathlib
import re
import sys

HUNK = re.compile(r'^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@')
CHAR_BUDGET = 24000
CONTEXT = 6


def changed_ranges(diff_text):
    files, current = {}, None
    for line in diff_text.splitlines():
        if line.startswith('+++ b/'):
            current = line[6:]
            files.setdefault(current, [])
        elif line.startswith('@@') and current:
            match = HUNK.match(line)
            if match:
                start = int(match.group(1))
                count = int(match.group(2) or 1)
                files[current].append((start, start + max(count - 1, 0)))
    return files


def excerpt(root, path, ranges):
    target = root / path
    if not target.exists():
        return ['(new file: no post-image to excerpt)']
    lines = target.read_text(encoding='utf-8', errors='replace').splitlines()
    keep = set()
    for start, end in ranges:
        keep.update(range(max(1, start - CONTEXT), min(len(lines), end + CONTEXT) + 1))
    out, previous = [], None
    for number in sorted(keep):
        if previous is not None and number != previous + 1:
            out.append('...')
        out.append(f'{number:>5} | {lines[number - 1]}')
        previous = number
    return out


def main(diff_path, repo_root):
    diff_text = pathlib.Path(diff_path).read_text(encoding='utf-8')
    root = pathlib.Path(repo_root)
    parts = ['## Patch (verbatim)', '```

diff', diff_text.strip(), '

```', '']
    budget = CHAR_BUDGET - sum(len(p) for p in parts)
    for path, ranges in changed_ranges(diff_text).items():
        block = '\n'.join(
            [f'## {path} (changed lines: {ranges})', '```

', *excerpt(root, path, ranges), '

```', '']
        )
        if len(block) > budget:
            parts.append(f'## {path}\n(omitted: exceeds remaining packet budget)')
            continue
        budget -= len(block)
        parts.append(block)
    sys.stdout.write('\n'.join(parts))


if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

The invocation is deliberately boring, because it has to be repeatable on a branch that changes hourly:

git fetch origin main
git diff origin/main...HEAD > /tmp/patch.diff
python3 review_packet.py /tmp/patch.diff . > /tmp/packet.md
wc -c /tmp/packet.md

The omitted-file marker matters more than it looks. When a packet is truncated, the reviewer should be told which files it never saw, so that a thin review of file nine is not read as a clean bill of health. Verify the packet on a patch with a renamed file before trusting it, since rename headers do not always produce a post-image path the script can open.

Step 2: a fixed finding vocabulary

Matching two reviews requires a shared label set, and a short one is enough. The taxonomy below covers most defects in small patches while keeping style out of the comparison:

• bug for wrong behaviour on a reachable path.
• error-handling for ignored or narrowed failures.
• resource for leaks, missing cleanup, unbounded growth.
• concurrency for shared state and ordering assumptions.
• test-gap for a changed branch no test exercises.
• perf for work that is quadratic or repeated unnecessarily.

Style and naming comments are excluded on purpose. They rarely agree across runs, they inflate singleton counts, and a maintainer reading a style singleton learns nothing about the patch.

Step 3: bucket the disagreement

The bucket script reads two JSON arrays of findings with file, line, and category fields, then matches anchors on the same file and category within a three-line window. Everything unmatched becomes a singleton attributed to one side, which is the set worth reading with fresh eyes.

#!/usr/bin/env python3
# bucket.py - split two reviewers' findings into consensus, contested, and singletons
import json
import sys

WINDOW = 3


def load(path):
    with open(path, encoding='utf-8') as handle:
        return {(row['file'], int(row['line']), row['category'].strip().lower())
                for row in json.load(handle)}


def bucket(left, right):
    pairs, taken = [], set()
    ordered = sorted(right)
    for finding in sorted(left):
        for index, candidate in enumerate(ordered):
            if index in taken:
                continue
            same_anchor = finding[0] == candidate[0] and finding[2] == candidate[2]
            if same_anchor and abs(finding[1] - candidate[1]) <= WINDOW:
                taken.add(index)
                pairs.append((finding, candidate))
                break
    left_only = [f for f in sorted(left) if all(f != pair[0] for pair in pairs)]
    right_only = [f for i, f in enumerate(ordered) if i not in taken]
    return pairs, left_only, right_only


def main(left_path, right_path):
    pairs, left_only, right_only = bucket(load(left_path), load(right_path))
    for first, second in pairs:
        print(f'CONSENSUS    {first[0]}:{first[1]} ~ {second[0]}:{second[1]}  [{first[2]}]')
    for finding in left_only:
        print(f'SINGLETON-A  {finding[0]}:{finding[1]}  [{finding[2]}]')
    for finding in right_only:
        print(f'SINGLETON-B  {finding[0]}:{finding[1]}  [{finding[2]}]')
    total = len(pairs) + len(left_only) + len(right_only)
    print(f'\n{len(pairs)} consensus of {total} anchors')


if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2])

For two finding sets where both reviewers flagged src/net/conn.c as error-handling six lines apart, plus one private finding each, the output takes this shape:

CONSENSUS    src/net/conn.c:214 ~ src/net/conn.c:220  [error-handling]
SINGLETON-A  src/net/pool.c:88  [resource]
SINGLETON-B  src/net/conn.c:301  [test-gap]

1 consensus of 3 anchors

The example above is illustrative rather than a production run, so compare the printed anchors against the packet before acting on them. A consensus count of one out of three is the useful signal here, not the individual findings.

Step 4: spend attention by bucket

Bucket	Reading	Action	Time box
Consensus	Two independent runs landed on the same file, category, and neighbourhood	Open the code, write a failing test, then decide whether the finding survives it	10 minutes
Contested	Same file, different category, or anchors more than three lines apart	Treat it as a specification question and read the surrounding function before the finding	15 minutes
Singleton	Only one side raised it, often outside a changed hunk	Check the anchor against the packet; drop anything outside a changed range immediately	5 minutes
Empty bucket	Neither reviewer flagged a changed hunk with a test	Add the test yourself before approving	15 minutes

The time boxes are the point of the exercise. A bucket report that does not change how long a maintainer spends on a patch has produced nothing but extra output.

Where free access changes the economics

Running two reviewers per patch only works if the second run costs nothing to authorize, which is where free model access matters. MonkeyCode offers free model access and a free server option, which makes the two-reviewer pass a default rather than a budget conversation. Pick two models available on the free tier and keep the pairing stable across patches, because rotating models turns the bucket counts into noise. The disclosure for this article is that it was prepared as part of MonkeyCode's product outreach, and the scripts above are provider-neutral and run without it.

The free server option is a reasonable home for the harness if a team wants each packet to be reproducible by the next reviewer. A checkout, one diff, and two JSON files need very little, and keeping the run off a laptop makes the packet addressable later. Confirm the current limits before scheduling anything against it, and keep the packet and bucket scripts in the repository either way, since they are the durable part.

Limitations and who should skip this

The loop is a triage tool, not evidence. Neither bucket report proves a defect exists, and consensus between two models that share training data is weaker than consensus between a model and a human. Patches that touch authentication, cryptography, or memory ownership need a human reviewer with domain context regardless of what the buckets say. The scripts assume a unified diff with standard +++ b/ headers, so patches from unusual diff drivers or binary-heavy changes will produce a thin packet.

Teams without a fixed finding vocabulary should skip step three entirely and run one reviewer with a careful reading pass. Anyone who cannot read the patched code well enough to judge a finding should also skip the loop, because a bucket report consumed without code reading is a more organized form of guessing. Small patches under roughly fifty changed lines usually do not repay the setup, and a single careful review is faster.

Closing

The reusable asset here is the bucket report, which turns two opinions into a short list of places where the patch is actually contested. Teams that already review patches this way can add the second reviewer through free model access and keep the packet and bucket scripts under version control. The loop only pays off when the report changes the order in which a maintainer reads the diff.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.