Two Cheap Reviews That Must Agree: A Consensus Gate for AI-Assisted OSS Patches

A patch is only as good as the review it survives. Free model access makes review cheap, and cheap review produces noise. The fix is agreement between two independent passes, not trust in one.

This workflow runs two review passes over the same diff. Only findings both passes flag survive. Everything else is logged and dropped.

1. The failure mode: one pass is a coin flip

A single reviewer with a single prompt returns confident comments. Some are correct, and many are not. Temperature, prompt wording, and truncation all move the output.

Anchoring findings to diff lines fixes citation, not correctness. A comment can cite line 42 and still be wrong about line 42. Consensus attacks the other axis. Two passes must independently point at the same code.

2. Build a deterministic input pack first

No model touches this step. The pack is a hashable snapshot of exactly what reviewers see. Reproducibility starts here, before any prompt.

#!/usr/bin/env python3
"""Build a deterministic review input pack. No model calls."""
import hashlib, json, subprocess, sys
from pathlib import Path

def run(*cmd):
    return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout

def main(base, repo, out):
    diff = run("git", "-C", repo, "diff", "--unified=3", base)
    files = run("git", "-C", repo, "diff", "--name-only", base).split()
    log = Path(out).with_suffix(".test.log")
    pack = {
        "base": base,
        "diff_sha256": hashlib.sha256(diff.encode()).hexdigest(),
        "files": files,
        "diff": diff,
        "test_log_tail": log.read_text()[-4000:] if log.exists() else "",
    }
    Path(out).write_text(json.dumps(pack, indent=2))
    print(f"pack: {len(files)} files, diff {len(diff)} bytes")

if __name__ == "__main__":
    main(*sys.argv[1:4])

Run it against the merge base, not against the branch tip. The hash goes into every downstream log line.

python3 pack.py origin/main ../patched pack.json

3. Ask two different questions, not the same question twice

Two passes with identical prompts measure sampling noise. Two passes with disjoint prompts probe different failure classes. One reviewer owns behavior, and the other owns tests.

#!/usr/bin/env python3
"""Two review passes over one pack. Example only; the adapter is a placeholder."""
import json, os, sys
from openai import OpenAI  # any OpenAI-compatible endpoint

PROMPTS = {
    "behavior": "Flag behavior changes the diff does not describe. Cite file and line range.",
    "tests": "Flag missing or weakened test coverage. Cite file and line range.",
}
SCHEMA = '{"file":..., "start_line":..., "end_line":..., "category":..., "claim":...}'

def ask(client, model, prompt, pack):
    msg = f"{prompt}\nReturn JSON lines matching {SCHEMA}.\n\n{pack['diff'][:20000]}"
    r = client.chat.completions.create(
        model=model, messages=[{"role": "user", "content": msg}])
    for line in r.choices[0].message.content.splitlines():
        line = line.strip().strip(",")
        if line.startswith("{"):
            try:
                yield json.loads(line)
            except json.JSONDecodeError:
                continue

def main(pack_path, out_path):
    pack = json.load(open(pack_path))
    client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["MODEL_KEY"])
    with open(out_path, "a") as f:
        for name, prompt in PROMPTS.items():
            for item in ask(client, os.environ["MODEL"], prompt, pack):
                f.write(json.dumps({"reviewer": name, "diff": pack["diff_sha256"], **item}) + "\n")

if __name__ == "__main__":
    main(*sys.argv[1:3])

The snippet is illustrative and has not been executed as written. The model name stays in an environment variable on purpose. Hardcoded model strings age badly.

4. Merge on overlap, not on wording

Two reviewers rarely phrase a finding the same way. They do point at the same lines. Overlap is the signal, and phrasing is noise.

#!/usr/bin/env python3
import json, sys
from collections import defaultdict

def overlap(a, b):
    return (a["file"] == b["file"]
            and a["start_line"] <= b["end_line"]
            and b["start_line"] <= a["end_line"])

items = [json.loads(l) for l in open(sys.argv[1])]
by_rev = defaultdict(list)
for it in items:
    by_rev[it["reviewer"]].append(it)

first, second = list(by_rev.values())[:2]
kept = [x for x in first for y in second
        if overlap(x, y) and x["category"] == y["category"]]

print(json.dumps({"raw": len(items), "kept": len(kept),
                  "keep_rate": round(len(kept) / max(len(items), 1), 3)}, indent=2))
for x in kept:
    print(x["file"], x["start_line"], x["category"], x["claim"][:80])

Set the overlap tolerance deliberately. A tolerance of one line suits refactors. A tolerance of zero suits single-statement fixes.

5. Where free model access and a free server fit

A second pass is only free if the first pass is free. Metered calls push teams toward one expensive review, which recreates the original problem. Free model access removes that trade-off, so two passes become the default rather than a luxury.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Both are availability claims from the operator. Quotas and limits change, so check the current figures on the product site instead of trusting any blog post, including this one.

The free server matters more than the models here. The review shift runs on a disposable box with a clean toolchain. Nothing depends on a laptop having the right Python version.

export REPO_URL=<repository-url>
ssh box "git clone --depth 1 $REPO_URL work && cd work && git worktree add ../patched $BRANCH"
ssh box "cd work && python3 pack.py origin/main ../patched pack.json"
ssh box "cd work && BASE_URL=$BASE_URL MODEL=$MODEL MODEL_KEY=$MODEL_KEY \
  python3 review.py pack.json findings.jsonl && python3 consensus.py findings.jsonl"

One shift produces one findings.jsonl. The box is deleted afterward. Teams that want to try the free tier can start with a single repository and a single afternoon.

6. Measure the gate before trusting it

Consensus is a filter. Filters need a known precision and recall. Build a labeled set from twenty merged findings in the repository history. Mark each one as real or noise.

Run three configurations over that set: single pass, line-anchored pass, and consensus. Record counts, not impressions.

Configuration	Kept	Real	Precision	Noise found
Single pass	—	—	—	—
Anchored only	—	—	—	—
Consensus	—	—	—	—

Empty cells are intentional. The numbers depend on the repository, model, and prompts, and any published figure would be someone else's baseline. Two hours of labeling buys a defensible threshold.

7. Limitations and who should skip this

The two passes often share a model. Shared training data means shared blind spots. Consensus raises precision and lowers recall, so a real finding caught by one reviewer gets dropped.

Labeling costs time that small projects do not have. Repos with fast human review gain little from a filter. Security review belongs to humans and dedicated tooling, not to a consensus gate.

Skip this workflow for typo fixes, dependency bumps, and single-line changes. The pack step alone costs more than the review.

Start with one repository, twenty labels, and one remote shift. Extend the gate only after the precision table shows a real gain.