Score Your AI Reviewer Before You Trust It: A Three-Persona Drill

An AI reviewer never sets its own quality bar. It answers, you decide, and the conversation usually ends there. This week's DEV discussion made the side effect visible: AI promoted everyone to reviewer, while the reviewer itself never went through a single test.

That gap is cheap to close. The drill below is a small experiment built from one diff. Three personas interview that diff, each answer gets a score, and the result is a number you can attach to your pull request.

The drill needs a chat endpoint with a free model. MonkeyCode's free tier is one practical route: free models, a free server option, and, at draft time, a 10M free token allowance that covers many full runs of this probe set. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script is endpoint-agnostic, so the same workflow works against any chat-completions-compatible server.

One diff, three interviews

Pick a real pull request from your current branch. Keep it small — a few hundred lines at most. Small diffs make false AI confidence easy to spot, because your own baseline is still within reach.

git fetch origin
git diff origin/main...HEAD > patch.diff
git log --oneline --first-parent -10 > context.txt
sed -n '1,80p' README.md >> context.txt
wc -l patch.diff

The context file is the map. A reviewer without context pattern-matches on syntax; a reviewer with two thousand lines of context loses the signal. Ten commits and eighty lines of README give the model just enough to check claims. Anything larger buries the signal.

Three personas, three questions

Each persona has one job. The gatekeeper decides what blocks the merge. The test-designer predicts what breaks first. The contradiction-hunter looks for claims the repo itself can refute.

The gatekeeper prompt is strict: return two to four issues, cite a diff line for each, and name the repo rule it breaks — or state that no rule exists. The test-designer prompt forces concreteness: predict the first failing scenario and write the test that proves it. The contradiction-hunter prompt asks the model to compare every claim against the context file.

Save the probe script as review_score.py.

#!/usr/bin/env python3
# Score an AI review: three personas, one diff, one scorecard.
import json, os, sys, urllib.request

PATCH = sys.argv[1] if len(sys.argv) > 1 else 'patch.diff'
CTX = sys.argv[2] if len(sys.argv) > 2 else 'context.txt'

diff = open(PATCH).read()
ctx = open(CTX).read()

BASE = os.environ.get('MC_BASE_URL', 'http://localhost:8080/v1')
MODEL = os.environ.get('MC_MODEL', '')
KEY = os.environ.get('MC_API_KEY', '')

PERSONAS = {
    'gatekeeper': (
        'You decide if a merge is blocked. Return 2-4 issues. '
        'For each issue, cite a diff line and the repo rule it breaks, '
        'or state no rule found.'
    ),
    'test-designer': (
        'This diff breaks something. Predict the first failing scenario '
        'and write one test that proves it, with a filename.'
    ),
    'contradiction-hunter': (
        'Flag every claim in the diff that the context file contradicts '
        'or cannot confirm. Be explicit.'
    ),
}

os.makedirs('.review-drill', exist_ok=True)

for name, system in PERSONAS.items():
    payload = {
        'model': MODEL,
        'temperature': 0.2,
        'messages': [
            {'role': 'system', 'content': system},
            {'role': 'user', 'content': 'Context:' + chr(10) + ctx + chr(10) + chr(10) + 'Diff:' + chr(10) + diff},
        ],
    }
    headers = {'Content-Type': 'application/json'}
    if KEY:
        headers['Authorization'] = 'Bearer ' + KEY
    req = urllib.request.Request(
        BASE.rstrip('/') + '/chat/completions',
        data=json.dumps(payload).encode(),
        headers=headers,
    )
    with urllib.request.urlopen(req) as resp:
        answer = json.load(resp)['choices'][0]['message']['content']
    path = '.review-drill/' + name + '.md'
    open(path, 'w').write(answer)
    print(f'wrote {path}')

Run it, then read the three files before you touch the code.

python3 review_score.py patch.diff context.txt

The scorecard

Read each answer three times: once for location, once for evidence, once for next steps. Then score.

Criterion	2 points	1 point	0 points
Precision	cites a diff line	names the file only	general advice
Evidence	confirmed by the context file	plausible best practice	invented rule
Action	concrete patch or test	vague suggestion	praise or filler

Six points is a perfect review. Apply the verdict table next.

Total	Verdict
0-2	do not forward
3-4	verify every item, then act
5-6	useful, but route through a human anyway

The verdict table is the part most teams skip. They paste AI comments into the PR and wait. The scorecard forces you to state, in one line, why this answer earned forwarding.

Three signals that the reviewer is guessing

The first signal is empty precision: every sentence is plausible, none of them cite a line. That is not a review, it is a summary. The second signal is a citation to unchanged code — a line the diff never touched. That is pattern-matching, not reading. The third signal is a confident rule that the context file neither states nor implies. Confidence makes it worse, not better.

When you see these signals, open the file and look at the line yourself. No forwarding until you can repeat the rule from memory.

Limits and who should skip this

The scorecard tests specificity, not correctness. A precise, well-cited comment can still be technically wrong. It also generalizes poorly across domains: a database migration needs different rubric rows than a UI patch. Adjust the criteria when the diff type changes.

Free allowances shift. Verify the current terms before you build a workflow on top of them. And respect data rules: never send proprietary code to an external endpoint. If export controls are strict, skip hosted routes entirely and run a self-hosted server.

Teams that already require citations in review tools do not need this drill — they can demand citations instead of scoring them. Everyone else gets value from one run: the scorecard shows how much of the AI answer is evidence and how much is pattern.

The review you can actually trust

One diff, three interviews, six points. That is the whole experiment. Run it on your next small PR, keep the scorecard in the description, and the next AI comment either arrives pre-verified or does not arrive at all.

If you want to run the drill on free models, MonkeyCode's free server option is the fastest setup I know.