Field Notes: For 48 Hours I Had a Free Model Review Its Own Reviewing Rules

"AI promoted every developer to reviewer. Nobody tested the reviewer." That sentence from a recent discussion kept circling in my head while I reviewed pull requests last week. We trust AI to write review comments, but who reviews the reviewer? Free models are notorious for producing plausible but inconsistent judgments. So I spent 48 hours doing something slightly recursive: using a free model to evaluate the consistency of its own code review outputs. Here are the field notes.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Setup: One Template, Two Passes

I wanted to know whether the same review prompt, run twice on the same pull request, would give meaningfully similar feedback. If the model can't agree with itself, then its review comments are closer to random suggestions than trustworthy analysis.

I wrote a simple review template that asked for three things:

You are a senior backend reviewer. For this PR, return:
1. One list of blocking issues (max 3)
2. One list of non-blocking suggestions (max 3)
3. A verdict: APPROVE / REQUEST_CHANGES

Be specific. Quote line numbers.

Using MonkeyCode's free model access and a free server, I looped that template over a handful of recent PRs. For each PR I called the model twice with identical inputs and compared the outputs.

The script was deliberately boring:

import json
from difflib import SequenceMatcher

# Replace with your own call_llm() implementation using your provider's SDK
def review_once(prompt: str) -> dict:
    raw = call_llm(prompt, temperature=0.0)  # force determinism as much as possible
    return json.loads(raw)

prompts = [build_review_prompt(pr) for pr in my_small_sample]


def consistency_score(rev_a: dict, rev_b: dict) -> float:
    a_text = " ".join(rev_a["blocking"]) + " " + " ".join(rev_a["non_blocking"])
    b_text = " ".join(rev_b["blocking"]) + " " + " ".join(rev_b["non_blocking"])
    return SequenceMatcher(None, a_text, b_text).ratio()

for pr, prompt in zip(my_small_sample, prompts):
    first = review_once(prompt)
    second = review_once(prompt)
    score = consistency_score(first, second)
    print(f"{pr['id']}: {score:.2f} · verdict: {first['verdict']}/{second['verdict']}")

Yes, I forced temperature=0.0 and still saw variance. That was the first red flag.

What Broke: The Model's Own Rules Were Interpreted Differently

The scores ranged from a depressing 0.18 to a still-worrying 0.61. On one PR, the first pass said REQUEST_CHANGES because of a missing timeout; the second pass approved it while calling timeout handling "acceptable for this service." Same prompt. Same commit. Same "temperature" setting.

Digging deeper, I noticed the template allowed interpretation. "Quote line numbers" — sometimes quoted, sometimes not. "Blocking issues (max 3)" — one run returned two, the other returned one. The model wasn't checking the code carefully; it was guessing what I meant by each rule.

So I rewrote the template as a decision tree, forcing JSON and removing all adjectives:

{
  "type": "object",
  "properties": {
    "blocking": {"type": "array", "items": {"type": "string"}, "maxItems": 3},
    "verdict": {"enum": ["APPROVE", "REQUEST_CHANGES"]}
  },
  "required": ["blocking", "verdict"]
}

The second iteration improved consistency to 0.67–0.82. Still not great, but enough to be usable as a filtering step, not a final reviewer.

What I'd Repeat Tomorrow

• Restrict the reviewers' vocabulary. Every free-form adjective turned into bias. "Critical", "important", "minor" — the model's mood changed what those meant run to run.
• Ask for only one output type. JSON with enums beats prose every time. You can't get REQUEST_CHANGES vs Approved vs "lgtm" in the same field if you don't allow free text.
• Run every PR twice and diff. Even with the JSON schema, two passes caught roughly 20% more real bugs than a single pass. Doubling the model call is cheap; merging the two lists into a final human review is not much extra work.

Limitations (Read Before Copying My Process)

My sample was small: six PRs, each reviewed twice, all from my own repos. I didn't randomize order, didn't test different prompt phrasings, and didn't control for the underlying model's updates mid-experiment. The free tier may route to different model versions at different times — I didn't verify that. Also, temperature=0 isn't a guarantee of determinism on every provider; some servers quantize or use sampling quirks.

If you're reviewing security-sensitive or legal-bound code, do not rely on this technique as a gate. Use a free model to pre-sort obvious issues, but keep a human accountable for the verdict.

The One Soft CTA I'll Allow Myself

If you want to run this consistency harness against your own PRs, MonkeyCode's free model access and free server make the experiment cheap to replicate. But the point isn't the product — it's the habit of testing the reviewer before trusting the review. My next 48 hours will be spent running this same loop against another provider, just to see which free model lies to me the least.

The reviewer is only as good as the rules you give it. And if you don't test those rules, the model will happily review them for you.