Your AI Reviewer Disagrees With Itself. Measure the Drift.

Send the same pull request to an AI reviewer twice.

Do you get the same verdict? Probably not. Nobody wants to admit that.

I stopped guessing. I wrote a tiny probe that does two reviews of one diff. Then I compared the finding lists.

The two reviews disagreed. Not once. Constantly.

This is not noise. This is the measurement your review pipeline was missing.

What "reviewer drift" actually means

Drift is the gap between two responses to the same input.

Free model endpoints are samplers, not lookup tables. Same prompt, same diff, different tokens. Most teams design their CI gate as if that weren't true.

They do one review, extract findings, and act like the verdict is stable.

It isn't. And the cost of pretending is flaky review gates, phantom regressions, and silent misses.

The myth list

Here are four claims I hear in every workflow review. Here is what the probe shows.

Myth 1: Same input, same review

"With temperature 0, the reviewer is deterministic."

I ran the probe with the settings most teams copy from each other's docs. The two outputs still differed.

Reason: free endpoints often run behind shared infrastructure. Load balancing, batching, and provider-side sampling all introduce variance.

Your zero temperature does not control the server. It only controls one sampling knob.

Evidence: two runs, one diff, different findings. Happens more often than you want.

Myth 2: A second review catches everything the first one missed

"If one pass misses a bug, the second pass will find it."

Sometimes yes. Often no.

The second review misses different things than the first one. The union of two passes is bigger than one pass. That part is real.

But the second pass does not see what the first one saw. It sees its own new blind spot.

Evidence: my overlap rate between two runs lands between 60% and 90%. A whole slice of findings exists in only one run.

Myth 3: More findings means better quality

"The run with 14 findings is better than the run with 9."

No. A reviewer that finds 14 today and 6 tomorrow is not thorough. It is unstable.

A stable reviewer finds 9 and 9. You can build a workflow around stable behavior.

You cannot build anything around randomness.

Evidence: the probe counts findings per run. The count alone tells you nothing about correctness.

Myth 4: Drift is a model problem, not a workflow problem

"We cannot fix variance, so we ignore it."

Wrong. Drift is a workflow signal. It tells you where your review pipeline needs redundancy.

You cannot make the model deterministic. You can make your process resilient to variance.

That is the part engineers control.

Reproducible artifact: a 15-line drift probe

The probe sends one diff to an OpenAI-compatible endpoint twice. It extracts the findings, counts them, and computes the overlap.

Save this script as review_drift.py.

"""review_drift.py: measure how unstable one AI code review is."""
import argparse
import os
import requests

SYSTEM = (
    "You are a senior code reviewer. "
    "List findings as '- [severity] path:line message'. "
    "Answer only."
)
USER = "Review this diff:\n\n{patch}"


def review(patch, api_url, api_key, model):
    resp = requests.post(
        api_url,
        headers={"Authorization": f"Bearer {api_key}"},
        json={
            "model": model,
            "messages": [
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": USER.format(patch=patch)},
            ],
        },
        timeout=120,
    )
    resp.raise_for_status()
    content = resp.json()["choices"][0]["message"]["content"]
    return [line.strip() for line in content.splitlines() if line.strip().startswith("-")]


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("patch")
    parser.add_argument("--api-url", required=True)
    parser.add_argument("--api-key", default=os.environ.get("FREE_MODEL_API_KEY"))
    parser.add_argument("--model", default=os.environ.get("FREE_MODEL_NAME"))
    args = parser.parse_args()

    patch = open(args.patch, encoding="utf-8").read()
    run_one = review(patch, args.api_url, args.api_key, args.model)
    run_two = review(patch, args.api_url, args.api_key, args.model)

    print(f"run 1: {len(run_one)} findings")
    print(f"run 2: {len(run_two)} findings")

    set_one, set_two = set(run_one), set(run_two)
    overlap = len(set_one & set_two)
    union = len(set_one | set_two)
    if union:
        print(f"exact-line overlap: {overlap}/{union} (Jaccard {overlap / union:.2f})")
    else:
        print("no findings in either run")


if __name__ == "__main__":
    main()

Generate the diff under review:

git diff HEAD~1 -- . ':(exclude)*.lock' > sample.patch

Then run the probe:

python review_drift.py sample.patch \
  --api-url "$FREE_MODEL_URL" \
  --api-key "$FREE_MODEL_KEY" \
  --model "$FREE_MODEL_NAME"

The script is deliberately small. The point is not sophistication. The point is repetition: same input, two runs, one honest comparison.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

How to read the output

Use the Jaccard score as your review-confidence budget.

• 0.90 and above: stable reviewer. Use a single pass for cheap feedback.
• 0.60 to 0.90: usable opinion, unstable gate. Never fail a build on one review.
• Below 0.60: treat the reviewer as a suggestion engine. Human review or a second run is mandatory.

Your numbers will differ. Different free endpoints, different models, different prompts. That is fine.

The method matters more than the exact score.

Why free endpoints change the workflow math

Drift testing costs twice as much as normal reviewing. That is the old math.

Free model access flips the equation. A second review stops being a cost decision and becomes a confidence decision.

I used the free model access and free server option from MonkeyCode for this probe. The open question is not "can I afford two runs?" It is "is my review pipeline stable enough to trust one run?"

That is a better question to ask before every merge.

Limitations and who should skip this

Drift is not quality. A stable reviewer can be confidently wrong.

To measure quality you need labeled test cases with known bugs. This probe measures predictability, not accuracy.

Free endpoints are often multi-tenant. Your drift can change by hour and by server. Do not publish your score as a universal benchmark.

Skim this whole approach if your review process already uses human sign-off on every finding. Then the variance costs you nothing.

Also skip it if you only need one honest opinion from a second engineer. This is for teams that made a machine part of the merge gate.

The corrected mental model

Stop treating your AI reviewer as a fact engine.

Treat it as a noisy team member. Give it redundant sampling, set a confidence budget, and remember what we can actually verify:

• The second review will disagree with the first.
• The disagreement is measurable.
• The measurement is the part you can ship.

When you stop trusting the single verdict, the review stops being a coin flip. It becomes a monitoring problem. And monitoring problems can be automated, alerted, and fixed.

No response is a response, they say. Well, two responses are the only review worth trusting.