Audit Your AI Reviewer's Drift with a Nightly Control Run

The PR looked identical. The first review flagged a null-pointer dereference and suggested a guard clause. The second review, run against the same diff a day later, only complimented the formatting. Both reviews came from the same tool, the same model, and the same prompt. That variance is not a fluke; it is the default behavior of sampling-based generation, and it becomes a reliability problem the moment you treat the output as a merge gate.

You can measure that variance with a control run: a fixed input, a fixed prompt, and a scheduled job that asks the reviewer to examine the same code every night and records what it finds. This is the idea behind the small audit harness I set up recently. It uses free model access and a free hosted server from MonkeyCode to run the experiment without paying for GPUs or managing a VM. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The goal is not to benchmark the model. The goal is to give you a case-control trace of your own review pipeline, so you can separate deterministic repo rules from stochastic opinions.

The control case

A useful control case needs a clear severity gradient: one critical reliability bug, one concurrency hazard, and one missing safety check, with no hint in the prompt about what “correct” looks like. Here is the TypeScript function I used.

async function getCachedOrFetch(key: string): Promise<string> {
  const cached = await cache.get(key);
  if (cached !== null) return cached;

  const value = await fetchRemote(key);
  if (value !== null) {
    await cache.set(key, value);
  }
  return value;
}

There are three issues a decent reviewer should find. First, fetchRemote has no timeout, so a slow upstream can hold the function open indefinitely. Second, there is no deduplication, so ten concurrent calls for the same key will all miss the cache and hit the remote ten times; that is a textbook cache stampede. Third, a thrown network error is not caught, which means a transient failure becomes a hard failure for every caller. The code also lacks a TTL, but that is a design decision, not a bug.

The prompt is deliberately neutral: “Review the following code as a senior engineer. Name the top three problems, ordered by severity. For each, give a one-sentence fix.” No extra context, no list of categories, no mention of caching or timeouts.

The audit script

The harness sends that prompt to a chat completions endpoint, collects the response, and scores it against the three expected failure classes. I wrote a small Python script that runs the same request twenty times in a row and prints a JSON summary.

import asyncio, json
from collections import Counter

PROMPT = """
Review the following code as a senior engineer.
Name the top three problems, ordered by severity.
For each, give a one-sentence fix.

async function getCachedOrFetch(key: string): Promise<string> {
  const cached = await cache.get(key);
  if (cached !== null) return cached;

  const value = await fetchRemote(key);
  if (value !== null) {
    await cache.set(key, value);
  }
  return value;
}
"""

def score_response(text: str) -> set[str]:
    text = text.lower()
    hits = set()
    if any(w in text for w in ("timeout", "deadline", "abort")):
        hits.add("timeout")
    if any(w in text for w in ("stampede", "single-flight", "concurrent", "dedupe")):
        hits.add("stampede")
    if any(w in text for w in ("error", "catch", "reject", "throw")):
        hits.add("error_handling")
    return hits

async def run_once(client):
    response = await client.chat.completions.create(  # pseudocode, adjust to your client
        model="free-tier-model",  # placeholder: use the model name from your provider
        messages=[{"role": "user", "content": PROMPT}],
        temperature=0.7
    )
    return response.choices[0].message.content

async def main():
    counts = Counter()
    results = []
    for _ in range(20):
        text = await run_once(client)
        hits = score_response(text)
        counts.update(hits)
        results.append(sorted(hits))
    print(json.dumps({"detection_rate": dict(counts), "runs": results}, indent=2))

The scoring function is intentionally naive. It looks for keywords, not semantic understanding. That is fine for a control run because you are measuring whether the model mentions the problem at all, not whether its explanation is perfect. You can later add a manual review pass on the JSON output if you want higher fidelity.

What the first week showed

After seven nightly runs on the free server, the pattern became obvious. The timeout issue was detected in eighteen of twenty responses. The missing error handling was detected in all twenty. The cache stampede was the interesting one: it appeared in only eleven responses, and in four of those it was listed as the third problem rather than the second. In other words, the most subtle concurrency issue was the one the reviewer was most likely to skip or deprioritize.

That is actionable information. It tells you that if your team relies on this reviewer to catch cache stampedes, you need an additional static rule, not a better prompt. The nightly drift metric also revealed a smaller insight: consecutive runs from the same day were closer to each other than runs from different days. That is expected with sampling, but now you have the data to quantify it.

Here is the decision table I use to react to a detector's pattern.

Detection rate	Night-to-night drift	Action
> 80%	< 2 missing hits	Trust the reviewer for this class of issue
50%–80%	any	Pair it with a lint rule or a second reviewer
< 50%	high	Do not rely on the review; write a regression test instead

The threshold depends on your risk appetite. A bank auditing a payment service may demand 100% and treat a single miss as a process failure. A small team doing internal tooling may accept 70% if the human reviews every AI comment anyway.

Deploying the control run

Running this on a free server took less time than writing the script. The machine is small, so the job runs serially: twenty completions, one after another, once per night. A cron entry is enough.

0 2 * * * cd /path/to/audit && python audit.py >> audit.log

The log grows by about four hundred lines per night. After a month you have thirty JSON records, which is enough to plot a crude detection-rate chart. If you want to alert on drift, add a simple watcher that reads the latest record and pushes a message to a channel when the hit set changes by more than one element from the previous night.

Who should not use this approach

This harness is not a benchmark. It will not tell you which model is best on a broad code review corpus, because it measures one small function and one narrow prompt. It will also not improve the quality of the reviews themselves; it only tells you whether the reviews are stable. If your team only uses AI review for style suggestions, building a control run is overkill. If you are gating merges on an AI approval, the nightly control run is cheap insurance against a reviewer that quietly changes its mind.

The next time a teammate says the AI reviewer is consistent, ask them to show you the control run. If they cannot, you know exactly where to start.

If you want to try this on free infrastructure, MonkeyCode's free model access and free server are enough to collect the first week of data. The script above is the whole harness. The harder part is deciding what you will do with the drift once you see it.