A Mutation-Based Regression Test for Your AI Code Reviewer

AI code assistants have turned every developer into a reviewer, and the uncomfortable side effect is that our review process itself has no test suite. A model gets updated, a prompt gets tweaked, and suddenly the comments stop catching the one bug that matters. That drift needs to be detected before it reaches production, so this article proposes a cheap, repeatable regression harness.

The harness measures how well your AI reviewer detects known defects in a fixed set of mutated diffs. It runs on a free server, uses free models, and returns a single detection rate that you can chart over time. Small teams that already depend on AI-assisted review will find this useful because it gives them evidence instead of vibes.

A hidden assumption behind every AI reviewer

Every AI code review tool ships with an implicit promise: the model will keep being useful. In practice, usefulness drifts when the underlying model changes, when the system prompt is edited, or when the provider adjusts its default behavior. Human reviewers drift too, but they leave comments you can discuss. Model behavior is a black box until you measure it.

Regression testing for code works by keeping a set of failing examples and re-running them against new changes. The same idea works for reviewers. You keep a set of small diffs, each containing one planted defect, and you ask your reviewer to find the defect. The ratio of found defects to total planted defects is your reviewer's detection rate.

What you need to build it

Building this harness takes four pieces.

• A set of mutation fixtures, which are small diffs with one known bug.
• A script that runs the reviewer against each fixture.
• A free server to schedule the run and store results.
• A simple dashboard or log file to track detection rate over time.

We used MonkeyCode for the reviewer, because it offers free models and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The same method works with any other tool that exposes a CLI or API, so you are not locked into a single vendor.

Designing the mutation fixtures

Each fixture is a tiny code change that introduces a real defect. Good fixtures are small enough to read in seconds but specific enough that missing them matters. We currently keep ten fixtures in a JSON file, with one fixture per language we care about.

{
  'fixtures': [
    {
      'id': 'off-by-one',
      'file': 'fixtures/off_by_one.py',
      'diff': 'def first(items): return items[1]',
      'expected': 'off by one in indexing'
    },
    {
      'id': 'missing-null-check',
      'file': 'fixtures/null_check.py',
      'diff': 'def trim(input): return input.strip()',
      'expected': 'possible None input'
    }
  ]
}

The expected field is the human-labeled defect. During the evaluation, the script checks whether the review output mentions that label or a close synonym.

The evaluation script

The core script is deliberately small so that anyone can extend it. The command below is pseudocode because the exact CLI may change; check the current MonkeyCode documentation before copying it.

import json
import subprocess

FIXTURES = 'fixtures.json'
REVIEW_OUTPUT = 'latest_review.json'

def run_review(diff_path):
    subprocess.run(
        ['monkeycode', 'review', '--diff', diff_path,
         '--output', REVIEW_OUTPUT],
        check=True
    )
    with open(REVIEW_OUTPUT, 'r') as f:
        return json.load(f)

def detect(fixture, review):
    text = json.dumps(review).lower()
    return fixture['expected'].lower() in text

def detection_rate(fixtures):
    caught = [f for f in fixtures if detect(f, run_review(f['file']))]
    return len(caught) / len(fixtures)

with open(FIXTURES) as f:
    fixtures = json.load(f)['fixtures']

print('current detection rate', detection_rate(fixtures))

The script returns a number between 0 and 1. Run it once after setting up the fixtures to get a baseline. Every later run can be compared against that baseline.

Tracking changes with a decision table

A single number is not enough; you need a rule for acting on it. Our team uses this decision table.

Detection rate change	Action
Drop of less than 5 percent	Accept the change but log it
Drop of 5 to 20 percent	Investigate the failing fixtures and adjust prompts
Drop of more than 20 percent	Block the model or prompt update, then rerun
Increase of more than 5 percent	Update baseline and celebrate quietly

This table is deliberately conservative. A 20 percent drop may be acceptable for a personal project, but it is dangerous for a team that trusts the reviewer in production.

Scheduling it on a free server

You do not need a paid CI plan for a daily check. The test runs in less than a minute for ten fixtures, so a free server with a cron job is enough. Here is a sample cron entry.

0 6 * * * cd /home/user/review-harness && python monitor.py >> history.log

The free server stores the history file. If you want a visual chart, point a static site generator at that log file. No database is required.

Limitations and who should skip this

This harness does not measure false positives. A reviewer that never flags anything will have a perfect detection rate if the planted defects are not in its threat model. To catch false positives, you would need a second fixture set of clean diffs and a check that the reviewer stays silent on them.

The harness also cannot detect deep architectural problems. It only checks known, localized defects. If your team needs review guidance for large refactors, this method gives you a weak signal.

Skip this workflow if your code cannot leave your network. The reviewer needs access to the diff, so it is not suitable for regulated environments where code sharing is prohibited. Also skip it if your team is not willing to act on the numbers, because a dashboard nobody reads is just another dashboard.

Where to start

Pick ten real bugs from your recent pull requests and turn them into fixtures. Run the script once to get a baseline, then automate it on a free server. When the detection rate drops by more than your chosen threshold, you will know before your users do.

If you maintain an AI review workflow, add a mutation fixture set before your next model upgrade. The free models and free server from MonkeyCode give you enough room to try this without touching your budget. If you build something similar, share your decision thresholds; the community still needs more public numbers.