Your AI Code Reviewer Needs a Test Suite Too

🤖 The Problem

You plugged an LLM into your PR pipeline. It leaves comments about SQL injection, N+1 queries, missing null checks. Everyone's thrilled for about two sprints, and then someone notices it stopped catching the exact same bug pattern it used to flag. A prompt got tweaked. A model version got bumped upstream. Someone added "be more concise" to the system prompt and it started skipping the security section entirely.

Nobody noticed because nobody was checking. The AI reviewer is running in production, making judgment calls on every PR in your org, and it has zero test coverage. That's the part that should bother you more than it probably does.

🕳️ Why This Gap Exists

We test the code the AI reviews. We don't test the reviewer itself, because it feels fuzzy — "it's an LLM, how do you even assert against prose?" But that's a cop-out. You don't need to assert on exact wording. You need to assert on behavior: given a diff with a known bug, does the reviewer flag it, in the right file, with the right severity, without three false positives burying the real issue?

That's a testable claim. It just requires building fixtures the same way you'd build fixtures for any other system with non-deterministic-ish output — like testing a search ranking algorithm or a fraud-detection model.

🧪 Building a Golden-Diff Suite

The core idea: collect real PRs (or synthetic ones) with known, labeled defects, and treat them like golden files. Run each one through your reviewer, then assert the output contains — or doesn't contain — specific findings.

Here's a minimal structure using Python and pytest, since most review pipelines are just a script wrapping an LLM call:

python

reviewer/client.py

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

SYSTEM_PROMPT = open("prompts/review_system.md").read()

def review_diff(diff_text: str) -> str:
response = client.chat.completions.create(
model="gpt-4.1",
temperature=0,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Review this diff:\n\n{diff_text}"},
],
)
return response.choices[0].message.content

Note temperature=0. You're not eliminating nondeterminism, but you're minimizing it — this matters a lot when you're about to assert against the output.

📁 Structuring the Fixtures

Each fixture is a known-bad diff plus a manifest describing what should get flagged:

fixtures/
sql_injection_raw_query/
diff.patch
expected.yaml
missing_await_async_call/
diff.patch
expected.yaml
hardcoded_secret_in_config/
diff.patch
expected.yaml

yaml

fixtures/sql_injection_raw_query/expected.yaml

must_flag:

• category: security file: app/db/queries.py line_range: [42, 45] keywords: ["injection", "parameteriz", "sanitiz"] must_not_flag:
• category: style keywords: ["variable naming"] max_total_comments: 4

The must_not_flag block matters as much as must_flag. A reviewer that comments on everything technically "catches" every bug, but it's useless noise. You want to pin down both precision and recall.

✅ Writing the Assertions

Since the output is prose, not structured data, you have two options: force structured output (JSON mode, function calling) or parse loosely with keyword/semantic matching. I'd push hard for structured output — it makes the whole test suite dramatically less brittle.

python

reviewer/schema.py

from pydantic import BaseModel

class Finding(BaseModel):
category: str
file: str
line_start: int
line_end: int
message: str
severity: str

class ReviewResult(BaseModel):
findings: list[Finding]

python

tests/test_golden_diffs.py

import yaml
import pytest
from pathlib import Path
from reviewer.client import review_diff
from reviewer.schema import ReviewResult

FIXTURE_DIR = Path("fixtures")

def load_fixtures():
for folder in FIXTURE_DIR.iterdir():
diff = (folder / "diff.patch").read_text()
expected = yaml.safe_load((folder / "expected.yaml").read_text())
yield pytest.param(diff, expected, id=folder.name)

@pytest.mark.parametrize("diff,expected", load_fixtures())
def test_reviewer_catches_known_bug(diff, expected):
raw = review_diff(diff)
result = ReviewResult.model_validate_json(raw)

for must in expected.get("must_flag", []):
    matches = [
        f for f in result.findings
        if f.category == must["category"]
        and f.file == must["file"]
        and any(kw in f.message.lower() for kw in must["keywords"])
    ]
    assert matches, f"Reviewer missed expected finding: {must}"

for forbidden in expected.get("must_not_flag", []):
    matches = [
        f for f in result.findings
        if any(kw in f.message.lower() for kw in forbidden["keywords"])
    ]
    assert not matches, f"Reviewer raised noise it shouldn't have: {forbidden}"

max_comments = expected.get("max_total_comments")
if max_comments is not None:
    assert len(result.findings) <= max_comments

This is a regression suite in the truest sense: every time someone edits the system prompt, swaps the model, or adjusts temperature, this runs and tells you exactly what broke. "Prompt change reduced recall on SQL injection cases from 100% to 60%" is a real, actionable CI failure — not a vibe.

🔄 Running It in CI

The expensive part is the LLM calls, so don't run this on every commit to every branch. Run it:

• On any PR that touches prompts/, reviewer/, or model config
• Nightly, to catch silent drift from provider-side model updates
• Before promoting a new prompt version to production, as a hard gate

yaml

.github/workflows/reviewer-regression.yml

name: reviewer-regression
on:
pull_request:
paths:
- "prompts/"
- "reviewer/"
schedule:
- cron: "0 6 * * *"
jobs:
golden-diffs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: pip install -r requirements.txt
- run: pytest tests/test_golden_diffs.py -v
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

⚖️ Trade-offs Worth Naming

This isn't free, and it's not perfectly deterministic even with temperature=0 — different model versions can still shift slightly. A few honest trade-offs:

• Fixture rot: real-world bug patterns evolve. Budget time to add new fixtures whenever a bug slips past the reviewer in production — that miss becomes your next test case.
• Cost: a suite of 50 fixtures run nightly against GPT-4-class models adds up. Consider a cheaper model for the regression suite if it's representative enough, and reserve the expensive model for prod.
• False confidence: passing the golden-diff suite doesn't mean the reviewer is good, only that it hasn't regressed on the specific patterns you've thought to encode. Treat it as a floor, not a ceiling.
• Structured output constraints: forcing JSON schemas can sometimes make models slightly less thorough in free-form reasoning. Worth A/B testing structured vs. prose output against your fixture set before committing.

None of these are reasons to skip this. They're reasons to scope it like any other test suite — start with the five bug patterns that have bitten you hardest in production, not fifty hypothetical ones.

🚀 Wrap Up

If your AI reviewer has opinions about your code quality, it deserves the same scrutiny you'd apply to any other piece of logic sitting between a developer and a merge button. A golden-diff suite is cheap to start — a handful of real bugs pulled from your git history and a YAML file describing what "catching it" looks like.

What's the bug pattern your AI reviewer has already let through that you haven't turned into a test case yet? That's usually fixture #1.