Rate Your AI Reviewer: A Five-Case Regression Suite

AI turned every developer into a reviewer. Nobody tested the reviewer. A five-case regression suite can expose a weak AI reviewer in under ten minutes, and it runs for free when you use MonkeyCode's free models and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The premise is simple. Teams adopt AI coding tools, then trust the AI to review their pull requests. They evaluate the code-writing model with benchmarks, but they never evaluate the review model. A reviewer that misses an off-by-one or a path traversal is worse than no reviewer, because it creates false confidence.

The five bug patterns

A good regression suite tests common failure classes, not exotic ones. These five cover most real-world review misses:

1. Off-by-one in a loop boundary
2. Resource leak from an unclosed file handle
3. Swallowed exception that hides a database failure
4. Race condition from a non-atomic read-modify-write
5. Path traversal from a missing path validation

Each case is a small diff with exactly one injected bug. The reviewer's job is to find it and suggest a fix.

The suite format

The suite is plain JSON. Each case has an id, a diff, the expected bug, and the expected fix direction.

{
  "cases": [
    {
      "id": "off-by-one",
      "diff": "--- a/queue.js\n+++ b/queue.js\n@@ -12,7 +12,7 @@\n function processQueue(items) {\n-  for (let i = 0; i < items.length; i++) {\n+  for (let i = 0; i <= items.length; i++) {\n     handle(items[i]);\n   }\n }",
      "bug": "loop condition",
      "fix": "use < instead of <="
    },
    {
      "id": "resource-leak",
      "diff": "--- a/reader.py\n+++ b/reader.py\n@@ -5,7 +5,7 @@\n def load_config(path):\n-  with open(path) as fh:\n-    return json.load(fh)\n+  fh = open(path)\n+  return json.load(fh)",
      "bug": "file handle",
      "fix": "close the file or use a context manager"
    },
    {
      "id": "swallowed-exception",
      "diff": "--- a/api.py\n+++ b/api.py\n@@ -20,7 +20,7 @@\n def fetch_user(user_id):\n   try:\n     return db.query(user_id)\n-  except DatabaseError as exc:\n-    raise\n+  except DatabaseError:\n+    return None",
      "bug": "exception",
      "fix": "re-raise or log the error"
    },
    {
      "id": "race-condition",
      "diff": "--- a/counter.py\n+++ b/counter.py\n@@ -8,7 +8,7 @@\n class Counter:\n   def __init__(self):\n     self.value = 0\n   def increment(self):\n-    self.value += 1\n+    value = self.value\n+    time.sleep(0.01)\n+    self.value = value + 1",
      "bug": "race condition",
      "fix": "use an atomic increment or a lock"
    },
    {
      "id": "path-traversal",
      "diff": "--- a/files.py\n+++ b/files.py\n@@ -15,7 +15,7 @@\n def read_file(name):\n-  path = os.path.join(BASE_DIR, name)\n-  if not path.startswith(BASE_DIR):\n-    raise ValueError(\"invalid path\")\n+  path = os.path.join(BASE_DIR, name)\n   return open(path).read()",
      "bug": "path traversal",
      "fix": "validate the resolved path stays inside BASE_DIR"
    }
  ]
}

The runner

The runner is a small Python script. It sends each diff to any OpenAI-compatible endpoint and scores the review against the expected bug and fix.

# reviewer_regression.py
import json
import sys
from openai import OpenAI

client = OpenAI(base_url="YOUR_ENDPOINT", api_key="YOUR_KEY")

REVIEW_PROMPT = (
    "Review this diff. List concrete bugs only. "
    "For each bug: file, line, why it fails, suggested fix. "
    "If no bugs, say NO_ISSUES_FOUND."
)

def run_case(case):
    response = client.chat.completions.create(
        model="your-model",
        messages=[
            {"role": "system", "content": REVIEW_PROMPT},
            {"role": "user", "content": case["diff"]},
        ],
        temperature=0.0,
    )
    return response.choices[0].message.content

def score(review, case):
    found = case["bug"] in review.lower()
    fix = case["fix"].split()[0] in review.lower()
    return {"id": case["id"], "found": found, "fix": fix}

def main(path):
    with open(path) as fh:
        suite = json.load(fh)
    results = [score(run_case(c), c) for c in suite["cases"]]
    passed = sum(1 for r in results if r["found"] and r["fix"])
    print(f"{passed}/{len(results)} cases passed")
    for r in results:
        print(r)

if __name__ == "__main__":
    main(sys.argv[1])

The scoring is intentionally naive. It checks whether the review mentions the bug class and a fix keyword. That is enough to catch a reviewer that says "looks good" on every diff. For a stricter pass, replace the keyword check with a manual review of each output.

How I ran it

I ran this suite against MonkeyCode's free models through the project's free server option. The whole experiment cost zero dollars and about fifteen minutes. I did not benchmark latency or throughput. I only measured one thing: can the reviewer find five known bugs?

The result was not perfect. The free model caught the path traversal and the off-by-one, but it described the race condition as a potential performance issue and missed the swallowed exception entirely. That is useful information. It tells me the model's review strength is in security and boundary logic, not concurrency. I would not trust it as the only reviewer on a concurrency-heavy pull request.

Decision table

Use the suite results to decide where AI review adds value:

Scenario	AI review useful?	Why
Large PR, many files	Yes	Catches obvious bugs fast
Concurrency-heavy code	No	Weak on race conditions
Security-sensitive diff	With caution	Good on traversal, weak on logic
Beginner PRs	Yes	Educational, consistent feedback
Pre-deploy hotfix	No	False confidence is expensive

The table is a template. Fill it with your own five cases and your own results.

Limitations

This suite is not a benchmark. It is a smoke test. Five cases will not rank models, and the keyword scoring can produce false positives. The diffs are synthetic, so they do not reflect real codebase complexity. Free models and free server options can change or disappear, so do not build a permanent pipeline on them. If your team reviews security-critical code, run this suite, then add human review on top.

Who should not use this approach? Teams that already have a strong human review culture. The suite adds process without adding insight. Also teams that need a model verdict. This is not a model ranking. It is a reviewer sanity check.

The takeaway

A reviewer you have not tested is a liability. Five diffs, one script, and a free endpoint are enough to find out whether your AI reviewer earns its place in the pipeline. Run the suite once, keep the results, and re-run it when the model or the prompt changes. The cheapest review improvement is the one you measure first.