I was one merge away from shipping. The AI reviewer had already spoken: "No issues found. Looks good to me."
And that's when the doubt hit. What does "looks good" mean when the reviewer is a model I've never tested?
So I stopped reviewing the code and started reviewing the reviewer. I planted a bug I already knew about, asked the model to find it, and scored the answer. The question was simple: can a free model reviewer catch a bug I already know is there? And if it can't, how would I ever find out?
Here's the function I used. It's small, it's realistic, and it contains exactly one logic bug.
def price_with_discount(price, discount_pct):
if discount_pct < 0 or discount_pct > 100:
raise ValueError("discount must be between 0 and 100")
factor = 1 + discount_pct / 100 # seeded bug: should be minus
return round(price * factor, 2)
Read it once. If you said "the sign is flipped," you just passed the test. The real question is whether a model reviewer says the same thing when this function is buried in a longer file.
There's a discussion on DEV this week about who reviews the reviewer now that AI writes the first draft. I'm the nobody in that discussion. I'm a student; my review process is reading my own code and hoping. A free model reviewer sounds like the upgrade I can actually afford — until I remember that hope is not a test plan.
So I built one. The goal was a reviewer regression test. Real software has regression tests, and a reviewer is software, so it deserves a test too. The test is brutally simple: seed a known bug, ask for a review, check whether the bug gets flagged. Run it once and you have a snapshot. Run it on a schedule and you have a trend — because models change, and a reviewer that worked last month might not work this month.
Prerequisites: Python 3.10+, an endpoint that speaks the OpenAI chat-completions format, and an API key. I used the open-source project MonkeyCode's free model access for the endpoint and their free server option to host the scheduled run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script itself doesn't care which provider you point it at.
Here's the harness:
# review_harness.py
import json
import os
import sys
import urllib.request
ENDPOINT = os.environ["ENDPOINT"]
API_KEY = os.environ["API_KEY"]
MODEL = os.environ["MODEL"]
# (buggy_line, symptom) pairs — the ground truth you already know
SEEDED_BUGS = [
("factor = 1 + discount_pct / 100", "increases the price"),
]
def load_code(path):
with open(path) as f:
return f.read()
def review(code):
payload = json.dumps({
"model": MODEL,
"messages": [{
"role": "user",
"content": (
"You are reviewing a pull request. "
"Find logic bugs, not style nits. "
"For each bug, quote the line and explain the impact.\n\n"
f"```
{% endraw %}
python\n{code}\n
{% raw %}
```"
),
}],
"temperature": 0,
}).encode()
req = urllib.request.Request(ENDPOINT, data=payload, headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}",
})
with urllib.request.urlopen(req) as resp:
data = json.load(resp)
return data["choices"][0]["message"]["content"]
def main():
code = load_code(sys.argv[1])
response = review(code)
caught = any(symptom in response for _, symptom in SEEDED_BUGS)
print(json.dumps({"caught": caught, "response": response}))
if __name__ == "__main__":
main()
Run it:
export ENDPOINT=... API_KEY=... MODEL=...
python review_harness.py price.py
Expected output — yours will differ in the response text, but the shape is the same:
{"caught": true, "response": "Line 4: `factor = 1 + discount_pct / 100` — the sign is flipped. A 20% discount increases the price by 20% instead of reducing it."}
The harness checks for a symptom phrase, not a perfect match. That's a deliberate tradeoff. It's easy to read, but it can also produce a false negative: if the model catches the bug and says "the math is backwards" instead of "increases the price," my check misses it. The model was right and my test still failed. Remember that every time you see a green or red checkmark — it's a heuristic wearing a uniform.
The one-off run answers one question. The scheduled run answers a better one: is this reviewer still any good? I put the harness on MonkeyCode's free server and let cron do the rest.
0 9 * * * cd ~/reviewer-check && python3 review_harness.py price.py >> results.jsonl
Every morning it appends one line to results.jsonl. After a week you have seven data points; after a month you have thirty. That's when the real insight shows up — not in any single review, but in the shape of the line. If cron's minimal environment can't find python3, replace it with the full path from which python3.
The run that made me keep the harness was a confident miss. The model said "no logic errors found" on a file that contained a planted sign flip. Nothing in the code had changed; the model had. That's the exact moment the setup earns its keep: confidence and accuracy are two different numbers, and only one of them was being measured before.
The exact numbers won't transfer to your setup. Your endpoint, your prompt, and your code will produce different ones. What transfers is the shape of the failure: a reviewer can be right on Monday and confidently wrong on Tuesday, and you won't know unless you're measuring.
Three lessons stuck with me. First, a reviewer you can't test is just an opinion with better grammar. The harness gave me a number I could reason about, and a number beats a vibe. Second, seeding bugs is calibration, not sabotage. You're not trying to trick the model; you're trying to map its edge, and every red line is a point on that map. Third, a one-off test goes stale. Models change, prompts drift, and "it worked last month" is not evidence. The free server turned a snapshot into a time series, and the time series is what actually changed my behavior.
Now the part nobody likes: who should not use this. If you're reviewing code you can't modify, this whole approach is out — you can't seed bugs into someone else's production PR. If you need a guarantee, this is still a heuristic; it measures one narrow capability, catching a planted logic bug, and it says nothing about architecture, security, or whether the model would have found the bug without a prompt that says "find logic bugs." Free tiers also come with real constraints: rate limits, latency, and the occasional outage. The harness will happily log those as failures too, so read the raw responses before you blame the model.
What should you take away? One sentence: an AI reviewer is software, and software needs a regression test. The test is a seeded bug, the assertion is a symptom phrase, and the schedule is a cron job. Build that before you trust any review — from a model, or from yourself.
If you want to run this exact setup, MonkeyCode's free model access and free server option are one way to get both pieces without a credit card. The value is in the harness, not the provider — point it at whatever endpoint you have and see what your reviewer is actually made of.
Top comments (0)