Free-Model Code Review: A 48-Hour False Positive Autopsy

Have you ever trusted a code review bot because it found something you missed? I did the opposite: I ran a free model on my own commit history and ended up with a long list of vulnerabilities that never existed. This is not a story about an unusable AI; it's a story about the difference between a useful warning and a real one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and a free server option to run the experiment described below. Everything else here is my own observation.

The Setup

I wanted to answer a simple question: can a free model act as a lightweight security reviewer for a small personal repository? I wrote a scheduler that runs hourly on a free server, pulls the latest 20 commits from git, extracts the diffs, and sends them to the model with a strict prompt asking for a JSON array of findings.

The repository is a toy blogging engine with around 300 commits, plenty of test fixtures, and one deliberate async bug that I seeded for the experiment. The model did not know that the bug was planted, because the diff never contained the word "bug" or "error" in the obvious places.

Here is the core loop, simplified to keep this article honest:

import json
import subprocess
import requests

def get_diffs(commit_count=20):
    log = subprocess.run(
        ["git", "log", f"-{commit_count}", "-p", "--max-count=1"],
        capture_output=True, text=True
    ).stdout
    return log[:8000]  # truncate to fit the free model's context

def ask_free_model(prompt):
    # Endpoint and headers come from your MonkeyCode free model setup.
    response = requests.post(
        "https://your-monkeycode-endpoint.example/v1/chat",
        json={"messages": [{"role": "user", "content": prompt}]},
        timeout=60,
    )
    return response.json()["choices"][0]["message"]["content"]

PROMPT = """
You are a security reviewer. Look at the diff below and return a JSON array of findings.
Each finding must have fields: title, severity, file, reason.
If there are no issues, return an empty array.

diff
{DIFF}

"""

diff = get_diffs()
raw = ask_free_model(PROMPT.replace("{DIFF}", diff))
try:
    findings = json.loads(raw)
except json.JSONDecodeError:
    # Sometimes the model wraps JSON in a markdown code block.
    import re
    match = re.search(r"```

(?:json)?\s*(\[.*?\])\s*

```", raw, re.DOTALL)
    findings = json.loads(match.group(1))
print(findings)

That parser saved my server, because without it a good portion of the responses would have crashed the job.

What Actually Happened

I collected 48 hourly runs, roughly 960 diffs reviewed. Here are the patterns that stood out.

• Phantom hardcoded secrets: the model flagged the same test fixture token 14 times as a "hardcoded credential". The token had test_ as a prefix and was defined in a fixture file, but the model saw token and jumped to high severity.
• The console.log panic: a standard console.info("user created") was reported as an "information disclosure" for three consecutive runs. It never mattered where the log appeared.
• The seeded bug survived: I planted a missing await on an async database call in the commit log. The model never once mentioned it, even though the diff was right there with a visible omission.
• Format drift: about 11% of responses were valid JSON wrapped in markdown fences, and two responses were actually a plain sentence plus an array. The regex parser handled those, but it introduced a small risk of silently swallowing real findings.
• The same issue, many names: a null check in the auth middleware was reported as "missing validation", "possible NPE", "unsafe access", and "logic error" across different runs. Deduplication based on title alone would have failed.

The numbers are not a benchmark; they are a sample of one repo. But they were enough to convince me that a raw model output is not a review.

The False Positive Pattern

Reading through the logs, the pattern became obvious. The free model relies on lexical signals: if it sees key, secret, token, or password, a high-severity flag is almost guaranteed. It does not simulate a developer browsing the codebase; it pattern-matches against common vulnerability snippets and hedges by reporting everything plausible.

That behavior makes sense when you think about what the model was trained to do. The instruction "find issues" rewards thoroughness, and there is no penalty for being wrong. Without a negative label for false positives, the model's optimal strategy is to raise everything that resembles a known issue.

A Quick Decision Table

After this experiment, I wrote a small decision table for myself. You can use it too.

Use case	Let the free model decide?	Why
Triage bucket for a human reviewer	Yes	It can highlight files with high signal if you add contextual instructions.
Direct commit blocker	No	Phantom findings will block unrelated changes and train people to ignore alerts.
Scanning test fixtures	No	It will drown you in fake secrets.
First pass on unfamiliar code	Maybe	Useful only if you accept a high noise floor.

What I'd Repeat and What I'd Skip

I would repeat the part where the free server ran the loop for two days without crashing. The scheduler, the parser, and the storage all held up. I would also repeat the practice of logging the raw response before parsing it, because without those logs I could never have reconstructed the duplicates.

I would skip the idea of auto-formatting results into issues without a review step. The next time I run this, every finding will go into a simple queue with a "confidence count" — the number of runs where the same file was flagged. Only findings with a confidence count above two and a severity label of high will reach human eyes.

Limitations

This is a narrow study: one repository, one prompt, one free model variant on a free server. My seeded bug detection is not a general measurement, and your repo's naming conventions could produce very different results. Compliance-sensitive codebases should stay far away from this pattern, and so should teams that cannot afford to manually triage a steady stream of false positives.

If you try something like this, start with the decision table and expect the model to confidently invent problems. That is not a reason to stop using free models for review; it is a reason to treat every alert as a hypothesis, not a conclusion.