I Let a Free-Tier Model Review PRs for 48 Hours. The Golden Set Survived.

Does a free-tier model make a decent pull request reviewer? I wanted a real answer, not a demo, so I ran one on a free server for two days and watched it drift in front of me.

I built a tiny service that watches new pull requests, pulls the diff, and asks a model for a verdict: approve, request changes, or skip. The model access came from MonkeyCode's free model tier, and the whole thing lived on their free server option, which meant the only cost was my attention. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I will not quote model names or quotas, because I did not measure them rigorously, and you should not trust numbers without a primary source.

On day one I wrote a golden set: five pull requests from my own repositories, each with a diff, a description, and a comment I had already written by hand. I stored them as JSON and wrote a small script that replays the same prompt through the model and compares the output against the expected label. That script became the only thing I trusted by the end.

[
  {
    "id": "pr-101",
    "title": "fix: validate email before sending",
    "diff": "+ if (!isValid(email)) return 400;",
    "expected": "request_changes",
    "reason": "missing test coverage"
  },
  {
    "id": "pr-102",
    "title": "refactor: extract parse_config",
    "diff": "+ config = parse_config(path)",
    "expected": "approve",
    "reason": "small, well-named change"
  }
]

# evaluator.py (pseudocode)
def evaluate(model_call, golden_set):
    results = []
    for case in golden_set:
        verdict = model_call(build_prompt(case))
        results.append({
            "case": case["id"],
            "expected": case["expected"],
            "got": verdict,
            "ok": verdict == case["expected"]
        })
    return results

Here is what happened in the next 48 hours, written as field notes rather than a polished post-mortem.

Hour 6 — Everything looked fine

The first reviews were genuinely useful. The model caught a missing null check, and it correctly approved a tiny refactor. I started to believe the experiment would work, which should have been my first warning.

Hour 14 — The server restarted, and my bot went quiet

The free server restarted overnight, and my polling loop died without a single log line. I only noticed because a colleague asked why the bot had stopped commenting. A supervisor process and a health check would have saved me two hours of confusion.

Hour 26 — The model started ignoring the diff

The comments became generic: 'consider adding tests' and 'this could be more robust' appeared on every pull request, even ones with no code changes. The model was reviewing the title and the description, not the actual diff, and the golden set caught it immediately.

Hour 40 — The golden set exposed the drift

I reran the evaluator and the agreement rate had fallen from 70% to 40%. The same prompts that produced sharp comments on day one now produced vague praise. Without the golden set, I would have blamed the repositories instead of the model.

Hour 48 — I removed the auto-approve path

I deleted the code that could approve a pull request and kept only the comment generator. The bot still writes suggestions, but a human has to click the merge button. That one change removed most of the risk.

What broke and what I would repeat

Let me summarize the failures and the fixes in a table, because I found myself re-learning the same lesson every few hours.

Symptom	Likely cause	What I did
All comments turned positive	Output drift	Re-ran the golden set
Comments ignored the diff	Missing context in prompt	Added the diff explicitly
No reviews after restart	No supervisor process	Added a health check
Verdicts flipped between runs	Sampling temperature	Set temperature to 0

Three things I would repeat without hesitation. First, build the golden set before the bot, because it turns vague anxiety into a number. Second, version the prompt file, because a one-line change can silently alter behavior. Third, log every raw response, because you cannot debug a model that does not keep its own history.

Limitations

Who should not use this approach? Teams that need a binding review, compliance teams, and anyone who cannot tolerate a silent outage. The free tier is fine for a personal side project, but it is not a production gate, and treating it as one will teach you the same lesson I learned.

If you are about to ship an AI reviewer, build the golden set first. It is the only part of this experiment I would repeat, and it is the only part that kept me honest.