Free AI PR Review on a Free Server: A Case Study in Testing the Reviewer Before Deployment

AI tools have promoted every developer to reviewer, but almost nobody runs a test on the reviewer itself. This case study covers one small project end to end: a PR-review bot that used free model tokens and a free server to watch a small Python repository. The bot only earned a permanent spot after passing a seeded-bug test, and the most valuable artifact was not the bot but the test.

Background: why a reviewer bot, and why now

The current conversation around AI-assisted review usually stops at "the AI writes comments, the human checks them." That framing skips the harder question: how do you know the AI's comments are worth checking? A reviewer that produces confident noise is worse than no reviewer, because it trains humans to ignore the channel entirely.

So the case study had a simple premise: build the cheapest possible reviewer, give it a tiny job, and measure it before letting it speak on real pull requests. The repository was a small Python CLI tool with about two thousand lines, three active contributors, and a handful of open PRs at any time. The workload was small enough for a free tier to cover, and the stakes were low enough that a wrong comment was annoying rather than dangerous.

Goal and constraints

The project had four explicit constraints:

• Zero operating cost: no model bill, no server bill, no hidden hourly charges.
• Unattended operation: the bot had to run on a free server and recover from crashes without human help.
• Human-judgeable output: every comment had to fit in a few lines with a clear severity tag.
• Testable before deployment: the bot had to pass a seeded-bug evaluation before commenting on real PRs.

MonkeyCode is an open-source project that bundles free model access with a free server option, and that combination fit the constraints directly. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The current offer is a 10-million-token allowance plus the free server, and a typical review in this project used roughly 1,000 to 3,000 tokens depending on diff size. That puts the allowance at thousands of reviews before it becomes a constraint, which is more than enough for a small repository.

Implementation: the bot

The bot itself is deliberately boring. It polls the GitHub API for open pull requests, fetches each diff, sends the diff to a model endpoint with a fixed checklist prompt, and posts a comment only when the model returns something other than OK. The script reads configuration from environment variables, so it does not care which provider backs the endpoint.

The polling script

# review_bot.py
import json
import os
import sys
import time
import urllib.request

GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
REPO = os.environ["REPO"]
MODEL_BASE_URL = os.environ["MODEL_BASE_URL"]
MODEL_API_KEY = os.environ["MODEL_API_KEY"]
MODEL_NAME = os.environ["MODEL_NAME"]

REVIEW_PROMPT = """You are reviewing a pull request diff.
Reply with exactly one line in this format:
SEVERITY: one-line reason
SEVERITY is one of BLOCKER, SHOULD-FIX, NIT, or OK.
Only mention real problems. Do not suggest style changes."""


def gh(path, method="GET", payload=None):
    req = urllib.request.Request(
        f"https://api.github.com/repos/{REPO}{path}",
        method=method,
        headers={
            "Authorization": f"Bearer {GITHUB_TOKEN}",
            "Accept": "application/vnd.github+json",
            "User-Agent": "review-bot",
        },
    )
    data = json.dumps(payload).encode() if payload else None
    with urllib.request.urlopen(req, data=data) as resp:
        return json.load(resp)


def gh_diff(pr_number):
    req = urllib.request.Request(
        f"https://api.github.com/repos/{REPO}/pulls/{pr_number}",
        headers={
            "Authorization": f"Bearer {GITHUB_TOKEN}",
            "Accept": "application/vnd.github.diff",
            "User-Agent": "review-bot",
        },
    )
    with urllib.request.urlopen(req) as resp:
        return resp.read().decode()


def review_diff(diff):
    body = {
        "model": MODEL_NAME,
        "messages": [
            {"role": "system", "content": REVIEW_PROMPT},
            {"role": "user", "content": diff},
        ],
        "temperature": 0.2,
    }
    req = urllib.request.Request(
        f"{MODEL_BASE_URL}/chat/completions",
        data=json.dumps(body).encode(),
        headers={
            "Authorization": f"Bearer {MODEL_API_KEY}",
            "Content-Type": "application/json",
        },
    )
    with urllib.request.urlopen(req) as resp:
        return json.load(resp)["choices"][0]["message"]["content"].strip()


def main():
    prs = gh("/pulls?state=open")
    for pr in prs:
        diff = gh_diff(pr["number"])
        if not diff:
            continue
        verdict = review_diff(diff)
        if not verdict.startswith("OK"):
            gh(f"/issues/{pr['number']}/comments", method="POST", payload={"body": verdict})
        print(f"PR #{pr['number']}: {verdict}")


if __name__ == "__main__":
    if "--once" in sys.argv:
        main()
    else:
        while True:
            main()
            time.sleep(300)

The polling loop is intentionally simple: run a pass, sleep five minutes, repeat. The environment file holds the credentials, and the bot reads them once at startup. Save it as /etc/review-bot.env with one KEY=value pair per line.

GITHUB_TOKEN=ghp_xxx
REPO=you/your-project
MODEL_BASE_URL=https://your-provider.example
MODEL_API_KEY=sk-xxx
MODEL_NAME=your-model-name

The server unit

On the free server, a systemd unit keeps the process alive and restarts it after crashes, which is the part that makes unattended operation real.

# /etc/systemd/system/review-bot.service
[Unit]
Description=Free-tier PR review bot
After=network-online.target

[Service]
EnvironmentFile=/etc/review-bot.env
ExecStart=/usr/bin/python3 /opt/review-bot/review_bot.py
Restart=on-failure
RestartSec=30

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now review-bot
journalctl -u review-bot -f

The reviewer test: the artifact that matters

The bot is easy to build; the test is what separates this project from a toy. Before the bot could comment on real pull requests, it had to pass an evaluation with two fixtures: one diff with a real bug and one clean refactor.

Fixture one: a real bug

This diff introduces a divide-by-zero in a totals function:

@@ -42,7 +42,7 @@ def calculate_total(items):
-    return sum(item.price for item in items)
+    return sum(item.price for item in items) / len(items)

Fixture two: a clean refactor

This diff is a pure variable rename that changes no behavior:

@@ -10,7 +10,7 @@ def get_user(name):
-    user = db.query(f"SELECT * FROM users WHERE name = '{name}'")
-    return user
+    record = db.query(f"SELECT * FROM users WHERE name = '{name}'")
+    return record

The test runner and acceptance rule

Save the fixtures as fixtures/divide_by_zero.diff and fixtures/clean_rename.diff. The test runner sends both through the same review function and compares each verdict against the known answer:

# test_reviewer.py
from review_bot import review_diff

CASES = [
    {"name": "divide_by_zero", "diff": open("fixtures/divide_by_zero.diff").read(), "should_flag": True},
    {"name": "clean_rename", "diff": open("fixtures/clean_rename.diff").read(), "should_flag": False},
]

for case in CASES:
    verdict = review_diff(case["diff"])
    flagged = not verdict.startswith("OK")
    status = "PASS" if flagged == case["should_flag"] else "FAIL"
    print(f"{status} {case['name']}: {verdict}")

The acceptance rule is deliberately strict: the bot must flag the buggy fixture with SHOULD-FIX or BLOCKER, and it must stay silent on the clean rename. A reviewer that cannot pass a two-case sanity check will not earn trust with a hundred-case suite.

Results and the decision framework

The seeded-bug run produced the expected split. The buggy fixture came back as SHOULD-FIX: division by zero when items is empty, and the clean rename came back as OK. The full test consumed roughly 1,500 tokens and took a couple of minutes, which puts the 10-million-token allowance into perspective: thousands of these evaluations fit inside the free tier.

Whether the bot stays is not a single yes or no; it is a small decision table you can apply to your own run:

Signal	Verdict
Flags seeded bugs with correct severity	Enable on real pull requests
Flags clean diffs with NIT or style noise	Restrict the prompt to BLOCKER and SHOULD-FIX only
Returns OK on everything, including bugs	Do not enable; the model is not suitable for review
Produces long essays instead of one-line verdicts	Tighten the prompt format before retesting

In this project, the bot passed the first row, so it earned a trial on real pull requests with one rule: every comment still gets a human acknowledgment before it counts as actioned. That rule matters more than the bot itself.

Lessons learned

Three lessons came out of the project. First, prompt design did more work than model choice: the one-line severity format made output easy to judge, while a free-form prompt produced paragraphs that nobody read. Second, the free server changed the deployment math, because a long-running poller on a paid VM would have quietly cost more than the model tokens. Third, the test was the product: the two-fixture evaluation took twenty minutes to write and saved the repository from a week of confident noise.

Who should not use this approach? Teams with high pull-request volume will exhaust the free allowance quickly and should budget for a paid tier. Repositories with security-sensitive diffs should not rely on a free model for review, period. And if nobody on the team will read the bot's comments, the bot is not free; it is a tax on attention.

Limitations

The bot has real limits. It reviews diffs, not branches, so it cannot catch integration failures or test gaps. It does not run the code, so a bug that only appears at runtime will pass. The token estimate depends heavily on diff size, and large pull requests can burn thousands of tokens per review. The free server is fine for a poller, but it is not a substitute for a CI runner or a production host. Finally, the 10-million-token figure and the free server option are the current offer at the time of writing; check the project's documentation before you rely on either one.

If you want to try the same setup, MonkeyCode's free tier is a reasonable place to start because the token allowance and the server are both free, and the script in this article does not care which provider backs it. Build the bot, run the two fixtures, and let the test decide.