Weekend Build Log: I Finally Tested My AI Code Reviewer

Friday, 9 PM. I merged a pull request. My AI reviewer had left 14 comments. Three were useful. The rest praised my variable names.

The PR was small. The review was loud. The merge was fine. The review was not.

Nobody had tested that reviewer. Not once. So I gave myself one weekend to fix that.

This is the build log. It covers the scope cut, the working demo, and everything I skipped.

The original plan was a trap

My first plan looked great on paper. A dashboard. Metrics per repository. Trends over time. Slack alerts. A Postgres database.

That is a two-month project. I had two days.

So I cut it down to one question. Which review comments are noise?

That question fits in a single script. Everything else got deleted. I wrote the full plan on a sticky note. Then I drew a line under one sentence. Everything below the line became "later".

Step 1: Collect real comments

I started with a public repository. The GitHub API returns pull request review comments as JSON.

# fetch_reviews.py
import json
import urllib.request

url = "https://api.github.com/repos/psf/requests/pulls/comments?per_page=50"
req = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json"})
comments = json.load(urllib.request.urlopen(req))

with open("comments.json", "w") as f:
    json.dump(comments, f, indent=2)

print(f"Saved {len(comments)} comments")

Run it once. Save the file. That is the whole data pipeline.

No authentication. No database. The unauthenticated GitHub API allows 60 requests per hour. One request is all this project needs.

Step 2: Define the rubric

You cannot measure noise without a definition. I used three labels.

Label	Meaning
blocking	Points to a bug, security issue, or broken behavior
actionable	Suggests a specific change with a location or example
noise	Praise, restatement, or a question with no next step

The rubric is the real artifact. The model just applies it. If your team disagrees on the labels, fix the rubric first.

I stole this rubric from how I review code by hand. Every comment gets one label. No comment gets two. Blocking comments stop a merge. Actionable comments improve the next commit. Noise comments only add scroll time. A healthy review is mostly actionable, some blocking, and very little noise.

Step 3: Classify every comment

I wrote a small classifier. The model call is an adapter. Swap in any endpoint.

# classify.py
import json

RUBRIC = """
blocking: points to a bug, security issue, or broken behavior
actionable: suggests a specific change with a location or example
noise: praise, restatement, or a question with no next step
"""

def classify(comment: str, complete) -> str:
    # Pseudocode: complete() is your model call
    prompt = f"{RUBRIC}\n\nLabel this review comment:\n{comment}"
    return complete(prompt).strip().lower()

with open("comments.json") as f:
    comments = json.load(f)

results = []
for c in comments:
    body = c.get("body", "")
    results.append({"label": classify(body, complete), "text": body})

with open("results.json", "w") as f:
    json.dump(results, f, indent=2)

The script reads the saved JSON. It never touches the network twice. That keeps the experiment reproducible.

I ran this through MonkeyCode's free model access. The free tier includes 10 million tokens. That covered the whole weekend of experiments.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The adapter took twenty minutes to write. Any OpenAI-compatible endpoint works.

Step 4: Generate a boring report

Dashboards are overrated. I printed a text report.

# report.py
import json

with open("results.json") as f:
    results = json.load(f)

counts = {}
for r in results:
    counts[r["label"]] = counts.get(r["label"], 0) + 1

lines = ["# Review Noise Report", ""]
lines.append(f"Total comments: {len(results)}")
for label in ("blocking", "actionable", "noise"):
    lines.append(f"- {label}: {counts.get(label, 0)}")
print("\n".join(lines))

My sample of 50 comments came back as 12 blocking, 17 actionable, 21 noise.

The noise rate matched my gut feeling. That was the whole point. I did not tune the prompt. I did not cherry-pick the repo. The first run is the one you see here.

I saved the output as markdown. Then I converted it to HTML with a one-line script. Read the noise bucket first. Those comments waste the most attention. If the noise rate drops next month, your reviewer improved. If it rises, your prompts drifted.

Step 5: Publish it on a free server

A report on your laptop is invisible. I rendered the output to a single HTML file.

Then I copied it to MonkeyCode's free server option. One static file. No Docker. No database. No CI.

Teammates opened a URL and saw the numbers. Deployment took less time than writing this paragraph. The HTML report had one table and three numbers. It answered the question better than any dashboard I have used. The server stayed up all weekend. It cost nothing. That is the point of a free tier for a side project.

What I skipped on purpose

• Authentication and multi-repo support. One repo was enough to answer the question.
• History and trend charts. I needed today's numbers, not a time series.
• Slack or Discord notifications. A URL in the team channel works fine.
• Retry and rate-limit handling. The GitHub API allows one fetch. I made one fetch.
• Tests beyond one golden sample. The rubric is the spec. I reviewed every label by hand.
• A web framework. The report is static HTML. There is nothing to crash.

Every item on that list is a real feature. None of them answer the core question.

The one surprise

The pipeline was fast. Fifty comments were classified while my coffee cooled.

The bottleneck was never the model. It was writing the rubric. Get that right and the rest is glue code. I expected the model to overuse "actionable". It did not. The distribution looked like a human review.

Limitations

The sample is 50 comments from one repository. That is not a benchmark.

The labels are subjective. Another rubric produces another verdict. The classifier can also be wrong. Treat it as triage, not as a judge. One model produced these labels. Another model may disagree. Run the same script twice and compare.

The free server option is for demos. Do not point production traffic at it.

Who should not use this

Skip this approach if you need audit-grade review metrics. Skip it if you will trust the labels without reading the comments.

This is a weekend artifact. It answers one question about one repo. That is exactly what it should do. Use it as a starting point, not a verdict. The value is in the conversation the numbers start.

If you want to run the same experiment, the code above is the whole project. MonkeyCode's free tier is enough to finish it in one weekend. Steal it.