Last Tuesday I opened PR #47 on our final-year term project, skimmed the diff, and nearly clicked merge. The diff touched utils.py and removed a few lines from tests/test_utils.py that I assumed were redundant. A classmate caught it during a late review: the deleted test was the only one covering parse_date with an empty string. The merge would have shipped quietly because our CI only required that tests still pass, not that they cover the same branch. I wondered whether a tiny, always-on second reviewer could catch that earlier than a tired human at 11pm.
That question became a small case study. I wanted a cheap second set of eyes, so I tried MonkeyCode's free model access and free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The current offer mentions a 30,000,000-token monthly allowance and a free server option. I treated those as an experiment budget, not as production infrastructure, because free tiers can change and I did not want my coursework to depend on a quota I did not control.
The goal was narrow. I did not want to build an AI code reviewer. I wanted a single-question gate that could read a diff file and tell me whether it removed or weakened a test without adding an equivalent one. That is easier to evaluate than "is this code good" and easier to debug when the model goes wrong.
I wrote a small Python script, diff_audit.py, that reads a saved diff and sends it to a chat-compatible endpoint. The endpoint and model are read from environment variables, so the same script can run against a local server or a free endpoint without changing code.
import os
import sys
import json
import requests
MC_BASE_URL = os.getenv("MC_BASE_URL", "http://localhost:8000/v1")
MC_API_KEY = os.getenv("MC_API_KEY")
MC_MODEL = os.getenv("MC_MODEL")
PROMPT_TEMPLATE = """You are a cautious code-review assistant for a student project.
Read the diff below and answer exactly one question:
Does this diff remove or weaken tests without adding an equivalent test?
Return only JSON:
{"risk": "low|medium|high", "reason": "short explanation"}
Diff:
__DIFF__
If the diff is empty or unreadable, return:
{"risk": "unknown", "reason": "empty or unreadable diff"}
"""
def ask_model(diff):
if not diff.strip():
return {"risk": "unknown", "reason": "empty or unreadable diff"}
prompt = PROMPT_TEMPLATE.replace("__DIFF__", diff[:12000])
payload = {
"model": MC_MODEL,
"messages": [
{"role": "system", "content": "You return only JSON."},
{"role": "user", "content": prompt},
],
"temperature": 0.1,
}
headers = {"Authorization": f"Bearer {MC_API_KEY}"}
resp = requests.post(
f"{MC_BASE_URL}/chat/completions",
headers=headers,
json=payload,
timeout=45,
)
resp.raise_for_status()
content = resp.json()["choices"][0]["message"]["content"]
try:
return json.loads(content)
except json.JSONDecodeError:
return {"risk": "unknown", "reason": f"model returned non-JSON: {content[:80]}"}
if __name__ == "__main__":
if len(sys.argv) != 2:
print("usage: python diff_audit.py path/to/filename.diff")
sys.exit(1)
with open(sys.argv[1], encoding="utf-8") as f:
diff = f.read()
result = ask_model(diff)
print(json.dumps(result, indent=2))
The first run used a diff saved with git diff main..pr-47 > pr_47.diff. The command python diff_audit.py pr_47.diff returned the JSON below.
{
"risk": "high",
"reason": "tests/test_utils.py: parse_date empty string test removed and no replacement present in diff"
}
The most useful test was not a good diff but a bad input. I ran echo "not a diff" > fake.diff and python diff_audit.py fake.diff. The model returned a non-JSON answer, and the fallback caught it. Then I tried an empty file and the script returned unknown before spending a single token.
{
"risk": "unknown",
"reason": "empty or unreadable diff"
}
That small guard saved more time than any clever prompt. It also made me realize that a reviewer tool should fail closed, not generate a confident answer from garbage.
To stop my laptop from being the always-on machine, I moved the same function into a tiny FastAPI service and ran it with uvicorn on the free server. The deployment was the same two commands I use locally: pip install -r requirements.txt and uvicorn review_api:app --host 0.0.0.0 --port 8000. Then I could post a diff to /review instead of keeping a terminal open.
from fastapi import FastAPI
from diff_audit import ask_model
app = FastAPI()
@app.post("/review")
def review(payload: dict):
diff = payload.get("diff", "")
return ask_model(diff)
I ran the script on three PRs from our last sprint. PR #47 earned a high risk with the missing empty-string test. PR #52 was low because it added a new helper and a corresponding test. PR #58 came back medium, but for the wrong reason: the diff included a generated lockfile, and the model treated the huge unrelated deletion as suspicious. That false positive was useful, because it taught me to pre-filter generated files before sending a diff.
Three lessons stood out. First, truncating the diff to a few thousand characters kept responses fast and predictable; long files made the model wander. Second, temperature matters. Setting it to 0.1 gave JSON most of the time, but not always, so I kept the fallback parser. Third, the free server cold-start delay was real but tolerable for a review that runs a few times a day. I would not put this on a latency-sensitive path.
Who should not use this? If you are reviewing proprietary code, do not send unredacted diffs to a third-party endpoint. If you need a deterministic rule like "any deleted test line is a fail," use a script, not a language model. If you depend on a free tier as a permanent CI gate, you will eventually be surprised by rate limits or changes. This was a second opinion, not a merge button.
If you want to try the same experiment, the large free token allowance meant I could rerun the same diff many times while tuning the prompt without worrying about cost. Start with one narrow question and one saved diff, then move it to a server only after the script earns your trust.
Top comments (0)