A Free Server, a Cheap Model, and a Scoreboard: One Team's Code Review Audit

A small team ran a public Node API with a code review bot on every pull request. The bot caught real issues. It also produced a monthly bill that grew faster than the team's income. A new low-cost model had just appeared in the operator's feed. The immediate answer was to swap. The maintainer said no. First they needed a number.

The team had no budget for a formal benchmark. They did have ninety days of pull request history. The history contained bot comments, human corrections, and labels. That looked like a training set. It could also be a test set. The team decided to treat their own repository as the only benchmark that mattered.

They needed a neutral way to run two models side by side. MonkeyCode's free model access and free server option made that possible. The audit did not require a credit card. That mattered because the whole point was to stop paying before measuring.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The team did not compare marketing numbers. They compared comments on their own diffs. The plan had five steps.

Step 1: Build a real corpus

The team exported ninety days of closed pull requests. Each case included the diff, the repository context, the model's comment, and a human verdict. The verdict marked each comment as useful, useless, correct, incorrect, actionable, or noise. Synthetic examples were not allowed. If a comment could not be tied to a real code path, it was dropped.

Step 2: Split by date, not shuffle

The team split the corpus into sixty days for calibration and thirty days for the final score. The split was by date, not by random shuffle. Random shuffling leaks future changes into the past.

Step 3: Keep provider calls behind one function

The provider APIs changed too often for the audit to depend on one SDK. The team kept all provider calls behind one function.

// providers.mjs
// This adapter is illustrative. The real SDK call stays in one file.
export async function callModel(provider, prompt) {
  const adapter = createAdapter(process.env[provider.toUpperCase() + '_KEY']);
  const started = Date.now();
  const raw = await adapter.complete({
    prompt,
    maxTokens: 500,
    temperature: 0,
  });
  return {
    raw,
    latencyMs: Date.now() - started,
  };
}

Step 4: Run both models through the same prompt

The prompt asked for no pleasantries. It asked for a JSON list of findings. Each finding had a file, line, severity, and a one-line explanation. The script parsed the output and saved the run to disk.

// run-audit.mjs
import fs from 'node:fs/promises';
import { callModel } from './providers.mjs';

const cases = JSON.parse(await fs.readFile('./pr-cases.jsonl', 'utf8'));

async function runOne(c) {
  const prompt = buildPrompt(c.diff, c.context);
  const budget = await callModel('budget', prompt);
  const frontier = await callModel('frontier', prompt);
  return {
    caseId: c.id,
    budget: parseFindings(budget.raw),
    frontier: parseFindings(frontier.raw),
    latency: {
      budget: budget.latencyMs,
      frontier: frontier.latencyMs,
    },
  };
}

const results = [];
for (const c of cases) {
  results.push(await runOne(c));
}
await fs.writeFile('./run-072.json', JSON.stringify(results, null, 2));

Step 5: Score precision, recall, and noise separately

The team did not use a single leaderboard number. They used three columns. First, recall on high-severity findings. Second, false positives per one thousand lines. Third, median latency. A model that catches every bug but floods the review with noise loses the false positive column.

// score.mjs
import fs from 'node:fs/promises';

const cases = JSON.parse(await fs.readFile('./pr-cases.jsonl', 'utf8'));
const truth = new Map(cases.map((c) => [c.id, c.human]));
const run = JSON.parse(await fs.readFile('./run-072.json', 'utf8'));

for (const model of ['budget', 'frontier']) {
  let tp = 0;
  let fp = 0;
  let fn = 0;
  for (const r of run) {
    const expected = truth.get(r.caseId).findings;
    const predicted = r[model].findings;
    for (const e of expected) {
      if (predicted.some((p) => sameFinding(e, p))) tp += 1;
      else fn += 1;
    }
    for (const p of predicted) {
      if (!expected.some((e) => sameFinding(e, p))) fp += 1;
    }
  }
  const precision = tp / (tp + fp);
  const recall = tp / (tp + fn);
  console.log(model, {
    precision: precision.toFixed(2),
    recall: recall.toFixed(2),
    f1: ((2 * precision * recall) / (precision + recall)).toFixed(2),
    falsePositives: fp,
  });
}

The first run surprised the team. The new budget model matched the expensive bot on recall. It caught the same null checks and missing awaits. Then the scoring script printed the false positive count. The budget model emitted almost twice as many comments. Most were true but trivial. They were the kind of comments that made reviewers stop reading. The team decided not to replace the bot. They used the budget model as a second reader on risky paths. The expensive bot stayed on the default branch.

The decision table was simple. If the budget model missed more than two high-severity bugs in the final thirty days, stop. If the false positive rate stayed above the old bot by more than thirty percent, do not replace. If both passed, test it on one repository for two weeks before changing the default. This table kept the decision from becoming a vibe.

The free server mattered in a specific way. The audit needed stable latency numbers. A laptop run had background tasks that made timing useless. The free server also let the team run the full thirty-day slice overnight without paying for API minutes. That turned a one-off demo into a repeatable monthly audit.

The limitations were real. The corpus only covered one team's code style. The labels came from the same maintainers who were tired of the old bot's noise. That bias could not be removed. Model behavior drifts. A model that passed in July may regress in August. The team also did not compare security findings at enough depth. A code review audit is not a penetration test.

This approach is not for every team. A team with no closed pull requests has no corpus. A team with compliance rules may not be able to send private diffs to a third-party server. A team that needs subsecond latency should not rely on a free tier that may throttle under load. A team that wants a one-line answer should not build this. The scoreboard only helps if someone reads the columns.

The useful artifact is not the conclusion. It is the small, boring harness. It turns a model swap into a number. The team still runs the audit before any new model enters the repository. The script is short enough to keep in a private branch. That is all a small team needs before they believe the hype.