Judge AI Code Review With a Scoreboard, Not a Demo: A Repeatable Experiment That Runs on Free Access

There's a familiar ritual when a team considers an AI review assistant: someone opens the product, feeds it a file from the current sprint, watches it produce a confident-sounding paragraph, and reports back that it "seems pretty good." Two months later nobody can say whether the subscription is catching real bugs or just generating plausible noise.

The failure isn't in the tool — it's in the test. A one-shot impression can't separate a reviewer that finds defects from one that merely sounds like a reviewer. What you need instead is a scoreboard: a fixed set of inputs with known answers, a scoring rule you wrote down in advance, and a script that reruns the whole thing on demand. This post walks through building exactly that, using nothing but free-tier model access, and ends with an honest account of where the approach breaks.

Why impressions fail as evidence

A code review assistant has to do three distinct jobs, and a casual demo conflates all of them:

• Detection — flag the lines where behavior is actually wrong.
• Explanation — say why it's wrong in a way a maintainer can act on.
• Restraint — stay quiet about the ninety percent of the file that's fine.

Most demo-style testing only exercises the first two, because a pasted snippet full of defects invites comments. Restraint — the property that determines whether your team will mute the tool within a week — is invisible until you measure false alarms on clean code. So the experiment below tests all three jobs separately.

The artifact: a mutation-graded scoreboard

The design has three pieces: a mutation corpus, a clean-file control group, and a scoring script.

1. Mutate real code, don't write fake buggy code

Handwritten "buggy example" files tend to look like textbook exercises. Real defects hide inside otherwise-plausible logic. So start from files in a project you maintain and apply one small mutation per copy:

• delete one line of a guard clause (if (!user) return; disappears),
• flip a boundary (<= becomes <),
• swap the order of two dependent statements,
• remove one element from a cleanup list (a close() call vanishes from a finally block),
• change one argument in a call to a similar-typed wrong value.

Each mutation produces a diff of one or two lines against code that previously worked. That's the closest you can get to "a bug a tired teammate would write." Save the original as the control file. For every mutated copy, record the expected verdict in a manifest:

{
  "case_id": "cart-total-boundary",
  "file": "mutated/cart_total_03.js",
  "control": false,
  "expect_finding": true,
  "severity": "high",
  "hint_terms": ["off-by-one", "boundary", "total"]
}

The control files get the same manifest entries with control: true and expect_finding: false. Aim for roughly a 2:1 ratio of mutated to clean files — enough clean cases that false positives actually show up in your numbers.

2. A runner that scores severity, not vibes

Here's the harness, in Node this time, since most teams evaluating review tools live in a JS or TS codebase anyway. It posts each file to a configurable endpoint, applies a severity-weighted rubric, and — this is the part most evaluations skip — it reruns each mutated case after renaming the local variables, to check whether findings survive a superficial rewrite:

#!/usr/bin/env node
// review-scoreboard.mjs
// Usage: REVIEW_URL=... REVIEW_TOKEN=... node review-scoreboard.mjs manifest.json
// Provider-agnostic: adjust buildRequest/parseReply to your endpoint's shape.

import { readFileSync } from "node:fs";

const URL_ = process.env.REVIEW_URL;
const TOKEN = process.env.REVIEW_TOKEN ?? "";

const SYSTEM =
  "You are a code reviewer. Report only correctness defects. " +
  "Format each finding as: @<line> [severity:low|med|high] <reason>. " +
  "If the code is correct, reply with exactly: CLEAN";

const SEVERITY_POINTS = { high: 3, med: 2, low: 1 };

async function review(code) {
  const res = await fetch(URL_, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${TOKEN}`,
    },
    body: JSON.stringify({ system: SYSTEM, input: code }),
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  const body = await res.json();
  return body.output; // adapt to your provider
}

// Superficial rename: swap identifier spellings so we can test
// whether the model is reasoning about structure or pattern-matching names.
function renameNoise(code) {
  return code
    .replaceAll("total", "acc")
    .replaceAll("items", "rows")
    .replaceAll("user", "acct");
}

function grade(reply, testCase) {
  if (testCase.control) {
    const noisy = reply.trim().toUpperCase() !== "CLEAN";
    return { kind: "control", falseAlarm: noisy };
  }
  const mentionsHint = testCase.hint_terms.some((t) =>
    reply.toLowerCase().includes(t.toLowerCase()),
  );
  const sevMatch = reply.match(/severity:(high|med|low)/i);
  const sev = sevMatch ? sevMatch[1].toLowerCase() : null;
  return {
    kind: "mutated",
    found: mentionsHint,
    severityRight: sev === testCase.severity,
    points: mentionsHint ? SEVERITY_POINTS[testCase.severity] : 0,
  };
}

const manifest = JSON.parse(readFileSync(process.argv[2], "utf8"));
const rows = [];

for (const c of manifest.cases) {
  const code = readFileSync(c.file, "utf8");
  const first = await review(code);
  const g1 = grade(first, c);

  let stable = null;
  if (!c.control) {
    const second = await review(renameNoise(code));
    stable = grade(second, c).found;
  }
  rows.push({ id: c.case_id, ...g1, stableAfterRename: stable });
}

const mutated = rows.filter((r) => r.kind === "mutated");
const controls = rows.filter((r) => r.kind === "control");

console.log(`detection rate : ${mutated.filter((r) => r.found).length}/${mutated.length}`);
console.log(`severity match : ${mutated.filter((r) => r.severityRight).length}/${mutated.length}`);
console.log(`false alarms   : ${controls.filter((r) => r.falseAlarm).length}/${controls.length}`);
console.log(`rename-stable  : ${mutated.filter((r) => r.stableAfterRename).length}/${mutated.length}`);
console.table(rows);

Three design choices are worth explaining:

1. The rename pass. If a finding evaporates when total becomes acc, the tool was keying on the identifier, not the logic. A reviewer that only catches bugs in conventionally-named variables will underperform on your real codebase, where names are idiosyncratic. This is a cheap proxy for robustness that most evaluations never run.
2. Severity weighting. Missing a low-severity nit and missing a data-corruption bug should not count the same. The rubric doesn't have to be sophisticated — it has to be written down before you look at the results, so you can't nudge it afterward.
3. The control group is mandatory. A tool that flags everything scores 100% on detection and is useless in practice. The false-alarm row is often the number that actually decides the purchase.

3. What a run looks like

A corpus of 15 cases (10 mutated, 5 clean) takes a few minutes end to end, including the rename pass. The output might read:

detection rate : 7/10
severity match : 4/10
false alarms   : 1/5
rename-stable  : 5/10

That single block tells a richer story than any demo: decent recall, weak severity calibration, acceptable noise, and — the worrying line — half the findings don't survive a variable rename. Whether that's a dealbreaker depends on your codebase, and now you can have that argument with numbers instead of adjectives.

Why this works on free tiers (and where MonkeyCode fits)

Notice the arithmetic: the whole experiment above is about 25 requests of moderate size. That's evaluation-scale traffic, not production-scale — precisely the gap that free access exists to fill. You should not need a credit card to find out whether a review model can tell < from <=.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, MonkeyCode currently provides free model access plus a free server option, so it's a practical backend for REVIEW_URL during this stage — the script above doesn't care what's behind the endpoint as long as you adapt the request and response shape. If you'd like a zero-cost target for your first scoreboard run, point the harness at it and see what the four summary lines say; keep the corpus around either way, because you'll want to re-run it against whatever you evaluate next.

The methodology is the point, though. Free access is for deciding; the decision criteria should be provider-neutral.

Should you rely on a free setup? A quick self-check

Answer yes or no to each:

• [ ] You're comparing tools or models before committing budget.
• [ ] Your review traffic is bursty (a PR here and there), not continuous.
• [ ] Your diffs fit comfortably in one request — no monorepo mega-files.
• [ ] Losing the service tomorrow would be an inconvenience, not an outage.
• [ ] The provider's data-handling terms cover the code you'd send.

Mostly yes: a free tier is a legitimate home for your evaluation and light ongoing use. Two or more no answers — especially the last two — and you already know the experiment's conclusion: you'll need a paid arrangement with guarantees, and the scoreboard's job is simply to tell you which one earns it.

Where this method falls short

• Building the corpus is the expensive part. Fifteen well-chosen mutations plus controls is a solid half day of work. A sloppy corpus produces confident, meaningless numbers — worse than no numbers, because they're harder to argue with.
• Hint-term matching undercounts paraphrase. A model that writes "the loop skips the final element" instead of "off-by-one" gets marked wrong by this scorer. Treat the script's output as a lower bound and hand-read every miss once before trusting the rate.
• Fifteen cases rank tools coarsely. You can separate "clearly broken" from "plausibly good," not "2% better than the alternative." Fine-grained ranking needs hundreds of cases, which usually outgrows free access by itself.
• Free offerings are moving targets. Lineups, limits, and terms on any free plan can change; never wire one into a path your team depends on.
• If you already have a funded, functioning review pipeline, this experiment answers a question you've already settled — skip it.

Closing thought

The scoreboard, not the subscription, is the asset. Once you have a mutated corpus, a control group, and a rubric you wrote down in advance, every future evaluation — a new model, a new vendor, a prompt change — becomes a five-minute rerun instead of a fresh round of gut feelings. Start it on free access, and let the four summary lines make the budget case for you.