DEV Community: Riley Zhu

Every Week a New Model Drops. Here's the 30-Minute Eval I Run Before Believing the Hype

Riley Zhu — Mon, 10 Aug 2026 11:02:29 +0000

A new open model release trends, the benchmarks look incredible, and by the time I've read the third hot take I still have no idea whether it's actually good for my work. The recent chatter around MiniMax's H3 release is a perfect example: lots of excitement, lots of leaderboard screenshots, and very little about how it behaves on the boring tasks I actually do every day.

So I stopped reading takes and built a tiny ritual instead: a 30-minute, reproducible smoke test I run against any newly hyped model before I let it anywhere near a real project. This post is that ritual, plus the exact script.

The problem with launch-week benchmarks

Public benchmarks answer "is this model smart?" I need answers to different questions:

Does it follow my prompt style, or does it need babysitting?
Does it degrade gracefully on ambiguous input, or confidently invent APIs?
Is it consistent across runs, or did I get a lucky sample?

These are cheap to test. The only real blocker used to be access: spinning up an environment and getting API keys for every new release is friction, and friction means I skip the eval and just trust the hype. These days I run the eval through MonkeyCode, which offers free model access and a free server option, so the cost of satisfying my curiosity is basically zero.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The artifact: a fixed five-task gauntlet

The trick is that the tasks never change. Only the model does. I keep five prompts that map to my actual daily work:

#	Task	What I'm really measuring
1	Explain a unfamiliar stack trace	Reading comprehension, not just generation
2	Refactor a deliberately smelly function	Taste: does it over-engineer?
3	Write a function against an underspecified docstring	Hallucination under ambiguity
4	Fix a bug where the "obvious" fix is wrong	Resistance to suggestion
5	Summarize a 200-line diff	Long-context fidelity

Task 4 deserves a note: I plant a red herring in the prompt ("I think the issue is the cache invalidation") when the actual bug is an off-by-one. Models that blindly agree with me fail. Models that push back with reasoning pass.

Here's the harness. It's deliberately boring — stdlib only, so it runs anywhere:

#!/usr/bin/env python3
"""model_gauntlet.py — run a fixed eval suite against any OpenAI-compatible endpoint.
Usage: MODEL=minimax-h3 python model_gauntlet.py
"""
import json, os, time, urllib.request

ENDPOINT = os.environ.get("MC_BASE_URL", "http://localhost:8000/v1")  # free server default
API_KEY = os.environ.get("MC_API_KEY", "none")
MODEL = os.environ["MODEL"]

TASKS = [
    {"id": "trace",    "prompt_file": "tasks/01_stacktrace.md"},
    {"id": "refactor", "prompt_file": "tasks/02_refactor.md"},
    {"id": "ambig",    "prompt_file": "tasks/03_ambiguous_spec.md"},
    {"id": "redherring","prompt_file": "tasks/04_red_herring_bug.md"},
    {"id": "diffsum",  "prompt_file": "tasks/05_diff_summary.md"},
]
RUNS_PER_TASK = 3  # consistency check: same prompt, 3 runs

def chat(prompt: str) -> str:
    req = urllib.request.Request(
        f"{ENDPOINT}/chat/completions",
        data=json.dumps({
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.7,
        }).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r)["choices"][0]["message"]["content"]

results = {}
for task in TASKS:
    prompt = open(task["prompt_file"]).read()
    runs = []
    for i in range(RUNS_PER_TASK):
        t0 = time.time()
        out = chat(prompt)
        runs.append({"latency_s": round(time.time() - t0, 2), "output": out})
    results[task["id"]] = runs

with open(f"results_{MODEL}_{int(time.time())}.json", "w") as f:
    json.dump(results, f, indent=2)
print(f"done -> results_{MODEL}_*.json (review manually, see rubric below)")

Notice what's not here: automated scoring. I tried LLM-as-judge scoring and it mostly measured how much the judge model liked its own writing style. For five tasks, reading fifteen outputs takes me twenty minutes and my judgment is the metric I actually care about.

The scoring rubric (this is the part that matters)

For each task, I grade each run on three axes, 0–2 each:

Correctness — would I ship this with light editing?
Calibration — when uncertain, does it say so?
Consistency — do the three runs agree in approach, or is it a slot machine?

A model that scores 2/2/0 is worse for me than one scoring 1/1/2, because unpredictability compounds in an agentic loop. This is the insight launch-week benchmarks never give you.

Where the free server comes in

Because the harness only needs an OpenAI-compatible endpoint, I point it at whatever I'm testing. When a release like H3 starts trending, I spin it up on MonkeyCode's free server option, run the gauntlet, read the outputs over coffee, and write three bullet points in my notes. Total cost: half an hour and zero dollars. The "open source spirit" angle matters here too — the value of open-weight releases is precisely that anyone can poke at them this way instead of trusting a vendor blog post, and tooling that lowers the barrier to doing that is doing the ecosystem a genuine favor.

Limitations, and who should skip this

Five tasks is not a benchmark. This tells you whether a model fits your workflow. It says nothing about its global ranking, and it shouldn't be cited as one.
My rubric encodes my biases. I weight consistency heavily because I run agentic workflows. If you do one-shot creative writing, you'd weight it differently — and you should build your own task list, not copy mine verbatim.
Free tiers have limits. Availability, throughput, and which models are offered can change. Don't build CI infrastructure that depends on a free tier staying free; do use it for exactly this kind of disposable exploration.
If you need rigorous evals — compliance, safety, regression gating for production — you want a real harness like promptfoo or inspect_ai with versioned datasets, not a coffee-break script.

The actual takeaway

The next time a model release dominates your feed, don't argue about it — test it. A fixed task list, three runs each, a rubric you wrote yourself, and thirty minutes. You'll end up with something no leaderboard can give you: evidence about how the model behaves in your hands.

If you've got a similar ritual, I'd genuinely like to hear what tasks made your list — the red-herring-bug test is the one I'm most curious to see other people adapt.

Judge AI Code Review With a Scoreboard, Not a Demo: A Repeatable Experiment That Runs on Free Access

Riley Zhu — Mon, 10 Aug 2026 08:37:41 +0000

There's a familiar ritual when a team considers an AI review assistant: someone opens the product, feeds it a file from the current sprint, watches it produce a confident-sounding paragraph, and reports back that it "seems pretty good." Two months later nobody can say whether the subscription is catching real bugs or just generating plausible noise.

The failure isn't in the tool — it's in the test. A one-shot impression can't separate a reviewer that finds defects from one that merely sounds like a reviewer. What you need instead is a scoreboard: a fixed set of inputs with known answers, a scoring rule you wrote down in advance, and a script that reruns the whole thing on demand. This post walks through building exactly that, using nothing but free-tier model access, and ends with an honest account of where the approach breaks.

Why impressions fail as evidence

A code review assistant has to do three distinct jobs, and a casual demo conflates all of them:

Detection — flag the lines where behavior is actually wrong.
Explanation — say why it's wrong in a way a maintainer can act on.
Restraint — stay quiet about the ninety percent of the file that's fine.

Most demo-style testing only exercises the first two, because a pasted snippet full of defects invites comments. Restraint — the property that determines whether your team will mute the tool within a week — is invisible until you measure false alarms on clean code. So the experiment below tests all three jobs separately.

The artifact: a mutation-graded scoreboard

The design has three pieces: a mutation corpus, a clean-file control group, and a scoring script.

1. Mutate real code, don't write fake buggy code

Handwritten "buggy example" files tend to look like textbook exercises. Real defects hide inside otherwise-plausible logic. So start from files in a project you maintain and apply one small mutation per copy:

delete one line of a guard clause (if (!user) return; disappears),
flip a boundary (<= becomes <),
swap the order of two dependent statements,
remove one element from a cleanup list (a close() call vanishes from a finally block),
change one argument in a call to a similar-typed wrong value.

Each mutation produces a diff of one or two lines against code that previously worked. That's the closest you can get to "a bug a tired teammate would write." Save the original as the control file. For every mutated copy, record the expected verdict in a manifest:

{
  "case_id": "cart-total-boundary",
  "file": "mutated/cart_total_03.js",
  "control": false,
  "expect_finding": true,
  "severity": "high",
  "hint_terms": ["off-by-one", "boundary", "total"]
}

The control files get the same manifest entries with control: true and expect_finding: false. Aim for roughly a 2:1 ratio of mutated to clean files — enough clean cases that false positives actually show up in your numbers.

2. A runner that scores severity, not vibes

Here's the harness, in Node this time, since most teams evaluating review tools live in a JS or TS codebase anyway. It posts each file to a configurable endpoint, applies a severity-weighted rubric, and — this is the part most evaluations skip — it reruns each mutated case after renaming the local variables, to check whether findings survive a superficial rewrite:

#!/usr/bin/env node
// review-scoreboard.mjs
// Usage: REVIEW_URL=... REVIEW_TOKEN=... node review-scoreboard.mjs manifest.json
// Provider-agnostic: adjust buildRequest/parseReply to your endpoint's shape.

import { readFileSync } from "node:fs";

const URL_ = process.env.REVIEW_URL;
const TOKEN = process.env.REVIEW_TOKEN ?? "";

const SYSTEM =
  "You are a code reviewer. Report only correctness defects. " +
  "Format each finding as: @<line> [severity:low|med|high] <reason>. " +
  "If the code is correct, reply with exactly: CLEAN";

const SEVERITY_POINTS = { high: 3, med: 2, low: 1 };

async function review(code) {
  const res = await fetch(URL_, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${TOKEN}`,
    },
    body: JSON.stringify({ system: SYSTEM, input: code }),
  });
  if (!res.ok) throw new Error(`HTTP ${res.status}`);
  const body = await res.json();
  return body.output; // adapt to your provider
}

// Superficial rename: swap identifier spellings so we can test
// whether the model is reasoning about structure or pattern-matching names.
function renameNoise(code) {
  return code
    .replaceAll("total", "acc")
    .replaceAll("items", "rows")
    .replaceAll("user", "acct");
}

function grade(reply, testCase) {
  if (testCase.control) {
    const noisy = reply.trim().toUpperCase() !== "CLEAN";
    return { kind: "control", falseAlarm: noisy };
  }
  const mentionsHint = testCase.hint_terms.some((t) =>
    reply.toLowerCase().includes(t.toLowerCase()),
  );
  const sevMatch = reply.match(/severity:(high|med|low)/i);
  const sev = sevMatch ? sevMatch[1].toLowerCase() : null;
  return {
    kind: "mutated",
    found: mentionsHint,
    severityRight: sev === testCase.severity,
    points: mentionsHint ? SEVERITY_POINTS[testCase.severity] : 0,
  };
}

const manifest = JSON.parse(readFileSync(process.argv[2], "utf8"));
const rows = [];

for (const c of manifest.cases) {
  const code = readFileSync(c.file, "utf8");
  const first = await review(code);
  const g1 = grade(first, c);

  let stable = null;
  if (!c.control) {
    const second = await review(renameNoise(code));
    stable = grade(second, c).found;
  }
  rows.push({ id: c.case_id, ...g1, stableAfterRename: stable });
}

const mutated = rows.filter((r) => r.kind === "mutated");
const controls = rows.filter((r) => r.kind === "control");

console.log(`detection rate : ${mutated.filter((r) => r.found).length}/${mutated.length}`);
console.log(`severity match : ${mutated.filter((r) => r.severityRight).length}/${mutated.length}`);
console.log(`false alarms   : ${controls.filter((r) => r.falseAlarm).length}/${controls.length}`);
console.log(`rename-stable  : ${mutated.filter((r) => r.stableAfterRename).length}/${mutated.length}`);
console.table(rows);

Three design choices are worth explaining:

The rename pass. If a finding evaporates when total becomes acc, the tool was keying on the identifier, not the logic. A reviewer that only catches bugs in conventionally-named variables will underperform on your real codebase, where names are idiosyncratic. This is a cheap proxy for robustness that most evaluations never run.
Severity weighting. Missing a low-severity nit and missing a data-corruption bug should not count the same. The rubric doesn't have to be sophisticated — it has to be written down before you look at the results, so you can't nudge it afterward.
The control group is mandatory. A tool that flags everything scores 100% on detection and is useless in practice. The false-alarm row is often the number that actually decides the purchase.

3. What a run looks like

A corpus of 15 cases (10 mutated, 5 clean) takes a few minutes end to end, including the rename pass. The output might read:

detection rate : 7/10
severity match : 4/10
false alarms   : 1/5
rename-stable  : 5/10

That single block tells a richer story than any demo: decent recall, weak severity calibration, acceptable noise, and — the worrying line — half the findings don't survive a variable rename. Whether that's a dealbreaker depends on your codebase, and now you can have that argument with numbers instead of adjectives.

Why this works on free tiers (and where MonkeyCode fits)

Notice the arithmetic: the whole experiment above is about 25 requests of moderate size. That's evaluation-scale traffic, not production-scale — precisely the gap that free access exists to fill. You should not need a credit card to find out whether a review model can tell < from <=.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, MonkeyCode currently provides free model access plus a free server option, so it's a practical backend for REVIEW_URL during this stage — the script above doesn't care what's behind the endpoint as long as you adapt the request and response shape. If you'd like a zero-cost target for your first scoreboard run, point the harness at it and see what the four summary lines say; keep the corpus around either way, because you'll want to re-run it against whatever you evaluate next.

The methodology is the point, though. Free access is for deciding; the decision criteria should be provider-neutral.

Should you rely on a free setup? A quick self-check

Answer yes or no to each:

[ ] You're comparing tools or models before committing budget.
[ ] Your review traffic is bursty (a PR here and there), not continuous.
[ ] Your diffs fit comfortably in one request — no monorepo mega-files.
[ ] Losing the service tomorrow would be an inconvenience, not an outage.
[ ] The provider's data-handling terms cover the code you'd send.

Mostly yes: a free tier is a legitimate home for your evaluation and light ongoing use. Two or more no answers — especially the last two — and you already know the experiment's conclusion: you'll need a paid arrangement with guarantees, and the scoreboard's job is simply to tell you which one earns it.

Where this method falls short

Building the corpus is the expensive part. Fifteen well-chosen mutations plus controls is a solid half day of work. A sloppy corpus produces confident, meaningless numbers — worse than no numbers, because they're harder to argue with.
Hint-term matching undercounts paraphrase. A model that writes "the loop skips the final element" instead of "off-by-one" gets marked wrong by this scorer. Treat the script's output as a lower bound and hand-read every miss once before trusting the rate.
Fifteen cases rank tools coarsely. You can separate "clearly broken" from "plausibly good," not "2% better than the alternative." Fine-grained ranking needs hundreds of cases, which usually outgrows free access by itself.
Free offerings are moving targets. Lineups, limits, and terms on any free plan can change; never wire one into a path your team depends on.
If you already have a funded, functioning review pipeline, this experiment answers a question you've already settled — skip it.

Closing thought

The scoreboard, not the subscription, is the asset. Once you have a mutated corpus, a control group, and a rubric you wrote down in advance, every future evaluation — a new model, a new vendor, a prompt change — becomes a five-minute rerun instead of a fresh round of gut feelings. Start it on free access, and let the four summary lines make the budget case for you.