Build a Personal Model Bake-Off: Testing Free AI Assistants on Your Real Bugs

Last month I almost subscribed to an AI coding tool based on a comparison chart someone posted. Then I caught myself: that chart was built on competitive-programming puzzles and greenfield demos. My day job is gluing endpoints onto a five-year-old codebase with custom lint rules and a module nobody wants to touch. The chart couldn't tell me anything about that.

So instead of picking a model from someone else's data, I ran a small bake-off on my own closed issues — using only free access, so the whole experiment cost nothing and committed me to nothing. This post is the recipe. The tooling is trivial; the discipline is the actual content.

The problem with borrowed benchmarks

Public leaderboards answer a question I rarely have: "which model writes the best sorting algorithm from scratch?" My real questions are different:

• Does the model invent packages that aren't in my package.json?
• Can it follow my repo's conventions (named exports, our logger, our error wrapper) without being told twice?
• When it touches the legacy module, does it produce a minimal patch or a confident rewrite?

None of those show up in generic evals, because they depend on context only I have. Which means the only benchmark that matters for the buying decision is one I run myself, on tasks where I already know what correct looks like.

Step 1: Assemble a task deck from your own history

I pulled seven recently closed issues and turned each into a card:

---
id: retry-regression-214
prompt: |
  The retry helper in src/net/retry.ts double-fires onAbort when
  the request times out. Fix it without changing the public API.
context_files: [src/net/retry.ts, src/net/__tests__/retry.test.ts]
known_good: one-line guard on settled flag; all existing tests pass
trap: models love rewriting the whole function and breaking test #4
---

Rules I set for the deck:

1. Small enough to judge in minutes. If a wrong answer isn't obviously wrong, the task is too big for this experiment.
2. Solved already. I need to know the correct fix so grading is fast and grounded.
3. Mixed shape. Mine was roughly half "wire X into existing Y", a quarter "fix this regression", a quarter "explain this behavior". Match your own workload, not mine.

The trap field is the most valuable part: I wrote down, in advance, how I expected models to fail. That turns vague impressions ("model B felt sloppy") into checkable predictions.

Step 2: Collect outputs, then blind the grading

I ran each card against every model I could reach for free, saved each raw output to its own file, and — this part matters — renamed the files so I graded without knowing which model wrote what. Self-graded, unblinded comparisons are mostly astrology.

Each graded output becomes one JSON record:

{
  "task": "retry-regression-214",
  "output_file": "out-07.patch",
  "compiles": true,
  "tests_pass": false,
  "invented_dependencies": [],
  "review_notes": "rewrote retry.ts wholesale; breaks test #4 as predicted",
  "would_merge": false
}

Two of those fields must never be filled in by eye: compiles and tests_pass come from actually executing the patch. I run that in a throwaway environment (I wrote about a free sandbox harness in an earlier post) so half-broken candidate code never touches my daily machine.

Step 3: Score with explicit, arguable weights

Here's the whole scorer — dependency-free Node, deliberately boring:

// bakeoff.mjs — node bakeoff.mjs graded/
import { readFileSync, readdirSync } from 'node:fs';
import { join } from 'node:path';

const dir = process.argv[2] ?? 'graded';

// My priorities. Yours will differ — that is the feature.
const score = (r) =>
  (r.compiles ? 1 : 0) +
  (r.tests_pass ? 3 : 0) +
  (r.would_merge ? 3 : 0) -
  2 * (r.invented_dependencies?.length ?? 0);

const rows = readdirSync(dir)
  .filter((f) => f.endsWith('.json'))
  .map((f) => JSON.parse(readFileSync(join(dir, f), 'utf8')));

const perModel = new Map();
for (const r of rows) {
  const m = perModel.get(r.output_file.slice(0, 3)) ?? [];
  m.push({ task: r.task, score: score(r) });
  perModel.set(r.output_file.slice(0, 3), m);
}

for (const [model, entries] of perModel) {
  const avg = entries.reduce((s, e) => s + e.score, 0) / entries.length;
  const worst = entries.sort((a, b) => a.score - b.score)[0];
  console.log(`${model}  avg=${avg.toFixed(2)}  weakest=${worst.task} (${worst.score})`);
}

The weights are an argument with myself, written down: passing my tests and being mergeable count triple, a fabricated dependency is a hard penalty. Your weights should encode your pain — if you mostly want explanation quality, would_merge shouldn't dominate. What matters is that the rubric exists before you look at results, not that it's perfect.

Running it for exactly zero dollars

The two things that usually make this annoying are needing several models to compare and needing a machine that isn't yours to run candidate code on.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode for both legs here: it currently offers free access to a selection of models and a free server option, which covered the full loop — same task deck pointed at multiple models, compile/test checks executed on their server, no credit card and nothing installed locally. The scorer above is indifferent to where outputs come from, though; pasting from any chat UI into the JSON files works identically. One deliberate omission: I recorded nothing about speed. Free-tier latency shifts with load and provider decisions, so a number I published would age badly — my rubric judges patch quality only.

What the bake-off actually told me

The headline wasn't a winner. It was three quieter findings:

• Task type dominated model choice. Everything I tested handled "add a guard clause" fine, and everything flailed at diagnosing a flaky test — until I pasted the last three failure logs into the prompt, at which point every model improved. My context-gathering was the bottleneck, not the models.
• One model kept inventing an internal-sounding package name. Not a famous library — a plausible, boring name that would pass a skim and fail npm install. That check now lives permanently in my rubric; a public benchmark would never have surfaced it against my dependency tree.
• A passing patch still got rejected. One output passed all tests by duplicating logic that belongs in our shared module. tests_pass said yes, would_merge said no. Keep both columns; CI alone would have crowned that output.

Honest limitations

• Seven tasks is a vibe with structure, not statistics. Treat the outcome as "which model deserves a longer trial", not a ranking. Anything within a point or two on my scale is noise.
• Blinding is partial. Model styles leak through phrasing. Having a teammate grade a few cards is strictly better; I couldn't, so I graded before unblinding and accepted the bias.
• Free access is a moving target. Which models are available, and on what terms, is whatever the provider says that week. This is a disposable experiment — do not wire it into CI or make purchasing commitments that assume any free tier persists.
• Skip the whole thing if you're on a common stack where any mainstream assistant already does fine, or if you're a solo dev who can just trial tools sequentially. The bake-off earns its overhead when you're choosing for a team, or when generic model advice keeps failing on your codebase.

The smallest useful version

Three cards, two models, one afternoon, zero spend. If you want a no-cost place to run the loop, MonkeyCode's free model access plus free server covers it end to end — but the deck, the blinded grading, and the explicit weights are the real artifact, and they travel with you to any provider. Build the benchmark that knows your codebase, because nobody else's does.