Every couple of weeks my feed lights up with a new coding model and the same conversation restarts from zero. Someone posts a screenshot, someone else posts a counter-screenshot, and the thread ends with no shared evidence about the only thing that matters: how the model behaves on the work sitting in our queue right now.
This piece is my answer to that loop. Instead of arguing about models in the abstract, I keep a small, permanent evaluation rig that treats every new release as just another contestant. The rig doesn't care what's trending this week — swap one adapter, rerun the suite, get a fresh verdict. What follows is the reasoning behind the design, a working skeleton you can adapt, and an honest list of the ways it can mislead you.
Why ad-hoc testing keeps failing teams
The typical evaluation looks like this: open the new tool, paste in something from memory, nod or shrug, form an opinion. There are three structural problems with that.
First, the test task is chosen after seeing the hype, which quietly biases toward tasks where the new model is rumored to be strong. Second, there is no control run — nothing to compare the output against except vibes. Third, nothing is written down, so when the next model arrives two weeks later, the whole exercise repeats from scratch with a slightly different gut feeling.
The alternative is to separate the fixture from the contestant. The fixture — your task set, your prompt wording, your scoring sheet — stays frozen. Only the contestant changes. That single discipline converts an opinion-forming exercise into a measurement you can rerun, share, and defend.
Building a fixture that survives the hype cycle
Before touching any model, assemble the fixed half of the experiment.
Task selection. Mine your own history: last quarter's pull requests, incident write-ups, and code review threads. I aim for roughly a dozen items spanning distinct failure modes — a race condition buried in a concurrency test, a refactor that must preserve a public API, a review comment demanding a surgical reply, a docstring for a function full of edge cases. Real tasks beat synthetic ones because they carry the ambiguity and missing context that production work always has.
Sanitization. Everything that goes into a prompt leaves your machine, so strip credentials, internal hostnames, customer names, and anything your compliance posture forbids. If a task can't be cleaned without losing its essence, drop it — the suite is a sample, not a census.
Prompt wording. Freeze one phrasing per task and treat that text as part of the fixture. If you tweak wording between contestants, you're measuring prompt sensitivity, not model quality. Version the prompt file in git like you would a database migration.
Scoring. Decide, in advance, what a passing answer must contain and what disqualifies it. Write it down per task. Scoring rubrics invented after reading the output are just rationalization with extra steps.
The rig: one runner, pluggable contestants
Label: proposal/skeleton — this is a design artifact for you to adapt, not something I executed while writing this article. Keep credentials in environment variables, never in source.
The idea is deliberately unglamorous: each contestant is a tiny adapter script that reads a task envelope from stdin and writes an answer envelope to stdout. The runner walks the frozen suite and invokes every adapter identically.
# bakeoff.py — boring on purpose; rerun it every hype cycle
import json
import os
import subprocess
import time
from pathlib import Path
ROOT = Path(__file__).parent
TASKS = ROOT / "tasks.jsonl"
RESULTS = ROOT / "results.jsonl"
# Each contestant: a command that consumes a task JSON on stdin
# and emits {"answer": ..., "patch": ..., "caveats": ...} on stdout.
CONTESTANTS = {
"incumbent": {
"cmd": ["python", "adapters/incumbent.py"],
"env": {"INCUMBENT_URL": os.getenv("INCUMBENT_URL", "")},
},
"newcomer": {
# Wire this to the model of the week, a local endpoint,
# or any free-tier coding model you're permitted to call.
"cmd": ["python", "adapters/newcomer.py"],
"env": {"NEWCOMER_URL": os.getenv("NEWCOMER_URL", "")},
},
}
def run_one(name, task):
spec = CONTESTANTS[name]
envelope = json.dumps({
"task_id": task["task_id"],
"kind": task["kind"],
"prompt": task["prompt"],
"context_files": task.get("context_files", []),
})
started = time.time()
proc = subprocess.run(
spec["cmd"],
input=envelope,
text=True,
capture_output=True,
timeout=task.get("timeout_s", 120),
env={**os.environ, **spec["env"]},
)
return {
"task_id": task["task_id"],
"contestant": name,
"clean_exit": proc.returncode == 0,
"elapsed_s": round(time.time() - started, 3),
"stdout": proc.stdout[-4000:],
"stderr": proc.stderr[-2000:],
}
def main():
with TASKS.open() as src, RESULTS.open("w") as out:
for line in src:
task = json.loads(line)
for name in CONTESTANTS:
out.write(json.dumps(run_one(name, task)) + "\n")
if __name__ == "__main__":
main()
One entry from tasks.jsonl, so the shape is concrete:
{"task_id":"race-03","kind":"concurrency_diagnosis","prompt":"This integration test deadlocks intermittently under parallel execution. Identify the most likely lock-ordering problem and describe one instrumentation change that would confirm it. Do not rewrite the test yet.","context_files":["integration/test_job_queue.py"],"rubric":{"must":["identifies a specific lock pair","proposes observable evidence"],"must_not":["guesses without a mechanism","invents files not in context"]},"timeout_s":90}
The payoff of the stdin/stdout contract: when the next model trends, you write maybe forty lines of adapter and rerun the exact same suite. The evaluation infrastructure never gets rebuilt; only the contestant roster changes.
Scoring with a checklist, not a vibe
I grade each output against a short checklist drawn from the task's rubric, plus a standing set of gates that apply to every contestant:
| Gate | Green when | Red when |
|---|---|---|
| Scope discipline | Touches only what the task asked | Refactors half the module unprompted |
| Context honesty | Flags missing information explicitly | Hallucinates a config key or endpoint |
| Idiom match | Follows the repo's existing patterns | Imports its own stylistic universe |
| Failure visibility | Wrongness trips CI or review | Wrongness compiles and passes silently |
| Latency sanity | Repeated runs cluster tightly | Random multi-minute stalls, no error |
| Evidence trail | Raw output logged to version control | Record exists only as a screenshot |
The last row matters more than it looks. A bake-off whose artifacts live in git — tasks, prompts, rubrics, raw logs — can be challenged, extended, and rerun by a skeptical teammate. One that lives in a chat window is marketing, regardless of who ran it.
A low-cost venue: MonkeyCode
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The most common objection I hear to structured evaluation is cost: why spend money benchmarking a model you might reject? Two operator-supplied facts make this experiment cheap to try: MonkeyCode offers free model access and a free server option. I'm intentionally not naming specific models, quotas, hardware, or durations — check MonkeyCode's current documentation before depending on any of that, because free tiers change.
In practice, the free model access can fill one contestant slot — incumbent or newcomer, whichever side of the comparison you need — and the free server option can host the runner, the adapters, the task suite, and the logs for a small team. That's sufficient to complete a dozen-task bake-off without a procurement request, which removes the excuse rather than the judgment.
Ways this rig will lie to you
- Benchmark leakage. If a task resembles something in public training corpora, you may be measuring memorization. Prefer tasks from private history over famous puzzles.
- Prompt fragility. Rephrasing a task can flip the ranking. Freeze wording, version it, and resist tuning prompts per contestant — that's rigging the election.
- Plausible wrongness. Models produce confident diffs against incorrect root causes. Wherever possible, grade with executable checks (tests, linters, type checks) rather than reading comprehension.
- Zero cost is not zero risk. A free endpoint still receives your code. Whether anything may leave your perimeter is a security and legal decision; the pricing page cannot answer it.
- Small samples, big conclusions. Twelve tasks can rank two models for your codebase, this month. They cannot crown an industry winner. Publish your log, not a leaderboard.
When to skip the whole thing
Skip this approach if you can't produce sanitized test data, if candidate output can't be quarantined behind CI before touching a real branch, if the workload is a latency-critical path with hard guarantees, or if endpoint selection requires formal sign-off regardless of cost. Also skip it if the goal is to retroactively justify a hot take — a frozen rubric is remarkably good at exposing that.
The durable habit here isn't any particular script; it's treating each model launch as a rerun of an existing experiment rather than a brand-new debate. If you build your own version, the most useful thing you can share isn't a verdict — it's the shape of your task suite and your gate list. If MonkeyCode's free tier lowers the cost of your first run, take it; the rig outlives whichever contestant wins this month.
Top comments (0)