Scroll DEV this week and you'll see the same pattern: agents everywhere, and everyone arguing about which model "codes better." One recent discussion made a sharp point — metrics from sub-agents aren't comparable to main-thread metrics — and the replies were full of people realizing their own comparisons were just vibes with extra steps.
That stung, because it's true of most of us. We paste the same prompt into two models, skim the outputs, and declare a winner. That's not an evaluation; it's a coin flip with markdown.
This article builds a tiny, boring, repeatable harness: 20 fixed coding tasks, deterministic scoring where possible, structured human rubric scoring where not, and a runner you can re-execute against any OpenAI-compatible endpoint. Total setup is one Python file and one JSONL file. No framework, no database, no dashboard.
Why a harness instead of a leaderboard
Public benchmarks measure what their authors cared about. You care about your failure modes: does the model hallucinate imports in your stack? Does it write tests that pass but assert nothing? Does it mangle SQL edge cases? A 20-task suite you own answers those questions in an afternoon, and — critically — you can re-run it every time a model updates, because model behavior drifts.
The other half of the problem is cost. Running a suite 3 times across 4 candidate models is 240 generations. That's exactly the situation where a free tier matters more than raw leaderboard position: you want iteration volume, not prestige.
This is where I've been pointing the harness at MonkeyCode. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The two things that make it practical here are the free model access (so a full sweep doesn't show up on a bill) and the free server option (so the runner has somewhere to live without me provisioning anything). Everything below works against any OpenAI-compatible API, though — the harness doesn't care who hosts the model.
The artifact: a minimal repeatable runner
Two files. First, tasks.jsonl — one task per line. Three scored types: unit (deterministic pass/fail), rubric (you grade 0–3), and diff (output must match a golden patch). Keep tasks small and specific to your work:
{"id":"py-01","type":"unit","prompt":"Write parse_iso_week(s) returning (year, week). Raise ValueError on bad input.","tests":"assert parse_iso_week('2024-W09')==(2024,9)\ntry:\n parse_iso_week('W09')\n assert False\nexcept ValueError: pass"}
{"id":"sql-03","type":"rubric","prompt":"Given tables orders(id,user_id,total,created_at), write a query for each user's second-most-recent order.","rubric":"uses window function or correct self-join; handles ties; no N+1"}
{"id":"js-07","type":"unit","prompt":"Implement debounce(fn, ms) that flushes trailing call.","tests":"let n=0;const f=debounce(()=>n++,50);f();f();f();setTimeout(()=>{if(n!==1)throw Error(n)},120)"}
Then run.py:
import json, subprocess, sys, tempfile, time
from openai import OpenAI # works with any OpenAI-compatible base_url
client = OpenAI(base_url="http://localhost:8000/v1", api_key="local") # free server endpoint
MODEL = sys.argv[1]
def run_task(t):
for attempt in range(3): # 3 samples: consistency is part of the score
resp = client.chat.completions.create(
model=MODEL, temperature=0.2,
messages=[{"role": "user", "content": t["prompt"] + "\nReturn only code."}],
)
code = resp.choices[0].message.content
if t["type"] == "unit":
lang = "python3" if t["id"].startswith("py") else "node"
with tempfile.NamedTemporaryFile("w", suffix=".py" if lang=="python3" else ".js", delete=False) as f:
f.write(code + "\n" + t["tests"]); f.flush()
ok = subprocess.run([lang, f.name], capture_output=True, timeout=30).returncode == 0
yield {"id": t["id"], "attempt": attempt, "pass": ok}
else: # rubric: save for human grading, never auto-judge
yield {"id": t["id"], "attempt": attempt, "output": code}
results = []
for line in open("tasks.jsonl"):
results.extend(run_task(json.loads(line)))
json.dump({"model": MODEL, "ts": time.time(), "results": results},
open(f"results-{MODEL}.json", "w"), indent=2)
unit = [r for r in results if "pass" in r]
print(f"unit pass-rate: {sum(r['pass'] for r in unit)}/{len(unit)}")
Run it per model: python run.py some-model-a, python run.py some-model-b. Then compare the result files.
Three design choices that matter more than the code
- Three samples per task at fixed temperature. A model that solves a task 1/3 times and one that solves it 3/3 have the same "best of" demo and completely different production value. Single-shot comparisons hide this.
- Deterministic scoring only where it's honest. Unit tests can auto-grade. "Is this SQL idiomatic?" cannot — save rubric outputs and grade them yourself in one sitting, blind if you can. Auto-judging quality with another LLM just moves the vibes somewhere you can't see them.
- Freeze everything. Same prompts, same temperature, same system message (none), same task file in git. The moment you tweak a prompt between models, you're benchmarking your prompting, not the models.
When this harness is the wrong tool
| Situation | Do this instead |
|---|---|
| Choosing a model for a one-off script | Just use whichever is already open |
| Evaluating agentic, multi-step workflows | You need trajectory logging, not single completions — different harness entirely |
| Regulated/compliance-bound code | 20 tasks can't establish fitness; use your org's review process |
| Comparing published benchmark scores to your results | Don't. Different distributions; only compare within your own runs |
| Tasks under NDA or proprietary code | Check the provider's data terms before sending anything; a free tier is not a data-handling guarantee |
Limitations, honestly
Twenty tasks is a smoke test, not a proof. Pass/fail on toy functions won't predict behavior on a 40-file refactor. Temperature 0.2 is a compromise — real usage varies. Rubric grading is still subjective; it just makes the subjectivity explicit and re-checkable. And any free tier — MonkeyCode's included — can change limits or availability, so keep the endpoint configurable rather than hard-coding your workflow around one provider.
The point
The agents conversation this week keeps circling the same question: how do you know any of this actually works on your problems? The answer isn't a better leaderboard; it's a smaller, dumber, more honest loop that you can re-run whenever something changes. Two files, twenty tasks, three samples. If you want a zero-cost place to start sweeping candidate models, MonkeyCode's free models and free server are a reasonable first endpoint — but the harness is the part worth keeping, and it goes wherever your models do.
What tasks would go in your tasks.jsonl? The failure modes you pick say more about your stack than any benchmark will.
Top comments (0)