A new open-weight coding model shipped this week, and right on schedule my feed filled with radar charts, victory laps, and disappointed rebuttals. I've watched this movie enough times to know how it ends: everyone argues from the same four benchmark suites, nobody runs the model against the work they actually do, and by next week the chart is obsolete anyway.
A while back I stopped participating. Not because benchmarks are worthless — they answer somebody's question — but because they don't answer mine. If you want authoritative details about a specific release, read the vendor's official model card and repository; everything else, including community reaction, is secondhand. The question I care about is narrower: will this model help with my code, on my machine, at a cost I can sustain? That question has a testable answer, and this article is about how I test it.
Aggregate scores describe someone else's workload
Benchmark suites sample from a distribution, and that distribution is not mine. My week involves resurrecting a CMake setup that three deprecated dependencies just broke, hunting a race condition in a logging thread, and reading template errors that scroll past my terminal's scrollback. A single coding score averages all of that into noise. Worse, it averages in front-end scaffolding, LeetCode-style puzzles, and docstring generation — tasks I rarely need help with.
So the metric I optimize for is unglamorous: replay bugs I've already solved, and see whether the model reaches the same resolution I eventually did. I know the ground truth because I lived it.
Cost is the other half of the equation, and it's where open weights quietly change the game. When the weights are public, many providers can serve the identical model. Hosting becomes a commodity, prices fall, and free tiers start appearing.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Concretely, MonkeyCode offers free access to open models plus a free server option — enough headroom to run a personal evaluation without ever opening a billing page. But nothing below depends on that particular provider. The whole point of the exercise is that it targets any OpenAI-compatible endpoint; the moment your evaluation is tied to one vendor, you've rebuilt the leaderboard problem with extra steps.
The loop: replay, grade, diff
The workflow has three moving parts. Curate a case file from your own debugging history — real failures, where you already know the correct fix. Fire those cases at whatever model you're evaluating. Grade the outputs deterministically, so that re-running the suite next month against the next hot release produces numbers you can diff, not vibes you can't.
#!/usr/bin/env python3
"""replay_eval.py — grade a coding model against YOUR solved bugs.
A template from my own workflow. The example cases are tuned for my
systems-heavy history; build yours to hurt the models where YOUR work lives.
"""
import json, os, subprocess, tempfile, time, urllib.request
ENDPOINT = os.environ.get("EVAL_ENDPOINT", "https://api.example.com/v1")
API_KEY = os.environ.get("EVAL_API_KEY", "")
MODEL_ID = os.environ.get("EVAL_MODEL", "replace-me")
def query(prompt: str) -> str:
body = json.dumps({
"model": MODEL_ID,
"messages": [{"role": "user", "content": prompt}],
"temperature": 0.0, # as close to deterministic as inference gets
}).encode()
req = urllib.request.Request(
f"{ENDPOINT}/chat/completions", data=body,
headers={"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=180) as r:
return json.load(r)["choices"][0]["message"]["content"]
def score(case: dict, answer: str) -> dict:
"""Deterministic grading: required concepts, forbidden claims,
and a compile check when the case ships a code snippet."""
hits = [k for k in case["must_mention"] if k.lower() in answer.lower()]
slips = [k for k in case["must_not_say"] if k.lower() in answer.lower()]
result = {"case": case["id"],
"concepts": f"{len(hits)}/{len(case['must_mention'])}",
"bad_claims": slips}
if "snippet" in case: # e.g. a proposed patch that should compile
with tempfile.NamedTemporaryFile("w", suffix=".c", delete=False) as f:
f.write(case["snippet"].replace("{{MODEL_FIX}}", answer))
path = f.name
ok = subprocess.run(["gcc", "-fsyntax-only", path],
capture_output=True).returncode == 0
result["compiles"] = ok
return result
def main():
with open("my_cases.json") as f:
cases = json.load(f)
rows = []
for case in cases:
t0 = time.time()
answer = query(case["question"])
row = score(case, answer)
row["latency_s"] = round(time.time() - t0, 2)
rows.append(row)
print(json.dumps(row))
fname = f"results_{MODEL_ID.replace('/', '_')}.json"
with open(fname, "w") as f:
json.dump(rows, f, indent=2)
if __name__ == "__main__":
main()
And here's what one entry in my_cases.json looks like — adapted from a genuinely bad Tuesday of mine:
{
"id": "ring-buffer-off-by-one",
"question": "A fixed-size ring buffer in C uses head and tail indices and reports 'full' when (tail + 1) % capacity == head. A teammate argues this wastes one slot and suggests reporting full when tail == head. Explain what ambiguity that creates, and show the minimal fix that keeps all slots usable. Under 250 words.",
"must_mention": ["empty", "full", "indistinguishable", "count"],
"must_not_say": ["both designs are equivalent", "no fix is needed"]
}
The grader is deliberately unsophisticated: substring checks, plus a syntax-only compile where the task produces code. That simplicity is the asset, not the compromise. Next quarter, when another release dominates the discourse, I point the identical case file at it, run the script, and diff two JSON files. The internet's argument becomes my experiment.
Why free serving changes behavior, not just price
A personal suite is maybe thirty cases times three candidate models — a hundred requests, give or take. In dollars, that's nearly nothing. In friction, it's everything: the evaluations that never happen die at the billing form, the card verification step, or the vague anxiety about an unknown quota. A free server tier — MonkeyCode's is one instance, and others exist — reduces setup to exporting two environment variables. For a student comparing models before committing to a subscription, or a hobbyist kicking tires on a weekend, that difference decides whether the evaluation happens at all.
Zoom out and this is the real promise of open weights. The license text alone doesn't help you ship anything. What helps is the ecosystem that open weights permit: interchangeable hosts, nobody gatekeeping your evaluation, and every launch-day claim demoted from verdict to hypothesis — one you can test tonight, for free.
When this loop earns its setup cost
| Situation | Do public scores suffice? | Build the loop? |
|---|---|---|
| Summaries, prose, casual chat | Yes, mostly | No |
| Niche stack — legacy build systems, embedded targets, CUDA | No, wrong distribution | Yes, urgently |
| Code that must stay on your network | Irrelevant | Yes, against self-hosted weights |
| Picking a long-term daily assistant | Directional at best | Yes — feed it your past bugs |
| Forming a launch-day opinion | Everyone already has one | Resist the urge |
What this method gets wrong
- Substring grading undervalues good answers. A correct fix using different vocabulary scores poorly. Treat the report as triage; read the failures manually before concluding anything.
- Temperature zero is not determinism. Inference stacks vary run to run. If two models land within a couple of points, rerun before believing the gap.
- Free tiers are a moving target. I'm intentionally not citing quotas, model rosters, or latency numbers — those drift, and I won't invent stability. Verify current terms before depending on any of them.
- My case file flatters nobody's work but mine. It's heavy on C and build-system archaeology because that's my history. A case file built from someone else's bugs will flatter the model in all the wrong places.
Who should skip this entirely
If your bar is "an assistant that works well enough," take the popular option and go ship something. This is half a day of setup that only pays off when you're choosing a long-term tool, considering self-hosting, or working in territory the mainstream benchmarks never sample.
The takeaway
Whatever model is trending this week is just the occasion; the durable skill is the loop itself: your solved bugs, your grading rules, any compatible endpoint. Open weights make the comparison possible, and free serving makes it cost nothing but an afternoon.
If you want a zero-cost place to start, the script runs fine against MonkeyCode's free model access and free server — but any OpenAI-compatible endpoint works equally well, and that interchangeability is the entire argument. If you build a case file from your own bug history, I'd genuinely like to hear which model surprised you — in either direction. The gap between leaderboard rank and "actually fixed my code" is where the useful information always hides.
Top comments (0)