Open model releases have turned into a weekly event. Another checkpoint, another chart, another round of confident replies under it — and somewhere in that noise I'm supposed to decide whether this thing deserves a slot in my daily workflow. For a long time my process was: open a chat, throw it a prompt I half-remember, and let my mood write the review.
That process failed me more than once. A model I dismissed after one bad answer turned out fine. A model I adopted on a good first impression hallucinated its way through a real refactor. The common thread: I was reacting, not measuring.
So I built a scoring loop that fits in a coffee break and produces a written verdict instead of a feeling. This post walks through the whole thing — the task file, the runner, the scoring discipline — with code you can lift directly.
Why first impressions lie
Three biases make casual testing useless for model selection:
- Prompt lottery. A single prompt tells you how the model handled that prompt, at that phrasing. Rephrase it and you might get a different model entirely.
- Recency of the last demo. Whatever the last screenshot in your feed showed becomes your anchor, regardless of what you actually need.
- Readability as a proxy for correctness. Fluent, well-formatted wrong answers feel right. Code that looks clumsy but passes feels wrong. My gut consistently scores style over substance.
The antidote is unglamorous: fix the inputs, automate the run, and score against criteria you wrote before you saw any output.
Design: tasks live in a file, verdicts live in a log
Instead of a hardcoded list, I keep my tasks in a plain JSONL file. One line per task. It diffs cleanly in git, it's easy to extend, and the runner doesn't care what model it's pointed at.
{"id": "fix-off-by-one", "kind": "code", "prompt": "Return ONLY Python code. Fix the bug: this should sum values at even indices.\n\ndef even_sum(nums):\n total = 0\n for i in range(1, len(nums), 2):\n total += nums[i]\n return total\n", "check": "assert even_sum([1,2,3,4,5]) == 9\nassert even_sum([]) == 0\nassert even_sum([7]) == 7\nassert even_sum([2,9,2,9]) == 4\n"}
{"id": "recall-buried-fact", "kind": "manual", "prompt": "Read this document and answer: which port does the staging worker bind to? [paste a long doc with the answer '8471' on line ~200]", "rubric": "states 8471 = 1pt; hedges instead of inventing when unsure = 1pt"}
{"id": "schema-strict", "kind": "manual", "prompt": "Reply with JSON only, keys: name, count, tags. Item: a pack of 12 black gel pens.", "rubric": "valid JSON = 1pt; exactly 3 keys = 1pt; tags is an array = 1pt"}
Two categories, deliberately:
-
codetasks are self-scoring. The output gets executed against assertions. No interpretation, no generosity. -
manualtasks carry a pre-written rubric. The rubric exists before the answer does. That's the whole trick — it turns "did I like this?" into "did it hit these specific points?"
I keep roughly ten tasks total, drawn from what I actually did last month: a debugging case from a real ticket, a summarization over a genuinely long document, one strict-formatting task, one "please don't do this" safety-adjacent task. If a task doesn't mirror real work, it gets cut.
The runner
Small, standard-library-only, works against any OpenAI-compatible endpoint:
#!/usr/bin/env python3
"""score_loop.py — run tasks.jsonl against a model, log a verdict.
BASE_URL=http://localhost:8000/v1 MODEL=new-hotness \
python score_loop.py tasks.jsonl
"""
import json, os, subprocess, sys, tempfile, time, urllib.request
BASE, MODEL = os.environ["BASE_URL"].rstrip("/"), os.environ["MODEL"]
KEY = os.environ.get("API_KEY", "x")
def ask(prompt):
body = json.dumps({
"model": MODEL, "temperature": 0,
"messages": [{"role": "user", "content": prompt}],
}).encode()
req = urllib.request.Request(
f"{BASE}/chat/completions", data=body,
headers={"Content-Type": "application/json",
"Authorization": f"Bearer {KEY}"})
with urllib.request.urlopen(req, timeout=240) as r:
return json.load(r)["choices"][0]["message"]["content"]
def grade_code(answer, check):
# strip a fenced block if the model added one
if "```
" in answer:
inner = answer.split("
```")[1]
answer = inner.split("\n", 1)[1] if "\n" in inner else inner
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
f.write(answer + "\n" + check)
path = f.name
try:
subprocess.run([sys.executable, path], timeout=10,
capture_output=True, check=True)
return 1
except Exception:
return 0
log = {"model": MODEL, "at": time.strftime("%Y-%m-%d %H:%M"), "rows": []}
for line in open(sys.argv[1]):
t = json.loads(line)
out = ask(t["prompt"])
row = {"id": t["id"], "kind": t["kind"]}
if t["kind"] == "code":
row["score"] = grade_code(out, t["check"])
else:
row["rubric"] = t["rubric"] # score these by hand, rubric first
row["answer"] = out[:2000]
log["rows"].append(row)
print(t["id"], "->", row.get("score", "needs manual grade"))
name = f"verdict_{MODEL.replace('/', '_')}_{int(time.time())}.json"
json.dump(log, open(name, "w"), indent=2)
print("wrote", name)
Three details I'd keep in any rewrite:
- Deterministic settings. Temperature 0 won't make a model deterministic, but it removes the most obvious source of run-to-run noise.
- Execution beats inspection for code. The model doesn't get partial credit for a persuasive explanation of a broken fix.
- Every run leaves an artifact. The verdict files accumulate. Six months from now, "why are we on this model?" has a paper trail.
Manual rubric scoring takes me maybe five minutes for the whole file, and because the rubric was written up front, it's grading rather than rationalizing.
The verdict rule
After scoring, one sentence goes into the log: adopt, trial, or skip.
- Adopt — beats or matches my current model on code tasks and produces zero fabricated answers on the recall/safety rows.
- Trial — mixed results but wins somewhere specific; it gets one week of real side-task usage before a final call.
- Skip — anything else. No "it seemed promising." Promising is how mediocre tools colonize your workflow.
One nuance I now score explicitly: failure shape. On the recall task, a wrong confident answer is worse than a wrong hedged one, and both are worse than a correct "the document doesn't say." I'd rather run a model that loses loudly than one that loses politely.
Running it for free
The loop needs an endpoint and nothing else. I've been running mine through MonkeyCode, which currently offers free model access together with a free server option — enough to point BASE_URL at it and let the runner work through the file. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What I like about this pattern existing at all is that it echoes how the open model ecosystem works in the first place. Weights get published, compatible APIs get shared, and suddenly the interesting question shifts from "can you afford to try this" to "what did you find when you did." Free inference and a free machine to run it on are the same instinct showing up in commercial form — the eval stops being a procurement decision and becomes a half-hour experiment. Anyone who wants to check a benchmark claim instead of quote-tweeting it can. The fully local route works too; the runner above doesn't care whether the endpoint is llama.cpp on your own box or a hosted free tier.
Where this breaks down
- Ten tasks is a filter, not a benchmark. It answers "worth a real trial?" and nothing more. Don't cite your personal pass rates as model rankings.
- The suite ages. Tasks you wrote last quarter may no longer represent your work. I prune mine when I notice I'm scoring models on problems I no longer have.
- Greedy decoding flatters consistency. A model that looks stable at temperature 0 may wander at the settings you use for drafting or brainstorming. Spot-check at real settings before adopting for those tasks.
- Free options are moving targets. Lineups, rate limits, and availability change; read the current terms, and never wire a free tier into anything production-shaped.
- Formal contexts need formal methods. Procurement, compliance, or research comparisons call for established suites and real methodology — this loop is for personal tooling choices.
- Don't run it weekly. Most releases aren't relevant to your workload. Trigger the loop on a concrete reason — a license you'd actually use, a size you can self-host, a capability gap in your current setup — not on hype volume.
Closing thought
The release cadence isn't slowing down, so the choice is between faster opinions and better filters. A task file, a runner that executes instead of admires, and a rubric written before the answers exist — that's the whole system. If you want a zero-cost endpoint to run it against, MonkeyCode's free model access and server are one place to start; a local server works identically. The point isn't where it runs — it's that next time a checkpoint trends, you reach for the runner instead of the reply button.
If you've built something similar, I'd love to hear which tasks made your personal cut — that's the part I'm always tempted to steal.
Top comments (0)