Every model launch triggers the same ritual on my timeline: screenshots of impressive one-liners on one side, screenshots of embarrassing failures on the other, and a hundred hot takes in between. After my third cycle of switching daily-driver models based on screenshots, I admitted the uncomfortable part — I wasn't evaluating anything. I was adopting other people's anecdotes.
So I built myself a scorecard: a small, frozen set of tasks, a clean room to run them in, and a written grading policy. New release lands, I spend a couple of hours running it through the scorecard, and I walk away with notes instead of feelings. Here's the full setup, the reasoning behind it, and the honest list of ways it can lie to you.
The two ways casual testing fooled me
Before the scorecard, my evaluation process was "open a chat, try a few prompts, form an opinion." That process has structural defects I couldn't prompt my way around.
You're sampling blind. Three prompts is three draws from a distribution you can't see. Pick tasks the model has effectively memorized and it looks superhuman. Pick tasks outside its comfort zone and it looks useless. Both impressions are artifacts of your prompt choice, not properties of the model.
Your environment contaminates the result. When I tested inside my working editor, the model silently benefited from my open files, my earlier corrections, and half-finished edits. I wasn't scoring the model — I was scoring the model wearing my project as a life-support system. Great for shipping, worthless for comparison.
The fix isn't clever. It's the boring discipline of any experiment: same stimulus, isolated environment, more than one repetition.
Cost and safety were the real blockers
Two practical problems kept me from running proper evaluations for months.
First, cost discipline fights measurement discipline. A serious run means repeating each task several times, and on a metered API every repetition is a small financial decision. I kept cutting trials to save money, which destroyed exactly the statistical confidence I was paying for.
Second, I didn't want an agent executing generated commands on the machine that holds my SSH keys, cloud credentials, and browser sessions.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What unblocked me was MonkeyCode, an open-source coding agent. Its free model access lets me point a fresh release at my scorecard without setting up billing, and its free server option means the agent loop runs in a throwaway remote environment instead of my laptop. Since the agent is open source, I can also confirm that trial runs don't share memory or leftovers — something I'd have to take on faith with a closed runner.
None of that is load-bearing, though. Any agent harness you can inspect, plus any model endpoint you don't have to ration, works. The scorecard below doesn't care what executes it.
Scorecard structure: eight tasks, repeated four times
The scorecard has three pieces. Swap my placeholder tasks for sanitized fragments of your own work — that's the version that actually tells you something.
tasks.yml — the frozen task list. Eight tasks split across three skill areas, each one sized to be small enough that a pass/fail verdict is unambiguous:
- id: fix-01
area: bugfix
sandbox: ./sandboxes/order-pipeline
prompt: |
Orders placed between 23:59 and 00:01 are assigned to the wrong
fulfillment date. Find the defect, repair it, and add a regression
test that fails on the old code and passes on yours.
verify: npm test -- --grep boundary-date
- id: feat-03
area: feature
sandbox: ./sandboxes/pastebin-lite
prompt: |
Add an expiry feature: pastes accept a TTL in minutes and return
HTTP 410 after expiry. Test the exact expiry boundary.
verify: cargo test expiry
- id: explain-02
area: analysis
sandbox: ./sandboxes/worker-queue
prompt: |
Under burst load, queue latency spikes then never recovers.
Explain the mechanism in writing. Do not modify any file.
verify: manual # graded against rubric.md
score.py — the harness. Only two design decisions really matter: every repetition starts from a byte-identical copy of the sandbox, and the full conversation transcript gets saved so I can re-read failures months later.
import shutil, subprocess, time
from pathlib import Path
def repetition(task, rep, model):
workdir = Path(f"results/{task['id']}/rep{rep}")
if workdir.exists():
shutil.rmtree(workdir)
shutil.copytree(task["sandbox"], workdir)
started = time.time()
transcript = run_agent(model=model, prompt=task["prompt"], cwd=workdir)
elapsed = round(time.time() - started, 1)
if task["verify"] == "manual":
outcome = None # graded by hand against the rubric
else:
result = subprocess.run(task["verify"], shell=True, cwd=workdir,
capture_output=True)
outcome = result.returncode == 0
return {"task": task["id"], "rep": rep, "model": model,
"outcome": outcome, "elapsed_s": elapsed,
"transcript": transcript}
# 8 tasks x 4 reps = 32 runs per model. Not science. Enough for a decision.
rubric.md — grading rules for the analysis tasks, since they have no automated check. Each explanation gets 0–2 points on three dimensions: identifies the true mechanism, proposes a workable mitigation, and invents nothing. Five of six points is a pass, and any fabricated log output or nonexistent config option zeroes that dimension instantly. A model that hallucinates evidence is more dangerous than one that admits confusion.
Read the failure pattern, not the percentage
Thirty-two runs give you a grid of outcomes. The aggregate pass rate is the single least useful cell in that grid. What changes my behavior is the geometry of the failures:
| Pattern in the grid | Probable cause | How I respond |
|---|---|---|
| Strong on fix/feature, weak on analysis | Retrieval strength, weak causal modeling | Use for implementation, keep architecture decisions human |
| Rep 1 passes, reps 2–4 wobble | Sampling instability at default settings | Distrust any single demo of this model; add reps |
| Passes my public sandboxes, fails my private ones | Has seen the public code before | Discount heavily for proprietary codebases |
| Verify step fails, transcript looks right | My sandbox or check script is broken | Fix the scorecard, not the model |
| Passes everything, takes forever | Capability without latency | Fine for overnight batch jobs, wrong for pairing |
The fourth row is the one I re-learn constantly. "The model broke" and "my test harness broke" produce identical output until you open the transcript. Archive everything.
Why reproducibility beats verdicts
There's a reason I lean on open components end to end. Open weights can't be silently swapped between my repetitions. An open agent loop lets me prove exactly what context the model received. And an open scorecard means anyone who doubts my conclusion can rerun it instead of arguing with it. When the next open-weight model drops, the most valuable community contribution isn't another hot take — it's a task suite with transcripts attached.
Opinions expire with the next release. A frozen task list doesn't, because the next model walks into the same room and faces the same eight tasks.
Where the scorecard fails
- Thirty-two runs is not a study. It ranks models for my own workflow. Procurement or platform decisions need far more data plus human review of every failure.
- Small sandboxes generalize poorly. A model that shines on a 1,500-line fixture may drown in a 500k-line monorepo. If that's your reality, build sandboxes from your actual code (scrubbed of secrets).
- Free tiers are borrowed, not owned. Free model access and free server capacity can be throttled, re-priced, or withdrawn at any time. Use them for weekend evaluations, never wire them into CI, and read the current terms before assuming anything.
- Don't bother with this at all if stakeholders need an audit-grade benchmark, if your code legally cannot leave your network and you have no local harness, or if your real question is answerable by simply using the tool for a week. Daily lived experience is a legitimate instrument — it's just not a calibrated one.
The actual takeaway
Launch-week discourse is a machine for manufacturing confidence from other people's single prompts. A scorecard converts "who should I trust?" into "what did I measure?" — eight tasks, four repetitions, one written rubric. If the two blockers I described (metered trials, unsafe execution) are what's stopping you, MonkeyCode's free model access and free server option are one workable way to start this weekend. But the scorecard is the asset that outlives any vendor, and it plugs into whatever tooling you already have.
If you maintain your own task list, I'd genuinely like to know what skill areas you test that mine misses — especially anything involving large multi-file refactors, which I still haven't found a good way to grade.
Top comments (0)