Every week there's a new model that "crushes" coding benchmarks, and every week the conversation drifts toward giving agents more tools, more permissions, more trust. The problem: benchmark scores don't tell you how a model behaves on your tasks, with your constraints, under your failure modes. And running serious evals against frontier models gets expensive fast, which pushes most of us toward vibes-based decisions.
This post is a cheaper middle path: a small, self-owned task suite, a scoring script, and free-tier resources to run it on. It's not a benchmark. It's a smoke alarm.
The setup
The task suite lives in a plain directory. Each task is a folder with three files:
tasks/
01-fix-off-by-one/
prompt.md # what the agent is told
starter/ # broken code it starts from
check.py # exit 0 = pass, exit 1 = fail
02-add-validation/
03-refactor-without-behavior-change/
check.py is the part that matters. It runs the code, asserts on behavior, and prints a JSON verdict:
import json, subprocess, sys
result = subprocess.run(
[sys.executable, "starter/main.py"],
capture_output=True, text=True, timeout=10
)
verdict = {
"task": "01-fix-off-by-one",
"passed": result.returncode == 0 and "sum=5050" in result.stdout,
"stderr_tail": result.stderr[-500:],
}
print(json.dumps(verdict))
Behavioral checks only. No "did the code look clean" scoring — that part you do yourself, later, on the failures.
The runner loops over tasks, calls whatever model endpoint you're evaluating, drops the returned code into a throwaway copy of starter/, and executes check.py. Run it in a container or a VM, because you are executing model-generated code. I've written about the sandboxing side of this before; the short version is: no host mounts, no network unless the task needs it, and a hard timeout.
Where the free tier fits
The reason this workflow is practical at all is that the per-task cost is zero. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which maps neatly onto the two things this harness needs: a model endpoint to point the runner at, and a machine I don't care about to execute the output on. I run the scoring there so the blast radius of a bad generation stays off my laptop entirely.
Two honest caveats: I don't control how long free tiers stay free, so the runner keeps the model behind a thin client function I can re-point at any endpoint in a few lines. And free compute is fine for tens of tasks, not thousands — if your suite grows, that's a good problem, but it's a different budget conversation.
What I actually score
Three columns, no composite score:
| Column | What it measures | Why separate |
|---|---|---|
| Pass rate | Behavioral correctness via check.py
|
The floor |
| Diff size | Lines changed vs. starter | Agents that rewrite everything are risky in real repos |
| Failure mode | Timeout / crash / wrong-output / refused | Tells you how it fails, not just that it fails |
That last column is the one benchmark leaderboards never show. A model that fails by producing plausible-but-wrong output is more dangerous in an agentic loop than one that crashes loudly — the loud failure gets retried, the quiet one gets committed. After a run, I read the quiet failures by hand. Ten minutes, and I learn more about a model than from any aggregate number.
Limitations
- A 10–30 task suite measures your tasks. It will not generalize, and that's the point — don't quote your pass rate as if it's a public benchmark.
- Single-shot prompting is not the same as an agentic loop with tool calls. This harness tests the model's raw code generation; it says nothing about multi-step planning.
- Behavioral checks miss style, security smells, and dependency choices. The manual failure review is load-bearing; don't skip it.
Who should skip this
If you're evaluating models for a purchase decision at a company scale, you need a real eval framework with statistical rigor, not a weekend harness. And if your tasks can't be expressed as runnable code with checkable output — design work, exploratory debugging — this structure won't capture what you care about.
Closing
The goal isn't to crown a winner among free models. It's to stop being surprised. A tiny suite you own, re-run whenever a model updates, turns "the new model feels worse" into "task 07 regressed, here's the diff." If you want to try it, scaffold three tasks from bugs you've personally fixed in the last month — MonkeyCode's free tier is one zero-cost place to run the first pass, but any sandboxed box with a model endpoint works. The harness is the asset; the endpoint is replaceable.
Top comments (0)