Most developers I know treat model choice as binary: either you pay for the strongest model you can get, or you use whatever is free and accept the quality hit. Both are lazy. The honest answer is that task difficulty is not uniform — the model you need to untangle a race condition in a legacy scheduler is not the model you need to rename a field across forty call sites.
The problem is that "which tasks can a free model handle?" is a question about your codebase, not about leaderboard scores. Public benchmarks run on strangers' repos tell you almost nothing about whether a model understands your naming conventions, your test layout, or your framework's quirks. I wrote before about building a benchmark harness from your own repo's bugs; this article is the practical follow-up: a small, repeatable workflow for deciding which categories of your daily work you can safely route to a free model, and which ones still justify a paid one.
The idea: a canary task suite
Mine your last month of actual work for small, self-contained tasks. Pull requests, commit messages, and your own chat history with coding assistants are all good sources. You want roughly 10–15 tasks across categories like:
- mechanical refactors (rename, extract, move)
- test authoring for existing code
- bug fixes with a clear reproduction
- boilerplate generation (migrations, configs, CLI scaffolding)
- explanation/review tasks ("why is this slow", "review this diff")
Each task needs a verifiable outcome: a test that should pass, a diff that should apply, or a rubric you can score in under two minutes. If a task has no checkable outcome, it doesn't belong in the suite.
Here's the corpus format I use:
# canary_tasks.yaml
- id: rename-retry-param
category: mechanical-refactor
prompt: "Rename the `max_attempts` parameter to `max_retries` across src/ and update call sites."
verify:
- "grep -rq 'max_attempts' src/ && exit 1 || exit 0"
- "pytest tests/ -x -q"
- id: add-timeout-test
category: test-authoring
prompt: "Write a pytest test proving `fetch_with_timeout` raises TimeoutError after 2s. Use monkeypatching; no real network."
verify:
- "pytest tests/test_fetch_timeout.py -q"
- id: flaky-lock-bug
category: bugfix
prompt: "tests/test_cache.py::test_concurrent_set flakes ~1 in 20 runs. Find and fix the race."
verify:
- "for i in $(seq 1 30); do pytest tests/test_cache.py::test_concurrent_set -q || exit 1; done"
And the runner is deliberately boring — subprocesses and exit codes, no framework:
#!/usr/bin/env python3
"""Score a model against your canary suite. Label: proposal, adapt before running."""
import subprocess, yaml, json, time
def run_task(task, workdir):
start = time.time()
# 1. Send task['prompt'] to your model/agent of choice, let it edit `workdir`.
# (Integration point left to you — this harness only scores outcomes.)
results = []
for check in task["verify"]:
p = subprocess.run(check, shell=True, cwd=workdir, capture_output=True)
results.append(p.returncode == 0)
return {
"id": task["id"],
"category": task["category"],
"passed": all(results),
"seconds": round(time.time() - start, 1),
}
tasks = yaml.safe_load(open("canary_tasks.yaml"))
scores = [run_task(t, "./sandbox_copy_of_repo") for t in tasks]
print(json.dumps(scores, indent=2))
Run each candidate model against the suite twice (models are nondeterministic; one run is anecdote). Then aggregate by category, not by overall score. The overall number is useless — the routing table is the point.
The routing table is the artifact
After running this, my decision matrix looked structurally like this (your rows and results will differ — that's the entire point):
| Task category | Free model pass rate | Paid model pass rate | Route to |
|---|---|---|---|
| Mechanical refactors | high enough | high | Free |
| Test authoring | high enough | high | Free |
| Boilerplate/config | high enough | high | Free |
| Subtle concurrency bugs | low | medium | Paid + review |
| Ambiguous design questions | low | medium | Paid + review |
Two things surprised me when I did this. First, the failure mode of free models on hard tasks wasn't wrong answers — it was confident, plausible, wrong answers, which is exactly why "just try it and see" is dangerous without verifiable checks. Second, roughly half my weekly prompt volume fell into categories where I genuinely could not tell the outputs apart. That half is pure savings.
For the free side of the experiment I used MonkeyCode, which currently offers free model access plus a free server option, so the whole evaluation cost me nothing to run — which matters, because a routing study you can't afford to rerun is a routing study that goes stale. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness, corpus format, and scoring method above are model-agnostic, and you should absolutely run the same suite against any free tier you have access to, including whatever new releases come out this month — treat every "cheap and great" launch claim as a hypothesis your canary suite can confirm or kill in an afternoon.
Limitations and who shouldn't bother
- Sample size is tiny. 12 tasks gives you directional signal, not statistics. Rerun monthly; model behavior drifts.
- Nondeterminism is real. A task that passes once and fails once goes in the "paid" column. Flaky passing is failing.
- Free tiers change. Availability, rate limits, and which models are offered can shift without notice. Build the harness so swapping the model under test is a one-line change.
- Don't route sensitive code this way. Proprietary algorithms, credentials handling, or anything under a strict NDA should not go to any third-party service, free or paid, without your org's sign-off.
- Skip this if your work is one category. If 90% of your prompting is already hard, ambiguous design work, a routing table saves you nothing — just pay for the good model and move on.
The meta-lesson from my last few articles holds here too: the useful unit of AI evaluation is your own repo, your own tasks, your own failure history. Leaderboards are marketing. A canary suite you control is engineering.
If you build a version of this, I'd genuinely like to hear what your routing table looks like — especially which categories surprised you. The corpus format above is a starting point, not a standard; steal it and make it worse in your own way.
Top comments (0)