In my last two posts I argued for running your own harness before trusting any coding model, and for refusing to merge model output just because a benchmark says it's good. This post is the practical middle step I skipped: how do you decide whether a free-tier model belongs in your daily loop at all?
The wrong way to answer that is vibes. You paste three prompts, one looks impressive, and suddenly the model is writing half your PRs. The right way is a staged gate: cheap tests first, expensive tests only for survivors. Here's the exact sequence I use, plus the scoring artifact you can copy.
The staged gate, in one picture
Stage 0: Task taxonomy → does this model even fit my work?
Stage 1: Static smoke test → syntax, imports, obvious hallucination
Stage 2: Sandbox execution → does it run? do my tests pass?
Stage 3: Diff review → would I approve this PR from a junior?
Stage 4: Shadow week → model proposes, human disposes
A model must pass each stage to reach the next. Most free models fail at Stage 2, which is fine — the whole point of staging is that Stage 2 costs you an afternoon, not a production incident.
Stage 0: Write your task taxonomy before touching any model
This is the step everyone skips, and it's why their conclusions are useless. Write down the five to eight task types that actually make up your week, weighted by frequency. Mine, as an example:
| Task type | Share of my week | Failure cost if model is wrong |
|---|---|---|
| Boilerplate / CRUD scaffolding | 25% | Low |
| Test writing for existing code | 20% | Medium |
| Refactoring within one file | 15% | Medium |
| Cross-file bug diagnosis | 15% | High |
| Config / CI / YAML wrangling | 10% | High (silent breakage) |
| Unfamiliar-library usage | 10% | Medium |
| Code review / explanation | 5% | Low |
Your table will differ. The weights matter because a model that's great at boilerplate and terrible at cross-file diagnosis might still be a net win if you restrict it to the first row. Free tiers especially tend to be uneven — strong on common patterns, shakier on long-context reasoning. The taxonomy turns "is this model good?" (unanswerable) into "which rows of my table is this model good for?" (testable).
Stage 1–2: A minimal scoring script
For each row of the taxonomy, pick two or three real tasks from your own git history — not from a public benchmark, because public tasks leak into training data. Then score outputs mechanically. Here's a stripped-down version of the scorer I run (adapt freely; this is illustrative, not a product):
#!/usr/bin/env python3
"""Score model outputs per task type. Usage: score.py results.jsonl"""
import json, sys, subprocess, tempfile, os
def run_tests(code: str, test_file: str, lang: str = "python") -> bool:
with tempfile.TemporaryDirectory() as d:
src = os.path.join(d, "solution.py")
with open(src, "w") as f:
f.write(code)
r = subprocess.run(["pytest", test_file, "-x", "-q"],
cwd=d, capture_output=True, timeout=120)
return r.returncode == 0
def main(path):
rows = {}
for line in open(path):
rec = json.loads(line)
# rec: {"task_type": ..., "code": ..., "test_file": ..., "review_pass": bool}
t = rec["task_type"]
exec_ok = run_tests(rec["code"], rec["test_file"])
score = (0.6 if exec_ok else 0.0) + (0.4 if rec["review_pass"] else 0.0)
cur = rows.setdefault(t, [])
cur.append(score)
for t, scores in sorted(rows.items()):
print(f"{t:35s} n={len(scores)} mean={sum(scores)/len(scores):.2f}")
if __name__ == "__main__":
main(sys.argv[1])
Two deliberate choices here:
- Execution is weighted above review (0.6/0.4) because "it compiles and passes tests" is objective, while my review judgment is not. When the two disagree, the task goes into a manual pile.
- Scores are per task type, never aggregated. An overall mean would hide exactly the failure modes you need to see. A model scoring 0.9 on scaffolding and 0.3 on cross-file bugs is not a "0.6 model" — it's a scaffolding tool.
Stage 3–4: The human gates
Stage 3 is a diff review with a single question: would I approve this if a junior teammate submitted it? Same bar, no discounts for "the AI did it." If anything, review harder — model errors are systematic, not random, so one weird import usually means a whole class of confident wrongness.
Stage 4 is a shadow week: the model proposes changes, you write or heavily edit everything yourself, and you log how often its proposal was usable. If the usable rate doesn't beat your baseline after a week, the model exits the loop for that task type.
Where free infrastructure fits
The expensive part of this workflow isn't the model calls — it's the iteration. Running Stage 1–3 across six task types and three candidate models means a few hundred executions, and you'll want to rerun it every time a model updates. Paying per-token for exploratory evaluation is how people end up evaluating on three prompts and calling it done.
This is where I've been using MonkeyCode: it offers free access to a set of coding models and a free server option, which maps neatly onto this workflow — free model access covers the candidate pool, and the free server is enough to host the sandbox executor for Stage 2 without standing up paid infrastructure. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I won't quote quotas, model lists, or limits here because they change; check the current offering yourself before designing around it. If your evaluation volume is small, any free tier — theirs or a provider's own — works equally well for Stages 0–3. The workflow is the point, not the vendor.
Decision table: when a free model earns a seat
| Outcome across your taxonomy | Verdict |
|---|---|
| Passes stages 1–4 on ≥2 high-frequency task types | Adopt, restricted to those task types |
| Passes on low-frequency or low-risk types only | Keep for drafts; never let it touch CI config or cross-file changes |
| Passes execution but fails diff review repeatedly | Do not adopt; systematic style/logic drift will eat your review time |
| Fails sandbox execution broadly | Drop it and retest in a few months; models move fast |
Limitations, and who should skip this
- Sample size is your enemy. Two tasks per type is a smoke screen, not a study. If you can't assemble at least 15–20 real tasks, your results are noise. Be honest about that before acting on them.
- Free tiers change without notice. Rate limits, model availability, and server capacity can shift mid-evaluation. Never wire a free tier into CI or anything with an SLA.
- This doesn't transfer to regulated or high-blast-radius code. If a wrong suggestion can touch auth, payments, medical logic, or infra-as-code that deletes things, free-model gating is the wrong conversation entirely — you need audited tooling and human authorship guarantees.
- Solo maintainers gain the most. Teams with established review culture already have a Stage 3; adding model evaluation is cheap. If you're solo with no tests, build the tests first — the harness is useless without them.
The gate is boring on purpose. Boring is what keeps a confident, free, occasionally-wrong model from becoming a line item in your incident review. If you've run a similar staged evaluation, I'd genuinely like to hear where your models failed — the taxonomy rows where free tiers collapse are more useful shared than hoarded.
Top comments (0)