Every few weeks a new coding model shows up with a polished demo, and every few weeks I watch someone wire it straight into their editor because the demo looked good. Then the surprises arrive: it hallucinates an API that doesn't exist, it rewrites a file it wasn't asked to touch, or it burns an afternoon on a task a grep would have finished in thirty seconds.
The fix isn't cynicism — it's measurement. You don't need paid API access or a fancy eval suite to find out whether a model actually helps your work. You need a small, repeatable harness and an hour of honesty.
This article walks through the harness I use. Everything below is runnable, and the whole thing works with free tiers — so cost is not an excuse to skip it.
Why a harness instead of vibes
The recent discussions here about AI agents and tool boundaries circle around the same problem: we grant tools trust before we've measured what they do with it. Anecdotal evaluation has three known failure modes:
- Primacy bias — the first impressive output anchors your judgment of the model.
- Task drift — you start evaluating on whatever the model happens to be good at, instead of your actual work.
- No baseline — you never compare against "just do it by hand," so a net loss still feels like a win.
A tiny harness fixes all three, because the task list is fixed before you see any output.
Step 1: Build a task list from your git history
Don't use someone else's benchmark. Mine your own repo for tasks you've actually done:
# Find your last 20 non-trivial commits as candidate tasks
git log --oneline --since="3 months ago" \
-- . ':(exclude)*.lock' ':(exclude)*.min.*' | head -20
Pick 8–12 commits that represent your real distribution of work. For each one, write a task card:
## Task 04: retry logic for flaky HTTP client
- Input given to model: the commit message + the diff of the *parent* commit
- Expected output: a patch equivalent to the real fix
- Pass criteria: existing tests still pass; retry only on 5xx/timeout;
no change to public function signatures
- Category: bugfix
- Human time it originally took: ~40 min
The "human time it originally took" field is your baseline. Without it, you're measuring speed against nothing.
Step 2: The scoring rubric
Keep it binary and boring. Fancy rubrics invite self-deception.
| Dimension | Pass condition | Weight |
|---|---|---|
| Correctness | Patch applies and your test suite passes | 3 |
| Scope discipline | Touches only files/functions the task implies | 2 |
| Fabrication | References no nonexistent APIs, files, or flags | 2 |
| Time | Faster than your recorded human baseline | 1 |
A model that scores high on correctness but fails scope discipline is a liability in a shared codebase — that's the tool-boundary problem showing up in miniature. Record results in a plain CSV so you can diff runs across model versions later.
Step 3: Run it without paying anything
This is where free access matters. I'm currently using MonkeyCode for this loop — Disclosure: This article was prepared as part of MonkeyCode's product outreach. — because it offers free model access plus a free server option, which covers the two things this harness needs: a model to evaluate and somewhere to run the harness repeatedly. I can't speak to quotas, specific model lineups, or how long the free tier lasts, so treat it as "free right now" and design accordingly.
The harness itself is deliberately portable — a shell script, not a platform feature:
#!/usr/bin/env bash
# eval.sh <task_dir> — proposed structure, adapt paths to your setup
set -euo pipefail
TASK_DIR="$1"
# 1. Snapshot the repo state
git stash -u 2>/dev/null || true
# 2. Feed task card to the model (via your client of choice),
# capture the proposed patch to $TASK_DIR/output.patch
# 3. Score it
git apply --check "$TASK_DIR/output.patch" && echo "APPLIES: yes" || echo "APPLIES: no"
git apply "$TASK_DIR/output.patch"
npm test --silent > "$TASK_DIR/test.log" 2>&1 && echo "TESTS: pass" || echo "TESTS: fail"
git diff --name-only HEAD > "$TASK_DIR/files_touched.txt"
git checkout -- . && git clean -fd
The key property: the model never sees the scoring step. If your eval tool and your eval rubric live in the same prompt, you're grading homework with the answer key visible.
Step 4: What to do with the results
After one run you'll have a per-category picture. The useful output isn't "model X is good" — it's a routing table for your own work:
- Bugfixes pass, refactors fail → use the model for triage, not for restructuring.
- Fabrication failures cluster on one library → paste that library's real API docs into context, or stop using the model for that dependency.
- Everything passes but slower than baseline → the model is a correctness aid, not a speed aid. Use it for review, not generation.
Re-run the harness whenever you switch models or the provider updates one. Because it's just a script plus a CSV, the rerun costs you an hour, not a migration.
Limitations — read this before adopting
- 8–12 tasks is a small sample. This harness tells you about your work, not about the model in general. Don't publish your CSV as a benchmark; it's a personal routing table.
- Your git history is biased toward what you already attempted solo. Tasks you avoided entirely (unfamiliar languages, big greenfield design) are invisible to this method.
- Free tiers change. Any setup that depends on a specific provider's free offering should be treated as temporary. The script above is provider-agnostic for exactly this reason — swap the client, keep the scoring.
- Time-to-pass isn't value. A model that's 20% faster but needs your full attention while it works may be a net loss compared to 40 uninterrupted minutes of your own focus. The rubric can't see that; you have to.
Who should skip this
If you write code fewer than a few hours a week, the setup cost exceeds the payoff — just try a model on two real tasks and trust the result. And if your work involves code you can't send to any third-party endpoint, free cloud models are off the table entirely; run something local or don't run anything.
For everyone else: the demo will always be flattering. Your git history won't be. Measure against the second one.
Top comments (0)