Stop Guessing: A Reproducible Harness for Evaluating Free AI Coding Models on Your Own Repo

Most "which AI coding model is best?" debates I see devolve into vibes. Someone pastes a cherry-picked diff, someone else counters with a different cherry-picked diff, and nobody learns anything transferable. The problem isn't the models — it's that we almost never evaluate them on our code, with our constraints, using a method we could rerun tomorrow.

This article is the harness I wish more teams built before arguing. It's a small, language-agnostic evaluation loop you can point at any model you have access to — including free tiers — and get a defensible answer to a narrow question: does this model help with the tasks I actually do?

The evaluation trap

Public benchmarks (HumanEval-style tasks, leaderboard scores) measure performance on curated problems with clean specifications. Your work is rarely that. Real tasks look like:

• "Add retry logic to this half-migrated HTTP client without breaking the old call sites."
• "Write tests for a function whose behavior depends on a config file three directories up."
• "Refactor this 200-line function, but the ORM calls must stay in the same transaction."

These tasks share a trait: correctness is checkable, but only by you. Your test suite, your type checker, your lint rules. That's actually good news — it means evaluation can be automated against artifacts you already have.

The artifact: a task-runner harness

The core idea is dumb on purpose. Define a set of tasks as directories. Each task has a prompt, a snapshot of the relevant code, and a verification command. The harness applies a model's patch and runs the verifier. No scoring model, no LLM-as-judge — just your own build.

eval/
├── tasks/
│   ├── 001-retry-http-client/
│   │   ├── prompt.md
│   │   ├── repo/            # snapshot of the relevant files
│   │   └── verify.sh        # exit 0 = pass
│   ├── 002-test-config-loader/
│   └── 003-split-billing-fn/
└── run_eval.py

Here's a minimal runner (Python 3.10+, stdlib only):

#!/usr/bin/env python3
"""run_eval.py — apply a model-produced patch to each task and verify.

Usage:
    python run_eval.py --tasks eval/tasks --patches patches/<model-name>/

The patch for task N lives at patches/<model-name>/<task-dir-name>.diff
This script never calls the model itself; generation is a separate step,
so you can diff outputs across models or re-verify later.
"""
import argparse
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path


def run(cmd: list[str], cwd: Path) -> subprocess.CompletedProcess:
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=300)


def evaluate_task(task_dir: Path, patch_file: Path) -> dict:
    with tempfile.TemporaryDirectory() as tmp:
        work = Path(tmp) / "repo"
        shutil.copytree(task_dir / "repo", work)

        applied = run(["git", "apply", "--check", str(patch_file)], cwd=work)
        if applied.returncode != 0:
            return {"task": task_dir.name, "result": "patch_rejected",
                    "detail": applied.stderr.strip()[:300]}

        run(["git", "apply", str(patch_file)], cwd=work)
        verified = run(["bash", str(task_dir.resolve() / "verify.sh")], cwd=work)
        return {
            "task": task_dir.name,
            "result": "pass" if verified.returncode == 0 else "fail",
            "detail": (verified.stderr or verified.stdout).strip()[:300],
        }


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--tasks", type=Path, required=True)
    ap.add_argument("--patches", type=Path, required=True)
    args = ap.parse_args()

    rows = []
    for task_dir in sorted(p for p in args.tasks.iterdir() if p.is_dir()):
        patch = args.patches / f"{task_dir.name}.diff"
        if not patch.exists():
            rows.append({"task": task_dir.name, "result": "no_patch", "detail": ""})
            continue
        rows.append(evaluate_task(task_dir, patch))

    passed = sum(r["result"] == "pass" for r in rows)
    for r in rows:
        print(f"{r['result']:>14}  {r['task']}  {r['detail'][:80]}")
    print(f"\n{passed}/{len(rows)} tasks passed")
    return 0


if __name__ == "__main__":
    sys.exit(main())

A verify.sh is just your normal quality gates. For a Python repo:

#!/usr/bin/env bash
set -euo pipefail
python -m pytest tests/ -x -q
python -m mypy src/ --strict
grep -q "retry" src/http_client.py   # crude intent check; refine per task

The grep line is doing something important: verifying the change happened, not just that nothing broke. A model that returns your code unchanged passes pytest trivially.

Picking tasks that discriminate

Five to ten tasks beat fifty. Choose along three axes:

Axis	Why it matters	Example
Context size	Does the model need to respect code it can only partially see?	Cross-module refactor
Verification sharpness	Can a wrong-but-plausible answer slip past your tests?	Concurrency fix with a race test
Task type mix	Generation vs. modification vs. explanation-driven edits	New endpoint vs. bugfix vs. migration

Include at least one trap task: a task where the obvious solution is wrong in your codebase (e.g., "use the standard retry decorator" — except your repo pins an old version without it). Trap tasks are where free and paid models diverge most visibly, because they punish confident pattern-matching.

Where free model access fits

Running this harness requires generating patches from several models, ideally more than once per model to catch variance. That's exactly where cost becomes the excuse for never doing it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This is a reasonable use case for MonkeyCode's free model access: you can route the generation step through models that are available at no cost, which makes a 3-models × 8-tasks × 2-runs matrix (48 generations) something you can do on a weekday afternoon instead of a budget request. If you don't want your repo snapshots leaving your own infrastructure, MonkeyCode also offers a free server option, so the same harness can run against a deployment you control — relevant if any of your task snapshots contain code you wouldn't paste into a hosted tool. I'm deliberately not claiming specific model names, quotas, or uptime here; check what's actually available in your account before designing around it.

The harness doesn't care which backend produced the diff. That separation — generation as a swappable step, verification as fixed ground truth — is the whole design.

Reading results honestly

A few rules I'd suggest:

1. Run everything twice. Single-run results on small task sets are noise. If a model passes 6/8 then 3/8, report the range, not the better number.
2. Classify failures, don't just count them. patch_rejected (couldn't follow the diff format) is a different signal from fail on tests (plausible but wrong) vs. fail on the intent check (did nothing). Each suggests a different remediation.
3. Weight by your work distribution. If 70% of your week is modifications to existing code, a model that aces greenfield generation but mangles refactors is a net negative for you, whatever the leaderboard says.

Limitations, and who shouldn't bother

• Small samples, wide error bars. Eight tasks cannot rank two close models. The harness answers "is this model usable for my tasks," not "is model A globally better than model B."
• Your tests are the ceiling. Weak verification means weak evaluation. If your suite wouldn't catch a regression a junior dev would spot, it won't catch a model's either.
• It measures one-shot patch quality. Interactive, multi-turn workflows (where a lot of real value lives) need a different, messier methodology.
• Don't build this if you use AI assistance occasionally for boilerplate, your tasks are all greenfield with no testable invariants, or you can't isolate task snapshots without dragging in proprietary code you have nowhere acceptable to send. A spreadsheet and honest notes will serve you better.

Takeaway

The useful question was never "which model is best." It's "which model passes my verification on my task mix, at a cost I can sustain." A harness like this turns that from an argument into a rerun.

If you build a version of this, I'm curious which trap tasks you chose — that part of the task set tends to reveal the most about both the models and the codebase. And if free access (e.g., via MonkeyCode) is what makes the experiment affordable for you, the multi-model comparison section is where to start.