A Repeatable Harness for Evaluating Free AI Coding Models Before You Trust Them

Every week another coding model shows up, and the demo always looks great. The problem is that demos are chosen to look great. When I want to know whether a model is worth wiring into my actual workflow, I don't watch demos — I run the same small set of tasks against every candidate and score the outputs myself.

This post is the harness I use. It's deliberately boring: three fixed tasks, a fixed prompt, an automated check where possible, and a short rubric where it isn't. Because it's fixed, I can rerun it whenever a new model becomes available — including free-tier options — and get a comparable answer in under an hour.

Why a fixed harness beats vibes

Two failure modes dominate casual model testing:

1. You test with problems you already know the answer to. You unconsciously steer the model, and every model looks fine.
2. You test once, with one prompt. Coding models are sensitive to phrasing, so a single run tells you almost nothing.

A fixed harness addresses both: the tasks and prompts are written down before you see any output, and the scoring is mechanical enough that two runs of the same model land close together.

The harness

Three artifacts: a task directory, a runner script, and a scoresheet.

1. The tasks

Keep three tasks that cover the failure modes you actually care about. Mine are:

• Task A — targeted bug fix. A small repo with one real bug and a failing test. Measures whether the model can read existing code without rewriting it.
• Task B — feature addition under constraints. "Add pagination to this endpoint, don't change the response shape for existing clients." Measures instruction-following.
• Task C — explain-then-change. Ask the model to explain what a function does, then modify it. If the explanation is wrong, the change usually is too — this is a cheap proxy for hallucination risk.

Here's a minimal Task A you can copy (a classic off-by-one in a date helper):

# task_a/dates.py
def add_business_days(start, days):
    """Add `days` business days to `start` (a datetime.date).
    Skips Saturdays and Sundays."""
    current = start
    added = 0
    while added < days:
        current = current.replace(day=current.day + 1)  # BUG: blows up at month end
        if current.weekday() < 5:
            added += 1
    return current

# task_a/test_dates.py
from datetime import date, timedelta
from dates import add_business_days

def test_month_boundary():
    # Jan 31 + 1 business day should be Feb 1, not a crash
    assert add_business_days(date(2026, 1, 30), 1) == date(2026, 2, 2)  # Fri -> Mon

def test_simple():
    assert add_business_days(date(2026, 8, 3), 3) == date(2026, 8, 6)

The bug (replace(day=current.day + 1) instead of + timedelta(days=1)) is realistic — it's the kind of thing that survives code review. A good model fixes it with timedelta and doesn't touch anything else.

2. The runner

The runner's only job is to make runs comparable: same prompt template, same files, same timeout, output saved to a timestamped folder.

# run_eval.py
import json, subprocess, sys, time
from pathlib import Path

PROMPT_TEMPLATE = """You are working in the directory shown below.
Fix the failing test. Change as little code as possible.
Do not modify the test file.

---
{code}
---

Failing test output:
{test_output}
"""

def run_task(task_dir: Path, call_model):
    code = "\n".join(
        f"# {p.name}\n{p.read_text()}"
        for p in task_dir.glob("*.py")
        if not p.name.startswith("test_")
    )
    test_output = subprocess.run(
        [sys.executable, "-m", "pytest", str(task_dir), "-x", "-q"],
        capture_output=True, text=True, timeout=60,
    ).stdout
    prompt = PROMPT_TEMPLATE.format(code=code, test_output=test_output)
    start = time.time()
    response = call_model(prompt)          # plug in whichever model you're testing
    elapsed = time.time() - start
    return {"prompt": prompt, "response": response, "seconds": round(elapsed, 2)}

# call_model is the only thing you swap between candidates.

The important property isn't the code, it's that call_model is the only variable between runs. Same prompt, same tasks, same machine if you can manage it.

3. The scoresheet

Automated tests catch Task A, but Tasks B and C need judgment. I use a four-line rubric, scored 0–2 each:

Criterion	0	1	2
Correctness	Wrong or untested	Mostly right, edge cases broken	Passes tests / works as specified
Diff discipline	Rewrites unrelated code	Some scope creep	Minimal, targeted change
Constraint following	Ignores stated constraints	Partial	Follows all stated constraints
Explanation accuracy (Task C)	Hallucinated behavior	Vague but not wrong	Precise and verifiable

Max 8 points. I write the score down before I let myself rationalize. Anything below 6 doesn't get a second look; 6–7 gets a rerun with a rephrased prompt; 8 gets tried on a real ticket.

Where free tiers fit in

Running this harness costs tokens, which is exactly why free access matters: evaluation should be cheap enough that you actually do it, repeatedly, for every new model that appears.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option, which maps neatly onto this workflow — the models under evaluation are free to call, and if you don't want the runner's test execution on your own machine, the free server can host the harness. That's the extent of what I can claim: I haven't verified quotas, rate limits, specific model availability, or how long the free tier lasts, and you shouldn't assume any of those from this post. Check the current terms before building anything on top of it.

The broader point is product-agnostic: whatever free tier you use, the harness itself is ~100 lines and portable. If the free access disappears tomorrow, you point call_model somewhere else and your historical scores are still comparable.

Limitations and honest caveats

• Three tasks is a smoke test, not a benchmark. It will catch "this model can't follow constraints at all" but won't rank two good models reliably. If you need that, you need dozens of tasks and multiple runs per task — at which point look at published eval methodology rather than a personal script.
• Free tiers change. Quotas shrink, models rotate, servers get busy. Treat any free option as evaluation infrastructure, never production infrastructure.
• Your rubric is still subjective. Scoring before rationalizing helps, but a second pair of eyes on borderline scores helps more.
• Don't evaluate on private code on a hosted free server unless you've actually read the data-handling terms. Use synthetic tasks like the one above for the first pass.

Who shouldn't bother with this

If you already have an eval pipeline at work, use that. If you only ever use one model and switching costs are high, a harness optimizes a decision you're not making. And if your real bottleneck is prompt quality rather than model choice, spend the hour on your prompts instead.

For everyone else: write down your three tasks before the next model launch, so the next launch can't impress you for free.

If you try this harness — on MonkeyCode's free tier or anywhere else — I'd genuinely like to hear what scoring criteria you ended up adding. Mine drift every few months as models fail in new and creative ways.