A Reproducible Harness for Evaluating Free AI Coding Models on Your Own Codebase

Most "which AI coding model should I use?" advice is based on someone else's codebase, someone else's prompts, and someone else's tolerance for wrong-but-confident answers. With free model access and free hosted servers becoming common across coding platforms, the real bottleneck is no longer access — it's evaluation. This article shows how to build a small, reproducible harness that scores models against tasks drawn from your repository, so your choice is based on evidence you can rerun.

Why public benchmarks don't answer your question

Public benchmarks measure aggregate performance on curated tasks. Your day-to-day work is narrower and stranger: your naming conventions, your framework versions, your tests. A model that tops a leaderboard can still mangle your ORM migrations or hallucinate your internal utility functions. The fix is cheap: a fixed prompt suite, a fixed scoring rubric, and a runner script you can point at any OpenAI-compatible endpoint.

Step 1: Build a task suite from your own repo

Pick 6–10 tasks that represent what you actually ask a coding assistant to do. Good candidates:

• Bug localization: paste a failing test plus the relevant file, ask for the root cause.
• Small feature diff: describe a change, ask for a patch in unified diff format.
• Refactor with constraints: "extract this function, keep the public signature, no new dependencies."
• Test authoring: ask for tests for a function with known edge cases.
• Explain-the-code: ask for a concise explanation of a module you understand well, so you can judge accuracy yourself.

Store each task as JSON:

{
  "id": "bugfix-session-timeout",
  "type": "bug_localization",
  "prompt": "Given this failing test and module, identify the root cause and propose a minimal fix...",
  "context_files": ["src/session.py", "tests/test_session.py"],
  "rubric": {
    "correctness": "Identifies the stale-cache bug, not the red herring in retry logic",
    "minimality": "Patch touches fewer than 15 lines",
    "no_hallucination": "Does not reference functions that don't exist in the repo"
  }
}

The rubric is the part most people skip, and it's the part that makes results comparable across models and across runs.

Step 2: A minimal runner script

The harness below is a starting point you can adapt. It assumes an OpenAI-compatible chat-completions API — which most hosted coding platforms, including free tiers, expose in some form. Treat it as a template, not a finished product:

import json, time, pathlib
from openai import OpenAI  # pip install openai

def run_suite(endpoint, api_key, model, suite_path):
    client = OpenAI(base_url=endpoint, api_key=api_key)
    tasks = json.loads(pathlib.Path(suite_path).read_text())
    results = []
    for task in tasks:
        context = "\n\n".join(
            pathlib.Path(f).read_text() for f in task["context_files"]
        )
        start = time.time()
        resp = client.chat.completions.create(
            model=model,
            temperature=0,
            messages=[
                {"role": "system", "content": "You are a careful coding assistant."},
                {"role": "user", "content": context + "\n\n" + task["prompt"]},
            ],
        )
        results.append({
            "task": task["id"],
            "model": model,
            "latency_s": round(time.time() - start, 2),
            "output": resp.choices[0].message.content,
        })
    return results

Key decisions, deliberately:

• temperature=0 so runs are as reproducible as the model allows.
• Raw outputs are saved, not just scores, so you can audit disagreements later.
• Latency is recorded because a slightly better answer that's 4x slower changes your workflow choice.

Step 3: Score against the rubric, not vibes

Score each output 0–2 per rubric criterion (0 = wrong, 1 = partially right, 2 = fully meets criterion). Two passes help:

1. You score it for correctness and hallucination on tasks where you know the answer.
2. A stronger model scores it as a second opinion — but only for style and completeness, never as the sole judge of correctness, since model-graded correctness tends to inflate scores for fluent-but-wrong answers.

Step 4: Decide with a table, not a feeling

Here's an empty decision table as a template — fill it with your own results:

Criterion	Weight	Model A	Model B
Rubric score (avg)	40%
Hallucination incidents	25%
Latency p50	15%
Cost (here: free vs. paid)	10%
Self-host / data control	10%

Weights are yours to set. The point is that "Model B felt smarter" becomes "Model B hallucinated on 3 of 8 repo tasks, so it stays out of my bug-fix loop."

Where free tiers fit in

Free model access is exactly what makes this workflow practical: you can run a 10-task suite against three or four candidate models without spending anything, and rerun it monthly as models change. This is also where I've been pointing the harness at MonkeyCode — Disclosure: This article was prepared as part of MonkeyCode's product outreach. It offers free access to coding models plus a free server option, which means you can run the suite against hosted endpoints without standing up your own infrastructure first. The free-server angle matters for a specific reason: evaluation harnesses are chatty, and having a hosted endpoint means your laptop isn't the bottleneck while a suite runs in the background.

A few honest caveats: free tiers change. Rate limits, available models, and server availability can shift without notice, so design your harness to treat endpoints and model names as configuration, not constants. And a free server is not the right place to paste proprietary code without checking the provider's data-handling terms — that decision table row about "data control" exists for a reason.

Limitations and who should skip this

• Small suites have high variance. Eight tasks can't rank two close models definitively. Treat results as "good enough for my workflow," not as a leaderboard.
• Reproducibility is approximate. Even at temperature 0, hosted models can drift between versions. Keep your raw outputs and timestamps.
• This is overkill if you only use AI for throwaway snippets, or if your organization has already mandated a tool — in that case, put the effort into prompt quality instead.
• Don't do this against production secrets or code you can't share with the hosting provider, whatever the tier costs.

Closing

The durable skill here isn't picking today's best free model — it's having a harness that tells you, in an afternoon, when the answer changes. If you want a zero-cost starting point for the endpoints, MonkeyCode's free models and free server are one option to plug into the runner above; the suite and rubric are what make the choice yours.