A Reproducible Baseline for Comparing Free LLM Coding Models on Your Own Repo

Scrolling DEV this week, roughly every third post is about AI coding tools, agents, or model comparisons. Most of them share the same weakness: the benchmarks happened on someone else's machine, on someone else's code, with prompts I can't rerun. When the result disagrees with my own experience, I have no way to check why.

So I stopped reading benchmarks and built a tiny harness that lets me compare models against my own repository, with fixed prompts, fixed scoring, and a log I can diff later. The trick that makes it cheap: you don't need paid API tiers to do this. You need any provider that gives you free model access and a free server option to run the harness on. In my case I used MonkeyCode's free model access plus its free server option — Disclosure: This article was prepared as part of MonkeyCode's product outreach. — but the harness below is provider-agnostic. Swap the endpoint and it works with whatever free tier you have.

Why vendor benchmarks fail on your codebase

Three reasons, consistently:

1. Distribution mismatch. A model that's great at greenfield React snippets may be mediocre at reading your 4-year-old Django codebase with custom middleware.
2. Prompt drift. Vendor demos use tuned prompts. Your real usage is messier.
3. No regression tracking. Even if a model works today, you won't notice when a silent update changes its behavior on your code.

A baseline harness fixes all three by making the comparison yours: your tasks, your rubric, your reruns.

The artifact: a minimal, rerunnable comparison harness

The design constraints:

• Fixed task set drawn from your repo (real functions to explain, real bugs to find, real tests to write).
• Deterministic scoring where possible (tests pass / fail), rubric scoring where not.
• One command to rerun everything, output committed to git so diffs are visible.

Here's the core runner in Python (~60 lines, stdlib + requests):

import json, subprocess, time, pathlib, requests

TASKS = json.loads(pathlib.Path("tasks.json").read_text())
RESULTS = pathlib.Path("results")
RESULTS.mkdir(exist_ok=True)

def call_model(endpoint, model, prompt):
    r = requests.post(endpoint, json={
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }, timeout=180)
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

def grade(task, output):
    if task["type"] == "write_test":
        # deterministic: save the generated test, run pytest, pass/fail
        pathlib.Path("scratch/test_gen.py").write_text(output)
        p = subprocess.run(["pytest", "scratch/test_gen.py", "-q"],
                           capture_output=True, text=True)
        return 1 if p.returncode == 0 else 0
    # rubric tasks (explain, find_bug): score = did output contain the key phrase?
    return int(task["expected_key"].lower() in output.lower())

def run(endpoint, model):
    rows = []
    for t in TASKS:
        out = call_model(endpoint, model, t["prompt"])
        rows.append({"task": t["id"], "score": grade(t, out),
                     "latency_s": None})  # add timing if you care
        time.sleep(1)  # be polite to free tiers
    path = RESULTS / f"{model.replace('/', '_')}.json"
    path.write_text(json.dumps(rows, indent=2))
    return path

if __name__ == "__main__":
    ENDPOINT = "https://your-free-endpoint.example/v1/chat/completions"
    for m in ["model-a", "model-b"]:
        print(run(ENDPOINT, m))

And a sample tasks.json built from real repo content:

[
  {
    "id": "explain-middleware",
    "type": "explain",
    "prompt": "Explain what this middleware does and name one edge case it mishandles:\n<paste your actual code here>",
    "expected_key": "connection reset"
  },
  {
    "id": "test-parsers",
    "type": "write_test",
    "prompt": "Write a pytest test for this function, including its failure mode:\n<paste function>"
  }
]

The whole loop is: curate 5–10 tasks from your repo → run per model → commit results → diff across runs and models. Because results live in git, when a provider quietly changes a model, your next rerun shows exactly which tasks regressed.

Where the free server actually matters

The harness is light, but two parts benefit from running on a server instead of your laptop:

• Consistency. Same machine, same network path, same timezone — latency numbers (if you record them) become comparable across runs weeks apart.
• Scheduling. A cron job that reruns the suite nightly turns your baseline into a regression monitor.

This is where MonkeyCode's free server option slotted in for me: I deployed the runner there once, pointed it at the free model endpoints, and let cron handle the rest. Nothing in the setup is MonkeyCode-specific — any always-on box with Python works — but "free model access + free server in one place" removed the two excuses I had for not doing this (cost and a machine to run it on).

Decision table: is a self-run baseline worth it for you?

Your situation	Worth building?
Picking one model for a team workflow	Yes — this is the cheapest evidence you'll get
Solo, casual AI use	Probably overkill; just try models on real tasks
Regulated/audited environment	Yes, and keep the git history as your audit trail
You only need a one-time choice	Run it once, skip the cron

Limitations and who shouldn't do this

• Rubric grading is weak. Keyword matching is a proxy, not understanding. Treat rubric scores as smoke signals; only the pytest-style tasks give hard pass/fail.
• Free tiers shift. Quotas, model availability, and rate limits change without notice — which is exactly why you want rerunnable results, but also why you shouldn't hard-depend on any single free endpoint. Don't build production systems on assumptions about free capacity.
• Small task sets lie. Five tasks will not rank models globally. They will rank models for your five tasks, which is the point — just don't generalize beyond it.
• If you need statistically meaningful model evaluation, use an established eval framework instead. This harness answers "which of these works on my repo this month," nothing more.

The takeaway

The most useful benchmark is the one you can rerun. Curate a handful of tasks from your real codebase, score them deterministically where you can, commit the results, and rerun on a schedule. Free model access plus a free server — MonkeyCode's or anyone else's — is enough infrastructure to start this week.

If you build a similar harness, I'd genuinely like to see your tasks.json structure in the comments — the task curation step is where I think most of the signal lives.