Comparing AI Coding Models Without Burning Budget: A Reproducible Harness on Free Compute

Every few weeks a new coding model drops, the timeline fills with cherry-picked screenshots, and I'm left with the same question: is it actually better for my repo, or just better at demo prompts?

In my last article I described a sandbox loop for safely executing AI-generated code: generate, isolate, assert. This post is the natural next step — using that same loop as a comparison harness so I can evaluate candidate models on my own tasks instead of trusting leaderboard vibes. The trick that makes this sustainable is doing it on free model access and free server time, so the cost of curiosity is zero and I can rerun the whole thing whenever a new model appears.

Why leaderboards don't answer my question

Public benchmarks measure performance on tasks I don't have. My actual workload is boring and specific:

• Small refactors inside a legacy Flask app with inconsistent naming
• Writing pytest fixtures for functions with messy side effects
• Explaining stack traces from a CI job I can only partially reproduce

A model that wins a competitive-programming benchmark can still be mediocre at "add a regression test for this 200-line view function without breaking the other 40 tests." So the unit of evaluation has to be my tasks, my assertions, my pass criteria.

The setup: one harness, swappable models

The harness has three parts:

1. A task file — real prompts drawn from my actual work, each with an assertion script.
2. A runner — sends each prompt to whatever model endpoint is configured, captures the output.
3. The sandbox loop from my previous post — executes each generated solution in isolation and runs the assertions.

For compute, I've been running this through MonkeyCode, which offers free model access and a free server option, so the runner and the sandbox can live on their side without me paying for idle time or burning a paid API quota on experiments. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness itself is plain Python and works against any OpenAI-compatible endpoint — nothing below is locked to one provider.

tasks.yaml — real tasks, mechanical assertions

- id: regression_test_view
  prompt: |
    Here is a Flask view function (pasted below). Write a pytest
    regression test that covers the ValueError branch. Do not modify
    the view. Use only pytest and the standard library.
    ...
  assert_script: checks/test_regression_view.py
  pass_criteria: exit_code_zero

- id: explain_ci_failure
  prompt: |
    This CI log tail shows a failure (pasted below). Identify the
    most likely root cause in two sentences and name the file to fix.
  assert_script: checks/keyword_and_file.py
  pass_criteria: mentions_expected_file

Note that not every assertion is "run the code." For explanation tasks I check cheaper mechanical signals (does it name the right file? does it avoid inventing a config key that doesn't exist in the repo?) and reserve manual review for borderline cases.

runner.py — the swappable part

import subprocess, tempfile, pathlib, yaml

def run_task(client, model, task):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": task["prompt"]}],
        temperature=0,
    )
    output = resp.choices[0].message.content
    code = extract_code_block(output)  # returns None if no code block

    with tempfile.TemporaryDirectory() as tmp:
        workdir = pathlib.Path(tmp)
        if code:
            (workdir / "solution.py").write_text(code)
        result = subprocess.run(
            ["python", task["assert_script"]],
            cwd=workdir, capture_output=True, timeout=60,
        )
        return {
            "task": task["id"],
            "model": model,
            "passed": result.returncode == 0,
            "stderr_tail": result.stderr.decode()[-500:],
        }

def compare(clients_and_models, tasks_file="tasks.yaml"):
    tasks = yaml.safe_load(open(tasks_file))
    rows = []
    for client, model in clients_and_models:
        for task in tasks:
            rows.append(run_task(client, model, task))
    return rows

Everything is deterministic-ish by design: temperature=0, a fixed prompt file, fixed assertions. When a new model shows up, I add one line to the client list and rerun — that rerun is free, which is the whole point.

A decision table, not a verdict

After a couple of runs, I stopped asking "which model is best" and started recording where each one fails. My current table looks like this (model names anonymized because your table should be yours):

Task type	Model A	Model B	Model C
Regression test generation	pass 4/5	pass 5/5	pass 2/5
Refactor with constraints	pass 3/5	pass 2/5	pass 4/5
CI log diagnosis	wrong file 2/5	correct 4/5	correct 3/5
Hallucinated imports/config keys	rare	occasional	frequent

The actionable output isn't a winner — it's a routing rule: "use B for test generation, C for constrained refactors, never trust C's imports without a compile check." That rule survives the next model release; a hype-driven ranking doesn't.

Limitations, honestly

• Small-N problem. Five tasks per category is a smoke test, not statistics. I treat failures as signals to investigate, not proof.
• Temperature 0 isn't fully deterministic. Same prompt can still vary run to run. I rerun failures once before recording them.
• Mechanical assertions miss quality. A test can pass my checker and still be unreadable. I spot-review anything I'm going to actually adopt.
• Free tiers have limits. Rate limits and queue times mean I batch runs instead of iterating interactively, and I don't assume today's free access is permanent — the harness is provider-agnostic precisely so I can move it.
• Who shouldn't bother: if you only use AI autocomplete for single lines, a task-level harness is overkill. This pays off when you're choosing a model for sustained, multi-step work.

Where I'd start

If you want to try this, don't build the whole thing at once. Pick five prompts from work you did last week, write one assertion script, and run two models against them. If you're looking for somewhere to run it without a budget, MonkeyCode's free model access and free server option is one place to host the loop — but the harness above will run against whatever endpoint you already have.

The durable skill here isn't picking the right model this month. It's owning the evaluation, so every "new model just dropped" announcement becomes a ten-minute rerun instead of a leap of faith.