DEV Community: Riley Zhang

Score Coding Models With a 60-Line Harness Before You Spend a Cent

Riley Zhang — Wed, 05 Aug 2026 10:12:06 +0000

Everyone on DEV this week is talking about agents, orchestration, and multi-agent pipelines. But there's a boring question that comes before all of that: can the model you picked actually do your task? Most of us answer it by pasting a prompt into a chat UI, squinting at the output, and saying "looks fine." That's not an evaluation — and if you iterate against a paid API, it's also a slow leak of money while you decide.

Here's a cheaper pattern I reach for: a tiny, fixed task set with deterministic checks, run against any OpenAI-compatible endpoint. It takes an afternoon to set up, costs nothing if you point it at a free tier, and gives you a pass/fail table instead of a vibe.

The idea

A prompt regression harness has three parts:

Fixed tasks — a handful of small coding prompts that represent what you actually ask models to do.
Deterministic verification — each generated answer is extracted and executed against real assert statements. No human squinting.
Repeatability — run each task N times at temperature: 0 and report a pass rate. One lucky generation shouldn't count.

Because the harness speaks the OpenAI chat-completions format, you can point it at almost anything: a paid API, a local server like llama.cpp or vLLM, or a hosted free tier.

The harness

Stdlib-only Python. Save as model_harness.py:

#!/usr/bin/env python3
"""Score a coding model against a small fixed task set.

Point it at any OpenAI-compatible endpoint:

    export LLM_BASE_URL="https://your-endpoint/v1"
    export LLM_API_KEY="..."          # or a placeholder for local servers
    export LLM_MODEL="model-name"
    python model_harness.py
"""
import json, os, re, subprocess, sys, tempfile, time, urllib.request

BASE_URL = os.environ.get("LLM_BASE_URL", "http://localhost:8000/v1").rstrip("/")
API_KEY  = os.environ.get("LLM_API_KEY", "none")
MODEL    = os.environ.get("LLM_MODEL", "change-me")
RUNS     = int(os.environ.get("RUNS", "3"))

TASKS = [
    {
        "name": "dedupe_keep_order",
        "prompt": (
            "Write a Python function `dedupe(items)` that returns the list "
            "with duplicates removed, preserving first-occurrence order. "
            "Reply with only a single python code block."
        ),
        "test": "from solution import dedupe\n"
                "assert dedupe([1, 2, 1, 3, 2]) == [1, 2, 3]\n"
                "assert dedupe([]) == []\n"
                "assert dedupe(['a', 'a', 'b']) == ['a', 'b']\nprint('ok')\n",
    },
    {
        "name": "fizzbuzz",
        "prompt": (
            "Write a Python function `fizzbuzz(n)` returning the classic "
            "FizzBuzz list from 1 to n. Reply with only a python code block."
        ),
        "test": "from solution import fizzbuzz\n"
                "assert fizzbuzz(5) == [1, 2, 'Fizz', 4, 'Buzz']\n"
                "assert fizzbuzz(15)[14] == 'FizzBuzz'\nprint('ok')\n",
    },
    {
        "name": "parse_keyvals",
        "prompt": (
            "Write a Python function `parse(s)` that parses a string like "
            "'a=1, b = 2 ,c=3' into {'a': '1', 'b': '2', 'c': '3'}. "
            "Reply with only a python code block."
        ),
        "test": "from solution import parse\n"
                "assert parse('a=1, b = 2 ,c=3') == {'a': '1', 'b': '2', 'c': '3'}\n"
                "assert parse('x=y') == {'x': 'y'}\nprint('ok')\n",
    },
]

def chat(prompt):
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=120) as r:
        return json.load(r)["choices"][0]["message"]["content"]

def extract_code(text):
    blocks = re.findall(r"```

(?:python)?\s*\n(.*?)

```", text, re.S)
    return blocks[0] if blocks else text

def run_once(task):
    code = extract_code(chat(task["prompt"]))
    with tempfile.TemporaryDirectory() as d:
        with open(os.path.join(d, "solution.py"), "w") as f:
            f.write(code)
        p = subprocess.run([sys.executable, "-c", task["test"]],
                           cwd=d, capture_output=True, text=True, timeout=30)
        return p.returncode == 0

if __name__ == "__main__":
    total_pass = total_runs = 0
    for task in TASKS:
        passed = 0
        start = time.time()
        for _ in range(RUNS):
            try:
                passed += run_once(task)
            except Exception as e:
                print(f"  error on {task['name']}: {e}", file=sys.stderr)
        total_pass += passed
        total_runs += RUNS
        print(f"{task['name']:20s} {passed}/{RUNS} passed "
              f"({time.time() - start:.1f}s)")
    print(f"\nOverall: {total_pass}/{total_runs} "
          f"({100 * total_pass / max(total_runs, 1):.0f}%)")

Run it against a local server, then against a hosted endpoint, and compare the tables. The tasks are deliberately trivial — that's a starting point, not a benchmark. Replace them with prompts pulled from your last month of actual usage and the scores start meaning something.

Safety note: this executes model-generated code. For anything beyond toy tasks, run the subprocess inside a container or disposable VM with no network and no credentials.

Where a free tier fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One practical endpoint option for this workflow is MonkeyCode, which currently offers free model access and a free server option. That combination is genuinely useful for exactly the phase described above: you're iterating on prompts and task definitions, you'll burn a lot of calls, and you don't yet know which model deserves your money. Pointing the harness at a zero-cost endpoint means your experimentation budget is time, not dollars. Set LLM_BASE_URL, LLM_API_KEY, and LLM_MODEL per its docs and the script above works unchanged.

Two honest caveats: free tiers can change their quotas, model lineup, or availability, so treat current terms as the source of truth rather than anything written here — and don't route proprietary code or secrets through any third-party endpoint, free or paid, without checking its data policy.

Decision table: which endpoint for which phase

Phase	Free hosted tier	Paid API	Self-hosted / local
Exploring whether a model fits your tasks	✅ Best fit — zero marginal cost per experiment	Wasteful at high iteration volume	Good if you already have the GPU
CI prompt-regression on every commit	Fine if rate limits allow	✅ Predictable quotas	✅ No external dependency
Proprietary / regulated code	⚠️ Check data policy first	⚠️ Check data policy first	✅ Best fit
Latency-sensitive production traffic	❌ Not the right tool	✅ SLAs exist	✅ If you can operate it
Offline / air-gapped work	❌	❌	✅ Only option

Limitations

Small samples lie. Three tasks times three runs tells you almost nothing statistically. Treat early scores as directional and grow the task set from real failures you observe.
Passing tests ≠ good code. A model can pass asserts with unreadable, unmaintainable output. Add human review for anything you'd ship.
Contamination is real. Classic tasks like FizzBuzz are almost certainly in training data, which inflates scores relative to your novel, private tasks.
Temperature 0 isn't determinism. Providers can and do return different outputs across runs and versions. Re-run periodically and record results with the model name and date.
Free-tier availability can change. Don't hard-wire a free endpoint into anything you'd miss if it disappeared tomorrow.

Who should skip this

If you already have an eval framework (like a proper benchmark suite or an LLM-judged pipeline), this is a downgrade. If your decision is purely about production latency or compliance, toy pass rates won't answer it. And if you'd only ever ask a model one question, ever — just use the chat UI.

For everyone else in the "which model is even worth paying for" phase: steal the script, swap in your own tasks, and let a table make the decision instead of a hunch. If you don't have spare GPU capacity or an API budget, MonkeyCode's free model access and free server option are one zero-cost way to get an endpoint for exactly this experiment.

A Reproducible Way to Evaluate Free AI Coding Models Before You Commit

Riley Zhang — Wed, 05 Aug 2026 06:40:18 +0000

Free tiers and free model access are everywhere right now, and that's genuinely useful — but it creates a new problem: how do you compare models you haven't paid for, without burning a weekend on vibes-based testing?

Most people evaluate a coding model the same way: paste one prompt, skim the output, decide it "feels smart" or "feels dumb." That's not an evaluation. That's a coin flip with extra steps. The model that impresses you on a greenfield snippet might fall apart on a messy legacy refactor, and the one that bored you might be the most reliable at writing tests.

This post lays out a small, reproducible harness you can run against any free model endpoint in under an hour, plus a scoring rubric that survives your own mood. It works whether you're comparing hosted free tiers, local models, or a mix of both.

The core idea: fixed tasks, fixed prompts, fixed rubric

The entire method rests on three constraints:

The same tasks for every model, stored as files — never retyped from memory.
The same prompt template, so differences in output come from the model, not from how you happened to phrase things at 11pm.
A written rubric you fill in before reading any model's name attached to an output, if you can manage it.

That's it. No benchmark suite, no GPU cluster. Just discipline, packaged as a script.

The task set

Pick five tasks that mirror your actual work. Here's a starter set that covers the failure modes I see people hit most often:

#	Task type	What it reveals
1	Greenfield function with a spec doc	Spec fidelity, hallucinated APIs
2	Bug fix in an unfamiliar file	Reading comprehension, minimal diffs
3	Refactor with a constraint ("no behavior change")	Whether it respects invariants
4	Test generation for existing code	Edge-case awareness
5	"This code is slow, why?" explanation	Reasoning vs. pattern-matching

Task 2 and 3 are where weak models get exposed. Greenfield generation is the easiest thing to fake; surgical edits are not.

The harness

Save each task as a directory with an input/ (the code) and a prompt.md. Then a small runner applies the same prompt to each model endpoint and writes outputs to separate folders so you can score them blind:

#!/usr/bin/env python3
"""Minimal model-eval runner. Bring your own endpoint config."""
import json, subprocess, sys
from pathlib import Path

MODELS = {
    "model_a": {"cmd": ["python", "clients/client_a.py"]},
    "model_b": {"cmd": ["python", "clients/client_b.py"]},
    # Add or remove freely. Each client reads a prompt on stdin,
    # prints the model's raw response on stdout.
}

def run_model(client_cmd: list[str], prompt: str) -> str:
    result = subprocess.run(
        client_cmd, input=prompt, capture_output=True, text=True, timeout=300
    )
    if result.returncode != 0:
        return f"[ERROR] {result.stderr.strip()}"
    return result.stdout

def main(task_dir: str):
    task = Path(task_dir)
    prompt = (task / "prompt.md").read_text()
    code_context = "\n\n".join(
        f"### {p.name}\n{p.read_text()}" for p in sorted((task / "input").glob("*"))
    )
    full_prompt = f"{prompt}\n\n## Code\n{code_context}"

    for name, cfg in MODELS.items():
        out_dir = task / "outputs" / name
        out_dir.mkdir(parents=True, exist_ok=True)
        response = run_model(cfg["cmd"], full_prompt)
        (out_dir / "response.md").write_text(response)
        print(f"{name}: done ({len(response)} chars)")

if __name__ == "__main__":
    main(sys.argv[1])

The important part isn't the Python — it's that the prompt is assembled identically for every model, and outputs land in folders you can review without knowing which is which.

For task 2 and 3, add one more step: apply the model's suggested diff and run the project's existing test suite. A response that reads beautifully but breaks pytest scores zero. That single check eliminates most "looks right" false positives.

The rubric

Score each output 0–2 on four axes, written down before you check which model produced it:

Correctness: does it run / pass existing tests? (0 = broken, 1 = partially, 2 = yes)
Minimality: did it touch only what it needed to?
Spec fidelity: did it follow the constraints in the prompt, or quietly drop one?
Explanation quality: when asked why, is the reasoning verifiable?

Max score per task is 8; five tasks means 40 points per model. Below ~24, the model will cost you more time than it saves on real work.

Where free access fits in

This harness is exactly the situation where free model access is most valuable: you want breadth of comparison without committing budget to a model you haven't validated on your tasks.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access plus a free server option, which makes it a reasonable candidate to slot into the MODELS table above as one of the endpoints under test — especially if you'd rather not run everything locally. The honest way to treat it, though, is the same as every other candidate: run it through the five tasks blind and let the rubric decide. If it scores well on your task mix, keep it; if it doesn't, the free tier cost you nothing but an hour.

If you want to try this, the lowest-friction starting point is running the harness against two or three free endpoints on a single task type you care about most, then expanding from there.

Limitations and who shouldn't use this

Sample size is tiny. Five tasks tell you about your workflow, not about the model in general. Don't publish your rubric scores as a universal benchmark — they aren't one.
Blind scoring is hard to maintain. If you wrote the client config, you'll recognize output styles. Recruit a teammate to shuffle folders if rigor matters.
Latency and reliability aren't measured here. A model that scores 38/40 but times out every third request is a production liability. Measure that separately over a longer window.
Free tiers change. Availability, rate limits, and which models are included can shift without notice. Re-run the harness before you build anything load-bearing on a free option.

If you're choosing a model for a large team with compliance or data-residency requirements, this process is a starting filter at best — you still need the procurement-grade evaluation. And if your work is mostly one-off scripts you'll never revisit, the hour of setup may genuinely not pay for itself.

Takeaway

Free model access is only useful if you can tell the good outputs from the fluent ones. Fix the tasks, fix the prompts, score blind, run the existing tests. An hour of structure beats a week of vibes.