A Free-Model Harness for Choosing Which Coding Assistant to Trust

I keep landing in the same argument. One developer says just let the AI drive the tools. Another says never give an agent a tool. Neither position survives contact with a real repo.

The missing piece is measurement. Benchmarks tell me a model's average. They don't tell me whether it will handle my error messages, my SQL dialect, and my habit of writing half-finished tests.

So I designed a small harness. It sends the same coding problems to any OpenAI-compatible model, collects the responses, and scores them with a rubric. I can run it against free endpoints before spending a cent.

Why local tests beat public vibes

Public leaderboards are useful, but they hide a lot.

• They grade on tasks I rarely do.
• They don't show how a model handles follow-up questions.
• They don't reveal how a model behaves with my specific stack.
• They change too fast for my memory to keep up.

A personalized harness flips the order. It makes the model answer my work, not someone else's benchmark.

Two model IDs that keep showing up in my notes are deepseek-v4-pro-0813 and grok-4.6. I don't treat those as winners. I treat them as candidates to run through the same gauntlet.

What the harness looks like

MonkeyCode's free model access and free server option makes this experiment cheap enough to repeat weekly. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The code below is the skeleton. It doesn't depend on a particular SDK. It uses the standard library, so it runs anywhere Python 3 does.

import json
import os
import urllib.request

CANDIDATES = [
    {
        "name": "candidate-a",
        "base_url": os.getenv("MC_BASE_URL", "http://localhost:8000/v1"),
        "api_key": os.getenv("MC_API_KEY", "local"),
        "model": os.getenv("MC_MODEL", "replace-with-model-id"),
    },
    # Add another candidate with a different model id, for example:
    # "deepseek-v4-pro-0813" or "grok-4.6" if your provider exposes them.
]

PROMPTS = [
    {
        "id": "bug_triage",
        "text": (
            "You are reviewing a Python web app. A user reports: "
            "sometimes the checkout page redirects to login after payment. "
            "Ask me the three most useful questions before suggesting a fix."
        ),
    },
    {
        "id": "sql_nulls",
        "text": (
            "Write a PostgreSQL query that returns customers with no orders. "
            "Use NOT EXISTS, then explain why you chose it over NOT IN."
        ),
    },
    {
        "id": "refactor",
        "text": (
            "Refactor this function so it is easier to test without changing "
            "its public behavior: "
            "def apply(price, user): "
            "    if user.plan == 'pro': return price * 0.9 "
            "    return price"
        ),
    },
    {
        "id": "test_first",
        "text": (
            "Given this failing behavior, write the smallest failing test first. "
            "Do not write the implementation yet."
        ),
    },
]

def call(endpoint, api_key, model, prompt):
    body = json.dumps({
        "model": model,
        "messages": [
            {"role": "user", "content": prompt},
        ],
        "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(
        endpoint.rstrip("/") + "/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": "Bearer " + api_key,
        },
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        data = json.loads(resp.read())
    return data["choices"][0]["message"]["content"]

for candidate in CANDIDATES:
    print()
    print("=== " + candidate["name"] + " ===")
    for prompt in PROMPTS:
        try:
            reply = call(
                candidate["base_url"],
                candidate["api_key"],
                candidate["model"],
                prompt["text"],
            )
            print()
            print("# " + prompt["id"])
            print(reply[:600])
        except Exception as exc:
            print()
            print("# " + prompt["id"] + " ERROR: " + str(exc))

This script is intentionally unglamorous. You read the output, then score it. No hidden AI judge that can be gamed by another AI.

Score with a stupidly simple rubric

Don't ask which model is better. Ask four smaller questions.

Dimension	0 points	1 point	2 points
Correctness	Wrong or refuses	Partially right	Correct or clearly fixable
Explanation	No reasoning	Vague reasoning	Specific, causal reasoning
Safety	Dangerous steps	Warning missing	Notes risk and rollback
Actionability	Nothing to do	General advice	Exact next patch or command

Max score per prompt: 8 points. Total possible: 32 across four prompts.

Score two models side by side.

The model with a higher total is not the best. It's just better for these four tasks on this day. That's still more useful than a leaderboard.

Three things this harness does not measure

A personal harness is a filter, not a proof.

• Long-context work. Four short prompts don't show how a model handles a 40-file refactor.
• Tool reliability. It won't reveal whether an agent deletes the wrong file at step nine.
• Rate-limit behavior. Free endpoints can throttle. That's a cost, not a bug in the model.

For those cases, I need a separate test: a sandboxed repo, a time budget, and a rollback plan.

Who should skip this

Skip this harness if you need:

• production procurement evidence,
• legal or compliance review,
• deterministic benchmark numbers for a paper,
• guarantees about tomorrow's model availability.

Free access can change. Model IDs can disappear. The value is the workflow, not the endpoint.

Start smaller than you think

Pick one annoying task from this week.

Then run one prompt.

You'll learn more from a bad answer on your own bug than from ten public charts.

If MonkeyCode's free server is available in your console, use it for the first run. When you hit a limit, stop and read the output instead of hunting for another key.