I Stopped Trusting My Gut on AI Coding Models. Here's the 30-Minute Test Rig I Use Instead

Quinn Zhu — Mon, 10 Aug 2026 08:31:38 +0000

Every few weeks, a new model lands on the front page with a slick demo, and every few weeks I watch someone adopt it because the demo felt impressive. Then, a sprint later, the same person is untangling hallucinated imports from their diff history.

I used to do this too. My evaluation method was: paste a tricky problem, squint at the answer, form an opinion. That method has a blind spot the size of a truck, and this post is about the tiny rig I built to replace it — a task pack, a batch runner, and a dumb-but-honest grader. It takes about half an hour to set up and runs against any OpenAI-compatible endpoint, including free ones.

The flaw in vibe-based adoption

Think about what a single test prompt actually measures:

You picked the task, and you picked one you understand. So you're scoring plausibility to you, not correctness in the places you can't check.
You ran it once. Generative output is a distribution, not a function. One draw tells you nothing about the tails.
You have no snapshot. Hosted models get silently swapped or re-tuned. Without a stored baseline, "it got worse" is just a mood.

A rig with frozen tasks, repeated sampling, and archived outputs addresses each of these directly.

Step one: build a task pack from your own scars

Generic benchmark prompts are useful for leaderboards and useless for you, because your failure modes are specific. My task pack has four buckets, each drawn from a way generated code has actually burned me:

Bucket A — constrained implementation. Give a spec with a hard constraint the model loves to ignore:

{
  "id": "A-rate-limiter-no-deps",
  "prompt": "Implement a token-bucket rate limiter in Go as a single struct with Allow() bool. Standard library only, goroutine-safe, no external packages.",
  "checks": [
    {"kind": "must_contain", "value": "func"},
    {"kind": "must_contain", "value": "sync."},
    {"kind": "must_not_contain", "value": "golang.org/x/"}
  ]
}

Bucket B — code reading. Present broken code and see if the diagnosis is right:

{
  "id": "B-closure-loop",
  "prompt": "This Python prints 3, 3, 3 instead of 0, 1, 2. Explain why and give the one-line fix.\n\nfuncs = [lambda: i for i in range(3)]\nprint([f() for f in funcs])",
  "checks": [
    {"kind": "regex", "value": "late binding|closure|i=i|default arg", "note": "names the closure/late-binding issue"}
  ]
}

Bucket C — the trap. This is the bucket nobody builds and everybody needs:

{
  "id": "C-fictional-lib",
  "prompt": "Show me how to stream-parse CSV with the `chunkreader-ex` Python package.",
  "checks": [
    {"kind": "regex", "value": "not aware|doesn't exist|can't find|unfamiliar|verify", "note": "refuses or flags uncertainty"}
  ]
}

chunkreader-ex does not exist. I invented it. A model that cheerfully writes you an import block and a fluent API for it will do exactly the same thing with your company's internal SDKs, where no Stack Overflow answer will save you. A pass here counts double in my scoring.

Bucket D — your real bugs. Paste in a sanitized version of an actual bug from your issue tracker and ask for a fix. This is the least gameable task you'll ever write, because it can't be in anyone's training data yet.

Step two: a runner that doesn't care what's behind the URL

The runner is deliberately unglamorous. It reads the task pack, fires each prompt at an endpoint several times, and dumps raw completions to a timestamped file:

# rig.py — template. Point it at your own endpoint and run it.
import json, glob, time, urllib.request

BASE_URL = "http://localhost:8080/v1/chat/completions"  # any OpenAI-compatible server
MODEL_ID = "model-under-test"
SAMPLES = 4

def ask(prompt: str) -> str:
    payload = json.dumps({
        "model": MODEL_ID,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0.7,
    }).encode()
    req = urllib.request.Request(
        BASE_URL, data=payload,
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=180) as resp:
        return json.load(resp)["choices"][0]["message"]["content"]

out = []
for f in sorted(glob.glob("pack/*.json")):
    task = json.load(open(f))
    for i in range(SAMPLES):
        out.append({"id": task["id"], "sample": i,
                    "text": ask(task["prompt"]), "t": int(time.time())})
        time.sleep(2)  # courtesy pause, especially on free endpoints

stamp = int(time.time())
json.dump(out, open(f"run-{stamp}.json", "w"), indent=2)
print(f"{len(out)} completions -> run-{stamp}.json")

Keep the output files forever. They're small, and they are the entire reason you can detect drift later.

Step three: grade mechanically, read manually

# grade.py
import json, re, glob

def passes(text, chk):
    k = chk["kind"]
    if k == "must_contain":     return chk["value"] in text
    if k == "must_not_contain": return chk["value"] not in text
    if k == "regex":            return bool(re.search(chk["value"], text, re.I))
    return False

run_file = sorted(glob.glob("run-*.json"))[-1]
pack = {t["id"]: t for t in
        (json.load(open(f)) for f in glob.glob("pack/*.json"))}

buckets = {}
for row in json.load(open(run_file)):
    task = pack[row["id"]]
    frac = sum(passes(row["text"], c) for c in task["checks"]) / len(task["checks"])
    buckets.setdefault(row["id"], []).append(frac)

for tid, scores in sorted(buckets.items()):
    print(f"{tid:26s} worst={min(scores):.2f} mean={sum(scores)/len(scores):.2f}")

Two deliberate choices here:

Print the worst sample first. Means lie about risk. A model that nails a task three times and produces garbage once will eventually hand you that garbage in a real PR. The worst-case column is what you're actually buying.
The checks are shallow on purpose. A regex can confirm the model mentioned late binding; it cannot confirm the explanation is correct. The grader's job is to eliminate obviously-failing samples so your human review time goes only to the plausible ones. Filtering, not verdicts.

Where free access fits

This rig needs an endpoint but almost no volume — a dozen tasks times four samples is a tiny, bursty workload with no latency requirements. That's exactly the shape free tiers handle well.

On the hosted side, MonkeyCode currently offers free model access along with a free server option, which is sufficient to point BASE_URL at it and get real numbers without a billing account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The more important point is structural: nothing in the rig depends on which provider answers. Judge a free endpoint with the same pack you'd judge a paid one, and archive every run so you can re-execute the identical suite the day the provider rotates models — or the day you rotate providers.

When this is the wrong tool

Be honest about the boundaries:

Four samples is directional, not statistical. Meaningful variance estimates start around ten or more runs per task, which may blow through free quotas. Treat low-sample results as a filter, not a ranking.
Free terms move. Quotas shrink, model lineups change, and "free" hosted inference sometimes comes with data-usage clauses. Read the actual policy before any proprietary code crosses the wire, and never assume the offering is permanent.
Don't put this in CI. Rate limits plus a blocking merge gate is how you get a red pipeline at 5pm on a Friday.
This measures single-shot text quality only. If the model will drive tools — shells, browsers, ticket systems — output correctness is the easy half of the problem, and you need sandboxing and permission scoping that this rig doesn't touch.
Your pack rots. Tasks that circulate get absorbed into training corpora. Refresh Bucket D from fresh bugs every month or two.

What I'd actually ask you to take away

The rig is not the point — you could rebuild it in an afternoon from scratch and lose nothing. The point is the habit: never let a demo video, a benchmark chart, or a pricing page substitute for evidence on your tasks, and never adopt without a baseline you can re-run. Thirty minutes of mechanical checking up front is cheaper than one afternoon reverse-engineering confidently wrong generated code.

If you're looking for a zero-cost endpoint to run a first pass against, MonkeyCode's free access works with the runner above as-is — but whatever you point it at, save the run files. The question "did this model change, or did I?" is only answerable if past-you left receipts.

DEV Community: Quinn Zhu