From Six Questions to a Script: My 30-Minute Eval Harness for Every New Model Release

A few weeks ago I wrote about the six questions I ask every new model. The response I got most often in private was: "Okay, but how do you actually run those questions without spending a weekend on it?"

Fair criticism. A checklist you can't execute quickly is trivia. So this is the follow-up: the small, ugly, reproducible harness I now run whenever a new model drops and the timeline starts screaming that it's cheap and great. It takes about 30 minutes of wall-clock time, most of which is me drinking coffee while prompts run.

The constraint that shaped everything

I'm not a lab. I don't need MMLU deltas. I need to know one thing: should this model replace the one currently wired into my side projects? That's a comparative question, so the harness compares the candidate against my current default on my tasks, not on benchmarks someone else designed for a press release.

The other constraint: cost. Evaluation season is expensive if every new release means spinning up paid API credits and a VPS. I currently run the harness using MonkeyCode's free model access for the candidate side and its free server option as the runner box, which keeps a full eval round at effectively zero marginal cost. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Everything below works with any OpenAI-compatible endpoint and any always-on machine, though — the harness doesn't care where the tokens come from.

The artifact: one file, three phases

The harness is a single Python script. It runs a fixed prompt suite against two endpoints (candidate and incumbent), records raw outputs, and then scores them with a rubric that maps to my six questions.

#!/usr/bin/env python3
"""model_gauntlet.py — compare a candidate model against your current default.

Usage:
    CANDIDATE_BASE=https://... CANDIDATE_KEY=... \
    INCUMBENT_BASE=https://... INCUMBENT_KEY=... \
    python model_gauntlet.py

No external deps beyond `httpx`. Deliberately boring on purpose.
"""
import json, os, time, hashlib, pathlib
import httpx

SUITE = [
    # (id, category, prompt, what_good_looks_like)
    ("refactor_1", "code_edit",
     "Here's a 40-line function with a hidden off-by-one and a swallowed exception. "
     "Find both, explain them in one sentence each, then rewrite. CODE: <paste yours>",
     "names both bugs before rewriting; rewrite compiles mentally"),
    ("explain_1", "comprehension",
     "Explain what this regex does, line by line, then give one input where it catastrophically backtracks.",
     "correct backtracking case, not a hand-wavy one"),
    ("honesty_1", "calibration",
     "What's the default max heap size of the JVM on a container with 512MB RAM on JDK 21? "
     "If you're not sure, say so explicitly.",
     "either correct-with-caveat or an explicit 'not sure'"),
    ("instruction_1", "constraint_following",
     "Summarize the following in EXACTLY three bullet points, no intro, no outro: <paste>",
     "exactly three bullets, zero filler"),
    ("drift_1", "long_context",
     "I'm going to give you a config file. Remember the value of RETRY_LIMIT. "
     "[600 lines of noise] ... Now: what was RETRY_LIMIT, and what line was it on?",
     "correct value; line number approximately right"),
    ("taste_1", "judgment",
     "Should I use a message queue or just a Postgres table with SKIP LOCKED for a job "
     "system doing ~50 jobs/minute? Push back if my framing is wrong.",
     "answers the actual question; doesn't gold-plate"),
]

def call(base, key, model, prompt):
    r = httpx.post(
        f"{base.rstrip('/')}/chat/completions",
        headers={"Authorization": f"Bearer {key}"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}],
              "temperature": 0.2},
        timeout=120,
    )
    r.raise_for_status()
    d = r.json()
    return d["choices"][0]["message"]["content"], d.get("usage", {})

def main():
    out_dir = pathlib.Path(f"runs/{time.strftime('%Y%m%d_%H%M')}")
    out_dir.mkdir(parents=True)
    for pid, cat, prompt, rubric in SUITE:
        row = {"id": pid, "category": cat, "rubric": rubric}
        for side in ("candidate", "incumbent"):
            text, usage = call(
                os.environ[f"{side.upper()}_BASE"],
                os.environ[f"{side.upper()}_KEY"],
                os.environ[f"{side.upper()}_MODEL"],
                prompt,
            )
            row[side] = {"text": text, "usage": usage}
            row[f"{side}_sha"] = hashlib.sha1(text.encode()).hexdigest()[:8]
        (out_dir / f"{pid}.json").write_text(json.dumps(row, indent=2))
        print(f"✓ {pid}")
    print(f"\nRaw outputs in {out_dir}. Now read them and score against the rubric.")

if __name__ == "__main__":
    main()

Three things to notice:

1. The prompts are mine. Two of the six are drawn from bugs I actually hit. A prompt suite built from your own scar tissue beats any public benchmark for the "should I switch?" question.
2. I score manually. No LLM-as-judge. With six prompts, judging takes 15 minutes and I learn things by reading the outputs — an auto-judge would hide exactly the failure modes I'm hunting for. I tried automated judging early on and it systematically overrated verbose, confident answers.
3. Raw outputs are archived with hashes. When someone says "that model regressed," I can diff my own archive instead of trusting vibes.

The decision table

After scoring, every candidate lands in one of four cells. This is the actual output of the whole exercise:

Result	Candidate wins on honesty + my-code tasks	Candidate wins only on generic tasks	Candidate loses on honesty	Tie everywhere
Action	Switch for code work, keep incumbent for drafting	Keep incumbent; revisit in one release	Disqualify regardless of benchmarks	Keep incumbent (switching cost isn't zero)
Why	Those two categories predict my real usage	Generic wins don't survive contact with my codebase	Calibration failures are disqualifying, full stop	Migration has real cost even when it's "free"

The honesty row is the one people argue with me about. I don't care how strong a model is on leaderboards if it confabulates a confident answer to my honesty_1 prompt — that failure mode costs me more than any capability gain pays back. Your table can weight things differently; mine reflects that I mostly review and edit generated code rather than accept it wholesale.

Where the free tier actually helps

The practical bottleneck for running this every release cycle is that it's just annoying enough to skip when it costs money or setup time. What made it a habit instead of an intention:

• The candidate side runs through MonkeyCode's free model access, so I can point CANDIDATE_BASE at whatever just launched without a billing decision first. If this kind of recurring eval is your bottleneck too, it's a reasonable place to start; the script above works unchanged against it.
• The free server option hosts the runner and the output archive, so runs/ accumulates in one place and I can kick off a round from my laptop or a phone browser.

Both could be swapped for anything else — that's the point of keeping the harness dependency-free.

Limitations, and who shouldn't bother

• Six prompts is a smell test, not a measurement. It will not detect a 3% regression in your specific domain. If you're gating a production rollout, you need a real eval set with hundreds of cases. This harness decides whether to investigate further, nothing more.
• My suite is contaminated by me. The prompts reflect my tasks (backend code review, config archaeology). A frontend engineer or a data scientist should write their own six; copying mine verbatim just measures how much your work resembles mine.
• Temperature 0.2 single-run. One sample per prompt means I'm comparing point estimates. I accept this because I'm looking for disqualifying failures, not fine rankings. If two models both pass cleanly, the harness deliberately says "tie" rather than pretending it can rank them.
• If your incumbent already handles your work and you ship rarely, skip this entirely. The best eval is the one your production traffic runs for free. This harness exists for people whose work (like mine) involves constantly auditioning tools.

The actual lesson

The six questions only became useful when they stopped being a mindset and became a script with an archive folder. Any new release — this week's included — now gets the same treatment: thirty minutes, six prompts, one decision-table cell, and the timeline's hype gets replaced by a folder of hashed outputs I can argue with later.

Write your own six prompts. The ones in the script are mine, and that's exactly why they work for me.