Every Week a New Model Is "Cheaper and Better" — Here's the 30-Minute Harness That Settles It

#ai #tutorial #testing #productivity

Somewhere right now, a model release thread is telling you that a new checkpoint is dramatically cheaper and noticeably better than whatever you're running. Next week there will be another one. The claims aren't always wrong — that's what makes them dangerous. Sometimes the new model really is better for the benchmarks the poster ran, and quietly worse for the eleven tasks you actually do every day.

I got tired of making default-model decisions off vibes, so I built a tiny A/B harness. It takes about 30 minutes to set up, runs on a fixed set of prompts drawn from my own real work, and ends in a one-line decision: switch, stay, or split. This post is that harness, plus the decision table I use to read the results. It's a workflow, not a benchmark — your prompt set is the whole point.

Why your own prompts beat public scores

Public evals measure what eval authors care about. My day is a narrow distribution: refactoring mid-size Python diffs, writing migration scripts, summarizing error logs, and generating test scaffolding. A model can win every leaderboard and still mangle my SQL migrations, because migrations reward tedious exactness, not cleverness.

So the harness starts with 20–30 prompts I have actually sent in the last month, each with a short note on what a good answer looks like. Not formal unit tests — just enough to score consistently:

# eval_set.yaml — prompts from real work, not from a benchmark
- id: migration-01
  task: "Convert this Django migration to raw SQL for Postgres 15"
  pass_if: "runs without error; preserves the index; no ORM left in output"
- id: log-summary-04
  task: "Summarize this 400-line traceback into 3 bullet points"
  pass_if: "identifies root cause line; no invented stack frames"
- id: refactor-02
  task: "Split this 200-line function without changing behavior"
  pass_if: "same public signature; all existing tests referenced by name"

The runner

The runner hits any OpenAI-compatible chat endpoint twice — once with my current default model, once with the candidate — and dumps raw outputs side by side for scoring. Label this a template: fill in your own endpoints and keys, and expect to tweak it.

# ab_harness.py — minimal template, not production code
import json, time, urllib.request, yaml

def chat(base_url, api_key, model, prompt):
    req = urllib.request.Request(
        f"{base_url}/chat/completions",
        data=json.dumps({
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.2,          # keep it low so runs are comparable
        }).encode(),
        headers={"Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json"},
    )
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        body = json.load(r)
    return body["choices"][0]["message"]["content"], round(time.time() - t0, 2)

def main():
    cases = yaml.safe_load(open("eval_set.yaml"))
    results = []
    for c in cases:
        row = {"id": c["id"], "pass_if": c["pass_if"]}
        for label, model in [("current", "my-current-model"),
                             ("candidate", "new-candidate-model")]:
            out, secs = chat("https://your-endpoint/v1", "YOUR_KEY", model, c["task"])
            row[label] = out
            row[f"{label}_secs"] = secs
        results.append(row)
    json.dump(results, open("ab_results.json", "w"), indent=2)
    print(f"scored {len(results)} cases -> ab_results.json")

if __name__ == "__main__":
    main()

Two deliberate choices here: temperature pinned low so differences come from the model rather than sampling luck, and wall-clock time recorded per call so the "cheaper" claim has to survive contact with "slower." A model that costs 40% less but takes 3× longer on log summaries is not cheaper for a workflow where I'm waiting on it.

Scoring without lying to yourself

I score each pair blind when I can — shuffle the outputs, grade against the pass_if note, then unblind. Each case ends in one of three states: current wins, candidate wins, or tie. Then the decision table:

Outcome across your prompt set	Decision
Candidate wins ≥ 40% and loses ≤ 10%	Switch default, keep old model config for a week
Candidate wins on one task family only (e.g., summaries)	Split: route that task, keep default elsewhere
Within ±10% overall	Stay. Migration cost exceeds any saving
Candidate faster but ties on quality	Only switch if latency is your actual bottleneck

The "stay" row is the one hype threads never mention. Even a genuinely better model has switching costs: different failure modes to relearn, prompt phrasing that no longer applies, edge cases you had already patched around. A marginal quality win doesn't cover that.

Running candidates without a billing surprise

There's a practical problem with this harness: it only works if you can actually run the candidate model on your 25 prompts, repeatedly, at temperature 0.2, without watching a meter. Paid API access makes "just test it yourself" an expensive hobby when a new release shows up every week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This is the specific spot where MonkeyCode fits the workflow: it offers free model access plus a free server option, which is exactly what a release-day A/B harness wants — a place to stand up the candidate model and throw your prompt set at it without provisioning anything or doing cost arithmetic first. I treat it as the candidate lane in the script above, while the current-default lane stays wherever it already lives. I wouldn't run production traffic through a free tier, but for evaluation traffic that's precisely the right tool: ephemeral, low-stakes, and disposable.

Limitations, and who shouldn't do this

25 prompts is a smell test, not a measurement. It catches "obviously worse at my work" and "obviously better," which is usually all a default-model decision needs. It will not detect a 3% regression.
No load, no concurrency, no long-context stress. If you're choosing a model for a production service rather than a personal default, you need real latency-percentile and cost-per-million-token testing; this harness is the wrong tool.
Free access changes. Don't build a repeatable CI eval on top of any free tier without a fallback plan, and re-verify what's actually offered before each release cycle.
Blind scoring by hand doesn't scale. Past ~50 cases, add an LLM judge with a fixed rubric — and then distrust the judge a little too.

If you're in a regulated domain, or your "prompt set" contains anything sensitive, sanitize before anything touches a third-party endpoint — free or otherwise.

The actual habit

The harness matters less than the ritual: when a release thread claims "cheaper and better," the response is no longer reading forty replies — it's spending 30 minutes running the set and letting the table decide. Most weeks the answer is "stay," and that answer, backed by your own outputs instead of someone else's screenshots, is worth the half hour. If you try this, steal the decision table before you steal the script — the script is replaceable, but having a written rule for switch, stay, or split is what keeps release hype from making your architecture decisions for you.