Another week, another model announcement dominating my feed. The pattern is predictable by now: launch post, impressive benchmark chart, a dozen screenshots of one-shot successes, and a comment section arguing about whether everything else is obsolete. I've been in those comment sections. I've also been burned by acting on them.
What finally cured me of release-day switching was a simple realization: every piece of launch-day evidence is measured against someone else's workload. Public benchmarks use curated tasks. Screenshots are the best run out of many. Neither tells me anything about the prompts I actually send — the messy refactors, the half-documented schemas, the tracebacks from my own dependency hell. So I built a fixed evaluation I can rerun the moment any new model appears, and I let that — not the timeline — make the call.
This post is the whole setup: what goes in the fixture set, how the runner works, how I read the results, and where I run it so the habit costs me nothing.
Fix the yardstick before the next launch
The whole approach collapses if you assemble your test after the hype arrives, because you'll unconsciously pick tasks where the new model shines. Everything below gets defined once, on a quiet week, and then frozen.
1. A frozen fixture set. I keep roughly twenty prompts pulled from my real history: an actual bug I asked a model to diagnose, a genuine refactor from a repo I maintain, a docstring task, a data-munging script, a "why is this test flaky" question. Each entry carries a checkable expectation — never just "good output."
2. Mechanical checks first, model judging last. My checks are ranked by trust: exact string or structural assertions where possible, then pattern matching, then — only when nothing else works — a separate judge model scoring against a written rubric. Judge scores get flagged in the output because a model grading a model is a signal, not a verdict.
3. A shadow metric for cost and speed. Every run records tokens and wall-clock time. A challenger that wins on quality but triples your latency on long prompts is often a downgrade in disguise, and you'll only see it if you measure it.
The runner
The harness is deliberately boring — plain Python against any OpenAI-compatible endpoint. Boring is the point: when the plumbing is trivial, provider changes stay trivial too.
# model_check.py — frozen fixtures vs. candidate models
import json, re, time, sys
from openai import OpenAI
client = OpenAI() # endpoint and key come from environment variables
def check(expect, answer):
mode = expect["mode"]
if mode == "contains":
return all(fragment in answer for fragment in expect["fragments"])
if mode == "pattern":
return re.search(expect["regex"], answer) is not None
if mode == "judge":
return judge(expect["rubric"], answer)
raise ValueError(f"unknown check: {mode}")
def judge(rubric, answer):
# A second model applies the rubric. Flagged as judge-graded downstream
# because judges carry their own biases.
r = client.chat.completions.create(
model=JUDGE_MODEL, temperature=0,
messages=[{"role": "user", "content":
f"Rubric: {rubric}\n\nAnswer: {answer}\n\nDoes it satisfy the rubric? Reply 1 or 0 only."}],
)
return r.choices[0].message.content.strip() == "1"
def evaluate(model, item):
t0 = time.time()
r = client.chat.completions.create(
model=model, temperature=0,
messages=[{"role": "user", "content": item["prompt"]}],
)
answer = r.choices[0].message.content
return {
"model": model, "item": item["key"],
"ok": check(item["expect"], answer),
"graded_by_judge": item["expect"]["mode"] == "judge",
"seconds": round(time.time() - t0, 2),
"tokens": r.usage.prompt_tokens + r.usage.completion_tokens,
}
if __name__ == "__main__":
candidates = sys.argv[1:] # model_check.py incumbent challenger
fixtures = [json.loads(line) for line in open("fixtures.jsonl")]
rows = [evaluate(m, it) for m in candidates for it in fixtures]
stamp = int(time.time())
json.dump(rows, open(f"results_{stamp}.json", "w"), indent=2)
for m in candidates:
mine = [r for r in rows if r["model"] == m]
wins = sum(r["ok"] for r in mine)
secs = sum(r["seconds"] for r in mine) / len(mine)
toks = sum(r["tokens"] for r in mine)
print(f"{m}: {wins}/{len(mine)} passed | {secs:.1f}s avg | {toks} tokens")
A fixture entry carries its own expectation:
{"key": "flaky-test-03", "prompt": "This pytest passes alone but fails in the full suite. List the three most likely causes and how to confirm each:\n```
\n...\n
```", "expect": {"mode": "pattern", "regex": "(?i)(state|order|isolat)"}}
A word of honesty about the checks: substring and regex assertions are blunt instruments, and judge-graded items inherit the judge's blind spots. That's acceptable — the goal is a consistent ruler, not a perfect one. What isn't acceptable is trusting the summary line blindly. I always read the actual outputs on any fixture where the challenger failed and the incumbent passed. That's the ten minutes where the real information lives.
Reading results without fooling myself
A single pass-rate number has misled me before, so every run gets forced through the same five gates:
| Gate | Green light to adopt | Yellow — retest later |
|---|---|---|
| Fixture pass rate | beats incumbent by 5+ points | within 5 points either way |
| Unexplained regressions | none | at most two |
| Speed on long prompts | no worse than incumbent | under 1.5x slower |
| Token cost per full run | no higher than incumbent | under 2x |
| My three make-or-break tasks | all pass | two pass |
That final gate does the heavy lifting. Averages conceal the specific tasks your daily work depends on. A model can be better across the board and still fail the one refactor pattern you hit every Tuesday — which makes it worse for you, regardless of what the aggregate says.
Anything that lands below the yellow column goes into a passed-on.md log with the date and the numbers. That file has saved me twice already: months later, when a model's reputation had grown and I started doubting my earlier call, I could check whether I'd rejected it on evidence or on a bad day.
Keeping the habit at zero cost
Each evaluation is a few dozen API calls per candidate — cheap individually, but it becomes real money if you evaluate every launch on paid frontier pricing. Free capacity is what turns this from an occasional project into a reflex.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
My runs go through MonkeyCode, which provides free model access plus a free server option, so each new release costs me some waiting time and nothing else, and the runs happen off my laptop. Because the harness only speaks the OpenAI-compatible API, pointing it at any provider is an environment-variable swap — no code changes. Two cautions if you take the same route: shared free capacity makes timing numbers jittery, so never adopt or reject on latency without a confirmation run; and free access can change or disappear, which is exactly why the workflow is designed to migrate with a config edit rather than a rewrite.
If you build your own version, the fixture set is the part worth protecting — the script above is an afternoon of plumbing, but a set of twenty tasks that genuinely represent your work is an asset that appreciates every time you run it.
Skip this if
- You use a model casually and nothing downstream consumes its output. The ceremony costs more than just trying the new release directly. This pays for itself when model output flows into shipped code, published content, or unattended pipelines.
- Your work is pure taste. If the real question is "which phrasing feels right," no automated fixture will save you from human review. Automate what's checkable and budget human time for the rest.
- You're planning to run production on free capacity. Evaluating on free tiers is sensible; depending on capacity you don't control for production traffic is a different and much riskier decision.
Why bother
The ritual converts an emotional question — "everyone says this model is incredible, am I falling behind?" — into a measurement with a written answer. Most launches, the answer comes back "stay put," and reaching that conclusion calmly in one sitting is the whole point. And the one launch in ten that genuinely is better on your tasks at your latency? You find out the day it drops, with receipts, instead of three months later by accident.
Freeze the fixtures. Automate the check. Let the screenshots argue without you.
Top comments (0)