I Don't Vibe-Check New Local Models Anymore — I Run This 30-Minute Eval Instead

A new "best open model ever" drops roughly every week now. And for most of this year, I handled every release the same dumb way: download it, ask it three questions in a terminal, nod at the answers, think "yeah, this one's smart" — then spend the next two weeks finding out where it actually breaks, in my real work, one task at a time.

That's a vibe-check. It feels like evidence. It isn't. A model that nails a clever one-liner and then mangles JSON in my log pipeline is useless to me, and no amount of terminal flirting reveals that until something breaks at 2 AM.

For context: my daily drivers are a 9B model on my Mac Mini and a 30B coder model on the RTX 3060 in my PC. Swapping either of them isn't free — every model has quirks baked into my prompts by now. So I stopped vibe-checking. Now every model goes through a small eval suite before it's allowed anywhere near my stack. Thirty minutes to run, built in an afternoon, and it has saved me from three bad migrations this year.

Why the public benchmarks don't help me

I read benchmark posts. I just don't make decisions with them, because they measure the average internet's tasks, not mine.

My actual workload on a normal day:

• Extracting structured JSON out of messy log lines and API responses
• Editing Monkey C for my watch face projects (a language most models barely know)
• Writing small glue Python scripts against local APIs
• Summarizing long docs where specific facts have to survive, not just the vibe

A model can top every leaderboard and still be the wrong model for me if it can't emit valid JSON without a chaperone. The reverse is true too — the "boring" model that always follows the output format is worth more to my setup than the genius that occasionally freestyles.

What's in the suite

One folder, 40 cases, all lifted from real work:

• 12 extraction tasks — real (scrubbed) log lines; the model must return valid JSON with specific keys
• 10 code edit tasks — "here's a function, add retry with backoff, keep it under 30 lines" style
• 8 summarization tasks — checked for must-mention facts, not tone
• 6 format-adherence tasks — strict output shapes, no prose allowed
• 4 trap tasks — ones that look easy but contain a subtle gotcha. Models that bluff instead of flagging the problem fail here, which is exactly what I want to catch

The rule for growing it: every time a model messes something up in daily use and I have to fix it by hand, that failure becomes a new case. The suite grows from my own scars. It's the most valuable folder on my disk and it's maybe 60 KB of text.

The harness

60-ish lines of Python. No framework, no LLM-as-judge (a judge model just adds its own preferences and noise). Each case runs three times at temperature 0, and the score is the average pass rate:

import json, re, sys, requests

OLLAMA = "http://localhost:11434/api/chat"

def run_case(model, case):
    r = requests.post(OLLAMA, json={
        "model": model,
        "messages": [{"role": "user", "content": case["prompt"]}],
        "stream": False,
        "options": {"temperature": 0.0},
    })
    return r.json()["message"]["content"]

def check(case, output):
    kind = case["checker"]
    if kind == "json":
        try:
            data = json.loads(output)
            return all(k in data for k in case["expect_keys"])
        except ValueError:
            return False
    if kind == "regex":
        return bool(re.search(case["pattern"], output, re.S))
    if kind == "contains":
        return all(s.lower() in output.lower() for s in case["must_contain"])
    return True

def score(model, cases, runs=3):
    total = 0
    for case in cases:
        passed = sum(check(case, run_case(model, case)) for _ in range(runs))
        total += passed / runs
    return round(100 * total / len(cases), 1)

if __name__ == "__main__":
    cases = [json.loads(l) for l in open("cases.jsonl")]
    print(sys.argv[1], "->", score(sys.argv[1], cases))

A few decisions baked in:

• Temperature 0, always. Evaluating at temperature 0.7 is judging a model on moods.
• Three runs per case catches flaky formatting. One model I tested passed a JSON case on run 1 and failed runs 2 and 3. That's not a model I trust with nightly jobs.
• One number out. If a model scores 87.5, it goes into a plain text file next to the previous scores. That file is now my real benchmark history.

What running it taught me

Newer isn't automatically better. The suite caught a hyped release that scored 9 points below my incumbent, even though everyone on my timeline was calling it the new default. On my tasks, it wasn't.

Quantization damage shows up in format, not intelligence. Q4 quants pass the logic checks fine, then blow output formatting at weird, inconsistent rates. If I'd only vibe-checked, I'd never have seen it — the prose looks equally smart either way.

Wins are task-shaped, not model-shaped. One challenger beat my incumbent by 15 points on extraction and lost by 8 on code edits. I didn't switch — I routed. That result is literally why extraction jobs in my setup now go to a different model.

My prompts drift too. I run the suite after big prompt changes, not just model changes. Twice this year a "clever prompt improvement" quietly regressed three cases. A vibe-check would never have caught that.

The switch rule

A challenger replaces my current model only if it wins by 10+ points overall AND doesn't drop more than 5 points in any single category. Below that, the migration isn't worth it — rewriting prompts, re-learning quirks, updating the docs I keep for myself.

The one time I ignored my own rule was that hyped release I mentioned. The timeline got me. I spent a weekend migrating prompts, ran the suite afterward out of curiosity, and the old model won by 9. I crawled back and re-deployed the old setup Sunday night. The suite forgave me. My Saturday didn't.

Wrapping up

If you run local models for anything that matters, don't vibe-check. Build a suite of YOUR tasks — 20 cases is enough to start — run new models against it at temperature 0, and let a number make the decision. It's 30 minutes of compute versus two weekends of regret.

If you've built something similar, I'm curious what made it into your cases. Drop your weirdest eval case in the comments — always looking to steal ideas for mine.

Comments

[Tae Kim]

We've got a multi-hop graph question as one of our trap cases. We give the model a tiny schema, five types and a handful of edges, then ask it to trace a two-hop path in plain English without any query syntax. Models that ace our extraction cases routinely bomb it because they're trying to match output patterns rather than actually holding state. And your Q4 format point is real, we'd get the right value wrapped in a code block and our parser would die just as hard as if the logic was completely wrong.