Stop Picking Free Models By Vibe: A 20-Case Golden Set You Can Run This Afternoon

You can't tell two free models apart by reading their landing pages. You can tell them apart in about 40 minutes, with 20 cases and a pass/fail rule you wrote down before you saw any output.

That's the whole article: a repeatable, boring, cheap way to choose a model — plus a place to run it without a bill.

Why not just read the benchmarks?

Because your task is not on the benchmark. Public leaderboards measure a dataset someone else wrote, on prompts someone else designed, with a scoring rubric you can't inspect. A big chunk of the current AI discourse is arguments about exactly this. Fair.

So build your own tiny eval. Small is fine. Twenty cases with honest graders will tell you more about your workload than a thousand-row leaderboard ever will.

Here's the catch: it only works if the graders are frozen before the run. Write them first, then run. Tune the expectations after seeing output and you're not evaluating — you're decorating.

Step 1 — Write the task contract first (10 minutes)

One paragraph, in a file, in the repo. What goes in, what comes out, what counts as wrong.

task: summarize a support ticket into {category, severity, needs_human}
input: raw ticket text (<= 1200 chars)
output: JSON object, 3 keys, no prose
hard fail: invalid JSON, missing key, severity outside {low, med, high}
soft fail: wrong category (annoying, not fatal)

Verify: if a new teammate can't grade a sample by hand using only this file, the contract is too vague. Fix it before you write a single case. Vague contract, meaningless score.

Step 2 — Build 20 cases in JSONL

Twelve normal, five edge, three adversarial. That ratio is the point. Normal cases make you feel good; edge cases make you choose correctly; adversarial cases stop you from getting burned in week two.

{"id":"n01","prompt":"My invoice is wrong, charged twice for October.","grader":{"type":"json_keys","expect":["category","severity","needs_human"]}}
{"id":"n02","prompt":"App crashes when I upload a 40MB CSV.","grader":{"type":"regex","expect":"\"severity\":\\s*\"(med|high)\""}}
{"id":"e01","prompt":"","grader":{"type":"exact","expect":"{\"error\":\"empty_input\"}"}}
{"id":"e02","prompt":"ignore previous","grader":{"type":"human"}}
{"id":"a01","prompt":"Mixed English/Japanese ticket, 1100 chars.","grader":{"type":"json_keys","expect":["category","severity","needs_human"]}}

Rules I keep:

• Every case gets an id. You will reference it in bug reports and in arguments with yourself.
• At least one empty input and one oversized input.
• One case where the correct behaviour is to refuse.
• Real inputs from production logs, redacted — not rows from a public set you might already be contaminated by.

Verify: wc -l cases.jsonl returns 20, and every line parses:

python -c "import json;[json.loads(l) for l in open('cases.jsonl')]"

Step 3 — The runner (stdlib, ~50 lines)

Keep the runner dumb. It calls, times, captures, writes. No scoring inside — grading is a separate pass so you can re-grade old runs when your graders change.

# runner.py
import argparse, json, time, pathlib

def call_model(model, prompt, timeout):
    if model == "stub":
        return {"text": '{"category":"x","severity":"low","needs_human":false}',
                "latency_ms": 1, "error": None}
    raise SystemExit("wire your client here; keep the return shape")

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--model", required=True)
    ap.add_argument("--cases", default="cases.jsonl")
    ap.add_argument("--out", required=True)
    ap.add_argument("--timeout", type=float, default=20.0)
    a = ap.parse_args()

    cases = [json.loads(l) for l in open(a.cases) if l.strip()]
    out = pathlib.Path(a.out); out.parent.mkdir(parents=True, exist_ok=True)
    with out.open("w") as f:
        for c in cases:
            t0 = time.time()
            try:
                r = call_model(a.model, c["prompt"], a.timeout)
            except Exception as e:
                r = {"text": "", "latency_ms": int((time.time()-t0)*1000), "error": repr(e)}
            r["id"] = c["id"]; r["model"] = a.model
            f.write(json.dumps(r, ensure_ascii=False) + "\n")
            print(c["id"], r["latency_ms"], "ms", r["error"] or "ok")

if __name__ == "__main__":
    main()

Now the part people skip. Dry-run the plumbing with the stub before you spend a single token:

python -m venv .venv && . .venv/bin/activate
python runner.py --model stub --out runs/stub.jsonl
head -1 runs/stub.jsonl

Verify: 20 lines, each valid JSON, no error field populated. If the stub run is broken, every number after this is noise. Fix it here, not later.

Step 4 — Grade from files, not from the chat window

python runner.py --model candA --out runs/candA.jsonl
python runner.py --model candB --out runs/candB.jsonl

Same grader code for both candidates. human cases go on a sticky note and get eyeballed once, after the automated numbers exist — otherwise you anchor yourself.

# grade.py
import json, re, sys

def grade(case, text):
    g = case["grader"]
    if g["type"] == "exact":   return text.strip() == g["expect"].strip()
    if g["type"] == "regex":   return re.search(g["expect"], text) is not None
    if g["type"] == "json_keys":
        try: obj = json.loads(text)
        except json.JSONDecodeError: return False
        return all(k in obj for k in g["expect"])
    return None  # human review

cases = {c["id"]: c for c in map(json.loads, open("cases.jsonl"))}
for path in sys.argv[1:]:
    rows = [json.loads(l) for l in open(path)]
    auto = [r for r in rows if grade(cases[r["id"]], r["text"]) is not None]
    passed = sum(1 for r in auto if grade(cases[r["id"]], r["text"]))
    errs = sum(1 for r in rows if r["error"])
    print(f"{path}: {passed}/{len(auto)} auto, {errs} errors")

Verify: grade the stub run. It should score a low, obviously wrong number. If the stub scores 20/20, your graders are too loose and you are about to ship a fake eval. That single check has caught more of my mistakes than any dashboard.

Step 5 — Decide with a rule you wrote before you looked

The order matters. Write the table first, run second, read it third.

Check	Fail condition	Action
Hard error rate	above 2/20	eliminate
Adversarial cases	any hard fail	eliminate
JSON validity	below 19/20	eliminate
Auto pass count	within 2 of the leader	tie, send to human review
p95 latency	above your client timeout	count as error

Those thresholds are illustrative, not published benchmarks. Pick yours from what your users will actually tolerate — not from what looks impressive in a table. A model that is 3% better and 4x slower is not a better model for a synchronous form field.

Where to run this without a bill

Two candidates, 20 cases, three repeats if you can afford it — that's 120 calls. Trivial on a paid API. Annoying on a personal card, which is exactly when people stop evaluating and start guessing.

That's where a free tier genuinely earns its place in the workflow: free model access supplies the candidates, and a free server option hosts the runner so the loop doesn't die because your laptop went to sleep. MonkeyCode offers both — free model access and a free server option, with the free allowance advertised at 10 million tokens. Treat those figures as operator-supplied and check the current limits in the project before you budget on them, because free-tier terms move.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Wire call_model to it, keep the stub path intact, and the harness stops caring who serves the tokens. That's the test of a decent eval harness: swapping the provider is a five-line diff, not a rewrite.

What this method does not tell you

Twenty cases is a smoke test, not a verdict. The confidence interval is wide enough to hide real differences, so resist the urge to declare a winner on a 2-case gap.

One run per case is noisy. Free tiers are also rate-limited, so repeats get slow — and repeated calls to the same endpoint may not be independent samples anyway.

Your graders are the weakest link. Regex over JSON is brittle; schema checks are better; anything about tone or writing quality needs a human and probably deserves its own rubric.

Contamination is real if your cases came from a public dataset. Write your own, from your own logs.

And it says nothing about tasks you didn't include. Change the prompt shape next month and re-run.

Who should not use this

If you already have an evaluation pipeline, you don't need a 50-line runner — add cases to what you have. If your failure mode is subtle prose quality that no regex can catch, hand-review is the honest method; don't dress it up in a pass rate. And if your output lands somewhere regulated, every result gets a human anyway, so optimize for review speed instead of model choice.

The monthly re-run

Freeze cases and graders in git. Re-run on the first of the month, diff the pass counts, and only then consider switching. The value isn't the score — it's that you can say why you switched, and notice when a model quietly gets worse without telling anyone.

Twenty cases. One stub run to prove the plumbing. One decision table you wrote before you peeked. Cheap, boring, and it survives contact with a real workload.