I Built a Personal Regression Suite for LLMs — Here's the Design, Not Just the Code

Every time a new open model trends, the same ritual plays out: impressive demos on Monday, horror stories by Wednesday, and by Friday nobody remembers what they concluded. I used to ride that wave. What pulled me off it wasn't a better opinion — it was realizing I already solved this exact problem years ago, just for code instead of models. We don't upgrade a database because a benchmark tweet said so. We run our own regression suite first.

So I built one for language models. It's opinionated, deliberately small, and it costs me nothing to execute. This post is about the design decisions — what to test, what to refuse to test, and how to score it — because those choices matter far more than the plumbing, which I'll show anyway.

The mistake most homegrown evals make

Before the design, three failure modes I see in every "I tested the new model" post:

1. Testing capabilities instead of workflows. Asking a model to "write a function that reverses a linked list" measures nothing you do at work. What you actually do is hand it a messy context and expect a usable artifact back.
2. Grading by reading. The moment you eyeball two outputs and pick a favorite, you've reintroduced the same bias you were trying to escape. Scoring has to be mechanical or it's theater.
3. No incumbent control. Comparing the new model against your memory of the old one is meaningless. Both must answer the same frozen prompts in the same session.

My suite is built to make all three mistakes impossible to commit accidentally.

What goes in the suite (and what doesn't)

I keep fifteen prompts, frozen in version control, drawn from work I've actually done. They split into three buckets, each engineered so that grading requires zero taste:

Bucket 1 — Executable truth (5 prompts). Tasks where the answer can be verified by running something. Examples from my own history: a prompt containing a broken SQL query and its error message, where the fix is graded by executing it against a scratch SQLite database; a prompt with a misconfigured GitHub Actions YAML, graded by whether a linter accepts the patch; a prompt asking for a regex that must match a fixed list of strings and reject another fixed list. The model's output either passes the harness or it doesn't. I never read these answers.

Bucket 2 — Constraint compliance (5 prompts). This is the bucket nobody demos, and it's where models silently rot. Each prompt demands an awkward structural contract: "output only a unified diff," "reply with exactly three bullet points, each under twelve words," "emit JSON matching this schema with no prose." Grading is a parser, not an opinion. Format drift is the failure mode that breaks my downstream scripts in production, so a single miss here carries real weight.

Bucket 3 — Refusal and honesty (5 prompts). Prompts built around traps from my past incidents: a question whose premise is wrong ("why does this Django query return duplicates" when the ORM call shown can't produce them), a request to modify a function in a way that would break an invariant documented in the context, an underspecified ticket where the correct move is to ask a clarifying question. Grading here is a short binary rubric I wrote once, before ever seeing model output — writing the rubric after seeing answers is how you rationalize.

What I deliberately exclude: creative writing, general knowledge, summarization quality, and anything I'd score with an adjective. If I can't grade it with a script or a one-line rubric, it doesn't belong in a suite meant to drive a yes/no decision.

The harness, kept boring on purpose

The runner exists to serve the design, so it's as generic as I can make it. It speaks plain HTTP to any endpoint that implements the chat completions shape — no vendor SDK, because tooling lock-in is how comparisons quietly stop being reproducible.

#!/usr/bin/env python3
"""replay.py — run a frozen prompt suite against any chat-completions endpoint.

Usage:
    MC_BASE=https://host/v1 MC_KEY=token MC_MODEL=some/model python replay.py

Writes results_<model>.jsonl. Grading lives in separate check scripts,
not here — the runner's only job is faithful recording.
"""
import json, os, sys, time, urllib.request

BASE  = os.environ["MC_BASE"].rstrip("/")
KEY   = os.environ.get("MC_KEY", "none")
MODEL = os.environ["MC_MODEL"]

def complete(system, user):
    t0 = time.monotonic()
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "system", "content": system},
                     {"role": "user", "content": user}],
        "temperature": 0,
        "seed": 42,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    with urllib.request.urlopen(req, timeout=240) as r:
        out = json.loads(r.read())
    return out["choices"][0]["message"]["content"], round(time.monotonic() - t0, 2)

suite = [json.loads(line) for line in open("suite.jsonl") if line.strip()]
out_path = f"results_{MODEL.replace('/', '_')}.jsonl"
with open(out_path, "w") as f:
    for i, item in enumerate(suite, 1):
        text, secs = complete(item["system"], item["prompt"])
        f.write(json.dumps({"task": item["task"], "bucket": item["bucket"],
                            "output": text, "latency": secs}) + "\n")
        print(f"[{i}/{len(suite)}] {item['task']} — {secs}s", file=sys.stderr)
print(f"wrote {out_path}")

The suite itself is newline-delimited JSON so individual tasks diff cleanly in git:

{"task": "sql-fix-01", "bucket": "executable", "system": "Return only the corrected SQL query, no explanation.", "prompt": "Schema: orders(id, user_id, total, created_at). Error: 'column o.name does not exist'. Query: SELECT o.name, SUM(total) FROM orders o GROUP BY o.name;"}
{"task": "fmt-diff-02", "bucket": "format", "system": "You emit unified diffs and nothing else.", "prompt": "Change the retry constant from 3 to 5 in this file: <paste>"}
{"task": "trap-premise-01", "bucket": "honesty", "system": "You are reviewing a bug report.", "prompt": "This query returns duplicate rows — explain why: SELECT DISTINCT email FROM users;"}

One invocation per model, one output file per model, and the grading scripts — a SQLite executor, a JSON schema validator, a word-count checker — consume those files without knowing which model produced them. Blind scoring matters more than people expect.

Keeping the budget at zero

The reason personal eval suites die isn't engineering effort — it's that nobody wants to open a billing page to settle an argument with a trending post. So the whole loop runs on free infrastructure, through two paths that both work with the script above:

• Hosted free model access — someone else runs the weights; you point MC_BASE at their endpoint and you're done. Fastest route for a model you just want to check.
• A free server you control — for open-weight releases, self-hosting the exact quantized build means nothing shifts between runs: no silent provider-side updates, no context window surprises, no mid-experiment drift.

I used MonkeyCode for both paths, since it currently offers free model access alongside a free server option, which let me run the candidate and my incumbent side by side without provisioning anything paid.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One property worth insisting on whatever you use: the harness shouldn't know which path served a response. Portability is what keeps the suite honest across providers, and it's what lets you re-run the identical comparison months later when the next release drops.

Scoring into a decision, not a vibe

After both runs I fill in a fixed table. The order matters — I stop at the first failure.

Gate	Rule	Why it's ordered this way
1. Regressions	Any task my incumbent passes that the candidate fails → reject immediately	A new model must not lose ground I depend on today
2. Executable bucket	Candidate must pass at least one more than the incumbent	This is the only bucket measuring real work
3. Format bucket	Zero misses tolerated	One structural break poisons every pipeline downstream
4. Honesty bucket	No new sycophancy failures vs. incumbent	Traps caught late are incidents, not quirks
5. Latency	Median under ~1.5x incumbent at temperature 0	Only checked after correctness — speed never buys back wrongness

"Better on average" is not a criterion. A model that shines on four tasks and fumbles the one that mirrors my ugliest recurring bug stays on the shelf, and I re-test at the next point release instead of negotiating with myself.

Honest limitations

• Fifteen prompts is a compass, not a study. It answers "is this clearly better for my work" and nothing larger. Small global quality shifts are invisible at this scale, and that's an accepted trade for something I'll actually run every release cycle.
• Temperature zero isn't determinism. Provider batching, quantization, and hardware variance all leak through. Any gate result close enough to change my mind gets re-run three times on different days.
• Free tiers move. Free model access and free server plans come with provider-set limits and can be revised or withdrawn. Treat them as ephemeral: never wire them into CI without a fallback, and confirm current terms yourself before relying on them.
• Self-hosted ≠ the full model. A 4-bit quant answers questions about that quant. Record the exact build you ran or your future self will compare against a ghost.
• Skip this approach if your prompts contain data that can't leave your machines (self-host only, and check the model license), if your output feeds safety-critical systems, or if you'll never re-curate the suite. A stale suite is worse than none — it emits confident verdicts about work you no longer do.

The point

The open-model ecosystem's best feature isn't any single release — it's that falsification got cheap. A viral claim about a model used to be something you either believed or ignored. Now it's something you can replay against your own bug history in the time it takes to drink a coffee, at zero cost, with tooling you fully control.

If you want to stand this up without spending anything, MonkeyCode's free models and free server option are one practical way to get both endpoints; but the suite above speaks plain chat completions, so it works against anything you point it at — which is exactly the property worth keeping.

The harness, suite format, and scoring gates here were written for this article. Fill the suite with tasks from your own project history — mine would only measure my work, not yours — and verify any provider's current free-tier terms before building on them.