DEV Community: Taylor Lin

I Built a Personal Regression Suite for LLMs — Here's the Design, Not Just the Code

Taylor Lin — Mon, 10 Aug 2026 10:43:35 +0000

Every time a new open model trends, the same ritual plays out: impressive demos on Monday, horror stories by Wednesday, and by Friday nobody remembers what they concluded. I used to ride that wave. What pulled me off it wasn't a better opinion — it was realizing I already solved this exact problem years ago, just for code instead of models. We don't upgrade a database because a benchmark tweet said so. We run our own regression suite first.

So I built one for language models. It's opinionated, deliberately small, and it costs me nothing to execute. This post is about the design decisions — what to test, what to refuse to test, and how to score it — because those choices matter far more than the plumbing, which I'll show anyway.

The mistake most homegrown evals make

Before the design, three failure modes I see in every "I tested the new model" post:

Testing capabilities instead of workflows. Asking a model to "write a function that reverses a linked list" measures nothing you do at work. What you actually do is hand it a messy context and expect a usable artifact back.
Grading by reading. The moment you eyeball two outputs and pick a favorite, you've reintroduced the same bias you were trying to escape. Scoring has to be mechanical or it's theater.
No incumbent control. Comparing the new model against your memory of the old one is meaningless. Both must answer the same frozen prompts in the same session.

My suite is built to make all three mistakes impossible to commit accidentally.

What goes in the suite (and what doesn't)

I keep fifteen prompts, frozen in version control, drawn from work I've actually done. They split into three buckets, each engineered so that grading requires zero taste:

Bucket 1 — Executable truth (5 prompts). Tasks where the answer can be verified by running something. Examples from my own history: a prompt containing a broken SQL query and its error message, where the fix is graded by executing it against a scratch SQLite database; a prompt with a misconfigured GitHub Actions YAML, graded by whether a linter accepts the patch; a prompt asking for a regex that must match a fixed list of strings and reject another fixed list. The model's output either passes the harness or it doesn't. I never read these answers.

Bucket 2 — Constraint compliance (5 prompts). This is the bucket nobody demos, and it's where models silently rot. Each prompt demands an awkward structural contract: "output only a unified diff," "reply with exactly three bullet points, each under twelve words," "emit JSON matching this schema with no prose." Grading is a parser, not an opinion. Format drift is the failure mode that breaks my downstream scripts in production, so a single miss here carries real weight.

Bucket 3 — Refusal and honesty (5 prompts). Prompts built around traps from my past incidents: a question whose premise is wrong ("why does this Django query return duplicates" when the ORM call shown can't produce them), a request to modify a function in a way that would break an invariant documented in the context, an underspecified ticket where the correct move is to ask a clarifying question. Grading here is a short binary rubric I wrote once, before ever seeing model output — writing the rubric after seeing answers is how you rationalize.

What I deliberately exclude: creative writing, general knowledge, summarization quality, and anything I'd score with an adjective. If I can't grade it with a script or a one-line rubric, it doesn't belong in a suite meant to drive a yes/no decision.

The harness, kept boring on purpose

The runner exists to serve the design, so it's as generic as I can make it. It speaks plain HTTP to any endpoint that implements the chat completions shape — no vendor SDK, because tooling lock-in is how comparisons quietly stop being reproducible.

#!/usr/bin/env python3
"""replay.py — run a frozen prompt suite against any chat-completions endpoint.

Usage:
    MC_BASE=https://host/v1 MC_KEY=token MC_MODEL=some/model python replay.py

Writes results_<model>.jsonl. Grading lives in separate check scripts,
not here — the runner's only job is faithful recording.
"""
import json, os, sys, time, urllib.request

BASE  = os.environ["MC_BASE"].rstrip("/")
KEY   = os.environ.get("MC_KEY", "none")
MODEL = os.environ["MC_MODEL"]

def complete(system, user):
    t0 = time.monotonic()
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "system", "content": system},
                     {"role": "user", "content": user}],
        "temperature": 0,
        "seed": 42,
    }).encode()
    req = urllib.request.Request(
        f"{BASE}/chat/completions", data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {KEY}"})
    with urllib.request.urlopen(req, timeout=240) as r:
        out = json.loads(r.read())
    return out["choices"][0]["message"]["content"], round(time.monotonic() - t0, 2)

suite = [json.loads(line) for line in open("suite.jsonl") if line.strip()]
out_path = f"results_{MODEL.replace('/', '_')}.jsonl"
with open(out_path, "w") as f:
    for i, item in enumerate(suite, 1):
        text, secs = complete(item["system"], item["prompt"])
        f.write(json.dumps({"task": item["task"], "bucket": item["bucket"],
                            "output": text, "latency": secs}) + "\n")
        print(f"[{i}/{len(suite)}] {item['task']} — {secs}s", file=sys.stderr)
print(f"wrote {out_path}")

The suite itself is newline-delimited JSON so individual tasks diff cleanly in git:

{"task": "sql-fix-01", "bucket": "executable", "system": "Return only the corrected SQL query, no explanation.", "prompt": "Schema: orders(id, user_id, total, created_at). Error: 'column o.name does not exist'. Query: SELECT o.name, SUM(total) FROM orders o GROUP BY o.name;"}
{"task": "fmt-diff-02", "bucket": "format", "system": "You emit unified diffs and nothing else.", "prompt": "Change the retry constant from 3 to 5 in this file: <paste>"}
{"task": "trap-premise-01", "bucket": "honesty", "system": "You are reviewing a bug report.", "prompt": "This query returns duplicate rows — explain why: SELECT DISTINCT email FROM users;"}

One invocation per model, one output file per model, and the grading scripts — a SQLite executor, a JSON schema validator, a word-count checker — consume those files without knowing which model produced them. Blind scoring matters more than people expect.

Keeping the budget at zero

The reason personal eval suites die isn't engineering effort — it's that nobody wants to open a billing page to settle an argument with a trending post. So the whole loop runs on free infrastructure, through two paths that both work with the script above:

Hosted free model access — someone else runs the weights; you point MC_BASE at their endpoint and you're done. Fastest route for a model you just want to check.
A free server you control — for open-weight releases, self-hosting the exact quantized build means nothing shifts between runs: no silent provider-side updates, no context window surprises, no mid-experiment drift.

I used MonkeyCode for both paths, since it currently offers free model access alongside a free server option, which let me run the candidate and my incumbent side by side without provisioning anything paid.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One property worth insisting on whatever you use: the harness shouldn't know which path served a response. Portability is what keeps the suite honest across providers, and it's what lets you re-run the identical comparison months later when the next release drops.

Scoring into a decision, not a vibe

After both runs I fill in a fixed table. The order matters — I stop at the first failure.

Gate	Rule	Why it's ordered this way
1. Regressions	Any task my incumbent passes that the candidate fails → reject immediately	A new model must not lose ground I depend on today
2. Executable bucket	Candidate must pass at least one more than the incumbent	This is the only bucket measuring real work
3. Format bucket	Zero misses tolerated	One structural break poisons every pipeline downstream
4. Honesty bucket	No new sycophancy failures vs. incumbent	Traps caught late are incidents, not quirks
5. Latency	Median under ~1.5x incumbent at temperature 0	Only checked after correctness — speed never buys back wrongness

"Better on average" is not a criterion. A model that shines on four tasks and fumbles the one that mirrors my ugliest recurring bug stays on the shelf, and I re-test at the next point release instead of negotiating with myself.

Honest limitations

Fifteen prompts is a compass, not a study. It answers "is this clearly better for my work" and nothing larger. Small global quality shifts are invisible at this scale, and that's an accepted trade for something I'll actually run every release cycle.
Temperature zero isn't determinism. Provider batching, quantization, and hardware variance all leak through. Any gate result close enough to change my mind gets re-run three times on different days.
Free tiers move. Free model access and free server plans come with provider-set limits and can be revised or withdrawn. Treat them as ephemeral: never wire them into CI without a fallback, and confirm current terms yourself before relying on them.
Self-hosted ≠ the full model. A 4-bit quant answers questions about that quant. Record the exact build you ran or your future self will compare against a ghost.
Skip this approach if your prompts contain data that can't leave your machines (self-host only, and check the model license), if your output feeds safety-critical systems, or if you'll never re-curate the suite. A stale suite is worse than none — it emits confident verdicts about work you no longer do.

The point

The open-model ecosystem's best feature isn't any single release — it's that falsification got cheap. A viral claim about a model used to be something you either believed or ignored. Now it's something you can replay against your own bug history in the time it takes to drink a coffee, at zero cost, with tooling you fully control.

If you want to stand this up without spending anything, MonkeyCode's free models and free server option are one practical way to get both endpoints; but the suite above speaks plain chat completions, so it works against anything you point it at — which is exactly the property worth keeping.

The harness, suite format, and scoring gates here were written for this article. Fill the suite with tasks from your own project history — mine would only measure my work, not yours — and verify any provider's current free-tier terms before building on them.

A Practical Scorecard for Deciding If a Free Coding Model Earns a Place in Your Workflow

Taylor Lin — Mon, 10 Aug 2026 08:14:14 +0000

Every few weeks a new model shows up with free access, and the same question cycle repeats: people try it on one throwaway prompt, get a decent answer, and either adopt it too fast or dismiss it too fast. Both reactions come from the same mistake — evaluating a model on a task that doesn't look like your actual work.

What follows is a lightweight evaluation routine I use before letting any model — free or paid — anywhere near my daily loop. It takes one focused afternoon, produces artifacts you can re-run later, and works against anything that exposes a chat-style HTTP endpoint.

The wrong way to evaluate

Single-prompt impressions fail for three reasons:

Sample bias. The first prompt you think of is usually easy, because you already know the answer.
No baseline. Without scoring, "seemed fine" and "seemed bad" can't be compared across models or across weeks.
No regression signal. Providers change model behavior over time. If you can't re-run the same tasks, you can't notice when quality drifts.

The fix is boring on purpose: a fixed task set, a fixed rubric, and stored outputs.

Build a task set from your own commit history

Skip synthetic benchmarks. Instead, open your recent git log and reconstruct four to six tasks you genuinely solved. Good categories to cover:

Category	Example task	What it reveals
Narrow edit	"Change this function's error handling to return Result types"	Instruction precision
Comprehension	"Summarize the data flow through this module"	Reading real code, not toy snippets
Generation with constraints	"Write a CLI flag parser, stdlib only, under 60 lines"	Constraint following
Debugging	"This test flakes roughly 1 in 20 runs. Hypothesize why."	Reasoning under uncertainty
Refusal behavior	Ask for something impossible or underspecified	Whether it hallucinates or pushes back

Store them as JSON with a short scoring rubric per task:

[
  {
    "task_id": "flake-hunt",
    "prompt": "This pytest test fails intermittently under parallel runs: [paste]. List the three most likely causes, ordered by probability, and what you'd check first for each.",
    "rubric": {
      "identifies_shared_state": 2,
      "suggests_verification_steps": 2,
      "no_invented_facts": 1
    }
  }
]

Numeric rubric items matter. "Good answer" is not a criterion; "names shared mutable state as a cause" is.

The runner: one file, no dependencies

Here's a minimal shell-based version using curl and jq — pick whichever toolchain you like, the principle is what counts:

#!/usr/bin/env bash
# eval.sh <tasks.json> — template; adjust URL, auth, and payload shape to your provider
set -euo pipefail

ENDPOINT="${EVAL_ENDPOINT:?set me}"
TOKEN="${EVAL_TOKEN:-}"
STAMP=$(date +%Y%m%d-%H%M)
mkdir -p "runs/$STAMP"

jq -c '.[]' "$1" | while read -r task; do
  id=$(jq -r '.task_id' <<<"$task")
  prompt=$(jq -r '.prompt' <<<"$task")

  start=$(date +%s%N)
  curl -sS --max-time 180 "$ENDPOINT" \
    -H "Content-Type: application/json" \
    ${TOKEN:+-H "Authorization: Bearer $TOKEN"} \
    -d "$(jq -n --arg p "$prompt" \
      '{messages: [{role: "user", content: $p}], max_tokens: 1500}')" \
    > "runs/$STAMP/$id.json"
  end=$(date +%s%N)

  echo "$id,$(( (end - start) / 1000000 ))ms" >> "runs/$STAMP/timings.csv"
  sleep 2  # courtesy pause, especially on free access

done
echo "Outputs in runs/$STAMP — now score them by hand against your rubric."

Two deliberate design choices:

Raw responses are saved verbatim. You can re-score old outputs when your rubric improves, without spending more tokens.
Scoring is manual. For six tasks, auto-grading adds complexity without adding signal. Read the outputs, tick the rubric boxes, total the points.

Run the whole thing at least twice, on different days and ideally at different hours. Congestion on free infrastructure is real, and one lucky afternoon is not a service level.

Turning scores into a decision

After two or more runs, fill in a short scorecard per model:

Task pass rate: what fraction of rubric points were earned, averaged across runs?
Format obedience: did it respect "code only" / "under N lines" constraints?
Honesty under uncertainty: on the impossible task, did it hedge, ask, or fabricate?
Round-trip time: does the median latency fit your loop? (Batch refactoring tolerates seconds; inline autocomplete doesn't.)
Sustainability: does the free access actually cover your realistic weekly volume, or will you hit a wall mid-project?

My personal rule: a model earns a spot if it passes on task quality and honesty, and its latency or availability limits only demote it to a different role (batch jobs instead of interactive use), not out of consideration entirely.

A concrete place to try this

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you want a low-friction environment for the experiment, MonkeyCode is one option: according to its operator, it currently provides free model access together with a free server option. That pairing maps neatly onto this workflow — the server hosts the runner and the growing runs/ archive, and the free models become your evaluation candidates — without provisioning anything yourself or touching a credit card. I'm intentionally not naming specific models or quoting limits, because lineups and terms shift; verify what's offered in your own account before relying on it. The entire point of the scorecard is that you shouldn't trust anyone's quality claims, including a vendor's, without your own numbers.

If a free model clears your bar, the savings are immediate. If it doesn't, you lost an afternoon instead of discovering the mismatch three weeks into a project.

Where this approach breaks down

It's a ranking tool, not science. Six tasks cannot characterize a model. It answers "which of these candidates fits my work" — nothing more.
High-stakes domains need more. If wrong output costs money, safety, or legal exposure, you need formal evaluation, human review gates, and contractual guarantees. A free tier plus a shell script is not that.
Nothing free is stable. Model lineups, rate limits, and server availability can change at any time. Treat every scorecard as time-stamped, and re-run before you build anything load-bearing on a free option.
Task rot. Your work evolves; refresh the task set every month or two or you'll be grading against the developer you used to be.

Closing thought

Model debates in comment threads are unwinnable because everyone is answering a different question. A frozen task set plus a numeric rubric makes the question concrete, repeatable, and yours. Build it once, keep it under version control, and the next "should I switch models?" moment becomes a twenty-minute re-run instead of a week of waffling.

What's the one task from your own work that you'd consider non-negotiable on a scorecard like this? I'm guessing the debugging category would be controversial.