Catch LLM regressions before your users do — a tiny CI gate for LLM output

You ship an LLM feature. Weeks later you tweak a prompt, or the provider rolls the model forward under you, and something breaks — not loudly, not in a stack trace, just three answers that used to be right and now aren't. Nobody notices until a user does.

I evaluate LLM output for a living, and this is the failure mode I see most. The fix isn't a platform or a dashboard. It's treating your eval like a test: a file you run on every change, that fails the build when it should. Here's how I do it, with runnable, dependency-free code.

Make your eval a file you run

Keep your evaluation data as plain JSON so anyone on the team can edit it without touching code. Each item is a question, the known-good answer, and — the important part — how to judge it:

[
  {"id": "q1", "question": "What year was the first moon landing?", "answer": "1969", "match": "contains"},
  {"id": "q2", "question": "What is pi to two decimals?", "answer": "3.14", "match": "numeric", "tol": 0.01},
  {"id": "q3", "question": "Capital of France?", "answer": "Paris", "match": "contains"}
]

Your model's answers are a simple {id: answer} object:

{
  "q1": "The first moon landing was in 1969.",
  "q2": "Pi is about 3.14159",
  "q3": "The capital of France is Paris."
}

Now score them. This is the open-source llm-eval-harness — one file, standard library only:

python llm_eval.py gold.json answers.json

accuracy: 100%  (3/3)

The match rule is what makes this trustworthy. contains passes "The first moon landing was in 1969." against gold 1969 — an exact-match-only script would mark that wrong and send you chasing a non-bug. numeric reads the first number in each and compares within a tolerance, so 3.14159 matches 3.14. The scorer normalizes first (lowercase, trim, collapse whitespace, drop trailing punctuation), which kills a whole class of false negatives.

And it exits non-zero the moment anything fails, so it drops straight into CI:

# .github/workflows/eval.yml
- name: LLM eval gate
  run: python llm_eval.py eval/gold.json eval/answers.json

That alone catches "did this change break a known-good answer?" on every PR.

Accuracy is the wrong number to watch

Here's the trap: a change can raise your average score and still break the three questions your biggest customer depends on. The number that actually matters on a change isn't the score — it's which items that used to pass now fail.

So compare two runs and diff them. This is one of the pieces the LLM-Eval Starter Kit adds on top of the free harness:

from llm_eval_kit import run, diff_runs, load_gold, load_answers

gold   = load_gold("eval/gold.json")
before = run(gold, load_answers("eval/answers_main.json"), name="main")
after  = run(gold, load_answers("eval/answers_pr.json"),   name="pr")

diff = diff_runs(before, after)
print(f"accuracy {diff['before_accuracy']:.0%} -> {diff['after_accuracy']:.0%}")

if diff["regressed"]:
    print("REGRESSIONS:", ", ".join(diff["regressions"]))
    raise SystemExit(1)   # fail the build

accuracy 100% -> 67%
REGRESSIONS: q3, q5

diff_runs gives you the regressed ids (passed before, fail now), the fixed ids, and the score delta. Gate CI on regressed and a prompt change can't quietly ship a regression again — the PR goes red with the exact ids that broke.

Open-ended answers: an honest LLM-as-judge

String rules run out fast. "Is this summary faithful to the source?" "Did it follow the format?" "Is the tone right?" You can't contains-match those. The stable approach is an explicit rubric graded by a model at temperature 0 — not a vibe check.

The kit keeps that honest: the rubric is data (criteria + a numeric scale + a pass threshold), and the judge is any ask(prompt) -> str callable, so you bring your own model and key. There's also a deterministic offline judge so your test suite never needs an API key:

from llm_eval_kit import score, StubJudge, make_openai_judge

gold = [{
    "id": "sum1",
    "match": "rubric",
    "question": "Summarize the release note in one sentence.",
    "context": "v2.3 adds retry-with-backoff to the uploader and fixes a memory leak.",
    "rubric": {
        "criteria": [
            "Faithful to the source (no invented features)",
            "One sentence, plain language",
        ],
        "scale": [1, 5],
        "threshold": 4,
    },
}]
answers = {"sum1": "v2.3 adds automatic upload retries and fixes a memory leak."}

# Offline + deterministic — perfect for CI/tests, no key:
acc, failures = score(gold, answers, judge=StubJudge(must_include=["retry", "memory leak"]))

# Real judge — bring your own key (OpenAI / OpenRouter / a local server):
judge = make_openai_judge(model="gpt-4o-mini", temperature=0)  # reads OPENAI_API_KEY
acc, failures = score(gold, answers, judge=judge)

One rule I never skip: spot-check the judge against a handful of human labels. An LLM judge is a measuring instrument, and an uncalibrated instrument lies confidently. Grade ~20 answers yourself, compare, and only then trust it at scale.

When one attempt isn't the whole story

For sampled or agentic systems you often care about "did any of k attempts pass?" rather than a single greedy answer. That's pass@k:

from llm_eval_kit import score_pass_at_k
rate, details = score_pass_at_k(gold, samples_by_id, k=5)   # samples_by_id: {id: [answer, ...]}

Same gold set, same match rules — you just feed it multiple samples per id.

The takeaway

You don't need a platform to stop shipping LLM regressions. You need:

1. Your eval as a versioned JSON file with a match rule per item.
2. A scorer that exits non-zero so CI can gate on it.
3. A regression diff so you watch which items flip, not just the average.
4. For open-ended answers, a rubric + a temperature-0 judge you've spot-checked.

The scoring core is open-source and free (MIT): llm-eval-harness. If you want the rubric LLM-as-judge, pass@k, regression diffs, and shareable HTML/Markdown/JSON reports ready to run — still zero dependencies — that's the LLM-Eval Starter Kit, launch price $19 for the first two weeks with code LAUNCH (then $39). There's also a short book on the whole method, Practical LLM Evaluation.

How are you catching regressions when a model updates under you? I'd genuinely like to hear it.