Catch LLM regressions in CI before your users do

Litmus — Wed, 11 Nov 2026 11:46:14 +0000

You change one line of a prompt. The demo still works. You merge. Two days later a user screenshots a nonsense answer and asks what happened.

What happened: nothing caught the regression, because nothing ran the prompt against real cases before merge.

Unit tests do not cover this. A prompt is not a function with one right answer - it has a distribution of acceptable answers, and the only cheap way to pin that distribution down is a small golden dataset plus a runner that scores every change against it.

The 20-minute setup

Write 10-30 cases from real traffic. Each case: the input, plus what a good answer must contain (or must not contain). JSONL is fine:

{"input": "refund request, order late by 3 weeks", "must_include": ["refund", "apolog"], "must_not_include": ["as an AI"]}
{"input": "ask for discount on enterprise plan", "must_include": ["annual"], "max_words": 120}

Run your app against the cases on every PR. Score with simple checks first (must_include, must_not_include, length, JSON validity). Add model-graded checks later if you need them - most teams never do.
Fail the build when the pass rate drops below the last known-good run.

That is the whole pattern. The tooling should stay out of the way: a YAML file for config, a JSONL file for cases, one command in CI.

What this looks like with Litmus

I maintain Litmus, a zero-dependency harness for exactly this (single Python file, Python 3.8+, standard library only, MIT license):

{
  "name": "support-bot-suite",
  "provider": "openai",
  "cases": [
    {
      "name": "refund-request",
      "prompt": "Reply to this customer: {{message}}",
      "vars": {"message": "my order is 3 weeks late, I want a refund"},
      "checks": {
        "contains": ["refund"],
        "not_contains": ["as an AI"],
        "max_latency_ms": 8000
      }
    }
  ]
}

$ python3 litmus.py run tests.json --baseline .litmus-baseline.json
Litmus 1.0.0 - suite 'support-bot-suite' (provider: openai)
  [FAIL] refund-request
        failed check: contains 'refund'
  [PASS] discount-question
1/2 cases passed

REGRESSIONS vs baseline:
  - refund-request: PASS -> FAIL (contains 'refund')
# exit code 1 - the merge is blocked

In CI the non-zero exit blocks the merge, and JUnit XML output feeds your CI dashboard. There is a deterministic offline mock provider, so you can test your wiring without spending tokens. The full setup guide is in the free starter pack (link on my profile).

The mistakes everyone makes first

Testing with vibes. "I tried it three times" is not a signal. Thirty saved cases beat three fresh tries because they are the same thirty every time.
Golden datasets that never grow. Add a case every time a user reports a bad answer. The dataset is the memory of every failure you have already paid for.
Model-graded everything on day one. String and length checks catch the majority of regressions and cost nothing. Start there.

Prompts change weekly now. The teams that treat them like code - versioned, reviewed, tested - ship faster precisely because they break less.

If you want the runner already built: Litmus Starter is a free sample pack (config, sample dataset, full CLI) and the complete version is $12 with the CI templates included. Links on my profile.

I got tired of breaking my LLM app with "one small prompt change", so I built a 450-line eval harness

Litmus — Wed, 11 Nov 2026 09:45:27 +0000

Every LLM developer I know has the same workflow: tweak the prompt, eyeball three outputs, ship it. Then a user reports that the refund flow has been answering wrong for two days.

The fix is evals, but the popular options all asked for more than I wanted to give: a SaaS account, a framework dependency, a YAML religion. I wanted pytest energy: one file, run it, exit code.

So I wrote Litmus: a single Python file, stdlib only, that evals an LLM app against a JSONL dataset and gates regressions in CI.

What a case looks like

{"id": "refund-q", "input": "How do I get a refund?",
 "expect": {"contains": ["30 days"], "regex": "refund"}}

That's it. Input, plus what a good answer must contain. Run it:

python3 litmus.py run --dataset cases.jsonl --endpoint https://api.openai.com/v1 --model gpt-4o-mini

  LITMUS REPORT  ============================================
  pass rate   97.4%  (76/78)
  failed      1
  errored     1
  latency     mean 0.84s  p50 0.71s  p95 1.9s
  ============================================================
  [FAIL] refund-question (0.91s)
        contains: missing: 30 days

The part that actually saves you: compare

Run Litmus against your production prompt, save the JSON report as the baseline, and check every change against it:

python3 litmus.py compare baseline.json current.json --threshold 0.02

It lists the exact cases that used to pass and now fail, and exits 1 - so GitHub Actions blocks the merge. The workflow file is 15 lines, included in the guide.

Design rules I stuck to

One file, no dependencies. pip install is where side tools go to die. Litmus is Python 3.8+ stdlib only. You can read all of it before running it.
No lock-in. Your dataset is JSONL in your repo. If you stop using Litmus tomorrow, your evals still mean something.
Any app. Pipe mode tests anything that reads a case on stdin and prints an answer - Python, Node, a shell script. Endpoint mode talks to any OpenAI-compatible server, including local Ollama models, which makes the whole loop free.
Cheap scorers first, judge last. contains and regex are deterministic and free. LLM-as-judge is there for tone and helpfulness, with your rubric - and judge failures are reported as errors, never silently passed.

What I learned building it

Mine real failures. Every user-reported bad answer becomes a case the same day. Ten real failures beat a hundred synthetic cases.
Small cases. One expectation per case, so the failure list reads like a bug report.
Latency is a regression too. A prompt change that doubled p95 response time is a production incident you haven't had yet.

Litmus is $12, one time: https://litmustools.gumroad.com/l/litmus. You get the tool, the field guide (eval-set patterns, LLM-judge rules, the CI workflow), and a runnable example. If it saves one bad deploy, it paid for itself.

DEV Community: Litmus