The bug you cannot see
Here is how prompt work usually goes. You write a prompt. You paste in a couple of inputs. The outputs look fine. You ship it. Two weeks later someone asks you to also handle a new edge case, so you add a sentence to the prompt. The new case works. You ship again.
What you did not notice is that your new sentence quietly broke three of the old cases. Nobody caught it, because nobody re-ran the old cases. There was no old case to re-run. It lived in your head and your scrollback, and both are gone now.
This is the whole problem with tuning by feel. It does not scale past a handful of examples, and it regresses in silence. A code change that breaks something usually throws. A prompt change that breaks something just returns slightly worse text, and slightly worse text passes the eye test on a Tuesday afternoon when you are tired.
The fix is boring and it works. Write the cases down. Turn them into assertions. Run them on every change. Below is the smallest version of that I have found useful, and it is framework agnostic, so it does not matter what model or SDK you call.
A minimal eval set is smaller than you think
You do not need a benchmark. You need a list of real inputs paired with things that must be true about the output. Ten cases is plenty to start. Pull them from actual usage if you have any, because invented inputs miss the weird stuff real users type.
Store them as plain data. A JSON or YAML file is fine. Each entry is an input and a set of expected properties, not an expected exact string. Something like this shape:
[
{
"name": "refund question",
"input": "how long do refunds take",
"expect": { "mentions": ["business days"], "max_words": 80 }
},
{
"name": "off topic",
"input": "what is the capital of France",
"expect": { "refuses": true }
}
]
The expect block is the whole trick. You are not checking that the model wrote one specific paragraph. You are checking that the output has the properties you care about. Does it mention the right fact. Is it short enough. Does it decline when it should. Is the JSON parseable. Those are the things that break in production, and those are the things you can assert on without a brittle string match.
If your prompt returns structured output, this gets even easier. Parse the JSON. Assert on fields. A response that fails to parse is a hard failure, full stop, and that alone catches a surprising number of regressions.
Assert on properties, not exact text
The runner is about forty lines. Loop over cases, call your model, check each property, collect failures. In pseudocode:
fails = []
for case in load_cases():
out = call_model(case.input)
for key, want in case.expect.items():
if not check(key, want, out):
fails.append((case.name, key))
report(fails)
exit(1 if fails else 0)
check is a little dispatch table. mentions does a substring or regex scan. max_words counts. refuses looks for the shape of a refusal (an apology, a "I do not have that", a handoff phrase) rather than exact wording. valid_json tries to parse. Add matchers as you need them. Keep each one dumb and readable.
Two honest cautions. First, model output is not deterministic, so a single run can flake. For properties that matter, either set temperature low for the test run or run the case a few times and require it to pass most of them. Second, do not write assertions so tight that any reasonable rewording fails. You want to catch real regressions, not punish the model for using a synonym. If a test fails and you read the output and it is actually good, the test was wrong. Loosen it.
Wire it into CI, and yes it is annoying
Put the runner behind one command. Make that command a required check on any pull request that touches a prompt file. Now a prompt change is a code change. It gets reviewed, it gets tested, and a regression shows up as a red X instead of a support ticket.
I will be straight with you about the cost. Evals are annoying to maintain. Cases go stale when the product changes. A model upgrade can flip a bunch of them at once and you have to sit there deciding which flips are fine and which are real. Every new failure mode is a new case someone has to write, and writing cases is nobody's favorite afternoon. The suite is never done.
It is still worth it. The first time the harness catches a one word prompt tweak that would have broken your JSON parsing for every user, you stop arguing with yourself about whether to keep it. The value is not that the tests are clever. The value is that the knowledge stops living in your scrollback and starts living in a file that runs on every change, whether or not anyone remembers to look.
Start with ten cases this week. Add one every time something breaks. That is the whole practice.
AGINE Academy is an independent product by AGINE AI (not affiliated with Anthropic). We teach building with Claude by doing the work, not watching lectures.
Top comments (1)
The property check point is the bit I keep coming back to. Exact string evals rot fast, but a tiny matcher for valid JSON, required fields, and refusal shape catches the regressions that usually slip through prompt reviews.