yaml
title: "You Don't Have a Prompt Problem. You Have an Eval Problem."
published: false
description: "If you can't tell automatically whether an output is good, every prompt change is a vibe. Here's the smallest eval setup that actually pays for itself."
tags: ai, testing, python, devops
series: Measuring AI Workflows
canonical_url:
(Leave canonical_url empty. Flip published when you've run the code.)
Here's a workflow I bet you recognise.
Output isn't quite right. You tweak the prompt. Looks better. Ship it. Two weeks later something else is off, you tweak again, and you have no idea whether you fixed the new problem or reintroduced the old one because there is no record of what "working" looked like.
That's not a prompting problem. That's what software development looks like without tests.
The reason it persists is that LLM output feels unbounded and untestable. It isn't. Most of it is far more testable than people assume, and the useful assertions are boring ones you can write in an afternoon.
The economics first, because they decide what's worth building
I timed [NUMBER] of my own tasks with and without AI last month. It lost on [NUMBER] of them, and the cause was never model quality. It was verification the human time spent checking output before it could be used.
net_per_run = manual − (generation + verification + retries)
If verification approaches manual time, net gain approaches zero no matter how fast generation is. A better model doesn't help. This gets misdiagnosed as a capability problem constantly.
An eval is a machine that does part of your verification for you. That's the entire value proposition, and it's why evals pay back faster than almost anything else you can build: they attack the term in that equation that a better model can't touch.
Start with assertions, not judges
The instinct is to reach for LLM-as-judge. Resist it for one more day. Most of what you need to know is checkable with plain code, and plain code is deterministic, instant, and free.
python
from dataclasses import dataclass
from typing import Callable
@dataclass
class Case:
name: str
inputs: dict
checks: list[Callable[[str], bool | str]] # True, or a failure message
def run_evals(cases, generate):
failures = []
for c in cases:
out = generate(**c.inputs)
for check in c.checks:
r = check(out)
if r is not True:
failures.append(f"{c.name}: {r or check.name}")
return failures
Now the checks. These are unglamorous and they catch a genuinely surprising share of real regressions:
python
import json, re
def is_valid_json(out):
try:
json.loads(out); return True
except json.JSONDecodeError as e:
return f"invalid JSON: {e.msg}"
def has_required_keys(*keys):
def check(out):
try: d = json.loads(out)
except Exception: return "not JSON"
missing = [k for k in keys if k not in d]
return True if not missing else f"missing keys: {missing}"
check.name = f"has_keys{keys}"
return check
def no_hedging(out):
banned = ["as an ai", "i cannot", "it's important to note", "delve into"]
hit = [p for p in banned if p in out.lower()]
return True if not hit else f"hedging: {hit}"
def within_length(lo, hi):
def check(out):
n = len(out.split())
return True if lo <= n <= hi else f"length {n} outside [{lo},{hi}]"
return check
def cites_only(allowed_urls):
def check(out):
found = set(re.findall(r"https?://[^\s)]]+", out))
extra = found - set(allowed_urls)
return True if not extra else f"invented URLs: {extra}"
return check
That last one — checking for URLs that weren't in the source material — has caught more real problems for me than any sophisticated method. Fabricated citations are common, high-embarrassment, and trivially detectable.
The cases matter more than the checks
Your eval set is not a random sample. It's a museum of everything that has already gone wrong.
Rule: every time you catch a bad output in production, it becomes a case. That's the whole discipline. Ten cases collected this way beat a hundred synthetic ones, because they're drawn from the actual distribution of your failures rather than your imagination of them.
python
CASES = [
Case("empty input", {"text": ""}, [is_valid_json]),
Case("very long input", {"text": LONG_SAMPLE}, [within_length(50, 300)]),
Case("adversarial", {"text": INJECTION_SAMPLE},[no_leaked_instructions]),
Case("the one from [DATE] that fabricated a source",
{"text": THAT_INPUT}, [cites_only(THAT_SOURCES)]),
]
Name the cases after what went wrong. "the one from March that fabricated a source" is a better test name than test_citation_accuracy_3 because in six months you'll know why it exists.
Then, and only then, an LLM judge
For the genuinely subjective dimensions tone, relevance, whether an answer actually addressed the question — use a model. Three rules that make the difference between a useful judge and an expensive random number generator:
Binary, not scored. "Rate 1–10" produces noise. Ask a yes/no question with a stated criterion.
One dimension per call. A judge asked about accuracy, tone and completeness at once will collapse them into a general vibe.
Judge against a reference, where you have one. Comparative judgement is far more stable than absolute judgement.
python
JUDGE = """You are checking one property of a piece of text.
PROPERTY: {property}
TEXT:
{text}
Answer with exactly one word: PASS or FAIL.
If the property does not clearly hold, answer FAIL."""
def judge(property_desc):
def check(out):
v = call_model(JUDGE.format(property=property_desc, text=out)).strip().upper()
return True if v.startswith("PASS") else f"judge failed: {property_desc}"
check.name = "judge"
return check
Validate your judge before you trust it. Hand-label twenty outputs, run the judge, compare. If it disagrees with you on more than a couple, the judge prompt is the problem fix it before you use it to evaluate anything else. A judge you haven't checked is just a second unverified model in the loop, which is the opposite of what you're building.
Wire it to a diff, not a dashboard
The value isn't a score. It's did this change make things worse.
python
def compare(cases, old_gen, new_gen):
before = set(run_evals(cases, old_gen))
after = set(run_evals(cases, new_gen))
return {
"fixed": sorted(before - after),
"broken": sorted(after - before), # the only column that matters
"still": sorted(before & after),
}
Run it on every prompt change and every model swap. broken is the column that should stop a deploy. Model upgrades are the underrated case here — "newer model" is not "better for your task," and this is how you find out in four seconds instead of four weeks.
Keep it fast and cheap enough to run every time. An eval suite that takes ten minutes and costs real money gets skipped exactly when you're in a hurry, which is exactly when you're most likely to break something.
What this doesn't do
Be honest about the boundary. Evals catch regressions on failure modes you've already seen. They don't catch novel ones, they don't tell you whether the workflow is worth having, and a green suite is not a guarantee of a good output.
They shift verification from every run to every change, which is where the leverage is. The residual human check gets shorter, not eliminated.
And if verification is still most of your run time after building these, that's a real signal about the task itself some work genuinely takes as long to check as to do, and the correct answer there is to keep it human rather than to build more tooling around it. I've written up the decision framework for that separately.
The 30-line version
If you build nothing else:
[ ] A list of cases, each named after a real failure
[ ] is_valid_json / has_required_keys, if you parse output
[ ] within_length
[ ] cites_only(sources), if the output makes claims
[ ] A diff of before/after failures on every prompt change
Five checks, one afternoon. It will catch things a month of careful reading wouldn't, and more importantly it will catch them the moment you introduce them, which is the only time fixing them is cheap.
I test AI tools and workflows honestly, including the ones that don't earn their place: AiStackGuru.
Top comments (0)