You paste a prompt into an agent, it produces a great result, and you wire it into a workflow. Two weeks later it starts failing on inputs that look nothing like what broke it before, and you can't tell if the prompt is bad or your inputs just got unlucky.
That's not a prompt-quality problem. It's a testing problem — you never actually tested the prompt, you tested one lucky run of it.
The test
Before you rely on a prompt for anything repeated, run it three times on real inputs — not the clean sample case you first tried it on, the messy actual data it'll see in production.
Then check two things:
- Output shape stability. Does the structure of the response stay consistent across the three runs — same sections, same format, same level of detail — or does it wander?
- Failure mode, not just failure rate. When it does go wrong, how does it go wrong? Missing a section? Hallucinating a field that doesn't exist? Silently truncating? Write it down.
The second point is the one people skip, and it's the one that matters. A prompt that fails the same predictable way every time is easy to guard against — you know what to check for. A prompt that fails a different way each time is a liability, because you can't write a check for "unknown unknown."
Judge on variance, not best case
This is the actual insight: a prompt that produces excellent output once and unusable output twice is worse than a plain, consistent prompt that's merely fine three times out of three.
The excellent-once prompt tempts you to ship it because your first impression was strong. But you can't build a repeatable workflow on a coin flip. The consistent-but-plain prompt is boring, and boring is exactly what you want in a pipeline step you're not going to babysit.
A worked example
Take a code-review prompt: "Review this diff for bugs, security issues, and style problems."
Run it on three real diffs from your actual repo (not toy examples):
- Run 1 (a straightforward refactor): clean, three-section output, correctly flags nothing since the diff is clean.
- Run 2 (a diff touching auth logic): flags a real issue, but the output format changes — it switches to a numbered list instead of the three sections from run 1.
- Run 3 (a large diff, 400+ lines): truncates silently around line 200 and reviews only the first half, with no indication it stopped early.
That's your annotation: this prompt's failure mode is silent truncation on large diffs, and its output format isn't stable once it finds an issue worth flagging. Now you know to cap diff size before calling it, and to not parse the output assuming a fixed structure. That's actionable. "Sometimes it's not great" is not.
Why this matters more for production than for one-off use
If you're using a prompt once, manually, and reading the output yourself, variance barely matters — you're the check. The moment you chain it into something that runs unattended, or that a teammate relies on without reading every output, variance is the whole risk. Free prompt lists are usually fine for one-off tasks like summarizing or rewriting, because a bad run just means you try again. They're not designed to survive being embedded in a pipeline, because nobody tested them that way.
Doing this for 150 prompts
Running this test yourself is the right call for prompts you use constantly and want to trust. It's also real work — three runs, honest read of the failures, written annotation — per prompt, for every prompt you rely on.
If you'd rather start from prompts where that work is already done, AgentKit Works' Prompt Ops pack is 150 production prompts annotated this way — when each one works, what to expect, and how it fails — across debugging, code review, refactoring, and more. It's one way to skip the setup; the variance test above is the thing to apply regardless of where your prompts come from.
Top comments (0)