There is a wide, expensive gap between a prompt that works in a playground and a feature real users can rely on.
The playground is forgiving. You know what to ask, you accept the occasional miss, and nothing depends on the answer. Production is none of those things.
Closing that gap comes down to four layers: retrieval, prompts, testing, and guardrails. Here is how each one works in practice.
Why does a working prompt fail in production?
Because you were the only user, and you already knew what to ask.
Real users phrase things strangely, ask about things you never documented, and occasionally try to break it on purpose. A wrong answer now has a consequence attached: a bad refund, a wrong medical instruction, a lost customer.
"It usually works" is a fine standard for a demo and not a standard you can ship. Everything below exists to raise it.
Why is retrieval the first thing to fix?
Because the model does not know your business, and most failures start here.
The most common AI feature answers questions from your own content: support documents, product data, internal knowledge. The instinct is to paste everything into the prompt. That is slow, expensive, and stops working once your content grows.
The real technique is retrieval. Store your content so it can be searched by meaning rather than exact words, fetch only the few relevant pieces for each question, and give the model just those to answer from.
Here is the part teams miss. If the search returns the wrong passages, the model answers confidently from the wrong context, and you get a fluent, plausible, incorrect answer.
Most complaints about a model "making things up" are really complaints about retrieval fetching the wrong documents. Before you change models or rewrite prompts, look at what the search actually returned.
When an answer is wrong, read the retrieved documents first. If the right document was never fetched, no prompt and no larger model will fix it. You are debugging search, not AI.
What makes a production prompt good?
Prompt writing has a mystique it does not deserve. A good one is clear instructions, relevant context, and explicit limits.
Four things matter far more than clever phrasing:
A specific role and task. "Answer only from the provided documents, as a support assistant for this product" beats a vague, chatty personality every time.
What to do when unsure. Tell the model directly to say it does not know rather than guess. This single instruction prevents a large share of confident wrong answers.
The exact output shape. If you need structured data, ask for it precisely and check what comes back before using it.
The boundaries. What it must never do, never claim, and never reveal, regardless of what a user asks.
Keep prompts in version control and treat a change to one as a change to code, because that is exactly what it is. A small edit shifts behaviour for every user at once, and if it is not tracked you cannot explain a problem later.
How do you know a change made it better?
You measure it. There is no shortcut here, and this is the layer most teams skip.
Build a set of real inputs paired with known-good answers. Include the awkward ones: ambiguous questions, hostile users, questions your documents genuinely do not cover. Then run the whole set every time you change a prompt, swap a model, or adjust retrieval.
This is unglamorous and it is the highest-value work in the entire project. It is also what lets you move to a newer, cheaper or faster model later with confidence rather than dread.
A small test set you actually run beats a large one you keep meaning to build.
What guardrails does a live feature need?
The defining question is not how good the answer is when it is right. It is what happens when it is wrong, because sometimes it will be.
Check structured output before anything acts on it. Raw model output should never drive a real action unchecked.
Limit what it can touch. If the feature can take actions, give it the fewest possible, and require confirmation for anything expensive or irreversible.
Filter in both directions. Watch for private data leaking out, for the conversation drifting into territory you are liable for, and for text that tries to give the model new instructions.
Keep a person involved wherever a wrong answer is expensive. A feature that drafts something for a human to approve has a completely different risk profile from one that acts alone.
Documents, emails, web pages and user messages are data, not commands. If your system treats retrieved text as instructions, anyone who can get text into your documents can control your feature.
What about latency and token use?
Both are design decisions, and both surprise teams who leave them until after launch.
Route by difficulty. The easy majority of requests can go to a small fast model, with a larger one kept for genuinely hard cases. Cache repeated questions, because there are far more of them than anyone expects.
For speed, stream the answer so the user sees it forming instead of watching a spinner. Do independent work at the same time rather than one step after another. And decide honestly which parts need an instant answer, because plenty of useful work can happen in the background.
The thread running through all of it
Every layer here is the same idea applied in a different place. A language model is capable and unreliable, so you build a system around it that makes the capability usable and the unreliability safe.
The model is the easy part now. The engineering around it is the product, and it is the bulk of what our AI work actually involves. If you are still deciding what to build, we wrote about choosing between an agent, retrieval and a single call.
The short version
Fix retrieval first, because that is where most wrong answers come from. Write prompts as instructions with clear limits, and version them like code.
Build a test set and run it on every change, so improvements are measured rather than hoped for. Then add the guardrails that make being wrong survivable, and decide cost and speed on purpose.
Top comments (0)