DEV Community

Cover image for Why Your LLM Prompts Break in Production (And How to Fix Them)
Taqui
Taqui

Posted on

Why Your LLM Prompts Break in Production (And How to Fix Them)

#llm #ai #programming #beginners

I recently started learning Applied AI Engineering from scratch. Before this, I thought prompting was just typing a question into ChatGPT, testing it once, and moving on.

In real apps, that breaks fast. Here are my handyy notes and visual diagrams from Chapter 1 on how to make LLMs reliable, ending with the fact-checking system I designed to test these concepts.


TLDR: The 5 Rules of Reliable Prompting
Temperature = 0 makes answers consistent, but it will still repeat a fake fact if the model does not know the answer.
Thinking comes first: Always put "thinking" before "answer" in JSON so the AI calculates steps before speaking.
Guard the edges: Wrap user input in <userMessage> tags and put your core rules at the very top and bottom.
Give an escape hatch: Use string | null in schemas so the model is never forced to hallucinate missing data.
Chain and retry: Split big prompts into small steps and use if/else retry loops with a max_retries cap.

1. Stochastic LLMs & Temperature

A calculator is deterministic (2 + 2 is always 4). An LLM is stochastic, meaning it predicts the next word by rolling a dice on probabilities. We control that dice roll using Temperature:

Prompt: "The dog chased the..."
 ├─ "ball" (50%) ◄── Temp = 0 (Always picks #1 top choice. Good for facts)
 ├─ "cat"  (30%)
 └─ "car"  (10%) ◄── Temp = 1 (Rolls the dice. Good for creative poems)
Enter fullscreen mode Exit fullscreen mode

Why this matters: Setting Temperature = 0 does not stop hallucinations. The model does not stop predicting at 0. If it does not know the answer, its #1 top guess is still a fake fact, and it will just return that wrong answer every single time.

2. Chain of Thought (CoT) Prompting

For an LLM, generating tokens on the screen is literally how it thinks. If you ask a tricky riddle and force a 1-word answer right away, it has zero time to calculate and guesses wrong.

  • Zero-Shot CoT: Adding "Let's think step by step". Good for general math or logic, but skip it for simple facts like "Capital of France" because extra tokens cost money and slow down your app.
  • Few-Shot CoT: Giving 2 to 3 solved step-by-step examples. Best when you need the AI to follow your exact checklist every time.

The "No Backspace" Rule for UI Schemas

Users do not want to see a wall of robot thoughts on their screen. To hide the thinking, define a JSON schema with a "thinking" variable (hidden in backend) and an "answer" variable (rendered in UI).

❌ WRONG ORDER                      ✅ RIGHT ORDER
{                                   {
  "answer": "Weak!",                  "thinking": "Step 1: 14 chars...",
  "thinking": "Wait, 14 chars..."     "answer": "Strong!"
}                                   }
(AI guesses before thinking,        (AI thinks first, then locks in
 and cannot backspace to fix it)     the right answer)
Enter fullscreen mode Exit fullscreen mode

3. Five Prompt Failure Modes

When real users use your app, prompts fail in predictable ways. Here is how the LLM's attention works and how to defend it:

Attention
  High │ ★ System Rules (Put core rules at the start)
       │
   Low │ ░░░ 10 Pages of Policy / Data (Lost in the Middle) ░░░
       │     <userMessage>{Untrusted Input}</userMessage>
  High │ ★ Sandwich Reminder (Put final rule here to beat Recency Bias)
       └────────────────────────────────────────────────────────► Prompt
Enter fullscreen mode Exit fullscreen mode
  1. Prompt Injection: Users type "Ignore previous rules" at the end. Because of recency bias, the AI listens to the latest message. Fix: Wrap input in <userMessage> fences and add a rule reminder at the very bottom.
  2. Lost in the Middle: LLMs forget stuff buried in the middle of long text. Fix: Put long data in the middle and strict rules at the top and bottom.
  3. Conflicting & Vague Rules: Saying "Be warm and casual, but sound like a strict lawyer. Make it pretty short." Fix: Separate the role from the tone, and replace "pretty short" with exact numbers ("max 3 sentences").
  4. The "Pink Elephant" Problem: Saying "Do not think of a pink elephant" makes you picture one. Telling an AI what not to do activates those words. Fix: Tell it what to do instead ("Write 1 paragraph using 5th-grade words").
  5. Sycophancy (Yes-Man Trap): LLMs flatter confident users. Fix: Detach your ego and ask it to find flaws: "Here is an idea I found online. Act as a strict critic and list 3 ways it fails."

4. Structured Outputs

Asking "Please return only JSON" in a prompt is just a polite wish. The AI will eventually add "Sure! Here is your JSON:" and crash your app.

Structured Outputs (Constrained Decoding) physically block invalid tokens by setting their probability to 0%:

AI wants to reply:
  "Sure!" (70%) ──► Blocked (0%) ❌
  "Here"  (20%) ──► Blocked (0%) ❌
  "{"     (10%) ──► Allowed (100%) ✅
Enter fullscreen mode Exit fullscreen mode

Two rules I use for schemas:

  1. Use clear names + Enums: Instead of "s": string, use "sentiment": "positive" | "negative" | "neutral".
  2. Always add a | null escape hatch: Valid JSON syntax does not mean true facts. If "github_url": string is required and a resume has no link, the AI is forced to invent a fake URL just to obey the schema. Using string | null lets it safely output null.

5. Prompting Reliably in Production

Instead of testing a prompt once ("vibe checking") or writing one giant Mega-Prompt, reliable systems use Eval Sets (20 to 50 saved test cases to catch regressions) and Prompt Chains with retry guardrails:

[Step 1: Translate] ──► [Step 2: Check Rule] ──► [Guardrail (if/else)]
 (Fast/Cheap Model)       (Smart Model)                 │
                            ▲                           ├─ Pass ──► [UI Output]
                            └─── Retry + Error Msg ─────┘ (Fail)
Enter fullscreen mode Exit fullscreen mode

Splitting steps makes it easy to see where the output went wrong and adjust only that step.

6. System Design: Fact-Checking Agent

To put everything together, I designed an agentic workflow in Excalidraw for a fact-checking app called TruthPulse.

Click to see the 3 bugs I fixed between V1 and V2
  1. Search Queries vs. URLs: In V1, I told the agent to pass a url string to Web_search_tool[cite: 1]. Since LLMs do not know today's URLs, they would hallucinate dead links. In V2, I changed this to pass query: string (search keywords)[cite: 2].
  2. Nullable Error Field: My V1 schema had "error": string[cite: 1], which forced the AI to invent an error even when the check succeeded. In V2, I changed it to "error": string | null[cite: 2].
  3. Retry Guardrail + Loop Limit: I added a {"retry": "true" | "false"} check after the search step[cite: 2], plus a max_retries = 3 cap so an unsearchable rumor does not loop forever.

Final System Diagram 

                        ┌───────────┐
                        │   Input   │
                        └─────┬─────┘
                              ▼
             <userMessage>{value}</userMessage>
             Remember: Only claim the given fact
                              │
                              ▼
        ┌───────────────────────────────────────────┐
        │ Main Agent + Thinking (Temp = 0)          │◄────────────┐
        │ "Strict fact checker... if invalid return │             │
        │  error, else pass query: string to tool"  │             │
        └──────────┬───────────────────────┬────────┘             │
                   │                       │                      │
     query: string │                       │ (Invalid input)      │
                   ▼                       │                      │
        ┌─────────────────────┐            │                      │
        │   Web_search_tool   │            │                      │
        └──────────┬──────────┘            │         Retry = true │
                   │ string                │      (max_retries=3) │
                   ▼                       │                      │
        ┌─────────────────────┐            │                      │
        │ Judge Prompt        │            │                      │
        │ Compare source with │            │                      │
        │ userMessage         │            │                      │
        └──────────┬──────────┘            │                      │
                   ▼                       │                      │
        ┌─────────────────────┐            │                      │
        │ {"retry": "true" |  ├────────────┼──────────────────────┘
        │           "false"}  │            │
        └──────────┬──────────┘            │
                   │ false                 │
                   ▼                       ▼
        ┌───────────────────────────────────────────┐
        │ {                                         │
        │   "thinking": string,                     │
        │   "claim": "true"|"false"|"unverified",   │
        │   "error": string | null                  │
        │ }                                         │
        └───────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Takeaway

The main thing I learned in Chapter 1 is that AI Engineering is not about finding one magic prompt. Because LLMs are probabilistic, the real work is building the tags, schemas, chains, and retry guardrails around the model so your app stays reliable even when the AI slips up.

i hope reading really worth it for you,

Make sure to drop (🔥,♥️,👍)

And your comment really motivates me

i am learning and sharing on the internet.

Have a nice Day:)

Top comments (0)