Debugging a Misbehaving Prompt in Production

So you have a prompt in production and a customer reports a bug: the prompt is misbehaving. What do you do?

LLM-reported issues are among the most frustrating challenges in shipping AI to production. Your system works fine internally, then a customer hits an edge case, their data doesn't match your assumptions, or variance in model output reveals a problem hiding in your prompt chain. This two-part article walks you through the full flow: reproducing and localizing it, fixing it with confidence, and deploying without breaking everything else.

Part 1: Reproduce, Localize, Classify

1. Reproduce the error

Run it multiple times. You're checking whether this is a one-off (LLM variance) or a real systematic bug.

Example: a customer says your booking bot skipped a required question. You rerun the same input 5 to 10 times. If it fails once in ten, that's variance, worth a note but not urgent. If it fails consistently, you have a real bug.

2. Localize where the problem actually is

Your production system is rarely one prompt. Your agent depends on a chain: multiple prompts and handovers, MCP calls, API calls. The bug could be anywhere in that chain, so before you touch anything, you need to know which link broke. leveraging the traces and logs that your system should have in place to observe and monitor your agent

3. Run a triaging prompt

Build a triaging agent that reads your agent's full trace and finds the failure point. Pass it: the customer input, every intermediate output, the LLM's reasoning at each step, and which tool or decision prompt fired at which stage. The triage agent compares what actually happened against what should have happened and tells you which step broke.

Think of the triage agent as a skill, not a one-off prompt: a main prompt that identifies where and why the failure happened, run on a frontier-class model with access to your observability and monitoring tools. It runs from within your codebase, since prompts are code, and it needs that code access to do proper discovery: figuring out which service or step actually took over when things went wrong.

Pass it: the customer input, every intermediate output, the LLM's reasoning at each step, and which tool or decision prompt fired at which stage. The triage prompt compares what actually happened against what should have happened and tells you which step broke.

4. Classify: customer-side or system-side

Once localized, you're in one of two situations.

Customer-side. The bug is config or input on the customer's bot, something that doesn't match your system's capability. Here you work with the customer to fix it on their end, without changing your system.

System-side. The bug is yours. Now you use what the triage prompt surfaced to decide the fix: change an instruction (add, remove, or reword it), or rework a tool-call prompt.

One more check before you fix anything

Is this bug isolated to one bot, or does it show up across several? That distinction matters. A single bot means you're looking at a patch, reproduce it, fix it, validate it (that's Part 2 below). Several bots hitting the same failure means you're not looking at a patch anymore, you're looking at a systemic regression, and that needs proper evaluation infrastructure, not a quick fix.

flowchart TD
    A[Customer reports bug] --> B["1. Reproduce: run multiple times"]
    B -->|Not reproducible| B1[Log as variance, monitor]
    B -->|Reproducible| C["2. Localize: walk the chain - prompts, MCP calls, APIs"]
    C --> D["3. Triage Agent analyzes traces: input, output, reasoning, decisions"]
    D -->|Localized| F{"4. Classify"}
    F -->|Customer-side: config or input mismatch| G[Work with customer, no system change]
    F -->|System-side: our prompt or tool| H[Tune instruction or tool-call prompt]
    H --> I{Reproducible on other bots?}
    I -->|No, isolated| J["Single-bot patch"]
    I -->|Yes, pattern| K["Systemic regression"]

Part 2: Fix, Validate, Ship

Part 1 ended once you've localized the bug and know it's on your system, one bot, not a pattern across many. Now you actually fix it and prove the fix works before it goes anywhere near production traffic.

1. Write a test for the edge case

Create an integration test around the new prompt. It should mimic the real environment as closely as possible: mock the same steps your code goes through, make a real call to the LLM. Run it multiple times, not once.

Example: the triage in Part 1 showed your booking bot's confirmation step drops a required field when the customer message is ambiguous. You write a test that replays that exact ambiguous input against the updated prompt, 10 times.

Set a pass threshold tuned to your test suite, not a fixed number like 100%. LLM output has variance, so you're balancing how much false-positive tolerance you can live with against how long you're willing to let this test run in CI. Too strict and every PR trips it on noise. Too loose and it stops meaning anything.

2. Confirm the fix against the original bug

Check the test output against the expected behavior, the one the customer's bug report described in the first place. Some ways to check are: an LLM-as-judge call, or deterministic matching against keywords or sentences that must be present in the response. Deterministic where you can, judge where the output is too open-ended to pin to exact strings.

3. If the test fails

The fix didn't work. Feed the failure back to the model and let it attempt the fix again. If that second attempt also fails, stop looping. Escalate to human review: an engineer looks at the trace and edits the prompt manually.

4. Check for regressions

Once the new test passes, run the rest of your integration suite. You're confirming the fix for this bug didn't break something else the prompt was already handling correctly.

5. Ship it gradually

Roll out to the reporting customer first to validate with live data, then to everyone, depending on how big the change is. If it's a big enough change, an A/B test might be worth it.

6. Turn the bug into a permanent test case

Add it to your offline dataset, the set you run against for major prompt edits or model swaps, not small patches.

flowchart TD
    A["From Part 1: single-bot patch confirmed"] --> B["1. Write integration test: mock env, real LLM call, multiple runs, tuned threshold"]
    B --> C["2. Check against original bug: LLM-as-judge or deterministic match"]
    C -->|Fails| D[Model attempts fix again]
    D --> C
    D -->|Fails again| E[Escalate to human, engineer edits manually]
    E --> C
    C -->|Passes| F["4. Run full regression suite"]
    F -->|Regression found| B
    F -->|Clean| G["5. Gradual rollout: customer first, then everyone"]
    G --> H{Big enough change?}
    H -->|Yes| I["A/B test"]
    H -->|No| J[Full rollout]
    I --> K["6. Add to offline dataset"]
    J --> K

Conclusion: What's next?

This workflow reproduce, localize, triage, fix, validate, ship. lets you move with confidence when a customer's bug lands. But it raises deeper questions:

• How do you build and scale the eval infrastructure to catch these bugs before a customer finds them? (Systemic regressions and cross-bot pattern detection)
• When the bug is on the customer's side, how do you guide them through the fix without touching your system? (Customer-side failure modes)
• What about A/B testing the rollout to catch regressions early? (Staged rollout and validation)

For now: ship the fix, learn from the incident, and add that test case to your offline dataset so you don't ship the same bug twice.

---