My LLM pipeline passed every eval. Then it started lying to users in production.

My LLM pipeline passed every eval. Then it started lying to users in production.

Not dramatically. Quietly. Confidently wrong.

Here's what happened.

I shipped a RAG pipeline for one of our SaaS products. Tested it on 40 documents. Response quality was sharp — I was genuinely proud of it. Onboarded the first real users. Within 72 hours, the system was returning answers that sounded authoritative but were just... fabricated. Hallucinated policy details. Made-up clause numbers.

I dug into the traces. The context window was silently overflowing. When the retrieved chunks exceeded the limit, the model didn't throw an error. It didn't truncate cleanly. It just started confabulating to fill the gap — and nothing in my eval suite caught it because my test docs were small.

The fix took 45 minutes. A token counter, a hard limit, a fallback message. Done.

But that 72-hour window cost me real trust with early users.

The lesson I keep relearning: LLMs don't fail loudly. They fail smoothly. The model will always return something — it will just sometimes be fiction dressed as fact. If you're shipping LLM features without per-call tracing and a token budget enforced at retrieval time, you are not testing for the failure mode that will actually hurt you.

Build the guardrails before you need them. Not after you read a support ticket, wondering why your product confidently told someone the wrong thing.