You built a RAG system. It scores say 0.91 faithfulness on your eval set. You deploy it. A user asks about a return policy that isn't in your knowledge base. The chatbot answers confidently, fluently, and completely wrong. Sounds familiar right !!
This isn't a prompt engineering problem. It's a retrieval architecture bug that almost every RAG tutorial skips.
What actually happens
When a user query has no good match in your vector store, cosine similarity doesn't return "nothing found." It returns the least-bad chunk, whatever is closest in embedding space, even if it's completely irrelevant. Your reranker then picks the best of these bad candidates. The LLM receives context that's wrong-but-plausible and does exactly what it's trained to do: generate a fluent, coherent answer. No hallucination flag. Just wrong.
Your eval set never caught this because you built it from documents in your knowledge base. Every golden query always had a valid retrieval match. The failure mode only surfaces on out-of-distribution queries — which is most of production traffic.
The fix: similarity thresholding before generation
results = vectorstore.similarity_search_with_score(query, k=5)
top_score = results[0][1] # cosine similarity of best match
if top_score < 0.72: # calibrate this on your domain
return "I couldn't find information about this in our knowledge base."
context = [doc for doc, score in results if score >= 0.65]
The threshold isn't universal, i.e 0.72 is a starting point. Calibrate it by running your golden eval set and finding the score below which faithfulness drops sharply. For most general-purpose corpora it's between 0.65–0.78.
Why this matters for interviews too
This exact scenario "your RAG system hallucinates on queries about content not in the knowledge base, how do you fix it?" Well, shows up verbatim in AI engineering interview loops at companies building production RAG systems. The answer they're looking for isn't "better prompting." It's similarity thresholding, out-of-distribution detection, and a graceful fallback path.
If you're prepping for AI engineering roles, the full RAG system design question set (including this one) is covered in depth on KnowEaseAI (https://knoweaseai.com/blog/rag-system-design-interview-questions)
— with the production failure modes that actually show up in interviews, not just the happy path.
Top comments (1)
Static cosine thresholding is definitely a step up from naive top-k, but raw cosine scores in high-dimensional dense embeddings can be deceiving in production.
Because of embedding hubness and score compression, an out-of-domain query can easily produce a 0.73 cosine score against a generic corporate disclaimer or FAQ chunk. Relying on a fixed float often fails when chunk lengths vary across the corpus.
A more reliable production guard is checking the unnormalized logit from a cross-encoder reranker. Because the cross-encoder computes full attention across the query and chunk tokens rather than comparing isolated vectors, its margin between relevant text and arbitrary drift is much sharper.