The RAG Bug That Isn't an Error: Bad Retrieval

OrienSpec — Thu, 30 Jul 2026 15:34:47 +0000

Most broken RAG pipelines don't crash. They run fine and quietly feed the LLM the wrong context — so you get confident, slightly-wrong answers. The two fixes that catch most of it:

Add a relevance floor. Vector search always returns your top-k, even when nothing is actually relevant. Reject weak matches instead of feeding the model noise:

results = store.similarity_search_with_score(query, k=5)

relevant = [doc for doc, score in results if score >= 0.75]

if not relevant:

return "I couldn't find anything relevant." # better than hallucinating

Rewrite the query before retrieving. Users ask messy questions. One cheap LLM call to clean it up sharpens the query vector a lot: python

search_query = llm.invoke(

f"Rewrite as a clear, standalone search query. Return only the query.\n\n{user_question}"

).content

results = store.similarity_search(search_query, k=5)

I build full stack and AI systems. My work is on GitHub: github.com/OrienSpec.

DEV Community: OrienSpec

The RAG Bug That Isn't an Error: Bad Retrieval