Most broken RAG pipelines don't crash. They run fine and quietly feed the LLM the wrong context — so you get confident, slightly-wrong answers. The two fixes that catch most of it:
- Add a relevance floor. Vector search always returns your top-k, even when nothing is actually relevant. Reject weak matches instead of feeding the model noise:
results = store.similarity_search_with_score(query, k=5)
relevant = [doc for doc, score in results if score >= 0.75]
if not relevant:
return "I couldn't find anything relevant." # better than hallucinating
- Rewrite the query before retrieving. Users ask messy questions. One cheap LLM call to clean it up sharpens the query vector a lot: python
search_query = llm.invoke(
f"Rewrite as a clear, standalone search query. Return only the query.\n\n{user_question}"
).content
results = store.similarity_search(search_query, k=5)
I build full stack and AI systems. My work is on GitHub: github.com/OrienSpec.
I build full stack and AI systems. My work is on GitHub: github.com/OrienSpec.
Top comments (1)
that's great !