Most broken RAG pipelines don't crash. They run fine and quietly feed the LLM the wrong context — so you get confident, slightly-wrong answers. The two fixes that catch most of it:
- Add a relevance floor. Vector search always returns your top-k, even when nothing is actually relevant. Reject weak matches instead of feeding the model noise:
results = store.similarity_search_with_score(query, k=5)
relevant = [doc for doc, score in results if score >= 0.75]
if not relevant:
return "I couldn't find anything relevant." # better than hallucinating
- Rewrite the query before retrieving. Users ask messy questions. One cheap LLM call to clean it up sharpens the query vector a lot: python
search_query = llm.invoke(
f"Rewrite as a clear, standalone search query. Return only the query.\n\n{user_question}"
).content
results = store.similarity_search(search_query, k=5)
I build full stack and AI systems. My work is on GitHub: github.com/OrienSpec.
I build full stack and AI systems. My work is on GitHub: github.com/OrienSpec.
Top comments (3)
the relevance floor is right but there's a nasty failure mode: the threshold you calibrate in dev becomes meaningless the moment the embedding model changes. we had a 0.78 floor calibrated on
text-embedding-ada-002, shipped fine, then the client switched embedding providers and scores drifted 15–20 points. the floor held but wasn't filtering the same thing anymore.the query rewrite is solid. one optimization: skip it entirely for short pronoun free queries — saves a round trip on the happy path. we added a heuristic (< 6 words, no "it"/"they"/"that") that skips rewriting ~40% of queries in prod.
are you running rewrite and retrieval sequentially, or caching the rewritten query for session level context?
Silent bad retrieval is the failure mode I trust the least, precisely because nothing errors out. I started logging the retrieved chunks next to the answer so I can eyeball whether the context was even relevant before blaming the model. Retrieval quality gets treated as an afterthought far too often.
that's great !