⭐ Star the repo: ClinicaQuery-AI on GitHub
🔔 Follow me here on Dev.to and on GitHub. I build full-stack AI systems and write up every ugly detail, including the parts that took four hours to fix.
I asked my medical research assistant a question about a paper it had never seen.
It answered anyway.
Fluent, confident, correctly formatted, with a citation and a green "high confidence" badge. Every part of that answer was invented. The paper it cited was about cardiovascular exercise. The question was about antibiotic dosing.
Nothing in my system was broken. That is the terrifying part. Retrieval ran, found the five least-irrelevant paragraphs it had, handed them to the LLM, and the LLM did what LLMs do: it produced fluent text from whatever you give it. There was no error, no exception, no log line. Just a wrong answer wearing a lab coat.
This is the failure mode nobody warns you about when you build your first RAG app. Your system cannot tell the difference between "I found the answer" and "I found five paragraphs." It has no concept of failure, so it can never report one.
I spent two weeks fixing that. Here is everything I learned, including the dependency conflict that had no solution and the DLL error that forced me to build a better architecture than I originally planned.
First, what actually is RAG? (skip if you know)
If you are new to this, here is the whole idea in one paragraph.
An LLM does not know about your PDF. So before you ask it anything, you cut your PDF into chunks, convert each chunk into a list of numbers that captures its meaning (an "embedding"), and store those numbers in a database. When a question comes in, you convert the question into numbers too, find the chunks whose numbers are closest, and paste those chunks into the prompt. The model answers from text you handed it rather than from memory. That is Retrieval-Augmented Generation.
Standard RAG is three steps:
embed the question -> find the 5 closest chunks -> generate
It works well enough in a demo to make you think you are done. You are not done. Those three steps have at least four serious weaknesses, and I want to walk through each one, because the fixes are the interesting part.
Weakness 1: your chunks don't know where they came from
Here is a real chunk from a paper in my system:
"The reduction was 6.9 mmHg (95% CI: 5.2-8.6, p<0.001)."
Reduction in what? Measured in whom? After which intervention? You cannot tell, and neither can the embedding model. That chunk gets converted into a vector that means roughly "a number went down a bit," which is useless. It will never be retrieved for the question it actually answers.
The fix, from Anthropic's contextual retrieval research: before embedding a chunk, ask an LLM to write one sentence explaining where the chunk sits in its document, and glue that sentence to the front.
The same chunk becomes:
"From a meta-analysis of aerobic exercise trials, section on blood pressure outcomes in hypertensive adults. The reduction was 6.9 mmHg (95% CI: 5.2-8.6, p<0.001)."
Now the embedding actually means something.
_CONTEXT_PROMPT = """<document>
{doc_preview}
</document>
Here is a chunk taken from that document:
<chunk>
{chunk}
</chunk>
Write ONE short sentence (max 25 words) that situates this chunk within the
document, so the chunk can be understood and retrieved on its own."""
It costs one small LLM call per chunk at upload time, and it runs in a thread pool so a 14-chunk paper takes about 6 seconds. Anthropic measured a 49% reduction in retrieval failures from this technique alone.
One detail that matters: I store the original text separately in node.metadata["raw_text"]. The context sentence helps the machine find the chunk. The user should still read what the paper actually said.
Weakness 2: vector search is bad at exact terms
Ask a vector database about "eNOS phosphorylation at Ser1177" and it will cheerfully return a passage about nitric oxide signalling that never mentions Ser1177, because in embedding space those two things sit right next to each other.
Embeddings capture meaning, and meaning is fuzzy by design. That is usually a feature. In medicine it is a liability, because medicine runs on exact strings: drug names, gene symbols, ICD codes, dosages, p-values. Those are precisely the tokens embeddings blur together.
The fix: run a keyword search alongside the vector search. BM25 is the classic algorithm here and it finds literal tokens instantly.
_TOKEN_RE = re.compile(r"[a-z0-9]+(?:[-_][a-z0-9]+)*")
def tokenize(text: str) -> list:
return [t for t in _TOKEN_RE.findall(text.lower())
if len(t) > 2 and t not in _STOPWORDS]
That regex is deliberately built to keep ser1177, covid-19, and il-6 intact instead of shredding them into meaningless fragments.
I used rank_bm25, which runs in-process and persists to a pickle file. No Elasticsearch cluster, no Docker, no second service to keep alive. For a single-instance app that is exactly the right amount of infrastructure.
Weakness 3: you cannot just add the two scores together
So now you have two ranked lists. Merging them is where most people quietly break their own system.
The obvious move is to normalise both scores to 0-1 and add them. Do not do this. Cosine similarity is bounded and roughly normal. BM25 is unbounded and heavily skewed by document length. Min-max normalising them means a single outlier BM25 score squashes everything else into a narrow band, and your ranking silently becomes noise.
The fix: throw the scores away and keep only the ranks.
def reciprocal_rank_fusion(ranked_lists, k=60):
scores = {}
for results in ranked_lists:
for rank, item in enumerate(results, start=1):
nid = item["node_id"]
scores[nid] = scores.get(nid, 0.0) + 1.0 / (k + rank)
...
That is the entire algorithm. Reciprocal Rank Fusion, from a 2009 SIGIR paper. The constant 60 is a damper: rank 1 contributes 1/61 and rank 2 contributes 1/62, so no single list can dominate and a document has to place well in several lists to win.
Elasticsearch, Weaviate, and Azure AI Search all use this. It beats carefully tuned score normalisation in most published comparisons. Six lines.
There is a free bonus here too. If a chunk appears in both the vector results and the keyword results, that is a strong signal, so I track it and surface it in the UI as a "both" badge:
item["retriever_agreement"] = len(ranks[nid])
Weakness 4: your retriever was never trained to compare
This one is architectural and it is the biggest win available.
Your vector database uses a bi-encoder. It embedded the chunk at upload time, long before anyone asked anything, so that chunk's vector is a compressed summary produced with zero knowledge of the future question. Then it embeds your question separately and compares the two vectors. Fast, because everything is precomputed. Lossy, for the same reason.
A cross-encoder reads the question and the chunk together, in one pass, and outputs a single relevance number. It can see that "150 minutes" in the passage answers "how many minutes" in the question. Nothing was compressed away in advance.
The catch is that nothing can be precomputed, so it is one model call per candidate. You never run it over your whole corpus. The pattern is always:
retrieve 30 candidates cheaply -> rerank those 30 precisely -> keep 5
In my UI, any source that got promoted by the reranker shows a green +4 badge. Expand it and you see rank 7 to 2. That is the reranker catching something vector search ranked seventh and putting it where it belongs.
Weakness 5: the one that actually scared me
Everything above makes retrieval better. None of it helps when the answer simply is not in your documents.
CRAG (Corrective RAG) adds a grader between retrieval and generation, and crucially, gives the pipeline somewhere to go when the grade is bad:
if max_score >= 0.70:
action = "correct" # generate now
elif max_score >= 0.35:
action = "ambiguous" # rewrite the query, retrieve again, merge
else:
action = "incorrect" # do not generate at all
That last branch is the whole reason I rebuilt this thing.
if action == "incorrect" or not documents:
with trace.span("7_generation") as span:
span.attributes["skipped"] = True
span.attributes["reason"] = "crag_graded_retrieval_as_incorrect"
for word in NO_ANSWER_RESPONSE.split(" "):
yield word + " ", None
The generator never runs. There is no answer to generate, so none gets generated.
One design note I went back and forth on: I grade on the best document, not the average. One excellent source is enough to answer well, and averaging punishes a good retrieval that also happened to pull in a couple of loosely related chunks.
Ask my cardiology paper about amoxicillin dosing now and it tells you, plainly, that the answer is not in the indexed documents. That is not a limitation. That is the feature.
Bonus: HyDE, the technique that sounds like a bug
There is a quiet asymmetry in dense retrieval. You embed a question and search against embeddings of answers. Look at what those look like:
question: "How much does exercise lower blood pressure?"
passage : "Systolic blood pressure decreased by a mean of 6.9 mmHg
across 89 trials."
Almost no shared vocabulary. Different grammar, different register. Their embeddings sit further apart than you would like, and the retriever is being asked to bridge a gap nobody trained it to bridge.
HyDE closes it by moving the query into answer-space first. Ask the LLM to write a plausible answer, ignore whether it is true, and embed that.
Yes, you are deliberately generating a hallucination and searching with it. It works because the fake passage is shaped like the real passage, so it lands near it in vector space and drags the true source out of the index.
The hallucination never reaches the user. It is discarded the moment retrieval finishes. Its only job is to have the right shape.
Now the part where everything broke
Six clean layers. Beautiful architecture. Then I ran pip install -r requirements.txt.
Act 1: the DLL that would not load
ImportError: DLL load failed while importing onnxruntime_pybind11_state
ChromaDB imports onnxruntime at module load for a default embedding function I never use, because my embeddings come from Gemini. Did not matter. The import has to succeed for ChromaDB to import at all, and the whole app died at line one.
Pinning onnxruntime<1.20 fixed it. Which immediately caused:
Act 2: the fix that broke something else
opentelemetry-proto 1.42.1 requires protobuf<7.0,
but you have protobuf 7.35.1 which is incompatible.
Installing onnxruntime had pulled the newest protobuf, which broke ChromaDB's telemetry instrumentation. Pin protobuf. Move on.
Act 3: the conflict with no solution
This is the one I want to talk about.
llama-index 0.14.23 requires llama-index-llms-openai>=0.7.0
llama-index-llms-openai-like 0.5.3 requires llama-index-llms-openai<0.6
Read those two lines again. Package A needs version 7 or higher. Package B needs below version 6. Both are in my tree because llama-index-llms-groq depends on openai-like.
There is no version of anything that satisfies both. I watched pip ping-pong between them four times, each install "fixing" the conflict by recreating the other half of it.
I nearly downgraded the whole framework. Then I actually looked at what that package was doing for me.
One thing. One single call:
Settings.llm.stream_complete(prompt)
That was it. An entire branch of dependency hell to wrap an HTTP request I could make myself. And I was already using the official Groq SDK directly in four other files: guardrails, HyDE, CRAG, and contextual chunking.
So I deleted the dependency:
stream = _get_llm().chat.completions.create(
model=_LLM_MODEL,
messages=[{"role": "user", "content": prompt}],
temperature=0.1,
stream=True,
)
for chunk in stream:
token = chunk.choices[0].delta.content or ""
if token:
full_answer += token
yield token, None
Conflict gone. Permanently. Fewer packages, less abstraction, more control over the stream.
The lesson I actually took from this: when a dependency conflict has no solution, stop solving it and start asking what the dependency was buying you. Sometimes it is a framework. Sometimes it is a wrapper around six lines you can write yourself, and you have been renting a dependency tree to avoid typing them.
Act 4: the error that improved the architecture
Last one, and it is my favourite, because the error made the system genuinely better.
[reranker] local model unavailable
([WinError 1114] A dynamic link library (DLL) initialization
routine failed. Error loading torch\lib\c10.dll)
PyTorch would not load on Windows. Which meant my cross-encoder would not load. Which meant the single highest-impact stage in the entire pipeline was dead, on a machine where nothing was actually wrong with my code.
I could have fought the DLL. Instead I asked a better question: why is my most important stage sitting behind a 2GB native dependency with a known platform failure mode?
So I added a third tier. The reranker now picks the best backend available at boot:
- Cohere rerank-v3.5 if an API key is set. Best accuracy money can buy.
- Local BGE cross-encoder via sentence-transformers. Free, offline, fast.
- Groq LLM-as-reranker. Pure HTTP. No torch, no model download, no new key.
Tier 3 scores each candidate with a small parallel LLM call:
with ThreadPoolExecutor(max_workers=8) as pool:
scores = list(pool.map(
lambda c: _groq_score_one(query, c.get("text", "")),
candidates,
))
Latency here is entirely network-bound, so 20 sequential calls would add seconds while 20 concurrent calls cost roughly one. It is less precise than a real cross-encoder and burns more API quota, but it works on any machine with an internet connection.
And if all three fail, the pipeline falls back to RRF order rather than crashing, and the trace records which backend actually ran.
That is the real lesson of the whole project. A DLL error on one laptop should degrade one stage, not delete a feature. Enterprise systems are not defined by having the best possible component. They are defined by what happens when the best possible component is unavailable.
You cannot improve what you cannot see
Every stage is wrapped in a span:
with trace.span("5_rerank") as span:
documents, backend = rerank(question, candidates, top_n=5)
span.attributes["backend"] = get_reranker_name()
span.attributes["promoted"] = len(promoted)
span.attributes["biggest_jump"] = max(...)
Which produces a per-query timeline showing exactly where the milliseconds went:
| Stage | Typical |
|---|---|
| HyDE | 400 to 800ms |
| Vector search | 200 to 400ms |
| BM25 | under 10ms |
| RRF fusion | under 5ms |
| Rerank | 150 to 400ms |
| CRAG grading | 300 to 600ms |
| Generation | 1 to 3s |
No LangSmith account, no Phoenix instance. A ring buffer, a JSONL file, and a dashboard page. About 200 lines.
Two things it immediately taught me that I would never have guessed:
BM25 is essentially free. Under 10 milliseconds against 14 chunks. I had been mentally budgeting it as a real cost. It is a rounding error, and it is catching exact-term matches that vector search misses entirely.
HyDE is my second most expensive stage. It costs more than reranking. That is a genuine trade-off I now get to make deliberately, with a number in front of me, instead of guessing.
The thing I am proudest of
I put the pipeline toggles in the product.
There is a panel above the chat input with four switches: HyDE, hybrid search, reranking, CRAG. Turn reranking off, ask the same question again, and watch the source ordering degrade in real time.
That is an ablation study. In the UI. Running on the user's own documents.
class PipelineSettings(BaseModel):
use_hyde: bool = True
use_hybrid: bool = True
use_rerank: bool = True
use_crag: bool = True
Four booleans on the request body. Anyone can now verify my claims about this architecture on their own data instead of taking my word for it, and I can debug a bad answer by bisecting the pipeline live rather than editing code and restarting.
Where it ended up
1. HyDE write a hypothetical answer, embed that instead
2. Dense retrieval vector search, top 20
3. Sparse retrieval BM25 keyword search, top 20
4. RRF fusion merge both lists by rank, not score
5. Cross-encoder rerank the top 30, keep the best 5
6. CRAG grading score relevance, rewrite and retry, or refuse
7. Generation stream the answer
8. Guardrails confidence score from lexical overlap plus LLM check
From three steps to eight. Roughly 3 to 5 seconds per query instead of 1 to 2.
Worth every millisecond, because the system can now do the one thing it could not do before: notice that it does not know.
If you take one thing from this post
Not the six techniques. Those are Googleable and half of them will have better versions in a year.
Take this: your RAG system's most dangerous state is not being wrong. It is being wrong with no mechanism to notice.
Retrieval quality, reranking, hybrid search, all of it exists to reduce how often you land in that state. But reducing it is not the same as detecting it. Until you build something that can grade its own retrieval and refuse, you have not built a system that can fail safely. You have built one that fails silently, which in a medical context is considerably worse.
Try it
Everything here is open source and runs locally.
github.com/Sowaiba-01/ClinicaQuery-AI
Stack: Next.js 15, FastAPI, ChromaDB, rank-bm25, Groq (Llama 3.3 70B), Gemini embeddings.
Papers worth reading if this interested you:
- Contextual Retrieval (Anthropic, 2024)
- Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, Gao et al. 2022)
- Corrective Retrieval Augmented Generation (Yan et al. 2024)
- Reciprocal Rank Fusion (Cormack et al., SIGIR 2009)
If this was useful
⭐ Star the repo so I know it landed
Follow me for the next one. I am working on GraphRAG and multi-document reasoning
Tell me in the comments what your RAG system does when it does not know the answer. I genuinely want to know how other people solved this.
And if you have ever shipped a RAG app that hallucinated confidently at a real user, say so. I would like this comment section to be the place people admit that.
Top comments (0)