Curing LLM Hallucinations: Building a Production-Grade Medical RAG with PubMed and Hybrid Search

Ever asked a general-purpose LLM for a specific clinical dosage or the latest treatment protocol for a rare condition? If you have, you’ve likely encountered the "Confidence Gap"—where the model provides an answer that sounds incredibly professional but is, in fact, dangerously wrong.

In the world of Medical RAG (Retrieval-Augmented Generation), hallucinations aren't just bugs; they are liabilities. To build a reliable Clinical Decision Support system, we need more than just a vector database. We need a multi-layered approach combining real-time PubMed API integration, Hybrid Search (BM25 + Vector), and rigorous data engineering. In this guide, we’ll dive into building an advanced RAG pipeline that solves knowledge staleness and accuracy issues using LlamaIndex, Pinecone, and Elasticsearch.

For those looking to scale these patterns into production environments, I’ve found that the architectural deep-dives at WellAlly Tech Blog provide excellent supplementary material on maintaining data integrity in high-stakes AI applications. 🩺

---

The Architecture: Why "Vector Only" Fails in Medicine

In medicine, keywords matter. A vector search might realize that "Myocardial Infarction" is semantically similar to "Heart Attack," but it might fail to distinguish between "Type 1 Diabetes" and "Type 2 Diabetes" if the embedding space isn't granular enough.

Our solution? A Hybrid Retrieval Pipeline. We combine the semantic power of Pinecone with the precise keyword matching of Elasticsearch (BM25), then top it off with real-time fetches from the PubMed API.

System Data Flow

graph TD
    User([User Query]) --> Rewriter[Query Engine / Rewriter]
    Rewriter --> VectorSearch[Pinecone: Semantic Search]
    Rewriter --> KeywordSearch[Elasticsearch: BM25]
    Rewriter --> LiveFetch[PubMed API: Real-time Papers]
    VectorSearch --> Merger[Hybrid Reranker / Reciprocal Rank Fusion]
    KeywordSearch --> Merger
    LiveFetch --> Merger
    Merger --> Context[Ranked Context Window]
    Context --> LLM[GPT-4o / Claude 3.5]
    LLM --> Response[Evidence-Based Answer with Citations]

---

Prerequisites

To follow this tutorial, you'll need:

• LlamaIndex: Our orchestration framework.
• Pinecone: For high-speed vector embeddings.
• Elasticsearch: To handle the BM25 keyword-based retrieval.
• PubMed API Key: To fetch the latest clinical abstracts.

---

Step 1: Setting up the Hybrid Index

We start by defining our storage context. By using both a vector store and a document store (for BM25), we ensure that we don't miss specific medical terminology.

from llama_index.core import StorageContext, VectorStoreIndex, SummaryIndex
from llama_index.vector_stores.pinecone import PineconeVectorStore
from llama_index.retrievers.bm25 import BM25Retriever
from pinecone import Pinecone

# Initialize Pinecone
pc = Pinecone(api_key="YOUR_PINECONE_KEY")
pinecone_index = pc.Index("medical-rag")

# Setup Vector Store
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Assuming 'documents' is a list of LlamaIndex Document objects
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)

# Initialize BM25 for keyword precision
bm25_retriever = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=2)

---

Step 2: Injecting Real-Time PubMed Data

Static databases go out of date the moment a new clinical trial is published. To solve "Knowledge Lag," we implement a tool that queries the PubMed API on the fly.

import requests
from llama_index.core.tools import FunctionTool

def fetch_pubmed_abstracts(query: str, max_results: int = 3):
    """Fetches real-time paper abstracts from PubMed."""
    base_url = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi"
    params = {"db": "pubmed", "term": query, "retmode": "json", "retmax": max_results}

    # 1. Get IDs
    id_resp = requests.get(base_url, params=params).json()
    id_list = id_resp.get("esearchresult", {}).get("idlist", [])

    # 2. Fetch Abstracts (Simplified for this example)
    # In production, use efetch.fcgi to get full text/abstracts
    return f"Fetched {len(id_list)} recent papers regarding {query}."

pubmed_tool = FunctionTool.from_defaults(fn=fetch_pubmed_abstracts)

---

Step 3: The "Doctor-in-the-Loop" Query Engine

Now, we combine everything into a QueryEngine. We use a custom reranker to ensure that the most relevant documents—whether from our local vector store or the live PubMed feed—are prioritized.

from llama_index.core.query_engine import CustomQueryEngine
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.postprocessor import LLMRerank

class MedicalQueryEngine(CustomQueryEngine):
    retriever: BaseRetriever
    reranker: LLMRerank

    def custom_query(self, query_str: str):
        # 1. Retrieve from Hybrid Sources
        nodes = self.retriever.retrieve(query_str)

        # 2. Apply LLM Reranking for Clinical Relevance
        reranked_nodes = self.reranker.postprocess_nodes(nodes, query_str=query_str)

        # 3. Synthesize Final Answer
        response = llm.predict(
            "As a medical assistant, answer the query based ONLY on the context: "
            f"Query: {query_str} \nContext: {reranked_nodes}"
        )
        return response

---

Advanced Optimization: Guardrails and Citations

When building for healthcare, you cannot afford "black box" answers. Every claim must be backed by a source.

Pro-Tip: Use Pydantic programs to force the LLM to output a JSON structure containing answer, source_nodes, and confidence_score.

For more production-ready patterns on structured LLM outputs and RAG evaluation, the experts at WellAlly Tech Blog have curated an excellent series on RAGas (RAG Assessment) and observability that I highly recommend for any senior developer in this space. 🚀

---

Conclusion: The Path to Clinical-Grade AI

Building a RAG system for the medical domain is a journey of balancing Recall (finding all relevant info) and Precision (not hallucinating). By combining Pinecone's semantic search with Elasticsearch's keyword rigor and the PubMed API's real-time updates, you create a system that doesn't just "chat"—it informs.

What's next?

1. Fine-tuning: Consider fine-tuning your embedding model on medical corpuses (like BioBERT).
2. Privacy: Ensure your HIPAA compliance if you're handling patient data (PHI).
3. Human-in-the-loop: Always have a "Report Error" button for clinicians to flag bad retrievals.

Are you building AI for healthcare? Drop a comment below with your biggest challenges! 👇