DEV Community

wellallyTech
wellallyTech

Posted on

Talk to Your Medical History: Building a Personal EHR RAG with Milvus and Unstructured.io 🩺

#ai #python #rag #vectordatabase

We’ve all been there: digging through a mountain of crumpled hospital printouts, blurry scans, and nested PDFs just to find out what that specific blood test result was three years ago. Medical data is messy, unstructured, and—let's be honest—doctor's handwriting is the final boss of OCR.

In this tutorial, we are building a Personal Electronic Health Record (EHR) RAG system. We will transform those chaotic PDFs and scanned images into a searchable, intelligent knowledge base. By using a Vector Database like Milvus and powerful document partitioning, we'll achieve a seamless Personal Electronic Health Record experience where you can literally "talk" to your medical history. 🚀

Why is this hard? (The Problem)

Standard RAG (Retrieval-Augmented Generation) often fails on medical documents because:

  1. Layouts are complex: Labs use tables; prescriptions use weird grids.
  2. Scanned Images: Many records are just photos of paper.
  3. Semantic Context: A "normal" result in 2021 might be "concerning" given a 2024 diagnosis.

Our solution combines Unstructured.io for "intelligent" PDF shredding, BGE Embeddings for high-precision medical semantics, and Milvus for industrial-grade vector storage.

The Architecture 🏗️

Before we dive into the code, let's look at the data flow. We are moving from raw pixels to structured semantic insights.

graph TD
    A[Raw Medical PDFs/Scans] --> B{Unstructured.io}
    B -->|OCR & Partitioning| C[Clean Text Chunks]
    B -->|Table Extraction| D[Structured Data]
    C & D --> E[BGE Embeddings Model]
    E --> F[(Milvus Vector DB)]
    G[User Query: 'What was my glucose trend?'] --> H[Query Embedding]
    H --> I[Milvus Similarity Search]
    I --> J[LlamaIndex Context Synthesis]
    J --> K[LLM Response]
Enter fullscreen mode Exit fullscreen mode

Prerequisites 🛠️

To follow along, you'll need:

  • Milvus: (Local via Docker or Zilliz Cloud)
  • Unstructured API Key: For high-quality OCR partitioning.
  • Python Stack: pymilvus, llama-index, unstructured, sentence-transformers.

Step 1: Shredding the Chaos with Unstructured.io

Standard PDF loaders often break tables or ignore images. Unstructured.io treats a document like a collection of elements (Title, NarrativeText, Table).

from unstructured.partition.pdf import partition_pdf

# This handles OCR and Table Extraction automatically!
elements = partition_pdf(
    filename="medical_report_2023.pdf",
    strategy="hi_res",           # Best for scanned documents
    extract_images_in_pdf=False,
    infer_table_structure=True,  # Keeps those lab results organized
    chunking_strategy="by_title",# Maintains semantic grouping
    max_characters=1000,
    combine_text_under_n_chars=200
)

# Convert to LlamaIndex-ready TextNodes
from llama_index.core.schema import TextNode

nodes = []
for el in elements:
    nodes.append(TextNode(text=el.to_dict().get("text"), metadata=el.to_dict().get("metadata")))
Enter fullscreen mode Exit fullscreen mode

Step 2: Embedding with BGE & Storing in Milvus

For medical data, we need high-dimensional accuracy. BGE-M3 is currently a top-tier choice for retrieval. We'll store these in Milvus, which allows us to scale as our medical history grows over decades. 🥑

from llama_index.vector_stores.milvus import MilvusVectorStore
from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

# Initialize Milvus (The powerhouse)
vector_store = MilvusVectorStore(
    uri="http://localhost:19530", 
    collection_name="personal_ehr", 
    dim=1024  # BGE-Large dimension
)

# Set up the embedding model
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-large-en-v1.5")

storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex(nodes, storage_context=storage_context, embed_model=embed_model)
Enter fullscreen mode Exit fullscreen mode

Step 3: The Query Logic

Now we can ask complex questions across multiple documents.

query_engine = index.as_query_engine(similarity_top_k=5)

response = query_engine.query(
    "Compare my cholesterol levels between the 2021 checkup and the 2023 report. Is there an improving trend?"
)

print(f"Medical Assistant: {response}")
Enter fullscreen mode Exit fullscreen mode

Scaling to Production-Ready EHR 🏥

While this local setup is great for a weekend project, building a HIPAA-compliant or production-grade medical AI requires much deeper architectural considerations—specifically regarding data privacy and advanced reranking.

For more production-ready examples and advanced patterns on handling sensitive healthcare data within RAG architectures, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover the nuances of scaling vector search and ensuring data integrity that go beyond the basics of this tutorial.

Why Milvus? ⚡

You might ask: "Why not just use a simple local vector store?"

  • Time-Travel: Milvus handles metadata filtering brilliantly. You can filter by year > 2020 before doing the vector search, making queries lightning-fast.
  • Persistence: Your medical history is for life. Milvus ensures your data is indexed, backed up, and ready even as you add thousands of pages of records.

Conclusion

By combining Unstructured.io's ability to "see" documents with Milvus's ability to "remember" them, we've turned a pile of useless paper into a life-saving personal assistant. No more digging through drawers; just query and find.

Next Steps:

  1. Try adding a Reranker (like Cohere) to improve accuracy on subtle medical terms.
  2. Implement a frontend using Streamlit to upload PDFs via your phone.
  3. Subscribe to the WellAlly technical newsletter for more insights on high-performance AI systems.

Happy coding, and stay healthy! 🩺💻

Top comments (0)