DEV Community

wellallyTech
wellallyTech

Posted on

Quantified-Self RAG: Turning 5 Years of Apple Health XML into a Personal Health AI

#rag #python #llamaindex #ai

Have you ever tried to export your Apple Health data? You’re met with a monolithic, multi-gigabyte export.xml file that looks like a digital archeology project. For those of us in the Quantified Self movement, this data is a goldmine, but querying it is a nightmare. Today, we are going to build a Personal Health Knowledge Base using Retrieval-Augmented Generation (RAG) to turn those messy XML logs into a conversational AI.

By leveraging LlamaIndex for orchestration, Qdrant as our vector database, and DuckDB for lightning-fast data processing, we can move beyond static charts. We will implement a pipeline that allows you to ask, "How has my deep sleep quality trended over the last three years compared to my caffeine intake?" and get a data-backed answer. This approach to personal health data RAG and vectorized health analytics is the future of proactive wellness. 🚀

The Architecture 🏗️

Processing 5 years of health data requires more than just a simple script. We need an ETL (Extract, Transform, Load) pipeline that can handle nested XML, flatten it into structured tables, and then index it for semantic search.

graph TD
    A[Apple Health export.xml] --> B{DuckDB / Pandas}
    B -->|Flatten & Clean| C[Structured Parquet/CSV]
    C --> D[LlamaIndex Document Ingestion]
    D --> E[Embedding Model: OpenAI/HuggingFace]
    E --> F[(Qdrant Vector DB)]
    G[User Query] --> H[LlamaIndex Query Engine]
    H --> F
    F -->|Context Retrieval| H
    H --> I[Final Personalized Health Insight]
Enter fullscreen mode Exit fullscreen mode

Prerequisites 🛠️

Before we dive in, ensure you have your export.xml from your iPhone (Health App -> Profile -> Export All Health Data). You’ll also need:

  • LlamaIndex: The framework for connecting LLMs to your data.
  • Qdrant: A high-performance vector search engine.
  • Pandas & DuckDB: For high-speed XML parsing and relational querying. 
pip install llama-index qdrant-client pandas duckdb
Enter fullscreen mode Exit fullscreen mode

Step 1: Taming the XML Beast with DuckDB 🦆

Apple Health XML is notoriously nested. While Pandas is great, DuckDB allows us to treat the XML/Parquet files like a SQL database, which is much more memory-efficient for 500MB+ files.

import pandas as pd
import duckdb

# Load the XML (Pro tip: Convert to CSV/Parquet first for speed)
def parse_health_data(xml_path):
    # This is a simplified logic to extract 'Record' tags
    # In reality, you'd use an iterative parser like lxml
    print("💎 Extracting records from XML...")

    # We use DuckDB to handle the structured extraction
    con = duckdb.connect()
    con.execute(f"""
        CREATE TABLE health_records AS 
        SELECT * FROM read_csv_auto('processed_health_data.csv')
    """)

    df = con.execute("SELECT type, value, unit, startDate FROM health_records").df()
    return df

# Example: Filtering for Sleep and Heart Rate
# df_filtered = df[df['type'].str.contains('SleepAnalysis|HeartRate')]
Enter fullscreen mode Exit fullscreen mode

Step 2: Vectorizing with LlamaIndex and Qdrant 🔍

Once the data is cleaned, we need to turn these rows into "Nodes" that an LLM can understand. We’ll use Qdrant to store these embeddings so we don't have to re-index every time we ask a question.

from llama_index.core import VectorStoreIndex, StorageContext, Document
from llama_index.vector_stores.qdrant import QdrantVectorStore
import qdrant_client

# 1. Initialize Qdrant Client
client = qdrant_client.QdrantClient(path="./qdrant_health_db")

# 2. Setup Vector Store
vector_store = QdrantVectorStore(client=client, collection_name="apple_health_logs")
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# 3. Create Documents from your Health Data
documents = [
    Document(
        text=f"On {row['startDate']}, my {row['type']} was {row['value']} {row['unit']}.",
        metadata={"date": row['startDate'], "type": row['type']}
    ) for _, row in df.iterrows()
]

# 4. Build the Index
index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context
)
Enter fullscreen mode Exit fullscreen mode

Step 3: Natural Language Health Queries 💬

Now for the magic. Instead of looking at a tiny graph on your phone, you can query your entire history.

query_engine = index.as_query_engine()

response = query_engine.query(
    "Analyze my resting heart rate trends over the last 3 summers. "
    "Is there a correlation with higher temperatures or activity levels?"
)

print(f"🥑 AI Health Consultant: {response}")
Enter fullscreen mode Exit fullscreen mode

The "Official" Way to Build Production RAG 🥑

While this DIY project is great for personal use, scaling RAG systems for production requires handling data privacy, complex metadata filtering, and high-concurrency retrieval.

For more production-ready examples and advanced architectural patterns on how to handle sensitive data in RAG pipelines, I highly recommend checking out the deep dives over at WellAlly Tech Blog. They cover everything from hybrid search strategies to optimizing LLM latency in health-tech contexts.

Conclusion & Next Steps

Building a Quantified-Self RAG pipeline transforms your "dead" data into an active advisor. By combining LlamaIndex for its robust query engine and Qdrant for efficient retrieval, you've essentially built a private, local health consultant.

What's next?

  1. Time-Series Augmentation: Use DuckDB to calculate weekly averages before sending data to the LLM.
  2. Privacy First: Use a local LLM (like Llama 3 via Ollama) to keep your health data 100% offline.

Are you tracking your health data? Drop a comment below or share your export.xml horror stories! 👇

Top comments (0)