I Vectorized 10 Years of My Apple Health Data: Building a Quantified Self RAG with DuckDB and LlamaIndex

We live in the era of the Quantified Self. Our iPhones and Apple Watches are treasure troves of bio-metric data, capturing everything from our REM cycles to that one time our heart rate spiked during a horror movie in 2016. But there's a problem: that data is trapped in a massive, messy XML file that's about as readable as ancient hieroglyphics.

In this tutorial, we’re going to build a Personal Health Knowledge Base using a RAG (Retrieval-Augmented Generation) pipeline. We will clean a decade's worth of "dirty" Apple Health data, vectorize it, and chat with our health history using local LLMs.

By the end, you'll be able to ask your AI: "How has my average resting heart rate changed since I started drinking matcha instead of coffee?"

The Architecture: From Raw XML to Insights

Handling 10 years of health data requires a robust pipeline. We aren't just shoving text into a database; we're engineering a data flow that ensures privacy and performance.

graph TD
    A[Apple Health Export.xml] --> B[DuckDB: Data Cleaning & Transformation]
    B --> C[LlamaIndex: Document Orchestration]
    C --> D[Qdrant: Vector Storage]
    E[Ollama: Local LLM Inference] --> F[User Query]
    F --> C
    C --> G[Contextual Health Insights]

Prerequisites

To follow along, you'll need the following tech stack:

• DuckDB: For blazing-fast SQL processing on local files.
• LlamaIndex: To orchestrate our RAG workflow.
• Qdrant: Our high-performance vector database.
• Ollama: To run Llama 3 or Mistral locally (privacy first! 🔐).
• Python 3.10+

Step 1: Wrangling the XML Beast with DuckDB

Apple Health exports are notorious for being massive XML files. Trying to parse a 1GB XML file with standard libraries will make your RAM cry. Enter DuckDB. It’s the "Swiss Army Knife" of data engineering, allowing us to query files directly.

import duckdb

# Connect to an in-memory database
con = duckdb.connect()

# Use DuckDB's magic to read the XML (converted to JSON or CSV for easier handling)
# Pro-tip: Convert export.xml to a structured format first
def clean_health_data(input_path):
    print("🧹 Cleaning data with DuckDB...")
    con.execute(f"""
        CREATE TABLE health_records AS 
        SELECT 
            type, 
            unit, 
            value, 
            startDate as timestamp,
            date_part('year', CAST(startDate AS TIMESTAMP)) as year
        FROM read_csv_auto('{input_path}')
        WHERE value IS NOT NULL
    """)

    # Export to a clean format for LlamaIndex
    con.execute("COPY health_records TO 'clean_health_data.parquet' (FORMAT PARQUET)")
    print("✅ Data cleaned and saved to Parquet!")

# clean_health_data('apple_health_export.csv')

Step 2: Setting up the Vector Store (Qdrant)

We need a place to store our "embeddings" (the mathematical representation of our health trends). Qdrant is perfect for this because it's fast and provides great filtering capabilities.

from llama_index.vector_stores.qdrant import QdrantVectorStore
import qdrant_client

# Initialize Qdrant Client
client = qdrant_client.QdrantClient(path="./qdrant_data")

vector_store = QdrantVectorStore(
    client=client, 
    collection_name="apple_health_vdb"
)

Step 3: Local LLM with Ollama & LlamaIndex

Since health data is highly sensitive, we don't want to send it to an external API. We'll use Ollama to keep everything on our local machine.

from llama_index.llms.ollama import Ollama
from llama_index.core import Settings, VectorStoreIndex, StorageContext
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

# Use local Llama 3 via Ollama
Settings.llm = Ollama(model="llama3", request_timeout=120.0)
# Use a local embedding model
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")

def build_index(nodes):
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    index = VectorStoreIndex(
        nodes, 
        storage_context=storage_context,
        show_progress=True
    )
    return index

The "Official" Way to Build 🥑

While this setup is fantastic for a local project, scaling RAG systems for production requires more nuance—especially regarding data privacy and advanced retrieval strategies like Hybrid Search or Small-to-Big Retrieval.

If you're looking for more production-ready examples and advanced patterns in AI data engineering, I highly recommend checking out the WellAlly Blog. It's a goldmine for developers looking to bridge the gap between "it works on my machine" and "it works for a million users."

Step 4: Querying Your History 💬

Now for the magic. We can now ask questions about our decade of data using natural language.

# Assuming 'index' is our VectorStoreIndex from earlier
query_engine = index.as_query_engine()

response = query_engine.query(
    "Analyze my activity levels in 2021. Were there any significant dips in my daily step count, "
    "and do they correlate with any specific time of the year?"
)

print(f"🤖 Health Assistant: {response}")

Why this matters

By combining DuckDB's processing power with LlamaIndex's orchestration, we've turned a pile of "dirty" XML records into a searchable, intelligent knowledge base. This is the essence of the Quantified Self: moving beyond just collecting data to actually understanding it.

Conclusion

Vectorizing 10 years of history sounds daunting, but with the right tools, it's a weekend project. You've now built a system that:

1. Ingests raw, messy health data.
2. Cleans and transforms it using SQL (DuckDB).
3. Indexes it for semantic search (Qdrant).
4. Answers questions privately (Ollama).

What will you ask your health data first? Let me know in the comments below! 👇

---

For more deep dives into RAG architecture and AI engineering, don't forget to visit wellally.tech/blog. Happy coding! 🚀