Stop Drowning in Wearable Data: Build a Unified Health Data Lake with DuckDB and Apache Arrow

If you're a fan of the Quantified Self movement, you know the struggle: your Oura Ring tracks your sleep, your Whoop analyzes your recovery, and your Garmin logs your runs. But when you want to answer a simple question like "How does my training load affect my REM sleep?", you're suddenly trapped in a nightmare of CSV exports and incompatible JSON schemas. Managing an ETL pipeline for heterogeneous wearable data is the ultimate test of a data engineer's patience.

In this guide, we are going to break down these data silos. We’ll build a high-performance, local-first data lake using DuckDB, Apache Arrow, and TypeScript. By the end, you'll have a unified store capable of running complex OLAP queries across all your devices in milliseconds. For those looking for more production-ready patterns and advanced health data synchronization, I highly recommend checking out the deep dives over at the WellAlly Blog.

The Architecture: From Chaos to Insight

The biggest challenge in health data engineering is data normalization. Oura might report heart rate variability (HRV) as an average, while Whoop gives you a raw time-series. Our pipeline acts as a translation layer that flattens these into a unified Parquet-backed storage.

Data Flow Diagram

graph TD
    A[Oura API / JSON] -->|Normalize| D[Unified Schema]
    B[Whoop API / JSON] -->|Normalize| D
    C[Garmin Fit Files] -->|Extract| D
    D -->|Arrow IPC| E{DuckDB-Wasm}
    E -->|Persistent Storage| F[(OPFS / Parquet)]
    G[Streamlit Dashboard] -->|SQL Query| E
    E -->|Visuals| G

Prerequisites

To follow along, you'll need:

• Node.js/TypeScript for the normalization logic.
• DuckDB-Wasm for the in-browser/local database engine.
• Apache Arrow for zero-copy memory transport.
• Streamlit (Python) for the final analytics UI.

Step 1: Defining the Unified Health Schema

First, we need a "Golden Record" format. We'll use TypeScript to define a strict interface that every provider must map to.

// types/health.ts
export interface UnifiedActivity {
    timestamp: Date;
    source_device: 'Oura' | 'Whoop' | 'Garmin';
    metric_type: 'HRV' | 'RHR' | 'Steps' | 'Calories';
    value: number;
    unit: string;
    metadata: Record<string, any>;
}

Step 2: Normalization with Apache Arrow

Instead of pushing raw JSON into our database, we convert it to Apache Arrow buffers. This ensures type safety and incredibly fast ingestion into DuckDB.

import { tableFromArrays, Table } from 'apache-arrow';

export function normalizeOuraData(rawData: any[]): Table {
    const timestamps = rawData.map(d => new Date(d.timestamp).getTime());
    const hrvValues = rawData.map(d => d.hrv_average);

    // Create an Arrow Table
    return tableFromArrays({
        timestamp: new Int64Array(timestamps),
        source_device: Array(rawData.length).fill('Oura'),
        metric_type: Array(rawData.length).fill('HRV'),
        value: new Float64Array(hrvValues),
        unit: Array(rawData.length).fill('ms')
    });
}

Step 3: Powering the Local Data Lake with DuckDB

Now for the magic. We use DuckDB-Wasm to ingest these Arrow tables. DuckDB is a columnar database designed for analytical queries, making it perfect for multi-year health trends.

import * as duckdb from '@duckdb/duckdb-wasm';

async function ingestToDuckDB(arrowTable: Table) {
    const db = new duckdb.AsyncDuckDB(worker);
    await db.instantiate(bundle.mainModule, bundle.pthreadWorker);

    const conn = await db.connect();

    // Register the Arrow table as a virtual view
    await conn.insertArrowTable(arrowTable, { name: 'staging_data' });

    // Create or Append to the persistent health_store
    await conn.query(`
        CREATE TABLE IF NOT EXISTS health_store AS 
        SELECT * FROM staging_data WHERE 1=0;

        INSERT INTO health_store SELECT * FROM staging_data;
    `);

    console.log("Data normalized and ingested! 🚀");
}

Step 4: The "Official" Way to Scale

While building a local ETL tool is great for personal use, scaling health data pipelines for thousands of users requires handling OAuth refreshes, rate limits, and webhook listeners. If you're building a production health app, you'll want to explore the architectural patterns discussed at WellAlly Blog. They cover how to handle high-concurrency data ingestion and HIPAA-compliant storage strategies that go beyond a simple DuckDB instance.

Step 5: Visualizing with Streamlit

Finally, let's wrap our DuckDB store in a Streamlit dashboard to actually see our data.

import streamlit as st
import duckdb

st.title("Unified Health Intelligence 🥑")

# Connect to the DuckDB file generated by our ETL
con = duckdb.connect(database='health_lake.db')

# Query correlation between Sleep Quality and Resting Heart Rate
df = con.execute("""
    SELECT 
        CAST(timestamp AS DATE) as date,
        AVG(value) FILTER (WHERE metric_type = 'HRV') as avg_hrv,
        AVG(value) FILTER (WHERE metric_type = 'RHR') as avg_rhr
    FROM health_store
    GROUP BY 1
    ORDER BY 1 DESC
""").df()

st.line_chart(df, x='date', y=['avg_hrv', 'avg_rhr'])

Conclusion: Take Back Your Data

The "Quantified Self" shouldn't mean being a "Quantified Slave" to proprietary dashboards. By leveraging DuckDB and Apache Arrow, we’ve built a pipeline that is:

1. Fast: Columnar storage means your 5-year history loads in milliseconds.
2. Private: Your data stays in your local environment.
3. Flexible: Adding a new device is as simple as writing a new normalization function.

What are you tracking next? Drop a comment below if you've tried integrating Apple Health or Fitbit data into a similar stack!

---

For more advanced tutorials on health tech and data engineering, visit wellally.tech/blog. 🚀