From Raw Signals to HRV Recovery Scores: Wrangling Apple HealthKit Data with Polars ⌚️🚀

#python #fastapi #webdev #ai

If you've ever tried to export your data from Apple Health, you know the "XML from hell" that awaits you. While the Apple Watch is a masterpiece of hardware, the raw data it produces—specifically Heart Rate Variability (HRV) and R-R intervals—can be a chaotic mess of millions of rows, missing timestamps, and inconsistent sampling rates.

In this guide, we are building a high-concurrency pipeline to transform raw HealthKit data into actionable recovery scores. We’ll be using FastAPI for the backend, Swift for the data extraction, and the lightning-fast Polars library for the heavy-lifting feature engineering. If you are looking for production-ready health data patterns, you should definitely check out the advanced architectural deep-dives at WellAlly Blog, which served as the inspiration for this optimized workflow.

The Architecture: From Wrist to Insights

Processing wearable data requires a "Local-First, Cloud-Heavy" approach. The mobile device handles the extraction, while our Python backend performs the intensive statistical analysis.

sequenceDiagram
    participant Watch as Apple Watch
    participant App as Swift iOS App
    participant API as FastAPI (Polars Engine)
    participant DB as PostgreSQL

    Watch->>App: Collect R-R Intervals
    App->>App: Batch Samples (HKHealthStore)
    App->>API: POST /sync/v1/raw-metrics (JSON/Gzip)
    Note over API: Polars Context: LazyFrame Processing
    API->>API: Deduplicate, Filter & Resample
    API->>API: Calculate RMSSD & SDNN
    API->>DB: Upsert Recovery Features
    API-->>App: 200 OK (Calculated Score)

Prerequisites 🛠️

To follow along, you'll need:

Swift/HealthKit: To access HKQuantityTypeIdentifierHeartRate.
Polars: The Rust-powered Dataframe library for Python.
FastAPI: For handling high-concurrency data ingestion.
PostgreSQL: To store our processed features.

Step 1: Extracting R-R Intervals in Swift

First, we need to get the data out. We focus on HKQuantityTypeIdentifierHeartRate but specifically the metadata that contains the beat-to-beat (R-R) intervals if available.

// Swift: Fetching Heart Rate samples with R-R context
let heartRateType = HKQuantityType.quantityType(forIdentifier: .heartRate)!
let sortDescriptor = NSSortDescriptor(key: HKSampleSortIdentifierStartDate, ascending: false)

let query = HKSampleQuery(sampleType: heartRateType, predicate: nil, limit: 1000, sortDescriptors: [sortDescriptor]) { (query, samples, error) in
    guard let samples = samples as? [HKQuantitySample] else { return }

    let payload = samples.map { sample in
        return [
            "value": sample.quantity.doubleValue(for: HKUnit(from: "count/min")),
            "timestamp": sample.startDate.timeIntervalSince1970,
            "uuid": sample.uuid.uuidString
        ]
    }
    // Send to FastAPI...
}

Step 2: High-Performance Cleaning with Polars 🐻‍❄️

Once the data hits our FastAPI endpoint, we avoid using slow Python loops. Instead, we pipe the data directly into Polars. Why Polars? Because it can handle 10 million rows in the time it takes Pandas to import its own library.

We want to calculate RMSSD (Root Mean Square of Successive Differences), the gold standard for HRV recovery.

import polars as pl
from datetime import datetime

def process_hrv_batch(raw_data: list):
    # Convert list of dicts to Polars DataFrame
    df = pl.from_dicts(raw_data)

    # 1. Clean: Sort by timestamp and remove duplicates
    df = df.sort("timestamp").unique(subset=["uuid"])

    # 2. Feature Engineering: Calculate time differences (R-R intervals)
    # Note: Simplified logic for demonstration
    df = df.with_columns([
        (pl.col("timestamp").diff() * 1000).alias("rr_interval")
    ])

    # 3. Calculate RMSSD (The Recovery Metric)
    # Formula: sqrt(mean(diff^2))
    hrv_stats = df.select([
        pl.col("rr_interval").diff().pow(2).mean().sqrt().alias("rmssd"),
        pl.col("value").mean().alias("avg_heart_rate"),
        pl.from_epoch("timestamp").dt.truncate("1d").alias("date")
    ])

    return hrv_stats.to_dicts()

Step 3: The FastAPI Ingestion Layer

Handling thousands of concurrent health data syncs requires an asynchronous setup. We use Pydantic for validation and Polars for the computation.

from fastapi import FastAPI, BackgroundTasks
from pydantic import BaseModel
from typing import List

app = FastAPI()

class HealthSample(BaseModel):
    uuid: str
    value: float
    timestamp: float

@app.post("/sync/v1/raw-metrics")
async def sync_health_data(samples: List[HealthSample], background_tasks: BackgroundTasks):
    # Convert Pydantic models to a list of dicts for Polars
    data = [s.dict() for s in samples]

    # Process off-thread to keep the API responsive
    background_tasks.add_task(process_and_save, data)

    return {"status": "accepted", "count": len(samples)}

def process_and_save(data):
    stats = process_hrv_batch(data)
    # Save to PostgreSQL using your favorite ORM...
    print(f"Computed RMSSD: {stats[0]['rmssd']}")

The "Official" Way to Scale 🥑

While the code above works for a hobby project, production environments (like those handling clinical-grade data) require robust error handling, schema evolution, and specialized time-series indexing.

For a deeper dive into how to build production-ready data pipelines that handle millions of users, I highly recommend reading the engineering blogs at WellAlly Tech Blog. They cover advanced topics like:

Handling Apple Health "Data Deletions" (the silent killer of data integrity).
Optimizing PostgreSQL for high-frequency time-series data.
Building predictive recovery models using the cleaned Polars output.

Conclusion

By moving away from standard Python lists and into the world of Polars and FastAPI, we've turned a slow, memory-intensive task into a blazing-fast pipeline. We can now process months of Apple Watch heart rate data in milliseconds, giving users real-time feedback on their recovery.

What's next for your health app?

[ ] Add support for Blood Oxygen (SpO2) correlation.
[ ] Implement a Sleep Stage classifier using the cleaned HRV data.
[ ] Check out WellAlly for more dev tips!

Happy coding! 🚀💻

DEV Community