wellallyTech

Posted on Aug 24

From Silos to Insights: Building a Personal Health Data Lake with Airbyte, BigQuery, and Grafana 🚀

#ai #python #react #discuss

Have you ever looked at your wrist and realized your health data is scattered across three different ecosystems? Your sleep is trapped in an Oura ring, your steps are locked in Google Health Connect, and your heart rate variability is floating somewhere in a proprietary cloud.

As developers, we don't just want apps; we want data ownership. In this guide, we’re going to build a robust Personal Health Data Lake using a modern ETL Automation Pipeline. We will leverage Airbyte for seamless integration, Google Cloud Functions for custom extraction, BigQuery as our scalable warehouse, and Grafana for that sweet, sweet visualization.

By the end of this post, you'll have a production-grade Health Data Integration system that turns fragmented metrics into actionable longitudinal insights.

The Architecture: How the Data Flows 🛠️

Before we dive into the code, let's look at the "big picture." We need to pull data from diverse APIs, normalize it, and store it in a way that allows for complex SQL queries.

graph TD
    A[Oura Ring API] -->|Webhook/Polling| B(Google Cloud Functions)
    C[Google Health Connect] -->|Android Sync| B
    B -->|JSON Raw Data| D{Airbyte}
    D -->|CDC / Incremental Sync| E[Google BigQuery]
    E -->|SQL Queries| F[Grafana Dashboard]

    style B fill:#f96,stroke:#333,stroke-width:2px
    style D fill:#6c5ce7,stroke:#333,stroke-width:2px
    style E fill:#4285F4,stroke:#333,stroke-width:2px

Prerequisites

To follow this tutorial, you'll need:

Airbyte (Open Source or Cloud)
A Google Cloud Project with Billing enabled (for BigQuery and GCF)
An Oura Personal Access Token
Basic knowledge of Python and SQL

Step 1: The Extraction Bridge (Google Cloud Functions)

Since Google Health Connect lives on-device (Android), and Oura provides a REST API, we need a "bridge" to aggregate this data before pushing it to our lake. We'll use a Google Cloud Function as a lightweight scraper/adapter.

import functions_framework
import requests
import json
from google.cloud import bigquery

# Configuration
OURA_API_URL = "https://api.ouraring.com/v2/usercollection/sleep"
HEADERS = {'Authorization': 'Bearer YOUR_OURA_TOKEN'}

@functions_framework.http
def fetch_health_metrics(request):
    """
    HTTP Cloud Function to fetch Oura data and stage it for Airbyte/BigQuery.
    """
    response = requests.get(OURA_API_URL, headers=HEADERS)

    if response.status_code == 200:
        data = response.json().get('data', [])
        # In a real scenario, you'd push this to a GCS bucket 
        # for Airbyte to pick up, or write directly to BigQuery
        return json.dumps({"status": "success", "count": len(data)}), 200
    else:
        return json.dumps({"status": "error", "message": "API Fail"}), 500

Step 2: Automating with Airbyte 🥑

Airbyte is the secret sauce here. Instead of writing custom cron jobs to handle API rate limits and schema changes, we use Airbyte’s BigQuery Destination connector.

Source: Set up a "Custom HTTP Source" or use the "S3/GCS" source if your Cloud Function saves files.
Destination: Select BigQuery. Provide your dataset_id and service account credentials.
Sync Frequency: Set it to "Every 24 hours" (Health data doesn't change by the millisecond!).

Airbyte ensures that if the Oura API adds a new field (like respiratory_rate_v2), it gets automatically appended to your BigQuery table without breaking your pipeline.

Step 3: Leveling Up Your Data Engineering

Building a basic pipeline is easy, but handling data quality and anomaly detection (e.g., "Why did my sleep score drop to 0?") requires more advanced patterns.

💡 Pro-Tip: For more production-ready examples and advanced data modeling patterns for health metrics, check out the detailed technical deep-dives over at WellAlly Blog. They cover everything from DBT transformations to real-time bio-metric monitoring.

Step 4: Visualizing in Grafana 📊

Now that our data is sitting neatly in BigQuery, we can connect Grafana.

Add Google BigQuery as a Data Source in Grafana.
Write a simple SQL query to track your Readiness Score over time:

SELECT
  timestamp_trunc(day, DAY) as time,
  AVG(readiness_score) as avg_readiness
FROM 
  `your_project.health_data.oura_sleep`
WHERE 
  day > DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY 1
ORDER BY 1

In Grafana, use the Time Series panel. You can now overlay your "Steps" (from Health Connect) on top of your "Sleep Score" (from Oura) to see if that late-night run actually ruined your recovery!

Conclusion: Data Sovereignty is Key

By building your own Personal Health Data Lake, you are no longer a passenger in your own health journey. You have the raw data to run your own correlations and the infrastructure to scale as you add more devices (like CGMs or smart scales).

What’s next?

Try adding dbt to transform raw JSON into clean fact tables.
Set up Slack alerts when your resting heart rate exceeds its 7-day moving average.

Are you building something similar? Drop a comment below or share your dashboard screenshots! 👇

For more advanced tutorials on health tech and data engineering, visit wellally.tech/blog. 🥑💻

DEV Community