DEV Community: Beck_Moulton

Skin Health on the Edge: Porting EfficientNet to iOS with CoreML for Instant, Private Analysis

Beck_Moulton — Mon, 05 Oct 2026 00:06:00 +0000

In the world of digital health, privacy isn't just a feature—it’s a requirement. When dealing with sensitive data like skin lesion images, sending every pixel to a cloud server isn't just slow; it's a potential privacy nightmare. 🛡️

By leveraging on-device machine learning, we can achieve sub-100ms latency while keeping user data strictly on their device. In this tutorial, we’ll explore how to take a pre-trained EfficientNet model from PyTorch, convert it using CoreML Tools, and deploy it to an iOS app using the Vision Framework and SwiftUI.

Whether you're building a wellness app or a clinical prototype, mastering CoreML conversion and iOS machine learning is a superpower for any modern developer.

The Architecture: From PyTorch to the Palm of Your Hand

The transition from a research-focused Python environment to a production-ready iOS app involves several optimization steps. We need to ensure our model architecture is compatible with Apple’s Neural Engine (ANE) for maximum performance.

Logic Flow

graph TD
    A[PyTorch Pre-trained EfficientNet] -->|Trace/Script| B(TorchScript)
    B -->|coremltools| C{.mlpackage Model}
    C -->|Bundled| D[Xcode Project]
    D -->|Vision Framework| E[iOS App / ANE]
    F[Camera Feed] -->|VNImageRequestHandler| E
    E -->|Classification| G[UI Update]

    style E fill:#f96,stroke:#333,stroke-width:2px
    style C fill:#007AFF,stroke:#fff,color:#fff

Prerequisites

To follow along, you'll need:

Python 3.9+ with torch, torchvision, and coremltools.
Xcode 15+ and an iOS device (iPhone 12 or newer recommended for Neural Engine tests).
A basic understanding of SwiftUI.

Step 1: Converting EfficientNet to CoreML

Standard PyTorch models use dynamic graphs, but CoreML thrives on static definitions. We’ll use coremltools to bridge the gap.

import torch
import torchvision.models as models
import coremltools as ct

# 1. Load a pre-trained EfficientNet-B0
model = models.efficientnet_b0(pretrained=True)
model.eval()

# 2. Trace the model with a dummy input
# EfficientNet-B0 expects 224x224 images
example_input = torch.rand(1, 3, 224, 224)
traced_model = torch.jit.trace(model, example_input)

# 3. Convert to CoreML (.mlpackage)
classifier_config = ct.ClassifierConfig(['Healthy', 'Melanoma', 'Basal Cell Carcinoma', 'Nevus']) # Simplified classes

coreml_model = ct.convert(
    traced_model,
    inputs=[ct.ImageType(name="colorImage", shape=example_input.shape, scale=1/255.0, bias=[-0.485/0.229, -0.456/0.224, -0.406/0.225])],
    classifier_config=classifier_config,
    minimum_deployment_target=ct.target.iOS17
)

# 4. Save the model
coreml_model.save("SkinLesionClassifier.mlpackage")
print("✅ Model converted and saved!")

Pro Tip: Notice the scale and bias parameters. These are crucial! They perform the ImageNet normalization on-device, saving you from writing manual preprocessing code in Swift.

Step 2: Integrating into iOS with Vision Framework

Once you drop the .mlpackage into Xcode, it automatically generates a Swift class. Now, let’s wrap it in a Vision request to handle the camera buffer.

import Vision
import CoreML

class SkinAnalyzer: ObservableObject {
    @Published var classificationResult: String = "Align lesion in frame"

    func performInference(image: CVPixelBuffer) {
        // 1. Load the generated model class
        guard let config = try? MLModelConfiguration(),
              let coreMLModel = try? SkinLesionClassifier(configuration: config),
              let visionModel = try? VNCoreMLModel(for: coreMLModel.model) else {
            return
        }

        // 2. Create the Request
        let request = VNCoreMLRequest(model: visionModel) { request, error in
            if let results = request.results as? [VNClassificationObservation],
               let topResult = results.first {
                DispatchQueue.main.async {
                    self.classificationResult = "\(topResult.identifier): \(Int(topResult.confidence * 100))%"
                }
            }
        }

        // 3. Set Crop and Scale option to match EfficientNet requirements
        request.imageCropAndScaleOption = .centerCrop

        // 4. Run the request
        let handler = VNImageRequestHandler(cvPixelBuffer: image, options: [:])
        try? handler.perform([request])
    }
}

🚀 The "Official" Way to Optimize AI Deployments

While this tutorial covers the basics of conversion, scaling these systems for high-traffic healthcare platforms requires advanced architectural patterns. We're talking about quantization strategies, A/B testing on-device models, and secure model delivery.

For more production-ready examples and deep-dives into advanced mobile vision patterns, I highly recommend checking out the engineering guides at WellAlly Tech Blog. It's the go-to resource for developers moving from prototypes to full-scale AI products.

Step 3: Low-Latency Camera Feed in SwiftUI

To make this "real-time," we hook into AVFoundation. By using the Vision Framework, we ensure the inference runs on the Apple Neural Engine (ANE), leaving the GPU free for rendering the UI.

Sequence of Real-time Inference

sequenceDiagram
    participant C as Camera Feed
    participant V as Vision Framework
    participant A as Neural Engine (ANE)
    participant U as SwiftUI View

    C->>V: CMSampleBuffer (Video Frame)
    V->>A: Perform VNCoreMLRequest
    Note over A: EfficientNet Inference (<20ms)
    A->>V: Return Probabilities
    V->>U: Update @Published State
    U->>U: Re-render classification label

Conclusion: Privacy-First AI is the Future

By moving our EfficientNet model to the edge, we’ve built a system that:

Respects Privacy: No images ever leave the iPhone.
Works Offline: Essential for remote clinics or areas with poor connectivity.
Feels Instant: No "Loading..." spinners while waiting for a cloud API.

Transitioning from Python-based research to iOS-based deployment is a journey of optimization. Don't stop at just classification—consider adding Saliency Maps to show users why the model made a decision!

What are you building with CoreML? Let me know in the comments below! 👇

If you enjoyed this tutorial, don't forget to ❤️ and 🦄. For advanced AI deployment strategies, visit wellally.tech/blog.

Bye-Bye Server Costs: Building a Real-Time AI Fitness Coach with WebLLM, WebGPU, and MediaPipe

Beck_Moulton — Sun, 04 Oct 2026 00:05:00 +0000

The dream of "Edge AI" has finally arrived in the browser. High server costs for hosting Large Language Models (LLMs) and computer vision models are a massive hurdle for developers. But what if you could run a full-scale vision-to-reasoning pipeline locally? In this tutorial, we are exploring how to leverage WebLLM, WebGPU, and MediaPipe to build a real-time fitness coach that corrects your posture without a single byte of video data ever leaving your machine.

By utilizing real-time posture correction and LLM-on-device techniques, we can achieve millisecond-level latency while maintaining 100% user privacy. This approach uses the power of WebGPU to accelerate Llama-3 directly in the browser, transforming your laptop into a high-performance AI engine. For developers looking to master these Edge AI implementations, this guide provides the blueprint for the next generation of web applications.

The Architecture: Vision meets Reasoning

Before we dive into the code, let's look at the data flow. We aren't just detecting dots on a body; we are passing those spatial relationships to a localized LLM to generate human-like coaching feedback.

graph TD
    A[User Webcam] -->|Video Stream| B(MediaPipe Pose)
    B -->|3D Keypoints| C{Posture Logic Engine}
    C -->|Trigger: 'Squat Too Shallow'| D(WebLLM Context)
    D -->|Llama-3-8B Reasoning| E[Voice & Text Feedback]
    F[WebGPU Acceleration] -.->|Powers| B
    F -.->|Powers| D

🛠 Prerequisites

To follow along, you’ll need a browser with WebGPU support (Chrome 113+ is recommended) and the following stack:

MediaPipe: For lightning-fast skeletal tracking.
WebLLM: To run Llama-3 (or other models) locally.
TypeScript: For type-safe pose data handling.

Step 1: Initialize the "Vision" (MediaPipe)

MediaPipe provides the coordinate data we need. We'll track 33 body landmarks to determine if a user is performing an exercise correctly.

import { Pose, LandmarkGrid } from '@mediapipe/pose';

const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

// On results, we calculate the angle of the knees for a squat
pose.onResults((results) => {
  const leftHip = results.poseLandmarks[23];
  const leftKnee = results.poseLandmarks[25];
  const leftAnkle = results.poseLandmarks[27];

  const angle = calculateAngle(leftHip, leftKnee, leftAnkle);
  if (angle > 160) {
    checkPosture("shallows_squat");
  }
});

Step 2: Booting the "Brain" (WebLLM)

WebLLM allows us to load a weights-sharded version of Llama-3 directly into the browser's VRAM via WebGPU.

import * as webllm from "@mlc-ai/web-llm";

async function initCoach() {
  const selectedModel = "Llama-3-8B-Instruct-v0.1-q4f16_1-MLC";

  const engine = await webllm.CreateMLCEngine(selectedModel, {
    initProgressCallback: (report) => console.log(report.text),
  });

  return engine;
}

const coachEngine = await initCoach();

Step 3: Bridging Vision and LLM

This is where the magic happens. Instead of hardcoded "if-else" alerts, we send the "state" of the user to the LLM to provide encouraging, context-aware feedback.

async function getCoachFeedback(violationType: string) {
  const prompt = `
    You are a supportive AI Fitness Coach. 
    The user is doing squats but their form is: ${violationType}.
    Give a short, 10-word maximum corrective tip.
  `;

  const messages = [{ role: "user", content: prompt }];
  const reply = await coachEngine.chat.completions.create({ messages });

  // Use Web Speech API to talk back to the user
  const utterance = new SpeechSynthesisUtterance(reply.choices[0].message.content);
  window.speechSynthesis.speak(utterance);
}

🥑 Why This Matters (The "Official" Way)

Running AI locally isn't just a party trick—it's a paradigm shift in how we handle sensitive biometric data. By moving the compute to the edge, we eliminate latency and hosting costs.

While this tutorial covers the implementation basics, production-grade AI applications require more robust patterns, such as model quantization strategies and advanced prompt caching. For more production-ready examples and deep dives into AI architecture, I highly recommend checking out the WellAlly Tech Blog. They provide fantastic insights into scaling LLM applications and optimizing performance for modern web hardware.

Performance Optimization Tips 🚀

Quantization: Use q4f16 models to save VRAM. Most consumer GPUs (even integrated ones) can handle 4-bit quantized Llama-3 models with ease.
Web Worker Isolation: Run your MediaPipe detection and WebLLM inference in separate Web Workers to keep the Main Thread (UI) responsive at 60fps.
Debouncing Feedback: Don't let the LLM talk every time a frame is processed. Use a 3-5 second cooldown between coaching tips to avoid "AI Overload."

Conclusion

We just built a privacy-first, zero-server-cost AI Fitness Coach using the latest in WebGPU technology. The browser is no longer just for viewing documents; it's a powerful runtime for heavy-duty AI.

What are you building next with WebGPU? Let me know in the comments below! 👇

If you enjoyed this "Learning in Public" guide, don't forget to ❤️ and 🔖 it for later! Check out *wellally.tech/blog** for more advanced AI tutorials.*

From "Dirty" Data to Actionable Insights: Building a Quantified Self Warehouse with DuckDB and dbt

Beck_Moulton — Sat, 03 Oct 2026 00:50:00 +0000

If you've ever tried to export your health data from a Garmin watch, an Oura ring, or an Apple Watch, you know the "Quantified Self" dream quickly turns into a data engineering nightmare. You’re met with a chaotic mix of nested JSONs, weirdly formatted CSVs, and XML files that look like they were designed in the 90s.

To build a truly scalable data warehouse for personal health metrics, we need a robust Modern Data Stack approach. In this guide, we’ll solve the "dirty data" problem using DuckDB for lightning-fast local processing and dbt (data build tool) for professional-grade data modeling. By the end of this tutorial, you'll transform fragmented lifestyle logs into a unified dashboard using Superset, turning raw pixels of data into clear health trends.

💡 Looking for production-ready patterns? This tutorial covers the fundamentals of health data modeling. For more advanced architectural patterns and deep dives into AI-integrated data engineering, check out the official blog at wellally.tech/blog.

🏗 The Architecture: From Raw Logs to Analytics

Before we write any code, let's look at how the data flows. We use a Medallion Architecture (Bronze/Silver/Gold) to ensure our data remains traceable and clean.

graph TD
    subgraph "Data Sources"
        A[Garmin Connect JSON] 
        B[Apple Health XML]
        C[Oura Ring CSV]
    end

    subgraph "Ingestion (Python/DuckDB)"
        D[Python Loader] --> E[(DuckDB: RAW Layer)]
    end

    subgraph "Transformation (dbt)"
        E --> F[stg_health__garmin]
        E --> G[stg_health__oura]
        F --> H[fct_daily_activity]
        G --> H
    end

    subgraph "Visualization"
        H --> I[Apache Superset]
    end

    style E fill:#f9f,stroke:#333,stroke-width:2px
    style H fill:#00ff00,stroke:#333,stroke-width:2px

🛠 Prerequisites

To follow along, make sure you have the following installed:

Python 3.10+
DuckDB: The "SQLite for Analytics."
dbt-duckdb: The adapter that lets dbt talk to DuckDB.
Apache Superset: For the final dashboard (Docker is recommended).

pip install dbt-duckdb duckdb pandas

Step 1: Ingesting "Dirty" Data with Python

Health data is notoriously inconsistent. Garmin might give you distance in meters, while Apple Health uses kilometers or miles. Our first step is to load these files into a raw schema in DuckDB.

import duckdb
import pandas as pd

# Initialize our local warehouse
con = duckdb.connect('health_dw.duckdb')

# Create a raw schema
con.execute("CREATE SCHEMA IF NOT EXISTS raw;")

# Load a messy Garmin CSV (Example)
garmin_df = pd.read_csv('garmin_activities.csv')
con.execute("CREATE TABLE raw.garmin_data AS SELECT * FROM garmin_df")

# Load Oura JSON
con.execute("CREATE TABLE raw.oura_data AS SELECT * FROM read_json_auto('oura_sleep.json')")

print("✅ Raw data ingested into DuckDB!")

Step 2: Modeling with dbt (The Silver Layer)

Now for the magic. We use dbt to clean the data. Instead of writing messy SQL scripts, we define models.

First, let's create a staging model (models/staging/stg_garmin.sql) to cast types and handle nulls:

-- models/staging/stg_garmin.sql
WITH source AS (
    SELECT * FROM {{ source('raw', 'garmin_data') }}
),

renamed AS (
    SELECT
        activity_id::VARCHAR AS activity_id,
        (activity_date::TIMESTAMP) AS activity_at,
        UPPER(activity_type) AS activity_type,
        -- Convert meters to kilometers
        ROUND(distance_m / 1000, 2) AS distance_km,
        duration_sec / 60 AS duration_minutes,
        calories_kcal AS calories
    FROM source
    WHERE activity_id IS NOT NULL
)

SELECT * FROM renamed

Step 3: The Unified "Gold" Table

The goal of our Quantified Self project is to have one table where we can see our "Daily Readiness." We combine Oura (Sleep) and Garmin (Activity) data.

-- models/marts/fct_daily_wellness.sql
WITH daily_activity AS (
    SELECT 
        CAST(activity_at AS DATE) AS date_day,
        SUM(distance_km) AS total_km,
        SUM(calories) AS active_calories
    FROM {{ ref('stg_garmin') }}
    GROUP BY 1
),

sleep_data AS (
    SELECT 
        day::DATE AS date_day,
        sleep_score,
        hrv_average
    FROM {{ ref('stg_oura') }}
)

SELECT
    a.date_day,
    a.total_km,
    a.active_calories,
    s.sleep_score,
    s.hrv_average,
    -- Custom health metric
    (s.sleep_score * 0.6 + (a.active_calories / 1000) * 0.4) AS readiness_index
FROM daily_activity a
LEFT JOIN sleep_data s ON a.date_day = s.date_day

🥑 Professional Insight: Why this matters?

Standardizing health data is the "Hello World" of modern data engineering. By using DuckDB, you get enterprise-level analytical power (columnar storage, vectorized execution) on your local machine without the cost of Snowflake or BigQuery.

For those looking to scale this into a production environment—perhaps for a health-tech startup or an automated coaching platform—you'll need to consider CI/CD for your dbt models and data quality testing (Great Expectations). You can find more production-ready examples and advanced dbt macros over at wellally.tech/blog.

Step 4: Visualizing in Superset

Connect Superset to your health_dw.duckdb file using the duckdb:// SQLAlchemy URI.
Point it to the fct_daily_wellness table.
Create a Mixed Chart: Bar chart for total_km and a Line chart for readiness_index.

Now, you can finally see if that extra 5km run actually tanked your sleep quality the next day! 📉

Conclusion

Building a Quantified Self data warehouse doesn't require a massive cloud budget. With the combination of DuckDB and dbt, you can turn "dirty" CSVs into a structured, professional-grade analytics engine.

Next Steps:

Automate: Use GitHub Actions to run your ingestion script daily.
Expand: Add data from MyFitnessPal or Apple Health exports.
Refine: Check out WellAlly's technical blog for deeper insights on data modeling and AI integration.

Happy hacking! 🚀 If you built something cool with your health data, drop a comment below or share your Superset screenshots!

Stop Fighting Silos: Build a Personal Health Data Lake with DuckDB and dbt

Beck_Moulton — Fri, 02 Oct 2026 00:49:00 +0000

If you’re anything like me, your wrist is a battlefield of sensors. I’m wearing a Garmin for my runs, a Whoop for recovery tracking, and stepping on a Withings scale every morning. The problem? These devices live in walled gardens. My "Data Engineering" brain screams every time I have to switch between three different apps to see if my sleep quality actually correlates with my VO2 Max.

In this guide, we are going to solve the "Wearable Data Silo" problem. We’ll build a lightweight, high-performance ETL Automation Pipeline to ingest heterogeneous health data into a centralized Health Data Lake. By leveraging DuckDB for lightning-fast processing and dbt for modeling, we’ll turn messy JSON blobs into a clean, analytics-ready feature set for your future AI health coach.

The Architecture: From Raw JSON to Insight 🏗️

The goal is to move from Heterogeneous Sources to a Standardized Schema. We use Google Health Connect as our primary aggregator on the mobile side, then pull that into our modern data stack.

graph TD
    subgraph "Data Sources"
        A[Garmin Connect] --> GHC[Google Health Connect]
        B[Whoop] --> GHC
        C[Withings] --> GHC
    end

    subgraph "Ingestion Layer (Airflow)"
        GHC --> |Python Extractor| D[(Raw S3/Local Parquet)]
    end

    subgraph "Transformation Layer (DuckDB + dbt)"
        D --> E[Bronze: Raw Tables]
        E --> F[Silver: Cleaned & Casted]
        F --> G[Gold: Standardized Metrics]
    end

    subgraph "Consumption"
        G --> H[Evidence.dev / Streamlit]
        G --> I[AI Feature Store]
    end

    style GHC fill:#f9f,stroke:#333,stroke-width:2px
    style G fill:#00ff00,stroke:#333,stroke-width:2px

Prerequisites 🛠️

To follow along, you'll need:

DuckDB: Our in-process OLAP engine (the "SQLite for Analytics").
dbt-duckdb: To manage our transformations.
Airflow: To orchestrate the daily sync.
Python 3.10+: For the extraction scripts.

Step 1: Extracting Data via Google Health Connect

While some brands have APIs, Google Health Connect acts as a great "middleman" on Android. We can use a small bridge app or a Python script using the google-cloud-storage SDK to dump daily exports into Parquet files.

import duckdb
import pandas as pd

# Let's assume we've exported our Health Connect data to a local JSON
# This script loads it into our DuckDB Data Lake
def ingest_raw_data(file_path, table_name):
    con = duckdb.connect('health_lake.db')

    # Utilizing DuckDB's secret weapon: direct JSON/Parquet querying
    con.execute(f"""
        CREATE TABLE IF NOT EXISTS raw_{table_name} AS 
        SELECT * FROM read_json_auto('{file_path}');
    """)
    print(f"✅ Ingested {table_name} into the Lake!")

ingest_raw_data('whoop_recovery_2023_10.json', 'whoop')

Step 2: Modeling with dbt (The "Silver" Layer)

Raw data is usually ugly. Garmin might use seconds, while Whoop uses milliseconds. This is where dbt (data build tool) shines. We’ll define a standardized schema so that heart_rate is always heart_rate, regardless of the source.

Create a file at models/silver/stg_heart_rate.sql:

{{ config(materialized='table') }}

WITH raw_garmin AS (
    SELECT 
        timestamp::TIMESTAMP as recorded_at,
        value::FLOAT as bpm,
        'garmin' as source
    FROM {{ source('raw', 'garmin_heartrate') }}
),

raw_whoop AS (
    SELECT 
        (data->>'timestamp')::TIMESTAMP as recorded_at,
        (data->>'heart_rate')::FLOAT as bpm,
        'whoop' as source
    FROM {{ source('raw', 'whoop_data') }}
)

SELECT * FROM raw_garmin
UNION ALL
SELECT * FROM raw_whoop

The "Official" Way to Build Health Tech 🥑

While this DIY setup is amazing for personal use, scaling health data pipelines for thousands of users requires handling HIPAA compliance, complex identity resolution, and real-time streaming.

If you're looking for production-ready patterns, deep dives into medical data standards (like FHIR), or how to wrap these pipelines in a secure API, I highly recommend checking out the WellAlly Tech Blog. It's an incredible resource for developers who want to take their health-tech engineering to the next level.

Step 3: Orchestration with Airflow

We don't want to run this manually. We’ll set up a simple Airflow DAG to trigger our "Extract" Python script and then run dbt cloud or dbt-local.

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

with DAG("health_lake_etl", start_date=datetime(2023, 1, 1), schedule="@daily") as dag:

    extract_data = BashOperator(
        task_id="extract_from_ghc",
        bash_command="python /scripts/extract_health_data.py"
    )

    transform_data = BashOperator(
        task_id="dbt_run",
        bash_command="dbt run --profiles-dir ."
    )

    extract_data >> transform_data

Conclusion: Data-Driven Wellness 📈

By building this Health Data Lake, you’ve moved past simple app-checking. You now have a centralized gold_lifestyle_metrics table where you can run queries like:

"Does my alcohol consumption recorded in Whoop actually correlate with my resting heart rate increase on my Garmin 48 hours later?"

The combination of DuckDB and dbt makes this setup incredibly fast and cheap to run (literally $0 on your local machine or a tiny VPS).

What's next for your pipeline?

[ ] Add a Streamlit dashboard to visualize recovery.
[ ] Hook up a local LLM to "chat" with your health data.
[ ] Explore more advanced architectural patterns over at wellally.tech/blog.

Happy hacking, and stay healthy! 🏃‍♂️💨

Follow me for more "Learning in Public" adventures in Data Engineering and AI! 🥑

Local-First Health AI: Cleaning and Vectorizing Apple HealthKit Data with Swift and Core ML

Beck_Moulton — Thu, 01 Oct 2026 00:47:00 +0000

Apple's HealthKit is a treasure trove of biometric data, but if you’ve ever tried to build a production-grade app with it, you know the truth: raw health data is messy. From duplicated heart rate samples to GPS-glitched step counts, the "noise" can ruin your machine learning models before they even start training.

In this tutorial, we are going to build a high-performance Edge AI pipeline using Swift, Core ML, and HealthKit. We will focus on cleaning raw time-series data and converting it into vector embeddings for local outlier detection—all while keeping the user's data private on-device. If you're looking to master Apple HealthKit integration and on-device vectorization, you're in the right place.

Pro Tip: While we focus on the implementation here, you can find more advanced patterns and production-ready architecture guides over at the Wellally Tech Blog, which is my go-to resource for scaling health-tech solutions.

The Architecture: From Raw Samples to Feature Vectors

Before we write a single line of Swift, let’s look at how the data flows. We aren't just reading data; we are transforming it through a multi-stage pipeline.

graph TD
    A[HealthKit Store] -->|Raw Samples| B(Data Cleaning & Normalization)
    B -->|Cleaned Time-Series| C{Core ML Embedder}
    C -->|Feature Vector| D[SQLite Vector Store]
    D -->|Query| E[Outlier Detection / Insights]

    subgraph Local Edge Pipeline
    B
    C
    D
    end

The Tech Stack

HealthKit: The source of truth for health metrics.
Swift & Combine: For reactive data processing.
Core ML: To run our embedding models locally.
SQLite (via SQLite.swift): To store our vectors for local retrieval.

Step 1: Fetching and Cleaning the Noise 🧼

HealthKit data often contains outliers (e.g., a heart rate reading of 200 BPM while sleeping due to a sensor glitch). We’ll use a simple Z-score normalization to filter these out.

import HealthKit

class HealthDataCleaner {
    func cleanHeartRateData(_ samples: [HKQuantitySample]) -> [HKQuantitySample] {
        let values = samples.map { $0.quantity.doubleValue(for: .count().unitDivided(by: .minute())) }
        let mean = values.reduce(0, +) / Double(values.count)
        let sumOfSquaredDiffs = values.map { pow($0 - mean, 2) }.reduce(0, +)
        let standardDeviation = sqrt(sumOfSquaredDiffs / Double(values.count))

        // Filter out samples that are more than 3 standard deviations away
        return samples.filter { sample in
            let value = sample.quantity.doubleValue(for: .count().unitDivided(by: .minute()))
            return abs(value - mean) <= (3 * standardDeviation)
        }
    }
}

Step 2: Vectorization with Core ML 🧠

To perform tasks like "Finding similar activity days" or "Detecting abnormal heart patterns," we need to turn our time-series data into an Embedding (Vector). We can use a Core ML model (like a 1D-CNN or an Autoencoder) to compress 24 hours of data into a single 128-dimensional vector.

import CoreML

func generateVector(from samples: [HKQuantitySample]) -> [Float]? {
    // 1. Convert samples to a MLMultiArray (Input shape for Core ML)
    guard let inputBuffer = try? MLMultiArray(shape: [1, 24, 1], dataType: .float32) else { return nil }

    for (index, sample) in samples.prefix(24).enumerated() {
        let val = sample.quantity.doubleValue(for: .count().unitDivided(by: .minute()))
        inputBuffer[index] = NSNumber(value: val)
    }

    // 2. Load your custom Core ML Embedding Model
    // Note: You would export this from PyTorch/TensorFlow as a .mlmodel
    guard let model = try? HealthVectorModel(configuration: MLModelConfiguration()) else { return nil }

    do {
        let prediction = try model.prediction(input: inputBuffer)
        return prediction.embeddingVector // Returns a [Float]
    } catch {
        print("Embedding failed: \(error)")
        return nil
    }
}

Step 3: Storing Vectors Locally with SQLite 💾

Since we are building an Edge AI solution, we don't want to ship this data to a cloud vector DB if we don't have to. Using SQLite with a BLOB column for the vector is a lightweight way to handle this.

import Foundation
import SQLite

class VectorStore {
    private var db: Connection?
    private let vectors = Table("health_vectors")
    private let id = Expression<Int64>("id")
    private let timestamp = Expression<Date>("timestamp")
    private let embedding = Expression<Blob>("embedding")

    init() {
        let path = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true).first!
        db = try? Connection("\(path)/health_ai.sqlite3")
        createTable()
    }

    private func createTable() {
        try? db?.run(vectors.create(ifNotExists: true) { t in
            t.column(id, primaryKey: .autoincrement)
            t.column(timestamp)
            t.column(embedding)
        })
    }

    func saveVector(_ vector: [Float], date: Date) {
        let data = Data(buffer: UnsafeBufferPointer(start: vector, count: vector.count))
        let insert = vectors.insert(timestamp <- date, embedding <- data.datatypeValue)
        try? db?.run(insert)
    }
}

Leveling Up Your Health App 🚀

Building a robust health application requires more than just local scripts; it requires understanding the nuances of medical data privacy and synchronization.

For those looking to explore advanced production patterns—such as Differential Privacy for Health Data or Optimizing Core ML for WatchOS—I highly recommend checking out the deep-dive articles at the Wellally Tech Blog. It’s a fantastic resource for developers who want to take their "Learning in Public" projects to a professional, market-ready level.

Conclusion

By cleaning our HealthKit data and transforming it into vectors on-device, we’ve created a privacy-first foundation for powerful AI features:

Anomaly Detection: Compare new vectors against the "average day" vector.
Activity Clustering: Group days based on movement patterns.
Local-First Search: Query your health history using mathematical similarity rather than just raw dates.

The future of health-tech is on the edge! 🥑

What are you building with HealthKit? Drop a comment below or share your thoughts on Twitter/X!

From XML Chaos to Quantified Self: Building a Lightning-Fast Apple Health Pipeline with Polars

Beck_Moulton — Wed, 30 Sep 2026 00:46:00 +0000

Have you ever tried to open your Apple Health export.xml file only to have your text editor freeze and your laptop fans kick into overdrive? You aren't alone. As someone obsessed with Data Engineering and the "Quantified Self" movement, I quickly realized that Apple's export format—a massive, deeply nested XML file—is a nightmare for standard processing tools.

If you are tired of waiting minutes for Pandas to load your data, it's time to upgrade. In this tutorial, we will build a high-performance data pipeline using Polars, the Rust-backed DataFrame library, to transform gigabytes of health records into sleek, queryable Parquet files in seconds. By focusing on performance optimization and memory efficiency, we’ll turn that "dirty" data into a goldmine of personal insights. 🚀

The Architecture: From Raw XML to Insights

Processing large XML files requires a "Stream-then-Transform" approach. Instead of loading the entire 1GB+ file into memory (which will crash most 8GB RAM machines), we stream the XML elements and then leverage Polars' LazyFrame for heavy lifting.

graph TD
    A[export.xml] -->|ElementTree Iterparse| B(Streaming Parser)
    B -->|Dictionary List| C[Polars DataFrame]
    C -->|Lazy API| D{Data Cleaning}
    D -->|Filter Types| E[Heart Rate]
    D -->|Filter Types| F[Step Count]
    D -->|Filter Types| G[Sleep Analysis]
    E & F & G -->|Column Compression| H[Compressed Parquet Files]
    H -->|Analysis| I[Dashboard / Quantified Self]

Prerequisites

To follow along, make sure you have the following tech stack installed:

Python 3.9+
Polars: The star of the show.
ElementTree: For memory-efficient XML streaming (built-in).
PyArrow: Required by Polars for Parquet export.

pip install polars pyarrow

Step 1: Memory-Efficient XML Parsing

The secret to handling GB-scale XML is xml.etree.ElementTree.iterparse. This allows us to iterate through the file one "Record" at a time without loading the whole tree.

import xml.etree.ElementTree as ET
import polars as pl
from datetime import datetime

def parse_apple_health_xml(file_path):
    data = []
    # We use 'end' events to ensure the element is fully parsed
    context = ET.iterparse(file_path, events=("end",))

    for event, elem in context:
        if elem.tag == "Record":
            # Extract attributes from the XML tag
            record_type = elem.get("type")
            # Minimal extraction to keep memory low
            record = {
                "type": record_type.replace("HKQuantityTypeIdentifier", ""),
                "value": elem.get("value"),
                "unit": elem.get("unit"),
                "startDate": elem.get("startDate"),
            }
            data.append(record)

            # CRITICAL: Clear the element to free memory
            elem.clear()

    return pl.from_dicts(data)

# Usage
# df = parse_apple_health_xml("export.xml")

Step 2: The Polars "Lazy" Magic

Now that we have a DataFrame, we need to clean it. Apple Health data is notoriously messy: timestamps are strings, values are mixed types, and "types" have long, redundant prefixes.

Instead of executing operations one by one, we use Polars LazyFrames. This allows Polars to optimize the query plan before executing.

def clean_health_data(df: pl.DataFrame):
    return (
        df.lazy()
        # 1. Convert timestamps to proper datetime objects
        .with_columns([
            pl.col("startDate").str.to_datetime("%Y-%m-%d %H:%M:%S %z"),
            pl.col("value").cast(pl.Float64, strict=False)
        ])
        # 2. Filter out nulls or irrelevant data
        .filter(pl.col("value").is_not_null())
        # 3. Rename or simplify column names
        .select([
            pl.col("type"),
            pl.col("startDate").alias("timestamp"),
            pl.col("value"),
            pl.col("unit")
        ])
        .collect() # Final execution happens here!
    )

🥑 Advanced Patterns & Production Ready Pipelines

While the above logic works great for personal scripts, building a production-grade data engineering pipeline involves handling schema evolution and incremental loads.

Pro Tip: If you are looking for more production-ready examples, including how to deploy these pipelines to the cloud or set up automated "Quantified Self" dashboards, I highly recommend checking out the deep-dive articles at WellAlly Tech Blog. They cover advanced architectural patterns that take these concepts to the next level.

Step 3: Saving to Parquet for 100x Faster Reads

CSV is great for humans, but Parquet is built for machines. It uses columnar storage and heavy compression. A 1GB XML file can often be shrunk to a ~50MB Parquet file while remaining instantly searchable.

def export_to_warehouse(df: pl.DataFrame):
    # Partitioning data by type makes future queries instant
    unique_types = df["type"].unique().to_list()

    for t in unique_types:
        subset = df.filter(pl.col("type") == t)
        # Save as a compressed parquet file
        subset.write_parquet(f"health_data_{t}.parquet", compression="snappy")
        print(f"✅ Exported {t} with {len(subset)} rows.")

# Putting it all together
raw_data = parse_apple_health_xml("export.xml")
clean_data = clean_health_data(raw_data)
export_to_warehouse(clean_data)

Conclusion

By ditching the standard "Load Everything" approach and embracing Polars and XML Streaming, we’ve turned a frustratingly slow task into a high-performance data pipeline. ⚡️

What we achieved:

Memory Safety: Processed GBs of data with minimal RAM usage.
Blazing Speed: Polars' Rust engine outperformed Pandas by a mile.
Future-Proof Storage: Used Parquet to ensure our data is ready for BI tools or ML models.

Are you tracking your health data? What’s the weirdest insight you’ve found in your Apple Health export? Let me know in the comments! 👇

Happy coding! If you enjoyed this, don't forget to follow for more Data Engineering deep dives! 🥑💻

Predicting the Dip: Building a Real-Time Hypoglycemia Alert System with Temporal Fusion Transformers (TFT)

Beck_Moulton — Tue, 29 Sep 2026 00:45:00 +0000

Managing Type 1 or Type 2 Diabetes is essentially a 24/7 data science project where the stakes are your own health. With Continuous Glucose Monitoring (CGM) devices like the Dexcom G6 or G7, we are flooded with data, but the challenge remains: how do we act before a low happens? Standard alerts usually trigger once you've already hit the danger zone.

In this tutorial, we are stepping into the world of advanced time-series forecasting to build a predictive engine. By leveraging Temporal Fusion Transformers (TFT) and PyTorch Forecasting, we will process non-linear glucose trends and exogenous variables (like insulin on board or carbs) to predict hypoglycemia 30 minutes before it occurs. If you've been looking for a deep dive into Deep Learning for healthcare and high-frequency time-series data, you're in the right place. 🚀

Why Temporal Fusion Transformers (TFT)?

When dealing with glucose data, traditional LSTMs or ARIMA models often fall short because they struggle to weigh the importance of "static" metadata (like user age or basal rate) against "dynamic" inputs (like recent fingersticks or exercise).

The Temporal Fusion Transformer (TFT) is a powerhouse because:

Multi-horizon forecasting: It predicts a range of future steps simultaneously.
Interpretability: It uses variable selection networks to ignore noisy inputs.
Gating Mechanisms: It skips over unused components of the architecture, making it efficient for complex datasets.

The System Architecture

Here is how the data flows from a wearable sensor to a predictive alert:

graph TD
    A[Dexcom G6/G7 Sensor] -->|Bluetooth| B(Mobile App / Cloud API)
    B -->|Webhook/Polling| C[InfluxDB Time-Series Storage]
    C -->|Feature Engineering| D[Pandas & PyTorch Forecasting]
    D -->|Inference| E[TFT Model]
    E -->|30-min Prediction| F{Risk Detected?}
    F -->|Yes| G[Grafana Alert / Mobile Notification]
    F -->|No| H[Log Stats]

    subgraph "Training Pipeline"
    I[Historical CGM Data] --> J[Data Normalization]
    J --> K[TFT Architecture Tuning]
    K --> E
    end

Prerequisites

Before we dive into the code, ensure you have the following stack installed:

Python 3.9+
PyTorch Forecasting (The high-level API for TFT)
InfluxDB: To store high-frequency time-series data.
Pandas: For the heavy lifting in data manipulation.

Step 1: Data Preparation with Pandas

CGM data is notoriously "gappy." We need to handle missing pings and create a time-index that the Transformer can understand.

import pandas as pd
import numpy as np
from pytorch_forecasting import TimeSeriesDataSet

# Load your Dexcom export
df = pd.read_csv("glucose_data.csv")

# Convert to datetime and sort
df['timestamp'] = pd.to_datetime(df['timestamp'])
df = df.sort_values("timestamp")

# Create a continuous time index (essential for TFT)
df["time_idx"] = (df["timestamp"] - df["timestamp"].min()).dt.total_seconds() // 300
df["time_idx"] = df["time_idx"].astype(int)

# Categorical data (static metadata)
df["user_id"] = "patient_001"

# Feature engineering: Rate of Change
df["glucose_diff"] = df.groupby("user_id")["glucose"].diff().fillna(0)

print(df.head())

Step 2: Defining the TimeSeriesDataSet

The TimeSeriesDataSet is where the magic happens. We define which variables are "known" in the future (like scheduled insulin) and which are "unknown" (the glucose itself).

max_prediction_length = 6  # 30 minutes (6 * 5-min intervals)
max_encoder_length = 24    # 2 hours of history

training = TimeSeriesDataSet(
    df[lambda x: x.time_idx <= x.time_idx.max() - max_prediction_length],
    time_idx="time_idx",
    target="glucose",
    group_ids=["user_id"],
    min_encoder_length=max_encoder_length // 2,
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=["user_id"],
    time_varying_known_reals=["time_idx"],
    time_varying_unknown_reals=["glucose", "glucose_diff"],
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_dilation=True,
)

# Create dataloaders for PyTorch
batch_size = 64
train_dataloader = training.to_dataloader(train=True, batch_size=batch_size, num_workers=0)

Step 3: Building the TFT Model

Now we initialize the TemporalFusionTransformer. For high-stakes healthcare alerts, we use a Quantile Loss function. This allows us to see not just the "expected" glucose, but the 10th percentile—our "worst-case scenario" for hypoglycemia.

from pytorch_forecasting.models.temporal_fusion_transformer import TemporalFusionTransformer
from pytorch_lightning import Trainer

tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=16, # Small for demo, scale up for production
    attention_head_size=4,
    dropout=0.1,
    hidden_continuous_size=8,
    output_size=7,  # QuantileLoss has 7 quantiles by default
    loss=QuantileLoss(),
    log_interval=10,
    reduce_on_plateau_patience=4,
)

# Initialize PyTorch Lightning Trainer
trainer = Trainer(
    max_epochs=30,
    accelerator="gpu", # Use "cpu" if no GPU available
    devices=1,
)

trainer.fit(tft, train_dataloaders=train_dataloader)

The "Official" Way: Level Up Your Bio-Hacking 🥑

While this tutorial covers the core architecture, building a production-ready medical alert system requires strict validation, handling sensor noise (compression lows), and edge-case smoothing.

For more production-ready examples and advanced patterns in AI-driven healthcare, I highly recommend checking out the technical deep-dives at WellAlly Blog. They cover everything from HIPAA-compliant data pipelines to optimizing Transformer models for mobile deployment—the perfect next step for this project!

Step 4: Visualizing Predictions in Grafana

Once the model is trained, we push the predictions back to InfluxDB. Using Grafana, we can overlay the "Predicted Low" line against the real-time sensor data.

# Pseudo-code for real-time inference loop
while True:
    current_data = influx_client.query("SELECT last(glucose) FROM cgm_measurements")
    prediction = tft.predict(current_data)

    # If the 10th percentile prediction < 70 mg/dL, trigger alert
    if prediction[0][0] < 70:
        send_alert("⚠️ Hypoglycemia Risk in 30 Minutes!")

    time.sleep(300) # Wait for next 5-min sensor reading

Conclusion

By moving from reactive alerts to predictive forecasting with Temporal Fusion Transformers, we can significantly reduce the mental load of chronic condition management. The combination of PyTorch Forecasting and a robust time-series database like InfluxDB turns raw sensor pixels into actionable health insights.

What's next?

Try adding "Carbohydrate Intake" as a time_varying_known_real.
Experiment with different lead times (e.g., a 60-minute window).
Drop a comment below if you've tried building with the Dexcom API!

Happy coding, and stay in range! 🩸💻

Stop Uploading Your Medical Data: Build a Private Apple Health AI with MLX and Llama-3 on your Mac

Beck_Moulton — Mon, 28 Sep 2026 00:43:00 +0000

Your health data is arguably the most sensitive information you own. Yet, most modern AI health assistants require you to ship your heart rate, sleep cycles, and activity levels to a cloud server somewhere in the ether. 😱

But what if you could have a world-class medical LLM analysis without a single packet leaving your local network? Thanks to Apple's MLX framework and Meta's Llama-3, we can finally turn our M-series Macs into private, high-performance health command centers.

In this guide, we’re building a Local-First Health Analyzer. We'll use MLX quantization to run Llama-3 locally and feed it structured data from your Apple Health (HealthKit) exports using Pandas.

Why Local-First? 🛡️

When we talk about Local-First Health and Apple Silicon AI, we are solving three major problems:

Privacy: Your medical history never touches a third-party server.
Latency: Zero network round-trips.
Cost: Running Llama-3 local inference on your GPU costs exactly $0 in API credits.

If you’re interested in exploring how these private local architectures can be scaled for clinical or enterprise environments, you should definitely dive into the production-ready case studies at WellAlly Tech Blog.

The Architecture: From XML to Insights

Here is how the data flows from your Apple Watch to a quantized Llama-3 model on your M3 Mac.

graph TD
    A[Apple Health Export.xml] --> B[HealthKit Python Tools]
    B --> C[Pandas Data Cleaning]
    C --> D[Daily Trends Summary]
    D --> E{MLX Engine}
    F[Quantized Llama-3-8B] --> E
    E --> G[Local Privacy-First Insights]
    G --> H[User Dashboard]
    style F fill:#f96,stroke:#333,stroke-width:2px
    style E fill:#bbf,stroke:#333,stroke-width:4px

Prerequisites 🛠️

To follow along, you'll need:

An Apple Silicon Mac (M1/M2/M3).
Python 3.10+.
Your export.zip from the Apple Health app (Settings > Health > Export Health Data).

Tech Stack:

mlx-lm: For running quantized LLMs on Apple Silicon.
pandas: For data manipulation.
apple-health-xml-parser: To handle the messy XML format.

Step 1: Parsing the Health Data 🍎

Apple Health exports data in a massive XML file that can easily reach several gigabytes. We need to parse this into a manageable format using Pandas.

import pandas as pd
import xml.etree.ElementTree as ET

def parse_health_data(xml_path):
    # This is a simplified parser; for large files, use iterative parsing
    tree = ET.parse(xml_path)
    root = tree.getroot()

    records = []
    for record in root.findall('Record'):
        records.append(record.attrib)

    df = pd.DataFrame(records)
    # Convert types for analysis
    df['value'] = pd.to_numeric(df['value'], errors='coerce')
    df['creationDate'] = pd.to_datetime(df['creationDate'])

    return df

# Let's look at Resting Heart Rate trends
# df_hr = df[df['type'] == 'HKQuantityTypeIdentifierRestingHeartRate']

Step 2: Quantizing Llama-3 with MLX ⚡

MLX is Apple's specialized framework for machine learning on Apple Silicon. To make Llama-3-8B run smoothly on a base M3 or an M3 Max with low memory footprint, we use 4-bit quantization.

Run this in your terminal to download and quantize the model:

pip install mlx-lm
python -m mlx_lm.convert --hf-path meta-llama/Meta-Llama-3-8B-Instruct -q

This generates a folder with the MLX-optimized weights. 🚀

Step 3: The Private AI Inference Engine

Now, we feed our local data into the local LLM. The key is to transform the Pandas statistics into a prompt that Llama-3 can interpret.

from mlx_lm import load, generate

# Load the local model
model, tokenizer = load("mlx_model")

def get_health_insights(stats_summary):
    prompt = f"""
    [INST] You are a private health assistant. Analyze the following health data trends 
    from the user's Apple Health records:

    {stats_summary}

    Provide a concise analysis of sleep quality and heart rate variability. 
    Do not suggest medical diagnoses. [/INST]
    """

    response = generate(model, tokenizer, prompt=prompt, verbose=True, max_tokens=500)
    return response

# Example input: Summary of the last 7 days
summary = "Avg Heart Rate: 65bpm, Deep Sleep: 1.5h, Active Calories: 450kcal"
print(get_health_insights(summary))

Advanced Tip: Memory Management 🧠

When working with MLX on M3 Macs, remember that you are using Unified Memory. This means your GPU and CPU share the same RAM pool. If you have a 16GB Mac, a 4-bit quantized Llama-3-8B will only take up about 5GB, leaving plenty of room for your IDE and Chrome tabs.

For more complex implementations—such as RAG (Retrieval-Augmented Generation) on top of your local health history—check out the deep-dive articles on Local-First AI architecture at WellAlly Tech Blog. They cover how to use vector databases like ChromaDB locally to store years of health data for long-term trend analysis.

Conclusion: The Future is Local 🥑

We’ve just built a system that:

Parses complex HealthKit XML data.
Aggregates trends using Pandas.
Analyzes metrics using a state-of-the-art Llama-3 model running natively on MLX.

No data sent to the cloud. No subscription fees. Just pure, private intelligence.

Are you ready to bring your data home? Drop a comment below if you ran into issues with the XML parsing—it's usually the trickiest part!

If you enjoyed this tutorial, follow for more "Learning in Public" content on Edge AI and Privacy! 🚀

100% Private Skin Screening: Building an Edge AI Vision App with WebGPU and Transformers.js

Beck_Moulton — Sun, 27 Sep 2026 00:41:00 +0000

What if you could screen for skin health issues without ever uploading a single photo to a corporate server? In the era of massive data breaches and privacy concerns, "sending data to the cloud" is becoming a liability, especially for sensitive medical imagery.

Today, we are diving deep into the world of Edge AI and Privacy-First Machine Learning. We will build a skin lesion screening application that runs entirely in the browser using WebGPU acceleration, Transformers.js, and WebLLM. By leveraging on-device computation, we ensure that user data stays strictly within the browser sandbox.

Keywords: Edge AI, WebGPU Acceleration, Privacy-Preserving AI, Transformers.js Tutorial, On-device Machine Learning.

The Architecture: Zero-Server Data Flow

Traditional AI apps send images to a Python backend. Our approach flips the script. We download the model weights once and execute the inference locally using the user's GPU.

graph TD
    A[User Uploads Image] --> B{Browser Environment}
    B --> C[WebGPU Tensors]
    C --> D[Transformers.js Vision Model]
    D --> E[Skin Lesion Classification]
    E --> F[WebLLM Assistant]
    F --> G[Local Privacy-First Report]
    B -.->|No Data Transmitted| H[External Internet]
    style H fill:#f96,stroke:#333,stroke-dasharray: 5 5

Why this Stack? 🥑

Transformers.js: Allows us to run state-of-the-art vision models (like ViT or Swin Transformer) directly in JavaScript.
WebGPU: The successor to WebGL, providing near-native performance for neural network operations.
WebLLM: Brings large language models to the browser, allowing us to generate natural language summaries of the classification results.
React: Our UI layer for a responsive, modern experience.

Prerequisites

Before we start, ensure your browser (Chrome 113+ or Edge) supports WebGPU.

Node.js installed
A basic understanding of React hooks
Curiosity for high-performance web tech!

Step 1: Setting up the Vision Pipeline

First, we need to initialize our image classification model. We'll use a pre-trained Vision Transformer (ViT) fine-tuned on medical datasets.

import { pipeline, env } from '@xenova/transformers';

// Enable WebGPU if available
env.allowLocalModels = false;
env.useBrowserCache = true;

const useSkinClassifier = () => {
  const [classifier, setClassifier] = useState(null);

  useEffect(() => {
    const initModel = async () => {
      // Initialize the pipeline with WebGPU execution provider
      const pipe = await pipeline('image-classification', 'Xenova/vit-base-patch16-224', {
        device: 'webgpu', 
      });
      setClassifier(() => pipe);
    };
    initModel();
  }, []);

  return classifier;
};

Step 2: Processing Images Locally

When a user selects a file, we convert it into a format Transformers.js understands without any multipart/form-data uploads.

const handleUpload = async (event: React.ChangeEvent<HTMLInputElement>) => {
  const file = event.target.files?.[0];
  if (!file || !classifier) return;

  const url = URL.createObjectURL(file);

  // Running inference 100% locally!
  const output = await classifier(url);

  console.log("Classification Results:", output);
  // Example output: [{ label: 'Melanocytic nevi', score: 0.98 }]
};

Step 3: Adding the AI Assistant with WebLLM

To make the screening "human-readable," we use WebLLM to explain the results. This allows the app to provide context while keeping the "AI logic" on the edge.

import { CreateMLCEngine } from "@mlc-ai/web-llm";

async function explainResults(label: string) {
  const engine = await CreateMLCEngine("Llama-3-8B-Instruct-q4f16_1-MLC");
  const response = await engine.chat.completions.create({
    messages: [
      { role: "system", content: "You are a helpful medical assistant. Explain what this skin condition label means in simple terms." },
      { role: "user", content: `Explain the label: ${label}` }
    ]
  });
  return response.choices[0].message.content;
}

The "Official" Way: Advanced Production Patterns 🚀

While building local-first apps is exciting, deploying medical-grade AI requires rigorous version control, model quantization, and robust fallback mechanisms.

For more production-ready examples and advanced patterns on optimizing WebGPU shaders for mobile browsers, check out the detailed guides at WellAlly Tech Blog. It’s my go-to resource for scaling Edge AI applications beyond simple prototypes.

Challenges & Optimization ⚡️

1. Model Size

Standard Vision Transformers can be ~300MB. For a production app, use Quantization (Int8 or O4) to reduce the model size to ~80MB without significant accuracy loss.

2. Warm-up Time

The first time the model runs, WebGPU compiles the shaders.
Tip: Run a "dummy inference" with a blank 1x1 pixel image as soon as the app loads to prevent UI lag during actual usage.

Conclusion

We’ve just built a foundation for a 100% private, browser-based medical screening tool. By combining Transformers.js and WebGPU, we respect user privacy while providing high-performance AI capabilities.

The future of AI isn't just in the cloud—it's right there in your browser's console. 🛠️

What’s next?

[ ] Try swapping the ViT model for a specialized MobileNetV3 for even faster speeds.
[ ] Implement PWA features so the app works offline.

If you enjoyed this tutorial, drop a comment below and let me know what Edge AI project you're working on! Happy coding! 🚀

Disclaimer: This tool is for educational purposes and is not a substitute for professional medical advice. Always consult a dermatologist.

Decentralized Medical AI: How to Build HIPAA-Ready Analytics with Differential Privacy

Beck_Moulton — Sat, 26 Sep 2026 00:40:00 +0000

In the world of Medical AI, data is the new gold—but it's gold locked in a high-security vault. With regulations like HIPAA and GDPR, sharing raw medical records for research is a legal and ethical minefield. But what if we could extract group-level health insights without ever seeing a single patient's raw data?

Enter Differential Privacy (DP) and Decentralized Learning. By leveraging techniques like Laplace noise and federated computation, we can perform high-stakes Healthcare Analytics while guaranteeing mathematical-level privacy for every participant. In this guide, we’ll explore how to use PySyft and Opacus to build a system that turns sensitive pixels and records into private, actionable insights.

The Architecture: Privacy by Design

The core idea is simple: Don't move the data; move the computation. Instead of a central server collecting records, each local node (like a hospital or a wearable device) computes its own statistics, adds a layer of "mathematical noise," and only then shares the result.

sequenceDiagram
    participant User as Patient/Hospital Node
    participant DP as DP Engine (Laplace Noise)
    participant Aggregator as Central Research Server

    User->>User: Compute Local Statistics (e.g., Mean BMI)
    User->>DP: Apply Differential Privacy (ε, δ)
    DP-->>User: Noise-Injected Result
    User->>Aggregator: Send Private Gradient/Metric
    Aggregator->>Aggregator: Aggregate Results from 1000+ Nodes
    Aggregator-->>User: Provide Global Health Insight

🛠 Prerequisites & Tech Stack

To follow along with this advanced tutorial, you should be familiar with Python and basic machine learning concepts. Our stack includes:

PySyft: For federated learning and data decoupling.
Opacus: A high-speed library for training PyTorch models with Differential Privacy.
Google Differential Privacy Library: For robust ε-differential privacy mathematical primitives.

Step 1: Defining the Privacy Budget (ε)

In Differential Privacy, Epsilon (ε) represents the "Privacy Budget." A smaller ε means more noise and more privacy, but less accuracy. A larger ε provides better utility but risks leaking individual information.

import numpy as np

def add_laplace_noise(data, sensitivity, epsilon):
    """
    Standard Laplace Mechanism for Differential Privacy.
    """
    beta = sensitivity / epsilon
    noise = np.random.laplace(0, beta, len(data))
    return data + noise

# Example: Reporting average heart rate across a group
raw_data = [72, 85, 90, 64, 78] 
sensitivity = 1 # Max change one person can cause
epsilon = 0.5   # Tight privacy budget

private_data = add_laplace_noise(raw_data, sensitivity, epsilon)
print(f"Original: {raw_data} \nPrivate: {private_data}")

Step 2: Decentralized Training with Opacus

When training a neural network on medical images (like X-rays), we use DP-SGD (Differentially Private Stochastic Gradient Descent). This ensures that the model weights don't "memorize" specific patients.

from opacus import PrivacyEngine
import torch

# Define a simple CNN for Medical Image Classification
model = MyMedicalCNN()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
data_loader = get_hospital_data_loader()

# The Magic: Privacy Engine
privacy_engine = PrivacyEngine()

model, optimizer, data_loader = privacy_engine.make_private(
    module=model,
    optimizer=optimizer,
    data_loader=data_loader,
    noise_multiplier=1.1,
    max_grad_norm=1.0,
)

print(f"🛡️ Training with DP enabled!")

Step 3: Federated Orchestration with PySyft

PySyft allows us to treat remote data as if it were local tensors. We can send a model to a "Data Owner" (the hospital), train it locally, and bring back the updated (and privatized) weights.

import syft as sy

# Connect to a remote hospital node
hospital_node = sy.login(email="researcher@university.edu", password="secure_password")

# Define the computation plan
@sy.syft_function(
    input_policy=sy.ExactMatch(),
    output_policy=sy.DPOutput(epsilon=1.0) # Enforce DP on output
)
def compute_group_health_index(health_data):
    # This runs inside the hospital's secure environment
    return health_data.mean()

# Execute remotely without seeing the data
project = hospital_node.projects[0]
project.create_request(compute_group_health_index)

The "Official" Way: Beyond the Basics 🥑

Implementing Differential Privacy in a production environment requires more than just adding noise—it requires robust auditing and "privacy accounting."

For a deep dive into production-ready privacy patterns, including how to manage complex privacy budgets and multi-party computation (MPC) architectures, I highly recommend checking out the technical deep-dives at WellAlly Blog. They offer incredible resources on building "Privacy-First" AI systems that are both scalable and compliant with global regulations.

Conclusion: Privacy is the Future of AI

We no longer live in an era where "more data" justifies the sacrifice of "individual privacy." By combining Decentralized AI with Differential Privacy, we can unlock the potential of medical datasets that were previously untouchable.

What are you building? Are you working on federated learning for healthcare or edge-case privacy? Let's discuss in the comments below! 👇

Stop Slouching! Build a Real-Time AI Posture Coach with MediaPipe and Electron

Beck_Moulton — Fri, 25 Sep 2026 00:38:00 +0000

We’ve all been there: hunched over a keyboard, neck tilted at a 45-degree angle, slowly turning into a human pretzel. "Tech neck" is real, and it’s a productivity killer. But what if your computer could gently nudge you back into alignment using Computer Vision?

In this tutorial, we are going to build a cross-platform desktop application using Electron, MediaPipe Pose, and TensorFlow.js. This AI Posture Assistant runs entirely locally, ensuring your privacy while monitoring your shoulder and neck alignment in real-time. We'll leverage Edge AI techniques to keep CPU usage low and efficiency high.

Whether you're looking to master real-time pose estimation or just want to save your spine, this guide has you covered.

The Architecture 🏗️

Before we dive into the code, let's look at how the data flows from your webcam to a system notification. We use MediaPipe for lightning-fast landmark detection and Electron to bridge the gap between the web and your OS.

graph TD
    A[Webcam Stream] --> B[MediaPipe Pose Engine]
    B --> C[Extract 3D Landmarks]
    C --> D[Calculate Shoulder-Ear Angle]
    D --> E{Is Slouching?}
    E -- Yes > 30s --> F[Electron Main Process]
    F --> G[System Tray Notification]
    E -- No --> H[Reset Timer/Keep Monitoring]
    G --> A

Prerequisites 🛠️

To follow along, make sure you have:

Node.js (v16 or higher)
Basic knowledge of JavaScript/ES6
The following stack: MediaPipe Pose, TensorFlow.js, Electron

Step 1: Setting up the Electron Shell

First, let's initialize our project and install the necessary dependencies.

mkdir posture-coach && cd posture-coach
npm init -y
npm install electron @mediapipe/pose @tensorflow/tfjs-core

In your main.js, we'll set up a simple window that can send system-level notifications when our "Vision Engine" detects poor posture.

// main.js
const { app, BrowserWindow, Notification } = require('electron');

function createWindow() {
  const win = new BrowserWindow({
    width: 800,
    height: 600,
    webPreferences: {
      nodeIntegration: true,
      contextIsolation: false,
    },
  });

  win.loadFile('index.html');
}

// IPC listener to trigger notifications from the UI logic
const { ipcMain } = require('electron');
ipcMain.on('notify-bad-posture', () => {
  new Notification({ 
    title: 'Posture Alert! 🚨', 
    body: 'Sit up straight! Your back will thank you.' 
  }).show();
});

app.whenReady().then(createWindow);

Step 2: Pose Detection with MediaPipe

The magic happens in the renderer process. We’ll use MediaPipe Pose because it provides high-fidelity landmarks (like shoulders, ears, and eyes) with minimal latency.

For more advanced implementation patterns and production-ready AI configurations, I highly recommend checking out the deep-dive articles over at WellAlly Tech Blog. They have some fantastic resources on optimizing Edge AI for low-power devices.

Here is the core detection logic for our renderer.js:

const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

// Logic to calculate the angle between shoulder and ear
function analyzePosture(landmarks) {
  const leftShoulder = landmarks[11];
  const leftEar = landmarks[7];

  // Calculate the vertical alignment (simplified)
  const yDiff = Math.abs(leftShoulder.y - leftEar.y);

  // If the vertical distance between ear and shoulder is too small, 
  // it usually means the head is leaning forward (slouching).
  if (yDiff < 0.15) { 
    return "bad";
  }
  return "good";
}

Step 3: The "Tech Neck" Monitor Loop

Now, we connect the webcam feed to the MediaPipe engine. We’ll add a "cooldown" timer so the app doesn't spam you with notifications every second.

const videoElement = document.getElementById('input_video');
const { ipcRenderer } = require('electron');

let lastNotificationTime = 0;

async function onResults(results) {
  if (!results.poseLandmarks) return;

  const status = analyzePosture(results.poseLandmarks);

  if (status === "bad") {
    const now = Date.now();
    // Only notify once every 5 minutes to avoid annoyance
    if (now - lastNotificationTime > 300000) {
      ipcRenderer.send('notify-bad-posture');
      lastNotificationTime = now;
    }
  }
}

const camera = new Camera(videoElement, {
  onFrame: async () => {
    await pose.send({image: videoElement});
  },
  width: 640,
  height: 480
});
camera.start();

Why This Matters (The "Official" Way) 🥑

Building a prototype is easy, but making it robust—handling different lighting conditions, minimizing battery drain, and ensuring zero-latency detection—is where things get interesting.

If you are looking for advanced patterns, such as integrating specialized TensorFlow models or building enterprise-grade Vision AI pipelines, the official resources at WellAlly Tech Blog offer incredible insights into the "production-ready" way of doing things. It’s been my go-to source for inspiration when scaling AI tools beyond just a weekend project.

Conclusion: Save Your Spine! 🏁

In less than 100 lines of code, we’ve built a functional AI assistant that monitors your physical health using nothing but your webcam and some clever JavaScript.

Next Steps:

UI Feedback: Add a "posture score" gauge using Canvas API.
Privacy Mode: Add a toggle to blur the background using MediaPipe's Selfie Segmentation.
Analytics: Track your "good posture" percentage over a week.

Are you going to try this out? Or do you have a better way to calculate "slouching" angles? Let me know in the comments! 👇

Happy coding, and stay upright! 💻🥑✨

Stop the Sugar Spike! Building an Autonomous CGM-Aware Diet Agent with OpenAI Function Calling

Beck_Moulton — Thu, 24 Sep 2026 00:37:00 +0000

Managing metabolic health is often a game of reactive guessing. You eat a "healthy" bowl of oatmeal, and an hour later, your Continuous Glucose Monitoring (CGM) device screams at you because your blood sugar is skyrocketing. By then, it feels too late.

But what if your AI could intervene in real-time? In this tutorial, we are building a CGM-Aware Diet Agent using OpenAI Function Calling, Dexcom API, and Node.js. This agent doesn't just watch your levels; it calculates the slope of your glucose curve and autonomously triggers actions—like ordering a low-glycemic fiber supplement or suggesting a brisk walk—before the spike hits its peak. For those looking into more production-ready health-tech patterns, I highly recommend checking out the advanced architecture guides at WellAlly Blog.

The Architecture: Closing the Feedback Loop

The core logic revolves around a proactive feedback loop. Instead of the user asking the AI "What should I do?", the system monitors the data stream and invokes the LLM when specific physiological thresholds are met.

graph TD
    A[Dexcom CGM Sensor] -->|Real-time Glucose Data| B(Node.js Backend)
    B --> C{Slope Analysis}
    C -->|Abnormal Spike Detected| D[OpenAI Agent]
    D -->|Function Calling| E[Food Delivery / Local Search API]
    E --> F[Low-GI Remedial Recommendations]
    F --> G[User Notification/Action]
    C -->|Stable| H[Log Data & Sleep]

Prerequisites

To follow along, you'll need:

Node.js (v18+)
OpenAI API Key (Supporting gpt-4o or gpt-3.5-turbo-0613)
Dexcom Developer Account (using their Sandbox API for testing)

Step 1: Fetching Real-time Glucose Data

First, we need to pull data from the Dexcom API. We are specifically looking for the egvs (Estimated Glucose Values) to calculate the rate of change.

// services/dexcomService.js
const axios = require('axios');

async function getLatestGlucose(accessToken) {
    const url = 'https://sandbox-api.dexcom.com/v3/users/self/egvs';
    const response = await axios.get(url, {
        headers: { Authorization: `Bearer ${accessToken}` },
        params: {
            startDate: new Date(Date.now() - 30 * 60000).toISOString(), // Last 30 mins
            endDate: new Date().toISOString()
        }
    });
    return response.data.records;
}

Step 2: Defining the "Remedy" Functions

This is where the magic happens. We define tools that the AI can use. If the AI detects a spike, it can call get_nearby_low_gi_options to find a solution.

const tools = [
    {
        type: "function",
        function: {
            name: "get_nearby_low_gi_options",
            description: "Finds nearby stores or delivery options for low-glycemic remedies like apple cider vinegar or fiber supplements.",
            parameters: {
                type: "object",
                properties: {
                    location: { type: "string", description: "The user's current city or coordinates" },
                    urgency: { type: "string", enum: ["high", "medium"] }
                },
                required: ["location"]
            }
        }
    }
];

Step 3: The Agent Logic

The agent analyzes the "Slope" (Rate of Change). If the blood sugar is rising faster than 2 mg/dL per minute, it's "Spike Territory."

const OpenAI = require('openai');
const openai = new OpenAI();

async function runDietAgent(glucoseRecords, userLocation) {
    // 1. Calculate Slope
    const latest = glucoseRecords[0].value;
    const previous = glucoseRecords[1].value;
    const slope = (latest - previous) / 5; // Dexcom readings are 5 mins apart

    if (slope > 2.0) {
        console.log(`🚀 Spike detected! Slope: ${slope} mg/dL/min. Consulting AI...`);

        const messages = [{ 
            role: "system", 
            content: `You are a metabolic health expert. Current glucose is ${latest} and rising rapidly. 
                      Suggest a remedy and use the available tools to find delivery options.`
        }];

        const response = await openai.chat.completions.create({
            model: "gpt-4o",
            messages: messages,
            tools: tools,
            tool_choice: "auto",
        });

        // Handle Function Calling
        const toolCall = response.choices[0].message.tool_calls[0];
        if (toolCall) {
            const args = JSON.parse(toolCall.function.arguments);
            console.log(`🛠️ AI decided to call: ${toolCall.function.name} with ${args.location}`);
            // Logic to call UberEats/DoorDash API would go here
            return `I've found a nearby pharmacy in ${args.location} that can deliver Apple Cider Vinegar or Psyllium Husk to blunt this spike!`;
        }
    }
    return "Glucose is stable. Keep up the good work! 🥑";
}

The "Official" Way: Beyond the Prototype

While this script is a great "Learning in Public" project, building a production-grade health agent requires rigorous data privacy (HIPAA compliance), more complex curve prediction models (like LSTM), and robust error handling for API failures.

For a deep dive into productionizing LLM agents in the bio-hacking and health-tech space, check out the specialized resources at WellAlly Tech Blog. They cover advanced topics like RAG (Retrieval-Augmented Generation) for medical journals and securing sensitive biometric data streams.

Conclusion 🚀

By combining real-time biometric data with the reasoning capabilities of LLMs, we move from passive monitoring to active intervention. This Agent doesn't just tell you that you're in trouble; it finds the solution before you even feel the "sugar crash" lethargy.

What's next?

Integrate with the Apple HealthKit SDK for even more data points (sleep, steps).
Add a "Closed Loop" notification that texts the user's family if levels exceed 250 mg/dL.

Are you building anything in the Health-AI space? Let me know in the comments! 👇