DEV Community: wellallyTech

Your Health Data is Sacred: Build a Private AI Health Coach with Llama-3 and Apple MLX 🍎🛡️

wellallyTech — Thu, 02 Jul 2026 01:05:00 +0000

In an era where data breaches are the norm, uploading your sensitive medical history or daily step counts to a cloud-based LLM feels like a massive gamble. Privacy-conscious developers are increasingly turning to Edge AI and Local LLMs to handle sensitive information. If you own an Apple Silicon Mac, you are sitting on a goldmine of compute power optimized for exactly this.

In this tutorial, we’ll explore how to build a privacy-preserving AI health assistant using the Apple MLX framework and Llama-3-8B. We will parse exported Apple HealthKit XML data and use a local model to perform trend analysis—all without a single packet of health data leaving your machine. For those looking to dive deeper into advanced production-ready AI patterns and enterprise-grade privacy architectures, I highly recommend checking out the technical deep-dives at WellAlly Blog.

The Local-First Architecture

The goal is simple: transform raw, messy XML health exports into actionable insights using a quantized Llama-3 model optimized for Metal (Apple’s GPU).

Data Flow Overview

graph TD
    A[HealthKit Export.xml] --> B{Local Python Script}
    B --> C[XML Parser & Summarizer]
    C --> D[MLX Quantized Llama-3-8B]
    D --> E[Local Health Insights]
    subgraph Privacy Boundary (Your Mac)
    B
    C
    D
    E
    end
    style B fill:#f9f,stroke:#333,stroke-width:2px

Prerequisites

Before we start, ensure you have an M1/M2/M3 Mac and the following tools installed:

Python 3.10+
Apple MLX: A framework for machine learning research on Apple silicon.
Hugging Face CLI: To download the quantized weights.

pip install mlx-lm xmltodict pandas

Step 1: Parsing the HealthKit "Monster" XML

Apple Health exports data in a massive export.xml file. It’s too large to feed directly into a model's context window, so we need a pre-processing step to extract specific metrics like Heart Rate, Sleep, and Active Energy.

import xml.etree.ElementTree as ET
import pandas as pd

def parse_health_data(file_path):
    # We only care about specific record types for this demo
    target_records = [
        'HKQuantityTypeIdentifierStepCount',
        'HKQuantityTypeIdentifierHeartRate',
        'HKQuantityTypeIdentifierActiveEnergyBurned'
    ]

    context = ET.iterparse(file_path, events=("end",))
    data = []

    for event, elem in context:
        if elem.tag == 'Record' and elem.get('type') in target_records:
            data.append({
                'type': elem.get('type').replace('HKQuantityTypeIdentifier', ''),
                'value': elem.get('value'),
                'date': elem.get('startDate')[:10] # Keep only YYYY-MM-DD
            })
        elem.clear() # Free memory

    return pd.DataFrame(data)

# Usage
# df = parse_health_data('export.xml')
# summary = df.groupby(['date', 'type'])['value'].astype(float).sum().unstack()

Step 2: Running Llama-3 locally with MLX

The mlx-lm library makes it incredibly easy to run 4-bit or 8-bit quantized models. Using a quantized version of Llama-3-8B ensures it runs lightning-fast even on 8GB or 16GB RAM machines.

from mlx_lm import load, generate

model_path = "mlx-community/Meta-Llama-3-8B-Instruct-4bit"
model, tokenizer = load(model_path)

def ask_local_llama(prompt):
    formatted_prompt = f"<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

    response = generate(
        model, 
        tokenizer, 
        prompt=formatted_prompt, 
        max_tokens=500,
        temp=0.7
    )
    return response

Step 3: Generating Health Insights

Now, we combine our parsed data with a structured prompt. We want the AI to look at the numbers and tell us if we're slacking or overtraining.

# Sample data snippet passed to the LLM
health_summary = """
Date: 2023-10-24 | Steps: 12000 | Avg Heart Rate: 72 | Calories: 450
Date: 2023-10-25 | Steps: 3000 | Avg Heart Rate: 65 | Calories: 120
"""

prompt = f"""
You are a private medical data analyst. Analyze the following health data:
{health_summary}

Tasks:
1. Identify trends in physical activity.
2. Provide 2 actionable suggestions for better health.
3. Maintain a professional and encouraging tone.
"""

print(ask_local_llama(prompt))

Why Local-First Matters (The "Official" Way) 🥑

Building "Local-First" isn't just a gimmick; it's the future of personalized AI. When you keep data on the edge, you eliminate latency and maximize user trust. While this tutorial provides a functional baseline, scaling this for production—handling multi-gigabyte XML files or integrating real-time streaming—requires more robust engineering.

For a deeper dive into Production-Grade Edge AI architectures and how to optimize MLX for high-concurrency environments, you should check out the engineering guides at wellally.tech/blog. They cover everything from RAG (Retrieval Augmented Generation) on encrypted local databases to advanced prompt engineering for healthcare.

Conclusion

By leveraging Apple MLX and Llama-3, we’ve turned a standard Mac into a powerful, private health consultant. No cloud, no subscription, and most importantly, no data leaks.

Key Takeaways:

MLX is the gold standard for AI on Apple Silicon.
Quantization (4-bit) allows Llama-3-8B to run smoothly on consumer hardware.
Local parsing ensures PII (Personally Identifiable Information) stays in your control.

What are you building with local LLMs? Drop a comment below or share your latest MLX benchmarks! 🚀💻

If you enjoyed this, don't forget to follow for more "Learning in Public" tutorials!

From Pixels to Calories: Building an AI Nutritionist with GPT-4o and SAM 2 🥗

wellallyTech — Wed, 01 Jul 2026 01:00:00 +0000

Ever tried logging your lunch in a fitness app, only to find that "Chicken Salad" has 500 different entries? 🥗 We can do better. With the rise of Multimodal LLMs and advanced Computer Vision, we can now build a pipeline that doesn't just guess, but actually analyzes the geometry and ingredients of your meal in real-time.

In this tutorial, we are going to build a "Pixels to Calories" pipeline. We’ll use Meta’s Segment Anything Model 2 (SAM 2) for precise object detection and GPT-4o for high-level visual reasoning and nutritional estimation. This combination allows us to handle complex, multi-object plates where traditional models usually fail. For those interested in scaling these types of AI workflows into production-grade systems, you can find more advanced patterns and production-ready examples over at the WellAlly Tech Blog. 🥑

The Challenge: Why traditional CV isn't enough

Most nutrition apps use simple classification. They see a "round orange object" and say "Orange: 60kcal." But what if it’s a bowl of pasta with mixed toppings? We need to:

Isolate each food item (Segmentation).
Identify the ingredients within those segments.
Estimate volume and density to calculate calories.

The Architecture 🏗️

Our pipeline follows a "Segment-then-Reason" logic. Instead of sending one giant, messy image to an LLM, we segment the individual components first to provide the model with "focal points."

graph TD
    A[User Image Upload] --> B[FastAPI Endpoint]
    B --> C[SAM 2: Multi-mask Generation]
    C --> D[Object Cropping & Preprocessing]
    D --> E[GPT-4o: Multimodal Reasoning]
    E --> F[Nutritional Data Extraction]
    F --> G[JSON Output: Calories/Macros]
    G --> H[Frontend Visualization]

Prerequisites 🛠️

To follow along, you'll need:

PyTorch (for SAM 2 execution)
SAM 2 Weights (from Meta's official repo)
OpenAI API Key (for GPT-4o)
FastAPI (for the backend)

Step 1: Segmenting the Plate with SAM 2

SAM 2 is a game-changer because it treats images as collections of masks. Unlike its predecessor, SAM 2 is faster and more robust at handling overlapping food items.

import torch
from sam2.build_sam import build_sam2
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

# Initialize SAM 2
sam2_checkpoint = "sam2_hvit_b.pt"
model_cfg = "sam2_hvit_b.yaml"

sam2 = build_sam2(model_cfg, sam2_checkpoint, device="cuda" if torch.cuda.is_available() else "cpu")
mask_generator = SAM2AutomaticMaskGenerator(sam2)

def get_food_segments(image_path):
    image = load_image(image_path) # Helper to load image
    masks = mask_generator.generate(image)

    # Filter masks by size to remove noise (small crumbs or shadows)
    valid_masks = [m for m in masks if m['area'] > 1000]
    return valid_masks

Step 2: Prompting GPT-4o for Visual Reasoning

Now that we have our segments, we send the original image and the segmentation metadata to GPT-4o. We use a structured prompt to ensure the output is a clean JSON object.

import openai

def analyze_nutrition(image_base64, segments_metadata):
    client = openai.OpenAI()

    prompt = f"""
    You are an expert nutritionist. I will provide an image of a meal and 
    segmentation data indicating {len(segments_metadata)} distinct items.

    Task:
    1. Identify each food item.
    2. Estimate the weight in grams based on visual volume.
    3. Calculate Calories, Protein, Carbs, and Fats.

    Return ONLY a JSON array of objects.
    """

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}}
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Wrapping it in FastAPI

We need an API to make this usable. FastAPI's asynchronous nature is perfect for handling the heavy lifting of SAM 2 and the external API calls to OpenAI.

from fastapi import FastAPI, UploadFile, File

app = FastAPI()

@app.post("/analyze-meal")
async def analyze_meal(file: UploadFile = File(...)):
    # 1. Save and Load Image
    contents = await file.read()

    # 2. Run SAM 2 (GPU Intensive)
    masks = get_food_segments(contents)

    # 3. Call GPT-4o for Nutritional Analysis
    nutrition_json = analyze_nutrition(encode_base64(contents), masks)

    return {
        "status": "success",
        "segments_found": len(masks),
        "data": nutrition_json
    }

The "Official" Way to Scale 🚀

While the code above is a great starting point for a "Learning in Public" project, building a production-grade health tech app requires more:

Distributed Task Queues: Running SAM 2 on a GPU during a web request is risky. Use Celery or Redis Queues.
Vector Databases: Store food embeddings to avoid calling GPT-4o for common, identical-looking meals.
Model Quantization: Using FP16 or INT8 for SAM 2 can significantly reduce latency.

For a deeper dive into these optimization strategies and to see how to integrate this with a full-stack dashboard, check out the specialized guides at wellally.tech/blog. They cover everything from LLM observability to high-performance inference.

Conclusion

Combining SAM 2 and GPT-4o bridges the gap between raw pixels and semantic understanding. We’ve moved from "this is a picture of food" to "this is 150g of grilled salmon with approximately 300 calories." 🚀

What would you build with this? A smart fridge? An automated restaurant billing system? Let me know in the comments below! 👇

Happy coding! 💻🥑

Beyond Averages: Predicting Burnout with Temporal Fusion Transformers (TFT) and Wearable Data 📉⌚

wellallyTech — Tue, 30 Jun 2026 01:00:00 +0000

We’ve all been there: You wake up feeling like a zombie 🧟‍♂️, despite your Apple Watch telling you that you slept 8 hours. The problem? Most wearable apps give you a reactive view of your health. They tell you what happened, not what is going to happen.

In this tutorial, we are diving deep into time-series forecasting and predictive analytics to transform raw Heart Rate Variability (HRV) data into a 24-hour stress forecast. By leveraging the Temporal Fusion Transformer (TFT)—a state-of-the-art deep learning model—we can move beyond simple moving averages to capture complex, long-range dependencies in physiological data. If you've been looking to implement Heart Rate Variability (HRV) monitoring with PyTorch Forecasting, you're in the right place! 🚀

The Architecture: From Pulse to Prediction 🧠

To build a production-grade health monitoring system, we need a robust pipeline that handles high-frequency data ingestion, complex modeling, and real-time visualization.

graph TD
    A[Wearable Devices: Oura/Apple Watch] -->|Raw HRV/ECG| B(InfluxDB: Time-Series Storage)
    B --> C{Feature Engineering}
    C -->|Static Covariates: Age, Sex| D[Temporal Fusion Transformer]
    C -->|Dynamic Covariates: Steps, Sleep| D
    D --> E[Multi-Horizon Forecast: Next 24h Stress]
    E --> F[Grafana Dashboard]
    E --> G[CoreML Export: On-Device Inference]
    style D fill:#f9f,stroke:#333,stroke-width:4px

Prerequisites 🛠️

Before we start coding, ensure you have the following stack ready:

PyTorch & PyTorch Forecasting: Our engine for the TFT model.
InfluxDB: Optimized for high-write loads of time-series health data.
Grafana: For visualizing our "Burnout Index."
CoreML Tools: To squeeze that heavy model onto an iPhone.

Step 1: Data Ingestion with InfluxDB 📥

We need a database that doesn't choke on millisecond-level heart data. InfluxDB is perfect here. We'll pull data from our wearable API and push it to a bucket.

from influxdb_client import InfluxDBClient, Point, WritePrecision
from influxdb_client.client.write_api import SYNCHRONOUS

# Initialize InfluxDB Client
token = "YOUR_TOKEN"
org = "QuantifiedSelf"
bucket = "hrv_data"

client = InfluxDBClient(url="http://localhost:8086", token=token, org=org)
write_api = client.write_api(write_options=SYNCHRONOUS)

# Example: Writing a single HRV data point
point = Point("physiological_metrics") \
    .tag("user_id", "dev_advocate_01") \
    .field("hrv_ms", 65.4) \
    .field("stress_score", 42) \
    .time(datetime.utcnow(), WritePrecision.NS)

write_api.write(bucket, org, point)

Step 2: Modeling with Temporal Fusion Transformer (TFT) 🤖

The Temporal Fusion Transformer (TFT) is the "Golden Child" of time-series forecasting. Unlike standard LSTMs, TFT uses specialized "Gating Mechanisms" and "Variable Selection Networks" to decide which inputs (like sleep quality vs. caffeine intake) are actually important at any given time.

Here is how we define our TimeSeriesDataSet for PyTorch Forecasting:

import torch
from pytorch_forecasting import TimeSeriesDataSet, TemporalFusionTransformer

# Define the dataset
max_prediction_length = 24  # Predict next 24 hours
max_encoder_length = 168    # Look back at the last 7 days

training = TimeSeriesDataSet(
    data,
    time_idx="time_idx",
    target="stress_index",
    group_ids=["user_id"],
    min_encoder_length=max_encoder_length // 2,
    max_encoder_length=max_encoder_length,
    min_prediction_length=1,
    max_prediction_length=max_prediction_length,
    static_categoricals=["user_id"],
    time_varying_known_reals=["hour_of_day", "day_of_week"],
    time_varying_unknown_reals=["hrv_ms", "stress_index", "sleep_score"],
    target_normalizer=GroupNormalizer(groups=["user_id"], transformation="softplus"),
    add_relative_time_idx=True,
    add_target_scales=True,
    add_encoder_generation_idx=True,
)

# Initialize the model
tft = TemporalFusionTransformer.from_dataset(
    training,
    learning_rate=0.03,
    hidden_size=16,  # Keep it small for mobile deployment later!
    attention_head_size=4,
    dropout=0.1,
    hidden_continuous_size=8,
    output_size=7,  # Quantile regression (7 quantiles)
    loss=QuantileLoss(),
    log_interval=10,
    reduce_on_plateau_patience=4,
)

The "Official" Way (Advanced Patterns) 🥑

While this tutorial gets you from zero to a working model, productionizing health AI requires handling missing data, sensor noise, and privacy-preserving federated learning.

If you're looking for more production-ready patterns, advanced architecture deep-dives, or enterprise-scale time-series implementations, I highly recommend checking out the technical deep-dives at Wellally Tech Blog. It’s where I get most of my inspiration for building robust health-tech systems!

Step 3: Edge Inference with CoreML 📱

We don't want to send our heart data to a server every 5 seconds. Let's export our trained PyTorch model to CoreML for on-device prediction.

import coremltools as ct

# Trace the model with a dummy input
dummy_input = torch.randn(1, max_encoder_length, n_features)
traced_model = torch.jit.trace(tft, dummy_input)

# Convert to CoreML
model = ct.convert(
    traced_model,
    inputs=[ct.TensorType(shape=dummy_input.shape)]
)

model.save("BurnoutPredictor.mlpackage")

Step 4: Visualizing Stress Forecasts in Grafana 📊

Connect InfluxDB to Grafana and use the Flux query language to overlay your Predicted Stress vs. Actual HRV. This creates a powerful feedback loop for the user.

from(bucket: "hrv_data")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "physiological_metrics")
  |> filter(fn: (r) => r["_field"] == "stress_index" or r["_field"] == "predicted_stress")
  |> yield(name: "mean")

Conclusion: Stop Reacting, Start Predicting 🏁

By combining the Temporal Fusion Transformer with wearable data, we move away from "dumb" dashboards and towards "intelligent" companions that can warn us of burnout before it happens.

Summary of what we built:

A high-performance data pipeline using InfluxDB.
An advanced TFT model using PyTorch Forecasting.
An edge-ready deployment strategy with CoreML.

Are you working on wearable tech or time-series AI? Drop a comment below or share your favorite feature engineering hacks!

Don't forget to visit Wellally.tech for more advanced tutorials on AI and health-tech engineering! 🥑✨

From Messy Medical PDFs to Structured FHIR: Building an LLM-Powered Healthcare Pipeline 🩺💻

wellallyTech — Mon, 29 Jun 2026 01:50:00 +0000

If you've ever worked in the healthcare space, you know the "Data Boss" from hell: the unstructured medical report PDF. Whether it's a scanned physical exam or a legacy system export, getting clean, actionable data out of these documents usually involves a nightmare of Regex, brittle OCR templates, and manual data entry.

But the game has changed. By leveraging LLM-powered ETL techniques, we can now transform chaotic pixels into the FHIR (Fast Healthcare Interoperability Resources) standard with surgical precision. In this guide, we'll build a pipeline using Unstructured.io, GPT-4-turbo, and Python to automate medical data extraction. We'll be focusing on high-volume keywords like medical data extraction, LLM-powered ETL, and FHIR standard implementation to ensure our solution is both modern and scalable.

The Architecture: From Pixels to Interoperability

The secret sauce isn't just "throwing the PDF at ChatGPT." It's about a structured pipeline that handles document partitioning, semantic cleaning, and schema mapping.

graph TD
    A[Medical PDF/Scan] --> B[Unstructured.io]
    B -->|Partitioning| C[Clean Text & Tables]
    C --> D[GPT-4-turbo Parser]
    D -->|Semantic Extraction| E[FHIR Mapping Logic]
    E --> F[Structured FHIR JSON]
    F --> G[(Healthcare Database/HIE)]

    subgraph "The LLM Brain"
    D
    E
    end

Prerequisites

Before we dive into the code, ensure you have your environment ready:

Python 3.10+
Unstructured.io (for document pre-processing)
OpenAI API Key (GPT-4-turbo recommended for long-context medical terms)
FHIR Resources knowledge (specifically Observation and Patient resources)

Step 1: Parsing the Chaos with Unstructured.io

Standard OCR often loses the "context" of tables. Unstructured.io helps us partition the document into logical elements like titles, narrative text, and most importantly, tables.

from unstructured.partition.pdf import partition_pdf

# Partitioning the PDF into manageable chunks
elements = partition_pdf(
    filename="patient_report_001.pdf",
    infer_table_structure=True,
    strategy="hi_res" # Uses layout analysis for better table detection
)

# Extracting only the tables and text for the LLM
raw_content = "\n".join([str(el) for el in elements])
print(f"Extracted {len(raw_content)} characters of medical data.")

Step 2: Defining the FHIR Target (Pydantic)

FHIR is strict. To ensure our LLM output is valid, we use Pydantic to define the structure for a lab result (an Observation in FHIR-speak).

from pydantic import BaseModel, Field
from typing import List, Optional

class ObservationResource(BaseModel):
    resourceType: str = "Observation"
    status: str = "final"
    code_display: str = Field(..., description="Name of the test, e.g., Blood Glucose")
    value_quantity: float = Field(..., description="Numeric value of the result")
    unit: str = Field(..., description="Measurement unit, e.g., mg/dL")
    reference_range: Optional[str] = Field(None, description="Normal range string")

class FHIRBundle(BaseModel):
    observations: List[ObservationResource]

Step 3: Semantic Extraction with GPT-4-turbo

Now for the magic. We feed the raw text into GPT-4-turbo with a system prompt that enforces the FHIR mapping.

Pro-Tip: For more production-ready examples and advanced patterns in AI-driven data engineering, I highly recommend checking out the engineering deep-dives at WellAlly Blog. They cover excellent strategies for handling PII and HIPAA compliance in LLM pipelines.

import openai

def extract_fhir_data(text_content):
    client = openai.OpenAI()

    response = client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[
            {"role": "system", "content": "You are a clinical data engineer. Extract lab results from the text and format them strictly as a JSON object following the FHIR Observation standard."},
            {"role": "user", "content": f"Extract data from this report: {text_content}"}
        ],
        response_format={"type": "json_object"}
    )

    return response.choices[0].message.content

# Execute the extraction
structured_json = extract_fhir_data(raw_content)

Step 4: Validating and Loading

Once the LLM returns the JSON, we validate it against our Pydantic model. This ensures that even if the LLM "hallucinates" a field name, our pipeline will catch the error before it hits the database.

import json

try:
    data = json.loads(structured_json)
    validated_bundle = FHIRBundle(**data)
    print("✅ Successfully converted to FHIR!")
    # Proceed to save to your FHIR server (e.g., HAPI FHIR or Azure Health Data Services)
except Exception as e:
    print(f"❌ Validation failed: {e}")

Why This Matters: The Big Picture 🌍

Converting "trapped" data in PDFs to the FHIR standard is the first step toward true healthcare interoperability. By using an LLM-powered ETL process:

Accuracy increases: GPT-4 understands context (e.g., it knows "Hgb" is Hemoglobin).
Developer Velocity: You stop writing 50 different parsers for 50 different hospital formats.
Scalability: New report types can be added just by updating the system prompt.

Conclusion

Building a medical data pipeline doesn't have to be a manual slog. By combining Unstructured.io for layout analysis and GPT-4-turbo for semantic mapping, we can turn a pile of PDFs into a clean, FHIR-compliant data lake.

If you're interested in scaling this to millions of documents or adding a Human-in-the-loop (HITL) layer, explore the advanced architectural patterns at wellally.tech/blog. They have some fantastic resources on handling large-scale unstructured data in regulated industries.

What are you building with LLMs and Healthcare? Let me know in the comments! 👇

Stop Forgetting Your Meds: Build an Edge-AI Smart Medicine Cabinet with YOLOv10 and Raspberry Pi 💊🤖

wellallyTech — Sun, 28 Jun 2026 01:50:00 +0000

We’ve all been there—staring at a pill bottle at 11 PM, wondering, "Did I already take this today?" Forgetting medication isn't just a minor inconvenience; it’s a global healthcare challenge. While smartphone reminders help, they don't actually know if you opened the box.

In this tutorial, we are diving into Edge AI vision to build a hardware-integrated solution. We'll be using YOLOv10 object detection, Raspberry Pi automation, and OpenCV to transform a standard medicine cabinet into a smart guardian. By deploying a lightweight vision model at the edge, we can achieve real-time computer vision tracking without sending private video feeds to the cloud.

Why YOLOv10 for the Edge?

YOLOv10 is a game-changer for edge devices like the Raspberry Pi. Unlike its predecessors, it introduces a NMS-free (Non-Maximum Suppression) training strategy, significantly reducing inference latency. For our smart cabinet, this means faster detection and lower CPU overhead on our Pi.

The Architecture

Our system follows a "Capture -> Detect -> Validate -> Notify" loop. Here is how the data flows from the camera lens to your notification center:

graph TD
    A[Camera Module] -->|Raw Frame| B[OpenCV Pre-processing]
    B -->|Resized Image| C[YOLOv10 Inference]
    C -->|Detected Labels| D{Logic Engine}
    D -->|Match Schedule?| E[Update Database]
    D -->|Missing/Wrong Dose?| F[MQTT Broker]
    F -->|Alert| G[Mobile App/Speaker]
    E -->|Log Entry| H[Local Dashboard]

Prerequisites

To follow along, you'll need:

Hardware: Raspberry Pi 4B or 5, USB Camera or Pi Cam.
Tech Stack:
- YOLOv10 (weights exported to ONNX or NCNN format)
- OpenCV for image handling
- Paho-MQTT for messaging
- Python 3.9+

Step 1: Exporting YOLOv10 for Edge Performance

First, we need to train (or fine-tune) YOLOv10 on medicine box datasets. Once trained, we export it to a format the Raspberry Pi can handle efficiently.

from ultralytics import YOLO

# Load your custom-trained YOLOv10 model
model = YOLO("yolov10n_meds.pt")

# Export to ONNX format for better compatibility with OpenCV DNN or ONNX Runtime
model.export(format="onnx", imgsz=640, optimize=True)

Step 2: The Core Detection Engine

On the Raspberry Pi, we use a loop to capture frames and run inference. We use MQTT to decouple the vision logic from the notification logic, making the system highly modular.

import cv2
from ultralytics import YOLO
import paho.mqtt.client as mqtt

# Initialize MQTT Client
client = mqtt.Client("MedicineCabinet")
client.connect("localhost", 1883)

# Load the exported model
model = YOLO("yolov10n_meds.onnx")

cap = cv2.VideoCapture(0)

print("🚀 Smart Cabinet Active...")

while cap.isOpened():
    ret, frame = cap.read()
    if not ret: break

    # Run YOLOv10 Inference
    results = model(frame, conf=0.5)

    detected_items = []
    for result in results:
        for box in result.boxes:
            label = model.names[int(box.cls)]
            detected_items.append(label)

            # Draw for local debugging
            x1, y1, x2, y2 = map(int, box.xyxy[0])
            cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
            cv2.putText(frame, label, (x1, y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)

    # Logic: If 'Aspirin' is detected, publish to MQTT
    if "Aspirin" in detected_items:
        client.publish("home/meds/detected", "Aspirin_Taken")

    cv2.imshow("Edge Vision Feed", frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

Step 3: Integrating with the "Source of Truth"

A standalone camera is cool, but a production-ready medical assistant needs a robust backend to manage schedules and dosage histories. While we are building the edge component here, managing complex medical logic requires proven architectural patterns.

💡 Pro-Tip: For advanced production patterns, such as handling multi-device synchronization or secure medical data storage, I highly recommend checking out the WellAlly Official Blog. They provide excellent deep dives into how to bridge the gap between hobbyist IoT projects and enterprise-grade healthcare solutions.

Step 4: Setting Up the MQTT Subscriber

Your "Notification Hub" (which could be another script or a Home Assistant instance) listens for these events:

def on_message(client, userdata, message):
    payload = str(message.payload.decode("utf-8"))
    print(f"✅ Event Logged: {payload}")
    # Trigger local TTS (Text-to-Speech)
    # os.system(f"say 'Thank you for taking your {payload}'")

client.subscribe("home/meds/detected")

The Result: A Truly Smart Cabinet

By combining YOLOv10 with Raspberry Pi, we've created a low-latency, private, and effective health monitor. The NMS-free architecture of YOLOv10 allows the Pi to maintain a steady 10-15 FPS, which is more than enough for monitoring a medicine cabinet.

Challenges to Consider:

Lighting: Medicine cabinets are often in dark bathrooms. Consider adding an LED ring triggered by a PIR motion sensor.
Occlusion: If two boxes are stacked, the camera might miss one. Positioning is key!

Conclusion

Building with Edge AI is all about balancing performance and privacy. This project demonstrates that you don't need a massive GPU server to implement meaningful, life-saving technology at home.

What’s next for your Smart Home? Are you going to add a smart scale to measure liquid dosages, or perhaps a facial recognition layer to ensure the right person is taking the right meds?

Let me know in the comments! And don't forget to visit wellally.tech/blog for more inspiration on building high-impact AI applications. Happy hacking!

Your Grandma’s New Best Friend: A Private, Offline AI Medication Assistant on Raspberry Pi 🥑

wellallyTech — Sat, 27 Jun 2026 01:45:00 +0000

Privacy is not just a feature; it's a human right—especially when it comes to healthcare data. In a world where every "Smart Home" device seems to be a glorified data-harvesting machine, building a private voice assistant that runs entirely offline is the ultimate flex.

Today, we are building a Local Medication Reminder & Interaction Checker. This system uses Edge AI to help elderly users track their pills, check for dangerous drug interactions, and log daily intake—all without a single packet of data leaving your home network. By leveraging Raspberry Pi, Ollama, and Whisper.cpp, we're turning a $70 computer into a life-saving, privacy-first healthcare companion.

Why Edge AI for Healthcare?

When dealing with sensitive medical logs, "the cloud" is just someone else's computer that can be hacked or sold to advertisers. By using a Local LLM and offline Speech-to-Text (STT), we ensure:

Zero Latency: No waiting for a server in Virginia to tell you if you can take Aspirin with Warfarin.
Total Privacy: Your medical history stays on the SD card.
Resilience: It works even if the Wi-Fi goes down.

The System Architecture

To make this work on a resource-constrained device like the Raspberry Pi 5 (or even a Pi 4 with 8GB RAM), we need an efficient pipeline. We use Whisper.cpp for its incredible C++ optimization and Piper for lightning-fast text-to-speech.

graph TD
    A[User Speaks] -->|Audio Capture| B(Whisper.cpp STT)
    B -->|Text String| C{Local LLM - Ollama}
    C -->|Drug Interaction Check| D[Medication Database/Knowledge]
    C -->|Response Text| E(Piper TTS)
    E -->|Audio Out| F[Speaker]
    C -->|Log Entry| G[(Local SQLite/JSON Log)]
    style C fill:#f96,stroke:#333,stroke-width:2px

Prerequisites

Before we dive in, make sure you have:

Raspberry Pi 5 (Recommended) or Pi 4 (8GB).
USB Microphone & a Speaker (3.5mm or USB).
Ollama installed on your Pi.
Python 3.10+.
Tech Stack: Whisper.cpp, Ollama, Piper TTS, Python.

Step 1: Speech-to-Text with Whisper.cpp

We don't want the heavy Python overhead of the original Whisper. Whisper.cpp is optimized for ARM processors.

# Clone and build whisper.cpp
git clone https://github.com/ggerganov/whisper.cpp
cd whisper.cpp
make
# Download the "base" or "tiny" model for speed
bash ./models/download-ggml-model.sh base.en

You can now run a command to transcribe audio locally. In our final Python script, we'll use a subprocess or a Python binding to capture the "Hey, I just took my Ibuprofen" command.

Step 2: The Brain - Ollama & Drug Interaction Logic

We need a model that is small but smart. Phi-3 or Llama-3-8B (quantized) works wonders here. We will give the LLM a specific "System Prompt" to act as a medical safety checker.

The System Prompt:

"You are a private medical assistant. You track medications. If a user says they are taking a drug, check it against their history for interactions. Be concise and prioritize safety."

import ollama

def check_medication_safety(new_drug, history):
    prompt = f"""
    User wants to take: {new_drug}
    Patient History: {history}
    Are there any known interactions? Answer in 2 sentences max. 
    If safe, say 'SAFE'. If dangerous, say 'WARNING'.
    """

    response = ollama.generate(model='phi3', prompt=prompt)
    return response['response']

# Example usage
current_history = ["Warfarin", "Vitamin D"]
new_input = "Aspirin"
print(check_medication_safety(new_input, current_history))

Step 3: Giving it a Voice with Piper TTS

Google TTS requires an API key. Piper is a local, neural TTS that sounds remarkably human and runs in milliseconds on a Pi.

# Install Piper
echo "I am checking your medication logs now." | \
  ./piper/piper --model en_US-lessac-medium.onnx --output_file welcome.wav

Step 4: The "Glue" Script

Here is a simplified logic flow of our main Python application. This handles the "Listen -> Think -> Speak" loop.

import subprocess
import json
import datetime

def record_and_transcribe():
    # Call whisper.cpp to listen for 5 seconds
    # Simplified for demonstration
    result = subprocess.run(["./whisper.cpp/main", "-m", "models/ggml-base.en.bin", "-t", "4"], capture_output=True)
    return result.stdout.decode('utf-8')

def main_loop():
    print("🚀 Local Health Assistant Active...")
    while True:
        user_text = record_and_transcribe()

        if "took" in user_text.lower():
            # Extract drug name and log it
            response = check_medication_safety(user_text, "Current Logs...")

            # Speak the safety check
            subprocess.run(["./piper/piper", "--model", "model.onnx", "--text", response])

            # Save to local log
            with open("med_logs.json", "a") as f:
                json.dump({"time": str(datetime.datetime.now()), "event": user_text}, f)

if __name__ == "__main__":
    main_loop()

Going Beyond the Prototype 🚀

Building a "hello world" for Edge AI is easy, but making it production-ready for an actual elderly care scenario requires more robust handling of medical ontologies and fail-safe mechanisms.

For more advanced patterns on Edge AI deployment, LLM quantization for ARM, and secure healthcare data architectures, I highly recommend checking out the deep-dive articles at WellAlly Tech Blog. They provide excellent resources on how to scale these local models while maintaining strict HIPAA-like privacy standards in a DIY environment.

Conclusion

We’ve successfully outlined a system that:

Listens using Whisper.cpp.
Reasons about drug safety using Ollama (Phi-3).
Speaks back via Piper TTS.
Stores data locally for 100% privacy.

This project isn't just about code; it's about using Edge AI to provide dignity and safety to those who might struggle with complex medication schedules.

What would you add to this? A pill-bottle camera using the Pi Camera? A notification sent to a family member's phone? Let's discuss in the comments! 👇

💡 From Glucose Spikes to Low-GI Menus: Building a Closed-Loop Health Agent with LangGraph

wellallyTech — Fri, 26 Jun 2026 01:50:00 +0000

Have you ever looked at a fitness app and thought, "I wish this didn't just tell me I'm doing poorly, but actually fixed my dinner for me?" 🥗

Most health apps are passive. They monitor, they alert, and then they leave the heavy lifting of behavior change to you. But we are in the era of AI Agents. Today, we’re going to build a Closed-Loop Health Agent using LangGraph. This agent doesn't just watch your blood sugar; it listens to your Dexcom CGM (Continuous Glucose Monitor) data, identifies anomalies, and proactively generates a personalized, low-GI recipe and shopping list in Notion.

If you're interested in building production-ready health tech, we’ll be using LangGraph state management, LLM Tool Calling, and a bit of Python magic.

Pro-Tip: For more production-ready patterns and advanced AI architecture deep-dives, definitely check out the engineering guides over at WellAlly Blog. They specialize in scaling these types of health-centric LLM workflows.

🏗️ The Architecture: A Reactive Health Loop

Unlike a simple chatbot, a health agent needs to maintain "state." It needs to know what your current glucose level is, what the previous reading was, and whether an action has already been taken.

Here is the logic flow of our LangGraph agent:

graph TD
    A[Start] --> B{Check Glucose Level}
    B -- Level Normal --> C[Log & Sleep]
    B -- Level High/Spiking --> D[Invoke LLM: Nutritionist Persona]
    D --> E[Generate Low-GI Recipe]
    E --> F[Tool: Search Grocery Inventory]
    F --> G[Tool: Push to Notion]
    G --> H[End Loop]
    C --> A

🛠️ Tech Stack

LangGraph: To handle the cyclical graph logic and state management.
OpenAI GPT-4o: Our "Brain" for nutritional reasoning.
Dexcom API (Simulated): For real-time glucose telemetry.
Notion API: Our UI for the final shopping list and meal plan.

👨‍💻 Step 1: Defining the Agent State

In LangGraph, the State is a shared schema that evolves as it moves through the nodes. We need to track the current glucose value and the generated recommendations.

from typing import TypedDict, List, Annotated
import operator

class HealthAgentState(TypedDict):
    # Tracks the current glucose reading
    glucose_level: int
    # History of messages for the LLM
    messages: Annotated[List[str], operator.add]
    # The final recipe to be pushed to Notion
    recipe_plan: str
    # Whether an intervention is needed
    alert_triggered: bool

🛠️ Step 2: Creating the Tools (Dexcom & Notion)

We need to give our agent "hands." We'll define a tool to check the glucose and a tool to write to Notion.

from langchain_core.tools import tool

@tool
def check_glucose_telemetry():
    """Fetches the latest reading from the Dexcom CGM."""
    # In a real scenario, you'd use the Dexcom Share API
    # Here, we simulate a 'high' reading of 185 mg/dL
    return 185 

@tool
def push_to_notion(content: str):
    """Pushes the Low-GI meal plan to the Notion Dashboard."""
    # Notion API Logic here...
    print(f"✅ Successfully pushed to Notion: {content[:50]}...")
    return "Notion Updated Successfully"

tools = [check_glucose_telemetry, push_to_notion]

🧠 Step 3: The Reasoning Node

This is where the agent decides what to do. If the glucose is high, it triggers the "Nutritionist" logic.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools(tools)

def nutrition_expert_node(state: HealthAgentState):
    glucose = state['glucose_level']
    prompt = f"The user's glucose is {glucose} mg/dL. This is high. Generate a low-GI meal plan."

    response = llm.invoke(prompt)
    return {"messages": [response], "alert_triggered": True}

🕸️ Step 4: Connecting the Graph

Now we wire everything together. We define the nodes and the conditional edges.

from langgraph.graph import StateGraph, END

workflow = StateGraph(HealthAgentState)

# Define the Nodes
workflow.add_node("monitor", lambda state: {"glucose_level": check_glucose_telemetry()})
workflow.add_node("nutritionist", nutrition_expert_node)
workflow.add_node("notifier", lambda state: {"recipe_plan": push_to_notion(state['messages'][-1].content)})

# Define the Edges
workflow.set_entry_point("monitor")

def should_intervene(state):
    if state["glucose_level"] > 140:
        return "intervene"
    return "wait"

workflow.add_conditional_edges(
    "monitor",
    should_intervene,
    {
        "intervene": "nutritionist",
        "wait": END
    }
)

workflow.add_edge("nutritionist", "notifier")
workflow.add_edge("notifier", END)

app = workflow.compile()

🚀 The Result: Real-Time Intervention

When this agent runs, it doesn't just output text to a console. It:

Observes: Hits the Dexcom API.
Analyzes: Realizes the user's blood sugar is spiking.
Acts: Drafts a recipe (e.g., Lemon Herb Grilled Salmon with Quinoa) specifically designed to flatten the glucose curve.
Delivers: Pushes that recipe directly to the user’s "Meal Planning" database in Notion.

Why this matters

The "stateful" nature of LangGraph ensures that if the Notion push fails, the agent knows where it left off. It turns a simple LLM into a reliable health system.

📚 Going Further

Building agents for healthcare requires more than just a clever prompt. You need to consider data privacy (HIPAA), low-latency monitoring, and robust error handling.

For a deeper dive into how to secure these agents and integrate them with more complex medical data sources, I highly recommend checking out the technical articles at wellally.tech/blog. They have some fantastic resources on the intersection of AI and preventative health.

What are you building with LangGraph? Drop a comment below or share your latest automation flow! 👇

Stop Guessing Calories! Build a Self-Calibrating AI Nutritionist with SegFormer & VLMs

wellallyTech — Thu, 25 Jun 2026 01:45:00 +0000

Let's be honest: manual diet tracking is a nightmare. You spend more time searching for "medium-sized avocado" in a database than actually eating it. Most apps rely on user input, which is notoriously inaccurate because humans are terrible at estimating portions.

What if we could use Computer Vision and Vision Language Models (VLM) to turn a simple photo into a detailed nutritional breakdown? In this guide, we’re building a system that uses Image Segmentation (SegFormer) and Object Detection (Grounding DINO) to estimate food volume and automate nutrient labeling. If you're looking to master automated nutrition tracking or VLM API integration, you're in the right place!

The Architecture: From Pixels to Proteins

To get accurate results, we can't just look at a 2D image. We need depth or a reference. Our pipeline uses a "Dual-Perspective" approach: a top-down view and a side view (or a reference object like a coin/credit card).

graph TD
    A[User Takes 2 Photos] --> B[Grounding DINO: Identify Food Items]
    B --> C[SegFormer: Pixel-Perfect Segmentation Masks]
    C --> D[Volume Estimation Engine]
    D --> E[VLM API: Contextual Refinement]
    E --> F[Nutritionix API: Nutrient Retrieval]
    F --> G[FastAPI Response: Calories/Macros]

    subgraph "The Vision Core"
    B
    C
    D
    end

Prerequisites

To follow along, you'll need a basic understanding of Python and the following stack:

FastAPI: For the high-performance backend.
SegFormer: For precise semantic segmentation.
Grounding DINO: To detect specific food items via text prompts.
VLM API (GPT-4o or Claude 3.5 Sonnet): For high-level reasoning.

Step 1: Detecting and Segmenting the Plate

Generic object detection isn't enough; we need to know exactly which pixels belong to the steak and which to the fries. We use Grounding DINO to find the items and SegFormer to generate the mask.

from transformers import AutoModelForSemanticSegmentation, TrainingArguments, Trainer

# Loading a pre-trained SegFormer model for fine-tuning on food datasets
model_name = "nvidia/mit-b0"
model = AutoModelForSemanticSegmentation.from_pretrained(
    model_name, 
    num_labels=150 # ADE20K classes or custom food classes
)

def get_segmentation_mask(image):
    # Logic to process image through SegFormer
    # Returns a mask where each pixel is labeled
    inputs = feature_extractor(images=image, return_tensors="pt")
    outputs = model(**inputs)
    return outputs.logits.argmax(1)

Step 2: Volume Estimation (The "Secret Sauce")

By taking two photos (Top and Side), we can approximate the geometry. We calculate the area ($A$) from the top view and the height ($h$) from the side view.

$$Volume \approx Area_{top} \times Height_{avg}$$

We calibrate these measurements by detecting a known object in the frame (like a 25-cent coin) to establish a "Pixels-to-Centimeters" ratio.

Step 3: Integrating the VLM for Smart Labeling

The Vision Language Model (VLM) acts as the brain. It takes the cropped images and the estimated volume to verify if the "Apple" detected is a "Granny Smith" or a "Honeycrisp," which have different caloric densities.

For more production-ready patterns on how to handle complex VLM prompts and structured data extraction, I highly recommend checking out the advanced tutorials over at WellAlly Tech Blog. They have fantastic deep dives into AI-integrated health tech.

import openai
from pydantic import BaseModel

class FoodAnalysis(BaseModel):
    food_name: str
    estimated_weight_grams: float
    confidence_score: float
    ingredients: list[str]

async def refine_with_vlm(image_url, volume_est):
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": f"This food has an estimated volume of {volume_est}cm3. Identify it precisely."},
                    {"type": "image_url", "image_url": {"url": image_url}}
                ],
            }
        ],
        response_format={ "type": "json_object" }
    )
    return response.choices[0].message.content

Step 4: The FastAPI Implementation

Finally, we wrap everything in a FastAPI endpoint to handle the requests from our mobile frontend.

from fastapi import FastAPI, File, UploadFile

app = FastAPI(title="NutriVision AI")

@app.post("/analyze-meal")
async def analyze_meal(top_view: UploadFile = File(...), side_view: UploadFile = File(...)):
    # 1. Run Grounding DINO to find the 'plate' and 'reference object'
    # 2. Run SegFormer for masks
    # 3. Calculate Volume
    # 4. Refine with VLM API
    # 5. Fetch data from Nutritionix

    result = {
        "item": "Grilled Chicken Salad",
        "calories": 450,
        "protein": "35g",
        "carbs": "12g",
        "fat": "22g",
        "accuracy_index": 0.94
    }
    return result

Challenges & Solutions

Overlapping Food: SegFormer struggles when the broccoli is hiding under the salmon.
- Solution: Use the VLM to "infer" the hidden volume based on common plate arrangements.
Lighting and Shadows: Can distort height estimation.
- Solution: Use a Canny edge detector as a pre-processing step to better define the food boundaries in the side view.

Conclusion

Building a self-calibrating diet tracker is a perfect project to transition from "Basic AI" to "Advanced Vision Systems." By combining the precision of SegFormer with the reasoning capabilities of GPT-4o/VLMs, we move from "guessing" to "measuring."

If you're interested in the full source code or want to see how we handle edge cases like "soup" or "transparent containers," head over to wellally.tech/blog for the deep-dive version of this architecture.

What are you building with Vision Models? Drop a comment below!

Secure Your Genome: Calculating Genetic Risk Scores Using Homomorphic Encryption and TenSEAL 🔐🧬

wellallyTech — Wed, 24 Jun 2026 01:40:00 +0000

Imagine a world where you can receive a life-saving medical diagnosis without ever showing your DNA to a doctor, a lab, or a tech giant. Sounds like science fiction? Well, thanks to Homomorphic Encryption (HE), it’s becoming our new reality.

In this tutorial, we’re diving deep into Genomic Data Privacy and Secure Multiparty Computation. We will explore how to calculate a Genetic Risk Score (GRS) using TenSEAL and Python, ensuring that sensitive genetic markers remain encrypted even during the computation phase. Whether you are building the next generation of HealthTech or you're a privacy advocate, mastering Homomorphic Encryption in Python is a superpower you need in your stack. 🚀

The Privacy Paradox in Digital Health

Genomic data is the ultimate PII (Personally Identifiable Information). Unlike a password, you can't "reset" your DNA if it gets leaked. However, to calculate risks for diseases like Type 2 Diabetes or Heart Disease, we often need to run Logistic Regression models over these datasets.

The challenge? How do we let a third-party cloud provider run these complex calculations without actually seeing the raw data?

The Solution: Homomorphic Encryption (CKKS Scheme)

Homomorphic Encryption allows us to perform mathematical operations (addition and multiplication) directly on ciphertexts. The result, when decrypted, matches the result of operations performed on the plaintext. For floating-point numbers (like genetic weights), we use the CKKS (Cheon-Kim-Kim-Song) scheme.

The Architecture of Private Computation

Here is how the data flows between the Data Owner (The Patient) and the Computation Provider (The Cloud):

sequenceDiagram
    participant Patient as Patient (Data Owner)
    participant Cloud as Cloud (Processor)

    Note over Patient: Generates Secret & Public Keys
    Patient->>Patient: Encrypts Genomic Markers (Vector)
    Patient->>Cloud: Sends Public Context + Encrypted Vector
    Note over Cloud: Loads Pre-trained Model Weights (Plaintext)
    Cloud->>Cloud: Performs Homomorphic Dot Product
    Cloud->>Patient: Returns Encrypted Risk Score
    Patient->>Patient: Decrypts with Secret Key
    Note over Patient: Result: Genetic Risk Score Revealed!

Prerequisites

To follow along, you'll need a Python environment with the following libraries:

TenSEAL: A library for MS SEAL, making HE accessible in Python.
NumPy: For vector manipulations.

pip install tenseal numpy

Step-by-Step Implementation

1. Setting up the Encryption Context

The "Context" defines the parameters of our encryption scheme (polynomial degree, coefficient modulus, etc.). This is the "rulebook" for our math.

import tenseal as ts
import numpy as np

def create_context():
    # CKKS scheme is perfect for floating-point numbers
    context = ts.context(
        ts.SCHEME_TYPE.CKKS,
        poly_modulus_degree=8192,
        coeff_mod_bit_sizes=[60, 40, 40, 60]
    )
    # This scale determines the precision of our fractional parts
    context.global_scale = 2**40
    # Create the public key for the Cloud to use
    context.generate_galois_keys()
    return context

context = create_context()

2. Encrypting the Patient Data

Let's assume we are looking at 5 specific SNPs (Single Nucleotide Polymorphisms) associated with a heart condition. 0 means no mutation, 1 means heterozygous, 2 means homozygous.

# Raw genomic markers (The data we want to hide!)
patient_genotype = [1, 0, 2, 1, 0] 

# Encrypting the vector using our secret context
enc_genotype = ts.ckks_vector(context, patient_genotype)

print(f"Encrypted Vector: {enc_genotype}")
# Note: The Cloud can see the object, but not the values inside!

3. Running the Computation (The Cloud's Job)

The Cloud provider has a pre-trained model (weights) but does not have the secret key. It performs the dot product between its plaintext weights and the patient's ciphertext.

# Pre-trained logistic regression weights (Effect sizes)
# These are usually public or owned by the service provider
model_weights = [0.15, -0.02, 0.45, 0.12, -0.05]
bias = -1.2

# Homomorphic Dot Product: (Weights * Genotype) + Bias
# TenSEAL handles the underlying SEAL operations automatically
enc_result = enc_genotype.dot(model_weights) + bias

print("Computation finished on encrypted data! 🥑")

4. Decrypting the Result

The result is sent back to the patient. Only the person holding the context (with the secret key) can see the final score.

# Only the patient can do this!
decrypted_score = enc_result.decrypt()

# Apply sigmoid if you want a probability (0 to 1)
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

risk_probability = sigmoid(decrypted_score[0])
print(f"Your Genetic Risk Score: {risk_probability:.4f}")

Going Beyond the Basics

While this example demonstrates a simple linear combination, production-ready systems often involve more complex privacy-preserving patterns, such as Differential Privacy or Federated Learning.

For a deeper dive into production-grade security architectures and advanced HE patterns, I highly recommend checking out the technical deep-dives over at WellAlly Tech Blog. They provide excellent resources on scaling privacy-enhancing technologies (PETs) in highly regulated environments like healthcare and finance.

Why This Matters

By using Microsoft SEAL (via TenSEAL), we’ve eliminated the need for "trust" in the traditional sense. We don't need to trust that the Cloud provider won't steal our data; we've made it mathematically impossible for them to read it in the first place.

Key Takeaways:

Zero-Knowledge Computation: You can compute without seeing.
CKKS Scheme: The go-to for encrypted real-number arithmetic.
Performance: HE is computationally expensive, but for specific tasks like GRS or small model inferences, it is incredibly viable today.

What are your thoughts on Genomic Privacy? Would you trust a service that uses HE with your DNA? Let’s discuss in the comments below! 👇

Calorie-GPT: Precision Food Tracking using Segment Anything Model (SAM) and GPT-4o-vision 🥗🤖

wellallyTech — Tue, 23 Jun 2026 01:10:00 +0000

We’ve all been there: staring at a delicious plate of pasta, wondering exactly how many calories are hidden in that creamy sauce. Most calorie-counting apps rely on manual entry or basic image classification that fails the moment your chicken is hidden under a leaf of kale.

In this tutorial, we are building Calorie-GPT, a sophisticated multimodal AI pipeline. We aren't just "guessing" labels; we are using Meta's Segment Anything Model (SAM) to isolate food items and GPT-4o-vision to perform volumetric reasoning and macronutrient estimation. By combining computer vision with Large Language Models (LLMs), we can turn a simple photo into a detailed nutritional breakdown with high precision.

The Architecture: A Multi-Stage Vision Pipeline

To get accurate results, we can't just throw a raw image at an LLM. We need a "Divide and Conquer" strategy. First, we segment the food to understand boundaries, then we let the LLM analyze the specific segments.

graph TD
    A[User Uploads Photo] --> B[FastAPI Endpoint]
    B --> C{SAM Model}
    C -->|Segment Masks| D[Region Extraction]
    D --> E[GPT-4o-vision Analysis]
    E -->|JSON Response| F[PostgreSQL Storage]
    F --> G[Frontend Display]

    subgraph "The AI Engine"
    C
    D
    E
    end

Why this stack?

Segment Anything Model (SAM): Precise zero-shot segmentation to identify exactly where the food is.
GPT-4o-vision: The gold standard for visual reasoning and estimating portion sizes.
FastAPI: High-performance Python framework to glue our microservices together.
PostgreSQL: To store user history and nutritional logs for long-term tracking.

Prerequisites

Before we dive in, ensure you have:

Python 3.9+
An OpenAI API Key
The segment-anything library and weights (sam_vit_h).

Step 1: Precise Segmentation with SAM

SAM allows us to generate masks for every object in the image. This prevents the AI from getting confused by the table, the cutlery, or the background.

import numpy as np
from segment_anything import SamPredictor, sam_model_registry

# Load the SAM model
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth")
predictor = SamPredictor(sam)

def get_food_segments(image_array):
    predictor.set_image(image_array)
    # For simplicity, we use the center point as a prompt
    # In production, use automatic mask generation
    masks, scores, logits = predictor.predict(
        point_coords=np.array([[image_array.shape[1]//2, image_array.shape[0]//2]]),
        point_labels=np.array([1]),
        multimask_output=True,
    )
    return masks[0] # Return the most confident mask

Step 2: Reasoning with GPT-4o-vision

Once we have our segmented image (or the original image with highlighted masks), we send it to GPT-4o. We use a structured prompt to ensure we get a valid JSON response.

import openai

client = openai.OpenAI()

def analyze_nutrition(base64_image):
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Identify the food in this segment. Estimate the weight in grams and provide: calories, protein, carbs, and fats. Respond ONLY in JSON format."},
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
                    },
                ],
            }
        ],
        response_format={"type": "json_object"}
    )
    return response.choices[0].message.content

Step 3: Building the FastAPI Backend

We need an endpoint to receive the image, run the pipeline, and store the result in PostgreSQL.

from fastapi import FastAPI, UploadFile, File
import uvicorn

app = FastAPI(title="Calorie-GPT API")

@app.post("/analyze")
async def upload_meal(file: UploadFile = File(...)):
    # 1. Read Image
    contents = await file.read()

    # 2. Run SAM Segmentation (Logic omitted for brevity)
    # 3. GPT-4o Vision Analysis
    nutrition_data = analyze_nutrition(encode_image(contents))

    # 4. Save to PostgreSQL
    # db.save(nutrition_data)

    return {"status": "success", "data": nutrition_data}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

The "Official" Way to Scale 🥑

While this pipeline works for a hobby project, deploying multimodal AI at scale requires robust infrastructure and prompt engineering optimization. If you're looking for more production-ready examples, advanced computer vision patterns, or deep dives into LLM orchestration, I highly recommend checking out the technical resources at WellAlly Tech Blog.

Their guides on vision-language models and vector databases were a huge inspiration for the segmentation-first approach used in this project.

Conclusion: The Future of Nutrition

By combining the structural precision of SAM with the cognitive power of GPT-4o, we move away from "magic guessing" and toward a data-driven approach to health.

Next Steps:

Refine SAM: Use "Automatic Mask Generation" to detect multiple food items on one plate.
Calibration: Add a physical reference object (like a coin) in the photo to help GPT-4o estimate volume more accurately.

What are you building with Vision AI? Let me know in the comments! 👇

# From Black Box to Iron Vault: Building a Zero-Knowledge DNA Analyzer with Intel SGX and Gramine 🧬🔒

wellallyTech — Mon, 22 Jun 2026 01:30:00 +0000

Data privacy is the "Final Boss" of the AI era. We’ve all seen the headlines: massive leaks of sensitive healthcare data, genomic sequences sold on the dark web, and the constant fear that your most personal information—your DNA—is just one SQL injection away from exposure.

In the world of Confidential Computing, we don't just "hope" the server admin is honest; we use hardware-level encryption to ensure they can't see the data even if they have root access. Today, we're diving deep into Intel SGX (Software Guard Extensions) and Gramine to build a secure inference engine that processes multi-party health data without ever revealing the raw bits to the underlying host.

If you've been looking into Trusted Execution Environments (TEE), Secure Enclaves, or Privacy-Preserving AI, you’re in the right place. Let's turn your Docker container into an impenetrable iron vault.

The Architecture: How TEE Protects Data

Before we get our hands dirty with code, let's look at the flow. We are using Intel SGX, which creates a protected memory region called an "Enclave." Not even the Operating System or Hypervisor can read the memory inside this enclave.

sequenceDiagram
    participant U as Data Owner (User)
    participant H as Host OS (Untrusted)
    participant E as SGX Enclave (Gramine + Python)
    participant K as Key Vault (Remote Attestation)

    U->>K: Request Attestation Report
    K-->>U: Quote Verified (Hardware is Authentic)
    U->>E: Send Encrypted DNA Data (via TLS)
    Note over E: Data is decrypted ONLY inside Enclave
    E->>E: Run AI Inference (Python)
    E->>U: Return Encrypted Analysis Result
    Note over H: Host sees nothing but encrypted noise

Prerequisites

To follow along with this advanced tutorial, you’ll need:

A CPU with Intel SGX support (and SGX enabled in BIOS).
Docker installed.
Gramine: The lightweight OS that allows us to run unmodified Linux binaries inside SGX.
Basic knowledge of Python and Linux security.

Step 1: The "Genomic Analyzer" (Python)

Let's write a simple Python script that simulates a complex genomic analysis. In a real-world scenario, this would be a PyTorch or TensorFlow model.

# analyzer.py
import os

def analyze_genomic_data(file_path):
    # In reality, this would be an encrypted stream or a secure mount
    with open(file_path, "r") as f:
        raw_data = f.read()

    print(f"--- [SECURE ENCLAVE] Processing DNA Sequence ---")
    # Simulating a sophisticated AI analysis
    risk_score = len(raw_data) % 100 

    return f"Analysis Complete. Genetic Risk Score: {risk_score}/100"

if __name__ == "__main__":
    # The path to the data provided by the user
    input_data = "/data/dna_sequence.txt"
    if os.path.exists(input_data):
        result = analyze_genomic_data(input_data)
        print(result)
    else:
        print("Error: No data found in the secure mount.")

Step 2: The Gramine Manifest (`python.manifest.template`)

Gramine requires a manifest file to define what the Enclave is allowed to do. This is where we define our "Trusted Files."

# python.manifest.template
loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ entrypoint }}"

loader.log_level = "error"

# Define the environment
loader.env.LD_LIBRARY_PATH = "/lib:/lib/x86_64-linux-gnu"
loader.env.PYTHONPATH = "/usr/lib/python3.10"

# Mount points for our code and data
fs.mounts = [
  { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
  { path = "/usr/lib", uri = "file:/usr/lib" },
  { path = "/etc", uri = "file:/etc" },
  { path = "/data", uri = "file:data" }, # Our sensitive data
]

# Mark our script as a trusted file (it will be measured)
sgx.trusted_files = [
  "file:{{ gramine.libos }}",
  "file:{{ entrypoint }}",
  "file:analyzer.py",
  "file:/usr/lib/python3.10/",
]

sgx.enclave_size = "2G"
sgx.thread_num = 4

Step 3: Containerizing with Confidentiality

We need to build a Docker image that includes the SGX drivers and the Gramine runtime.

# Dockerfile
FROM gramineproject/gramine:latest

# Install Python
RUN apt-get update && apt-get install -y python3

WORKDIR /app
COPY analyzer.py .
COPY python.manifest.template .

# Create a data directory for our sensitive input
RUN mkdir /app/data

# Generate the SGX-specific files (Sigstruct and Token)
RUN gramine-sgx-gen-sigstruct python.manifest.template > python.sig \
    && gramine-sgx-get-token --output python.token --sig python.sig

ENTRYPOINT ["gramine-sgx", "python"]

The "Official" Way to Production 🛡️

While this setup gets you running locally, production-grade Confidential Computing requires robust Remote Attestation. This is the process where the hardware proves to the user, "Hey, I really am an Intel SGX enclave running specifically this code."

If you are looking for more production-ready patterns, advanced attestation workflows, or deep dives into the performance overhead of TEEs, I highly recommend checking out the technical deep dives at WellAlly Tech Blog. They provide extensive resources on securing distributed AI workloads and building zero-trust architectures for healthcare.

Running the Enclave

Once you've built your image, you can run it. Note that you must pass the SGX devices from the host to the container:

docker run --rm \
  --device=/dev/sgx_enclave \
  --device=/dev/sgx_provision \
  -v $(pwd)/my_sensitive_dna.txt:/app/data/dna_sequence.txt \
  my-confidential-analyzer

What just happened?

Isolation: The Python interpreter started inside an SGX Enclave.
Integrity: Gramine verified that analyzer.py hadn't been tampered with by checking its hash against the manifest.
Privacy: The host OS (even with sudo) could not peek into the memory addresses where the dna_sequence.txt was being processed.

Conclusion: The Future is Confidential

Building with Intel SGX and Gramine feels like giving your code a superpower. You can now process the world's most sensitive data with the mathematical certainty that it remains private.

As AI continues to eat the world, the developers who understand Privacy-Enhancing Technologies (PETs) will be the ones building the platforms users actually trust.

What's next for you?

Are you implementing SGX in the cloud (Azure DC-series or AWS Nitro Enclaves)?
Have you tried Fully Homomorphic Encryption (FHE) as an alternative?

Let's discuss in the comments! 🚀🥑

For more advanced guides on TEE and secure systems, visit wellally.tech/blog.

Privacy-First Healthcare AI: Building Decentralized Disease Prediction Models with Flower and PyTorch 🛡️🏥

wellallyTech — Sun, 21 Jun 2026 01:30:00 +0000

In the era of Big Data, healthcare AI faces a massive paradox: we need massive datasets to build accurate disease prediction models, but patient privacy laws like HIPAA and GDPR make sharing raw medical data a legal and ethical nightmare. This is where Federated Learning comes to the rescue! 🚀

By leveraging Edge AI and decentralized training techniques, we can train high-performing models on sensitive physiological metrics without a single byte of raw data ever leaving the user's device. In this deep dive, we’ll explore how to use the Flower (flwr) framework and PyTorch to build a privacy-preserving system that aggregates intelligence, not data.

💡 Pro Tip: If you're looking for production-grade architectural patterns for secure healthcare systems, I highly recommend checking out the deep dives over at wellally.tech/blog. They have some fantastic resources on scaling privacy-preserving AI.

The Architecture: Bringing the Model to the Data

Traditional AI sends data to the model (Centralized). Federated Learning sends the model to the data (Decentralized).

Here is how the data flow looks when using a framework like Flower to coordinate multiple hospitals or edge devices:

sequenceDiagram
    participant S as Central Aggregator (Flower Server)
    participant C1 as Hospital A (Edge Client)
    participant C2 as Hospital B (Edge Client)

    S->>C1: Send Global Model Weights (v1)
    S->>C2: Send Global Model Weights (v1)
    Note over C1: Train on Local Patient Data
    Note over C2: Train on Local Patient Data
    C1->>S: Send Updated Weights/Gradients
    C2->>S: Send Updated Weights/Gradients
    Note over S: Aggregate Weights (FedAvg Algorithm)
    S->>S: Update Global Model (v2)
    S->>C1: Send New Global Model (v2)

In this flow, the raw physiological data (heart rate, glucose levels, etc.) stays on the client. Only the "learnings" (weight updates) are shared.

Tech Stack 🛠️

To follow this tutorial, you'll need:

PyTorch: To define our neural network.
Flower (flwr): The orchestration framework for Federated Learning.
Docker: For simulating multiple clients in an isolated environment.
Python 3.9+

Step 1: Defining the Disease Prediction Model

We'll start with a standard PyTorch nn.Module. For healthcare applications, we often deal with tabular data or time-series metrics.

import torch
import torch.nn as nn
import torch.nn.functional as F

class HealthPredictor(nn.Module):
    def __init__(self):
        super(HealthPredictor, self).__init__()
        # Input: 10 physiological markers (e.g., BP, BMI, Age, etc.)
        self.fc1 = nn.Linear(10, 64)
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 2) # Binary classification: Healthy vs. At Risk

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

# Initialize the model
net = HealthPredictor()

Step 2: Implementing the Flower Client 🥑

The "Client" is the code that runs on the hospital's local server. It handles the local training loop and reports updates back to the central server.

import flwr as fl
from collections import OrderedDict

class DiseasePredictionClient(fl.client.NumPyClient):
    def get_parameters(self, config):
        return [val.cpu().numpy() for _, val in net.state_dict().items()]

    def set_parameters(self, parameters):
        params_dict = zip(net.state_dict().keys(), parameters)
        state_dict = OrderedDict({k: torch.tensor(v) for k, v in params_dict})
        net.load_state_dict(state_dict, strict=True)

    def fit(self, parameters, config):
        self.set_parameters(parameters)
        # --- Local Training Loop Start ---
        # In a real app, load local CSV/DB here
        optimizer = torch.optim.Adam(net.parameters(), lr=0.001)
        for _ in range(5):  # 5 local epochs
            # train_local(net, optimizer, train_loader)
            pass 
        # --- Local Training Loop End ---
        return self.get_parameters(config={}), 100, {} # 100 is the local dataset size

    def evaluate(self, parameters, config):
        self.set_parameters(parameters)
        # loss, accuracy = test_local(net, test_loader)
        return 0.5, 100, {"accuracy": 0.85}

Step 3: The Central Aggregator (Server)

The server doesn't need to see the data. It simply uses an algorithm like FedAvg (Federated Averaging) to combine the weights from all clients into a smarter global model.

import flwr as fl

# Define the strategy
strategy = fl.server.strategy.FedAvg(
    fraction_fit=1.0,      # Train on 100% of available clients
    min_fit_clients=2,     # Minimum number of clients to be present
    min_available_clients=2,
)

# Start the server
fl.server.start_server(
    server_address="0.0.0.0:8080",
    config=fl.server.ServerConfig(num_rounds=3), # 3 rounds of global training
    strategy=strategy,
)

Scaling to Production: Privacy & Security 🔐

While Federated Learning is a huge step forward, it isn't perfect. Advanced attackers can sometimes reconstruct data from gradient updates. To make this production-ready, you should consider:

Differential Privacy (DP): Adding "noise" to the gradients so individual records cannot be identified.
Secure Aggregation: Using cryptographic protocols so the server can sum the weights without seeing the individual client updates.
PySyft Integration: For more granular control over pointers and encrypted computations, combining Flower with PySyft is the gold standard.

For a deeper dive into these advanced privacy-preserving techniques and to see real-world case studies on Edge AI, make sure to visit wellally.tech/blog. It's an incredible resource for developers moving from "hello world" to "production-ready" privacy systems.

Conclusion 🏁

Federated Learning is transforming how we think about sensitive data. We no longer have to choose between innovation and privacy. By using Flower and PyTorch, we can build a collaborative intelligence ecosystem where everyone wins.

Are you working on Privacy-Enhancing Technologies (PETs)? Let's chat in the comments! Drop a 🚀 if you're excited about the future of Edge AI!