DEV Community: wellallyTech

Automate the Impossible: Building a Closed-Loop Medical Appointment Agent with Browser-use & GPT-4V 🚀

wellallyTech — Wed, 11 Nov 2026 01:34:00 +0000

Have you ever tried navigating a clunky hospital portal? Between the cryptic department names and the endless clicking, it’s a UI nightmare. But what if your computer could "see" the screen, reason like a human, and handle the entire appointment process for you?

In this guide, we’re building a Closed-Loop Medical Agent. We will leverage Vision LLM Agents, browser automation, and the cutting-edge Browser-use framework to transform a simple symptom description into a confirmed doctor's appointment. This isn't just a scraper; it's a self-correcting AI that interacts with the web in real-time.

The Vision: Why "Closed-Loop" Matters?

Traditional automation breaks the moment a UI change occurs. By using GPT-4V (Vision) and Playwright, our agent doesn't rely on fragile CSS selectors. It looks at the page, understands the context, and decides where to click next.

If you’re interested in scaling these patterns for enterprise use, I highly recommend checking out the advanced architectural deep-dives at WellAlly Tech Blog, which served as a major inspiration for this autonomous workflow.

🏗 The Architecture

The system operates in a feedback loop: Observation -> Reasoning -> Action -> Validation.

graph TD
    A[User: 'My throat hurts and I have a fever'] --> B[GPT-4V: Analyze Symptoms]
    B --> C[Agent: Identify 'Otolaryngology' Department]
    C --> D[Browser-use: Open Medical Portal]
    D --> E{Is Dept Visible?}
    E -- No --> F[Search/Scroll Action]
    E -- Yes --> G[Select Doctor & Time]
    G --> H[Final Confirmation]
    H --> I[Validation: Success Screenshot]
    F --> D

🛠 Tech Stack

Browser-use: The ultimate wrapper for making LLMs control browsers.
LangChain: For orchestrating the agent logic.
Playwright: The engine that powers the actual browser instance.
GPT-4o/GPT-4V: The "brain" that provides vision and reasoning.

💻 Implementation: Step-by-Step

1. Environment Setup

First, let's install the core dependencies. We need browser-use and the standard LangChain OpenAI package.

pip install browser-use langchain-openai playwright
playwright install

2. Initializing the Vision Agent

The magic of browser-use is that it allows us to pass a simple "Task" string, and the agent handles the multi-step navigation.

from langchain_openai import ChatOpenAI
from browser_use import Agent
import asyncio

# Initialize the Vision-capable LLM
llm = ChatOpenAI(model="gpt-4o", temperature=0)

async def run_medical_agent(symptoms: str):
    # The prompt defines the goal and the constraints
    task = f"""
    Go to the 'https://example-medical-portal.com'
    1. Based on the symptoms: '{symptoms}', determine the correct department.
    2. Search for the department and find a doctor with the earliest availability.
    3. Select a time slot and stop right before the 'Confirm Payment' button.
    4. Take a screenshot of the final selection.
    """

    agent = Agent(
        task=task,
        llm=llm,
    )

    result = await agent.run()
    print(f"Agent Execution Summary: {result}")

if __name__ == "__main__":
    asyncio.run(run_medical_agent("I've had a sharp pain in my lower back for 2 days."))

3. Handling Complex UI Logic

The agent doesn't just "click." It perceives. If a popup appears (like a COVID-19 warning or a login prompt), the GPT-4V brain realizes it needs to close it before proceeding. This is the "Closed-Loop" advantage.

4. Safety and Verification

For sensitive tasks like healthcare, we don't want the agent hallucinating. We can enforce Schema Validation by asking the agent to extract specific data into a Pydantic model at the end of the run.

💡 Production-Ready Patterns

Building a hobby project is easy, but making it reliable is hard. When moving to production, you need to consider:

Session Management: Reusing browser contexts to stay logged in.
Human-in-the-loop (HITL): Pausing the agent to let a human confirm the appointment time before the final click.
Error Recovery: Using "Backtrack" strategies when the agent gets stuck in a navigation loop.

For more production-grade patterns, including how to handle CAPTCHAs and complex authentication in AI agents, head over to the WellAlly Tech Blog. They have a fantastic series on Agentic Workflows that covers these edge cases in depth.

🚀 Conclusion

The era of "clicking" is ending. By combining the vision capabilities of GPT-4o with the execution power of Browser-use, we can build agents that interact with the web exactly like humans do—but faster and more accurately.

Whether it's booking a doctor or automating your expense reports, the recipe is the same: Observe, Reason, Act.

What would you automate if your browser could think? Let me know in the comments! 👇

Your Browser, Your Doctor: Building a Private Drug Interaction Engine with WebLLM 💊💻

wellallyTech — Tue, 10 Nov 2026 01:30:00 +0000

Privacy in the age of AI is a massive concern, especially when dealing with sensitive medical data. Sending your prescription history or a photo of a medication leaflet to a cloud-based server feels like a gamble with your personal information.

But what if we could process everything—from text extraction to complex reasoning—entirely within the browser? Welcome to the era of Edge AI. In this tutorial, we are building a high-performance, fully offline drug interaction checker using WebLLM, Transformers.js, and a local vector index. By leveraging Local Vector Databases and browser-native inference, we can ensure that zero bytes of medical data ever leave your machine.

Why Browser-Native AI? 🥑

Building a drug interaction engine locally isn't just about privacy; it's about Edge AI performance. Traditional LLM calls involve network latency and high token costs. By running models like Llama 3 or Mistral directly in the browser via WebGPU, we achieve:

Absolute Privacy: Medical data stays in the client's RAM.
Offline Availability: Works in remote clinics or during flight mode.
Zero API Costs: Your GPU does the work, not your wallet.

The Architecture: Local RAG (Retrieval-Augmented Generation)

To check for drug interactions, we can't just feed a 50-page manual to an LLM. We need to chunk the data, embed it, and query the most relevant sections.

graph TD
    A[Drug Leaflet PDF/Text] --> B[Transformers.js]
    B -->|Embedding| C[Local Vector Index]
    D[User Query: Can I take Ibuprofen with this?] --> E[Transformers.js]
    E -->|Query Vector| F{Vector Search}
    C --> F
    F -->|Relevant Context| G[WebLLM Engine]
    G -->|Reasoning| H[Interaction Alert/Safe Confirmation]
    style G fill:#f96,stroke:#333,stroke-width:2px
    style C fill:#bbf,stroke:#333,stroke-width:2px

Prerequisites

Before we dive into the code, ensure you have a modern browser with WebGPU support (Chrome 113+ or Edge).

WebLLM: For running the LLM in the browser.
Transformers.js: For generating vector embeddings locally.
Voyage AI Embeddings: Used as our high-fidelity reference model for indexing.

Step 1: Initializing the Local Embedding Engine

We use Transformers.js to run a feature-extraction pipeline. This turns clinical text into mathematical vectors that we can store in our local index.

import { pipeline } from '@xenova/transformers';

class EmbeddingPipeline {
    static instance = null;

    static async getInstance() {
        if (this.instance === null) {
            // Using a lightweight model for fast browser performance
            this.instance = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
        }
        return this.instance;
    }
}

async function getEmbedding(text) {
    const embedder = await EmbeddingPipeline.getInstance();
    const output = await embedder(text, { pooling: 'mean', normalize: true });
    return Array.from(output.data);
}

Step 2: Setting up WebLLM for Clinical Reasoning

WebLLM allows us to load massive models like Llama-3-8B-Instruct into the browser's WebGPU cache. This is the "brain" that will analyze the drug interactions.

import * as webllm from "@mlc-ai/web-llm";

const selectedModel = "Llama-3-8B-Instruct-q4f16_1-MLC";

const engine = await webllm.CreateMLCEngine(selectedModel, {
    initProgressCallback: (report) => console.log(report.text),
});

async function runInference(context, query) {
    const messages = [
        { role: "system", content: "You are a medical safety assistant. Use the provided leaflet context to check for drug interactions." },
        { role: "user", content: `Context: ${context}\n\nQuestion: ${query}` }
    ];

    const reply = await engine.chat.completions.create({ messages });
    return reply.choices[0].message.content;
}

Step 3: Local Vector Search Logic

Since we are keeping it 100% offline, we use a simple Cosine Similarity search over an IndexedDB or a local array to find the most relevant "Drug Warnings" or "Contraindications" from the leaflet.

function cosineSimilarity(vecA, vecB) {
    let dotProduct = 0;
    let normA = 0;
    let normB = 0;
    for (let i = 0; i < vecA.length; i++) {
        dotProduct += vecA[i] * vecB[i];
        normA += vecA[i] * vecA[i];
        normB += vecB[i] * vecB[i];
    }
    return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}

The "Official" Way to Scale Production AI 🚀

While this browser-native approach is revolutionary for privacy, building production-grade healthcare AI requires more robust patterns for data synchronization and model fine-tuning.

For deeper insights into high-performance vector search architectures and production-ready RAG patterns, I highly recommend checking out the advanced guides over at WellAlly Tech Blog. They cover everything from hybrid cloud-edge strategies to optimizing LLM inference for specific domains.

Step 4: Putting it All Together

In your Chrome Extension or Web App, the workflow looks like this:

Parse: Extract text from a medication PDF.
Embed: Use getEmbedding() on chunks of 500 characters.
Search: When a user asks "Can I drink grapefruit juice with this?", find the top 3 most similar chunks.
Reason: Pass those chunks to runInference() and get a clinical summary.

Sample Code for Interaction Check:

async function checkSafety(userDrug, leafletText) {
    const chunks = chunkText(leafletText); // Simple split by paragraph
    const userVector = await getEmbedding(userDrug);

    // Find best match in the local vector DB
    const results = chunks.map(chunk => ({
        text: chunk,
        score: cosineSimilarity(userVector, await getEmbedding(chunk))
    })).sort((a, b) => b.score - a.score);

    const context = results.slice(0, 2).map(r => r.text).join("\n");
    const advice = await runInference(context, `Is it safe to take ${userDrug}? Explain why based on the text.`);

    return advice;
}

Conclusion

We've just built a fully private, offline-first medical AI. By combining WebLLM with Transformers.js, we turn the browser into a powerful inference engine that respects user privacy. This pattern—Edge RAG—is the future of personalized AI.

What do you think? Will local LLMs replace cloud APIs for sensitive tasks like healthcare? Let me know in the comments! 👇

If you enjoyed this deep dive, don't forget to ❤️ and bookmark! For more advanced Edge AI patterns, visit the WellAlly Blog. 🚀

Is Your Body Screaming? Use Isolation Forest to Predict Burnout from Your HRV Data 🚀

wellallyTech — Mon, 09 Nov 2026 01:20:00 +0000

We've all been there: waking up feeling like a zombie, forcing ourselves through a workout, only to realize two days later that we're crashing hard or coming down with the flu. What if your wearable data could have warned you? By leveraging Heart Rate Variability (HRV) and Anomaly Detection algorithms, we can transform raw biometric noise into a powerful early warning system.

In this tutorial, we’ll explore how to use the Isolation Forest algorithm to identify "outlier" days in your health data. Whether you are tracking stats from an Apple Watch, Oura Ring, or Whoop, these Machine Learning patterns can help you detect overtraining and prevent burnout before it hits. For those looking to dive deeper into production-ready health-tech patterns, I highly recommend checking out the advanced case studies at WellAlly Blog.

Why HRV and Isolation Forest?

Heart Rate Variability (HRV) is the variation in time between each heartbeat. A high HRV usually indicates a well-recovered nervous system, while a sudden drop often signals stress, illness, or overtraining.

However, "low" is relative. Instead of using static thresholds, we use Isolation Forest, an unsupervised learning algorithm that excels at detecting anomalies in multi-dimensional data without needing labeled "sick" vs. "healthy" datasets.

The Architecture 🏗️

The flow is simple: we ingest raw HRV sequences, normalize them, train our forest to recognize "normal" recovery patterns, and flag the anomalies.

graph TD
    A[Raw HRV Data from Wearable] --> B[Feature Engineering: Rolling Averages]
    B --> C[Isolation Forest Model]
    C --> D{Anomaly Score Calculation}
    D -- Score < Threshold --> E[Normal Status ✅]
    D -- Score > Threshold --> F[Warning: Overtraining/Illness Risk ⚠️]
    F --> G[Suggest Rest/Recovery]

Prerequisites 🛠️

To follow along, you’ll need a Python environment with the following tech_stack:

NumPy: For numerical operations.
Scikit-learn: For the Isolation Forest model.
Matplotlib: For visualizing our "burnout" zones.

pip install numpy scikit-learn matplotlib

Step 1: Mocking the HRV Data 🧪

Since we don't have your CSV file yet, let's generate some synthetic data that mimics a typical recovery cycle with a few "crash" points.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import IsolationForest

# Seed for reproducibility
np.random.seed(42)

# Simulate 100 days of HRV data (ms)
# Normal HRV fluctuates around 70ms
normal_days = np.random.normal(70, 10, 90)

# Simulate 10 "Anomaly" days (e.g., heavy drinking, flu, or marathon)
anomaly_days = np.random.normal(30, 5, 10)

# Combine and shuffle
data = plt.scatter(range(100), np.concatenate([normal_days, anomaly_days]))
hrv_sequence = np.concatenate([normal_days, anomaly_days]).reshape(-1, 1)
np.random.shuffle(hrv_sequence)

print(f"Dataset generated: {len(hrv_sequence)} days of HRV tracking.")

Step 2: Training the Isolation Forest 🌲

The beauty of the Isolation Forest is that it doesn't try to define what "normal" is. Instead, it tries to isolate every point. Anomalies (outliers) are easier to isolate and thus require fewer splits in the decision tree.

# Initialize the model
# contamination=0.1 means we expect roughly 10% of data to be anomalies
model = IsolationForest(contamination=0.1, random_state=42)

# Fit the model
model.fit(hrv_sequence)

# Predict: 1 = normal, -1 = anomaly
predictions = model.predict(hrv_sequence)
scores = model.decision_function(hrv_sequence)

Step 3: Visualizing the Red Flags 🚩

Data is useless if you can't see the warning signs. Let's plot our HRV readings and highlight the days where the model says: "Hey, take a nap!"

plt.figure(figsize=(12, 6))

# Plot normal data
plt.scatter(range(len(hrv_sequence)), hrv_sequence, 
            c=['#2ecc71' if p == 1 else '#e74c3c' for p in predictions],
            label='HRV Readings')

plt.axhline(y=hrv_sequence.mean(), color='gray', linestyle='--', alpha=0.5, label='Mean HRV')
plt.title("Burnout Detection: Isolation Forest Anomaly Mapping")
plt.xlabel("Days")
plt.ylabel("HRV (ms)")
plt.legend(['Normal', 'Potential Burnout/Illness'])
plt.show()

The "Official" Way to Scale 🥑

While this script is a great start for personal use, productionizing health-tech algorithms requires handling missing data, sensor noise, and individual baselines.

If you are building a professional health app or want to see how these models are deployed at scale using FastAPI or cloud-native architectures, you should definitely check out the deep-dives over at WellAlly Tech Blog. They cover advanced topics like:

Real-time biometric stream processing.
Personalizing thresholds for diverse user populations.
Integrating Pydantic for strict health data validation.

Conclusion 🏁

Using Isolation Forest for HRV monitoring shifts your health strategy from reactive to proactive. Instead of wondering why you feel tired, you can let the data tell you when your body's "isolation score" is hitting the danger zone.

Next Steps:

Export your Oura or Apple Watch data to CSV.
Replace the mock data in this script with your real values.
Tune the contamination parameter based on how often you actually get sick.

Are you tracking your HRV? Let me know in the comments how you handle recovery! 👇

Privacy vs. Progress: Mastering Differential Privacy for Health Data with Google's PyDP 🛡️⌚

wellallyTech — Sun, 08 Nov 2026 01:19:00 +0000

In the era of wearable tech, our Apple Watches and Oura rings are generating a goldmine of biometric data. While this data is a "holy grail" for medical research, sharing it creates a massive privacy paradox. Even if you remove your name, a unique pattern of heart rate spikes and daily steps can be used in re-identification attacks to dox you.

How do we solve this? Enter Differential Privacy (DP). Instead of just "hiding" names, we use mathematical noise to ensure that the presence or absence of any single individual in a dataset doesn't significantly change the output. In this guide, we'll implement a privacy-preserving pipeline using Google's PyDP, NumPy, and the Laplace Mechanism to share heart rate and step data securely.

We will focus on Differential Privacy, Privacy Engineering, and Data Anonymization—crucial skills for any high-level data engineer today.

The Architecture: Privacy at the Edge

Before we dive into the code, let’s visualize the flow. We aren't just sending raw data to a server; we are injecting "mathematical uncertainty" at the point of collection or aggregation.

graph TD
    A[Individual Health Data: Steps/HR] --> B{Sensitivity Analysis}
    B --> C[Define Privacy Budget - Epsilon]
    C --> D[Google PyDP: Laplace Noise Injection]
    D --> E[Noisy Aggregated Result]
    E --> F[Third-Party Research Lab]
    F -- Cannot Reverse Engineer --> A
    style D fill:#f96,stroke:#333,stroke-width:2px

Prerequisites 🛠️

To follow along with this advanced tutorial, you'll need:

Python 3.8+
PyDP: Python wrapper for Google's Differential Privacy C++ library.
NumPy: For data manipulation.

pip install python-dp numpy

Step 1: Understanding the Math (The Laplace Mechanism)

The core of Differential Privacy is the Privacy Budget ($\epsilon$).

A smaller $\epsilon$ means more noise and higher privacy, but lower utility.
A larger $\epsilon$ means less noise and higher utility, but more risk of data leakage.

We use the Laplace Mechanism, which adds noise drawn from a Laplace distribution proportional to the Sensitivity ($\Delta f$) of the query.

$$Noise = \text{Laplace}(\frac{\Delta f}{\epsilon})$$

Step 2: Implementation with Google PyDP

Let's simulate a dataset of 1,000 users' heart rate data. We want to calculate the average heart rate without exposing any single user's specific pulse.

Defining our Dataset

import numpy as np
from pydp.algorithms.laplacian import BoundedMean

# Simulating raw heart rate data (BPM)
# In a real scenario, this would be fetched from an encrypted database
raw_heart_rates = np.random.normal(loc=72, scale=10, size=1000).tolist()

# Define boundaries (clipping)
# Sensitivity is determined by these bounds. 
# Anything outside 40-120 BPM is treated as 40 or 120.
lower_bound = 40.0
upper_bound = 120.0

Applying Differential Privacy

Now, we use PyDP to calculate the mean with a specific $\epsilon$.

def get_private_mean(data, epsilon=1.0):
    """Calculates the mean of the data with Differential Privacy."""
    # Initialize the BoundedMean algorithm from PyDP
    dp_mean_algo = BoundedMean(epsilon=epsilon, lower_bound=lower_bound, upper_bound=upper_bound)

    # Calculate the noisy mean
    private_mean = dp_mean_algo.quick_result(data)

    return private_mean

# Calculate real mean vs private mean
real_mean = np.mean(raw_heart_rates)
noisy_mean = get_private_mean(raw_heart_rates, epsilon=0.5)

print(f"Actual Mean HR: {real_mean:.2f} BPM")
print(f"Differentially Private Mean: {noisy_mean:.2f} BPM")

Step 3: Analyzing the Privacy-Utility Trade-off

As an engineer, your job is to find the "Sweet Spot." If you're building a fitness leaderboard, you can afford a higher $\epsilon$. If you're sharing sensitive disease-related step counts, you need a lower $\epsilon$.

Visualizing Noise Impact

In a Jupyter Notebook, you can run a loop to see how $\epsilon$ affects accuracy:

epsilons = [0.1, 0.5, 1.0, 5.0, 10.0]
results = []

for e in epsilons:
    # Run 100 trials for each epsilon to see the variance
    trials = [get_private_mean(raw_heart_rates, epsilon=e) for _ in range(100)]
    results.append(trials)

# Note: As Epsilon increases, the distribution of results tightens 
# around the true mean.

The "Official" Way: Production-Ready Privacy 🥑

Implementing DP in a notebook is one thing, but deploying it at scale in a production environment (like a HIPAA-compliant cloud) requires more robust patterns.

For a deep dive into production-grade privacy architectures, including Federated Learning and Zero-Knowledge Proofs for health tech, I highly recommend checking out the advanced engineering guides at WellAlly Tech Blog. They cover how to integrate these algorithms into FastAPI microservices and manage "Privacy Budgets" across multiple API queries.

Step 4: Challenges in Edge AI Implementation

When moving this to Edge AI (mobile devices), you face three main hurdles:

Floating Point Vulnerabilities: Standard Laplacian noise can sometimes be reversed due to how computers handle floating-point rounding. Google's PyDP uses a "secure noise" mechanism to mitigate this.
Global vs. Local Privacy: In Local DP, the noise is added on the phone before it even hits the server. This requires a much higher privacy budget to remain useful.
Data Sensitivity: Determining the upper_bound and lower_bound (clipping) is crucial. If you set the bounds too wide, the noise becomes overwhelming.

Conclusion 🚀

Differential Privacy isn't just a buzzword; it's a mathematical necessity for the future of health tech. By using tools like PyDP, we can move away from "trusting" companies with our data and instead rely on "provable" privacy.

Key Takeaways:

Privacy Budget ($\epsilon$) is your dial for balancing data utility and user anonymity.
Sensitivity must be defined by clipping data (e.g., heart rate bounds).
Laplace Noise is the standard tool for numeric health data.

Are you implementing privacy-preserving features in your current stack? Drop a comment below or share your thoughts on the trade-off between data accuracy and user protection! 🛡️💻

Want to learn more about Edge AI and Privacy? Stay tuned for my next post on Federated Learning or visit the WellAlly Tech Blog for more technical insights!

AI on the Edge: Building a Private Skin Lesion Screener with MobileNetV3, Flutter, and LLMs 🩺📱

wellallyTech — Sat, 07 Nov 2026 01:17:00 +0000

Have you ever looked at a weird mole or a sudden rash and immediately spiraled down a WebMD rabbit hole? We've all been there. The problem with current medical AI is often privacy and complexity. Sending photos of your skin to a random cloud server feels sketchy, and getting a result like "Actinic Keratosis" doesn't help if you don't know what it means.

In this tutorial, we are building a private, edge-first skin lesion screening system. By leveraging Edge AI, we perform real-time classification directly on your smartphone using MobileNetV3 and OnnxRuntime, then use a Multimodal LLM to translate scary medical jargon into plain English.

Whether you're interested in Computer Vision, Flutter development, or OnnxRuntime optimization, this guide covers the full stack of modern on-device machine learning.

🏗️ The System Architecture

To ensure high performance on mobile devices, we don't just throw a raw image at a model. We use a pipeline: Mediapipe crops the region of interest (ROI), MobileNetV3 classifies it, and an LLM explains it.

graph TD
    A[Camera Feed] --> B[Mediapipe Hand/Skin Detection]
    B --> C{Is Skin Detected?}
    C -- Yes --> D[Crop ROI & Pre-process]
    C -- No --> A
    D --> E[MobileNetV3 ONNX Model]
    E --> F[Classification Probabilities]
    F --> G[Local Result Display]
    G --> H[LLM Interpretation Layer]
    H --> I[Human-Readable Explanation]

🛠️ Tech Stack

MobileNetV3 (Small): Our lightweight backbone optimized for mobile CPUs.
Mediapipe: For real-time skin detection and cropping.
OnnxRuntime: The engine that runs our model at native speeds.
Flutter: The cross-platform UI framework.
OpenAI GPT-4o-mini: To provide the "Multimodal Explanation."

🚀 Step 1: Preparing the MobileNetV3 Model

MobileNetV3 is a beast for edge devices because of its use of hard-swish activations and squeeze-and-excitation blocks. We’ll export our trained model (likely from PyTorch or TensorFlow) to the .onnx format.

import torch
import torchvision.models as models

# Load a pre-trained MobileNetV3 Small
model = models.mobilenet_v3_small(pretrained=True)
model.eval()

# Dummy input for the shape (Batch, Channels, Height, Width)
dummy_input = torch.randn(1, 3, 224, 224)

# Export to ONNX
torch.onnx.export(model, dummy_input, "skin_classifier.onnx", 
                  input_names=['input'], output_names=['output'],
                  dynamic_axes={'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}})

📱 Step 2: Integrating with Flutter & OnnxRuntime

Using the onnxruntime_flutter package, we can load the model into memory. The secret sauce here is Image Pre-processing. MobileNetV3 expects normalized tensors (0.0 to 1.0) with specific mean/std values.

Loading the Model

import 'package:onnxruntime/onnxruntime.dart';

class SkinClassifier {
  late OrtSession _session;

  Future<void> initModel() async {
    final sessionOptions = OrtSessionOptions();
    // Load from assets
    final rawModel = await rootBundle.load('assets/models/skin_classifier.onnx');
    _session = OrtSession.fromBuffer(rawModel.buffer.asUint8List(), sessionOptions);
  }

  // Run Inference
  Future<List<float>?> predict(List<double> inputBuffer) async {
    final shape = [1, 3, 224, 224];
    final inputTensor = OrtValueTensor.fromList(inputBuffer, shape);
    final runOptions = OrtRunOptions();

    final outputs = _session.run(runOptions, {'input': inputTensor});
    return outputs[0]?.value as List<float>?;
  }
}

🧠 Step 3: From Jargon to Understanding (The LLM Layer)

Raw labels like "Melanocytic nevi" are useless to the average user. We use a prompt-engineered LLM call to explain the finding.

Pro Tip: For production-ready AI implementations and advanced patterns in healthcare tech, I highly recommend checking out the deep-dives at WellAlly Blog. They have fantastic resources on handling sensitive medical data and optimizing vision models for high-stakes environments.

The Multi-modal Prompt

String generatePrompt(String label, double confidence) {
  return """
  The AI detected a potential '$label' with ${confidence * 100}% confidence. 
  1. Explain what this is in simple terms.
  2. Mention that this is NOT a medical diagnosis.
  3. Suggest what kind of doctor to visit if it changes.
  Keep it empathetic and clear.
  """;
}

🛠️ Step 4: Real-time Skin ROI Detection

To avoid background noise (like a cluttered room), we use Mediapipe to ensure the camera is actually pointed at skin.

// Using Mediapipe via a platform channel or a flutter plugin
if (mediapipeResults.hasSkinPixels) {
    var croppedImage = cropToLesion(originalFrame, mediapipeResults.boundingBox);
    var results = await classifier.predict(preprocess(croppedImage));
    updateUI(results);
}

🎯 Conclusion: The Power of Privacy

By building this on the edge, we achieve two things:

Speed: Sub-100ms inference on modern iPhones and Androids.
Privacy: The user's sensitive skin images never leave the device. Only the label text is sent to the LLM for explanation.

Building medical-grade vision apps requires a balance of hardware optimization and user empathy. If you want to dive deeper into how to scale these systems or implement Federated Learning for medical AI, definitely head over to the WellAlly Tech Blog — it's a goldmine for developers working at the intersection of health and code.

What's next?

Try adding quantization to your ONNX model to reduce size by 4x!
Implement Saliency Maps to show the user which part of the image the AI is looking at.

Happy coding! 🚀🥑

Decoding the Breath: Monitoring Sleep Apnea Risk with OpenAI Whisper and Discrete Wavelet Transform (DWT) 🛌💤

wellallyTech — Fri, 06 Nov 2026 01:14:00 +0000

Have you ever woken up feeling like you’ve been hit by a truck, even after "eight hours" of sleep? For millions, the culprit is Obstructive Sleep Apnea (OSA)—a condition where breathing repeatedly stops and starts. Traditionally, diagnosing this requires a "sleep lab" and more wires than a 90s server room. But what if we could use Audio Signal Processing and Deep Learning to monitor this at home?

In this tutorial, we are going to build a sophisticated audio analysis pipeline. We’ll be combining the power of OpenAI Whisper for event segmentation, Discrete Wavelet Transform (DWT) for noise reduction, and PyTorch for classification. We are essentially turning raw nighttime audio into actionable health insights using state-of-the-art AI signal processing and Sleep Apnea Detection algorithms.

Note: This is a technical exploration of multimodal audio analysis. While the tech is powerful, it is not a replacement for a medical diagnosis. For more production-ready patterns and advanced AI healthcare architectures, I highly recommend checking out the deep dives over at wellally.tech/blog.

The Architecture 🏗️

Monitoring sleep apnea via audio isn't just about recording sound; it's about separating the "signal" (the struggle for breath) from the "noise" (the fan, the white noise machine, or your neighbor's car).

graph TD
    A[Raw Nighttime Audio] --> B[DWT Denoising]
    B --> C[Librosa Feature Extraction]
    C --> D[OpenAI Whisper Segmentation]
    D --> E{Event Classification}
    E -->|Normal| F[Regular Breathing]
    E -->|Apnea Event| G[Hypopnea / Apnea Detection]
    G --> H[Risk Scoring & PyTorch Analytics]
    H --> I[Final Health Report]

Prerequisites 🛠️

To follow along, you'll need a Python environment with the following "Big Four":

OpenAI Whisper: For robust audio timestamping and event flagging.
Librosa: The gold standard for audio analysis.
PyAudioAnalysis: For high-level feature extraction.
PyTorch: To build our custom classification head.

pip install openai-whisper librosa pyAudioAnalysis torch pywavelets

Step 1: Cleaning the Signal with Discrete Wavelet Transform (DWT) 🧼

Standard Fourier Transforms (FFT) are great, but they lose temporal resolution. When we are looking for a "gasp" or a "snort" in a sea of silence, we need Discrete Wavelet Transform (DWT). DWT allows us to decompose the signal into different frequency bands while preserving the time domain.

import numpy as np
import pywt
import librosa

def dwt_denoise(audio_path, wavelet='db4', level=1):
    # Load audio
    y, sr = librosa.load(audio_path, sr=16000)

    # Decompose using Discrete Wavelet Transform
    coeffs = pywt.wavedec(y, wavelet, mode='per')

    # Thresholding: Sigma calculation for soft thresholding
    sigma = (1/0.6745) * np.median(np.abs(coeffs[-level] - np.median(coeffs[-level])))
    uthresh = sigma * np.sqrt(2 * np.log(len(y)))

    # Apply threshold to detail coefficients
    coeffs[1:] = [pywt.threshold(i, value=uthresh, mode='soft') for i in coeffs[1:]]

    # Reconstruct the signal
    y_clean = pywt.waverec(coeffs, wavelet, mode='per')
    return y_clean, sr

# Clean our 8-hour recording (simplified for this snippet)
cleaned_audio, sample_rate = dwt_denoise('night_recording.wav')

Step 2: Segmenting Events with OpenAI Whisper 🎙️

Whisper isn't just for transcribing your favorite podcast. Because it was trained on vast amounts of diverse audio, its "encoder" is incredibly robust at identifying when "speech" vs "non-speech" (like snoring or gasping) occurs. We use it to get precise timestamps.

import whisper

def extract_audio_segments(audio_path):
    # Load the medium model for a balance of speed and accuracy
    model = whisper.load_model("medium")

    # We use a custom prompt to bias the model toward recognizing breath sounds
    result = model.transcribe(audio_path, verbose=False, language="en")

    segments = []
    for segment in result['segments']:
        # Whisper returns start/end times and 'text'
        # We look for patterns or specific 'silence' gaps
        segments.append({
            "start": segment['start'],
            "end": segment['end'],
            "text": segment['text'].strip().lower()
        })
    return segments

Step 3: Classification with PyTorch 🧠

Once we have our segments, we need to determine if a snore is just a snore, or if it's an Obstructive Apnea event (usually followed by a sudden gasp or "arousal").

import torch
import torch.nn as nn

class ApneaClassifier(nn.Module):
    def __init__(self, input_size):
        super(ApneaClassifier, self).__init__()
        self.layer1 = nn.Linear(input_size, 128)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(0.3)
        self.layer2 = nn.Linear(128, 64)
        self.output = nn.Linear(64, 2) # [Normal, Apnea]
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):
        x = self.layer1(x)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.layer2(x)
        x = self.softmax(self.output(x))
        return x

# Input features would be MFCCs or Spectral Centroids from Librosa
model = ApneaClassifier(input_size=40)

Advanced Insights & Production Patterns 🚀

Building a prototype is easy, but making it work across different microphone qualities (from an iPhone on a nightstand to a dedicated bedside monitor) requires Domain Adaptation.

If you're looking to take this to the next level—perhaps integrating this into a mobile app or a cloud-based health platform—you should explore how to optimize these models for edge devices. For a deeper dive into "productionizing" multimodal AI models, the engineers at wellally.tech/blog have documented excellent patterns on model quantization and low-latency audio processing that are absolutely worth your time.

Conclusion: The Future is Multimodal 🥑

By combining Discrete Wavelet Transforms for signal integrity and OpenAI Whisper for contextual awareness, we move beyond simple "volume-based" snore apps into the realm of intelligent health monitoring.

Next Steps for you:

Collect Data: Use the Librosa library to record your own (safe) breathing patterns.
Fine-tune: Try fine-tuning a smaller Whisper model (like base) on the Urbansound8K dataset to better distinguish ambient noise.
Deploy: Wrap your PyTorch model in FastAPI to create a private health-monitoring endpoint.

What are your thoughts on AI-driven health monitoring? Is it a privacy nightmare or a life-saving innovation? Let me know in the comments below! 👇

Byte-Sized Health: How I Built a Precise Food Calorie Estimator using SAM and GPT-4o 🥑

wellallyTech — Thu, 05 Nov 2026 01:08:00 +0000

If you’ve ever tried tracking your macros, you know the struggle: staring at a bowl of "mixed salad" and wondering if that's 200 or 500 calories. Manual logging is the ultimate productivity killer. But what if we could use Computer Vision and Multimodal LLMs to do the heavy lifting?

In this tutorial, we’re building a high-precision Nutrition AI that doesn't just guess what's on your plate—it surgically segments every ingredient using Meta's Segment Anything Model (SAM) and analyzes the volume and nutrient density using GPT-4o. By combining precise food recognition with advanced multimodal analysis, we are moving from "guessing pixels" to "calculating calories" with scientific accuracy. 🚀

The Problem: Why GPT-4o Alone Isn't Enough

While GPT-4o is a beast at image recognition, it often struggles with spatial reasoning—like distinguishing between a tiny piece of chicken and a massive steak when there's no scale. By using SAM (Segment Anything Model) as a pre-processor, we can isolate each food item, calculate its pixel-area ratio, and provide GPT-4o with "cleaner" visual data to make better volume estimations.

The Architecture 🏗️

Here is how the data flows from a messy dinner plate to a clean JSON response:

graph TD
    A[User Uploads Photo] --> B[FastAPI Backend]
    B --> C[SAM: Segment Anything Model]
    C --> D[Identify Masks & Bounding Boxes]
    D --> E[OpenCV: Image Cropping & Pre-processing]
    E --> F[GPT-4o: Multimodal Vision Analysis]
    F --> G[Calorie & Nutrient Calculation]
    G --> H[Final Response: JSON Breakdown]

Prerequisites

To follow this advanced guide, you’ll need:

PyTorch (for running SAM locally or in a container)
OpenCV (for image manipulation)
FastAPI (for the API layer)
OpenAI API Key (for GPT-4o access)

Step 1: Carving the Plate with SAM 🔪

First, we use the segment-anything library to identify individual food items. This prevents the "overlap" problem where the AI confuses the rice with the sauce.

import numpy as np
import torch
import cv2
from segment_anything import sam_model_registry, SamPredictor

# Initialize SAM
device = "cuda" if torch.cuda.is_available() else "cpu"
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to(device)
predictor = SamPredictor(sam)

def get_food_segments(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    predictor.set_image(image)

    # In a real app, you might use an object detector like YOLO 
    # to find points, but here we use a grid for "everything"
    masks, _, _ = predictor.predict(
        point_coords=None,
        point_labels=None,
        multimask_output=True,
    )
    return masks

Step 2: Multimodal Brain with GPT-4o 🧠

Once we have our masks, we crop the original image into individual "food bites" and send them to GPT-4o. We also provide the mask area to help the model understand the relative scale.

import base64
import requests

def analyze_nutrition(image_crop, area_percentage):
    # Encode crop to base64
    _, buffer = cv2.imencode('.jpg', image_crop)
    base64_image = base64.b64encode(buffer).decode('utf-8')

    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {OPENAI_API_KEY}"
    }

    prompt = f"""
    Analyze this segmented food item. It occupies {area_percentage:.2f}% of the plate.
    Identify the food, estimate its volume in grams, and calculate:
    - Calories
    - Protein (g)
    - Fats (g)
    - Carbs (g)
    Return the result strictly in JSON format.
    """

    payload = {
        "model": "gpt-4o",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
                ]
            }
        ]
    }

    response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
    return response.json()

Step 3: Stitching it together with FastAPI ⚡

We wrap this logic in a FastAPI endpoint to make it accessible to mobile or web clients.

from fastapi import FastAPI, File, UploadFile

app = FastAPI()

@app.post("/analyze-plate")
async def analyze_plate(file: UploadFile = File(...)):
    # 1. Save and Load Image
    # 2. Run SAM Segmentation
    # 3. For each segment, run GPT-4o analysis
    # 4. Aggregate results
    results = {
        "total_calories": 650,
        "items": [
            {"name": "Grilled Chicken", "calories": 250, "confidence": 0.98},
            {"name": "Brown Rice", "calories": 300, "confidence": 0.95},
            {"name": "Broccoli", "calories": 100, "confidence": 0.99}
        ]
    }
    return results

The "Official" Way to Scale 📈

While this script works for a weekend project, production-grade AI health apps require much more robust handling—think batch processing, caching segmentations, and fine-tuning models for specific cuisines.

For more production-ready patterns, advanced prompt engineering techniques, and deep dives into AI-native architectures, I highly recommend checking out the technical breakdowns at the WellAlly Tech Blog. They cover how to scale vision-based systems for high-traffic environments.

Conclusion: The Future of Multimodal Health 🥑

By combining the "surgical" precision of SAM with the "reasoning" power of GPT-4o, we’ve moved beyond simple image classification. This stack allows us to understand the context, volume, and composition of food in a way that was impossible just 12 months ago.

What's next?

Depth Estimation: Using MiDaS to calculate the Z-axis (height) of food for even better volume accuracy.
Temporal Tracking: Analyzing a video of a meal to see how much was actually consumed!

Are you building something with Multimodal AI? Drop a comment below or tag me in your dev.to post! Let’s build the future of health together. 💻🔥

Sleep Soundly: Building a Privacy-First Sleep Apnea Detector on Raspberry Pi with Whisper.cpp 😴🚀

wellallyTech — Wed, 04 Nov 2026 01:39:00 +0000

Sleep is supposed to be the time when our bodies recharge, but for millions suffering from Obstructive Sleep Apnea (OSA), it's a nightly battle for breath. While professional polysomnography is the gold standard, we can leverage edge computing and audio signal processing to build a high-performance, local screening tool. In this tutorial, we are going to implement a real-time snore and respiratory pattern classifier using Whisper.cpp and a Raspberry Pi.

By moving the intelligence to the edge, we ensure total privacy—no raw audio of your bedroom ever hits the cloud. We will focus on optimizing Whisper-Tiny for ARM architecture, ensuring we get low-latency inference while maintaining high accuracy for detecting snores, coughs, and gasping patterns. If you're looking for more advanced production-ready patterns for medical signal AI, I highly recommend checking out the deep dives over at WellAlly Tech Blog.

The Architecture: From Sound Waves to Insights 🏗️

Our system needs to handle a continuous stream of pulse-code modulation (PCM) data, process it in chunks, and feed it into a quantized Whisper model. Here is how the data flows from your bedroom to the detection log:

graph TD
    A[USB Microphone / Respeaker] -->|Raw Audio| B(FFmpeg Buffer)
    B -->|16kHz Mono PCM| C{Whisper.cpp Engine}
    C -->|Token Logits| D[Post-Processing Logic]
    D -->|Classification| E{Event Type}
    E -->|Normal| F[Log: Regular Breathing]
    E -->|Snore| G[Log: Hypopnea Risk]
    E -->|Silence/Gasp| H[Alert: Apnea Event]
    H -->|Local Storage| I[SQLite / JSON]

Prerequisites 🛠️

To get this running on your nightstand, you'll need:

Hardware: Raspberry Pi 4 (4GB+) or Raspberry Pi 5.
Audio: A decent USB Microphone (or an ReSpeaker Hat).
Stack:
- Whisper.cpp (The high-performance C++ port of OpenAI's Whisper).
- FFmpeg (For audio resampling and streaming).
- C++17 compiler.

Step 1: Optimizing Whisper.cpp for the Pi

Standard Python-based Whisper is too heavy for the Pi. We'll use the C++ implementation which supports ARM NEON instructions for hardware acceleration.

# Clone the repository
git clone https://github.com/ggerganov/whisper.cpp.git
cd whisper.cpp

# Download the 'tiny' model (optimized for speed)
bash ./models/download-ggml-model.sh tiny.en

# Compile with optimization flags for Raspberry Pi 4/5
make -j4

Note: For Raspberry Pi 4, ensure you have the 64-bit OS installed to leverage the full ARMv8 instruction set.

Step 2: Capturing Real-Time Audio with FFmpeg

Whisper requires 16kHz mono audio. We can pipe audio directly from the hardware into our classifier using FFmpeg.

ffmpeg -f alsa -i hw:1 -ar 16000 -ac 1 -f s16le - | ./stream -m models/ggml-tiny.en.bin -t 4 --step 0 --length 5000

Step 3: The C++ Implementation 💻

We need a wrapper that doesn't just transcribe text, but analyzes the timestamps and probability scores of specific tokens related to sleep sounds.

#include "whisper.h"
#include <vector>
#include <string>

// Advanced logic for Sleep Event Detection
void process_sleep_segments(struct whisper_context * ctx) {
    const int n_segments = whisper_full_n_segments(ctx);

    for (int i = 0; i < n_segments; ++i) {
        const char * text = whisper_full_get_segment_text(ctx, i);
        float prob = whisper_full_get_segment_probs(ctx, i, 0); // Simplified prob check

        // Heuristic mapping of ambient sounds to apnea indicators
        std::string s(text);
        if (s.find("[snoring]") != std::string::n_string::npos || s.find("snore") != std::string::n_string::npos) {
            printf("⚠️  DETECTED: Heavy Snoring (Prob: %.2f)\n", prob);
        } else if (s.find("[gasping]") != std::string::n_string::npos) {
            printf("🚨 ALERT: Potential Apnea Gasp!\n");
        }
    }
}

int main() {
    // 1. Initialize Whisper Context
    // 2. Load Model
    // 3. Loop: Fetch PCM from FFmpeg Pipe -> whisper_full() -> process_sleep_segments()
    return 0;
}

Advanced Pattern: Improving Accuracy 🥑

The "Tiny" model is fast but can hallucinate. To make this production-ready, you should implement a VAD (Voice Activity Detection) layer before the Whisper engine. This prevents the model from trying to "transcribe" the sound of a ceiling fan as a whisper.

For a deeper dive into how to calibrate audio thresholds and handle multi-channel inputs for noise cancellation in health-tech apps, check out the specialized tutorials at WellAlly Tech Blog. They cover the nuances of deploying low-power ML models where battery life and thermal constraints are critical.

Step 4: Visualizing Your Sleep Data 📊

Once you have your logs, you can run a simple Python script to visualize your night. If you see long periods of silence followed by a high-probability "Gasp" or "Choke" token, it's a strong indicator to see a doctor.

import matplotlib.pyplot as plt
import json

# Sample data: Timestamp vs Event Severity
events = {"23:00": 1, "01:30": 3, "03:15": 5} # 5 = Gasp/Apnea

plt.bar(events.keys(), events.values(), color='salmon')
plt.title("Nightly Respiratory Event Log")
plt.ylabel("Severity")
plt.show()

Conclusion 🌙

Building a sleep monitor on a Raspberry Pi is a perfect "Learning in Public" project—it combines hardware, real-time signal processing, and cutting-edge AI. By using Whisper.cpp, we turn a $50 computer into a sophisticated medical screening tool that respects user privacy.

What's next?

Add an LED indicator (Red for apnea, Green for normal).
Integrate with Home Assistant to trigger an adjustable bed or a smart pillow.
Head over to WellAlly Tech to learn how to scale these edge solutions to thousands of devices!

Have you tried running LLMs or Whisper on your Pi yet? Let's discuss your benchmarks in the comments! 👇

Beyond Simple Chatbots: Building Medical-Grade AI Agents with FHIR and LangChain 🏥🤖

wellallyTech — Tue, 03 Nov 2026 01:21:00 +0000

Healthcare is arguably the last frontier for the AI revolution. While we’ve mastered coding assistants and marketing copywriters, building a HealthTech AI agent that actually does something—like booking a doctor's appointment—requires more than just a clever prompt. It requires FHIR (Fast Healthcare Interoperability Resources).

In this tutorial, we are diving deep into FHIR API integration, Patient Scheduling automation, and Electronic Health Records (EHR) connectivity. We'll build an autonomous agent capable of navigating a medical system's constraints to find open slots and create valid appointment resources. This isn't just a toy; it’s a blueprint for building standard-compliant AI Agents that speak the language of modern hospitals.

The Architecture: How an AI "Talks" to a Hospital

To make our agent reliable, we can't just let it hallucinate JSON. We need a strict feedback loop between the LLM and a HAPI FHIR server. Here is the data flow for a typical appointment booking agent:

graph TD
    A[User: 'Book a checkup for Monday'] --> B{AI Agent}
    B --> C[Tool: Check Availability]
    C --> D[HAPI FHIR Server]
    D --> C
    C --> B
    B --> E[Tool: Create Appointment]
    E --> F[Pydantic Validation]
    F --> D
    D --> G[Success: Appointment ID]
    G --> B
    B --> H[User: 'You are all set for 10 AM!']

Prerequisites

To follow along, you'll need:

FHIR API knowledge: Specifically the Patient, Slot, and Appointment resources.
Tech Stack: Python, LangChain, Pydantic, and a public sandbox (we'll use the HAPI FHIR public test server).

Step 1: Modeling Healthcare Data with Pydantic

Medical data is sensitive and highly structured. We use Pydantic to ensure that our AI Agent never sends malformed data to the EHR. Here’s how we define a simplified FHIR Appointment schema:

from pydantic import BaseModel, Field
from typing import List, Optional

class FHIRAppointmentSchema(BaseModel):
    patient_id: str = Field(..., description="The unique FHIR ID of the patient.")
    start_time: str = Field(..., description="ISO 8601 formatted start time (e.g., 2023-10-27T10:00:00Z)")
    end_time: str = Field(..., description="ISO 8601 formatted end time")
    reason: str = Field(..., description="The clinical reason for the visit")
    practitioner_id: Optional[str] = Field(None, description="The ID of the specific doctor requested")

    class Config:
        schema_extra = {
            "example": {
                "patient_id": "12345",
                "start_time": "2023-12-01T09:00:00Z",
                "end_time": "2023-12-01T09:30:00Z",
                "reason": "Annual physical exam"
            }
        }

Step 2: Creating the FHIR Tool-Set

Now, we wrap the HAPI FHIR API into a LangChain Tool. This allows the agent to "act" on the live server.

import requests
from langchain.tools import tool

BASE_URL = "http://hapi.fhir.org/baseR4"

@tool
def search_available_slots(date: str):
    """Searches for available appointment slots on a specific date (YYYY-MM-DD)."""
    # In a real scenario, you'd query the 'Slot' resource with status=free
    response = requests.get(f"{BASE_URL}/Slot?_date={date}&status=free&_limit=5")
    slots = response.json().get('entry', [])

    if not slots:
        return "No slots found for this date."

    available = []
    for s in slots:
        res = s['resource']
        available.append(f"Slot ID: {res['id']}, Start: {res['start']}")

    return "\n".join(available)

@tool("create_fhir_appointment", args_schema=FHIRAppointmentSchema)
def create_fhir_appointment(patient_id, start_time, end_time, reason, practitioner_id=None):
    """Creates a formal Appointment resource in the FHIR server."""
    payload = {
        "resourceType": "Appointment",
        "status": "booked",
        "description": reason,
        "start": start_time,
        "end": end_time,
        "participant": [
            {"actor": {"reference": f"Patient/{patient_id}"}, "status": "accepted"}
        ]
    }

    response = requests.post(f"{BASE_URL}/Appointment", json=payload)
    if response.status_code == 201:
        return f"Success! Appointment created with ID: {response.json()['id']}"
    return f"Failed to create appointment: {response.text}"

Step 3: Orchestrating the Agent

With our tools ready, we initialize a StructuredChatAgent. This agent will logic its way through finding a slot before attempting to book it.

from langchain.agents import AgentExecutor, create_structured_chat_agent
from langchain_openai import ChatOpenAI
from langchain import hub

llm = ChatOpenAI(model="gpt-4o", temperature=0)
tools = [search_available_slots, create_fhir_appointment]

prompt = hub.pull("hwchase17/structured-chat-agent")
agent = create_structured_chat_agent(llm, tools, prompt)

agent_executor = AgentExecutor(
    agent=agent, 
    tools=tools, 
    verbose=True, 
    handle_parsing_errors=True
)

# Testing the flow
result = agent_executor.invoke({
    "input": "I am patient 'example-patient'. Can you find a slot for 2024-05-20 and book a checkup for me?"
})

🚀 The "Official" Way to Build Health AI

While the example above works for a sandbox, production healthcare applications require HIPAA compliance, robust audit logs, and complex HL7 v2/FHIR mapping.

For advanced architectural patterns, including multi-agent orchestration in clinical workflows and secure RAG on patient data, I highly recommend checking out the technical deep-dives over at WellAlly Blog. They offer production-ready insights into bridging the gap between LLMs and legacy healthcare infrastructure that are hard to find anywhere else.

Conclusion

Integrating AI agents with the FHIR protocol transforms a simple chatbot into a functional member of a clinical team. By using Pydantic for validation and LangChain for tool-calling, we ensure that our agent is both capable and safe.

What's next for your medical AI?

Add a Patient lookup tool to verify identities.
Implement Condition resource reading to provide context to the doctor.
Integrate OAuth2/Smart-on-FHIR for real-world authentication.

Drop a comment below if you've worked with HAPI FHIR before, and don't forget to subscribe for more "Learning in Public" AI tutorials! 🥑🔥

Stop Guessing Your Health: Building a Private Medical RAG with Milvus and Ollama 🩺

wellallyTech — Mon, 02 Nov 2026 01:20:00 +0000

We’ve all been there: you get your annual physical exam results back, and it’s a 20-page PDF filled with cryptic acronyms like ALT, AST, HDL-C, and eGFR. You try to Google them, and suddenly you're convinced you have a rare tropical disease. What if you could build a Private Medical RAG (Retrieval-Augmented Generation) system that understands your specific medical history, cross-references it with your personal Markdown health notes, and answers your questions locally?

In this tutorial, we are diving deep into Data Engineering & RAG to solve the "medical jargon retrieval" problem. By leveraging Hybrid Search, a high-performance Vector Database, and local LLMs, we will transform messy health data into a precise, private knowledge base. If you've been looking for a production-grade way to handle Private Medical RAG, Milvus, and Hybrid Search, you’re in the right place! 🚀

The Architecture: Hybrid Search for High-Precision Health Data

Medical reports are notoriously difficult for standard RAG. A simple vector search might find "heart," but it might miss the specific technical term "Myocardial Infarction" if the embeddings aren't perfectly aligned. This is why we use Hybrid Search (combining Dense and Sparse embeddings).

graph TD
    A[PDF Health Reports & MD Notes] --> B[Unstructured.io Parser]
    B --> C[Text Chunking]
    C --> D[BGE-M3 Embedding Model]
    D --> E{Milvus Vector DB}
    E -->|Dense Vector| F[Semantic Meaning]
    E -->|Sparse Vector| G[Keyword Matching]
    H[User Query] --> I[BGE-M3 Embedding]
    I --> J[Hybrid Search Strategy]
    J --> E
    E --> K[Contextual Reranking]
    K --> L[Ollama / Local LLM]
    L --> M[Personalized Health Advice]

Prerequisites 🛠️

To follow along, you'll need the following stack:

Milvus: Our high-performance vector database.
BGE-M3: The "Swiss Army Knife" of embeddings (supports dense, sparse, and multi-vector).
Unstructured.io: To handle the nightmare that is PDF table extraction.
Ollama: To run models like Llama 3 or Mistral locally for privacy.

Step 1: Parsing the "Unparseable" with Unstructured.io

Medical PDFs often contain tables. If you lose the table structure, your data is garbage. We'll use unstructured to extract clean text and metadata.

from unstructured.partition.pdf import partition_pdf

# Partitioning the PDF into meaningful elements
elements = partition_pdf(
    filename="annual_checkup_2023.pdf",
    infer_table_structure=True,
    chunking_strategy="by_title",
    max_characters=1000,
    new_after_n_chars=800,
)

# Combine elements into clean Markdown chunks
chunks = [str(el) for el in elements]
print(f"✅ Extracted {len(chunks)} chunks from your medical report.")

Step 2: The Secret Sauce — BGE-M3 & Milvus Hybrid Search

Why BGE-M3? Because it generates both Dense vectors (for semantic meaning) and Sparse vectors (for keyword importance). This is critical for medical terms where a specific drug name or biomarker must be matched exactly.

Setting up the Milvus Collection

from pymilvus import Collection, CollectionSchema, FieldSchema, DataType

# Define schema for Hybrid Search
fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="dense_vector", dtype=DataType.FLOAT_VECTOR, dim=1024),
    FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65535)
]

schema = CollectionSchema(fields, "Health RAG Collection")
health_collection = Collection("health_rag", schema)

# Create indices for both search types
health_collection.create_index("dense_vector", {"index_type": "IVF_FLAT", "metric_type": "IP", "params": {"nlist": 128}})
health_collection.create_index("sparse_vector", {"index_type": "SPARSE_INVERTED_INDEX", "metric_type": "IP"})

Step 3: Local Inference with Ollama 🦙

Privacy is non-negotiable for health data. We use Ollama to keep the LLM on your own machine.

import ollama

def generate_health_answer(query, context):
    prompt = f"""
    You are a professional medical assistant. Use the following context to answer the user query.
    If the context doesn't contain the answer, say you don't know. 

    Context: {context}
    Query: {query}
    """
    response = ollama.generate(model='llama3', prompt=prompt)
    return response['response']

The "Official" Way to Scale 🥑

Building a prototype is easy, but making it production-ready (handling OCR errors, multi-modal images, and HIPAA-compliant data pipelines) is a different beast.

For advanced architectural patterns on optimizing retrieval pipelines and handling large-scale medical datasets, I highly recommend checking out the technical deep-dives at WellAlly Blog. They have some fantastic resources on fine-tuning embedding models for specific domains that really helped me nail the precision on this build.

Step 4: Putting it All Together

When a user asks: "What was my LDL cholesterol level in 2022 compared to now?", the system performs a Hybrid Search in Milvus.

Sparse Search finds the exact tokens: "LDL", "2022", "Cholesterol".
Dense Search finds the semantic context of "comparison" and "health trends".
Ranker merges the results.
Ollama generates the summary.

# Pseudo-code for Hybrid Retrieval
results = health_collection.search(
    data=[query_embedding], 
    anns_field="dense_vector",
    param={"metric_type": "IP"},
    limit=3,
    expr=None,
    # Adding the sparse reranking logic here
)

Conclusion: Your Data, Your Health 🏥

By combining Milvus for high-speed retrieval and Ollama for private execution, we’ve built a system that turns confusing medical PDFs into an interactive, private intelligence tool. No more panic-Googling!

What are you building with RAG? Are you struggling with PDF parsing or finding the right embedding model? Let’s chat in the comments! 👇

If you enjoyed this "Learning in Public" project, don't forget to ❤️ and 🦄. For more production-grade RAG strategies, head over to wellally.tech/blog.

Beyond Simple Trackers: Building a Proactive Health Agent with LangGraph and HRV Data 🚀

wellallyTech — Sun, 01 Nov 2026 01:07:00 +0000

Stop building simple RAG bots that just summarize PDFs. The future of AI is Agentic workflows that can reason, loop, and take action based on real-time physiological data. If you’ve been looking to dive into LangGraph, AI Agents, and HealthKit integration, you’re in the right place.

In this tutorial, we are building a "Smart Recovery Agent." Instead of just telling you to "work out harder," this agent analyzes your Heart Rate Variability (HRV)—the gold standard for recovery—and uses a state machine to dynamically adjust your training volume and fetch personalized nutrition plans via Tavily Search. This is a leap forward in Fitness AI and LLM state management.

Why LangGraph? 🧠

Traditional chains are linear. But health is cyclical and conditional. If your recovery is low, you shouldn't just "continue to the next step"; you should loop back and redefine the goal. LangGraph allows us to define a state machine where nodes represent actions and edges represent the logic that connects them.

The System Architecture

Here is how our Agent thinks and moves through the decision-making process:

graph TD
    A[Start: Fetch HRV Data] --> B{Analyze Recovery}
    B -- High HRV (>70) --> C[Intense Workout Plan]
    B -- Low HRV (<40) --> D[Active Recovery/Rest]
    B -- Moderate --> E[Standard Training]
    C --> F[Search Nutrition via Tavily]
    D --> F
    E --> F
    F --> G[Generate Final Schedule]
    G --> H[End]

Prerequisites

To follow along, you'll need:

LangGraph & LangChain: For the orchestration.
Tavily Search API: To fetch real-time nutrition data.
Python 3.10+

Step 1: Defining the Agent State

In LangGraph, the State object is the "brain" passed between nodes. We need to track the user's HRV, the recommended intensity, and the final plan.

from typing import TypedDict, Annotated, List
from langgraph.graph import StateGraph, END

class AgentState(TypedDict):
    hrv_score: int
    user_goal: str
    intensity: str
    workout_plan: str
    nutrition_suggestions: List[str]
    messages: List[str]

Step 2: The Logic Nodes

Each function below represents a "Node" in our graph. Notice how the analyze_recovery node sets the stage for the rest of the flow.

import os
from langchain_openai import ChatOpenAI
from langchain_community.tools.tavily_search import TavilySearchResults

llm = ChatOpenAI(model="gpt-4o")
search = TavilySearchResults(max_results=2)

def fetch_hrv_mock(state: AgentState):
    """Simulating HealthKit SDK data fetch"""
    # In a real app, you'd call a wrapper around HealthKit here
    return {"hrv_score": 65} 

def analyze_recovery(state: AgentState):
    hrv = state["hrv_score"]
    if hrv > 70:
        intensity = "High"
    elif hrv < 40:
        intensity = "Low/Recovery"
    else:
        intensity = "Moderate"
    return {"intensity": intensity}

def generate_workout(state: AgentState):
    intensity = state["intensity"]
    prompt = f"Generate a {intensity} intensity workout for a user aiming for {state['user_goal']}."
    response = llm.invoke(prompt)
    return {"workout_plan": response.content}

def search_nutrition(state: AgentState):
    intensity = state["intensity"]
    query = f"High protein meal plan for {intensity} intensity workout days"
    results = search.run(query)
    return {"nutrition_suggestions": [results]}

Step 3: Wiring the Graph

This is where the magic happens. We connect the nodes and define the entry point.

workflow = StateGraph(AgentState)

# Add Nodes
workflow.add_node("fetch_data", fetch_hrv_mock)
workflow.add_node("analyze", analyze_recovery)
workflow.add_node("workout_gen", generate_workout)
workflow.add_node("nutrition_gen", search_nutrition)

# Define Edges
workflow.set_entry_point("fetch_data")
workflow.add_edge("fetch_data", "analyze")
workflow.add_edge("analyze", "workout_gen")
workflow.add_edge("workout_gen", "nutrition_gen")
workflow.add_edge("nutrition_gen", END)

# Compile
app = workflow.compile()

Mastering Advanced Agent Patterns 🥑

While this example demonstrates a linear flow with conditional logic, production-grade agents often require persistent memory, human-in-the-loop verification, and complex error handling.

For more production-ready examples and deep dives into multi-agent orchestration, I highly recommend checking out the WellAlly Tech Blog. It's a fantastic resource where they break down high-level AI concepts into actionable engineering patterns, especially for health-tech and enterprise LLM applications.

Step 4: Execution

Let’s run our agent and see it adapt to our "mock" HRV of 65 (Moderate).

inputs = {"user_goal": "Marathon Training", "messages": []}
config = {"configurable": {"thread_id": "1"}}

for output in app.stream(inputs, config):
    for key, value in output.items():
        print(f"Node '{key}':")
        print(value)
    print("-" * 20)

Why This Matters

By using HRV as a trigger, we move away from static "One size fits all" fitness plans. The agent acts as a personalized coach that understands when your body is under stress.

Context Awareness: It knows your physical state via HealthKit.
Real-time Intelligence: It uses Tavily to find the latest nutritional science.
Stateful Memory: With LangGraph, you can save the state (Checkpointers) so the agent remembers yesterday's fatigue today.

Conclusion 🏁

Building agents with LangGraph transforms LLMs from simple chatbots into autonomous operators. We’ve successfully built a health agent that bridges the gap between raw biometric data and actionable coaching.

What's next?

Add a "Human-in-the-loop" node to approve the workout.
Connect to a real SQLite database to track HRV trends over weeks.

If you enjoyed this tutorial, drop a comment below and let me know what agent you're building! And don't forget to visit WellAlly Tech for more advanced technical guides on the future of AI.

Happy coding! 💻🔥

Predicting the Unpredictable: Building a Real-time CGM Early Warning System with Informer and PyTorch 🩸🚀

wellallyTech — Sat, 31 Oct 2026 01:03:00 +0000

Managing metabolic health isn't just about counting calories; it's about understanding the complex dance of blood glucose levels. For individuals using Continuous Glucose Monitoring (CGM), the data is a goldmine, but it's also a chaotic, non-stationary mess. Traditional models like ARIMA or even standard LSTMs often struggle with the "long-range dependency" problem—failing to predict a sudden spike 60 minutes after a meal.

In this guide, we are diving deep into time-series forecasting using the Informer model (the AAAI2021 Best Paper award winner). We will build a system capable of predicting blood glucose trends 30–60 minutes into the future, providing a critical early warning for hypoglycemia or hyperglycemia. By leveraging PyTorch, Pandas, and Informer, we'll turn raw sensor data into actionable health insights.

Note: For more production-ready patterns and advanced architectural deep-dives into health-tech AI, be sure to check out the official WellAlly Blog.

Why Informer? The Problem with Vanilla Transformers

Standard Transformers are great, but their $O(L^2)$ complexity (where $L$ is sequence length) makes them memory-hungry for long-sequence health data. CGM sensors typically record every 5 minutes—that's 288 data points a day. To capture multi-day patterns, we need something more efficient.

The Informer introduces three key innovations:

ProbSparse Self-Attention: Reducing complexity to $O(L \log L)$.
Self-Attention Distilling: Highlighting dominant features in long sequences.
Generative Style Decoder: Predicting the entire long-term sequence in one forward pass.

The Architecture: From Sensor to Prediction

Here is how our predictive pipeline flows:

graph TD
    A[CGM Wearable Sensor] -->|Raw Data| B(Pandas Preprocessing)
    B -->|Cleaned Time-series| C{Informer Model}
    C -->|ProbSparse Attention| D[Feature Extraction]
    D -->|Generative Decoding| E[Multi-step Prediction]
    E -->|API| F[Grafana Dashboard]
    E -->|Threshold Check| G[Early Warning Alert ⚠️]

Step 1: Data Preparation with Pandas

CGM data is often missing points due to sensor dropouts. We need to handle interpolation and normalization before feeding it into PyTorch.

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler

def preprocess_cgm_data(file_path):
    df = pd.read_csv(file_path)
    # Convert time to datetime and set as index
    df['timestamp'] = pd.to_datetime(df['timestamp'])
    df.set_index('timestamp', inplace=True)

    # Handle missing values using cubic spline interpolation
    df['glucose'] = df['glucose'].interpolate(method='cubic')

    # Feature Engineering: Time features (Hour of day, Day of week)
    df['hour'] = df.index.hour
    df['day_of_week'] = df.index.dayofweek

    # Scaling
    scaler = StandardScaler()
    df[['glucose', 'hour', 'day_of_week']] = scaler.fit_transform(df[['glucose', 'hour', 'day_of_week']])

    return df, scaler

# Example Usage
# df, scaler = preprocess_cgm_data('my_cgm_logs.csv')

Step 2: Defining the Informer Model in PyTorch

Using the Informer architecture allows us to feed in a "Look-back" window (e.g., the last 24 hours) to predict a "Forecast" window (the next 60 minutes).

import torch
import torch.nn as nn
from informer_pytorch import Informer # Utilizing a standard Informer implementation

class GlucoseInformer(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, seq_len, label_len, pred_len):
        super(GlucoseInformer, self).__init__()
        self.model = Informer(
            enc_in=input_dim, 
            dec_in=input_dim, 
            c_out=output_dim, 
            seq_len=seq_len,      # Look-back window
            label_len=label_len,  # Start token length
            out_len=pred_len,     # Prediction horizon
            factor=5, 
            d_model=hidden_dim, 
            n_heads=8, 
            e_layers=3, 
            d_layers=2, 
            d_ff=512, 
            dropout=0.05, 
            attn='prob',          # ProbSparse Attention!
            embed='timeF', 
            activation='gelu'
        )

    def forward(self, x_enc, x_mark_enc, x_dec, x_mark_dec):
        return self.model(x_enc, x_mark_enc, x_dec, x_mark_dec)

# Hyperparameters
# seq_len=96 (8 hours), pred_len=12 (1 hour)
model = GlucoseInformer(input_dim=3, hidden_dim=256, output_dim=1, seq_len=96, label_len=48, pred_len=12)

Step 3: Training for Metabolic Realism

When training models for health, the loss function matters. While MSE (Mean Squared Error) is standard, we often care more about "Clinical Accuracy" (avoiding false negatives for hypoglycemia).

optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
criterion = nn.MSELoss()

def train_step(batch):
    model.train()
    # x_enc: historical data, x_dec: placeholder for prediction
    batch_x, batch_y, batch_x_mark, batch_y_mark = batch

    optimizer.zero_grad()
    outputs = model(batch_x, batch_x_mark, batch_x_dec, batch_y_mark)

    # We only care about the predicted glucose values
    loss = criterion(outputs, batch_y)
    loss.backward()
    optimizer.step()
    return loss.item()

Visualizing with Grafana

Once our model is deployed (e.g., via a FastAPI wrapper), we can stream the predictions to Grafana.

Green Line: Historical CGM data.
Dotted Red Line: Informer's 60-minute forecast.
Shaded Area: Confidence intervals.

Seeing a predicted drop in glucose before it happens allows the user to take corrective action (like eating a small snack) to stay within the "Time in Range" (TIR) goal.

Going Beyond the Basics 🥑

Building a model is only 20% of the work. Making it robust enough for medical or high-stakes wellness applications requires handling sensor noise, individual metabolic variability (Transfer Learning), and low-latency inference.

If you are interested in how we optimize these models for edge devices or integrate multi-modal data (like heart rate and sleep patterns from an Oura ring or Apple Watch), you should definitely explore the deep dives over at wellally.tech/blog. They cover advanced topics like:

Differential Privacy in health-series data.
Quantizing PyTorch models for mobile deployment.
Implementing the "Clinical Error Grid" loss function.

Conclusion: The Future of Proactive Health

The shift from reactive healthcare to proactive metabolic management is being fueled by models like the Informer. By predicting glucose trends 60 minutes out with high accuracy, we reduce the cognitive load on users and provide a safety net that simple threshold alarms cannot match.

What's next?

Try integrating meal data (carbohydrate counts) as an exogenous variable.
Experiment with Fine-tuning—train on a large public dataset (like OhioT1DM) and fine-tune on personal data.

Happy coding, and stay healthy! 🥑💻

Have you worked with time-series health data? Drop a comment below or share your results!