wellallyTech

Posted on Aug 16

Stop Sending Your Vitals to the Cloud: Running Llama-3 Locally in the Browser with WebLLM & WebGPU 🥑

#ai #beginners #tutorial #python

Privacy is the ultimate "final boss" in HealthTech. When users record sensitive medical logs, the last thing they want is their data being used to train a massive corporate model. Today, we are pushing the boundaries of Edge AI by building a 100% private, client-side health log analyzer. By leveraging WebGPU acceleration and WebLLM, we can run a full Llama-3 instance directly in the browser.

In this tutorial, we will explore how to combine Transformers.js for lightweight feature extraction and WebLLM for complex reasoning. This approach ensures that your privacy-first health apps remain performant without a single byte of personal health information (PHI) ever leaving the user's device. Let’s dive into the world of local LLM inference and browser-based machine learning! 🚀

🏗 The Architecture: 100% Data Sovereignty

Traditional AI apps follow a Client-Server model. We are flipping the script. Our architecture keeps the data, the model, and the compute inside the browser's sandbox.

graph TD
    A[User Inputs Health Log] --> B{Local Processing}
    B --> C[Transformers.js: Entity Extraction]
    B --> D[WebLLM: Llama-3-8B Reasoning]
    C --> E[Structured Health Data]
    D --> F[Clinical Insights & Summary]
    E --> G[IndexedDB: Local Storage]
    F --> G
    G --> H[Privacy-Safe UI View]
    style B fill:#f9f,stroke:#333,stroke-width:4px

🛠 Prerequisites

Before we start coding, ensure your environment meets these requirements:

Tech Stack: React (Vite), WebLLM, Transformers.js.
Hardware: A device with a GPU that supports WebGPU (Chrome 113+, Edge, or Safari Technology Preview).
Difficulty: Advanced (Buckle up! 🏎️).

1. Setting Up the WebLLM Engine

WebLLM is a high-performance in-browser LLM inference engine. It uses the WebGPU API to execute model weights compiled with TVM.

First, install the dependency:

npm install @mlc-ai/web-llm

Now, let's create a hook to manage our Llama-3 instance. We’ll use the Llama-3-8B-Instruct-q4f16_1-MLC variant, which is optimized for 4-bit quantization to fit in browser memory.

import { useState, useEffect } from 'react';
import * as webllm from "@mlc-ai/web-llm";

export function useWebLLM() {
  const [engine, setEngine] = useState<webllm.MLCEngine | null>(null);
  const [loadingProgress, setLoadingProgress] = useState(0);

  const initEngine = async () => {
    const engine = new webllm.MLCEngine();

    // Callback to track model downloading/loading progress
    engine.setInitProgressCallback((report) => {
      setLoadingProgress(Math.round(report.progress * 100));
      console.log(report.text);
    });

    const selectedModel = "Llama-3-8B-Instruct-q4f16_1-MLC";
    await engine.reload(selectedModel);
    setEngine(engine);
  };

  return { engine, initEngine, loadingProgress };
}

2. Extracting Metadata with Transformers.js

While Llama-3 handles the heavy reasoning, we can use Transformers.js for fast, local Named Entity Recognition (NER). This is great for identifying medications or symptoms before passing them to the LLM.

import { pipeline } from '@xenova/transformers';

const analyzeLogBasics = async (text) => {
  // Use a tiny, efficient model for fast extraction
  const extractor = await pipeline('token-classification', 'Xenova/bert-base-NER');
  const results = await extractor(text);

  // Filter for medical-related entities locally
  return results.filter(entity => ['MED', 'SYMPTOM'].includes(entity.entity));
};

3. The "Official" Way to Build Secure AI 🛡️

When building production-grade healthcare applications, simply running a model isn't enough. You need to handle state management, local encryption, and sophisticated prompt engineering.

For a deeper dive into production-ready Edge AI patterns and advanced security protocols for health data, I highly recommend checking out the technical deep-dives at WellAlly Blog. They offer incredible resources on how to bridge the gap between "cool browser demos" and "HIPAA-compliant local software."

4. Implementing the Health Log Logic

Now, let's combine everything into a React component. The user types their log, we extract entities, and then Llama-3 provides a clinical summary—all on the GPU.

import React, { useState } from 'react';
import { useWebLLM } from './hooks/useWebLLM';

const HealthAnalyzer = () => {
  const { engine, initEngine, loadingProgress } = useWebLLM();
  const [input, setInput] = useState("");
  const [output, setOutput] = useState("");

  const handleAnalyze = async () => {
    if (!engine) return;

    const messages = [
      { role: "system", content: "You are a private health assistant. Analyze the user's log for potential trends. Keep it professional." },
      { role: "user", content: input }
    ];

    const reply = await engine.chat.completions.create({ messages });
    setOutput(reply.choices[0].message.content);
  };

  return (
    <div className="p-8 max-w-2xl mx-auto">
      <h2 className="text-2xl font-bold mb-4">Local Health Log 🩺</h2>

      {!engine ? (
        <button 
          onClick={initEngine}
          className="bg-blue-600 text-white px-4 py-2 rounded"
        >
          Load Llama-3 ({loadingProgress}%)
        </button>
      ) : (
        <div className="space-y-4">
          <textarea 
            className="w-full border p-2"
            placeholder="e.g., Feeling dizzy after taking 20mg Lisinopril..."
            onChange={(e) => setInput(e.target.value)}
          />
          <button 
            onClick={handleAnalyze}
            className="bg-green-600 text-white px-4 py-2 rounded"
          >
            Analyze Privately
          </button>
          <div className="mt-4 p-4 bg-gray-100 rounded">
            <strong>Insight:</strong> {output}
          </div>
        </div>
      )}
    </div>
  );
};

💡 Why This Matters

Zero Latency: Once the model is loaded, inference happens at the speed of your GPU. No more waiting for "Server Busy" errors.
Cost Efficiency: You stop paying for OpenAI/Anthropic tokens. The user provides the compute!
GDPR/HIPAA by Default: Since the data never leaves the browser, your compliance surface area shrinks dramatically.

Conclusion

Running Llama-3 in the browser isn't just a party trick; it's a paradigm shift for Edge AI and Privacy. By using WebLLM and WebGPU, we give power back to the users while maintaining the "magic" of LLMs.

Are you ready to move your AI workloads to the edge? Let me know in the comments if you've tried running local models! And don't forget to visit wellally.tech/blog for more advanced AI architecture guides. 🥑💻

Happy coding!

DEV Community