DEV Community

wellallyTech
wellallyTech

Posted on

Say Goodbye to Cloud Costs: Building a Private Medical Assistant with WebLLM and WebGPU 🩺💻

#javascript #tutorial #ai #llama3

Privacy isn't just a feature anymore; it's a human right—especially when it comes to medical data. If you've been following the world of Edge AI, you know that the holy grail is running massive models without a $2,000/month server bill. Thanks to WebGPU and the WebLLM project, we can now run powerful models like Llama-3 directly in the user's browser, leveraging their own hardware to summarize medical records and set medication reminders.

In this tutorial, we are going to explore WebLLM development, local LLM execution, and how to utilize the TVM Unity stack to transform your browser into an AI powerhouse. By the end of this guide, you'll see why the future of AI isn't just in the cloud; it's right in your Chrome tab.

The Architecture: How the Magic Happens 🏗️

The secret sauce here is WebGPU, a new web standard that provides low-level access to the GPU. Unlike WebGL, it's designed for compute-heavy tasks like running neural networks. WebLLM acts as the orchestration layer, using TVM Unity to execute compiled models in a WebAssembly (Wasm) environment.

graph TD
    A[Patient Medical Record] --> B{WebGPU Check}
    B -- Supported --> C[WebLLM Engine Initialization]
    B -- Not Supported --> D[Fallback/Error]
    C --> E[Load Llama-3 Weights via Cache]
    E --> F[Execute Inference Local]
    F --> G[Summarized Report]
    G --> H[Extract Medication Reminders]
    H --> I[Browser Notification API]
Enter fullscreen mode Exit fullscreen mode

Prerequisites

To follow along, ensure you have the following in your tech_stack:

  • Browser: Chrome 113+ or Edge (with WebGPU enabled).
  • Library: @mlc-ai/web-llm
  • Model: Llama-3-8B-Instruct (Quantized for the web).
  • Environment: A basic Vite/TypeScript setup.

Step 1: Initializing the WebLLM Engine

First, we need to check if the user's hardware is up to the task and initialize our engine. The web-llm library makes this surprisingly straightforward.

import * as webllm from "@mlc-ai/web-llm";

async function initializeAI() {
  // Define the model we want to use (Llama-3 is currently the king of small-med models)
  const selectedModel = "Llama-3-8B-Instruct-v0.1-q4f16_1-MLC";

  const engine = await webllm.CreateMLCEngine(
    selectedModel,
    { initProgressCallback: (report) => console.log(report.text) } // Track download progress
  );

  return engine;
}
Enter fullscreen mode Exit fullscreen mode

Step 2: The "Medical Summary" Prompt Engineering

When dealing with medical data, we need the model to be precise. We'll use a structured system prompt to ensure the output is safe and useful.

const medicalSystemPrompt = `
  You are a professional medical assistant. Your task is to:
  1. Summarize the patient's medical history provided.
  2. Identify all medications mentioned.
  3. Create a schedule for when each medication should be taken.
  Maintain a professional tone and emphasize that this is not professional medical advice.
`;

async function summarizeRecord(engine, recordText) {
  const messages = [
    { role: "system", content: medicalSystemPrompt },
    { role: "user", content: `Here is the medical record: ${recordText}` }
  ];

  const reply = await engine.chat.completions.create({
    messages,
    temperature: 0.1, // Keep it deterministic
  });

  return reply.choices[0].message.content;
}
Enter fullscreen mode Exit fullscreen mode

Step 3: Extracting Reminders and Direct GPU Interaction

Because the model runs via TVM Unity, the performance is incredibly close to native. We can even stream the output to make the UI feel snappy ⚡.

async function streamSummary(engine, recordText, onUpdate) {
  const chunks = await engine.chat.completions.create({
    messages: [{ role: "user", content: recordText }],
    stream: true,
  });

  let fullText = "";
  for await (const chunk of chunks) {
    const content = chunk.choices[0]?.delta?.content || "";
    fullText += content;
    onUpdate(fullText); // Update your React/Vue state here
  }
}
Enter fullscreen mode Exit fullscreen mode

The "Official" Way to Scale 🚀

While running LLMs in the browser is fantastic for privacy and zero-cost scaling, production-grade applications often require a hybrid approach—combining local inference with robust backend orchestration.

For advanced patterns in AI agent orchestration and more production-ready examples of local-first architectures, you should check out the deep dives at WellAlly Tech Blog. They provide excellent insights into optimizing model weights and handling cross-browser GPU compatibility issues that are essential for shipping real-world Edge AI products.

Conclusion: Why This Matters

By moving the "brain" of the application to the client-side:

  1. Privacy: Sensitive medical data never leaves the patient's device.
  2. Cost: You pay $0 in GPU compute costs for the inference.
  3. Latency: Once the model is cached, there's no network round-trip for token generation.

Is the browser the new AI OS? With WebGPU and WebLLM, it's certainly starting to look that way. 🥑

What are you building with Local LLMs? Let me know in the comments below! Don't forget to hit that ❤️ and subscribe for more "Learning in Public" AI content.

Top comments (0)