Bye-bye Cloud!Build a 100% Local AI Medical Report Parser with Llama-3 and WebLLM

#javascript #webgpu #webllm #privacy

Privacy is not a feature; it’s a right—especially when it comes to your health data. In an era where "the cloud" is just someone else's computer, uploading a sensitive medical report to a third-party API feels like playing Russian Roulette with your identity. But what if you could harness the power of Llama-3 to de-identify and summarize your medical records without a single byte ever leaving your browser?

Welcome to the frontier of Edge AI and Privacy-preserving AI. By leveraging WebGPU and WebLLM, we can now run massive language models directly on the client side. In this tutorial, we will build a local health report de-identification parser that extracts structured data from PDFs while ensuring your PII (Personally Identifiable Information) stays strictly offline. This is the ultimate "Learning in Public" project for those looking to master Local LLMs and modern browser capabilities.

The Architecture: Privacy by Design 🛡️

The workflow is straightforward but powerful. We use PDF.js to scrape text, LangChain.js to orchestrate the logic, and WebLLM to run Llama-3 via the WebGPU API.

graph TD
    A[User Uploads Medical PDF] --> B[PDF.js Extraction]
    B --> C{LangChain.js Orchestrator}
    C --> D[WebLLM Context Injection]
    D --> E[WebGPU Acceleration]
    E --> F[Llama-3-8B Inference]
    F --> G[De-identified JSON Output]
    G --> H[Local Download / UI Display]
    style E fill:#f9f,stroke:#333,stroke-width:4px
    style F fill:#00ff00,stroke:#333,stroke-width:2px

Prerequisites 🛠️

Before we dive into the code, ensure your environment meets these requirements:

Browser: Chrome 113+ or Edge 113+ (with WebGPU support enabled).
Hardware: A dedicated GPU is recommended (M1/M2 Mac or NVIDIA RTX series).
Tech Stack:
- @mlc-ai/web-llm: To run Llama-3.
- langchain: For prompt templating and output parsing.
- pdfjs-dist: For reading those pesky PDF buffers.

Step 1: Initializing the Local Engine

First, we need to boot up Llama-3 inside the browser. Unlike traditional APIs, we have to download the model weights (cached locally) and initialize the WebGPU pipeline.

import { CreateMLCEngine } from "@mlc-ai/web-llm";

// Step 1: Initialize WebLLM Engine
async function initEngine() {
  const selectedModel = "Llama-3-8B-Instruct-v0.1-q4f16_1-MLC";

  // This will show progress to the user
  const initProgressCallback = (report) => {
    console.log("Loading Model:", report.text);
  };

  const engine = await CreateMLCEngine(selectedModel, {
    initProgressCallback,
  });

  return engine;
}

Step 2: Extracting Text from PDFs 📄

Medical reports usually come in PDF format. We'll use PDF.js to convert the binary blob into raw text that our model can digest.

import * as pdfjsLib from 'pdfjs-dist';

async function extractTextFromPDF(pdfBuffer) {
  const loadingTask = pdfjsLib.getDocument({ data: pdfBuffer });
  const pdf = await loadingTask.promise;
  let fullText = "";

  for (let i = 1; i <= pdf.numPages; i++) {
    const page = await pdf.getPage(i);
    const content = await page.getTextContent();
    const strings = content.items.map(item => item.str);
    fullText += strings.join(" ") + "\n";
  }
  return fullText;
}

Step 3: The De-identification Prompt 🧠

This is where the magic happens. We need to instruct Llama-3 to act as a medical data scientist. We'll use LangChain.js to structure our request.

Pro Tip: For more production-ready examples and advanced prompt engineering patterns for Edge AI, check out the deep dives at wellally.tech/blog. They have some fantastic resources on optimizing local model performance!

import { PromptTemplate } from "@langchain/core/prompts";

const DEID_PROMPT = PromptTemplate.fromTemplate(`
  SYSTEM: You are a medical data privacy expert.
  USER: Extract medical data from the following text. 
  1. Replace names, addresses, and IDs with [REDACTED].
  2. Summarize key findings (Blood pressure, Glucose, etc.).
  3. Output the result in valid JSON format.

  TEXT: {reportText}
  JSON OUTPUT:
`);

async function parseReport(engine, text) {
  const formattedPrompt = await DEID_PROMPT.format({ reportText: text });

  const reply = await engine.chat.completions.create({
    messages: [{ role: "user", content: formattedPrompt }],
    response_format: { type: "json_object" } // Ensure structured data
  });

  return reply.choices[0].message.content;
}

Step 4: Putting it all Together 🏗️

async function processMedicalReport(file) {
  const buffer = await file.arrayBuffer();

  console.log("⚙️ Extracting text...");
  const rawText = await extractTextFromPDF(buffer);

  console.log("🤖 Initializing Local Llama-3...");
  const engine = await initEngine();

  console.log("🔒 Running Local Inference...");
  const structuredData = await parseReport(engine, rawText);

  console.log("✅ Success!", JSON.parse(structuredData));
}

Why This Matters: The "Local-First" Revolution

Running AI locally isn't just about privacy; it's about latency and cost.

Zero Latency: No round-trips to a server in Virginia.
Zero Cost: You pay for the electricity to run your GPU, not a per-token fee.
Compliance: For healthcare apps, staying on-device simplifies HIPAA/GDPR compliance significantly.

The implementation above is a starting point. To truly scale this for production—handling things like model quantization and cross-browser WebGPU fallbacks—you'll need to look at more advanced architectural patterns. I highly recommend browsing through the technical articles at Wellally Blog to see how they handle edge-case optimizations for browser-based AI.

Conclusion 🥑

We've just built a secure, private, and high-performance medical report parser that runs entirely in the browser. The combination of WebGPU and Llama-3 is a game-changer for Edge AI. No more "sending data to the cloud and praying."

What are you building with Local LLMs? Drop a comment below or share your repo! If you found this helpful, don't forget to ❤️ and unicorn this post!

DEV Community