Bye-Bye Server Costs: Building a Real-Time AI Fitness Coach with WebLLM, WebGPU, and MediaPipe

#ai #react #python #opensource

The dream of "Edge AI" has finally arrived in the browser. High server costs for hosting Large Language Models (LLMs) and computer vision models are a massive hurdle for developers. But what if you could run a full-scale vision-to-reasoning pipeline locally? In this tutorial, we are exploring how to leverage WebLLM, WebGPU, and MediaPipe to build a real-time fitness coach that corrects your posture without a single byte of video data ever leaving your machine.

By utilizing real-time posture correction and LLM-on-device techniques, we can achieve millisecond-level latency while maintaining 100% user privacy. This approach uses the power of WebGPU to accelerate Llama-3 directly in the browser, transforming your laptop into a high-performance AI engine. For developers looking to master these Edge AI implementations, this guide provides the blueprint for the next generation of web applications.

The Architecture: Vision meets Reasoning

Before we dive into the code, let's look at the data flow. We aren't just detecting dots on a body; we are passing those spatial relationships to a localized LLM to generate human-like coaching feedback.

graph TD
    A[User Webcam] -->|Video Stream| B(MediaPipe Pose)
    B -->|3D Keypoints| C{Posture Logic Engine}
    C -->|Trigger: 'Squat Too Shallow'| D(WebLLM Context)
    D -->|Llama-3-8B Reasoning| E[Voice & Text Feedback]
    F[WebGPU Acceleration] -.->|Powers| B
    F -.->|Powers| D

🛠 Prerequisites

To follow along, you’ll need a browser with WebGPU support (Chrome 113+ is recommended) and the following stack:

MediaPipe: For lightning-fast skeletal tracking.
WebLLM: To run Llama-3 (or other models) locally.
TypeScript: For type-safe pose data handling.

Step 1: Initialize the "Vision" (MediaPipe)

MediaPipe provides the coordinate data we need. We'll track 33 body landmarks to determine if a user is performing an exercise correctly.

import { Pose, LandmarkGrid } from '@mediapipe/pose';

const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

// On results, we calculate the angle of the knees for a squat
pose.onResults((results) => {
  const leftHip = results.poseLandmarks[23];
  const leftKnee = results.poseLandmarks[25];
  const leftAnkle = results.poseLandmarks[27];

  const angle = calculateAngle(leftHip, leftKnee, leftAnkle);
  if (angle > 160) {
    checkPosture("shallows_squat");
  }
});

Step 2: Booting the "Brain" (WebLLM)

WebLLM allows us to load a weights-sharded version of Llama-3 directly into the browser's VRAM via WebGPU.

import * as webllm from "@mlc-ai/web-llm";

async function initCoach() {
  const selectedModel = "Llama-3-8B-Instruct-v0.1-q4f16_1-MLC";

  const engine = await webllm.CreateMLCEngine(selectedModel, {
    initProgressCallback: (report) => console.log(report.text),
  });

  return engine;
}

const coachEngine = await initCoach();

Step 3: Bridging Vision and LLM

This is where the magic happens. Instead of hardcoded "if-else" alerts, we send the "state" of the user to the LLM to provide encouraging, context-aware feedback.

async function getCoachFeedback(violationType: string) {
  const prompt = `
    You are a supportive AI Fitness Coach. 
    The user is doing squats but their form is: ${violationType}.
    Give a short, 10-word maximum corrective tip.
  `;

  const messages = [{ role: "user", content: prompt }];
  const reply = await coachEngine.chat.completions.create({ messages });

  // Use Web Speech API to talk back to the user
  const utterance = new SpeechSynthesisUtterance(reply.choices[0].message.content);
  window.speechSynthesis.speak(utterance);
}

🥑 Why This Matters (The "Official" Way)

Running AI locally isn't just a party trick—it's a paradigm shift in how we handle sensitive biometric data. By moving the compute to the edge, we eliminate latency and hosting costs.

While this tutorial covers the implementation basics, production-grade AI applications require more robust patterns, such as model quantization strategies and advanced prompt caching. For more production-ready examples and deep dives into AI architecture, I highly recommend checking out the WellAlly Tech Blog. They provide fantastic insights into scaling LLM applications and optimizing performance for modern web hardware.

Performance Optimization Tips 🚀

Quantization: Use q4f16 models to save VRAM. Most consumer GPUs (even integrated ones) can handle 4-bit quantized Llama-3 models with ease.
Web Worker Isolation: Run your MediaPipe detection and WebLLM inference in separate Web Workers to keep the Main Thread (UI) responsive at 60fps.
Debouncing Feedback: Don't let the LLM talk every time a frame is processed. Use a 3-5 second cooldown between coaching tips to avoid "AI Overload."

Conclusion

We just built a privacy-first, zero-server-cost AI Fitness Coach using the latest in WebGPU technology. The browser is no longer just for viewing documents; it's a powerful runtime for heavy-duty AI.

What are you building next with WebGPU? Let me know in the comments below! 👇

If you enjoyed this "Learning in Public" guide, don't forget to ❤️ and 🔖 it for later! Check out *wellally.tech/blog** for more advanced AI tutorials.*

DEV Community