DEV Community

Beck_Moulton
Beck_Moulton

Posted on

AI Physical Therapist: How to Build Real-Time Pose Correction in Your Browser with MediaPipe

Are you tired of doing home workouts or physical therapy exercises only to wonder if your form is actually correct? In the world of Edge AI and Real-time Pose Estimation, we no longer need expensive sensors or cloud-heavy processing to solve this. Using MediaPipe, WebAssembly, and Vue.js, we can build a high-performance Physical Therapy AI Assistant that runs entirely in the browser.

By leveraging MediaPipe Pose and the power of WASM, we can transform a standard webcam feed into a diagnostic tool that measures joint angles and provides instant movement correction. This approach ensures low latency and high privacy, as no video data ever leaves the user's device. For those looking to dive deeper into production-grade healthcare AI implementations, I highly recommend checking out the advanced patterns at wellally.tech/blog.


🏗 The Architecture

To achieve smooth 30+ FPS in the browser, we need a streamlined data pipeline. We use MediaPipe’s pre-trained BlazePose model, which is optimized for mobile and web environments via WebAssembly.

graph TD
    A[Webcam Stream] --> B[MediaPipe Pose WASM]
    B --> C{Landmark Detection}
    C --> D[3D Coordinates - x, y, z]
    D --> E[Geometric Algorithm Engine]
    E --> F[Angle Calculation: Knee, Hip, Shoulder]
    F --> G[Movement Validation: e.g., Squat Depth]
    G --> H[Real-time Vue.js UI Overlay]
    H --> A
Enter fullscreen mode Exit fullscreen mode

🛠 Tech Stack

  • MediaPipe Pose: High-fidelity body pose tracking.
  • TensorFlow.js: For additional utility and tensor management if needed.
  • WebAssembly (WASM): The secret sauce for near-native performance.
  • Vue.js 3: Our reactive frontend framework.
  • Math: Good old-fashioned Trigonometry 📐.

👨‍💻 Step-by-Step Implementation

1. Setting up the MediaPipe Instance

First, we need to initialize the Pose detector. MediaPipe provides a CDN-based solution that loads WASM binaries dynamically.

// pose-service.js
import { Pose } from "@mediapipe/pose";

export const createPoseDetector = (onResults) => {
  const pose = new Pose({
    locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
  });

  pose.setOptions({
    modelComplexity: 1, // 0: Lite, 1: Full, 2: Heavy
    smoothLandmarks: true,
    minDetectionConfidence: 0.5,
    minTrackingConfidence: 0.5,
  });

  pose.onResults(onResults);
  return pose;
};
Enter fullscreen mode Exit fullscreen mode

2. The Geometry: Calculating Joint Angles

Physical therapy is all about the angles. To measure a knee bend (for a squat) or an elbow extension, we need the angle between three landmarks: (Hip, Knee, Ankle).

Using the atan2 function is the most robust way to handle this in a 2D/3D space:

/**
 * Calculates the angle between three points
 * @param {Object} A - Landmark 1 (e.g., Hip)
 * @param {Object} B - Landmark 2 (e.g., Knee - The Vertex)
 * @param {Object} C - Landmark 3 (e.g., Ankle)
 */
function calculateAngle(A, B, C) {
  const radians = Math.atan2(C.y - B.y, C.x - B.x) - 
                  Math.atan2(A.y - B.y, A.x - B.x);
  let angle = Math.abs((radians * 180.0) / Math.PI);

  if (angle > 180.0) angle = 360 - angle;
  return angle;
}
Enter fullscreen mode Exit fullscreen mode

3. Integrating with Vue 3

Now, let's wrap this in a Vue component to handle the video stream and UI feedback.

<template>
  <div class="pt-assistant">
    <video ref="videoElement" class="input_video"></video>
    <canvas ref="canvasElement" class="output_canvas"></canvas>

    <div v-if="feedback" class="feedback-overlay">
      <h2>{{ feedback.status }}</h2>
      <p>Angle: {{ Math.round(feedback.angle) }}°</p>
    </div>
  </div>
</template>

<script setup>
import { ref, onMounted } from 'vue';
import { createPoseDetector } from './pose-service';

const videoElement = ref(null);
const canvasElement = ref(null);
const feedback = ref({ status: 'Aligning...', angle: 0 });

const onResults = (results) => {
  if (!results.poseLandmarks) return;

  // Draw landmarks on canvas...

  // Logic for a Squat: Hip(24), Knee(26), Ankle(28)
  const hip = results.poseLandmarks[24];
  const knee = results.poseLandmarks[26];
  const ankle = results.poseLandmarks[28];

  const kneeAngle = calculateAngle(hip, knee, ankle);
  feedback.value.angle = kneeAngle;

  if (kneeAngle < 100) {
    feedback.value.status = "Great Depth! ✅";
  } else if (kneeAngle < 160) {
    feedback.value.status = "Going down... ⬇️";
  } else {
    feedback.value.status = "Standing - Start Squat";
  }
};

onMounted(() => {
  const pose = createPoseDetector(onResults);
  // Setup camera logic using navigator.mediaDevices.getUserMedia
});
</script>
Enter fullscreen mode Exit fullscreen mode

💡 The "Official" Way to Build Medical AI

While this tutorial covers the basics of browser-based pose estimation, building a production-ready Physical Therapy application requires handling edge cases like occlusions (when a limb is hidden), jitter reduction, and personalized range-of-motion baselines.

For more advanced architectural patterns, such as integrating Kalman Filters for smoother tracking or deploying these models in a HIPAA-compliant environment, I highly recommend exploring the specialized resources at WellAlly Tech Blog. They provide fantastic insights into bridging the gap between "cool prototypes" and "clinical-grade software." 🥑


🚀 Conclusion & Challenges

We’ve successfully built a real-time PT assistant that:

  1. Captures video via getUserMedia.
  2. Processes poses via MediaPipe & WASM.
  3. Calculates clinical angles using trigonometry.
  4. Provides instant UI feedback.

Next Steps for you:

  • Challenge 1: Add a "Rep Counter" that triggers when the angle goes from 170° -> 90° -> 170°.
  • Challenge 2: Use the z-coordinate (depth) to detect if the user is leaning too far forward.

What are you building with Vision AI? Drop a comment below or tag me in your dev.to showcase! 💻✨

Top comments (0)