DEV Community

Beck_Moulton
Beck_Moulton

Posted on

From Pixels to Physical Therapy: Building a Real-Time Pose Correction System with MediaPipe and React

#ai #react #machinelearning #opencv

In the era of telehealth, the gap between a patient's living room and the physiotherapy clinic is closing fast. Thanks to breakthroughs in Computer Vision and Edge AI, we no longer need expensive motion-capture suits to track movement. Using MediaPipe, Pose Estimation, and WebAssembly, we can turn a standard webcam into a sophisticated clinical tool that provides real-time biomechanical feedback.

This tutorial will show you how to build a lightweight skeleton tracking system capable of analyzing exercises like squats or lunges. We will leverage TensorFlow.js and the high-performance MediaPipe runtime to calculate joint angles directly in the browser. If you're looking for more advanced production-ready patterns and deep dives into AI-driven healthcare, be sure to check out the technical resources at WellAlly Tech Blog, which served as the inspiration for this architecture.

The Architecture: From Frame to Feedback

To achieve "real-time" performance (30+ FPS) on a browser, we need a pipeline that minimizes latency. By using WebAssembly (WASM), MediaPipe processes video frames at near-native speeds.

graph TD
    A[Webcam Stream] --> B[React Camera Component]
    B --> C[MediaPipe Pose Engine]
    C --> D{Keypoint Detection}
    D --> E[Angle Calculation Logic]
    E --> F[Visual Overlay / Canvas]
    E --> G[Speech Synthesis Feedback]
    G --> H[User: 'Straighten your back!']
Enter fullscreen mode Exit fullscreen mode

Prerequisites

Before we dive into the code, ensure you have a React environment ready. We will be using:

  • MediaPipe Pose: For high-fidelity body tracking.
  • React: For the UI layer.
  • WebAssembly: To run the ML models efficiently in-browser.

Step 1: Initializing the Pose Engine

First, we need to set up the MediaPipe Pose listener. This engine detects 33 3D landmarks on the human body.

import { Pose } from "@mediapipe/pose";
import * as cam from "@mediapipe/camera_utils";

const pose = new Pose({
  locateFile: (file) => {
    return `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`;
  },
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});
Enter fullscreen mode Exit fullscreen mode

Step 2: The Math - Calculating Joint Angles

For physical therapy, knowing "where" a joint is isn't enough; we need to know the angle. For a squat, we calculate the angle between the Hip, Knee, and Ankle.

/**
 * Calculates the angle between three points (A, B, C)
 * B is the vertex (e.g., the Knee)
 */
function calculateAngle(a, b, c) {
    const radians = Math.atan2(c.y - b.y, c.x - b.x) - 
                    Math.atan2(a.y - b.y, a.x - b.x);
    let angle = Math.abs((radians * 180.0) / Math.PI);

    if (angle > 180.0) angle = 360 - angle;
    return angle;
}

// Usage in the MediaPipe results callback:
const kneeAngle = calculateAngle(results.poseLandmarks[24], // Hip
                                 results.poseLandmarks[26], // Knee
                                 results.poseLandmarks[28]);// Ankle
Enter fullscreen mode Exit fullscreen mode

Step 3: Real-Time Feedback Logic

Now, let's turn those numbers into actionable advice. If the user's knee angle suggests they are squatting too shallowly or their form is collapsing, the system will trigger a voice command.

const provideFeedback = (angle) => {
  const synth = window.speechSynthesis;

  if (angle > 160) {
    // Standing up
    return "Lower your hips more.";
  } else if (angle < 90) {
    // Too deep for some patients
    const utter = new SpeechSynthesisUtterance("Great depth, keep your back straight!");
    synth.speak(utter);
  }
};
Enter fullscreen mode Exit fullscreen mode

The "Official" Way to Scale

While this implementation is great for a weekend project, production-grade Remote Physiotherapy platforms require much more: data persistence, HIPAA compliance, and advanced filtering to handle "noisy" environments.

For developers looking to take this to the next level—such as implementing multi-person tracking or integrating with backend HL7 FHIR standards—I highly recommend exploring the deep-dive articles at wellally.tech/blog. They offer incredible insights into the intersection of Edge AI and modern healthcare infrastructure.

Conclusion: Your Browser is a Kinesiologist

By combining MediaPipe's skeleton tracking with simple trigonometric functions, we've built a system that can literally "see" and "correct" human movement. This is the power of modern Vision AI—bringing specialized expertise to anyone with a browser and a webcam.

What's next?

  1. Add a "Rep Counter" using a simple state machine.
  2. Integrate a Chart.js dashboard to track knee mobility over time.
  3. Share your progress in the comments below! 👇

Top comments (0)