DIY AI Physical Therapist: Real-Time Pose Correction with React Native and MediaPipe

#ai #react #healthtech #computervision

Have you ever tried doing physical therapy exercises at home, only to wonder if your form is actually helping or just making things worse? 🤕 Traditional home rehabilitation often lacks the "watchful eye" of a professional. However, with the rise of on-device computer vision and React Native development, we can now build powerful, low-latency movement correction tools that run directly on a smartphone.

In this tutorial, we are diving deep into MediaPipe pose estimation and mobile AI integration to build a "Smart Rehab Coach." We’ll explore how to capture real-time landmarks, calculate joint angles, and provide instant feedback—all while maintaining 60 FPS performance. By leveraging TensorFlow Lite and on-device processing, we ensure user privacy while delivering a seamless experience. 🚀

The Architecture 🏗️

To achieve real-time feedback, we need a pipeline that minimizes the "bridge" overhead in React Native. We use a frame processor to pipe camera data directly into the MediaPipe inference engine.

graph TD
  A[Camera Stream] -->|Native Frame| B(MediaPipe Pose Landmarker)
  B -->|33 3D Landmarks| C{Angle Calculation Engine}
  C -->|Compare| D[Reference Library]
  D -->|Feedback| E[React Native UI Overlay]
  E -->|Visual Cues| F[User]

  subgraph "On-Device Processing"
  B
  C
  D
  end

Prerequisites 🛠️

Before we start coding, ensure your environment is ready:

React Native: 0.70+ (using VisionCamera or similar)
MediaPipe Tasks-Vision: For the Pose Landmarker model
TensorFlow Lite: Optimized for mobile (GPU/NNAPI delegates)
Math Logic: Basic trigonometry for joint angle calculation

Step 1: Initializing the Pose Landmarker

The heart of our application is the MediaPipe Pose Landmarker. Unlike cloud-based solutions, this runs locally on the device's NPU/GPU.

import { PoseLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

async function createPoseLandmarker() {
  const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
  );

  const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath: `pose_landmarker_lite.task`,
      delegate: "GPU" // Critical for real-time performance ⚡
    },
    runningMode: "VIDEO",
    numPoses: 1
  });

  return poseLandmarker;
}

Step 2: The Geometry of Correction 📐

To tell a user their arm isn't straight enough, we need to calculate the angle between three points (e.g., Shoulder, Elbow, Wrist). We use the Law of Cosines or the atan2 function.

// Helper to calculate angle between three landmarks
const calculateAngle = (a: Landmark, b: Landmark, c: Landmark): number => {
  const radians = Math.atan2(c.y - b.y, c.x - b.x) - 
                  Math.atan2(a.y - b.y, a.x - b.x);
  let angle = Math.abs((radians * 180.0) / Math.PI);

  if (angle > 180.0) {
    angle = 360 - angle;
  }
  return angle;
};

// Implementation for a Bicep Curl or Overhead Press
const checkForm = (landmarks: Landmark[]) => {
  const shoulder = landmarks[11];
  const elbow = landmarks[13];
  const wrist = landmarks[15];

  const elbowAngle = calculateAngle(shoulder, elbow, wrist);

  if (elbowAngle > 160) return "Fully Extended";
  if (elbowAngle < 45) return "Good Contraction";
  return "Keep Moving!";
};

Step 3: Real-Time Feedback Loop

In React Native, we use react-native-vision-camera with a frame processor. This allows us to run our logic on every single frame captured by the lens.

function RehabCamera() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet';
    const poses = detectPose(frame); // Native call to MediaPipe

    if (poses.length > 0) {
      const angle = calculateAngle(poses[0].landmarks[11], ...);

      // Update UI via Shared Values
      if (angle < 90) {
        feedbackText.value = "Lower your hips!";
      }
    }
  }, []);

  return (
    <Camera
      device={device}
      isActive={true}
      frameProcessor={frameProcessor}
    />
  );
}

Deep Dive: Advanced Optimization 🥑

Building a production-ready vision app involves more than just landmark detection. You need to handle jitter (using a Kalman Filter), varying light conditions, and different body types.

For those looking to scale this into an enterprise-grade solution, check out the specialized patterns on WellAlly Tech Blog. They cover advanced topics like:

Optimizing TFLite models for heterogeneous mobile hardware.
Advanced smoothing algorithms for noisy pose data.
Production-level state machines for exercise repetition counting.

Integrating these patterns ensures that your app doesn't just "detect" poses, but actually "understands" human movement at a clinical level.

Conclusion 🏁

On-device AI is transforming how we approach healthcare and fitness. By combining React Native for the UI and MediaPipe for the intelligence, we can create low-latency, private, and highly effective rehabilitation tools.

The future of physical therapy isn't just in the clinic—it's in the pocket of every patient. 📱💪

Are you building something with Pose Estimation? Drop a comment below or share your repo! I’d love to see how you’re handling landmark smoothing!

DEV Community