wellallyTech

Posted on Jan 23

AI Physical Therapist: Build a Real-Time Posture Correction App with Mediapipe and Vue.js 🦾

#ai #nextjs #api #webdev

Let's face it: physical therapy is expensive, and doing rehab exercises at home is... well, boring and prone to "cheating." We’ve all been there—slouching during a squat or not fully extending our arms during a bicep curl. But what if your laptop could talk back and tell you exactly how to fix your form?

In this tutorial, we are diving into the world of AI motion correction using Pose Estimation. We’ll build a system that calculates joint angles in real-time and provides instant feedback. By leveraging Mediapipe, OpenCV logic, and Vue.js, we are transforming a standard webcam into a high-precision clinical tool.

Whether you're interested in real-time posture analysis, building a fitness app, or exploring Mediapipe tutorials, this guide covers the full pipeline from pixels to actionable data.

🏗️ The System Architecture

Before we jump into the code, let's visualize how data flows from your webcam to the "AI Coach" logic. We need a low-latency pipeline to ensure the feedback feels instantaneous.

graph TD
    A[Webcam Stream] --> B[Vue.js Frontend]
    B --> C{Mediapipe Pose Engine}
    C -->|Landmarks| D[Angle Calculation Logic]
    D --> E[Comparison vs. PT Standards]
    E -->|Correction Data| F[UI Overlays / Canvas]
    E -->|Audio Feedback| G[Web Speech API]
    F --> B
    G --> B

🛠️ Prerequisites

To follow along, you'll need a basic understanding of JavaScript and Vue.js. Here is our "Physio-Tech" stack:

Mediapipe (@mediapipe/pose): The heavy lifter for landmark detection.
Vue.js 3: For a reactive and snappy UI.
Webcam API: To grab frames.
Web Speech API: To give our AI a voice.

🚀 Step 1: Setting up the Pose Engine

First, we need to initialize Mediapipe. Unlike heavy deep learning models, Mediapipe is incredibly optimized for the browser.

import { Pose } from "@mediapipe/pose";
import { Camera } from "@mediapipe/camera_utils";

const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

// This will trigger every time a frame is processed
pose.onResults(onResults);

📐 Step 2: The Geometry of Movement

The secret sauce of AI motion correction isn't just finding points on a body; it's the math between them. To check if a patient is performing a "Bicep Curl" correctly, we need to calculate the angle at the elbow.

We treat the Shoulder, Elbow, and Wrist as vectors.

function calculateAngle(a, b, c) {
    const radians = Math.atan2(c.y - b.y, c.x - b.x) - 
                    Math.atan2(a.y - b.y, a.x - b.x);
    let angle = Math.abs((radians * 180.0) / Math.PI);

    if (angle > 180.0) angle = 360 - angle;
    return angle;
}

// Usage in our onResults function:
const elbowAngle = calculateAngle(
    results.poseLandmarks[11], // Left Shoulder
    results.poseLandmarks[13], // Left Elbow
    results.poseLandmarks[15]  // Left Wrist
);

🎙️ Step 3: Real-Time Feedback Logic

Numbers are great for developers, but users need instructions like "Straighten your back!" or "Go lower!" We use the Web Speech API to make the app interactive.

function giveFeedback(angle) {
    const synth = window.speechSynthesis;

    if (angle > 160 && !isExtended) {
        const utter = new SpeechSynthesisUtterance("Great extension!");
        synth.speak(utter);
        isExtended = true; // State management to prevent spamming
    } else if (angle < 40 && isExtended) {
        const utter = new SpeechSynthesisUtterance("Good squeeze!");
        synth.speak(utter);
        isExtended = false;
    }
}

🌟 The "Official" Way to Scale

While this "Learning in Public" project is a great start, building production-ready medical or fitness AI requires handling edge cases like occlusion (when a limb is hidden), varying lighting conditions, and complex 3D kinematic chains.

If you're looking for advanced motion analysis patterns or more production-ready computer vision examples, I highly recommend checking out the engineering deep-dives at WellAlly Tech Blog. They cover how to take these "Intermediate" concepts and scale them into robust healthcare solutions.

🎨 Step 4: Visualizing the Skeleton

Finally, we draw the landmarks back onto a <canvas> so the user can see their "Digital Twin."

function onResults(results) {
    canvasCtx.save();
    canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);

    // Draw the video frame
    canvasCtx.drawImage(results.image, 0, 0, canvasElement.width, canvasElement.height);

    if (results.poseLandmarks) {
        // Use Mediapipe's helper to draw connectors
        drawConnectors(canvasCtx, results.poseLandmarks, POSE_CONNECTIONS,
                       {color: '#00FF00', lineWidth: 4});
        drawLandmarks(canvasCtx, results.poseLandmarks,
                      {color: '#FF0000', lineWidth: 2});
    }
    canvasCtx.restore();

    // Trigger our AI Logic
    processRehabLogic(results.poseLandmarks);
}

🏁 Conclusion

By combining Mediapipe's Pose Estimation with a bit of trigonometry and the Webcam API, we've built a functional home rehabilitation tool! This setup is just the beginning. You could extend this to:

Rep counting: Automating the tracking of sets.
Form scoring: Comparing the user's movement curve against a "perfect" pro athlete's curve.
Gamification: Adding XP for every correct movement.

What are you building next? Let me know in the comments! If you found this helpful, don't forget to ❤️ and 🦄 this post!

Follow me for more "Learning in Public" AI tutorials. For deep technical architecture on AI-driven health, visit wellally.tech/blog. 🥑

DEV Community