Physiotherapy is one of those things we all know we should do, but "doing it at home" usually results in terrible form and zero progress. Whether you're recovering from an ACL tear or just trying to fix that "tech neck," accuracy is everything.
In this tutorial, we are building a Real-Time Pose Correction Engine. We'll leverage Computer Vision and Pose Estimation to calculate Range of Motion (ROM) directly in the browser. Using Mediapipe for landmark detection and WebSockets for a reactive feedback loop, we’ll turn a standard webcam into a professional-grade physical therapy assistant.
If you are looking for more production-ready healthcare AI implementations or advanced computer vision patterns, definitely check out the deep dives over at WellAlly Tech Blog.
The Architecture 🏗️
The system follows a "Client-Side Processing, Server-Side Validation" pattern. We process the heavy video frames on the client using WebAssembly (via Mediapipe) to ensure zero latency, while using WebSockets to sync data with a Node.js backend for logging or advanced clinical analysis.
graph TD
A[Webcam Feed] --> B[Mediapipe Pose Engine]
B --> C{Landmark Extraction}
C --> D[Angle Calculation Logic]
D --> E[React UI Overlay]
D --> F[WebSocket Bridge]
F --> G[Node.js Backend]
G --> H[Clinical Data Validation]
H --> |Correction Data| F
F --> |Real-time Alert| E
Prerequisites 🛠️
To follow along, you'll need:
- Mediapipe:
@mediapipe/posefor the heavy lifting. - React: For the reactive UI state.
- Socket.io: To handle the bidirectional communication.
- Node.js: Our backend "referee."
Step 1: Calculating the Joint Angle 📐
The heart of physical therapy is the Angle. To tell a user if their squat is deep enough or their arm is straight, we need to calculate the angle between three points (e.g., Shoulder, Elbow, Wrist).
We use the 2D coordinates provided by Mediapipe and a bit of trigonometry:
/**
* Calculates the angle between three landmarks
* @param {Object} A - Landmark 1 (e.g., Shoulder)
* @param {Object} B - Landmark 2 (Middle point, e.g., Elbow)
* @param {Object} C - Landmark 3 (e.g., Wrist)
*/
const calculateAngle = (A, B, C) => {
const radians = Math.atan2(C.y - B.y, C.x - B.x) -
Math.atan2(A.y - B.y, A.x - B.x);
let angle = Math.abs((radians * 180.0) / Math.PI);
if (angle > 180.0) {
angle = 360 - angle;
}
return angle;
};
Step 2: The Mediapipe + React Integration 💻
We need to hook into the webcam feed and pass every frame to the Mediapipe Pose engine. We use requestAnimationFrame to keep the UI buttery smooth.
import React, { useEffect, useRef } from 'react';
import { Pose } from '@mediapipe/pose';
import * as cam from '@mediapipe/camera_utils';
const PoseEngine = () => {
const webcamRef = useRef(null);
const canvasRef = useRef(null);
const onResults = (results) => {
if (!results.poseLandmarks) return;
// Extract specific landmarks (e.g., Left Arm)
const shoulder = results.poseLandmarks[11];
const elbow = results.poseLandmarks[13];
const wrist = results.poseLandmarks[15];
const currentAngle = calculateAngle(shoulder, elbow, wrist);
// Logic: If angle < 160 during a stretch, send a correction
if (currentAngle < 160) {
socket.emit('pose-update', { angle: currentAngle, type: 'BICEP_STRETCH' });
}
drawCanvas(results); // Function to draw landmarks on UI
};
useEffect(() => {
const pose = new Pose({
locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});
pose.setOptions({
modelComplexity: 1,
smoothLandmarks: true,
minDetectionConfidence: 0.5,
minTrackingConfidence: 0.5,
});
pose.onResults(onResults);
if (webcamRef.current) {
const camera = new cam.Camera(webcamRef.current.video, {
onFrame: async () => {
await pose.send({ image: webcamRef.current.video });
},
width: 640,
height: 480,
});
camera.start();
}
}, []);
return (
<div className="relative">
<video ref={webcamRef} className="hidden" />
<canvas ref={canvasRef} className="rounded-lg shadow-2xl" />
<div className="absolute top-4 left-4 bg-black/50 text-white p-4">
Live Feedback: <span className="text-green-400">Keep your arm straight!</span>
</div>
</div>
);
};
Step 3: Real-time Correction with WebSockets 🔌
Why use WebSockets instead of just local state? Because in a clinical setting, a remote therapist might need to monitor the session live, or a backend AI model might need to compare the current movement against a "Golden Standard" trajectory stored in a database.
Server-side (Node.js):
const io = require('socket.io')(3001, { cors: { origin: "*" } });
io.on('connection', (socket) => {
console.log('Client connected 🥑');
socket.on('pose-update', (data) => {
const { angle, type } = data;
// Business Logic: Check against PT requirements
if (type === 'BICEP_STRETCH' && angle < 165) {
socket.emit('correction', {
message: "Straighten your elbow slightly more!",
severity: "low"
});
}
});
});
The "Official" Way 🥑
Building a basic pose estimator is easy; building a clinically accurate rehabilitation tool is hard. Handling occlusion (when a body part is hidden), varying lighting conditions, and precise depth estimation requires a more robust architectural approach.
For more production-ready examples, including how to handle 3D coordinate mapping and integrating with Electronic Health Records (EHR), check out the technical guides at WellAlly Tech Blog. They cover the nuances of deploying AI in regulated environments that we couldn't fit into this quick tutorial!
Conclusion 🏁
We’ve just turned a few lines of JavaScript and a webcam into a functional Physical Therapy Engine. By combining Mediapipe's pose estimation with WebSocket's real-time capabilities, we can provide users with immediate biofeedback, making home exercises safer and more effective.
What's next?
- Add a "Rep Counter" logic using a simple state machine.
- Integrate 3D landmarks for better depth perception.
- Store session data to visualize progress over time.
Drop a comment below if you want a part 2 on 3D pose reconstruction! Happy coding! 💻🔥
Top comments (0)