Goodbye Bad Posture: Building a Real-Time Scoliosis Risk Assessment with MediaPipe and Three.js 🚀

#webdev #javascript #machinelearning #computervision

We’ve all been there—slouching over our desks for eight hours until our spines resemble a question mark. But for those undergoing physical therapy or monitoring scoliosis, getting professional feedback usually means an expensive trip to the clinic. What if we could bring that clinical expertise to your browser using Computer Vision and Real-time Pose Estimation?

In this tutorial, we are building a "Digital Spine Assistant." By leveraging MediaPipe Pose and WebRTC streaming, we’ll calculate a simulated Cobb Angle (the gold standard for scoliosis measurement) and visualize it in a 3D space using Three.js. This approach brings Physical Therapy AI directly to the edge, ensuring privacy and low-latency feedback.

🏗 The Architecture

To achieve real-time feedback, we need a pipeline that handles video ingestion, keypoint extraction, geometric processing, and 3D rendering.

graph TD
    A[User Webcam] -->|WebRTC Stream| B(MediaPipe Pose Engine)
    B -->|3D Landmarks| C{Geometry Engine}
    C -->|Calculate Shoulder/Hip Tilt| D[Cobb Angle Simulation]
    C -->|Coordinate Mapping| E[Three.js 3D Avatar]
    D -->|Correction Feedback| F[UI Dashboard]
    E -->|Visual Overlay| F
    F -->|Real-time Alert| A

🛠 Prerequisites

Before we dive into the math, ensure you have these tools in your belt:

MediaPipe: For high-fidelity body tracking.
Three.js: To render our 3D skeletal twin.
NumPy: To handle the vector math for spinal alignment.
WebRTC: To ensure smooth, low-latency video input.

👨‍💻 Step 1: Capturing the Spine with MediaPipe

MediaPipe provides 33 3D landmarks. For scoliosis assessment, we care most about the alignment of the shoulders (Landmarks 11, 12), hips (23, 24), and the midpoint "spine" line.

// Initialize MediaPipe Pose
const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 2, // High complexity for better 3D depth
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5
});

// Capture landmarks via WebRTC
pose.onResults((results) => {
  if (!results.poseLandmarks) return;

  // Extract landmarks for Cobb calculation
  const leftShoulder = results.poseLandmarks[11];
  const rightShoulder = results.poseLandmarks[12];
  const leftHip = results.poseLandmarks[23];
  const rightHip = results.poseLandmarks[24];

  analyzeSpineAlignment(leftShoulder, rightShoulder, leftHip, rightHip);
  update3DModel(results.poseWorldLandmarks); 
});

📐 Step 2: The Geometry of Correction

Clinical Cobb angles are measured via X-rays, but we can simulate the Coronal Plane Alignment by measuring the angle between the shoulder-line vector and the hip-line vector relative to the gravity axis.

import numpy as np

def calculate_simulated_cobb(shoulder_l, shoulder_r, hip_l, hip_r):
    """
    Calculates the deviation angle between the shoulder axis and the pelvis.
    Values > 10° often trigger a 'Risk' flag.
    """
    # Create vectors
    shoulder_vector = np.array([shoulder_r.x - shoulder_l.x, shoulder_r.y - shoulder_l.y])
    hip_vector = np.array([hip_r.x - hip_l.x, hip_r.y - hip_l.y])

    # Calculate angle relative to the horizontal (1, 0)
    unit_vector = np.array([1, 0])

    def get_angle(v):
        dot_product = np.dot(v, unit_vector)
        magnitude = np.linalg.norm(v)
        return np.degrees(np.arccos(dot_product / magnitude))

    shoulder_angle = get_angle(shoulder_vector)
    hip_angle = get_angle(hip_vector)

    # The 'Risk Factor' is the delta between these two tilts
    imbalance_score = abs(shoulder_angle - hip_angle)
    return imbalance_score

🎨 Step 3: 3D Visualization with Three.js

Numerical data is boring. We want to show the user their "Digital Twin." We use Three.js to map the poseWorldLandmarks (which are in meters) to a 3D skeletal mesh.

const scene = new THREE.Scene();
const geometry = new THREE.SphereGeometry(0.05);
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });

// Mapping function
function update3DModel(worldLandmarks) {
  worldLandmarks.forEach((landmark, index) => {
    // MediaPipe uses a right-handed coordinate system
    // We map X, Y, Z to our Three.js scene
    let sphere = scene.getObjectByName(`joint_${index}`);
    if (sphere) {
      sphere.position.set(landmark.x, -landmark.y, -landmark.z);

      // Change color to red if the angle is high!
      if (globalImbalanceScore > 10) sphere.material.color.set(0xff0000);
    }
  });
}

🏥 The "Official" Way: Production Patterns

While this DIY approach is a great start, building a production-ready medical AI involves HIPAA compliance, filtering sensor noise (Kalman Filters), and more robust biomechanical models.

For a deeper dive into production-grade healthcare AI patterns, advanced pose correction algorithms, and enterprise-scale vision deployments, I highly recommend checking out the engineering deep-dives at WellAlly Blog. They cover the bridge between "it works on my machine" and "it works for thousands of patients." 🥑

💡 Conclusion

By combining MediaPipe for edge-based inference and Three.js for interactive feedback, we've created a tool that can democratize spinal health monitoring. No servers, no expensive hardware—just a browser and a dream of better posture.

What's next?

Add a "Plumb Line" overlay to check vertical center of gravity.
Integrate a "Session Timer" to track posture over time.
Implement a voice-assistant to say "Sit up straight!" when you slouch. 🤖

Are you working on AI-powered rehab? Let’s chat in the comments! 👇

DEV Community