Beck_Moulton

Posted on Feb 20

Save Your Spine: Building a Real-Time AI Posture Coach with MediaPipe and Electron

#ai #machinelearning #webdev #javascript

Are you reading this while hunched over your keyboard like a gargoyle? Don't worry, you’re not alone. As developers, we spend thousands of hours in front of screens, often sacrificing our spinal health for that one elusive bug fix.

In this tutorial, we are going to build "SpineGuard"—a cross-platform desktop application that uses Computer Vision, MediaPipe, and TensorFlow.js to monitor your posture in real-time. We'll leverage AI Posture Detection and Desktop Ergonomics logic to trigger alerts whenever you start slouching.

By the end of this guide, you'll have a functional app that calculates your neck angle and lumbar curvature, sending "sit up straight!" notifications via WebSocket.

The Architecture

Before we dive into the code, let's look at how the data flows from your webcam to your desktop notifications. We use MediaPipe's Pose Landmarker to identify 33 body keypoints, focusing specifically on the ears, shoulders, and hips.

graph TD
    A[Webcam Feed] --> B[MediaPipe Pose Engine]
    B --> C{Keypoint Detection}
    C -->|Coordinates| D[Ergonomic Logic Engine]
    D --> E[Calculate Neck & Spine Angles]
    E --> F{Threshold Exceeded?}
    F -->|Yes| G[WebSocket Trigger]
    G --> H[Electron Main Process]
    H --> I[System Desktop Notification]
    F -->|No| J[Continue Monitoring]

Prerequisites

To follow along, ensure you have the following in your tech_stack:

MediaPipe: For high-performance pose estimation.
TensorFlow.js: The backbone for running ML models in the browser/Electron.
Electron: To package our app for Windows/macOS/Linux.
WebSocket: To decouple the vision processing from the UI alerts.

Step 1: Setting up the Vision Engine (MediaPipe)

First, we need to initialize the Pose detection. We’ll use the @mediapipe/pose library because it’s incredibly lightweight and runs directly in the browser thread or an Electron renderer process.

// vision-engine.js
import { Pose } from "@mediapipe/pose";

const pose = new Pose({
  locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose/${file}`,
});

pose.setOptions({
  modelComplexity: 1,
  smoothLandmarks: true,
  minDetectionConfidence: 0.5,
  minTrackingConfidence: 0.5,
});

// This function processes the camera frame
export const detectPosture = async (videoElement) => {
  await pose.send({ image: videoElement });
};

pose.onResults((results) => {
  if (results.poseLandmarks) {
    analyzeErgonomics(results.poseLandmarks);
  }
});

Step 2: The Math of "Slouching"

The secret sauce of an AI posture coach is calculating the angle between key body parts. To detect a "Forward Head" position, we calculate the angle between the Ear, the Shoulder, and a vertical line.

function calculateAngle(a, b, c) {
    const radians = Math.atan2(c.y - b.y, c.x - b.x) - Math.atan2(a.y - b.y, a.x - b.x);
    let angle = Math.abs((radians * 180.0) / Math.PI);
    if (angle > 180.0) angle = 360 - angle;
    return angle;
}

function analyzeErgonomics(landmarks) {
    // Landmark 7: Left Ear, Landmark 11: Left Shoulder
    const ear = landmarks[7];
    const shoulder = landmarks[11];

    // We create a virtual point directly above the shoulder to measure vertical tilt
    const verticalPoint = { x: shoulder.x, y: shoulder.y - 0.5 };

    const neckAngle = calculateAngle(ear, shoulder, verticalPoint);

    // If the angle is too large, you're leaning too far forward!
    if (neckAngle > 25) {
        sendWarning("Hey! Sit up straight! 🦴");
    }
}

Step 3: Triggering Desktop Alerts via Electron & WebSocket

We want our app to run in the background. When the "Vision Engine" detects poor posture, it sends a message via WebSocket to the Electron Main process to trigger a native notification.

The WebSocket Bridge

// Inside the Renderer Process
const socket = new WebSocket('ws://localhost:8080');

function sendWarning(message) {
    socket.send(JSON.stringify({ type: 'POSTURE_ALARM', message }));
}

The Electron Main Process

// main.js
const { Notification } = require('electron');
const { WebSocketServer } = require('ws');

const wss = new WebSocketServer({ port: 8080 });

wss.on('connection', (ws) => {
  ws.on('message', (data) => {
    const payload = JSON.parse(data);
    if (payload.type === 'POSTURE_ALARM') {
      new Notification({ 
          title: 'Posture Alert! 🚨', 
          body: payload.message 
      }).show();
    }
  });
});

Level Up: Production Patterns

While this setup works for a MVP, production-grade AI vision tools require more robust state management and optimized rendering cycles to prevent CPU spikes.

Pro Tip: For advanced patterns on optimizing TensorFlow.js memory management or implementing multi-person tracking for office environments, I highly recommend checking out the deep dives over at WellAlly Blog. They have some fantastic resources on scaling Computer Vision apps in real-world production scenarios.

Conclusion

Building a posture corrector isn't just a cool weekend project—it's a tool that adds years of health to your career. By combining MediaPipe's powerful landmark detection with Electron's desktop capabilities, we've created a personal health assistant that lives right in your taskbar.

What's next?

Add a "Squat Counter" for those quick 5-minute desk breaks.
Integrate a "Daily Ergonomic Score" dashboard.
Use a more robust backend for data logging.

Have you built something similar or have questions about the math behind the angles? Let's chat in the comments! 👇

DEV Community