Building RepWise: A Live AI Gym Coach That Rescues Your Workout

#ai #programming #gemini

Building RepWise: A Live AI Gym Coach That Rescues Your Workout

This article was created for the purposes of entering the Gemini Live Agent Challenge.

The Problem

Most people don't quit workouts because they're lazy. They quit because something breaks.

The bench is taken. Energy crashes mid-session. A movement hurts. You don't recognize the exercise your plan says to do next. You only have 15 minutes left instead of 30.

These are real friction points that happen in every gym, every day. And no fitness app handles them. They generate a plan, hand it to you, and disappear.

I wanted to build something that stays with you — and adapts when things go wrong.

What RepWise Does

RepWise is a beginner-friendly, friction-aware live gym companion that runs in the browser.

Here's the flow:

Quick Setup — You tell RepWise your level (beginner/intermediate), goal (build muscle, lose fat, quick session), available time, equipment, energy level, and context (crowded gym, low confidence, soreness).
Mission Generation — Gemini generates a personalized workout plan with exercises, sets, reps, warmup, and coaching cues tailored to your inputs.
Live Coaching — Your phone camera tracks your movement using MediaPipe pose detection. RepWise counts reps, tracks sets with rest intervals, detects form issues (leaning forward, insufficient depth), and provides real-time
coaching cues from Gemini.
Workout Rescue — This is the core differentiator. When friction hits, you tap Rescue and select a reason: Equipment Busy, Low Energy, Short on Time, Discomfort, Don't Know This Exercise, or Easier Please. RepWise instantly
rewrites the remaining workout while preserving your original goal.
Mission Complete — A grounded summary showing actual exercises performed, sets completed, form score, and a Gemini-generated coaching debrief with next-step recommendations.

Why Workout Rescue Matters

Rep counting and form detection are useful, but they're expected. Every fitness AI does some version of that.

Workout Rescue solves a different problem: keeping the session alive when real-world conditions change. It's the moment where the AI stops being a tracker and starts being a coach — one that makes a decision under changing
constraints, explains it, and keeps you moving.

That's the kind of agency the Gemini Live Agent Challenge is about.

How Gemini Is Used

RepWise uses Gemini in three distinct ways:

Gemini Live API (via ADK bidi-streaming) — Real-time coaching during the workout session. Pose events and session context are streamed to Gemini, which responds with contextual coaching cues via WebSocket.
Gemini 2.5 Flash (structured planning) — Workout generation, warmup creation, and exercise adaptation. The model returns structured JSON that the backend parses into exercise plans with sets, reps, and cues.
Gemini 2.5 Flash (reasoning) — Post-workout debrief generation and workout rescue logic. When rescue is triggered, Gemini evaluates the remaining plan against the new constraint and produces a coherent adaptation.

How Google Cloud Is Used

Cloud Run — The FastAPI backend is containerized and deployed to Cloud Run. Handles WebSocket connections for live coaching, REST APIs for session management, and Gemini API calls.
Cloud Firestore — Stores workout sessions with full rep event data, form scores, coaching summaries, and anonymous user profiles for continuity across sessions.
Automated deployment — A single deploy.sh script enables, provisions, and deploys the entire stack.

Tech Stack

Layer	Technology
Frontend	JavaScript, HTML, CSS
Pose Detection	MediaPipe Tasks Vision (in-browser)
Backend	Python, FastAPI
AI Coaching	Gemini Live API + ADK
AI Planning	Gemini 2.5 Flash
Deployment	Google Cloud Run, Docker
Persistence	Google Cloud Firestore
Testing	Pytest (111 tests)

What I Learned

MediaPipe in the browser is surprisingly capable. Running pose detection at 10fps with joint angle computation and rep counting — all client-side — keeps the architecture simple and latency low.

Semantic events beat raw data. Sending Gemini a structured pose event like {"exercise": "squat", "rep": 8, "form_issue": "leaning_forward"} produces much better coaching than sending raw landmark coordinates.

Fallbacks are non-negotiable for demos. Every Gemini call has a deterministic fallback. Workout generation, rescue, warmup, debrief — all have local logic that activates if the API is slow or unavailable. The demo never breaks.

Beginner/intermediate branching changes everything. The same app feels meaningfully different for a beginner (3 simple exercises, reassuring cues, longer rest) versus an intermediate user (more exercises, tighter language,
performance framing). Small backend logic, big UX impact.