How I Built an AI Fitness Coach with Gemini, React and Three.js

How I Built an AI Fitness Coach with Gemini, React and Three.js

Building a website is one thing. Building a website that can actually interact with users is something completely different.

While working on my fitness project under ASForge, I wanted to create more than a traditional fitness website. I wanted to build an AI fitness coach that could communicate with users, answer questions, and feel like an actual character inside the website.

That's how Titan was created — an AI-powered fitness coach built with React, Three.js and Gemini.

The Idea Behind Titan

The goal was simple:

Create a fitness website where the AI coach is not just a chatbot sitting inside a box, but a 3D character that interacts with the user.

Titan was designed to provide an interactive experience where users can:

Ask fitness-related questions
Chat with the AI
Use voice interaction
Receive spoken responses
Switch between different languages
Interact with a 3D AI character

The project also gave me an opportunity to combine frontend development, 3D graphics and generative AI in one application.

Technologies I Used

The main technologies behind the project are:

React — User interface and application logic
Vite — Development environment and build tool
Three.js — 3D graphics
React Three Fiber — Using Three.js inside React
@react-three/drei — Useful Three.js helpers
Gemini API — AI responses
Web Speech API — Voice recognition and text-to-speech
JavaScript — Application logic and interactions

The combination of these technologies made it possible to create both the visual and conversational parts of Titan.

Creating the 3D AI Coach

The first challenge was getting a 3D character into the React application.

I used a GLB model for Titan and loaded it using useGLTF.

A simplified version looks like this:

import { useGLTF } from "@react-three/drei";

function Robot() {
const { scene } = useGLTF("/models/ai-coach.glb");

return (
object={scene}
scale={1.35}
position={[0.15, -0.85, 0]}
/>
);
}

I then placed the model inside a React Three Fiber Canvas.

<Canvas
camera={{
position: [0, 0.1, 3.8],
fov: 40,
}}

position={[5, 5, 5]}
intensity={2}
/>

Getting the correct camera position, model scale and character placement took quite a bit of experimentation.

A model can look perfect in a 3D viewer but appear too small, too large or partially outside the screen when placed inside a real webpage.

Adding Animation

The character should not feel like a static image.

The GLB model contains animations, so I used useAnimations to play them.

const { scene, animations } =
useGLTF("/models/ai-coach.glb");

const { actions } =
useAnimations(animations, group);

useEffect(() => {
if (!actions) return;

Object.values(actions).forEach((action) => {
action.reset().fadeIn(0.5).play();
});
}, [actions]);

This made Titan feel much more alive inside the website.

Connecting Titan to Gemini

The next part was giving Titan a brain.

Instead of hardcoding responses, I connected the application to the Gemini API.

The basic flow is:

User
↓
Titan UI
↓
React
↓
Backend API
↓
Gemini
↓
AI Response
↓
Titan

I kept the Gemini API key on the server side instead of exposing it directly in the React frontend.

The frontend sends the user's message to my backend:

const response = await fetch("/api/chat", {
method: "POST",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
message: text,
language: language.name,
}),
});

The backend then communicates with Gemini and returns the generated response.

This separation is important because API keys should not be placed directly inside frontend code.

Giving Titan a Voice

A normal chatbot only shows text.

I wanted Titan to actually speak.

For that, I used browser speech capabilities.

The browser's speech synthesis API can convert text into spoken audio:

const utterance = new SpeechSynthesisUtterance(text);

utterance.rate = 0.88;
utterance.pitch = 0.55;
utterance.volume = 1;

speechSynthesis.speak(utterance);

Now the interaction becomes:

User asks a question
↓
Gemini generates the answer
↓
Titan receives the answer
↓
Speech Synthesis speaks it

The exact available voices depend on the user's browser and operating system, so voice characteristics can vary between devices.

Voice Input

I also wanted users to be able to talk to Titan instead of typing everything.

The browser's Speech Recognition API can convert speech into text.

The basic concept is:

const recognition =
new SpeechRecognition();

recognition.lang = "en-IN";

recognition.onresult = (event) => {
const transcript =
event.results[0][0].transcript;

console.log(transcript);
};

recognition.start();

The recognized text can then be sent to Gemini just like a normal typed message.

Handling Multiple Languages

Another goal of Titan was making the experience more accessible to users who are more comfortable communicating in Indian languages.

The interface includes language selection for languages such as:

Hindi
English
Bengali
Telugu
Tamil
Marathi
Gujarati
Kannada
Malayalam
Punjabi
Odia
Assamese
Urdu
Nepali
Konkani
Maithili
Sanskrit
Kashmiri
Sindhi
Dogri
Manipuri
Santali

The selected language is passed along with the user's request so Titan can respond in the selected language.

However, speech recognition and available voices are dependent on browser and device support, so multilingual voice behavior can vary.

Making Titan Feel Like an Assistant

One of the most interesting parts of the project was designing the interaction.

I didn't want the AI to behave like a simple form:

Question → Answer

Instead, I wanted the experience to feel more like talking to an assistant.

The interaction became:

Speak
↓
Speech Recognition
↓
Gemini
↓
Titan Response
↓
Text-to-Speech
↓
User

I also worked on interruption handling so that when the user genuinely starts another question while Titan is speaking, the current speech can be stopped and the new request can be processed.

This part was more difficult than I initially expected because the microphone can sometimes pick up the computer's own speaker output.

Using microphone echo cancellation and noise suppression helped, but the behavior can still depend on the hardware and environment.

Designing the AI Interface

I wanted the AI controls to stay separate from the 3D character.

The basic structure became:

┌─────────────────────────────┐
│ Titan AI Coach │
│ │
│ Chat / Talk / Language │
│ │
│ Messages │
│ │
│ 3D Titan │
│ 🤖 │
└─────────────────────────────┘

The 3D character is rendered in its own layer while the chat and controls are placed above or behind it using CSS positioning and z-index.

This creates the impression that Titan is actually standing inside the interface rather than being another ordinary HTML element.

Problems I Faced

The project wasn't straightforward.

Some of the biggest challenges were:

1. 3D Model Positioning

The character was sometimes too small or moved outside the visible area.

I had to adjust:

Camera position
Field of view
Model scale
Model position
Canvas dimensions

1. Voice Interruption

Speech recognition could sometimes detect audio coming from Titan itself.

This created unexpected interruptions.

I experimented with:

Echo cancellation
Noise suppression
Automatic gain control
Microphone monitoring
Speech recognition confirmation

1. API Availability

AI APIs can occasionally return temporary errors or become unavailable during periods of high demand.

This made retry handling and proper error messages important.

1. Keeping API Keys Secure

I initially focused on making the AI work, but quickly realized that API credentials should never be exposed in frontend code.

Moving the API call to a backend was an important architectural improvement.

What I Learned

This project taught me that building an AI application isn't only about calling an AI API.

There are several layers involved:

UI/UX
+
3D Graphics
+
AI
+
Voice
+
Backend
+
Browser APIs

Each layer has its own challenges.

Three.js taught me about cameras, lighting, models and 3D positioning.

React taught me how to manage application state and UI.

Gemini introduced me to integrating generative AI into an actual application.

The Web Speech API showed me how powerful browser capabilities can be for voice interfaces.

And debugging all of these systems together taught me perhaps the most important lesson:

Building real products involves solving problems that don't appear in tutorials.

What's Next for Titan?

Titan is still a work in progress.

Some of the things I want to explore next include:

Better conversational memory
More personalized workout guidance
Improved voice interaction
Better multilingual support
Fitness progress tracking
More natural AI conversations
More advanced 3D character interactions
Better mobile experience

The long-term goal is to turn Titan from an AI feature into a complete AI fitness experience.

Final Thoughts

What started as a fitness website became an experiment in combining AI, 3D graphics, voice interaction and modern web development.

I'm building these projects under ASForge, where my goal is to explore and create practical AI-powered technology.

Titan is just one step in that journey.

If you're also building something with React, AI or Three.js, I'd love to hear about it.

Keep building. Keep learning. Keep experimenting. 🚀