DEV Community

927tanmay
927tanmay

Posted on

I Built a 3D AI Voice Avatar That Runs Entirely in Your Browser — No Servers, No API Keys, No GPU Cloud

#ai #opensource #webgpu #webdev

Speech recognition, voice synthesis, and lip-synced 3D animation — running inside Web Workers on your desktop browser. Here's how.

TL;DR: I open-sourced a React component that drops a fully conversational, lip-syncing 3D avatar into any web app. Built primarily as a drop-in 3D frontend for your existing cloud LLMs (OpenAI, Claude, custom backends), it handles speech recognition (Whisper), voice synthesis (Kokoro TTS), and real-time ARKit facial blendshape animation in-browser via WebAssembly and WebGPU. It can also run completely offline with on-device models. One npm install, one component, done.

🔗 Live Demo · 📦 NPM Package · 🐙 GitHub

The Problem That Kept Bugging Me

Every time I explored building a conversational AI interface — the kind where a character actually talks back to you — I hit the same wall:

  • Cloud TTS APIs charge per character and add 200–500ms of round-trip latency.
  • WebSocket video streaming from GPU servers is fragile, expensive, and adds heavy infrastructure overhead.
  • Existing avatar libraries just give you a static 3D model. You still have to wire up speech, lip-sync, turn-taking, and microphone handling yourself (which takes months of integration).

I wanted something different: a single React component where I write <AiVoiceAvatar/> and it just... works. The avatar listens, thinks, speaks with a natural voice, and moves its mouth in perfect sync — serving as the perfect visual layer for your AI backend.

So, I built it.

What It Actually Does

react-ai-voice-avatar is a React + React Three Fiber component that orchestrates an entire voice conversation pipeline inside the browser:

🎤 Microphone → Whisper ASR → LLM Reasoning → Kokoro TTS → 3D Lip-Sync → 🔊 Speaker

Every heavy ML stage runs inside dedicated Web Workers. This ensures the main thread stays buttery smooth at 60 FPS while the neural networks execute quietly in the background.

The Two-Brain Architecture

While the package supports fully offline execution, it was built first and foremost to plug into your existing cloud infrastructure:

Brain Mode How It Works Best Used For
🧠 Connected Brain (Primary) Route transcribed speech to your existing backend (OpenAI, Claude, custom FastAPI, Vercel AI SDK, etc.) using the onSubmit prop. Production web apps, SaaS products, and enterprise AI assistants.
🔒 On-Device Brain (Offline) A 0.5B parameter LLM (Qwen 2.5) runs via WebGPU locally. No API keys or network calls required. Privacy-sensitive web apps, kiosks, or demo environments with spotty WiFi.

For the Connected Brain, the avatar handles all the complex frontend tasks: listening, transcribing, audio playing, and lip animation. Your backend just streams the text back:

// Connected Brain: Your backend handles the thinking
<AiVoiceAvatar
  avatarPreset="ananya"
  ttsVoice="af_heart"
  onSubmit={async (userSpeech) => {
    const res = await fetch('/api/chat', {
      method: 'POST',
      body: JSON.stringify({ prompt: userSpeech })
    });
    return res.body; // Streams natively!
  }}
/>
Enter fullscreen mode Exit fullscreen mode

That's it. The 3D character loads over CDN, the TTS and ASR models download once (cached in IndexedDB permanently), and your app gets an interactive 3D character connected directly to your existing AI API.

Under the Hood: How It Works

I spent weeks getting this architecture right. Here are the most interesting technical hurdles I had to cross.

1. Web Worker Isolation

Running Whisper or Kokoro on the main thread would freeze the web app instantly. To fix this, every model runs in isolation:

  • ML Pipeline Worker: Handles Whisper ASR transcription and (optionally) the local Qwen LLM. Streams LLM tokens back to the main thread as they're generated.
  • Kokoro TTS Worker: Converts text chunks into 24kHz Float32 audio. Runs the Kokoro-82M ONNX model using multi-threaded WASM or WebGPU acceleration.

These workers are pre-bundled using esbuild at build time and stringified into the package. You don't have to host worker files or configure Webpack. It works instantly with Vite, Next.js, or Create React App.

2. Real-Time Lip Synchronization (80 FPS)

This was the hardest part. The avatar needs to move its mouth naturally and in perfect sync with the audio stream. I built a dual-source blending approach:

  • Phoneme Timing Engine: We extract phoneme-level timing data from Kokoro and map it to 15 standard viseme shapes. This gives us predictive mouth shapes that lead the audio slightly, mimicking real human speech.
  • Audio Amplitude Fallback: A Web Audio API AnalyserNode reads real-time frequency data, providing a secondary signal blended for amplitude-driven jaw movement.
  • Procedural Facial Dynamics: The avatar has continuous idle micro-animations (randomized Poisson interval blinks, breathing, head drift) so it feels alive even when silent.

All of this targets the 52 standard Apple ARKit blendshapes, meaning any humanoid .glb model rigged with these morph targets works out of the box.

3. Safari & WASM Limits: The Invisible OOM Safety Net

Safari and restricted browser runtimes impose strict WebAssembly memory limits. Running heavier TTS models (~90MB) under tight WASM memory constraints can occasionally trigger Out of memory errors.

Instead of letting the component crash or fail silently, I built an automatic failover system. If Kokoro initialization encounters memory restrictions, the engine transparently switches to a lightweight MMS TTS model (~30MB). The voice quality drops slightly, but the avatar keeps talking without breaking the user session. No crashes, no frozen UI, and zero developer intervention required.

4. Hard Interruption

Real conversations are messy. If a user taps "Stop" mid-sentence, everything must halt instantly. I implemented a coordinated interrupt system across both Web Workers that clears all internal queues, aborts in-flight LLM generation via a sentinel error, and flushes the audio context.

The Developer Experience

Zero Configuration, Genuinely

I'm allergic to "zero config" tools that require 14 setup steps. Here is the actual install process:

npm install react-ai-voice-avatar three @react-three/fiber @react-three/drei
Enter fullscreen mode Exit fullscreen mode 
import { AiVoiceAvatar } from 'react-ai-voice-avatar';

// Inside your R3F Canvas:
<AiVoiceAvatar avatarPreset="ananya" />
Enter fullscreen mode Exit fullscreen mode

No Vite optimizeDeps overrides. No worker file hosting. The NPM footprint is only ~3.3 MB (including pre-bundled workers and viseme mapping). The heavy 3D models and neural network weights load on-demand over the network and cache in the browser.

Imperative Control

For times when you need programmatic control, the component exposes a clean Ref API:

const avatarRef = useRef<AiVoiceAvatarHandle>(null);

// Make the avatar speak programmatically
avatarRef.current?.speak("Hello! How can I help you?");

// Submit text as if the user spoke it
avatarRef.current?.sendText("What's the weather like?");

// Hard-interrupt mid-speech
avatarRef.current?.interrupt();
Enter fullscreen mode Exit fullscreen mode

What Surprised Me Building This

  • Numbers break TTS models: Kokoro chokes on symbols like % or $. I had to build a sanitizeForSpeech preprocessor that translates "95% complete" to "ninety-five percent complete" before it hits the model.
  • Web Worker Bundling is a Nightmare: Standard worker approaches break in library distribution. My solution (pre-compiling with esbuild and reconstructing as a Blob URL at runtime) isn't elegant, but it is universally compatible across all modern web bundlers.

What's Next

The core engine is stable and production-ready for web interfaces. Next on the roadmap:

  • Hindi & Indic Language Voices: The phoneme engine supports retroflex and aspirated consonants, and the visemeTable has Devanagari mappings. We just need to train the TTS voices!
  • Ready Player Me Integration: Official support for RPM avatars.
  • Conversation Memory: APIs like addContext() for injecting dynamic knowledge mid-conversation.

Try It Out

If you want to see the 80FPS lip-sync and streaming TTS integration in action, check out the demo:

🌐 Live Demo — react-ai-voice-avatar.vercel.app

The GitHub repo includes four complete example apps, ranging from a 30-line quickstart to a full hybrid-cloud OpenAI streaming integration.

If you build something with this, I'd genuinely love to see it. Open an issue, tag me, or drop a comment below, or connect with me on LinkedIn.

Top comments (0)