Imagine asking an AI a complex question and hearing it think, pausing naturally as it formulates the next thought, and speaking the answer back to you in real-time. This isn't a sci-fi movie; it's the power of streaming text-to-speech (TTS).
In modern web development, specifically within the Next.js ecosystem, bridging the gap between Large Language Models (LLMs) and user audio perception creates a revolutionary user experience. By combining the Vercel AI SDK, React Server Components (RSC), and the native Web Speech API, we can build a "talking assistant" that feels alive.
This guide explores the architecture behind real-time audio synthesis and provides a complete, copy-pasteable code example to get you started.
The Architecture: From Tokens to Audio
To build a truly responsive assistant, we must abandon the "stop-and-wait" model. If we wait for the LLM to generate a full paragraph before converting it to audio, the latency ruins the immersion.
Instead, we implement a streaming audio synthesis pipeline. Here is the theoretical breakdown of how the data flows:
- The LLM (The Composer): Generates text incrementally, one token at a time.
- The Vercel AI SDK (The Conductor): Manages the stream, pushing tokens from the server to the client instantly.
- The Web Speech API (The Instrumentalist): The browser's native synthesizer receives these tokens and converts them into sound waves immediately.
The "Orchestra" Analogy
Think of this architecture like a live orchestra performance:
- The LLM is the composer writing the score note-by-note.
- The SDK is the conductor reading the notes as they appear and cueing the musicians instantly.
- The Web Speech API is the musician playing the instrument in real-time.
The goal is Zero-Latency Auditory Feedback. The user hears the AI speaking milliseconds after the first text token is generated.
The Challenge: Streaming vs. Synthesis
The Web Speech API (window.speechSynthesis) is designed to speak complete sentences. However, LLMs stream tokens (often sub-word chunks like "ing" or "pre"). If you feed every tiny token to the synthesizer as a separate utterance, the result is a robotic, stuttering mess.
To solve this, we need a Buffering Strategy:
- Lookahead Buffering: Accumulate tokens in a buffer.
- Boundary Detection: Flush the buffer to the synthesizer when a natural break is detected (punctuation like
.,?, or a space). - Timeout Fallback: If the buffer gets too large or too old (e.g., 200ms), flush it to prevent latency buildup.
This ensures the synthesizer receives intelligible phrases rather than disjointed syllables.
Implementation: The Code
Below is a self-contained Next.js Client Component. It simulates a server stream (using a mock async function) and plays the audio in real-time using the native browser API.
TalkingAssistant.tsx
'use client';
import React, { useState, useEffect, useRef } from 'react';
// Define the shape of the streamable text token
type StreamToken = {
type: 'text';
content: string;
};
export default function TalkingAssistant() {
// State for the visual UI
const [displayText, setDisplayText] = useState<string>('');
// State to track audio status
const [isSpeaking, setIsSpeaking] = useState<boolean>(false);
// Ref to buffer text tokens for smoother audio
const bufferRef = useRef<string>('');
/**
* 1. SIMULATE SERVER STREAM
* In a real app, replace this with `useChat` or `useStreamableUI` from Vercel AI SDK.
*/
const simulateStream = async (): Promise<void> => {
// Reset previous state
setDisplayText('');
bufferRef.current = '';
window.speechSynthesis.cancel(); // Clear any existing queue
const mockTokens = [
'Hello, ', 'developer! ', 'I am ', 'your AI assistant. ',
'I am processing ', 'your request ', 'right now. ',
'This ', 'audio ', 'is ', 'streaming ', 'in real-time.'
];
for (const token of mockTokens) {
// Simulate network latency
await new Promise(resolve => setTimeout(resolve, 300));
// 2. Update Visual State
setDisplayText(prev => prev + token);
// 3. Buffer and Speak Audio
handleAudioStream(token);
}
};
/**
* 2. AUDIO SYNTHESIS LOGIC
* Handles buffering and queuing to the Web Speech API.
*/
const handleAudioStream = (token: string) => {
if (!window.speechSynthesis) {
console.error('Web Speech API not supported.');
return;
}
// Add token to buffer
bufferRef.current += token;
// Check for natural break points (punctuation or spaces)
// In a production app, you might want a more robust regex or a timer.
const hasBreak = /[.!?]\s|,\s|\s$/.test(bufferRef.current);
if (hasBreak) {
speakText(bufferRef.current);
bufferRef.current = ''; // Clear buffer
}
};
/**
* 3. THE SPEAKER
* Creates an utterance and adds it to the browser's queue.
*/
const speakText = (text: string) => {
const utterance = new SpeechSynthesisUtterance(text);
// Optional: Select a specific voice
const voices = window.speechSynthesis.getVoices();
const preferredVoice = voices.find(v => v.lang === 'en-US');
if (preferredVoice) utterance.voice = preferredVoice;
// Event Listeners for UI Sync
utterance.onstart = () => setIsSpeaking(true);
utterance.onend = () => {
// Only set to idle if the queue is empty
if (window.speechSynthesis.pending === 0 && window.speechSynthesis.speaking === false) {
setIsSpeaking(false);
}
};
window.speechSynthesis.speak(utterance);
};
// Controls
const pauseSpeech = () => {
window.speechSynthesis.pause();
setIsSpeaking(false);
};
const resumeSpeech = () => {
window.speechSynthesis.resume();
setIsSpeaking(true);
};
const stopSpeech = () => {
window.speechSynthesis.cancel();
setIsSpeaking(false);
bufferRef.current = '';
};
// Cleanup on unmount
useEffect(() => {
return () => window.speechSynthesis.cancel();
}, []);
return (
<div style={{ padding: '20px', maxWidth: '600px', margin: '0 auto', fontFamily: 'system-ui' }}>
<h2>AI Talking Assistant</h2>
{/* Visual Output */}
<div style={{
border: '1px solid #ddd',
padding: '15px',
minHeight: '80px',
marginBottom: '20px',
borderRadius: '8px',
background: '#f9f9f9'
}}>
<p style={{ color: '#333' }}>
{displayText || <span style={{ color: '#999' }}>Click "Start Stream" to begin...</span>}
</p>
</div>
{/* Controls */}
<div style={{ display: 'flex', gap: '10px', flexWrap: 'wrap' }}>
<button
onClick={simulateStream}
disabled={isSpeaking}
style={{ padding: '10px', background: '#0070f3', color: 'white', border: 'none', borderRadius: '4px', cursor: 'pointer' }}
>
Start Stream
</button>
<button
onClick={pauseSpeech}
disabled={!isSpeaking}
style={{ padding: '10px', background: '#f59e0b', color: 'white', border: 'none', borderRadius: '4px', cursor: 'pointer' }}
>
Pause
</button>
<button
onClick={resumeSpeech}
style={{ padding: '10px', background: '#10b981', color: 'white', border: 'none', borderRadius: '4px', cursor: 'pointer' }}
>
Resume
</button>
<button
onClick={stopSpeech}
style={{ padding: '10px', background: '#ef4444', color: 'white', border: 'none', borderRadius: '4px', cursor: 'pointer' }}
>
Stop
</button>
</div>
<div style={{ marginTop: '15px', fontSize: '0.85rem', color: '#666' }}>
Status: {isSpeaking ? 'Speaking...' : 'Idle'}
</div>
</div>
);
}
Key Technical Concepts Explained
1. The 'use client' Directive
The Web Speech API (window.speechSynthesis) is a browser-only API. It does not exist in the Node.js environment where server components run. Therefore, any component interacting with audio must be marked as a Client Component.
2. The SpeechSynthesisUtterance Queue
The browser handles a queue of utterances automatically. When we call window.speechSynthesis.speak(utterance), it is added to the queue.
- The Pitfall: If you fire this for every single token without buffering, you will create a cacophony of overlapping syllables.
- The Solution: Our
handleAudioStreamfunction acts as a gatekeeper. It waits for a "natural break" (like a space or punctuation) before releasing the text to the synthesizer.
3. Voice Loading Race Conditions
A common issue with the Web Speech API is that window.speechSynthesis.getVoices() returns an empty array initially. Voices load asynchronously. In a production app, you should listen for the onvoiceschanged event to ensure the voice is available before attempting to speak.
Common Pitfalls to Avoid
- Auto-play on Mount: Mobile browsers (iOS Safari) strictly block audio from playing without a direct user interaction (like a click). Never call
speak()inside auseEffectwithout user input. - Token Fragmentation: If your LLM streams character-by-character, your buffer logic must be smart enough to group them. A 200ms timer is a good fallback to group rapid-fire tokens.
- Memory Leaks: Always call
window.speechSynthesis.cancel()in your component's cleanup function (useEffectreturn) to stop audio and clear the queue when the user navigates away.
Conclusion
By decoupling the generation of text from the synthesis of audio, we can create highly responsive, accessible, and immersive web applications. The combination of Next.js RSC for security, Vercel AI SDK for real-time data streaming, and the native Web Speech API for client-side synthesis provides a powerful, lightweight stack for building the next generation of voice interfaces.
The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the book The Modern Stack. Building Generative UI with Next.js, Vercel AI SDK, and React Server Components
Take a look at my eBooks
JavaScript & TypeScript
Foundations
OpenAI API, Zod, and LangChain.js
The Modern Stack
Building Generative UI with Next.js, Vercel AI SDK, and React Server Components.
Master Your Data
Production RAG, Vector Databases, and Enterprise Search.
Autonomous Agents
Building Multi-Agent Systems and Workflows with LangGraph.js
The Edge of AI
Local LLMs (Ollama), Transformers.js, WebGPU, and Performance Optimization
The AI-Ready SaaS Boilerplate. Auth, Database with Vector Support, and Payment Stack
Auth, Database with Vector Support, and Payment Stack.
Backend for Frontend & Intelligent APIs. tRPC, Edge Functions, and LLM Data Transformation
tRPC, Edge Functions, and LLM Data Transformation.
The Monetization Engine. Stripe, Smart Dunning, and AI Customer Support Agents
Stripe, Smart Dunning, and AI Customer Support Agents.
AI-Driven Growth Engineering. Programmatic SEO with GPT-4, Content Automation, and Analytics.
Programmatic SEO with GPT-4, Content Automation, and Analytics.
No More Localhost. Mastering Docker, Linux, and Containerization for JS & AI Apps
Mastering Docker, Linux, and Containerization for JS & AI Apps.
The Perfect Pipeline. Advanced CI/CD with GitHub Actions, Automated Testing, and AI Code Reviews
Advanced CI/CD with GitHub Actions, Automated Testing, and AI Code Reviews.
Kubernetes & Orchestration. Deploying Scalable Node.js & AI Clusters without Tears
Deploying Scalable Node.js & AI Clusters without Tears.
React Native for Web Developers
From Next.js to Expo, NativeWind, and Universal App
Offline AI & Local LLMs. Running Llama 3 and Vector Search directly on the Smartphone
Running Llama 3 and Vector Search directly on the Smartphone.
App Store Engineering. CI/CD for Mobile (EAS), OTA Updates, and AI-Driven App Store Optimization
CI/CD for Mobile (EAS), OTA Updates, and AI-Driven App Store Optimization.
The TypeScript-First Architect. Building Robust Applications with Effect, Zod, and Drizzle
Building Robust Applications with Effect, Zod, and Drizzle.
The Native Era. Modern Node.js, Bun & Deno without Bundlers or Transpilers
Modern Node.js, Bun & Deno without Bundlers or Transpilers.
Local-First Systems in TypeScript. Collaborative & Offline-Ready Web Apps with CRDTs and WASM DBs
Collaborative & Offline-Ready Web Apps with CRDTs and WASM DBs.
TypeScript Metaprogramming. Advanced Type Gymnastics, Modern Decorators, and Compiler Internals
Advanced Type Gymnastics, Modern Decorators, and Compiler Internals.
Standardizing Tool Integration, Vision-Driven Browser Automation, and Agent Governance in TypeScript.
Generative Media & Visual Workflow Engines. Node-Based AI Canvases, Real-Time Media Streaming Pipelines, and WebGPU Processing in TypeScript
Node-Based AI Canvases, Real-Time Media Streaming Pipelines, and WebGPU Processing in TypeScript.
Neuro-Symbolic AI & Knowledge Graphs. Deterministic Solvers, GraphDBs, Ontologies, and Zero-Hallucination Architectures
Deterministic Solvers, GraphDBs, Ontologies, and Zero-Hallucination Architectures in TypeScript.
Event-Driven Architecture & DDD in TypeScript. Event Sourcing, CQRS, and Microservices at Scale
Event Sourcing, CQRS, and Microservices at Scale.
Building Desktop Apps & Developer Tools with Tauri 2.0, Rust, and TypeScript
Cross-platform desktop tools with Tauri, Rust, and TypeScript.
FinTech Architecture in TypeScript. Precision Math, Double-Entry Ledgers, and High-Reliability Payment Pipelines
Precision Math, Double-Entry Ledgers, and High-Reliability Payment Pipelines.
Hardened TypeScript. Passkeys, Supply Chain Defense, and Zero-Trust Architectures
Passkeys, Supply Chain Defense, and Zero-Trust Architectures.
Spatial Web Development. Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript
Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript.
Multiple-choice test book for: Foundations (Volume 1)
Spatial Web Development. Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript
Building Interactive 3D and WebXR Experiences with React Three Fiber & TypeScript.
Jev: The Definitive Guide to System One AI
Building Sub-100ms Decision Engines, Calibrated Guardrails, and Two-Speed Architectures with Jev and Generative LLMs
Python
Data Structures and the Standard Library
Web Development with Python
Building backend services and dynamic websites with a framework like Flask
Advanced Python & AI Integration
Deep dive into OOP, decorators, asyncio, and orchestrating LLMs with LangChain.
Gemini 3 Python Programming - The Complete Guide
Agents, Veo 3.1, Lyria, Nano Banana/Pro, Function Calling, Grounding, Computer Use and Robotics
AI Autonomous Agents with Python Programming
Master LangGraph, CrewAI, and RAG to Build Self-Correcting Swarms and Autonomous Digital Workers
Finance & AI Trading with Python Programming
Master Algorithmic Trading, Financial NLP, and Vectorized Backtesting to Build Autonomous 'News + Math' Strategies
Cloud-Native Python, DevOps & LLMOps. Containerization, Kubernetes, and Serving AI Models at Scale
From Docker and Kubernetes to Serving LLMs with Pulumi
Defensive Cybersecurity with Python Programming
A Practical Guide to System Monitoring, Network Defense, and Automated Security Hardening
Data Science & Analytics with Python Programming
Neural Networks & Deep Learning with Python Programming
Architecting Neuro-Symbolic Agents with Python Programming
Integrating LLMs, Wolfram Alpha, IBM Watson and Open Source Stacks for Near-Zero Hallucination Systems
Bioinformatics & AI with Python Programming
Master Genomic Data Science, Protein Folding with AlphaFold, and AI-Driven Drug Discovery
Geospatial AI (GeoAI) with Python Programming
Building Autonomous GIS Agents, Deep Learning Models, and Interactive Dashboards
Astrophysics & AI with Python Programming
Building Research Agents for Astronomy, Cosmology, and SETI
Open-Source LLMs & Local Fine-Tuning
Mastering LoRA, vLLM, Ollama, and Custom SLMs
Unsloth: Efficient Fine-Tuning for Large Language Models
Methods and Workflows for Fine-Tuning and Deploying Large Language Models on Limited Hardware
Hermes Agent: The Self-Evolving AI Workforce
Architecting Autonomous Systems that Learn, Remember, and Grow.
Frontier AI Safety, Mechanistic Interpretability & Alignment Engineering
Inspecting Neural Circuits, Steering Vectors, Autonomous Capability Evals, and Scalable Oversight for Superintelligent Systems.
C# / .NET
Get all the Ten C# & AI volumes at a discounted price, or choose an ebook:
The Foundations
Syntax, Type System, and Logic for Modern Developers.
Advanced OOP & AI Data Structures
Modeling Complex Systems and Tensors.
Data Manipulation, LINQ & Vectors
From Collections to AI Embeddings
Asynchronous AI Pipelines
Async/Await, Parallelism, and Streaming LLM Responses.
Building AI Web APIs with ASP
NET Core. Serving Models and Chat Endpoints
Intelligent Data Access with EF Core
Vector Databases, RAG, and Memory Storage.
Cloud-Native AI & Microservices
Containerizing Agents and Scaling Inference.
The Core of AI Engineering: Microsoft Semantic Kernel & Agentic Patterns
Edge AI & Local Inference
Running LLMs (Llama/Phi) locally with C# and ONNX.
High-Performance C# for AI
Span, SIMD, and Optimizing Token Processing
Full Stack AI with Blazor. Building Interactive Copilots and WASM AI
Building Interactive Copilots and WASM AI.
Enterprise AI Integration & Process Automation. Connecting LLMs to legacy systems, internal APIs, and real-world business processes
Connecting LLMs to legacy systems, internal APIs, and real-world business processes.
AI for Game Development & Interactive Simulation. Using LLMs and generative AI to create dynamic worlds and intelligent characters in Unity
Using LLMs and generative AI to create dynamic worlds and intelligent characters in Unity.
Swift & Apple Platform
Core ML & Vision Framework
On-device image classification, object detection, and custom model integration with Core ML and Vision.
Apple Intelligence & Foundation Models
Building apps with Apple's on-device LLM APIs, Writing Tools, and the Apple Intelligence framework
Natural Language & Speech
NLP, sentiment analysis, text classification, and Speech-to-Text with Apple's Natural Language and Speech frameworks.
SwiftUI for AI Apps
Building reactive, intelligent interfaces that respond to model outputs, stream tokens, and visualize AI predictions in real time
Create ML Studio
Training custom models without Python: tabular, image, sound, and motion classifiers using Create ML in Swift.
MLX Swift & Local LLMs. Deep dive into Apple's MLX framework for high-performance machine learning.
Building custom inference engines, fine-tuning local models (LoRA), and leveraging Unified Memory directly from Swift.
visionOS & Spatial AI with Swift
Swift + OpenAI & LangChain
Integrating external LLM APIs, RAG pipelines, and agentic workflows in iOS and macOS apps
CoreData, CloudKit & Vector Search
Shipping AI Apps to the App Store
Kotlin & Android
On-Device GenAI with Android Kotlin
Mastering Gemini Nano, AICore, and local LLM deployment using MediaPipe and Custom TFLite models
Edge AI Performance with Android Kotlin
Optimizing hardware acceleration via NPU, GPU, and DSP. Advanced quantization and model pruning
Android AI Agents
Building autonomous apps that use Tool Calling, Function Injection, and Screen Awareness to perform tasks for the user
Rust
Rust Advanced Memory Patterns for AI
Mastering Lifetimes, Smart Pointers, and custom allocators for managing large models and datasets
Extending Python with Rust. Creating high-performance Python modules with PyO3.
Creating high-performance Python modules with PyO3 for data processing, tokenization, and inference, replacing slow Python code.
Top comments (0)