Exploring the Voice AI Stack

The voice AI landscape becomes easier to understand when you map it as a stack.

I reached that conclusion after building one working voice-agent stack, then exploring the alternatives. LiveKit, Deepgram, ElevenLabs, and Vapi often appear in the same landscape, but each controls a different part of the system. One provides real-time media and an agent framework. Another began with speech recognition. Another became known for generated voices. Another packages orchestration, telephony, models, and operations.

The practical question is: Which layers do I want to own?

Why Voice AI Is Gaining Traction

Voice already has distribution. It runs through every telephone, browser, mobile device, car, and contact center. The new opportunity comes from changing what happens after someone starts speaking.

Customer service is an early market for voice AI because many customers still choose the phone when they need help. A 2025 Dynata survey of 1,000 US consumers commissioned by PolyAI found that 65% preferred a phone call as their primary way to contact retail and travel brands. In the same survey, 71% said they would speak with an intelligent voice assistant if it could accurately complete the task.

That last condition matters. A 2026 Gartner survey of 3,566 customers found that 87% expect companies using generative AI for customer service to provide access to a human agent. Customers want faster resolution while retaining a human escape hatch.

Three changes are expanding what voice systems can do:

1. The conversation is less scripted. Native audio models such as Amazon Nova 2 Sonic, OpenAI Realtime, and Gemini Live combine live audio, reasoning, interruptions, and tool use inside a persistent session.
2. The infrastructure is easier to assemble. LiveKit Agents and Pipecat provide programmable frameworks for media, models, and tools. Vapi, Retell, and the newer agent platforms from speech providers package more of that work into managed services.
3. Service organizations are increasing technology investment. Gartner predicts that more than half of customer-service organizations will double technology spending by 2028. Its October 2025 survey of 321 service leaders found that only 20% had reduced agent headcount because of AI. Investment is rising, but full labor replacement is not the current baseline.

The broader AI adoption curve adds momentum. McKinsey's 2025 global survey of 1,993 participants found that 88% reported regular AI use in at least one business function. Twenty-three percent reported scaling an agentic AI system, while another 39% were experimenting.

The economics still need careful measurement. Voice agents can answer concurrently, operate outside business hours, and absorb call spikes without matching staffing growth. That does not guarantee lower cost per completed task. Gartner predicts that generative AI cost per customer-service resolution could exceed $3 by 2030, higher than some offshore human-agent resolutions. Tool failures, retries, supervision, and human escalation belong in the calculation.

How Large Could the Market Become?

There is no single number for the voice AI market because researchers define the category differently.

Market definition	Starting estimate	Forecast	CAGR
AI voice agents	$2.5B in 2025	$35.2B in 2033	39.0%
AI voice agents, alternate estimate	$2.4B in 2024	$47.5B in 2034	34.8%
Conversational AI	Broader than voice	$41.4B in 2030	23.7% from 2025
Voice and speech recognition	$20.2B in 2023	$53.7B in 2030	14.6%
AI voice generators	$3.6B in 2023	$21.8B in 2030	29.5%

These figures overlap and should not be added together. Voice agents include orchestration and task completion. Conversational AI includes text assistants. Voice and speech recognition includes transcription, biometrics, automotive, and other uses. Voice generation includes synthesis, cloning, media, and agent speech.

The useful range is therefore directional. Two current estimates place the narrow voice-agent category near $2.4B to $2.5B in 2024-2025 and between $35B and $48B by 2033-2034. Adjacent speech and conversational markets are already larger, but they measure different revenue pools.

The 35% to 39% forecast growth rates explain the attention around the category. They remain analyst forecasts, not observed future revenue. The strongest evidence of traction today is the combination of an established voice channel, expanding enterprise AI use, and a stack that now supports open-ended conversation and tool execution.

The Stack Hidden Behind the Demo

A production voice agent is easier to understand when the animated waveform disappears.

Channel
Phone, browser, mobile app, contact center
        |
Real-time transport
WebRTC, WebSocket, SIP, RTP, PSTN
        |
Turn detection
Is the user pausing, or finished?
        |
Speech understanding
Streaming transcription or native audio
        |
Reasoning and action
Model, tools, retrieval, memory, workflows
        |
Speech generation
Streaming synthesis or native audio
        |
Operations
Hosting, tests, traces, analytics, governance

Products enter this stack from different starting points:

Starting point	What it primarily controls	Examples
Media infrastructure	Real-time audio, rooms, WebRTC, SIP	LiveKit, Daily
Agent framework	The flow of audio, model events, tools, and responses	Pipecat, LiveKit Agents
Speech recognition	Streaming transcription, endpointing, turn detection	Deepgram, AssemblyAI
Speech generation	Streaming voices, voice design, cloning	ElevenLabs, Cartesia
Native audio model	Audio understanding, reasoning, and audio generation in one session	Amazon Nova 2 Sonic, OpenAI Realtime, Gemini Live
Managed voice-agent platform	Telephony, orchestration, models, tools, tests, and operations	Vapi, Retell

A flat vendor comparison hides these boundaries. Use the table to identify which products belong in the same purchasing decision.

Three Architectures, Three Control Models

The stack can be assembled in three broad ways.

Chained

Audio -> speech-to-text -> text model -> text-to-speech -> audio

A chained system exposes each stage. The builder can choose Deepgram or AssemblyAI for recognition, a text model for reasoning, and ElevenLabs, Cartesia, or another speech model for the voice.

That separation buys control. Each component can be measured and replaced. The transcript creates an explicit audit trail. Pronunciation, moderation, model choice, and exact spoken wording can be handled at separate layers.

The cost is coordination. Every boundary adds buffering, network travel, credentials, failure handling, and another place to diagnose. Tone and hesitation can also be flattened into text before the reasoning model receives them.

Native Speech-to-Speech

Audio -> native real-time audio model -> audio

Amazon Nova 2 Sonic, OpenAI Realtime, and Gemini Live collapse speech understanding and generation into a persistent model session. These APIs can still emit transcripts and call tools, but text is no longer the only representation passed between understanding and speech.

This can preserve timing, tone, pacing, and interruption behavior. It also moves more control into one provider-specific session protocol. Voice choice, stage-by-stage diagnosis, and model portability become functions of that provider's interface.

Hybrid

Audio -> native audio understanding and reasoning -> specialist voice -> audio

A hybrid keeps speech-native input but sends the response through a separate speech generator. LiveKit documents this as a distinct pipeline, alongside chained and native real-time architectures.

The pattern matters when conversational understanding and brand voice have different owners. A native model can handle the live interaction while a specialist speech model controls voice identity, pronunciation, or language coverage.

Each architecture creates a different control model.

Every Layer Is Expanding Vertically

The voice AI landscape looks unstable because the product boundaries are moving.

Deepgram now offers a Voice Agent API on top of recognition, turn detection, and speech generation. AssemblyAI now documents stored voice agents, voices, tools, turn detection, and interruption handling alongside transcription.

ElevenLabs has expanded into ElevenAgents, including workflows, tools, knowledge, telephony, testing, and analytics. Cartesia Line similarly extends beyond speech generation into orchestration and deployment.

Infrastructure companies are moving upward too. LiveKit Agents combines its media layer with agent orchestration. Pipecat Cloud adds managed deployment, scaling, secrets, logs, and session APIs to the open-source framework.

Managed platforms are moving downward. Vapi exposes the underlying transcriber, model, and voice as configurable parts. Retell supports managed orchestration while also allowing custom telephony and custom model integrations.

The pattern is vertical convergence: each provider starts at one layer, then expands into adjacent layers to own more of the working system.

The result is several competing bundles with different escape hatches.

The Real Decision Is the Ownership Boundary

There are three practical places to draw the line.

Ownership boundary	The builder owns	The provider owns	Main tradeoff
Assemble the stack	Pipeline behavior, providers, deployment, observability	Individual model or transport services	Maximum control, maximum integration work
Use a framework	Agent logic, tools, model choices	Streaming primitives and possibly hosting	Replaceable components inside a framework contract
Buy a managed platform	Business workflow, prompts, tools, integrations	Telephony, orchestration, deployment, tests, monitoring	Fast path to production, larger platform dependency

The right amount of ownership depends on the application. A browser assistant with custom media behavior has different requirements from an outbound telephone agent that must transfer calls, detect voicemail, replay failures, and support an operations team.

The boundary should follow the part of the system that creates differentiation.

If the value is a proprietary conversation engine, keep orchestration and model choice replaceable. If the value is a business workflow, buying the media and telephony layers can remove undifferentiated work. If voice identity is central to the product, keep the speech-generation layer independent even when the rest of the agent is managed.

Measuring Latency Across the Full System

Latency is often presented as one number. A conversation experiences a chain:

1. Network transport and buffering.
2. Detecting that the user finished speaking.
3. Producing a useful transcript or audio representation.
4. Generating the first useful response.
5. Running tools or retrieval.
6. Producing the first audible speech.
7. Delivering that audio back to the user.

A "75 ms" speech-generation result and a "sub-one-second" voice response can both be accurate while measuring different intervals. Neither describes the full call.

Turn detection is a good example. Deepgram documents endpointing separately from transcription, while AssemblyAI combines acoustic and semantic turn signals. An aggressive endpoint responds quickly but can cut off a caller who pauses before a number or name. A conservative endpoint captures the thought but feels slower.

My own Nova 2 Sonic and LiveKit experiment exposed a larger measurement trap. Retrieval took roughly half a second. The model then took 55 to 90 seconds to turn about 1,500 characters of dense retrieved material into a spoken response. Optimizing the vector lookup would have targeted less than 1% of the delay.

The useful metric is the distribution from user stops speaking to first useful audible response, measured on the real channel with real tools.

What to Compare Instead

I would evaluate a voice-agent architecture on eight questions:

1. Channel: Does it need a browser, mobile app, telephone network, contact center, or multi-party room?
2. Turn-taking: How does it distinguish a pause from a completed thought?
3. Interruption: Does playback stop immediately and recover cleanly when the user speaks?
4. Tool reliability: Can it select the right tool, produce valid arguments, recover from failure, and confirm before writes?
5. Auditability: Are transcripts, model events, tool traces, recordings, and replays available?
6. Portability: Which models, transports, phone numbers, prompts, and evaluation sets can move elsewhere?
7. Operations: Who owns deployment, scaling, regression tests, monitoring, and failure diagnosis?
8. Economics: What is the cost per completed task, including retries and human escalation, rather than the advertised cost per minute?

This changes the research agenda. The useful experiment holds the workflow constant while comparing competing ownership boundaries:

• LiveKit Agents versus Pipecat with the models held constant.
• Native speech-to-speech versus a chained pipeline with the task held constant.
• Vapi or Retell versus a framework-based implementation with the telephone workflow held constant.
• Specialist speech components versus each provider's integrated agent platform.

Without those controls, a vendor comparison measures several architectural changes at once.

What Is Still Missing

I have validated one path: LiveKit transport and orchestration with Amazon Nova 2 Sonic, browser and telephone channels, tools, persistence, and knowledge retrieval. I have not run the same workflow through the other frameworks, managed platforms, native audio models, or integrated agent platforms described above.

That boundary matters. Public documentation can establish what a product exposes. It cannot establish interruption quality, tool reliability, operational clarity, or cost per successful outcome under the same conditions.

The next useful map should be a controlled comparison of ownership boundaries.

Understanding voice AI as a stack leaves one open question: which layer becomes the durable control point? It could be the model session, the media network, the orchestration framework, the telephone platform, or the evaluation system that tells operators why a call failed.