A voice agent can have excellent speech recognition, a capable language model, and natural text-to-speech, then still feel broken.
The reason is simple: users experience the system as one conversation, not as four separate services.
They notice the pause after they stop speaking. They notice when the agent talks over them. They notice when a transcription error sends the conversation in the wrong direction. They notice when high-quality synthetic speech arrives too late.
That makes real-time voice AI an architecture problem as much as a model problem.
A typical system has four core layers:
- Speech-to-text (STT)
- A large language model (LLM)
- Text-to-speech (TTS)
- Telephony or another real-time audio transport
The interesting engineering happens in the boundaries between them.
The architecture in one sentence
At its simplest, the pipeline looks like this:
User speech
↓
Streaming STT
↓
LLM
↓
Streaming TTS
↓
Telephony / WebRTC
↓
User hears response
That diagram is useful, but it can also be misleading.
A production voice system should not behave like a serial batch-processing pipeline where STT finishes completely, then the LLM starts, then TTS starts, then audio is finally delivered.
If every stage waits for the previous one to finish, latency compounds.
Responsive systems stream and overlap work wherever possible.
1. STT turns audio into usable state
Speech-to-text, also called automatic speech recognition or ASR, converts incoming audio into text that the rest of the system can process. Developers evaluating this layer can compare the architecture against a production speech-to-text system rather than treating recognition as an isolated offline task.
For offline transcription, waiting for a complete recording may be fine. Real-time conversations do not have that luxury.
A streaming STT engine emits partial transcription hypotheses while the user is still speaking. That gives downstream components something to work with before the utterance has completely finished.
There is a tradeoff.
Partial transcripts can change as more audio arrives. If the system commits too early, one incorrectly recognized word can steer the LLM toward the wrong intent.
This is why STT quality is only part of the problem.
VAD can make a fast model feel slow
Voice Activity Detection, or VAD, determines whether the user is currently speaking.
It also helps answer a critical question:
When has the user actually finished their turn?
If the endpointer is conservative, it waits longer before deciding that speech has ended.
To the infrastructure, that might be a few hundred milliseconds of uncertainty.
To the user, it feels like the AI is thinking too slowly.
A surprising number of apparent model-latency problems are actually endpointing problems. Smallest AI's guide to Voice Activity Detection for real-time voice apps goes deeper into latency, false triggers, and production tuning around this layer.
Aggressive VAD settings create the opposite failure mode. The system may decide that the user has finished during a natural pause and start generating a response too early.
Real-time speech systems therefore have to balance responsiveness with turn-detection accuracy.
2. The LLM is the reasoning layer, not the whole voice system
Once enough transcript is available, the LLM determines what the system should say next.
For many voice architectures, this is the largest individual source of compute latency.
A large general-purpose model may provide excellent reasoning, but that capability comes with an inference cost. In a voice application, every extra delay becomes visible because the user is waiting for speech to resume.
Two architectural choices help.
First, structured workflows do not always require the largest available model. Customer support, appointment scheduling, qualification, routing, and similar flows may benefit from smaller task-focused models when their capabilities fit the workflow.
Second, the LLM should stream output.
If the application waits until the complete response has been generated, the TTS layer sits idle.
With token streaming, the system can send usable text fragments downstream while the remainder of the response is still being generated.
The LLM is important, but optimizing it while ignoring the rest of the stack is a mistake. Users judge the complete conversation.
3. TTS determines when the response becomes real
Text-to-speech converts the LLM response back into audio. A productionctext-to-speech layer has to be evaluated by how quickly it can begin returning usable audio, not only by the quality of a completed file.
For a real-time system, the critical question is not simply:
"How long does synthesis take?"
A more useful question is:
How quickly can the system produce the first playable audio?
This is commonly measured as Time to First Audio Byte, or TTFAB.
A streaming TTS system begins emitting audio chunks before the entire response is available. That allows playback to start while synthesis continues.
Without streaming, even a fast LLM can be followed by a noticeable pause while the speech engine waits for the complete sentence and synthesizes it.
This is another reason component benchmarks cannot be evaluated in isolation. What matters is how quickly usable output moves across the entire chain.
4. Telephony is part of the architecture, not plumbing
The final layer carries audio between your application and the user.
Depending on the product, that might involve:
- The public telephone network
- SIP
- VoIP infrastructure
- A PBX or cloud phone system
- WebRTC in a browser or application
This layer creates its own failure modes.
Networks introduce jitter and variable delay. Codecs compress audio. Packet loss affects intelligibility. Transcoding can alter the audio reaching the STT system.
Those effects often appear only under production traffic.
A speech recognition model that performs well on clean microphone recordings may behave differently after audio has passed through a telephone codec.
Similarly, perfect TTS output generated in the backend is irrelevant if the delivery path degrades it before the caller hears it.
Telephony therefore belongs inside the performance budget.
Streaming changes the shape of the pipeline
The most important architectural idea in real-time voice AI is overlap.
Instead of this:
STT completes
↓
LLM completes
↓
TTS completes
↓
Audio plays
you want behavior closer to this:
STT ===============
LLM =============
TTS =============
Playback ===========
The stages remain logically separate, but execution overlaps.
STT can emit partial text while the user is speaking.
The LLM can begin once enough stable context exists.
TTS can begin when it receives a usable text fragment.
Playback can begin as soon as the first synthesized audio reaches the transport layer.
This architecture turns latency from a simple sum into a coordination problem.
For a deeper treatment of this pattern, the Smallest AI guide to streaming architecture for real-time voice agents explains why streaming has to extend across the pipeline rather than being added to only one component.
Where the latency budget goes
An illustrative latency budget might look like this:
| Component | Typical contribution | Primary optimization lever |
|---|---|---|
| Streaming STT | 50-100 ms | Streaming transcription and VAD tuning |
| LLM inference | 150-300 ms | Smaller models, token streaming, caching |
| TTS synthesis | 50-150 ms to first audio | Streaming TTS and low-latency models |
| Telephony / network | 20-80 ms | Deployment location, codec choice, WebRTC |
These values should be treated as architectural estimates, not universal guarantees. Real performance varies with model size, infrastructure, geography, audio conditions, and the transport path.
The important pattern is the distribution.
The LLM is commonly the largest individual contributor, but optimizing only the LLM does not guarantee a responsive conversation.
You might reduce inference time and still lose the improvement because VAD waits too long.
You might deploy faster TTS and then add network latency between regions.
You might improve STT accuracy while choosing a model that processes audio too slowly for the desired interaction.
Voice latency is an end-to-end property.
What a production deployment actually looks like
Consider a contact-center workflow.
A caller enters through SIP into an existing PBX or cloud telephony environment.
Streaming STT begins processing audio as it arrives.
The LLM receives the transcript and determines the next response. When the workflow requires external information, the application can consult a knowledge base, CRM, scheduling system, or another business tool through application logic or function calling.
The response is streamed toward TTS.
Synthesized audio is then returned through the telephony layer.
If the conversation moves outside the system's permitted scope or requires human intervention, the call can be escalated according to the application's fallback logic.
The same architectural pattern can be used in healthcare scheduling, support, commerce, sales, and other conversational applications. The requirements change, but the underlying coordination problem remains similar.
For teams that prefer an integrated implementation path rather than assembling every layer independently, the Smallest AI voice-agent platform brings agent configuration, telephony, integrations, and deployment workflows into one environment. The related guide to voice bot architecture shows how the speech and orchestration layers can be connected inside a production SDK.
Three architecture misconceptions worth avoiding
Misconception 1: Better accuracy automatically creates a better real-time system
Accuracy and latency have to be evaluated together.
A larger speech model may improve recognition but require more processing per audio chunk.
For transcription workloads where accuracy dominates, that trade can make sense.
For a live customer conversation, a modest accuracy improvement may not justify a delay that repeatedly disrupts turn-taking.
There is no single correct tradeoff. It depends on the application.
The important point is that an offline accuracy benchmark cannot tell you whether a model is appropriate for a real-time conversation.
Misconception 2: The LLM is the voice AI
It is not.
A powerful LLM connected to poor STT can reason about the wrong transcript.
A powerful LLM connected to slow TTS still feels slow.
A powerful LLM behind badly configured VAD may constantly interrupt users or leave awkward pauses.
A powerful LLM sent through an unreliable telephony layer still produces an unreliable product.
The voice agent is the system formed by all of these components.
Misconception 3: Speech-to-speech eliminates the modular pipeline everywhere
Speech-to-speech models can accept audio and produce audio without exposing explicit text stages in the same way as a traditional STT + LLM + TTS architecture.
That can be useful in latency-sensitive applications.
It does not automatically make modular architectures obsolete.
Many production systems still require clear insertion points for business logic, predictable workflows, auditability, tool execution, and explicit control over individual stages.
For those systems, keeping STT, reasoning, and TTS as identifiable components can be valuable.
Speech-to-speech is therefore another architectural option, not a universal replacement.
Production problems start where the happy path ends
Choosing models is only the beginning.
Real users interrupt, hesitate, change their minds, speak over the system, call from noisy environments, and ask questions the application was never designed to answer.
Several decisions become critical once a system reaches production. Smallest AI's article on designing AI voice agents provides additional context on architecture, use cases, and safety guardrails beyond the basic STT + LLM + TTS pipeline.
Interruption handling
Users will talk while the AI is speaking.
The system needs to detect the new speech, cancel or stop the active TTS output, update the conversational state, and process the interruption.
If it cannot, the agent talks over people.
That immediately makes the experience feel mechanical.
Context management
Conversation history grows on every turn.
Sending an indefinitely growing context back to the LLM can increase latency and processing requirements.
Common approaches include summarizing older turns or maintaining a sliding context window while preserving important application state.
The right approach depends on how much historical context the workflow genuinely needs.
Fallback and escalation
A production system needs defined behavior for situations such as:
- Low-confidence transcription
- Unsupported requests
- Missing business data
- Tool failures
- Requests that require human handling
Fallback logic should be designed as part of the architecture, not added after deployment.
Edge versus cloud placement
Every network hop matters.
Moving audio between regions can consume part of the latency budget before the models have done any work.
Some architectures move latency-sensitive processing closer to the user while keeping other components in centralized cloud infrastructure.
The correct boundary depends on infrastructure, model requirements, geography, and operational complexity.
Codec and audio quality
Phone audio is not the same as clean studio audio.
Codecs such as G.711 and Opus affect what reaches the speech recognition system and what the caller ultimately hears.
When production accuracy suddenly differs from testing, inspect the audio path before assuming the model itself has regressed.
Prototyping the stack with Smallest AI
The current Smallest AI voice platform spans speech recognition, speech generation, speech-to-speech, and voice-agent workflows. For developers evaluating this architecture programmatically, the Smallest AI API provides the developer entry point for working with the relevant voice models and services.
The useful way to test a real-time architecture is not to evaluate only isolated model output.
Run the system using representative audio, your actual deployment geography, the codecs your application will use, realistic conversation lengths, and real interruption behavior.
Measure at least:
- Time from user speech ending to the first LLM output
- Time to first synthesized audio
- End-to-end turn latency
- STT errors under real audio conditions
- False VAD triggers
- Missed end-of-turn events
- Barge-in behavior
- Network and telephony delay
If authenticated API requests are added to your implementation, keep API credentials on the server. Do not expose them in browser JavaScript, mobile application code, public repositories, screenshots, URLs, or client-side logs.
The architecture is the product
The failure mode in real-time voice AI is rarely one component completely breaking.
More often, the experience degrades through accumulation.
A little VAD delay.
A slow first LLM token.
A TTS buffer that waits too long.
An unnecessary network hop.
A codec mismatch.
A barge-in handler that does not cancel playback quickly enough.
Each problem may look small in isolation. Together, they determine whether the conversation feels natural.
That is the central architectural lesson: optimize the handoffs, not just the parts.
STT has to stream.
The reasoning layer has to produce output early enough for downstream synthesis.
TTS has to return playable audio incrementally.
The delivery layer has to preserve both timing and audio quality.
And the entire system has to survive interruptions, imperfect networks, growing context, and unpredictable users.
If you are building this pipeline, the best test is your own application under realistic conditions. Create an API key and prototype the voice workflow with Smallest AI, then measure the complete path from live audio input to the first audio returned to the user.
Top comments (0)