An AI voice agent is software you can talk to. It listens to you speak, reasons about what you said, and talks back in real time — holding a natural spoken conversation instead of marching you through a phone menu.
That definition is the easy part. Every explainer on the internet will tell you a voice agent is speech-to-text, plus a language model, plus text-to-speech.
Here is the part almost nobody writes down: a voice agent is a latency and connectivity problem, not a prompt problem.
The language model is the easy 80%. Wiring three models together is a weekend project. The hard part — the part that decides whether your agent feels human or just goes silent in a customer's office — is the plumbing underneath: how the audio actually travels, how fast, and whether it can connect at all.
This guide covers the whole stack. You'll get the standard architecture, the honest 2026 trade-offs, a real latency budget with numbers we measured ourselves, and the transport layer that the top-ranking guides leave out entirely.
TL;DR: An AI voice agent hears you (speech-to-text), thinks (an LLM), and speaks back (text-to-speech), coordinated by an orchestrator that manages turns and interruptions — with every stage streamed to stay under the ~800 ms that keeps a conversation feeling natural. The layer most guides skip is transport: the audio rides WebRTC, and on real-world networks it needs a TURN relay, or the agent connects and goes silent.
What Is an AI Voice Agent?
An AI voice agent is an autonomous, voice-first system that holds a natural spoken conversation, reasons about it in real time with a large language model, and takes action across connected systems — without a human scripting each turn. It combines speech-to-text, an LLM, and text-to-speech, connected over a real-time media transport like WebRTC.
That last clause is the one you rarely see, and it is the whole reason this article exists.
The difference between a voice agent and the older systems it replaces is intent. An IVR forces callers down a fixed menu tree — "press 1 for sales." A chatbot handles typed text. A voice agent understands natural spoken language, decides what to do, calls your backend, and answers in speech, per the distinction drawn across Aircall's 2026 explainer and Deepgram's 2026 guide.
So a voice agent is a loop, not a black box. It hears, it thinks, it speaks, and something has to conduct all three in the right order, fast enough that you don't notice the seams.
That conductor is the orchestrator, and it does more than pass data between stages. Let's break the loop down stage by stage.
How Do Voice Agents Work? The Four Stages
A voice agent runs four real-time stages in a tight loop: speech-to-text transcribes what you say, an LLM decides the reply and calls any tools, text-to-speech speaks it, and an orchestrator manages when a turn ends and when to stop for an interruption. Streaming overlaps the stages so the reply starts before the model has finished thinking.
Here's each stage and what it's actually responsible for.
Speech-to-text (STT) turns your audio into words the model can read. In a good agent it runs continuously, emitting partial transcripts as you talk rather than waiting for you to finish.
The LLM reads the transcript, decides what to say, and — when needed — calls tools: look up an order, book the appointment, check inventory. This is the "brain," but as you'll see in the latency section, it is rarely the slow part.
Text-to-speech (TTS) turns the reply back into audio. Streaming TTS starts speaking the first words while the rest of the sentence is still being generated, which is what keeps the pause short.
The orchestrator is the unsung hero. It decides when your turn has ended (turn detection), starts the reply, and — critically — stops the agent mid-sentence the instant you interrupt. That interruption behavior is called barge-in, and getting it right is most of what makes an agent feel human rather than robotic.
Notice the theme: everything streams. A voice agent that waits for each stage to fully finish before starting the next one feels broken, because the silence stacks up.
AssemblyAI's April 2026 architecture breakdown puts a number on it: a naive, non-streaming pipeline adds two to four seconds of dead air per turn. Nobody waits four seconds for a "hello."
That's the standard model. But in 2026 there's a real architectural fork in the road — one model or three?
Cascaded Pipeline vs Speech-to-Speech
There are two ways to build the "hear-think-speak" loop, and the choice is a genuine engineering trade-off in 2026. A cascaded pipeline chains three separate models — STT, then LLM, then TTS — with readable text between each step. A speech-to-speech (S2S) model does it in one shot: audio in, audio out, no text in the middle.
Cascade gives you a text artifact at every step, so you can log it, moderate it, filter it, and route on it. Speech-to-speech often feels more natural and can be faster, but it's harder to debug, more expensive, and less transparent.
As of April 2026, cascade still dominates production, per Deepgram and Softcery's lab tests. Here is how the two stack up on the numbers people actually argue about.
| Dimension | Cascaded (STT → LLM → TTS) | Speech-to-speech (one model) |
|---|---|---|
| Time to first audio | ~1.5–3 s in production (Softcery/Deepgram, 2026) | 0.78–2.98 s across models (Softcery, Apr 2026) |
| Cost per minute | ~$0.05–0.15 (Softcery, Apr 2026) | ~$0.15–0.60 for premium realtime (Softcery, Apr 2026) |
| Transparency | High — text at every stage to log and filter | Low — audio in, audio out, harder to audit |
| Debuggability | Easy — inspect the transcript and the reply | Hard — no intermediate text artifact |
| Best for | Transactional, regulated, tool-heavy agents | Natural-feeling, latency-sensitive chat |
The cost gap is not a rounding error. Softcery's April 2026 measurements show a spread of up to 182× between the cheapest cascaded stack and premium realtime speech-to-speech models — the difference between a fraction of a cent and roughly thirty cents a minute.
Cascade runs about $0.05–$0.15 per minute and produces a clean text log at every step; premium speech-to-speech models run roughly $0.15–$0.60 per minute and feel more natural but are far harder to audit. Their time-to-first-audio ranges from about 0.78 seconds on the fastest model to nearly 3 seconds on others, versus roughly 1.5–3 seconds end-to-end for a well-built cascade.
Here's what most teams actually do: a hybrid. Use a speech-to-speech model for the natural-feeling opener, then fall back to a cascaded pipeline for the transactional turns where you need a text log and tool calls.
Now, every source above measures the models. None of them measures the wire the audio travels on. That's the layer we're built to explain.
The Layer Most Guides Skip: How the Audio Actually Travels
Here is the gap. The best voice-agent architecture guides — AssemblyAI's (April 2026) and Deepgram's (2026) among them — walk through STT, LLM, and TTS in detail and then stop. Deepgram names WebRTC as a transport and moves on. None of them explains the layer underneath: how the audio actually gets from a user's microphone to your agent and back.
So let's fill it in.
Audio has to ride something. There are three real choices for moving real-time voice, and they are not interchangeable.
WebRTC is the transport built for exactly this job. It runs over UDP, was designed for low-latency real-time media, handles packet loss gracefully, and includes the machinery to punch through firewalls. The emerging consensus for production voice agents is WebRTC over WebSockets.
WebSockets run over TCP. They're perfect for control messages, transcripts, and prototypes — but TCP's head-of-line blocking means one late packet stalls everything behind it, which is exactly the wrong property for live audio. Fine for a demo; not what production settles on.
SIP is the telephony path. If your agent answers actual phone calls over the PSTN, SIP is in the picture — it's a live transport decision for phone-based agents, as relinns' 2026 comparison lays out. For a web or app-based agent, WebRTC is the default.
So for the agents most people are building — web and app — the answer is WebRTC. But choosing WebRTC is where the connectivity problem begins, not where it ends.
Because WebRTC prefers a direct UDP path, and on a lot of real-world networks, that direct path simply doesn't exist. Hold that thought — we'll get to why in two sections. First, let's put real numbers on "fast enough."
The Real Latency Budget (With Our Measured Numbers)
Here's the target: keep the end-to-end response under about 800 milliseconds at the 95th percentile, from the moment you stop talking to the moment the agent starts. Under 500–700 ms feels natural; past that, it starts to feel like a bad phone connection.
Those thresholds come from practitioners. AssemblyAI's April 2026 breakdown budgets 600–900 ms for a fully streamed pipeline; Prodinit's 2026 production guide puts the reliability floor at sub-800 ms p95 and calls sub-250 ms p50 achievable. AssemblyAI also notes that responses beyond 500–700 ms "start to feel unnatural."
Here's a dated budget, stage by stage, from the published sources next to our own measured build.
| Stage | Industry budget (AssemblyAI, 2026-04-29) | Our measured run (2026-07-14, M1 Pro, cloud) |
|---|---|---|
| Speech-to-text | 200–500 ms | ~1,200–1,800 ms (the bottleneck) |
| LLM (first token) | 150–400 ms | ~800–1,400 ms |
| Text-to-speech (first audio) | 200–400 ms | ~500–700 ms |
| Network / transport | 50–150 ms | included above |
| End-to-end | 600–900 ms | 2,500–3,900 ms to first audio |
| Barge-in (interruption) | — | ≈ 1 ms (server-side VAD) |
Let me be honest about our numbers. We built a real voice agent and measured it on 2026-07-14 — a MacBook Pro, 16 GB RAM, on a residential network, running a cloud stack of OpenAI GPT plus Whisper STT plus OpenAI TTS with streaming enabled. These are our numbers on our hardware, not a vendor performance guarantee. The full build and method are in our TypeScript voice-agent tutorial.
End to end, we measured roughly 2.5–3.9 seconds to first audio on the default buffered pipeline. That's slower than the streamed-ideal industry budget — roughly 200–500 ms for STT, 150–400 ms for the LLM's first token, 200–400 ms for TTS, and 50–150 ms of network, about 600–900 ms end to end (AssemblyAI, 2026-04-29). The gap between that textbook 600–900 ms and a real 2.5–3.9 s is buffering and cold starts, not model quality — which is exactly the point.
And here's the insight that reorganizes how you optimize: STT dominates. Buffered Whisper alone ate 1.2–1.8 seconds — the single biggest slice. The LLM (~0.8–1.4 s) and TTS (~0.5–0.7 s) were not our bottleneck; transcription was. If you're trying to make an agent feel faster, the streaming STT model is usually the highest-leverage lever, not a bigger LLM.
One more number worth staring at: barge-in. When we talked over the agent, the server's voice-activity detection cancelled its speech within about 1 millisecond of detecting our voice. Interruption handling isn't a latency problem — it's an architecture problem, and it belongs on the server, close to the media.
Which brings us back to the connectivity thread. All of this latency assumes the audio connects in the first place. On a lot of networks, it doesn't.
Why Voice Agents Fail on Corporate, Hospital, and Mobile Networks
This is the failure mode that ships to production and blindsides teams: the agent works perfectly on your Wi-Fi and goes silent in a customer's office. The call connects, the transcript even flows — and there's no audio.
It's not a bug in your code. It's the network, and it's predictable.
WebRTC wants a direct peer-to-peer path over UDP. Restrictive networks break that in two ways. Many corporate, hospital, and mobile carrier networks use symmetric NAT, which scrambles the address mapping so the two sides can't agree on where to send packets. Others simply block the UDP ports WebRTC reaches for.
Here's the pattern we see over and over: it works on the developer's home Wi-Fi and dies the moment a real user is on a corporate LAN or on cellular. Same code, different network, silent call.
Now add the twist that makes this worse for agents specifically. A browser-to-browser call can sometimes fall back to a direct path between two consumer networks. A server-side voice agent has no peer-to-peer fallback — one end is a machine in a data center — so when the direct path fails, there is no plan B on the same wire. The audio has to be relayed, which is why agent media is almost always relayed in production. We wrote up that reasoning in depth in TURN for AI voice agents.
So the uncomfortable truth is that "it works on my machine" is the default state of a WebRTC voice agent, and it lies to you. The networks where it fails are precisely the networks your paying customers sit on.
The fix is a relay. And that's where STUN and TURN come in.
STUN, TURN, and What a Production Agent Actually Needs
Two acronyms do the connectivity work, and they are not the same thing. STUN helps a client discover its own public address so two peers can try a direct connection. TURN is the fallback that actually relays the media through a server when the direct path fails. STUN discovers; TURN relays. Confusing the two is the most common mistake in this whole topic.
For a production voice agent, you need TURN. The direct path fails often enough on real networks — and a server-side agent has no peer-to-peer fallback — that a relay isn't a nice-to-have, it's the thing standing between "works in the demo" and "works for customers."
There's a detail that matters for the hardest networks: TURN over TLS on port 443. Locked-down corporate and hospital firewalls that block everything else usually still allow outbound 443, because that's where normal HTTPS lives. A TURN server that speaks TURNS on 443 looks like ordinary web traffic and gets through where raw UDP is dead on arrival.
So how do you get a TURN server? You have two honest paths, and both are legitimate.
Run your own with an open-source server like coturn. It's free software, but you own the config, the TLS certificates, the ports, the capacity planning, and the bandwidth bill. It's real DevOps work, and the bandwidth adds up.
Use a managed TURN service and skip the operations. You can start free — Open Relay gives you 20 GB/month of TURN bandwidth at no cost, on ports 80 and 443 with TURNS, which is plenty for small workloads. When you outgrow it, Metered's TURN service starts with a 500 MB free trial and then runs Growth at $99 for 150 GB, Business at $199 for 500 GB, and Enterprise at $499 for 2 TB, across 31+ regions and 100+ edge locations.
You don't have to decide today. The point is that a TURN relay is part of the voice-agent stack, full stop — not an optional extra you bolt on after launch, but the layer that makes the other four work on real networks. If you want to check whether your own agent's path holds up, you can test a TURN connection before your users do it for you.
That's the whole stack, actually. Let's map it.
The Voice-Agent Stack You Actually Need
A voice-agent stack has six layers, and most guides only cover the first four. You need speech-to-text, an LLM, text-to-speech, an orchestrator for turn-taking and barge-in, a media-transport layer (WebRTC) to move the audio, and — for production reliability — a TURN relay plus signalling to establish the connection.
The first four are the conversation. The last two are the connection. Skip the connection layers and you get an agent that demos beautifully and fails in the field.
Metered is not a voice-agent platform, and this isn't a pitch to replace your STT, LLM, or TTS. It is the infrastructure under the stack — the transport and connectivity layer that sits beneath any voice agent, whether you built it yourself or bought a hosted one.
That layer is two things. TURN relays the media so the agent connects on real networks; Open Relay covers the free tier and the Metered TURN service covers scale, regions, and analytics. And signalling is the coordination channel that helps the two sides find each other and exchange connection details before the media flows.
On signalling, one honest note so you can plan the whole connection layer: Metered Realtime is a managed signalling service you can start free, with an MIT-licensed open-source client.
Read the stack top to bottom and the thesis of this whole guide falls out. The top four layers are where the intelligence lives, and they're mostly solved by picking good models. The bottom two are where agents actually break, and they're an infrastructure problem — latency and connectivity — not a prompt problem.
So how do you assemble all six? You have three routes.
How to Build a Voice Agent
build it open-source.** Assemble it yourself for full control and provider freedom. Our own free, open-source TypeScript SDK, LLMRTC, handles the voice-agent hard parts — WebRTC transport, server-side voice-activity detection, natural barge-in, tool calling, and a provider-agnostic pipeline so you can swap OpenAI, Anthropic, Gemini, or local models by config. It's Apache-2.0 and truly free, built by a team that runs production WebRTC infrastructure. Its own docs tell you to put a TURN server in front of it for users behind NAT — because, as this article has hammered, you need one.
We proved that route end-to-end: our TypeScript voice-agent tutorial builds a real agent and swaps the entire stack from cloud to 100% local with one config change. And if you're wiring a browser straight to a hosted realtime model, our walkthrough of OpenAI Realtime over WebRTC shows exactly when TURN enters the picture — the direct browser-to-OpenAI path connected with no iceServers at all, and TURN came back the moment we owned a leg of the connection.
Route 2: hybrid. Use a platform or a hosted model for the conversation, and own the connectivity layer yourself — WebRTC transport, TURN relay, signalling — so you control quality, regions, and cost on the part that actually breaks. This is where most serious deployments land, and it's the route where an independent TURN service earns its keep.
Whichever route you take, the connection layer is yours to get right. No platform makes the network problem disappear if you own any leg of the WebRTC path.
Frequently Asked Questions
What is an AI voice agent?
An AI voice agent is an autonomous, voice-first system that holds a natural spoken conversation, reasons about it in real time with a large language model, and takes action across connected systems — without a human scripting each turn. It combines speech-to-text, an LLM, and text-to-speech, connected over a real-time media transport like WebRTC.
How do AI voice agents work?
They run four real-time stages: speech-to-text transcribes the user, an LLM decides the response and calls tools, text-to-speech speaks it, and an orchestrator manages turn-taking and interruptions. Streaming overlaps the stages to cut latency. A media-transport layer — usually WebRTC — carries the audio between the user and the agent.
How much latency is acceptable for a voice agent?
Keep end-to-end response under about 800 ms at p95, from end of speech to first audio; under 500–700 ms feels natural, and sub-250 ms p50 is achievable with a fully streamed stack. A naive, non-streaming pipeline adds 2–4 seconds of dead air, which breaks the conversation. Transport is part of that budget.
Do voice agents need a TURN server?
In production, usually yes. A server-side agent's WebRTC media has no peer-to-peer fallback, and many corporate and mobile networks block direct UDP, so the audio must be relayed through a TURN server. You can start free with Open Relay's 20 GB/month and move to a managed TURN service for regions, capacity, and analytics as you scale.
Why does my voice agent work locally but fail on office or hospital Wi-Fi?
Restrictive networks use symmetric NAT and block the UDP ports WebRTC needs, so the direct media path can't form. A server-side agent has no peer-to-peer fallback, so its audio is almost always relayed — which means it needs a TURN server. Without one, the call connects and then goes silent.
What's the difference between speech-to-speech and a cascaded pipeline?
A cascaded pipeline chains three models — STT, LLM, TTS — giving you a readable text artifact at each step to log, filter, or route. Speech-to-speech uses one model from audio in to audio out: often more natural, but harder to debug, more expensive, and less transparent. As of April 2026, cascade still dominates production.
What's the difference between a voice agent and an IVR or chatbot?
An IVR forces callers down fixed menus ("press 1 for sales"); a chatbot handles typed text. A voice agent understands natural spoken language, reasons with an LLM, takes actions in backend systems, and replies in natural speech — no scripted menu tree, and it works over the phone or the web.
The Bottom Line
An AI voice agent is four models in a loop — hear, think, speak, and a conductor to run them — riding on two layers of connectivity most guides never mention. Get the models right and you have a demo. Get the transport and relay right and you have a product.
That's the whole argument: voice agents are a latency and connectivity problem, not a prompt problem. The prompt is the part you'll finish first. The media path is the part that decides whether your agent feels human or goes silent on the exact networks your customers use.
So build the conversation however you like — a platform, open-source, or a hybrid. But own the connection layer. Start free on Open Relay's 20 GB/month, and when real users behind real firewalls show up, Metered's TURN service relays the audio across 31+ regions so your agent connects everywhere — not just on your Wi-Fi.
Top comments (1)
Thank you for reading. I hope you like the article