Search "best voice agent API" and you'll get a beauty contest. Everyone's the fastest, everyone's the easiest, everyone has the slickest dashboard. It's noise. And it points you at the wrong question.
Here's the question that actually decides whether your voice agent works in production: does it hear your user correctly? Because a voice agent is a pipeline — speech-to-text feeds the LLM, the LLM decides what to say, and text-to-speech says it. If the first step gets the words wrong, everything downstream is confidently answering a question the user never asked. The model can be brilliant. The voice can be gorgeous. Doesn't matter. Garbage in, garbage out — at the speed of conversation.
So the best voice agent API isn't the flashiest one. It's the one built on the most accurate foundation. Let's talk about how to actually evaluate that, and where the real options land.
What "best voice agent API" actually means
When most people picture a voice agent, they picture the LLM — the "brain." That's the fun part. But the brain only ever sees text. It never hears your customer. It reads a transcript that some speech-to-text model produced a few hundred milliseconds ago, and it acts on that transcript as if it were ground truth.
Think about what that means when someone says their name is "Nguyen," reads back a 16-digit account number, or talks through a drive-thru with road noise in the background. If the STT mishears "Nguyen" as "when," the agent doesn't know it misheard. It just runs with the wrong input. The user has to repeat themselves, the conversation stalls, and trust evaporates.
That's why we frame the whole category around accuracy. The foundation of any voice agent is how well it hears — because that's the one error every other component inherits and none of them can fix. Pick your API on that basis and most of the "which platform is prettiest" debate stops mattering.
The criteria that decide production quality
A demo that works once in a quiet room tells you almost nothing. Here's what separates a voice agent that ships from one that gets pulled after week one:
- Speech accuracy on real audio. Not clean, scripted benchmark clips — messy conversational audio with names, account numbers, accents, cross-talk, and background noise. This is the tiebreaker, and we'll come back to it.
- Turn detection. Knowing when the user has actually finished speaking versus just pausing mid-thought. Get this wrong and your agent either interrupts constantly or sits there dead-air awkward.
- Interruption and barge-in handling. Real conversations aren't turn-based. People cut in. A production agent has to stop talking the instant the user does and pick up the new thread.
- Latency. End-to-end, mouth to ear. Above roughly a second and a half, the conversation starts feeling like a bad phone connection and people talk over the agent.
- Billing model. One unified bill, or three vendors you're metering and reconciling separately? This is a cost-predictability question as much as an accounting one.
- Concurrency and scale. Can you run 500 calls at once during a Monday-morning spike, or are you hitting a concurrency cap and rate limits?
- Developer experience. Standard protocols and clear docs, or an SDK maze and a proprietary conversation-design tool you have to bend your app around?
Weight these however your use case demands. But notice that accuracy sits underneath most of them — bad transcripts make turn detection harder, force retries that add latency, and generate the "sorry, can you repeat that?" loops that wreck the experience.
API vs platform: which are you actually buying?
This is the fork in the road, and it's worth being honest about.
No-code platforms — Vapi, Retell, Synthflow — give you an opinionated way to design a conversation. Drag-and-drop flows, hosted orchestration, a dashboard for non-engineers. If you're a small team that needs a working phone agent by Friday and your logic is fairly standard, that's a genuinely good fit. No shame in it. You're trading control for speed, and sometimes speed is exactly what you need.
But here's where it gets interesting. Those same platforms bundle a speech-to-text vendor you often can't choose, wrap your logic in their conversation model, and constrain what you can customize when your requirements get weird — and in production, requirements always get weird. You inherit their accuracy, their latency budget, and their abstractions. When voice agent stacks start showing their limits, it's usually because the thing you most need to tune is the thing the platform hid from you.
A voice agent API gives you the pipeline and the control. You own the logic, the prompts, the integration, and — critically — the foundation your agent hears through. That's the path if you're building something differentiated, if accuracy is load-bearing, or if you just don't want a vendor's opinions baked into your product.
The reframe: don't ask "which platform wins." Ask "am I buying a control surface or a black box?" — and then, whichever you pick, ask what it's hearing your users through.
How the main options compare
Let's be objective about the field. Every option here is a real, capable product. They just optimize for different things.
AssemblyAI Voice Agent API. One WebSocket handles the full STT → LLM → TTS pipeline, built on Universal-3.5 Pro Realtime for speech accuracy. Flat $4.50/hour all-in, one bill. Around one second end-to-end latency, unlimited concurrency, no rate limits, and standard JSON over WebSocket with no SDK required. You can update the system prompt, tools, and settings mid-conversation without reconnecting. It's invisible infrastructure — the foundation your agent stands on, not another dashboard to learn.
OpenAI Realtime. A multimodal model that also does voice. But that's the framing to keep in mind: it's a model that speaks, not a pipeline engineered end-to-end for conversation quality. It's token-priced, so your cost varies with usage in ways that are hard to forecast, and you're assembling the production concerns — turn-taking, barge-in, scale — yourself.
Deepgram. Ships a unified voice agent API. . The differentiator comes down to accuracy on real agent audio, which is where the benchmark numbers below get pointed. We put the full breakdown in our AssemblyAI vs Deepgram comparison.
ElevenLabs. If lifelike TTS is your top priority, they're have a good option. The tradeoffs are concurrency (capped around 30, which matters at scale) and, on the input side, transcription accuracy. More in our Voice Agent API vs ElevenLabs Conversational AI writeup.
Vapi and Retell. The leading no-code platforms. Fast to a working agent, great for standard flows and non-engineering teams. You trade control and vendor choice for that speed, per the section above.
One more thing worth naming: LiveKit and Pipecat are open-source pipeline frameworks, not competitors. We ship drop-in plugins for both. If you're already building on them, you can slot AssemblyAI in as the STT layer without rearchitecting anything.
| Option | Foundation | Pricing | Concurrency | Best for |
|---|---|---|---|---|
| AssemblyAI Voice Agent API | Universal-3.5 Pro Realtime | Flat $4.50/hr all-in | Unlimited | Accuracy-critical production agents |
| OpenAI Realtime | Multimodal model | Token-priced (variable) | Model-dependent | Multimodal prototyping |
| Deepgram | Own STT | Usage-based | High | Developer-controlled pipelines |
| ElevenLabs | Own STT + best-in-class TTS | Usage-based | ~30 | Voice-output-first agents |
| Vapi / Retell | Bundled STT vendor | Platform + passthrough | Platform-dependent | No-code, standard flows |
Why speech accuracy is the tiebreaker
Here's the part everyone skips, and it's the part that decides production quality.
Pipecat runs an open STT benchmark built from real agent conversations — the messy, name-heavy, number-heavy audio your agents actually field, not scripted clean speech. Lower is better. On word error rate, Universal-3.5 Pro Realtime comes in at 6.99%, against Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%.
Sit with that gap for a second. Roughly half the word errors of the next-closest option on realistic agent audio. Every one of those avoided errors is a "can you repeat that?" that never happens.
Now the number that matters even more for agents: entity error rate — how often the model botches the things that carry the meaning, like names, account numbers, and confirmation codes. Universal-3.5 Pro Realtime lands at 15.31%. Deepgram Flux: 50.50%. When someone reads back a credit card number or spells their last name, that's the metric deciding whether your agent got it right or is about to charge the wrong account.
And there's a lever most stacks don't have. Pass the agent's current question to the transcription model as context — we call it agent_context — and word error rate drops another 10.2%, measured across 20,000 voice agent audio files. The intuition is simple: if the agent just asked "what's your ZIP code?", the model should expect five digits, not a sentence. Context makes it hear better. You can see the full methodology on our benchmarks page and the model details in the Universal-3.5 Pro Realtime announcement.
This is the whole argument in one place. When two APIs both promise low latency and a nice developer experience, accuracy on real audio is what's left to actually choose on.
What you'll pay
Cost in this category is sneaky, because most options make you assemble it yourself.
Wire up a voice agent from separate STT, LLM, and TTS providers and you're metering three usage-based bills, each priced differently — per-minute here, per-token there, per-character somewhere else — and each with its own concurrency terms and overage behavior. Your cost per conversation becomes a function of how chatty the call was, how long the LLM's replies ran, and which tier you tripped into that month. Forecasting it is a spreadsheet exercise, and it drifts.
AssemblyAI's Voice Agent API is a flat $4.50 per hour, all-in — STT, LLM, and TTS on one bill. An hour of conversation costs $4.50 whether the user rambled or kept it tight. That predictability is the point: you can price your own product against it without modeling three vendors' token math. Full numbers are on the pricing page, and new accounts get $50 in free credits to start.
Token-priced options like OpenAI Realtime can look cheap in a demo and then scale in ways that surprise you, because your bill tracks token volume rather than wall-clock conversation time. Neither model is wrong — but know which one you're signing up for before you're at 10,000 calls a day.
Getting started
The developer experience is deliberately boring, in the best way. One WebSocket connection, standard JSON, no SDK to install and no proprietary framework to learn. If you can open a socket, you can build on it — which is also why it works natively with coding agents like Claude Code.
Here's a whole agent:
curl -X POST https://agents.assemblyai.com/v1/agents \
-H "Authorization: $ASSEMBLYAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Quickstart Assistant",
"system_prompt": "You are a friendly assistant having a casual voice conversation. Keep replies short and natural.",
"greeting": "Hey there, what can I help with?",
"voice": { "voice_id": "alba" }
}'
That's it. Name it, prompt it, give it a greeting and a voice, and you've got a running agent on a foundation that hears your users right. Want to go deeper on tools, live configuration, and production patterns? The build guide and the launch announcement walk through the rest.
The best voice agent API is the one that gets the words right the first time, every time, at scale, on one bill. Everything else is a preference. Accuracy is the foundation.
Frequently asked questions
What is the best voice agent API?
The best voice agent API is the one that hears your users most accurately, because speech-to-text is the foundation the entire agent stands on — if the transcript is wrong, the LLM responds to the wrong thing. On Pipecat's open benchmark of real agent conversations, AssemblyAI's Voice Agent API, built on Universal-3.5 Pro Realtime, posts a 6.99% word error rate, roughly half the next-closest option, which is why we'd point accuracy-critical builders there. The "best" for you also depends on your priorities across turn-taking, latency, cost, and concurrency.
What's the difference between a voice agent API and a voice agent platform?
A platform (like Vapi or Retell) gives you an opinionated, often no-code way to design conversations, trading developer control for speed. An API gives you the pipeline and full control over logic, prompts, and the STT foundation your agent hears through. Platforms fit standard flows and non-engineering teams; APIs fit differentiated products where accuracy and customization are load-bearing.
How much does a voice agent API cost?
**** It depends on the pricing model. Assembling STT, LLM, and TTS from separate vendors means three usage-based bills that vary with conversation length and token volume. AssemblyAI's Voice Agent API is a flat $4.50 per hour, all-in on one bill, so cost per conversation is predictable regardless of how the call goes. Token-priced options like OpenAI Realtime vary with usage and can be harder to forecast at scale.
Which voice agent API is most accurate?
**** On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime leads with a 6.99% word error rate versus Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. It's even further ahead on entity error rate — the names, account numbers, and codes that carry meaning — at 15.31% versus Deepgram Flux's 50.50%. Passing the agent's question as context cuts word error rate another 10.2%.
Do I need separate STT, LLM, and TTS providers?
**** No. You can wire up three vendors yourself for maximum control over each layer, but that means orchestrating three systems and reconciling three bills. A unified voice agent API handles the full STT → LLM → TTS pipeline over a single connection, on one bill, which removes the orchestration and cost-forecasting overhead for most teams.
Can I change the agent's behavior mid-conversation?
Yes. AssemblyAI's Voice Agent API supports live configuration — you can update the system prompt, tools, and settings mid-conversation without reconnecting, so the agent can adapt as the call moves between steps without dropping the session.
Top comments (0)