You've decided to build a voice product. Maybe it's a support agent, maybe an AI companion, maybe a voice layer on the app you already shipped. Either way, you're staring at a blank editor and a very fast-moving landscape, and the question in front of you isn't "which model is most accurate." It's "how do I get a working demo by Friday?"
If that's you, this guide is for you. Most people building their first voice product aren't voice agent power users. Roughly 80% are net-new to this. You're demo-driven, you're code-forward, and you don't want to spend your first week learning someone's proprietary conversation-design UI. So let's skip the theory and talk about the decision you're actually making.
What a voice agent API bundles, and why the pipeline is the hard part
A voice agent is a loop. Someone talks, your system transcribes it (speech-to-text), an LLM decides what to say, and a text-to-speech engine says it back. STT to LLM to TTS. That's the whole pipeline.
Sounds simple. It isn't.
The hard part isn't any single box in that diagram. It's the seams between them. When does the user stop talking? (Harder than it sounds - people pause mid-thought all the time.) What happens when they interrupt the agent? How do you keep latency low enough that the conversation doesn't feel like a walkie-talkie? And here's the one that quietly sinks most first builds: if the transcription is wrong, the LLM responds to the wrong thing. Garbage in, confident garbage out.
That last point matters more than founders expect. Your LLM prompt can be perfect, your TTS voice can sound gorgeous, but if the STT hears "Karen" as "Aaron" or drops the last four digits of an account number, your agent is now confidently wrong. Accuracy at the input is the foundation everything else sits on. If you want the deeper version of this argument, we wrote about what actually makes a good voice agent - the short version is that the listening layer is where good agents are won or lost.
A voice agent API bundles that whole pipeline - plus turn detection, interruption handling, and voice activity detection - so you don't have to hand-build the seams. That's the category. The question is how you get one.
The three paths for a startup
There are basically three ways to ship your first voice product. Each has a real trade-off, and the "right" one depends less on your budget than on how much of the plumbing you want to own.
| Path | What you own | Trade-off | Best for |
|---|---|---|---|
| DIY multi-vendor | Separate STT, LLM, and TTS providers wired together yourself | Maximum control, but three vendors, three invoices, and you build turn detection, barge-in, and streaming orchestration | Teams whose differentiator is the pipeline itself |
| Platform (Retell, Vapi, etc.) | A managed environment, often no-code, with built-in telephony | Fast to launch, but opinionated conversation design, awkward custom integrations, and a real ceiling | No-code builders and telephony-first products |
| One API | Your system prompt, tools, and conversation logic — over a single connection | More control than a platform, far less overhead than DIY; you write code | Code-forward startups shipping a first voice product |
Path 1: DIY multi-vendor. You pick an STT provider, an LLM provider, and a TTS provider, and you wire them together yourself. Maximum control. But you're now managing three vendors, three invoices, three sets of rate limits, and three places to debug when a call goes sideways. You also own the hard middle - turn detection, barge-in, streaming orchestration. For a first product, that's a lot of undifferentiated engineering before you've validated that anyone wants the thing.
Path 2: A platform (Retell, Vapi, and similar). These give you a managed environment, often with a no-code builder and built-in telephony. Great for getting something live fast without much code. The catch is the ceiling. Platforms are opinionated about conversation design, custom integrations can get awkward, and - candidly - a lot of platform-built agents end up sounding the same because they're built on the same rails. Vapi in particular is orchestration middleware, which means your agent is only as good as the weakest third-party link in its chain. We wrote more about where voice agent stacks start showing their limits if you want the honest version.
Path 3: One API. A single API that handles STT to LLM to TTS through one connection. You still write your own system prompt, define your own tools, and control the conversation - but you don't manage three vendors or build the streaming plumbing. It's the middle path: more control than a platform, far less overhead than DIY.
For a first voice product, path 3 is usually the sweet spot. You get to spend your time on product logic - the thing that makes your agent yours - instead of on infrastructure nobody will ever thank you for.
What to actually evaluate
Before you pick, here's the short list of things that genuinely move the needle for a first build. Ignore the spec-sheet noise; these five are what you'll feel in production.
| Criterion | Why it matters | AssemblyAI Voice Agent API |
|---|---|---|
| Accuracy | Wrong transcription means the LLM answers the wrong thing — names, numbers, and addresses break flows | Built on Universal-3.5 Pro Realtime: 6.99% pooled WER and 15.31% entity error rate on Pipecat's open STT benchmark |
| Latency | End-to-end round-trip is what makes a conversation feel human | Around one second end-to-end |
| Developer experience | Time from "read the docs" to "talking to an agent" is the whole game for a first build | Standard JSON over one WebSocket, no SDK required; readable in ~10 minutes and works natively with Claude Code |
| Pricing | Predictable pricing lets you model unit economics before the product exists | Flat $4.50/hr billed by the minute — STT, LLM, and TTS included; roughly 4x cheaper than OpenAI's Realtime API (~$18/hr) |
| Lock-in | If you outgrow the tool, how painful is the exit? | A raw API over a standard WebSocket — you keep full control of conversation design, tools, VAD, and turn timing |
Speech accuracy. This is the garbage-in problem from earlier. Look at word error rate on real conversational audio, not clean read-aloud benchmarks, and pay attention to entity accuracy - names, numbers, addresses - because that's what breaks support and booking flows. If you want to go deeper on why the usual metric is shaky, word error rate is broken is a good read.
Latency. End-to-end round-trip time is what makes a conversation feel human or robotic. You want something around a second. Don't obsess over shaving milliseconds off one stage; obsess over the full loop.
Developer experience and time-to-ship. How long from "read the docs" to "talking to a working agent"? For a first-timer this is the whole ballgame. SDK sprawl, dozens of event types, and framework lock-in all add days. A clean JSON API you can read in ten minutes does not.
Pricing model. Is it flat and predictable, or are you doing token math across three invoices plus per-minute STT plus TTS characters? Unpredictable pricing is its own kind of tax when you're trying to model unit economics for a product that doesn't exist yet.
Lock-in. If you outgrow the tool, how painful is the exit? No-code platforms can be sticky in ways that hurt later. An API you call over a standard WebSocket is a lot easier to walk away from.
How AssemblyAI's Voice Agent API maps to that list
Full disclosure: this is our product. I'm going to be specific so you can check the claims, not just take my word.
The Voice Agent API is one WebSocket API that runs the entire STT to LLM to TTS pipeline. You connect to a socket, stream audio in, and get audio back. You write the system prompt. That's the mental model.
On the five criteria:
Accuracy. It's built on Universal-3.5 Pro Realtime, our flagship streaming model, which posts a 6.99% pooled word error rate on Pipecat's open STT benchmark of real agent conversations. For comparison on that same benchmark, entity error rate lands at 15.31% - names, places, phone numbers, the stuff that actually breaks agents. It also takes your agent's question as context, so a mumbled one-word reply or a spelled-out account ID resolves correctly instead of getting mangled.
Latency. Around one second end-to-end. Fast enough to feel like a conversation, not a transaction.
Developer experience. Standard JSON over a WebSocket. No SDK required, no framework to learn. You can read the full API reference in about ten minutes, and most developers have something working the same afternoon. It also works natively with Claude Code - copy the docs, paste them in, and build - which is exactly why we designed it for coding agents.
Here's roughly what connecting looks like - one socket, JSON messages, no ceremony:
const ws = new WebSocket("wss://agents.assemblyai.com/v1/ws", {
headers: { Authorization: `Bearer ${process.env.ASSEMBLYAI_API_KEY}` }
});
ws.on("open", () => {
ws.send(JSON.stringify({ type: "session.update", session: {
system_prompt: "You are a friendly booking assistant for a dental clinic.",
// stream your audio frames in from here
} }));
});
You can also update the system prompt, tools, and settings mid-conversation without reconnecting - handy when the flow branches.
Pricing. Flat $4.50/hr, billed by the minute, with STT, LLM, and TTS all included. One number, one bill, one set of logs. No reconciling three invoices. For context, that's roughly 4x cheaper than OpenAI's Realtime API (around $18/hr) and comes with a cleaner developer experience - their API has 30-plus event types to reason about.
Lock-in. It's a raw API over a standard WebSocket. We call it "invisible infrastructure" for a reason - your customers should feel like you built the thing from scratch, and you keep full control over conversation design, tools, VAD, and turn timing. We're not the agent. We're what you build the agent on.
It currently supports six languages - English, Spanish, French, German, Italian, and Portuguese - and ships drop-in plugins for LiveKit and Pipecat if you're already using those. When you're ready to go from "hello world" to something real, how to build with the Voice Agent API walks the full pipeline.
When a platform is the better call
I said this would be an honest guide, so here's the honest part. A one-API approach is not always right, and sometimes a platform genuinely is the better choice.
Reach for a platform like Retell or Vapi if:
- You want no-code. If you're a non-technical founder or you need business users to build and tweak agents without touching a WebSocket, a visual builder is worth a lot. Our API assumes you can write code.
- You need built-in telephony today. If your product is fundamentally about placing and receiving phone calls and you want that handled out of the box right now, platforms have mature telephony baked in. We're shipping telephony as a fast-follow, but "today" matters when you're shipping.
- Call routing and hosted orchestration are the product. If the routing, queueing, and call-center-style workflow is your value, a platform gives you that scaffolding for free.
There's no shame in starting on a platform to validate demand and moving to an API when you hit the ceiling. Just go in knowing where that ceiling is.
And if the real question is DIY versus one API? Unless stitching STT, LLM, and TTS together is itself your differentiator, one API replaces three vendors, three invoices, and three debugging surfaces. For a first product, that's almost always the right trade.
Quickstart and a live demo to close
Here's the thing about voice products: reading about them tells you very little. Talking to one tells you everything. Accuracy, latency, turn-taking, the feel of an interruption landing cleanly - you catch all of it in about thirty seconds of conversation.
So the move is simple. Grab a key, wire up the socket, point it at a system prompt, and talk to it. If you can talk to it and it feels right, you're most of the way to a demo.
The best voice agent API for your first product is the one that gets you to that working demo fastest without boxing you in later. For most code-forward startups, that's a single API with strong accuracy at the input, predictable pricing, and an exit that isn't a trap.
Frequently asked questions
What's the best voice agent API for a startup building its first voice product?
The best one is whatever gets you to a working demo fastest without locking you in. For code-forward startups, that usually means a single API that bundles STT, LLM, and TTS over one connection - so you write product logic instead of managing three vendors. AssemblyAI's Voice Agent API fits that profile: one WebSocket, standard JSON, flat pricing, and around one-second end-to-end latency.
What's the cheapest voice agent API?
Watch out for pricing that looks cheap per stage but adds up across three invoices (STT per minute, LLM per token, TTS per character). A flat rate is easier to model. AssemblyAI's Voice Agent API is $4.50/hr all-in - STT, LLM, and TTS included - which is roughly 4x cheaper than OpenAI's Realtime API at around $18/hr. Cheapest on paper isn't always cheapest in practice; predictability matters when you're modeling unit economics.
One API or a platform like Retell or Vapi - which should I choose?
Choose a platform if you want no-code building, need telephony out of the box today, or if hosted call routing is your actual product. Choose a single API if you want full control over conversation design and integrations without hitting a platform ceiling, and you're comfortable writing code. Platforms trade complexity for constraints; an API keeps you in control but expects you to build.
How good does the developer experience really need to be?
For a first voice product, it's the whole game. The gap between an API you can read in ten minutes and one with dozens of event types and required SDKs is measured in days of your life. AssemblyAI's Voice Agent API uses standard JSON over a WebSocket with no SDK required, works natively with Claude Code, and lets you update prompts and tools mid-conversation without reconnecting.
Which languages does the Voice Agent API support?
Six today: English, Spanish, French, German, Italian, and Portuguese. If you need broader coverage for pre-recorded workflows, Universal-3.5 Pro handles native code-switching across 18 languages on the async side.
How do I get started?
Open the API reference, connect to the WebSocket, and stream audio at a system prompt. Most developers have a working agent the same afternoon.
Top comments (0)