DEV Community: Mart Schweiger

Agent Context Carryover for Voice Agents on LiveKit

Mart Schweiger — Wed, 12 Aug 2026 15:00:48 +0000

If you've built a voice agent, you know the failure mode. The agent asks "What's your email?" and the caller says "j-dot-oconnor at gmail" — and the transcript comes back "j dot o connor at gmail dot com," or worse. Short replies, spelled-out entities, proper nouns, menu items, street names: these are exactly the moments where a caller expects the agent to just get it, and exactly where streaming speech-to-text tends to slip.

The usual fix is to feed the model hints — key terms, custom vocabulary, a running list of nouns you expect to hear. It works, but it's plumbing you have to build and maintain. With Agent Context Carryover on Universal-3.5 Pro Realtime, running on LiveKit, you skip the plumbing entirely. Flip one setting and the model transcribes each user turn while aware of what your agent just said. In this guide we'll build a working voice agent that puts it to the test on some deliberately hard-to-transcribe input.

What Agent Context Carryover actually does

Context carryover lets the speech-to-text model stay aware of the conversation while it transcribes new input. Universal-3.5 Pro Realtime preserves both sides of the dialogue:

Agent context — your assistant's most recent spoken reply.
User context — prior finalized user turns from the session.

That bidirectional awareness is what moves the needle on accuracy, especially for short answers and spelled-out entities. When your agent asks "What's your email address?", the model can anticipate the shape of the answer and format "user@assemblyai.com" correctly instead of producing "user at assemblyai dot com."

Here's the part that matters for how you build: on LiveKit, the AssemblyAI plugin feeds that agent context in automatically. There's no extra wiring to capture the agent's last message and shuttle it back to the transcriber. You enable one parameter and the plugin does the rest — no key-term lists to predefine, no context relay to maintain. As of this release, AssemblyAI is the only speech-to-text provider offering this kind of drag-and-drop context carryover.

What we'll build

We'll stand up a LiveKit voice agent from the official Python starter, wire in AssemblyAI for speech-to-text, enable Agent Context Carryover, and then stress-test it as a restaurant takeout line — a fictional spot called Le Périgord Fusion whose menu is a minefield of proper nouns: escargots à la bourguignonne, khachapuri adjaruli, bánh xèo, gnocchi alla sorrentina, mille-feuille, kouign-amann, yuzu kombucha. If the model can order all of that without a single custom vocabulary entry, it can handle your product names too.

Prerequisites

Python 3.10+ and uv installed
A free AssemblyAI API key
A LiveKit Cloud project (for LIVEKIT_URL, API key, and secret)
livekit-agents 1.6.6 or newer (needed for automatic agent-context forwarding)

Step 1 — Spin up the LiveKit Python starter

LiveKit publishes a starter project that scaffolds a complete Python agent — a voice pipeline on LiveKit Inference (Gemma for the LLM, Cartesia for TTS), the LiveKit turn detector, background noise cancellation, and an eval suite. The quickest way to create one is with the LiveKit CLI:

lk cloud auth
lk agent init my-agent --template agent-starter-python

The CLI clones the template and writes your LiveKit credentials into .env.local. Prefer to do it by hand? Clone the repo, install dependencies with uv, and copy the env template yourself:

git clone https://github.com/livekit-examples/agent-starter-python
cd agent-starter-python
uv sync
cp .env.example .env.local

Step 2 — Install the AssemblyAI plugin for LiveKit

Add the AssemblyAI plugin to the project:

uv add "livekit-agents[assemblyai]"

This gives you the assemblyai plugin namespace you'll use to configure speech-to-text in the agent.

Step 3 — Configure Universal-3.5 Pro and enable context carryover

Open the agent entrypoint at src/agent.py. Out of the box, the starter builds its AgentSession with Deepgram as the speech-to-text provider through LiveKit Inference:

# Before — the starter's default STT line
session = AgentSession(
    stt=inference.STT(model="deepgram/nova-3", language="multi"),
    # ... llm, tts, turn detection
)

Swap that one line for the AssemblyAI plugin and point it at the universal-3-5-pro model. Add the import at the top of the file — everything else in the session (LLM, TTS, turn detection) stays exactly as it was:

from livekit.plugins import assemblyai

# After — AssemblyAI STT with context carryover
session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    # ... llm, tts, turn detection unchanged
)

The agent_context and previous_context_n_turns parameters below apply only to universal-3-5-pro. That's the whole integration. On livekit-agents 1.6.6 and newer, AgentSession automatically forwards each assistant reply into the model's agent_context — the carryover happens without another line of code. (If a reply runs past the 1,750-character limit, the plugin truncates it silently and keeps the trailing content, which is usually the question you care about.)

Want the model primed before the caller says anything? Seed it with your agent's opening line:

stt = assemblyai.STT(
    model="universal-3-5-pro",
    agent_context="Hello, how can I help you today?",
)

Automatic forwarding takes over after the first real assistant reply.

Step 4 — Add your API key and run

Drop your AssemblyAI key into .env.local alongside your LiveKit credentials:

ASSEMBLYAI_API_KEY=your_api_key_here
LIVEKIT_URL=your_livekit_url
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret

Then start the agent in dev mode:

uv run python src/agent.py dev

Connect to your agent, say hello, and confirm it responds. Now the fun part.

Putting it to the test

With the agent live as our Le Périgord Fusion takeout line, we ordered straight down the menu — appetizer, drink, dessert — using the hardest names on it. The exchange went like this:

Agent: Welcome to Le Périgord Fusion. Would you like to place an order, or may I read through our menu for you?

Caller: Yeah, what do you have for appetizers?

Agent: We have escargots à la bourguignonne for $14.50, khachapuri adjaruli for $13, bánh xèo for $11.75, and gnocchi alla sorrentina for $12.25…

Caller: Let's do escargots à la bourguignonne… and a yuzu kombucha… and a kouign-amann.

Every item — French, Georgian, Vietnamese, Italian — came back transcribed correctly, and the order summary matched exactly: one escargots à la bourguignonne, one yuzu kombucha, one kouign-amann, subtotal $29.25. No custom vocabulary. No key-term list. The only thing we turned on was Agent Context Carryover, and because the model could see what the agent had just recited, it knew which unusual words to expect when the caller echoed them back.

The build actually splits the call across two agents — an order-taker that runs the menu, and a checkout host it hands off to for delivery and payment. That handoff is where the next hard case showed up: the caller read out a full delivery address, "1214 Madison Avenue, New York, 10012," and a payment method, and the checkout host read both back verbatim in its confirmation. Street addresses and spoken numbers are exactly the kind of input that trips up streaming transcription, and here the model's awareness of what the agent had just asked ("Please provide your full delivery address") is what kept the digits and the street name intact.

Tuning it: the parameters worth knowing

Automatic forwarding covers most builds, but two parameters give you finer control when you need it.

Parameter	Type	What it does
agent_context	string	Your agent's most recent spoken reply, up to 1,750 characters. Set at construction or updated mid-stream via update_options.
previous_context_n_turns	integer	How many prior conversation turns are carried forward automatically. Range 0–100; server default is 5. Connect-time only — it can't be changed mid-session.

A couple of rules to keep in mind: an explicit agent_context over 1,750 characters raises a ValueError (only automatic forwarding truncates silently), and previous_context_n_turns is fixed at connection time.

If you're on a version older than 1.6.6, or you want to manage the agent context yourself, disable automatic forwarding and push updates on each assistant turn:

from livekit.agents import AgentSession, ConversationItemAddedEvent
from livekit.plugins import assemblyai

AGENT_CONTEXT_MAX_CHARS = 1750

session = AgentSession(
    stt=assemblyai.STT(model="universal-3-5-pro"),
    stt_context_options={"forward_chat_context": False},
)

@session.on("conversation_item_added")
def _on_conversation_item_added(ev: ConversationItemAddedEvent) -> None:
    if ev.item.type != "message" or ev.item.role != "assistant":
        return

    agent_stt = session.stt
    if not isinstance(agent_stt, assemblyai.STT):
        return

    spoken = ev.item.text_content
    if not spoken:
        return

    agent_stt.update_options(agent_context=spoken[-AGENT_CONTEXT_MAX_CHARS:])

Why this matters for production voice agents

Transcription accuracy is the foundation every downstream step in a voice agent stands on — intent detection, tool calls, order summaries, CRM writes. A misheard proper noun early in a call cascades into the wrong action later. Agent Context Carryover attacks that at the source, and it does it without the maintenance burden of key-term prompting or a hand-rolled context relay. You get the accuracy gain on proper nouns and key terms, and you get it by flipping a switch.

LiveKit sees the same thing from the infrastructure side:

"We're excited to make AssemblyAI's Universal-3.5 Pro available on LiveKit Inference. What really stands out is their pace of innovation with Context Carryover — it intelligently applies conversation context to improve transcription accuracy in a way most speech models don't, removing the need for users to predefine key terms."

— David Zhao, Co-founder at LiveKit

Try it on your own agent

If you're already building on LiveKit, Agent Context Carryover is available today — turn it on and hear the difference on your hardest inputs. Follow the four steps above, and point it at the proper nouns your callers actually say. For the full parameter reference, see the Context Carryover docs, and if you're new to this stack, start with our guide to building a voice agent with LiveKit.

Frequently asked questions

What is Agent Context Carryover?

Agent Context Carryover is a feature of AssemblyAI's Universal-3.5 Pro Realtime model that lets the speech-to-text model transcribe each user turn while aware of the conversation so far — specifically the agent's most recent reply and prior finalized user turns. That context improves accuracy on short replies, proper nouns, and spelled-out entities like emails.

How is it different from key-term prompting or custom vocabulary?

Key-term prompting requires you to predefine the words you expect and maintain that list. Context carryover works from the live conversation instead, so on LiveKit it needs no predefined terms and no extra plumbing — the plugin forwards the agent's context automatically.

Do I need to write code to forward the agent's context on LiveKit?

No. On livekit-agents 1.6.6 and newer, AgentSession forwards each assistant reply into the model's agent_context automatically. You only write forwarding code if you're on an older version or want to manage the context manually.

Which model supports it?

The agent_context and previous_context_n_turns parameters apply only to universal-3-5-pro (Universal-3.5 Pro Realtime).

Is there a limit on how much context I can pass?

Yes. agent_context accepts up to 1,750 characters. Setting an explicit value over that limit raises a ValueError; automatic forwarding instead truncates silently, keeping the trailing content. previous_context_n_turns ranges from 0 to 100 (default 5) and is set at connection time only.

Does it work with speech-to-text providers other than AssemblyAI?

As of this release, AssemblyAI is the only speech-to-text provider offering drag-and-drop context carryover on LiveKit.

OpenAI Realtime API Alternatives in 2026 (How to Migrate)

Mart Schweiger — Wed, 12 Aug 2026 15:00:41 +0000

OpenAI's Realtime API made "talk to your app" easy to prototype: one model, speech in, speech out, low latency. If you're building voice agents in 2026, you've probably tried it.

Prototyping and shipping are different problems, though. Once real users start calling, the single-model architecture that demos well starts working against you — on cost, on transcription accuracy, and on the details of managing a real conversation. That's what sends teams looking for an OpenAI Realtime API alternative that holds up in production.

This post does two things. First, a clear-eyed rundown of the alternatives in 2026 — Gemini Live, ElevenLabs, Deepgram, a couple of open-source options, and AssemblyAI's Voice Agent API. Then a concrete migration walkthrough, because a list isn't much use if nobody tells you how to actually switch.

Why teams look for an OpenAI Realtime alternative

Let's be specific about what pushes people to look.

Cost you can't predict. OpenAI Realtime is token-priced. The current flagship, gpt-realtime-2.1, runs roughly $32 per million audio input tokens and $64 per million audio output tokens. At the base rate a call lands around $0.05 per minute — fine on a spreadsheet. But conversation context gets re-processed each turn, so on longer calls real bills commonly run 2x to 5x that base rate. Prompt caching helps, but now you're maintaining caching logic just to keep a voice bot affordable. There's a cheaper gpt-realtime-2.1-mini at about $0.016 per minute, but you trade accuracy for the discount. The through-line: your cost per call climbs with call length, and you can't quote a customer a number without a footnote.

Speech accuracy on real audio. This is the architectural crux. OpenAI Realtime is one multimodal model that also does voice, which means transcription isn't a dedicated step — it's one job the model does among many. In AssemblyAI's own side-by-side testing, OpenAI Realtime hallucinated on messy input, inventing words that were never said. On a support line or a drive-through, that's not a rounding error — that's a wrong order.

Barge-in and turn quality. Interruption handling is what separates a conversation from a walkie-talkie. In that same testing, OpenAI Realtime's barge-in logic triggered on background noise — a cough was enough to make it stop talking. An agent that yields the floor to a sneeze is an agent users don't come back to.

Thirty-plus event types. The Realtime API exposes 30+ event types you subscribe to and emit — session updates, input buffer commits, response deltas, and so on. Every one is a thing to handle, a thing to get wrong, and a thing to keep working as the API evolves.

Lock-in. One model doing everything means you can't swap the transcription layer, the reasoning layer, or the voice independently. When one piece underperforms, you don't have a knob — you have a migration.

The best OpenAI Realtime API alternatives in 2026

The right pick depends on what you're optimizing for. Short, factual rundowns, then a table.

AssemblyAI Voice Agent API

Instead of one model doing everything, the Voice Agent API wires together dedicated models for each step — speech-to-text, LLM, and text-to-speech — behind a single WebSocket and a single bill. Pricing is flat at $4.50/hr all-in, so cost doesn't drift with call length. End-to-end latency is around 1 second, it covers 6 languages (EN/ES/FR/DE/IT/PT), and concurrency is unlimited. It's built on Universal-3.5 Pro Realtime, which is the accuracy story below. Best for: production voice agents where transcription accuracy and predictable cost matter most.

Google Gemini Live

Gemini Live is Google's real-time multimodal offering — native audio, video input, and tight integration if you're already in Google's ecosystem. Architecturally it's the same shape as OpenAI Realtime: one large multimodal model, usage-based pricing, and the same "transcription is a side effect" tradeoff, so it inherits the same accuracy and cost-predictability questions. Best for: teams already standardized on Google Cloud who need audio plus vision in one model.

ElevenLabs Conversational AI

ElevenLabs is known for its text-to-speech, and its Conversational AI product leans on that voice-output quality. On the input side — the part that decides whether your agent hears the caller correctly — its Scribe v2 model lands at 9.76% WER on Pipecat's benchmark, behind the category leaders, and concurrency is capped (around 30), which becomes a constraint at scale. Best for: applications where expressive TTS output is the top priority and input accuracy is secondary. (More in our AssemblyAI vs. ElevenLabs comparison.)

Deepgram Voice Agent

Deepgram offers a single-API Voice Agent product in the same shape as AssemblyAI's. The differentiator is accuracy on real agent audio: its newer Flux model comes in at 15.58% WER on the Pipecat benchmark, well behind the leaders, and the gap widens on entity error — getting names, numbers, and addresses right — where Flux reaches 50.50%. Best for: teams already invested in Deepgram's stack. (Full breakdown in our AssemblyAI vs. Deepgram comparison.)

Open-source (Moshi, Qwen-Omni)

If you want to own the whole stack and self-host, the open-source options are worth a look. Kyutai's Moshi is a full-duplex speech model you can run yourself; Alibaba's Qwen-Omni is a multimodal model you can wire into orchestration frameworks like Pipecat or LiveKit. You avoid a per-minute fee, but you take on the hosting, scaling, reliability, and accuracy-tuning yourself. Best for: teams with ML infrastructure that need on-prem or full customization and can staff the maintenance.

Here's how they stack up:

Alternative	Pricing model	Speech accuracy (Pipecat WER)	Developer experience	Concurrency
OpenAI Realtime	Token-based, variable (~$0.05/min base, often 2x–5x)	Prone to hallucination on messy audio	30+ event types	Rate-limited
AssemblyAI	Flat $4.50/hr all-in	6.99%	One WebSocket + JSON	Unlimited
Gemini Live	Token-based, variable	One-model tradeoff	Multimodal API	Rate-limited
ElevenLabs	Usage-based	9.76% (Scribe v2)	Single API	Plan-tiered
Deepgram	Usage-based	15.58% (Flux)	Single API	Plan-tiered
Open-source	Self-hosted (infra cost)	Varies by model	DIY orchestration	Your hardware

Why AssemblyAI is the closest drop-in for production

The wedge is simple. OpenAI Realtime is one multimodal model that also does voice. The Voice Agent API is dedicated models per step, assembled for conversation quality. That difference shows up in four places that matter when real calls are coming in.

Accuracy that holds on real audio. On Pipecat's open STT benchmark — lower is better — Universal-3.5 Pro Realtime posts 6.99% WER. For context: Google Chirp3 sits at 9.04%, ElevenLabs Scribe v2 at 9.76%, and Deepgram Flux at 15.58%. The gap widens on entities — the names, order numbers, and addresses your business actually runs on. Universal-3.5 Pro Realtime's entity error rate is 15.31% versus Deepgram's 50.50%. When a caller says "my confirmation is A-as-in-apple 4 7 2," that's the number that decides whether the call works. More on the model in the Universal-3.5 Pro Realtime launch post.

Context that raises accuracy further. You can feed the agent live context — product names, customer vocabulary, whatever your domain needs. Across 20,000 files, using agent_context cut WER by 10.2%. That's a tuning knob a single-model architecture doesn't expose.

One WebSocket instead of 30+ events. No SDK required, standard JSON over a WebSocket. You're not subscribing to a taxonomy of event types — you send audio and handle responses. That's a big reason the Voice Agent API plays well with coding agents like Claude Code, which can scaffold an integration without wrestling a sprawling event model.

Predictable everything. Flat $4.50/hr all-in — STT, LLM, and TTS through one bill — so a 2-minute call and a 20-minute call cost what you'd expect. Around 1-second end-to-end latency. Unlimited concurrency, so a traffic spike doesn't mean a support ticket. And you can change configuration mid-conversation instead of tearing down and rebuilding a session. For where these differences bite, we wrote up where voice agent stacks start showing their limits.

Want to hear it before you read another word of code? Talk to a Live Agent — it's the fastest way to judge latency and interruption handling for yourself.

How to migrate from OpenAI Realtime to AssemblyAI

The good news: if you built on OpenAI Realtime, you already have the hard parts — audio capture, a client transport, and turn logic. Migration is mostly re-pointing them at a simpler surface.

Map the concepts

Here's how the mental model translates:

OpenAI Realtime	AssemblyAI Voice Agent API
session.update with dozens of fields	Create an agent once (name, prompt, greeting, voice), then connect
30+ typed events to emit and subscribe to	One WebSocket; send audio, receive messages
Instructions passed per session	system_prompt set on the agent, adjustable live mid-conversation
Manual VAD / turn-detection config	Turn detection handled server-side
Tool/function-calling via event schema	Tool-calling wired to your agent, invoked in the conversation flow
Audio buffers you commit and clear	Stream raw audio frames over the socket

The shift in one sentence: instead of configuring a session with a wall of parameters and events, you define an agent as a durable object, then open a realtime connection to it.

Create the agent

Define it once with a single API call:

curl -X POST https://agents.assemblyai.com/v1/agents \
-H "Authorization: $ASSEMBLYAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Quickstart Assistant",
"system_prompt": "You are a friendly assistant having a casual voice conversation. Keep replies short and natural.",
"greeting": "Hey there, what can I help with?",
"voice": { "voice_id": "alba" }
}'

That returns an agent_id. To go live, you connect to it over a realtime WebSocket by referencing that agent_id — and the same agent works unchanged from a server, a browser, or a phone. Define once, connect anywhere. For a fuller build, walk through how to build with the Voice Agent API.

The audio gotcha worth knowing before you ship

This one trips people up. For best accuracy, feed the model raw audio. If you're capturing in a browser, disable noiseSuppression and autoGainControl in your getUserMedia constraints — but keep echoCancellation on. Aggressive client-side processing is tuned to make audio sound nice to humans, and in doing so it smears the very signal a transcription model relies on. You want the microphone's honest output going over the wire, not a cleaned-up version that's lost detail. Turning that processing off is often the single biggest accuracy win in a migration.

Sessions and resumption

You don't rebuild an agent to change its behavior. Because configuration is live, you can adjust the system prompt or swap context mid-conversation without tearing down the connection — handy for escalating tone, injecting a looked-up account detail, or handing off between skills. And since the agent is a persistent object rather than an ephemeral session, reconnecting a dropped call is a matter of connecting to the same agent_id again, not reconstructing state from scratch.

That's the whole migration. Point your existing audio pipeline at one WebSocket, define your agent once, fix your browser audio constraints, and you're running.

Try it

You don't have to take the benchmark numbers on faith.

Talk to a Live Agent and judge latency and interruption handling with your own voice.
Read the docs to see the full agent and WebSocket reference.
Check pricing to confirm the flat $4.50/hr math against your current OpenAI bill.

Frequently asked questions

What is the best alternative to the OpenAI Realtime API?

It depends on what you're optimizing for, but for production voice agents where transcription accuracy and predictable cost matter most, AssemblyAI's Voice Agent API is the closest drop-in. It uses dedicated models per step behind one WebSocket, posts 6.99% WER on Pipecat's benchmark, and bills at a flat $4.50/hr. Gemini Live fits teams deep in Google Cloud, and ElevenLabs fits cases where TTS output quality outweighs input accuracy.

How much does the OpenAI Realtime API cost?

It's token-priced and variable. The flagship gpt-realtime-2.1 runs about $32 per million audio input tokens and $64 per million output tokens — roughly $0.05 per minute at the base rate. Because conversation context gets re-processed each turn, real bills commonly run 2x to 5x higher on longer calls. There's a cheaper gpt-realtime-2.1-mini at around $0.016 per minute. The catch is predictability: your cost per call climbs with call length.

Is there a cheaper alternative to OpenAI Realtime?

"Cheaper" is less useful than "predictable." AssemblyAI's Voice Agent API is a flat $4.50/hr all-in — STT, LLM, and TTS on one bill — so cost doesn't drift with call length the way token-based pricing does. Open-source options like Moshi or Qwen-Omni have no per-minute fee, but you take on the hosting, scaling, and accuracy-tuning costs yourself.

How do I migrate from OpenAI Realtime to AssemblyAI?

Reuse your existing audio capture and transport. Create an agent once with a single POST (name, system prompt, greeting, voice), then connect to the returned agent_id over a realtime WebSocket. Replace the 30+ event-type handling with a single send-audio/receive-message loop. One migration tip: if you capture audio in the browser, disable noiseSuppression and autoGainControl while keeping echoCancellation on, since aggressive client-side processing hurts transcription accuracy.

What's the difference between OpenAI Realtime and a cascaded voice pipeline?

OpenAI Realtime is one multimodal model that handles speech-to-text, reasoning, and speech output all at once — but transcription is just one job among many, which is where hallucination on messy audio creeps in. A cascaded (dedicated-models-per-step) pipeline like AssemblyAI's routes each stage to a purpose-built model, so you get category-leading transcription accuracy, the ability to feed in live context, and the option to tune each layer independently — all still behind one WebSocket.

Best Voice Agent API (2026): Accuracy, Cost & Options

Mart Schweiger — Wed, 12 Aug 2026 15:00:15 +0000

Search "best voice agent API" and you'll get a beauty contest. Everyone's the fastest, everyone's the easiest, everyone has the slickest dashboard. It's noise. And it points you at the wrong question.

Here's the question that actually decides whether your voice agent works in production: does it hear your user correctly? Because a voice agent is a pipeline — speech-to-text feeds the LLM, the LLM decides what to say, and text-to-speech says it. If the first step gets the words wrong, everything downstream is confidently answering a question the user never asked. The model can be brilliant. The voice can be gorgeous. Doesn't matter. Garbage in, garbage out — at the speed of conversation.

So the best voice agent API isn't the flashiest one. It's the one built on the most accurate foundation. Let's talk about how to actually evaluate that, and where the real options land.

What "best voice agent API" actually means

When most people picture a voice agent, they picture the LLM — the "brain." That's the fun part. But the brain only ever sees text. It never hears your customer. It reads a transcript that some speech-to-text model produced a few hundred milliseconds ago, and it acts on that transcript as if it were ground truth.

Think about what that means when someone says their name is "Nguyen," reads back a 16-digit account number, or talks through a drive-thru with road noise in the background. If the STT mishears "Nguyen" as "when," the agent doesn't know it misheard. It just runs with the wrong input. The user has to repeat themselves, the conversation stalls, and trust evaporates.

That's why we frame the whole category around accuracy. The foundation of any voice agent is how well it hears — because that's the one error every other component inherits and none of them can fix. Pick your API on that basis and most of the "which platform is prettiest" debate stops mattering.

The criteria that decide production quality

A demo that works once in a quiet room tells you almost nothing. Here's what separates a voice agent that ships from one that gets pulled after week one:

Speech accuracy on real audio. Not clean, scripted benchmark clips — messy conversational audio with names, account numbers, accents, cross-talk, and background noise. This is the tiebreaker, and we'll come back to it.
Turn detection. Knowing when the user has actually finished speaking versus just pausing mid-thought. Get this wrong and your agent either interrupts constantly or sits there dead-air awkward.
Interruption and barge-in handling. Real conversations aren't turn-based. People cut in. A production agent has to stop talking the instant the user does and pick up the new thread.
Latency. End-to-end, mouth to ear. Above roughly a second and a half, the conversation starts feeling like a bad phone connection and people talk over the agent.
Billing model. One unified bill, or three vendors you're metering and reconciling separately? This is a cost-predictability question as much as an accounting one.
Concurrency and scale. Can you run 500 calls at once during a Monday-morning spike, or are you hitting a concurrency cap and rate limits?
Developer experience. Standard protocols and clear docs, or an SDK maze and a proprietary conversation-design tool you have to bend your app around?

Weight these however your use case demands. But notice that accuracy sits underneath most of them — bad transcripts make turn detection harder, force retries that add latency, and generate the "sorry, can you repeat that?" loops that wreck the experience.

API vs platform: which are you actually buying?

This is the fork in the road, and it's worth being honest about.

No-code platforms — Vapi, Retell, Synthflow — give you an opinionated way to design a conversation. Drag-and-drop flows, hosted orchestration, a dashboard for non-engineers. If you're a small team that needs a working phone agent by Friday and your logic is fairly standard, that's a genuinely good fit. No shame in it. You're trading control for speed, and sometimes speed is exactly what you need.

But here's where it gets interesting. Those same platforms bundle a speech-to-text vendor you often can't choose, wrap your logic in their conversation model, and constrain what you can customize when your requirements get weird — and in production, requirements always get weird. You inherit their accuracy, their latency budget, and their abstractions. When voice agent stacks start showing their limits, it's usually because the thing you most need to tune is the thing the platform hid from you.

A voice agent API gives you the pipeline and the control. You own the logic, the prompts, the integration, and — critically — the foundation your agent hears through. That's the path if you're building something differentiated, if accuracy is load-bearing, or if you just don't want a vendor's opinions baked into your product.

The reframe: don't ask "which platform wins." Ask "am I buying a control surface or a black box?" — and then, whichever you pick, ask what it's hearing your users through.

How the main options compare

Let's be objective about the field. Every option here is a real, capable product. They just optimize for different things.

AssemblyAI Voice Agent API. One WebSocket handles the full STT → LLM → TTS pipeline, built on Universal-3.5 Pro Realtime for speech accuracy. Flat $4.50/hour all-in, one bill. Around one second end-to-end latency, unlimited concurrency, no rate limits, and standard JSON over WebSocket with no SDK required. You can update the system prompt, tools, and settings mid-conversation without reconnecting. It's invisible infrastructure — the foundation your agent stands on, not another dashboard to learn.

OpenAI Realtime. A multimodal model that also does voice. But that's the framing to keep in mind: it's a model that speaks, not a pipeline engineered end-to-end for conversation quality. It's token-priced, so your cost varies with usage in ways that are hard to forecast, and you're assembling the production concerns — turn-taking, barge-in, scale — yourself.

Deepgram. Ships a unified voice agent API. . The differentiator comes down to accuracy on real agent audio, which is where the benchmark numbers below get pointed. We put the full breakdown in our AssemblyAI vs Deepgram comparison.

ElevenLabs. If lifelike TTS is your top priority, they're have a good option. The tradeoffs are concurrency (capped around 30, which matters at scale) and, on the input side, transcription accuracy. More in our Voice Agent API vs ElevenLabs Conversational AI writeup.

Vapi and Retell. The leading no-code platforms. Fast to a working agent, great for standard flows and non-engineering teams. You trade control and vendor choice for that speed, per the section above.

One more thing worth naming: LiveKit and Pipecat are open-source pipeline frameworks, not competitors. We ship drop-in plugins for both. If you're already building on them, you can slot AssemblyAI in as the STT layer without rearchitecting anything.

Option	Foundation	Pricing	Concurrency	Best for
AssemblyAI Voice Agent API	Universal-3.5 Pro Realtime	Flat $4.50/hr all-in	Unlimited	Accuracy-critical production agents
OpenAI Realtime	Multimodal model	Token-priced (variable)	Model-dependent	Multimodal prototyping
Deepgram	Own STT	Usage-based	High	Developer-controlled pipelines
ElevenLabs	Own STT + best-in-class TTS	Usage-based	~30	Voice-output-first agents
Vapi / Retell	Bundled STT vendor	Platform + passthrough	Platform-dependent	No-code, standard flows

Why speech accuracy is the tiebreaker

Here's the part everyone skips, and it's the part that decides production quality.

Pipecat runs an open STT benchmark built from real agent conversations — the messy, name-heavy, number-heavy audio your agents actually field, not scripted clean speech. Lower is better. On word error rate, Universal-3.5 Pro Realtime comes in at 6.99%, against Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%.

Sit with that gap for a second. Roughly half the word errors of the next-closest option on realistic agent audio. Every one of those avoided errors is a "can you repeat that?" that never happens.

Now the number that matters even more for agents: entity error rate — how often the model botches the things that carry the meaning, like names, account numbers, and confirmation codes. Universal-3.5 Pro Realtime lands at 15.31%. Deepgram Flux: 50.50%. When someone reads back a credit card number or spells their last name, that's the metric deciding whether your agent got it right or is about to charge the wrong account.

And there's a lever most stacks don't have. Pass the agent's current question to the transcription model as context — we call it agent_context — and word error rate drops another 10.2%, measured across 20,000 voice agent audio files. The intuition is simple: if the agent just asked "what's your ZIP code?", the model should expect five digits, not a sentence. Context makes it hear better. You can see the full methodology on our benchmarks page and the model details in the Universal-3.5 Pro Realtime announcement.

This is the whole argument in one place. When two APIs both promise low latency and a nice developer experience, accuracy on real audio is what's left to actually choose on.

What you'll pay

Cost in this category is sneaky, because most options make you assemble it yourself.

Wire up a voice agent from separate STT, LLM, and TTS providers and you're metering three usage-based bills, each priced differently — per-minute here, per-token there, per-character somewhere else — and each with its own concurrency terms and overage behavior. Your cost per conversation becomes a function of how chatty the call was, how long the LLM's replies ran, and which tier you tripped into that month. Forecasting it is a spreadsheet exercise, and it drifts.

AssemblyAI's Voice Agent API is a flat $4.50 per hour, all-in — STT, LLM, and TTS on one bill. An hour of conversation costs $4.50 whether the user rambled or kept it tight. That predictability is the point: you can price your own product against it without modeling three vendors' token math. Full numbers are on the pricing page, and new accounts get $50 in free credits to start.

Token-priced options like OpenAI Realtime can look cheap in a demo and then scale in ways that surprise you, because your bill tracks token volume rather than wall-clock conversation time. Neither model is wrong — but know which one you're signing up for before you're at 10,000 calls a day.

Getting started

The developer experience is deliberately boring, in the best way. One WebSocket connection, standard JSON, no SDK to install and no proprietary framework to learn. If you can open a socket, you can build on it — which is also why it works natively with coding agents like Claude Code.

Here's a whole agent:

curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Quickstart Assistant",
    "system_prompt": "You are a friendly assistant having a casual voice conversation. Keep replies short and natural.",
    "greeting": "Hey there, what can I help with?",
    "voice": { "voice_id": "alba" }
  }'

That's it. Name it, prompt it, give it a greeting and a voice, and you've got a running agent on a foundation that hears your users right. Want to go deeper on tools, live configuration, and production patterns? The build guide and the launch announcement walk through the rest.

The best voice agent API is the one that gets the words right the first time, every time, at scale, on one bill. Everything else is a preference. Accuracy is the foundation.

Frequently asked questions

What is the best voice agent API?

The best voice agent API is the one that hears your users most accurately, because speech-to-text is the foundation the entire agent stands on — if the transcript is wrong, the LLM responds to the wrong thing. On Pipecat's open benchmark of real agent conversations, AssemblyAI's Voice Agent API, built on Universal-3.5 Pro Realtime, posts a 6.99% word error rate, roughly half the next-closest option, which is why we'd point accuracy-critical builders there. The "best" for you also depends on your priorities across turn-taking, latency, cost, and concurrency.

What's the difference between a voice agent API and a voice agent platform?

A platform (like Vapi or Retell) gives you an opinionated, often no-code way to design conversations, trading developer control for speed. An API gives you the pipeline and full control over logic, prompts, and the STT foundation your agent hears through. Platforms fit standard flows and non-engineering teams; APIs fit differentiated products where accuracy and customization are load-bearing.

How much does a voice agent API cost?

**** It depends on the pricing model. Assembling STT, LLM, and TTS from separate vendors means three usage-based bills that vary with conversation length and token volume. AssemblyAI's Voice Agent API is a flat $4.50 per hour, all-in on one bill, so cost per conversation is predictable regardless of how the call goes. Token-priced options like OpenAI Realtime vary with usage and can be harder to forecast at scale.

Which voice agent API is most accurate?

**** On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime leads with a 6.99% word error rate versus Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. It's even further ahead on entity error rate — the names, account numbers, and codes that carry meaning — at 15.31% versus Deepgram Flux's 50.50%. Passing the agent's question as context cuts word error rate another 10.2%.

Do I need separate STT, LLM, and TTS providers?

**** No. You can wire up three vendors yourself for maximum control over each layer, but that means orchestrating three systems and reconciling three bills. A unified voice agent API handles the full STT → LLM → TTS pipeline over a single connection, on one bill, which removes the orchestration and cost-forecasting overhead for most teams.

Can I change the agent's behavior mid-conversation?

Yes. AssemblyAI's Voice Agent API supports live configuration — you can update the system prompt, tools, and settings mid-conversation without reconnecting, so the agent can adapt as the call moves between steps without dropping the session.

The Hard Cases in Speaker Diarization: Overlap & Noise

Mart Schweiger — Wed, 12 Aug 2026 15:00:09 +0000

Speaker diarization looks solved right up until it isn't. Point a decent model at a clean two-person phone call and it'll nail who said what, turn after turn, with almost no effort. That's the demo everyone shows.

Now drop the same model into a dinner party. Four people, a cross-talk moment, somebody laughing mid-sentence, a "yeah, exactly" fired off while the host is still finishing a thought. The clean-call model falls apart—and it falls apart in specific, nameable ways.

This post is that list. Not "diarization is hard, be careful," but the actual taxonomy: the exact cases that break systems, why each one breaks them, and—the part almost nobody writes down—how you'd even know it's failing. Because here's the uncomfortable part: the metric most people quote barely notices some of the worst failures.

Why diarization is still hard

Diarization is two jobs stapled together. First, figure out how many distinct speakers exist. Second, draw the boundaries that say person A talked from here to here, then person B took over. Do both perfectly and you get a clean speaker-labeled transcript. Miss on either and the errors compound.

On a clean call, both jobs are easy. Two clearly different voices, one talking at a time, generous pauses between turns—the acoustic signal practically labels itself. If you want the full mechanics of how this works under the hood, we wrote a complete explainer on speaker diarization that walks through the pipeline.

Real conversations don't cooperate. People interrupt. They react. They talk over each other. They mumble from across a room. And the moment reality gets messy, the failures sort into a handful of recurring categories. Let's name them.

Overlapped speech

Two people talking at once is the single biggest source of diarization error. Full stop.

Here's why it's so brutal. Most diarization systems are built on an assumption baked deep into their architecture—that at any given instant, exactly one person is speaking. That assumption makes the math tractable. It also makes overlap a blind spot, because when two voices land in the same slice of audio, the system has to pick one. The other speaker's words just vanish.

Some systems handle overlap by masking it—detecting the overlapped region and excluding it from analysis so it doesn't corrupt the clustering. That keeps the speaker labels cleaner, sure. But think about what you just did: you threw away the audio where two people were talking. Every word in that region is gone from the transcript. For a meeting-summarization use case or a call-center QA workflow, those are often the words that matter most—the interruption, the objection, the correction.

So what does good look like? A model that treats overlap as a first-class event, not noise to be filtered. It should attribute overlapping speech to the right speakers and keep both sets of words, instead of picking a winner and dropping the loser. That's a real bar, and most legacy pipelines don't clear it.

Short turns and back-channels

"Yep." "No." "Right." A quick laugh. A "wait, what?" thrown in while someone else holds the floor.

These are back-channels—the tiny verbal reactions that keep a conversation moving—and they wreck diarization in a sneaky way. The system credits them to whoever was already speaking. Person A is mid-sentence, Person B says "exactly," and the transcript hands that "exactly" to Person A. The turn was too short and too embedded to trigger a speaker switch, so it gets swallowed.

Now here's the part that should genuinely change how you evaluate diarization. You might never see this failure in your metrics.

The standard metric, DER (diarization error rate), weights errors by seconds of audio. A back-channel is maybe 300 milliseconds. Misattribute it and DER shrugs—it barely moves. So you can have a system that systematically misfiles every short reaction in your data and still posts a DER that looks great on a slide.

We ran the numbers on this and the gap is almost comical. In our own testing, a catastrophic attribution output—words credited to the wrong people all over the place—scored just 15.1% DER. Sounds fine, right? The same output scored 30.7% cpWER. Twice as bad by the metric that actually tracks whether each person's words landed in the right place.

It cuts the other way too. A flawless transcript—every word right, every word attributed correctly—scored 51.5% DER but 0.0% cpWER. DER called a perfect result a failure. If you're optimizing for DER, you're optimizing for the wrong thing, and short turns are exactly where that bites. We broke this whole comparison down in DER vs cpWER if you want the full teardown.

Background noise and far-field audio

Not every conversation happens through a headset an inch from someone's mouth. A lot of the most valuable audio is captured in the worst possible conditions.

Think about a drive-thru: engine noise, wind, a speaker box, someone in the back seat. Or a self-service kiosk in a busy lobby. Or a conference room where the mic is in the center of the table and half the speakers are eight feet away. Or a phone on speaker sitting on a desk. This is far-field audio, and it stacks two problems at once.

First, the target voices are quieter and smeared by room reverberation, which blurs the acoustic fingerprint the system uses to tell speakers apart. Second, background noise and background speech—a TV, a nearby conversation, ambient chatter—give the model extra "voices" to trip over. It can invent speakers that don't exist or fail to separate the ones that do.

The fix isn't just a better microphone you don't control. It's a model that can suppress background speech and noise before it corrupts the speaker labels—tuned for whether the audio is near-field or far-field, because those are genuinely different acoustic problems. A single-mic room recording and a multichannel setup call for different strategies too; if you've got separate audio channels per speaker, multichannel diarization sidesteps a lot of this by design.

Getting the speaker count wrong

Every diarization output has a hidden claim baked into it: this conversation had N speakers. Get N wrong and everything downstream is wrong too.

There are two ways to blow it. Merging is when the system collapses two people into one—two similar-sounding colleagues become "Speaker A," and now half of one person's words are filed under the other. Splitting is the opposite: one person's voice shifts—they get excited, they lower their voice, the line quality changes—and the system decides that's a new person, so "Speaker A" and "Speaker C" are actually the same human.

Similar-sounding voices make merging worse. Two men with comparable pitch and cadence, or two people with the same regional accent, sit close together in the model's internal space, and the clustering can't pull them apart.

This is why you need a roster metric—speaker count error—that asks a blunt question: did the system get the number of people right? It's separate from word-level accuracy, and it catches structural failures that a per-word score can smear across the whole transcript. Merges and splits are cheap to make and expensive to ignore.

How to tell if your diarization is actually failing

By now the theme should be clear. The failures that hurt most—short turns, misattribution, cross-talk—are the ones DER is worst at seeing. So if you only track DER, you're flying blind on your hardest cases.

Here's the evaluation stack that actually works, in two numbers.

cpWER (concatenated minimum-permutation word error rate). For each person in the conversation, what fraction of their words did the system get wrong—misrecognized, dropped, or credited to someone else? You concatenate everything each speaker said, find the speaker-mapping that gives the best possible match, and score the errors. Because it's grounded in words, not seconds, a misattributed "exactly" costs exactly as much as it should. cpWER is the number that punishes the failures your users will actually notice.

Speaker count error. The roster metric from the last section—did we get the number of people right? Track it alongside cpWER so structural merges and splits can't hide.

That's the honest scorecard: cpWER for word-level attribution, speaker count error for the roster. If you want to run this yourself, we published a companion walkthrough on measuring cpWER in Python so you can score your own audio instead of trusting a vendor's slide.

How modern models handle the hard cases

Naming the failures is one thing. Building around them is another. Here's how we approach it.

The biggest architectural shift is refusing to treat transcription and diarization as separate steps. Universal-3.5 Pro produces the transcript and the speaker labels jointly—one model, one pass—instead of transcribing first and then bolting speaker labels on afterward. That joint modeling is exactly what lets it catch the short turns, rapid back-and-forth, and overlapped speech that break two-stage pipelines. And it's optimized for cpWER, not DER, because that's the metric that reflects real quality.

The numbers back it up. Universal-3.5 Pro averages 30.17 cpWER against Deepgram Nova-3 English at 37.92, ElevenLabs Scribe v2 at 35.26, and Gladia at 36.87. It runs async at $0.21/hour. Getting started is about as light as it gets:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

config = aai.TranscriptionConfig(speaker_labels=True)
transcript = aai.Transcriber().transcribe("./meeting.wav", config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

Real-time is where diarization gets genuinely thorny, because you can't cluster speakers you haven't heard yet. When someone new talks 30 seconds in, a naive streaming system either guesses early and gets it wrong or stalls waiting for certainty.

Universal-3.5 Pro Realtime handles this with streaming diarization plus revision. It labels speakers live as the audio comes in, then—once it has the full picture—re-clusters and sends a single correction within about half a second of the stream ending. You get low-latency labels during the call and a cleaned-up roster the moment it's over, handling up to 10 speakers. We wrote up the mechanics in streaming speaker diarization and the major streaming upgrade that made this possible.

For the far-field and noise problems, voice_focus with near_field and far_field settings suppresses background speech and noise before it can poison the speaker labels—the difference between a drive-thru transcript that's usable and one that's full of phantom speakers.

Want to see it on your own audio? Poke at it in the playground, read the speaker diarization feature page for the full spec, or just wire it into your stack and run your worst files through it.

Frequently asked questions

What are the limitations of speaker diarization?

The main limitations of speaker diarization are overlapping speech, short turns and back-channels, background noise and far-field audio, speaker-count errors, and similar-sounding voices. Overlapping speech is the single biggest source of error, because most systems assume one person talks at a time and drop words when two voices collide. Short reactions like "yep" or "no" get credited to the wrong speaker, and noisy or far-field audio blurs the acoustic differences the system relies on. Modern joint transcription-and-diarization models built and measured for cpWER handle these cases far better than legacy two-stage pipelines.

What is overlapping speech?

Overlapping speech is when two or more people talk at the same time, so their voices land in the same slice of audio. It's the #1 source of diarization error because most systems are built to attribute each moment to a single speaker, which forces them to pick one voice and lose the other's words. Systems that mask overlap keep cleaner speaker labels but delete the transcribed words in the overlapped region—often the interruptions and corrections that matter most.

How does background noise affect speaker diarization?

Background noise degrades diarization by masking the acoustic features a system uses to tell speakers apart and by adding stray sounds it can mistake for extra speakers. Far-field audio—kiosks, drive-thrus, conference rooms, speakerphones—makes it worse, since distance and reverberation blur each voice's fingerprint. Noise suppression tuned for near-field versus far-field capture, like AssemblyAI's voice_focus setting, strips out background speech and noise before it corrupts the speaker labels.

Why do diarization systems get the number of speakers wrong?

Diarization systems miscount speakers in two ways: merging and splitting. Merging collapses two similar-sounding people into one speaker, while splitting treats one person's changing voice—excitement, volume shifts, line-quality changes—as a brand-new speaker. Similar-sounding voices make merging especially common. Tracking speaker count error as a dedicated roster metric is the only reliable way to catch these structural mistakes.

How do you measure speaker diarization accuracy?

Measure diarization accuracy with cpWER (concatenated minimum-permutation word error rate) plus speaker count error, not DER alone. cpWER asks, for each person, what fraction of their words the system got wrong—misrecognized, dropped, or credited to someone else—so it catches short-turn misattribution that DER's seconds-based weighting misses. In our testing a catastrophic attribution output scored just 15.1% DER but 30.7% cpWER, while a flawless transcript scored 51.5% DER yet 0.0% cpWER—proof that DER can grade a perfect result as a failure and a broken one as fine.

Does Whisper Do Speaker Diarization? How to Add It

Mart Schweiger — Wed, 12 Aug 2026 14:59:43 +0000

Does Whisper do speaker diarization?

No. Whisper is OpenAI's automatic speech recognition (ASR) model — it turns audio into text, and that's the whole job. It doesn't tell you who spoke. Hand it a two-person interview and you'll get an accurate transcript with zero speaker labels, no "Speaker A" and "Speaker B," nothing that separates one voice from another.

So if you want speaker diarization — the "who spoke when" layer on top of the words — you have to bolt it on yourself. Most people reach for pyannote.audio (or WhisperX, which wraps a similar idea). That works. But it's a pipeline you assemble and own, and the seams show up exactly where diarization is hardest: short turns, rapid back-and-forth, and overlapping speech.

Let's build the DIY version first, then measure it honestly.

How people add diarization to Whisper

The standard recipe has three stages. Transcribe with Whisper to get words and timestamps. Run a separate diarization model — pyannote — to get speaker segments. Then align the two: for each word, figure out which speaker segment it belongs to.

Here's the shape of it in Python. This is illustrative — the point is the structure, not a production-ready alignment engine:

# pip install openai-whisper pyannote.audio torch
import whisper
from pyannote.audio import Pipeline

# 1. Transcribe with Whisper (word-level timestamps)
asr = whisper.load_model("large-v3")
result = asr.transcribe("meeting.wav", word_timestamps=True)

# 2. Diarize with pyannote (needs a free Hugging Face access token)
diarizer = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token="<HF_TOKEN>",
)
diarization = diarizer("meeting.wav")

# 3. Align: for each Whisper word, find the diarization segment its
#    midpoint falls in, and attach that speaker label. (You write this glue.)

Looks clean. Three steps, a few dozen lines. But here's where it gets interesting — every one of those steps hides real friction.

The pyannote models are gated. You need a free Hugging Face account, you have to accept the model's user conditions, and you generate an access token to pass into from_pretrained. Miss that and the download fails. It's not hard, but it's a step, and it's a credential you now manage.

You'll want a GPU. Whisper's large-v3 and pyannote both run painfully slow on CPU. For anything beyond a short clip, you're provisioning and maintaining a GPU — locally or in the cloud — plus the CUDA and PyTorch versions that keep them happy.

There's voice-activity detection in the mix too. pyannote leans on VAD to decide where speech is before it decides who's speaking, and tuning that on noisy or far-field audio is its own rabbit hole.

And then there's the alignment glue — step 3, the innocent-looking comment. That's your code. You're mapping Whisper's word timestamps onto pyannote's speaker segments, deciding what to do when a word straddles a boundary, what to do when the diarizer and the ASR disagree about where speech even is. The midpoint heuristic in the snippet is the naive version. It works until it doesn't, and when it doesn't, you own the bug.

None of this is a knock on Whisper or pyannote. They're solid open-source tools, and for an offline project or a weekend build, this pipeline is genuinely fine. The question is what happens to accuracy when the conversation gets messy — and how you'd even know.

Where the cracks show: measuring it with cpWER

Most diarization tutorials measure quality with DER — diarization error rate. DER asks, roughly, what fraction of audio time got assigned to the wrong speaker. It's the classic academic metric, and it has a blind spot big enough to drive a truck through: it barely notices short turns, and it can wave through catastrophic attribution errors.

The metric that actually reflects what your users experience is cpWER — concatenated minimum-permutation word error rate. Here's the plain-English version: for each person in the conversation, what fraction of their words did the system get wrong? Wrong meaning misrecognized, dropped, or credited to the wrong speaker. You concatenate everything each speaker actually said, compare it against what the system attributed to them (under the best speaker-to-speaker matching), and count the damage. It folds transcription errors and attribution errors into one honest number.

Why does the choice of metric matter so much? Because DER and cpWER can tell completely different stories about the same output. In AssemblyAI's testing, a catastrophic attribution failure — a case where speakers got badly swapped — scored a tidy 15.1% DER while landing at 30.7% cpWER. The DER made it look like a B-plus. The cpWER told the truth: nearly a third of the words were attributed to the wrong person. If you're reading a transcript, that's the difference between "mostly right" and "who said that?"

Now think about where the Whisper + pyannote pipeline is most likely to break.

Short turns. When someone jumps in with a quick "Right," "No, wait," or "Exactly," that turn might be half a second long. The midpoint-alignment heuristic has almost no signal to work with, and the interjection frequently gets swallowed into the neighboring speaker's block. DER shrugs — it's a sliver of audio. cpWER counts every one of those words as misattributed.

Overlap. Two people talking at once is the reality of real meetings, debates, and calls. A pipeline that runs VAD, then diarization, then alignment tends to collapse overlapped speech into a single speaker — one voice wins, the other's words vanish or get reassigned. Those dropped and misattributed words hit cpWER directly.

Speaker merging. On long recordings, a diarizer can decide two people are one, folding one speaker's turns into another. DER, again, can under-penalize this depending on the timing. cpWER makes the merge obvious, because a whole speaker's worth of words landed under the wrong label.

The uncomfortable takeaway: you can stand up the Whisper + pyannote pipeline, run DER, see a respectable number, and ship something that mangles exactly the moments people care about most. If you want to measure this properly on your own audio, AssemblyAI's companion walkthrough on measuring cpWER in Python shows how, and the DER vs. cpWER breakdown explains why the metric you pick changes the verdict.

The managed alternative: diarization built into the model

Here's the other way to do this. Instead of stitching an ASR model to a diarization model and writing the glue between them, you call one API that produces the transcript and the speaker labels together.

AssemblyAI's Universal-3.5 Pro is the async flagship, and it's built for exactly this. It generates the transcript and speaker labels jointly, and it's optimized for cpWER — the honest metric — not just DER. Because diarization is produced alongside the words rather than aligned after the fact, it holds up on the hard cases: short turns, rapid back-and-forth, and overlapped speech.

Here's the whole thing in Python:

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assembly.ai/wildfires.mp3"  # or "./meeting.wav"

config = aai.TranscriptionConfig(speaker_labels=True)

transcript = aai.Transcriber().transcribe(audio_file, config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

That's pip install assemblyai and a single speaker_labels=True flag. No Hugging Face token. No gated model downloads. No alignment heuristic to write and debug. No GPU to provision, patch, or keep alive. The utterances come back already labeled by speaker, in order, ready to print or store.

On price, Universal-3.5 Pro runs $0.21/hr, and standard diarization is an additional $0.02/hr — a rounding error against the cost of running your own GPU box and maintaining the pipeline around it. On accuracy, the numbers hold up where it counts: Universal-3.5 Pro averages 30.17 cpWER, compared with 37.92 for Deepgram Nova-3 English, 35.26 for ElevenLabs Scribe v2, and 36.87 for Gladia. Lower is better, and the gap is largest exactly on the short-turn and overlap cases that a DIY pipeline fumbles. The full Universal-3.5 Pro async release has the details.

This isn't magic — it's just diarization treated as a first-class part of the model instead of a post-processing step you assemble. And it means the messy moments in real conversations get attributed correctly instead of collapsed.

Whisper + pyannote vs. Universal-3.5 Pro: when to use which

Both are legitimate choices. Here's the honest comparison.

Factor	Whisper + pyannote	Universal-3.5 Pro (async)
Setup	Multiple libraries, gated models, HF token	pip install assemblyai, one API call
Diarization	Separate model, aligned after transcription	Produced jointly with the transcript
Short turns	Often swallowed by neighboring speaker	Captured
Overlapping speech	Frequently collapsed to one speaker	Handled
Optimized for	DER (typical tutorials)	cpWER — the honest metric
Avg cpWER	Varies; you tune and measure it yourself	30.17
Infra	GPU strongly recommended, you manage it	None — fully managed
Alignment code	You write and maintain it	None
Cost	"Free" software + your GPU + your time	$0.21/hr + $0.02/hr diarization
Offline / air-gapped	Yes	No (API)
Support	Community / GitHub issues	Docs, playground, support

So when should you reach for each?

Go with Whisper + pyannote if you need everything to run offline or air-gapped, if you're doing research or a hobby build where the pipeline itself is the point, if you have a GPU sitting idle, and if cost pressure means you'd rather spend engineering time than dollars. It's a fine tool for that.

Go with Universal-3.5 Pro if accuracy on real conversations matters — short turns, cross-talk, overlap — if you don't want to own a GPU and an alignment script, if you'd rather ship in an afternoon than tune VAD for a week, and if you want a number you can trust because it's measured with cpWER. For most production speech-to-text work, that's the trade that pays off.

The thing is, most teams reach for Whisper + pyannote because it looks free and turns out to be expensive — in GPU time, in glue code, and in the accuracy you quietly lose on the hardest turns. Priced against that, a managed API optimized for the honest metric is usually the cheaper answer.

Next steps

If you're building diarization into a product, the fastest way to compare is to run the same audio through both. Grab a messy clip — an interview with interruptions, a meeting with cross-talk — and see which one attributes the short turns correctly.

Try it live in the playground — paste a file, get labeled speakers, no code.
Read the speaker diarization docs for the async API details.
See the speaker diarization feature overview for how it fits into a full pipeline.
Compare options in our roundup of the top speaker diarization libraries and APIs and the top free speech-to-text APIs and open-source engines.

Frequently asked questions

Does Whisper support speaker diarization?

No. Whisper is an ASR model — it transcribes speech to text but doesn't identify who's speaking. To get speaker labels you have to pair it with a separate diarization model like pyannote.audio, or use a managed API that does both jointly.

How do you add speaker diarization to Whisper?

The common approach is a three-stage pipeline: transcribe with Whisper for word-level timestamps, run pyannote.audio (or WhisperX) to get speaker segments, then align each word to a speaker turn with your own glue code. pyannote's models are gated, so you'll need a free Hugging Face token, and a GPU is strongly recommended.

Is Whisper + pyannote free?

The software is open source, so there's no license fee. But "free" is misleading — you'll typically need a GPU to run it at any real speed, plus engineering time to build and maintain the alignment step, manage credentials, and tune voice-activity detection. The dollars move from a per-hour bill to your infrastructure and your team's hours.

How accurate is Whisper speaker diarization?

Whisper itself is a strong transcriber; the diarization accuracy depends on pyannote and your alignment code. Measured with cpWER — the metric that counts misattributed words — DIY pipelines tend to struggle on short turns, rapid back-and-forth, and overlapping speech, precisely the cases DER can hide. Always measure with cpWER, not just DER.

What's the difference between DER and cpWER?

DER (diarization error rate) measures the fraction of audio time attributed to the wrong speaker, and it under-penalizes short turns and attribution swaps. cpWER (concatenated minimum-permutation word error rate) measures, per speaker, the fraction of their words that were misrecognized, dropped, or credited to someone else. In one AssemblyAI test, a bad attribution output scored just 15.1% DER but 30.7% cpWER — same output, very different story.

What's a simpler alternative to Whisper + pyannote?

AssemblyAI's Universal-3.5 Pro async model produces the transcript and speaker labels together from a single API call — no Hugging Face token, no alignment script, no GPU to manage. It's optimized for cpWER, averages 30.17 cpWER, and costs $0.21/hr plus $0.02/hr for standard diarization.

How to Measure Diarization Accuracy with cpWER in Python

Mart Schweiger — Wed, 12 Aug 2026 14:59:37 +0000

A hands-on runbook: compute cpWER in Python to see how good your speaker diarization really is — then score a Universal-3.5 Pro transcript end to end.

Here's a scorer you can paste into a notebook right now:

import numpy as np
import jiwer
from scipy.optimize import linear_sum_assignment

def _error_count(ref, hyp):
    ref = ref.strip() or "<empty>"
    hyp = hyp.strip() or "<empty>"
    out = jiwer.process_words(ref, hyp)
    return out.substitutions + out.deletions + out.insertions

def cpwer(reference, hypothesis):
    """
    reference, hypothesis: dict mapping speaker label -> that speaker's full text.
    Returns (cpWER, best_mapping). cpWER is invariant to how speakers are named.
    """
    ref_speakers = list(reference.keys())
    hyp_speakers = list(hypothesis.keys())
    n = max(len(ref_speakers), len(hyp_speakers))
    ref_padded = ref_speakers + [None] * (n - len(ref_speakers))
    hyp_padded = hyp_speakers + [None] * (n - len(hyp_speakers))

    cost = np.zeros((n, n), dtype=int)
    for i, r in enumerate(ref_padded):
        for j, h in enumerate(hyp_padded):
            cost[i, j] = _error_count(reference.get(r, ""), hypothesis.get(h, ""))

    rows, cols = linear_sum_assignment(cost)
    total_errors = cost[rows, cols].sum()
    total_ref_words = sum(len(t.split()) for t in reference.values())
    mapping = {ref_padded[i]: hyp_padded[j] for i, j in zip(rows, cols)}
    return total_errors / max(total_ref_words, 1), mapping

That's the whole thing. By the end of this post you'll know exactly what it computes, why it beats the metric most people reach for, and how to point it at a real AssemblyAI transcript. Let's dig in.

Why "who spoke when" needs its own metric

Plain word error rate is great at one job: did the system get the words right? Take a two-person interview, transcribe it perfectly, and standard WER can read 0% even if the model credited every one of the host's sentences to the guest. WER never looks at the speaker labels, so speaker mistakes are invisible to it. That's a problem the second your product shows "Speaker A" and "Speaker B" to a user.

Our research team wrote the full argument for why cpWER is the honest way to measure speaker diarization and why DER quietly lies — read the DER-vs-cpWER pillar if you want the deep theory. This post is the other half: the code. So rather than send you off to read a definition, I'll restate it here and then we'll build it.

What cpWER actually measures

cpWER stands for concatenated minimum-permutation word error rate. Three moving parts, and the name tells you all of them.

Concatenated. For each speaker, glue all their words into one stream — one blob of text per person, in both the reference and the hypothesis.

Minimum-permutation. Your system's "Speaker A" isn't guaranteed to be the reference's "Speaker A." So try every way of matching hypothesis speakers to reference speakers and keep the pairing that produces the fewest total word errors.

Word error rate. Under that best pairing, compute a normal WER across everything.

In plain English: for each person in the conversation, what fraction of their words did the system get wrong — misrecognized, dropped, or handed to the wrong speaker? One number, and it only goes up when something a human would actually notice goes wrong.

Now contrast that with DER, the diarization error rate. DER scores seconds of audio, and it scores things a user never sees: silent pauses, segment padding, non-speech annotations. The gap between the two is not subtle. In our testing, a flawless transcript scored 51.5% DER but 0.0% cpWER — DER screaming failure at a perfect result. Flip it around and a catastrophic speaker-attribution output — words shoved onto the wrong people — scored just 15.1% DER but 30.7% cpWER. DER shrugged at output that was genuinely broken. If you're evaluating speech recognition models, that's the exact kind of inversion you can't afford.

Setting up

One install line:

pip install jiwer scipy numpy

jiwer does the word-level alignment, scipy gives you the optimal assignment solver, and numpy holds the cost matrix. That's it — no GPU, no models, no audio processing.

What you actually need to run an evaluation:

A reference transcript with speaker labels — your ground truth, shaped as {speaker: text}.
A system hypothesis in the same shape — whatever your diarizer produced.

And here's an underrated part of the pitch. To annotate for cpWER, a human just needs to write down who said what — a speaker-labeled transcript. To annotate for DER, someone has to mark frame-level speech boundaries down to the fraction of a second. That transcript-only labeling runs roughly 10x cheaper than DER's frame-level labels, which matters a lot when you're building an eval set by hand.

Computing cpWER step by step

Look back at the scorer. It does three things in order.

Per-speaker concatenation. The reference and hypothesis are dicts of {speaker: text}, so the concatenation is already done by the time text lands in the dict — one string per speaker.

The assignment step. This is the clever bit. We build an n x n cost matrix where cost[i, j] is the number of word errors you'd get if reference speaker i were matched to hypothesis speaker j. Then linear_sum_assignment — the Hungarian algorithm — finds the pairing that minimizes the total across the diagonal. Padding with None handles the case where the two sides have a different speaker count (the model found three speakers, the truth has two). That's the "minimum-permutation" in the name, done efficiently instead of by brute force.

Scoring with jiwer. Each cell's cost comes from jiwer.process_words, summing substitutions, deletions, and insertions. Total errors under the best mapping, divided by total reference words, gives you cpWER.

Now the payoff — a worked example that proves the metric doesn't care what you name your speakers:

reference = {
    "A": "hey did you finish the report",
    "B": "yeah i sent it this morning",
}

# Same words, correct attribution — but the labels are swapped.
hypothesis = {
    "spk_1": "yeah i sent it this morning",
    "spk_2": "hey did you finish the report",
}

score, mapping = cpwer(reference, hypothesis)
print(f"cpWER: {score * 100:.1f}%")   # cpWER: 0.0%
print(mapping)                         # {'A': 'spk_2', 'B': 'spk_1'}

Words correct, labels flipped, and the score is 0.0%. The assignment step lines A up with spk_2 and B with spk_1, then finds zero word errors. That's label-invariance — exactly what you want, because "Speaker A" is an arbitrary tag, not a fact about the world.

Break the attribution instead of the naming and the score moves. Merge both people into one speaker and cpWER jumps toward the high 80s; introduce a single misrecognized word and you'll see it tick up by one word's worth of error. The metric tracks the mistakes a listener would actually catch — nothing more, nothing less. If you've ever felt burned by how word error rate hides real failures, this is the diarization-aware fix.

Scoring a real transcript with AssemblyAI

Now let's feed it something real. AssemblyAI's speaker diarization runs on Universal-3.5 Pro, our async flagship at $0.21/hr, which produces the transcript and the speaker labels jointly and is optimized for cpWER — not DER. It's tuned to catch short turns, rapid back-and-forth, and overlapped speech, which is where diarizers usually fall apart. (Standard async diarization adds $0.02/hr.)

Install the SDK and transcribe with speaker labels on:

pip install assemblyai

import assemblyai as aai

aai.settings.api_key = "<YOUR_API_KEY>"

audio_file = "https://assembly.ai/wildfires.mp3"  # or a local path like "./meeting.wav"

config = aai.TranscriptionConfig(speaker_labels=True)

transcript = aai.Transcriber().transcribe(audio_file, config)

for utterance in transcript.utterances:
    print(f"Speaker {utterance.speaker}: {utterance.text}")

The full parameter list lives in the AssemblyAI docs.

The transcript comes back as a list of utterances, each with a speaker tag and text. To score it, fold those utterances into the {speaker: text} shape our scorer expects — concatenate every utterance per speaker:

from collections import defaultdict

hypothesis = defaultdict(str)
for utterance in transcript.utterances:
    hypothesis[utterance.speaker] += " " + utterance.text

# `reference` is your ground-truth transcript in the same {speaker: text} shape
score, mapping = cpwer(reference, dict(hypothesis))
print(f"cpWER: {score * 100:.1f}%")

That's the entire loop: transcribe, group by speaker, score against your ground truth. The mapping tells you which model speaker the scorer matched to each reference speaker, which is handy when you want to eyeball a specific person's errors.

Reading your score: what's good and the traps

So you've got a number. Now what?

For context, Universal-3.5 Pro posts an average cpWER of 30.17. Where do competitors land on the same measure? Deepgram Nova-3 English sits at 37.92, ElevenLabs Scribe v2 at 35.26, and Gladia at 36.87. Lower is better, and the spread between providers is real — this is the kind of gap you can only see once you're measuring the thing users actually experience.

A few things to keep in mind as you interpret your own runs:

Short turns and overlap dominate the error. A one-word "yeah" dropped in the middle of someone else's sentence is brutal for any diarizer, and those moments contribute a disproportionate share of cpWER. If your score is high, listen to the rapid back-and-forth sections first — that's usually where it's coming from.

Formatting choices swing DER but not cpWER. This is the trap that sends people down rabbit holes. In our testing, changing a single segmentation setting — max_gap — swung DER from 47.8% to 14.5% on identical model output. Toggling whether laughter was included in the reference swung DER from 23.4% to 1.1%. Same words out of the model, wildly different DER, purely from annotation and segmentation knobs. cpWER doesn't budge for any of that, because it scores words, not seconds. When your metric moves and the transcript didn't, that's DER noise — not a real quality change.

Compare like with like. cpWER is text-normalization sensitive (casing, punctuation, numbers). Keep your normalization identical across reference and hypothesis, and identical across the providers you're comparing, or you'll be measuring your preprocessing instead of the model.

Next steps

You've now got a self-contained, tested cpWER scorer and a way to point it at production diarization output. From here:

Build a small labeled eval set from your own audio — remember, transcript-only annotation is about 10x cheaper than DER labeling, so this is more achievable than it sounds.
Run the same set through multiple providers and rank them on cpWER, not DER or plain WER. If you're surveying options, our roundup of speaker diarization libraries and APIs is a good starting map.
Kick the tires without writing any setup code in the AssemblyAI playground, then wire the SDK into your pipeline.

When you're ready to score a real transcript, run the two code blocks above end to end. Measure the thing your users actually see — then go make it better.

Frequently asked questions

What is cpWER?

cpWER is concatenated minimum-permutation word error rate. It concatenates each speaker's words into one stream, finds the reference-to-hypothesis speaker mapping that minimizes total word error, and reports the word error rate under that best mapping. In plain terms: for each person, what fraction of their words did the system get wrong — misrecognized, dropped, or credited to the wrong speaker.

What's the difference between cpWER and DER?

cpWER scores words a listener actually reads; DER scores seconds of audio, including things users never see like silent pauses, segment padding, and non-speech annotations. The two can disagree dramatically — a flawless transcript scored 51.5% DER but 0.0% cpWER in our testing, while a badly mis-attributed one scored just 15.1% DER but 30.7% cpWER. cpWER is also cheaper to annotate, since it needs only a speaker-labeled transcript.

How do you measure speaker diarization accuracy in Python?

Represent your reference and hypothesis as {speaker: text} dicts, then use jiwer for word-level alignment and scipy's linear_sum_assignment to find the best speaker mapping. The cpwer() function in this post does exactly that — install with pip install jiwer scipy numpy and pass in your two dicts.

Does AssemblyAI support speaker diarization?

Yes. Set speaker_labels=True in your TranscriptionConfig and Universal-3.5 Pro returns per-utterance speaker labels alongside the transcript. It's optimized for cpWER and built to handle short turns, rapid back-and-forth, and overlapped speech. Standard async diarization adds $0.02/hr.

Why is diarization error rate (DER) misleading?

Because DER moves when the words don't. In our testing, changing one segmentation setting swung DER from 47.8% to 14.5% on identical model output, and toggling whether laughter was in the reference swung it from 23.4% to 1.1%. Those are annotation and formatting artifacts, not quality changes. cpWER ignores them because it scores words instead of seconds.

How much labeled data do I need to compute cpWER?

Just a speaker-labeled transcript of your audio — who said what, in order. You don't need frame-level speech boundaries, which is why cpWER annotation runs roughly 10x cheaper than DER annotation. Start with a handful of representative clips and grow the set as you compare providers.

AssemblyAI vs NVIDIA Parakeet & Canary for Production

Mart Schweiger — Wed, 05 Aug 2026 12:54:23 +0000

NVIDIA's speech models are having a moment, and it's earned. Parakeet TDT and Canary post excellent numbers on standard ASR benchmarks, they're fast, and if your stack already lives on NVIDIA GPUs and NIM, running them feels like the path of least resistance. Teams evaluating clinical and conversational transcription are right to put them on the shortlist.

So this isn't a takedown. It's the comparison you actually need before you build a product on one of them — because benchmark accuracy and production accuracy are different things, and in regulated, entity-heavy domains like healthcare, the difference is the whole ballgame.

Quick comparison

	NVIDIA Parakeet / Canary	AssemblyAI (Universal-3.5 Pro)
Delivery	Open weights you host (NIM / your GPUs)	Managed API, or self-hosted in your VPC
Cost model	GPU hours + ops + eval pipeline	$0.21/hr, per second, no minimums
Medical tuning	Build it yourself	Medical Mode (medical-v1), +$0.15/hr
Entity accuracy	Strong on clean benchmarks	Best-in-market on drugs, numbers, names
Diarization	Separate system	Joint transcript + speaker, cpWER-optimized
Compliance	Your responsibility	BAA available without a sales call, SOC 2, EU residency
Support	Community / your team	Forward-deployed engineers

Where NVIDIA is genuinely strong

Give the models their due. On clean, single-speaker English benchmarks, Parakeet and Canary are among the best open models available, and their inference speed on NVIDIA hardware is hard to beat. If you're doing high-volume offline batch on infrastructure you already own and operate, that combination is real leverage.

The question is what happens when the audio stops being clean and the domain starts being regulated.

Real-world clinical audio isn't a benchmark

A clinician dictating over a noisy ward, a patient with an accent describing symptoms, two voices overlapping across an exam room — this is the audio a medical scribe actually hears, and it's nothing like a read-speech leaderboard. Two things decide whether a transcript is safe to use downstream: getting the entities right and getting the speakers right.

Medical entity accuracy. Drug names, dosages, conditions, and procedures are exactly the tokens where a single error is clinically meaningful. AssemblyAI's Medical Mode — turned on with a single "domain": "medical-v1" parameter, no model switch — reduces the missed entity rate on drugs, conditions, and procedures by roughly 20%. And because Universal-3.5 Pro takes contextual prompting, feeding a patient's prior-visit note cut missed medical terms by 31% in our internal testing, even when the note came from an earlier visit. Getting that behavior out of a raw Parakeet checkpoint is a research project you'd own.

Diarization. Attributing each utterance to the right speaker is its own hard problem. Universal-3.5 Pro produces the transcript and speaker boundaries jointly and leads our published cpWER benchmarks across meetings, telephony, and conversational audio. Bolting a separate diarizer onto an NVIDIA checkpoint gives you the brittle timestamp-alignment approach that falls apart on the interruptions clinical audio is full of.

Compliance is a feature, not a footnote

For healthcare teams, the model is half the decision. The other half is whether you can legally put protected health information through it.

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the services to process protected health information. AssemblyAI is considered a business associate under HIPAA, and we offer a Business Associate Addendum (BAA) — required under HIPAA — that you can sign in minutes, without a sales call. Add SOC 2, EU data residency at the same price, and self-hosted deployment in your own VPC, and the compliance path is a signature, not a quarter of legal review. Self-hosting an open model means building and documenting all of that yourself.

When NVIDIA is the right call

If you already run a mature NVIDIA GPU and NIM stack, have an ML platform team that wants to own the model, and your workload is mostly offline batch where you can keep utilization high, Parakeet and Canary are a strong, cost-effective choice. Same if you have a hard requirement to keep everything on infrastructure you fully control and the team to operate it — though note a managed platform can meet strict data-isolation needs through self-hosted deployment too.

Verdict

NVIDIA's models are excellent engines. A production medical transcription product needs more than an engine — it needs entity accuracy on messy clinical audio, reliable speaker attribution, and a compliance path you can actually sign. Universal-3.5 Pro with Medical Mode delivers those out of the box; self-hosting Parakeet means building each of them and owning them forever.

The right way to decide is on your audio: turn on Medical Mode and run it against a batch of your real recordings — the noisy, accented, terminology-dense ones.

Frequently asked questions

What's the difference between NVIDIA Parakeet and Canary?

Parakeet and Canary are both open speech models from NVIDIA built on the NeMo framework, sharing a FastConformer encoder but using different decoders. Parakeet (for example, Parakeet-TDT-0.6B) is optimized for blazing-fast, high-throughput transcription at low latency. Canary (for example, Canary-1B-v2, ~1B parameters) is the larger multitask model, tuned for accuracy and multilingual translation across 25 European languages. In short: reach for Parakeet for speed, Canary for multilingual and translation accuracy.

Is AssemblyAI or NVIDIA Parakeet better for medical transcription?

For clinical audio, the deciding factors are medical entity accuracy, reliable speaker attribution, and a compliance path — not clean-benchmark WER. Parakeet and Canary are excellent engines on clean, read speech, but medical tuning, diarization, and HIPAA documentation are things you'd build and own yourself. AssemblyAI's Universal-3.5 Pro with Medical Mode (one "domain": "medical-v1" parameter) reduces the missed entity rate on drugs, conditions, and procedures by roughly 20%, produces diarization jointly with the transcript, and comes with a signable BAA. Test both on your own noisy, terminology-dense recordings.

Do NVIDIA Parakeet and Canary have an API?

Parakeet and Canary are released as open weights — via Hugging Face and NVIDIA NeMo/NIM — that you host and serve yourself, rather than a fully managed transcription API with an SLA. You can wrap them in your own endpoint or run them through NVIDIA NIM, but you own the GPUs, autoscaling, and uptime. A managed API like AssemblyAI bills per second of audio and runs that infrastructure for you.

Can NVIDIA Parakeet do real-time streaming transcription?

Parakeet is built for low-latency, high-throughput inference and NeMo offers streaming-capable variants, but production streaming — chunking, partial hypotheses, endpointing, and turn detection — is something you assemble and operate yourself. If you need live transcription for captions, dictation, or voice agents out of the box, a streaming-native API saves that work. AssemblyAI's Universal-3.5 Pro Realtime ships it at roughly 300ms end-of-turn.

Can I use NVIDIA Parakeet or Canary for HIPAA or PHI workloads?

NVIDIA's open models don't come with a Business Associate Addendum (BAA), so if you self-host them to process protected health information, the HIPAA safeguards, documentation, and vendor BAAs are your responsibility. AssemblyAI is considered a business associate under HIPAA and offers a standard BAA you can sign in minutes without a sales call, alongside SOC 2, EU data residency, and self-hosted VPC deployment. That turns the compliance path into a signature rather than a build-and-document project.

When should you self-host NVIDIA Parakeet instead of using a speech-to-text API?

Self-host Parakeet or Canary when you already run a mature NVIDIA GPU and NIM stack, your workload is mostly offline batch where you can keep utilization high, and you have an ML platform team that wants to own and fine-tune the model. Those cases make the economics work. For production features that need medical tuning, diarization, streaming, or a compliance path you can sign, a managed API is usually the better trade.

Self-Host STT on Baseten, Modal, Fireworks — or an API?

Mart Schweiger — Wed, 05 Aug 2026 12:54:16 +0000

Here's a decision more teams are facing: an open speech model like Whisper or Qwen3-ASR is free, and platforms like Baseten, Modal, and Fireworks make it genuinely easy to serve one. A few hours of work and you've got a transcription endpoint. So why pay a managed API per hour when you can host the model yourself for the cost of some GPU time?

It's a reasonable question — and the answer isn't "self-hosting is bad." These are capable platforms. The answer is that you're not comparing a model to a model. You're comparing renting GPUs to run a checkpoint yourself against calling an API where someone else owns the whole stack. Once you frame it that way, the comparison is about total cost of ownership, not sticker price.

What you're actually comparing

	Self-host on Baseten / Modal / Fireworks	AssemblyAI
Headline cost	GPU time (looks cheap)	$0.15–$0.21/hr async, per second
You also pay for	Idle GPUs, autoscaling eng, on-call, evals	Nothing — it's in the rate
Production features	Build diarization/streaming/entities yourself	Included
Concurrency	You provision and scale it	Unlimited, no rate limits
Reliability	Your pager	SOC 2, SLA, forward-deployed engineers

The three platforms, briefly

They're not identical, and the differences matter for the cost math:

Baseten gives you dedicated, autoscaled model deployments — great control, but you're paying for provisioned GPU capacity and tuning the autoscaling yourself.

Modal is serverless GPU with pay-per-execution. That fixes idle cost when traffic is quiet, but hands you a cold-start problem: scale to zero and the next request waits for a container and a multi-gigabyte model to load. Keep instances warm to avoid it and you're paying for idle again.

Fireworks offers fast hosted inference for open models on a per-token or per-second basis — convenient, but you're still assembling the production speech stack around it.

Different pricing shapes, same conclusion: the platform runs the checkpoint. Everything that turns a checkpoint into a transcription product is still yours.

Hidden cost #1: the GPU you pay for whether or not it's working

Open weights are free; GPUs are not, and utilization is the number nobody puts in the demo. Transcription traffic is spiky. A GPU idling at 15% still bills at 100%. Serverless softens idle but adds cold starts; dedicated capacity avoids cold starts but bills for idle. Autoscaling a GPU fleet against bursty voice traffic is a real, ongoing engineering project.

A managed API bills per second of audio — nothing for the gaps — because absorbing that utilization math is the vendor's problem.

Hidden cost #2: everything the checkpoint doesn't do

Hosting solves "how do I run the model." It does nothing for what the model doesn't produce: speaker diarization that survives cross-talk, entity accuracy on numbers and names, real-time streaming with endpointing, code-switching, PII redaction, formatting. Every one is a feature you build and maintain on top of the endpoint — or one that's already in the API price.

Hidden cost #3: reliability becomes your job

The demo runs one request on one GPU. Production is concurrency at volume, retries, latency under load, zero-downtime model upgrades, failover, monitoring, uptime. Self-hosting means all of it is yours and someone carries the pager. AssemblyAI runs hundreds of millions of inference calls a month with unlimited concurrency and no rate limits, plus forward-deployed engineers who embed with your team — because reliability at scale is the product.

When self-hosting on these platforms makes sense

If you're running massive offline batch where latency is irrelevant and you can pin GPUs at high utilization, the economics genuinely can favor self-hosting. Same if you have a hard requirement to keep audio on infrastructure you fully control, or an ML platform team that wants to own the model and has the cycles. If that's you, Baseten, Modal, and Fireworks are good tools — and note AssemblyAI also offers self-hosted deployment in your own VPC if control is the driver but you'd rather not rebuild the stack.

Verdict

The mistake isn't self-hosting. It's comparing the model's price to the API's price instead of comparing the fully loaded cost of running your own speech stack — GPUs, idle, engineers, evals, on-call — against a per-second bill where someone else owns all of it. Run that math honestly at your real volume and the answer flips more often than the afternoon-demo instinct suggests.

Frequently asked questions

Is self-hosting an open speech model cheaper than a managed API?

Not as often as the sticker price suggests. Open weights are free, but self-hosting means renting GPUs that bill whether or not audio is flowing, plus the engineering time to autoscale them and the on-call to keep them up. A managed API like AssemblyAI bills per second of audio ($0.15–$0.21/hr async, no minimums) and absorbs that utilization math. The honest comparison is your fully loaded cost — GPUs, idle, engineers, evals, on-call — versus a per-second bill, and at real production volume the answer flips more often than a quick demo suggests.

When does self-hosting speech-to-text actually become cheaper?

Self-hosting tends to win when you run large, steady, offline batch workloads where you can keep GPUs pinned at high utilization and latency doesn't matter. In that scenario the fixed GPU cost is spread across near-full usage, which is where owning the hardware pays off. It usually loses on spiky or real-time traffic, where GPUs sit idle between requests but still bill — and that math ignores the engineering and on-call cost of running the stack. Model it at your real volume and utilization, not on a single-GPU demo.

What's the difference between Baseten, Modal, and Fireworks for hosting a speech model?

All three make it easy to serve an open model, but their pricing shapes differ. Baseten gives you dedicated, autoscaled deployments — strong control, but you pay for provisioned GPU capacity and tune the autoscaling yourself. Modal is serverless GPU with pay-per-execution, which fixes idle cost but introduces cold starts when it scales to zero. Fireworks offers fast hosted inference on a per-token or per-second basis. In every case the platform runs the checkpoint; the production speech stack around it is still yours to build.

What is the cold-start problem with serverless GPU transcription?

A cold start is the delay when a serverless GPU platform has scaled to zero and your next request has to spin up a container and load a multi-gigabyte model before it can transcribe. It's the tradeoff for not paying for idle capacity — you skip the idle bill but add latency to the first request. Keeping instances warm removes the cold start but puts you back to paying for idle. A managed API absorbs this by running the fleet for you and billing only per second of audio.

What does a managed speech-to-text API price include that a raw checkpoint doesn't?

A managed API price covers the whole system around the model, not just inference. That includes speaker diarization, entity accuracy on numbers and names, real-time streaming with endpointing, code-switching, PII redaction, and formatting — plus the infrastructure: unlimited concurrency, autoscaling, uptime, and on-call. A raw checkpoint gives you a transcript; every one of those production features is something you build and maintain yourself, or one that's already in the per-second rate.

What's the best alternative to hosting Whisper yourself?

The best alternative to self-hosting Whisper is a managed speech-to-text API that runs the infrastructure and ships the production features for you. AssemblyAI, for example, bills per second of audio ($0.15–$0.21/hr async) with diarization, streaming, and entity accuracy included, unlimited concurrency, and a SOC 2 / SLA-backed platform — and offers self-hosted VPC deployment if data control was the reason you were considering hosting in the first place. Run it against your real traffic before committing engineering time to either path.

AssemblyAI vs Qwen3-ASR: Which Speech-to-Text to Ship?

Mart Schweiger — Wed, 05 Aug 2026 12:53:51 +0000

Qwen's audio models are some of the most interesting work coming out of open research. Qwen3-ASR is capable, multilingual, and if your team already builds on the Qwen ecosystem, reaching for it is a natural move. Strong model, real momentum — worth taking seriously.

And then worth pressure-testing against what production actually demands. Because a capable open checkpoint and a production speech platform solve overlapping but different problems, and the gap shows up fastest on the two things Qwen wasn't primarily built to own: everything around the transcript, and everything around running it.

Quick comparison

	Qwen3-ASR	AssemblyAI (Universal-3.5 Pro)
Delivery	Open weights you host	Managed API, or self-hosted in your VPC
Cost	GPU time + ops	$0.21/hr, per second, no minimums
Streaming	Build it yourself	Native, ~300ms end-of-turn
Diarization	Not production-ready	Joint transcript + speaker, cpWER-optimized
Code-switching	Varies by config	Native across 18 languages, no config
Entity accuracy	Raw output	Best-in-market on numbers, names, addresses
Support / SLA	Community	Forward-deployed engineers, SOC 2, BAA, EU residency

Multilingual is the right battleground — so let's use it

Qwen's appeal is heavily multilingual, so that's the fair place to test it. And multilingual transcription has a specific hard mode: code-switching, where a speaker changes language mid-sentence — Hinglish, Spanglish, and the like. It's one of the harshest tests in speech-to-text, and the spread between models is enormous.

Universal-3.5 Pro handles native code-switching across 18 languages with no configuration and no separate pass — the model transcribes each word in the language it was actually spoken. On our published code-switching benchmark it posts 7.69% average normalized WER, well ahead of other models on the same audio. That's the capability to beat, and building it out of a raw checkpoint is not a config flag.

The transcript is the start, not the finish

Qwen3-ASR gives you words. A production feature needs the layers around them.

Diarization that survives interruptions and overlap — produced jointly with the transcript, not stitched on from a second system. Entity accuracy on the tokens that break products when they're wrong: account numbers, emails, addresses, names. Streaming for anything live — Universal-3.5 Pro Realtime ships the chunking, partials, and endpointing you'd otherwise build from scratch, since most open checkpoints are async-first. Redaction and formatting on top of that.

You can assemble all of it around Qwen. The question is whether that assembly is your product or a distraction from it.

And it still has to run

Free weights come with a GPU bill you pay whether audio is flowing or not, autoscaling against spiky traffic, and an on-call rotation. A managed API bills per second and runs hundreds of millions of inference calls a month with unlimited concurrency — the full economics are in the anchor post, "the real cost of self-hosting open-source speech-to-text."

When Qwen3-ASR is the right choice

If you're already deep in the Qwen ecosystem, doing research or offline batch where you control the audio and can keep GPUs busy, or you have a platform team that wants to own and fine-tune the model, Qwen3-ASR is a solid, flexible choice. Those are legitimate reasons to self-host.

Verdict

Qwen3-ASR is a strong open model, especially for multilingual and research-flavored work. A production voice product needs code-switching that just works, diarization that holds up, reliable entities, streaming, and an SLA — and Universal-3.5 Pro delivers those without a GPU fleet or a stack to maintain. If you're shipping a product rather than studying a model, that's usually the trade worth making.

Settle it on your own audio: run Universal-3.5 Pro against your hardest multilingual recordings next to Qwen3-ASR.

Frequently asked questions

What is Qwen3-ASR?

Qwen3-ASR is a family of open-source automatic speech recognition models from Alibaba's Qwen team, released in early 2026. It includes two all-in-one models (Qwen3-ASR-1.7B and Qwen3-ASR-0.6B) built on the Qwen3-Omni foundation, with language identification and transcription across 52 languages and dialects. The weights are freely available on Hugging Face, and Alibaba also offers a hosted version (Qwen3-ASR-Flash) through its API.

Is Qwen3-ASR free, and what does it cost to run in production?

The Qwen3-ASR weights are free to download, but running them in production is not. Self-hosting means paying for GPU time — billed whether or not audio is flowing — plus the engineering to add streaming, diarization, and formatting and the on-call time to keep it up. A managed API like AssemblyAI instead charges $0.21 per hour of audio, billed per second with no minimums, so you only pay for what you process. Alibaba's hosted Qwen3-ASR-Flash API is a third option, priced separately from the open weights.

How does Qwen3-ASR compare to AssemblyAI for production speech-to-text?

Qwen3-ASR is a capable open checkpoint, while AssemblyAI is a managed speech platform — they solve overlapping but different problems. Qwen3-ASR gives you the transcript and the freedom to host and fine-tune it yourself; AssemblyAI's Universal-3.5 Pro adds the production layers around the transcript (native code-switching, joint speaker diarization, entity accuracy, streaming, and redaction) plus an SLA, SOC 2, a BAA, and per-second pricing. If you're shipping a product rather than studying a model, the managed path usually removes more work. Benchmark both on your own audio before deciding.

Does Qwen3-ASR support real-time streaming and speaker diarization?

Qwen3-ASR is primarily an offline, async-first model, so real-time streaming and production diarization aren't turnkey — you'd build the chunking, partial results, endpointing, and speaker separation yourself. AssemblyAI ships these natively: Universal-3.5 Pro Realtime streams partial transcripts with roughly 300 ms end-of-turn detection, and diarization is produced jointly with the transcript and optimized for cpWER. For live or conversational audio, that difference is usually where self-hosting gets expensive.

How good is Qwen3-ASR at multilingual and code-switched audio?

Qwen3-ASR covers 52 languages and dialects with built-in language identification, which makes it attractive for multilingual work, though code-switching behavior depends on configuration. Code-switching — where a speaker changes language mid-sentence, like Hinglish or Spanglish — is one of the hardest tests in speech-to-text. AssemblyAI's Universal-3.5 Pro handles native code-switching across 18 languages with no configuration and posts 7.69% average normalized word error rate on its published code-switching benchmark. Test both on your own multilingual recordings, since results vary widely by audio.

When should you use Qwen3-ASR instead of a managed speech-to-text API?

Qwen3-ASR is the right choice when you're already invested in the Qwen ecosystem, doing research or offline batch work where you control the audio and can keep GPUs busy, or you have a platform team that wants to own and fine-tune the model. Those are legitimate reasons to self-host. For real-time or customer-facing products that need reliable code-switching, diarization, entities, streaming, and an SLA without maintaining a GPU fleet, a managed API like AssemblyAI is usually the lower-cost, faster path to production.

AssemblyAI vs Whisper Large-v3: Which STT to Ship?

Mart Schweiger — Wed, 05 Aug 2026 12:53:44 +0000

Whisper is the model that made open-source speech-to-text a real option. It's free, it's genuinely good on clean English, and it's the first thing most teams reach for when they add transcription to a product. So let's be fair to it before we compare — Whisper Large-v3 is a strong baseline, and if you're transcribing tidy studio audio in one language, it might be all you need.

But "add transcription to a product" and "run transcription in production" are different jobs. The gap between them is where the Whisper-vs-managed-API decision actually gets made, and it's rarely about the clean-audio benchmark everyone quotes.

Here's the honest breakdown.

Quick comparison

	Whisper Large-v3	AssemblyAI (Universal-3.5 Pro)
Cost	"Free" weights + your GPU bill	$0.21/hr async, per second, no minimums
Hosting	You run it	Fully managed API (or self-hosted in your VPC)
Streaming	Async only — build it yourself	Native streaming, ~300ms end-of-turn
Diarization	Not built in	Joint transcript + speaker model, cpWER-optimized
Entity formatting	Raw text	Best-in-market on numbers, names, emails, addresses
Hallucination	Known to invent text on silence/noise	~30% lower than Whisper in our testing
Code-switching	Limited	Native across 18 languages, no config
Support / SLA	Community	Forward-deployed engineers, SOC 2, BAA, EU residency

Accuracy is fine on clean audio — and that's the trap

Whisper's headline accuracy comes from clean, read speech. Your users don't produce clean, read speech. They're on phones in cars, in call centers with cross-talk, using product names and jargon no model trained on.

The failure mode that costs you most there is hallucination — Whisper's tendency to generate fluent, confident text during silence or background noise. It's not a cosmetic bug. A transcript that's wrong in a plausible way sails straight past every downstream check, into your analytics, your summaries, your agent's next turn. In AssemblyAI's own testing, Universal-3 Pro's hallucination rate runs about 30% lower than Whisper's, and Universal-3.5 Pro is built specifically for the messy, real-world audio where Whisper slips. You can see the full accuracy breakdown here.

The transcript is where Whisper stops — and where your work begins

This is the part the "it's basically the same model" argument skips. Whisper gives you a transcript. A production voice feature needs more than a transcript.

Speaker labels. Whisper doesn't do diarization. Bolt on a separate system and you're aligning timestamps between two models that don't know about each other — brittle on exactly the audio that matters, like interruptions and overlapping speech. Universal-3.5 Pro produces the transcript and the speaker boundaries jointly, scored on cpWER rather than the looser metrics that flatter demos.

Entity accuracy. When a caller reads out a credit card, a confirmation code, or a medication name, one wrong character makes the transcript worthless. This is precisely where raw checkpoints struggle and where our models are tuned to win.

Streaming. Whisper Large-v3 is async. Building anything live — a voice agent, real-time captions, in-call intelligence — means building the whole streaming layer yourself: chunking, partials, endpointing. Universal-3.5 Pro Realtime ships it.

You can build all of this on top of Whisper. The real question is whether reimplementing diarization and a streaming stack is a better use of your engineers than shipping your product.

And someone has to run it

Free weights, metered GPUs. Serving Whisper at production latency means renting accelerators that bill whether or not audio is flowing, autoscaling them against bursty traffic, and carrying the pager when it falls over at 2am. A managed API bills per second of audio and absorbs the concurrency, uptime, and version-upgrade problems — we run hundreds of millions of inference calls a month with unlimited concurrency and no rate limits. We cover the full economics in the anchor post, "the real cost of self-hosting open-source speech-to-text."

When Whisper Large-v3 is the right choice

Credit where it's due. Reach for Whisper if you're prototyping and want full model control, running large offline batch jobs where latency doesn't matter and you can keep GPUs pinned at high utilization (which fixes the cost math), or you specifically need to fine-tune the weights on your own data and have the team to do it. Those are real, defensible cases.

And if you like Whisper's coverage but not the ops, note we also offer Whisper-Streaming as a managed endpoint at $0.30/hr — Whisper's reach without the GPU fleet.

Verdict

Whisper Large-v3 is a great place to start and a hard place to scale. The moment you need speaker labels, reliable entities, streaming, or an SLA, you're either building a platform around a free checkpoint or paying per second for one that already exists. For most teams shipping a real product, the second math wins.

The fastest way to settle it is on your own audio: run Universal-3.5 Pro against your worst recordings — the noisy, accented, jargon-heavy ones — and compare the transcripts side by side with Whisper's. That's the benchmark that decides it.

Frequently asked questions

Is AssemblyAI or Whisper Large-v3 more accurate for transcription?

On clean, read English audio, Whisper Large-v3 and a managed model like AssemblyAI's Universal-3.5 Pro are both strong. The difference shows up on real-world audio — phone calls, cross-talk, accents, and industry jargon — where Whisper is prone to hallucinating fluent but wrong text during silence or noise. In AssemblyAI's own testing, Universal-3 Pro's hallucination rate runs about 30% lower than Whisper's, and Universal-3.5 Pro is built specifically for that messy audio. The reliable way to decide is to run both on a batch of your own worst recordings.

Why does Whisper hallucinate, and how do you prevent it?

Whisper hallucinates because it's trained to always produce fluent text, so during silence or background noise it can invent words that were never spoken. This matters because a plausibly-wrong transcript passes every downstream check and corrupts your summaries, analytics, and an agent's next turn. You can reduce it with voice-activity detection and careful chunking, or use a model tuned for noisy, conversational audio — Universal-3.5 Pro posts roughly 30% lower hallucination than Whisper in our testing.

When should you use Whisper Large-v3 instead of a speech-to-text API?

Use Whisper Large-v3 when you're prototyping and want full model control, running large offline batch jobs where latency is irrelevant and you can keep GPUs at high utilization, or fine-tuning the weights on your own data with a team that can maintain it. Those are real, defensible cases. For production voice features that need streaming, diarization, reliable entity formatting, or an SLA, a managed API is usually the better trade.

Is it cheaper to self-host Whisper or use a speech-to-text API?

Whisper's weights are free, but self-hosting is not — you rent GPUs that bill whether or not audio is flowing, autoscale them against bursty traffic, and carry the pager when they fail at 2am. A managed API like AssemblyAI bills per second of audio ($0.21/hr async, no minimums) and absorbs the utilization, concurrency, and uptime problems. The honest comparison isn't $0 vs. $0.21/hr — it's your fully loaded engineering cost versus a per-second bill someone else owns.

Does Whisper Large-v3 support real-time streaming and speaker diarization?

No — Whisper Large-v3 is async-only and has no built-in speaker diarization. To run it live you build the streaming layer yourself (chunking, partial hypotheses, endpointing), and to label speakers you bolt on a separate diarization model and align timestamps between two systems that don't know about each other. AssemblyAI's Universal-3.5 Pro Realtime ships native streaming at ~300ms end-of-turn and produces the transcript and speaker boundaries jointly,

The Real Cost of Self-Hosting Open Source Speech-to-Text

Mart Schweiger — Wed, 05 Aug 2026 12:53:18 +0000

Open-source speech-to-text got good. That part isn't marketing spin, and pretending otherwise is how you lose an engineer's trust in the first paragraph.

Whisper Large-v3 transcribes clean English about as well as anything you can buy. Qwen3-ASR and NVIDIA's Parakeet and Canary models are genuinely strong. Mistral shipped Voxtral. And the checkpoints are free — pip install, download the weights, point it at an audio file, done. Spin one up on Baseten, Modal, or Fireworks and you've got a working transcription endpoint in an afternoon. For a growing number of teams, that afternoon ends with a reasonable question: why are we paying an API for this?

It's a fair question. Here's the more useful one: what does it actually cost to run that endpoint in production for the next two years?

Because the sticker price of an open model is zero, and the total cost of ownership is not. The gap between those two numbers is where most self-hosting decisions quietly go wrong.

What "free" actually buys you

A model checkpoint is one component of a transcription product. It is not the product.

When you call a managed speech-to-text API, the price covers the model plus the parts you'd otherwise have to build, operate, and keep alive yourself: the inference infrastructure, the accuracy work on messy real-world audio, speaker diarization, entity formatting, streaming, language handling, and the on-call rotation for when any of it breaks at 2am. Download a raw checkpoint and you own every one of those. The weights are free. The system around them is the job.

So the honest comparison isn't "$0 vs. $0.21 an hour." It's "your engineering org vs. $0.21 an hour." Let's break down what lands on your side of that trade.

Factor	Self-hosted open source (Whisper, Parakeet, Voxtral, Qwen3-ASR)	Managed API (AssemblyAI)
Upfront model cost	$0 — open weights, no license fee	$0 to start — free credits and playground, then pay-as-you-go
What you actually pay for	GPU hours (billed even when idle), plus engineering and DevOps time	Only the audio you process, billed per second
Typical run cost	~$2–4+/hr per on-demand A100/H100-class GPU, regardless of utilization	$0.15–$0.21/hr async; $0.15–$0.45/hr streaming
Idle / low utilization	You pay full price for idle GPUs — 15% utilized still bills 100%	No idle cost — you only pay for audio transcribed
Accuracy on real-world audio	Varies; higher hallucination and WER on noisy, accented, code-switched speech (e.g., Voxtral Mini ~18% WER code-switched)	Tuned for production audio; Universal-3.5 Pro <8% WER on code-switched speech, and Universal-3 Pro runs ~30% fewer hallucinations than Whisper
Speaker diarization	Build or integrate a separate system	Built in — transcript and speaker turns produced jointly
Real-time streaming	Build your own low-latency serving stack	Native streaming API with ~300 ms partial transcripts
Entity accuracy & PII redaction	Not included — build and maintain separately	Included: entity detection, PII redaction, and formatting
Scaling & concurrency	You manage autoscaling, queuing, and retries	Unlimited concurrency, no rate limits
Reliability & maintenance	Your team owns uptime, monitoring, model upgrades, and on-call	Managed uptime with 24/7 support
Data control & compliance	Full control; runs entirely in your own VPC or on-prem	Voice AI Cloud, EU data residency, self-hosted deployment, and a BAA available
Time to production	Weeks to months of setup and tuning	Minutes — sign up and get an API key
Best fit	High-utilization offline batch, research/prototyping, strict data isolation	Real-time and customer-facing apps needing accuracy, features, and scale without ops burden

Hidden cost #1: the GPU bill nobody quotes you

Open weights don't come with a price tag, but GPUs do. To serve transcription at production quality and latency you're renting accelerators — and you pay for them whether or not audio is flowing through them.

This is the detail that surprises teams. Utilization is rarely high. Traffic is spiky, models sit warm waiting for requests, and a GPU idling at 15% still bills at 100%. Serverless platforms like Modal soften the idle problem but hand you a cold-start problem instead: scale to zero and your next request waits for a container and a multi-gigabyte model to load. Keep instances warm to avoid that and you're back to paying for idle. Autoscaling a GPU fleet against bursty voice traffic is a real, ongoing engineering project, not a checkbox.

A managed API bills per second of audio — AssemblyAI's pricing runs $0.15–$0.21/hr for pre-recorded transcription with no minimums and no idle time — because absorbing that utilization math is the vendor's problem, not yours.

Hidden cost #2: everything the checkpoint doesn't do

Here's where the "it's basically the same model" argument falls apart. Raw ASR checkpoints give you a transcript. Production voice products need a lot more than a transcript.

Speaker diarization. Who said what. Most open checkpoints don't do it at all, and the ones that gesture at it don't hold up on the audio that matters — short back-and-forth turns, interruptions, overlapping speech. Getting diarization right is its own research problem; AssemblyAI's latest models produce the transcript and the speaker boundaries jointly, optimized for concatenated word error rate rather than the looser metrics that make demos look fine and production look broken.

Entity accuracy. In the real world people say credit card numbers, email addresses, medication names, and account IDs — the exact tokens where a single wrong character makes the transcript worthless. A base model transcribes the words; it doesn't reliably format an alphanumeric confirmation code spelled out over a phone line.

Streaming. Most open checkpoints are async-only. If you're building anything live — a voice agent, real-time captions, in-call intelligence — you get to build the streaming layer yourself: chunking, partial hypotheses, endpointing, the works. Real-time transcription is a different engineering problem than batch, not a config flag.

Code-switching, redaction, formatting. Multilingual speakers switch languages mid-sentence. Recordings need PII redacted. Numbers, dates, and punctuation need to render like a human wrote them. Every one of these is a feature you either buy once or build and maintain forever.

You can build all of it. The question is whether reimplementing diarization and a streaming stack is the best use of your team versus shipping your actual product.

Hidden cost #3: accuracy on audio that isn't clean

Benchmarks are run on clean audio. Your users are not clean audio.

They're on a phone in a moving car, in a call center with three conversations bleeding together, using product names and industry jargon no model saw in training, in accents the leaderboard underweighted. This is exactly where open checkpoints tend to slip — and where one specific failure mode gets expensive: hallucination. Whisper is known to invent fluent, confident text during silence or noise. It's not a rounding error; in AssemblyAI's own testing, Universal-3 Pro's hallucination rate runs about 30% lower than Whisper's, and on published head-to-head benchmarks, open models trail badly on the audio that resembles production — Voxtral Mini, for instance, posts nearly 18% word error rate on code-switched speech versus under 8% for a managed flagship. A transcript that's wrong in a plausible way is more dangerous than one that's obviously garbled, because nothing downstream flags it.

Closing that gap yourself means fine-tuning, building an eval set from your own audio, and re-running it every time you touch the model. That's a standing capability you'd be committing to own. And word error rate alone won't even tell you if you're winning — you need to measure the errors that actually break your product.

Hidden cost #4: reliability is now your pager

The afternoon demo runs on one GPU serving one request. Production is a different animal.

Concurrency at real volume, retries and graceful degradation, latency under load, model-version upgrades without downtime, region failover, monitoring, uptime — this is the operational surface of a speech platform, and self-hosting means it's yours. Someone carries the pager. A managed platform runs hundreds of millions of inference calls a month with unlimited concurrency and no rate limits, because reliability at scale is the product it's selling. When you self-host, reliability at scale is a headcount question.

When self-hosting is the right call

None of this means "never self-host." There are real cases where owning the stack is the correct decision:

You're doing research or a prototype where model control matters more than production polish. You're running massive offline batch jobs where latency is irrelevant and you can keep GPUs pinned at high utilization, which fixes the economics. You have a hard data-isolation requirement that rules out any external call — though note that a managed platform can meet strict needs too, through self-hosted deployment in your own VPC, EU data residency, and a signable Business Associate Addendum for teams handling protected health information. Or you have an ML platform team that wants to own this and has the cycles to.

If one of those is you, self-hosting is defensible. If none of them is you, the "free" model is the most expensive line item on the roadmap — it just doesn't show up on the invoice.

The number that actually matters

The mistake isn't choosing open source. The mistake is comparing the wrong two numbers.

The comparison that matters isn't the model's price against the API's price — it's the fully loaded cost of running your own speech stack, GPUs and engineers and eval pipelines and on-call included, against a per-second bill where someone else owns all of it. Run that math honestly and the answer flips more often than the afternoon-demo instinct suggests. Cheap models are real. Free ones don't exist.

If you want to see where a managed model lands on your actual audio before you commit engineering time to either path, run it against a batch of your worst recordings — the noisy ones, the accented ones, the jargon-heavy ones. That's the benchmark that decides it.

Frequently asked questions

Is open-source speech-to-text really free?

Open-source speech-to-text models are free to download but not free to run in production. Model weights like Whisper, NVIDIA Parakeet, Qwen3-ASR, and Mistral's Voxtral carry no license fee, but you still pay for the GPUs that serve them, the engineering time to add production features, and the on-call burden to keep them running. The real comparison isn't "$0 vs. an API bill" — it's your fully loaded infrastructure and engineering cost versus a managed per-second price.

How much does it cost to self-host an open-source speech-to-text model?

Self-hosting cost is driven by GPU time, not the model. A single on-demand cloud GPU in the A100/H100 class typically runs $2–4+ per hour and bills whether or not it's actively transcribing, so a model idling at 15% utilization still costs you 100%. On top of that you add engineering time to build diarization, streaming, and formatting, plus ongoing monitoring and model upgrades. By contrast, a managed API like AssemblyAI charges $0.15–$0.21 per hour of audio for pre-recorded transcription, billed per second with no idle cost.

Is self-hosted Whisper cheaper than a speech-to-text API?

It depends almost entirely on GPU utilization. Self-hosted Whisper can be cost-effective for large offline batch jobs that keep a GPU near 100% busy, but for spiky or low-volume workloads the idle GPU time usually makes a per-second managed API cheaper. Whisper also lacks production features and shows higher hallucination rates on real-world audio — AssemblyAI's Universal-3 Pro runs about 30% fewer hallucinations than Whisper. Benchmark both on your own audio and include engineering and infrastructure time, not just the sticker price.

What features do open-source speech-to-text models lack out of the box?

Raw open-source checkpoints transcribe audio but omit most of what production apps need. Commonly missing pieces include speaker diarization (who said what), real-time streaming, PII redaction, entity accuracy for data like credit cards and medication names, code-switching across languages, and consistent formatting. Each one has to be built, integrated, and maintained yourself, whereas a managed API ships them as part of the model or a single parameter.

When does self-hosting open-source speech-to-text make sense?

Self-hosting is a good fit when you have high, steady GPU utilization or strict control requirements. The clearest cases are large offline batch jobs that keep GPUs busy, research and prototyping where you need to modify the model, strict data-isolation rules, and teams that already run a dedicated ML platform. For most real-time or customer-facing products that need accuracy, features, and scale without an ops burden, a managed API is usually the lower-cost and faster path to production.

How much does a managed speech-to-text API cost?

AssemblyAI's pre-recorded speech-to-text starts at $0.15–$0.21 per hour of audio, billed per second with no minimums, and streaming transcription ranges from $0.15 to $0.45 per hour. There are no idle-GPU charges, no rate limits, and unlimited concurrency, so you pay only for the audio you actually process.

Best Voice Agent Platform for Developer Experience

Mart Schweiger — Wed, 05 Aug 2026 12:53:11 +0000

Every voice agent demo looks great. The gap shows up later — the afternoon you're three providers deep, staring at three dashboards, trying to work out why the agent heard "twenty" as "plenty" and confidently booked the wrong appointment.

That's the real test of developer experience for voice agents. Not how fast you get a demo talking, but how fast you ship something to production and how much control you keep once it's live. On that test, "best DX" usually gets handed to whichever platform has the most building blocks. We'd argue the opposite: the best developer experience is the one that gives you the fewest moving parts to manage and the most control over the parts that matter.

Let's compare the real options.

What "developer experience" actually means for voice agents

Strip away the marketing and DX for a voice agent comes down to five things:

Time to a working agent. How long from reading the docs to a real conversation?
Simplicity of the stack. How many providers, bills, and dashboards are you managing?
Control after launch. Can you change behavior, tools, and settings without rebuilding?
Accuracy of what the agent hears. If the transcription is wrong, everything downstream is wrong.
Freedom from lock-in. Can you shape the agent to your product, or does every agent end up sounding the same?

Notice that "how many features are on the pricing page" isn't on the list. More surface area is not more DX. Often it's the opposite.

The two paths most teams take (and the third one)

Path one: an orchestration platform like Vapi or Retell. You get a fast start and a lot of built-in scaffolding. The tradeoff shows up at the ceiling — opinionated conversation design, limited room for niche integrations, and a sameness where every agent built on the platform behaves alike. Great for getting going; frustrating when you need your agent to be genuinely yours.

Path two: do it yourself — wire up a separate speech-to-text provider, an LLM, and a text-to-speech provider. Maximum control, but now you own three vendors, three invoices, three sets of logs, and all the glue in between: turn detection, interruption handling, voice activity detection. You'll spend your first weeks on plumbing instead of product.

Path three: one API that runs the whole speech-to-speech pipeline — STT, LLM, and TTS — behind a single connection, while still letting you control conversation design, tools, and timing. This is where AssemblyAI's Voice Agent API sits. You own the hardest part of the stack (what the agent hears and how it responds) without owning the integration burden. It's infrastructure you build on, not a platform you build inside.

Quick comparison

	AssemblyAI Voice Agent API	Vapi / Retell	OpenAI Realtime API	DIY (STT + LLM + TTS)
Integration model	One WebSocket + JSON	Platform SDK/config	Realtime API (30+ event types)	Three separate providers
SDK required	No	Yes (platform SDK)	Yes	Multiple
Time to first agent	Same afternoon	Fast	Moderate	Slow
Pricing	Flat $4.50/hr, all-in	Platform fee + usage	~$18/hr	Three separate bills
STT accuracy (pooled WER)	6.99%	Varies by chosen STT	Multimodal, not STT-specialized	Depends on provider
Live mid-call config	Yes	Limited	Limited	You build it
Framework plugins	LiveKit, Pipecat	N/A	N/A	You build it
Lock-in	Low (infrastructure)	Higher (platform)	Model-tied	None, but you own everything

Pooled WER is from Pipecat's open STT benchmark of real agent conversations. Comparison reflects general positioning; benchmark your own audio.

DX, criterion by criterion

Time to a working agent

The Voice Agent API is a standard JSON API over a WebSocket — no SDK, no framework to learn. You can read the full API reference in about 10 minutes, and most developers have a working agent the same afternoon. It works with Claude Code out of the box: copy the docs, paste, build. Connect to the WebSocket, stream audio in, get audio back, write your system prompt, and focus on your product.

Compare that to standing up three providers yourself, or learning the OpenAI Realtime API's 30-plus event types before your agent says its first word.

Simplicity of the stack

This is the heart of it. One WebSocket replaces three providers. One bill — measured in hours, not token math across three invoices. One set of logs instead of three dashboards to correlate when something breaks. At a flat $4.50/hr, STT, LLM, and TTS are all included. When your agent misbehaves at 11pm, you're debugging one surface, not triangulating across three.

Control after launch

Shipping the demo is the easy part. The DX difference is what happens when you need to change the agent in production. With the Voice Agent API you can update the system prompt, tools, and settings mid-conversation without reconnecting. Custom functions are defined with plain JSON Schema. You control voice activity detection, turn timing, and barge-in. This is where platforms tend to hit their ceiling — you can configure what they expose, and no more.

Accuracy of what the agent hears

Here's the criterion that quietly determines everything: if the STT is wrong, the LLM responds to the wrong thing. The whole pipeline is built around getting the input right. It runs on Universal-3.5 Pro Realtime, which posts a 6.99% pooled word error rate on Pipecat's open STT benchmark of real agent conversations — ahead of Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%). It even takes the agent's own question as input via agent_context — which cut word error rate by 10.2% across a benchmark of 20,000 voice agent audio files — and keeps a rolling conversation memory so a one-word "yes" or a spelled-out account number resolves in context. Turn detection reads tonality and pacing, not just silence, so the agent knows the difference between a pause and a finished thought.

Most platforms treat transcription as a commodity you plug in. We treat it as the foundation, because it is.

Freedom from lock-in

The Voice Agent API is invisible infrastructure — "we're not the agent, we're the platform upon which you build." Your customers should feel like you built the whole thing from scratch. And you're not boxed in: there are drop-in plugins for LiveKit and Pipecat, the two most popular open-source voice pipeline frameworks, and sophisticated teams can hit the WebSocket directly. You're building on infrastructure, not renting space inside someone's platform.

When a platform is the better choice

If you need no-code configuration so non-technical teammates can build and edit agents, a platform like Vapi or Retell will serve you better today. If you need built-in telephony and call routing out of the box with zero setup, that's a platform strength (telephony is a fast-follow on our side, not shipping yet). And if you never need to go beyond what the platform exposes, its guardrails are a feature, not a limit.

The Voice Agent API is aimed at developers who want to ship fast and keep control — roughly 80% of new voice builders start net-new, and they consistently value simplicity and ownership over pre-built scaffolding. In a recent week, 11 of 12 new enterprise accounts were voice-agent or conversation-intelligence builders — the pattern is real.

The verdict

If DX means "how many features can I click together," a platform wins. If DX means "how fast can I ship a production voice agent, and how much control do I keep after launch," the answer is one API that owns the hardest part of the stack and gets out of your way on the rest.

The fastest way to judge developer experience is to build on it. So build one.

Frequently asked questions

What is the best AI voice agent platform?

There's no single best — it depends on whether you optimize for speed-to-demo or long-term control. No-code platforms like Vapi and Retell are fastest for non-technical teams and offer built-in telephony today. For developers who want to ship fast while keeping full control over conversation design and integrations, a single API like AssemblyAI's Voice Agent API offers the best balance: one WebSocket for the full STT-LLM-TTS pipeline, no SDK, flat $4.50/hr, built on a market-leading realtime speech model.

What's the difference between a voice agent platform and a voice agent API?

A platform (Vapi, Retell) is an environment you build inside — fast to start, but with ceilings on customization and a tendency toward sameness. An API (AssemblyAI's Voice Agent API) is infrastructure you build on — you keep control over conversation logic, tools, and timing while the API handles the speech pipeline. Platforms trade control for convenience; APIs trade a little setup for control.

How much do AI voice agents cost to build?

AssemblyAI's Voice Agent API is a flat $4.50/hr that includes speech-to-text, the LLM, and text-to-speech in one bill, billed by the minute. Building it yourself means paying three separate providers and reconciling three invoices, and OpenAI's Realtime API runs closer to $18/hr for a comparable pipeline. Flat, all-in pricing makes voice-agent costs far easier to forecast.

Which voice agent platform is best for startups and small businesses?

Startups building their first voice product usually value time-to-working-demo and predictable pricing over pre-built scaffolding. A single API with flat $4.50/hr pricing, no SDK, and a 10-minute API reference lets a small team ship an agent the same afternoon and keep control as they grow — without platform lock-in or a per-seat plan. Teams that need non-technical, no-code configuration may prefer a managed platform.

Do I need an SDK to build a voice agent?

Not with AssemblyAI's Voice Agent API. It's a standard JSON API over a WebSocket — connect, stream audio in, get audio back. You can read the full API reference in about 10 minutes, and it works with Claude Code out of the box.

Vapi vs. Retell vs. AssemblyAI — which has the best developer experience?

Vapi and Retell win on no-code configuration and built-in telephony. AssemblyAI wins on stack simplicity (one API, one bill, one set of logs), transcription accuracy (6.99% pooled WER on Pipecat's open benchmark), and control after launch (live mid-conversation config, custom tools via JSON Schema, no lock-in). Developers who want to own their product's behavior tend to prefer the API approach.