DEV Community: Mart Schweiger

Why Streaming Transcription Drifts to English — and How to Fix It

Mart Schweiger — Tue, 23 Jun 2026 21:26:42 +0000

You built a multilingual voice product, tested it on Spanish audio, and it worked. Then it hit production traffic and started handing back English. A caller says "necesito ayuda con mi factura" and the transcript reads "I need help with my invoice" — translated, not transcribed — or worse, a phonetic English mush that means nothing downstream.

If you're evaluating streaming speech-to-text for a multilingual product, this is the failure mode that quietly kills you in testing. It's not random, and it's not unfixable. Here's why streaming models drift to English, and how to steer them back.

The drift is a confidence problem, not a language problem

Most multilingual streaming models can transcribe your target languages. The trouble is what they do when they're unsure — and streaming makes them unsure far more often than batch processing does.

A pre-recorded model reads the whole file before committing. A streaming model has to decide what it heard within a few hundred milliseconds, working from a short rolling window of audio. Less evidence per decision means more uncertainty, and uncertainty is exactly where the default kicks in.

That default is usually English. Most ASR training data skews heavily English, so when a model can't confidently place a sound in Spanish or German, the safest bet — statistically — is English. The model isn't broken. It's guessing, and its prior says English.

Three situations push a streaming model into that low-confidence zone over and over:

Short utterances. "Sí." "Vale." "Mmhmm." A one-word turn gives the detector almost nothing to work with, so it falls back to its prior.
Code-switching boundaries. When a speaker drops an English word into a Spanish sentence — "mándame el invoice" — a model that detects language per turn can latch onto the English token and flip the whole turn.
Noise and accents. 8kHz phone audio, background chatter, or an underrepresented regional accent all lower confidence, and low confidence trends toward English.

So the drift isn't your audio being "too multilingual." It's the model hitting uncertainty and resolving it the wrong way.

Test drift on your own multilingual audio

Run your real production audio — short turns, code-switching, phone quality — through streaming transcription and see how it holds up. Start with a free account and clear docs.

Why "just force the language" usually backfires

The obvious instinct is to pin the language down — hard-code one language and be done with it. Sometimes that helps. Often it makes things worse, for one big reason: hard-coding one language breaks the moment a real conversation code-switches. Your Spanish caller says "my tracking number is..." and a single-language lock either drops it or mangles it. Real bilingual speech doesn't stay in one lane, so forcing one lane fights the audio.

The nuance worth getting right: there's a difference between blindly forcing a language and correctly telling the model what you already know. If your audio genuinely is monolingual — a support line in Osaka that runs in Japanese, a clinic intake in Madrid that runs in Spanish — committing the model to that language is now the recommended way to steer (more on the language_code parameter below). The mistake is forcing a single language onto audio that actually mixes languages. For that, you steer with context and let the model code-switch.

The fix isn't a bigger hammer. It's matching the model to your language reality and then giving its detector the signals it needs.

How to fix language steering

Think of language steering as five levers, in rough order of impact.

1. Match the model to how your users actually speak

This is the highest-leverage decision, and it's where most drift gets solved before it starts. AssemblyAI gives you streaming paths that behave differently on multilingual audio:

Universal-3.5 Pro Realtime (universal-3-5-pro) — native code-switching across 18 languages in a single stream: English, Spanish, French, German, Italian, Portuguese, Arabic, Danish, Dutch, Hebrew, Hindi, Japanese, Mandarin, Vietnamese, Finnish, Norwegian, Swedish, and Turkish. It treats a mid-sentence switch — Hinglish included — as ordinary speech instead of a language to re-detect, which is exactly the behavior that prevents drift on bilingual calls. This is the model that holds the line, and it's the new default for realtime transcription.
Universal-Streaming Multilingual (universal-streaming-multilingual) — a cost-effective path that covers a smaller set of languages with per-turn language detection. Per-turn detection is fine when speakers change languages between turns, but it's more prone to flipping on intra-sentence switches.
Whisper-Streaming (whisper-rt) — the 99+ language fallback when your languages fall outside the core set. Automatic language detection is built in and mandatory here.

Picking a per-turn model for an intra-sentence code-switching product is the single most common cause of drift we see. Match the model first.

2. Tell the model the language when you actually know it

Universal-3.5 Pro Realtime runs in multilingual mode by default — the right behavior when you don't know what's coming. But most production calls aren't a guessing game. When you know the session is monolingual, pass the new language_code connection parameter. It commits the model to one language instead of asking it to detect one, which is the cleanest way to head off the wrong-language slips that creep in on short or ambiguous audio. This is now the recommended way to bias toward a single language.

client.connect(
    StreamingParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        language_code="es",   # commit to Spanish when you know the call is monolingual
    )
)

Omit language_code and you keep full multilingual code-switching. For calls that genuinely mix languages, leave it off and steer with context instead — that's what the next lever is for.

3. Give the model the conversation as context

A big share of low-confidence drift comes from the model hearing each moment cold, with no sense of what came before. Universal-3.5 Pro Realtime fixes that two ways. It keeps a short, rolling memory of the conversation (Context Carryover) and uses it as context for whatever comes next — on by default, nothing to configure. And for voice agents, you can pass the agent's own question in with agent_context, so a mumbled or short reply resolves through the lens of what was just asked. More context per decision means fewer of the uncertain moments that resolve toward English. You can also describe the audio with the prompt parameter to prime the model for a noisy line or a specific domain.

4. Anchor vocabulary with keyterms — and hear the speaker, not the room

Drift often shows up first on the words that matter most: names, product terms, account types, medication names. Universal-3.5 Pro Realtime includes keyterms prompting at no extra cost, and it applies across all supported languages at once. Feeding the model your domain vocabulary keeps those terms anchored in the right language instead of getting "Englished" the moment confidence dips. And because background speech — a TV, a passenger — gets transcribed as phantom words that drag confidence down, voice_focus isolates the primary speaker and suppresses the rest (use near_field for headsets and phones, far_field for rooms and kiosks).

5. Give the detector room to work

A lot of self-inflicted drift comes from starving the model of context. If you've cranked silence thresholds down to chase latency, you're chopping audio into fragments too short to place a language confidently. Leave enough audio per turn for the model to commit, and you'll see fewer fallbacks — especially on the short acknowledgments that trip every system. Our real-time transcription guide shows a working streaming setup you can adapt to test these settings.

See language steering live

Drop multilingual and code-switched audio into the playground and watch real-time transcription hold the right language. No setup required.

Try playground

What "fixed" looks like in production

When the model matches your audio and has the signals it needs, the drift stops being a coin flip. A bilingual support call stays bilingual in the transcript. The same speaker keeps producing Spanish when they speak Spanish and English when they speak English — including mid-sentence — and your downstream intent detection, routing, and analytics stop choking on phantom English.

Universal-3.5 Pro Realtime does this without trading away responsiveness: end-of-turn detection reads tonality, pacing, and rhythm rather than silence alone and lands around 300ms, and the model posts a market-leading 6.99% pooled word error rate on Pipecat's open benchmark of real agent conversations. You don't have to choose between "stays in the right language" and "fast enough for a voice agent." If you're benchmarking options, our guide on how to evaluate speech recognition models walks through testing this properly — and the Streaming Speech-to-Text API docs cover the configuration.

The bottom line

English drift isn't a quirk of your audio — it's a model resolving uncertainty toward its English-heavy prior, and streaming creates that uncertainty constantly. You fix it by steering, not guessing: match the model to your real language mix, tell it the language with language_code when you actually know it, feed it conversation context, anchor your vocabulary with keyterms, and give the detector enough audio to commit. Blindly pin a single language onto mixed audio and you'll win the demo and lose production. Steer instead, and the messy, code-switched, real-world call stays in the language it was actually spoken in.

If multilingual reliability is a make-or-break for your product, test with the audio that actually breaks things — short turns, accents, mid-sentence switches, phone-quality lines. That's the audio that tells you whether a model steers or drifts.

Frequently asked questions

Why does my streaming transcription keep switching to English on non-English audio? Streaming models work from short audio windows, so they hit low-confidence moments often — short utterances, code-switching boundaries, noise, and accents. When confidence drops, most models fall back to an English-heavy statistical prior because that's what dominates ASR training data. The fix is to match the model to your language mix and feed it better signals — the language when you know it, plus conversation context — rather than guessing.

Should I force a single language to stop the drift? Only when your audio really is monolingual. In that case, pass language_code to commit Universal-3.5 Pro Realtime to one language — that's the recommended way to steer. But hard-coding one language onto audio that actually code-switches breaks the moment a speaker mixes languages, so for genuinely bilingual calls you leave it off and steer with context instead.

Which AssemblyAI model is best for multilingual audio that code-switches mid-sentence? Universal-3.5 Pro Realtime (universal-3-5-pro). It supports native code-switching across 18 languages in one stream — including Hinglish and other mixed-language speech — and treats mid-sentence switches as ordinary speech rather than re-detecting language each turn, which is what prevents English drift. For languages outside those 18, Whisper-Streaming (whisper-rt) covers 99+ languages.

What's the difference between native code-switching and per-turn language detection? Native code-switching transcribes language changes as they happen, including inside a sentence. Per-turn detection only identifies the language at a turn boundary, so a single English word inside a Spanish sentence can flip the whole turn to English. For natural bilingual conversation, native code-switching is the one that resists drift.

How do I reduce language drift in production? Match your audio to Universal-3.5 Pro Realtime, set language_code for monolingual sessions, and lean on the model's rolling conversation memory and agent_context so each turn is decided with context rather than cold. Pair that with keyterms prompting to keep domain vocabulary anchored in the correct language, and voice_focus to stop background speech from dragging confidence down.

Keyterm Prompting: Boost Names, Jargon & Product Terms

Mart Schweiger — Tue, 23 Jun 2026 21:26:17 +0000

Your voice agent nails 98% of the conversation and then fumbles the one word that actually mattered — the caller's last name, your product SKU, the medication. "Metoprolol" comes back as "metoprolal." "Byrne-Donoghue" becomes "Byrne Donahue." The transcript looks great in the demo and falls apart on the exact tokens your downstream logic depends on.

If you're evaluating real-time speech-to-text, this is the accuracy problem that decides whether the model is usable in production. And it's the one generic benchmarks hide, because they average it away. Here's the lever that fixes it — keyterm prompting — how it actually works, and how to use it without making things worse.

Why the words that matter are the hardest to get right

It's not random which words a model misses. Domain-specific terms fail at roughly 3–5x the rate of general speech in off-the-shelf models, and the reason is training data. Common English words show up millions of times; your company name, your SKUs, a clinician's surname, an alphanumeric policy ID — those barely appear, if at all. The model has no strong prior for them, so the moment audio gets noisy or a speaker has an accent, it falls back to a more common word that sounds similar.

That's the trap: the rare, high-value terms are exactly the ones most exposed to error, and they're the ones a transcript can least afford to get wrong. A misheard filler word costs nothing. A misheard account number breaks the call.

What keyterm prompting does

Keyterm prompting is the most direct accuracy lever AssemblyAI gives you for this. You pass a list of the words and phrases that matter — names, brands, jargon, product terms — through the keyterms_prompt parameter, and the model biases toward recognizing them. It's an array of strings, and it works on both pre-recorded and streaming transcription. It's part of the broader promptable interface AssemblyAI introduced with Universal-3 Pro and carried into the Universal-3.5 Pro Realtime flagship.

keyterms_prompt=["Kelly Byrne-Donoghue", "metoprolol", "Universal-3.5 Pro"]

That's the whole interface. The interesting part is what happens behind it.

Boost your hardest terms in real time

Test keyterm prompting on your own names, jargon, and product terms with Universal-3.5 Pro Realtime. Free account, clear docs, no credit card.

How it actually works: two boosting stages

This is the part that comes up constantly and rarely gets explained. Streaming keyterm prompting isn't one mechanism — it's two, working in sequence.

Word-level boosting happens live, during inference. The model is biased toward your keyterms as words are emitted, so recognition improves in real time as the audio streams in. This stage is on by default.

Turn-level boosting happens after each turn completes. A second pass re-examines the full turn against your keyterms list using metaphone-based matching — phonetic matching, not exact spelling. Metaphone encodes how a word sounds, so when the model hears "Byrne Donahue" and your keyterm is "Byrne-Donoghue," the phonetic codes line up and the term gets corrected to the spelling you specified. That's why keyterms fix names that are pronounced one way and spelled another. On Universal-3.5 Pro Realtime (universal-3-5-pro), turn-level boosting is always active; on Universal-Streaming English and Multilingual it kicks in when format_turns=true.

The two stages stack: word-level catches the term as it's spoken, turn-level cleans up anything that slipped through using sound-alike matching. Together they target the precise failure mode — a term the model heard almost right.

Using it in a streaming session

On streaming, keyterms_prompt is set when you open the WebSocket and can be changed mid-stream. Here's the shape with the Python SDK:

client.connect(
    StreamingParameters(
        sample_rate=16000,
        speech_model="universal-3-5-pro",
        keyterms_prompt=["Keanu Reeves", "AssemblyAI", "Universal-3.5 Pro"],
    )
)

A few limits worth knowing before you load it up, because they're easy to trip over:

Streaming caps at 100 keyterms per session, and each term must be 50 characters or less. Go over and the request errors; over-length terms are ignored. (Pre-recorded is far more generous — up to 1,000 words or phrases, max six words each.)
On Universal-3.5 Pro Realtime, keyterms are included at no extra cost, alongside the model's conversation context. On Universal-Streaming English and Multilingual, keyterm boosting is an add-on.
You can combine keyterms_prompt with a prompt in the same request — keyterms get appended to your prompt automatically. (For more on shaping output with prompts, see our prompt engineering guide.)

The real unlock for voice agents is dynamic keyterms. You don't have to commit to one list for the whole call. Send an UpdateConfiguration message and swap the terms as the conversation moves:

{ "type": "UpdateConfiguration", "keyterms_prompt": ["cardiology",
"echocardiogram", "Dr. Patel", "metoprolol"] }

So when your agent reaches the identity-verification step, you load names and date-of-birth terms; when it moves to a medical intake, you swap in clinical vocabulary. You're priming the model for exactly what it's about to hear, stage by stage — which is one of the most effective ways to lift mid-call accuracy. (Our real-time transcription guide shows a full working setup, and the Universal-3.5 Pro Realtime release post covers dynamic prompting in context.)

Keyterms aren't your only context lever anymore

Keyterm prompting tells the model which words to expect. Universal-3.5 Pro Realtime adds two more ways to give it context, and they stack with keyterms:

Conversation context (Context Carryover) is on by default. The model keeps a short, rolling memory of the call and uses it as context for the next turn, so it's no longer deciding each utterance cold. Across a 20,000-file voice agent benchmark, passing context cut word error rate by 10.2%, concentrated exactly where agents hurt: short utterances, names, and entities.
agent_context lets a voice agent pass in the question it just asked. Prime the model with "What's your email address?" and a mumbled reply resolves to user@assemblyai.com instead of "user at assembly a i dot com." Spelled-out account IDs, addresses, and one-word confirmations — the short utterances that wreck most realtime models — finally have the context to come out right.

Use keyterms for the specific high-value vocabulary your application can't lose, and use context to sharpen everything around it.

See keyterm boosting on your own audio

Drop in a recording with tricky names or jargon and watch keyterm prompting correct them live — no code required.

Try playground

How to use keyterms without making accuracy worse

More keyterms is not better. Overloading the list — or stuffing it with common words — causes overcorrection and hallucinations, where the model forces a keyterm onto audio that didn't contain it. The guidance that holds up in production:

Start with none. Run your real audio first, find the terms the model consistently misses, and add only those. Don't pre-load a dictionary.
Match spelling and capitalization exactly to the output you want. Keyterms double as a spelling instruction — "Byrne-Donoghue," not "byrne donoghue."
Skip common words. "Information," "account," "today" — the model already handles these, and boosting them just invites false matches.
Keep the list tight. A focused set of genuinely hard, genuinely important terms beats a long list every time.

Think of it as a scalpel for the specific words your application can't afford to lose, not a blanket over the whole transcript.

Where accents fit in

Keyterm prompting is also one of your best levers for accented speech — a question that comes up constantly, especially for heavier accents like Irish, Scottish, or strong regional English. Accents lower the model's confidence on individual words, and low confidence is exactly when a name or technical term slips toward a more common sound-alike. Because turn-level boosting matches phonetically, keyterms are well suited to catch those: the accented pronunciation still maps to the metaphone code of the term you specified.

That said, keyterms aren't the whole answer for accents. Pair them with the right model — Universal-3.5 Pro Realtime shows consistent improvements on accented speech, and its voice_focus option isolates the primary speaker so a noisy room or a second voice doesn't drag accuracy down further — and, critically, test with audio that matches your actual users. A model's accuracy on a clean American-English benchmark tells you almost nothing about how it handles an Irish caller on a noisy phone line. Boost the names and terms that matter, run your real accents through it, and measure what you actually get. Our guide on how to evaluate speech recognition models covers building that kind of test set, including Missed Entity Rate — the metric that actually captures whether your high-value terms survive.

Keyterms also aren't your only prompting lever. When the vocabulary you need to boost is unknown or varies call to call, open-field speech-to-text prompting gives you behavioral control — formatting, verbatim vs. clean, domain context — that a term list can't. On streaming you can run a prompt and keyterms_prompt together; on pre-recorded they're mutually exclusive, so you pick one per request.

Build accurate transcription into your product

Get real-time speech-to-text with keyterm prompting, strong accented-speech accuracy, and dynamic mid-call control. Start free and ship in an afternoon.

The bottom line

Headline accuracy numbers don't decide whether a transcription model works for you — accuracy on the handful of words your application depends on does. Keyterm prompting is the lever that targets exactly those words, through real-time biasing plus a phonetic cleanup pass that fixes sound-alike errors on names and jargon. Use it surgically: start empty, add the terms the model actually misses, spell them the way you want them, and update them as the conversation moves. Layer in conversation context and agent_context on top, and the demo that fell apart on a customer's name becomes the system that gets it right on the first try — which, for anything past the prototype stage, is the only accuracy that counts.

Frequently asked questions

What is keyterm prompting in speech-to-text? Keyterm prompting is a feature that improves transcription accuracy for specific words and phrases you supply through the keyterms_prompt parameter — names, brands, jargon, product terms, and other domain vocabulary. The model biases toward recognizing those terms, so the high-value words your application depends on are far less likely to be mis-transcribed.

How do I improve recognition accuracy for names, jargon, and product terms? Pass them as keyterms. With AssemblyAI you provide an array of terms via keyterms_prompt (up to 100 per streaming session, up to 1,000 for pre-recorded audio), spelled exactly as you want them to appear. The model applies real-time word-level boosting plus a metaphone-based turn-level pass that corrects phonetically similar mistakes — so "Byrne Donahue" becomes "Byrne-Donoghue." On Universal-3.5 Pro Realtime you can stack conversation context and agent_context on top.

How can I transcribe audio with heavy accents, like an Irish accent? Use a model with strong accented-speech accuracy (Universal-3.5 Pro Realtime), turn on voice_focus to isolate the primary speaker, then add keyterm prompting for the names and terms accents most often distort — phonetic boosting maps the accented pronunciation back to the spelling you specified. Most importantly, test with audio from your actual user demographics rather than clean benchmark clips, since accent accuracy is highly specific to the speakers you serve.

Does keyterm prompting work in real time? Yes. On streaming, keyterms are applied live through word-level boosting as audio arrives, then refined by a turn-level boosting pass after each turn. You can also update the keyterms list mid-stream with an UpdateConfiguration message — useful for voice agents that move through stages (verification, intake, payment) with different vocabulary at each.

Can adding too many keyterms hurt accuracy? Yes. Overloading the list or including common words can cause overcorrection and hallucinations, where the model forces a keyterm onto audio that didn't contain it. Start with no keyterms, add only the terms the model consistently misses, keep the list focused, and avoid common words the model already handles well.

How many keyterms can I use, and do they cost extra? Streaming allows up to 100 keyterms per session (50 characters max each); pre-recorded allows up to 1,000 words or phrases (six words max per phrase). Keyterm prompting is included at no extra cost on Universal-3.5 Pro Realtime, alongside conversation context; on Universal-Streaming English and Multilingual it's an add-on.

Real-Time STT Latency: What "Fast Enough" Means

Mart Schweiger — Tue, 23 Jun 2026 21:26:10 +0000

Every voice agent evaluation eventually lands on the same fight: latency. Someone pulls up a benchmark, sees one provider faster than another, and declares a winner. The problem is that the two numbers often aren't measuring the same thing — and even when they are, the faster one isn't always the right call.

If you're choosing a streaming speech-to-text model for a voice agent, latency is the metric most likely to make or break the decision. So it's worth getting precise about what to measure, what the real numbers are, and what "fast enough" actually means — because chasing the lowest number on a chart is how teams end up with a fast agent that mishears every account number.

"Fast enough" starts with how humans talk

Here's the target you're actually aiming for. Research on human conversation finds the average gap between turns is around 200 milliseconds — that's the rhythm people expect, and anything much slower starts to feel like a pause, a lag, a robot thinking.

But that 200ms is the gap between two humans. A voice agent has to do more in that window. When someone stops talking, the full response loop is speech-to-text, then the LLM deciding what to say, then text-to-speech producing the audio. STT is one slice of that budget — and usually not the biggest one. The LLM is often the long pole.

That reframes the whole latency question. The goal isn't to make STT as close to zero as possible. It's to make STT fast enough that it isn't the bottleneck, while leaving accuracy intact. A model that shaves 80ms off transcription but drops a digit in a phone number hasn't made your agent faster — it's made it wrong, faster. (If you'd rather not wire up STT, the LLM, and TTS yourself, the Voice Agent API bundles all three into one connection built on Universal-3.5 Pro Realtime.)

The metrics that actually matter

Most latency arguments fall apart because people use one word — "latency" — for several different measurements. Three matter for voice agents.

Time to first token (TTFT) is how quickly you get the first partial transcript after speech starts. It's what powers barge-in detection and speculative LLM inference — letting your agent start reasoning before the user finishes. With Universal-3.5 Pro Realtime, the interruption_delay parameter tunes this directly: lower it toward 0 for the earliest possible signal, raise it for fewer, more confident partials.

Time to complete turn (TTCT) is the one that decides how responsive your agent feels. It's the interval from when the speaker stops talking to when the final transcript segment arrives — the moment your LLM can actually act. Universal-3.5 Pro Realtime's end-of-turn detection reads how someone speaks — tonality, pacing, rhythm — not just silence, and lands around 300ms. That's the number to anchor on for turn-based conversation.

P50 vs. P95 is the distinction that separates demos from production. Median (P50) latency tells you the typical case. P95 tells you what one in twenty turns looks like — and in a real call, that tail is where conversations stall, agents talk over people, and the experience falls apart. A model with a great median and an ugly P95 will demo beautifully and frustrate users in production.

Benchmark latency on your own audio

Test real-time transcription speed and accuracy with Universal-3.5 Pro Realtime on your actual call audio. Free account, no credit card.

The real numbers

So where does Universal-3.5 Pro Realtime land? Accuracy claims are easy to make and hard to verify, so the honest test is real agent conversations, not clean read speech. On Pipecat's open STT benchmark — measured on actual voice agent audio — Universal-3.5 Pro Realtime posts a market-leading pooled word error rate of 6.99%, against Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%.

The gap widens on the tokens voice agents actually act on. Entity error rate tells you whether names, places, and numbers survive:

Metric	Universal-3.5 Pro Realtime	Deepgram Flux	ElevenLabs Scribe v2	Google Chirp3
Word error rate	6.99%	15.58%	9.76%	9.04%
Entity error rate	15.31%	50.50%	39.70%	21.51%
Names	16.92%	39.21%	38.03%	22.10%
Places	6.28%	14.86%	34.06%	10.04%
Phone numbers	3.55%	10.41%	4.78%	4.95%

A 3.55% phone-number error rate is the difference between an agent that calls the right line back and one that invents a digit. Lower is better on every row, and the full methodology lives on /benchmarks.

Here's the part that ties latency and accuracy together. At end-of-turn detection around 300ms, transcription isn't your bottleneck — your LLM and TTS will each cost more than that. Shaving STT to 250ms doesn't make the loop feel meaningfully faster, because you've optimized the smallest slice. But losing the customer's policy number does break the call. Fast enough plus accurate beats fastest-but-wrong every time the conversation contains something that matters — and with this model you no longer trade one for the other.

See real-time accuracy and speed live

Drop in call audio and watch Universal-3.5 Pro Realtime handle names, numbers, and turn-taking in real time — no code required.

Try playground

How to hit your latency target

"Fast enough" is also tunable, which most benchmark comparisons miss entirely. Instead of hand-tuning a stack of low-level flags, Universal-3.5 Pro Realtime lets you pick a mode when you open the WebSocket:

min_latency for the fastest transcripts, when responsiveness is everything.

balanced (the default) for strong all-around performance.
max_accuracy for noisy, far-field audio where getting the words right is worth a little more time.

From there, the levers that move the finer numbers (our real-time transcription guide shows how to set them in code):

interruption_delay controls TTFT — how soon the first partial lands. Drop it when you're running speculative inference or aggressive barge-in; raise it for fewer, more confident partials.
min_turn_silence controls how quickly an end-of-turn check fires. Lower means snappier turn completion; the trade-off is that setting it too low can split entities like phone numbers mid-sequence — exactly the kind of accuracy loss that masquerades as a latency win.
Speculative inference on partials lets your LLM start working off early partials instead of waiting for the final transcript, hiding STT latency behind reasoning you were going to do anyway.
Pass agent_context. The model takes your agent's question as input, so the reply resolves with more context and fewer re-tries. Across a 20,000-file voice agent benchmark, passing agent context cut word error rate by 10.2% — accuracy you get without spending latency.
Close your sessions. Streaming is billed on connection duration, not audio — a separate point from latency, but the same discipline of not leaving the pipe open.

The other free lever is infrastructure: Universal-3.5 Pro Realtime runs with unlimited concurrency, so your P95 doesn't degrade under load the way rate-limited services do. The latency you measure at one stream is the latency you get at a thousand. Production teams hit this wall when a model that benchmarked well on a single connection falls over at scale.

Don't optimize the wrong number

The most common mistake in voice-agent evals is picking the model with the lowest median latency on a slide and moving on. That decision ignores two things that decide real-world quality: the P95 tail that determines whether turns stall under load, and the entity accuracy that determines whether the transcript is worth acting on. For a deeper framework, our guide on how to evaluate speech recognition models covers measuring latency at the percentile and accuracy level you'll actually deploy at, and the Universal-3.5 Pro Realtime release notes detail the turn-detection design behind these numbers.

Set your latency budget from the full loop, not the STT line item. Decide what total response time feels natural for your agent, subtract realistic LLM and TTS costs, and you'll usually find STT has comfortable headroom around the 300ms range. Within that band, the right question stops being "which model is fastest" and becomes "which model is fast enough and gets the words right." That's the one that wins the eval that matters — the one your users run every time they talk to your agent.

The bottom line

Latency is the right thing to obsess over for voice agents, but "lowest number on the chart" is the wrong way to obsess over it. Anchor on time to complete turn, watch P95 not just the median, and measure latency and accuracy together — because a transcript that arrives 80ms sooner with a mangled account number is slower in every way that counts. Universal-3.5 Pro Realtime is built for that reality: end-of-turn detection around 300ms to stay out of your response budget's way, and a market-leading 6.99% pooled word error rate so the words it delivers in that time are the right ones. If you're evaluating, test it on streaming with your own audio at your own settings — the only benchmark that ever really mattered.

Frequently asked questions

What is the typical latency for real-time speech-to-text? It depends on what you measure. For voice agents, the metric that matters most is time to complete turn (TTCT) — the gap from when a speaker stops to when the final transcript arrives. Universal-3.5 Pro Realtime's end-of-turn detection reads tonality, pacing, and rhythm rather than silence alone and lands around 300ms. Fast streaming models generally aim for the sub-350ms range on turn completion.

How do latency requirements differ for real-time transcription versus batch? Batch (pre-recorded) transcription has no latency constraint — it optimizes purely for accuracy with the full file as context. Real-time streaming has to commit within a few hundred milliseconds from a short rolling window, so it's evaluated on TTFT and TTCT alongside accuracy. Don't compare a batch model's accuracy to a streaming model's; benchmark streaming at the latency you'll actually deploy at.

Can a voice agent respond in under 500ms? The full response loop is STT + LLM + TTS, so sub-500ms end-to-end is tight but the STT slice is small: Universal-3.5 Pro Realtime's end-of-turn detection lands around 300ms, and you can pull the first partial earlier (via interruption_delay) to start LLM inference sooner. Most of a sub-500ms budget is won or lost in the LLM and TTS stages and through speculative inference, not in shaving milliseconds off transcription.

Why isn't the lowest-latency model always the best choice for a voice agent? Because STT latency is only one slice of the response loop, and the words still have to be right. A model that's faster on median but misses entities — names, numbers, emails — produces wrong actions faster, not better conversations. Evaluate latency (especially P95) and entity accuracy together; on Pipecat's agent-conversation benchmark, Universal-3.5 Pro Realtime leads on both word error rate (6.99%) and entity error rate (15.31%).

What's the difference between P50 and P95 latency, and which should I track? P50 (median) is the typical turn; P95 is the slow one in twenty. For production voice agents, P95 is often more telling, because the tail is where conversations stall and the agent talks over the user. Universal-3.5 Pro Realtime runs with unlimited concurrency, so its tail latency holds under load instead of degrading the way rate-limited services do.

How do I tune Universal-3.5 Pro Realtime for my latency target? Start with a mode — min_latency, balanced (default), or max_accuracy — instead of hand-tuning low-level flags. From there, interruption_delay shapes how soon the first partial lands and min_turn_silence shapes how fast a turn completes. Pass agent_context to lift accuracy without spending latency, and run speculative inference on partials to hide STT latency behind LLM reasoning.

Build a Voice Agent Without Pipecat or LiveKit

Mart Schweiger — Tue, 23 Jun 2026 21:25:44 +0000

If you've started building a voice agent in the last year, you've hit this question fast: do I need Pipecat or LiveKit?

The internet says yes. Every tutorial reaches for an orchestration framework before it writes a line of agent logic. And for good reason — those frameworks are genuinely excellent, and AssemblyAI ships drop-in plugins for both. If you're already running one, keep running it.

But "do I need a framework" is the wrong first question. The real question is: what moves the audio, and what wires the pipeline together? Answer those two and the framework question answers itself. A lot of the time, the honest answer is: you don't need one.
This post is an architecture overview of the framework-free path. We'll cover the four things people actually mean when they ask about going without Pipecat or LiveKit:

What a voice pipeline framework actually does for you (and what it costs).
The transport question everyone conflates — SIP vs. WebSocket.
Connecting a phone number with Twilio.
Whether the framework-free architecture holds up at enterprise scale.

Let's start with what you'd be removing.

What a framework actually does for you

A voice agent has to do a few things in a tight loop, every few hundred milliseconds: turn speech into text, decide what to say, turn that back into speech, and move audio in and out — while handling interruptions when the caller talks over the bot.

Historically, each of those was a different vendor. Speech-to-text from one provider, an LLM from another, text-to-speech from a third. Something has to sit in the middle and conduct: route partial transcripts to the LLM, stream the LLM's tokens to the TTS engine, manage turn-taking, and handle barge-in. That conductor is what Pipecat and LiveKit Agents give you, plus a transport layer to get audio to and from the user.

That's real work, and the frameworks do it well. The thing is, it's only necessary work if your pipeline is actually a multi-vendor pipeline.

Here's the cost side, because it's easy to miss when you're following a quickstart. A framework is another dependency in your stack — one more thing to deploy, version, monitor, and reason about when something goes wrong at 2am. You're still wiring three model vendors together, so you still own three sets of API keys, three billing relationships, three failure modes, and three latency budgets that stack. The framework hides that complexity. It doesn't remove it.

So the question becomes: what if the pipeline weren't multi-vendor?

The architecture without a framework

Here's the shift. AssemblyAI's Voice Agent API collapses speech-to-text, the LLM, and text-to-speech into a single WebSocket connection. You stream audio in, you get audio out. Turn detection, interruption handling, and tool calling happen inside that one connection.

When the whole pipeline lives behind one API, there's nothing left to orchestrate. Your "conductor" becomes a single open socket.
Compare the two topologies.

With a framework (cascading, multi-vendor):

    ┌──────────── your orchestration framework ────────────┐

Caller ─▶│  transport ─▶ STT vendor ─▶ LLM vendor ─▶ TTS vendor │─▶ Caller
└──────────────────────────────────────────────────────┘
(you deploy, scale, and monitor this whole box)

Without a framework (single API):

Caller ─▶ thin transport relay ─▶ Voice Agent API ─▶ Caller
(STT + LLM + TTS, one socket)

The Voice Agent API runs on Universal-3.5 Pro Realtime for the speech layer — the model that leads Pipecat's agent-conversation benchmark on both word error rate (6.99%) and entity accuracy, and nails short-utterance handling ("yes," "no," "mmhmm"). It takes the agent's question as context, keeps a rolling memory of the conversation, isolates the speaker from the room, and runs across 18 languages with native code-switching. It's framed as invisible infrastructure on purpose: one connection, one bill, one set of logs. It runs around one second end-to-end, and because there's no SDK to adopt, it works with coding agents like Claude Code out of the box.

What you build instead of an orchestration layer is a thin relay: take audio from wherever your user is, forward it to the WebSocket, and play back what comes off it. That's it. Let's wire it up — first in code, then over the phone.

Connect from your own server

Server-side and native clients connect directly with your API key in the Authorization header. Create an agent once with a single REST call, then connect to it by agent_id:

curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support Assistant",
    "system_prompt": "You are a friendly support agent. Keep replies short and
natural.",
    "greeting": "Hey there, what can I help with?",
    "voice": { "voice_id": "ivy" }
  }'

That returns an id. Now open the WebSocket and bind to it. The agent's stored prompt, voice, and tools load automatically, so you don't resend them:

import asyncio, json, base64, websockets

URL = "wss://agents.assemblyai.com/v1/ws"
AGENT_ID = "7ad24396-b822-4dca-871a-be9cc4781cf9"  # from the create call above

async def main():
    headers = {"Authorization": "YOUR_API_KEY"}
    async with websockets.connect(URL, extra_headers=headers) as ws:
        # Bind to the stored agent; no inline config needed.
        await ws.send(json.dumps({
            "type": "session.update",
            "session": {"agent_id": AGENT_ID},
        }))

        async for raw in ws:
            event = json.loads(raw)
            t = event.get("type")
            if t == "session.ready":
                print("ready:", event.get("session_id"))
                # start streaming input.audio frames here
            elif t == "transcript.agent":
                print("agent:", event.get("text"))
            elif t == "reply.audio":
                pcm = base64.b64decode(event["data"])  # play this
            elif t in ("error", "session.error"):
                print("error:", event.get("message"))

asyncio.run(main())

Notice there's no pipeline graph, no service registry, no turn-detection plugin to configure. The events — session.ready, transcript.agent, reply.audio, input.speech.started for barge-in — are the whole protocol, and it's identical across every transport. For the browser, you'd mint a short-lived token server-side and open wss://agents.assemblyai.com/v1/ws?token= so your key never ships to the client. Same session.update, same events. The full browser walkthrough is in the Voice Agent API quickstart.

See real-time Voice AI in action

Drop in call audio and watch Universal-3.5 Pro Realtime handle names, numbers, and turn-taking in real time — no code required.

Try playground

The transport question: SIP vs. WebSocket

Here's where most of the confusion lives, so let's settle it.
Audio has to physically move between the user and your agent. There are three common ways to move it:

WebRTC — the browser-and-app real-time standard. It handles NAT traversal, jitter buffering, and echo cancellation. This is LiveKit's and Daily's home turf, and it's the right tool for multi-party rooms, video, and rich client SDKs.
SIP — the signaling protocol of the telephone network. If you're terminating raw PSTN calls or running your own telephony infrastructure, you're in SIP territory.
WebSocket — a plain, bidirectional socket. No media server, no SFU, no signaling dance. You send audio frames, you receive audio frames.

AssemblyAI takes audio over a WebSocket. That's the entire transport story for the STT layer and the Voice Agent API alike.

"But my users are on the phone — don't I need SIP?" This is the part people get wrong. You don't speak SIP yourself, and you don't stand up a WebRTC media server either. Your telephony provider does that for you. Twilio terminates the PSTN call, handles the SIP side, and hands you the call's audio over — you guessed it — a WebSocket. So the agent is WebSocket on the user side and WebSocket on the AssemblyAI side. It's sockets all the way through.

That's why you can skip the framework's transport layer for telephony: the carrier already converted the hard part into a WebSocket before it reaches you. Your job is just to bridge two sockets.

Connecting a phone number with Twilio

Let's make the phone case concrete, because it's the one that sounds like it should need the most machinery and actually needs the least.
The end-to-end path is four hops:

Caller ↔ Twilio Media Streams ↔ Your server ↔ Voice Agent API

Twilio handles the phone network. Your server is a thin bridge. The agent handles speech-to-speech. And here's the detail that removes a whole class of pain: Twilio's native G.711 μ-law audio is byte-compatible with the Voice Agent API's audio/pcmu encoding, so your bridge forwards audio as-is, with zero transcoding or resampling.
When a call comes in, Twilio hits a webhook on your server. You return TwiML that opens a Media Streams WebSocket:

app.post("/twiml", (req, res) => {
  const callId = newCallId();
  const hostname = process.env.HOSTNAME.replace(/^https?:\/\//, "");
  const streamUrl = `wss://${hostname}/media-stream/${callId}`;

  res.type("text/xml").status(200).send(
    `<Response>
  <Connect>
    <Stream url="${streamUrl}" />
  </Connect>
</Response>`,
  );
});

When Twilio opens that Media Streams socket, your server opens a parallel connection to the Voice Agent API and configures the session inline — telling it to speak μ-law in both directions to match Twilio:

const aaiWs = new WebSocket("wss://agents.assemblyai.com/v1/ws", {
  headers: { Authorization: process.env.ASSEMBLYAI_API_KEY },
});

aaiWs.send(JSON.stringify({
  type: "session.update",
  session: {
    system_prompt: "You are a helpful voice assistant.",
    greeting: "Hi, thanks for calling. How can I help?",
    input: { type: "audio", format: { encoding: "audio/pcmu" } },
    output: { type: "audio", voice: "ivy", format: { encoding: "audio/pcmu" } },
    tools: [/* your tool definitions */],
  },
}));

Once session.ready fires, the whole bridge is two forwarding rules plus one for interruptions:

// Caller → Agent: each Twilio media event becomes an input.audio event.
tw.on("media", (msg) => {
  if (msg.media.track !== "inbound") return;
  aaiWs.send(JSON.stringify({ type: "input.audio", audio: msg.media.payload }));
});

aaiWs.on("message", (data) => {
  const event = JSON.parse(data.toString());

  // Agent → Caller: each reply.audio event becomes a Twilio media action.
  if (event.type === "reply.audio" && event.data) {
    tw.send({ event: "media", streamSid: tw.streamSid, media: { payload:
event.data } });
  }

  // Barge-in: caller talks over the agent → clear Twilio's buffer so it stops
instantly.
  if (event.type === "input.speech.started") {
    tw.send({ event: "clear", streamSid: tw.streamSid });
  }
});

That's a production phone agent. No SIP stack, no media server, no orchestration framework — a webhook and a socket bridge. AssemblyAI maintains a complete Twilio example repo with inbound and outbound calling and tool handling wired up, and the full walkthrough lives in Connect to Twilio. The same pattern applies to other carriers — Twilio, Vonage, and the rest all hand you a media WebSocket.

What if you only want to swap the STT layer?

Not everyone is building net-new. Maybe you already have an LLM and a TTS voice you love, and you only want better transcription — the rip-and-replace-one-piece path. You can do that without a framework too.
Connect straight to the streaming speech-to-text WebSocket and keep the rest of your stack exactly as it is:

import json
from urllib.parse import urlencode
import websocket  # pip install websocket-client

API_KEY = "YOUR_API_KEY"
# Streaming uses speech_model (singular). universal-3-5-pro is the flagship realtime model.
params = {"speech_model": "universal-3-5-pro", "sample_rate": 16000}
ENDPOINT = f"wss://streaming.assemblyai.com/v3/ws?{urlencode(params)}"

def on_message(ws, message):
    data = json.loads(message)
    if data.get("type") == "Turn":
        if data.get("end_of_turn"):
            print("Final:", data.get("transcript"))   # → send to your own LLM
        else:
            print("Partial:", data.get("transcript"), end="\r")
    elif data.get("type") == "SpeechStarted":
        print("Barge-in: user started speaking")       # → interrupt your TTS

# Authenticate with the raw key in the header — no "Bearer" prefix.
ws = websocket.WebSocketApp(ENDPOINT, header={"Authorization": API_KEY},
                            on_message=on_message)
ws.run_forever()
# Stream 16 kHz PCM16 audio in ~50 ms chunks as binary WebSocket frames.

You hold the loop — partial transcripts to your LLM, tokens to your TTS, SpeechStarted to trigger barge-in. It's more wiring than the all-in-one Voice Agent API, but it's still a direct socket, and you're still not deploying a framework. If you want a longer build, our real-time transcription in Python walkthrough goes step by step.

A couple of details that trip people up: streaming uses speech_model (singular) — that's the opposite of the pre-recorded API, which takes a speech_models array for fallback routing. And universal-3-5-pro uses punctuation-based turn detection, so end_of_turn_confidence_threshold is a no-op on it. If your session is monolingual, pass language_code to commit the model to one language; leave it off to keep native code-switching.

Does this hold up at enterprise scale?

A fair pushback: "single API is fine for a demo, but does it survive production?" This is where collapsing the pipeline actually helps instead of hurts.****

Fewer moving parts, fewer failure modes. A three-vendor cascade has three things that can rate-limit you, three that can have an incident, and three latency budgets that compound. One connection has one of each. When you're the one carrying the pager, that math matters more than any benchmark.****

Concurrency that scales with you. Pay-as-you-go accounts start at 100 new streams per minute with no hard cap on concurrent sessions, and capacity scales up automatically — roughly a 10% bump every 60 seconds once you cross 70% utilization. AssemblyAI runs millions of hours of audio and 600M+ inference calls a month, with unlimited concurrency on the platform.****

Pricing you can forecast. The Voice Agent API is a flat $4.50/hr that bundles STT, the LLM, and TTS. There's no separate per-vendor metering to model out, which makes capacity planning a spreadsheet instead of a guessing game. (Prefer to bring your own LLM and TTS? The streaming STT layer is billed separately on its own pricing — $0.45/hr for Universal-3.5 Pro Realtime, with context and keyterm prompting included.)****

The compliance and residency boxes. SOC 2 Type 2, EU data residency via streaming.eu.assemblyai.com at the same price as the US, and a self-hosted option if the audio can't leave your VPC. For teams handling protected health information (PHI), AssemblyAI is a business associate under HIPAA and offers a Business Associate Addendum (BAA) you can sign in minutes, without a sales call.

None of this is a reason to avoid a framework — plenty of large deployments run Pipecat or LiveKit happily. It's a reason to know that the framework-free path isn't a toy. For an honest look at where any voice agent architecture starts to strain, see where voice agent stacks start showing their limits.

When you should use Pipecat or LiveKit

Skipping the framework is the right call for a large share of voice agents — especially phone agents, support bots, and anything where the job is a clean speech-to-speech loop. But not all of them. Reach for Pipecat or LiveKit when:

You need real WebRTC features — multi-party rooms, video alongside voice, screen share, or polished mobile and web client SDKs. This is LiveKit's core strength, and a WebSocket bridge won't replicate it.
You want best-of-breed per stage — a specific TTS voice, a particular LLM, or a custom turn-detection model, mixed and matched per use case. Frameworks are built for exactly this kind of swapping.
You already run one in production. If Pipecat or LiveKit is in your stack, don't rip it out. AssemblyAI ships first-class plugins for both, so you can run Universal-3.5 Pro Realtime as your STT inside the framework you already trust and get the same accuracy either way.
You need fine-grained control over the pipeline graph — custom processors, intricate branching, frame-level manipulation.

The decision is less "framework vs. no framework" and more "how much of the pipeline do I actually need to control."

Choosing your path

You want to…	Use	Why
Ship a phone or web voice agent fast	Voice Agent API (direct WebSocket)	One socket for STT + LLM + TTS; flat $4.50/hr; no framework to deploy
Keep your own LLM/TTS, upgrade only STT	Streaming STT (direct WebSocket)	Swap one piece; you keep the loop
Build multi-party rooms or add video	LiveKit Agents + AssemblyAI plugin	WebRTC features a socket bridge can't match
Mix-and-match vendors with fine control	Pipecat + AssemblyAI plugin	Framework-grade pipeline control
Add a phone number to any of the above	Your telephony provider (e.g. Twilio)	Carrier terminates SIP/PSTN, hands you a WebSocket

The takeaway

The voice agent ecosystem defaulted to orchestration frameworks for a sound reason: for years, building an agent meant gluing three vendors together, and that glue is real engineering. Frameworks made the glue manageable.

But the default quietly assumes the pipeline has to be multi-vendor. The moment one API owns speech-to-text, the LLM, and text-to-speech behind a single WebSocket — and your carrier hands you telephony audio over a WebSocket too — the orchestration layer doesn't get easier. It disappears. What's left is a socket on each side and a few lines to forward audio between them.

So before you add Pipecat or LiveKit to your next voice agent, ask the two questions that actually matter: what's moving my audio, and is my pipeline really multi-vendor? If the answers are "a WebSocket" and "no," you've already got your architecture.

Build your voice agent today

Get a free API key and $50 in credits to ship your first voice agent — one WebSocket for speech-to-text, the LLM, and text-to-speech, at a flat $4.50/hr, with no framework to deploy.

Frequently asked questions

Do you need Pipecat or LiveKit to build a voice agent?

No. Pipecat and LiveKit orchestrate a multi-vendor pipeline — they wire together separate speech-to-text, LLM, and text-to-speech services and manage turn-taking and transport between them. If a single API handles all three, like AssemblyAI's Voice Agent API does over one WebSocket, there's nothing left to orchestrate. Reach for a framework when you need WebRTC features like multi-party rooms or video, or fine-grained control over each pipeline stage.

Do voice agents use SIP or WebSocket for audio transport?

Voice agents can move audio over WebSocket, WebRTC, or SIP, and the three solve different problems. AssemblyAI takes audio over a plain WebSocket — no media server or SFU required. For phone calls you don't implement SIP yourself: your telephony provider (such as Twilio) terminates the PSTN and SIP side and hands you the call's audio over a WebSocket, so the whole agent ends up being WebSocket on both ends.

What's the difference between the AssemblyAI Voice Agent API and a framework like Pipecat or LiveKit?

An orchestration framework connects separate STT, LLM, and TTS vendors and coordinates turn-taking, interruptions, and transport across them. The Voice Agent API replaces those three providers with one WebSocket connection — one bill, one set of logs, and one latency budget instead of three. Frameworks give you per-stage swappability and deep pipeline control; the single API gives you fewer moving parts. You don't have to choose blindly, either: AssemblyAI ships plugins for both Pipecat and LiveKit, so you can run Universal-3.5 Pro Realtime inside whichever framework you already use.

Can you build a Twilio voice agent without a framework?

Yes. Twilio terminates the phone call and streams G.711 μ-law audio to your server over a Media Streams WebSocket, and your server bridges that audio to the Voice Agent API, which accepts μ-law (audio/pcmu) natively with zero transcoding. The whole bridge is a webhook that returns TwiML plus a few lines that forward audio in each direction — no SIP stack, no media server, and no orchestration framework. AssemblyAI maintains a complete Twilio example repo with inbound calling, outbound calling, and tool handling.

Does a voice agent without an orchestration framework scale to production?

Yes. AssemblyAI's streaming platform starts at 100 new streams per minute on pay-as-you-go with no hard cap on concurrent sessions, and capacity scales up automatically under load. Collapsing speech-to-text, the LLM, and text-to-speech into one connection actually reduces operational risk at scale — there are fewer vendors to rate-limit you, fewer services that can have an incident, and fewer latency budgets that stack. Pricing is a flat $4.50/hr, with EU data residency and a self-hosted deployment option for teams with strict data requirements.

How do I build a voice agent without Pipecat or LiveKit?

Create an agent with a single REST call to define its prompt, voice, and tools, then open a WebSocket to wss://agents.assemblyai.com/v1/ws and bind it by agent_id in your first session.update message. From there you stream PCM audio in, play the agent's audio back out, and handle barge-in when the input.speech.started event fires. The same connection and event protocol work from a browser (using a short-lived token), from your own server, or over the phone through Twilio.

Prompting Claude to Build Voice Agents

Mart Schweiger — Tue, 23 Jun 2026 21:25:37 +0000

Ask Claude to build you a voice agent and you'll get working code back in about thirty seconds. That part is easy now. The gap between that first draft and something you'd actually put in front of customers — that's where the prompting craft lives.

Most of the difference doesn't come from clever wording. It comes from what you put in front of the model before you ask for anything. A coding agent is only as good as its context, and for voice agents the two things that matter most — current API docs and a real sense of how your calls actually go — are exactly the two things the model doesn't have by default.

So this isn't a list of magic prompts. It's four techniques that consistently move a voice agent from "demo that compiles" to "agent that ships":

Ground the agent in live docs before you prompt anything.
Pick your coding model deliberately — but know what actually moves the needle.
Keep business context out of the first prompt.
Feed the model real call transcripts as context.

If you're brand new to this, start with our walkthrough on how to vibe code a voice agent — it covers the basics of what comes out the other side. This post assumes you've done that once and want the version that holds up.

Ground the agent in live docs first

Here's the failure mode that wrecks more voice agent builds than any other: the model writes confident, clean, completely outdated code.
Voice APIs move fast. Endpoints get versioned, parameters get renamed, whole capabilities get added. AssemblyAI's streaming endpoint moved to wss://streaming.assemblyai.com/v3/ws a while back — but ask a coding agent working from training data alone, and there's a real chance it reaches for the retired v2/realtime URL, writes it perfectly, and hands you something that simply won't connect. The code looks right. It isn't. And you lose an hour learning that the hard way. The same goes for the newest parameters: features like agent_context and conversation carryover on Universal-3.5 Pro Realtime don't exist in any model's training data, so an ungrounded agent can't use the levers that most improve accuracy.

The fix is to stop letting the model guess. Point it at the live docs before your first real prompt. AssemblyAI publishes a docs MCP server exactly for this — it lets a coding agent query current documentation instead of recalling stale training data. In Claude Code, add it once:

claude mcp add --transport http --scope user assemblyai-docs
https://mcp.assemblyai.com/docs

There's also a skill that bundles the correct patterns and gotchas:

npx skills add AssemblyAI/assemblyai-skill --global

Now when you ask Claude to wire up streaming transcription or the Voice Agent API, it pulls the current endpoint, the current parameters, and the current auth flow. The hallucinated-endpoint class of bug mostly disappears. This one step does more for code quality than any amount of prompt tuning.

It's not Claude-specific, either. Any MCP-aware tool — Cursor, Windsurf, Codex — can connect to the same server, and prompt-based builders like Lovable or v0 can take the docs URL (https://www.assemblyai.com/docs/voice-agents/voice-agent-api) right in the prompt. Which leads to the question everyone asks next.

Point your coding agent at the docs

Grab a free API key, connect the AssemblyAI docs MCP server, and let your agent write current, correct voice AI code instead of guessing from stale training data.

Get your free API key

Claude or Codex? The model matters less than you think

Developers ask this constantly: Claude or Codex for building voice agents? It's a fair question, and the honest answer is the less satisfying one — for this task, the coding agent you pick matters less than whether it can see current docs.

Both are strong. Codex is a capable coding agent and plenty of people ship good work with it. On the Claude side, if you want the most capable model for the kind of long, multi-step, agentic build a voice agent turns into, that's Claude Opus 4.8 — run it at xhigh effort, which is the default Claude Code uses for coding and agentic work. If you care more about speed and cost on a tighter loop, Claude Sonnet 4.6 is the better trade. We won't pretend to hand you a head-to-head leaderboard here; the relevant comparison shifts every few weeks, and you should run your own task on both.

But here's the thing that doesn't shift: a voice agent's correctness depends on getting transport, turn detection, and audio formats exactly right, and none of those live in any model's training data at the version you need. Both Claude and Codex write better voice-agent code with the AssemblyAI docs MCP server attached than either does without it. The grounding is the variable that actually controls your outcome. The model is a smaller one.

For what it's worth, the Voice Agent API was built to work with Claude Code out of the box — one WebSocket, JSON in and out, no SDK to adopt — so if you're starting fresh and undecided, that's the path of least resistance. If you already live in Codex, point it at the same MCP server and keep moving. Either way, ground it first.

Keep business context out of the first prompt

This one feels backwards, so stick with me.

When people sit down to build a customer-facing agent, the instinct is to pour everything into the opening prompt: the product catalog, the refund policy, the escalation rules, the twelve edge cases sales keeps hitting. It feels thorough. It produces a mess.

The problem is that you've asked the model to solve two different problems at once — get the real-time pipeline working and encode your entire business — before either of you has confirmed the pipeline even runs. You get a sprawling first draft where the architecture and the domain logic are tangled together, and when something breaks (it will), you can't tell whether it's the WebSocket bridge or your refund rules.

Split it into two phases. The first prompt is about the engineering, and you should spec that fully and up front: the transport, the model, the latency target, barge-in handling, the STT→LLM→TTS loop. Be specific and complete here — modern models like Opus 4.8 do their best work when the technical task is well-defined in one shot.

What you leave out is the domain:

Build a phone voice agent on AssemblyAI's Voice Agent API, bridged through Twilio over a WebSocket. Target
~1 second end-to-end latency, handle barge-in, and give me a clean way to plug in a system prompt and tools
later. Don't add any business logic yet — I want a working loop I can call and talk to first.

That last sentence is the whole technique. You get back a skeleton you can actually dial and have a conversation with. You verify the hard part — the real-time plumbing — in isolation. Then you layer in the business context, against a foundation you already trust. Which is exactly where the next technique comes in, because the best business context isn't written from memory.

Feed Claude real call transcripts as context

Once you've got a working skeleton, the most valuable thing you can hand the model isn't a document describing how your calls should go. It's a pile of transcripts showing how they actually go.

Think about what a real call contains that a spec never will: the exact way customers say their account numbers, the product names they mangle, the three different ways people ask for a refund, where they interrupt, what they say when they're confused. That's the raw material for a system prompt that sounds right, a keyterms list that catches the entities your agent will really hear, and a tool list that matches what customers actually ask for. You can guess at all of it. Or you can read it off real calls.

You almost certainly have the recordings sitting in your contact center or call platform already. Transcribe them with AssemblyAI's pre-recorded speech-to-text API — Universal-3 Pro for the highest accuracy, with speaker labels so the agent's lines and the caller's lines stay separate:

import os
import assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

# Universal-3 Pro for accuracy; speaker_labels splits agent vs. caller.
config = aai.TranscriptionConfig(
    speech_models=["universal-3-pro"],
    speaker_labels=True,
)
transcriber = aai.Transcriber(config=config)

# Transcribe a folder of real call recordings into plain-text transcripts.
os.makedirs("transcripts", exist_ok=True)
for filename in os.listdir("recordings"):
    transcript = transcriber.transcribe(os.path.join("recordings", filename))
    if transcript.error:
        print(f"Skipped {filename}: {transcript.error}")
        continue

    # Speaker-labeled turns read the way a conversation actually flows.
    lines = [f"{u.speaker}: {u.text}" for u in transcript.utterances]
    out_path = os.path.join("transcripts", filename.rsplit(".", 1)[0] + ".txt")
    with open(out_path, "w") as f:
        f.write("\n".join(lines))
    print(f"Wrote {out_path}")

Now point Claude at the transcripts/ folder and let the real conversations do the work:

Read these 50 real support call transcripts. From them: (1) draft a system prompt for a voice agent that
handles these calls, in the tone the reps actually use; (2) build a keyterms list of the product names,
account formats, and domain terms that show up, so streaming transcription gets them right; (3) list the
tools this agent would need to resolve these calls without a human.

The output is grounded in evidence instead of imagination. The keyterms list contains the SKUs your customers actually say, not the ones you remembered. The system prompt mirrors how your best reps actually talk. And because Universal-3.5 Pro Realtime lets you update keyterms mid-session — and carries conversation context from turn to turn on its own — you can keep feeding that list back into the live agent as you learn more.
This is the technique that's genuinely worth stealing. Most teams prompt their agent from a blank page. The teams whose agents sound human are reading from the transcript.

Transcribe your own calls

Run a real call recording through Universal-3 Pro in the playground and see the transcript accuracy — entities, speaker labels, and all — that your agent's context will be built on.

Try playground

The two-phase workflow, end to end

Put the four techniques in order and you get a repeatable way to build:

Phase	What you prompt	What you feed the model	What you get
Setup	—	AssemblyAI docs MCP server + skill	An agent that writes current, correct API code
Phase 1 — skeleton	The full engineering spec; no business logic	Live docs	A working STT→LLM→TTS loop you can call and verify
Phase 2 — ground it	"Build the prompt, keyterms, and tools from these"	Real call transcripts	A system prompt, keyterms, and tools grounded in real calls
Tune	Targeted fixes as you test	Fresh transcripts of your agent's own calls	An agent that keeps getting more accurate

Notice the model brand never appears in that table. Claude or Codex, Opus 4.8 or Sonnet 4.6 — the workflow is the same, and the workflow is what carries the quality.

The takeaway

The interesting shift in building voice agents isn't that AI can write the code. It's that the bottleneck moved. It used to be syntax and SDK archaeology. Now it's context — and context is something you control completely.
Stale docs and remembered business rules give you a demo. Live docs and real transcripts give you an agent. Whichever model you prompt, that's the lever. Ground it, spec the engineering cleanly, and let your own calls write the hard part.

Build your voice agent today

Get a free API key and $50 in credits, point your coding agent at the AssemblyAI docs, and ship your first voice agent on a single WebSocket — no SDK required.

Get your free API key

Frequently asked questions

Can you build a voice agent just by prompting Claude?

Yes. Claude — through Claude Code — can scaffold a complete voice agent on AssemblyAI's Voice Agent API, including the speech-to-text, LLM, and text-to-speech loop and the transport bridge. The quality depends far more on context than on wording: give it current documentation through the AssemblyAI docs MCP server and ground it in real call transcripts, rather than letting it work from training data alone. Spec the engineering task first, get a working loop, then layer in your business logic.

How do I stop an AI coding agent from writing outdated AssemblyAI code?

Connect the agent to the AssemblyAI docs MCP server so it queries current documentation instead of recalling stale training data. In Claude Code, run claude mcp add --transport http --scope user assemblyai-docs https://mcp.assemblyai.com/docs, and optionally add the skill with npx skills add AssemblyAI/assemblyai-skill --global. This eliminates the most common failure — confidently written code that uses a retired endpoint or a renamed parameter, or that misses newer capabilities like agent_context on Universal-3.5 Pro Realtime. Any MCP-aware tool, including Cursor and Codex, can use the same server.

Claude or Codex — which is better for building voice agents?

Both are capable coding agents, and you should run your own task on each rather than trust a leaderboard that shifts every few weeks. Among Claude models, Opus 4.8 is the most capable for the long, multi-step build a voice agent becomes — run it at xhigh effort, the default Claude Code uses for coding work — while Sonnet 4.6 trades some capability for speed and cost. For voice agents specifically, the bigger lever is whether the agent can see current API docs: both Claude and Codex write better code with the AssemblyAI docs MCP server attached, and the Voice Agent API works with Claude Code out of the box.

How do I use real call recordings to build a better voice agent?

Transcribe your existing call recordings with AssemblyAI's pre-recorded speech-to-text API — Universal-3 Pro with speaker labels turned on — then hand the transcripts to your coding agent as context. Real calls reveal the actual account-number formats, product names, phrasing, and turn-taking your agent will face, so the system prompt, keyterms list, and tool definitions get built from evidence instead of guesswork. Because Universal-3.5 Pro Realtime lets you update keyterms mid-session, you can keep feeding what you learn back into the live agent.

Should I include my business logic in the first prompt?

No — separate the engineering prompt from the domain prompt. Spec the full technical task (transport, model, latency target, barge-in handling) up front and get a working speech-to-text → LLM → text-to-speech loop you can call and verify first. Add the business context — system prompt, keyterms, tools — in a second phase, ideally generated from real call transcripts. Dumping everything into the opening prompt tangles the architecture with domain logic and makes it much harder to tell what broke when something does.

What do I need to prompt Claude to build a voice agent on AssemblyAI?

You need a free AssemblyAI API key, a coding agent (Claude Code is the smoothest path), and the AssemblyAI docs MCP server connected so the agent writes current code. From there, prompt for the engineering skeleton first, then ground it with transcripts of your real calls. The Voice Agent API is a single WebSocket with one flat rate and no SDK to adopt, which keeps the code the agent generates simple enough to verify quickly.

One Parameter, 20% Fewer Missed Medical Entities

Mart Schweiger — Tue, 23 Jun 2026 21:25:11 +0000

You already have a pipeline. You're already sending clinical audio to AssemblyAI. So this post isn't going to teach you what a transcript is or why drug names are hard. You know.

Here's the part that matters: there's a single config parameter that gets you about 20% fewer missed medical entities on the exact same audio you're already sending. No new model. No re-integration. No new API key. One line.

That parameter is domain: "medical-v1", and it turns on Medical Mode.

Let me show you what changes, what doesn't, and what it costs.

The one parameter

Here's your existing call to Universal-3 Pro, with Medical Mode added:

import assemblyai as aai

transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(
speech_models=["universal-3-pro"],
domain="medical-v1",   # the one parameter
)
transcript = transcriber.transcribe("clinical-audio.wav", config)

That's it. That's the diff. If you were already on Universal-3 Pro, you add one field to the config object you already build. Everything downstream of the transcript—your diarization handling, your redact_pii step, your storage—stays exactly where it is.

Notice what you didn't do. You didn't swap to a different model family. Medical Mode runs on top of Universal-3 Pro and Universal-3.5 Pro Realtime, so the decoder, the language coverage, the entity accuracy you already rely on—all still there. You're tuning the model you're already running, not replacing it.

What actually changes in the transcript

Universal-3 Pro is already strong on entities—drug names, proper nouns, rare words—because it uses an LLM-based decoder rather than a classic acoustic-only approach. Medical Mode pushes that further specifically for clinical vocabulary. The result: a 3.2% Missed Entity Rate (MER), roughly 20% fewer missed medical entities than Universal-3 Pro alone, and the lowest MER across the providers we benchmarked against—Deepgram, Speechmatics, AWS, and Google. The full numbers live on /benchmarks.

But abstract percentages don't tell you what to expect in your own output. So here's a concrete picture.
Illustrative example. The transcript pair below is constructed to show the kind of errors Medical Mode catches—it is not measured data. The measured numbers come from /benchmarks.

A clinician dictates a short medication summary. Without Medical Mode, a general model might produce:

"Patient continues on metformin 500 milligrams twice daily. Started hydrochlorothiazide for the
hypertension, and we'll reassess after the echocardiogram."

…but the errors a general model tends to make cluster exactly where it hurts:

"Patient continues on Metro Min 500 milligrams twice daily. Started hydrocortisone for the hypertension, and
we'll reassess after the echo cardiogram."

Read that second version as a downstream system would. "Metro Min" isn't a drug. "Hydrocortisone" is a real drug—just the wrong one, and a dangerous swap for a thiazide diuretic. "Echo cardiogram" splits a procedure into two tokens your coding logic won't recognize.

With Medical Mode on, the same audio resolves to:

"Patient continues on metformin 500 milligrams twice daily. Started hydrochlorothiazide for the
hypertension, and we'll reassess after the echocardiogram."

The dosage was never the hard part. The entities were. That's the whole point of measuring missed entities separately from raw word error—a model can nail the filler words and still hand you the wrong drug.****

Want to see it on your own audio? Activate Medical Mode in your dashboard and re-run a file you've already transcribed. Open your dashboard →

What does not change

This is the part I want to be loud about, because "improve medical accuracy" usually implies a migration project. Here it doesn't.

No model swap. You stay on Universal-3 Pro or Universal-3.5 Pro Realtime. Medical Mode is a setting on those models, not a separate endpoint.
No re-integration. Same SDK, same Transcriber(), same response shape. Your parsing code doesn't move.
No new API key. The key you're using right now works. There's nothing to provision.
No language regression. Medical Mode supports English, Spanish, German, and French—both pre-recorded and streaming. If your traffic is multilingual, the clinical tuning follows the language.

If you've ever scoped a "switch transcription providers for better medical accuracy" ticket, you know it usually runs weeks. This is a config change you can ship in an afternoon and roll back just as fast.

Async and streaming, same flag

The example above is pre-recorded. Streaming works the same way—you set domain: "medical-v1" on Universal-3.5 Pro Realtime and keep your existing socket handling.

This matters for ambient clinical use. If you're transcribing a live encounter—an ambient scribe, a telehealth visit, a nurse triage line—you get the same entity tuning at streaming latency. Universal-3.5 Pro Realtime's end-of-turn detection reads tonality, pacing, and rhythm rather than silence alone and lands around 300ms, and turning on Medical Mode doesn't change how you consume partial and final transcripts. You're not trading speed for accuracy here; you're getting the clinical vocabulary handling inline.

One nuance worth knowing if you use keyterms prompting: streaming supports up to 100 keyterms for free, mid-stream. Medical Mode and keyterms aren't mutually exclusive—Medical Mode handles the broad clinical vocabulary, and you can still prompt the handful of facility-specific terms (a local formulary name, a clinic's procedure shorthand) that no general medical model would know.

What it costs

Medical Mode is a $0.15/hr add-on. It stacks on top of your base model price:

Universal-3 Pro + Medical Mode = $0.36/hr (the $0.21/hr async base plus the add-on)
Universal-3.5 Pro Realtime + Medical Mode = $0.60/hr (the $0.45/hr streaming base plus the add-on)

Full breakdown on the pricing page.

Here's how I'd think about that fifteen cents. The cost of a missed entity isn't the audio minute—it's the downstream correction. A transcript that turns hydrochlorothiazide into hydrocortisone doesn't just need a re-listen; in a clinical workflow it can trigger a manual review, a clinician callback, or worse. Cutting missed entities by ~20% changes how much human QA your pipeline needs. For most teams running clinical audio at volume, that math closes fast.

A note on PHI handling

Since you're processing clinical audio, the compliance question comes up. AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA). Medical Mode runs on that same BAA-eligible infrastructure, BAA included.

Practically, that means the tools you'd reach for—diarization to separate clinician from patient, and redact_pii to strip PHI from stored transcripts—are available alongside Medical Mode, on the same models. You're not assembling a separate compliance stack.

Frequently asked questions

Do I have to migrate off my current model to use Medical Mode?

No. Medical Mode is a domain: "medical-v1" setting on Universal-3 Pro and Universal-3.5 Pro Realtime. If you're already on either, you add one parameter—no model swap, no re-integration, no new key.

Does Medical Mode work for live transcription?

Yes. It works on Universal-3.5 Pro Realtime with the same flag, so ambient scribes and telehealth pipelines get clinical entity tuning at streaming latency.

Which languages does Medical Mode support?

**** English, Spanish, German, and French, for both pre-recorded and streaming audio.

How much does it add to my bill?

$0.15/hr on top of the base model. That's $0.36/hr for Universal-3 Pro async and $0.60/hr for Universal-3.5 Pro Realtime.

Can I use keyterms prompting and Medical Mode together?

Yes. Medical Mode covers broad clinical vocabulary; keyterms cover your facility-specific terms—up to 100 free mid-stream on streaming, up to 1,000 on async for an additional $0.05/hr.

How We Measure Medical Transcription: MER vs. WER

Mart Schweiger — Tue, 23 Jun 2026 21:25:05 +0000

Word error rate has a flattering quality. It rolls every mistake into one clean percentage, and a clean percentage is easy to put on a slide.

It's also lying to you about medical audio.

Here's the problem in one sentence: WER treats every word as equal. Missing an "um" costs exactly the same as turning "hydrochlorothiazide" into "hydrocortisone." A filler word and a beta-blocker, weighted identically.

That's fine for a podcast. It's indefensible for a clinical transcript. So let me walk through why WER misleads evaluators, what we measure instead, and how the numbers actually shake out.

What WER counts, and what it ignores

WER is substitutions plus insertions plus deletions, divided by the number of reference words. Every error in that numerator carries weight one. The metric has no concept of which words matter.

Think about what a medical conversation actually contains. The vast majority of words are connective tissue—"the," "and," "patient," "we'll," "okay." A handful of words carry the entire clinical meaning: the drug, the dose, the diagnosis, the procedure. Maybe 5% of the tokens hold 95% of the risk.

WER averages across all of them. So a model can flub the 5% that matter and still post a great-looking score, because it nailed the 95% that don't. We've written before about how word error rate is broken and how your WER benchmark might be lying to you—this is the same disease in its most dangerous form.
Consider two transcription errors:

"uh, the patient" → "the patient" (dropped a filler)
"metoprolol" → "metformin" (a cardiac drug became a diabetes drug)

To WER, both are one error. One of them is noise. The other could change a treatment decision. A metric that can't tell them apart is the wrong metric for healthcare.

Missed Entity Rate, defined

So we measure the thing that actually matters: how often the clinically meaningful words come out wrong.
Missed Entity Rate (MER) is the percentage of medical entities not correctly transcribed. By "entities" we mean the words a clinician or a downstream coding system depends on:

Drug names—generic and brand
Diagnoses and conditions
Procedures
Dosages and units

MER ignores whether the model dropped an "um." It asks a narrower, harder question: when a drug name appeared in the audio, did the transcript get it right? When a procedure was named, did it survive intact, or did "echocardiogram" become "echo cardiogram" and fall out of your entity extraction?

This is the metric that maps to clinical risk. A model can sit at a respectable WER and still have a terrible MER—and if you're building anything that touches patient care, MER is the number you should be staring at.****

Want to see this on your own audio instead of ours?

Run the benchmark on representative files from your domain.

Try playground

The benchmark

Here's where the two metrics part ways. We benchmarked Universal-3 Pro with Medical Mode against the providers evaluators actually shortlist, measuring both MER and WER. Lower is better for each; MER is the share of entities not correctly transcribed.

Provider / configuration	MER	WER
AssemblyAI Universal-3 Pro w/ Medical Mode	3.2%	5.3%
Deepgram	3.6%	5.5%
Speechmatics Enhanced Medical	4.7%	6.1%
Deepgram Nova-3 Medical	8.7%	5.9%
AWS Transcribe Medical	24.4%	12.9%
Google Medical Conversation	—	—

Read the table for a second before the conclusion, because the rows tell the story better than I can.

Look at Deepgram Nova-3 Medical: 5.9% WER, 8.7% MER. If you'd evaluated on WER alone, you'd see a single point of difference from the top of the table and shrug. But its MER is more than double ours—it's missing entities at almost three times the rate while looking nearly identical on the headline metric. That's WER's flattery in action.
AWS Transcribe Medical makes the gap impossible to miss: 12.9% WER, but a 24.4% MER. Nearly a quarter of medical entities not transcribed correctly. The WER alone wouldn't scream that loudly.

Universal-3 Pro with Medical Mode posts the lowest MER in the set at 3.2%. That's the claim that matters here, and it's an entity claim, not a word-count claim. Full methodology and the rest of the numbers are on /benchmarks.

How we measure it, at a high level

You shouldn't take a benchmark on faith, so here's the shape of how MER is computed.

Start with reference transcripts—human-verified ground truth for real clinical-style audio. Identify the medical entities in each reference: the drugs, diagnoses, procedures, and dosages. Run each model over the same audio. Then, for every reference entity, check whether the model's output contains a correct match. The share that don't match is the MER.

The detail that does the work is entity alignment. "Echocardiogram" rendered as "echo cardiogram" is two tokens where the reference has one, so a naive comparison can misfire. Robust entity matching has to handle tokenization differences, casing, and generic-versus-brand naming, so that you're scoring clinical correctness rather than punishing formatting. This is the same care we describe in how to evaluate speech recognition models.

It's worth saying plainly: no public benchmark is your benchmark. Our test set reflects our distribution of accents, recording conditions, specialties, and drug frequencies. Yours will differ. The point of publishing MER isn't "trust our number"—it's "measure the right thing, then measure it on your audio."

Why this isn't just an AssemblyAI talking point

I'd make the MER argument even if we lost the benchmark, because the alternative is worse. An industry that evaluates medical transcription on WER alone is optimizing models to be confident and wrong about exactly the words that carry risk. A model trained and tuned to minimize WER has every incentive to get common words perfect and treat a rare drug name as a rounding error.

That's backwards. The rare words are the whole job. If you want to understand how far speech-to-text has come on the words that aren't rare, we cover that in how accurate is speech-to-text in 2026—but general accuracy and medical entity accuracy are different problems, and conflating them is how teams ship clinical tools that look fine in a demo and fail on the third encounter.

Frequently asked questions

What's the difference between MER and WER?

WER measures the share of all words transcribed incorrectly, weighting every word equally. MER measures only the share of medical entities—drug names, diagnoses, procedures, dosages—transcribed incorrectly. A model can have a low WER and a high MER if it gets common words right and clinical terms wrong.

Why is WER a poor metric for medical transcription?

Because it treats a dropped "um" the same as a wrong drug name. In clinical audio, a small fraction of words carry almost all the meaning and risk, and WER averages them away. See word error rate is broken for the longer argument.

Which model has the lowest MER?

In our benchmark, Universal-3 Pro with Medical Mode posts the lowest MER at 3.2%, ahead of Deepgram, Speechmatics, AWS, and Google. The full table is on /benchmarks.

Can I reproduce this on my own audio?

Yes—and you should. Public benchmarks reflect the publisher's data distribution, not yours. Run your own representative clinical files through the playground and compare entities against your ground truth.

Does a low MER mean I can skip human review?

No. A lower MER means fewer entities to catch and less QA burden, but clinical workflows still warrant human verification. The value of MER is telling you where the residual risk lives—in the entities—so you can review the right things.

Wrong Drug Name In, Wrong SOAP Note Out: Error Propagation

Mart Schweiger — Tue, 23 Jun 2026 21:24:40 +0000

A clinical AI pipeline looks deceptively simple on a whiteboard. Audio goes in. A speech-to-text model turns it into text. An LLM reads that text and writes the SOAP note, pulls the medication list, maybe suggests ICD-10 codes for billing. Three boxes, two arrows, done.

The problem lives in the arrows.

Every stage in that pipeline inherits whatever the stage before it produced. Not "mostly inherits." Inherits completely. The LLM that writes your assessment never hears the audio. It never sees the waveform. It sees text—and it treats that text as ground truth, because it has no other choice. If the transcription layer hands it the word "hydrocortisone" when the clinician said "hydrochlorothiazide," the model doesn't flag the discrepancy. It can't. There's nothing to compare against. It writes a fluent, confident, clinically coherent note about the wrong drug.

That's the failure mode nobody benchmarks for. Garbage in, fluent garbage out.

The error doesn't stay where it started

Here's what makes entity errors in clinical pipelines worse than they look at first glance: they don't fail loudly. A misrecognized drug name doesn't produce a typo or a [INAUDIBLE] tag. It produces a different real word that the downstream model accepts without suspicion.

Walk through one swap. A physician dictates a follow-up for a patient on hydrochlorothiazide for hypertension. The STT layer transcribes it as hydrocortisone—a real drug, spelled correctly, phonetically adjacent, and completely wrong therapeutically. One's a thiazide diuretic for blood pressure. The other's a corticosteroid.

Now watch the blast radius.

The medication list updates to hydrocortisone. The LLM writing the assessment reads "patient on hydrocortisone" and reasons accordingly—maybe it notes steroid considerations, maybe it adjusts the plan around a drug the patient never took. The coding step downstream maps to the wrong therapeutic class. The billing record reflects a medication that isn't in the chart for any clinical reason. And every one of those artifacts reads cleanly. A reviewer skimming the SOAP note sees fluent, plausible prose. Nothing screams "error" because the error has been laundered into grammatical, confident English at every stage.

Try another. The clinician says metformin—a first-line drug for type 2 diabetes. The transcript says metronidazole, an antibiotic. Suddenly the note implies a diabetic patient is being managed with an antimicrobial, the problem list drifts, and the coding logic follows the transcript off a cliff. The LLM can't recover metformin. It was never given metformin to recover.

This is the part that gets underappreciated: the LLM isn't the weak link here. A frontier model writing the note might be doing flawless work—faithfully, accurately summarizing the text it received. The accuracy ceiling for the entire pipeline was set upstream, at the transcription layer, before the LLM ever woke up.

Why fixing it downstream doesn't work

The intuitive instinct is to patch this at the LLM stage. Add a verification prompt. Ask the model to double-check medications. Build a reconciliation step.

It doesn't hold up, and the reason is information-theoretic, not a matter of prompt engineering. You cannot reconstruct information that was destroyed before it reached you. When the STT layer collapsed "hydrochlorothiazide" into "hydrocortisone," the signal that distinguished those two words—the acoustic detail in the audio—is gone by the time the text exists. The LLM has access to the wrong word and a high-quality language model's prior that the wrong word is perfectly reasonable in context. Asking it to catch the error is asking it to detect a problem it has no evidence for.

You could feed the audio to the LLM directly and skip the transcript. Some teams will. But for the vast majority of production clinical pipelines—ambient scribes, dictation workflows, coding automation—text is the interface, the audit trail, and the thing humans actually review. The transcript is load-bearing. Which means transcript accuracy, specifically entity accuracy, is the highest-leverage place in the entire system to prevent propagation.

Fix it at the source and everything downstream gets cheaper, safer, and more trustworthy. Fix it downstream and you're building increasingly elaborate machinery to compensate for a problem that should never have entered the pipeline.****

Want to see how entity accuracy holds up on your own clinical audio? Talk to our team or test a file in the playground.

Why WER isn't the metric that matters here

For years the industry has graded speech-to-text on word error rate. It's a useful number. It's also the wrong number for clinical work, and the reason traces directly back to error propagation.

WER treats every word as equally important. Miss "the," miss "hydrochlorothiazide"—both count as one error against the same denominator. A model can post an impressive WER while quietly fumbling the exact words that propagate downstream. We've written before about why WER is broken and why your WER benchmark might be lying to you—the short version is that a single aggregate number hides the distribution of which words get missed.
In a clinical pipeline, the distribution is everything. The words that matter are drug names, dosages, proper nouns, lab values, anatomical terms—the entities. A model that nails common words and drops entities will look fine on WER and fail catastrophically on the thing you actually care about.

That's why we track missed entity rate (MER) as a first-class metric, not an afterthought. MER measures how often the model drops or mangles the clinical entities that drive every downstream decision. If you're only watching WER, you're watching the wrong gauge. Our full methodology lives in how to evaluate speech recognition models.

What Medical Mode actually does about it

Universal-3 Pro already delivers best-in-market entity accuracy on drug names, proper nouns, and rare words. Medical Mode tightens that further for clinical audio specifically.

Turning it on is one parameter:

import assemblyai as aai

transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(
speech_models=["universal-3-pro"],
domain="medical-v1",
)
transcript = transcriber.transcribe("audio.wav", config)

That domain="medical-v1" flag works on both Universal-3 Pro and Universal-3.5 Pro Realtime, across English, Spanish, German, and French.

The numbers, measured on our benchmarks: Universal-3 Pro with Medical Mode posts a 3.2% MER—the lowest across every provider we tested, and roughly 20% fewer missed medical entities than Universal-3 Pro alone. Here's how that stacks up:

Provider	MER	WER
AssemblyAI Universal-3 Pro w/ Medical Mode	3.2%	5.3%
Deepgram	3.6%	5.5%
Speechmatics Enhanced Medical	4.7%	6.1%
Deepgram Nova-3 Medical	8.7%	5.9%
AWS Transcribe Medical	24.4%	12.9%

Look at the AWS row. A 24.4% MER means roughly one in four clinical entities gets missed or mangled. In a pipeline that propagates every error downstream, that's not a transcription model—it's an error generator with a confident downstream LLM amplifying its mistakes into polished, wrong documentation.
Pricing is a $0.15/hr add-on on top of Universal-3 Pro's $0.21/hr, so $0.36/hr all in. Full details on the pricing page.

The rest of a pipeline you can actually trust

Entity accuracy is the highest-leverage lever, but it's not the only one. Two more pieces matter for a clinical pipeline you'd put your name on.

First, speaker diarization. In an exam-room recording, who said what changes the meaning of the note. A symptom the patient reports and an instruction the clinician gives are different clinical objects, and diarization keeps them separated before the LLM ever tries to structure them. Get the speaker attribution wrong and you've introduced a different flavor of the same propagation problem.

Second, PII redaction. A trustworthy pipeline doesn't just transcribe accurately—it controls what flows downstream. Redacting protected identifiers at the transcript layer means the LLM and every system after it operate on the minimum data they need.

On the infrastructure side: AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA). It runs on BAA-eligible infrastructure with a BAA included.

Frequently asked questions

Why can't a good LLM just catch a wrong drug name?

Because it never receives the audio. The LLM only sees the transcript. Once the speech-to-text layer has substituted one real drug name for another, the acoustic information that distinguished them is gone. The model has no evidence anything is wrong, so it writes a fluent note around the incorrect entity.

Isn't a low word error rate enough to trust a clinical transcript?

No. WER weights every word equally, so a model can score well overall while consistently missing the drug names, dosages, and proper nouns that drive downstream decisions. Missed entity rate measures the words that actually propagate. That's why we report MER alongside WER.

How do I turn on Medical Mode?

Set one parameter: domain="medical-v1". It works on Universal-3 Pro and Universal-3.5 Pro Realtime, in English, Spanish, German, and French.

What does Medical Mode cost?

It's a $0.15/hr add-on. Combined with Universal-3 Pro's $0.21/hr, the total is $0.36/hr.

What if my audio has rare or facility-specific drug names?

Use keyterms prompting to load them—up to 1,000 terms on async, up to 100 on streaming, updatable mid-stream. It's the mechanism for specialized vocabulary that even a strong general model might not weight heavily enough.

Veterinary Transcription API: Species, Breeds & Vet Drugs

Mart Schweiger — Tue, 23 Jun 2026 21:24:33 +0000

A vet's exam room is one of the hardest audio environments in medicine, and almost nobody builds for it.

Think about what a general-purpose speech-to-text model has to handle. A clinician dictating over a barking dog. A tech reading back a weight-based dose. An owner answering questions from across the room. Drug names that overlap with human pharmacology but get used at different doses for different reasons. Breed names that sound like nothing in any training corpus built on human conversation. "Brachycephalic." "Maine Coon." "Enrofloxacin."

Most transcription APIs were trained on podcasts, call-center audio, and meetings. Drop a veterinary exam on them and they fall apart exactly where it matters—on the entities. The good news is that the same vocabulary-accuracy engine we built for human clinical work handles veterinary transcription remarkably well. Works for doctors, works for you. Let me show you why, and where the honest gaps are.

The three things that break veterinary transcription

Veterinary audio fails general models in three predictable places. Knowing them tells you exactly what to configure.

Species and breeds. This is the vocabulary humans-only training data never sees. A model that's never encountered "brachycephalic" won't render it correctly—it'll guess something phonetically close and wrong. The same goes for breed names that carry real clinical weight. A French Bulldog's breathing complaint and a Maine Coon's cardiac screening aren't trivia; they're context that shapes the whole note. If the breed is mangled, the record loses meaning.

Drug names, including the dual-use ones. Here's where it gets interesting. A lot of veterinary pharmacology overlaps directly with human medicine. Gabapentin, meloxicam, and metronidazole show up in both. But veterinary practice also leans on drugs and brand names that rarely appear in human dictation: carprofen (Rimadyl), maropitant (Cerenia), enrofloxacin (Baytril), firocoxib (Previcox). A model strong on human clinical terms gets you most of the way. The vet-specific names are the gap.

Clinic-floor audio. Barking. Multiple handlers. An owner, a tech, and a clinician all in one room. This is overlapping, noisy, multi-speaker audio—the kind that needs real diarization to untangle who said what.

The engine: entity accuracy first

The foundation is Universal-3 Pro, which delivers best-in-market accuracy on the words that matter most—drug names, proper nouns, and rare words. That's the category veterinary transcription lives and dies on. A model that handles common conversational English beautifully but drops the drug name is useless in a clinical record, human or animal. We've argued at length that word error rate is the wrong way to judge this—what you actually care about is whether the entities survive.

Universal-3 Pro supports English, Spanish, French, German, Italian, and Portuguese, with native code-switching, at $0.21/hr async.

Then there's Medical Mode. And here I want to be straight with you rather than oversell it.

Medical Mode is built on human clinical terminology. It's tuned for the pharmacology, anatomy, and clinical language of human medicine—and that terminology overlaps heavily with veterinary practice. The drug classes, the dosing language, the anatomical vocabulary, the structure of a clinical note: a large share of that is shared. So Medical Mode gives you a real lift on the overlapping clinical pharmacology that veterinary work shares with human medicine. It is not a veterinary-specific model, and I won't pretend it is. It's a clinical-language engine that happens to cover most of what a vet says, because most of what a vet says is clinical language.

Turning it on is one parameter—domain="medical-v1"—and it works on both Universal-3 Pro and Universal-3.5 Pro Realtime.

Want to try it on a real exam recording? Get your free API key and run a file in minutes.

Keyterms prompting: where you close the gap

So Medical Mode covers the overlap. What covers carprofen, brachycephalic, and the name of the specialty drug your practice stocks that nobody else does?

Keyterms prompting. This is the mechanism for specialized and custom vocabulary, and it's the single most important feature for veterinary work specifically. You hand the model a list of the exact terms you care about—breed names, species, vet-specific and brand drug names, your practice's house vocabulary—and it weights them during recognition.

The limits: up to 1,000 terms on async (a $0.05/hr add-on), up to 100 on streaming, free, and updatable mid-stream. That last detail matters more than it sounds. Mid-stream updates mean a live transcription session can load a new patient's breed and medication context as the exam moves room to room.

Here's the pattern:

import assemblyai as aai

transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(
speech_models=["universal-3-pro"],
domain="medical-v1",
keyterms_prompt=["carprofen", "maropitant", "brachycephalic"],
)
transcript = transcriber.transcribe("audio.wav", config)

Three terms there for illustration. In practice you'd load your full formulary, the breeds your clinic sees most, and any species-specific terms your dictation uses. Load enrofloxacin, firocoxib, Cerenia, Baytril, Previcox, French Bulldog, Maine Coon—whatever your records actually contain.

What the gap actually looks like

Here's an illustrative example—not a measured benchmark, just a concrete picture of the difference keyterms prompting makes on a vet-specific drug.

Illustrative example
Clinician says: "Started the Frenchie on carprofen, 2.2 milligrams per kilogram, twice daily."

Without keyterms prompting: "Started the Frenchie on car profen, 2.2 milligrams per kilogram, twice daily."

With keyterms_prompt=["carprofen"]: "Started the Frenchie on carprofen, 2.2 milligrams per kilogram, twice daily."

The drug name is the whole point of the sentence. Get it wrong and the medication record is wrong, the dose is attached to a non-word, and anything downstream that reads the note inherits the error. This is exactly the entity-accuracy problem at the center of every clinical pipeline—the difference is that in vet work, you've got a clean, simple lever to fix it.

Multi-speaker exam rooms: diarization

A vet exam usually isn't one voice. It's a clinician, a tech, and often the pet owner, all talking in the same room. To turn that into a usable record you need to know who said what.

That's speaker diarization. It separates the speakers so the owner's description of symptoms, the tech's readback, and the clinician's assessment land as distinct contributions instead of one undifferentiated wall of text. Combined with PII redaction for owner details, it gets you a transcript that's structured enough to actually build on.

The part most teams miss

The interesting thing about veterinary transcription is that it's not actually a harder problem than human clinical transcription—it's a different distribution of the same problem. The clinical-language backbone is shared. What changes is the long tail: the breeds, the species, the vet-specific drug names that no general training corpus weights heavily.

And the long tail is the one part of speech recognition you can configure directly. You can't hand-tune how a model handles general English, but you can hand it the exact 200 terms your practice uses every day and have it prioritize them. That makes veterinary transcription one of the more solvable specialized domains out there—not because the audio is easy, but because the hard part is a list you already know.

Write down your formulary and your breed list. That's most of your accuracy problem solved before you transcribe a single second.

Get your free API key and try it on your own exam audio: sign up here, or browse the docs to see the full keyterms prompting reference

Frequently asked questions

Is there a veterinary-specific transcription model?

Not as a separate model—and you don't need one. Universal-3 Pro's entity accuracy plus Medical Mode covers the large overlap between human and veterinary clinical language, and keyterms prompting covers the species, breed, and vet-specific drug gaps. That combination handles veterinary audio without a dedicated model.

Does Medical Mode actually help with animal patients?

Yes, because veterinary pharmacology and clinical language overlap heavily with human medicine—shared drug classes, dosing language, and anatomical terms. Medical Mode is built on human clinical terminology, so it lifts accuracy on everything shared. Keyterms prompting handles the rest.

How do I make sure carprofen, Cerenia, and breed names get transcribed correctly?

Load them into keyterms prompting—up to 1,000 terms on async, up to 100 on streaming, updatable mid-stream. It's the mechanism for custom and specialized vocabulary, and it's where you close the veterinary-specific gap.

Can it handle a noisy exam room with multiple people talking?

Yes. Speaker diarization separates the clinician, tech, and owner so the transcript attributes each statement to the right speaker, even in overlapping, noisy clinic-floor audio.

What does this cost to run?

Universal-3 Pro is $0.21/hr async. Medical Mode adds $0.15/hr ($0.36/hr combined). Keyterms prompting is a $0.05/hr add-on on async and free on streaming. Full details are on the pricing page.

Building Behavioral Health Documentation Clinicians Trust

Mart Schweiger — Tue, 23 Jun 2026 21:24:08 +0000

Trust is a strange thing to engineer into a transcript.

A clinician doesn't sit there grading your word error rate. They glance at the note, see that "sertraline" came through as "sertraline" and not "sir Tralee," and decide—usually in the first session—whether your product is something they can rely on or something they have to babysit. That decision is mostly subconscious, and it's almost always made on the words that carry clinical weight.

So if you're a product manager or founder building a behavioral health scribe, the real question isn't "how accurate is the transcription?" It's "is it accurate on the things a clinician would notice being wrong?" Those are very different bars.

We've written a step-by-step tutorial on building an AI scribe for therapy sessions—the actual code, the upload-and-transcribe loop, the configuration. This post is the companion to that. Less about how to wire it up, more about what earns the clinician's trust once it's wired up. Three things, in my experience, do most of the work.

Accuracy on the words that actually matter

Behavioral health vocabulary is unforgiving in a specific way. Psychiatric medication names are dense, similar-sounding, and dose-dependent. Sertraline, lamotrigine, quetiapine, bupropion—these aren't words a general speech-to-text model has heard a million times, and they sit next to numbers ("up to 200 milligrams," "split the 300 into two doses") where a single transposed entity changes the clinical meaning of the note.

Here's the trap a lot of teams fall into. They benchmark on overall word error rate, see a number that looks great, and ship. But overall WER averages across "the," "and," "um," and "lamotrigine." A model can post a lovely aggregate number while quietly fumbling the one word per minute that a prescriber will actually catch.

That's why we measure Medical Mode on Missed Entity Rate instead—how often the model drops or mangles a clinically meaningful entity like a drug, a dose, or a diagnosis. Medical Mode delivers a 3.2% MER, the lowest across the providers we benchmarked, and roughly 20% fewer missed medical entities than Universal-3 Pro running on its own. You can see the full breakdown on our benchmarks page.

Turning it on is one parameter.

import assemblyai as aai

transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(
speech_models=["universal-3-pro"],
domain="medical-v1",
)
transcript = transcriber.transcribe("clinical-audio.wav", config)

That domain="medical-v1" flag is the whole activation. It works on Universal-3 Pro for pre-recorded sessions and on Universal-3.5 Pro Realtime for live ones, so your documentation pipeline behaves the same whether you're processing a recorded intake overnight or transcribing a session as it happens.

And when a practice uses regional brand names or a niche formulary that even a medical model wouldn't expect, keyterms prompting lets you hand the model that vocabulary up front. Think of it as telling the model what to listen for before it ever hears the audio.****

Ready to test accuracy on your own clinical audio? Explore Voice AI for healthcare or grab an API key.

Knowing who said what

Now here's where behavioral health diverges from almost every other medical transcription use case.

A radiology dictation is one voice. A therapy session is at least two—and group, couples, and family sessions can be four, five, six people talking over each other, finishing each other's sentences, going quiet, coming back. If your transcript renders all of that as one undifferentiated wall of text, the clinical value collapses. A note that says "patient reported increased anxiety" is useless when there were three patients in the room and you can't tell which one said it.

This is exactly what speaker diarization solves—segmenting the audio by who's speaking so each utterance is attributed to the right person. In behavioral health it's not a nice-to-have feature sitting next to accuracy. It's part of the accuracy. And on Universal-3.5 Pro Realtime, live diarization gets a second pass: the model labels speakers during the session, then re-clusters every voice when the stream ends and sends a single revision correcting any labels it now knows were wrong—async-grade speaker accuracy within about half a second of the session ending, up to 10 speakers.

JotPsych, which builds documentation tooling for behavioral health, put it plainly. Jackson Bierfeldt, their Cofounder and CTO, told us: "In the medical context, accuracy is highly important…[and] there can be multiple people present. Separating them is key to accuracy. The biggest impact AssemblyAI has had has been in enabling our technical team to focus on workflow-specific features rather than a general speech-to-text pipeline."

Read that last sentence again, because it's the part product leaders should care most about. JotPsych didn't just want accurate diarization—they wanted to stop building and maintaining a speech pipeline at all, so their engineers could spend their time on the things that actually differentiate a behavioral health product. The transcription layer should be something you configure, not something you staff a team around.

NovoPsych, another team building in behavioral health, is solving the same shape of problem—turning sensitive, multi-speaker clinical conversation into structured documentation that a clinician will sign their name to. When the words and the speakers are both right, the clinician's review time drops, and that's where the trust compounds.

Handling sensitive sessions with care

Let's talk about privacy, because behavioral health is about the most sensitive category of data there is, and a clinician's trust evaporates fast if they suspect a session is being handled carelessly

I'll be direct about what this is and isn't. This isn't the headline of your product—no clinician chose your scribe because of an infrastructure diagram. But it's the floor underneath everything else, and if the floor isn't there, none of the accuracy matters.

AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI. The infrastructure is BAA-eligible, and the BAA is included.

On the engineering side, you've got PII redaction built in. With redact_pii, you can strip identifying information out of transcripts—names, contact details, the kinds of identifiers that turn an ordinary transcript into something you have to lock down. For behavioral health, where a single transcript might name a patient, their family members, and their employer in the first two minutes, redaction is a practical tool, not a checkbox.

The point is that data handling should be designed in from the first session, not retrofitted after your first enterprise customer asks the question in a security review.
Building for sensitive clinical settings? Start with a free API key and review the docs.

The trust isn't in the transcript

Here's the thing I'd leave you with, and it's the part that's easy to miss when you're heads-down on accuracy metrics.

Clinician trust isn't earned in the moment the transcript is correct. It's earned in the moment the clinician stops checking. The first few sessions, they read every line against their memory of what happened. Then one day they skim, sign, and move on—because the medication names have been right, the speakers have been attributed correctly, and nothing has surprised them. That shift from auditing to trusting is the entire ballgame, and it only happens if the foundation is boring and reliable across hundreds of sessions, not impressive in a demo.

Build for the hundredth session, not the first.****

See what Medical Mode does on your audio—explore Voice AI for healthcare or get your API key.

Frequently asked questions

How accurate is Medical Mode on psychiatric medication names?

Medical Mode posts a 3.2% Missed Entity Rate—the lowest across the providers we benchmarked—and around 20% fewer missed medical entities than Universal-3 Pro on its own. Because MER measures clinically meaningful entities specifically, including drug names and dosages, it's a better proxy for behavioral health accuracy than overall word error rate. The full numbers live on our benchmarks page.

Can it separate speakers in group or family therapy sessions?

Yes. Speaker diarization segments the audio by speaker and attributes each utterance to the right person, which is what makes a multi-party session usable as documentation rather than a single block of text. On Universal-3.5 Pro Realtime, live labels are refined by an end-of-stream revision for async-grade speaker accuracy, up to 10 speakers. It's the capability JotPsych called out as key to accuracy in the medical context.

Does it work for live sessions, or only recorded ones?

Both. Medical Mode runs on Universal-3 Pro for pre-recorded audio and on Universal-3.5 Pro Realtime for live transcription, so you can build the same accuracy into a real-time scribe and an after-hours batch pipeline with the same one-parameter activation.

How do you handle PHI and HIPAA?

AssemblyAI is considered a business associate under HIPAA and offers a standard Business Associate Addendum, which is required under HIPAA, on BAA-eligible infrastructure. You can also use redact_pii to strip identifying information from transcripts as part of your data handling.

What does Medical Mode cost?

Medical Mode adds $0.15/hr on top of the base model. Universal-3 Pro with Medical Mode comes to $0.36/hr, and Universal-3.5 Pro Realtime with Medical Mode is $0.60/hr. Full details are on the pricing page.

Medical Transcription in Spanish, German, and French

Mart Schweiger — Tue, 23 Jun 2026 21:24:01 +0000

Most multilingual transcription stories are about coverage. How many languages does the model support? Forty? A hundred?

That's the wrong question for medicine.

In a real clinic, the hard problem isn't supporting French. It's the moment a Spanish-speaking patient in a US clinic says "me duele el pecho, like a pressure" in one breath, and your transcript has to get both halves right—the Spanish symptom and the English qualifier—without anyone touching a language setting. Coverage counts languages. Clinical accuracy counts the words inside a single messy sentence. Those are not the same achievement.

Medical Mode launches benchmarked across four languages—English, Spanish, German, and French—for both pre-recorded and streaming audio. But the part I want to spend this post on is the part that actually breaks competing systems: what happens at the seam between languages.

Why multilingual medical transcription is genuinely hard

Let's be honest about the difficulty, because it's easy to wave at "multilingual support" as if it were a switch you flip.

First, medical vocabulary is treacherous across languages precisely because so much of it shares Latin and Greek roots. "Hypertension," "hypertensión," "Hypertonie," and "hypertension" look and sound like cousins—and that similarity is a trap, not a help. A model has to map each one to the right clinical entity in the right language, not blur them into an average. Near-identical isn't identical, and in a medication or diagnosis field, close is wrong.

Second, accents. A francophone clinician in Montreal, a Swabian physician, and a Madrid pharmacist don't pronounce their own languages the way a textbook does, let alone the way a model's training distribution assumes.

Third—and this is the one that quietly wrecks deployments—real clinical encounters are mixed-language. US Hispanic care, the DACH region with its English-heavy medical training, francophone clinics with English drug brand names. People code-switch. They start a sentence in one language and finish it in another, drop an English drug name into a Spanish sentence, or answer a German question in English because that's how they learned the term.

A pipeline built on "detect the language, then transcribe in that language" has no good answer here. By the time it's committed to Spanish, the English half of the sentence is already a casualty.

Code-switching is the whole differentiator

So here's where it gets interesting.

Universal-3 Pro handles native intra-utterance code-switching across all six of its languages—English, Spanish, French, German, Italian, and Portuguese. Intra-utterance is the operative phrase. Not "we detect a language per file," and not even "per sentence." Within a single utterance, the model can follow a speaker as they move between languages and keep transcribing accurately the entire way through.

For live audio, Universal-3.5 Pro Realtime extends that same native code-switching to 18 languages—so a streaming clinical encounter that mixes languages mid-sentence holds the thread across a much wider range. Medical Mode itself is benchmarked across the four launch languages (English, Spanish, German, and French), but the underlying code-switching the model relies on is built in across all of them.

No language toggle. No separate pipeline per language. No detect-then-route logic you have to build and maintain.
That means the Spanglish sentence—"me duele el pecho, like a pressure"—comes through intact, both halves, because the model was never forced to pick a lane. The clinician who slips into English for the drug name and back into French for the symptom doesn't break the transcript. The patient who answers in their first language mid-question doesn't either.

We went deeper on how this works in our piece on multilingual streaming and native code-switching, and it's worth reading if you're serving any population where two languages share a room.

The practical upshot for product teams: you build one integration. Not one per market.

Where the accuracy actually shows up

Code-switching is the headline, but it only matters if the underlying transcription is clinically accurate, so let's tie it back to numbers.

Medical Mode delivers a 3.2% Missed Entity Rate—the lowest across the providers we benchmarked—and roughly 20% fewer missed medical entities than Universal-3 Pro running on its own. MER is the right yardstick here because it measures what a clinician actually cares about: how often a clinically meaningful entity, a drug or a dose or a diagnosis, gets dropped or mangled. And critically, Medical Mode is benchmarked across all four launch languages, so that 3.2% isn't an English-only figure dressed up as multilingual. See the full breakdown on the benchmarks page.

Activation is the same single parameter in every language.

import assemblyai as aai

transcriber = aai.Transcriber()
config = aai.TranscriptionConfig(
speech_models=["universal-3-pro"],
domain="medical-v1",
)
transcript = transcriber.transcribe("clinical-audio.wav", config)

You can let the model auto-detect language, but you don't need a language toggle to get code-switching—it's native. The domain="medical-v1" flag is identical whether the audio is Spanish, German, French, English, or a mix of them in one recording.

This runs the same way for both modes. Pre-recorded audio goes through Universal-3 Pro at $0.21/hr, and live audio runs on Universal-3.5 Pro Realtime at $0.45/hr base, at streaming latency—end-of-turn detection lands around 300ms. Medical Mode adds $0.15/hr in either case—so Universal-3 Pro plus Medical Mode is $0.36/hr, and the streaming variant plus Medical Mode is $0.60/hr. The full table is on the pricing page.****

Want to hear it handle your languages? Get a free API key or try it in the playground.

Region-specific drug names and keyterms prompting

One more wrinkle that matters specifically for multilingual healthcare: drug brand names don't translate, they vary by market.

The same molecule ships under different brand names in the US, Germany, France, and Spain. A general medical model can't possibly anticipate every regional formulary. So when you're deploying into the DACH region or francophone care, keyterms prompting lets you hand the model the specific brand names, local terminology, and formulary your clinicians actually use—before it processes a single second of audio. It's the difference between a model that knows the active ingredient and one that also knows what your pharmacist calls it.

Pair that with diarization for multi-speaker encounters and redact_pii for sensitive data, and you've got a clinical pipeline that holds up across markets.

On the privacy side: AssemblyAI enables covered entities and their business associates subject to HIPAA to use the AssemblyAI services to process protected health information (PHI). AssemblyAI is considered a business associate under HIPAA, and we offer a standard Business Associate Addendum (BAA) that is required under HIPAA to ensure that AssemblyAI appropriately safeguards PHI—on BAA-eligible infrastructure, with the BAA included.

The integration you don't have to fork

Here's the insight worth sitting with if you're choosing an approach for a multilingual patient population.

The hidden cost of "supports N languages" architectures isn't accuracy—it's branching. A detect-then-route pipeline forces you to maintain a code path per language, test each one, and handle the seams between them, and those seams are exactly where mixed-language encounters live. Native code-switching collapses that whole tree into one path. You're not just getting a better transcript on Spanglish; you're getting an integration that doesn't fork every time you enter a new market.

Build one pipeline, serve every patient who walks in, regardless of which language they reach for mid-sentence.****

Ready to build it? Get your free API key or test your own clinical audio in the playground.

Frequently asked questions

Which languages does Medical Mode support?

Medical Mode is benchmarked across four languages at launch—English, Spanish, German, and French—for both pre-recorded and streaming audio. The underlying Universal-3 Pro model supports six languages for pre-recorded audio (adding Italian and Portuguese) with native code-switching, and Universal-3.5 Pro Realtime extends native code-switching to 18 languages for live audio.

What is intra-utterance code-switching, and why does it matter for clinics?

It means the model can follow a speaker who changes languages within a single utterance—not just per file or per sentence. In real clinical settings where patients and clinicians mix languages mid-thought, this keeps the transcript intact without a language toggle or a separate pipeline per language.

How accurate is multilingual medical transcription?

Medical Mode posts a 3.2% Missed Entity Rate, the lowest across the providers we benchmarked, with around 20% fewer missed medical entities than Universal-3 Pro alone—and it's benchmarked across all four launch languages, not English only. Details are on the benchmarks page.

How do I handle region-specific drug brand names?

Use keyterms prompting to supply the specific brand names, local terminology, and formulary your clinicians use before transcription runs. This is especially useful across markets where the same molecule ships under different brand names.

Does code-switching require a different setup than single-language transcription?

No. You activate Medical Mode with the single domain="medical-v1" parameter, and code-switching is native—there's no toggle to enable and no separate multilingual pipeline to configure. On streaming, if a session is genuinely monolingual you can pass language_code to bias the model toward one language; leave it off to keep native code-switching.

Best platforms for enterprise voice agents

Mart Schweiger — Mon, 15 Jun 2026 16:25:04 +0000

Every voice agent demo looks good. A founder opens a laptop, speaks a few sentences, and the agent responds with something reasonable in under a second. The room nods. But here's where it gets interesting: that same agent needs to handle 10,000 concurrent calls on a Tuesday morning when half your customer base decides to check their account balance at once. It needs to correctly capture email addresses, prescription numbers, and policy IDs—not approximate them. And it needs to do all of this while meeting the compliance requirements your legal team won't budge on.

The gap between a working demo and a production voice agent that enterprises actually depend on is enormous. Choosing the wrong platform means months of integration work followed by accuracy problems that surface only after you've committed.

So what should enterprise teams actually look for? And which platforms deliver? Let's break it down.

What enterprise voice agents actually need

Most platform comparison guides focus on surface-level features: language count, voice selection, basic latency numbers. That's table stakes. Enterprise voice agents have a different set of requirements that only become obvious once you're building for production.

Speech accuracy on entities—not just words

Overall word accuracy matters, but it's not the whole picture. Voice agents act on specific pieces of information: account numbers, email addresses, phone numbers, medication names, confirmation codes. If the speech-to-text layer gets "RX-7704132" wrong, the LLM downstream acts on bad data. The agent doesn't just mishear—it takes the wrong action. Missed entity rate is the metric that actually predicts whether your agent completes tasks or frustrates customers.

Sub-second latency that holds under load

A one-second response time in a controlled demo is easy. Maintaining that latency at scale—with thousands of concurrent sessions, tool calls mid-conversation, and noisy telephony audio—is a different engineering problem entirely. Ask any platform what their P95 latency looks like at peak concurrency, not just their best-case P50.

Turn detection that works in real conversations

This is the thing most teams underestimate. Basic voice activity detection (VAD) uses silence thresholds to decide when someone's done talking. But people pause mid-sentence. They say "um" while thinking. They say "uh-huh" to acknowledge without wanting to interrupt. If your voice agent architecture can't distinguish between a thinking pause and a completed turn, every conversation will feel awkward—and your users will notice immediately.

Compliance that's actually enforceable

For enterprise deployments, SOC 2 Type 2 certification is the baseline. If you're in healthcare, you need a Business Associate Addendum (BAA) in place before any patient data touches the platform. If you're in finance, you need audit trails and data retention controls. Don't settle for "we're working on it"—ask for the documentation.

Scalability without concurrency ceilings

Some platforms cap concurrent sessions or require concurrency commitments before you can scale. That's fine for a pilot. It's a problem when your contact center traffic spikes 3x on the first of the month and you're hitting rate limits. True enterprise platforms autoscale without renegotiation.

Mid-conversation flexibility

Production voice agents aren't static. You need to update the system prompt when a customer provides their account type. You need to swap tools when the conversation shifts from billing to technical support. You need to adjust VAD sensitivity when a caller is in a noisy environment. The ability to change configuration mid-session—without dropping the connection—separates infrastructure built for real use cases from platforms built for demos.

Build enterprise voice agents on the most accurate speech foundation

AssemblyAI's Voice Agent API handles STT, LLM, and TTS in a single WebSocket connection. $4.50/hr flat rate, no concurrency caps, SOC 2 Type 2 certified.

The major platforms compared

There are several serious options for building enterprise voice agents in 2026. Here's an honest look at how they stack up—what each does well, where they fall short, and which use cases they're best suited for.

Platform	Architecture	Word accuracy	Missed entity rate	Pricing	Turn detection	Concurrency
AssemblyAI Voice Agent API	Cascaded (STT + LLM + TTS, single WebSocket)	94.07%	16.7%	$4.50/hr flat	Semantic + neural network + VAD	No concurrency caps
AssemblyAI Universal-3 Pro Streaming (BYO stack)	Standalone STT (bring your own LLM + TTS)	94.07%	16.7%	$0.45/hr (STT only)	Provided by your orchestrator	Unlimited, autoscaling
OpenAI Realtime API	Native multimodal (GPT-4o)	93.13%	23.3%	~$18/hr, per-token billing	Basic VAD	99+ languages; complex event surface
Deepgram Voice Agent API	Cascaded (Nova-3)	92.10%	25.5%	~$4.50/hr, concurrency-metered	Traditional VAD	Requires concurrency tier commitments
ElevenLabs Conversational AI	TTS-first conversational stack	Not published	>25.2%	Varies	Standard VAD	30-agent concurrency cap

AssemblyAI Voice Agent API

AssemblyAI's Voice Agent API takes a cascaded architecture approach—dedicated best-in-class models for STT, LLM, and TTS, exposed through a single WebSocket. You stream audio in, you get audio back. One connection, one bill.

The speech understanding layer is built on Universal-3 Pro Streaming, which ranks #1 on the Hugging Face Open ASR Leaderboard. In benchmarks, it achieves 94.07% word accuracy with a 16.7% missed entity rate on names, emails, phone numbers, and credit card numbers. That entity accuracy gap is significant—we'll get into why shortly.

The developer experience is notably simple. It's a standard JSON API over WebSocket with no proprietary SDK required. Most developers get a working agent running the same afternoon. But simple doesn't mean limited: you get speech-aware turn detection (semantic + neural network + VAD), tool calling via JSON Schema, mid-session updates to prompt, voice, tools, and VAD settings, and 30-second session resumption if the WebSocket drops.

Pricing is a flat $4.50/hr covering the entire pipeline. No token math, no separate input/output charges. Six languages currently supported: English, Spanish, French, German, Italian, and Portuguese. On compliance, AssemblyAI holds SOC 2 Type 2 and ISO 27001 certifications, with a BAA available for healthcare use cases. Medical Mode is also available for improved accuracy on clinical terminology.

AssemblyAI Universal-3 Pro Streaming (bring-your-own-stack)

Not every team wants a managed pipeline. If you've already built an orchestration layer with LiveKit, Pipecat, or Vapi, you might want the best possible streaming speech-to-text without replacing the rest of your stack.

That's where Universal-3 Pro Streaming as a standalone STT API comes in. Same speech model, same accuracy, $0.45/hr for transcription only. Unlimited concurrent streams with autoscaling—no concurrency commitments required. It drops into existing pipelines as the STT layer, giving you full control over your LLM and TTS choices.

This is the best option for teams that already have an orchestrator and want to upgrade their speech understanding without rearchitecting everything.

OpenAI Realtime API

OpenAI takes a fundamentally different approach. Their Realtime API uses GPT-4o as a native multimodal model that handles audio as one of many modalities—text, images, video, and voice. It's not a pipeline designed specifically for speech conversations.

The upside is broad language support (99+ languages) and the ability to handle multimodal inputs. For prototyping conversational experiences, it's fast to get started.

The downsides become clear at enterprise scale. Pricing runs approximately $18/hr with per-token billing across 30+ event types—roughly 4x more expensive than cascaded alternatives. Word accuracy sits at 93.13% with a 23.3% missed entity rate, which reflects the trade-off of using a generalist model for speech-specific tasks. Turn detection relies on basic VAD rather than semantic understanding, which means more awkward interruptions in real conversations. And the developer surface area is complex—30+ event types to handle compared to a handful with purpose-built APIs.

OpenAI Realtime is a strong choice for demos, browser-first apps, and multilingual prototyping. But at enterprise scale, the cost and accuracy trade-offs add up quickly.

Deepgram Voice Agent API

Deepgram also uses a cascaded architecture, similar to AssemblyAI. Their voice agent offering runs approximately $4.50/hr but uses concurrency-metered billing—you'll need to commit to concurrency tiers as you scale.

Nova-3 achieves 92.10% word accuracy with a 25.5% missed entity rate—a meaningful gap compared to AssemblyAI's 16.7%. Turn detection relies on traditional VAD without the semantic layer that distinguishes thinking pauses from completed turns. Mid-session updates are limited to prompt and voice only. For enterprise deployments where accuracy on real-world data directly impacts outcomes, those gaps add up.

ElevenLabs Conversational AI

ElevenLabs built its reputation on voice synthesis, and their TTS remains excellent. Their Conversational AI product extends that focus into the voice agent space.

The enterprise limitation is the 30-agent concurrency cap. For contact centers or any high-volume deployment, that ceiling makes scaling impossible. ElevenLabs started as a TTS company—their speech understanding trails purpose-built STT providers, with a missed entity rate above 25.2%. For applications where voice output quality is the primary differentiator and concurrency demands are modest, it's a solid fit. For enterprise voice agents that need to scale, the constraints are significant.

Test voice agent accuracy on your own audio

See how Universal-3 Pro Streaming captures entities, handles accents, and detects turns. Compare the results to what you're getting from your current provider.

Try playground

Why speech accuracy is the deciding factor for enterprise

Here's the thing most teams don't realize until they're deep into production: transcription errors don't just reduce transcript quality. They cascade through the entire voice agent pipeline.

When your STT layer gets an email address wrong, the LLM doesn't know it's wrong. It processes the incorrect email as if it's fact, then takes action on it—sending a confirmation to the wrong address, looking up the wrong account, or storing incorrect contact information in your CRM. The agent didn't "make a mistake"—it did exactly what it was told with bad input.

The STT model you choose determines your agent's effective intelligence. A brilliant LLM fed wrong data will confidently do the wrong thing.

Consider a pharmacy refill scenario. A caller provides their prescription number, date of birth, medication name, dosage, address, and phone number in a single conversation. AssemblyAI's Voice Agent API correctly transcribes "RX-7704132" and "Metoprolol 80mg" while formatting the date of birth, address, and phone number accurately. In the same scenario, Deepgram's transcription produces "dash seven seven zero four one three two" without the RX prefix, drops date formatting, and garbles the medication dosage format.

That's not a cherry-picked example—it reflects the systematic accuracy advantage that comes from building an entire pipeline around a purpose-built speech model. When you're processing thousands of these conversations daily, the difference between a 16.7% missed entity rate and a 25.5% missed entity rate translates directly into failed tasks, repeat calls, and frustrated customers.

In our Voice Agent Report, 76% of respondents rated speech-to-text accuracy as the single most important factor when building voice agents—above latency, cost, and integration capabilities. The data backs up what builders already know intuitively: if the agent can't hear correctly, nothing else matters.

The two-path approach

One thing that sets AssemblyAI apart in the voice agent solutions space is that it offers two distinct paths to the same speech accuracy foundation.

The Voice Agent API is the fastest path to production. One WebSocket, one bill, working agent in an afternoon. It's purpose-built for teams that want to focus on their product logic rather than managing voice infrastructure. You write the system prompt, register your tools, and ship.

Universal-3 Pro Streaming as a standalone STT API is for teams that want full architectural control. If you've already invested in an orchestration framework, a specific LLM, or custom TTS, you can drop in the same speech model that powers the Voice Agent API without changing anything else in your stack.

Both paths share the same Universal-3 Pro speech recognition foundation. Transcription quality stays consistent regardless of which approach you choose. And because both use WebSocket connections and standard JSON, developers often prototype with the Voice Agent API for speed, then graduate to the bring-your-own-stack approach as their architecture matures and they want more control over LLM routing or TTS selection. Switching requires updating your connection endpoint and message handling—not rebuilding from scratch.

This two-path approach is a big part of why developers consistently rank AssemblyAI as the best voice agent API for developers—you get the simplicity of a managed pipeline when you want speed, and the flexibility of a raw STT API when you need control. You're not locked into a single integration pattern, and you're not forced to compromise on speech accuracy to get the architecture you want.

Choosing the right platform for your team

For enterprise voice agents, the decision comes down to three factors: accuracy, compliance, and scalability. Get any of them wrong and you'll feel it in production—either through failed tasks, blocked deployments, or infrastructure that can't keep up with demand.

AssemblyAI covers all three. The highest entity accuracy in the market means your agents complete tasks on the first attempt. SOC 2 Type 2, ISO 27001, and a BAA for healthcare mean your compliance team can sign off. Unlimited concurrency with flat-rate pricing means you scale without surprises.

But beyond the specs, there's a practical question worth asking: what does it feel like to build on this platform? AssemblyAI's developer experience is deliberately simple—standard JSON over WebSocket, no proprietary SDKs, documentation you can read in 10 minutes. That simplicity compounds over time. Fewer moving parts mean fewer failure surfaces, faster debugging, and less operational overhead.

The best way to evaluate any voice agent platform is to have a conversation with one. Try the live demo—no signup required—and judge the accuracy, turn detection, and conversation flow for yourself.

Start building enterprise voice agents today

Get your API key and have a working voice agent running this afternoon. Free tier available, no credit card required.

Frequently asked questions

What is the best API for building voice agents?

AssemblyAI's Voice Agent API is the best API for building voice agents in 2026, combining industry-leading speech accuracy (94.07% word accuracy, 16.7% missed entity rate) with a flat $4.50/hr rate that covers the entire STT, LLM, and TTS pipeline. It's a single WebSocket connection with no proprietary SDK required, and most developers ship a working agent the same day. For teams that want to bring their own LLM and TTS, Universal-3 Pro Streaming provides the same speech foundation at $0.45/hr as a standalone STT layer.

How much does it cost to build an enterprise voice agent?

Costs vary significantly by platform. AssemblyAI's Voice Agent API charges a flat $4.50/hr covering speech understanding, LLM reasoning, and voice generation—no token math or separate invoices. OpenAI's Realtime API runs approximately $18/hr with per-token billing. Deepgram's voice agent offering is roughly $4.50/hr but requires concurrency commitments as you scale. For high-volume enterprise deployments, the billing model matters as much as the per-unit cost.

What compliance certifications should a voice agent platform have?

At minimum, enterprise voice agent platforms should hold SOC 2 Type 2 certification, which verifies ongoing security controls. Healthcare organizations need a Business Associate Addendum (BAA) in place before processing any patient data. AssemblyAI holds SOC 2 Type 2 and ISO 27001 certifications, with a BAA available for healthcare use cases and Medical Mode for improved clinical terminology accuracy.

Can I use my own LLM with AssemblyAI's voice agent infrastructure?

Yes. AssemblyAI offers two paths. The Voice Agent API includes a managed LLM as part of its all-in-one pipeline, but if you want full control, Universal-3 Pro Streaming gives you AssemblyAI's industry-leading STT as a standalone API that drops into your existing orchestration stack—whether that's LiveKit, Pipecat, or a custom pipeline. You bring your own LLM and TTS while getting the same speech accuracy foundation.

What is the best one-API solution for voice agents?

AssemblyAI's Voice Agent API is the leading one-API solution for voice agents. It replaces three separate providers (STT, LLM, TTS) with a single WebSocket connection, one invoice measured in hours, and one set of logs. At $4.50/hr flat, it eliminates the token math and multi-vendor complexity that slows down development and creates fragile production systems.

How does turn detection work in voice agents?

Turn detection determines when a user has finished speaking and when they're trying to interrupt. Basic approaches use silence thresholds (VAD), which often cut users off mid-thought or create awkward pauses. AssemblyAI's Voice Agent API uses semantic turn detection that considers what the user actually said—not just silence—to decide if they're done. It distinguishes back-channel responses like "uh-huh" from real interruptions like "wait, stop," and adapts its timing to each user's speaking pace throughout the conversation.