Most speech-to-text demos use clean audio. One speaker, a good mic, a quiet room. Your production audio doesn't look like that. It's a call-center recording with a headset that clips, two people talking over each other, a customer reading a card number while a TV plays in the background, and a speaker who says "I, uh — I mean, the second one." That messy 20% is where transcription accuracy actually gets decided, and it's the part demos never show.
The good news: hard audio is mostly tunable. Universal-3.5 Pro is built for the "hard stuff" traditional models miss, and it gives you levers to control behavior instead of accepting whatever a black box hands back. Here's how to think about each failure mode — noise, overlap, and disfluencies — and what to actually do about it.
Clean-audio WER isn't your problem — the messy 20% is
A headline word error rate on a benchmark dataset tells you how a model does on relatively tidy speech. Useful, but it's not the number that predicts whether your product works. What predicts that is how the model handles the specific tokens your workflow depends on — the account number, the medication name, the email address, the "yes" buried under crosstalk — on audio that was never studio-clean.
That's why entity accuracy matters more than aggregate WER for real applications. Universal-3.5 Pro posts large gains over Universal-2 on exactly these tokens: a 37% relative improvement in missed entity rate on email addresses, 31% on locations, 29% on medical terms, and double-digit gains on organization names, phone numbers, and credit card numbers. Those are the words a wrong transcript makes expensive.
Background noise and low-quality recordings
Noise is the most common complaint and the most misunderstood one. The instinct is to clean the audio before it reaches the model — run it through aggressive noise cancellation first. Resist that instinct. Client-side noise reduction introduces artifacts, and those artifacts often confuse the model more than the original noise did. You can strip out the very speech cues the model relies on.
For real-time audio, the better approach is server-side Voice Focus in Universal-3.5 Pro Streaming, which isolates the primary speaker so background voices don't turn into phantom words or false interruptions. It has a near-field setting for close-talking headset mics and a far-field setting for distant capture. For pre-recorded audio, feed the model the cleanest original signal you have and let it do the separation — then use prompting and keyterms (below) to lock in the words that matter.
Overlapping speakers and crosstalk
Two people talking at once is genuinely hard, and it's made harder by a subtle issue: a sensitive model picks up everything, including audio bleeding between channels. A classic failure looks like this — an agent's headset echoes the customer's voice back into the agent's mic, and that echo gets transcribed as extra utterances on the agent's side, quietly wrecking any downstream auto-QA that assumes one speaker per channel.
The fix is architectural, not a setting you crank up. If you have separate audio channels for each party — which most call-center stacks do — use multichannel transcription so each speaker is transcribed from their own isolated channel. That prevents cross-channel bleed from being attributed to the wrong person. Diarization (labeling "who spoke when" within a single mixed channel) is the tool when you only have one combined track. They solve different problems: reach for multichannel when you have the channels, diarization when you don't. Confusing the two is a common source of "the transcript looks wrong" tickets.
Filler words and disfluencies: verbatim vs. clean
"Um," "uh," "you know," false starts, self-corrections — disfluencies are where teams disagree, because there's no universally correct answer. A medical or legal transcript may need every hesitation preserved for the record. A meeting summary wants them gone. So the real question isn't "does the model handle filler words?" — it's "can I choose?"
With Universal-3.5 Pro you can, through prompting. The important nuance: use contextual phrasing, not brittle commands. A directive like "transcribe everything you hear including every disfluency, filler, and false start" can actually backfire — pushing the model that hard can trigger hallucinations, where it starts emitting disfluencies that weren't spoken. The more reliable pattern is to describe the goal ("preserve the speaker's natural linguistic patterns") and let the model deliver a faithful verbatim transcript without over-correcting. For a clean transcript, prompt for the opposite. The point is that filler-word behavior is a dial you set per use case, not a fixed property of the model.
Getting names, numbers, and jargon right
The single highest-return lever on hard audio is telling the model which words to expect. Keyterms prompting lets you supply the names, product terms, account formats, and domain vocabulary specific to your calls, so the model recognizes them even when the audio is poor. If your agents say a product name a general model has never seen, a keyterm turns a guess into a correct transcription. Combine that with conversation context on streaming — where the agent's question ("what's your email?") primes the next answer — and the hardest tokens in your audio become the ones you've most reinforced.
A practical recipe for call-center audio
Putting it together, a robust starting configuration for messy two-party call audio looks like this: transcribe with Universal-3.5 Pro; use multichannel if you have per-speaker channels, diarization if you don't; add your domain vocabulary as keyterms; use a context-style prompt to set verbatim or clean output depending on your use case; and for real-time capture, enable Voice Focus rather than pre-cleaning the audio yourself. Then — and this is the step teams skip — measure it on your own audio.
Measure it on your own truth files
Don't trust a vendor's benchmark or ours as a stand-in for your data. Take a representative sample of your hardest recordings, create reference "truth" transcripts, and score the model against them — ideally with semantic WER, which counts meaning-changing errors more heavily than cosmetic ones. Our guides on how to evaluate speech recognition models and why raw WER can mislead walk through how to do this without fooling yourself. The teams that are happiest with their transcription accuracy are the ones who tuned against their own hard audio, not someone else's easy audio.
The takeaway
Hard audio feels like a wall, but most of it is a set of dials you haven't turned yet. Noise wants Voice Focus, not pre-cleaning. Overlap wants multichannel or diarization, depending on your source. Filler words want a context-style prompt, not a brittle command. Names and numbers want keyterms. Underneath all of it is a model built for the hard stuff — and the entity accuracy to prove it. Tune for your worst 20%, measure on your own files, and the messy audio stops being the thing that breaks your product. For more on why accurate transcripts pay off downstream, see the true cost of inaccurate transcription and how we think about accuracy versus quality.
Frequently asked questions
How accurate is speech-to-text on noisy call center recordings?
Accuracy on noisy recordings depends far more on entity accuracy and configuration than on headline WER. Universal-3.5 Pro is built for difficult audio and delivers large gains on the tokens that matter — a 37% relative improvement in missed entity rate on emails and 29% on medical terms versus Universal-2 — and its behavior on noise can be tuned with Voice Focus, multichannel transcription, and keyterms prompting.
Should I remove background noise before sending audio to a transcription API?
Usually not. Aggressive client-side noise cancellation introduces artifacts that often confuse the model more than the original noise. Send the cleanest original signal you have, and for real-time audio use server-side Voice Focus in Universal-3.5 Pro Streaming to isolate the primary speaker instead of pre-cleaning yourself.
What's the difference between multichannel transcription and speaker diarization?
Multichannel transcribes each speaker from their own separate audio channel, which prevents cross-channel bleed from being attributed to the wrong person — ideal when your call stack already separates agent and customer. Diarization labels who spoke when within a single mixed channel, which is the right tool when you only have one combined track.
How do I handle filler words and stuttering in transcripts?
Filler-word behavior is controllable through prompting in Universal-3.5 Pro, so you can produce a verbatim transcript for legal or medical needs or a clean one for summaries. Use context-style phrasing like "preserve the speaker's natural linguistic patterns" rather than forceful directives to transcribe every disfluency, which can push the model to hallucinate fillers that weren't spoken.
How can I improve transcription accuracy on names, account numbers, and jargon?
Use keyterms prompting to tell the model the specific names, product terms, and identifier formats to expect, so it recognizes them even on poor audio. On real-time streams, pair keyterms with conversation context so the agent's question primes the expected answer — together they give the biggest accuracy lift on hard tokens.
Top comments (0)