For most of the last decade, transcription was something you did after the fact.
You recorded a call, dumped the file into a queue, waited a few minutes (or a few hours), and got back a block of text. That was the deal. Batch, async, post-hoc—whatever you want to call it, the audio was already over by the time the model saw it. And for a long time, that was fine, because that's all the technology could reliably do.
Recently, the field has quietly crossed the line where the most interesting, highest-value speech-to-text work happens while people are still talking. Realtime isn't a niche feature you bolt on for a captions demo. It's becoming the default that everything else gets measured against.
Here's the argument.
The shift: the interesting STT workloads are now live
Look at where the energy is going in Voice AI right now.
Voice agents that answer the phone and actually resolve the issue. Agent-assist tools that whisper the next best action to a rep mid-call. Live captions that keep up with a fast talker. Ambient scribes that write the clinical note as the visit unfolds. Real-time conversation intelligence that flags a compliance risk or a churn signal before the call ends, not in a report the next morning.
Every one of those is realtime-first. Not realtime-optional—realtime-first. The value evaporates if you make the user wait.
Compare that to the classic batch workloads: transcribing a podcast back catalog, indexing a media archive, generating subtitles for content that already exists. Those are real, they matter, and they're not going away. But notice the pattern—they're all about audio that's already done. The frontier, the stuff people are building companies around in 2026, is about audio that's happening.
So the question flips. It used to be "when do I actually need realtime?" Now it's closer to "why am I making this a two-step process at all?"
What changed: latency and accuracy crossed the usability line at the same time
For years you could have fast transcription or accurate transcription in the streaming setting, but picking both felt like a trap. Realtime models were noticeably worse than their batch cousins. So teams defaulted to batch when quality mattered and only reached for streaming when they absolutely had to.
That tradeoff is mostly gone. And it went away because two curves crossed the usability threshold at roughly the same moment.
The first is latency. Universal-3.5 Pro Realtime, the current streaming flagship, returns partial transcripts in a few hundred milliseconds and finalizes in the same ballpark. The prior generation, Universal-3 Pro Streaming, already ran around 150ms P50 post-VAD, roughly 240ms at P90, with time-to-complete-transcript near 250ms. Those aren't "good for realtime" numbers. Those are numbers where a person on the other end of a phone call can't tell a machine is in the loop.
The second curve is accuracy. On Pipecat's open STT benchmark, run on real agent conversations, Universal-3.5 Pro Realtime posted a 6.99% pooled word error rate. For context, that same benchmark put Deepgram Flux at 15.58%, ElevenLabs Scribe v2 at 9.76%, and Google Chirp3 at 9.04%. The entity numbers are where it really shows up: 15.31% entity error rate versus 50.50% for the next-closest streaming competitor on that run, and 3.55% on phone numbers.
| Streaming model | Pooled word error rate |
|---|---|
| Universal-3.5 Pro Realtime | 6.99% |
| Google Chirp3 | 9.04% |
| ElevenLabs Scribe v2 | 9.76% |
| Deepgram Flux | 15.58% |
Sit with that for a second. A streaming model, transcribing live, is now clearing accuracy bars that batch models were bragging about a couple of years ago.
And it keeps getting smarter mid-conversation. Universal-3.5 Pro Realtime takes direction from the application through agent_context—you pass in what the agent just asked, so a mumbled reply, a spelled-out account ID, or a one-word "yep" resolves correctly instead of turning into garbage. Across a benchmark of 20,000 voice agent audio files, passing that context cut WER by 10.2%, with fabrications down 18.3% and hallucinations down 17.2%.
Then there's turn-taking. Old systems decided you were done talking by waiting for silence, which is why they interrupted you every time you paused to think. The end-of-turn detection here reads tonality, pacing, and rhythm—not just a silence timer—and fires in about 300ms. That's the difference between a conversation and a walkie-talkie.
When fast and accurate stop being a tradeoff, the reason to default to batch quietly disappears.
But here's the thing: realtime is genuinely harder
It may be easy to read the section above and conclude realtime is just batch with a lower latency setting. It's not. It's a different engineering problem, and pretending otherwise is how teams ship a demo that falls apart in production.
Batch has a luxury realtime never gets: the whole recording. A batch model can look ahead, reconsider, and clean up before it hands you a single finished answer. A streaming model has to commit to words it can't take back while the sentence is still being spoken.
That creates hard problems that simply don't exist in async:
Turn detection. Deciding when a person is done is its own discipline. Cut them off and the agent talks over them; wait too long and it feels laggy and dead. Getting this right is arguably harder than the transcription itself.
Partial versus final transcripts. Streaming emits guesses (partials) that it later confirms or revises (finals). Your application has to be built around that flicker—rendering something useful now, correcting it gracefully a moment later—without making the UI jump around like a broken slot machine.
Live context and memory. A batch model gets the full conversation for free. A streaming model has to carry context forward as it goes. Universal-3.5 Pro Realtime keeps a short rolling memory (Context Carryover, on by default) so it remembers what was said thirty seconds ago instead of treating every utterance as a cold start.
Diarization with revision. Figuring out who's speaking, live, is brutal—especially with overlap and short back-and-forth turns. The approach that actually works is to label speakers in real time and then send a single correction that re-clusters the whole session within about half a second of the stream ending. We wrote up the mechanics in streaming speaker diarization, and it's a good illustration of the general principle: realtime often means being provisionally right now and definitively right slightly later.
None of this is a reason to avoid realtime. It's a reason to take it seriously as its own category—and to be skeptical of any provider that treats streaming as an afterthought to their batch product.
The business case: latency is starting to look like a revenue metric
Here's where it gets interesting for the people who sign the invoices.
We've spent years talking about transcription accuracy as the thing that drives business outcomes, and it does—we've written a whole piece on the true cost of inaccurate transcription. But in realtime systems, latency starts behaving like its own outcome variable.
Think about a voice agent taking an inbound call. Every extra beat of dead air after the caller stops talking is a beat where the caller wonders if the line dropped, gets annoyed, or hangs up. I'd frame it as an open question the whole industry should be measuring more rigorously: does streaming latency move call conversion and hangup rates? My strong suspicion, from what we see across customers, is yes—and that the effect is larger than most teams assume. A voice agent that responds in about a second feels like a conversation. One that responds in three feels like a hold queue. Same transcript, wildly different business result.
That reframes the whole evaluation. If you're only benchmarking word error rate, you're measuring half the system. In a live setting, how fast the words arrive and how naturally the turns flow are load-bearing for the metric you actually care about, whether that's containment rate, CSAT, or conversion. This is part of why we keep arguing that word error rate alone is broken as a way to judge a model—it says nothing about whether the system feels alive.
Where batch still absolutely makes sense
I'd lose credibility fast if I told you batch is dead. It isn't, and I'd steer you toward it in plenty of cases.
| Realtime (streaming) | Batch (async) | |
|---|---|---|
| Best for | Interactive work — voice agents, agent assist, live captions, ambient scribes | Already-recorded audio — archives, media libraries, overnight analytics |
| Model | Universal-3.5 Pro Realtime | Universal-3.5 Pro |
| Pricing | $0.45/hr base, billed on session duration | $0.21/hr, billed on audio duration |
| Key advantage | Acts on speech while the conversation is still happening | Look-ahead over the whole file; simpler and cheaper at volume |
If the audio is already recorded and nobody's waiting on the result, batch is the right tool. Transcribing an archive of ten thousand old calls. Subtitling a media library. Running analytics over a quarter of sales conversations overnight. There's no live interaction to preserve, so paying the complexity tax of streaming buys you nothing.
Cost is real, too. Batch speech-to-text with Universal-3.5 Pro runs $0.21/hr of audio, billed on audio duration. Streaming with Universal-3.5 Pro Realtime is $0.45/hr base, billed on session duration—the whole time the WebSocket is open. For huge volumes of already-finished audio, the async path is simpler and cheaper, full stop.
And batch still gets that look-ahead luxury. When you can afford to wait, a model that sees the entire file can sometimes squeeze out accuracy that no streaming system will match on the hardest audio.
So the honest position isn't "realtime replaces batch." It's "realtime becomes the default for interactive work, and batch keeps owning the archive." Two tools, clearer boundaries.
What to look for in a realtime STT API
If you buy the argument that realtime is where the field is heading, here's what I'd actually evaluate—because "we support streaming" on a pricing page tells you almost nothing.
Ask about latency and accuracy together, on realistic audio. A model that's fast on clean read speech and falls apart on a noisy phone call in an accent isn't solving your problem. That's why we benchmark on real agent conversations, not lab recordings, and publish the numbers on our benchmarks page.
Ask how turn detection works. Silence-timer only? Or does it read tonality and pacing? This single choice determines whether your product feels human.
Ask whether the model can take live context. Can you feed it the agent's current question or a set of key terms mid-stream? agent_context and keyterm prompting are the difference between a model that guesses at "my ID is A-as-in-apple, 4, 4, 9" and one that gets it right.
Ask about diarization and whether it revises. Live speaker labels that never get corrected will drift. The revise-at-end-of-turn pattern is what keeps them honest.
And ask about the production stuff—concurrency limits, EU data residency, a signable BAA for healthcare workloads. Universal-3.5 Pro Realtime runs with unlimited concurrency and no rate limits, which matters the day your traffic spikes and you don't want to file a ticket to serve it.
For the full picture of how we think about the live path, the streaming speech-to-text product page is the place to start, and if you're weighing accuracy specifically, how accurate is speech-to-text in 2026 digs into the numbers.
The close: the transcript stops being the product
Here's the forward-looking part, and it's the thing I've been circling the whole time.
As realtime becomes the default, the transcript itself stops being the deliverable. For decades the output of speech-to-text was a document—text you'd read, search, or file. But a realtime transcript isn't something a human reads. It's something another system consumes, instantly, to make a decision: route the call, surface the answer, flag the risk, generate the next turn.
That changes what "good" means. The winning realtime models won't just be the ones with the lowest word error rate. They'll be the ones that emit the most machine-actionable stream—context-aware partials, confident finals, live speaker turns, end-of-turn signals—in a shape the next system can act on without waiting.
In other words, speech-to-text is quietly turning from a transcription tool into a real-time interface between human speech and software. The transcript is becoming plumbing. And once you see it that way, batch-versus-realtime stops being a feature comparison and starts looking like the difference between reading history and participating in the present.
That's the future I'd bet on. The rest of the field is already building for it.
Frequently asked questions
Is realtime speech-to-text replacing batch transcription?
No—and anyone who says so is oversimplifying. Realtime is becoming the default for interactive workloads like voice agents, agent assist, live captions, and ambient scribes, where making a user wait kills the value. Batch still wins for already-recorded audio: archives, bulk media, and overnight analytics, where it's simpler and cheaper. Batch async runs $0.21/hr with Universal-3.5 Pro; streaming runs $0.45/hr base. Think two tools with clearer boundaries, not a replacement.
What's the difference between realtime and streaming speech-to-text?
They're the same thing. "Streaming" describes the mechanism—audio flows over a WebSocket connection and the model returns partial and final transcripts as it listens. "Realtime" describes the outcome—results arrive fast enough to act on while the conversation is still happening. AssemblyAI's streaming API uses wss://streaming.assemblyai.com/v3/ws.
How low does latency need to be for a good voice experience?
For a natural back-and-forth, you want transcripts arriving in a few hundred milliseconds and turn detection firing quickly enough that the system doesn't talk over people or leave dead air. Universal-3.5 Pro Realtime returns partials in a few hundred ms, and its end-of-turn detection reads tonality and pacing to fire in about 300ms rather than waiting on a silence timer. End to end, a well-built voice agent lands around one second.
Do you sacrifice accuracy by going realtime?
Much less than you used to. On Pipecat's open STT benchmark of real agent conversations, Universal-3.5 Pro Realtime posted a 6.99% pooled word error rate—ahead of Deepgram Flux (15.58%), ElevenLabs Scribe v2 (9.76%), and Google Chirp3 (9.04%). Passing agent context cut WER by a further 10.2% across 20,000 files. Batch can still edge ahead on the hardest audio because it sees the whole file, but the gap has narrowed dramatically.
What are the main realtime speech-to-text use cases?
Voice agents, live agent assist for contact center reps, real-time conversation intelligence, live captioning, and ambient clinical scribes. The common thread: another system consumes the transcript instantly to make a decision, so post-call transcription wouldn't work.
How do I get started with realtime speech-to-text?
Open a WebSocket to the streaming endpoint and start sending audio—you'll get partial and final transcripts back right away. The streaming speech-to-text product page has the details.
Top comments (0)