Streaming ASR vs Whisper on mobile: when to switch

The problem: Whisper feels laggy for live apps

Voice-first apps became mainstream in 2024-2026. Voice agents, real-time captions, live meeting transcription, voice notes with instant feedback. Developers building this new generation of apps typically look at Whisper first (whisper.cpp, faster-whisper). It is the de facto default for on-device speech-to-text.

And for batch transcription (upload audio file, wait, get transcript), Whisper is genuinely great. Interview transcription, meeting notes, lecture recordings on MacBook or in an Android app. Whisper handles all these accurately and reliably.

But as soon as you try Whisper for live scenarios (user speaks, expects instant response), something feels off. A 300-500 ms delay minimum shows up, often up to 1-2 seconds. Words get "sliced" at chunk boundaries. Transcript comes out in jerks. For a voice agent that should respond like a human in conversation, this is a frustrating UX.

This is not a Whisper bug. It is a design mismatch: Whisper is a batch model retrofitted for streaming, while a real-time scenario fundamentally needs a different architecture.

This post is about what native streaming ASR is, how it differs from Whisper's approach, what alternatives exist, and when you should switch (or stay).

What "streaming" really means in ASR

Before comparing options, terminology first.

Batch ASR processes audio as a whole: the model receives the entire file (or a fixed-length chunk), runs inference in one pass, returns the full transcript. Classic architectures (Whisper, original Wav2Vec2, seq2seq encoder-decoder) are batch by nature.

Streaming ASR processes audio continuously: the model takes a stream of samples, emits text incrementally as speech arrives. Key streaming parameters: lookahead (how many milliseconds of future audio the model "sees" before emitting text) and chunk latency (how many milliseconds the model buffers between emissions).

Classification by lookahead:

• Infinite lookahead: batch (the whole file is known upfront). Whisper, original NeMo Conformer non-streaming.
• Large lookahead (1000ms+): pseudo-streaming batch with large chunks. Whisper.cpp streaming mode, faster-whisper VAD-based.
• Small lookahead (80-320 ms): native streaming. NeMo FastConformer streaming, streaming Conformer variants, Parakeet streaming, some Kaldi TDNN variants.
• Zero lookahead: causal streaming. Fastest response but usually strictly worse accuracy.

The difference between "pseudo-streaming" and "native streaming" is fundamental but often confused. Both technically "stream." But native streaming is trained with a constraint to see only limited future audio, while pseudo-streaming is a hack on top of a batch model.

Whisper's approach: chunked batch retrofit

Whisper is trained on 30-second audio chunks. To get streaming behavior, developers use one of two workarounds.

Whisper.cpp streaming mode: rolling window of 500 ms to 3 s, each chunk is processed as a mini-batch, outputs are concatenated. It works, but has several side effects:

• Chunk boundary artifacts: words at chunk borders can be cut, repeated, or misrecognized.
• Latency equals chunk size plus inference time. On SD662 with Whisper base.en at 500 ms chunks, perceived latency is 700-1500 ms.
• VAD and endpointing need separate tuning (Whisper has no native VAD).
• Punctuation can be inconsistent between chunks (the model "loses context" at borders).

VAD-based streaming wrappers (e.g., Whisper-Streaming or WhisperLive built on faster-whisper): VAD triggers, the accumulated segment gets transcribed, output emits. Cleaner output (no chunk boundary artifacts), but latency is even higher (you have to wait for end of speech for VAD trigger).

Neither of these approaches delivers what native streaming does: continuous text stream with sub-100 ms lookahead. This is an architectural limitation, not a bug.

Whisper is not bad. Whisper is not for live. These are two different task categories.

Native streaming ASR: how it works

Native streaming architectures (NeMo FastConformer streaming, streaming Conformer, Parakeet streaming variants) work differently.

Cache-aware inference. The model is trained knowing it will see only limited future audio (typically 80-320 ms lookahead). Internal hidden states cache between chunks. No need to re-process previous audio for each new segment.

Continuous emission. The model emits tokens incrementally as speech arrives. No "chunk boundaries" in perceived output. Text simply prints in real time.

Trade-off exists, but minimal. Native streaming models are usually slightly less accurate than their batch counterparts (the limited future audio constraint reduces context understanding). But the gap is small, typically 0.2-0.5% WER on LibriSpeech test-clean.

Example architecture: NeMo FastConformer streaming (stt_en_fastconformer_hybrid_medium_streaming_80ms_pc by NVIDIA). Roughly 32 million parameters, hybrid decoder (RNN-T + CTC), 80 ms cache-aware lookahead, WER 3.267% on LibriSpeech test-clean with the RNN-T decoder (our WER-500 measurement, see footnote in the comparison table).

Similar variants:

• Parakeet streaming: same NeMo family, different size variants.
• Streaming Conformer: general architectural family, different implementations from Google, Microsoft, and NVIDIA.
• Kaldi TDNN-F: old-school, but streaming, still used in Vosk.

If the task is live, these models deliver a qualitatively different experience than chunked Whisper.

Our implementation: VoxRT, a Rust runtime around NeMo streaming

We (VoxRT) built a Rust runtime that packages NeMo FastConformer streaming (the same cache-aware architecture described above) into ready-made SPM (iOS) and Gradle(Android) packages. The model is stt_en_fastconformer_hybrid_medium_streaming_80ms_pc from NVIDIA, ported into our .vxrt format without architectural changes (fp16 quantization). Our .vxrt format supports AES-256-GCM encryption for proprietary models, but the NeMo streaming ASR model here remains under CC-BY-4.0, not encrypted.

What this gives a developer compared to the "write your own wrapper" approach:

• One-line integration in Gradle/SPM. No custom Python + ONNX bridge to write for JNI or Objective-C interop.
• Symmetric API surface between iOS and Android. Kotlin and Swift code is nearly identical, same lifecycle pattern (init -> processPcm -> stop).
• Native inference via Rust runtime. No PyTorch, no ONNX runtime dependency, minimal APK/IPA footprint (native binary ~424 KB stripped on Android, ~500 KB framework compressed on iOS).
• Bundled Silero VAD for pre-filtering. No separate VAD integration to write, the 1.2 MB model works out of the box paired with ASR.

Honest trade-off: our runtime is proprietary (LICENSE-BINARY, Elephant Enterprises LLC), though the wrapper is Apache-2.0. The model is under CC-BY-4.0 (NVIDIA), which requires attribution in your app's credits. For commercial closed-source apps, no compliance issues. Pure GPL/AGPL projects will hit a license conflict with our proprietary runtime. In that case, writing your own wrapper around the open-source NeMo runtime is a better path.

From here on, all "streaming ASR" numbers in the post come from our implementation of this architecture. Whisper and Vosk numbers come from their upstream benchmarks (source cited).

Comparison table

Extended comparison versus the first post: added axes for streaming approach, chunk latency, and ready SDK availability. All numbers cited.

	whisper.cpp (base.en)	whisper.cpp (small.en)	Vosk (small-en-0.15)	VoxRT NeMo FastConformer streaming
Model type	Batch (encoder-decoder Transformer)	Batch	Streaming (Kaldi TDNN-F)	Native streaming (Conformer + cache)
Model size	~149 MB ¹	~489 MB ¹	40 MB (+ 1.6 GB for punctuation) ²	60.4 MB ³
WER (LibriSpeech test-clean)	4.27% ⁴	3.05% ⁴	9.85% ²	3.267% (RNN-T) / 4.895% (CTC) ⁵
Streaming approach	Chunked batch (~500 ms blocks)	Chunked batch	Native streaming	Native cache-aware (80 ms lookahead)
Latency pattern	500-1500 ms per chunk (persistent)	500-1500 ms per chunk	300-500 ms per chunk	1.12 s initial buffer, then continuous ~100-200 ms
Punctuation by default	✓ (in-model)	✓	✗ (needs +1.6 GB model)	✓ (in-model)
Ready iOS SPM package	✗ (only community ggml-org/whisper.spm)	✗	✗	✓
Ready Android Gradle package	✗	✗	✗	✓
License	MIT (code + weights)	MIT	Apache-2.0 (code + weights)	Apache-2.0 wrapper / proprietary runtime / CC-BY-4.0 model

¹ whisper.cpp README, ggml format disk size (source lists 142 MiB and 466 MiB, converted to MB).
² alphacephei.com/vosk/models, evaluation table.
³ voxrt-asr-models v0.1.2 downloads table.
⁴ HuggingFace model cards openai/whisper-base.en and openai/whisper-small.en, evaluation section.
⁵ Our WER-500 measurements on LibriSpeech test-clean.

Important latency nuance: whisper.cpp has persistent per-chunk latency (each chunk 500-1500 ms). VoxRT NeMo streaming has an initial buffer of 1.12 s (for cache warmup), then sustained latency of ~100-200 ms perceived. Meaning:

• For short queries (under 3 s): whisper.cpp can feel faster (VoxRT's initial buffer is just overhead).
• For longer live sessions (over 3 s): VoxRT streaming feels smoother (continuous), whisper.cpp remains "jerky."

Real numbers on mobile

Everything above is architectural comparison. But any dev reading this will ask one question: how heavy is this on the device?

VoxRT NeMo FastConformer streaming:

• Snapdragon 662 (mid-range Android from 2020): RTF 0.302 (file replay) / 0.353 (live-mic). That is 300-350 milliseconds of CPU time per 1 second of audio.
• iPhone 13 Pro Max (Apple A15): RTF 0.08-0.10. Processor is roughly 10× faster than real time.
• Runtime memory: ~150 MB steady-state.
• APK/IPA impact: 60.4 MB (model) + ~500 KB (native framework) = ~61 MB added to app size.

VoxRT Silero VAD (used as pre-filter):

• SD662: RTF 3.05%, latency ~1 ms per 32-ms frame.
• iPhone A15: RTF 1.85%, latency ~0.6 ms.
• Model size: 1.2 MB.

Whisper.cpp benchmarks on mobile: no public RTF numbers for Snapdragon or iOS in the whisper.cpp README, only Apple Silicon benchmarks. Community measurements (posts in r/androiddev, r/iOSProgramming) usually put Whisper base.en RTF in the 40-70% range on mid-range Android SoCs from 2020-2022. That is roughly 2× heavier than VoxRT NeMo for comparable accuracy.

Practical conclusion: for always-on live transcription on mid-range Android, Whisper base.en often burns enough CPU to noticeably heat up the device and drain battery. Streaming ASR under 40% RTF is sustainable without visible performance impact.

Decision framework: when to switch, when to stay

Practical rules for the reader.

Stay with Whisper if:

• Batch transcription of files (audio, wait, transcript). Live UX not required. Whisper accuracy is great, community is huge, plenty of ready tools.
• Short one-shot voice notes (user hits record, speaks 10 seconds, releases, gets transcript). Streaming ASR initial buffer is not justified here.
• You do not care about battery or CPU load (e.g., transcription runs only on-demand, not always-on).
• You want maximum community support, example code, and integrations.

Switch to streaming ASR if:

• Live voice agents (chatbot that responds while user speaks). Every 500-1500 ms delay equals broken conversation.
• Real-time captions for accessibility (hearing-impaired users). Here latency directly impacts UX.
• Live meeting transcription with on-screen text visible as people speak.
• Always-on background listening (voice notes 24/7, wake-word context). CPU load matters.

Go with a hosted API (Deepgram, AssemblyAI, etc.) if:

• Concurrency: thousands of parallel sessions. Self-hosted infrastructure becomes overhead.
• Diarization required out of the box (who is speaking when). Native streaming ASR usually has no built-in diarization, you need a separate model (pyannote and similar).
• Multi-language mixed audio. Streaming models are usually English-focused, hosted APIs have multilingual support.
• Budget allows and privacy is not a hard requirement.

Go with self-hosted streaming ASR (native streaming architecture) if:

• Privacy is required (data cannot leave device or your infrastructure).
• Mobile deployment: apps must work offline.
• Scale exists, but budget matters. Hosted APIs get expensive at high volume.
• You want low sustained latency without a hosted API dependency.

Wrapping up

Streaming vs batch in ASR is not a runtime tweak, it is an architectural choice. Whisper for batch is a valid default. For live it is a mismatch. Native streaming ASR (NeMo FastConformer, streaming Conformer, Parakeet variants) is a different tool for a different job.

Three things worth remembering when picking a stack:

• Latency pattern matters more than absolute latency. Persistent 500 ms per chunk (Whisper chunked) subjectively feels worse than 1 s initial buffer + continuous ~100 ms (native streaming).
• Punctuation by default is critical for UX. Whisper and native streaming provide it. Vosk requires a separate 1.6 GB model.
• Ready-made mobile SDK. The only project that ships symmetric SPM + Gradle packages for streaming ASR is us (VoxRT). Whisper.cpp has a community SPM (ggml-org/whisper.spm) but no Gradle. Vosk offers only bindings, wrap yourself.

We build a Rust runtime for on-device voice models. Our ASR repos: github.com/orgs/VoxRT/repositories?q=asr+sort:stars. If you are building a live voice app for iOS, Android, or Linux, this stack plus a native streaming model give you everything needed.

Our first post covered building a full on-device voice notes pipeline (VAD + ASR + local storage) with working Kotlin and Swift snippets: dev.to/voxrtio/building-private-voice-notes-for-ios-and-android-without-the-cloud-54a8.

Learn more about the runtime and product family: voxrt.com.

Attribution: the streaming ASR model is a derivative of NVIDIA NeMo stt_en_fastconformer_hybrid_medium_streaming_80ms_pc, released under CC-BY-4.0.