The default is to send voice to the cloud
If you are shipping a voice feature in your app today, odds are you are sending audio to a server. That is how most existing solutions work. The mobile app captures sound from the microphone, packages it as WAV or opus, streams it to an API endpoint, and gets back JSON with the recognized text. Your business logic then operates on text.
This pattern dominates not because it is the best approach, but because historically it was the only one that worked at real production model sizes. Open ASR models were too big. On-device model quality was noticeably worse than the cloud equivalents. Mobile hardware could not sustain ML inference in the background without destroying battery life inside an hour.
The trade-off did not satisfy us for very long. We built the VoxRT stack for on-device from day one. We did not migrate from the cloud. We picked a different path from the start. This article covers why that decision keeps paying off, what trade-offs we accepted, and where cloud still legitimately wins.
The economics that changed our thinking
Picture a hypothetical voice-notes app with 100,000 monthly active users. The average user dictates 20 minutes a month. That is 2 million minutes of audio to transcribe.
Take a typical cloud ASR provider rate of roughly $0.004 per minute, which is on the lower end of mainstream pricing today. The arithmetic is simple. 2,000,000 x $0.004 = $8,000 a month. $96,000 a year. For transcription alone. No wake-word, no keyword spotting, no TTS.
Now consider on-device. The engineering work is upfront, not per-user. Integrating an on-device SDK is usually 2 to 6 weeks for a straightforward single-platform start, longer when you ship across multiple platforms or need custom optimization. There is ongoing cost too: SDK upgrades when model versions improve, device-specific regressions to debug, new OS releases to validate against. None of it scales linearly with user minutes. The per-unit variable cost drops to zero.
One factor we do not want to hide: on-device SDKs are not universally free. Our stack ships under a hybrid license (open wrappers, proprietary runtime), and other on-device voice vendors use per-seat, per-deployment, or revenue-share pricing. For most teams the engineering cost plus any license fee combined still beats per-minute cloud pricing at scale, but the full picture requires factoring both sides, not just the per-minute delta.
The breakeven point depends on the user base. For our hypothetical app, a few thousand paying users are enough for on-device to beat cloud economics. If you expect your app to reach hundreds of thousands or millions of users, on-device wins over the long run almost without exception.
The calculation also ignores second-order effects. No cold-start latency from the network. No rate limits during peak load. No scaling cost when you grow. No backend team to monitor the uptime of your ASR pipeline. Those effects compound differently per product. For a voice-notes app running at scale, the first two matter most. For an occasional-use feature in a bigger app, the engineering commitment to a mobile SDK may not pay off even if the per-minute math eventually does.
Three cases where on-device wins cleanly
Privacy is a legal requirement. Healthcare apps with voice notes. Legal tools recording client meetings. Personal journals. For these use cases, sending audio to a third-party server is not a performance question. It is a compliance question. HIPAA, GDPR, and the EU AI Act are all simpler to operate under when the audio never physically leaves the device. It is easier to prove to a lawyer that user voice was never sent to a server when the stack is architecturally incapable of making network calls.
The app has to work offline. Field-work tools for inspectors, construction workers, and surveyors. Travel apps on airplanes or in countries without data roaming. Workflows for medics and first responders in remote areas. If a user needs voice input in these contexts, cloud-only is not an option. You can build an offline fallback through a cache, but if you are already building the on-device part, why keep the cloud path running in parallel?
Latency has to be below 100 ms. For wake-word and hot-word detection, the cycle "user said it, app responded" must feel instant. Any network round-trip breaks the UX. Even on perfect 5G, you lose 50 to 150 ms to RTT alone, plus server-side inference. On-device wake-word inference typically lands between 50 and 200 ms end-to-end. For a wake-detection flow, that is the difference between "this works" and "this is annoying."
These three cases are not edge cases. They are a substantial slice of real voice use cases: privacy-sensitive apps, offline-first tools, and always-listening trigger layers inside a larger voice stack.
What is genuinely hard about on-device
This article would not be honest without a section on the trade-offs we accepted.
Model size versus quality. To fit a model into 100 KB (our wake-word), you cut the architecture deep. For our wake-word we accepted a fixed trigger phrase ("Hey Assistant") and ship one proven model, not a custom-trained one per user. That worked for our scenario, but it means we cannot offer custom trigger phrases without a training pipeline that we have not released publicly. The streaming ASR model is larger (around 60 MB of weights for the medium model), still well under what cloud providers run server-side on their multi-gigabyte models.
Hardware fragmentation is real. We ship across iOS, Android, browser (via WASM), and Linux (Raspberry Pi, Jetson, servers). The shared Rust runtime compiles to each target, but every target still needs its own benchmark run and build tuning for local SIMD extensions, memory layout, and audio API conventions. Publishing a single "RTF of 0.05" number across all platforms is not honest. The reality is 5 to 10 benchmark points on different devices, each with its own caveats. That is more work than cloud providers face, because their infrastructure is uniform while ours is not.
Updates ship through the SDK release cycle. If you have a bug in a cloud ASR, you push a fix to the server and everything is current tomorrow. For us, a bug means a new SDK version, npm/pip/CocoaPods updates, and users have to upgrade. Mobile developers know this pain from any native library. A stable C ABI between the runtime and the SDK bindings reduces the breaking-change frequency, but it does not remove the user-upgrade step. That is a real friction we have not solved.
Benchmarking requires discipline. It is easy to cherry-pick one device and publish a pretty number. It is hard to publish a multi-device matrix with measured RTF, different CPU budgets on different cores, and p99 chunk latency. We make this harder on ourselves than we have to because we think the alternative misleads integrators who then build on expectations that do not match their hardware. The problem compounds every time a new flagship phone ships.
On-device debugging in production is thinner. When a cloud ASR regresses on a specific user, the server team has logs, metrics, and the ability to replay the audio. On-device, your telemetry surface is whatever you can get back from the user's device without violating the privacy premise that made you pick on-device in the first place. We ship RTF counters and detection scores back via opt-in telemetry only, and even that cuts off at what a user is comfortable sharing. Diagnosing a regression on one specific chipset family often means buying that device and reproducing the issue locally.
Where this trade-off actually lands
We share the numbers below not as a pitch but as evidence that the on-device path has a real engineering shape in 2026, not just an aspirational one. Each of these came out of an actual measurement that we had to defend internally before publishing.
A wake-word model that fits in 100 KB and sustains 5.3% of one A53 core on a $15 Raspberry Pi Zero 2 W is on the extreme end of what on-device looks like today. On a Pi 4 B or Pi 5 it drops to 1 to 2% CPU. The constraints that forced the 100 KB target (single trigger phrase, fixed, no custom training in-SDK) are real limitations, not features. We shipped them because the alternative was not shipping at all on hardware this cheap.
A voice activity detector running at 3% CPU on a mid-range Android (Snapdragon 662 class) is the ceiling that lets us run always-on microphone capture without destroying battery. We only ship VAD on mobile, not in browser, because the WASM profile for continuous audio capture is different enough that we have not justified the engineering investment yet.
Keyword spotting across 14 fixed English commands at 0.97% RTF in Chrome WASM and 16% RTF on Snapdragon 662 is what makes a browser voice-UX feel instant. The full browser stack (runtime included) bundles to about 1.47 MB. That is small enough to not matter for most apps and large enough to matter for landing pages optimized for mobile 3G.
Streaming ASR (NeMo FastConformer, 60 MB weights) runs real-time on mobile. Partial transcripts arrive while the user is still speaking, with finalization after a silence timeout. This is the model that would have been impossible to ship on-device three years ago. Today it is a shipping choice.
These numbers live in the READMEs at github.com/VoxRT. We publish them with the test set, the hardware, the methodology, and the caveats. We do not publish subset measurements next to another vendor's full-set measurements. If our wake-word gets compared with another approach, both numbers need to be on the same test set, or it is not a comparison.
Where cloud still wins honestly
Cloud ASR is the right call for several scenarios:
Multilingual coverage across 50-plus languages. Open ASR models for rarer languages are often dramatically worse than cloud equivalents. If you need voice in Swahili, Thai, or Tagalog, cloud providers are still ahead. On-device multilingual coverage is catching up, but slower.
Fast iteration on models. A cloud team can push an improved model today, and tomorrow every user gets better accuracy without any change to their app. On-device apps have to re-ship. For products that iterate fast on model quality, that is a meaningful velocity difference.
Enterprise compliance tooling around the API. Major cloud ASR providers come with SOC2, HIPAA, and PCI-DSS compliance infrastructure, audit logs, and enterprise SLA agreements. If you sell into the Fortune 500, buying compliance through a vendor is sometimes easier than proving it yourself for on-device deployment.
Open-ended complex workflows. If your voice UX is "ask AI a complex question via voice, run it through ASR, pipe it to an LLM, get a voice response," the entire pipeline already lives on a server. Adding transcription on the same server costs you very little extra latency. The on-device win on the first step is small when every other step already runs in the cloud.
These cases are real. On-device is not a universal answer. It is the right answer for a specific set of product scenarios that are becoming more common.
Where this is going
To understand why we think on-device picks up more market share, it helps to see how much ground has already shifted. Capabilities that were impractical or research-only three years ago are standard engineering choices today:
| Capability | 2022-2023 state | 2026 state |
|---|---|---|
| Streaming ASR on mobile | Whisper-class models (1-3 GB, batch architecture) impractical for real-time mobile streaming | FastConformer-class models (~60 MB weights) run real-time on commodity mobile hardware |
| Browser voice inference | Pre-SIMD128 WASM, 2-4x realtime typical, Safari lacked SIMD128 until March 2023 | SIMD128 across all major browsers, sub-1% RTF achievable |
| Wake-word on cheap hardware | Reproducible open benchmarks rare on Pi Zero class, mostly proprietary paid tiers | Open benchmarks published, 5.3% RTF measured on $15 Pi Zero 2 W |
| Voice-cloning TTS on-device | On-device voice-cloning was research-stage, production voice-cloning was cloud-only | Shipping products exist, single-step flow sampling made fast synthesis realistic |
Three trends we see on the two-to-three year horizon:
Mobile silicon for ML becomes standard. Apple Neural Engine in iPhones, Google Tensor TPU, Qualcomm Hexagon NPU in Snapdragon. Every flagship now has dedicated silicon for ML inference. What was a feat in 2022 (running a transformer on a phone) is a standard capability in 2026.
Browser as a first-class target. WASM SIMD128 (available in Chrome 91+, Firefox 89+, and Safari 16.4+) opened the door to serious ML inference in the browser. Our 100 KB wake-word runs at 0.16% RTF in Chrome M4. A few years ago that was not possible without a native wrapper. Fully-in-browser voice UX is now a real option.
Privacy becomes a compliance requirement. The EU AI Act for GPAI models started applying with enforcement teeth. Regulators in the US and Asia are following. For voice-app companies, proving to a regulator that user voice is never sent to the cloud is easier when the architecture makes that impossible by construction. On-device moves from nice-to-have to a compliance posture.
Our prediction is that cloud ASR stays the default for broad applications over the next 2-3 years, but on-device picks up market share in privacy-sensitive, offline-first, and latency-critical segments. This is not "on-device will kill cloud." It is a reasonable coexistence of two architectural approaches for different use cases.
Repos and resources
Our runtime and SDKs are on github.com/VoxRT. Platform-specific bindings:
- Linux (Pi, Jetson, servers): voxrt-wake-word-linux, voxrt-kws-linux, voxrt-asr-linux
- Browser (WASM): voxrt-wake-word-browser, voxrt-kws-browser
- Mobile: iOS and Android bindings in the respective repos
More technical articles from our team on on-device voice are on dev.to/voxrtio.
If you looked at on-device voice recently and chose cloud instead, what was the deciding factor? The honest answers are the ones worth hearing.
Top comments (0)