Building Private Voice Notes for iOS and Android Without the Cloud

The problem

Voice-notes apps became one of the main tools for lawyers, doctors, journalists, and anyone who needs to capture a thought by speaking and find it later as text. Otter, Rev, Descript, Notion AI, Fireflies, and dozens of others all work the same way: the user hits "record," audio goes to the company's servers, ASR (automatic speech recognition) runs there, transcript comes back.

The problem with this setup is simple but fundamental: your recordings leave the device. For a casual user, that is annoying. For a lawyer discussing case strategy with a client, it is an attorney-client privilege violation. For a doctor, it is a HIPAA problem. For a journalist, it can burn a source. And even without any incident on the provider's side, for many use cases the mere fact of audio being sent to a third-party server is already unacceptable.

There is an alternative, but it needs a different architecture: transcription has to happen on the device itself. Not a single byte of audio leaves the phone. No API keys. No network calls. No dependency on whether your provider's backend is up.

This post walks through how to do it on iOS and Android in roughly one evening of wiring. Preview: the code fits in about 50 lines per platform, the SDK is under 500 KB, and the transcription model is 60 MB.

What "on-device" actually means

Before jumping into code, let's break down what a voice-notes pipeline actually contains. Understanding the architecture makes it easier to pick the right SDK.

Three stages:

1. Microphone. Provides a continuous stream of PCM (uncompressed audio samples): 16 kHz mono, 16-bit signed integers. This is the standard format for speech recognition. The sampling rate covers the human voice range.

2. VAD (Voice Activity Detection). Determines whether the current audio chunk contains speech. A light model (~1 MB), but critical. Without VAD, you would run silence, air-conditioner hum, and chair squeaks through your ASR, which either produces garbage text or just burns CPU and battery for no reason. VAD cuts the input stream into speech segments and drops everything else.

3. ASR (Automatic Speech Recognition). The actual speech-to-text step. A heavier model (~60 MB) that takes an audio segment and returns text. Modern streaming models emit text incrementally as audio comes in, without waiting for the recording to end. That is important for UX: users see the transcript "typing" in real time.

Flow:

[Microphone] → [PCM: 16 kHz mono] → [VAD] → speech present?
    → yes → [segment buffer] → [ASR streaming] → [text]
    → no  → discard

All of this runs locally. The final text goes into a local database (SQLite, CoreData, whatever you use), optionally encrypted. Zero network traffic during transcription.

The key metric for on-device work is RTF (Real-Time Factor): the ratio of processing time to audio length. RTF 5% means the processor spends 50 milliseconds on one second of audio. Lower RTF, less CPU and battery burn. For always-on voice notes, RTF above 50% starts being a problem: the phone heats up and the battery drains. A decent solution keeps RTF under 40% for transcription and under 5% for VAD.

Comparing the alternatives

Before showing VoxRT code, let's look at the honest set of on-device options a mobile developer has in 2026. Four SDK ecosystems worth knowing:

• whisper.cpp: a C++ port of OpenAI Whisper, MIT-licensed. Wide model zoo (tiny/base/small/medium/large). Community iOS SPM package. No official Android Gradle package.
• Vosk: Kaldi-based, Apache-2.0. Broad language coverage. Bindings for many languages but no ready-made SPM or Gradle package for mobile.
• sherpa-onnx: Apache-2.0, from the k2-fsa team. Fully open source. Ships iOS SPM and Android AAR packages. Runs several streaming architectures (Zipformer, Paraformer, and NVIDIA transducer variants). Punctuation is an optional separate model.
• VoxRT: Apache-2.0 wrappers on top of a proprietary Rust runtime, running NeMo FastConformer streaming with in-model punctuation.

There are also the platform-native recognizers: SFSpeechRecognizer on iOS and Android's SpeechRecognizer. Both can run on-device on modern OS versions, but with caveats. Session lengths are capped (typically ~1 minute on iOS, shorter on Android), on-device mode has to be explicitly requested and depends on the user having downloaded the language pack, and vocabulary/accuracy are outside your control. Fine for short dictation, awkward for continuous voice notes.

The table below focuses on the four SDK options, because that is where you actually make an architectural choice.

	whisper.cpp (base.en)	Vosk (small-en-0.15)	sherpa-onnx (streaming Zipformer, en)	VoxRT ASR (streaming-medium-pc)
Model size	142 MB (whisper.cpp fp16 GGML ggml-base.en.bin)	40 MB (+ 1.6 GB for punctuation = 1.64 GB total)	~250 MB (fp32) / ~68 MB (int8) ³	60.4 MB
WER on LibriSpeech test-clean	4.27% ¹	9.85% ²	varies per model	6.96% ⁴
Streaming	✓ (chunked via whisper-stream, ~500 ms blocks)	✓	✓ (native)	✓ (native cache-aware, 80 ms lookahead)
Ready-made iOS SPM package	✓ (community ggml-org/whisper.spm)	✗	✓	✓ (from the main org)
Ready-made Android Gradle/AAR package	✗ (only .android example)	✗ (bindings only)	✓	✓
Punctuation by default	✓ (Whisper is trained with p&c)	✗ (needs +1.6 GB model)	✗ (separate post-processor)	✓ (in-model)
License	MIT (code + weights)	Apache-2.0 (code + weights)	Apache-2.0 (code + weights)	Apache-2.0 wrapper / proprietary runtime / CC-BY-4.0 model

¹ Source: HuggingFace model card openai/whisper-base.en, evaluation section.
² Source: alphacephei.com/vosk/models, evaluation table.
³ Source: sherpa-onnx pre-trained models page (sherpa-onnx-streaming-zipformer-en-2023-06-26).
⁴ Source: HuggingFace model card nvidia/stt_en_fastconformer_hybrid_medium_streaming_80ms_pc, RNN-T on LibriSpeech test set.

Accuracy. No single option is a strict winner. Whisper base.en (4.27%) is currently the most accurate of the numbers we have side-by-side, at 2.4× the model size of VoxRT. sherpa-onnx running a recent streaming Zipformer is in the same accuracy band as Whisper base.en, at roughly 4× the model size of VoxRT (fp32) or comparable to VoxRT (int8, with the usual quantization tradeoffs). VoxRT (6.96%) trades some accuracy for a much smaller model with punctuation baked in. Vosk small (9.85%) is the least accurate, and its punctuation model triples the "small" story once you actually need punctuated output.

Size. VoxRT is the smallest option that ships with punctuation in the box. Vosk small looks smaller (40 MB) until you add the 1.6 GB vosk-recasepunc-en-0.22 model to get usable output, at which point Vosk becomes the heaviest in the table by a wide margin.

SDK integration. Both sherpa-onnx and VoxRT ship SPM (iOS) + Gradle/AAR (Android) packages, which is the meaningful gap versus whisper.cpp and Vosk. Between them: sherpa-onnx has a broader model zoo and a fully open-source runtime (heavier native binary, but no proprietary components). VoxRT has a smaller native runtime (~500 KB per platform on top of the model) and in-model punctuation, at the cost of a proprietary native library.

Licensing. whisper.cpp, Vosk, and sherpa-onnx are fully open source in code and weights, permissive licenses across the board. VoxRT is a hybrid stack. The Kotlin/Swift wrapper is Apache-2.0, safe to embed in closed-source products. The native Rust runtime is proprietary (LICENSE-BINARY, Elephant Enterprises LLC), shipped as part of the SDK. The model is under NVIDIA's CC-BY-4.0, which requires an attribution line in your app's credits section.

On decoders. VoxRT's .vxrt file ships both an RNN-T and a CTC decoder in the same weights, switchable via an API constant. RNN-T is what the table's WER reflects, and it's the more accurate of the two. CTC runs faster at inference and is useful when latency budget is tighter than accuracy budget.

For the rest of this post I'll use VoxRT as the example. The two-line dependency setup keeps the tutorial short, and the pipeline logic (VAD → ASR → text) is the same for any of the four SDK options.

Installing the SDK

Android. Add JitPack as a repository source in settings.gradle.kts:

dependencyResolutionManagement {
    repositories {
        maven { url = uri("https://jitpack.io") }
    }
}

Two dependencies in your app module's build.gradle.kts:

dependencies {
    implementation("com.github.VoxRT:voxrt-silero-android:v0.1.2")
    implementation("com.github.VoxRT:voxrt-asr-android:v0.1.1")
}

Drop the model files (.vxrt) into your app's assets/: silero_vad.vxrt (~1.2 MB) for VAD and streaming_medium_pc.vxrt (60.4 MB) for ASR. Download them from the releases of the corresponding repos at github.com/VoxRT.

One more thing for Android: tell Gradle not to compress the .vxrt assets, so the SDK can memory-map them directly from the APK. Add this to the android { ... } block in your app's build.gradle.kts:

android {
    androidResources {
        noCompress.add("vxrt")
    }
}

Skipping this leads to slower init and, on some setups, runtime errors when opening the model file descriptor.

Minimum supported Android is API 26 (Android 8.0). Architecture: arm64-v8a (which covers 99% of modern devices).

iOS. Add two packages to Package.swift:

dependencies: [
    .package(url: "https://github.com/VoxRT/voxrt-silero-ios.git", from: "0.1.3"),
    .package(url: "https://github.com/VoxRT/voxrt-asr-ios.git", from: "0.1.2"),
]

Or through the Xcode UI: File → Add Packages → paste URL. Model files go into the app bundle (Copy Bundle Resources).

Minimum iOS is 16.0, arm64 devices (iPhone 8 and newer, plus Apple Silicon simulators). iOS 16 dropped support for the older 64-bit iPhones (5s / 6 / 6 Plus / 6s / 6s Plus / 7 / 7 Plus / SE 1st gen), so those don't run this stack.

The basic workflow: code

Both platforms below. The logic is fully symmetric.

Kotlin (Android):

import com.voxrt.silero.VoxrtSileroVadEngine
import com.voxrt.silero.VadEvent
import com.voxrt.asr.VoxrtAsrStreamingEngine

class VoiceNotesRecorder(context: Context) {
    private val vad = context.assets.openFd("silero_vad.vxrt").use { fd ->
        VoxrtSileroVadEngine.fromAssetFd(fd)
    }
    private val asr = context.assets.openFd("streaming_medium_pc.vxrt").use { fd ->
        VoxrtAsrStreamingEngine.fromAssetFd(fd)
    }

    private var inSpeech = false

    fun processAudioFrame(pcm: ShortArray) {
        val events = vad.processPcm(pcm)

        for (event in events) {
            when (event) {
                is VadEvent.SpeechOnset -> {
                    asr.reset()  // fresh K/V cache + LSTM state for the new utterance
                    inSpeech = true
                }
                is VadEvent.SpeechOffset -> {
                    val tail = asr.stop()  // drain any final text the model was still holding
                    if (tail.isNotEmpty()) onTranscript(tail)
                    inSpeech = false
                }
            }
        }

        // Feed audio to the ASR only while VAD says we're inside speech.
        // processPcm returns the text emitted during this call — often empty
        // until ~1 s of audio has accumulated, then non-empty each chunk.
        if (inSpeech) {
            val floatPcm = FloatArray(pcm.size) { i -> pcm[i] / 32768f }
            val delta = asr.processPcm(floatPcm)
            if (delta.isNotEmpty()) onTranscript(delta)
        }
    }

    fun stop() {
        if (inSpeech) {
            val tail = asr.stop()
            if (tail.isNotEmpty()) onTranscript(tail)
        }
        asr.close()
        vad.close()
    }

    private fun onTranscript(text: String) {
        // Save to your local DB: SQLite, Room, whatever
        // Nothing goes over the network
    }
}

All you have to do from the outside: feed processAudioFrame with PCM frames from the microphone (16 kHz mono, Int16). The standard AudioRecord API on Android delivers exactly that format. onTranscript fires whenever the model emits a chunk of text: during speech as audio accumulates, and once more at the end of each utterance to flush the tail.

Swift (iOS):

import VoxrtSilero
import VoxrtAsr

class VoiceNotesRecorder {
    private let vad: VoxrtSileroVadEngine
    private let asr: VoxrtAsrStreamingEngine
    private var inSpeech = false

    init() throws {
        guard let vadURL = Bundle.main.url(forResource: "silero_vad", withExtension: "vxrt"),
              let asrURL = Bundle.main.url(forResource: "streaming_medium_pc", withExtension: "vxrt") else {
            fatalError(".vxrt files not found in bundle")
        }
        vad = try VoxrtSileroVadEngine(modelURL: vadURL)
        asr = try VoxrtAsrStreamingEngine(modelURL: asrURL)
    }

    func processAudioFrame(_ pcm: [Int16]) throws {
        let events = try vad.processPcm(pcm)

        for event in events {
            switch event {
            case .speechOnset(_):
                try asr.reset()  // fresh K/V cache + LSTM state for the new utterance
                inSpeech = true
            case .speechOffset(_):
                let tail = try asr.stop()  // drain any final text the model was still holding
                if !tail.isEmpty { onTranscript(tail) }
                inSpeech = false
            }
        }

        // Feed audio to the ASR only while VAD says we're inside speech.
        // processPcm returns the text emitted during this call — often empty
        // until ~1 s of audio has accumulated, then non-empty each chunk.
        if inSpeech {
            let floatPcm = pcm.map { Float($0) / 32768.0 }
            let delta = try asr.processPcm(floatPcm)
            if !delta.isEmpty { onTranscript(delta) }
        }
    }

    // iOS relies on ARC to release the ASR engine when this instance
    // deallocates — VoxrtAsrStreamingEngine has no explicit close() on iOS.
    // The VAD engine ships close() from an earlier API; call it here.
    func stop() throws {
        if inSpeech {
            let tail = try asr.stop()
            if !tail.isEmpty { onTranscript(tail) }
        }
        vad.close()
    }

    private func onTranscript(_ text: String) {
        // Save locally: CoreData / SwiftData / SQLite
    }
}

Audio on iOS comes from AVAudioEngine via installTap. Converting to 16 kHz Int16 mono is a standard task via AVAudioConverter, documented by Apple.

What this code does:

1. Audio from the microphone goes into VAD frame by frame.
2. When VAD signals speech started, we call asr.reset() to give the ASR fresh K/V cache and LSTM state for the new utterance.
3. During speech, each PCM frame is fed straight into asr.processPcm(). The ASR returns the delta of text emitted during that call. It's usually empty for the first ~1 s while the encoder accumulates audio, then non-empty every chunk boundary. The transcript grows on screen while the user is still talking.
4. When VAD signals speech ended, we call asr.stop() to drain any tail the model was still holding, then wait for the next onset. The same engine handles the next utterance after another reset().
5. On recorder stop(), we flush any in-flight utterance and release resources.

Skipped for brevity: error handling, microphone permissions (declare in AndroidManifest.xml and Info.plist), background execution, and converting mic audio to the right format. Standard mobile plumbing, not specific to voice recognition.

Real numbers: device performance

Any developer reading this and thinking about integration will ask: how heavy is this on the device? Numbers below come from real measurements, hardware specified for each.

VAD (Silero via the VoxRT runtime):

• Snapdragon 662 (mid-range Android from 2020): RTF 3.05%, latency ~1 ms per 32-ms frame.
• iPhone 13 Pro Max (Apple A15): RTF 1.85%, latency ~0.6 ms per 32-ms frame.

Both numbers mean the same thing: VAD is essentially free. Even on a mid-range Android from five years ago, it takes ~3% of one CPU core.

ASR (streaming FastConformer via the VoxRT runtime):

• Snapdragon 662: RTF 0.302 (file replay) / 0.353 (live-mic). In practice, the processor handles one second of audio in about 300-350 milliseconds.
• iPhone 13 Pro Max: RTF 0.08-0.10. On A15 it processes roughly 10× faster than real time.
• First-output latency: ~1.12 s. This is the sum of two separate config values, not one. The encoder consumes speech in 1040 ms streaming chunks, and uses an 80 ms cache-aware lookahead to peek ahead within a chunk. Combined, that's the minimum audio you need before the model emits its first tokens. After that first output, new text flows every ~1 s as each chunk completes.

Accuracy (WER on LibriSpeech test):

• RNN-T decoder (default): 6.96%, from the NVIDIA model card on the LibriSpeech test set. This is the number we run in production. We don't fine-tune the upstream weights.
• The same .vxrt file also ships a CTC decoder, switchable via API constant. CTC runs faster than RNN-T at some accuracy cost. We publish only the RNN-T number.

Punctuation and capitalization come from the model directly, no post-processing. The output reads like normal English, not one long lowercase stream.

File sizes (what ends up in your APK/IPA):

• VAD model: 1.2 MB
• ASR model: 60.4 MB
• VAD native binary (Android, stripped): ~424 KB
• VAD framework (iOS, compressed): ~500 KB

Total impact on app size: ~62 MB (mostly the ASR model). Comparable to what an average App Store game takes, and smaller than the full whisper.cpp base.en (142 MB) or Vosk with punctuation (1.64 GB).

What else a production app needs

The tutorial above is the minimum working setup. A production app needs a few more things I won't walk through line by line but should mention.

Permissions. Android: RECORD_AUDIO in the manifest plus a runtime permission request. iOS: NSMicrophoneUsageDescription in Info.plist. Standard procedure, documented by both Apple and Google.

Background recording. iOS: background modes → audio. Android: foreground service with a notification. Without these, iOS kills your recorder when the app goes to background, and Android does the same after 5-10 minutes.

Local encryption for transcripts. Since we promised the user privacy, it is worth encrypting saved notes. iOS: Keychain for the key plus CryptoKit for AES-GCM. Android: EncryptedSharedPreferences, or Keystore plus Cipher. Simple plumbing, but must-have for privacy-first positioning.

Microphone interruption handling. An incoming call arrives, you have to stop the recording, release the audio session, then restore it. iOS: AVAudioSession.interruptionNotification. Android: AudioManager.OnAudioFocusChangeListener.

Model lifecycle. ASR models take ~150 MB RAM in the loaded state. If your app goes to background for a long time, call close() on the engine and reinitialize on return. VAD is light enough to keep loaded all the time.

Wrapping up

Private voice notes without the cloud are not an abstract idea or a heavy engineering effort. In one evening you can integrate a full pipeline (VAD → ASR → local storage) into an existing iOS/Android app. Result: the user gets an Otter-like experience, but not a single byte of audio leaves the device.

Three things worth remembering when picking a stack:

• RTF is the target metric for on-device work. A good solution keeps transcription under 40% RTF on a mid-range Android and under 10% on iPhone.
• Punctuation by default saves you 1.6 GB (Vosk's punctuation model) or the need to write custom post-processing.
• Ready-made SPM (iOS) and Gradle (Android) packages from a single project save several days of boilerplate compared to wrapping a C++ library yourself, especially on Android where whisper.cpp and Vosk still require your own wrapper.

Tools used in this post:

• Silero VAD wrapper: github.com/VoxRT/voxrt-silero-android / voxrt-silero-ios
• Streaming ASR wrapper: github.com/VoxRT/voxrt-asr-android / voxrt-asr-ios
• Models: releases in the corresponding -models repos in the github.com/VoxRT organization.
• Main VoxRT site with a full stack overview: voxrt.com.

Full example code, including the wiring around AudioRecord / AVAudioEngine, will land as a small demo app in a follow-up post.

Attribution: the streaming ASR model is a derivative of NVIDIA NeMo stt_en_fastconformer_hybrid_medium_streaming_80ms_pc, released under CC-BY-4.0.