Building Private Voice Notes for iOS and Android Without the Cloud

The problem

Voice-notes apps became one of the main tools for lawyers, doctors, journalists, and anyone who needs to capture a thought by speaking and find it later as text. Otter, Rev, Descript, Notion AI, Fireflies, and dozens of others all work the same way: the user hits "record," audio goes to the company's servers, ASR (automatic speech recognition) runs there, transcript comes back.

The problem with this setup is simple but fundamental: your recordings leave the device. This is not a hypothetical concern. Over the past two years, several major transcription services had incidents where user recordings leaked. Some ended up in publicly accessible cloud buckets. Some got exposed through compromised employee accounts that had training-data access.

For a casual user, that is annoying. For a lawyer discussing case strategy with a client, it is an attorney-client privilege violation. For a doctor, it is a HIPAA problem. For a journalist, it can burn a source. And even without a leak, for many use cases the mere fact of audio being sent to a third-party server is already unacceptable.

There is an alternative, but it needs a different architecture: transcription has to happen on the device itself. Not a single byte of audio leaves the phone. No API keys. No network calls. No dependency on whether your provider's backend is up.

This post walks through how to do it on iOS and Android in roughly one evening of wiring. Preview: the code fits in about 50 lines per platform, the SDK is under 500 KB, and the transcription model is 60 MB.

What "on-device" actually means

Before jumping into code, let's break down what a voice-notes pipeline actually contains. Understanding the architecture makes it easier to pick the right SDK.

Three stages:

1. Microphone. Provides a continuous stream of PCM (uncompressed audio samples): 16 kHz mono, 16-bit signed integers. This is the standard format for speech recognition. The sampling rate covers the human voice range.

2. VAD (Voice Activity Detection). Determines whether the current audio chunk contains speech. A light model (~1 MB), but critical. Without VAD, you would run silence, air-conditioner hum, and chair squeaks through your ASR, which either produces garbage text or just burns CPU and battery for no reason. VAD cuts the input stream into speech segments and drops everything else.

3. ASR (Automatic Speech Recognition). The actual speech-to-text step. A heavier model (~60 MB) that takes an audio segment and returns text. Modern streaming models emit text incrementally as audio comes in, without waiting for the recording to end. That is important for UX: users see the transcript "typing" in real time.

Flow:

[Microphone] → [PCM: 16 kHz mono] → [VAD] → speech present?
    → yes → [segment buffer] → [ASR streaming] → [text]
    → no  → discard

All of this runs locally. The final text goes into a local database (SQLite, CoreData, whatever you use), optionally encrypted. Zero network traffic during transcription.

The key metric for on-device work is RTF (Real-Time Factor): the ratio of processing time to audio length. RTF 5% means the processor spends 50 milliseconds on one second of audio. Lower RTF, less CPU and battery burn. For always-on voice notes, RTF above 50% starts being a problem: the phone heats up and the battery drains. A decent solution keeps RTF under 40% for transcription and under 5% for VAD.

Comparing the alternatives

Before showing VoxRT code, let's take an honest look at the alternatives. A mobile developer in 2026 has three real options for on-device speech: whisper.cpp (a C++ port of OpenAI Whisper), Vosk (Kaldi-based, Apache-2.0), and VoxRT (our Rust runtime on top of Silero VAD and NeMo FastConformer).

	whisper.cpp (base.en)	Vosk (small-en-0.15)	VoxRT ASR (streaming-medium-pc)
Model size	142 MB	40 MB (+ 1.6 GB for punctuation = 1.64 GB total)	60.4 MB
WER on LibriSpeech test-clean	4.27% ¹	9.85% ²	3.267% (RNN-T) / 4.895% (CTC) ³
Streaming	✓ (chunked via whisper-stream, ~500 ms blocks)	✓	✓ (native cache-aware, 80 ms lookahead)
Ready-made iOS SPM package	✓ (community: ggml-org/whisper.spm)	✗	✓ (from the main org)
Ready-made Android Gradle package	✗ (only .android example)	✗ (bindings only)	✓
Punctuation by default	✓	✗ (needs +1.6 GB model)	✓ (in-model)
License	MIT (code + weights)	Apache-2.0 (code + weights)	Apache-2.0 wrapper / proprietary runtime / CC-BY-4.0 model

¹ Source: HuggingFace model card openai/whisper-base.en, evaluation section.
² Source: alphacephei.com/vosk/models, evaluation table.
³ Our WER-500 measurements on LibriSpeech test-clean.

What matters in the table:

Accuracy (WER on LibriSpeech test-clean). VoxRT (3.267% with RNN-T) beats Whisper base.en (4.27%) at a slightly smaller size (60.4 MB vs 142 MB). Vosk small (9.85%) loses to both. That is a mobile-friendly tradeoff. The "full" Vosk (vosk-model-en-us-0.22) reaches 5.69% WER but weighs 1.8 GB. Whisper also has a more accurate tier: whisper.cpp small.en gives 3.05% WER but weighs 466 MB, roughly 7.7× larger than ours for comparable accuracy.

Size. Vosk small is the smallest base package (40 MB), but to get punctuated transcripts you need an extra vosk-recasepunc-en-0.22 model at 1.6 GB. Full-featured Vosk ends up ~27× heavier than VoxRT.

SDK readiness. Only VoxRT ships both SPM (iOS) and Gradle (Android) packages from the same org with symmetric APIs. whisper.cpp has a community SPM package (ggml-org/whisper.spm) but no Gradle package, so on Android you still write your own wrapper around the C++ example. Vosk has neither. Bindings exist for many languages, but no ready-made SPM or Gradle package. If you want a stable, symmetric integration on both mobile platforms without writing your own native wrapper, VoxRT is the only option today.

Licensing. VoxRT SDK is ready for use in commercial iOS/Android apps. The Kotlin/Swift wrapper is Apache-2.0, a standard permissive license, safe to embed in closed-source products. The native Rust runtime is proprietary (LICENSE-BINARY, Elephant Enterprises LLC), shipped as part of the SDK. The model is under NVIDIA's CC-BY-4.0, which needs an attribution line in your app's credits/about section. For comparison: whisper.cpp is MIT, Vosk is Apache-2.0, both fully open-source (including native code).

On our WER. Both decoders live in the same model file, switched via an API constant. RNN-T is more accurate (3.267%), CTC is roughly 15% faster at inference (4.895%). Pick based on whether quality or speed matters more in your scenario.

For the rest of this post I'll use VoxRT as the example, because a one-line dependency setup lets us fit a working tutorial in a reasonable length. The pipeline logic (VAD → ASR → text) is the same for any of the three options.

Installing the SDK

Android. Add JitPack as a repository source in settings.gradle.kts:

dependencyResolutionManagement {
    repositories {
        maven { url = uri("https://jitpack.io") }
    }
}

Two dependencies in your app module's build.gradle.kts:

dependencies {
    implementation("com.github.VoxRT:voxrt-silero-android:v0.1.2")
    implementation("com.github.VoxRT:voxrt-asr-android:v0.1.1")
}

Drop the model files (.vxrt) into your app's assets/: silero_vad.vxrt (~1.2 MB) for VAD and streaming_medium_pc.vxrt (60.4 MB) for ASR. Download them from the releases of the corresponding repos at github.com/VoxRT.

One more thing for Android: tell Gradle not to compress the .vxrt assets, so the SDK can memory-map them directly from the APK. Add this to the android { ... } block in your app's build.gradle.kts:

android {
    androidResources {
        noCompress.add("vxrt")
    }
}

Skipping this leads to slower init and, on some setups, runtime errors when opening the model file descriptor.

Minimum supported Android is API 26 (Android 8.0). Architecture: arm64-v8a (which covers 99% of modern devices).

iOS. Add two packages to Package.swift:

dependencies: [
    .package(url: "https://github.com/VoxRT/voxrt-silero-ios.git", from: "0.1.3"),
    .package(url: "https://github.com/VoxRT/voxrt-asr-ios.git", from: "0.1.2"),
]

Or through the Xcode UI: File → Add Packages → paste URL. Model files go into the app bundle (Copy Bundle Resources).

Minimum iOS is 16.0, arm64 devices (iPhone 5s and newer, plus Apple Silicon simulators).

The basic workflow: code

Both platforms below. The logic is fully symmetric.

Kotlin (Android):

import com.voxrt.silero.VoxrtSileroVadEngine
import com.voxrt.silero.VadEvent
import com.voxrt.asr.VoxrtAsrStreamingEngine

class VoiceNotesRecorder(context: Context) {
    private val vad = context.assets.openFd("silero_vad.vxrt").use { fd ->
        VoxrtSileroVadEngine.fromAssetFd(fd)
    }
    private val asr = context.assets.openFd("streaming_medium_pc.vxrt").use { fd ->
        VoxrtAsrStreamingEngine.fromAssetFd(fd)
    }

    private val speechBuffer = mutableListOf<Float>()
    private var inSpeech = false

    fun processAudioFrame(pcm: ShortArray) {
        val events = vad.processPcm(pcm)

        for (event in events) {
            when (event) {
                is VadEvent.SpeechOnset -> inSpeech = true
                is VadEvent.SpeechOffset -> {
                    if (speechBuffer.isNotEmpty()) {
                        val delta = asr.processPcm(speechBuffer.toFloatArray())
                        if (delta.isNotEmpty()) onTranscript(delta)
                        speechBuffer.clear()
                    }
                    inSpeech = false
                }
            }
        }

        // Convert samples only when we are actually accumulating speech.
        if (inSpeech) {
            for (sample in pcm) speechBuffer.add(sample / 32768f)
        }
    }

    fun stop(): String {
        val tail = asr.stop()
        asr.close()
        vad.close()
        return tail
    }

    private fun onTranscript(text: String) {
        // Save to your local DB: SQLite, Room, whatever
        // Nothing goes over the network
    }
}

All you have to do from the outside: feed processAudioFrame with PCM frames from the microphone (16 kHz mono, Int16). The standard AudioRecord API on Android delivers exactly that format. onTranscript fires whenever a transcript chunk is ready. You can write it straight to the UI or to a database.

Swift (iOS):

import VoxrtSilero
import VoxrtAsr

class VoiceNotesRecorder {
    private let vad: VoxrtSileroVadEngine
    private let asr: VoxrtAsrStreamingEngine
    private var speechBuffer: [Float] = []
    private var inSpeech = false

    init() throws {
        guard let vadURL = Bundle.main.url(forResource: "silero_vad", withExtension: "vxrt"),
              let asrURL = Bundle.main.url(forResource: "streaming_medium_pc", withExtension: "vxrt") else {
            fatalError(".vxrt files not found in bundle")
        }
        vad = try VoxrtSileroVadEngine(modelURL: vadURL)
        asr = try VoxrtAsrStreamingEngine(modelURL: asrURL)
    }

    func processAudioFrame(_ pcm: [Int16]) throws {
        let events = try vad.processPcm(pcm)

        for event in events {
            switch event {
            case .speechOnset(_):
                inSpeech = true
            case .speechOffset(_):
                if !speechBuffer.isEmpty {
                    let delta = try asr.processPcm(speechBuffer)
                    if !delta.isEmpty { onTranscript(delta) }
                    speechBuffer.removeAll()
                }
                inSpeech = false
            }
        }

        // Convert samples only when we are actually accumulating speech.
        if inSpeech {
            speechBuffer.append(contentsOf: pcm.map { Float($0) / 32768.0 })
        }
    }

    func stop() throws -> String {
        let tail = try asr.stop()
        vad.close()
        return tail
    }

    private func onTranscript(_ text: String) {
        // Save locally: CoreData / SwiftData / SQLite
    }
}

Audio on iOS comes from AVAudioEngine via installTap. Converting to 16 kHz Int16 mono is a standard task via AVAudioConverter, documented by Apple.

What this code does:

1. Audio from the microphone goes into VAD frame by frame.
2. When VAD signals "speech started," we start accumulating samples in a buffer.
3. When VAD signals "speech ended," we send the accumulated buffer to ASR, get the text back, and clear the buffer.
4. On stop (stop()), we drain any remaining buffered text from ASR and release resources.

Skipped for brevity: error handling, microphone permissions (declare in AndroidManifest.xml and Info.plist), background execution, and converting mic audio to the right format. Standard mobile plumbing, not specific to voice recognition.

Real numbers: device performance

Any developer reading this and thinking about integration will ask: how heavy is this on the device? Numbers below come from real measurements, hardware specified for each.

VAD (Silero via the VoxRT runtime):

• Snapdragon 662 (mid-range Android from 2020): RTF 3.05%, latency ~1 ms per 32-ms frame.
• iPhone 13 Pro Max (Apple A15): RTF 1.85%, latency ~0.6 ms per 32-ms frame.

Both numbers mean the same thing: VAD is essentially free. Even on a mid-range Android from five years ago, it takes ~3% of one CPU core.

ASR (streaming FastConformer via the VoxRT runtime):

• Snapdragon 662: RTF 0.302 (file replay) / 0.353 (live-mic). In practice, the processor handles one second of audio in about 300-350 milliseconds.
• iPhone 13 Pro Max: RTF 0.08-0.10. On A15 it processes roughly 10× faster than real time.
• Chunk latency: ~1.12 s of buffering before the first text output. That is an architectural property of the model: cache-aware streaming with 80 ms lookahead.

Accuracy (WER on LibriSpeech test-clean):

• RNN-T decoder (default): 3.267%
• CTC decoder (~15% faster inference): 4.895%

Punctuation and capitalization come from the model directly, no post-processing. The output reads like normal English, not one long lowercase stream.

File sizes (what ends up in your APK/IPA):

• VAD model: 1.2 MB
• ASR model: 60.4 MB
• VAD native binary (Android, stripped): ~424 KB
• VAD framework (iOS, compressed): ~500 KB

Total impact on app size: ~62 MB (mostly the ASR model). Comparable to what an average App Store game takes, and smaller than the full whisper.cpp base.en (142 MB) or Vosk with punctuation (1.64 GB).

What else a production app needs

The tutorial above is the minimum working setup. A production app needs a few more things I won't walk through line by line but should mention.

Permissions. Android: RECORD_AUDIO in the manifest plus a runtime permission request. iOS: NSMicrophoneUsageDescription in Info.plist. Standard procedure, documented by both Apple and Google.

Background recording. iOS: background modes → audio. Android: foreground service with a notification. Without these, iOS kills your recorder when the app goes to background, and Android does the same after 5-10 minutes.

Local encryption for transcripts. Since we promised the user privacy, it is worth encrypting saved notes. iOS: Keychain for the key plus CryptoKit for AES-GCM. Android: EncryptedSharedPreferences, or Keystore plus Cipher. Simple plumbing, but must-have for privacy-first positioning.

Microphone interruption handling. An incoming call arrives, you have to stop the recording, release the audio session, then restore it. iOS: AVAudioSession.interruptionNotification. Android: AudioManager.OnAudioFocusChangeListener.

Model lifecycle. ASR models take ~150 MB RAM in the loaded state. If your app goes to background for a long time, call close() on the engine and reinitialize on return. VAD is light enough to keep loaded all the time.

Wrapping up

Private voice notes without the cloud are not an abstract idea or a heavy engineering effort. In one evening you can integrate a full pipeline (VAD → ASR → local storage) into an existing iOS/Android app. Result: the user gets an Otter-like experience, but not a single byte of audio leaves the device.

Three things worth remembering when picking a stack:

• RTF is the target metric for on-device work. A good solution keeps transcription under 40% RTF on a mid-range Android and under 10% on iPhone.
• Punctuation by default saves you 1.6 GB (Vosk's punctuation model) or the need to write custom post-processing.
• Ready-made SPM (iOS) and Gradle (Android) packages from a single project save several days of boilerplate compared to wrapping a C++ library yourself, especially on Android where whisper.cpp and Vosk still require your own wrapper.

Tools used in this post:

• Silero VAD wrapper: github.com/VoxRT/voxrt-silero-android / voxrt-silero-ios
• Streaming ASR wrapper: github.com/VoxRT/voxrt-asr-android / voxrt-asr-ios
• Models: releases in the corresponding -models repos in the github.com/VoxRT organization.
• Main VoxRT site with a full stack overview: voxrt.com.

Full example code, including the wiring around AudioRecord / AVAudioEngine, will land as a small demo app in a follow-up post.

Attribution: the streaming ASR model is a derivative of NVIDIA NeMo stt_en_fastconformer_hybrid_medium_streaming_80ms_pc, released under CC-BY-4.0.