---
title: "On-Device Whisper on Android: NNAPI Delegation, INT8 Tradeoffs, and the Real-Time Latency Ceiling"
published: true
description: "Wire a quantized Whisper TFLite model to Android's NNAPI delegate with the right INT8 tradeoffs, buffer sizing strategy, and runtime fallback chain — and actually hit real-time transcription on device."
tags: [android, kotlin, mobile, architecture]
canonical_url: https://blog.mvpfactory.co/on-device-whisper-nnapi-int8-latency-android
---
## What We Are Building
By the end of this tutorial you will have a working Android pipeline that runs OpenAI's Whisper `tiny.en` model fully on-device using TFLite and Android's Neural Networks API (NNAPI). We will cover delegate selection with a runtime fallback chain, mel-spectrogram preprocessing off the audio thread, INT8 vs FP16 weight tradeoffs, and buffer chunk sizing for both streaming and utterance-level inference.
The target: sub-250 ms transcription latency on a mid-range Snapdragon device, no server round trip required.
---
## Prerequisites
- Android Studio Flamingo or later
- A physical device with a Snapdragon 8 Gen 1 or comparable SoC (emulator will not exercise NNAPI meaningfully)
- TFLite runtime dependency: `org.tensorflow:tensorflow-lite:2.13.0`
- TFLite NNAPI delegate: `org.tensorflow:tensorflow-lite-gpu-delegate-plugin:0.4.4`
- Whisper `tiny.en` quantized to INT8 as a `.tflite` flatbuffer — the [whisper.tflite](https://github.com/usefulsensors/openai-whisper) project has a ready-made conversion pipeline
- Basic familiarity with Kotlin coroutines and `AudioRecord`
---
## Step 1 — Build the Delegate Fallback Chain at Runtime
Most teams treat NNAPI as binary: either it works or it does not. In production, that breaks fast. You need a tiered fallback chain detected at runtime.
Let me show you a pattern I use in every project.
kotlin
fun buildInterpreter(model: MappedByteBuffer): Interpreter {
val nnapi = NnApiDelegate(
NnApiDelegate.Options().apply {
acceleratorName = "qti-dsp" // Qualcomm DSP; null = auto
useNnapiCpu = false
allowFp16PrecisionForFp32 = true
}
)
val gpu = GpuDelegate(
GpuDelegate.Options().apply { precisionLossAllowed = true }
)
val options = Interpreter.Options().apply {
try {
addDelegate(nnapi)
} catch (e: Exception) {
try { addDelegate(gpu) } catch (ignored: Exception) {
setNumThreads(4) // CPU fallback
}
}
}
return Interpreter(model, options)
}
Run `android.os.Build` checks and `NnApiDelegate.getNnApiErrno()` post-inference to catch silent delegation failures. NNAPI will fall back to CPU on unsupported ops without throwing — which is a fun bug to discover in production.
Cache the winning delegate in `SharedPreferences` after first-launch detection. Skip the detection cost on all subsequent runs.
---
## Step 2 — Choose Your Precision Tier
Here is the tradeoff table you actually need when selecting a model variant:
| Precision | Model size | Avg latency (tiny, 30s audio) | WER delta vs FP32 | DSP compatible |
|-------------|------------|-------------------------------|-------------------|----------------|
| FP32 | ~150 MB | 620 ms | baseline | No |
| FP16 | ~75 MB | 340 ms | +0.3% | Partial |
| INT8 (PTQ) | ~38 MB | 190 ms | +1.1–1.8% | Yes |
Post-training quantization at INT8 is the sweet spot for NNAPI DSP delegation. The WER increase is real — do not let anyone tell you quantization is free — but it stays acceptable for voice UI workloads. FP16 is your best option when targeting GPU delegation on Mali or Adreno without DSP support.
Quantize against a representative English speech calibration set. The WER penalty stays below 2% for `tiny.en` and `base.en`, which is acceptable for command-and-control and caption use cases.
---
## Step 3 — Keep Mel-Spectrogram Preprocessing Off the Audio Thread
The audio thread budget on Android is tight — typically 4–8 ms per callback at 16 kHz. Mel-spectrogram extraction for Whisper requires 80 mel bins over a 25 ms window with 10 ms hop. That is CPU-expensive. Do not block the audio callback.
Here is the minimal setup to get this working:
kotlin
// AudioRecord callback → ring buffer → coroutine on IO dispatcher
audioRecord.setRecordPositionUpdateListener(object : AudioRecord.OnRecordPositionUpdateListener {
override fun onPeriodicNotification(recorder: AudioRecord) {
val chunk = ShortArray(CHUNK_SIZE)
recorder.read(chunk, 0, CHUNK_SIZE)
ringBuffer.offer(chunk) // lock-free hand-off
}
override fun onMarkerReached(recorder: AudioRecord) {}
}, audioHandler)
// Separate coroutine
launch(Dispatchers.Default) {
for (chunk in ringBuffer) {
val mel = computeMelSpectrogram(chunk, sampleRate = 16000)
inferenceQueue.send(mel)
}
}
The lock-free ring buffer is the critical piece. Any blocking call inside `onPeriodicNotification` risks an audio underrun and you will never hear the end of it from your QA team.
---
## Step 4 — Size Your Buffers for the Latency Target You Actually Have
Whisper was trained on 30-second fixed-length audio. That is the core tension for streaming inference.
**Utterance-level batching (simplest):** 30-second chunks give best accuracy with the least engineering overhead. Good fit for voice memos or dictation where latency above 1 second is acceptable.
**Streaming inference:** 320–480 ms chunks with ~50% overlap and silence detection to gate inference calls. You will fire the model 3–4× more often, but perceived latency drops below 500 ms.
Use VAD — WebRTC VAD via JNI or `SileroVAD` TFLite — to suppress inference during silence. Gate on voice activity, not a fixed timer. Firing the model on silence wastes 190 ms of DSP time and heats the device for nothing. A lightweight VAD model costs under 5 ms and halves your inference call volume in real-world usage.
For most production voice UIs, a 1.5-second VAD-gated chunk through the INT8 NNAPI pipeline lands comfortably under 250 ms. That is the ceiling where transcription starts feeling instantaneous rather than merely responsive.
---
## Gotchas
**Silent NNAPI fallback to CPU.** NNAPI will silently downgrade unsupported ops to CPU without throwing an exception. Always call `NnApiDelegate.getNnApiErrno()` after your first inference in a session and log the result. If you skip this, you will benchmark a "DSP run" that is actually running on four CPU cores.
**Delegate construction exceptions are swallowed by the pattern above.** The nested try-catch in `buildInterpreter` is intentional — delegate constructors can throw on unsupported hardware — but make sure you log each fallback level. Silent CPU fallback on a flagship device is a performance regression you want to catch in CI.
**30-second padding requirement.** Whisper's encoder expects exactly 30 seconds of mel input (3000 frames). Shorter audio must be zero-padded. Forgetting this gives you garbage output with no obvious error — the model just hallucinates text to fill the gap.
**`AudioRecord` buffer sizing.** Your `CHUNK_SIZE` must be a multiple of the minimum buffer size returned by `AudioRecord.getMinBufferSize()`. Sizes below the minimum will cause `AudioRecord.read()` to return an error code silently.
---
## Conclusion
The pipeline is: INT8 model + NNAPI DSP delegate + lock-free ring buffer handoff + VAD-gated chunking. Get those four things right and on-device Whisper is a production-viable choice for offline, privacy-sensitive, or latency-constrained voice features.
The docs do not mention this, but the hardest part is not the model — it is the plumbing around it. The delegate fallback chain, silent NNAPI failures, and audio thread budget constraints are where projects actually stall.
**Further reading:**
- [TFLite NNAPI delegate documentation](https://www.tensorflow.org/lite/performance/nnapi)
- [Whisper.tflite conversion tooling](https://github.com/usefulsensors/openai-whisper)
- [Android AudioRecord best practices](https://developer.android.com/reference/android/media/AudioRecord)
- [SileroVAD](https://github.com/snakers4/silero-vad) for lightweight voice activity detection
Top comments (0)