---
title: "TFLite Delegates on Android: GPU, NNAPI, and Fallback Chains That Actually Work"
published: true
description: "GPU delegates crash off-thread. NNAPI silently falls back to CPU. Emulators lie by 3–5x. Here is the production delegate selection strategy that fixes all three."
tags: kotlin, android, mobile, architecture
canonical_url: https://mvpfactory.co/blog/tflite-delegates-android-gpu-nnapi-fallback
---
## What We Are Building
By the end of this tutorial, you will have a crash-free TFLite delegate selection strategy covering GPU thread-safety, NNAPI compatibility across API levels, and a fallback chain that tells you exactly which delegate is actually running. Everything here is tested on physical hardware — Pixel 7 and Galaxy S22 — with TensorFlow Lite 2.14, Kotlin 1.9, and minSdk 24.
## Prerequisites
- Android project targeting minSdk 24+
- TensorFlow Lite 2.14 added to your dependencies
- A physical Android device for benchmarking (the emulator will actively mislead you — I will show you the numbers)
---
## The Core Problem
Most teams treat delegate selection as a configuration problem. It is not. It is a **runtime environment problem**. The delegate you initialize is only as good as the hardware underneath it, the API level it runs on, and — critically — the thread it was born on.
Here is the trap you hit on day two: you wire up a `GpuDelegate`, run your integration tests, ship to QA — and half the test devices crash on cold start. The other half are mysteriously slow. Your emulator numbers looked great.
Let me show you a pattern I use in every project that handles all three failure modes at once.
---
## Step 1: Pin the GPU Delegate to a Single Thread
The GPU delegate wraps an OpenGL ES or OpenCL context. That context is thread-local. Create it on one thread, call `interpreter.run()` from another — you get a silent crash or undefined behavior.
kotlin
// Must run entirely on the same thread
val gpuDelegate = GpuDelegate(
GpuDelegate.Options().apply {
isPrecisionLossAllowed = true // enables fp16, ~1.4x faster on Mali/Adreno
}
)
val options = Interpreter.Options().apply {
addDelegate(gpuDelegate)
}
val interpreter = Interpreter(model, options)
If you are using coroutines, pin this to a dedicated `Dispatcher` backed by a single thread — not `Dispatchers.IO`, which is a pool.
---
## Step 2: Gate NNAPI on the Right API Level
NNAPI is available from API 27, but the number that actually matters in production is **API 28**, where op coverage became meaningful. Below that, `NnApiDelegate` initializes without error and accelerates nothing — or worse, partial graph acceleration with overhead that exceeds the CPU baseline (observed in our testing on API 27 emulation layers).
| API Level | NNAPI Status | Practical Acceleration |
|-----------|-------------|------------------------|
| < 27 | Not available | None — must skip |
| 27 | Available, sparse ops | Marginal or negative |
| 28 | Improved op set | Conv layers, basic MobileNet |
| 29+ | Full acceleration profile | Most standard architectures |
| 31+ | NNAPI 1.3, int8 support | Production-grade for quantized models |
The key option that buys real throughput on fp32 models targeting API 28+:
kotlin
val nnApiDelegate = NnApiDelegate(
NnApiDelegate.Options().apply {
allowFp16PrecisionForFp32 = true
executionPreference =
NnApiDelegate.Options.EXECUTION_PREFERENCE_FAST_SINGLE_ANSWER
}
)
---
## Step 3: Build the Fallback Chain
GPU wins on latency, CPU wins on predictability. Here is the minimal setup to get this working in production:
kotlin
fun buildDelegate(context: Context): Delegate? = runCatching {
GpuDelegate(GpuDelegate.Options().apply { isPrecisionLossAllowed = true })
}.getOrElse {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.P) {
runCatching { NnApiDelegate() }.getOrNull()
} else null
// null = Interpreter falls through to multithreaded CPU
// CPU inference is thread-safe by default — no pinning required here
}
Log which delegate was actually selected. Silent CPU fallback is the leading cause of "why is inference slow on that device" bugs in production.
---
## Step 4: Benchmark on Physical Hardware Only
Here is the gotcha that will save you hours: emulator numbers are not just inaccurate — they actively lie about delegate performance. Use the [TFLite Benchmark Tool](https://www.tensorflow.org/lite/performance/measurement) on physical hardware:
bash
adb shell /data/local/tmp/benchmark_model \
--graph=/data/local/tmp/model.tflite \
--use_gpu=true \
--num_runs=50 \
--warmup_runs=5
MobileNetV3-Small classification results across delegates, measured on physical hardware:
| Delegate | Pixel 7 (ms) | Galaxy S22 (ms) | Emulator (ms) |
|----------|-------------|-----------------|---------------|
| CPU (4 threads) | 46 | 58 | 190 |
| GPU (fp16) | 11 | 16 | N/A |
| NNAPI (API 31) | 14 | 20 | 820 |
The emulator NNAPI number is not a typo. It routes through a software emulation layer that is catastrophically slow. Teams that skip the physical device harness ship with 3–4x worse latency than they measured in CI.
*(Side note: real-device benchmarking means long stretches at your workstation. I run [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) in the background for break reminders and guided desk exercises — worth it on a heavy testing day.)*
---
## Gotchas
- **`Dispatchers.IO` is a pool, not a single thread.** Your GPU delegate will crash unpredictably under load. Use `newSingleThreadContext` or a `HandlerThread`.
- **NNAPI on API 27 can produce negative acceleration.** Partial graph execution overhead can exceed your CPU baseline. Gate at API 28, not API 27.
- **CPU fallback is thread-safe by default.** The single-thread pinning requirement is GPU-only — do not over-apply it to your CPU path.
- **Always log the active delegate.** `NnApiDelegate` initializes without throwing even when it accelerates nothing. You will not know you have silently fallen to CPU unless you instrument it.
- **The docs do not mention this, but** `isPrecisionLossAllowed = true` on the GPU delegate enables fp16 and typically yields around 1.4x faster inference on Mali and Adreno hardware. Leave it off only if your model is sensitive to precision loss.
---
## Conclusion
Delegate selection is a runtime contract, not a config flag. Pin your GPU inference to a dedicated thread. Gate NNAPI at API 28+. Log the active delegate on every cold start. Benchmark exclusively on physical hardware with warm-up runs — the emulator numbers are fiction that will cost you in production.
**Resources:**
- [TFLite GPU delegate docs](https://www.tensorflow.org/lite/performance/gpu)
- [NNAPI delegate docs](https://www.tensorflow.org/lite/android/delegates/nnapi)
- [TFLite Benchmark Tool](https://www.tensorflow.org/lite/performance/measurement)
Top comments (0)