---
title: "Android GPU Inference for Real-Time Semantic Search: NNAPI, Vulkan, and HNSW on Pixel 9"
published: true
description: "Wire Android GPU compute to INT8 sentence-transformers with NNAPI, Vulkan delegate, and HNSW indexing for sub-40ms semantic search — no cloud round-trip required."
tags: android, kotlin, architecture, mobile
canonical_url: https://mvpfactory.co/blog/android-gpu-inference-semantic-search
---
## What We Are Building
By the end of this tutorial you will have a fully on-device semantic search pipeline that runs in under 40ms on a Pixel 9. We will quantize a sentence-transformer to INT8, wire NNAPI and Vulkan compute delegates with runtime routing, build an in-process HNSW index using hnswlib-kotlin, and profile the full token→embedding→nearest-neighbor path.
No cloud round-trip. No 200–400ms server tax.
**Prerequisites:** Android Studio Hedgehog or later, a Python environment with TensorFlow 2.x for the quantization step, and a physical Android device for profiling (emulators will not exercise NNAPI or Vulkan paths meaningfully).
---
## Step 1: Quantize the Model to INT8
Start with `all-MiniLM-L6-v2` — 384-dimensional output, compact enough to ship on-device. Export to TFLite with full INT8 quantization using a representative calibration dataset:
python
def representative_dataset():
for sample in calibration_samples:
yield [np.array(sample, dtype=np.float32)]
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
tflite_model = converter.convert()
This is the pattern I use in every on-device inference project. The calibration dataset is non-negotiable for full INT8 — without it, TFLite falls back to dynamic-range quantization and your delegate acceleration degrades. Model size drops from ~90MB (FP32) to ~23MB, and the arithmetic shifts to integer ops that NNAPI and Vulkan compute shaders are built to accelerate.
---
## Step 2: Route Delegates at Runtime
Most teams pick one delegate and ship it. Here is the gotcha that will save you hours: hard-coding NNAPI works fine on Pixel and breaks silently on devices without an NPU. Detect and route explicitly.
| Delegate | Best on | Latency (Pixel 9) |
|---|---|---|
| NNAPI | Devices with DSP/NPU | ~8–14ms |
| Vulkan compute | GPU-capable devices | ~10–18ms |
| CPU (XNNPACK) | Fallback | ~35–60ms |
Define capability checks explicitly:
kotlin
fun isNnapiSupported(): Boolean =
Build.VERSION.SDK_INT >= Build.VERSION_CODES.P
fun isVulkanSupported(context: Context): Boolean =
CompatibilityList().isDelegateSupportedOnThisDevice(GpuDelegateFactory.Options())
Then route dynamically — use explicit variable assignment to avoid delegate chaining ambiguity:
kotlin
fun buildInterpreter(context: Context): Interpreter {
val options = Interpreter.Options()
return when {
isNnapiSupported() -> {
val delegate = NnApiDelegate()
options.addDelegate(delegate)
Interpreter(model, options)
}
isVulkanSupported(context) -> {
val gpuOptions = GpuDelegate.Options().apply {
inferencePreference = GpuDelegate.Options.INFERENCE_PREFERENCE_FAST_SINGLE_ANSWER
}
val delegate = GpuDelegate(gpuOptions)
options.addDelegate(delegate)
Interpreter(model, options)
}
else -> {
options.setNumThreads(4)
Interpreter(model, options)
}
}
}
On Pixel 9, NNAPI with the Tensor Processing Unit lands at ~8ms median for a 128-token input. Vulkan runs ~12ms but with lower variance — important when you are optimizing for P99.
---
## Step 3: Size Your HNSW Index
Here is the minimal setup to get this working. Add the dependency first:
kotlin
// build.gradle.kts
implementation("io.github.jelmerk:hnswlib-kotlin:1.1.0")
Then build and tune the index:
kotlin
val index = HnswIndex.newBuilder(384, DistanceFunctions.FLOAT_COSINE_DISTANCE, 100_000)
.withM(16)
.withEfConstruction(200)
.withEf(64)
.build()
The docs do not make this obvious, but `ef` is a query-time parameter you should tune against your end-to-end P99 budget, not a micro-benchmark. `ef=64` delivers ~97% recall at ~2–4ms query latency. Dropping to `ef=32` halves that to ~1ms at the cost of ~3% recall — acceptable for most retrieval-augmented pipelines.
On memory: 100k vectors at 384 dims in FP32 runs ~147MB for raw vectors plus ~60–80MB HNSW graph overhead. Keep embeddings in INT8 (drops to ~37MB) and cap your index at 50k entries to stay comfortably under 100MB total — important on devices where an edge LLM may hold 1–2GB of KV cache.
---
## Gotchas
**Silent NNAPI fallback.** If the delegate is unsupported, TFLite falls back to CPU without throwing. Instrument each path separately in production telemetry — otherwise your P99 charts will mislead you.
**Calibration dataset quality.** A poor representative dataset produces a quantized model with measurable accuracy regression. Use real samples from your domain, not random noise.
**Tokenization overhead is real.** It is easy to benchmark only the HNSW query or only the embedding call. Profile the full pipeline. Tokenization and marshaling add ~2ms at P50 and contribute meaningfully to P99.
**Delegate chaining.** Do not add multiple delegates to the same `Interpreter.Options` instance without testing. Some combinations produce no error and degrade silently.
---
## End-to-End Latency on Pixel 9
Profiling on Pixel 9 (Tensor G4, 8GB RAM, Android 15):
| Stage | P50 | P99 |
|---|---|---|
| Tokenization | 1ms | 2ms |
| Embedding (NNAPI INT8) | 9ms | 14ms |
| HNSW query (ef=64) | 3ms | 6ms |
| Result marshal | <1ms | 1ms |
| **Total** | **~14ms** | **~23ms** |
Sub-40ms is achievable and comfortable. Even on mid-range devices without a dedicated NPU, the Vulkan path plus XNNPACK fallback keeps P99 under 60ms.
**Resources:** [TFLite quantization guide](https://www.tensorflow.org/lite/performance/post_training_integer_quant) · [hnswlib-kotlin](https://github.com/jelmerk/hnswlib-kotlin) · [TFLite GPU delegate docs](https://www.tensorflow.org/lite/performance/gpu)
The latency ceiling here is a system property — measure the whole pipe, set `ef` based on your end-to-end budget, and route delegates at runtime. That combination is what gets you to keystroke-level semantic search without leaving the device.
Top comments (0)