---
title: "Two-Tier On-Device ML for Android: Routing Between ML Kit and TFLite INT8"
published: true
description: "Build a confidence-gated inference pipeline on Android — ML Kit as the fast path, a custom TFLite INT8 model as fallback — enforced within a strict per-frame CameraX budget."
tags: android, mobile, architecture, kotlin
canonical_url: https://mvpfactory.co/blog/tflite-int8-mlkit-android-inference-pipeline
---
## What we are building
By the end of this tutorial you will have a working two-tier on-device inference pipeline: ML Kit handles the common case fast, a quantized TFLite INT8 model catches the edge cases your primary model misses, and a token-bucket frame gate keeps you inside a 33 ms per-frame budget on mid-range hardware.
This is the pattern I use in every project where on-device ML needs to be production-reliable, not just demo-reliable.
## Prerequisites
- Android project targeting API 26+
- CameraX `1.3.x` on the classpath
- A TFLite INT8 model (post-training quantization via TensorFlow's converter)
- ML Kit Vision dependency
---
## Step 1 — Design the routing layer first
Keep the router dead simple. One threshold comparison per tier.
CameraX Frame → ML Kit Inference
│
confidence ≥ 0.85? ──YES──▶ emit result
│
NO
│
TFLite INT8 Model
│
confidence ≥ 0.60? ──YES──▶ emit result
│
NO
▶ drop frame / fallback label
The complexity belongs in model training and delegate selection — not here.
## Step 2 — Select the right delegate per device tier
INT8 quantized models are a prerequisite for NNAPI acceleration. Most NNAPI implementations reject FP32 graphs on mid-range silicon. Here is how the tiers map out:
| Delegate | Best For | INT8 Support | Notes |
|---|---|---|---|
| NNAPI | Snapdragon 6xx, Dimensity 7xx | Strong post-Android 10 | Use `allowFp16PrecisionForFp32=true` |
| GPU | Snapdragon 8xx, Tensor G3+ | Partial pre-Android 12 | OEM driver consistency varies |
| CPU (XNNPACK) | All devices, fallback | Full | Most predictable, highest latency |
The docs do not mention this, but NNAPI performance swings wildly across OEMs. Probe at startup, time a warm-up inference, cache the winner in `SharedPreferences`:
kotlin
fun selectDelegate(context: Context): List {
return try {
val nnapi = NnApiDelegate()
listOf(nnapi)
} catch (e: Exception) {
emptyList() // falls back to XNNPACK via default interpreter
}
}
Do this once on first launch and revisit whenever you ship a model architecture change.
## Step 3 — Enforce the per-frame budget at the camera layer
At 30 fps you have 33 ms per frame. `STRATEGY_KEEP_ONLY_LATEST` alone is not enough — it buffers one frame, not zero. Pair it with a token-bucket gate:
kotlin
val analysis = ImageAnalysis.Builder()
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setTargetRotation(Surface.ROTATION_0)
.build()
analysis.setAnalyzer(inferenceExecutor) { imageProxy ->
if (frameGate.tryAcquire()) { // token bucket, refills every 33ms
runInferencePipeline(imageProxy)
} else {
imageProxy.close() // explicit drop
}
}
On a Dimensity 700-class device, this combination holds 28–30 fps with the two-tier pipeline active versus 18–22 fps without frame gating.
## Step 4 — Calibrate your confidence thresholds
Do not pick 0.85 and 0.60 out of thin air. Plot precision-recall curves on a held-out validation set. The ML Kit threshold should sit at the knee of your precision curve — near-zero false positives, because the TFLite fallback owns the uncertain cases. For the INT8 fallback, a lower threshold is acceptable since that model is specifically trained on your hard negatives.
When both models return low confidence, emit a "no result" signal. Surface it to the UI. Do not guess.
---
## Gotchas
**NNAPI is not a free upgrade.** A warm-up probe can still return a delegate that silently degrades on specific chipsets. Always benchmark on Snapdragon 680 or Dimensity 700 hardware — not a flagship emulator. I run [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) on mid-range test devices specifically to validate background inference stays within battery and CPU envelopes real users will accept.
**`STRATEGY_KEEP_ONLY_LATEST` buffers, it does not drop.** You still need the token-bucket gate on the analyzer side. This is the gotcha that will save you hours of profiling confusion.
**GPU delegate INT8 support is fragmented pre-Android 12.** If you need broad device coverage, NNAPI-or-XNNPACK is your reliable path, not GPU.
**Thresholds are not universal.** The 0.85 / 0.60 split is a starting point. Plot your curves and choose thresholds that match the cost asymmetry of your specific false-positive vs false-negative trade-off.
---
## Conclusion
ML Kit is a great first-pass filter, not a terminal decision. Wire a TFLite INT8 model behind a confidence gate, enforce your frame budget at the `ImageAnalysis` layer with a token-bucket dropper, and probe NNAPI at startup rather than assuming it works. This architecture keeps inference fast for the common case, handles edge cases entirely on-device, and gives you the control that production systems actually need.
**Resources:**
- [TensorFlow Lite post-training quantization guide](https://www.tensorflow.org/lite/performance/post_training_quantization)
- [CameraX ImageAnalysis docs](https://developer.android.com/training/camerax/analyze)
- [NnApiDelegate reference](https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/nnapi/NnApiDelegate)
Top comments (0)