---
title: "Wiring CameraX to a Quantized Hand Gesture Classifier for Real-Time Sign Language Recognition Under 30ms"
published: true
description: "Wire CameraX to MediaPipe Hands and a quantized TFLite INT8 classifier with GPU delegate. Full pipeline, memory layout, and EMA smoothing for sub-30ms latency on mid-range Android."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/android-camerax-mediapipe-tflite-sign-language-30ms
---
## What We Are Building
By the end of this tutorial, you will have a working Android pipeline: CameraX frame delivery → MediaPipe Hands landmark extraction → quantized TFLite gesture classifier with GPU delegate acceleration — end-to-end under 30ms on mid-range Snapdragon hardware.
Let me show you a pattern I use in every real-time vision project: treat the pipeline as a single latency budget, not a collection of independent stages. Most teams measure each stage at 15ms, then wonder why end-to-end is 80ms. The overhead lives in the handoffs.
## Prerequisites
- Android Studio Giraffe or later
- A physical device, Snapdragon 7xx class or equivalent, for realistic timing numbers
- MediaPipe Hands dependency in your Gradle build
- A quantized INT8 TFLite model for your gesture vocabulary (26-class ASL in our benchmarks)
## The Pipeline at a Glance
CameraX ImageAnalysis
└─► ImageProxy (YUV_420_888) → Bitmap conversion
└─► MediaPipe Hands (CPU, landmark extraction)
└─► 21 × (x, y, z) keypoints → FloatArray
└─► Temporal smoother (EMA, α=0.6)
└─► TFLite INT8 classifier (GPU delegate)
└─► Gesture label + confidence
Total budget: 30ms. Here is how each stage earns its slice.
---
## Step 1 — CameraX with the Right Backpressure Strategy
kotlin
val analysisUseCase = ImageAnalysis.Builder()
.setTargetResolution(Size(640, 480))
.setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_YUV_420_888)
.build()
`STRATEGY_KEEP_ONLY_LATEST` is non-negotiable. With `STRATEGY_BLOCK_PRODUCER`, a slow inference frame stalls the camera queue and you end up processing stale frames — the worst possible outcome for gesture recognition. Drop frames aggressively; temporal smoothing in Step 3 handles the gaps.
## Step 2 — MediaPipe Hands Landmark Extraction
MediaPipe Hands produces 21 3D landmarks per hand. On a Pixel 6a (Tensor G2) expect ~12ms; on a Snapdragon 778G, ~16–18ms. This is your most expensive stage and it runs on CPU.
The docs do not mention this, but avoid re-encoding to JPEG before passing frames to MediaPipe. That single mistake adds 8–12ms.
kotlin
val frameMetadata = FrameMetadata.Builder()
.setWidth(bitmap.width)
.setHeight(bitmap.height)
.setRotation(rotationDegrees)
.build()
hands.send(bitmap, frameMetadata, SystemClock.uptimeMillis())
## Step 3 — Temporal Smoothing: Do Not Skip This
Raw landmark sequences are noisy. A single dropped or occluded frame produces a landmark spike that maps to the wrong gesture. An exponential moving average over the sequence costs ~0.1ms and eliminates most false positives.
kotlin
class LandmarkSmoother(private val alpha: Float = 0.6f) {
private var prev: FloatArray? = null
fun smooth(current: FloatArray): FloatArray {
val p = prev ?: current.copyOf()
val smoothed = FloatArray(current.size) { i -> alpha * current[i] + (1 - alpha) * p[i] }
prev = smoothed
return smoothed
}
}
`α = 0.6` balances responsiveness and stability. Lower values (0.3–0.4) suit slower, deliberate signs; higher values (0.8+) suit fast fingerspelling.
## Step 4 — TFLite INT8 Classifier with GPU Delegate
The classifier is a 3-layer MLP: 63 float inputs (21 landmarks × x, y, z) → Dense(128, ReLU) → Dense(64, ReLU) → Dense(num_classes, Softmax). Intentionally shallow — deeper models add latency without meaningful accuracy gains on a 63-feature input, and the flat landmark vector gives you no spatial hierarchy to exploit with convolutions.
| Config | Inference (Snapdragon 778G) | Top-1 Accuracy (ASL 26-class) |
|---|---|---|
| FP32, CPU | 11.2ms | 97.4% |
| INT8, CPU | 6.8ms | 95.1% |
| **INT8, GPU delegate** | **3.1ms** | **95.1%** |
| INT8, NNAPI | 4.4ms | 94.8% |
GPU delegate on INT8 wins. Skip NNAPI — it introduces driver inconsistency across OEMs, and I have seen 2x variance on the same chipset across firmware versions.
Always wrap GPU delegate initialization. It fails silently on roughly 10% of devices due to broken OEM drivers:
kotlin
val options = Interpreter.Options()
try {
options.addDelegate(GpuDelegate())
} catch (e: Exception) {
// Fall back to INT8 CPU — still 6.8ms, well within budget
}
val interpreter = Interpreter(modelBuffer, options)
---
## End-to-End Latency Budget
| Stage | Mid-range (778G) |
|---|---|
| CameraX frame delivery | ~2ms |
| YUV → Bitmap | ~3ms |
| MediaPipe Hands | ~17ms |
| EMA smoothing | ~0.1ms |
| TFLite INT8 + GPU | ~3.1ms |
| **Total** | **~25ms** |
That leaves ~5ms of headroom before the 30ms budget — enough to absorb GC pauses without dropping user-visible frames. On the CPU fallback path, total latency rises to ~29ms, which still clears the target.
---
## Gotchas
Here is the gotcha that will save you hours: **memory layout has no safety net**. TFLite's GPU delegate requires the input `ByteBuffer` to be direct-allocated with floats interleaved in `[landmark_index][x, y, z]` order. A heap-allocated buffer or wrong stride produces wrong predictions with no exception thrown — just quietly bad results.
kotlin
val inputBuffer = ByteBuffer.allocateDirect(63 * 4).order(ByteOrder.nativeOrder())
landmarks.forEach { lm ->
inputBuffer.putFloat(lm.x)
inputBuffer.putFloat(lm.y)
inputBuffer.putFloat(lm.z)
}
A few more to watch for:
- **Post-classification label smoothing causes boundary jitter.** Smooth the landmark sequence before the classifier, not the label after it.
- **JPEG re-encoding is a hidden tax.** Pass the raw `Bitmap` directly to MediaPipe, always.
- **`STRATEGY_BLOCK_PRODUCER` is the default trap.** Set `STRATEGY_KEEP_ONLY_LATEST` unconditionally for any real-time vision `ImageAnalysis` pipeline.
---
## Conclusion
Here is the minimal setup to get this working: `STRATEGY_KEEP_ONLY_LATEST` on CameraX, raw Bitmap to MediaPipe, EMA smoothing at α=0.6, INT8 quantization with GPU delegate, and a direct-allocated `ByteBuffer` with the right memory layout. That combination consistently lands around 25ms on Snapdragon 7xx hardware — 5ms under budget.
The INT8 quantization costs ~2–3% accuracy on hand keypoints and recovers ~40% of your inference budget. That tradeoff is almost always worth taking.
Side note: sessions spent profiling tight loops like this are long ones. I keep [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) running during deep ML work — the guided desk-exercise breaks are easy to dismiss but genuinely help when you are chasing single-digit milliseconds for hours at a stretch.
**Further reading:**
- [MediaPipe Hand Landmarker](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker)
- [TFLite GPU delegate guide](https://www.tensorflow.org/lite/performance/gpu)
- [CameraX ImageAnalysis reference](https://developer.android.com/training/camerax/analyze)
Top comments (0)