Wiring Android's CameraX Frame Analysis to a Quantized LLM Vision Encoder

---
title: "Wiring CameraX to a Quantized TFLite Vision Encoder Under 40ms"
published: true
description: "Wire CameraX ImageAnalysis to a quantized TFLite CLIP encoder. Covers YUV conversion without allocation spikes, executor isolation, and frame-drop strategies with Pixel 8 benchmarks."
tags: kotlin, android, mobile, architecture
canonical_url: https://mvpfactory.co/blog/camerax-tflite-vision-encoder-under-40ms
---

What We Are Building

By the end of this tutorial you will have a real-time on-device vision pipeline that connects CameraX's ImageAnalysis use case to a quantized CLIP-style TFLite encoder — running end-to-end under 40ms on a Pixel 8. We will cover YUV-to-RGB conversion without allocation spikes, a two-executor threading strategy that keeps the UI thread free, and the drop-not-queue frame strategy that prevents compounding latency when the model cannot keep up.

Prerequisites

• Android project targeting API 24+
• CameraX 1.3.x added to your build.gradle
• A quantized INT8 TFLite model (ViT-B/32 CLIP-style encoder works for the benchmarks below)
• Android Studio's allocation profiler open alongside your emulator or device

---

The Problem Most Teams Hit

The usual mistake with real-time on-device vision: treat the camera pipeline and the ML pipeline as independent concerns and bolt them together with a shared thread pool. The result is latency jitter, dropped UI frames, and an inference queue that grows unbounded under load.

The architecture has to be deliberate from the start.

---

Pipeline Architecture

Here is the minimal setup to get this working. Five stages, each with a clear contract:

CameraX ImageAnalysis
        │
        ▼
  YUV→RGB Converter  (pre-allocated ByteBuffer)
        │
        ▼
  TFLite Inference   (single-threaded dedicated executor)
        │
        ▼
  Result Channel     (conflated — drops stale frames)
        │
        ▼
  UI / Compose State

The key constraint: CameraX at 30fps gives you ~33ms per frame. An INT8-quantized ViT-B/32 via TFLite runs ~28–35ms on the Pixel 8's NPU delegate, and 55–80ms on a mid-range Snapdragon 6-series without NPU acceleration. You cannot process every frame. You should not try.

---

Step 1 — Fix YUV-to-RGB Without Allocation Spikes

ImageProxy delivers frames in YUV_420_888. The naive path — converting via Bitmap.createBitmap() on every frame — allocates approximately 3MB of YUV input and produces a ~6MB RGB Bitmap per frame at 1080p. At 30fps that is upwards of 270MB/s of GC pressure. You will see it immediately in the allocation profiler as sawtooth spikes.

Let me show you a pattern I use in every project — pre-allocate once, reuse forever:

class YuvToRgbConverter(private val inputSize: Int) {
    private val rgbBuffer = ByteBuffer
        .allocateDirect(inputSize * inputSize * 3)
        .also { it.order(ByteOrder.nativeOrder()) }

    // reusableBitmap is allocated once at construction and recycled on every frame
    val reusableBitmap: Bitmap = Bitmap.createBitmap(inputSize, inputSize, Bitmap.Config.ARGB_8888)

    fun convert(image: ImageProxy) {
        rgbBuffer.rewind()
        val yPlane = image.planes[0].buffer
        val uPlane = image.planes[1].buffer
        val vPlane = image.planes[2].buffer
        // nativeYuvToRgb() is a JNI convenience; teams avoiding JNI can substitute
        // CameraX's androidx.camera.core.internal.utils.ImageUtil or a RenderScript
        // YuvToRgb kernel — the pre-allocation pattern remains identical.
        nativeYuvToRgb(yPlane, uPlane, vPlane, rgbBuffer, inputSize)
        reusableBitmap.copyPixelsFromBuffer(rgbBuffer.also { it.rewind() })
    }
}

Reuse both the ByteBuffer and the target Bitmap. Allocate once at startup, reuse on every frame. Steady-state allocation drops to near zero after warmup.

---

Step 2 — Two Executors, No Sharing

The docs do not mention this, but a shared thread pool between camera and inference is the most reliable way to introduce jitter. Two dedicated single-thread executors, hard boundary between them:

val cameraExecutor = Executors.newSingleThreadExecutor()
val inferenceExecutor = Executors.newSingleThreadExecutor()

The ImageAnalysis use case runs its analyze() callback on cameraExecutor. Inside that callback, gate on inference availability using an AtomicBoolean and submit to inferenceExecutor only if the previous inference has completed. The camera thread never blocks:

private val inferenceRunning = AtomicBoolean(false)
private val converter = YuvToRgbConverter(inputSize = 224)

override fun analyze(image: ImageProxy) {
    if (!inferenceRunning.compareAndSet(false, true)) {
        image.close() // drop frame — do not queue
        return
    }
    converter.convert(image)
    image.close()

    inferenceExecutor.submit {
        try {
            val result = tfliteModel.run(converter.reusableBitmap)
            _sceneState.value = result
        } finally {
            inferenceRunning.set(false)
        }
    }
}

Drop, don't queue. A LinkedBlockingQueue as the executor's backing store will build up frames and introduce compounding latency. A conflated StateFlow on the result side ensures the UI always sees the latest embedding, never a stale queued one.

---

Latency Breakdown: Pixel 8 vs. Mid-Range

Stage	Pixel 8 (NPU delegate)	Snapdragon 6s (CPU)
YUV → RGB conversion	~1.5ms	~3ms
TFLite inference (INT8 ViT-B/32)	~28ms	~68ms
StateFlow emission + Compose recompose	~1ms	~1.5ms
Total end-to-end	~30.5ms	~72.5ms

The Pixel 8 lands comfortably under 40ms with the NNAPI/NPU delegate enabled. The mid-range device at ~72.5ms processes roughly 1 in every 2.2 frames (~14fps). That is still workable for scene understanding — the drop-not-queue architecture keeps it predictable even when throughput is lower.

---

Gotchas

Forgetting to call image.close() — If you drop a frame via the AtomicBoolean gate but forget image.close(), CameraX stalls the analysis pipeline entirely. Always close before returning.

Sharing the reusable Bitmap across threads — The converter writes to reusableBitmap on cameraExecutor, and inference reads it on inferenceExecutor. The AtomicBoolean gate serializes this handoff. Remove the gate and you have a data race.

Skipping the NPU delegate — NPU acceleration is the difference between under-40ms and over-60ms. Ship two profiling builds — one with and one without the delegate — and gate on device capability at runtime. The CPU fallback exists for correctness, not performance.

Profiling on emulator only — The allocation profiler sawtooth pattern only shows up clearly on physical devices under real camera load. Test on hardware before drawing conclusions about GC pressure.

---

Conclusion

Three things make this pipeline work:

1. Pre-allocate your YUV buffers and Bitmap at startup. A reused ByteBuffer and Bitmap pair eliminates the ~3MB YUV input allocation and the ~6MB RGB Bitmap allocation per frame. Measure with Android Studio's allocation profiler before shipping.

2. Two dedicated single-thread executors, never a shared pool. Camera callbacks and inference must be isolated. The AtomicBoolean gate on the camera thread is the simplest correct mechanism for frame dropping without queue buildup.

3. Enable the NNAPI delegate with CPU fallback and benchmark on your actual device tier.

The architecture is deliberate from the start — or it costs you in ways that are difficult to profile later. Get the pipeline right before optimizing the model.