Wiring Android's CameraX to a Quantized Segmentation Model for Real-Time Background Replacement

---
title: "CameraX + DeepLab v3+: Real-Time Segmentation on Android"
published: true
description: "Wire CameraX ImageAnalysis to a quantized DeepLab v3+ model via TFLite GPU delegate. Learn tensor layout, stride bitmaps, and memory trade-offs for 30fps on-device segmentation."
tags: android, kotlin, mobile, performance
canonical_url: https://mvpfactory.co/blog/camerax-deeplab-realtime-segmentation-android
---

## What We Are Building

By the end of this tutorial you will have a production-grade on-device background replacement pipeline running at 30fps on Android. We will wire CameraX `ImageAnalysis` to a quantized DeepLab v3+ model through the TFLite GPU delegate — covering tensor input layout, stride-aligned bitmap conversion, per-frame latency budgeting, and the memory constraints that force you to choose the right model backbone before you write a single line of inference code.

Here is the gotcha that will save you hours: this is not a model problem. It is a data plumbing problem.

## Prerequisites

- Android Studio Hedgehog or later
- A device running API 26+ (GPU delegate requires API 28+ for full coverage)
- TFLite GPU delegate AAR in your Gradle deps
- The MobileNetV2-backbone INT8 quantized DeepLab v3+ `.tflite` model ([download from TF Hub](https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/2))
- CameraX `1.3.x` dependencies

## Step 1 — Get the Tensor Layout Right First

DeepLab v3+ expects input in **NHWC format** — `[1, H, W, 3]`. The default resolution is 513×513; the 257×257 variant trades accuracy for latency on tighter hardware.

The docs do not mention this clearly, but you must match your `ByteBuffer` type to the model's quantization scheme. The `uint8` path uses `[0, 255]`; the `int8` path uses `[-128, 127]`. The GPU delegate handles scale and zero-point remapping internally, but a mismatch produces silently wrong masks — no crash, just bad output.

kotlin
// Pre-allocate once at startup — never allocate per-frame
val inputBuffer = ByteBuffer.allocateDirect(1 * 513 * 513 * 3)
.order(ByteOrder.nativeOrder())

A single `allocateDirect` at startup with a manual `rewind()` per frame is the difference between 28ms and 45ms average inference on a Pixel 6.

## Step 2 — Fix the Stride-Aligned Bitmap Conversion

CameraX delivers `YUV_420_888` frames via `ImageProxy`. Here is the trap: the Y plane row stride often does **not** equal the image width. Ignore this and your input tensor is sheared and corrupted — every row is offset by invisible padding bytes.

Let me show you a pattern I use in every project:

kotlin
fun ImageProxy.toStrideCorrectedBitmap(): Bitmap {
val yPlane = planes[0]
val yBuffer = yPlane.buffer
val rowStride = yPlane.rowStride
val width = width
val height = height

val yData = ByteArray(height * width)
for (row in 0 until height) {
    yBuffer.position(row * rowStride)
    yBuffer.get(yData, row * width, width)
}

val pixels = IntArray(width * height) { i ->
    val y = yData[i].toInt() and 0xFF
    0xFF000000.toInt() or (y shl 16) or (y shl 8) or y
}
val bitmap = Bitmap.createBitmap(pixels, width, height, Bitmap.Config.ARGB_8888)
return Bitmap.createScaledBitmap(bitmap, 513, 513, false)

}

Always read `rowStride` from the plane object, not from `image.width`. This loop discards stride padding and copies only valid pixel columns per row. On flagship devices stride happens to equal width — which is exactly why this bug survives code review.

## Step 3 — Configure the GPU Delegate

kotlin
val gpuDelegate = GpuDelegate(
GpuDelegate.Options().apply {
inferencePreference = INFERENCE_PREFERENCE_SUSTAINED_SPEED
precisionLossAllowed = true // enables FP16 path on Adreno and Mali
}
)

val interpreter = Interpreter(
loadModelFile(context, "deeplab_v3_plus_int8.tflite"),
Interpreter.Options().addDelegate(gpuDelegate)
)

`precisionLossAllowed = true` unlocks the FP16 execution path on Adreno and Mali GPUs. Combined with INT8 quantization this is your primary lever for staying under 33ms on mid-range silicon.

## Step 4 — Know Your Memory Budget

Here is the minimal setup to get this working within Android's ~400MB ceiling on a typical mid-range device:

| Component | Approx. Memory |
|---|---|
| INT8 MobileNetV2 model | ~2.5–3 MB |
| GPU delegate tensor buffers | ~30–60 MB |
| CameraX preview surface | ~25–40 MB |
| Output mask + compositing | ~15–20 MB |
| Framework overhead | ~80–120 MB |

The Xception backbone at >200MB is a non-starter. The MobileNetV2 INT8 variant is the production-correct choice — comfortable on 4GB devices with headroom for the rest of your app stack.

At 30fps you have **33ms** per frame. On a mid-range Snapdragon 778G, expect:

- YUV→NHWC conversion: ~4–6ms
- GPU delegate inference (INT8, 513×513): ~18–24ms
- Mask post-processing + compositing: ~4–6ms

That leaves 1–5ms of slack. If your compositing uses `Canvas.drawBitmap` in software mode, you have already lost that slack. Use `RenderEffect` (API 31+) or a GLSL shader via `SurfaceTexture` to keep compositing on the GPU.

## Gotchas

**Silent accuracy regression from stride.** Stride bugs do not crash — they just produce progressively wrong masks. Always verify `rowStride` handling before debugging anything else in the pipeline.

**Wrong quantization type, wrong buffer.** `uint8` models need unsigned byte buffers. `int8` models need signed. The GPU delegate will not throw; it will infer garbage.

**Benchmarking on the wrong device.** Benchmark on a Pixel 6a or Galaxy A54, not a Pixel 9 Pro. Your install base is mid-range hardware.

**CPU compositing kills throughput.** Any buffer copy back to CPU for compositing serializes the pipeline and forfeits the delegate's latency advantage. Invest in GLES compositing early — retrofitting it is expensive.

## Conclusion

The pipeline works when all three components — CameraX, stride-correct conversion, and the GPU delegate — operate in lockstep. Fix stride first, default to INT8 quantization, and keep all tensor operations GPU-side end to end. That combination is what holds 30fps within a realistic memory budget on the devices that actually matter.

For deeper reading: [TFLite GPU delegate docs](https://www.tensorflow.org/lite/performance/gpu) and the [CameraX ImageAnalysis guide](https://developer.android.com/training/camerax/analyze).