SoftwareDevs mvpfactory.io

Posted on Sep 22 Originally published at mvpfactory.io

Wiring Android's CameraX to a Quantized Monocular Depth Model for Real-Time Scene Reconstruction

#webdev #programming

---
title: "CameraX + MiDaS v3 Small: Real-Time Depth at 30fps on Android"
published: true
description: "Wire CameraX ImageAnalysis to a quantized MiDaS v3 Small depth model via TFLite GPU delegate. Covers YUV conversion, depth normalization, ring buffer point clouds, and staying under 28ms on mid-range hardware."
tags: [android, kotlin, mobile, architecture]
canonical_url: https://blog.mvpfactory.co/camerax-midas-depth-30fps-android
---

What We Are Building

By the end of this tutorial you will have a working Android pipeline that feeds CameraX frames into a quantized MiDaS v3 Small depth model at 30fps on mid-range hardware — without triggering the OOM killer mid-session. The pipeline looks like this:

CameraX ImageAnalysis → YUV→RGB → Resize/Normalize → TFLite GPU (MiDaS v3 Small) → Depth Map → Ring Buffer

Here is the gotcha that will save you hours: a GPU delegate that silently falls back to CPU will cost you 40ms per frame and show up only as a 12fps complaint in a Play Store review. This post is about preventing exactly that — and the two other pipeline killers standing between you and sustained 30fps depth estimation.

Prerequisites

Android project targeting API 21+
CameraX 1.3.x added to build.gradle
tensorflow-lite-gpu and tensorflow-lite dependencies
MiDaS v3 Small .tflite model (quantized, 256×256 input) in assets/

Frame Budget Reality Check

Before writing a single line, know your budget on a Snapdragon 7-series class device:

Stage	CPU Path	GPU Path	Target Budget
YUV→RGB conversion	6–10ms	2–4ms	≤4ms
TFLite inference	30–50ms	8–15ms	≤18ms
Depth normalization	<1ms	<1ms	≤1ms
Point cloud update	1–3ms	1–2ms	≤3ms
Total	~40–67ms	~13–24ms	≤28ms

GPU inference is not optional. CPU alone blows the 33ms budget before normalization even starts.

Step 1: Pre-Allocate Your YUV Buffer

Let me show you a pattern I use in every project. CameraX delivers YUV_420_888 frames. The trap most teams fall into is calling ImageProxy.toBitmap() inside the analyzer callback — that allocates a new Bitmap on every frame and you are looking at an OOM event within minutes.

Allocate once at construction, reuse across every frame:

// Allocate ONCE at construction — never inside the callback
val rgbBuffer: ByteBuffer = ByteBuffer.allocateDirect(MODEL_WIDTH * MODEL_HEIGHT * 3)

analyzer.setAnalyzer(executor) { imageProxy ->
    convertYuvToRgb(imageProxy, rgbBuffer) // writes into pre-allocated buffer
    imageProxy.close()                     // critical — CameraX stalls if omitted
    runInference(rgbBuffer)
}

Step 2: Wire the TFLite GPU Delegate Correctly

val gpuDelegate = GpuDelegate(GpuDelegate.Options().apply {
    isPrecisionLossAllowed = true  // enables FP16; 30–40% latency reduction on Adreno/Mali
})
val options = Interpreter.Options().addDelegate(gpuDelegate)
val interpreter = Interpreter(loadModelFile(context), options)

Pre-allocate the input tensor buffer with explicit native byte order — this is the exact constraint that determines whether the GPU delegate actually runs:

val inputBuffer = ByteBuffer
    .allocateDirect(1 * MODEL_HEIGHT * MODEL_WIDTH * 3 * Float.SIZE_BYTES)
    .order(ByteOrder.nativeOrder()) // 4-byte alignment guarantee for GPU delegate

If your ByteBuffer is not direct-allocated and natively ordered, the delegate silently falls back to CPU. No exception. Just 4× slower inference.

Step 3: Normalize Your Depth Output Per-Frame

MiDaS v3 outputs inverse relative depth — higher values mean closer geometry. Normalize per-frame to [0, 1]:

val min = outputArray.min()
val max = outputArray.max()
val range = (max - min).coerceAtLeast(1e-5f)  // avoid division by zero in static scenes
outputArray.forEachIndexed { i, v ->
    normalizedDepth[i] = (v - min) / range
}

Do not apply a global running normalization across frames unless the scene is stationary. Temporal drift from a global normalizer introduces flickering that makes ring-buffer accumulation incoherent.

Step 4: Cap Your Ring Buffer

A 256×256 depth map at 30fps generates roughly 2 million depth values per second. Here is the minimal setup to get this working without exhausting heap memory:

class DepthRingBuffer(private val capacity: Int) {
    private val frames = ArrayDeque<FloatArray>(capacity)

    fun push(depthMap: FloatArray) {
        if (frames.size >= capacity) frames.removeFirst()
        frames.addLast(depthMap)
    }

    fun snapshot(): List<FloatArray> = frames.toList()
}

Downsample depth maps to 64×64 before accumulation. The difference is stark:

Resolution	60-frame buffer
256×256 (full)	~15MB
64×64 (downsampled)	~960KB

That is a 16× reduction. On a mid-range device with a ~300MB app memory budget, that gap determines whether your app survives a five-minute session.

Gotchas

Silent GPU fallback. The docs do not mention this, but Tensor.device() is not stable across TFLite versions — do not gate production logic on it. Use a latency probe at startup instead:

val warmupMs = measureTimeMillis { repeat(2) { interpreter.run(inputBuffer, outputBuffer) } } / 2
if (warmupMs > GPU_LATENCY_THRESHOLD_MS) {
    Log.w(TAG, "TFLite GPU delegate may have fallen back to CPU (${warmupMs}ms avg)")
    analytics.logEvent("tflite_gpu_fallback_suspected")
}

On Adreno 6xx and Mali-G7x hardware, GPU inference completes under 20ms for a 256×256 model. CPU execution reliably exceeds 35ms. That warmup probe will catch device-tier issues you will never reproduce in your own test lab.

Forgetting imageProxy.close(). CameraX stalls the entire analyzer pipeline if you skip this. It is easy to miss when you are focused on the inference path.

Using MiDaS v3 Large on mid-range. The Large variant at 384×384 input adds 30–60% to inference time. It requires a higher-tier device to hit 30fps. Start with Small.

Conclusion

Three things actually matter here: pre-allocate everything at startup, use a latency probe to detect GPU fallback rather than a string check, and downsample before accumulating. Get those three right and you will have a pipeline that stays under 28ms and survives a full session on mid-range hardware.

Further reading:

DEV Community