---
title: "Real-Time AR Occlusion on Android: CameraX + TFLite GPU Under 33ms"
published: true
description: "Wire CameraX ImageAnalysis to a quantized MiDaS/Depth Anything V2 model via TFLite GPU delegate for real-time AR occlusion meshes — covering delegate selection, YUV→RGB cost, and ring-buffer recovery."
tags: kotlin, android, mobile, architecture
canonical_url: https://blog.mvpfactory.co/real-time-ar-occlusion-android-camerax-tflite-gpu
---
## What We Are Building
By the end of this tutorial you will have a working Android pipeline that takes live camera frames, runs monocular depth estimation on-device at 30fps, and feeds the result into SceneView/ARCore as a per-pixel occlusion mesh. No depth sensor required.
The target: every stage completes inside a **33ms frame budget**.
## Prerequisites
- Android project targeting API 24+ (API 31+ for the zero-copy GPU path)
- CameraX `1.3+` and TFLite `2.14+` added to your dependencies
- A quantized INT8 MiDaS 384×384 or Depth Anything V2 model (`.tflite`)
- SceneView or ARCore set up for your AR scene
- Basic familiarity with Kotlin coroutines and Android GPU concepts
---
## The Pipeline at a Glance
Here is the minimal setup to get this working — treat the whole chain as a **single GPU memory budget problem**:
CameraX ImageAnalysis
│ (YUV_420_888, non-blocking)
▼
YUV → RGB Conversion (GPU Bitmap or shader)
│
▼
TFLite Interpreter (GPU Delegate)
└── INT8 quantized MiDaS / Depth Anything V2
│
▼
Depth Map Buffer (ring buffer, 3 frames)
│
▼
SceneView / ARCore Occlusion Mesh
Each stage competes for the same GPU bus. Getting even one stage on the wrong executor blows your frame budget entirely.
---
## Step 1 — Choose Your Delegate (and Test It Under Real Load)
Let me show you a pattern I use in every project: benchmark delegates under concurrent camera and render load, not in isolation. That's where NNAPI will burn you.
| Delegate | Avg Inference (INT8) | Jitter (p99) | Notes |
|---|---|---|---|
| CPU (4 threads) | ~90ms | High | Unusable at 30fps |
| NNAPI | ~28ms | High under load | Driver-dependent, risky |
| GPU Delegate | ~18–22ms | Low | Consistent, composable |
The GPU delegate wins because it shares memory space with the render pipeline — no cross-bus copies. Configure it like this:
kotlin
val options = GpuDelegateV2.Options().apply {
setPrecisionLossAllowed(true)
setInferencePriority1(GpuDelegateV2.Options.INFERENCE_PRIORITY_MIN_LATENCY)
}
val gpuDelegate = GpuDelegateV2(options)
val interpreterOptions = Interpreter.Options().apply {
addDelegate(gpuDelegate)
}
val interpreter = Interpreter(modelBuffer, interpreterOptions)
NNAPI's p99 latency is its weakness — you will not see it until CameraX and SceneView are both running at full tilt.
---
## Step 2 — Fix the YUV→RGB Path First
Here is the gotcha that will save you hours: CameraX delivers `YUV_420_888` frames and CPU conversion costs **5–12ms per frame**. That can break your budget before inference even starts.
On API 31+, use a zero-copy GPU texture hand-off:
kotlin
// API 31+ path: zero-copy GPU texture hand-off
val imageReader = ImageReader.newInstance(
width, height,
ImageFormat.YUV_420_888, 3,
HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_CPU_READ_RARELY
)
This lands the RGB data directly in GPU memory so the TFLite GPU delegate reads it without a host-side copy.
On pre-31 devices, fall back to a RenderScript intrinsic (`ScriptIntrinsicYuvToRGB`). CPU-side `Bitmap` conversion is a last resort — always benchmark it on your **minimum target device** before shipping.
Fixing this single stage can recover 8–12ms of frame budget before you touch the model or delegate configuration. It is the highest-leverage optimization in the pipeline.
---
## Step 3 — Add a Ring Buffer From Day One
Thermal throttling and driver hiccups are production realities, not edge cases. Without a recovery strategy, missed depth frames cause visible occlusion flicker as stale geometry snaps to new camera poses.
A three-slot ring buffer decouples inference timing from render timing at the cost of ~45KB of memory:
kotlin
class DepthRingBuffer(size: Int = 3) {
private val slots = Array(size) { FloatArray(WIDTH * HEIGHT) }
private val writeIdx = AtomicInteger(0)
fun write(depth: FloatArray) {
val slot = writeIdx.getAndIncrement() % slots.size
depth.copyInto(slots[slot])
}
fun readLatest(): FloatArray = slots[writeIdx.get() % slots.size]
}
The render thread calls `readLatest()` every frame. If inference missed its slot, the prior depth map is reused — imperceptible at 30fps unless the camera moves fast.
---
## Step 4 — Manage the GPU Memory Budget
On a mid-range SoC (Adreno 6xx, Mali G78), the shared GPU memory budget for a camera + AR workload is roughly **200–300MB** before the OS starts evicting. Your INT8 MiDaS 384×384 model occupies ~15MB in the delegate's tensor arena. SceneView's shadow maps and environment textures consume another 80–120MB.
The docs do not mention this, but FP16 intermediate activations in the GPU delegate can hurt latency if your driver does not handle them efficiently. Measure with:
bash
adb shell dumpsys gfxinfo
Quantize aggressively: INT8 over FP32 cuts model size by ~4x with minimal accuracy loss for occlusion. You do not need millimeter precision — you need correct layering order.
---
## Gotchas
- **Do not benchmark delegates in isolation.** Run CameraX and SceneView simultaneously before committing. NNAPI's scheduling latency only shows up under concurrent GPU load.
- **CPU bitmap conversion is a silent budget killer.** It looks fine in microbenchmarks and destroys frame timing in production.
- **Non-blocking mode on `ImageAnalysis` is mandatory.** Blocking mode will queue frames and introduce compounding latency. Set `setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)`.
- **FP16 activations are driver-dependent.** What works on a Pixel 7 may jank on a Samsung mid-range. Always test on your minimum supported device.
---
## Conclusion
Monocular depth for AR occlusion is solvable at 30fps on-device if you treat the pipeline as a unified GPU memory problem. The three changes with the highest ROI: profile delegate latency under real load (not synthetic benchmarks), eliminate the CPU YUV→RGB path before touching anything else, and ship a ring buffer from day one to absorb thermal and driver variance.
**Relevant resources:**
- [TFLite GPU Delegate docs](https://www.tensorflow.org/lite/performance/gpu)
- [CameraX ImageAnalysis reference](https://developer.android.com/training/camerax/analyze)
- [SceneView occlusion API](https://github.com/SceneView/sceneview-android)
- [Depth Anything V2 model hub](https://huggingface.co/depth-anything)
Top comments (0)