---
title: "CameraX + Quantized CLIP: Zero-Shot Vision at 30fps on Android"
published: true
description: "Wire Android's CameraX to a quantized CLIP vision encoder via TFLite and XNNPACK for real-time zero-shot classification under 30ms per frame. Memory layout, quantization strategy, and label discipline are what actually determine whether you succeed."
tags: [android, kotlin, mobile, architecture]
canonical_url: https://blog.mvpfactory.co/camerax-quantized-clip-zero-shot-30fps
---
## What We Are Building
A real-time zero-shot visual classifier on Android. CameraX feeds frames into a quantized CLIP vision encoder running through TFLite with the XNNPACK delegate. Each frame produces an embedding we score against a frozen label matrix using cosine similarity — and we do it in under 30ms, without triggering the OOM killer.
Let me show you a pattern I use in every project: treat this as a **pipeline problem**, not a model problem. Most teams swap in smaller models and still miss their frame budget because the bottleneck is preprocessing, memory allocation, or interpreter lifecycle — not the model itself.
---
## Prerequisites
- Android project targeting API 21+
- CameraX 1.3+ (`androidx.camera:camera-camera2`, `camera-lifecycle`, `camera-view`)
- TFLite runtime with XNNPACK delegate (`org.tensorflow:tensorflow-lite`, `tensorflow-lite-gpu`)
- A quantized CLIP vision encoder exported to `.tflite` (INT8 PTQ or QAT)
- Pre-computed label embeddings (run your text prompts through the CLIP text encoder offline, L2-normalize, freeze as a `[N×512]` float32 buffer)
---
## Step 1 — Enable the XNNPACK Delegate at Interpreter Construction
Here is the minimal setup to get this working. Pass the delegate when building the interpreter — this is when TFLite pre-packs weight tensors into XNNPACK's tiled memory layout. This happens **once at load time**, not per inference.
kotlin
val xnnpackDelegate = XNNPackDelegate(
XNNPackDelegate.Options().apply { numThreads = 2 }
)
val interpreter = Interpreter(
modelBuffer,
Interpreter.Options().addDelegate(xnnpackDelegate)
)
Never recreate the interpreter between frames. Interpreter creation costs ~40ms and forces a full re-pack cycle. Interpreter lifetime should match session lifetime — allocate once, release only when the use case is torn down.
---
## Step 2 — Wire CameraX with the Right Resolution Strategy
`setTargetResolution` is deprecated as of CameraX 1.3. Use `ResolutionSelector` with `ResolutionStrategy`. Run the analyzer on a dedicated single-thread `Executor` — frame analysis must not compete with the main thread or Compose recomposition.
kotlin
val resolutionSelector = ResolutionSelector.Builder()
.setResolutionStrategy(
ResolutionStrategy(
Size(224, 224),
ResolutionStrategy.FALLBACK_RULE_CLOSEST_HIGHER_THEN_LOWER
)
)
.build()
val imageAnalysis = ImageAnalysis.Builder()
.setResolutionSelector(resolutionSelector)
.setBackpressureStrategy(STRATEGY_KEEP_ONLY_LATEST)
.setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
.build()
.also { it.setAnalyzer(executor, ::analyzeFrame) }
Use `STRATEGY_KEEP_ONLY_LATEST` — you want the freshest frame, not a queue of stale ones.
---
## Step 3 — Quantization Strategy
The docs do not always make this trade-off explicit, but here is the table that actually matters for CLIP on a mid-range Snapdragon:
| Precision | Model Size | Inference (Pixel 7) | Accuracy Drop |
|------------|------------|---------------------|---------------|
| FP32 | ~350MB | ~55ms | Baseline |
| FP16 | ~175MB | ~38ms | <0.5% |
| INT8 (PTQ) | ~88MB | ~19ms | 1–3% |
| INT8 (QAT) | ~88MB | ~19ms | <1% |
Use INT8 PTQ as your baseline. Invest in QAT only if the accuracy regression exceeds your product threshold — model size and latency are identical between the two. The text encoder is only run offline, so its precision is irrelevant to the runtime budget.
---
## Step 4 — Memory Layout Under 512MB
At 30fps your budget is 33ms per frame. A quantized CLIP vision encoder costs 18–22ms on a mid-range Snapdragon. That leaves 11–15ms for YUV→RGB conversion (use libyuv, not deprecated RenderScript on API 31+), similarity scoring, and UI dispatch.
Two rules keep you inside that margin:
- Allocate input/output tensors as `ByteBuffer.allocateDirect()` — off-heap, GC-invisible
- Keep your frozen label embedding matrix in native memory and treat it as immutable
XNNPACK's pre-packed weight buffers live in native memory and do not count against your Java heap. The OOM killer on Android targets the largest contiguous Java heap allocation first — native buffers are your friend here.
---
## Gotchas
**Recreating the interpreter between frames.** This is the one that will burn you. Every teardown forces a repack cycle that costs 4–8ms and spikes GC pressure at exactly the wrong moment — 99th-percentile frame time spikes, visible stutters, and an angry product manager.
**Reallocating the label embedding matrix at runtime.** Freeze it at init. Any runtime reallocation introduces GC pauses that blow your frame budget. Compute it once from your text prompts, L2-normalize, and never touch it again.
**Running the analyzer on the main thread.** CameraX will let you do this. Do not. Dedicate a single-thread executor to frame analysis.
**Using `setTargetResolution`.** Deprecated since CameraX 1.3. Use `ResolutionSelector` or you will get a compiler warning today and a runtime surprise later.
---
## Conclusion
The pipeline — CameraX → YUV→RGB → TFLite + XNNPACK → cosine similarity → label — is straightforward once you understand that the hard constraints are memory layout and interpreter lifecycle, not model architecture. Enable the XNNPACK delegate at construction, freeze your label embeddings at init, allocate off-heap, and never recreate the interpreter mid-session. Hit those three and you will hold 30fps on mid-range hardware with headroom to spare.
**Resources:**
- [TFLite XNNPACK Delegate docs](https://www.tensorflow.org/lite/performance/xnnpack)
- [CameraX ResolutionSelector API](https://developer.android.com/reference/androidx/camera/core/resolutionselector/ResolutionSelector)
- [libyuv on Android](https://chromium.googlesource.com/libyuv/libyuv)
Top comments (0)