On-Device VLM Receipt Parsing with Android NNAPI Delegate

What We Are Building

Running a Vision-Language Model on mid-range Android hardware is viable — but only if you treat memory, tokens, and thermals as first-class constraints from day one. In this tutorial we wire a quantized PaliGemma 3B INT4 model to Android's NNAPI delegate for real-time receipt parsing from camera frames. You will walk away with a working six-stage OCR-to-token pipeline, a locked 512-token budget, a thermal gate, and a smart ML Kit fallback that cuts VLM inference frequency by 40–60%.

Prerequisites

• Android project targeting API 29+
• TensorFlow Lite with NNAPI delegate
• ML Kit Text Recognition v2
• CameraX
• A mid-range test device — Snapdragon 6xx or Dimensity 7xx series recommended. Do not benchmark only on flagship hardware.

---

Step 1: Configure the NNAPI Delegate

Here is the minimal setup to get this working. Target the DSP explicitly — CPU fallback defeats the purpose entirely.

val nnApiDelegate = NnApiDelegate(
    NnApiDelegate.Options().apply {
        acceleratorName = "qti-dsp"         // target DSP, not CPU fallback
        useNnapiCpu = false
        allowFp16PrecisionForFp32 = true
        executionPreference = NnApiDelegate.Options.EXECUTION_PREFERENCE_SUSTAINED_SPEED
    }
)

val interpreterOptions = Interpreter.Options().apply {
    addDelegate(nnApiDelegate)
    setNumThreads(2)                        // leave threads for UI and camera pipeline
}

Use EXECUTION_PREFERENCE_SUSTAINED_SPEED, not FAST_SINGLE_ANSWER. On streaming camera frames, fast mode spikes the DSP clock and triggers thermal throttling within 90 seconds.

---

Step 2: Lock Your Token Budget at 512

A naively loaded PaliGemma 3B in FP16 requires ~6GB VRAM equivalent — a non-starter on mid-range hardware. INT4 quantization brings the weight footprint to ~1.6GB. Even then, your usable RAM ceiling on a Snapdragon 695 is roughly 3.5–4GB after OS and app overhead. Here is what the budget looks like, measured via Android Studio Memory Profiler:

Token Budget	Working Mem	Viable on 4GB?	Quality
256 tokens	~210 MB	Yes	Partial (totals, date)
512 tokens	~380 MB	Yes	Good (line items + totals)
768 tokens	~520 MB	Marginal	Excellent
1024 tokens	~680 MB	No — OOM risk	—

My production default is 512. The prompt template enforces the output cap explicitly:

val prompt = buildString {
    append("<image>")
    append("Extract receipt JSON: {merchant, date, line_items:[{desc,qty,price}], total, currency}. ")
    append("Be terse. Max 400 output tokens.")
}

The inline cap reduces the probability of the model running long on verbose receipts.

---

Step 3: Build the Six-Stage OCR Pipeline

Feeding raw 1280×960 frames directly into the VLM is a token budget disaster. Split responsibilities:

1. Capture — CameraX ImageProxy (1280×960)
2. Pre-crop — OpenCV candidate receipt regions
3. Confidence gate — ML Kit Text Detection 4a. High-confidence path — Tile + Resize (224×224) → PaliGemma INT4 via NNAPI → structured JSON 4b. Low-confidence fallback — ML Kit OCR → raw text strings
4. Post-process — Regex merge → final structured receipt

ML Kit handles degraded regions. Reserve the VLM for structured extraction where it actually adds value.

---

Step 4: Build the Thermal Gate Before Anything Else

Continuous camera inference is a thermal trap. Here is a pattern I use in every project:

private val inferenceScope = CoroutineScope(Dispatchers.Default + SupervisorJob())
private var lastInferenceMs = 0L
private val MIN_INTERVAL_MS = 800L  // ~1.25 fps inference rate

fun onFrameAvailable(frame: ImageProxy) {
    val now = SystemClock.elapsedRealtime()
    if (now - lastInferenceMs < MIN_INTERVAL_MS) {
        frame.close(); return
    }
    if (thermalStatusProvider.isThrottling()) {
        frame.close(); return  // hard stop, not degraded inference
    }
    lastInferenceMs = now
    inferenceScope.launch { runInference(frame) }
}

Use PowerManager.getThermalHeadroom() (API 29+) as your throttle signal. When headroom drops below 0.7, halt inference entirely. Partial results on a receipt are worse than no results.

---

Gotchas

ML Kit is a pre-filter, not just a fallback. ML Kit Text Recognition v2 processes a 224×224 crop in under 30ms on the same mid-range hardware. For line items where OCR confidence is above 0.92, a regex post-processor over ML Kit output produces equivalent structured results at a fraction of the VLM cost. On a corpus of ~1,200 grocery and retail receipts, this routing strategy cut VLM inference frequency by roughly 40–60% — though your exact split depends on receipt quality and merchant format diversity.

The docs do not mention this, but FAST_SINGLE_ANSWER is the wrong preference for streaming frames. It is designed for single-shot queries, not sustained inference. Use SUSTAINED_SPEED.

Do not tune the token budget post-launch. Set it at 512 before you write model code. Memory pressure on mid-range devices is a design constraint, not a tuning knob. Size your pipeline around the ceiling, not the ideal case.

A dropped frame is recoverable. A throttled NPU mid-transaction is not. Monitor thermal headroom, not just frame rate.

---

Conclusion

The VLM is the expensive last resort — reserved for ambiguous layouts, handwritten annotations, and multi-currency edge cases. Route everything else through ML Kit. Lock the token budget at 512. Wire the thermal gate before you wire the model. That ordering matters more than any individual optimization.