---
title: "Running Gemma-3 2B on Android: KV-Cache, Prefill Latency, and the Token Budget That Actually Fits"
published: true
description: "KV-cache math, Snapdragon 7 vs 8 prefill benchmarks, and INT4 vs INT8 tradeoffs for shipping Gemma-3 2B on mid-range Android devices with MediaPipe."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/gemma3-android-mediapipe-kvcache
---
## What we are building
By the end of this walkthrough, you will understand how to wire Android's MediaPipe LLM Inference API to a quantized Gemma-3 2B model — and more importantly, *why* the configuration choices you make at initialization will either ship a smooth on-device experience or quietly OOM-kill your process mid-session.
Let me show you a pattern I use in every on-device LLM project: budget memory from first principles before writing a single line of integration code.
---
## Prerequisites
- Android project targeting API 26+
- MediaPipe Tasks dependency added to your Gradle build
- A `gemma3-2b-it-int4.task` model file (available via Google's model page)
- Familiarity with Kotlin coroutines or callback-based async patterns
---
## Step 1: Understand where your memory actually goes
Most teams benchmark the model weights and forget the KV-cache. That is where the memory surprises come from.
The KV (Key-Value) cache stores intermediate attention state for every token in context. It enables autoregressive decoding without recomputing the full attention matrix on every step — a worthwhile tradeoff, but one with a **linear memory cost**.
Here is the formula:
KV cache (bytes) = 2 × num_kv_heads × head_dim × num_layers × context_len × dtype_bytes
For Gemma-3 2B — grouped query attention, 4 KV heads, 256-dimensional heads, 26 transformer layers, FP16:
2 × 4 × 256 × 26 × 1 token × 2 bytes = 106,496 bytes ≈ 104 KB per token
| Context Length | KV-Cache Size |
|---|---|
| 256 tokens | ~26 MB |
| 512 tokens | ~52 MB |
| 1,024 tokens | ~104 MB |
| 2,048 tokens | ~208 MB |
At 2,048 tokens the cache alone consumes 208 MB. Add weights and activations and you are threatening the foreground memory limit on a 4 GB device.
---
## Step 2: Choose your quantization tier
| Quantization | Weight Size | 4 GB Device Feasible | Quality Delta |
|---|---|---|---|
| FP16 | ~4 GB | No | — |
| INT8 | ~2 GB | Marginal (6 GB+ recommended) | <1% perplexity |
| INT4 | ~1 GB | Yes (with context budget) | 2–4% perplexity |
INT4 is the only practical path on 4 GB devices. The quality regression is real but tolerable for most inference workloads. MediaPipe bundles quantization into the `.task` model format — you select the correct file at integration time and the runtime handles the rest.
---
## Step 3: Wire the MediaPipe LLM Inference API
Here is the minimal setup to get this working:
kotlin
val options = LlmInference.LlmInferenceOptions.builder()
.setModelPath("/data/local/tmp/gemma3-2b-it-int4.task")
.setMaxTokens(1500) // pre-allocated KV-cache budget
.setMaxTopK(40)
.setTemperature(0.8f)
.build()
val llmInference = LlmInference.createFromOptions(context, options)
llmInference.generateResponseAsync(prompt) { partial, done ->
// stream tokens to UI
}
`setMaxTokens` is your single most important configuration knob. It determines how much KV-cache MediaPipe pre-allocates at initialization. Set it too high on a 4 GB device and the OOM killer terminates your process before the first inference. For mid-range INT4 deployments, start at 1,200–1,500.
---
## Step 4: Profile before you ship
The `.task` file size is not your runtime footprint — do not use it as a proxy.
Measure the RSS delta before and after `LlmInference.createFromOptions()`:
bash
adb shell dumpsys meminfo
Android Studio's Memory Profiler visualizes the allocation spike in real time. Perfetto gives you a system-level view of memory pressure across the full stack.
Working backwards from a ~1.6 GB ceiling on a 4 GB foreground app:
- INT4 weights: ~1,000 MB
- Activations + MediaPipe runtime: ~200 MB
- Remaining for KV-cache: ~400 MB
- Safe operational limit: **1,200–1,500 tokens**
---
## Gotchas
**Prefill latency will wreck your UX on 7-series SoCs.** Prefill is the phase where the model processes your entire prompt before generating the first token. The gap between device tiers is severe:
| SoC | 128-tok Prefill | 512-tok Prefill | 1024-tok Prefill |
|---|---|---|---|
| Snapdragon 8 Gen 3 | ~0.6s | ~2.0s | ~4.5s |
| Snapdragon 7 Gen 2 | ~1.8s | ~6.5s | ~14s |
| Snapdragon 7s Gen 3 | ~2.0s | ~7.2s | ~16s |
A 14-second blank screen before the first token is a UX failure. On 7-series devices, keep your system prompt under 300 tokens and design your loading state accordingly. Users tolerate slower token generation far better than a long blank wait at the start.
**MediaPipe routes through the GPU delegate, not the NPU.** The LLM Inference API dispatches via OpenGL ES / Vulkan compute by default. Direct NPU/HTP access is not exposed through the public MediaPipe LLM API. If you need Hexagon Tensor Processor acceleration on Snapdragon silicon, Qualcomm's AI Hub SDK is the alternative — at the cost of a heavier integration and different model formats.
**OOM is silent until it is catastrophic.** Memory is allocated upfront at `createFromOptions`. There is no graceful recovery path from a mid-session OOM — load-test at initialization on your lowest-spec target device before releasing.
---
## Conclusion
Running Gemma-3 2B on a 4 GB Android device is feasible, but only with INT4 quantization and a carefully budgeted context window. Keep your combined token budget under 1,500, your system prompt under 300 tokens on 7-series hardware, and measure RSS delta — not file size — before you commit to a device tier.
(On the topic of building thoughtful developer tools: if you spend long sessions working through problems like this, [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) is worth a look for keeping your desk-time sustainable with break reminders and guided exercises.)
The docs do not mention this, but the engineers who ship stable on-device LLM experiences all do the same thing: they budget memory on a whiteboard before they open Android Studio.
---
*Figures in the prefill benchmarks are estimates derived from relative GPU compute throughput ratios. Verify against Qualcomm's official SoC specification pages and capture device-specific baselines with Perfetto before citing in production documentation.*
Top comments (0)