DEV Community

Viacheslav Volkov
Viacheslav Volkov

Posted on

Shipping a real on-device LLM app on Android: 5 LiteRT-LM production bugs nobody warns you about

We ship an offline-first LLM chat app on Google Play. Everything runs on-device with Google's LiteRT-LM runtime (com.google.ai.edge.litertlm:litertlm-android) — no server, no account, no data leaving the phone. Here's what actually broke in production and how we fixed it.

1. The GPU delegate will get your app killed

The obvious move is "enable GPU, it's faster". In LiteRT-LM ≤0.17 the GPU delegate allocates RSS an order of magnitude above the model size — we measured ~11 GB resident for a 0.8B int4 model on a Galaxy S24 Ultra. lmkd kills the app in the foreground.

Fix: CPU is the default for every device. GPU is an opt-in "experimental" switch, plus two fallback layers:

  • init-time: if Engine creation with GPU throws, retry with CPU;
  • generation-time: if inference fails with known GPU markers, flip a permanent force_cpu flag and retry.

2. dlopen("libOpenCL.so") fails on Android 12+ even when the driver exists

On Samsung/One UI (Vulkan-only GPUs) the GPU path dies at runtime with Can not find OpenCL library. The device has the library — it's in /vendor/etc/public.libraries.txt — but the driver has private dependencies unavailable to the app namespace.

Fix in AndroidManifest.xml:

<uses-native-library android:name="libOpenCL.so" android:required="false"/>
<uses-native-library android:name="libOpenCL-car.so" android:required="false"/>
<uses-native-library android:name="libOpenCL-pixel.so" android:required="false"/>
Enter fullscreen mode Exit fullscreen mode

Without the required="false" declarations the app won't even install on devices that lack the libs.

3. Naive chat templates serialize JSON into your context

LiteRT-LM SDK passes message content as a parts array, and the engine stores model replies as multi-part content. Templates that do {{ message.content }} without a content is string branch (huihui-family finetunes are the usual suspects) dump raw JSON straight into the conversation:

[{"type":"text","text":"…"}]
Enter fullscreen mode Exit fullscreen mode

The next turn then echoes the prompt back because the context is poisoned.

Fix, two layers:

  • at session creation, build a content-safe copy of the embedded template with a parse_content macro that iterates parts and joins text fields — substitute it into ConversationConfig.chatTemplate;
  • at the sink, run a sanitizer that unwraps [{"type":"text",…}] envelopes in already-generated text before persisting to Room.

Verify with a real multi-turn test on the affected model — the bug only appears on turn 3+ when the stored message re-enters the context.

4. .litertlm bundles are version-fragile

Bundles exported for SDK 0.13.1 refused to run on 0.16.1: prefill work group size exceeds available state entries, on both GPU and CPU. The model catalog had to be re-converted wholesale for the new runtime. Keep a minRuntimeVersion field in your model JSON and treat bundle regeneration as part of the SDK bump, not an afterthought.

The good news: xnnpack weight caches are portable — same model file (mtime+size identity), different device, cache still valid. We ship no cache, but users' second device inits are faster than the first download+compile.

5. Engine/Conversation are not thread-safe — serialize everything

Any concurrent native call during streaming risks use-after-free → SIGSEGV. We wrapped every native interaction — initializeModel, prepareConversation, generateResponseStream, closeModel, even title generation — under a single Mutex. An AtomicInteger of in-flight native ops gates the idle auto-unload timer so the engine is never closed mid-stream.

Bonus: reuse the live Conversation across turns. We verify append-only history growth (seeded params + native turn count + suffix check on the wrapped prompt) and skip re-creating the session — prefill then runs only on the new tokens, not the whole history.

Numbers worth knowing (measured, device + runtime noted)

  • 0.8B warm init: ~40s on Snapdragon S20-class, ~98s on a $200 Nord N200
  • 4B warm init: ~168s
  • The native LITERTLM container load phase — not your Kotlin — is where init time lives

If you're building on-device AI on Android, I'd love to compare notes. I'm also building LocalClaw (offline agent runtime) and LocalAI (the chat client described above) — both free, both no-account.

All measurements device-verified; runtime versions: litertlm-android 0.16.1–0.18.0, minSdk 26, targetSdk 36.

Top comments (0)