<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Viacheslav Volkov</title>
    <description>The latest articles on DEV Community by Viacheslav Volkov (@handydude).</description>
    <link>https://dev.to/handydude</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4252358%2F85b0237c-b889-40f0-a1ee-fa3017e9ce27.png</url>
      <title>DEV Community: Viacheslav Volkov</title>
      <link>https://dev.to/handydude</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/handydude"/>
    <language>en</language>
    <item>
      <title>Shipping a real on-device LLM app on Android: 5 LiteRT-LM production bugs nobody warns you about</title>
      <dc:creator>Viacheslav Volkov</dc:creator>
      <pubDate>Wed, 18 Nov 2026 17:17:41 +0000</pubDate>
      <link>https://dev.to/handydude/shipping-a-real-on-device-llm-app-on-android-5-litert-lm-production-bugs-nobody-warns-you-about-km3</link>
      <guid>https://dev.to/handydude/shipping-a-real-on-device-llm-app-on-android-5-litert-lm-production-bugs-nobody-warns-you-about-km3</guid>
      <description>&lt;p&gt;We ship an offline-first LLM chat app on Google Play. Everything runs on-device with Google's LiteRT-LM runtime (&lt;code&gt;com.google.ai.edge.litertlm:litertlm-android&lt;/code&gt;) — no server, no account, no data leaving the phone. Here's what actually broke in production and how we fixed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The GPU delegate will get your app killed
&lt;/h2&gt;

&lt;p&gt;The obvious move is "enable GPU, it's faster". In LiteRT-LM ≤0.17 the GPU delegate allocates RSS an order of magnitude above the model size — we measured ~11 GB resident for a 0.8B int4 model on a Galaxy S24 Ultra. &lt;code&gt;lmkd&lt;/code&gt; kills the app in the foreground.&lt;/p&gt;

&lt;p&gt;Fix: &lt;strong&gt;CPU is the default for every device&lt;/strong&gt;. GPU is an opt-in "experimental" switch, plus two fallback layers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;init-time: if &lt;code&gt;Engine&lt;/code&gt; creation with GPU throws, retry with CPU;&lt;/li&gt;
&lt;li&gt;generation-time: if inference fails with known GPU markers, flip a permanent &lt;code&gt;force_cpu&lt;/code&gt; flag and retry.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  2. &lt;code&gt;dlopen("libOpenCL.so")&lt;/code&gt; fails on Android 12+ even when the driver exists
&lt;/h2&gt;

&lt;p&gt;On Samsung/One UI (Vulkan-only GPUs) the GPU path dies at runtime with &lt;code&gt;Can not find OpenCL library&lt;/code&gt;. The device &lt;em&gt;has&lt;/em&gt; the library — it's in &lt;code&gt;/vendor/etc/public.libraries.txt&lt;/code&gt; — but the driver has private dependencies unavailable to the app namespace.&lt;/p&gt;

&lt;p&gt;Fix in &lt;code&gt;AndroidManifest.xml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight xml"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;uses-native-library&lt;/span&gt; &lt;span class="na"&gt;android:name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"libOpenCL.so"&lt;/span&gt; &lt;span class="na"&gt;android:required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"false"&lt;/span&gt;&lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;uses-native-library&lt;/span&gt; &lt;span class="na"&gt;android:name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"libOpenCL-car.so"&lt;/span&gt; &lt;span class="na"&gt;android:required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"false"&lt;/span&gt;&lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;uses-native-library&lt;/span&gt; &lt;span class="na"&gt;android:name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"libOpenCL-pixel.so"&lt;/span&gt; &lt;span class="na"&gt;android:required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"false"&lt;/span&gt;&lt;span class="nt"&gt;/&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without the &lt;code&gt;required="false"&lt;/code&gt; declarations the app won't even install on devices that lack the libs.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Naive chat templates serialize JSON into your context
&lt;/h2&gt;

&lt;p&gt;LiteRT-LM SDK passes message content as a parts array, and the engine stores model replies as multi-part content. Templates that do &lt;code&gt;{{ message.content }}&lt;/code&gt; without a &lt;code&gt;content is string&lt;/code&gt; branch (huihui-family finetunes are the usual suspects) dump raw JSON straight into the conversation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"…"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The next turn then echoes the prompt back because the context is poisoned.&lt;/p&gt;

&lt;p&gt;Fix, two layers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;at session creation, build a &lt;em&gt;content-safe&lt;/em&gt; copy of the embedded template with a &lt;code&gt;parse_content&lt;/code&gt; macro that iterates parts and joins &lt;code&gt;text&lt;/code&gt; fields — substitute it into &lt;code&gt;ConversationConfig.chatTemplate&lt;/code&gt;;&lt;/li&gt;
&lt;li&gt;at the sink, run a sanitizer that unwraps &lt;code&gt;[{"type":"text",…}]&lt;/code&gt; envelopes in already-generated text before persisting to Room.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Verify with a real multi-turn test on the affected model — the bug only appears on turn 3+ when the stored message re-enters the context.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. &lt;code&gt;.litertlm&lt;/code&gt; bundles are version-fragile
&lt;/h2&gt;

&lt;p&gt;Bundles exported for SDK 0.13.1 refused to run on 0.16.1: &lt;code&gt;prefill work group size exceeds available state entries&lt;/code&gt;, on both GPU and CPU. The model catalog had to be re-converted wholesale for the new runtime. Keep a &lt;code&gt;minRuntimeVersion&lt;/code&gt; field in your model JSON and treat bundle regeneration as part of the SDK bump, not an afterthought.&lt;/p&gt;

&lt;p&gt;The good news: &lt;code&gt;xnnpack&lt;/code&gt; weight caches are portable — same model file (mtime+size identity), different device, cache still valid. We ship no cache, but users' second device inits are faster than the first download+compile.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. &lt;code&gt;Engine&lt;/code&gt;/&lt;code&gt;Conversation&lt;/code&gt; are not thread-safe — serialize everything
&lt;/h2&gt;

&lt;p&gt;Any concurrent native call during streaming risks use-after-free → SIGSEGV. We wrapped every native interaction — &lt;code&gt;initializeModel&lt;/code&gt;, &lt;code&gt;prepareConversation&lt;/code&gt;, &lt;code&gt;generateResponseStream&lt;/code&gt;, &lt;code&gt;closeModel&lt;/code&gt;, even title generation — under a single &lt;code&gt;Mutex&lt;/code&gt;. An &lt;code&gt;AtomicInteger&lt;/code&gt; of in-flight native ops gates the idle auto-unload timer so the engine is never closed mid-stream.&lt;/p&gt;

&lt;p&gt;Bonus: reuse the live &lt;code&gt;Conversation&lt;/code&gt; across turns. We verify append-only history growth (seeded params + native turn count + suffix check on the wrapped prompt) and skip re-creating the session — prefill then runs only on the new tokens, not the whole history.&lt;/p&gt;

&lt;h2&gt;
  
  
  Numbers worth knowing (measured, device + runtime noted)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;0.8B warm init: ~40s on Snapdragon S20-class, ~98s on a $200 Nord N200&lt;/li&gt;
&lt;li&gt;4B warm init: ~168s&lt;/li&gt;
&lt;li&gt;The native LITERTLM container load phase — not your Kotlin — is where init time lives&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you're building on-device AI on Android, I'd love to compare notes. I'm also building &lt;a href="https://ochandydude.dev/apps/localclaw/" rel="noopener noreferrer"&gt;LocalClaw&lt;/a&gt; (offline agent runtime) and &lt;a href="https://ochandydude.dev/apps/localai/" rel="noopener noreferrer"&gt;LocalAI&lt;/a&gt; (the chat client described above) — both free, both no-account.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;All measurements device-verified; runtime versions: litertlm-android 0.16.1–0.18.0, minSdk 26, targetSdk 36.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>android</category>
      <category>ai</category>
      <category>kotlin</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
