DEV Community

SoftwareDevs mvpfactory.io
SoftwareDevs mvpfactory.io

Posted on Originally published at mvpfactory.io

On-Device LLM Inference on Android with Microsoft Decision-1 and Qwen3.5-9B

#webdev #programming 
---
title: "On-Device LLM Inference on Android: Running Qwen3.5-9B with NNAPI and Vulkan Delegate"
published: true
description: "A production guide to deploying Qwen3.5-9B on Snapdragon 8 Gen 3 — covering Q4_K_M vs Q5_K_S quantization tradeoffs, mmap weight loading, and achieving sub-400ms TTFT."
tags: android, kotlin, mobile, architecture
canonical_url: https://blog.mvpfactory.co/on-device-llm-android-qwen3-nnapi
---
Enter fullscreen mode Exit fullscreen mode

What You Will Build

By the end of this tutorial you will have a working Android inference pipeline that runs Qwen3.5-9B on a Snapdragon 8 Gen 3 device, stays under 2.5 GB of LPDDR5 memory, and delivers time-to-first-token under 400ms. Let me show you the pattern I use in every production on-device LLM deployment.

Open-weight 9B models are no longer a server-only story. The hardware caught up: Snapdragon 8 Gen 3 ships with ~77 GB/s LPDDR5 bandwidth and a Hexagon NPU rated at 73 TOPS INT8. The remaining question is purely about quantization and delegation strategy — and that is exactly what we are covering here.

Prerequisites

  • Android device with Snapdragon 8 Gen 3 (or simulator for architecture work)
  • llama.cpp Android port with JNI bindings configured
  • Qwen3.5-9B GGUF weights downloaded (Q4_K_M format — more on why below)
  • Android GPU Inspector installed for delegation profiling
  • minSdkVersion 29 or above (required for NNAPI stable paths)

Step 1 — Choose Q4_K_M, Not Q5

Here is the gotcha that will save you hours: the quantization format is not a quality tradeoff on Android. It is a deployment gate.

Your app gets approximately 2.5 GB of LPDDR5 (the OS reserves ~1.5 GB). Here is what the numbers actually look like at runtime:

Format GGUF File Size Runtime RSS (2K ctx) PPL Delta vs FP16 Avg Token Latency TTFT
Q4_K_M ~5.3 GB ~2.1 GB +0.18 ~28 ms/tok ~340 ms
Q5_K_S ~6.2 GB ~2.5 GB +0.09 ~34 ms/tok ~410 ms
Q5_K_M ~6.7 GB ~2.7 GB +0.07 ~38 ms/tok ~460 ms

Q5_K_S consumes the full 2.5 GB budget with zero headroom for KV cache (which costs ~280 MB at 2K context, plus ~350 MB for activation buffers). Q4_K_M leaves ~150 MB of runtime margin — enough to survive background memory pressure without an OOM kill.

Do not debate Q5 until you have a device with 12 GB RAM.

Step 2 — Memory-Map Your Weights

Do not load model weights into the heap. Here is the minimal setup to get this working:

val weightFile = File(modelDir, "qwen3.5-9b-q4_k_m.gguf")
val channel = FileInputStream(weightFile).channel
val mappedBuffer = channel.map(
    FileChannel.MapMode.READ_ONLY,
    0L,
    channel.size()
)
// Pass mappedBuffer directly to llama.cpp JNI binding
nativeLoadFromMappedBuffer(mappedBuffer)
Enter fullscreen mode Exit fullscreen mode

The OS page cache handles eviction gracefully under memory pressure. Cold-start latency drops from 3–4 seconds to under 800ms because the kernel only faults in pages touched during the actual forward pass.

Step 3 — Split Delegation by Operator Type

The docs do not mention this clearly, but llama.cpp's default "auto" mode often leaves performance on the table. Mixed delegation is the right call:

  • NNAPI — handles quantized INT8 matrix multiplications efficiently on the Hexagon DSP and manages power budget better during sustained sessions. Route MLP layers here.
  • Vulkan compute — lower driver overhead for attention operations and handles irregular memory access patterns more gracefully. Route the attention mechanism here.

Profile with Android GPU Inspector before committing. SoC vendor implementations vary enough that assumptions will cost you latency.

Step 4 — Chunk-Prefill Batching for Sub-400ms TTFT

Time-to-first-token is dominated by prompt prefill. Process the prompt in 64-token batches with double-buffering on weight reads:

val CHUNK_SIZE = 64
val promptTokens = tokenizer.encode(userPrompt)

promptTokens.chunked(CHUNK_SIZE).forEach { chunk ->
    nativePrefillChunk(chunk.toIntArray())
}
// First decode step now starts from warm KV cache
val firstToken = nativeDecode()
Enter fullscreen mode Exit fullscreen mode

This trades roughly 10–20% higher total prefill time for dramatically lower perceived latency. Users experience responsiveness; wall-clock throughput is a secondary metric.

Gotchas

android:largeHeap="true" will silently kill your mmap. On 32-bit ABI targets, largeHeap expands the heap address space region and directly competes with your mmap virtual address range. The model mmap can fail with ENOMEM even when physical RAM is available. The default is false — verify yours explicitly rather than assuming. Even on 64-bit targets, the expanded heap increases GC pressure during inference.

Thermal throttle window is real. The Snapdragon 8 Gen 3 sustains full performance for roughly 8–12 minutes before throttling. Design your session length expectations accordingly, especially for production chat interfaces.

GGUF file size and runtime RSS are not the same. mmap loads pages lazily, but the kernel still maps the full virtual address range. Always measure RSS, not file size, when evaluating your memory budget.

Conclusion

Q4_K_M, mmap loading, mixed NNAPI/Vulkan delegation, and chunk-prefill batching — that is the complete stack for production Qwen3.5-9B on Android. Each decision has a concrete reason behind it rooted in the hardware constraints of Snapdragon 8 Gen 3. Start here, profile on your specific target device, and do not optimize further until the data tells you to.

Resources:

Top comments (0)