Wiring Android's ExecuTorch Runtime to Compose

---
title: "Wiring ExecuTorch to Jetpack Compose: Streaming LLaMA 3.2 on Android"
published: true
description: "Memory-map your .pte model, configure XNNPACK for ARM CPUs, and pipe token-by-token output into a Kotlin StateFlow. The setup that delivers 12+ tokens/sec without GC pauses."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/executorch-compose-llama-android
---

## What We Are Building

By the end of this tutorial you will have a working on-device inference pipeline: a 4-bit quantized LLaMA 3.2 3B model loaded via memory-mapped I/O, an XNNPACK delegate tuned for sustained ARM throughput, and token-by-token output flowing into a Jetpack Compose UI through a `StateFlow`. Target numbers on Snapdragon 8 Gen 3: 12–15 tokens/sec sustained, cold-start under 2.2 seconds.

Get any one of three things wrong — model loading strategy, thread pinning, or the coroutine dispatcher — and you will hit GC pauses, thermal throttling, or out-of-order token rendering. Let me show you a pattern I use in every project.

---

## Prerequisites

- Android project targeting API 26+
- ExecuTorch Android AAR added to your Gradle dependencies
- A `.pte` (ExecuTorch Program) file — 4-bit quantized LLaMA 3.2 3B
- Basic familiarity with Kotlin coroutines and Jetpack Compose

---

## Step 1: Memory-Map the Model File

ExecuTorch uses `.pte` files — not ONNX, not TFLite. This format is designed for memory-mapped I/O. The gotcha that will save you hours: do not load the model into a `ByteArray` with standard Java I/O. That forces a 2–4 GB weight file onto the JVM heap and triggers constant GC. Use `FileChannel.map()` instead.

kotlin
val fd = context.assets.openFd("llama32_4bit.pte")
val channel = FileInputStream(fd.fileDescriptor).channel
val mappedBuffer = channel.map(
FileChannel.MapMode.READ_ONLY,
fd.startOffset,
fd.declaredLength
)
val module = Module.load(mappedBuffer) // stays in native memory

`mmap` keeps weights in native memory, outside the GC-managed heap. On an 8 GB device, this alone cuts inference-start latency 40–60% versus heap-loaded alternatives.

---

## Step 2: Configure the XNNPACK Delegate

The docs do not mention this, but the default XNNPACK config leaves significant performance on the table. Here is the minimal setup to get this working at production throughput:

kotlin
val xnnpackOptions = XnnpackDelegate.Options.Builder()
.setNumThreads(Runtime.getRuntime().availableProcessors() / 2)
.setFlags(XnnpackDelegate.FLAG_ENABLE_SUBGRAPH_RESHAPING)
.setWorkspaceSize(256 * 1024 * 1024L)
.build()

The thread count is deliberate. Half-core pinning sustains 10–14 tokens/sec across a full multi-turn session. Full-core utilization triggers aggressive thermal management on Snapdragon and Dimensity chips within 30–90 seconds — you get a spike then a cliff.

---

## Step 3: Stream Tokens via StateFlow

ExecuTorch emits tokens synchronously on the inference thread. The correct bridge to Compose is a `MutableStateFlow` updated from a single-threaded coroutine dispatcher — not `Dispatchers.Default`, which reorders emissions under load.

kotlin
class LlamaInferenceEngine(private val module: LlamaModule) {

private val inferenceDispatcher = Executors.newSingleThreadExecutor()
    .asCoroutineDispatcher()

private val _tokenStream = MutableStateFlow("")
val tokenStream: StateFlow<String> = _tokenStream.asStateFlow()

fun generate(prompt: String, scope: CoroutineScope) {
    scope.launch(inferenceDispatcher) {
        module.generate(prompt) { token ->
            _tokenStream.update { current -> current + token }
        }
    }
}

}

In your Composable, use `collectAsStateWithLifecycle`. This stops collection when the screen leaves the composition — critical for a long-running inference job.

kotlin
@Composable
fun InferenceScreen(engine: LlamaInferenceEngine) {
val output by engine.tokenStream.collectAsStateWithLifecycle()
Text(
text = output,
modifier = Modifier.verticalScroll(rememberScrollState())
)
}

---

## Step 4: Set Your KV Cache Budget at Load Time

The KV cache is where most on-device LLM integrations fall apart at scale. `max_seq_len` must be set once at load time — it is immutable per runner instance.

kotlin
val runner = LlamaRunner.Builder()
.setModelPath(modelPath)
.setMaxSeqLen(2048)
.build()

Exceeding this silently truncates context. A 4-bit LLaMA 3.2 3B at 2048 tokens requires approximately 180 MB of KV cache on top of model weights. Plan your memory budget before you set this value.

---

## Gotchas

**Heap-loading the model.** This is the single most common regression I see. Always use `FileChannel.map()`.

**Using all available cores.** Looks great in a 60-second benchmark, catastrophic after 90 seconds of real use. Pin to half.

**`Dispatchers.Default` for token emission.** Under inference load, this produces out-of-order token appends. One dedicated thread, one stream, no surprises.

**Setting `max_seq_len` per call.** It is a builder parameter, not a per-generation argument. Set it wrong once at construction and every subsequent call inherits the wrong context window.

---

## Conclusion

On-device LLM inference on Android is production-viable today with ExecuTorch — but only when the memory model, delegate configuration, and streaming architecture are all correct. Memory-map your weights, pin XNNPACK to half your cores, and route token emissions through a single-threaded dispatcher. The benchmark numbers hold across a full multi-turn session, not just a cold-start spike.

For the full ExecuTorch Android setup, the [official ExecuTorch docs](https://pytorch.org/executorch/) cover delegate registration and model export in detail.