Wiring Android's MediaPipe LLM Inference API to a Streaming UI

---
title: "Streaming LLM Tokens in Android with Jetpack Compose: Back-Pressure and the Frame Budget"
published: true
description: "Wire MediaPipe's LLM Inference API to Jetpack Compose using Kotlin Flow back-pressure, correct buffer sizing, and a 16ms recomposition budget that keeps mid-range devices jank-free at 20+ tokens/sec."
tags: kotlin, android, mobile, architecture
canonical_url: https://mvpfactory.co/blog/streaming-llm-tokens-android-compose
---

What You Will Build

By the end of this tutorial you will have a production-ready pattern for streaming on-device LLM token output into a Compose Text component — without dropped tokens, without jank on mid-range hardware, and without the recomposition storm that trips up most first implementations.

Here is the gotcha that will save you hours: MediaPipe does not give you a suspend function. It gives you a callback on a thread you do not own. The naive bridge will work fine on your Pixel 9 and silently drop frames on your users' Helio G88.

---

Prerequisites

• Android project with MediaPipe LLM Inference API configured
• Kotlin coroutines and Flow basics
• Jetpack Compose set up in your module

---

Step 1 — Understand the Streaming Contract MediaPipe Gives You

MediaPipe exposes token output like this:

inferenceSession.addQueryChunk(prompt)
inferenceSession.generateAsync(
    { partialResult, done ->
        onToken(partialResult, done)
    }
)

Two constraints that matter:

1. The callback fires on an internal MediaPipe thread — not main, not a dispatcher you control
2. There is no back-pressure signal — the model pushes tokens as fast as it decodes them

Ignore these two constraints and you get frame drops that only appear under thermal throttling on mid-range hardware. That is exactly the device your users own.

---

Step 2 — Bridge the Callback to Kotlin Flow

The cleanest bridge is a Channel with a bounded buffer. Here is the minimal setup to get this working:

fun LlmInferenceSession.tokenFlow(prompt: String): Flow<String> =
    channelFlow {
        val buffer = Channel<String>(capacity = 64, onBufferOverflow = BufferOverflow.SUSPEND)

        addQueryChunk(prompt)
        generateAsync { partial, done ->
            runBlocking { buffer.send(partial) }
            if (done) buffer.close()
        }

        for (token in buffer) send(token)
    }.flowOn(Dispatchers.Default)

A capacity of 64 covers roughly 2–3 seconds of burst at 25 tok/sec before SUSPEND kicks in. runBlocking inside the callback is intentional — it blocks the producer when the consumer falls behind, without dropping tokens. The docs do not mention this, but this is the only safe primitive here.

---

Step 3 — Solve the Recomposition Budget Problem

Let me show you a pattern I use in every project. Compose does not batch state writes from collectAsState the same way it batches layout passes. Each emission can schedule an independent recomposition.

Device tier	Vsync budget	Max safe recompositions/sec	Safe token batch window
Flagship (Snapdragon 8 Gen 3)	8.3ms (120Hz)	~60	16ms
Mid-range (Snapdragon 6 Gen 1)	16ms (60Hz)	~30	33ms
Budget (Helio G88)	16ms (60Hz)	~15	66ms

At 25 tokens/sec on a budget device, one recomposition per token misses frames. The fix is chunked emission with a time window:

fun Flow<String>.chunkByTime(windowMs: Long = 80L): Flow<String> =
    flow {
        val accumulator = StringBuilder()
        var lastEmit = System.currentTimeMillis()

        collect { token ->
            accumulator.append(token)
            val now = System.currentTimeMillis()
            if (now - lastEmit >= windowMs) {
                emit(accumulator.toString())
                accumulator.clear()
                lastEmit = now
            }
        }
        if (accumulator.isNotEmpty()) emit(accumulator.toString())
    }

An 80ms window caps recompositions at ~12/sec — safely below the budget-device ceiling.

---

Step 4 — Wire It Into Compose

@Composable
fun StreamingResponse(session: LlmInferenceSession, prompt: String) {
    val text by remember(prompt) {
        session.tokenFlow(prompt)
            .chunkByTime(windowMs = 80L)
            .runningReduce { acc, chunk -> acc + chunk }
    }.collectAsState(initial = "")

    Text(text = text, modifier = Modifier.fillMaxWidth())
}

runningReduce accumulates the full response string without holding a mutable StringBuilder in composition scope. remember(prompt) restarts the flow cleanly on new prompts, avoiding stale emissions.

---

Gotchas

Do not use MutableStateFlow to bridge the MediaPipe callback. Only a Channel with BufferOverflow.SUSPEND prevents dropped tokens from a native producer thread.

Tune to your minimum supported device, not your desk Pixel. 80ms is a safe default. Tighten to 33ms only if your floor is mid-range 60Hz and you have profiler data to back it up.

Before shipping, verify in Android Studio's Compose Layout Inspector: skipped frames should be zero during active streaming; recomposition count on Text should be ≤15/sec on your lowest-supported device.

---

Conclusion

The full pattern is three composable pieces: a Channel-based callback bridge, a time-windowed chunking operator, and a stateless composable that folds chunks with runningReduce. Each piece solves one specific constraint — thread ownership, back-pressure, and frame budget — and they stay decoupled enough to test independently.

Profile on your lowest-supported hardware before shipping. Flagship benchmarks will lie to you.