Running an LLM fully offline on Android with Kotlin coroutines

#android #kotlin #llm #ai

A 0.5B-parameter chat model, 4-bit quantized, is about 490 MB. That fits on any recent phone, and llama.cpp runs it on the CPU with no GPU, no server and no API key. The hard part is not the inference. It's making it feel like a normal Kotlin API inside a normal Android app.

This post is about the three problems that took most of the time when I built llama-kotlin, a small Kotlin library over llama.cpp, and a fully offline chat app on top of it.

The API I wanted

val model = LlamaModel.load(File(filesDir, "qwen2.5-0.5b-instruct-q4_k_m.gguf"))

model.newSession().use { session ->
    session.chat(
        listOf(
            ChatMessage.system("You are a concise assistant."),
            ChatMessage.user("Explain coroutines in one sentence."),
        )
    ).collect { piece -> print(piece) }

    println("\n${session.lastStats?.tokensPerSecond} tokens/s")
}

chat() returns a cold Flow<String>. In Compose, that means the answer streams into the UI with a plain collect, and the ViewModel never touches JNI.

Problem 1: native code is not thread-safe, coroutines jump threads

A llama.cpp context (the KV cache plus the sampler) must not be used from two threads at the same time. Coroutines, on the other hand, can resume on any thread of a dispatcher. If two screens call the same session, or one call starts while another is still generating, you get corrupted state or a crash deep in native code.

The fix is boring, and that's the point: each session owns one dedicated thread.

fun newSessionDispatcher(): ExecutorCoroutineDispatcher =
    Executors.newSingleThreadExecutor { r -> Thread(r, "llama-session").apply { isDaemon = true } }
        .asCoroutineDispatcher()

Every native call goes through flowOn(dispatcher) or withContext(dispatcher), and a Mutex makes a second generation wait for the first instead of interleaving with it. The caller can be on Dispatchers.Main, IO, anything: the session is safe to call from any coroutine.

Problem 2: "Stop" has to actually stop

Users tap Stop. On a phone, a generation that keeps running in the background after Stop is burning battery and holding the CPU.

Because the generation is a flow { } builder, cancellation comes almost for free. The loop checks it before every token:

while (generated < maxTokens && !closing) {
    currentCoroutineContext().ensureActive()
    val bytes = engine.nextToken() ?: break
    generated++
    val text = decoder.push(bytes)
    if (text.isNotEmpty()) emit(text)
}

Cancel the coroutine that collects the flow and the loop throws at the next token, the finally block still records the stats, and the mutex is released for the next call. In the app, Stop keeps whatever was already written on screen.

Problem 3: tokens are bytes, not characters

This one surprised me. The first version printed � in the middle of answers, mostly with emoji and accented words.

A model token is a piece of UTF-8, not a character. A 4-byte emoji can be split across two tokens. If you decode each token on its own, the first half is invalid UTF-8 and becomes �.

The fix is a tiny streaming decoder: keep the unfinished tail of a character until the rest of its bytes arrive.

fun push(bytes: ByteArray): String {
    val all = pending + bytes
    val cut = completePrefixLength(all)   // where the last complete character ends
    pending = all.copyOfRange(cut, all.size)
    return String(all, 0, cut, Charsets.UTF_8)
}

completePrefixLength walks back over at most three continuation bytes (10xxxxxx), reads the lead byte to know how long that character should be, and cuts before it if it's not complete yet. It's 40 lines and it's fully unit tested on the JVM, no device needed.

Chat templates come from the model

Every model family formats a conversation differently (Qwen, Llama, Gemma, Phi each have their own special tokens). GGUF files carry their chat template inside, so chat() formats the messages with the template of the model you loaded. You keep a plain List<ChatMessage>, append the user's turn, call chat(), append the answer.

Testing without a phone

The JNI layer sits behind a small Engine interface. The session logic (streaming, limits, cancellation, stats, the UTF-8 decoder) runs against a fake engine in plain JVM unit tests, so CI checks it on every push without an emulator.

Try it

Library: ThalzeronLabs/llama-kotlin (Apache-2.0)
App: ThalzeronLabs/android-on-device-llm: the APK is in Releases. On first launch (on Wi-Fi) it downloads Qwen2.5-0.5B-Instruct, checks the SHA-256, and then works with no internet. Phones with more RAM can switch to SmolLM2 1.7B or Qwen2.5 1.5B, or import any .gguf.

If you run it, I'd like to know which phone you used and the tokens/s the app shows. There are a few "good first issue" tickets open if you want to contribute.

DEV Community