MediaPipe LLM Inference is maintenance-only: start on LiteRT-LM (and add this ABI check)

Running a language model on the phone itself is the most private option you have and the most demanding. The model file is large, it needs memory while it runs, and it turns battery into words. In return, the feature works in airplane mode on most recent 64-bit phones, and you choose the model.

LiteRT-LM replaces the MediaPipe LLM Inference API

For the last couple of years the usual way to do this was MediaPipe's LLM Inference API. Google's MediaPipe documentation now carries a notice that the LLM Inference API (Android, iOS and Web) is in maintenance-only mode, that new features and optimizations are focused on LiteRT-LM, and that projects should migrate.

New code should start on LiteRT-LM.

LiteRT-LM is Google's runtime for language models on top of LiteRT. Its Kotlin API is small:

• an Engine loads a .litertlm model
• a Conversation holds the chat state
• sendMessageAsync streams the reply as a Kotlin Flow

The current release on Google Maven at the time of writing (October 2026) is 0.18.0, still below 1.0:

dependencies {
    implementation("com.google.ai.edge.litertlm:litertlm-android:0.18.0")
}

Getting a model

LiteRT-LM reads .litertlm files. Google points to the LiteRT Community organization on Hugging Face for ready-made ones, including Gemma builds.

The author ships Android apps on Google Play as Hartmann Studios, and his ollama-android-client keeps a curated catalog in LocalModelCatalog.kt. Its rule: prefer the smallest CPU-friendly quantization (int4 or q8) over f32 "so downloads stay reasonable on mobile data and the model actually fits in RAM". Two entries from that file:

Catalog entry	File	Approx. size
Gemma 3 270M IT (q8)	gemma3-270m-it-q8.litertlm	304 MB
Gemma 3 1B IT (int4)	gemma3-1b-it-int4.litertlm	584 MB

For the rest of this article, assume the file is in your app's private storage and you have its path.

Engine and conversation

The documentation is explicit that engine.initialize() can take a significant amount of time (it gives "up to 10 seconds" as an example) and should run on a background thread or coroutine. Load the engine once and keep it. Create a new conversation per chat.

class LocalLlm(private val context: Context, private val modelPath: String) {
    private var engine: Engine? = null
    private val lock = Mutex()

    suspend fun engine(): Engine = lock.withLock {
        engine ?: withContext(Dispatchers.Default) {
            Engine(EngineConfig(
                modelPath = modelPath,
                backend = Backend.CPU(),
                cacheDir = context.cacheDir.path, // speeds up later loads
            )).also { it.initialize() }
        }.also { engine = it }
    }

    fun release() { engine?.close(); engine = null }
}

A conversation takes an optional system instruction, earlier messages and sampling settings:

val config = ConversationConfig(
    systemInstruction = Contents.of("You are a concise, friendly assistant."),
    initialMessages = listOf(
        Message.user("What is a homelab?"),
        Message.model("A few computers at home that run your own services."),
    ),
    samplerConfig = SamplerConfig(topK = 10, topP = 0.95, temperature = 0.8),
)

Streaming a reply as a Flow

sendMessageAsync(text) without a callback returns a Flow<Message>, which the documentation recommends for coroutine users. In the book this sits behind a small TextGenerator interface so the UI does not care which backend answers:

override fun generate(messages: List<ChatMessage>): Flow<String> = flow {
    val history = messages.dropLast(1).filter { it.role != Role.System }
        .map {
            if (it.role == Role.User) Message.user(it.text)
            else Message.model(it.text)
        }
    val system = messages.firstOrNull { it.role == Role.System }?.text
    val config = ConversationConfig(
        systemInstruction = system?.let { Contents.of(it) },
        initialMessages = history,
    )
    llm.engine().createConversation(config).use { conversation ->
        conversation.sendMessageAsync(messages.last().text)
            .collect { emit(it.toString()) }
    }
}.flowOn(Dispatchers.Default)

Use flowOn, not withContext, inside a flow { } builder. ollama-android-client's LiteRtChatService.kt documents why in a comment: a flow { } builder cannot emit from a different dispatcher than the one that collected it, and wrapping the body in withContext throws "Flow invariant is violated" at runtime.

The ABI check is not optional

This is the part that comes from a real crash report.

LiteRtChatService.kt notes that the litertlm-android AAR versions it uses only ship native libraries for arm64-v8a and x86_64, with no 32-bit ARM build. Phones without those ABIs threw UnsatisfiedLinkError the moment the native library loaded.

The fix has two parts. First, check Build.SUPPORTED_ABIS up front:

override suspend fun availability(): Availability =
    if (Build.SUPPORTED_ABIS.any { it == "arm64-v8a" || it == "x86_64" })
        Availability.Ready
    else Availability.Unavailable("This phone's processor is not supported")

Second, catch Throwable rather than Exception around engine creation. UnsatisfiedLinkError is an Error, not an Exception, and it slipped past the old catch.

GPU first, CPU as the safety net

The GPU backend is usually faster for language models, but it does not work on every phone or with every model. To use it, the LiteRT-LM Android guide says to declare the OpenCL-related native libraries in the manifest, marked as not required:

<application>
    <uses-native-library android:name="libvndksupport.so" android:required="false"/>
    <uses-native-library android:name="libOpenCL.so" android:required="false"/>
</application>

Then try GPU and fall back to CPU. ollama-android-client builds a list of backend attempts and, inside each, a list of history lengths to try: the full compacted history, then half, then one exchange, then none. Only when every rung fails does the user see an error.

Fitting the conversation into the context window

A language model on a phone has a small context window, and the conversation grows with every message. When the history no longer fits, generation fails. ollama-android-client trims history to a token budget before every request:

1. It estimates tokens cheaply as characters divided by four, plus a small overhead per message.
2. It allows history to use 55 percent of the model's token limit, leaving the rest for the new message and the reply.
3. It keeps the newest exchanges that fit, then adds the very first exchange back if there is room, then up to a couple of older exchanges that share keywords with the new message.

The chat itself stays in Room. Only the copy sent to the model is trimmed.

Releasing the model

A loaded model holds a large amount of native memory. Close the conversation when a chat ends (the use block above does this) and close the engine when the user leaves the AI part of the app or the system signals memory pressure:

class AssistantApp : Application() {
    lateinit var localLlm: LocalLlm

    override fun onTrimMemory(level: Int) {
        super.onTrimMemory(level)
        if (level >= ComponentCallbacks2.TRIM_MEMORY_BACKGROUND) {
            localLlm.release()
        }
    }
}

Checklist

• New code on LiteRT-LM, not the MediaPipe LLM Inference API.
• engine.initialize() off the main thread, once.
• flowOn, never withContext, inside flow { }.
• Check Build.SUPPORTED_ABIS for arm64-v8a or x86_64 before loading.
• Catch Throwable around engine creation.
• GPU first, CPU fallback, with the manifest entries marked not required.
• Release the engine on memory pressure.

This is adapted from my book On-Device AI for Android (Android the Right Way, Book 5). The free sample is here and the full book is here.

Drafted with AI help and edited by the author; based on his book.