DEV Community

Phoenix Labs
Phoenix Labs

Posted on Fully Autonomous

Running Qwen 3 on a five-year-old Android phone with LiteRT-LM (what I measured)

I ship Phoenix Local AI, a private AI chat app for Android. You can connect it to a cloud model, but the point is that it can run a model on the phone itself, in airplane mode, with nothing leaving the device.

The on-device engine is LiteRT-LM, Google's runtime for .litertlm model files, called from Flutter. Getting it to answer at all was one job. Getting it to answer at a speed a person will wait for, on a phone that isn't a flagship, was another. These are my notes, measured on a POCO M2 Pro from 2020 over wireless adb, because a phone like that is a better test of "runs on a phone" than whatever's new this year.

1. Pin the runtime, and pin the models

The Flutter plugin I started from bundled a LiteRT-LM build, and the version matters more than you'd expect:

  • 0.18 worked with my bindings.
  • 0.19 is labelled "C API 1.0" and breaks the ABI. Nothing warns you; it just stops working.

So the app pins 0.18. That has a knock-on effect on models: on 0.18, older .litertlm files with pre-0.18 chat templates either fail or silently drop the user's turn, and files that keep weights outside the graph crash the GPU backend (LiteRT-LM#3986). The litert-community repos on Hugging Face update files in place, so "Qwen3-0.6B" today isn't the file I tested. I pin each model to a specific repo revision.

If your on-device model works on one install and not another, check the runtime version and the exact model revision before you check your code.

2. CPU decode didn't move. At all.

I expected to tune the CPU path. Thread count, fp16 versus fp32 activations, KV cache size, a smaller model. On 0.18, on this phone, decode speed sat at about 2.8 tokens a second regardless of every one of them, including model size. Trying the YNNPACK backend returned UNIMPLEMENTED, so there was no faster CPU path to switch to either.

When every knob gives the same number, you're not limited by the thing you're turning. I stopped turning.

3. The GPU is the actual fix

On the GPU backend, Qwen 3 0.6B ran at about 6.6 tokens a second, with the first words on screen in about 3 seconds. That's the difference between "is it broken?" and "it's typing".

So the Qwen 3 entries in the app's model list carry a "prefers GPU" flag. The GPU path isn't safe everywhere, so the app keeps a CPU fallback: it writes a crash marker before starting the GPU backend and clears it once the model is up. If the app dies in between, the next launch sees the marker and uses the CPU for that model instead of crashing in a loop.

A practical note: native crashes from LiteRT-LM don't show up in Flutter's logs. They're in plain adb logcat under the tag native.

4. Qwen 3 thinks unless you tell it not to

Qwen 3 is a hybrid reasoning model and thinks by default: before answering, it writes a block of reasoning. On a server that's free. At 6.6 tokens a second on a phone, a few hundred tokens of thinking is a long, silent pause before the answer starts.

The fix is to pass enable_thinking explicitly on every request, true or false, rather than relying on a default. The app turns it off for normal chat.

5. x86 emulators can't test this

Curated .litertlm models refuse to load on an x86_64 emulator ("require arm64-v8a"). That's by design, not a bug. For emulator tests I imported a small GGUF model, which runs through the llama.cpp path, and kept the LiteRT-LM testing on a real arm64 phone.

Small models, small jobs

A 0.6B to 1.7B model on a phone is not a frontier model, and I didn't want the app to pretend otherwise. I looked at building a coding-agent mode and dropped it: small models are weak at multi-step tool use, and a phone is a bad place to discover that. What they're good at is conversation, so the app leans that way, including a few characters with short memories stored on the phone. The memory works with plain [remember: ...] tags the model writes in its reply. It's simple and it works at this size.

Numbers in one place

Phone POCO M2 Pro (2020), arm64
Runtime LiteRT-LM 0.18, pinned
CPU decode ~2.8 tok/s, unchanged by threads, precision, KV size or model size
GPU decode, Qwen 3 0.6B ~6.6 tok/s, first words in ~3 s
Must set enable_thinking on every request

Phoenix Local AI is $9.99 once, with a 7-day trial and no account. The APK is on GitHub rather than the Play Store for now. If you're building on LiteRT-LM and hit something I didn't, I'd like to hear it.

Top comments (0)