Memory, not speed, is the hard part of running an LLM on a phone

Charles-Antoine Fournel — Tue, 22 Sep 2026 12:12:35 +0000

I ship an Android app (Onira) that generates a personalized hypnosis/relaxation
script on-device with Gemma 4 E2B, through LiteRT-LM, then narrates it with
on-device TTS. Nothing the user types, and nothing the model generates, ever
leaves the phone. This is the part that was actually hard to get right, and it
was not the part I expected.

Speed was fine. Memory nearly killed it.

A mid-range phone writes 300-500 words in tens of seconds — acceptable for a
relaxation app where narration masks generation latency. The real problem: the
runtime loads the 2.6GB model into the native heap and does not mmap it.
Measured on a Pixel 7 (7.6GB RAM), mid-generation:

~6GB PSS
zram 100% full
kswapd0 at 72% CPU

The process thrashes, generation crawls, and Android's low-memory killer takes
the app the instant it leaves the foreground. On a 6GB device it's worse.

Fix #1: stop holding the model once you're done needing it

The obvious mistake would be keeping the Engine/Conversation alive for the
whole session "just in case." But my output is consumed by TTS over the next
30-40 minutes, and for all of that time nothing needs the model resident. So:
close it the instant the last block is generated, and let the rest of the
session narrate with the model unloaded. That window is also, not
coincidentally, exactly when a user is most likely to background the app to do
something else — which is exactly when the low-memory killer was taking it.

The one deliberate exception: a mediaPlayback foreground service keeps the
process alive for the whole narration, started right when the user taps
"Begin Session." Without it, backgrounding the app during the first few
generating minutes (model still resident) got the whole session killed
outright. Android 12+ also refuses a foreground-service start from the
background — exactly the moment that request usually comes — so it has to
start earlier than you'd want, while the model is still loaded. The accepted
cost: the device may thrash and lmkd kills other background apps instead,
until the model unloads a few minutes in.

Fix #2: generate in blocks, not one call

A session is an ordered sequence of blocks — induction, deepening, metaphor,
suggestions, anchoring, repeat, emergence. Each is generated as its own turn
on the same Engine/Conversation, so later blocks stay thematically
consistent with earlier ones without re-stating prior text in the prompt.
Two consequences that matter more than the consistency:

Playback starts before the model finishes anything. Induction and emergence are static templates — neither depends on the category or the user's free text, so there's nothing to personalize — which means narration can start as soon as the model is loaded, without waiting on a single generation call. On a mid-range device the first real generation can take minutes; a user hearing a voice at second three is a different product than one staring at a spinner.
Failure is local, not fatal. A block that fails generation gets a short pre-written bridging line instead of aborting the session. The listener never notices; they just got a slightly shorter passage.

The body is capped at 6 blocks — a repeating cycle of
[deepening, metaphor, suggestions, anchoring]. This bound is what keeps any
single generation call short regardless of total session length, and it's
also exactly what the context window is sized against:

private const val MAX_BODY_BLOCKS = 6
When the target word budget doesn't fill a whole cycle, the scheduler keeps
the highest-priority block types and drops the rest — suggestions and
anchoring survive, metaphor goes first:

fun buildBodyPlan(targetBodyWords: Int): List<SessionBlockType> {
    val bodyBlocks = ((targetBodyWords + WORDS_PER_BLOCK_TARGET / 2) / WORDS_PER_BLOCK_TARGET)
        .coerceIn(1, MAX_BODY_BLOCKS)
    return buildList {
        repeat(bodyBlocks / CORE_CYCLE.size) { addAll(CORE_CYCLE) }
        addAll(partialCycle(bodyBlocks % CORE_CYCLE.size))
    }
}

Rounded to the nearest block, not floored — at the short end of the range,
flooring to whole cycles costs four blocks at once and undershoots the target by far more than rounding up overshoots it.

The thing I got wrong for a while: sizing by word count
My first version budgeted a fixed word count for the whole script — "4200
words ≈ 30 minutes at 140 wpm." It shipped ~90-minute sessions at default
settings. The bug: narration speed and the user's "pause between phrases"
setting (3-15s, applied at every clause boundary) swing real spoken duration
by more than 3x. A fixed word count cannot hold duration steady when the
seconds-per-word ratio itself varies that much per user.

The fix budgets from a target duration, converted to words using the user's
own playback settings:

private fun secondsPerWord(speechRate: Float, sentencePauseSeconds: Float): Double {
    val userScale = (speechRate.coerceAtLeast(0.1f) / REFERENCE_USER_RATE).toDouble()
    val wordsPerMinute = BASE_WORDS_PER_MINUTE * NARRATION_BASE_RATE * userScale
    val speakingSeconds = 60.0 / wordsPerMinute
    val pause = sentencePauseSeconds.coerceIn(3f, 15f)
    val pausePerClause =
        (SENTENCE_BOUNDARY_SHARE * pause + (1 - SENTENCE_BOUNDARY_SHARE) * NON_SENTENCE_PAUSE_SECONDS) / userScale
    return speakingSeconds + pausePerClause / AVG_WORDS_PER_CLAUSE
}

fun wordsForSeconds(targetSeconds: Int, speechRate: Float, sentencePauseSeconds: Float = 10f): Int =
    (targetSeconds / secondsPerWord(speechRate, sentencePauseSeconds)).toInt().coerceAtLeast(1)

AVG_WORDS_PER_CLAUSE and SENTENCE_BOUNDARY_SHARE aren't guesses — they're
measured off the shipped template corpus (~10k words): one clause boundary
every ~17 words, ~85% of them sentence ends that take the user's pause
setting; the rest (ellipsis, semicolon) stay on a fixed, shorter pause. Only
sentence boundaries scale with the user's setting, because that's the only
pause the setting is actually supposed to control.

The result: total session length is the thing held constant across every
speech-rate and pause-length combination a user can pick, instead of word
count — which is what actually matters for a relaxation session that's
supposed to run 30-40 minutes, not 90.

The honest costs of on-device
A 2.6GB first-run model download — by a wide margin the largest drop-off in
the funnel.
No server-side moderation layer in front of a model writing
psychologically-framed text for someone who typed in whatever's actually
on their mind. Handled with a deliberately high-recall keyword gate that
runs before the model is ever invoked — false positives (blocking a
benign message) are cheap here; false negatives are not.
Happy to go deeper on the safety gate design or the LiteRT-LM integration
specifically, if there's interest. The app is Onira
on the Play Store if anyone wants to see the output — but the above is the
part I think is worth discussing.

DEV Community: Charles-Antoine Fournel

Memory, not speed, is the hard part of running an LLM on a phone

Speed was fine. Memory nearly killed it.

Fix #1: stop holding the model once you're done needing it

Fix #2: generate in blocks, not one call