---
title: "Continuous Batching for On-Device LLM Inference on Android"
published: true
description: "Implement continuous batching in a local llama.cpp Android server — KV cache slot management, dynamic batch assembly, and queue architecture that holds p50 latency under concurrent load."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/continuous-batching-on-device-llm-android
---
## What We Are Building
By the end of this tutorial, you will have a working request scheduler, a per-sequence KV cache slot allocator, and a dynamic batch assembly loop — the three components you need to serve multiple concurrent LLM requests from a single llama.cpp instance on Android without your throughput falling off a cliff.
## Prerequisites
- Android project with llama.cpp integrated via JNI or a Kotlin/Native bridge
- Familiarity with Kotlin coroutines and `Channel`
- Basic understanding of how autoregressive token generation works
---
## The Problem Nobody Warns You About
Most teams treat the inference engine like a function call: one request in, wait for completion, next request in. That works at one user. At five concurrent users on the same device — think a local Android server powering multiple app features simultaneously — you hit a throughput cliff so steep it looks like a wall.
The culprit is **prefill starvation**. While a long generation decodes token-by-token, every new request sits idle. GPU/NPU compute is underutilized, KV cache memory sits half-empty, and your p50 latency balloons proportionally to queue depth.
The server-side world solved this years ago with **continuous batching** (also called iteration-level scheduling). The on-device world is only catching up now.
Here is the comparison that will make this concrete:
| Property | Static Batching | Continuous Batching |
|---|---|---|
| Scheduling unit | Full request | Single decode iteration |
| New request joins | After current batch completes | Next available iteration |
| KV cache allocation | Fixed at batch start | Dynamic per sequence slot |
| GPU utilization | Spiky, often <50% | Sustained, typically 70–90% |
| p50 latency under load | Grows linearly with queue | Near-flat to moderate concurrency |
| Implementation complexity | Low | Moderate |
With static batching, a 5-request queue behind a 500-token generation means request 5 waits for ~2,500 decode steps before its prefill even begins. With continuous batching, that same request joins the batch at the next iteration boundary.
---
## Step 1: The Request Queue
Let me show you a pattern I use in every project — decouple the queue from the inference loop with typed data and coroutine channels.
kotlin
data class InferenceRequest(
val id: String,
val prompt: String,
val maxTokens: Int,
val responseChannel: Channel
)
class RequestScheduler(private val maxConcurrent: Int = 4) {
private val pending = ArrayDeque()
private val active = mutableMapOf()
fun enqueue(request: InferenceRequest) {
pending.addLast(request)
tryPromote()
}
private fun tryPromote() {
while (active.size < maxConcurrent && pending.isNotEmpty()) {
val req = pending.removeFirst()
val slot = SlotAllocator.acquire() ?: return // KV cache full
active[req.id] = SequenceSlot(req, slot)
}
}
}
`maxConcurrent` is not arbitrary — it is bounded by your KV cache capacity. Exceed it and you either evict sequences mid-generation or OOM. Size it at initialization from available VRAM or shared memory, not at runtime.
---
## Step 2: Per-Sequence KV Cache Slot Manager
llama.cpp exposes `llama_kv_cache_seq_rm` and `llama_kv_cache_seq_cp` for sequence-level control. Here is the minimal setup to get this working:
kotlin
class SlotAllocator(private val totalSlots: Int) {
private val free = ArrayDeque((0 until totalSlots).toList())
private val inUse = mutableSetOf()
@Synchronized
fun acquire(): Int? = free.removeFirstOrNull()?.also { inUse.add(it) }
@Synchronized
fun release(slot: Int) {
inUse.remove(slot)
free.addLast(slot)
// Signal scheduler via coroutine channel
}
}
Each active sequence owns exactly one slot. When a generation completes or is cancelled, `release()` fires and the scheduler immediately promotes the next pending request. No idle cycles.
---
## Step 3: Dynamic Batch Assembly
At each decode step, re-assemble the batch from all active sequences:
kotlin
suspend fun runBatchStep(active: Map) {
val batch = llama_batch_init(active.size, 0, 1)
active.values.forEachIndexed { i, seq ->
llama_batch_add(batch, seq.nextToken, seq.position, intArrayOf(seq.slot), i == active.size - 1)
}
llama_decode(ctx, batch)
active.values.forEach { seq ->
val logits = llama_get_logits_ith(ctx, seq.batchIndex)
val token = sample(logits, seq.samplingParams)
seq.emit(token)
if (token == eosToken || seq.length >= seq.request.maxTokens) {
seq.complete()
}
}
llama_batch_free(batch)
}
The loop runs continuously as long as any sequence is active. New requests slot in; completed sequences drain out. The engine never idles waiting for a single long generation to finish.
---
## Gotchas
**Size the slot allocator at startup, not at runtime.** Calculate maximum concurrent sequences from available KV cache memory before accepting any requests. Fail fast rather than evict mid-generation — eviction destroys the sequence state you spent compute building.
**Decouple your queue from your batch loop using coroutines.** The `RequestScheduler` and the decode loop should communicate through channels, not shared mutable state. This keeps cancellation and backpressure composable.
**Benchmark p50, not just throughput.** The docs do not mention this, but continuous batching trades marginal single-request latency for better tail behavior under load. Measure at the 50th and 95th percentile under realistic concurrent load — that is where you see the real gains.
**The sweet spot is 3–6 concurrent sequences.** Below that, batch management overhead can slightly inflate single-request latency. Above ~8 on a mobile GPU, you hit memory pressure before CPU scheduling becomes the bottleneck. Tune `maxConcurrent` empirically against your target device tier.
---
## Conclusion
Naive serial inference on Android will not survive concurrent load. The three components you built here — the request scheduler, the slot allocator, and the dynamic batch loop — give you iteration-level scheduling with per-sequence KV cache control. That is the same architecture that keeps server-side LLM inference healthy under concurrent users, brought to your Android process.
Your next step is profiling `maxConcurrent` on your actual target device. Start at 4, measure p50 and p95 under 5 simultaneous requests, and adjust from there.
**Further reading:**
- [llama.cpp batching documentation](https://github.com/ggerganov/llama.cpp/blob/master/docs/batched.md)
- [Continuous Batching: How HuggingFace TGI handles it](https://huggingface.co/docs/text-generation-inference/conceptual/scheduling)
- [Android NNAPI and GPU delegate memory constraints](https://developer.android.com/ndk/guides/neuralnetworks)
Top comments (0)