DEV Community

Roronoa
Roronoa

Posted on

Measure Before You Choose: On-Device, Cloud API, or a Small Server for Your Mobile LLM Feature

#mobile #ai #android #performance

A few weeks ago I sketched three deployment options for a summarization feature in a React Native app: run a small model on-device, call a hosted model API, or put a small model on a cheap server I control. Every option had a plausible-sounding argument. None of the arguments survived contact with the question that actually matters on mobile: what does this cost in energy, latency, and failure modes on a real phone, on a real network?

This article is the measurement harness I built to answer that question honestly. It is a reproducible method, not a benchmark report — I am deliberately not publishing numbers here, because numbers from my test device would be useless for your device class, your model, and your network. What transfers is the procedure, the instrumentation, and the decision table at the end.

The three placements, framed as mobile constraints

Placement Latency driver Energy driver Failure mode to test
On-device inference Model size vs. SoC/NPU Sustained CPU/GPU/NPU load, thermal throttle App backgrounded mid-inference, low power mode
Hosted model API Network RTT + queueing Radio (cellular radio is expensive), TLS, retries Network switch (Wi-Fi → LTE), permission revoke
Small model on your own server Same as API + cold start Radio + keeping a server reachable Server sleep/restart, stale responses after reconnect

The third row is where free tiers matter for prototyping. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option, which makes it practical to stand up rows two and three without spending money during the measurement phase — a hosted model call for the API row, and a free server to host a small open model (or even just a mock endpoint) for the self-hosted row. Treat both as prototyping infrastructure, not a production commitment: verify current availability and limits yourself before designing around them, and do not assume any free tier is permanent.

The experiment design

One device, one task, three placements, four lifecycle conditions. The task must be identical across placements — same prompt, same expected output class — so the only variable is where execution happens.

Fixed conditions:

  • Device: one physical phone (I used a mid-range Android; note your exact model, SoC, and OS build)
  • Battery: start each run at 80%, unplugged, screen at fixed brightness
  • Network: first pass on stable Wi-Fi, second pass on cellular with a Wi-Fi → LTE handover mid-run
  • App state: foreground for baseline; then repeat with the app backgrounded for 10 s mid-task, and once more with low power mode enabled

The lifecycle matrix (kept small on purpose):

  1. Foreground, stable Wi-Fi — your baseline
  2. Foreground, Wi-Fi → LTE handover during the request
  3. Backgrounded for 10 s mid-task, then resumed
  4. Low power mode enabled

Four conditions is enough to expose the interesting differences. If a placement survives these, expand from there.

Instrumentation: energy on Android

Use batterystats as a coarse but repeatable energy diff. Reset, run one placement N times, dump:

adb shell dumpsys batterystats --reset
# run the task 20 times from the app, same prompt each time
adb shell dumpsys batterystats > placement_ondevice.txt
adb bugreport bugreport_placement_ondevice.zip  # for Battery Historian
Enter fullscreen mode Exit fullscreen mode

Record, per placement: estimated power use (mAh) attributed to your app's UID, mobile radio active time, and Wi-Fi radio active time. The radio-active-time row is where the on-device option usually wins, and where the two network options get separated by retry behavior. On iOS, use Xcode Instruments' Energy Log with the same fixed-brightness, fixed-charge protocol, and capture network traffic via the Network instrument.

Caveat: batterystats energy attribution is an estimate. It is good enough for relative comparison across placements on the same device, not for publishing absolute numbers.

Instrumentation: latency that survives backgrounding

Wall-clock timers lie when the app is backgrounded — the process may be suspended while your timer keeps conceptually running. Stamp events monotonically and persist them, so a killed process still leaves a trail:

data class TaskTrace(
    val taskId: String,
    val placement: String,          // "ondevice" | "api" | "server"
    val startedMs: Long,            // SystemClock.elapsedRealtime()
    var networkSentMs: Long? = null,
    var responseReceivedMs: Long? = null,
    var finishedMs: Long? = null,
    var interruptedBy: String? = null,  // "backgrounded" | "network_switch" | "process_death"
    var outcome: String? = null         // "completed" | "restarted" | "lost"
)
// Persist each mutation immediately (Room/DataStore), not in memory.
Enter fullscreen mode Exit fullscreen mode

Log a trace row per attempt, and at the end of each condition compute per-placement: p50/p95 end-to-end latency, and — more important — the outcome distribution: how many tasks completed, restarted, or silently disappeared. A placement that is fast but loses 30% of tasks when backgrounded loses to a slower one that always completes.

What to look for in the results

  • On-device row: does latency degrade across 20 consecutive runs? That is thermal throttling, and it means your single-run demo was flattering. Check whether low power mode caps performance hard enough to break your UX.
  • API row: compare radio active time against payload size. If you are sending large context, the upload cost on cellular can dominate.
  • Server row: if your free server sleeps or restarts, the first request after idle will show it. That is fine for prototyping — but measure the cold-start penalty explicitly and decide whether a keep-alive ping is worth its own battery cost. (If you use the MonkeyCode free server here, this cold-start measurement is exactly the thing to run before building on it.)
  • All rows, condition 2: a mid-request Wi-Fi → LTE handover. Does your HTTP client retry transparently, or does the user see a dead spinner? Reproducible kill-switch testing helps here — run the API/server rows against an endpoint you control (this is where having your own free server is genuinely useful: you can kill it on demand).

Decision table

Signal from your measurements Choose
On-device p95 acceptable, no thermal collapse, task loss ≈ 0 when backgrounded On-device — best privacy, no radio cost
On-device throttles or exceeds latency budget; payload small; network mostly stable Hosted API
You need control over the model/endpoint, can tolerate cold starts, want a kill-switch for failure testing Your own small server
Task loss > a few % in any network row and you cannot add client-side persistence + resume None of the network options until the resume path exists

Limitations and who should skip this

  • Single-device results do not generalize across device classes. Repeat on at least one low-end device before committing.
  • batterystats attribution is approximate; treat it as relative signal.
  • Free tiers (including MonkeyCode's free models and free server) are prototyping tools. Do not ship production traffic on assumptions about free-tier permanence, quotas, or SLAs I have not stated here because I cannot verify them for you.
  • If your feature requires guaranteed offline behavior, the network rows are disqualifying by definition — skip straight to on-device.
  • If you cannot persist task state client-side, no placement choice will save you from backgrounding losses; fix that first.

If you run this harness, I would like to compare notes: what device and OS build, which lifecycle transition you used, and whether interrupted tasks recovered, restarted, or silently disappeared. That outcome distribution is the number I trust least from any single setup — including mine.

Top comments (0)