Benchmark Cloud LLM Calls Across Mobile Lifecycle Conditions Without Paying for the Backend

#mobile #testing #ai #android

On-device inference gets most of the attention in mobile AI discussions, but plenty of shipped features still call a hosted model. The problem is that most "should we call a cloud model?" decisions are made from a laptop on office Wi-Fi, and the answer changes the moment your app is backgrounded mid-request, the user revokes network access via a data-saver toggle, or the device switches from Wi-Fi to a congested cell tower.

This post is a reproducible test harness for measuring cloud LLM behavior under real mobile lifecycle conditions, plus a decision table for when a remote call is acceptable at all. I ran it against a free hosted endpoint, so the whole experiment costs nothing to repeat. Everything below is labeled as either measured on my device or a proposed step you should run on yours.

Environment

Device: Pixel 7 (Tensor G2), Android 14, battery at 62%, battery saver off unless noted
App: minimal Kotlin client, OkHttp 4.12, coroutines, targetSdk 34
Backend: a free hosted model endpoint (details below)
Network: Wi-Fi, then forced LTE via airplane-mode toggle, then a throttled profile via Android emulator network settings for the 3G comparison

The free backend problem

Benchmarking cloud LLM calls usually means burning API credits just to learn that your timeout handling is wrong. For this experiment I used MonkeyCode, which currently offers free model access and a free server option, so I could stand up a throwaway endpoint and point the Android client at it without a billing account.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I have not verified quotas, model lineup, or how long the free tier stays free, so treat it as a zero-cost test backend, not infrastructure you should depend on. The harness below works against any OpenAI-compatible endpoint — swap the base URL and key and nothing else changes.

The harness: measure the lifecycle, not the happy path

The interesting failures happen at lifecycle transitions, so the harness drives transitions explicitly. This is Kotlin for Android; the same structure ports to iOS with URLSession and background tasks.

class LifecycleBenchmark(private val client: OkHttpClient) {

    data class Sample(
        val scenario: String,
        val ttftMs: Long?,      // time to first token, null if failed
        val totalMs: Long?,
        val outcome: Outcome
    )

    enum class Outcome { COMPLETED, CANCELLED_BY_SYSTEM, TIMEOUT, STALE_ON_RESUME }

    suspend fun run(prompt: String, scenario: String): Sample {
        val start = SystemClock.elapsedRealtime()
        var ttft: Long? = null
        return try {
            val call = client.newCall(buildRequest(prompt))
            call.execute().use { resp ->
                resp.body!!.byteStream().bufferedReader().useLines { lines ->
                    lines.forEach { line ->
                        if (ttft == null && line.isNotBlank())
                            ttft = SystemClock.elapsedRealtime() - start
                    }
                }
            }
            Sample(scenario, ttft, SystemClock.elapsedRealtime() - start, Outcome.COMPLETED)
        } catch (e: IOException) {
            Sample(scenario, ttft, null, classify(e))
        }
    }

    private fun classify(e: IOException) = when {
        e is java.net.SocketTimeoutException -> Outcome.TIMEOUT
        e.message?.contains("Canceled") == true -> Outcome.CANCELLED_BY_SYSTEM
        else -> Outcome.STALE_ON_RESUME
    }
}

The STALE_ON_RESUME case is the one most apps get wrong: the request was in flight, the app went to the background, the OS suspended or killed the socket, and on resume you hold a response object that will never complete. The harness logs it as its own outcome so you can count how often it actually happens.

Test matrix

Run each row at least 10 times. Record device, OS build, battery level, and network state alongside every sample.

#	Scenario	Transition under test	Expected observation
1	Foreground, Wi-Fi, screen on	None (baseline)	Stable TTFT distribution
2	Background mid-request	Home button 2s after request starts	Socket killed or response delayed until resume
3	Wi-Fi → LTE handoff	Toggle airplane mode mid-stream	Stream breaks; measure how OkHttp reports it
4	Throttled 3G profile	Emulator network throttling	TTFT inflates; find your real timeout floor
5	Battery saver on	Enable before request	Background network deferred; request may queue
6	Permission-adjacent loss	Revoke then restore network access via data-saver exemption	App sees failure with no signal why
7	Process death + restore	"Don't keep activities" toggle mid-request	Request gone; does your UI recover or hang?

Row 7 is the one that decides whether your feature is shippable. If the answer is "the spinner hangs forever," you don't have a latency problem, you have a state-recovery problem, and no amount of model tuning fixes it.

What I measured vs. what you should verify

Measured on the Pixel 7 above: baseline TTFT on Wi-Fi was consistent enough to be useful; the background-mid-request scenario produced CANCELLED_BY_SYSTEM in most runs, which confirms that a streaming request cannot be trusted to survive backgrounding on stock Android 14 without a foreground service or WorkManager handoff. The Wi-Fi → LTE handoff broke the stream every time — no silent recovery.

I am deliberately not publishing the raw numbers here, because TTFT against a free shared endpoint reflects that endpoint's current load, not a property of your future production backend. The numbers that matter are yours, on your device class, against your endpoint. If you want a zero-cost starting point for that, MonkeyCode's free model access and free server are enough to run this matrix today; for paid production planning, re-run the same matrix against your real provider before drawing conclusions.

Decision table: is a cloud call acceptable for this feature?

Feature property	Cloud call OK?	Why
Result needed within one foreground session	Yes	Lifecycle risk is bounded
Must work offline or in airplane mode	No	On-device or queued-only
Streams for >10s and user may background app	Risky	Needs foreground service or resumable design
Input is sensitive (health, children, journal)	Prefer on-device	Privacy and backup-exclusion concerns dominate
Failure must be silent and auto-recovered	Only with WorkManager retry + idempotency key	Otherwise duplicates or hangs

Limitations and who should not use this approach

Free hosted endpoints have unknown rate limits and no latency SLA. Use them to validate your handling of failure, not to estimate production p95.
This harness tests one device and one OS. iOS background execution rules differ substantially; port before generalizing.
I did not measure energy impact here. TTFT under battery saver (row 5) hints at it, but a real claim needs a power profiler run.
If your feature requires guaranteed delivery, deterministic cost per request, or data residency, a free shared test backend is the wrong tool from day one.

Ask

If you run this matrix, post your row 2 and row 7 outcomes with device, OS, and network state: did the in-flight request recover, restart, or silently disappear? That pair of rows tells you more about shippability than any benchmark average.