Benchmark Your Mobile LLM Fallback Across Network Loss Before You Ship It

Last month I was testing an LLM-powered note-summarization feature on a Pixel 7 (Android 14, March 2026 patch level) when I walked into an elevator mid-request. The app did not crash. It did something worse: it hung for 94 seconds, then showed a success toast with an empty summary. The network had dropped, the retry logic had silently swallowed the failure, and the UI layer never found out.

This is the class of bug that passes every emulator test and every Wi-Fi office demo. It only appears under real lifecycle transitions: network loss, backgrounding, Doze, permission changes. This post is a reproducible test plan for the fallback path between an on-device model and a remote model endpoint — the moment your app decides "the cloud is gone, what do I do now?"

The setup and the constraint

The feature shape I keep seeing in production apps:

1. Try the remote model first (better quality, no download cost).
2. Fall back to a small on-device model when the network is unavailable or the request times out.
3. Never lose the user's input, regardless of which path runs.

Step 2 is where apps break. Not because the fallback logic is hard, but because nobody tests it under the transitions that actually trigger it. To run these tests you need a remote endpoint you can hammer with failure scenarios without worrying about per-request cost. For this round of testing I used MonkeyCode's free model access hosted on its free server tier as the remote endpoint, which meant I could run hundreds of forced-failure requests without a billing meter influencing how many edge cases I bothered to test.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am treating its free tier as test infrastructure here, not evaluating it as a production backend — more on that in the limitations.

The test matrix

Every fallback test I run crosses one network transition with one app-state transition. These six cover most of the failure modes I have actually seen ship to users:

#	Network transition	App state	Expected behavior
1	Wi-Fi → airplane mode mid-request	Foreground	Timeout in ≤5s, fall back to on-device, input preserved
2	Airplane mode at request start	Foreground	Skip remote attempt entirely, go straight to on-device
3	Wi-Fi → cellular handoff mid-request	Foreground	Request survives or cleanly retries once, no duplicate submission
4	Request sent, app backgrounded, network drops	Background → foreground on return	No phantom success; result reconciles on resume
5	Doze/adaptive battery restricts network	Background	Work deferred, not silently failed; user notified or state persisted
6	Remote returns 5xx (not a network failure)	Foreground	Treated as fallback-worthy, not retried forever

Row 4 is the one that produced my elevator bug. The request's callback fired while the app was backgrounded, the response was an error, and the error handler assumed a foreground UI existed.

Forcing the transitions reproducibly

You cannot test row 1 by walking into elevators. Use adb to script the transitions:

# Start the request, then kill connectivity 800ms later
adb shell svc wifi disable
adb shell svc data disable
sleep 0.8  # adjust to land mid-request for your latency profile

# Restore for the next run
adb shell svc wifi enable

For row 5, force Doze without waiting for the real thing:

adb shell dumpsys deviceidle force-idle
adb shell dumpsys battery unplug
# ... run scenario ...
adb shell dumpsys deviceidle unforce
adb shell dumpsys battery reset

Wrap each scenario in a script that (a) starts a request via a deep link or UI Automator, (b) applies the transition at a fixed offset, (c) captures logcat filtered to your networking and inference tags, and (d) screenshots the final UI state. The screenshot matters — "no crash" is not "correct behavior," as my empty success toast proved.

The fallback gate, in Kotlin

The core artifact is a single decision point that owns the remote-vs-local choice and, critically, owns the user's input until a result exists. This is simplified from my test harness (strip the logging before shipping):

sealed class SummaryResult {
    data class Remote(val text: String) : SummaryResult()
    data class OnDevice(val text: String) : SummaryResult()
    data class Failed(val preservedInput: String) : SummaryResult()
}

suspend fun summarize(
    input: String,
    remote: RemoteSummarizer,
    local: OnDeviceSummarizer,
    connectivity: ConnectivityProbe,
): SummaryResult {
    // Row 2: don't even attempt remote with no network.
    if (!connectivity.isUsable()) {
        return runOnDevice(local, input)
    }

    return try {
        withTimeout(5_000) {
            SummaryResult.Remote(remote.summarize(input))
        }
    } catch (e: Exception) {
        when (e) {
            is TimeoutCancellationException,
            is IOException,          // rows 1 and 3
            is Http5xxException -> { // row 6
                Log.i("Fallback", "remote failed: ${e::class.simpleName}, going on-device")
                runOnDevice(local, input)
            }
            else -> SummaryResult.Failed(preservedInput = input)
        }
    }
}

private suspend fun runOnDevice(
    local: OnDeviceSummarizer,
    input: String,
): SummaryResult = try {
    SummaryResult.OnDevice(local.summarize(input))
} catch (e: Exception) {
    SummaryResult.Failed(preservedInput = input)
}

Two things to notice:

• Failed carries the input. The UI can always offer a retry or at minimum not silently discard what the user typed. The empty-toast bug existed because the error path returned Unit.
• The result is typed by source. Your analytics should distinguish Remote from OnDevice — if 40% of your real-world requests land on-device, that is a product-quality signal, not just an engineering detail.

For row 4 (backgrounding), do not deliver results via a callback that assumes an alive UI. Write the result to a DataStore or database from a WorkManager task or a coroutine scoped to the application, and let the UI observe it. Then "app backgrounded during request" stops being a special case at all.

Measuring, not vibes

For each matrix row, record: device, OS build, app version, network state before/after, time from request start to visible result, which path produced the result, and whether the input survived. On the Pixel 7 with a mid-range on-device model, my on-device fallback path consistently completed within a few seconds for short inputs — but I am deliberately not publishing a number here, because a single-device, single-model figure generalizes poorly and the method is the point, not my hardware's score. Run the matrix on the lowest-end device in your support list; that is where fallback latency decides whether the feature feels broken.

Limitations and who should not do this

• The free server is test infrastructure, not an SLA. I used MonkeyCode's free tier because cost-free failure-injection is genuinely useful during development — you can fire row 6 (forced 5xx via a bad endpoint config) and row 1 timeouts all afternoon. I have no verified information about its quotas, rate limits, uptime guarantees, or how long the free tier lasts, so do not architect a production dependency on it. Point the same RemoteSummarizer interface at your real backend before release and re-run the matrix.
• Emulators lie about rows 3–5. Cellular handoff and Doze behave differently on real hardware with carrier stacks and OEM battery managers (looking at you, every OEM with an aggressive app killer). Emulators are fine for rows 1, 2, and 6 only.
• This matrix assumes a remote-first design. If your feature is privacy-sensitive enough that input must never leave the device, skip the fallback architecture entirely and go on-device-only — a fallback path is also a data-flow path you have to audit.
• Battery cost is untested here. Six scenarios × many iterations of on-device inference is a real energy draw; I did not measure it in this round and you should before calling fallback "free."

What I'd ask you to check

If you have a remote-first AI feature in the wild, run row 4 tonight: send a request, background the app, kill the network, come back. Report what happened — device, OS, and whether the result recovered, restarted, or silently disappeared. My bet is that at least a third of apps quietly lose the input. If you want a cost-free endpoint to break against while you find out, MonkeyCode's free tier is where I ran mine; the matrix above works against any endpoint you control.