Should an LLM Run On the Phone, at the Edge, or in the Cloud?

“On-device AI” sounds like a deployment choice. For a mobile team it is a product constraint that affects download size, memory pressure, battery, startup time, privacy, and which devices receive the feature.

The right placement is often not purely on-device or purely cloud. Decide per task, then measure on the slowest supported hardware and network.

Begin with the task envelope

Write down what one operation requires:

input:       up to 30 seconds of transcribed speech
output:      title plus five action items
deadline:    first useful result within the product's target
offline:     desirable, not mandatory
sensitivity: meeting content may be confidential
frequency:   several times per day
devices:     the actual supported OS and hardware range

Without this envelope, “run a small model locally” is not an engineering requirement. Input length and output size affect memory and compute. Frequency affects battery. Sensitivity affects whether a cloud path is permitted.

Compare three placements

Placement	Strengths	Constraints
On-device	can work offline; input may stay local; no inference round trip	model package size, RAM, thermal load, device fragmentation
Cloud	larger model choices; centralized updates; predictable server tooling	network latency, service cost, data transfer and retention concerns
Hybrid/edge	local routing, redaction, or fallback plus remote generation	two implementations, synchronization, more failure states

Hybrid does not automatically mean “best of both.” It adds a boundary that must be tested. A local classifier followed by cloud generation can reduce unnecessary uploads, but the product now needs a rule for classifier uncertainty and offline behavior.

Budget memory before model quality

A model’s file size is not its complete runtime footprint. Budget for:

• weights after the chosen quantization;
• runtime and graph overhead;
• key-value cache, which grows with context and generation;
• tokenizer and supporting assets;
• input buffers and application UI;
• temporary memory during model loading or conversion.

Test memory warnings and background/foreground transitions. Mobile operating systems can terminate an application that behaves acceptably during a short foreground demo but exceeds pressure limits during longer use.

Do not publish one device result as a universal benchmark. Record model artifact hash, runtime version, device, OS version, power state, thermal state, prompt length, output length, and whether the run was cold or warm.

Measure the experience in phases

For interactive generation, collect:

cold model load
warm request setup
time to first output
output rate
time to completion
peak memory
energy or battery impact over a repeatable workload
thermal behavior across repeated runs

A fast first run can be misleading. Sustained use may trigger thermal throttling, and a model kept resident for speed may increase memory pressure. Test a session that resembles actual product frequency.

For cloud inference, add DNS/connect time, upload time, network type, retry count, and server timing when available. Test high latency and packet loss, not only good office Wi-Fi.

Treat model delivery as application delivery

Bundling weights increases the base application download. Downloading after install creates a second product flow:

1. explain the size and purpose;
2. check storage and network conditions;
3. support pause, resume, and cancellation;
4. verify the artifact’s integrity;
5. keep compatibility between app, runtime, tokenizer, and model versions;
6. remove obsolete artifacts safely.

Avoid silently downloading a large model over a metered connection. If the model is optional, make the feature’s storage visible and removable.

On Apple platforms, Core ML model deployment documents downloading and compiling models on a user’s device. Other runtimes use different packaging mechanisms, but model artifacts still need versioning and lifecycle management.

Make the privacy claim precise

“Runs on device” does not guarantee that data stays on device. Check every path:

• analytics and crash reports;
• prompt or output logs;
• cloud fallback;
• remote safety or moderation checks;
• synchronization and backups;
• shared clipboard or exported files;
• third-party keyboards, extensions, or SDKs involved in input.

Describe the actual boundary: “Audio and transcript remain on this device unless the user chooses cloud fallback,” for example. Then test that boundary and make fallback consent explicit.

Also decide whether model outputs contain sensitive derived data. Keeping input local while uploading the generated summary may still violate the intended privacy property.

Design interruption and recovery

Mobile work is interrupted. Calls arrive, apps enter the background, connectivity changes, and the OS reclaims resources. Define behavior for:

• backgrounding during model load or generation;
• audio-session interruption;
• low-memory termination;
• loss of network during a cloud request;
• switching from Wi-Fi to cellular;
• low-power mode and thermal warnings;
• app updates while an optional model is downloading.

Persist only the state required to recover. A generation operation needs an ID and clear terminal state; replaying it after restart must not duplicate an external action.

Use a placement gate

Choose on-device only if the evaluated model fits the supported device envelope and the privacy/offline benefit justifies packaging and runtime cost. Choose cloud only if network, data policy, and operating cost meet the task requirement. Choose hybrid when a clearly defined local stage changes the outcome enough to justify a second execution path.

The final decision should include measured device results, a fallback behavior, artifact delivery plan, data-flow diagram, and a list of unsupported devices or states. Placement becomes an engineering decision when the team can say exactly which constraint it satisfies—and what it costs elsewhere.