TL;DR
Running a language model on a device, a tablet, a handset, an edge box, is not just the cloud without the network. It is a different budget. Memory is fixed, precision is a lever, latency includes a cold start, and there is no autoscaler to hide your mistakes. This post walks through the tradeoffs that decide whether on-device inference is a demo or a product.
Why on-device at all
Three reasons push inference onto the device: data that cannot leave, environments with no reliable network, and latency that must not depend on a round trip. A factory floor, a secure facility, or a field device often has all three at once. When any of them holds, the cloud is not an option, and the question becomes how much capability you can fit locally.
Memory is the hard wall
In the cloud you rent more memory. On a device you do not. The model weights, the key-value cache for context, and the runtime all have to fit in what the device has, at the same time.
This makes context length a budget decision, not a free parameter. A longer prompt grows the cache, which competes with the weights for the same fixed memory. The practical move is to size context to the device, not to the model's theoretical maximum.
Precision is the main lever
Lower numeric precision shrinks the model and speeds it up, at some cost to quality. The art is choosing a precision that fits memory and still holds the behavior you need.
Two cautions. First, the same model at a lower precision can behave differently, so benchmark the precision you will actually ship, not a higher one you tested in the lab. Second, quality loss from quantization is not uniform across tasks, so test on your task, not a generic one.
Latency includes a cold start
Cloud latency is usually measured warm. On a device, the model often has to load into memory first, and that cold start can dominate the first user experience. A request that arrives before the model is ready will fail or wait.
The fix is to treat loading as a visible state. Show that the model is warming up, retry the first request with backoff, and do not report the warm-up error as a crash. A 503 during load is a normal signal, not a fault.
No autoscaler, no hiding
In the cloud a bad capacity estimate gets papered over by scaling. On a device there is nothing to scale into. That raises the bar on three things: measure real memory headroom before you ship, estimate throughput with the mean rather than the median so one slow case does not get lost, and watch which resource is actually saturating under load rather than trusting a ratio.
A short checklist
- Does the model, cache, and runtime fit in device memory at your target context length?
- Did you benchmark the exact precision you will ship, on your task?
- Is cold start a visible, retried state rather than an error?
- Did you size context to the device, not the model maximum?
- Did you measure headroom and throughput under real load, not at idle?
FAQ
Is on-device always worth it? No. It is worth it when data cannot leave, the network is unreliable, or latency cannot depend on a round trip. Otherwise the cloud is simpler.
How low can precision go? As low as your task tolerates. That is an empirical question you answer by testing on your task, not a fixed number.
Why does cold start matter so much? Because it often dominates the first request, which is the one that shapes the user's impression.
What breaks most often? Memory. Context length quietly grows the cache until it collides with the weights.
Further reading
On-device inference rewards humility about fixed resources. Fit memory first, ship the precision you tested, make cold start a visible state, and measure under real load. The device will punish every shortcut.
Top comments (0)