How to run compact LLMs on phones and edge hardware with quantization, latency trade-offs, and framework options.
A small language model (SLM) is a compact neural network, typically between 1 billion and 13 billion parameters, designed to run directly on edge devices including smartphones, robots, and IoT hardware. Unlike frontier models such as GPT-4, which require cloud servers and cost cents per inference, SLMs trade some reasoning capacity for the ability to execute locally, offline, and with latencies under 500 milliseconds per token. This shift from cloud-first to edge-first AI is reshaping how products handle real-time inference, privacy, and cost at scale.
Why this matters now
Through 2026, the economics and capability of edge AI have reached an inflection point. Device hardware, particularly mobile neural processors, has crossed a threshold: flagship phones now pack 300+ TOPS of AI compute, enough to handle quantized models in the 3B to 7B parameter range. Simultaneously, techniques like 4-bit quantization and grouped query attention have reduced model weight without crippling quality. The result is that teams building mobile products, industrial robotics, and deployed sensors now have a realistic option to avoid the latency, cost, and privacy exposure of cloud APIs.
This matters because the cost structure flips. A cloud-based inference pipeline that handles 100 million daily requests might cost hundreds of thousands monthly in API calls. An on-device SLM, deployed once, costs nearly nothing per inference after the initial development and device storage footprint. In regulated industries such as healthcare and financial services, edge inference also sidesteps data residency and compliance concerns entirely. At the same time, the quality and reasoning gaps between SLMs and frontier models remain real: teams must be honest about when a 7B model is sufficient and when you need GPT-4.
Model size, RAM, and device constraints
Photo by Godfrey Atima on Pexels.
The relationship between model parameters, quantization, and available device memory is non-negotiable: understand it or your deployment will fail in production.
A model's size in memory depends on two variables: parameter count and precision. A 7B parameter model stored in full 32-bit floating-point format occupies roughly 28GB in RAM (7 billion parameters × 4 bytes per parameter). That fits no phone. Quantization shrinks this dramatically:
FP32 (32-bit float): baseline, no compression. 7B model = 28GB. Not viable on mobile.
FP16 (16-bit float): 7B model = 14GB. Still too large for most phones.
Int8 (8-bit integer): 7B model = 7GB. On the edge of large flagship phones with aggressive compression elsewhere.
Int4 (4-bit integer): 7B model = 3.5GB. Fits modern flagships with headroom. Quality loss is typically 1-3% on benchmarks when done carefully.
An iPhone 16 Pro or Snapdragon 8 Gen 3 flagship typically offers 8-12GB of RAM. The OS reserves 2-4GB, leaving 4-8GB for your application. A quantized 7B SLM at 3.5GB is therefore plausible, but it fills most of the available space and leaves little room for other app operations. In practice, many teams deploy 3B to 5B models to stay comfortably under 2GB and reduce memory pressure on the OS.
For inference itself, models do not need to load the entire weight matrix into RAM at once. Techniques like "streaming" weights from disk or using segment-by-segment execution can reduce peak memory usage below the full model size, at the cost of added latency. This matters on budget phones (2-4GB total RAM) where 1B parameter models are the practical ceiling.
Robotics and edge servers offer more flexibility. An NVIDIA Jetson Orin Nano has 8GB VRAM and can handle 7B models comfortably. An Orin NX with 16GB VRAM can run models up to 20B parameters with quantization. Industrial applications with no size or power constraint can be more generous with model scale, enabling better quality at the cost of higher power draw.
Latency, throughput, and token generation speed
Latency is the wall-clock time to the first token (time-to-first-token or TTFT) and the time per subsequent token (tokens-per-second or TPS). These matter because they directly affect user experience in real-time applications.
On a flagship mobile phone (e.g., A17 Pro with 16-core Neural Engine), a quantized 3B SLM typically generates the first token in 100-200ms and subsequent tokens at 5-15 tokens per second, depending on the specific model architecture and whether the device supports batch or speculative decoding. A 7B model under the same conditions takes 200-400ms for the first token and 3-8 tokens per second. These are rough ranges; actual performance varies with implementation, whether you are using optimized kernels (like ONNX Runtime's QNN provider for Snapdragon), and what else is running on the device.
For comparison, a cloud API call (e.g., to Claude or GPT-4) typically incurs 500-2000ms of latency due to network round-trip, server queue, and processing time. For short requests, the edge device can be 5-10x faster. However, this advantage erodes if the edge model needs to generate long outputs: a 100-token response on-device (generating at 8 TPS) takes 12.5 seconds, while the cloud model might take only 3-4 seconds if it has higher throughput.
The practical implication is that edge SLMs excel at latency-sensitive, short-output tasks: autocomplete, real-time translation, local search, or sentiment analysis. They are less ideal for tasks requiring long, multi-paragraph generations where total time-to-completion matters more than first-token latency.
Memory bandwidth is another constraint rarely discussed but critical in practice. Mobile phones are designed for multimedia, not sustained matrix math. The peak memory bandwidth on an iPhone 16 Pro is around 120 GB/s. A 7B model at 4-bit precision requires roughly 3.5GB of weight data. During a single forward pass, this data must move from memory to compute. If the compute itself takes only a few billion operations (which is common in inference, not training), the model quickly becomes memory-bandwidth limited rather than compute-limited. This means faster hardware does not always translate to proportionally faster inference.
Quantization: the core technique for edge deployment
Photo by Daniil Komov on Pexels.
Quantization is not one technique but a family of methods to reduce model precision. For on-device SLMs, the most common approaches are post-training quantization (PTQ) and quantization-aware training (QAT).
Post-training quantization happens after the model is fully trained. You take a model in FP32 and convert weights and activations to lower precision (e.g., Int8 or Int4) using calibration data. The appeal is speed: you do not retrain. The downside is that aggressive quantization (especially Int4) can degrade accuracy by 2-5% on reasoning tasks if not done carefully. Techniques like symmetric vs. asymmetric quantization, per-channel vs. per-layer scaling, and learned quantization parameters all affect the output quality.
Quantization-aware training, by contrast, simulates quantization during training so the model learns to operate in lower precision. QAT typically preserves accuracy better than PTQ (degradation under 1% even at Int4) but requires access to training infrastructure and labeled data, making it less accessible for practitioners working with open-source or pre-trained models.
For most edge AI teams, the practical workflow is: download a pre-trained SLM, apply post-training quantization using a tool like GPTQ, AutoGPTQ, or ONNX Runtime quantization, benchmark on your target task, and iterate. If accuracy is insufficient, either fine-tune the quantized model on your task or move to a larger base model. Complete retraining from scratch is rarely necessary.
A note on bit-width: Int4 and Int8 are the sweet spots for edge deployment. Int4 is aggressive and introduces more quantization error but cuts model size in half versus Int8. Some vendors also promote "mixed-bit" schemes where some layers stay at Int8 while others use Int4. The gains are incremental and the complexity is higher; Int4 uniformly is usually simpler and sufficient.
Frameworks and deployment ecosystems
Choosing a framework for SLM deployment depends on your target hardware and the depth of optimization you need. No single framework dominates all scenarios.
ONNX Runtime is the most hardware-agnostic option. You convert your model to ONNX format, then run it on ONNX Runtime, which supports CPU, GPU (via TensorRT or CoreML), and NPUs. ONNX is well-suited for teams that want a single model to work across Android, iOS, and desktop. The downside is that ONNX does not always expose the very latest optimizations from specialized accelerators, so latency can lag behind native solutions. For most SLM use cases, ONNX Runtime is fast enough.
TensorFlow Lite is Google's framework for mobile and edge deployment. It excels on Android and supports Android Neural Processing Unit (NNPU) acceleration. TensorFlow Lite has strong tooling for quantization and is widely used in production. The ecosystem is mature and documentation is good. On iOS, TensorFlow Lite works but is less natural than Core ML.
Core ML is Apple's native framework for on-device inference. It integrates tightly with iOS and can accelerate inference using the Neural Engine, GPU, or CPU as needed. For iOS-first products, Core ML is the fastest and most power-efficient option. Conversion from other formats (PyTorch, TensorFlow) to Core ML is straightforward via tools like coremltools.
TensorRT-LLM is NVIDIA's inference engine optimized for LLMs on CUDA-capable GPUs. It is not a mobile framework but essential for Jetson and data-center edge deployments. TensorRT-LLM applies aggressive kernel fusion, dynamic batching, and specialized optimizations for LLM patterns. If you are deploying on Jetson hardware, TensorRT-LLM is the path to best performance.
llama.cpp is an open-source inference engine for llama-family models that has become a de facto standard. It runs on CPU on almost any device (including older phones) and supports quantization. The appeal is simplicity and no framework overhead. The downside is CPU-only inference, which is slower than GPU or NPU acceleration. For learning and small-scale deployment, llama.cpp is excellent; for production on modern hardware, a framework that can leverage accelerators is preferable.
In practice, the choice often comes down to: iOS primary? Use Core ML. Android primary? Use ONNX Runtime or TensorFlow Lite. Jetson-based robotics? Use TensorRT-LLM. Cross-platform but willing to optimize separately per platform? Use all three.
Quality gaps: when SLMs are insufficient
SLMs trade capacity for deployability. Understanding what you lose is crucial to avoid building products that fail silently in production.
On factual recall and retrieval tasks, SLMs perform comparably to larger models if they have seen the relevant data. A 7B model fine-tuned on your domain can retrieve facts as accurately as GPT-4.
On multi-step reasoning and novel problem-solving, the gaps are real. OpenAI's evals show that a 7B SLM reaches roughly GPT-3.5 capability on complex tasks like math (solving a SAT-level algebra problem) or code synthesis (writing a complete function from a specification). Below 7B, the degradation accelerates sharply. A 3B model is closer to GPT-3 or "good davinci" in 2023 terms.
For tasks that require long context (>8k tokens), smaller models can struggle due to attention mechanism limitations and training data. Many SLMs are trained on sequences only up to 4k tokens, making them unsuitable for long-document summarization or context-heavy QA.
The honest approach is to benchmark your specific task against your target SLM before committing to edge deployment. If your use case is autocomplete, sentiment analysis, or factual QA on your own data, a 3B-7B SLM is likely sufficient. If your use case involves chain-of-thought reasoning, open-ended creative writing, or solving unseen problems, you may need a larger model or a hybrid approach where the edge SLM handles preprocessing and the cloud handles the reasoning.
Common pitfalls and when edge AI fails
Edge AI is not a universal solution. Several patterns lead to failed deployments:
Underestimating power draw: SLMs on mobile generate heat, and continuous inference drains battery rapidly. A 7B model generating 10 tokens per second for 30 seconds can consume 5-10% of battery on a flagship phone. For applications that run sporadically (like a user typing a prompt), this is acceptable. For always-on or background inference, edge deployment becomes impractical without optimization.
Forgetting about latency variance: Benchmark latency under best-case conditions (device idle, cool CPU, no other apps running) and you will be surprised by real-world performance. When the user is already running Chrome, Slack, and a video call, inference on a shared CPU can be 2-5x slower. Always test under realistic load.
Deploying models too large for the target hardware: Testing on a flagship phone and then deploying to a mid-range device is a classic mistake. A 5B model that runs in 100ms on an A17 Pro may take 500ms on a Snapdragon 6 Gen 1. If your app needs to support a wide range of devices, either stick to 1-3B models or implement adaptive selection logic that downgrades to a smaller model on lower-end hardware.
Ignoring the update problem: Once deployed, updating SLM weights to a newer version can be challenging. App updates that swap a 2GB model file require re-shipping the entire app and user re-download. Teams building products that need to improve or fix models over time often choose a hybrid approach: ship a small edge model for latency-critical inference, but allow the backend to update and A/B test new models without breaking all users.
Missing the real-time constraint: Some tasks do not actually require on-device inference. If your SLM is generating a summary that the user will read in 5 seconds, a 2-second cloud call followed by local caching is often simpler and higher-quality than building custom on-device inference infrastructure. Edge AI excels at sub-100ms latency requirements. For tasks with looser timing, the cloud is often faster to market.
Practical implementation: a workflow
For a team starting with edge AI, a typical workflow is:
Choose a base model. Start with an open SLM such as Llama 2 7B, Mistral 7B, or Phi-3. These have proven deployability and available quantized versions.
Quantize and profile. Use AutoGPTQ or ONNX Runtime to quantize to Int4. Profile latency and memory on your target device (e.g., a mid-range Android phone and a modern iPhone).
Benchmark on your task. Run your inference workload (e.g., 100 real user prompts) and measure accuracy, latency, and memory usage. If latency exceeds your target, move to a smaller model. If accuracy is insufficient, consider fine-tuning or retrieval augmentation.
Choose a framework. Based on your target platform, select ONNX Runtime, Core ML, or TensorFlow Lite. Get an end-to-end hello-world running, then integrate into your app.
Test offline and on-device. Disable network and run inference. Measure battery drain and heat generation. If power is a concern, experiment with batching requests or running inference only when plugged in.
Plan for updates. Decide whether you will ship model updates as app updates, incremental downloads, or keep a cloud fallback. Each has trade-offs in size, latency, and maintenance burden.
Most teams take 4-8 weeks from "we want on-device AI" to a beta build ready for internal testing. Scaling to production adds another 4-8 weeks of performance tuning, memory profiling, and battery testing.
Edge AI is no longer experimental. The combination of hardware capability, quantization techniques, and mature frameworks makes on-device SLM inference practical for many real-world applications. The key is matching the right model size, quantization strategy, and framework to your hardware and latency budget, then testing ruthlessly on actual devices under realistic conditions. Teams that do this work rigorously can ship responsive, private, offline-first AI products at scale. Teams that skip it will burn cycles debugging mysterious slowdowns and out-of-memory crashes in production.
This article was originally published on AI Glimpse.
Top comments (0)