---
title: "Speculative Decoding on Mobile: 2-3x Faster LLM Inference on Android and iOS"
published: true
description: "Speculative decoding cuts mobile LLM latency 2-3x. Learn how to tune acceptance rate, manage dual-model memory, and structure token trees for Neural Engine and Snapdragon NPU hardware."
tags: [mobile, android, ios, architecture]
canonical_url: https://mvpfactory.co/blog/speculative-decoding-mobile-llm
---
## What We Are Building
By the end of this tutorial you will understand how to apply speculative decoding to on-device LLM inference on Android and iOS — pairing a fast 1B draft model with a 7B verifier to cut mean token latency by 2-3x. We will cover the speedup formula, token tree construction, memory budgeting for Neural Engine and Snapdragon NPU hardware, and how to tune acceptance rate before you ship.
---
## Prerequisites
- Familiarity with transformer inference (autoregressive decoding, KV cache)
- Experience deploying quantized models on Android (NNAPI) or iOS (Core ML / ANE)
- Basic understanding of INT4 quantization
---
## Step 1: Understand Why Mobile Inference Is Memory-Bound
Let me show you a pattern I use in every project: profile before you optimise.
Every autoregressive token loads the full weight matrix from DRAM. That is the bottleneck, not compute. A 7B INT4 model sits at roughly 3.5 GB. Each forward pass touches all of it. On an Apple A17 Pro with 68 GB/s memory bandwidth — after accounting for KV cache, activation buffer, and OS overhead — you land at 50-120ms mean token latency. Painful for real-time conversation.
---
## Step 2: Introduce the Draft-Verifier Architecture
Speculative decoding adds a second, smaller model to the pipeline:
- **Draft model** (1B parameters, INT4): ~5ms per token
- **Verifier model** (7B parameters, INT4): ~80ms per token
The draft generates *K* candidate tokens speculatively. The verifier evaluates all *K+1* positions in a single forward pass — transformer attention is parallelisable across the sequence dimension. If the draft tokens fall within the verifier's acceptance threshold, you keep all *K* tokens. If not, you truncate at the first rejection and emit a corrected token.
The theoretical speedup ceiling is `1 / (1 - α)` where `α` is the acceptance rate. The docs do not mention this, but the practical formula is:
speedup ≈ (α·K + 1) / (1 + r·(K + 1))
With `r = 0.06` (5ms draft / 80ms verifier), `K = 7`, and `α = 0.7`, realised speedup is ~2.1x — not the theoretical 3.3x ceiling, but a meaningful production gain.
---
## Step 3: Move From Linear Drafts to Token Trees
Linear drafting wastes the verifier's parallel capacity. Here is the minimal structure to get token trees working:
text
Draft token tree — depth 3, branching factor 2
[START]
├── "The"
│ ├── "quick"
│ │ ├── "fox"
│ │ └── "dog"
│ └── "slow"
└── "A"
└── "fast"
Instead of a single chain, the draft model generates a tree of continuations — branching where token probability is spread across multiple candidates. The verifier scores all leaf paths in one attention pass using a structured attention mask, then accepts the highest-probability valid path.
On Apple ANE and Qualcomm Hexagon NPU — hardware that supports batched attention efficiently — tree verification increases accepted tokens per verifier call by 40-60% compared to linear drafts.
---
## Step 4: Budget Your Memory Carefully
Here is the hard constraint: running two models simultaneously on mobile.
| Model | Params | INT4 Weights | KV Cache (2K ctx) | Total |
|---|---|---|---|---|
| Draft (1B) | 1B | ~0.5 GB | ~0.1 GB | ~0.6 GB |
| Verifier (7B) | 7B | ~3.5 GB | ~0.4 GB | ~3.9 GB |
| Combined | | | | ~4.5 GB |
iOS Neural Engine on A-series chips allows ~6 GB for model execution on high-end devices. Android headroom on Snapdragon 8 Gen 3 reaches 4-6 GB — but that depends on LPDDR5 allocation policies set by OEMs, not NNAPI itself. NNAPI is an inference dispatch API, not a memory allocator.
Design for your p25 device, not your p75.
---
## Step 5: Tune Acceptance Rate Before You Ship
Acceptance rate α is not fixed. A 1B draft model on general text may hit α = 0.65 on conversational prompts and drop to α = 0.40 on domain-specific technical content. Your levers:
1. **Match sampling temperatures** between draft and verifier. Mismatched temperatures tank acceptance rate immediately.
2. **Fine-tune the draft on your prompt distribution.** A domain-adapted 1B model can match a general 3B model at half the memory cost.
3. **Dynamically adjust draft length K.** If rolling α < 0.5, reduce K to 3. If α > 0.75, push K to 8.
---
## Gotchas
**Token trees only pay off on NPU-capable hardware.** On GPU fallback, use linear drafts. Implement a runtime capability check and fall back gracefully — do not assume tree support.
**The correct draft model is the largest that fits your remaining memory budget** after the verifier and KV cache are allocated. Not the one with the best standalone benchmark. Benchmark scores are irrelevant if the model does not fit.
**Profile α on your production prompt sample before committing to a draft model.** If α falls below 0.55, either fine-tune the draft or select a larger model that better tracks the verifier's distribution. This is the gotcha that will save you hours of post-launch tuning.
---
## Conclusion
Speculative decoding is one of the highest-leverage inference optimisations available on mobile today. Pair a 1B draft with a 7B verifier, move to token trees on NPU hardware, size your models to your real device memory budget, and profile acceptance rate against your actual prompt distribution before release. The 2-3x latency reduction is real — but only if you tune for your hardware and your users.
For further reading: [Apple ANE documentation](https://developer.apple.com/documentation/coreml) and [Qualcomm AI Hub](https://aihub.qualcomm.com/).
Top comments (0)