POCKET-35B: Running a 35B MoE Model on iPhone and CPU-Only PCs via ~5 Memory Bandwidth Reduction

#ondeviceai #moelanguagemodel #edgeinference #pocket35b

POCKET-35B: Running a 35B MoE Model on iPhone and CPU-Only PCs via ~5× Memory Bandwidth Reduction

TL;DR: VIDRAFT Lab's FINAL-Bench team released POCKET, a 35-billion-parameter Mixture-of-Experts (MoE) model designed to run fully on-device — including on iPhones and PCs with no discrete GPU. By activating only a sparse subset of parameters per token, POCKET cuts per-token memory transfer to roughly 0.66 GB versus ~3.5 GB for a comparable dense model, making real-world on-device inference practical for the first time at this parameter scale.

What it is

POCKET (Personal On-device Compact Knowledge and Execution Transformer) is a 35B-parameter MoE language model published by FINAL-Bench (VIDRAFT Lab) on July 23, 2026. Its primary design goal is on-device, private inference — no cloud API required.

Key characteristics from the announcement:

Architecture: Mixture-of-Experts (MoE) with sparse activation per token
Total parameters: 35B (only a fraction are active per forward pass)
Target hardware: iPhone (Apple Neural Engine), GPU-less PCs, and high-performance GPU rigs
Distribution: Open-weight, available on Hugging Face under a research/personal-use license
Multiple quantization builds provided so users can pick the right quality/resource trade-off for their hardware

FINAL-Bench's blog post accompanying the release explicitly states that POCKET "sacrifices nothing" — their claim is that MoE sparsity combined with aggressive quantization lets the model hit usable quality and speed and accessibility simultaneously, rather than compromising one for the others.

This follows FINAL-Bench's earlier release of Aether-7B-5Attn, a 100% open-source sovereign foundation model, continuing the lab's stated mission of building an open, private AI ecosystem.

How it works

The efficiency story here is straightforward MoE math, applied to an extreme hardware constraint:

Sparse activation: In a standard (dense) transformer, every token activates all parameters on every forward pass. In a MoE model, each token is routed to only a small subset of "expert" sub-networks. Parameters that aren't selected for a given token never need to be loaded into active compute — they sit in memory unread.
Memory bandwidth is the bottleneck on edge devices: On a phone or a CPU-only laptop, you're not compute-bound in the same way a GPU datacenter is. You're bandwidth-bound — the bottleneck is how fast you can stream weights from storage/RAM into compute units. POCKET's MoE design attacks exactly this bottleneck.
The reported result: Per-token memory transfer drops to approximately 0.66 GB, compared to approximately 3.5 GB for a dense model of equivalent capability — roughly a 5× reduction.
Apple Neural Engine (ANE) optimization: The iPhone build is specifically tuned for ANE execution, combining extreme quantization with hardware-aware operator scheduling to hit practical inference speeds on mobile silicon.
Quantization tiers: Multiple builds are shipped, from full-precision (for GPU-equipped machines) to aggressively quantized variants (for CPU-only PCs and iPhones). Users select the build that fits their hardware budget.

Benchmarks & results

The source article does not publish specific benchmark scores (e.g., MMLU, HumanEval, or throughput numbers in tokens/sec). What is stated qualitatively:

Practical inference speed is achieved on iPhone hardware — the framing implies usable, not merely "technically runnable" performance
Memory transfer per token: ~0.66 GB (POCKET MoE) vs. ~3.5 GB (comparable dense model) — a ~5× reduction cited as the key enabler
The FINAL-Bench blog describes the result as a technical milestone: 35B parameters running on an iPhone is positioned alongside Apple OpenELM and Microsoft Phi series as a meaningful step in on-device LLM progress, notable because 35B is considerably larger than typical on-device model targets

For independently verified benchmark numbers, watch the FINAL-Bench Hugging Face organization page where POCKET's model card should carry evaluation results.

How to try it

POCKET is listed as publicly available on Hugging Face under the FINAL-Bench organization. The source cites a Hugging Face Blog post published July 23, 2026 as the primary reference.

To find the model:

# Browse or search the FINAL-Bench org on Hugging Face
huggingface-cli search models --author FINAL-Bench POCKET

The release includes multiple quantization builds — check the model repository's README for the build matrix and pick the variant that matches your hardware (full-precision, quantized CPU, or iPhone-optimized).

⚠️ Specific model repository slugs, quantization format names, and any CLI or API endpoints are not confirmed in this source. Check the official Hugging Face Blog post and FINAL-Bench org page directly before scripting any downloads.

No OpenAI-compatible API endpoint is mentioned in the source — POCKET is positioned as a local/on-device model, not a hosted API service.

FAQ

Q: What's the practical difference between 0.66 GB and 3.5 GB per-token transfer for on-device inference?
A: On a phone or CPU-only laptop, DRAM bandwidth is typically 50–100 GB/s — orders of magnitude lower than a high-end GPU. Reducing per-token weight loading by ~5× translates roughly linearly into faster token generation on bandwidth-constrained hardware. This is why MoE is especially compelling for edge deployment, not just for saving VRAM.

Q: Is this model free to use in commercial products?
A: The source describes POCKET as "open-weight, for research and personal use." Commercial licensing terms are not specified in the coverage — check the model card on Hugging Face for the exact license before building production applications.

Originally reported by AI Chronicle (日本) (2026-07-28) — source article.