A 35B model that runs on a GPU-less PC — and on a phone — with stock llama.cpp

#llm #opensource #localllama #machinelearning

There is a quiet, crowded race in AI right now: not in the cloud, but on the device in your hand. On-device models run without a network, keep your data local, and cost nothing per token. Everyone from big tech to the open-source community is pushing that way.

One of the most-cited names in that space is Bonsai — a 27B-class model compressed hard enough to run on a phone. Its family has passed 2 million downloads on Hugging Face, and it became the symbolic "big model in a small box."

A Korean deep-tech startup, VIDRAFT, just shipped an answer: POCKET — a 35B model that runs on a GPU-less PC, and on a phone, using stock llama.cpp. No fork. No CUDA. No cloud.

That sounds like a stretch. 35B usually means a data-center GPU. So the interesting part isn't the claim — it's that VIDRAFT measured it against Bonsai on the same machine, same stock tools, and published where it loses too.

The numbers (measured, same machine, stock llama.cpp)

Generation speed — the metric that decides on-device UX:

	POCKET-35B (IQ1_M)	Bonsai-27B (Q1_0)
CPU generate (Xeon, 16 threads)	27.0 tok/s	10.1	🟢 2.69×
GPU generate (H100)	197 tok/s	89	🟢 2.22×
Apple Metal generate	25.4 tok/s	12.8	🟢 1.99×
Laptop CPU generate (8 threads)	13.8 tok/s	4.4	🟢 3.13×

Quality — essentially tied, and we say so:

	POCKET	Bonsai
HellaSwag (400q)	61.0%	60.0%	⚪ tie (CI overlaps)

Where Bonsai wins — disclosed, not hidden:

	POCKET	Bonsai
Prefill on H100 (server GPU)	753 tok/s	1816	🔴 0.41×

That last row matters. On a data-center H100, Bonsai processes long prompts faster. On the hardware POCKET is actually built for — a laptop, a mini-PC, a phone — that advantage disappears: POCKET wins prefill too (Metal 3.28×, CPU 4.75×). The one place Bonsai leads is the one place on-device users don't run.

Publishing your losing rows is the tell. If you disclose where you lose, the rows where you win become believable.

Why a bigger model runs faster

POCKET is 35B; Bonsai is 27B. POCKET is the larger model, yet it generates faster in most settings. The reason is architecture.

POCKET is a Mixture-of-Experts model. Bonsai is like one generalist who answers every question. POCKET keeps many specialists and, for each token, calls only the few it needs. The total headcount is larger, but the work done per token is smaller.

That gap matters most exactly where there's no GPU. On a CPU or a phone, throughput is dominated by how much data you must read and process each step. A dense model reads everything, every time. A sparse-MoE model reads a fraction. Hence "bigger, but faster."

The part that's easy to miss: stock tooling

On-device models die on friction. If a model needs a custom-built runtime or a private fork, adoption stalls no matter how good the weights are.

Here's the concrete contrast: the same-size rival Ternary-Bonsai-27B-Q2_0 (7.2 GB) does not load in upstream llama.cpp — it needs the vendor's fork. POCKET loads in stock llama.cpp as-is — which means LM Studio, Ollama, and the rest of the standard ecosystem, day one. For Apple silicon, there's a native MLX build for iPhone and Mac.

What POCKET is actually saying

Three things:

Access — you don't need an expensive GPU server to run a 35B model. A GPU-less PC or a phone is enough.
Privacy — nothing leaves the device. That's the whole point for sensitive work, field data, and air-gapped environments.
Reproducibility — the speed comes from sparse-MoE plus ordinary quantization, measured on stock tools. Download the same files, run the same tool, check the numbers yourself.

It isn't flawless — prefill on a server H100 goes to Bonsai, and real-world feel varies with RAM, storage, quantization, and thermals. But VIDRAFT put those caveats in the table instead of hiding them. "We measured under the same conditions and here's what came out" is a stronger claim than "we won everything."