POCKET: Running a 35B Model on CPU with llama.cpp, ~3.4x Over Bonsai

POCKET: Running a 35B Model on CPU with llama.cpp, ~3.4x Over Bonsai

A 35B model with no GPU

POCKET is an on-device LLM built to run a 35B-parameter model without a GPU, using standard llama.cpp. No custom runtime, no exotic accelerator: the target is commodity CPUs and the widely used llama.cpp inference stack. In on-device measurements it reached roughly 3.4x the throughput of Bonsai, an on-device baseline. The headline is not a capability score but a deployment fact: a model this size running acceptably on hardware that has no graphics card.

POCKET Box

The hardware companion is POCKET Box, a dedicated CPU-only device with no GPU. The idea is to package the CPU-inference approach into a fixed appliance so that on-device deployment does not depend on whatever accelerator a user happens to have. It is the same bet as the software: push capability onto standard CPU silicon rather than scarce, power-hungry GPUs.

Distribution

POCKET ships through the channels developers already use:

• Ollama
• Docker
• ModelScope
• Hugging Face GGUF

That spread matters for on-device work, where the friction is usually packaging and format rather than the model itself. GGUF plus Ollama and Docker covers most local-inference workflows without a bespoke installer.

The quantization trade-off

Running large models on CPU leans hard on quantization, and this is where honesty is required. Extreme low-bit quantization, specifically IQ1, degrades Korean-language output noticeably. The recommendation is Q4 or higher for usable Korean generation. That is a real constraint, not a footnote: the same trick that makes a 35B model fit on a CPU can break the language quality that Korean users care about, if pushed too far.

Why CPU-first is a real strategy

GPU scarcity and cost are the main barriers to on-device and edge deployment. A stack that runs a 35B model on CPU through llama.cpp sidesteps that barrier for a large class of use cases: private local inference, air-gapped environments, and appliances where adding a GPU is impractical. The ~3.4x over Bonsai suggests the engineering is doing real work, not just relabeling a baseline.

Honest scope

The ~3.4x figure is measured against Bonsai, an on-device baseline, not against GPU inference or frontier throughput; it is a relative on-device comparison. CPU inference of a 35B model is viable but still slower than GPU serving in absolute terms, and quality depends heavily on quantization: IQ1 is not recommended for Korean, and Q4+ is the practical floor. POCKET Box is a CPU-only appliance, so it inherits both the strengths and the limits of CPU inference. Treat POCKET as a strong on-device and edge story, scoped to local, no-GPU deployment, rather than a general replacement for GPU serving.

More: https://vidraft.net