DEV Community

AI OpenFree
AI OpenFree

Posted on

POCKET-35B-GGUF: Running a 35B LLM on CPU-Only PCs and Mobile Devices Without a GPU

#localllm #cpuinference #ggufquantization #pocket35b

POCKET-35B-GGUF: Running a 35B LLM on CPU-Only PCs and Mobile Devices Without a GPU

TL;DR: VIDRAFT's FINAL-Bench team has released POCKET-35B-GGUF on Hugging Face — a family of heavily quantized 35B-parameter models designed to run entirely on CPU, with no CUDA or cloud dependency. Variants cover everything from 32 GB RAM desktop servers down to Android phones and Apple Silicon devices via MLX. If you've been waiting for a 35B model that fits on a mini-PC or iPhone, this is worth a look.

What it is

POCKET-35B is a 35-billion-parameter language model released by VIDRAFT's FINAL-Bench team as a series of GGUF-format quantized weights on Hugging Face. The core design goal is CPU-first, GPU-optional local inference — meaning you don't need an NVIDIA card, a cloud subscription, or any CUDA installation to run it.

The release gained significant traction when a post to Reddit's r/LocalLLaMA community went viral, subsequently spreading to Chinese-language tech media including 桃子快讯.

The model family ships in multiple quantization tiers and language-specific branches:

Core GGUF variants (35B backbone):

  • Q4_K_M — 21 GB file, targets 32 GB RAM PCs or servers, highest output quality
  • Q2_K ⭐ (team-recommended) — 13 GB file, runs on a GPU-less mini-PC as a daily driver
  • IQ1_M — 8.2 GB file, targets 16 GB RAM devices, smallest footprint while preserving full model architecture

Language- and platform-specific branches:

  • POCKET-KR-GGUF IQ2_M — 5.1 GB, Korean-language variant for Android devices with 8 GB+ RAM
  • POCKET-KR-MLX 2-bit — 5.1 GB, Korean variant optimized for Apple Silicon via the MLX framework (iPhone, iPad, Mac)
  • POCKET-EN-GGUF iPhone-mix — 5.3 GB, English variant for iPhone via PocketPal
  • POCKET-EN-GGUF PC-mix — 6.8 GB, English variant for PC and Android

How it works

At a conceptual level, POCKET-35B follows the established GGUF + llama.cpp local inference pipeline that the open-source LLM community has standardized around:

  • Quantization reduces the numerical precision of model weights (e.g., from 16-bit floats down to 4-bit, 2-bit, or even sub-2-bit representations), dramatically shrinking both file size and memory footprint at the cost of some output quality.
  • GGUF format is the file container standard used by llama.cpp, enabling CPU inference with optional GPU offload. No CUDA installation is required for CPU-only operation.
  • MLX path (for Apple Silicon variants) takes a different route: instead of llama.cpp, it leverages Apple's MLX framework to run matrix operations natively on the unified memory architecture of M-series chips, which is particularly efficient for the KR and EN mobile variants.
  • The IQ1_M tier uses extremely aggressive quantization (sub-2-bit), which allows the full 35B model structure to be preserved while still fitting in 8.2 GB — a significant engineering trade-off that the team acknowledges comes with measurable quality degradation.

Benchmarks & results

The team published Korean-language perplexity (PPL) scores as the primary quality reference metric. Lower PPL = better quality:

Variant File Size PPL (Korean)
Q4_K_M 21 GB 5.79
Q2_K ⭐ 13 GB 6.49
IQ1_M 8.2 GB 9.69
POCKET-KR IQ2_M / MLX 2-bit 5.1 GB 7.95

A few important caveats the source itself flags:

  • English-language PPL scores were not published for the EN variants.
  • The r/LocalLLaMA post claimed 59 tokens/s on CPU, but the source notes that hardware configuration and test conditions for this figure were not fully disclosed in the original post.
  • The IQ1_M tier shows a meaningful quality jump (PPL 9.69 vs. 6.49 for Q2_K), so treat it as a size-constrained fallback, not a first-choice option.

How to try it

The POCKET-35B-GGUF series is publicly available on Hugging Face under the FINAL-Bench organization. You can browse and download the weights using the standard Hugging Face CLI:

pip install huggingface_hub
huggingface-cli download FINAL-Bench/POCKET-35B-GGUF
Enter fullscreen mode Exit fullscreen mode

Once downloaded, any GGUF-compatible inference runtime will work — llama.cpp is the primary target. For Apple Silicon users, the MLX-format KR variant is available on the same Hugging Face page. For iPhone users, the EN iPhone-mix variant is designed for use with PocketPal.

No OpenAI-compatible API endpoint has been publicly announced for this release. This is a self-hosted, local-first distribution.

FAQ

Q: Do I really need no GPU at all, or is that marketing?
A: The GGUF format genuinely supports CPU-only inference via llama.cpp. The Q2_K (13 GB, 32 GB RAM recommended for headroom) and IQ1_M (8.2 GB, 16 GB RAM) tiers are specifically sized for GPU-less machines. GPU offloading remains optional if you have VRAM available.

Q: How bad is the quality loss at IQ1_M vs. Q4_K_M?
A: Significant. The published Korean PPL scores jump from 5.79 (Q4_K_M) to 9.69 (IQ1_M) — roughly a 67% increase in perplexity. The team's own recommended tier is Q2_K, which hits a reasonable 6.49 PPL at 13 GB. Use IQ1_M only when 16 GB RAM is a hard constraint.

Q: Is the base model weights or training data publicly documented?
A: The source article notes that the original model provenance and full technical details were not disclosed in the r/LocalLLaMA post. Check the FINAL-Bench Hugging Face model card directly for the most complete information available.

Originally reported by 桃子快讯 (중국) (2026-07-26) — source article.

Top comments (0)