What actually moves the needle for on-device LLMs (lessons from 1.18M GGUF downloads)

We publish an on-device LLM series. Cumulative downloads recently passed 1.18M. Some of what we learned contradicts the usual advice.

1. The market is not chasing the biggest model

From a snapshot of the top 300 text-generation repos on Hugging Face by 30-day downloads:

Size bucket	Share of downloads
under 3B	30.1%
3-15B	31.2%
15-70B	26.8%
70B+	10.8%

Sub-15B accounts for 61.3%. Quantized repos are 51% of the list by count and 37.2% by downloads.

People download what they can actually run.

2. Quantization format is a distribution decision

GGUF alone is 13.4% of downloads in that snapshot. The reason is boring and important: it is what local runtimes consume. A brilliant model in a format nobody's runtime loads gets zero adoption.

3. Measure the whole process, not the cache

We once reported a KV-cache reduction of about 41% and had to retract the framing. Cache-level savings did not translate: process-level memory was down only about 11.9% at 32K context. The buffers we forgot to count were real memory on the user's device.

If you publish a compression number, publish the resident set, not the component you optimized.

4. Vocabulary pruning is free only in the languages you measured

We pruned vocabulary and verified no regression in Korean, English and code: identical token counts, identical retrieval and tool-calling scores.

Then we measured other scripts:

Language	Token inflation
Japanese	+31%
Chinese	+33%
Arabic	+129%

Nothing was broken. The model just became 2.3x more expensive for Arabic users. Put a different writing system in your eval set.

5. The runtime is part of the artifact

We shipped an attention modification whose savings only materialize in our own fork. Loaded by a stock runtime, the file loads fine and silently delivers zero benefit, which is worse than failing loudly.

Gate the runtime, not just the file. If two builds produce byte-identical outputs when they should not, that is a symptom, not a result.

Checklist

• Ship GGUF if you want local adoption
• Report process RSS, not cache deltas
• Include a non-Latin, non-CJK language in evals
• Version and verify the runtime alongside the weights

---

From VIDRAFT, a Korean deep-tech company running an **AI Foundry* — we diagnose, breed and optimize AI models for specific industries. Open models: Hugging Face · vidraft.net*