Someone on r/LocalLLaMA benchmarked Qwen3.8-Flash-Next in llama.cpp from CPU-only all the way to 96GB of VRAM. The model file is 87.2 GiB. On a plain CPU it decodes at 8.34 tok/s. With the full 96GB, 109 tok/s.
The number that looks like a party trick is the CPU one. It is a compression story. The weights are 4-bit. At 16-bit the file would be around 4x bigger, north of 300GB, and no consumer setup loads that, GPU or not.
Two more findings from the benchmark worth stealing:
- At 245K context, the 96GB advantage over 24GB shrinks from 2.80x to 1.45x. Long context flattens everything.
- Dropping the model's 27.2 GiB embedding table onto the GPU cut decode from 108.5 to 1.95 tok/s. RAM placement was 55.6x faster. Where a tensor lives matters as much as how much VRAM you have.
Why this matters here: embeddings hit the same wall at smaller scale. Every f32 embedding at 768 dims costs 3,072 bytes. A 100k-vector agent memory is about 300MB. Fine on a server, real money on a phone.
We built vecq for that. Training-free vector quantization in Rust (Apache-2.0, crate: vecq-core). No calibration data, deterministic results across platforms. On real EmbeddingGemma vectors: 4.78x smaller at default width (642 bytes/vector) with recall@10 of 0.979, and the index builds 14x faster than HNSW.
Honest trade: search is brute force, about 14x slower than HNSW at default width. Made for on-device indexes in the thousands of vectors, not million-vector servers. The file format is versioned and readers accept older versions, so indexes keep loading across releases.
Full breakdown with the benchmark tables is on the blog:
👉 https://blog.codecora.dev/quantization-cpu-to-96gb-and-embeddings/
Repo: https://github.com/codecoradev/vecq
Crate: https://crates.io/crates/vecq-core
Top comments (0)