Support for Qwen3.8-Flash-Next merged into llama.cpp on August 27, 2026, bringing the architecture behind Alibaba's next Qwen generation to the software most people use to run models on their own machines. The pull request is unusually large -- 65 commits touching 28 files and adding 2,881 lines -- and its centrepiece is machinery for a 97.7 GiB per-layer n-gram lookup table, a slab of model that is read from rather than computed on.
Key facts
- Pull request #27742 by Unsloth's Daniel Han merged into ggml-org/llama.cpp at 19:32 UTC on August 27, 2026, adding a converter, text graph, sparse attention, vision support and three quantizer fixes.
- The architecture carries a 97.7 GiB n-gram hash table handled through host-side row indices rather than GPU tensors.
- Reported perplexity on wikitext-2 is 4.0068 against 4.0126 for the reference implementation, with 98.0% top-1 agreement on a prose sample.
- Primary source: llama.cpp pull request #27742.
Qwen3.8-Flash-Next is the model that put a 20-million-entry n-gram table inside a language model -- billions of parameters that are looked up rather than multiplied. That idea is elegant on paper and a nightmare for inference software, because every existing loader assumes a model's parameters are tensors you push onto an accelerator. A table this size cannot go on a consumer GPU, and it does not need to: a lookup only needs the handful of rows relevant to the tokens in front of you.
The merge solves that by keeping the table's indexing on the host and pulling rows on demand, and by streaming the table during conversion instead of assembling it in memory. It also adds a 64-bit integer case to the model loader, because the hash multipliers the architecture uses do not fit in 32 bits. The most telling line in the pull request is a negative result: "git diff master --stat -- ggml/ is empty: no new ggml op, and no change to any existing one." Everything new was expressible in the operations llama.cpp already had, which is the difference between a port that lands and a port that forks the engine.
The rest of the architecture is handled in familiar pieces: a gated delta-net on three of every four layers, a mixture of experts with 512 experts choosing ten at a time, and a new sparse attention graph with its own cache. Vision runs through the existing image path.
Now the part that decides whether you can actually run this. The Unsloth GGUF repository publishes eleven quantized builds. The smallest, at roughly 1-bit, totals about 72.5 GB on disk. The 4-bit build most people would reach for comes to about 111 GB, and the unquantized set is about 354 GB. No official VRAM requirement is published for any of them. What exists instead are community measurements in the repository's discussion thread: one user reports running a 4-bit build on a 16 GB RTX 5070 Ti with roughly 100 GB of combined RAM and VRAM in use, getting about 22 tokens per second, and another reports a build running entirely in system memory peaking at 109.3 GiB and generating 7.71 tokens per second at very long context.
Read that carefully, because it is the whole point of the design. This is not a 16 GB model. It is a model whose bulkiest component was deliberately made cheap to keep in ordinary system RAM, so a modest graphics card can do the compute while a large pile of DDR5 holds the lookups. That is a bet on offloading as an architecture decision rather than a fallback -- and an uncomfortable bet this particular week, given that DRAM contract prices have roughly doubled in a quarter.
The caveats are in the pull request itself, which is more candid than most. The bit-identical agreement between sparse and dense attention holds at full precision but not through quantization, where the 1-bit build shows a measurable logit difference. The automated architecture test is weaker than it looks because its synthetic model carries no lookup-table tensors, so that code path never runs during the check. And the author opened the work as a draft precisely because the weights were not public when the accuracy numbers were produced, meaning nobody outside could reproduce them at the time.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)