Qwen 3.8 arrived as two different releases with two different licences, and only one of them is something you can put on a card you own. The 2.4 trillion parameter A95B opened up on 12 August under Alibaba's own qwen3.8-max terms. The one that matters for local work is Qwen 3.8 27B, whose safetensors went up on 13 August at 08:23 UTC with an Apache 2.0 LICENSE file following the next morning. Both dates are off the Hugging Face commit log, not a launch post.
Here is the practical picture: what it needs, why its long context is unusually cheap, and the one setting that makes people think they downloaded a broken quant.
The shape of the model decides everything
27B dense parameters across 64 layers, hidden size 5120. The interesting part is in config.json, where layer_types reads 48 linear attention layers and 16 full attention layers, alternating three to one (full_attention_interval: 4). Only those 16 layers keep a KV cache.
The rest of the shape: 24 attention heads with head_dim 256 and 4 KV heads, a 248,320 token vocabulary, and max_position_embeddings of 262,144. It is a native vision language model, so images and video go in without a wrapper, and the ggml-org pack also ships a multi token prediction head as a separate file.
The numbers
Sizes below are the file sizes Hugging Face reports for unsloth/Qwen3.8-27B-GGUF, read on 14 August 2026. Packs differ by a few hundred megabytes, so check the repo you actually pull from. lmstudio-community has Q4_K_M at 16.8 GB and ggml-org at 19.0 GB for the same nominal quant.
| Quant | Size on disk | Realistic home |
|---|---|---|
| UD-IQ2_XXS | 9.0 GB | 12 GB cards, visible quality cost |
| UD-Q2_K_XL | 10.7 GB | 12 GB cards, almost no context left |
| UD-Q3_K_XL | 13.4 GB | 16 GB cards |
| Q3_K_M | 13.8 GB | 16 GB cards |
| IQ4_XS | 15.7 GB | largest quant that stays whole on 16 GB |
| Q4_K_M (sweet spot) | 17.1 GB | 24 GB cards |
| Q5_K_M | 19.8 GB | 24 GB, less context headroom |
| Q6_K | 22.9 GB | 24 GB barely, or 32 GB |
| Q8_0 | 29.0 GB | 32 GB or a two card split |
BF16 (from ggml-org) |
53.8 GB | server cards, or CPU and patience |
| mmproj | 0.9 GB | the vision encoder, separate file |
Machine classes, honestly:
- 24 GB GPU: Q4_K_M whole, with real context headroom. This is the card the model was sized for.
- 16 GB GPU: IQ4_XS fits whole; Q4_K_M works with a few layers offloaded and costs you speed.
- 12 GB GPU: only the 2-bit quants, and you will feel it. A 3060 runs it, slowly.
- Apple Silicon: 32 GB unified memory is comfortable at Q4_K_M, 24 GB works if nothing else is open.
Where the hybrid layout pays off
Every full attention layer stores a KV cache that grows with the context. At fp16 one token costs 2 x 4 heads x 256 dim x 2 bytes = 4 KB per layer. A conventional 64 layer model pays that on all 64 layers, which is 256 KB per token. Qwen 3.8 27B pays it on 16 layers, so 64 KB per token.
| Context | KV cache | Q4_K_M total |
|---|---|---|
| 8K | 0.5 GB | 17.6 GB |
| 32K | 2.0 GB | 19.1 GB |
| 128K | 8.0 GB | 25.1 GB |
| 262K | 16.4 GB | 33.5 GB |
That is the difference between a long window on the spec sheet and one you actually turn on. A 128K session that would need 32 GB of cache on a dense model needs 8 GB here.
With a current llama.cpp:
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
--jinja \
-ngl 99 \
-c 32768
The template trap
That --jinja flag is not optional, and it is the single biggest source of "this quant is broken" reports.
Qwen 3.8 ships its own chat template. Load the model without it and the model has no reliable marker for where your turn ends and its answer begins. The two failure modes both look like a bad conversion: either it rambles past the stop token, or it answers in a clipped voice and loses the conversation between turns.
There is a second, sharper version of this. The official template wraps each assistant turn in a think block even when the reasoning is empty, then opens another one when generation starts. Across several turns those nest and the history gets truncated. Several GGUF packs already ship a corrected chat_template.jinja. If you converted the weights yourself, swap the template before you blame the quantization.
Vision needs the second file
The vision encoder is not inside the language GGUF. Download the mmproj file, about 0.9 GB, and load it alongside:
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
--mmproj mmproj-F16.gguf \
--jinja -ngl 99
Skip it and you have a strong text model that will politely tell you it cannot see the image you just pasted.
Thinking is a dial, not a switch
The model reasons before answering by default. Lower reasoning_effort from the default to medium or low for quick answers, or pass chat_template_kwargs with enable_thinking: false to skip the reasoning pass entirely. Locally the cost of thinking is not money, it is your own time watching tokens arrive, which is the more annoying currency.
The no-terminal route
If you would rather click than type flags, Locally Uncensored (open source, AGPL) wraps this: install, open the Model Manager, paste a Qwen 3.8 27B GGUF repo, pick the quant that fits your card, chat. It carries the llama.cpp engine, handles the offload split and the template, and keeps everything on your machine with no account and no telemetry.
And the 2.4T reality check
The big Qwen 3.8, Qwen/Qwen3.8-2.4T-A95B, has open weights but not an open licence. Hugging Face reports license: other with license_name: qwen3.8-max, so read the terms before you build a product on it, and do not repeat the "Apache" line that has been going around. At 2.4 trillion total parameters with 95 billion active it is a data center model regardless.
The split that works: run the 27B at home for anything private, offline, or repetitive, and reach the A95B through a hosted API when a task genuinely needs that scale. It is on DeepInfra, and both it and the 27B class of models sit in LU Labs Cloud if you want them next to each other in one picker.
FAQ
Is 27B enough? For local work it is the interesting size: big enough for real coding and agent loops, small enough to sit on one consumer card. The hybrid attention means long context does not price you out the way it does on a dense model of the same parameter count.
Ollama or LM Studio? Both pick up the shipped template automatically, which spares you the trap above. The bare-GGUF-plus-llama.cpp route is where people get bitten.
Uncensored builds? Apache 2.0 makes finetunes and abliterations fully legal, and community variants started appearing within a day of the weights. The stock model carries standard alignment.
How do I check the sizes myself? https://huggingface.co/api/models/<repo>/tree/main returns every file with its byte count. That is where every number in this post came from.
Top comments (1)
Local model posts are most useful when they include the boring constraints: real file sizes, KV cache cost, template gotchas, and where the setup fails before inference starts.