DEV Community

sheephess9527
sheephess9527

Posted on AI-assisted

Local coding LLM on a 24GB GPU: Qwen2.5-Coder 32B vs Qwen3-Coder 30B-A3B (it's all about context)

If you want a local coding model on an RTX 4090 / 3090 / 7900 XTX, the question isn't only "does it load", it's "does it load with enough context to read my files".

Qwen2.5-Coder 32B (dense)

Build 4K 8K 16K 32K
GGUF Q4_K_M 21.7 GB (90%, tight) 95% doesn't fit doesn't fit
AWQ INT4 18.1 GB (75%) 80% 89%, tight doesn't fit
EXL2 3.5bpw 16.0 GB

So the dense 32B is great for chat-length prompts. AWQ via vLLM gives you comfortable 8K (Linux / WSL2 only).

Qwen3-Coder 30B-A3B (MoE)

Build 4K 32K
GGUF Q4_K_M 19.8 GB (83%) 22.7 GB (95%, tight)
AWQ INT4 16.4 GB 19.2 GB (80%, comfortable)

Only ~3B active parameters per token, so its bandwidth ceiling is far higher than a dense 32B. Note: the site has no measured run of it and runs no coding benchmarks, so this is about fit, not code quality.

Want even longer context?

North Mini Code 1.0 30B-A3B GGUF Q4_K_M: ~20.8 GB at 32K, ~23.5 GB at 128K (tight). Only 13 of its 49 layers keep a growing cache.

Quick start

# dense 32B, default 4K window
ollama pull qwen2.5-coder:32b
# AWQ, 8K comfortable
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ --max-model-len 8192 --host 127.0.0.1 --port 8000
Enter fullscreen mode Exit fullscreen mode

TL;DR

  • Short prompts, best dense coder: Qwen2.5-Coder 32B AWQ.
  • Whole-file / repo context on one card: an A3B MoE coder.

Full guide with GPU checks and troubleshooting: https://quantized.uk/cookbook/qwen-coder-32b-single-4090/
Model page with all quants: https://quantized.uk/quant-hub/qwen2.5-coder-32b/

Top comments (1)

Collapse
 
jaxsinclair profile image
Jax Sinclair •

Good table, and it is internally consistent in a way most fit posts are not. I checked your two KV numbers against the architectures: Qwen3-Coder-30B-A3B is 48 layers with 4 KV heads at head_dim 128, so 96 KiB per token, and 32K costs 3.0 GiB, which is exactly your 19.8 to 22.7. The dense 32B is 64 layers and 8 KV heads, so 256 KiB per token: 1 GiB at 4K, 4 GiB at 16K. That is why the AWQ build is the only one with room at 16K.

Then the part you leave open, because it is the part that decides whether the thing is usable. The ceilings differ by an order of magnitude. 1008 GB/s over 21.7 GB of dense weights is 46 tok/s at batch 1, before anyone loses a third to scheduling. The A3B reads about 2.2 GB per token, so its arithmetic roof is near 470 tok/s on the same card. Nobody gets either number, and the distance between roof and reality is the only thing on these cards worth measuring.

So, if one of these is your card and not a spec sheet: what does ollama report on the dense Q4 at 4K, -n 256, no other load? Your row goes first in the table I am building, with your handle on it.