DEV Community

Jovan Chan
Jovan Chan

Posted on • Originally published at aifoss.dev

Qwen3.6-35B-A3B Local Setup 2026: Ollama and 24GB VRAM

#ollama #llm #coding #selfhosted

This article was originally published on aifoss.dev

TL;DR: Qwen3.6-35B-A3B is a 35B Mixture-of-Experts model with only ~3B active parameters per token, so it loads like a 35B model but runs at the speed of a 3B one. At Q4_K_M it needs roughly 21 GB, so a single 24 GB GPU runs it with usable context. The catch: long context blows up the KV cache fast, and you have to manage that.

What you'll have running after this guide:

  • Qwen3.6-35B-A3B serving a local OpenAI-compatible API through Ollama
  • Cline or Continue.dev wired to that endpoint for in-editor agentic coding
  • A llama.cpp fallback with MoE CPU offload for GPUs smaller than 24 GB

Why this model is worth the setup

Open-weight coding models usually force a tradeoff: dense 30B+ models give you frontier-ish quality but want 20–40 GB of VRAM, while the 7–14B dense models fit a consumer card but trail on real repo-level work.

Qwen3.6-35B-A3B routes around that. It's a sparse MoE from Alibaba's Qwen team, released April 16, 2026 under the Apache 2.0 license — commercial use unrestricted. There are 35B total parameters spread across 256 experts (8 routed plus 1 shared active per token), but only about 3B parameters fire on any given forward pass. You pay roughly the inference cost of a 3B dense model while the full 35B weight pool supplies the knowledge.

The numbers Qwen published back that up:

Benchmark Score
SWE-bench Verified 73.4%
Terminal-Bench 2.0 51.5%
AIME 2026 92.6%
Context (native) 262K tokens
Context (with YaRN) up to 1M
License Apache 2.0

The architecture mixes Gated DeltaNet linear attention with standard gated attention and the sparse MoE block, and it's trained with Multi-Token Prediction. The model is also natively multimodal — it accepts text and images — which matters less for pure coding but is there if you want it.

If you've already set up Qwen3-Coder-Next locally, the workflow here is nearly identical, just with a smaller memory footprint.

Hardware requirements

MoE models front-load memory. Even though only ~3B parameters activate per token, all 35B weights sit in memory at load time. Quantization is what makes this fit on a single card. Here's the realistic picture, drawn from community VRAM measurements and the GGUF file sizes on Hugging Face:

Quant Approx. size Min. VRAM (short ctx) Fits on
UD-Q4_K_XL (Unsloth dynamic) ~19–20 GB ~22 GB 24 GB GPU
Q4_K_M ~21 GB ~24 GB 24 GB GPU / 32 GB Mac
Q5_K_M ~25 GB ~28 GB 32 GB Mac / 2× GPU
Q8_0 ~37 GB ~40 GB 48 GB GPU / 64 GB Mac
BF16 ~70 GB ~80 GB A100 80GB / 96 GB Mac

For a single 24 GB card — an RTX 3090, RTX 4090, or RTX 5090 — Q4_K_M is the sweet spot. Unsloth's dynamic UD-Q4_K_XL quant trims another 1–2 GB by quantizing non-critical tensors harder, which buys you headroom for context. On Apple Silicon, a Mac with 32 GB unified memory handles Q4_K_M comfortably.

One thing to plan for: that ~21 GB is weights only. The KV cache grows with context length, and at the model's full 262K window it can add tens of GB. On a 24 GB card you'll run Q4_K_M with maybe 16K–32K of context, not the full 262K. More on that in the OOM section.

If you don't own a 24 GB card and don't want to buy one, renting an RTX 4090 or A100 on RunPod costs a fraction of a dollar per hour and skips the local memory math entirely. For a deeper look at which GPU to actually buy for local LLMs, see the home lab GPU guides on runaihome.com.

Step 1: Install Ollama and pull the model

Ollama is the lowest-friction path. Install it (Linux shown; macOS and Windows have native installers):

curl -fsSL https://ollama.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Confirm it's running:

$ ollama --version
ollama version is 0.17.1
Enter fullscreen mode Exit fullscreen mode

Then pull the model. The default tag is the Q4_K_M-class build:

ollama pull qwen3.6:35b-a3b
Enter fullscreen mode Exit fullscreen mode

That download is about 24 GB, so give it a few minutes. Ollama publishes several tags if you want a specific footprint:

  • qwen3.6:35b-a3b — default, ~24 GB
  • qwen3.6:35b-a3b-q4_K_M — explicit Q4_K_M
  • qwen3.6:35b-a3b-mxfp8 — MXFP8, larger, higher quality
  • qwen3.6:35b-a3b-bf16 — full precision, needs 80 GB-class memory

Stick with the default unless you have a reason not to.

Step 2: Run it and check the output

Start an interactive session:

$ ollama run qwen3.6:35b-a3b
>>> Write a Python function that returns the nth Fibonacci number using memoization.

def fib(n, _cache={0: 0, 1: 1}):
    if n not in _cache:
        _cache[n] = fib(n - 1) + _cache.setdefault(n - 2, fib(n - 2))
    return _cache[n]
Enter fullscreen mode Exit fullscreen mode

On a 24 GB GPU you should see generation in the ballpark of 40–70 tokens/second once the model is loaded — that MoE sparsity is the reason a "35B" model feels this quick. The first response is slower because the weights are loading into VRAM.

To run it as a background server with the OpenAI-compatible API (what the editor integrations need):

ollama serve
Enter fullscreen mode Exit fullscreen mode

That exposes an endpoint at http://localhost:11434/v1. Quick sanity check with curl:

curl http://localhost:11434/v1/chat/completions -d '{
  "model": "qwen3.6:35b-a3b",
  "messages": [{"role": "user", "content": "Say hello in one word."}]
}'
Enter fullscreen mode Exit fullscreen mode

Sampling settings

Qwen ships recommended sampling parameters on the model card — check it rather than guessing, because the 3.x series separates "thinking" and "non-thinking" defaults. As a rule, lower temperature (around 0.6–0.7) suits coding and agentic work where you want determinism; greedy decoding (temperature 0) tends to cause repetition loops on this architecture, so avoid it. Set these in your client or in an Ollama Modelfile with PARAMETER temperature 0.7.

Step 3: Wire it into your editor

The point of a local coding model is using it where you write code. Two solid open-source options, both covered in depth on our sister site for AI coding tools, aicoderscope.com:

Cline (VS Code agent): Install the Cline extension, open settings, choose "Ollama" as the API provider, set the base URL to http://localhost:11434, and pick qwen3.6:35b-a3b from the model list. Our Cline setup guide walks through the agentic config in detail.

Continue.dev (inline + chat): Add a block to ~/.continue/config.json:

{
  "models": [
    {
      "title": "Qwen3.6-35B-A3B",
      "provider": "ollama",
      "model": "qwen3.6:35b-a3b",
      "apiBase": "http://localhost:11434"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Continue then uses it for chat, edits, and codebase questions. The Continue.dev + Ollama guide covers autocomplete config too.

For agentic tasks, the 262K context window is the real draw — you can feed a whole feature branch and let the model reason across files. Just remember the KV cache cost.

Step 4: The low-VRAM fallback (llama.cpp + CPU offload)

No 24 GB card? Because only ~3B parameters are active per token, you can keep the bulk of the MoE experts in system RAM and only stream the active path onto the GPU. llama.cpp supports this with --n-cpu-moe. One community writeup reported running this model at roughly 30 tokens/second on just 6 GB of VRAM by offloading MoE layers to CPU, given enough system RAM.

Build llama.cpp, grab a GGUF (Unsloth's Qwen3.6-35B-A3B-GGUF repo), then:

./llama-server \
  -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
  --n-cpu-moe 28 \
  --ctx-size 16384 \
  --host 0.0.0.0 --port 8080
Enter fullscreen mode Exit fullscreen mode

Tune --n-cpu-moe to how many MoE layers you push to CPU: higher number = less VRAM, slower generation. You'll want 32+ GB of system RAM for

Top comments (0)