I got Qwen3.8-27B running on dual RTX 3090s (no NVLink) under WSL2 — every pitfall I hit

DigitalMarket-World — Tue, 18 Aug 2026 09:02:23 +0000

Note: English translation assisted by an LLM. I'm not a native English speaker; all technical content, commands, and measurements come from my own setup.

I spent a full week getting Qwen3.8-27B (hybrid GDN architecture, 48 linear-attention + 16 full-attention layers, built-in MTP head) running on my dual RTX 3090 (24GB × 2) box — no NVLink, PCIe Gen4, Windows 10 + WSL2 (Ubuntu-24.04). I hit basically every trap in the book. This is the complete record: what broke, why, and the config that finally gave me 170-210 tok/s on code/JSON.

My setup

Item	Configuration
GPU	2× NVIDIA RTX 3090 (24GB GDDR6X, SM86/Ampere)
GPU interconnect	No NVLink (PCIe Gen4 x8 bridge)
CPU / RAM	Ryzen 9 5950X / 64GB DDR4
OS	Windows 10 + WSL2 (Ubuntu-24.04)
Inference engine	SGLang 0.5.17
CUDA toolchain	CUDA 13.0 (this exact version matters, see Pitfall 2)
Model	Qwen3.8-27B-AWQ-INT4 (cyankiwi quant, ~21GB)
Speculative decoding	DSpark (1.4B draft model)
Context length	245,760 tokens

The timeline

vLLM: compilation hell

vLLM was the obvious first try. It did not go well:

vLLM 0.25.1 (Docker): loads, but MTP speculation gives zero benefit on two GPUs without NVLink (56 vs 58 tok/s) and pushes first-token latency from 8.7s to 15s.
vLLM 0.27.1: hits missing libnvrtc.so.13 (nvidia-cuda-nvrtc was a 0.0.0a0 placeholder package). After force-reinstalling, flashinfer's sampling JIT demands the CUDA-13-only --host-stub-linkage-explicit while local nvcc is 12.8. Engine init crashes every time.

Verdict: skip vLLM 0.27+ for Qwen3.8 on 3090s. The old Docker 0.25.1 works but gives no speedup.

SGLang: frozen at weight loading

Switching to SGLang was worse: the model froze during weight loading — process hung, port never listened.

I suspected parameters and tried dozens of combinations. Useless. Comparing keys in model.safetensors.index.json finally revealed the root cause:

cyankiwi's Qwen3.8-AWQ-INT4 is a "half-quantized" checkpoint — the MTP head and some linear_attn layers use plain weight instead of weight_packed, unlike Qwen3.6's fully-quantized checkpoints. Same architecture config: Qwen3.6 runs, Qwen3.8 dies on load.

The breakthrough: CUDA 13.0 + SGLang 0.5.17

The real fix was toolchain matching:

CUDA	Result
12.8	weight loading hangs
13.3	`__NV_ATOMIC_RELAXED` macro removed → verify-graph compilation fails
13.0 ✅	perfect match for torch cu130; keeps the cccl macros flashinfer 0.6.15 needs

Plus the no-NVLink triple fix:

--disable-custom-all-reduce     # avoid "peer access is not supported"
--mm-feature-transport cpu      # bypass pidfd_getfd crash in CUDA IPC under WSL
NCCL_P2P_DISABLE=1              # must disable P2P without NVLink

And a SymmMem patch: set MultimemAllGatherer.enabled=False in logits_processor.py to bypass the SIGFPE from torch.distributed._symmetric_memory.rendezvous() on 3090s.

It ran! At ~10 tok/s. Unusable.

Speed optimization: systematic A/B of every speculative decoding option

Approach	Acceptance	Speed	Verdict
No speculation (baseline)	—	~63 tok/s	start
MTP/EAGLE (built-in head)	1.07-1.27	~37 tok/s	❌ negative — INT4 quantization damaged the BF16 head
NGRAM	—	~56 tok/s	❌ +50% on JSON only, -30% on normal text
ReplaySSM	1.45→1.55	40→43 tok/s	❌ negligible
DSpark (1.4B draft model)	3.7-4.2	170-210 tok/s	✅✅✅

DSpark was the only thing that worked. An independent 1.4B draft model proposes a 7-token candidate block per step; the main model verifies the block in one pass. Acceptance of 3.7-4.2 means ~4 tokens accepted per verification — 3x better than MTP on this quantized model.

Pitfalls (each one cost me real hours)

1. Lost line-continuation backslashes — the sneakiest one

Symptom: service starts, but server_args= log shows default values for everything — mm_feature_transport='cuda_ipc', speculative_algorithm=None.

Cause: a trailing \ went missing in the launch script; bash truncates the command there and silently discards every parameter after it. The service boots with bare defaults, which crashes on WSL without NVLink.

Lesson: after editing the launch script, verify with grep server_args= that your parameters actually took effect before tuning anything. This may be the truth behind 80% of "my parameters don't work" reports.

2. CUDA toolchain version is life or death

Qwen3.8's GDN kernels need CUDA-13 compilation. 12.8 hangs on load, 13.3 fails to compile, 13.0 is exactly right. Check which cu version your SGLang torch was built against (cu130), then match the toolchain exactly.

3. SM86 (Ampere) hard ceiling

flashinfer's GDN kernels require SM90+; the 3090 is SM86 → only the Triton linear-attention backend works (hard architecture limit, not config).
--enable-torch-compile crashes on GDN (launcher() missing '_grid_2' — inductor can't compile causal_conv1d).
Don't waste time on these.

4. MTP is doomed on INT4 weights

Qwen3.8's built-in MTP head is BF16, but this INT4 quant quantized the head too — it misjudges on INT4 hidden states, acceptance never rises (1.07-1.27). Not a parameter problem; a quantization problem. Either use a W4A16 model that keeps the BF16 head (limited gain without P2P) or go DSpark.

5. "Officially recommended" isn't right for your workload

--mamba-radix-cache-strategy extra_buffer (recommended to avoid GDN degradation): halves decode speed (62→30 tok/s). Permanent penalty to avoid a rare failure. Not worth it.
--chunked-prefill-size 2048 (recommended for concurrency): for single-request + 50-90K token agent contexts it quadruples prefill iterations and slows TTFT. I use 8192.

Test your own workload before copying official parameters.

6. Memory allocation is the lifeline of CUDA graph capture

--mem-fraction-static too high + large context → no headroom for CUDA graph capture → verify-graph hangs (log stalls at Capturing batches 0%, GPU 100% but CPU time frozen). Fix: 0.8 (DSpark also needs VRAM for the 1.4B draft), or --cuda-graph-backend-prefill disabled.

How to tell a hang from progress: ps -o pid,pcpu,time — CPU TIME growing = healthy, frozen = deadlock. Don't kill the process prematurely; graph capture legitimately takes 90-120s.

7. `--served-model-name` must be a short name

Clients send short model names; if the server only exposes the full path → detokenizer hangs, health checks fail, everything times out. Add --served-model-name <short-name>.

8. Multiple launch scripts resurrect old configs

I had 6 launch entry points (.bashrc, systemd, watchdog, Windows VBS chain…). Editing parameters in one script while a watchdog pulls from another → old config "resurrects" after reboot. Audit all of them:

grep -r "sglang\|start-models" ~/.bashrc ~/.wsl-hermes/ /etc/systemd/

and point every entry point at the same config file.

Final config

python -m sglang.launch_server \
  --model-path /home/user2222/models/cyankiwiQwen3.8-27B-AWQ-INT4 \
  --served-model-name cyankiwiQwen3.8-27B-AWQ-INT4 \
  --port 9090 --tp-size 2 \
  --quantization compressed-tensors \
  --mem-fraction-static 0.8 \
  --kv-cache-dtype fp8_e4m3 \
  --chunked-prefill-size 8192 \
  --context-length 245760 \
  --dtype bfloat16 --mamba-ssm-dtype bfloat16 \
  --disable-custom-all-reduce \
  --enable-tf32-matmul --schedule-policy lpm \
  --trust-remote-code \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path /home/user2222/models/Qwen3.8-27B-DSpark \
  --speculative-dspark-block-size 7 \
  --speculative-draft-model-quantization unquant \
  --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --max-running-requests 2 --allow-auto-truncate \
  --cuda-graph-bs-decode 1 2 3 4 5 \
  --cuda-graph-backend-prefill disabled \
  --mm-feature-transport cpu --stream-interval 1

Measured results

Official-style benchmarks (DSpark vs no speculation):

Task	No speculation	DSpark	Speedup
Code/JSON	~63	170-210 tok/s	3x
Deep reasoning	~63	153 tok/s	2.4x
Greedy sampling	~63	215 tok/s	3.4x
Plain text	~63	57 tok/s	≈parity
Long context	~63	61 tok/s	≈parity
Speculative acceptance	—	3.7-4.2	3x MTP

End-to-end on real tasks (includes prefill + thinking tokens, so below pure decode peak):

Task	Time	Output tokens	End-to-end
Code (quicksort)	11.6s	800	~69 tok/s
JSON structured output	4.2s	263	~63 tok/s
Prose	6.2s	298	~48 tok/s
Math reasoning	6.3s	439	~69 tok/s

Stability: 245,760 context, GPU0 23.9GB / GPU1 21.3GB, running 24/7 as an agent backend (code, JSON, tool calls, deep reasoning) with no crashes.

TL;DR lessons

Read the official model card + cookbook after a new release; don't force-fit last gen's parameters.
Engine, toolchain, and quant format must match: CUDA 13.0 + SGLang 0.5.17 + AWQ-INT4 was the answer here.
When MTP fails, try DSpark — acceptance 3.7 vs 1.2 says everything.
Without NVLink, don't expect 2x scaling; 170-210 tok/s is already beyond this combo's expected ceiling.
Check the launch script's line continuations before changing any parameter. Saved me countless times.

Happy to answer questions in the comments — this took me a week and I'd rather you skip it.

Originally written for DigitalMarket.World (World Digital Economy Network). All data from real measurements on my own hardware. Free to share with credit.

DEV Community: DigitalMarket-World