Why Qwen2 Was Running at Half Speed

While building mini-vllm-rs, I unified Candle's quantized Llama and Qwen2 implementations into a shared transformer backend. When I later compared our backend against upstream Candle to see what we could upstream, one difference stood out immediately.

In candle-transformers, the quantized Llama backend routes single-token Metal decode steps through Candle's fused Scaled Dot-Product Attention (sdpa) kernel, which computes Grouped-Query Attention (GQA) directly over the original, unexpanded key/value heads.

However, the quantized Qwen2 backend was missing that fast path: every single-token decode step instead materialized expanded copies of K and V across all query heads, then ran separate matmul + softmax + matmul dispatches.

Routing that decode through fused SDPA (PR #4092):

• Speeds up isolated Metal decode attention by 4.1×–10.3× per-layer on Qwen2.5-0.5B and 6.3×–15.2× per-layer on Qwen2.5-7B.
• Improves end-to-end generation throughput in Candle's built-in quantized-qwen2-instruct example by ~1.28× (+28%) at short context lengths, with a separate 127-token decode benchmark reaching 2.02× at a 2K-token prompt length on an M2 Mac.

Why It Matters: 7× KV Expansion on Every Decode Step

At long context lengths, single-token attention becomes increasingly memory-bandwidth-bound: each new token must attend over the growing KV cache.

The two model configurations benchmarked here both use an aggressive 7:1 GQA ratio:

• Qwen2.5-0.5B: 14 query heads, 2 KV heads (head_dim = 64, 24 layers)
• Qwen2.5-7B: 28 query heads, 4 KV heads (head_dim = 128, 28 layers)

Without the fused SDPA path, every generated token had to:

1. Materialize temporary K and V tensors with 7× as many head elements to match the query head count.
2. Run three separate Metal kernel dispatches (Q × K^T, softmax, and Attn × V) that read back that expanded KV footprint.

To put concrete numbers on the memory traffic: at a 2,048-token context length on Qwen2.5-7B (F32 KV cache), a single layer's unexpanded KV cache is 8 MiB:

2 (K, V) × 4 KV heads × 2048 tokens × 128 dim × 4 bytes = 8 MiB / layer

Expanding both KV tensors by 7× allocates and writes 56 MiB per layer (or 1.53 GiB across 28 layers) for a single generated token, before attention even begins.

By contrast, Candle's fused Metal vector attention kernel indexes the 4 KV heads directly using a GQA stride factor in a single pass, completely avoiding the copies and intermediate attention matrix materialization.

This isn't the first time I've explored this trade-off. In mini-vllm-rs, I benchmarked several CPU attention implementations, including one that groups query heads instead of repeating KV heads.

Grouped Q improved decode throughput by 26% with F32 activations and 11% with F16, while significantly reducing peak memory usage in the F32 benchmark.

• F32 CPU paged-attention benchmark
• F16 CPU paged-attention benchmark

Benchmarks

All benchmarks below were run on an Apple M2 (8-core CPU, 10-core GPU, 16 GB unified memory) with Candle 0.11.1, using the Metal backend.

1. Candle's Built-in End-to-End Benchmark

Measured using Candle's built-in quantized-qwen2-instruct example (Qwen2-0.5B-Instruct q4_0 GGUF) with greedy decoding on two workloads:

Workload	Prompt Tokens	Generated Tokens	Upstream (tok/s)	Metal SDPA (tok/s)	End-to-End Speedup
Default prompt	20	292	118.24	150.94	1.28× (+28%)
Long generation	36	948	99.41	125.99	1.27× (+27%)

2. Isolated Decode Attention Microbenchmark

To isolate the attention cost from quantized MLP and LM-head projections, each row below measures 100 synchronized Metal iterations (after 20 warm-up iterations) for a single decode step across context lengths from 128 to 4,096 tokens.

2.1 Qwen2.5-0.5B Head Geometry

The last column estimates the total attention time saved across all layers for each generated token.

Context Length	Upstream (µs / layer)	Metal SDPA (µs / layer)	Layer Speedup	Estimated Time Saved per Token (ms)
128	66.72	16.27	4.10×	1.211
256	87.47	23.98	3.65×	1.524
512	111.94	19.66	5.69×	2.215
1,024	464.68	56.83	8.18×	9.789
2,048	518.39	55.43	9.35×	11.111
4,096	970.22	94.50	10.27×	21.017

2.2 Qwen2.5-7B Head Geometry

Context Length	Upstream (µs / layer)	Metal SDPA (µs / layer)	Layer Speedup	Estimated Time Saved per Token (ms)
128	97.62	15.46	6.31×	2.300
256	235.88	28.32	8.33×	5.812
512	451.58	34.46	13.10×	11.679
1,024	862.74	87.30	9.88×	21.712
2,048	1,926.05	148.10	13.01×	49.783
4,096	4,018.55	264.82	15.17×	105.104

At a 4,096-token context on Qwen2.5-7B, eliminating the KV expansion and fusing attention saves 105 ms per generated token in attention alone.

3. End-to-End Decode Scaling by Context Length

As the KV cache grows, attention accounts for an increasing share of total decode latency. Measuring 127 autoregressive decode steps across fixed prompt lengths on Qwen2.5-0.5B shows how the end-to-end speedup scales with context length:

Prompt Tokens	Upstream Decode (ms)	Metal SDPA Decode (ms)	Upstream (tok/s)	Metal SDPA (tok/s)	End-to-End Speedup
128	1,037.96	835.34	122.36	152.03	1.24×
512	1,302.36	899.51	97.52	141.19	1.45×
1,024	1,619.19	1,038.02	78.43	122.35	1.56×
2,048	2,237.02	1,106.48	56.77	114.78	2.02×

At 2,048 prompt tokens, upstream decode throughput drops by more than half (122.36 → 56.77 tok/s), whereas fused SDPA sustains 114.78 tok/s, a 2.02× end-to-end speedup.

What I learned

The interesting part wasn't that I had to invent a new attention kernel. Candle already had a fast, fused Metal implementation capable of handling GQA. Quantized Qwen2 simply wasn't using it.

Sometimes the biggest performance wins come from finding the fast path that was already there.

The change is in Candle PR #4092. It targets single-token decode on Metal, while prefill and other backends continue to use the existing path.