Kristiyan Stoyanov

Posted on Jul 18

Speculative Decoding on Local Hardware: Benchmarking N-Gram, MTP, EAGLE3, and DFlash on the NVIDIA DGX Spark

#ai #llm #benchmark #agents

Speculative Decoding on Local Hardware: Benchmarking N-Gram, MTP, EAGLE3, and DFlash on the NVIDIA DGX Spark

TL;DR: We benchmark four speculative decoding methods — N-Gram, MTP, EAGLE3, and DFlash — running Qwen3.5-122B-A10B-hybrid-int4-fp8 on a single NVIDIA DGX Spark (GB10, SM121). MTP-2 delivered the best balance of throughput and stability (~49 tok/s avg, ~51 tok/s peak). DFlash achieved the highest individual peaks (~78 tok/s) but with significant variance. Configurations shared.

Why Speculative Decoding Matters for Local AI

Squeezing every token per second out of expensive local hardware is not just a nice-to-have — on machines like the DGX Spark where you've invested in a system to run frontier-scale models locally, inference throughput directly translates to latency in agentic pipelines, coding assistants, and multi-turn conversations.

Speculative decoding is one of the few optimization techniques that can deliver meaningful speedups without any change to output quality. Because the target model's final verification step guarantees mathematically identical output distributions to standard autoregressive decoding, there is zero quality trade-off.

LLM Inference: The Memory-Bandwidth Bottleneck

Before diving into speculative decoding mechanics, it's worth understanding why LLM inference is slow in the first place.

Every time an LLM generates a single token, the GPU must load the full model weight matrix from VRAM into compute units. For a 120B+ parameter model, this means moving tens of gigabytes of data — often for just a handful of floating-point operations per weight. This makes inference fundamentally memory-bandwidth bound, not compute-bound. The GPU's tensor cores sit largely idle while the memory bus does the heavy lifting.

Each inference step consists of two distinct phases:

Prefill (Prompt Processing): The input prompt tokens are processed in parallel. This phase is compute-bound and relatively fast, even for long contexts.
Decode (Generation): The model generates one token at a time, autoregressively. Each step requires a full forward pass through the model, loading all weights from memory. This is the slow phase — and it scales linearly with output length.

On the DGX Spark with the Qwen3.5-122B-A10B-hybrid-int4-fp8 checkpoint, the unoptimized baseline sits around 36–37 tok/s generation throughput. That's the number we're trying to beat.

How Speculative Decoding Works

The core idea is to break the serial token-by-token bottleneck. A small, fast draft model proposes a sequence of K candidate tokens in rapid succession. The large target model then verifies all K tokens in a single forward pass — because verification (running the full model over a known input) is significantly cheaper than independent generation.

The effective speedup depends on two competing factors:

Draft speed: How fast the draft model can propose tokens
Acceptance rate (α): What fraction of draft tokens the target model accepts

A very cheap but low-quality drafter (low α) will generate tokens faster than the target can accept, wasting compute. A very accurate drafter that nearly matches the target model will have a high α but won't be meaningfully faster to run. The sweet spot is a lightweight drafter that accurately mimics the target model's distribution for the task at hand.

The expected number of tokens produced per target-model forward pass is:

where α is the per-token acceptance rate and K is the number of speculative tokens. This means even modest acceptance rates (e.g., 0.8) with K=2 yield ~2.4 tokens per target pass — a meaningful multiplier.

The Test Setup

All tests were run on a single NVIDIA DGX Spark (GB10, SM121, 128 GB unified memory) with the following base setup:

Model: Qwen3.5-122B-A10B-hybrid-int4-fp8 (Intel AutoRound INT4 MoE experts + FP8 dense shared expert layers)
Inference engine: vLLM 0.19.1 compiled for SM121
Attention backend: FlashInfer
MTP weights patch: Required for the Intel AutoRound checkpoint (see below)

The hybrid checkpoint is worth noting: the MoE expert weights are quantized to INT4 (via Intel's AutoRound), while the dense shared expert layers use FP8 from the official Qwen FP8 checkpoint. This hybrid approach avoids the accuracy penalty of INT4-quantizing high-utilization attention layers.

Albond Recipe for Hybrid Checkpoint

Method 1: N-Gram Speculative Decoding

Concept: N-Gram matching is the simplest possible speculative decoding approach — and the most instructive for understanding the acceptance rate dynamic. It uses no learned model at all. Instead, it scans the current context window for recurring token n-grams and proposes continuations based on frequency co-occurrence.

Speculative configuration in vLLM:

{
  "method": "ngram",
  "num_speculative_tokens": 5,
  "prompt_lookup_num_tokens": 4
}

prompt_lookup_num_tokens controls how many tokens form the lookup key. The algorithm finds all positions in the context where the last N tokens appeared before and proposes what came after them.

Why it fails for open-ended generation: N-gram matching works well when the output is highly repetitive relative to the input (e.g., the model is echoing back parts of the prompt, or doing summarization). For free-form generation, the context has no meaningful prefix matches, so the acceptance rate drops sharply.

docker run -it --name vllm-qwen35 \
  --gpus all --net=host --ipc=host \
  -v /home/krisitown/AI/models:/models \
  vllm-qwen35-v2 \
  serve /models/qwen35-122b-hybrid-int4fp8 \
  --served-model-name qwen/qwen3.5 \
  --max-model-len 196608 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.88 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend FLASHINFER \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --generation-config auto \
  --override-generation-config '{"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
  --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":4}'

Result: ~24–30 tok/s — below the 36–37 tok/s baseline. The overhead of proposing tokens that get rejected outweighs any gain. N-Gram speculative decoding is only practical in highly repetitive/retrieval-heavy workloads.

Method 2: MTP (Multi-Token Prediction)

Concept: MTP is a fundamentally different approach. Rather than using a separate external draft model, MTP leverages a lightweight head that was trained jointly with the target model and ships as part of its checkpoint. Because the MTP head has been optimized to match the target model's own output distribution — sharing its internal representations — acceptance rates are dramatically higher than any external drafter.

The MTP head adds extra transformer layers at the end of the main model that are trained with an auxiliary loss to predict the next token in the sequence, using the intermediate hidden states of the main model. During inference, these layers run as a cheap drafting pass that reuses the KV cache already computed by the main model.

The MTP weights patch for Intel AutoRound:

The Intel AutoRound INT4 checkpoint for Qwen3.5-122B ships the MTP weights in model_extra_tensors.safetensors (785 tensors, ~4.8 GB BF16) but does not register them in model.safetensors.index.json. vLLM reads the index to discover weights, so it never loads the MTP head unless you patch the index manually. The recipe at albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4 includes a script that registers all 785 tensor mappings:

python patches/02-mtp-speculative/add-mtp-weights.py \
    --source "$INTEL_DIR" \
    --target "$MODEL_DIR"

Without this patch, adding --speculative-config will silently have no effect.

vLLM configuration:

vllm serve /models/qwen35-122b-hybrid-int4fp8 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

⚠️ MTP requires --attention-backend FLASHINFER. The PyTorch backend does not support MTP in vLLM ≤0.19.

Tuning num_speculative_tokens:

The number of speculative tokens is the key tuning parameter. Testing with 1, 2, and 4 tokens revealed a clear pattern:

`num_speculative_tokens`	Avg tok/s	Peak tok/s	Notes
0 (baseline)	~36–37	~38	No speculative decoding
1	~44–45	~46	Solid improvement
2	~48–49	~51	Best balance
4	~34–35	~38	Acceptance rate drops, worse than baseline

The degradation at num_speculative_tokens=4 is expected. Each additional speculative token compounds the error: if the MTP head diverges from the target model at position i, all subsequent positions i+1 ... K are already wrong, so the overhead of generating and then discarding them hurts throughput. The acceptance rate at position 4 was measured in the low 50s%, meaning roughly half of fourth-position tokens were rejected — and the overhead of running the head + verification + token correction exceeded the benefit.

Result: MTP-2 delivers ~49 tok/s average, ~51 tok/s peak — approximately +32% over baseline with stable, consistent throughput.

Method 3: EAGLE3

Concept: EAGLE3 is the third iteration of the EAGLE family of speculative decoding drafters. Unlike MTP, the EAGLE drafter is a separately trained model — but it compensates for this by conditioning on the target model's internal hidden states, not just the output token embeddings.

The intuition is: if the draft model has access to the target model's intermediate representations at the current position, it already "knows" most of what the target was going to compute before producing the next token. This dramatically improves acceptance rates compared to a purely external drafter.

EAGLE3 architectural changes from EAGLE1/2:

Multi-layer hidden state inputs: The drafter consumes hidden states from multiple layers of the target model (not just the final layer). Earlier layers carry richer semantic information that the final layer collapses into a logit distribution.
Removal of next-feature prediction loss: EAGLE-2 trained the drafter to predict both the next token and the next hidden state. EAGLE3 drops the hidden state prediction auxiliary loss, freeing model capacity for the actual goal.
Inference-aligned training data augmentation: During training, the drafter is exposed to its own autoregressive rollouts (not ground-truth hidden states), reducing the distribution shift between training and inference.

Practical constraint: EAGLE3 heads are model-specific and must be trained for a particular target model checkpoint. For this test, no EAGLE3 head was available for the 122B MoE model, so the test was conducted on a separate Qwen3-30B-A3B baseline (~33 tok/s without speculative decoding) using an EAGLE3 head pulled from Hugging Face.

vLLM configuration:

{
  "method": "eagle",
  "model": "model-name/eagle3-head",
  "num_speculative_tokens": 1
}

Results on Qwen3-30B-A3B:

`num_speculative_tokens`	Avg tok/s	Notes
0 (baseline)	~33	Qwen3-30B-A3B baseline
1	~38–40	~+20% improvement
12	~15–17	Severe degradation (8-11% acceptance rate)

The num_speculative_tokens=12 result is instructive. Even though the EAGLE3 drafter was running at 85–86 tok/s draft speed, an acceptance rate of 8–11% means the target model was rejecting the vast majority of drafts. The net result was roughly halved generation throughput compared to baseline.

This is the fundamental constraint of autoregressive drafters: each additional speculative token must be generated sequentially by the drafter. Drafting 12 tokens takes 12 forward passes through the draft model, and if most are rejected, you've spent significant time generating tokens the target will discard.

Result: EAGLE3 with num_speculative_tokens=1 delivered a ~+20% improvement on the 30B-A3B model. The method scales poorly with higher speculative token counts unless the acceptance rate is very high (>85%).

Method 4: DFlash (Block Diffusion Speculative Decoding)

Concept: DFlash is the most architecturally novel method in this comparison, and addresses the core limitation of autoregressive drafters: the serial bottleneck. Instead of generating draft tokens one at a time, DFlash uses a lightweight block diffusion model that generates a block of up to 16 tokens in a single forward pass.

The method draws inspiration from image generation diffusion models. In image diffusion, all pixels are generated in parallel via iterative denoising. DFlash adapts this concept to token generation: the drafter generates an entire block of masked tokens simultaneously, denoised in a single step.

Architecture:

The target model processes the prompt and extracts hidden context features
These features are injected directly into the KV cache of the draft diffusion block
The diffusion head generates K tokens in one forward pass, conditioned on the target model's deep representations
The target model verifies the proposed block with a single forward pass

The key difference from EAGLE3: because the block is generated in parallel, the marginal cost of generating additional speculative tokens approaches zero. Generating 16 tokens takes roughly the same time as generating 4. This means DFlash can use far higher speculative token counts without the exponential cost increase that punishes autoregressive drafters.

Performance ceiling: Published benchmarks from the original DFlash paper (Chen et al., arXiv:2602.06036) show over 6x lossless acceleration on Qwen3 models, delivering up to 2.5x higher speedup than EAGLE3 on reasoning-heavy tasks. On coding benchmarks, acceptance lengths of 6–7 tokens per step were reported — double what EAGLE3 achieves.

vLLM configuration (DFlash):

vllm serve /models/qwen35-122b-hybrid-int4fp8 \
  --attention-backend FLASHINFER \
  --speculative-config '{
    "method": "dflash",
    "model": "z-lab/Qwen3.5-122B-DFlash",
    "num_speculative_tokens": 16
  }'

The DFlash head for the Qwen3.5-122B model was created by ZLab and is available on Hugging Face. Setup requires following a specific GitHub recipe (link in video description).

Observed behavior: DFlash showed high throughput variance. When the acceptance rate was strong (high-structure output like code), peak throughput hit ~78 tok/s — significantly exceeding MTP-2. However, for general-purpose prompts, throughput fell to ~30–33 tok/s average, with peaks to ~50 tok/s.

The variance is explained by DFlash's acceptance semantics. Unlike autoregressive drafters where individual token rejections are isolated, DFlash accepts a prefix of the diffusion block. If the target model accepts tokens 1–3 but rejects token 4, tokens 5–16 are discarded entirely even if they would have been accepted. This means low acceptance rate at any position in the block wastes the entire remaining draft. When the task is highly structured (code, JSON, math), the acceptance rate is consistently high across positions, and DFlash dominates. For open-ended generation, variance is high.

Result: DFlash achieved the highest peak throughput (~78 tok/s) but also the highest variance. Average throughput for general prompts was lower than MTP-2. For structured-output tasks (coding agents, JSON generation), DFlash is compelling. It is also a relatively new method and actively improving.

Results Summary

Method	Model	Avg tok/s	Peak tok/s	Acceptance Rate	Notes
None (baseline)	Qwen3.5-122B	~36–37	~38	—	Reference
N-Gram	Qwen3.5-122B	~28–30	~31	Very low	Worse than baseline
MTP-1	Qwen3.5-122B	~44–45	~47	Mid-to-high 80s%	Solid improvement
MTP-2	Qwen3.5-122B	~48–49	~51	~80%+	Best consistency
MTP-4	Qwen3.5-122B	~34–35	~38	Drops significantly	Worse than MTP-2
None (baseline)	Qwen3-30B-A3B	~33	~40	—	Separate model
EAGLE3 (1 token)	Qwen3-30B-A3B	~38–39	~41	Good	~20% improvement
DFlash	Qwen3.5-122B	~48–50 avg*	~78	Variable	High peaks, high variance

*DFlash average varies significantly by task type (structured vs. open-ended).

Practical Guidance: Choosing the Right Method

Use MTP if your model ships with a native MTP head (Qwen3.5, DeepSeek V3, Gemma 4). It is the lowest-friction option — no separate model download, no architecture mismatch risk, and acceptance rates are consistently high because the head was jointly trained. num_speculative_tokens=2 is a strong default for most Qwen3.5 deployments.

Use EAGLE3 if no MTP head is available and a well-matched EAGLE3 drafter exists for your target model on Hugging Face. Keep num_speculative_tokens conservatively at 1–4; beyond that, acceptance rates drop sharply for most tasks.

Use DFlash for structured-output workloads: code generation, function calling, JSON mode, math. The parallel drafting architecture gives DFlash a fundamental advantage when the output has strong syntactic structure and acceptance rates across the block are consistently high. Avoid it for open-ended chat where variance makes latency unpredictable.

Avoid N-Gram for general inference. It is primarily useful as a pedagogical example or in specialized summarization/extraction pipelines where output closely mirrors input.

Key Takeaways

Speculative decoding is lossless. Output quality is mathematically identical to standard autoregressive decoding. The only variable is throughput.
Acceptance rate is everything. More speculative tokens are not always better. The acceptance rate at each position compounds — even a single bad position in an autoregressive chain invalidates all subsequent drafts.
DFlash's parallel drafting breaks the serial constraint. By generating blocks in a single forward pass, DFlash eliminates the per-token cost overhead that limits autoregressive drafters at high K. This is the architectural reason its peak throughput is so high.
Task type matters. Structured output (code, JSON, math) consistently benefits more from speculative decoding than open-ended generation, because the conditional distribution is sharper and easier for any drafter to model.
Hardware matters. These numbers are specific to the DGX Spark GB10 (SM121, 128 GB unified memory). Results on H100, A100, or consumer GPUs will differ due to memory bandwidth, compute, and kernel optimization differences.