DEV Community: Kristiyan Stoyanov

Speculative Decoding on Local Hardware: Benchmarking N-Gram, MTP, EAGLE3, and DFlash on the NVIDIA DGX Spark

Kristiyan Stoyanov — Sat, 18 Jul 2026 18:20:03 +0000

Speculative Decoding on Local Hardware: Benchmarking N-Gram, MTP, EAGLE3, and DFlash on the NVIDIA DGX Spark

TL;DR: We benchmark four speculative decoding methods — N-Gram, MTP, EAGLE3, and DFlash — running Qwen3.5-122B-A10B-hybrid-int4-fp8 on a single NVIDIA DGX Spark (GB10, SM121). MTP-2 delivered the best balance of throughput and stability (~49 tok/s avg, ~51 tok/s peak). DFlash achieved the highest individual peaks (~78 tok/s) but with significant variance. Configurations shared.

Why Speculative Decoding Matters for Local AI

Squeezing every token per second out of expensive local hardware is not just a nice-to-have — on machines like the DGX Spark where you've invested in a system to run frontier-scale models locally, inference throughput directly translates to latency in agentic pipelines, coding assistants, and multi-turn conversations.

Speculative decoding is one of the few optimization techniques that can deliver meaningful speedups without any change to output quality. Because the target model's final verification step guarantees mathematically identical output distributions to standard autoregressive decoding, there is zero quality trade-off.

LLM Inference: The Memory-Bandwidth Bottleneck

Before diving into speculative decoding mechanics, it's worth understanding why LLM inference is slow in the first place.

Every time an LLM generates a single token, the GPU must load the full model weight matrix from VRAM into compute units. For a 120B+ parameter model, this means moving tens of gigabytes of data — often for just a handful of floating-point operations per weight. This makes inference fundamentally memory-bandwidth bound, not compute-bound. The GPU's tensor cores sit largely idle while the memory bus does the heavy lifting.

Each inference step consists of two distinct phases:

Prefill (Prompt Processing): The input prompt tokens are processed in parallel. This phase is compute-bound and relatively fast, even for long contexts.
Decode (Generation): The model generates one token at a time, autoregressively. Each step requires a full forward pass through the model, loading all weights from memory. This is the slow phase — and it scales linearly with output length.

On the DGX Spark with the Qwen3.5-122B-A10B-hybrid-int4-fp8 checkpoint, the unoptimized baseline sits around 36–37 tok/s generation throughput. That's the number we're trying to beat.

How Speculative Decoding Works

The core idea is to break the serial token-by-token bottleneck. A small, fast draft model proposes a sequence of K candidate tokens in rapid succession. The large target model then verifies all K tokens in a single forward pass — because verification (running the full model over a known input) is significantly cheaper than independent generation.

The effective speedup depends on two competing factors:

Draft speed: How fast the draft model can propose tokens
Acceptance rate (α): What fraction of draft tokens the target model accepts

A very cheap but low-quality drafter (low α) will generate tokens faster than the target can accept, wasting compute. A very accurate drafter that nearly matches the target model will have a high α but won't be meaningfully faster to run. The sweet spot is a lightweight drafter that accurately mimics the target model's distribution for the task at hand.

The expected number of tokens produced per target-model forward pass is:

where α is the per-token acceptance rate and K is the number of speculative tokens. This means even modest acceptance rates (e.g., 0.8) with K=2 yield ~2.4 tokens per target pass — a meaningful multiplier.

The Test Setup

All tests were run on a single NVIDIA DGX Spark (GB10, SM121, 128 GB unified memory) with the following base setup:

Model: Qwen3.5-122B-A10B-hybrid-int4-fp8 (Intel AutoRound INT4 MoE experts + FP8 dense shared expert layers)
Inference engine: vLLM 0.19.1 compiled for SM121
Attention backend: FlashInfer
MTP weights patch: Required for the Intel AutoRound checkpoint (see below)

The hybrid checkpoint is worth noting: the MoE expert weights are quantized to INT4 (via Intel's AutoRound), while the dense shared expert layers use FP8 from the official Qwen FP8 checkpoint. This hybrid approach avoids the accuracy penalty of INT4-quantizing high-utilization attention layers.

Albond Recipe for Hybrid Checkpoint

Method 1: N-Gram Speculative Decoding

Concept: N-Gram matching is the simplest possible speculative decoding approach — and the most instructive for understanding the acceptance rate dynamic. It uses no learned model at all. Instead, it scans the current context window for recurring token n-grams and proposes continuations based on frequency co-occurrence.

Speculative configuration in vLLM:

{
  "method": "ngram",
  "num_speculative_tokens": 5,
  "prompt_lookup_num_tokens": 4
}

prompt_lookup_num_tokens controls how many tokens form the lookup key. The algorithm finds all positions in the context where the last N tokens appeared before and proposes what came after them.

Why it fails for open-ended generation: N-gram matching works well when the output is highly repetitive relative to the input (e.g., the model is echoing back parts of the prompt, or doing summarization). For free-form generation, the context has no meaningful prefix matches, so the acceptance rate drops sharply.

docker run -it --name vllm-qwen35 \
  --gpus all --net=host --ipc=host \
  -v /home/krisitown/AI/models:/models \
  vllm-qwen35-v2 \
  serve /models/qwen35-122b-hybrid-int4fp8 \
  --served-model-name qwen/qwen3.5 \
  --max-model-len 196608 \
  --max-num-batched-tokens 32768 \
  --gpu-memory-utilization 0.88 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend FLASHINFER \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --generation-config auto \
  --override-generation-config '{"temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
  --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":4}'

Result: ~24–30 tok/s — below the 36–37 tok/s baseline. The overhead of proposing tokens that get rejected outweighs any gain. N-Gram speculative decoding is only practical in highly repetitive/retrieval-heavy workloads.

Method 2: MTP (Multi-Token Prediction)

Concept: MTP is a fundamentally different approach. Rather than using a separate external draft model, MTP leverages a lightweight head that was trained jointly with the target model and ships as part of its checkpoint. Because the MTP head has been optimized to match the target model's own output distribution — sharing its internal representations — acceptance rates are dramatically higher than any external drafter.

The MTP head adds extra transformer layers at the end of the main model that are trained with an auxiliary loss to predict the next token in the sequence, using the intermediate hidden states of the main model. During inference, these layers run as a cheap drafting pass that reuses the KV cache already computed by the main model.

The MTP weights patch for Intel AutoRound:

The Intel AutoRound INT4 checkpoint for Qwen3.5-122B ships the MTP weights in model_extra_tensors.safetensors (785 tensors, ~4.8 GB BF16) but does not register them in model.safetensors.index.json. vLLM reads the index to discover weights, so it never loads the MTP head unless you patch the index manually. The recipe at albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4 includes a script that registers all 785 tensor mappings:

python patches/02-mtp-speculative/add-mtp-weights.py \
    --source "$INTEL_DIR" \
    --target "$MODEL_DIR"

Without this patch, adding --speculative-config will silently have no effect.

vLLM configuration:

vllm serve /models/qwen35-122b-hybrid-int4fp8 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

⚠️ MTP requires --attention-backend FLASHINFER. The PyTorch backend does not support MTP in vLLM ≤0.19.

Tuning num_speculative_tokens:

The number of speculative tokens is the key tuning parameter. Testing with 1, 2, and 4 tokens revealed a clear pattern:

`num_speculative_tokens`	Avg tok/s	Peak tok/s	Notes
0 (baseline)	~36–37	~38	No speculative decoding
1	~44–45	~46	Solid improvement
2	~48–49	~51	Best balance
4	~34–35	~38	Acceptance rate drops, worse than baseline

The degradation at num_speculative_tokens=4 is expected. Each additional speculative token compounds the error: if the MTP head diverges from the target model at position i, all subsequent positions i+1 ... K are already wrong, so the overhead of generating and then discarding them hurts throughput. The acceptance rate at position 4 was measured in the low 50s%, meaning roughly half of fourth-position tokens were rejected — and the overhead of running the head + verification + token correction exceeded the benefit.

Result: MTP-2 delivers ~49 tok/s average, ~51 tok/s peak — approximately +32% over baseline with stable, consistent throughput.

Method 3: EAGLE3

Concept: EAGLE3 is the third iteration of the EAGLE family of speculative decoding drafters. Unlike MTP, the EAGLE drafter is a separately trained model — but it compensates for this by conditioning on the target model's internal hidden states, not just the output token embeddings.

The intuition is: if the draft model has access to the target model's intermediate representations at the current position, it already "knows" most of what the target was going to compute before producing the next token. This dramatically improves acceptance rates compared to a purely external drafter.

EAGLE3 architectural changes from EAGLE1/2:

Multi-layer hidden state inputs: The drafter consumes hidden states from multiple layers of the target model (not just the final layer). Earlier layers carry richer semantic information that the final layer collapses into a logit distribution.
Removal of next-feature prediction loss: EAGLE-2 trained the drafter to predict both the next token and the next hidden state. EAGLE3 drops the hidden state prediction auxiliary loss, freeing model capacity for the actual goal.
Inference-aligned training data augmentation: During training, the drafter is exposed to its own autoregressive rollouts (not ground-truth hidden states), reducing the distribution shift between training and inference.

Practical constraint: EAGLE3 heads are model-specific and must be trained for a particular target model checkpoint. For this test, no EAGLE3 head was available for the 122B MoE model, so the test was conducted on a separate Qwen3-30B-A3B baseline (~33 tok/s without speculative decoding) using an EAGLE3 head pulled from Hugging Face.

vLLM configuration:

{
  "method": "eagle",
  "model": "model-name/eagle3-head",
  "num_speculative_tokens": 1
}

Results on Qwen3-30B-A3B:

`num_speculative_tokens`	Avg tok/s	Notes
0 (baseline)	~33	Qwen3-30B-A3B baseline
1	~38–40	~+20% improvement
12	~15–17	Severe degradation (8-11% acceptance rate)

The num_speculative_tokens=12 result is instructive. Even though the EAGLE3 drafter was running at 85–86 tok/s draft speed, an acceptance rate of 8–11% means the target model was rejecting the vast majority of drafts. The net result was roughly halved generation throughput compared to baseline.

This is the fundamental constraint of autoregressive drafters: each additional speculative token must be generated sequentially by the drafter. Drafting 12 tokens takes 12 forward passes through the draft model, and if most are rejected, you've spent significant time generating tokens the target will discard.

Result: EAGLE3 with num_speculative_tokens=1 delivered a ~+20% improvement on the 30B-A3B model. The method scales poorly with higher speculative token counts unless the acceptance rate is very high (>85%).

Method 4: DFlash (Block Diffusion Speculative Decoding)

Concept: DFlash is the most architecturally novel method in this comparison, and addresses the core limitation of autoregressive drafters: the serial bottleneck. Instead of generating draft tokens one at a time, DFlash uses a lightweight block diffusion model that generates a block of up to 16 tokens in a single forward pass.

The method draws inspiration from image generation diffusion models. In image diffusion, all pixels are generated in parallel via iterative denoising. DFlash adapts this concept to token generation: the drafter generates an entire block of masked tokens simultaneously, denoised in a single step.

Architecture:

The target model processes the prompt and extracts hidden context features
These features are injected directly into the KV cache of the draft diffusion block
The diffusion head generates K tokens in one forward pass, conditioned on the target model's deep representations
The target model verifies the proposed block with a single forward pass

The key difference from EAGLE3: because the block is generated in parallel, the marginal cost of generating additional speculative tokens approaches zero. Generating 16 tokens takes roughly the same time as generating 4. This means DFlash can use far higher speculative token counts without the exponential cost increase that punishes autoregressive drafters.

Performance ceiling: Published benchmarks from the original DFlash paper (Chen et al., arXiv:2602.06036) show over 6x lossless acceleration on Qwen3 models, delivering up to 2.5x higher speedup than EAGLE3 on reasoning-heavy tasks. On coding benchmarks, acceptance lengths of 6–7 tokens per step were reported — double what EAGLE3 achieves.

vLLM configuration (DFlash):

vllm serve /models/qwen35-122b-hybrid-int4fp8 \
  --attention-backend FLASHINFER \
  --speculative-config '{
    "method": "dflash",
    "model": "z-lab/Qwen3.5-122B-DFlash",
    "num_speculative_tokens": 16
  }'

The DFlash head for the Qwen3.5-122B model was created by ZLab and is available on Hugging Face. Setup requires following a specific GitHub recipe (link in video description).

Observed behavior: DFlash showed high throughput variance. When the acceptance rate was strong (high-structure output like code), peak throughput hit ~78 tok/s — significantly exceeding MTP-2. However, for general-purpose prompts, throughput fell to ~30–33 tok/s average, with peaks to ~50 tok/s.

The variance is explained by DFlash's acceptance semantics. Unlike autoregressive drafters where individual token rejections are isolated, DFlash accepts a prefix of the diffusion block. If the target model accepts tokens 1–3 but rejects token 4, tokens 5–16 are discarded entirely even if they would have been accepted. This means low acceptance rate at any position in the block wastes the entire remaining draft. When the task is highly structured (code, JSON, math), the acceptance rate is consistently high across positions, and DFlash dominates. For open-ended generation, variance is high.

Result: DFlash achieved the highest peak throughput (~78 tok/s) but also the highest variance. Average throughput for general prompts was lower than MTP-2. For structured-output tasks (coding agents, JSON generation), DFlash is compelling. It is also a relatively new method and actively improving.

Results Summary

Method	Model	Avg tok/s	Peak tok/s	Acceptance Rate	Notes
None (baseline)	Qwen3.5-122B	~36–37	~38	—	Reference
N-Gram	Qwen3.5-122B	~28–30	~31	Very low	Worse than baseline
MTP-1	Qwen3.5-122B	~44–45	~47	Mid-to-high 80s%	Solid improvement
MTP-2	Qwen3.5-122B	~48–49	~51	~80%+	Best consistency
MTP-4	Qwen3.5-122B	~34–35	~38	Drops significantly	Worse than MTP-2
None (baseline)	Qwen3-30B-A3B	~33	~40	—	Separate model
EAGLE3 (1 token)	Qwen3-30B-A3B	~38–39	~41	Good	~20% improvement
DFlash	Qwen3.5-122B	~48–50 avg*	~78	Variable	High peaks, high variance

*DFlash average varies significantly by task type (structured vs. open-ended).

Practical Guidance: Choosing the Right Method

Use MTP if your model ships with a native MTP head (Qwen3.5, DeepSeek V3, Gemma 4). It is the lowest-friction option — no separate model download, no architecture mismatch risk, and acceptance rates are consistently high because the head was jointly trained. num_speculative_tokens=2 is a strong default for most Qwen3.5 deployments.

Use EAGLE3 if no MTP head is available and a well-matched EAGLE3 drafter exists for your target model on Hugging Face. Keep num_speculative_tokens conservatively at 1–4; beyond that, acceptance rates drop sharply for most tasks.

Use DFlash for structured-output workloads: code generation, function calling, JSON mode, math. The parallel drafting architecture gives DFlash a fundamental advantage when the output has strong syntactic structure and acceptance rates across the block are consistently high. Avoid it for open-ended chat where variance makes latency unpredictable.

Avoid N-Gram for general inference. It is primarily useful as a pedagogical example or in specialized summarization/extraction pipelines where output closely mirrors input.

Key Takeaways

Speculative decoding is lossless. Output quality is mathematically identical to standard autoregressive decoding. The only variable is throughput.
Acceptance rate is everything. More speculative tokens are not always better. The acceptance rate at each position compounds — even a single bad position in an autoregressive chain invalidates all subsequent drafts.
DFlash's parallel drafting breaks the serial constraint. By generating blocks in a single forward pass, DFlash eliminates the per-token cost overhead that limits autoregressive drafters at high K. This is the architectural reason its peak throughput is so high.
Task type matters. Structured output (code, JSON, math) consistently benefits more from speculative decoding than open-ended generation, because the conditional distribution is sharper and easier for any drafter to model.
Hardware matters. These numbers are specific to the DGX Spark GB10 (SM121, 128 GB unified memory). Results on H100, A100, or consumer GPUs will differ due to memory bandwidth, compute, and kernel optimization differences.

If you want more detail, here is a YT video I did, going through the tests:

Resources

albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4 — Hybrid checkpoint + MTP patch recipe
DFlash paper: arXiv:2602.06036 — Block Diffusion for Flash Speculative Decoding
ZLab DFlash models on Hugging Face — DFlash heads for Qwen3 models
vLLM Speculative Decoding docs — Configuration reference
EAGLE3 architecture reference

The AI agent habit that was quietly wasting my time and tokens

Kristiyan Stoyanov — Tue, 23 Jun 2026 20:39:55 +0000

The AI Agent Habit That Was Quietly Wasting My Time and Tokens

Tags: ai, localai, machinelearning, productivity, agents

I realized I had been using AI agents in a way that looked smart but was actually pretty wasteful.

The pattern was simple: I would ask the agent for something useful, it would go off and figure it out, and eventually I would get an answer. The problem is that if you keep asking the agent to rediscover the same process over and over, you are paying for repeated reasoning, repeated tool usage, and repeated trial and error. That means more tokens, more latency, and more opportunities for the agent to fumble.

What finally clicked for me was this: use LLM inference for decisions, not for repetition.

If a task has already been figured out once, I do not want the model burning context and tool calls to solve it again every time. I want the model to recognize the task, use a reliable tool, and move on.

That is the pattern I have been using with Hermes, and it has made my local agent setup much more useful.

The Setup I Am Running

Right now I am running Hermes on a DGX Spark. In the video, I show the machine with 128 GB of unified memory, and at that moment I had about 1 GB free because I had a quantized Qwen 3.5 model loaded locally.

Hermes is my current agent framework of choice. I have tried other options, but Hermes has been easy to install and easy to live with. One thing I especially like is that it supports Telegram through a gateway, so I can talk to my agent from my phone instead of only from a terminal window.

On the tool side, the ones that matter most for this workflow are:

Web search and scraping
Terminal access
File operations
Code execution
Sub-agent delegation

For web search, I am using Tavily. In the video, I mention the free tier gives about 1,000 requests per month, which is enough for experimentation but still limited enough that I notice when an agent wastes calls.

That matters, because this whole post is really about reducing unnecessary tool usage.

The Wasteful Version

I started with a normal prompt:

What is the weather going to be like in Sofia this weekend? Help me plan some activities based on it.

That is exactly the kind of thing I would send to an agent from Telegram while I am on the move.

Hermes did eventually answer, but watching the trace was the important part. It did a web search, then a web extract, then checked time and date, then stumbled a bit, then searched again after not getting what it wanted the first time. In the video, I call out the real cost: this simple request filled about 20k tokens of context.

And that is the issue.

The answer was fine. The process was not.

If I ask for weather and activity suggestions regularly, I do not want the model improvising a mini research project every single time.

The Better Version: Research Once, Automate Once, Reuse Forever

Instead of asking the agent the end question again, I switched to building a capability.

First, I asked Hermes to research free weather APIs that did not need a key and were easy to automate:

Research free and open APIs that give you weather forecasts. Look for APIs that do not need an API key and can be easily automated with a Python script. Do not write the script yet. Let me choose the API first.

Hermes went off, searched around, and came back with several options, including Open-Meteo, WeatherAPI, and met.no. It recommended Open-Meteo, and that was good enough for me.

So I moved to the next step and told it to build something concrete:

Let's use Open-Meteo. I want you to spawn an open code sub-agent and create a directory. Inside of that directory the sub-agent must implement an Open-Meteo API client wrapped by a CLI. Use Python. Make sure it uses real data. Use mocks only for unit tests. Report back when ready.

That “use real data, mocks only for unit tests” line is one I use a lot. If the agent can run against reality, it can verify its own work much better.

Hermes delegated the task to a coding-focused sub-agent, created the project, and implemented the CLI.

Then came the part that matters most.

Never Trust the Agent

When the agent said the project was complete, I did not just accept it.

I tested it with a real request:

Now let's test it with the real API. I want you to use the script to give me the weather forecast for Berlin.

This is the rule I keep coming back to: never trust your agent.

Read the code. Run the script. Verify the output. Make sure it is using the real API. Make sure it is not doing anything unexpected. Only after that should it move from “experiment” to “capability.”

In the demo, the script returned a 7-day Berlin forecast in about 0.4 seconds. That is the moment where the whole pattern becomes obvious. The slow, token-heavy part was discovering how to do the task. Once that is solved, the best move is to package it.

Turning a One-Off Script into a Permanent Skill

Once the weather CLI worked, I asked Hermes to wrap it as a reusable skill:

Now let's create a skill for you that wraps around this CLI script and uses it whenever I ask you about the weather in future sessions.

Hermes created the skill using its skill management flow, and that became part of its permanent skill set.

Then I started a completely new session.

That is the real test, because a fresh session has fresh context. No hidden memory from the earlier chat. No cheating.

I asked the same question again:

What is the weather going to be like in Sofia this weekend? Help me plan some activities based on it.

This time Hermes checked its skills, found the weather skill, executed the script, and gave me a clean answer with activity suggestions.

The difference was huge.

The first time, it burned through web searches, including two Tavily searches, and spent a lot of tokens figuring out how to answer. The second time, it reduced the whole thing to essentially one tool call to the script I had already verified.

That is the pattern in one line:

Explore once. Automate once. Wrap it as a skill. Reuse forever.

This Gets More Interesting Than Weather

Weather is a toy example, but it is useful because the waste is easy to see.

The more interesting example from the video is one I built from my phone over Telegram. I use my agent pretty often for stock-related questions, so I had it create a stock analyzer script that fetches stock or index data from an open API.

Same pattern:

Create a stock analyzer Python script that fetches stock or index data from a popular and open API. Use real data. Use mocks only for unit tests.

When it finished, I verified it with a real run for Microsoft one year back. The output included about 250 trading days of data, the latest price, some moving averages, technical indicators, and a short interpretation.

Then I turned that into a skill too.

In a brand-new session, I asked a vague question about the USO ETF. I did not mention the script. I did not explain the workflow again. Hermes picked the stock analyzer skill on its own and returned a useful summary with current data.

That is where this starts to feel less like chatting with a model and more like growing a personal assistant over time.

The Security Rule I Want to Keep

The core safety idea here is simple: verify before you automate.

If an agent writes a script, read it. If it claims something works, test it. If it needs access to real systems, expose only the operations you actually want it to perform.

For anything sensitive, I would keep the agent on the narrowest possible rails. In practice, that means preferring read-only capabilities where possible, using small purpose-built tools instead of broad access, and only promoting a workflow into a permanent skill after I have seen it behave correctly.

The more capable the agent gets, the more important this becomes.

The Bigger Picture

What I like most about this pattern is that it compounds.

Every time I notice a repeated agent task, I have a choice:

Keep paying for the agent to rediscover the solution.
Or turn the solution into a reusable capability.

Over time, that changes the shape of the whole setup.

I stop treating the model like a universal improviser and start treating it like a coordinator that knows when to call reliable tools. The model still provides the intelligence, but the repetitive parts move into code.

That opens the door to more domain-specific assistants too. A natural next step is something like a private realtor assistant that checks listings, pulls mortgage news, summarizes changes, and sends a Telegram update on a schedule. Same principle, just applied to a workflow that actually matters to someone day to day.

That is the part I find exciting. Not AI magic, but a steadily improving assistant that gets more useful because I keep teaching it durable skills.

If you want to see the full walkthrough, including the Hermes session, the weather skill build, and the Telegram-based stock example, watch the YouTube video here:

If you are experimenting with agents locally, I would love to hear what repeated task you would turn into a skill first.