DEV Community: Jessie Jia

Speculative Decoding and MTP: Why Guessing Is Free

Jessie Jia — Thu, 20 Aug 2026 15:32:51 +0000

I saw "MTP round-trip" on a checklist for a Megatron conversion pipeline and had no idea what it meant. Two acronyms, one hyphen, apparently important enough that someone had listed it as a thing to verify.

Working out what it meant took me somewhere I didn't expect. The interesting part turned out not to be MTP at all — it was the reason speculative decoding works in the first place, which rests on a fact about hardware that I had backwards.

TL;DR

Generating text is slow because it's sequential: one full forward pass per token.
But a forward pass over five tokens costs about the same as over one. Generation is bottlenecked by moving weights, not by arithmetic.
Speculative decoding exploits that: something cheap drafts k tokens, the big model verifies all of them in one pass.
It is exact, not an approximation. Same output distribution as normal decoding.
MTP (Multi-Token Prediction) is one way to produce those drafts — a small module trained into the model itself.
MTP has two separate lives: a training-time auxiliary loss you can throw away, and an inference-time draft head you can't.

Why generating text is slow

To produce token N+1, the model needs token N. There's no way around that ordering — it's what "language model" means.

So generating 100 tokens means 100 full passes through the network. For a model like GLM-5.2, that's 78 layers, 100 times over.

The obvious conclusion is that generation is 100 times as expensive as reading the prompt. The obvious conclusion is wrong, and the way it's wrong is the whole point.

The part that got me

A forward pass processing one token and a forward pass processing five tokens take roughly the same wall-clock time.

I had assumed compute scaled with tokens. It doesn't, because compute isn't the bottleneck.

Every forward pass has to read the model's weights out of memory and into the compute units. That's hundreds of gigabytes moving across a memory bus, and it happens whether you're processing one token or fifty. The actual arithmetic on a handful of tokens is rounding error next to the cost of fetching the weights to do it with.

So the accounting looks like this:

	weight reads
5 tokens, one at a time	5
5 tokens, in a single pass	1

Same tokens, same math, five times the memory traffic — purely because of the ordering.

That ratio is the prize. If you could somehow process five tokens at once, you'd get them roughly five times faster. You can't, because token 3 depends on token 2.

Unless you guess.

Speculative decoding: draft, then verify

The trick is to split generation into two roles.

Something cheap drafts k tokens ahead. These are guesses.
The big model does one forward pass over all k positions at once, checking each guess.
Accept the longest correct prefix. Throw away the rest.
Repeat.

Step 2 is the whole thing, and it hinges on a property that's easy to miss: verification is parallel even though generation isn't.

The sequential dependency exists because you don't know token 3 until you've produced token 2. But in verification you already have candidate tokens 1 through k — the drafter handed them over. So you can lay them all out and check them in a single pass. There's nothing left to wait for.

Guess three tokens correctly and you've produced four tokens for the price of one weight read. Guess wrong on the first one and you fall back to producing one token, which is what you'd have done anyway.

It's exact, not approximate

This is the part I assumed had a catch, and it doesn't.

Speculative decoding produces the same output distribution as ordinary decoding. It is not a quality-for-speed trade. Every token in the final output is one the big model itself endorsed — the drafter's guesses are only ever suggestions, and any suggestion the big model wouldn't have made is rejected.

The acceptance test is designed so that the surviving tokens are distributed exactly as if the big model had generated them alone. So there's no accuracy knob to tune and no quality regression to monitor. Either it's faster or it isn't.

The only cost of a bad drafter is wasted drafting work. Which means the metric that matters is acceptance rate: what fraction of guesses survive.

Where the drafts come from

Several options, and this is where MTP finally enters.

Draft source	How it works	Cost
A separate small model	A 1B model drafts for a 70B one	Train, ship and serve a second model; tokenizers must match
N-gram / lookup	Copy candidates straight from the context	Free, but only helps when output repeats input — code editing, summarisation
MTP head	A small module trained into the main model	One extra layer

The separate-model approach came first and has an awkward property: the two models were trained independently, so they don't necessarily think alike. When the drafter's instincts diverge from the big model's, acceptance rate falls, and a drafter whose guesses get rejected is pure overhead.

Which motivates building the drafter into the model.

What MTP actually is

Multi-Token Prediction. GLM-5.2's config says:

"num_hidden_layers": 78,
"num_nextn_predict_layers": 1

78 layers of main model, plus one extra module whose job is to predict the token after next. That module is MTP.

Two properties make it a good drafter:

It agrees with the main model, because it was trained alongside it and sits on top of the same internal representations. Its guesses are the guesses the main model would plausibly make, which is exactly what acceptance rate rewards. DeepSeek-V3, which uses the same design, reports 85–90% acceptance on the next token and around 1.8× end-to-end throughput.

It's cheap, because it reuses the main model's already-computed hidden states. Drafting costs one extra layer, not a second model's worth of forward pass.

And there's no second checkpoint to version, deploy or keep in sync. The drafter ships inside the model.

MTP has two separate lives

This is the distinction I'd have got wrong if I hadn't looked.

At training time, MTP is an auxiliary loss. Forcing the model to predict two tokens ahead rather than one pushes it toward representations that carry more forward-looking information, which improves the main model. Used this way, MTP is disposable — you can drop the module when training finishes and keep the benefit.

At inference time, MTP is the draft head. Used this way, you have to keep it, or you lose the speedup.

Same weights, two unrelated reasons to care about them. Whether you need MTP to survive your pipeline depends entirely on which of these you're after.

Back to the checklist

Which finally explains why "MTP round-trip" is its own line item.

A training pipeline converts the model between formats: HuggingFace to Megatron to train, Megatron back to HuggingFace to serve. The round-trip is that loop, and the question is whether the MTP module comes out the other side intact.

The reason it needs its own check is that losing it is silent. MTP isn't on the main forward path — it doesn't affect what the model says, only how fast it says it. So:

Conversion completes without error
Forward-parity tests pass
Training runs, loss curve looks normal
Export succeeds
Serving works, at half the throughput, with nothing in the logs

I looked at the verification script in the pipeline that prompted all this. Step 5 checks the HuggingFace to Megatron weight mapping by running both implementations on the same input and comparing outputs — cosine 0.9936, a clean pass. It also contains this line:

if hasattr(hc, "num_nextn_predict_layers"): hc.num_nextn_predict_layers = 0

MTP is switched off for the test. Which is the right call for what that test is for — it isolates the main trunk so a mismatch points somewhere specific. But it means the verified mapping says nothing about MTP, and a forward-parity check couldn't have caught the omission anyway, because MTP doesn't touch the forward path being compared.

The verification you'd actually want is different in kind: not "does the model still behave correctly" but "is every weight still here". Enumerate the tensors before and after, compare names and shapes, and require the numbers to match exactly — a format conversion does no arithmetic, so there's no floating-point slack to allow for. Closer to an inventory than a test.

What I took from it

I went in thinking MTP was the topic and speculative decoding was context. It's the reverse: speculative decoding is the idea, and MTP is one implementation of one of its parts.

The thing worth carrying, though, is the hardware fact underneath. The cost of a forward pass barely depends on how many tokens are in it. Once that lands, speculative decoding stops looking like a clever trick and starts looking obvious — you have four free seats in the car, so you may as well guess who's coming.

It also rhymes with something I ran into autoscaling vLLM replicas: the intuitive cost model was wrong in a way that only shows up when you check what the hardware is actually doing.

中文版：投机解码与 MTP：为什么"猜"是免费的

What Actually Gets Tokenized in SFT

Jessie Jia — Thu, 13 Aug 2026 07:17:00 +0000

tok.apply_chat_template(msgs, add_generation_prompt=True)

I have copied that line more times than I can count without thinking about what any of it does. The official example writes it that way, the output looks right, you move on.

Then you have to build your own SFT data and compute your own loss mask, and it turns out every piece of that line is load-bearing. I spent today pulling it apart. Notes below.

The counterintuitive part first: the model has no idea messages exists.

TL;DR

The model only ever sees the rendered string. messages is for Python. Roles are real tokens in the vocabulary.
A chat template is a Jinja2 program, stored in tokenizer_config.json or as a standalone chat_template.jinja. When both exist the file wins.
Training renders the full conversation with add_generation_prompt=False. Inference renders msgs[:-1] with add_generation_prompt=True.
The prefixes those two produce must be identical down to the byte. One assert catches it.
Don't let the tokenizer add special tokens after templating, or you get two BOS.

The model has no idea `messages` exists

I had some vague picture in my head where the role was structured metadata and something inside the model read it. Nope.

[{"role": "user", "content": "What is the Young's modulus of graphene?"}]

That list lives in your Python process and nowhere else. What reaches the model is:

[gMASK]<sop><|user|>
What is the Young's modulus of graphene?<|assistant|>

and after tokenization:

[151331, 151333, 151336, 198, ...content ids..., 151337, 198]

Those leading numbers are GLM-4's ids for [gMASK], <sop>, <|user|> and <|assistant|>. Every model family has its own, so don't hardcode them across models.

The role is a token, not a field. The model knows it is its turn to talk because it sees id 151337, the same way it learns any other pattern. No JSON at inference time, no key lookup, no schema. One sequence.

Nearly everything below follows from that.

Jinja2, the rendering layer

Jinja2 is a text templating engine originally built to render HTML for Flask. Three constructs, that's it:

Syntax	Purpose
`{{ expr }}`	print the value of an expression
`{% ... %}`	control flow: `for`, `if`, `set`
`{# ... #}`	comment, produces no output

HuggingFace borrowed it for one job: flatten structured messages into the string the model was trained on, and flatten it the same way every time. The minimal shape:

{%- for message in messages %}
    {{- '<|' + message['role'] + '|>\n' + message['content'] + '<|endoftext|>\n' }}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|assistant|>\n' }}
{%- endif %}

The dashes aren't style

{%- and -%} strip whitespace around the tag. Real templates are covered in them.

I assumed this was a formatting preference. It isn't. Templates get indented so humans can read them, and without stripping, every newline and indent in the source lands verbatim in the prompt. One extra \n and the token sequence no longer matches what training saw.

If you write your own, print the result with repr() rather than trusting your eyes.

What's in scope inside a template

transformers renders in a jinja2.sandbox.ImmutableSandboxedEnvironment, so arbitrary attribute access and side effects are blocked. You can't call random Python from in there. What you get:

messages, plus tools when tool definitions are passed
add_generation_prompt
bos_token, eos_token and friends, from the tokenizer
raise_exception(), for templates that reject malformed conversations, like a system message in the wrong slot
strftime_now(), for templates that inject the current date

Where the template lives

Two places, same string.

The old way is inline in tokenizer_config.json. That's the file AutoTokenizer.from_pretrained() reads, and note it does not hold the vocabulary. That's in tokenizer.json, or vocab.json plus merges.txt, or a SentencePiece *.model. It only records how to construct the tokenizer object:

{
  "tokenizer_class": "PreTrainedTokenizerFast",
  "model_max_length": 131072,
  "padding_side": "left",
  "bos_token": "[gMASK]",
  "eos_token": "<|endoftext|>",
  "pad_token": "<|endoftext|>",
  "clean_up_tokenization_spaces": false,
  "added_tokens_decoder": {
    "151329": { "content": "<|endoftext|>", "special": true, "normalized": false },
    "151336": { "content": "<|user|>",      "special": true, "normalized": false }
  },
  "chat_template": "{%- for message in messages %}..."   // ← inline, one giant line
}

For SFT work, a few fields are worth a look.

added_tokens_decoder decides whether <|user|> is one atomic token or gets shredded into BPE pieces. Shredded, the role marker stops being a clean signal and the model has to infer the boundary from "< then | then user". It can learn that. No reason to spend the capacity.

padding_side is right for training and must be left for batched generation. In a right-padded generation batch every sequence shorter than the longest one comes out garbage.

eos_token and pad_token govern both where generation stops and which positions get carved out of the loss.

The new way is a standalone chat_template.jinja, which newer transformers versions write by default from save_pretrained(). The reason is mundane: inside JSON every newline is \n and every quote is escaped, so a 200-line template becomes one unreadable line with no diff and no highlighting.

The file wins on precedence. Plenty of repos ship both for backward compatibility, which leaves a trap. Once the two drift apart, behavior depends on your transformers version, and that is a miserable thing to track down. Check they agree before you debug anything else.

add_generation_prompt: training vs inference

This is where I actually got stuck today. One SFT sample:

msgs = [
    {"role": "user",      "content": "What is the Young's modulus of graphene?"},
    {"role": "assistant", "content": "About 1 TPa."},
]

At training time you have the answer. That's the supervision signal, so you render the whole thing with no generation prompt:

tok.apply_chat_template(msgs, add_generation_prompt=False)

<|user|>
What is the Young's modulus of graphene?<|assistant|>
About 1 TPa.<|endoftext|>

Setting it to True would append a second, empty <|assistant|>\n after the answer. A dangling role marker, which the model would learn to emit.

At inference you don't have the answer. Producing it is the point, so you pass msgs[:-1]:

tok.apply_chat_template(msgs[:-1], add_generation_prompt=True)

<|user|>
What is the Young's modulus of graphene?<|assistant|>

add_generation_prompt=True is what appends that trailing <|assistant|>\n. Without it the last token the model sees is ?, and it will cheerfully continue the user's turn, inventing a follow-up question instead of answering. With it, the model is standing exactly where, during training, the next token was the start of an answer.

The prefixes have to match exactly

Line the two up:

training:   <|user|>\nWhat is ... graphene?<|assistant|>\n | About 1 TPa.<|endoftext|>
inference:  <|user|>\nWhat is ... graphene?<|assistant|>\n | ← generation starts here
            └──────────── must match exactly ───────────┘

Everything left of the bar is the model's conditioning context. Training taught it "given this prefix, emit About". If inference rebuilds that prefix with one extra newline, or a space the training path didn't have, the model is conditioning on something outside its training distribution.

What makes it nasty is that it degrades invisibly. Print both strings and they look identical. Nothing in the logs.

So just assert:

full   = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=False)
prefix = tok.apply_chat_template(msgs[:-1], tokenize=True, add_generation_prompt=True)
assert full[:len(prefix)] == prefix

Three lines, and it hands you the loss-mask boundary for free: len(prefix) is where the answer starts.

Cutting the loss mask

Roles are tokens, not fields, so there are no field boundaries to slice on. You locate the assistant span yourself.

The direct approach is the length difference:

labels = [-100] * len(prefix) + full[len(prefix):]

The boundary lands naturally after <|assistant|>\n. Clean.

The cleverer-looking alternative is to regex the rendered text for <|assistant|> and map character offsets back to token indices. Don't. The moment a message's content legitimately contains that string it's wrong, and wrong silently.

The length-diff trick has one real limit: it only handles the last turn. For multi-turn data where you want loss on every assistant reply, you'd render incrementally turn by turn, which gets ugly fast.

The proper fix is to mark the assistant span in the template itself:

{%- if message['role'] == 'assistant' %}
    {% generation %}{{- message['content'] + '<|endoftext|>' }}{% endgeneration %}
{%- endif %}

Then ask for the mask:

out = tok.apply_chat_template(
    msgs, tokenize=True, return_dict=True,
    return_assistant_tokens_mask=True,
)
mask = out["assistant_masks"]   # 1 on assistant tokens

Correct for any number of turns. The cost is that the template has to contain {% generation %} blocks, and a lot of published templates don't, which is why I haven't switched to it yet. Check first, or add the blocks to your own copy.

Things that bite

Double BOS. The template already emitted BOS. Call tok(text) after it and the default add_special_tokens=True adds another:

text = tok.apply_chat_template(msgs, tokenize=False)
ids  = tok(text, add_special_tokens=False)["input_ids"]   # ← required

Or skip the round trip with tokenize=True, which uses add_special_tokens=False internally anyway.

Hand-built strings. Writing f"<|user|>\n{q}<|assistant|>\n" in your data pipeline works, right up until the official template changes, or you swap base models, or someone adds a system prompt. Let the template be the single source of truth and stop thinking about it.

Inconsistent template branches. Some hand-edited templates emit <|assistant|>\n inside the loop and <|assistant|> in the add_generation_prompt branch, one newline short. The official GLM and Qwen templates are fine, but if you've touched a template, or added tool definitions or a system prompt, that assert is the only thing that will tell you.

Special tokens in user content. Jinja does plain string concatenation with no escaping. A content field containing the literal text <|assistant|> tokenizes into the real special token, and the model reads a genuine turn boundary. Prompt injection, training-data edition. If your corpus is scraped or model-generated, scan it first.

Wrapping up

It comes down to one thing: the byte string you train on and the byte string you infer on have to share an identical prefix. Jinja2 produces that string, tokenizer_config.json and chat_template.jinja hold the recipe, and add_generation_prompt is the only difference the two paths should have.

Honestly, none of this comes up most of the time. Use a stock model on a standard pipeline and apply_chat_template handles everything; you never need to know what's underneath. It only surfaces when you start building your own data, computing your own masks, or swapping base models. And when it does surface it doesn't throw. It just makes things slightly worse, which is exactly why it's worth knowing in advance.

There's an isomorphic problem on the serving side. The QPS-as-load-signal mistake in Little's Law and vLLM autoscaling has the same shape: a metric that looks reasonable and is quietly measuring something else.

Further reading: the chat templating guide in the transformers docs, and the whitespace-control section of the Jinja2 template designer reference.

中文版：Chat Template 到底 tokenize 了什么

DEV Community: Jessie Jia

Speculative Decoding and MTP: Why Guessing Is Free

TL;DR

Why generating text is slow

The part that got me

Speculative decoding: draft, then verify

It's exact, not approximate

Where the drafts come from

What MTP actually is

MTP has two separate lives

Back to the checklist

What I took from it

What Actually Gets Tokenized in SFT

TL;DR

The model has no idea messages exists

Jinja2, the rendering layer

The dashes aren't style

What's in scope inside a template

Where the template lives

add_generation_prompt: training vs inference

The prefixes have to match exactly

Cutting the loss mask

Things that bite

Wrapping up

The model has no idea `messages` exists