Multi-Token Prediction: DeepSeek's Built-In Draft Model

Speculative decoding needs two models: a small draft model that guesses ahead and a big verifier that checks. Multi-token prediction (MTP) collapses that into one. DeepSeek-V3 trains extra prediction heads directly into the main model, so it drafts its own future tokens — no separate draft model to train, align, or keep in memory. You get a denser training signal and a roughly 1.8x decoding speedup from the same weights.

If you've been treating "next-token prediction" as the only game in town, MTP is the part of modern LLM training that quietly changed.

TL;DR

Multi-token prediction (MTP) adds auxiliary heads that predict tokens t+2, t+3, … alongside the standard t+1 head, giving each training position more supervision.
DeepSeek-V3 uses sequential, causal MTP modules (each conditioned on the previous depth), not the independent parallel heads from Meta's earlier MTP paper.
The MTP heads are thrown away or reused: discard them and the base model runs normally, or keep them as a built-in draft model for self-speculative decoding.
DeepSeek reports a second-token acceptance rate around 85–90% and roughly 1.8x faster decoding, with no accuracy loss because every draft is verified.
MTP helps even without the speedup: predicting further ahead forces the model to plan, which improves benchmark scores.

What is multi-token prediction and how does it differ from next-token prediction?

Standard language models are trained with a single objective: given tokens t_1…t_i, predict t_{i+1}. One position, one label, one gradient signal per token. Multi-token prediction keeps that head and bolts on additional heads that predict t_{i+2}, t_{i+3}, and so on from the same hidden state.

The immediate payoff is training efficiency. In vanilla next-token prediction, position i only ever learns to produce one token. With MTP at depth D, that same position contributes D cross-entropy terms, so the representation at i has to encode information about several upcoming tokens, not just the next one. That pushes the model toward planning instead of myopic greedy continuation — which matters most for structured output like code, where the token three steps ahead constrains the one you emit now.

Meta's 2024 paper (Gloeckle et al., "Better & Faster Large Language Models via Multi-token Prediction") used n independent parallel heads sharing a trunk. DeepSeek-V3 took a different, more careful route.

How does DeepSeek-V3 implement MTP?

DeepSeek-V3 uses sequential MTP modules that preserve the full causal chain, rather than predicting all future tokens in parallel from one shared hidden state. Each module predicts exactly one additional token and is conditioned on the representation from the previous depth.

Concretely, for MTP depth k, module k combines the previous depth's hidden state with the embedding of the token it's now conditioning on:

# Sketch of one DeepSeek-V3 MTP module at depth k (per position i)
# Shared with the main model: Emb (embedding) and OutHead (unembedding)
# Per-depth and trainable: proj_k (R^{d x 2d}) and block_k (one transformer block)

def mtp_module_k(h_prev, next_tok_id, proj_k, block_k):
    # h_prev: hidden state from depth k-1 (h^0 = main model's final hidden state)
    # next_tok_id: the token t_{i+k} we now know and condition on
    a = rms_norm(h_prev)                 # normalize previous-depth state
    b = rms_norm(Emb(next_tok_id))       # normalize the newly-known token's embedding
    h = proj_k(concat([a, b], dim=-1))   # project 2d -> d
    h = block_k(h)                       # one transformer block (attention + MLP)
    logits = OutHead(h)                  # predicts t_{i+k+1}, using the SHARED head
    return h, logits

Two design choices carry the weight:

The embedding matrix and the output head are shared across the main model and every MTP module. Only the projection proj_k and one transformer block block_k are added per depth, so the parameter overhead is small.
Causality is kept intact. Module 1 sees the real token t_{i+1}, module 2 sees t_{i+2}, and so on. Each depth is conditioned on ground truth during training, which is what makes the heads good enough to draft from at inference time.

DeepSeek-V3 ships with D = 1 — a single MTP module predicting one extra token. That's a deliberately conservative depth; the marginal gain from deeper heads shrinks fast while training cost grows.

What does the MTP loss look like?

The MTP loss is just the mean cross-entropy across depths, added to the main objective with a small weight. For each depth k:

L_MTP_k = CrossEntropy( p_k(t_{i+k+1} | context) )   averaged over valid positions i

L_MTP   = (lambda / D) * sum_{k=1..D} L_MTP_k

L_total = L_main + L_MTP

The weight lambda is what keeps MTP an auxiliary objective rather than something that distorts the primary next-token distribution. DeepSeek-V3 used a larger lambda early in training and annealed it down over the run — hard early planning pressure, then back off so the main head dominates by the end. If you set lambda too high, the shared representation over-optimizes for the harder multi-step targets and single-token quality suffers; too low and the heads never get good enough to draft from.

How does MTP turn into a free draft model at inference?

Here's the part that makes MTP more than a training trick. Because the MTP module already predicts t_{i+2} from information available at step i, you can use it as a self-speculative draft — the model drafts ahead of itself, then verifies with its own main head.

The loop, per step:

Main head predicts t_{i+1}.
MTP module 1 predicts a candidate for t_{i+2} in the same forward pass.
Run one verification forward pass that scores the drafted token against the main head's distribution.
Accept the draft if it matches (greedy) or passes the speculative-sampling acceptance test; otherwise reject and fall back to the main head's token.

Every accepted draft means two tokens emerged from what is effectively one decode step. Because rejected drafts are discarded, output is identical in distribution to plain decoding — you never trade accuracy for speed, only wall-clock time. This is the same correctness guarantee as classic speculative decoding, minus the second model.

DeepSeek reports the second-token acceptance rate lands around 85–90% across generation domains, translating to roughly a 1.8x improvement in tokens per second. The acceptance rate is what to watch: it depends on how predictable your text is. Boilerplate code and repetitive structured output accept at very high rates; dense reasoning with high branching accepts less often.

Why not just use a separate draft model?

Because a separate draft model is a second system to train, distill, and keep aligned with the target. MTP folds drafting into the model you already have:

	Classic speculative decoding	MTP self-speculation
Draft source	Separate small model	Built-in MTP head
Extra memory at inference	Full draft model weights	One transformer block per depth
Distribution mismatch	Draft and target can diverge	Same trunk, tightly coupled
Training cost	Train/distill a draft model	Auxiliary loss during main run
Benchmark quality	Neutral	Improves from denser signal

The distribution-mismatch column is the subtle win. A distilled draft model can drift from the target and tank your acceptance rate; the MTP head shares the same trunk representation and output head, so its guesses stay close to what the main model would do. You spend a little training compute and a small amount of inference memory, and drafting quality comes along for free.

When does MTP not help?

MTP is not a universal speedup. Three failure modes to plan around:

Low-entropy vs high-entropy content. Acceptance rate collapses on genuinely unpredictable text (novel reasoning, adversarial prompts). At acceptance rates below ~40%, the verification overhead can eat the gains, and you're paying for drafts you throw away.
Batch pressure. Self-speculation shines at low batch sizes where you're latency-bound and the GPU is underutilized. At high batch sizes the machine is already compute-saturated, and the extra verification work competes with real throughput. Serving stacks gate MTP on batch size for exactly this reason.
Depth beyond 1. Deeper MTP (predicting t+3, t+4) sounds like more free tokens, but each extra depth is conditioned on a drafted token, so acceptance compounds downward. DeepSeek's choice of depth 1 reflects that the second drafted token rarely pays for itself.

If you're deploying, instrument the acceptance rate per request class before assuming MTP is a win. It's a distribution-dependent optimization, not a constant.

Direct answer: what is multi-token prediction and why does it matter?

Multi-token prediction is a training objective where an LLM predicts several future tokens at each position — not just the next one — using lightweight auxiliary heads that share the model's embedding and output layers. DeepSeek-V3's version chains those heads causally so each predicts one more token conditioned on the last, adding a small auxiliary cross-entropy loss during pretraining. The result is two-for-one: the denser supervision improves the model's planning and benchmark scores, and the same heads double as a built-in draft model for self-speculative decoding, delivering roughly 1.8x faster generation at an 85–90% acceptance rate with no change to the output distribution. Unlike classic speculative decoding, MTP needs no separate draft model — the model drafts and verifies itself, which is why it's showing up as a default in frontier training recipes rather than a niche inference hack.