A draft model is not simply a smaller model. It has to speak the target model’s token language exactly, and it has to be cheap enough that being wrong sometimes still leaves you ahead. Both constraints are checkable before you download anything.
What the runtime checks at load
llama.cpp does not take your word for it. Before speculation starts it compares the two vocabularies, and the comparison in common/speculative.cpp is specific:
- Vocabulary type. The tokenizer families must be the same kind — a SentencePiece target and a BPE draft are rejected outright, with a message naming both types.
- BOS and EOS. Not just whether each model adds them, but the token ids themselves. The error reports all four values: whether the target adds BOS, whether the draft does, and the id each uses.
- Vocabulary size. A difference is tolerated, but a small one — the constant is
SPEC_VOCAB_MAX_SIZE_DIFFERENCE, set to 128 tokens. This exists because families ship variants padded with a handful of extra special tokens, not to let you pair unrelated models. - Token text. Starting from
SPEC_VOCAB_CHECK_START_TOKEN_ID, which is 5, it compares the literal string each id decodes to and complains as soon as one differs, reporting the id and both strings. The first few ids are skipped precisely because that is where the special tokens live.
Read that list as a definition rather than as a list of hurdles: a usable draft model is one whose id-to-string mapping is the same function as the target’s, near enough that only padding differs.
Why the vocabulary has to match
The reason is not implementation convenience. Verification works by running the target model over the drafted token ids and asking whether it would have chosen the same ids. If the two models number their tokens differently, that comparison is meaningless — id 4711 is one string in the draft and another in the target, so an “accepted” token would silently be a different piece of text.
Nor can you translate between them. Detokenising the draft and retokenising for the target does not round-trip: tokenizers are greedy over different merge tables, so a string one model emits as three tokens the other may segment as two, and the position alignment that verification depends on is gone. There is no cheap fix here, which is why the check is a hard failure and not a warning.
The exception is a draft head trained against a specific target rather than a separate model — EAGLE-3, MTP and the diffusion heads. Those come with the target’s vocabulary by construction; llama.cpp’s conversion instructions for them pass --target-model-dir so the converted file inherits the target’s tokenizer and token embeddings.
The size ratio, derived
Compatibility gets you a draft model that runs. Whether it helps is arithmetic. Write f for the cost of one draft forward pass as a fraction of one target forward pass, n for the number of tokens drafted per step, and E for the mean number of those accepted. A speculative step costs n·f + 1 and returns E + 1tokens; a plain step costs 1 and returns 1. Speed-up is (E + 1) / (n·f + 1).
Two consequences fall straight out. First, f has to be small: at f = 0.5 — a draft half the target’s cost — drafting 4 tokens costs 3 target passes, so you need better than 2 accepted tokens per step on average before you break even, and that is a lot to ask. At f = 0.05, drafting 4 costs 1.2 and almost any acceptance is profit. The usual guidance of a tenth or less is this inequality, not folklore.
Second, f is not the parameter ratio. Both models are memory bound at batch size one, so the honest first approximation is the ratio of the two files’ resident sizes at the quantisation you will actually run. Derive it from the parameter counts rather than looking it up: a K-quant at four bits and a bit of overhead lands near 4.8 bits per weight, so a 0.5B is about 0.5e9 × 4.8 / 8 = 0.3 GB and an 8B about 8e9 × 4.8 / 8 = 4.8 GB, giving f ≈ 0.06 — and the ratio is what matters here, so the estimate being a few per cent out does not change the decision. Read the true sizes off the two files before committing, and remember this ignores the draft’s smaller KV cache and, if it is placed on another device, a different memory bandwidth entirely. Quantising the draft harder than the target, with llama-quantize, is usually a good trade: a draft that is wrong slightly more often but costs half as much moves n·f more than it moves E.
Same family is not the same as same tokenizer
Families rework their tokenizers between generations. A 1B from one generation and a 70B from the next often share a name and not a vocabulary, and the load-time check will tell you so in a message about vocabulary size or token text rather than about versions. Check the tokenizer, not the branding.
Instruction tuning matters in the other direction. Acceptance depends on the draft agreeing with the target token by token, so a base-model draft paired with an instruction-tuned target will diverge on exactly the formatting and refusal patterns the tuning installed — the vocabulary check passes and the acceptance rate quietly disappoints. Prefer the same tuning lineage where a choice exists, and where it does not, measure acceptance before deciding.
When no draft model fits
Plenty of models have no small sibling. llama.cpp’s --spec-type also offers strategies that need no second model at all: the n-gram family drafts by looking for repeats of the current suffix in what has already been generated or in the prompt. That is worthless on novel prose and very effective on the workloads where a long input is being quoted, edited or reformatted, because the answer is largely already present in the context. It costs no extra memory, which makes it the first thing to try when the alternative is loading a second set of weights you cannot afford.
If neither fits, the remaining levers are the ordinary ones — a smaller quant, more layers on the GPU via the -ngl flag, or batching more requests together so the hardware is doing arithmetic rather than waiting on memory. Speculation is one way to buy back idle arithmetic, not the only one.
Top comments (0)