The six-token run is a useful counterexample to optimizing acceptance length alone: the mean length increases while overall generation throughput falls. I would expose the convention behind mean len, especially because the four-token configuration reports 4.12 in turn three.
If that metric includes the target's own next token, it is not the number of accepted draft tokens and should not be used directly to choose the next draft maximum. Recording committed tokens, drafting time and verification time per round would make the proposed tuning rule easier to validate. The best draft length should maximize committed tokens per wall-clock time for that workload.
Engineers working on GPU cloud infrastructure in Japan. We write hands-on notes on LLM fine-tuning, quantization, and inference benchmarks, plus real infra tradeoffs of running AI workloads at scale.
Hi Ahmet, thanks for the close read — this is a fair catch.
You’re right about the convention: mean length here is defined as 1 (the target model’s own guaranteed token) plus the number of accepted draft tokens per verification round. So for a draft-n-max of 4, the practical ceiling is 5, not 4 — the “past the configured maximum of 4” phrasing in the post was imprecise and we’re correcting it to just report the 4.12 figure against that actual ceiling. We’ll update the article text itself to reflect this.
That also means you’re right that it shouldn’t be used on its own to pick the next draft-max — it conflates the bonus token with genuinely accepted draft tokens, so a config can look like it’s “improving” while real draft-token yield is flat or falling, which is exactly what the six-token case shows.
Good suggestion on logging committed tokens, drafting time, and verification time per round separately — that’s the right level of granularity to actually validate a tuning rule, and committed-tokens-per-wall-clock is a better objective than mean length alone. We’ll fold that into how we evaluate draft-length choices going forward. Appreciate you flagging it.
That is a fair correction to my reading: with one guaranteed target token plus accepted draft tokens, the ceiling is five, so 4.12 is not beyond it. Separating those terms makes the four-versus-six comparison much clearer.
For the timing breakdown, I would also retain rejected draft work per round. It explains how a longer draft can increase committed length yet lose wall-clock throughput. Comparing the same prompt set and output-length distribution would keep that extra work visible without treating mean acceptance length as the tuning objective.
Engineers working on GPU cloud infrastructure in Japan. We write hands-on notes on LLM fine-tuning, quantization, and inference benchmarks, plus real infra tradeoffs of running AI workloads at scale.
Thanks — good suggestion. Tracking rejected draft work per round, and holding the prompt set / output-length distribution constant across configs, would make the wasted-compute cost of a longer draft length explicit rather than something you have to infer from the throughput delta. We’ll look at adding that level of logging to how we evaluate draft-length choices going forward.
Per-round logging would also let you separate where the rejected work happens within a generation. A longer draft might help the repetitive tail while wasting more computation near a difficult branch, and the same mean acceptance length can hide that difference.
I would summarize accepted and rejected tokens by position in the output, then compare total generation time on the matched requests. That gives the draft-length decision a workload-specific explanation rather than assuming that a better average acceptance figure necessarily produces a faster response.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The six-token run is a useful counterexample to optimizing acceptance length alone: the mean length increases while overall generation throughput falls. I would expose the convention behind mean len, especially because the four-token configuration reports 4.12 in turn three.
If that metric includes the target's own next token, it is not the number of accepted draft tokens and should not be used directly to choose the next draft maximum. Recording committed tokens, drafting time and verification time per round would make the proposed tuning rule easier to validate. The best draft length should maximize committed tokens per wall-clock time for that workload.
Hi Ahmet, thanks for the close read — this is a fair catch.
You’re right about the convention: mean length here is defined as 1 (the target model’s own guaranteed token) plus the number of accepted draft tokens per verification round. So for a draft-n-max of 4, the practical ceiling is 5, not 4 — the “past the configured maximum of 4” phrasing in the post was imprecise and we’re correcting it to just report the 4.12 figure against that actual ceiling. We’ll update the article text itself to reflect this.
That also means you’re right that it shouldn’t be used on its own to pick the next draft-max — it conflates the bonus token with genuinely accepted draft tokens, so a config can look like it’s “improving” while real draft-token yield is flat or falling, which is exactly what the six-token case shows.
Good suggestion on logging committed tokens, drafting time, and verification time per round separately — that’s the right level of granularity to actually validate a tuning rule, and committed-tokens-per-wall-clock is a better objective than mean length alone. We’ll fold that into how we evaluate draft-length choices going forward. Appreciate you flagging it.
That is a fair correction to my reading: with one guaranteed target token plus accepted draft tokens, the ceiling is five, so 4.12 is not beyond it. Separating those terms makes the four-versus-six comparison much clearer.
For the timing breakdown, I would also retain rejected draft work per round. It explains how a longer draft can increase committed length yet lose wall-clock throughput. Comparing the same prompt set and output-length distribution would keep that extra work visible without treating mean acceptance length as the tuning objective.
Thanks — good suggestion. Tracking rejected draft work per round, and holding the prompt set / output-length distribution constant across configs, would make the wasted-compute cost of a longer draft length explicit rather than something you have to infer from the throughput delta. We’ll look at adding that level of logging to how we evaluate draft-length choices going forward.
Per-round logging would also let you separate where the rejected work happens within a generation. A longer draft might help the repetitive tail while wasting more computation near a difficult branch, and the same mean acceptance length can hide that difference.
I would summarize accepted and rejected tokens by position in the output, then compare total generation time on the matched requests. That gives the draft-length decision a workload-specific explanation rather than assuming that a better average acceptance figure necessarily produces a faster response.