<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Maddy Osman</title>
    <description>The latest articles on DEV Community by Maddy Osman (@mosman).</description>
    <link>https://dev.to/mosman</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3283402%2F4c7b7691-628c-4788-a28c-9f4313111f4f.png</url>
      <title>DEV Community: Maddy Osman</title>
      <link>https://dev.to/mosman</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mosman"/>
    <language>en</language>
    <item>
      <title>How do you tune speculative decoding in a production serving stack?</title>
      <dc:creator>Maddy Osman</dc:creator>
      <pubDate>Fri, 20 Nov 2026 17:52:33 +0000</pubDate>
      <link>https://dev.to/digitalocean/how-do-you-tune-speculative-decoding-in-a-production-serving-stack-6li</link>
      <guid>https://dev.to/digitalocean/how-do-you-tune-speculative-decoding-in-a-production-serving-stack-6li</guid>
      <description>&lt;p&gt;Tune speculative decoding against your own traffic, not a benchmark. Choose a method that fits your workload, set draft length from measured acceptance rate, and re-test at production concurrency, because the speedup shrinks as batches grow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Key takeaways&lt;/strong&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Watch acceptance rate—draft length is the main knob that moves it.
&lt;/li&gt;
&lt;li&gt;Concurrency decides whether speculation helps or hurts, so sweep load with speculation on and off.
&lt;/li&gt;
&lt;li&gt;Judge results on per-token latency and throughput, not just time to first token.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Pick the method first
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://docs.vllm.ai/en/latest/features/speculative_decoding/" rel="noopener noreferrer"&gt;vLLM speculative decoding docs&lt;/a&gt; say model-based methods such as EAGLE, MTP, and draft models give the best latency reduction, while n-gram and suffix decoding give modest speedups without adding workload at peak traffic. If your target model ships native MTP support, start there. Otherwise, EAGLE is a strong general-purpose default.&lt;/p&gt;

&lt;h2&gt;
  
  
  Set draft length from acceptance rate
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;num_speculative_tokens&lt;/code&gt; controls how many tokens the drafter proposes per step. Longer drafts promise more speedup, but later positions get rejected more often. The &lt;a href="https://vllm.ai/blog/2026-08-14-dspark-adaptive-verification" rel="noopener noreferrer"&gt;vLLM team reports&lt;/a&gt; that one model had the last token of a 7-token block survive less than 10% of the time, versus over 70% for the first token.&lt;/p&gt;

&lt;p&gt;Sweep a few values and read per-request acceptance metrics rather than guessing. Stop raising the number once extra tokens are mostly rejected.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"method"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"eagle3"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"model"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&amp;lt;your-draft-head&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"num_speculative_tokens"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Test at your real concurrency
&lt;/h2&gt;

&lt;p&gt;At batch size 1, the vLLM team says the GPU is memory-bound with spare compute, so draft tokens cost relatively little. At high concurrency, the draft tokens compete with real tokens for compute, and rejected drafts waste it. A March 2026 &lt;a href="https://arxiv.org/abs/2601.11580" rel="noopener noreferrer"&gt;arXiv study of speculative decoding in vLLM&lt;/a&gt; found speedups fall as batch size rises.&lt;/p&gt;

&lt;p&gt;Run your load test across low, medium, and peak concurrency, and include a run with speculation disabled. If traffic swings widely, look at vLLM's adaptive verification, which sizes verification per request.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep outputs trustworthy
&lt;/h2&gt;

&lt;p&gt;With the standard rejection sampler, vLLM describes speculative decoding as lossless up to hardware numerical precision. Lossy options such as FLy deliberately relax verification, so keep them out of quality-sensitive paths.&lt;/p&gt;

&lt;h2&gt;
  
  
  Skip the stack tuning: run vLLM on DigitalOcean
&lt;/h2&gt;

&lt;p&gt;Everything above applies wherever you run vLLM. But if you'd rather not run this yourself, DigitalOcean's &lt;a href="https://www.digitalocean.com/blog/dedicated-inference-technical-deep-dive" rel="noopener noreferrer"&gt;Dedicated Inference&lt;/a&gt; hosts models on dedicated GPUs using vLLM and llm-d, with orchestration, endpoints, and scaling handled for you. Your team keeps the decisions that drive latency and cost: model, GPU profile, replica count, and scaling behavior.&lt;/p&gt;

&lt;h2&gt;
  
  
  Speculative decoding FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What metrics matter for LLM inference observability (TTFT, TPOT, ITL)?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Track acceptance rate, per-token latency (TPOT and ITL), and throughput at each concurrency level. Speculation cuts decode steps, so TTFT is mainly a regression check.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does speculative decoding change model outputs?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Not with the standard rejection sampler, which vLLM describes as lossless up to hardware numerical precision. Lossy verification modes such as FLy do change the distribution.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is the best inference provider for teams who don't want to configure vLLM and don't have dedicated MLOps?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;DigitalOcean Dedicated Inference is built for that case. It runs vLLM and llm-d on dedicated GPUs and manages orchestration, endpoints, and scaling, while you choose the model, GPU profile, and replica count.&lt;/p&gt;

&lt;h2&gt;
  
  
  References &amp;amp; further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Learn more about method selection and config keys in the &lt;a href="https://docs.vllm.ai/en/latest/features/speculative_decoding/" rel="noopener noreferrer"&gt;vLLM speculative decoding docs&lt;/a&gt;. Options change fast, so review the latest details before building.
&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://vllm.ai/blog/2026-08-14-dspark-adaptive-verification" rel="noopener noreferrer"&gt;adaptive verification post from the vLLM team&lt;/a&gt; explains why no static draft length wins across concurrencies.
&lt;/li&gt;
&lt;li&gt;Read the &lt;a href="https://arxiv.org/abs/2601.11580" rel="noopener noreferrer"&gt;arXiv study on speculative decoding in vLLM&lt;/a&gt; for how speedups change with batch size.
&lt;/li&gt;
&lt;li&gt;Learn more about &lt;a href="https://www.digitalocean.com/blog/dedicated-inference-technical-deep-dive" rel="noopener noreferrer"&gt;Dedicated Inference architecture&lt;/a&gt; and the &lt;a href="https://www.digitalocean.com/products/inference-engine" rel="noopener noreferrer"&gt;DigitalOcean Inference Engine&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>startup</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
