DEV Community

Cover image for SGLang 0.5.23 Megakernels: When Does Fusing a Decode Step Into Persistent Kernels Actually Pay Off?
AI Tech News
AI Tech News

Posted on

SGLang 0.5.23 Megakernels: When Does Fusing a Decode Step Into Persistent Kernels Actually Pay Off?

TL;DR

  • SGLang 0.5.23 (released 2026-11-02, 1007 PRs from 257 contributors) adds an opt-in "low-latency mode": an NVIDIA megakernel plugin and an AMD monokernel plugin that run decode as fused persistent kernels. The release notes claim up to 44% faster single-user decode on GB300 and 75% on MI350X for DeepSeek-V4.1-Flash.
  • A speedup of 44% means 30.6% of the step time disappeared. A speedup of 75% means 42.9% disappeared. That is the number to reason about, because it is the launch and collective gap the fused kernel removed.
  • That gap is roughly fixed per step, while useful work grows with batch size. In a simple illustrative model the same fix gives about +44% at batch 1 and about +15% at batch 64. The shape matters more than the exact numbers.
  • Both plugins are labeled experimental, cover specific models, GPUs and tensor-parallel sizes, and are inert unless you pass a flag. Treat them as a latency-tier feature and canary them with a flag you can remove.

Source: SGLang v0.5.23 release notes, sgl-nvidia-mega, sgl-amd-mono.

What did SGLang 0.5.23 actually ship?

The headline item in the release notes reads: the opt-in NVIDIA megakernel and AMD monokernel plugins run DeepSeek-V4.1-Flash decode as fused persistent kernels, for up to 44% faster single-user decode on GB300 and 75% on MI350X. The pull requests behind it are #44945 (the --enable-nvidia-megakernel flag and a plugin layer), #44353 (one runtime switch for vendor monokernel plugins), #45450 and #44621.

Three details from the plugin READMEs shape how you should read that sentence.

  1. It is a plugin, not a core rewrite. Both packages load through SGLang's general plugin entry point (sglang.srt.plugins). The NVIDIA package registers each model's spec with SGLang's megakernel framework. Without the flag, the README says SGLang runs untouched, and the 0.5.23 notes say the CUDA Docker images ship the plugin "inert unless --enable-nvidia-megakernel is set".
  2. It replaces only some steps. With the flag on, SGLang checks the setup at argument resolution, before weights load. After the model runner prepares a covered model, SGLang builds the megakernel and replaces that model instance's forward. The replacement runs each covered step itself and leaves every other step to the model's own forward. For DeepSeek-V4.1-Flash, the README states that larger verify steps, prefill and extend run SGLang's standard path.
  3. It is narrow on purpose. The NVIDIA table lists DeepSeek-V4.1-Flash on GB300 and B300 at tensor-parallel 4, with speculative verify steps of 1 to 384 rows (batch 1 to 64 at 6 rows per request). On B300, the README says speed was measured at one request only. Other rows in the table cover Qwen3.5, Kimi K3, Kimi K3.1 and GLM-5.3 and GLM-5.2 with their own limits. Both READMEs carry an "Experimental" banner saying the kernels, covered setups and the switch may change.

Why would one fused kernel make small-batch decode faster?

The AMD README states the mechanism plainly: a monokernel is a persistent kernel that runs most of a decoder layer, including attention, the mixture-of-experts block and both tensor-parallel all-reduces, in one or two launches. The stated benefit is removing "the launch gaps and collective latency that dominate small-batch decoding".

At batch 1, a decode step moves a small amount of data per layer. The GPU finishes each small kernel quickly and then waits for the next launch, or waits for an all-reduce to complete before the next operator can start. The NVIDIA README describes the fix from the other side: kernels hand their outputs to each other through mailboxes and programmatic dependent launch instead of crossing per-operator boundaries.

For a model run at tensor-parallel 4, every layer pays for several collectives. Each one is a synchronization point where all four GPUs wait for the slowest. Fusing the all-reduce into the same persistent kernel lets the next stage start as soon as data arrives rather than after a separate launch.

What does "44% faster" mean in step time?

Speedups are easy to misread. Tokens per second going up 44% means each token takes 1/1.44 of the old time:

def step_time_removed(speedup):
    # speedup = 0.44 means 44% more tokens/s
    return 1 - 1 / (1 + speedup)

for name, s in [("GB300", 0.44), ("MI350X", 0.75)]:
    print(name, round(step_time_removed(s) * 100, 1), "% of step time removed")
# GB300 30.6 % of step time removed
# MI350X 42.9 % of step time removed
Enter fullscreen mode Exit fullscreen mode

Share of decode step time removed by the two plugins

So, on the best-case single-user workload, roughly a third of the GB300 step and about two fifths of the MI350X step was gap and synchronization overhead. That is a useful diagnostic by itself. If your own decode traces at batch 1 show a similar share of idle time between kernels, you have a candidate for this class of optimization. If your traces show the GPU busy end to end, you will not get anything like these numbers.

Why does the gain shrink as batch size grows?

The removed overhead is mostly per step, not per request. Useful work, such as expert matmuls and attention over more KV entries, grows with batch size. So the same absolute saving is a smaller share of a bigger step.

Here is a deliberately simple illustrative model, not a measurement. Assume a step costs 100 time units at batch 1, of which 30.6 are removable gap (matching the 44% claim). Assume the remaining work grows 3% for each extra request in the batch.

G = 30.6          # removable gap per step (units), from the 44% claim
W1 = 69.4         # useful work at batch 1
GROW = 0.03       # illustrative: +3% work per extra request

for b in (1, 2, 4, 8, 16, 32, 64):
    W = W1 * (1 + GROW * (b - 1))
    gain = (W + G) / W - 1
    print(b, f"{gain*100:.1f}%")
# 1 44.1%   2 42.8%   4 40.5%   8 36.4%   16 30.4%   32 22.9%   64 15.3%
Enter fullscreen mode Exit fullscreen mode

Illustrative speedup versus batch size

The growth rate of 3% is invented for illustration. The point is the direction: a fixed saving over a growing denominator. Real curves depend on expert routing, KV length and how well the standard path already overlaps its own gaps at larger batches. The README also says that on B300 speed was measured at one request only, and that larger batches rely on kernel tests. That is consistent with where the benefit is strongest.

One more wrinkle: with speculative decoding, the "rows" in a verify step are batch times drafted tokens. The NVIDIA table's 1 to 384 rows at 6 rows per request corresponds to batch 1 to 64. A fixed gap saving is therefore shared across more rows per step than a plain decode step, which already softens the relative gain at a given batch.

Who should care: the latency tier, not the throughput tier

Many fleets run two kinds of traffic. Interactive traffic (agents waiting on a tool loop, a developer in an editor) cares about per-user inter-token latency at low concurrency. Batch and background traffic cares about aggregate tokens per second per dollar at high concurrency.

The plugins target the first kind. A single user at batch 1 sees the full benefit. An autoscaled throughput pool running at batch 32 or 64 sees a smaller one, and the plugin's tested coverage is thinnest there. Matching each plugin to the pool where it was measured is cheaper than turning it on everywhere.

A related default change in the same release is worth separating from this one. For DeepSeek-V4.1-Flash under prefill/decode disaggregation on SM100 and SM103, both roles now resolve to the MegaAttention backend, and the handshake checks the KV layout, so prefill and decode must run the same attention backend. The release notes tell you to upgrade both roles together, or pass --dsv4-attn-backend flashmla on both to keep FlashMLA. That is a separate mechanism from the megakernel plugin, but both arrive in the same upgrade, so keep them apart when you attribute a latency change.

How do you canary it safely?

Because the plugin is inert by default and gated by one flag, the rollout is simple and reversible.

# Latency-tier canary, tensor-parallel 4 on a covered GB300 setup
python -m sglang.launch_server \
  --model-path <your-DeepSeek-V4.1-Flash-checkpoint> \
  --tp 4 \
  --enable-nvidia-megakernel
# On AMD Instinct: --enable-amd-monokernel instead
Enter fullscreen mode Exit fullscreen mode

Checklist:

  1. Confirm it engaged. The README says every rank logs NVIDIA megakernel: active (...). If you do not see that on all ranks, you are benchmarking the standard path and any difference is noise. The argument-resolution check runs before weights load, so a rejected setup fails fast rather than silently.
  2. Match the covered configuration. Model, GPU generation, tensor-parallel size and batch range all appear in the tables. Outside them, steps fall back to the standard path.
  3. Measure at your concurrency, not at one. Run the same prompts at concurrency 1, 4, 16 and 64 with and without the flag. Record p50 and p99 inter-token latency and aggregate tokens per second. The curve, not the batch-1 point, tells you where to draw the pool boundary.
  4. Compare quality with enough samples. The NVIDIA README reports that on GLM-5.2 NVFP4 (8 x B300) GSM8K-200 strict match scored 0.975 with the megakernel and 0.960 without. That is three questions out of 200, well inside sampling noise, so read it as "no regression observed", not as an improvement. Run your own eval with a larger set and fixed seeds.
  5. Keep the rollback trivial. Removing the flag restores the untouched path. Do not bake it into an image entrypoint until you have run it through at least one full release cycle, since the README warns the switch itself may change.

How is this different from CUDA graphs?

CUDA graphs already remove per-launch CPU overhead by replaying a recorded sequence. The plugins go further: they change what runs inside the sequence, merging operators and communication into persistent kernels that exchange data without per-operator boundaries. The AMD README also notes that each step is routed by its CUDA graph key alone, so every rank makes the same decision and a captured graph replays the decision made at capture. In other words, the plugins sit inside the graph machinery rather than replacing it. If you disabled graphs for debugging, expect different behavior.

FAQ

Is the 44% number what I will see in production?
Not by default. It is the release notes' upper bound for single-user decode on GB300 with DeepSeek-V4.1-Flash. Your batch mix, prompt lengths and speculative settings change it. Use the step-time view to estimate how much gap you have to remove.

Does it work on any model?
No. Each plugin lists covered models, GPUs and tensor-parallel sizes. Anything else runs the standard path. The NVIDIA table covers DeepSeek-V4.1-Flash, Qwen3.5-397B-A17B, Kimi K3 and K3.1, and GLM-5.3 and GLM-5.2. The AMD table is broader on models but limited to specific Instinct parts.

Does it speed up prefill?
The README says prefill and extend run SGLang's standard path for DeepSeek-V4.1-Flash. The plugin targets small-batch decode and speculative verify steps.

What flag turns it on?
--enable-nvidia-megakernel for NVIDIA and --enable-amd-monokernel for AMD. Both are described as experimental.

What is the first thing to check if I see no change?
Look for the NVIDIA megakernel: active log line on every rank. Then confirm your model, GPU and tensor-parallel size are in the supported table.

Bottom line

Persistent kernels attack a real cost: at batch 1, a large share of a decode step is gaps and collectives rather than math. SGLang 0.5.23 packages that idea as a reversible, flag-gated plugin. Convert the headline speedup into removed step time, expect the benefit to shrink as batch grows, put it on the pool where users feel inter-token latency, and let your own concurrency curve decide the rest.

Further reading

Top comments (0)