Speculative decoding guesses a few tokens ahead and throws away the wrong ones. On a hybrid Mamba model, that means saving a copy of every layer's state first. At large batch sizes, the copies cost more than the guessing saves.
Engineer's view. This is a database that rewrote the whole table file on every transaction so it could roll back. Real databases append each change to a log, checkpoint now and then, and roll back by truncating the log. A Mamba layer can store its state the same way.
TL;DR. A Mamba layer overwrites its state every token, so undoing a rejected guess needs a saved copy. At large batch sizes those copies cost more than speculation saves. Logging each token's inputs and checkpointing now and then makes undo a pointer move.
Explain Like I'm 5
You are writing a story on a whiteboard, and a friend guesses your next four words. You write them in. If a guess is wrong, you must erase back to where you were. On paper you just cross out the last words. On a whiteboard that gets wiped and redrawn every word, you need a photo of the board before each guess. Taking all those photos is slow. A faster trick: photograph the board once in a while, and keep a list of the words since then.
The Problem
You have shipped this bug before, in a system with no AI in it.
A collaborative editor needs undo. The first version saves a full copy of the document before every edit. It works in testing. In production, a hundred people type into large documents at once. Each keystroke now writes megabytes, and the disk spends its time copying documents nobody will ever restore. The edits are cheap. The copies are not. The fix everyone lands on is an operation log: store each edit, which is small, and save a full snapshot now and then. Undo drops the newest edits from the log.
Hybrid models have the same bug inside speculative decoding. Speculative decoding lets a cheap drafter guess the next few tokens, and the big model checks them all in one pass. Wrong guesses are thrown away. How a Model Writes Several Words per Step covered why that pays.
Hybrid models mix two kinds of layer. Attention layers keep a cache that only grows. A Mamba layer keeps one fixed-size state per request and overwrites it on every token. Nemotron-3-Super has 40 Mamba layers and 8 attention layers. Each Mamba state is 4 MiB per request. That is as much as one attention layer stores for about 8,000 tokens. How to Work Out How Much GPU Memory One Request Really Needs sized that fixed part. Here it is the traffic that hurts.
To throw away a guess, an attention layer moves a pointer back. A Mamba layer cannot, because the old state is gone. So the server saves a full copy of the state after every draft token. With four drafts, that is four copies per layer, per request, per step.
The fix: store each token's small inputs plus a periodic checkpoint, the way the editor stores an operation log. Then throwing away a guess is a pointer move again.
Read the full post on The AI Commit
Why Speculative Decoding Slows Hybrid Mamba Models at Large Batch Sizes carries the rest: the fix and why it works, the code to change for each role, how to tell it worked, when it is the wrong tool, and an interactive model you can play with.
Originally published at theaicommit.com, a daily AI engineering lab for software engineers. The original has an interactive version you can play with, and the full runnable code is on GitHub.
Top comments (0)