DEV Community: Jahn

The sm_120 shared-memory cliff: why FP8 KV cache crashes vLLM on workstation Blackwell

Jahn — Tue, 25 Aug 2026 21:02:33 +0000

If you run vLLM with --kv-cache-dtype fp8 on a DeepSeek-family (MLA) model and your GPU is a GB10, an RTX PRO 6000, or any workstation or consumer Blackwell card, there is a decent chance the engine dies on startup with:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 102400, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

We hit this, measured exactly where the 1,024 missing bytes come from, and posted the measurements on the upstream issue (vllm-project/vllm#53748). This post is the diagnostic method, because it applies to any Triton kernel on this hardware class, not just this one bug.

The hardware fact underneath

Datacenter Blackwell parts advertise a larger per-block shared-memory budget than the workstation and consumer silicon. On an RTX PRO 6000 Blackwell (sm_120), torch.cuda.get_device_properties reports:

property	bytes
`shared_memory_per_block_optin`	101,376
`shared_memory_per_multiprocessor`	102,400

GB10 (DGX Spark) reports the same 101,376-byte per-block opt-in limit. So a kernel tuned to fit 102,400 bytes fits the SM but not the block, and the whole workstation Blackwell class fails together. If your kernel config was validated on H100 or B200 numbers, this is the cliff you fall off when someone runs it on a desk machine.

The five-minute probe

Do not debug this inside the serving engine. Launch the suspect kernel directly with the exact tile configuration the engine would pin, sweep num_stages, and read the compiled kernel's shared-memory size. For vLLM's MLA decode kernel (_fwd_grouped_kernel_stage1) with the DeepSeek tile shape (BLOCK_DMODEL=512, BLOCK_DPE=64, BLOCK_DV=512, BLOCK_N=32, BLOCK_H=16, num_warps=4):

num_stages	KV bf16	KV fp8e4m3
1	59,392 (launches)	83,968 (launches)
2	63,488 (launches)	102,400 (OutOfResources)
3	63,616 (launches)	102,656 (OutOfResources)
4	63,616 (launches)	102,656 (OutOfResources)

The table says everything. With bf16 KV, an extra pipeline stage costs about 4 KB and even four stages sit far under the limit. With fp8 KV, a stage costs about 18 KB, and stage two lands on 102,400 bytes: exactly the full per-SM budget, and 1,024 bytes over the per-block cap. The crash is specific to the fp8 KV path; MLA with bf16 KV is fine at these tile sizes.

The probe is ordinary Triton: import the kernel function, build dummy tensors for the tile shape, launch once per (dtype, num_stages) combination, and catch OutOfResources. The compiled size is on kernel.metadata after a successful launch. No server, no model download, under a minute per configuration.

Reproducing end to end

DeepSeek-V2-Lite-Chat is the smallest convenient MLA model with the same Lk=576 head layout (512 latent plus 64 rope):

vllm serve deepseek-ai/DeepSeek-V2-Lite-Chat --trust-remote-code \
  --enforce-eager --kv-cache-dtype fp8 --max-model-len 4096

On sm_120 the backend selector reports TRITON_MLA as the only MLA decode option, so there is no fallback path: the engine dies during the startup dummy run (vLLM 0.27.1, torch 2.13.0+cu130, triton 3.7.1, CUDA 13.1).

What a durable fix looks like

The upstream patch direction pins num_stages=1 for MLA kernels on non-HIP devices, which makes the server start and generate correctly (we validated greedy output on sm_120 after applying it). But the sweep shows why a blanket is_mla condition costs something: bf16 KV at two stages fits comfortably (63,488 < 101,376), and the second stage exists for throughput. The robust rule is to compute the tile's actual shared-memory requirement and compare it against shared_memory_per_block_optin for the device you are actually on. That handles GB10, RTX PRO 6000, and whatever ships next, without a special case per chip.

The general lesson for anyone shipping Triton kernels: your shared-memory budget is a per-block, per-device property, not a compile-time constant, and the workstation cards your users actually own report smaller numbers than the datacenter parts your CI runs on. Probe first; it takes five minutes.

This came out of inference engineering work on Blackwell workstations. The benchmark that surrounds it, with raw data and reproduction scripts, is here: https://conatus.jahn.ai/ai-engineering/sample-report

Did FP8 make the model dumber? A per-prompt regression check for quantized serving

Jahn — Tue, 25 Aug 2026 18:12:24 +0000

FP8 gave us a clean 1.5x on Qwen3-8B serving throughput on an RTX PRO 6000 Blackwell (1,725 to 2,597 tok/s at concurrency 32, vLLM). The uncomfortable question is always the same: did the model get dumber. This post is the exact check we ran before recommending the switch, with numbers, so you can run the same one.

Why "run an eval suite" is usually the wrong first answer

Standard benchmarks (MMLU and friends) are noisy instruments for quantization deltas at 8B scale. Score movement inside the error bars tells you nothing about whether YOUR prompts changed behavior. What you actually want to know is narrower: on the workload you serve, does the FP8 checkpoint produce materially different outputs than BF16, and are any of the differences wrong.

That is answerable directly, cheaply, and per prompt.

The method

Both configurations run the same fixed workload: 20 prompts covering reasoning, code, summarization, translation, extraction, classification, math, and instruction following. Greedy decoding, temperature 0, 256-token cap, streamed. Greedy matters: it removes sampling noise, so any output difference is attributable to the numerics.

Then a three-stage comparison:

Byte equality. outputs_bf16[i] == outputs_fp8[i]. Anything identical is settled.
Similarity triage. For non-identical pairs, difflib.SequenceMatcher.ratio() sorts near-identical wording drift from real divergence.
Side-by-side review under a written rubric. Every non-identical pair gets read. The rubric asks one question: is there a factual or numerical claim that one precision gets right and the other gets wrong. Wording changes, reordering, and equally-defensible readings are recorded but not counted as regressions.

The core loop is small:

import difflib, json

bf16 = json.load(open("vllm_bf16_conc1.texts.json"))
fp8  = json.load(open("vllm_fp8_conc1.texts.json"))

for i, (a, b) in enumerate(zip(bf16, fp8)):
    if a == b:
        print(i, "identical")
        continue
    r = difflib.SequenceMatcher(None, a, b).ratio()
    print(i, f"similarity {r:.3f}")
    # non-identical pairs go to side-by-side review

What it found on Qwen3-8B FP8

7 of 20 outputs byte-identical.
9 differed only in wording or formatting; equivalent content on review.
3 minor regressions: a repeated word in a poem, one list item drifting off topic, one questionable tool suggestion.
1 extraction prompt was ambiguous and produced two defensible readings.
Zero cases where FP8 gave a wrong factual or numerical answer that BF16 answered correctly.

That last line is the acceptance bar. Minor stylistic wobble is expected from a numerics change; a flipped fact is a blocker. This profile passed, so the 1.5x was free for this workload.

Two honest caveats. First, this validates a prompt profile, not the model in general: different domains, longer contexts, or sampled decoding need their own pass. Second, greedy-decoding equality is a strict signal but not a complete one; if you serve with sampling, run the review stage on sampled pairs too and expect more (benign) divergence.

The wrinkle that almost blocked the whole thing

Getting FP8 to run at all on workstation-class Blackwell (sm_120) required routing around a kernel assertion in the default FP8 path. If you are on the same silicon and vLLM refuses to load the FP8 checkpoint, that is a known class of problem rather than something wrong with your setup.

The full report this check belongs to, with the raw CSVs, environment manifest, and the one-command reproduction script, is here: https://conatus.jahn.ai/ai-engineering/sample-report

Qwen3-8B on workstation Blackwell: vLLM vs SGLang vs llama.cpp, plus an FP8 pass

Jahn — Tue, 25 Aug 2026 16:56:08 +0000

Benchmarks of the same model on the same GPU across three serving stacks, then an FP8 pass on the winner. All numbers measured on our own hardware last week. Raw CSVs, the environment manifest and a one-command reproduction script exist for every figure; the script was re-run end to end after the report was written and all figures reproduced within 6 percent.

Setup

GPU: RTX PRO 6000 Blackwell, 96 GB, sm_120 (workstation Blackwell, not the datacenter parts)
Model: Qwen3-8B
Stacks: vLLM 0.27.1, SGLang 0.5.9, llama.cpp (CUDA build)
Method: identical prompt set and sampling settings across engines, greedy decoding, output token counts matched across engines before any timing was compared. Concurrency 32 for the aggregate numbers.

Matching token counts before comparing throughput matters more than it sounds. Engines disagree about stop conditions and detokenization, and a 10 percent difference in generated tokens quietly becomes a 10 percent throughput lie.

BF16, concurrency 32

stack	aggregate tok/s	TTFT p50	e2e p99
vLLM 0.27.1	1,725	39 ms	3.4 s
SGLang 0.5.9	1,327	42 ms	5.0 s
llama.cpp (CUDA)	428	316 ms	16.3 s

FP8 on the winner (vLLM, same settings)

precision	single-stream tok/s	batch tok/s	latency p50
BF16	86	1,725	0.74 s
FP8 (official checkpoint)	130	2,597	0.49 s

What the numbers say

Stack choice barely matters for one user. Single-stream decode sits in the 83 to 96 tok/s band on every stack. If your service has one concurrent user, pick whatever is easiest to operate.

Stack choice is a 4x decision under load. At concurrency 32 the spread is 428 to 1,725 tok/s for the same model on the same silicon. Time to first token spreads 4x as well (39 ms vs 316 ms p50), which dominates perceived snappiness for chat workloads.

FP8 was a clean 1.5x. 1,725 to 2,597 tok/s aggregate, 86 to 130 single-stream, p50 latency down a third. We ran a fixed 20-prompt factual check before and after: zero regressions. For an 8B model on a 96 GB card the VRAM saving is irrelevant, the win is throughput.

The sm_120 wrinkle

Workstation Blackwell is not datacenter Blackwell, and default paths tuned for the latter sometimes assert on the former. Getting FP8 to run here required routing around a kernel assertion in the default FP8 path (DeepGEMM asserting on sm_120, resolved by falling back to the CUTLASS path). Related example of the same hardware-class gap: vllm-project/vllm#53748, where the Triton MLA decode kernel overflows the 101,376-byte per-block shared-memory limit that GB10 and workstation Blackwell share, but datacenter parts do not.

If you are serving on RTX PRO 6000, GB10 or consumer Blackwell, budget time for this class of issue. The performance is there, the default configs often are not.

Method notes

Greedy decoding for determinism; throughput measured after warmup
Aggregate numbers are steady-state over the full prompt set, not peak
Each engine got the same max batch and context settings where the knob exists

Full methodology and the case study writeup: conatus.jahn.ai/ai-engineering