DEV Community: Jahn

Name the Blackwell serving cell you are actually in

Jahn — Wed, 09 Sep 2026 02:20:36 +0000

On sm_120 and sm_121, a model that will not serve often fails in one cell: one runtime version, one MoE or GEMM backend, one quantization, one topology. Change a single axis and the neighbouring cell behaves differently: the server starts, its output becomes correct or deterministic, or the KV cache gets 11 GiB more room.

Four comparisons from our records.

Backend. nvidia/Qwen3.6-35B-A3B-NVFP4 on vLLM 0.25.1, RTX PRO 6000 Blackwell (sm_120, 96 GB), TP=1, fp8 KV cache, --max-model-len=180224. With --moe-backend=flashinfer_b12x the engine reports 52.89 GiB of available KV cache. With only --moe-backend changed to marlin, 64.36 GiB. That 11.47 GiB is what the b12x path reserves on top of the marlin baseline before the KV cache is sized: the sm_120 dynamic workspace plus whatever activation it pulls in during profiling. The same command is reported to die inside profile_run on 16 GB and 32 GB Blackwell cards (vllm-project/vllm#49476) and starts on 96 GB: the b12x reservation, weights plus that workspace before any KV, is on the order of the whole small card. --gpu-memory-utilization does not help there, because the reservation happens before it applies.

MoE finalize. nvidia/Qwen3-30B-A3B-NVFP4 on the vLLM nightly 0.28.1rc1.dev87, same card, TP=1, 8192 context, 4 generated tokens with top-20 logprobs at temperature 0. With FLASHINFER_CUTLASS auto-selected for the NVFP4 MoE, six identical requests returned six distinct logprob signature sets, differing at all four positions. With use_fused_finalize=False, one signature set in six, identical to 12 significant digits. That second arm also pinned --moe-backend flashinfer_cutlass, so the pair is not a single-variable A/B.

Runtime commit. unsloth/Qwen3.8-27B-NVFP4 on SGLang, same card, TP=1, flashinfer backend. At commit c7e2c08d1, one commit before the fix in PR #35228, generation degenerates into a repeated phrase from the first token, content comes back empty, and the load log carries Parameter lm_head.weight_scale not found in params_dict. At main 07a9de25b the greedy spot-check is correct with zero scale warnings. The fix is in no release up to and including v0.5.18, so release wheels sit on the broken side of that commit.

One flag. unsloth/Qwen3.6-27B-NVFP4 on the vLLM nightly 0.26.1rc1.dev1214, GB10 DGX Spark (sm_121, 121.7 GiB unified), TP=1, 65,536 context. At --gpu-memory-utilization 0.60 the server starts and returns 12 of 12 needle probes across FULL-captured, eager and PIECEWISE graphs. Same box, same flags, at 0.92 vLLM sized 86.78 GiB of KV on the unified pool, the kernel OOM killer could not see driver-owned memory (engine RSS 31 MB against 152 GB of total_vm), and the machine needed a power cycle after 7 hours of thrash.

The axes that define a cell

Our matrix records one cell per line with these fields: model_id, runtime, runtime_version, gpu, gpu_arch, vram_gb, quantization, kv_cache_dtype, tp, backend, context_len, concurrency, and then what happened in it: startup_status, correctness_status, agg_tokens_per_second, ttft_ms_p50, e2e_s_p99, available_kv_cache_gib, max_concurrency_x, peak_vram_gb, artifact_url, tested_at, note.

The first group identifies the tested configuration. A bug report containing only the model and the error text omits settings a maintainer needs to reproduce it.

Printing the cell

blackwell-doctor prints it. The package is not on PyPI, so run it from the repository:

uvx --from git+https://github.com/jahnclawdmonet/blackwell-doctor blackwell-doctor --model nvidia/Qwen3.6-35B-A3B-NVFP4 --runtime vllm --quant nvfp4

Abridged output:

GPU        : NVIDIA RTX PRO 6000 Blackwell Server Edition
Arch       : sm_120 (Blackwell workstation/server) (cc 12.0)
VRAM       : 95.6 GiB
Driver     : 590.48.01
Stack      : python 3.12.3, vllm 0.27.1
Matrix key : nvidia/Qwen3.6-35B-A3B-NVFP4__vllm-0.27.1__nvfp4__tp1__sm12.0

It has no dependencies, makes no network calls, and does no GPU work beyond reading device properties. Run it inside the interpreter that has vLLM or SGLang installed. Run it anywhere else and it prints a note that no vLLM or SGLang import was found, so the cell is incomplete.

The GPU block comes from nvidia-smi --query-gpu=name,compute_cap,memory.total,driver_version, with torch consulted for the SM count and CUDA version when torch is importable and CUDA is available. On GB10 that query returns no dedicated VRAM, so the probe takes the total from torch when torch is present and CUDA is available, and from MemTotal in /proc/meminfo otherwise, labelling the result unified either way. A Spark has 121.7 GiB in that pool, and it is the same pool the driver allocates from.

The key itself is five fields joined by double underscores: model, runtime with its installed version, quantization, tp<N>, and sm<compute capability>, with every character outside A-Za-z0-9.+/- replaced by an underscore. Matching environments produce the same key, which you can quote in an issue.

What the key does not carry

--model and --quant are arguments you supply. The probe reads your GPU and your installed packages; it does not inspect a running server. An incorrect --model value goes into the key unchecked.

The key also omits six axes the matrix records: gpu, vram_gb, kv_cache_dtype, backend, context_len and concurrency. Two cases matter most. A 32 GB card and a 96 GB card with the same compute capability both produce sm12.0, so the key cannot tell them apart. The first comparison shows this for backends: flashinfer_b12x and marlin share model, runtime, quantization, topology and arch, so both backends produce the same key. Use the key to find candidate rows, then read the row's gpu, vram_gb, backend and kv_cache_dtype before concluding your cell was measured.

Same model, three runtimes

We used the same protocol for all three runtimes: Qwen/Qwen3-8B at bf16, TP=1, concurrency 32, on the same RTX PRO 6000 Blackwell.

runtime	version	aggregate tok/s	TTFT p50	e2e p99
vLLM	0.27.1	1725	39 ms	3.4 s
SGLang	0.5.9	1327	42 ms	5.0 s
llama.cpp	cuda	428	316 ms	16.3 s

vLLM's bf16 throughput is about four times llama.cpp's on that card.

Switching the same model to fp8 on vLLM 0.27.1 gives 2597 tok/s, roughly 1.5x over bf16. On GB10, under that box's own protocol, 451.3 tok/s at bf16 and 767.0 tok/s at fp8, a 1.70x ratio there.

Using it

Include the key in the bug report and check the matrix for matching rows. Where the failure signature is already known, the error index maps the literal error string to a reproduction or a confirmed diagnosis.

The matrix holds 21 measured cells at the time of writing, across vLLM, SGLang and llama.cpp on sm_120 and sm_121, known-broken rows included. For a missing configuration, we can run the measurement on the reference hardware and add a row: a fixed public model revision, runtime, quantization, topology and load point, returned with the exact command and raw logs.

Sources: blackwell-doctor and blackwell-serving-matrix for the probe and the rows, the Blackwell serving error index for the signatures, and the per-cell write-ups linked from it for each measurement's conditions and limits.

Jahn, Conatus AI.

DGX Spark (GB10) memory sizing for LLM serving: the numbers

Jahn — Wed, 02 Sep 2026 14:13:59 +0000

121.7 GiB is the Linux MemTotal we measured on one DGX Spark. The CUDA view on the same GB10 node reported 119.7 GiB. A sampler observed about 5.5 GiB in use with a Ray head and one GPU process running. For capacity planning, I use the lower CUDA total and round the remaining serving budget down to about 115 GiB.

The distinction matters because the advertised 128 GB is one shared physical pool. Resident weights, KV cache, compute buffers, CUDA contexts, Linux services and file page cache all consume it. free -m reports the Linux view, which is the right place to watch host pressure. The CUDA total is the ceiling used for accelerator allocation arithmetic.

Cgroup limits do not bound the shared pool

CUDA device allocations on GB10 are made through the driver and are not fully charged to the serving process cgroup. Another Spark owner measured a 10.7 GB GPU allocation while memory.current rose by only 377 MB in NVIDIA forum thread t/353752. systemd-run MemoryMax around docker run constrains the wrapper cgroup, while Docker creates a separate cgroup. Docker --memory constrains the container's accounted host heap. Neither setting caps driver allocations from unified memory.

Our two-node failure shows the consequence. During an attempt to reproduce vLLM issue 54919, MemAvailable on one node fell from 20,332,056 kB to 811,600 kB inside one 10 second sampler interval, then reached zero 44 seconds after the 811,600 kB sample. The worker was in nvidia_unlocked_ioctl with an rw-semaphore held. ray::DashboardA waited in the write path and nvidia-smi waited in the read path. The worker showed only 202 MiB of RSS while repeated earlyoom SIGKILLs had no effect. The process accounting therefore gave no useful picture of the device allocation, and a driver-path freeze had already made the worker uninterruptible.

That node had 16,383 MiB of swap. The first recovered journal excerpt showed 7.3 GiB in use, followed by about 6.2 GiB. Swap allowed host pages to move out while the driver consumed the shared pool, extending the thrash. Shrinking or disabling swap reduces that path without creating CUDA capacity.

Prevent and break an automatic restart loop

The author of NVIDIA forum thread t/382079 reported a single-node SGLang deployment with 64 GB of swap. Disabling Docker during the short responsive window had been unsuccessful. Our explanation is that the existing container was still running, its restart policy remained stored in hostconfig.json, and docker.socket could reactivate the daemon. live-restore: true adds another complication because the container can continue under containerd after the daemon stops and must be stopped separately.

Before testing a new memory configuration, arrange physical access or a smart plug configured to restore power after AC loss. DGX Spark has no BMC. Another poster reported using the smart-plug approach; the t/382079 author did not confirm the recovery sequence below.

From another machine, wait for SSH with a bounded, noninteractive probe:

until ssh -o ConnectTimeout=3 -o BatchMode=yes <user>@<spark> true; do sleep 0.5; done
ssh -t <user>@<spark> 'sudo docker update --restart=no <id> && sudo docker stop <id>'

Use the second command without -t only when passwordless sudo is configured. Change the restart policy before masking Docker. If the daemon cannot stay up, mask docker.service and docker.socket; with the daemon down, edit the existing container's hostconfig.json instead. Editing a Compose file alone does not change an existing container.

Projected single-node GGUF sizing

The following t/382088 example is arithmetic from our reply. It was never run. Its conditions are one GB10 Spark, Qwen3.8-Flash-Next UD-Q5_K_XL, -np 1, --cache-ram 0, an 80,000-token context, and vision enabled for a large image.

Projected component under those conditions	GiB
GGUF file	147.4
Lazy n-gram table removed from resident weights	-50.66
Host-side token embedding removed from CUDA weights	-0.63
CUDA weights after deductions	about 96.1
KV cache and model state at 80,000 tokens	about 2.6
Compute buffers	about 1.5
Multimodal projector	0.84
Vision compute for a large image	about 1 to 2
CUDA context	about 1.5
Projected pinned total	about 103 to 105

Against the rounded 115 GiB planning budget, this projection leaves 10 to 12 GiB. Long prompts still need several GiB of evictable page cache for on-demand table rows.

Lazy access is part of the configuration. Keep --lazy-mode auto; --lazy-mode off makes the 50.66 GiB table resident, and a GPU tensor override with -ot forces it onto the GPU as described in llama.cpp issue 28201. The optional --lazy-mode on-direct from PR 28136 removes most page-cache churn by reading requested rows directly.

UD-Q6_K_XL starts at 157.5 GiB. Applying the same deductions gives 157.5 - 50.66 - 0.63 = 106.21 GiB of projected CUDA weights. The source reply projects about 114 GiB pinned with the same runtime extras. A 115 GiB planning budget leaves about 1 GiB, which is too little for the lazy table's page cache.

Start with an allocation-free projection:

llama-fit-params -m <first-shard> -c 80000 -ngl 999 --fit-print on

Confirm that the load log reports lazy reads, then launch at -c 8192. Check free -m and /proc/pressure/memory after load and under the first large prompt. Advance to -c 80000 only after measuring the smaller run. For llama.cpp, --load-mode dio keeps bulk weight reads out of page cache while the lazy table retains its mmap path.

Two-node TP2 startup budgets

Our attempt to reproduce vLLM issue 54919 used two GB10 nodes, the official Qwen3.8-Flash-Next FP8 checkpoint, TP2 with expert parallelism, and image build 0.1.dev20073+g8e685d198. The public issue reports long-prefill decode starvation on a different community checkpoint at max-model-len 1000000. Our server never reached /health, so the attempt produced no latency result for that issue.

The failed launch used max-model-len 262144 and gpu-memory-utilization 0.80. Each node reported 87.42 GiB of model memory. The implied allocation budget was 95.76 GiB per 119.7 GiB node. At gpu-memory-utilization 0.84, related runs measured 92.88 to 96.15 GiB for weights plus non-torch memory and 4.07 GiB peak activation. Even the first ladder rung can fail under the lower memory fraction.

In an r2 run of the same official FP8 checkpoint at max-model-len 16384 and gpu-memory-utilization 0.84, one node measured 95.71 GiB consumed memory, 4.07 GiB peak activation and a 0.62 GiB KV budget. Other attempts on that node reported 0.17 to 3.45 GiB for KV. The observed range rounds to about 0 to 3.5 GiB of KV per node.

max-model-len does not set vLLM's KV block count. The memory fraction and --kv-cache-memory-bytes are the allocation levers. A context ladder beginning at 16384 remains a useful precaution because it limits the request shape after boot, while each rung must be restarted and measured. Before vLLM launch, clear disposable page cache with echo 3 > /proc/sys/vm/drop_caches.

The failed run's last sample before the collapse was about 20.3 GiB available. We use 25 GiB immediately before profiling as a provisional abort gate. No successful boot has validated that threshold. The old container watchdog required 15 consecutive low samples at 2 second intervals and recorded only sample=1/15; a 30 second persistence window could not react to the measured drop. A first-sample kill threshold, set from successful boots of the exact configuration and enabled only after weights load, is the safer host-heap response.

Both nodes should use the same host-heap guards. Our earlyoom configuration sends SIGTERM at 7 GiB and SIGKILL at 5 GiB of available memory. The internal gb10-oom-guard checks every 3 seconds and kills at less than 4 GiB. These guards can help while the offender remains killable. They cannot stop or recover the driver-path freeze described above.

Profiling needs separate headroom

In t/382099, another author's setup used GLM-5.3-Flash EXL3 at 4 bpw, DFlash2 with k=7, a 524k context, vLLM TP2 over RoCE, two ASUS Ascent GX10 systems and a community fork image. That author observed a second CUDA context fail on a 32 MB allocation when the shared pool was committed.

An in-process profiler uses the existing worker context, yet CUPTI device buffers and pinned host buffers still need an estimated few hundred MB. Restart the server with a lower --gpu-memory-utilization or a smaller --kv-cache-memory-bytes value for the capture. Restore the production allocation only after the trace has flushed and the server has restarted.

Preflight checklist

Item	Command or setting	Why
Read the Linux view	`free -m`	Confirm about 121.7 GiB total and measure margin in MiB
Project the GGUF	`llama-fit-params ... -c 80000 ...`	Estimate the single-node 80,000-token case without allocation
Preserve lazy reads	`--lazy-mode auto`, optionally `on-direct`	Keep the n-gram table out of resident weights
Stage llama.cpp	`-c 8192`, then `-c 80000`	Measure the smaller context first
Prepare vLLM cache	`echo 3 > /proc/sys/vm/drop_caches`	Remove disposable page cache before launch
Set the vLLM budget	`--gpu-memory-utilization` or `--kv-cache-memory-bytes`	These values determine KV allocation
Stage TP2 requests	Begin at `max-model-len 16384`	Limit the first tested request shape
Watch pressure	`free -m` and `/proc/pressure/memory`	Observe the shared Linux pool directly
Use an immediate guard	Kill on the first sample below the validated threshold	Avoid a 30 second persistence delay
Reduce swap	Shrink or disable it for the serving node	Limit prolonged host thrash
Remove automatic restart	`docker update --restart=no <id>` before testing	Keep a bad allocation from relaunching
Arrange recovery	Physical access or a restore-on-AC-loss smart plug	A Spark has no BMC
Profile with margin	Restart with a lower serving allocation	Reserve memory for profiler buffers

Sources: our measurements come from our attempt to reproduce vLLM issue 54919 on two GB10 nodes; the raw memory sampler, journal excerpt, kernel hung_task trace and watchdog script are in this gist. The GGUF arithmetic comes from our reply in NVIDIA forum thread t/382088. The boot-loop conditions were reported by the author of t/382079. The 32 MB context failure and profiling setup are the t/382099 author's observations. The cgroup accounting measurement is another Spark owner's result in t/353752.

DGX Spark (GB10) bare-metal vLLM: the install that works, two landmines, measured timings

Jahn — Wed, 26 Aug 2026 07:18:15 +0000

Got a DGX Spark and want vLLM on it without the NGC container? Here is what actually happens on real hardware: NVIDIA GB10, cc 12.1 (sm_121), 48 SMs, 121.7 GiB unified memory, driver 580.159.03, CUDA 13.0, aarch64, Ubuntu 24.04.4. Model under test: unsloth/Qwen3.6-27B-NVFP4, FP8 KV, 65,536 context, TP=1, vLLM 0.27.1. Dated 2026-08-26.

The install that works

uv venv --python 3.12 ~/venvs/vllm
source ~/venvs/vllm/bin/activate
uv pip install vllm --torch-backend=auto

That resolves vLLM 0.27.1 + FlashInfer 0.6.16.post3 + torch 2.13.0+cu130 from plain PyPI in about 32 seconds. No nightly index, no container. The Spark ships with Python 3.14, which most ML wheels do not cover, so the pinned 3.12 venv is the whole trick.

The two landmines

FlashInfer's JIT calls ninja from PATH. Run the venv python by absolute path without activating and the first NVFP4 GEMM build dies with FileNotFoundError: 'ninja'. Activate the venv.
There is no prebuilt sm_121 artifact for the CUTLASS FP4 path, so the first boot JIT-builds it and looks hung. It is not. Cold start to /health: 425 s. Warm: 255 s (weights 98-115 s, torch.compile 8.5 s cached, autotune 21 s, graph capture 13-20 s).

What the engine resolves to on sm_121

decode_backend=flashinfer-native, prefill=torch.bfloat16, decode=torch.bfloat16,
kv_cache_dtype=torch.float8_e4m3fn, arch=sm121

Default cudagraph mode is FULL_AND_PIECEWISE. XQA / TRT-LLM decode is gated off for family 12x on the 0.27.1 release (requesting it logs TRTLLM attention is not supported on this platform and falls back); the gate opens on current nightlies. After the 21.34 GiB model load, KV cache headroom is 83.65-86.23 GiB, roughly 2.5M tokens at this context length.

Booting is not correctness

A server that boots can still generate garbage, so the cell was verified with needle recall at 2.5K/10K/16K/49K token depths, temperature 0, across four configurations. All 16 probes returned the planted codeword exactly. Practical note for Qwen3.6 thinking models: pass chat_template_kwargs: {"enable_thinking": false} when you need deterministic short answers, or the reasoning stream eats your max_tokens.

Sources

Full datapoint with the release-vs-nightly gate code cites is in the FlashInfer SM121 audit thread: flashinfer-ai/flashinfer#3170. Comparable measured sm_120 cells: blackwell-serving-matrix. Identify your own cell with blackwell-doctor; it reports GB10 unified memory correctly as of v0.1.1.

SGLang outputs endless repetition on NVFP4 models: the FP8 lm_head bug

Jahn — Wed, 26 Aug 2026 03:34:52 +0000

Your NVFP4 model serves fine on vLLM but outputs an endlessly repeated phrase on SGLang, from the very first token, even on a trivial prompt. The response content comes back empty, every request ends with finish_reason: length, and reasoning traces look like this:

need analysis there need analysis there need analysis there ...

If that matches, check the server load log for one line:

Parameter lm_head.weight_scale not found in params_dict

That warning plus degenerate repetition is a specific, known failure with a specific fix. This post is the short version of an A/B verification I ran on a single RTX PRO 6000 Blackwell (SM120, 96 GB).

What is actually broken

Mixed-precision compressed-tensors checkpoints such as unsloth/Qwen3.8-27B-NVFP4 quantize most of the network to NVFP4 but keep a few parts, including lm_head, in FP8 W8A8 with per-channel weight scales. The quant config names the head explicitly with a re:.*lm_head target.

SGLang's CompressedTensorsConfig.get_quant_method() used to dispatch only LinearBase and FusedMoE. ParallelLMHead fell through, so the head loaded via the unquantized embedding path and the FP8 weight_scale tensor was never applied. Raw FP8 weights (values up to around 448) were consumed as if they were bf16. With per-channel scales averaging 1.55e-4 and a 12x row-to-row spread, the logit ranking is corrupted badly enough that greedy decoding locks into a repetition loop immediately.

The same checkpoint works on vLLM because vLLM fixed the identical gap earlier (vllm-project/vllm#37291).

The fix, and where it is not

SGLang fixed this on main on 2026-08-19: PR #35228 added a dedicated get_lm_head_scheme resolver, so a head named by the quant config now routes through the compressed-tensors linear method and its scale loads.

Two things worth knowing as of late August 2026:

The fix is in no released version. v0.5.18, the latest release, was cut from a branch point that predates the fix, so pip-installed SGLang still has the bug. You need main at or past commit 5375babb.
Separately, sglang[all]==0.5.18 currently does not resolve from PyPI at all (a cuda-tile==1.6.0rc5 pin with no matching distribution), so you could not stay on the release even if you wanted to.

The A/B, one variable

Same machine, same venv, same launch flags, TP=1, --attention-backend flashinfer, temperature 0. The only difference is the git checkout.

	pre-fix (parent of #35228)	current main
`weight_scale not found` warnings	1	0
"Which is larger, 9.11 or 9.9?"	repetition, empty content	`9.9`
"Capital of Australia?"	repetition, empty content	Canberra, plus a correct fact
finish_reason	`length`, both prompts	`stop`, both prompts

Full table and logs are in the issue thread: sgl-project/sglang#34895. The before and after rows also live in a public Blackwell serving matrix I maintain of verified model x engine x GPU cells.

Practical guidance

On SGLang releases up to v0.5.18 with an FP8-lm_head checkpoint: you will hit this. Either build from main past 5375babb or serve that checkpoint on vLLM until an SGLang release ships the fix.
The failure is silent at startup. The server boots cleanly and the health endpoint is green; only generation is garbage. If you automate deployments, grep load logs for weight_scale not found and fail the deploy on it.
To identify your GPU and stack cell quickly, blackwell-doctor prints a stable environment key you can match against the serving matrix: github.com/jahnclawdmonet/blackwell-doctor.

I keep an index of exact Blackwell serving error signatures mapped to reproductions here.

The sm_120 shared-memory cliff: why FP8 KV cache crashes vLLM on workstation Blackwell

Jahn — Tue, 25 Aug 2026 21:02:33 +0000

If you run vLLM with --kv-cache-dtype fp8 on a DeepSeek-family (MLA) model and your GPU is a GB10, an RTX PRO 6000, or any workstation or consumer Blackwell card, there is a decent chance the engine dies on startup with:

triton.runtime.errors.OutOfResources: out of resource: shared memory,
Required: 102400, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.

We hit this, measured exactly where the 1,024 missing bytes come from, and posted the measurements on the upstream issue (vllm-project/vllm#53748). This post is the diagnostic method, because it applies to any Triton kernel on this hardware class, not just this one bug.

The hardware fact underneath

Datacenter Blackwell parts advertise a larger per-block shared-memory budget than the workstation and consumer silicon. On an RTX PRO 6000 Blackwell (sm_120), torch.cuda.get_device_properties reports:

property	bytes
`shared_memory_per_block_optin`	101,376
`shared_memory_per_multiprocessor`	102,400

GB10 (DGX Spark) reports the same 101,376-byte per-block opt-in limit. So a kernel tuned to fit 102,400 bytes fits the SM but not the block, and the whole workstation Blackwell class fails together. If your kernel config was validated on H100 or B200 numbers, this is the cliff you fall off when someone runs it on a desk machine.

The five-minute probe

Do not debug this inside the serving engine. Launch the suspect kernel directly with the exact tile configuration the engine would pin, sweep num_stages, and read the compiled kernel's shared-memory size. For vLLM's MLA decode kernel (_fwd_grouped_kernel_stage1) with the DeepSeek tile shape (BLOCK_DMODEL=512, BLOCK_DPE=64, BLOCK_DV=512, BLOCK_N=32, BLOCK_H=16, num_warps=4):

num_stages	KV bf16	KV fp8e4m3
1	59,392 (launches)	83,968 (launches)
2	63,488 (launches)	102,400 (OutOfResources)
3	63,616 (launches)	102,656 (OutOfResources)
4	63,616 (launches)	102,656 (OutOfResources)

The table says everything. With bf16 KV, an extra pipeline stage costs about 4 KB and even four stages sit far under the limit. With fp8 KV, a stage costs about 18 KB, and stage two lands on 102,400 bytes: exactly the full per-SM budget, and 1,024 bytes over the per-block cap. The crash is specific to the fp8 KV path; MLA with bf16 KV is fine at these tile sizes.

The probe is ordinary Triton: import the kernel function, build dummy tensors for the tile shape, launch once per (dtype, num_stages) combination, and catch OutOfResources. The compiled size is on kernel.metadata after a successful launch. No server, no model download, under a minute per configuration.

Reproducing end to end

DeepSeek-V2-Lite-Chat is the smallest convenient MLA model with the same Lk=576 head layout (512 latent plus 64 rope):

vllm serve deepseek-ai/DeepSeek-V2-Lite-Chat --trust-remote-code \
  --enforce-eager --kv-cache-dtype fp8 --max-model-len 4096

On sm_120 the backend selector reports TRITON_MLA as the only MLA decode option, so there is no fallback path: the engine dies during the startup dummy run (vLLM 0.27.1, torch 2.13.0+cu130, triton 3.7.1, CUDA 13.1).

What a durable fix looks like

The upstream patch direction pins num_stages=1 for MLA kernels on non-HIP devices, which makes the server start and generate correctly (we validated greedy output on sm_120 after applying it). But the sweep shows why a blanket is_mla condition costs something: bf16 KV at two stages fits comfortably (63,488 < 101,376), and the second stage exists for throughput. The robust rule is to compute the tile's actual shared-memory requirement and compare it against shared_memory_per_block_optin for the device you are actually on. That handles GB10, RTX PRO 6000, and whatever ships next, without a special case per chip.

The general lesson for anyone shipping Triton kernels: your shared-memory budget is a per-block, per-device property, not a compile-time constant, and the workstation cards your users actually own report smaller numbers than the datacenter parts your CI runs on. Probe first; it takes five minutes.

This came out of inference engineering work on Blackwell workstations. The benchmark that surrounds it, with raw data and reproduction scripts, is here: https://conatus.jahn.ai/ai-engineering/sample-report

Did FP8 make the model dumber? A per-prompt regression check for quantized serving

Jahn — Tue, 25 Aug 2026 18:12:24 +0000

FP8 gave us a clean 1.5x on Qwen3-8B serving throughput on an RTX PRO 6000 Blackwell (1,725 to 2,597 tok/s at concurrency 32, vLLM). The uncomfortable question is always the same: did the model get dumber. This post is the exact check we ran before recommending the switch, with numbers, so you can run the same one.

Why "run an eval suite" is usually the wrong first answer

Standard benchmarks (MMLU and friends) are noisy instruments for quantization deltas at 8B scale. Score movement inside the error bars tells you nothing about whether YOUR prompts changed behavior. What you actually want to know is narrower: on the workload you serve, does the FP8 checkpoint produce materially different outputs than BF16, and are any of the differences wrong.

That is answerable directly, cheaply, and per prompt.

The method

Both configurations run the same fixed workload: 20 prompts covering reasoning, code, summarization, translation, extraction, classification, math, and instruction following. Greedy decoding, temperature 0, 256-token cap, streamed. Greedy matters: it removes sampling noise, so any output difference is attributable to the numerics.

Then a three-stage comparison:

Byte equality. outputs_bf16[i] == outputs_fp8[i]. Anything identical is settled.
Similarity triage. For non-identical pairs, difflib.SequenceMatcher.ratio() sorts near-identical wording drift from real divergence.
Side-by-side review under a written rubric. Every non-identical pair gets read. The rubric asks one question: is there a factual or numerical claim that one precision gets right and the other gets wrong. Wording changes, reordering, and equally-defensible readings are recorded but not counted as regressions.

The core loop is small:

import difflib, json

bf16 = json.load(open("vllm_bf16_conc1.texts.json"))
fp8  = json.load(open("vllm_fp8_conc1.texts.json"))

for i, (a, b) in enumerate(zip(bf16, fp8)):
    if a == b:
        print(i, "identical")
        continue
    r = difflib.SequenceMatcher(None, a, b).ratio()
    print(i, f"similarity {r:.3f}")
    # non-identical pairs go to side-by-side review

What it found on Qwen3-8B FP8

7 of 20 outputs byte-identical.
9 differed only in wording or formatting; equivalent content on review.
3 minor regressions: a repeated word in a poem, one list item drifting off topic, one questionable tool suggestion.
1 extraction prompt was ambiguous and produced two defensible readings.
Zero cases where FP8 gave a wrong factual or numerical answer that BF16 answered correctly.

That last line is the acceptance bar. Minor stylistic wobble is expected from a numerics change; a flipped fact is a blocker. This profile passed, so the 1.5x was free for this workload.

Two honest caveats. First, this validates a prompt profile, not the model in general: different domains, longer contexts, or sampled decoding need their own pass. Second, greedy-decoding equality is a strict signal but not a complete one; if you serve with sampling, run the review stage on sampled pairs too and expect more (benign) divergence.

The wrinkle that almost blocked the whole thing

Getting FP8 to run at all on workstation-class Blackwell (sm_120) required routing around a kernel assertion in the default FP8 path. If you are on the same silicon and vLLM refuses to load the FP8 checkpoint, that is a known class of problem rather than something wrong with your setup.

The full report this check belongs to, with the raw CSVs, environment manifest, and the one-command reproduction script, is here: https://conatus.jahn.ai/ai-engineering/sample-report

Qwen3-8B on workstation Blackwell: vLLM vs SGLang vs llama.cpp, plus an FP8 pass

Jahn — Tue, 25 Aug 2026 16:56:08 +0000

Benchmarks of the same model on the same GPU across three serving stacks, then an FP8 pass on the winner. All numbers measured on our own hardware last week. Raw CSVs, the environment manifest and a one-command reproduction script exist for every figure; the script was re-run end to end after the report was written and all figures reproduced within 6 percent.

Setup

GPU: RTX PRO 6000 Blackwell, 96 GB, sm_120 (workstation Blackwell, not the datacenter parts)
Model: Qwen3-8B
Stacks: vLLM 0.27.1, SGLang 0.5.9, llama.cpp (CUDA build)
Method: identical prompt set and sampling settings across engines, greedy decoding, output token counts matched across engines before any timing was compared. Concurrency 32 for the aggregate numbers.

Matching token counts before comparing throughput matters more than it sounds. Engines disagree about stop conditions and detokenization, and a 10 percent difference in generated tokens quietly becomes a 10 percent throughput lie.

BF16, concurrency 32

stack	aggregate tok/s	TTFT p50	e2e p99
vLLM 0.27.1	1,725	39 ms	3.4 s
SGLang 0.5.9	1,327	42 ms	5.0 s
llama.cpp (CUDA)	428	316 ms	16.3 s

FP8 on the winner (vLLM, same settings)

precision	single-stream tok/s	batch tok/s	latency p50
BF16	86	1,725	0.74 s
FP8 (official checkpoint)	130	2,597	0.49 s

What the numbers say

Stack choice barely matters for one user. Single-stream decode sits in the 83 to 96 tok/s band on every stack. If your service has one concurrent user, pick whatever is easiest to operate.

Stack choice is a 4x decision under load. At concurrency 32 the spread is 428 to 1,725 tok/s for the same model on the same silicon. Time to first token spreads 4x as well (39 ms vs 316 ms p50), which dominates perceived snappiness for chat workloads.

FP8 was a clean 1.5x. 1,725 to 2,597 tok/s aggregate, 86 to 130 single-stream, p50 latency down a third. We ran a fixed 20-prompt factual check before and after: zero regressions. For an 8B model on a 96 GB card the VRAM saving is irrelevant, the win is throughput.

The sm_120 wrinkle

Workstation Blackwell is not datacenter Blackwell, and default paths tuned for the latter sometimes assert on the former. Getting FP8 to run here required routing around a kernel assertion in the default FP8 path (DeepGEMM asserting on sm_120, resolved by falling back to the CUTLASS path). Related example of the same hardware-class gap: vllm-project/vllm#53748, where the Triton MLA decode kernel overflows the 101,376-byte per-block shared-memory limit that GB10 and workstation Blackwell share, but datacenter parts do not.

If you are serving on RTX PRO 6000, GB10 or consumer Blackwell, budget time for this class of issue. The performance is there, the default configs often are not.

Method notes

Greedy decoding for determinism; throughput measured after warmup
Aggregate numbers are steady-state over the full prompt set, not peak
Each engine got the same max batch and context settings where the knob exists

Full methodology and the case study writeup: conatus.jahn.ai/ai-engineering