Packing K/V into 576 bytes per token, fixing vLLM's hybrid cache planner, and measuring the result on an RTX PRO 6000 Blackwell.
I spent a fair amount of time getting NVFP4 KV storage working in my Qwen3.8-Flash-Next serving stack. The work covered the quantizer, a packed-cache writer, a sparse attention reader, and the vLLM integration needed to make them usable together.
The most useful result from the latest comparison was capacity: at almost the same KV-cache memory budget, the server reported a pool of 439,958 tokens with NVFP4, compared with 189,904 with BF16. That is about 2.32x as many tokens in the planner's pool.
Speed was less straightforward. BF16 was faster in the sequential test; NVFP4 finished the four-request batch faster. The small quality check produced the same aggregate classification score, but two answers changed. Those details matter just as much as the memory result.
This is about quantizing the live attention cache. Both sides of the comparison use the same model checkpoint. I am not comparing newly quantized weights against an original BF16 model.
The interesting integration failure happened before inference even started. My kernels used 576 bytes per token per full-attention layer, while an earlier planning step still accounted for 1,024. In a hybrid model, that disagreement was enough to prevent the server from allocating its cache.
The code and evaluation artifacts are now public: nvfp4-qsa on GitHub.
Why the hybrid architecture matters
The inspected configuration has 48 layers: three linear-attention layers followed by one full-attention layer, repeated twelve times. That gives 36 linear-attention layers and 12 full-attention layers. The latter use QSA sparse selection to choose positions for attention reads.
The main QSA K/V arrays store information for individual tokens. The linear-attention path has recurrent state. This implementation compresses the former; recurrent state remains float32 in the observed launch configuration. Indexer state is another allocation with its own accounting.
That distinction matters before discussing memory savings. A smaller K/V representation does not shrink the model weights, draft model, graph buffers or recurrent state by the same factor.
What changed on the server
On September 25, 2026 (UTC), I ran both cache modes in an isolated container during a maintenance window on the RTX PRO 6000 Blackwell. Production traffic was not routed to the test endpoint. Other services remained active, so this is a controlled comparison on a shared host.
The checkpoint, MTP=3, maximum length of 169,984, twelve-sequence limit, 8,192-token scheduler budget, memory utilization 0.94 and float32 recurrent state were held constant. Compilation and CUDA graphs were enabled. The changed parameter was the cache format. NVFP4 ran first, followed by BF16; the order was not randomized or repeated.
| Cache | Available KV budget | Reported planner token pool |
|---|---|---|
| BF16 | 5.85 GiB | 189,904 |
| NVFP4 | 5.84 GiB | 439,958 |
That is about 2.32 times the reported token pool at nearly the same KV budget. It is smaller than the 3.56x main-K/V format ratio because the hybrid model has other state and overhead. It is an allocator observation, not a sustained full-window concurrency test.
| Cache | Concurrency | Median TTFT | Aggregate E2E output |
|---|---|---|---|
| BF16 | 1 | 165.6 ms | 123.48 tok/s |
| NVFP4 | 1 | 160.8 ms | 88.91 tok/s |
| BF16 | 4 | 1070.9 ms | 91.88 tok/s |
| NVFP4 | 4 | 375.8 ms | 129.02 tok/s |
Each condition contains twelve requests for 128 output tokens and one measured batch. Throughput uses actual token usage and complete batch wall time. Two single-request warmups were performed; the four-way condition was not separately warmed. Unique user-message prefixes prevent identical requests, while shared chat-template prefixes can still be cached.
BF16 was faster for the single-request stream in this run. NVFP4's four-way batch finished faster, but startup delays are visible and there are no independent repeated batches. Picking only the four-way result would overstate the performance evidence. The storage and planner-capacity benefit is the clearest result here.
Did the answers change?
For quality I selected sixteen examples each from pinned AG News, SIB-200 English and SIB-200 Russian test files before evaluation. Both modes received the same questions and label order. Answers were scored by exact stripped label matching.
| Task | BF16 | NVFP4 |
|---|---|---|
| AG News | 13/16 | 13/16 |
| SIB-200 EN | 11/16 | 12/16 |
| SIB-200 RU | 13/16 | 12/16 |
| Synthetic code retrieval | 9/9 | 9/9 |
Both modes scored 37/48 on classification, but their predictions were not identical. NVFP4 corrected one English example and changed the Russian version of the same example from correct to incorrect. Across all tasks, 55 of 57 answers matched. Equal aggregate accuracy is not proof that quality is unchanged.
The nine retrieval questions placed one code at three positions in three text sizes. The maximum measured input was 9,280 tokens. This is a small regression check, not 120K retrieval validation or a full-dataset benchmark. Raw answers, token usage and dataset revisions are included. The chart's Wilson intervals expose the uncertainty from small subsets; translated examples also introduce dependence between the English and Russian observations.
How 2,048 bytes became 576
Each K or V vector has 256 elements. FP4 packs two elements into each byte, giving 128 data bytes. Groups of 16 elements each have one FP8 E4M3 scale, adding another 16 bytes. With two KV heads and both K and V, the total is:
(256 / 2 + 256 / 16) x 2 x 2 = 576 bytes/token/layer
| Main K/V representation | Bytes per token per layer |
|---|---|
| BF16 | 2,048 |
| FP8 | 1,024 |
| NVFP4, including group scales | 576 |
At an equal main-K/V byte budget, that means 1.78 times as many tokens as FP8, or 3.56 times as many as BF16. Equivalently, the main-K/V allocation is 43.75% smaller than FP8. These are format calculations, not measured gains for the entire server.
The FP4 values use E2M1 with magnitudes 0, 0.5, 1, 1.5, 2, 3, 4, 6 and a sign bit. Element 2j goes in the low nibble; element 2j+1 goes in the high nibble. Decoding applies fp4 * group_scale / global_scale.
The global scale must remain stable while older cache entries are live. Changing it when a new token arrives would change the interpretation of existing bytes unless the old scale were retained. The integrated cache initializes both K and V global scales to 1.0; each token carries its own group scales. Some standalone numerical tests instead choose a global scale from the tensor's absolute maximum. Their results cannot be treated as direct task-quality validation of the fixed-scale serving path.
Writing and reading the packed cache
The Triton writer takes BF16 values and a token-to-physical-slot mapping. It computes group maxima, converts scales to FP8, rounds values to E2M1 and packs pairs. Invalid slots do not write.
The reader loads the selected QSA positions and their group scales and reconstructs values inside the attention kernel. It does not build a full-context BF16 staging cache.
There is an important ownership contract: valid destination slots must be unique within a write call. Duplicate slots can cause conflicting writes. Standalone checks cover duplicate rejection and overwriting a physical slot with new data. The integrated scheduler must also satisfy this contract under append, prefix reuse, cancellation and preemption.
The bug between planning and allocation
The hybrid allocator needed attention pages compatible with a recurrent-state page of 3,207,168 bytes. At an early planning step, the attention specification implied 1,024 bytes per token. A 3,136-token attention block was selected.
Later, the physical allocation used the packed 576-byte representation:
3,136 x 576 = 1,806,336 bytes
That page was too small. The format could be correct and the read kernel could pass its tests while the server still could not allocate a valid hybrid cache.
The correction belongs at the point where the backend tells the planner what it will actually allocate. The QSA backend's customize_spec() now publishes packed content bytes before platform block-size adjustment:
packed_region = spec.head_size // 2 + spec.head_size // 16
return replace(spec, state_content_bytes=2 * packed_region)
This is 288 bytes per KV head. The specification accounts for the two heads, producing 576 bytes per token. The matching page becomes:
3,207,168 / 576 = 5,568 tokens
The final cache uses physical uint8 K/V regions of 144 bytes and quantization mode NONE. QSA owns the packing and unpacking; the finalized physical width must not be transformed again. The 5,568-token page is a consequence of this configuration, not a constant to copy into another model.
Checking the implementation separately from model quality
The earlier numerical checks below are from August 28, 2026 (UTC). I used an independent codec alongside the Python reference and CUDA quantizer. A separate comparison checks fused reading against dense attention over the same decoded NVFP4 values.
| Historical check | Result |
|---|---|
| Independent decode vs Python reference | cosine 0.99999726 |
| Independent codec vs CUDA packed bytes | 0.999634 match |
| Fused vs dense NVFP4 read | cosine 0.99999702 |
| Fused NVFP4 vs BF16 oracle | cosine approximately 0.99079 |
| Read boundary matrix | 27/27 |
| Write boundary matrix | 28/28 |
The byte comparisons are not universally exact. Against the Python reference, 123 of 567 differing nibbles were exact midpoint ties. The remaining differences involved values near rounding boundaries and scale arithmetic. Calling all differences “just tie-breaking” would hide part of the observation.
The fused-versus-dense comparison checks arithmetic and layout on the quantized representation. The comparison against BF16 includes quantization error. Neither number measures downstream task accuracy.
The archived Compute Sanitizer runs report zero errors in four modes on the tested small shapes. Larger functional boundary cases are a different gate. This does not prove every long-context serving path or concurrent execution pattern safe.
What the long-context run actually tells me
Before the paired comparison, I had also exercised the integrated server with a synthetic 120,064-token input. In that August run, a single request reached its first text chunk in 14.569 seconds; a group of seven took 101.953 seconds to finish. Each request generated only one output token.
That established that the tested path could process a long input. The prompt repeated a phrase, so it said little about whether the model could retrieve useful information from that context. The September retrieval check is more meaningful for correctness, but only reaches 9,280 input tokens.
The August run also used different settings, including MTP=2 rather than 3. I keep its raw observations in the package as a separate record; they are not a before-and-after speed comparison with the September results.
Integration problems that remain
Two earlier BF16 startup attempts with MTP disabled failed with a CUDA illegal memory access, including in eager mode. A standalone GDN warmup with the same head dimensions passed on Blackwell and did not locate the earlier asynchronous fault. The successful comparison above uses MTP=3. FP8 was not measured in this window.
The serving stack also needed invalid/padded expert-ID handling, workspace reuse between compatible sequential MoE layers, and separation of the target's NVFP4 B12x backend from its BF16 draft backend. Embedding, output-head and index-buffer references required their own fixes.
Those changes are packaged separately. Sharing scratch buffers is appropriate only when their uses do not overlap or are explicitly synchronized. CPU tests verify route masking and shared-object identity; they do not establish CUDA stream or graph safety for arbitrary overlapping wrappers.
Testing the cache without loading the model
I separated the writer, reader and independent codec into a Python package that does not import vLLM. That makes the format and page operations testable without loading the full model.
from nvfp4_qsa import PackedCache
cache = PackedCache(blocks=4, page_size=16, device="cuda")
The checked API rejects duplicate valid write slots, non-finite inputs and incompatible dimensions. Its validation synchronizes with the host, so this wrapper is not presented as a production fast path. It explicitly rejects CUDA graph capture. Raw-kernel callers still own stream ordering and cache lifetime.
The separately installed wheel passed 16 tests on an RTX 4050 Laptop: nine CPU tests and seven GPU cases. GPU coverage includes ordinary, zero, small and saturating values; slot reuse; invalid metadata; and sparse attention over fragmented pages compared with an independent CPU calculation using decoded cache values.
What can be reproduced today
The complete server measurements came from my model-specific vLLM build. Its historical base revision has not been verified against a public upstream commit, which limits how directly someone else can reproduce that full stack.
I have also prepared a narrow port against public vLLM revision e126687a9a828d513c01a07cd69f025f27d63280, where the model implementation uses the module name qwen4_exp. The patch applies cleanly and its BF16/NVFP4 physical-spec checks pass. I have not validated a full server run of that public port. The server benchmark numbers in this article belong to the tested model-specific build.
The GitHub repository contains the installable portable source, tests, the integration patch, raw request results, dataset revisions and scripts for reconstructing the evaluation prompts. It contains no model weights. Large checkpoint tensors were not rehashed during the maintenance window; the provenance record distinguishes download metadata from verified file hashes.
The next serving experiments need repeated, independently warmed batches, a matched FP8 baseline, and a larger quality evaluation. The failed MTP-disabled startup path also needs investigation. Those are separate pieces of work from validating the cache format.
For now, the strongest result is the extra room reported by the allocator, backed by working reads and writes and a small paired evaluation. The part I would carry into another implementation is the physical-byte accounting: the quantizer, reader and cache planner all need to agree on what a page actually contains.
Top comments (0)