DEV Community: Yuri Pocepaev

How I Took GLiClass FP8 from 59 ms to 16 ms on an RTX 4050

Yuri Pocepaev — Sun, 27 Sep 2026 14:40:04 +0000

Multilingual zero-shot classification on a 6 GB laptop GPU, with native FP8, Triton, CUDA Graphs, and reproducible measurements.

I quantized Knowledgator’s GLiClass Multilang Ultra into a custom FP8 W8A8 checkpoint. Then I wanted to find out how much of that compression could translate into faster local inference.

The weights already took up a third less disk space. The first native FP8 implementation, however, needed about 59 ms per request. The original BF16 model took roughly 24 ms. Reducing precision had made the artifact smaller, but the execution path still needed work.

After fusing auxiliary operations with Triton and adding CUDA Graphs, median request latency reached 16.10 ms. BF16 with the same graph wrapper measured 23.79 ms. On this particular workload, the optimized FP8 path was 1.48× faster, with roughly a third less memory allocated to tensors.

Here is what the model does, what I changed, and what these measurements actually establish. The weights, implementation, and benchmark report are on Hugging Face.

What GLiClass does

GLiClass Multilang Ultra, developed by Knowledgator, classifies text against labels supplied at inference time. You can provide categories such as “delivery,” “refund,” “payment,” and “other” without training a new classification head for that particular list.

For example:

text = "The courier has postponed my delivery for the second day."
labels = ["delivery", "payment", "refund", "other"]

The useful output is a class and its score. This makes classification a practical component before retrieval, application logic, or a more expensive model call.

Potential applications include support-ticket routing, document categorization, intent detection, and filtering text collections. Those are examples of where this interface can help; this experiment did not validate every one of those tasks.

I used the multilingual Ultra model. My quality evaluation covers English and Russian topic classification. It does not establish performance across every language supported by the upstream model or readiness for a particular production workflow.

What is inside the FP8 checkpoint?

I started from the original GLiClass model and built the FP8 W8A8 quantization used here. A separate phase then audited the tensors, measured quality, and optimized its inference path with Triton and CUDA Graphs. Knowledgator remains the developer of the original model; the quantization and local inference work are my derivative contribution.

The checkpoint quantizes 168 matrices across 24 mT5 encoder blocks: the attention projections q, k, v, and o, plus the feed-forward projections wi_0, wi_1, and wo.

Weights use FP8 E4M3 with a separate scale for each output channel. Activations are quantized dynamically for each token. Embeddings, normalization layers, and classification components remain in BF16.

W8A8 therefore describes the selected linear projections. It does not mean every tensor in the model uses eight bits.

Weight file	Size
Original BF16	3,416,522,340 bytes
FP8 W8A8	2,259,902,516 bytes

That is a 33.85% reduction, or 1.51× compression. The shared embedding matrix alone accounts for approximately 1.02 GB of the remaining size.

I also checked the unquantized tensors against a pinned snapshot of the original model. All 80 original tensors expected to remain unchanged matched byte for byte. The exact upstream revision used during the historical quantization was not recorded, so the report states that explicitly. The BF16 snapshot used for the comparison is pinned.

Why FP8 was initially slower

The storage format does not tell you which operations the GPU will execute.

One loading configuration reconstructed the weights in BF16 while retaining activation-quantization emulation. That path took around 264 ms per request. It was useful as a diagnostic, but it was not a native FP8 performance measurement.

The next adapter used actual FP8 matrix multiplication through torch._scaled_mm and cuBLAS. A separate profiler pass recorded 168 FP8 GEMM kernels for sm89, one for each replaced projection. Yet the complete request still took 59.23 ms.

The trace showed an obvious place to investigate: 3,269 GPU kernel executions for one request. Around the matrix multiplications, separate operations computed maxima, calculated scales, converted types, padded rows, and processed outputs.

At batch size 1, the cost of running that sequence matters. Matrix multiplication is only part of request latency. The profile suggested reducing auxiliary operations and launch overhead; the subsequent measurements tested those changes at the request level.

The changes that made a difference

Fuse activation quantization

For each token, the adapter needs an absolute maximum, a scale, and an FP8 representation of the activations. The original implementation expressed these as separate PyTorch operations.

I moved them into one Triton kernel. It performs the reduction, calculates the scale, converts values to E4M3, and fills the additional rows needed for matrix alignment.

Fuse the output scaling

The native FP8 GEMM produces an FP32 result. The adapter then applies the activation scale for each token and the weight scale for each output channel before converting back to BF16.

That sequence became a second Triton kernel. The stored FP8 weights are used directly, without another round of weight quantization. GEMM continues to use use_fast_accum=False.

Together, these changes reduced median latency to 37.97 ms.

Replay the encoder with CUDA Graphs

Next, I added CUDA Graphs around the encoder. A graph replays a captured sequence of GPU work, reducing the overhead of issuing its operations separately from Python.

Text lengths vary, so the adapter uses four buckets: 64, 128, 192, and 256 tokens. Inputs are padded to the appropriate bucket, and the extra positions are masked. Before classification, the encoder output is sliced back to the original token count.

Every request copies fresh tokens and a fresh mask into the graph's input buffers. Predictions are not cached. The output is cloned before it is returned to the caller.

One compatibility issue needed attention: in the tested Transformers version, attention-mask construction copied a scalar from CPU to GPU during graph capture. Preparing the mask outside the captured region resolved it.

With these changes, median latency reached 16.10 ms. The profiled request contained 1,425 GPU kernel executions, down from 3,269, while retaining all 168 native FP8 GEMMs.

How I measured it

The machine was an NVIDIA RTX 4050 Laptop with 6 GB VRAM, running Windows. The environment used Python 3.12.9, PyTorch 2.11.0 with CUDA 12.8, GLiClass 0.1.20, Transformers 5.14.1, and Triton Windows 3.6.0.post26.

Latency was measured on one fixed AG News example with four candidate labels, at batch size 1. After warmup and quality evaluation, I ran 50 additional requests. Each request was synchronized with CUDA before and after timing. The measurement includes tokenization and postprocessing.

Model loading, Triton compilation, and graph preparation are excluded from these steady-state timings. All four graph buckets were prepared in advance. FP8 graph preparation took 2.18 seconds in this run; that is not the complete cold-start time.

Runtime	Median	p95	Peak allocated tensor memory
BF16, eager	24.18 ms	29.67 ms	3.219 GiB
BF16 + CUDA Graphs	23.79 ms	24.45 ms	3.238 GiB
FP8, initial native adapter	59.23 ms	66.36 ms	2.145 GiB
FP8 + Triton	37.97 ms	46.62 ms	2.144 GiB
FP8 + Triton + CUDA Graphs	16.10 ms	16.44 ms	2.166 GiB

For the main comparison, I applied the same CUDA Graph wrapper to the original BF16 model. Optimized FP8 was 1.48× faster than that control. The 3.68× improvement compares the optimized implementation with my initial native FP8 adapter; it describes progress in the adapter implementation.

The memory column measures allocated PyTorch tensors. It is different from total device memory shown by nvidia-smi, which also reflects allocator reservations and other overhead. The current loader temporarily reconstructs weights in BF16 before replacing the projections, so 2.17 GiB is not a loading-memory requirement.

Laptop clocks and thermals were not locked. These are measurements of this configuration, not performance confidence intervals across independent sessions. I did not measure sustained throughput at larger batch sizes.

What happened to quality?

The paired evaluation used 664 identical examples: a seeded subset of 256 AG News test examples and the full English and Russian SIB-200 test splits, with 204 examples each. SIB-200 candidate labels were in English for both languages.

Dataset revisions, example indices, and hashes are recorded in the repository. The original and quantized tokenizers produced identical token IDs. No example was truncated at the 256-token limit.

Dataset	BF16 macro-F1	Optimized FP8 macro-F1	Difference
AG News	79.08%	79.49%	+0.41 pp
SIB-200 English	84.57%	84.04%	−0.53 pp
SIB-200 Russian	84.09%	83.42%	−0.67 pp

Optimized FP8 changed 5 of 664 top-1 predictions relative to BF16, for 99.25% agreement. Agreement measures whether the two variants choose the same label; it is not classification accuracy. BF16 with and without the graph wrapper produced identical top-1 predictions on all 664 examples.

I would not interpret the small positive AG News difference as an improvement in model quality. The other two datasets show declines, and this evaluation is too narrow to establish general quality preservation. A support-ticket router, for example, would need its own evaluation on representative tickets and candidate labels.

The single-label pipeline retained only the winning label and its probability. Full score distributions were not saved, so this report makes no claims about changes across all logits or probability calibration.

Try it locally

The Hugging Face repository contains the weights, tokenizer, dependencies, adapter, and an inference.py entry point.

After downloading the repository and installing the environment described in the model card, run:

python inference.py --text "NASA launched a spacecraft to study Mars." --labels space politics sports business

Or use the Python API:

import torch
from inference import load_pipeline

classifier = load_pipeline()

with torch.inference_mode():
    result = classifier(
        "NASA launched a spacecraft to study Mars.",
        ["space", "politics", "sports", "business"],
        batch_size=1,
        threshold=0.0,
    )

print(result)

The tested adapter supports sequential inference, batch size 1, and at most 256 input tokens, including labels and prompt formatting. The tokenizer truncates longer inputs. Concurrent requests to one instance require synchronization or separate runners.

This is currently a specialized inference path for the measured workload. Other GPUs, Linux execution, and installation in a fresh environment were not independently validated for this release. The measured configuration is Windows on an RTX 4050 Laptop.

What this experiment taught me

A quantized checkpoint has at least three separate properties: its size on disk, its runtime memory use, and the speed of a particular implementation. Here, the smaller FP8 file initially produced a slower request. The latency improvement came from optimizing the operations around GEMM and the way the encoder's GPU work was launched.

The result is a reproducible starting point: 16.10 ms on the selected batch-1 request, approximately 2.17 GiB of allocated tensor memory, and measured quality changes. Larger batches, longer inputs, and real application workloads are the next useful experiments. Their results need to be measured separately.

The main takeaway is simple: quantization gives you a smaller representation; making it faster is a separate engineering problem.

The model, source code, and report are available under Apache-2.0. Knowledgator developed the original model; this release provides an independent quantized representation, inference implementation, and evaluation.

About the author

I’m Yuri Pocepaev, a software engineer and lead developer at Neuroprem. This article is part of my public engineering log.

Building an NVFP4 KV Cache for a Hybrid Qwen Model

Yuri Pocepaev — Sun, 27 Sep 2026 14:33:27 +0000

Packing K/V into 576 bytes per token, fixing vLLM's hybrid cache planner, and measuring the result on an RTX PRO 6000 Blackwell.

I spent a fair amount of time getting NVFP4 KV storage working in my Qwen3.8-Flash-Next serving stack. The work covered the quantizer, a packed-cache writer, a sparse attention reader, and the vLLM integration needed to make them usable together.

The most useful result from the latest comparison was capacity: at almost the same KV-cache memory budget, the server reported a pool of 439,958 tokens with NVFP4, compared with 189,904 with BF16. That is about 2.32x as many tokens in the planner's pool.

Speed was less straightforward. BF16 was faster in the sequential test; NVFP4 finished the four-request batch faster. The small quality check produced the same aggregate classification score, but two answers changed. Those details matter just as much as the memory result.

This is about quantizing the live attention cache. Both sides of the comparison use the same model checkpoint. I am not comparing newly quantized weights against an original BF16 model.

The interesting integration failure happened before inference even started. My kernels used 576 bytes per token per full-attention layer, while an earlier planning step still accounted for 1,024. In a hybrid model, that disagreement was enough to prevent the server from allocating its cache.

The code and evaluation artifacts are now public: nvfp4-qsa on GitHub.

Why the hybrid architecture matters

The inspected configuration has 48 layers: three linear-attention layers followed by one full-attention layer, repeated twelve times. That gives 36 linear-attention layers and 12 full-attention layers. The latter use QSA sparse selection to choose positions for attention reads.

The main QSA K/V arrays store information for individual tokens. The linear-attention path has recurrent state. This implementation compresses the former; recurrent state remains float32 in the observed launch configuration. Indexer state is another allocation with its own accounting.

That distinction matters before discussing memory savings. A smaller K/V representation does not shrink the model weights, draft model, graph buffers or recurrent state by the same factor.

What changed on the server

On September 25, 2026 (UTC), I ran both cache modes in an isolated container during a maintenance window on the RTX PRO 6000 Blackwell. Production traffic was not routed to the test endpoint. Other services remained active, so this is a controlled comparison on a shared host.

The checkpoint, MTP=3, maximum length of 169,984, twelve-sequence limit, 8,192-token scheduler budget, memory utilization 0.94 and float32 recurrent state were held constant. Compilation and CUDA graphs were enabled. The changed parameter was the cache format. NVFP4 ran first, followed by BF16; the order was not randomized or repeated.

Cache	Available KV budget	Reported planner token pool
BF16	5.85 GiB	189,904
NVFP4	5.84 GiB	439,958

That is about 2.32 times the reported token pool at nearly the same KV budget. It is smaller than the 3.56x main-K/V format ratio because the hybrid model has other state and overhead. It is an allocator observation, not a sustained full-window concurrency test.

Cache	Concurrency	Median TTFT	Aggregate E2E output
BF16	1	165.6 ms	123.48 tok/s
NVFP4	1	160.8 ms	88.91 tok/s
BF16	4	1070.9 ms	91.88 tok/s
NVFP4	4	375.8 ms	129.02 tok/s

Each condition contains twelve requests for 128 output tokens and one measured batch. Throughput uses actual token usage and complete batch wall time. Two single-request warmups were performed; the four-way condition was not separately warmed. Unique user-message prefixes prevent identical requests, while shared chat-template prefixes can still be cached.

BF16 was faster for the single-request stream in this run. NVFP4's four-way batch finished faster, but startup delays are visible and there are no independent repeated batches. Picking only the four-way result would overstate the performance evidence. The storage and planner-capacity benefit is the clearest result here.

Did the answers change?

For quality I selected sixteen examples each from pinned AG News, SIB-200 English and SIB-200 Russian test files before evaluation. Both modes received the same questions and label order. Answers were scored by exact stripped label matching.

Task	BF16	NVFP4
AG News	13/16	13/16
SIB-200 EN	11/16	12/16
SIB-200 RU	13/16	12/16
Synthetic code retrieval	9/9	9/9

Both modes scored 37/48 on classification, but their predictions were not identical. NVFP4 corrected one English example and changed the Russian version of the same example from correct to incorrect. Across all tasks, 55 of 57 answers matched. Equal aggregate accuracy is not proof that quality is unchanged.

The nine retrieval questions placed one code at three positions in three text sizes. The maximum measured input was 9,280 tokens. This is a small regression check, not 120K retrieval validation or a full-dataset benchmark. Raw answers, token usage and dataset revisions are included. The chart's Wilson intervals expose the uncertainty from small subsets; translated examples also introduce dependence between the English and Russian observations.

How 2,048 bytes became 576

Each K or V vector has 256 elements. FP4 packs two elements into each byte, giving 128 data bytes. Groups of 16 elements each have one FP8 E4M3 scale, adding another 16 bytes. With two KV heads and both K and V, the total is:

(256 / 2 + 256 / 16) x 2 x 2 = 576 bytes/token/layer

Main K/V representation	Bytes per token per layer
BF16	2,048
FP8	1,024
NVFP4, including group scales	576

At an equal main-K/V byte budget, that means 1.78 times as many tokens as FP8, or 3.56 times as many as BF16. Equivalently, the main-K/V allocation is 43.75% smaller than FP8. These are format calculations, not measured gains for the entire server.

The FP4 values use E2M1 with magnitudes 0, 0.5, 1, 1.5, 2, 3, 4, 6 and a sign bit. Element 2j goes in the low nibble; element 2j+1 goes in the high nibble. Decoding applies fp4 * group_scale / global_scale.

The global scale must remain stable while older cache entries are live. Changing it when a new token arrives would change the interpretation of existing bytes unless the old scale were retained. The integrated cache initializes both K and V global scales to 1.0; each token carries its own group scales. Some standalone numerical tests instead choose a global scale from the tensor's absolute maximum. Their results cannot be treated as direct task-quality validation of the fixed-scale serving path.

Writing and reading the packed cache

The Triton writer takes BF16 values and a token-to-physical-slot mapping. It computes group maxima, converts scales to FP8, rounds values to E2M1 and packs pairs. Invalid slots do not write.

The reader loads the selected QSA positions and their group scales and reconstructs values inside the attention kernel. It does not build a full-context BF16 staging cache.

There is an important ownership contract: valid destination slots must be unique within a write call. Duplicate slots can cause conflicting writes. Standalone checks cover duplicate rejection and overwriting a physical slot with new data. The integrated scheduler must also satisfy this contract under append, prefix reuse, cancellation and preemption.

The bug between planning and allocation

The hybrid allocator needed attention pages compatible with a recurrent-state page of 3,207,168 bytes. At an early planning step, the attention specification implied 1,024 bytes per token. A 3,136-token attention block was selected.

Later, the physical allocation used the packed 576-byte representation:

3,136 x 576 = 1,806,336 bytes

That page was too small. The format could be correct and the read kernel could pass its tests while the server still could not allocate a valid hybrid cache.

The correction belongs at the point where the backend tells the planner what it will actually allocate. The QSA backend's customize_spec() now publishes packed content bytes before platform block-size adjustment:

packed_region = spec.head_size // 2 + spec.head_size // 16
return replace(spec, state_content_bytes=2 * packed_region)

This is 288 bytes per KV head. The specification accounts for the two heads, producing 576 bytes per token. The matching page becomes:

3,207,168 / 576 = 5,568 tokens

The final cache uses physical uint8 K/V regions of 144 bytes and quantization mode NONE. QSA owns the packing and unpacking; the finalized physical width must not be transformed again. The 5,568-token page is a consequence of this configuration, not a constant to copy into another model.

Checking the implementation separately from model quality

The earlier numerical checks below are from August 28, 2026 (UTC). I used an independent codec alongside the Python reference and CUDA quantizer. A separate comparison checks fused reading against dense attention over the same decoded NVFP4 values.

Historical check	Result
Independent decode vs Python reference	cosine 0.99999726
Independent codec vs CUDA packed bytes	0.999634 match
Fused vs dense NVFP4 read	cosine 0.99999702
Fused NVFP4 vs BF16 oracle	cosine approximately 0.99079
Read boundary matrix	27/27
Write boundary matrix	28/28

The byte comparisons are not universally exact. Against the Python reference, 123 of 567 differing nibbles were exact midpoint ties. The remaining differences involved values near rounding boundaries and scale arithmetic. Calling all differences “just tie-breaking” would hide part of the observation.

The fused-versus-dense comparison checks arithmetic and layout on the quantized representation. The comparison against BF16 includes quantization error. Neither number measures downstream task accuracy.

The archived Compute Sanitizer runs report zero errors in four modes on the tested small shapes. Larger functional boundary cases are a different gate. This does not prove every long-context serving path or concurrent execution pattern safe.

What the long-context run actually tells me

Before the paired comparison, I had also exercised the integrated server with a synthetic 120,064-token input. In that August run, a single request reached its first text chunk in 14.569 seconds; a group of seven took 101.953 seconds to finish. Each request generated only one output token.

That established that the tested path could process a long input. The prompt repeated a phrase, so it said little about whether the model could retrieve useful information from that context. The September retrieval check is more meaningful for correctness, but only reaches 9,280 input tokens.

The August run also used different settings, including MTP=2 rather than 3. I keep its raw observations in the package as a separate record; they are not a before-and-after speed comparison with the September results.

Integration problems that remain

Two earlier BF16 startup attempts with MTP disabled failed with a CUDA illegal memory access, including in eager mode. A standalone GDN warmup with the same head dimensions passed on Blackwell and did not locate the earlier asynchronous fault. The successful comparison above uses MTP=3. FP8 was not measured in this window.

The serving stack also needed invalid/padded expert-ID handling, workspace reuse between compatible sequential MoE layers, and separation of the target's NVFP4 B12x backend from its BF16 draft backend. Embedding, output-head and index-buffer references required their own fixes.

Those changes are packaged separately. Sharing scratch buffers is appropriate only when their uses do not overlap or are explicitly synchronized. CPU tests verify route masking and shared-object identity; they do not establish CUDA stream or graph safety for arbitrary overlapping wrappers.

Testing the cache without loading the model

I separated the writer, reader and independent codec into a Python package that does not import vLLM. That makes the format and page operations testable without loading the full model.

from nvfp4_qsa import PackedCache

cache = PackedCache(blocks=4, page_size=16, device="cuda")

The checked API rejects duplicate valid write slots, non-finite inputs and incompatible dimensions. Its validation synchronizes with the host, so this wrapper is not presented as a production fast path. It explicitly rejects CUDA graph capture. Raw-kernel callers still own stream ordering and cache lifetime.

The separately installed wheel passed 16 tests on an RTX 4050 Laptop: nine CPU tests and seven GPU cases. GPU coverage includes ordinary, zero, small and saturating values; slot reuse; invalid metadata; and sparse attention over fragmented pages compared with an independent CPU calculation using decoded cache values.

What can be reproduced today

The complete server measurements came from my model-specific vLLM build. Its historical base revision has not been verified against a public upstream commit, which limits how directly someone else can reproduce that full stack.

I have also prepared a narrow port against public vLLM revision e126687a9a828d513c01a07cd69f025f27d63280, where the model implementation uses the module name qwen4_exp. The patch applies cleanly and its BF16/NVFP4 physical-spec checks pass. I have not validated a full server run of that public port. The server benchmark numbers in this article belong to the tested model-specific build.

The GitHub repository contains the installable portable source, tests, the integration patch, raw request results, dataset revisions and scripts for reconstructing the evaluation prompts. It contains no model weights. Large checkpoint tensors were not rehashed during the maintenance window; the provenance record distinguishes download metadata from verified file hashes.

The next serving experiments need repeated, independently warmed batches, a matched FP8 baseline, and a larger quality evaluation. The failed MTP-disabled startup path also needs investigation. Those are separate pieces of work from validating the cache format.

For now, the strongest result is the extra room reported by the allocator, backed by working reads and writes and a small paired evaluation. The part I would carry into another implementation is the physical-byte accounting: the quantizer, reader and cache planner all need to agree on what a page actually contains.

Code and reproduction

Running Qwen Flash-Next NVFP4 in vLLM: PLE Loading, B12x Fixes, and Stable Inference

Yuri Pocepaev — Sun, 27 Sep 2026 14:32:29 +0000

Companion code: PLE and B12x compatibility patches, tests, and deployment notes. The repository contains extracted fixes and validation notes, not a complete runtime image.

The model loaded. The health endpoint returned a successful response. A few requests completed. Then concurrent traffic arrived, and the inference process ran out of GPU memory.

That was the most instructive part of bringing Qwen Flash-Next NVFP4 online. There was no single switch that made the deployment work. We had to get a large checkpoint onto disk, adapt a loader to its tensor layout, and find a memory configuration that survived actual generation. Each milestone removed one obstacle while exposing the next.

The eventual result was a server configured for 131,072 tokens of context, with a limit of 16 sequences, that completed three rounds of 16 concurrent requests without a recorded container restart. Getting there required giving up some of our initial context and concurrency targets—and being precise about what the successful test actually demonstrated.

Before getting into the details, a big shout-out to windowsxp811203. The quality of the work and the detailed reports deserve real credit. It is great to see someone put that much care into both the result and its documentation.

This account follows the initial bfloat16 KV-cache baseline and the additional B12x integration work needed by the developed serving stack. The measured 48-request test belongs to the initial baseline. The later B12x work is described separately, without transferring that test result to a different runtime. The loader and backend patches were required independently of KV-cache quantization.

The first practical problem was simply getting the checkpoint onto the machine. The recorded download comprised 31 files, approximately 186.4 GB in total. A single PLE shard accounted for roughly 102.4 GB. With artifacts of that size, storage preparation and download recovery were part of the deployment work rather than incidental setup.

We provisioned a dedicated volume with an ext4 filesystem and used the Hugging Face CLI with eight download workers. The transfer ran through a local proxy. At one point, progress stalled at around 14 GB. After restarting, the reported progress fell to approximately 4.7 GB before the transfer resumed at about 93 MiB/s.

The practical consequence was extra transfer time and uncertainty about completion. A progress counter was useful for watching the download, but it was not sufficient evidence that the checkpoint was ready. We waited for the final shard to finish before moving on to loading.

It is also important to keep disk size separate from GPU residency. The total checkpoint size was not a prediction of the amount of VRAM the process would consume. This deployment used PLE CPU offload, so some of the storage and loading work belonged on the host side. Later failures still depended on the memory available on the GPU during generation.

Before replacing the existing model server, we saved its container configuration and retained a rollback target. That gave the experiments a defined starting point and preserved a route back if the new model failed. It also prevented the previous deployment from becoming another variable while we were changing the new one.

The initial runtime was a model-specific vLLM container image. It provided a starting point, but it did not accept this checkpoint without local changes. Two incompatibilities surfaced before inference tuning could begin.

The first was in the attention configuration. Twelve layer entries used qwen_sparse_attention, while the runtime path we were using expected full_attention. We supplied a local configuration override that changed those entries.

That was an operational compatibility change, and it has an important limit: getting past configuration parsing does not establish that the two settings are computationally equivalent. This work did not include a controlled comparison of output quality or attention behavior before and after the override. The evidence supports the narrower claim that the override allowed this runtime to proceed with loading.

The second incompatibility involved the layout of the PLE embedding table. The loader expected checkpoint shards with numeric suffixes. The checkpoint instead exposed the complete table through a tensor name whose shard index was empty. There was also a naming mismatch: the loader path had to recognize both the longer prefixed form and the shorter form actually encountered during loading.

The distinction sounds small until the tensor behind the name is enormous. A loader cannot safely infer that an unrecognized name represents an interchangeable piece of the model. It needs to identify the layout and validate the shape before copying the data.

Our fix went through three iterations. The first recognized the empty suffix and tried to hand the tensor to the ordinary loading path. That was insufficient. The second added explicit handling for the complete table, including shape validation, but recognized only the longer prefix. The third also handled the shorter name used by the checkpoint. That was the version that loaded successfully.

The successful path checked the incoming tensor against the expected full embedding dimensions. It then used the embedding-copy helper with a checkpoint offset of zero and the relevant embedding partition boundaries. Numbered shards retained their existing handling, and other weights continued through the regular loader.

The shape check was an essential part of the change. Supporting an additional serialization layout should not mean accepting arbitrary tensor dimensions just because the name looks familiar. A mismatch should fail at loading, where the error can be explained, rather than being allowed to become an inference problem later.

We packaged the patch as a small layer over the original container image, replacing a single loader module. Keeping that change narrow made it easier to understand what differed between the starting runtime and the working one. The rest of the deployment could then be investigated without treating the entire runtime as a custom fork.

The PLE loader was only one part of the runtime work. The serving stack subsequently required a separate set of fixes at the vLLM–FlashInfer boundary to use the B12x MoE path on our Blackwell SM120 GPU. This is a separate stage from the initial concurrency test reported below; its patches should not be retroactively attributed to that earlier run.

That distinction changes how the deployment should be described. A single loader replacement explains the initial checkpoint-loading fix. The more developed serving image also carries backend integration, speculative-decoding, offload, and scheduling changes. Reproducing that stack requires preserving the complete working runtime, rather than assuming that a stock image plus the PLE patch is equivalent.

The B12x work addressed two very different problems: invalid expert routes and duplicated workspace allocations. We changed the integration around the backend. We did not rewrite the B12x CUDA kernels.

During dummy profiling and padding, the integration could pass an expert ID of -1. That value represented an inactive route on the caller's side, but it could reach a path that treated it as an expert index. Our deployment notes associate this failure with cudaErrorIllegalAddress, GPU-process failures, and Xid 31 reports.

The wrapper now validates each selected expert ID before calling the original backend. A route is valid only when its ID is nonnegative and below the expert count. An invalid route is redirected to expert zero and its routing weight is set to zero. Valid routes retain both their IDs and weights.

The essential operation is small:

valid = (expert_ids >= 0) & (expert_ids < num_experts)
safe_ids = where(valid, expert_ids, 0)
safe_weights = where(valid, routing_weights, 0)

Redirecting an invalid ID alone would be wrong: it would turn padding into a real contribution from expert zero. Clearing the routing weight is what preserves the intended zero contribution of an inactive route. The wrapper also catches IDs above the valid range, although that defensive handling is not a substitute for diagnosing an upstream routing bug if one appears outside padding or profiling.

The saved unit test makes the transformation explicit. Given IDs [-1, 7, 512] for a model with 512 experts, the wrapper passes [0, 7, 0]. Given routing weights [0.4, 0.5, 0.6], it passes [0.0, 0.5, 0.0]. That verifies the wrapper's treatment of invalid routes; it is not, by itself, a numerical validation of the complete GPU inference path.

The second problem was workspace ownership. Our deployment notes describe allocations of roughly 400 MiB per MoE layer across 48 layers. At that scale, allocating an independent workspace for every layer can consume approximately 18.75 GiB before considering the rest of the runtime. This is an estimate from the reported per-layer allocation, not a separately measured memory saving.

Those layers execute sequentially in the execution pattern for which the patch was built. Compatible wrappers can therefore reuse scratch storage instead of reserving a separate copy for every layer.

The implementation caches three buffers: the static workspace, the dynamic workspace, and the MoE output buffer. The cache key includes expert count, top-k, hidden and intermediate dimensions, maximum token count, local expert count, output dtype, device, activation, quantization mode, and source format. Wrappers with matching keys reuse the same buffers; a different key triggers a separate allocation.

Compatibility and execution order both matter. Matching dimensions alone would not make shared writable storage safe for overlapping executions. In this deployment, reuse relies on the sequential layer execution pattern. A change to scheduling or stream behavior would require checking that assumption again.

The saved workspace test verifies that two identically configured wrappers allocate once and share the same buffer objects. That is useful evidence about the Python integration. It does not replace a runtime test under CUDA graphs and real traffic.

This design is also reflected in the current FlashInfer B12x API documentation, which exposes optional shared static, dynamic, and output buffers for compatible wrappers. That describes the current upstream interface; it does not establish that the runtime version used in our deployment already provided an equivalent integration.

Why pursue B12x at all? Our deployment notes identify Marlin as the fallback and B12x as the preferred native execution path for the NVFP4 MoE workload on this GPU. This article does not include a controlled comparison between the two, so we do not attach a speedup number or a claim of identical output quality to that choice. Selecting a backend does not, by itself, requantize the checkpoint, but numerical equivalence still needs measurement.

The surrounding runtime also needed coordinated choices for CUDA graph shapes, prefill budget, and MTP fallback. The recorded constraints included the 2^31 - 1 memref limit in the affected path, autotuner allocations, and a narrow VRAM margin. These were additional integration constraints, not all manifestations of the same memory bug. The memref limit in particular should not be read as a universal limit on GPU memory or model context.

This is why the custom development runtime mattered. Sanitizing routes addressed invalid indexing. Workspace reuse addressed duplicate buffers. Graph, prefill, and speculative-decoding choices addressed other execution constraints. The PLE loader change solved a separate checkpoint-format problem. Replacing the full image with a stock runtime would discard that combination of fixes in the environment described here.

The resulting stack combines separate responsibilities:

NVFP4 MoE experts
  -> FlashInfer B12x
  -> route validation and compatible workspace reuse

PLE n-gram table
  -> CPU offload and checkpoint-layout compatibility

KV cache
  -> an independent precision and capacity decision

These patches were necessary for checkpoint compatibility and backend execution in our environment, before any decision to quantize the KV cache. A reproduction should preserve the complete working runtime and validate each change against that baseline.

With those later integration changes distinguished from the initial baseline, the earlier memory-tuning sequence is easier to read on its own terms.

Once the checkpoint loaded, memory became the main constraint. The initial goal was a context window close to 240,000 tokens with a relatively generous sequence limit. Our first configuration requested 245,760 tokens, used gpu-memory-utilization=0.948, and allowed up to 64 sequences.

It failed the KV-cache capacity check. The runtime required 6.85 GiB and had 6.79 GiB available. Its calculated maximum was 243,936 tokens, just below the requested context length.

That small difference encouraged a tempting adjustment: increase the memory-utilization setting. We tried 0.96. In this environment, that left too little physical headroom for other work during initialization, and problems appeared around autotuning and CUDA graphs. The setting increased the budget available to the runtime, but it did not create additional memory on the device.

Reducing the requested context to 240,000 while returning utilization to 0.948 got us further. The runtime reported KV capacity for 242,909 tokens, completed startup, and answered its health check.

Then the first real inference failed. The failing operation wanted another 192 MiB, but only 32.44 MiB was free.

At that point the distinction between startup success and inference success was concrete. The server had enough memory for the checks it had already performed. It did not have enough for an allocation reached during generation.

The full progression looked like this:

Maximum context	Memory utilization	Maximum sequences	Observed result
245,760	0.948	64	Failed KV-capacity validation: 6.85 GiB required, 6.79 GiB available
245,760	0.96	64	Insufficient headroom around autotuning and CUDA graphs
240,000	0.948	64	Started successfully, then failed during generation
200,000	0.94	32	With expandable segments enabled, failed during startup
200,000	0.94	32	Without that allocator setting, passed sequential requests but failed under concurrent traffic
131,072	0.92	16	Completed three rounds of 16 concurrent requests

The inference failure pointed into the GDN linear-attention path. The trace identified a temporary state-tensor allocation during forward execution. That allocation needed free memory beyond the resources already accounted for by the successful startup and KV-capacity checks.

The GPU was also shared with other workloads. Its nominal capacity was therefore not the same as the budget available to this model. The amount of free memory at a critical allocation depended on more than the model weights and configured KV cache.

Our working explanation was that the deployment had too little margin for temporary allocations during generation, and the concurrent workload exposed that margin more reliably than isolated requests. The failure locations and reported allocation sizes supported that explanation. We did not collect a full allocation-level memory profile, so we could not quantify every contributor to the peak.

The next configuration was particularly misleading. At 200,000 tokens of context, utilization of 0.94, and a sequence limit of 32, several sequential requests succeeded. Requests through the API gateway also completed. If the acceptance criterion had been “the endpoint answers,” we could have stopped there.

Concurrent traffic produced another OOM. This time the failing allocation requested 290 MiB with only 150 MiB free. The container restarted, making it clear that the earlier successful responses had not exercised the workload that mattered.

That result changed the way we evaluated subsequent attempts. A successful sequential request remained useful as a functional check, but it could no longer stand in for a concurrency test. We needed to send overlapping requests and observe the process while generation was in progress.

We also tried the allocator option expandable_segments. In this setup, it introduced a different failure mode: an OOM during memory mapping, followed by a pidfd_getfd: Operation not permitted error in the PLE offload worker. We removed the setting.

That experiment did not establish that expandable segments are generally unsuitable for inference. It established that the option did not solve this deployment's problem and made this particular startup path fail. Carrying it forward would have added another unresolved variable.

The configuration that passed our concurrent test reduced three limits together: maximum context, maximum sequences, and memory utilization. The resulting settings were:

max-model-len: 131072
kv-cache-dtype: bfloat16
gpu-memory-utilization: 0.92
max-num-seqs: 16
PLE CPU offload: enabled
MTP speculative tokens: 1
prefix caching: enabled

The runtime also used the Qwen reasoning and tool-call parsers. Multimodal request limits were configured, but the test described below used text requests; it did not validate image or video workloads.

Changing three memory-related parameters at once means we cannot attribute the success to one of them in isolation. We did not perform an exhaustive sweep that held every other factor constant. The objective at this stage was to find a usable baseline in the actual environment. The combination above supplied that baseline for the workload we ran.

We tested it with three rounds of 16 concurrently submitted chat-completion requests. Each request asked for a roughly 300-word essay about distributed systems. The generation limit was 700 tokens, temperature was 0.8, and the client timeout was 180 seconds. Each round finished before the next one began.

The script counted responses containing a finish_reason field and inspected the container restart count. It also sampled GPU memory approximately every two seconds during the beginning of each round.

The recorded results were:

Round	Responses with a finish reason	Recorded container restarts
1	16 / 16	0
2	16 / 16	0
3	16 / 16	0
Total	48 / 48	0

The highest sampled GPU memory usage was 92,712 MiB, approximately 90.54 GiB. The final memory snapshot reported 4,537 MiB free, approximately 4.43 GiB.

Those numbers need a little care. The samples describe device-wide usage, including other processes on the GPU. Sampling at intervals can miss short peaks, and the script did not continuously sample for the entire duration of every request. The free-memory reading was a final snapshot; it should not be presented as a continuously guaranteed reserve under load.

The completion count also has a specific meaning. All 48 responses passed the script's check for the presence of a finish reason. That check did not evaluate the quality of the essays or require that every response ended naturally rather than reaching its token limit. We did not derive a throughput claim, a latency percentile, or a quality score from this test.

Most importantly, a server configured for 131,072 tokens of context was not being tested with 131,072-token prompts. These were short prompts with bounded generation. The result established that the selected configuration handled this concurrent text workload without a recorded restart. Testing long-context prefill, mixed prompt lengths, sustained traffic, and multimodal requests would require additional runs.

After the concurrent test, requests through the API gateway also completed. That checked a separate part of the deployment: the model was reachable through the intended serving route after the container changes. It complemented the direct inference test, while leaving the same workload limits in place.

What made this exercise useful was the sequence of increasingly meaningful checks. Download completion established that the artifacts were present. Successful loading established that the runtime could interpret them. A health response established that the service was up. A generated response exercised inference. Concurrent requests exposed a memory problem that the earlier checks had missed.

The same distinction applied to the fixes. The loader patch addressed a checkpoint-layout mismatch. The configuration override got past a runtime compatibility issue but still needed separate quality validation. The final memory settings addressed the observed failures under a particular workload. None of those changes, on its own, justified calling every aspect of the deployment validated.

We ended with a smaller context target than we initially wanted, a lower sequence limit, and a configuration that completed the chosen concurrency check. That was a much more useful foundation for further experiments than a larger advertised context attached to a process that restarted when traffic overlapped.

Further experiments need separate baselines for the runtime and the representation being changed. Long-context and sustained-load tests should exercise the complete custom stack. Any KV-cache quantization experiment should retain the established backend integration and be evaluated independently. Keeping those changes distinguishable lets us measure memory consumption, reliability, and answer quality without attributing a result to the wrong patch.