NyoomFloat12: How a 12-Bit Fixed-Length Format Doubles LLM Inference Throughput Without Losing a Single Bit

Serving large language models at scale is fundamentally a memory problem. During autoregressive decoding, every generated token requires reading the entire weight matrix and the accumulated KV cache from GPU high-bandwidth memory (HBM). On an H100, that bandwidth ceiling is around 3.35 TB/s — fast, but finite. As context windows grow and batch sizes increase, the gap between what the GPU can compute and what it can actually move through memory widens.

Lossless weight compression is one lever practitioners can pull to ease this bottleneck. The idea is simple: store weights in a smaller format, decompress on the fly during inference, and net a bandwidth gain. The catch is that decompression itself must be cheap enough not to eat the savings. A new paper from researchers at Stanford and Together AI introduces NyoomFloat12 (NF12), a 12-bit fixed-length format that threads this needle by exploiting a structural property of trained BF16 weights that turns out to be nearly universal.

The Redundancy Hidden in BF16 Weights

BFloat16 uses 1 sign bit, 8 exponent bits, and 7 mantissa bits. The exponent field is where the redundancy lives. In a trained model, the distribution of weight magnitudes is tightly concentrated — most weights are small, and after applying a per-matrix power-of-two scale to normalize the distribution, the upper four bits of the exponent field are 0111 for more than 99.7% of weights across every tested LLM.

Those four bits are constant. Storing them is waste. NF12 drops them.

The result: eight BF16 weights (128 bits) pack into 96 bits — 12 bytes instead of 16. That is a 25% reduction in weight storage, achieved without any approximation.

For the small fraction of weights whose exponent falls outside the expected range, NF12 uses an escape mechanism. A zero meta byte signals that a group of eight weights is stored verbatim in a per-matrix BF16 buffer. The overhead is negligible, and the format remains strictly lossless.

Why Fixed-Length Matters for GPUs

The prior art in lossless LLM weight compression — most notably DFloat11, presented at NeurIPS 2025 — uses Huffman coding on the exponent bits. Huffman achieves slightly higher compression (around 10.79 bits per weight versus NF12's 12.02), but it produces variable-length encodings.

Variable-length codes are a poor fit for SIMT (Single Instruction, Multiple Threads) GPU architectures. When threads in a warp decode weights of different lengths, they diverge — some finish early and stall while others catch up. The decompression kernel becomes serialized, and the bandwidth savings evaporate into compute overhead. DFloat11's decode kernel takes around 1,771 microseconds on the benchmarks reported in the NF12 paper. NF12's fixed-length decoder takes 548 microseconds — a 3.2× speedup in decompression alone.

NF12's decode path is designed around GPU warp execution. A warp reconstructs groups of weights in lockstep: 13 register operations plus a three-instruction escape check. The common path costs less than a standard BF16 memory copy. That is the key property — the decoder is not just fast, it is faster than the alternative of not compressing at all.

Two Regimes, Two Wins

The NF12 paper evaluates performance across two distinct inference regimes, and the format delivers in both.

Bandwidth-bound inference (small batch, short context): Here the bottleneck is how fast weights can be streamed from HBM to the compute units. NF12's fused GEMV (general matrix-vector multiplication) kernel achieves 1.26× the throughput of a custom BF16 GEMV and 1.35× over cuBLAS on an H100. The 25% smaller weight footprint translates almost directly into a proportional bandwidth gain, minus the small decode overhead.

Capacity-bound inference (large batch, long context): Here the bottleneck is total GPU memory. The KV cache competes with model weights for HBM space, and as context grows, the cache wins — crowding out concurrency. By shrinking the weight footprint, NF12 frees HBM for KV cache. On Qwen3-32B, NF12 provides 4.5× more KV cache capacity compared to BF16. At high concurrency, this translates to 2.54× higher rollout throughput — the headline number from the paper.

That 2.54× figure is not a cherry-picked benchmark. It reflects a realistic serving scenario where many sequences are processed simultaneously and the KV cache is the binding constraint.

Practical Implications for Inference Engineers

NF12 is a drop-in optimization for existing model weights. No retraining, no fine-tuning, no accuracy degradation — the format is lossless by construction. The compression step is a one-time offline operation. At inference time, the fused GEMV kernel handles decompression transparently.

The format is most impactful in two situations practitioners encounter regularly:

1. Fitting larger models on existing hardware. A 32B model in NF12 occupies roughly the same HBM as a 24B model in BF16. That headroom can be used for longer contexts, larger batches, or simply running a model that previously required an extra GPU.

2. Improving throughput at long context. As context windows push into the hundreds of thousands of tokens, KV cache pressure becomes the dominant constraint. NF12's capacity gains compound with other KV cache optimizations like grouped query attention and prefix caching.

The comparison with DFloat11 is instructive for practitioners choosing between lossless methods. DFloat11 compresses slightly more aggressively and is well-suited for scenarios where the goal is fitting a model into a fixed memory budget and throughput is secondary. NF12 is the better choice when end-to-end inference speed is the objective — its fixed-length decoder keeps the GPU's SIMT execution efficient.

What This Tells Us About the Inference Optimization Landscape

NF12 is part of a broader pattern in 2026 inference research: the recognition that the memory wall is the primary constraint, and that solutions must be co-designed with GPU architecture rather than borrowed from general-purpose compression. The Invariant Bit Packing (IBP) work from earlier this year takes a similar approach for PCIe transfers. The common thread is that compression formats designed for CPUs — Huffman, ANS, LZ4 — do not map cleanly onto SIMT execution models.

NF12's contribution is demonstrating that a carefully chosen fixed-length format can beat variable-length alternatives on end-to-end throughput even when it compresses less. The 25% size reduction is not the point; the SIMT-aligned decoder is.

For teams running inference at scale, NF12 is worth evaluating as a complement to quantization-based approaches. Unlike INT8 or FP8 quantization, it introduces zero approximation error — useful in domains where numerical fidelity matters, or when quantization has already been applied to the KV cache and further lossy compression of weights is undesirable.

The paper is available on alphaXiv. The authors are Liberman, Fang, Zhang, Dao, and Fu — several of whom have prior work on FlashAttention and efficient GPU kernels, which explains the focus on SIMT-aligned design.

For teams already using vLLM or SGLang for serving, NF12 is designed to integrate at the kernel level without changes to the serving framework's scheduling or batching logic. The authors plan to release the CUDA kernels as open source, which should make adoption straightforward once the code is available.