GINIGEN AI

Posted on Oct 21 Originally published at ginigen-ai-edge-ai-news.static.hf.space

0.1 Bits Per Weight Is Not 0.1 Bits Per Weight: Where the Bytes Go in Sub-2-Bit LLMs

#ai #machinelearning #edgeai #quantization

TL;DR

Recent coverage of Samsung Research's LittleBit repeats its headline: Llama2-13B at 0.1 bits per weight, under 1 GB. The paper (arXiv 2506.13771, NeurIPS 2025) supports the number, but the number is a property of the transformer blocks only. Three things follow when you account for the whole file:

The file is not 0.1 bits per parameter. The embedding table and lm_head stay in FP16. By our reconstruction they are about 80% of the 0.84 GB Llama2-13B file and about 83% of the 0.63 GB Llama2-7B file. File-level bits per parameter are 0.52 (13B) and 0.75 (7B), not 0.10.
The quality cost is real and was measured by the authors. At 0.1 BPW, WikiText-2 perplexity on Llama2-7B is 15.92 against 5.47 for FP16, about 2.9x. At 0.3 BPW several zero-shot tasks sit at chance level.
The 11.6x speedup is one MLP layer on an A100, from a custom kernel the authors say is not yet at the level of industry libraries. It is not a phone benchmark.

This post is arithmetic on published numbers and file sizes. We did not run LittleBit. Every figure is either quoted from the paper, read from Hugging Face metadata on 2026-10-22, or computed by us with the script shown below, and we say which.

What does "0.1 bits per weight" actually mean?

LittleBit replaces each linear layer's weight matrix W (shape d_out x d_in) with a low-rank factorization whose two factors are binarized to +1 or -1. Scales in FP16 compensate for the lost magnitude. The paper stores two parallel paths, a primary one and a residual one, and counts bits per layer like this (the paper's Appendix D, Eq. 16):

bits = 2r(d_out + d_in) + 32(d_out + d_in) + 32r
b    = bits / (d_out * d_in)          # effective bits per weight of that layer

The first term is the sign bits of the two factors on two paths. The second and third are the FP16 row, column and latent scales on two paths. To hit a target b, the authors solve for the latent rank r (Eq. 17):

def rank(b, d_out, d_in):
    return (b * d_out * d_in - 32 * (d_out + d_in)) / (2 * (d_out + d_in) + 32)

rank(0.10, 4096, 4096)    # 86.2   -> a rank-86 factorization of a 4096 x 4096 matrix
rank(0.10, 11008, 4096)   # 133.1  -> Llama-2 7B gate/up projection
rank(0.30, 4096, 4096)    # 290.6
rank(0.55, 4096, 4096)    # 546.1

So "0.1 bits per weight" for Llama-2 7B means each attention projection is approximated by two paths of rank about 86, which is 2.1% of the full 4096 rank. We checked that plugging the rounded rank back into Eq. 16 returns 0.0998. The paper's KV-cache table (Table 4) lists a latent rank of 192 for 0.1 BPW and 1,624 for 0.8 BPW, roughly twice our per-path values, which looks like the two paths' ranks added together. We did not confirm that reading with the authors.

This is also why the method is quantization-aware training and not a drop-in post-training step. The factors are initialized from an SVD (the paper calls it Dual-SVID) and then trained with a distillation loss, using WikiText-2 plus C4 partitions, sequence length 2048 and 5 epochs. The compute cost of that training is not stated in the parts of the paper we read.

Why is a 0.1 BPW 13B model 0.84 GB and not 0.16 GB?

Because quantization is applied to the transformer blocks only. The paper says so itself in the caption of Table 3: "components like the embedding layer and the lm_head remain in FP16."

Llama-2 has a 32,000-token vocabulary. The embedding table and the output head each hold 32000 x d_model parameters, and neither is shrunk. At FP16 that is a fixed cost no matter how hard the blocks are squeezed. We rebuilt the file size from the public architecture shapes (layer count, hidden size, FFN size, KV width):

Model	Block params	FP16 embed + head	Blocks at 0.1 BPW	Our estimate	Paper Table 3	Embedding share of file
Llama2-7B	6.48B	0.524 GB	0.081 GB	0.605 GB	0.63 GB	83%
Llama2-13B	12.69B	0.655 GB	0.159 GB	0.814 GB	0.84 GB	78%
Llama2-70B	68.45B	1.049 GB	0.856 GB	1.904 GB	1.98 GB	53%

Our estimates land 3% to 4% below the paper's sizes. We did not chase the remainder, which is plausibly rank rounding, norm weights and file overhead. The same reconstruction also tracks the paper at the other settings for 13B: 1.53 GB estimated against 1.51 GB at 0.55 BPW, and 1.13 GB against 1.15 GB at 0.3 BPW.

Two practical consequences for an on-device team:

Below roughly 0.3 BPW, the next saving is not in the blocks. At 0.1 BPW on 13B, going all the way to zero bits in the blocks would still leave 0.655 GB of embeddings. Quantizing the embedding table and lm_head is the remaining lever, and vocabulary size matters. A model with a 150,000-token vocabulary pays about 4.7x this embedding cost at the same hidden size.
Compression factors flatter large models. The headline 31x for 13B and 69.7x for 70B are file-size ratios against FP16. They grow with model size because the fixed embedding cost is a smaller fraction of a bigger file.

How do other sub-2-bit releases look at the file level?

"Bits per weight" is reported inconsistently across projects: sometimes nominal for the quantized tensors, sometimes for the whole file. The honest comparison metric for deployment is the same one you pay for in RAM: file bytes times 8, divided by total parameters. Here it is for LittleBit's published sizes next to three GGUF files from PrismML's Bonsai family on Hugging Face (Apache-2.0, based on Qwen3.6-27B and Qwen3.8-27B), measured the same way:

File	Size	File-level bits per parameter
LittleBit Llama2-70B, 0.1 nominal	1.98 GB	0.23
LittleBit Llama2-13B, 0.1 nominal	0.84 GB	0.52
LittleBit Llama2-7B, 0.1 nominal	0.63 GB	0.75
`Bonsai-27B-Q1_0.gguf`	3.80 GB	1.13
`Ternary-Bonsai-2-27B-PTQ1_0.gguf`	5.95 GB	1.77
`Ternary-Bonsai-27B-Q2_0.gguf`	7.17 GB	2.13

The Bonsai rows use the GGUF metadata total of 26,895,998,464 parameters and the file sizes from the Hugging Face API. The LittleBit rows use 16 x size / FP16 size from the paper's Table 3. Here is the script for the GGUF side, which works for any GGUF repository that reports a parameter total:

import json, urllib.request

def file_bits_per_param(repo, filename):
    url = f"https://huggingface.co/api/models/{repo}?blobs=true"
    m = json.load(urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "x"})))
    size = {s["rfilename"]: s["size"] for s in m["siblings"]}[filename]
    return size * 8 / m["gguf"]["total"]

file_bits_per_param("prism-ml/Bonsai-27B-gguf", "Bonsai-27B-Q1_0.gguf")  # 1.131

This table is a size comparison and nothing more. The rows are different models, different base architectures, different training procedures and different benchmarks, so it says nothing about which one is better. We have not inspected the Bonsai tensor layouts, and we do not know which of their tensors, if any, are kept at higher precision. We also have not tested whether any of these files runs well on a specific phone.

What does the quality cost?

The authors report WikiText-2 perplexity across model families, with STBLLM, a sub-1-bit baseline, as the comparison. For Llama2-7B (paper Table 1), LittleBit keeps perplexity within a factor of three of FP16 all the way down to 0.1 BPW, while STBLLM collapses at 0.3 BPW:

Llama2-7B WikiText-2 perplexity	FP16	0.55 BPW	0.3 BPW	0.1 BPW
LittleBit	5.47	10.47 (1.9x)	12.00 (2.2x)	15.92 (2.9x)
STBLLM	5.47	30.67	about 1,800	not reported

The multiplier grows with model difficulty. At 0.1 BPW the table gives 15.09 against 4.88 for Llama2-13B (3.1x), 26.11 against 6.10 for Llama3-8B (4.3x) and 35.26 against 6.34 for QwQ-32B (5.6x). The authors themselves write that "a quantization cliff appears between 0.3 and 0.1 BPW" and recommend 0.3 to 0.55 BPW as the sweet spot.

Perplexity is a gentle metric. Zero-shot accuracy is harsher (paper Table 2, Llama2-7B, average over seven tasks):

FP16 averages 62.97%. LittleBit at 0.55 BPW averages 47.26% and at 0.3 BPW 45.20%. Look at the individual tasks at 0.3 BPW and the picture is stark: WinoGrande, a two-choice task, scores 51.30% against a chance level of 50%, and ARC-Challenge, a four-choice task, scores 25.09% against a chance level of 25%. Tasks that need broader knowledge or reasoning are close to guessing. PIQA, at 65.83% against 78.12% for FP16, and BoolQ, at 61.80% against 71.37%, retain more.

Two caveats on how to read these numbers. First, the training data includes WikiText-2 (presumably its training split), and WikiText-2 perplexity is the headline metric, so the perplexity figures are partly in-distribution for the training run. The paper says it follows the training protocol of an earlier binarization paper, but the setup still flatters perplexity compared with a held-out domain. Second, the zero-shot tasks are common-sense multiple choice benchmarks. They tell you nothing about long-context behavior, tool calling or code, which are what most on-device assistants need.

What is the 11.6x speedup, and what is it not?

The paper reports a custom 1-bit GEMV CUDA kernel running a single Llama2-70B MLP layer (8,192 x 28,672) on an NVIDIA A100, peaking at 11.6x over an optimized FP16 baseline at 0.1 BPW. That is a kernel micro-benchmark on a data-center GPU.

The authors add that "the discrepancy between theoretical computational gains and practical latency is likely attributable to memory access dominance during small-batch inference" and that the kernel "has not yet reached the optimization level of industry-standard libraries." They list hardware co-design for NPUs as future work. So there is no published number for end-to-end tokens per second on a phone, and you should not infer one from 11.6x.

The authors also publish a code repository, SamsungLabs/LittleBit, whose description says it contains the NeurIPS 2025 implementation and the follow-up LittleBit-2. That follow-up, arXiv 2603.00042, says its geometric preconditioning "establishes a new state-of-the-art in the sub-1-bit regime" from 1 down to 0.1 bits per parameter and matches the fidelity of leading 1-bit baselines. We have not read its tables, so we make no claim about its numbers here.

When does this matter for an edge deployment?

Our reading, which is an opinion and not a measurement:

If your budget is a few hundred megabytes of weights, the 0.3 to 0.55 BPW range is where the authors' own data suggests the trade is defensible. It pairs with a use case that tolerates a large quality gap, such as a classifier, a router or a drafter, and not with open-ended chat.
If your target is a phone with 8 to 12 GB of RAM, a 4-bit or 2-bit file of a stronger model is likely the better starting point, because the quality gap at 0.1 BPW is large and the runtime stack for factorized binary weights is not mature. Measure on your own tasks before choosing.
If you are comparing quantization headlines, convert every claim to file-level bits per parameter first. It takes two lines of Python, and it turns marketing numbers into numbers that predict your RAM bill.

What are the limits of this analysis?

We did not run LittleBit or any Bonsai model. Quality numbers are the authors' own, from the arXiv HTML of the paper (v5, February 2026).
Our file-size reconstruction uses public Llama-2 shapes and assumes every block linear layer sits exactly at the nominal rate. It is within 4% of the paper, and the gap is unexplained.
Table 1 mixes Llama (first generation) and Llama2 columns. We used the Llama2-7B and Llama2-13B columns, which the paper's own text confirms for the 12.00 and 15.92 figures.
The Bonsai comparison is file size only. We did not verify press figures about those models beyond what Hugging Face metadata shows.
The paper's license is CC BY-NC-ND 4.0, so we summarize and cite and do not reproduce its tables beyond the figures quoted above.

FAQ

What does 0.1 bits per weight mean for LittleBit?
It is the average cost per weight of the quantized transformer blocks, counting sign bits and FP16 scales, after replacing each matrix with two rank-r binary factorizations. It does not include the embedding table and output head, which stay in FP16.

How big is Llama2-13B with LittleBit at 0.1 BPW?
0.84 GB according to the paper, against 26.06 GB in FP16. That is 31x smaller, and about 0.52 bits per parameter for the whole file.

Does LittleBit need training?
Yes. It is quantization-aware training with distillation, initialized from an SVD of the original weights, using WikiText-2 and C4 data for 5 epochs at sequence length 2048 in the authors' setup. It is not a post-training step you can run in minutes.

How much accuracy is lost at 0.1 bits per weight?
Perplexity on Llama2-7B rises from 5.47 to 15.92. At 0.3 BPW, zero-shot WinoGrande and ARC-Challenge for Llama2-7B are at chance level. The authors call 0.3 to 0.55 BPW the sweet spot.

Can I run it on a phone today?
We found no published phone benchmark. The reported 11.6x speedup is a single-layer kernel on an A100 GPU, and the authors state that the kernel is not yet at the level of industry-standard libraries.

DEV Community