The headline is always the same: a model that runs on a laptop, at a fraction of the energy, with almost no quality loss. The number behind it is bits per weight, pushed from sixteen down to eight, four, and now below two.
The direction is right. The mechanism, however, is routinely misdescribed, and that misdescription leads teams to expect savings in the wrong place.
Part One: Where the energy actually goes
Generating a token is not an arithmetic-heavy operation. For a single user, batch size one, the accelerator spends most of its time waiting for weights to arrive from memory. The multiply-accumulate units are mostly idle.
That is the whole reason low-bit weights work so well. Halving the bits halves the bytes that must be moved to produce a token. If you are bandwidth-bound, halving the traffic roughly halves the time and the energy. The arithmetic gets cheaper too, but that part was never the bottleneck.
This has a sharp consequence: low-bit weights speed up generation far more than they speed up training or prefill, because those phases are compute-bound and already batch the arithmetic efficiently. A team that quantises weights and benchmarks on prefill will conclude the technique underdelivers. It did not; it was measured in the wrong place.
Part Two: What breaks below four bits
At eight bits, quantisation is close to free. At four bits, it costs a little on reasoning and almost nothing on fluency. Below three bits, the failure stops being uniform, and that is the dangerous part.
A model at 1.58 bits does not degrade evenly. Instruction following stays surprisingly intact, because the format is learned redundantly across many weights. Multi-step reasoning collapses first, because it depends on precise intermediate values that a ternary representation cannot hold. The model still sounds competent, which is exactly why the failure is easy to miss in a demo.
The second break is calibration. Quantisation needs a scale, and the scale is computed from a sample of data. If that sample does not resemble production traffic, the model loses quality on inputs the calibrator never saw. This is invisible in benchmarks and obvious in support tickets.
The third break is tooling. Kernels for sub-4-bit inference are specialised, and the fast path exists for a handful of shapes. A model that fits in 1.58 bits but hits an unoptimised kernel can end up slower than the 4-bit version it replaced.
Part Three: The business case, done properly
Suppose you serve a 70B model for a chat product. At FP16, the weights occupy about 140 GB, so you need multiple accelerators per replica. At 4 bits, the same model fits in about 35 GB, which fits on one device with room for cache.
The immediate effect is not "four times faster". It is that the replica count per device drops from four to one, and the deployment becomes four times denser. Cost per concurrent user improves by roughly that factor, before any speedup from bandwidth.
At 1.58 bits the weights fall to about 14 GB, and the model now competes with the cache for space. The marginal gain from 4 bits to 1.58 bits is much smaller than the gain from 16 to 4, because you have already crossed the threshold where the model fits on one device. Below that threshold, further compression mostly buys you more room for the KV cache — which is worth something, but not what the headline promised.
This is the general shape of the curve. Compression pays off in steps, not smoothly, and the biggest step is the one that makes the model fit on a single device.
Part Four: What to measure before shipping
Four checks, in order.
Measure time per output token at batch one, not throughput at batch thirty-two. Bandwidth savings only show up where bandwidth is the limit.
Measure the reasoning subset separately. Aggregate benchmark scores hide the collapse that matters, because fluency carries the average.
Measure the tail. Quantised models fail more often on unusual inputs, and the failures are confident rather than garbled. A red-team pass on edge cases is cheaper than a production incident.
Measure kernel support for your exact shape and batch size. The theoretical speedup assumes an optimised path exists, and for sub-4-bit it often does not.
What is actually transferable
The transferable idea is that the benefit of a compression technique is determined by which resource was the bottleneck, not by how much was compressed. Low-bit weights help because decoding is memory-bound. The same technique applied to a compute-bound training job yields far less, and applying it to a storage-bound pipeline yields almost nothing.
That single question — what is the bottleneck — predicts whether a compression result will reproduce in your stack far better than any published benchmark. It is also the question that benchmark tables are least likely to answer, because the vendor's bottleneck is rarely yours.
SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS
Creating this takes real effort — thank you for the support.
Top comments (0)