Take a 7B model, find the smallest half of its weights by magnitude and set them to zero. You now have a model that is half sparse, exactly the same size on disk, exactly the same speed at inference, and slightly worse at its job. The entire subject of pruning is the gap between that sentence and a useful one.
Zeroing weights changes nothing by itself
A weight matrix is a dense array of numbers. Setting some of them to zero does not remove them from the array. The file still contains one value per position, the tensor still occupies the same memory, and the matrix multiply still reads every element and multiplies it. A multiplication by zero costs exactly what a multiplication by 0.043 costs.
So “we pruned the model to 50 per cent sparsity” on its own reports a property of the numbers, not a saving. To convert it into a saving, one of two things has to happen: the zeros must be removed from storage in a format the compute kernel can read, or the sparsity must follow a pattern the hardware has a circuit for. Those are the only two doors, and both are narrow.
The storage arithmetic of a sparse format
Compressed sparse row is the standard general format: store the non-zero values, a column index for each, and a row pointer array. Cost it out for a 16-bit weight matrix pruned to 50 per cent:
Dense fp16:
2 bytes per weight
1,000,000 weights = 2,000,000 bytes
CSR at 50% sparsity, 32-bit indices:
500,000 values * 2 bytes = 1,000,000
500,000 indices * 4 bytes = 2,000,000
---------
3,000,000 bytes
The "compressed" version is 1.5x LARGER than the dense one.
With 16-bit indices it is 2,000,000 bytes — a tie, at the cost of a much worse memory access pattern. The break-even for general sparse formats sits somewhere below 25 per cent density, and to get a real win you want under 10 per cent, which is far more aggressive than a large language model tolerates without retraining.
And storage is the easy half. A sparse matmul kernel reads indices, gathers scattered values and cannot use the dense tensor-core instructions that make an accelerator fast. In practice an unstructured sparse kernel at 50 per cent density is slower than the dense kernel it replaced. This is the single fact most pruning write-ups leave out.
The one pattern the hardware accelerates
NVIDIA’s tensor cores from the Ampere generation onward support one specific structured pattern, usually written 2:4: in every contiguous group of four values along the reduction dimension, exactly two must be zero.
Every group of 4 weights: [w0, w1, w2, w3]
Allowed: exactly 2 non-zero, e.g. [w0, 0, w2, 0]
Storage: 2 values + a 2-bit index per surviving value
-> roughly half the weight bytes, with the metadata inline
Throughput: the sparse tensor-core path is specified at up to 2x
the dense matmul rate.
This is the pattern to aim for if hardware acceleration is the goal, and the constraints are worth stating clearly. It caps you at exactly 50 per cent — there is no 75 per cent version. It constrains which weights you may remove, so you cannot simply keep the largest ones globally; you must keep the largest two in every group of four, which is a strictly worse selection than unconstrained magnitude pruning and costs measurably more quality. And it needs a runtime and a kernel that support it end to end.
Structured pruning: remove whole things
The approach that actually ships is coarser and simpler: delete entire structural units so that what remains is a smaller dense model.
- Attention heads. Drop
kof the heads in a layer and the projections shrink accordingly. Published analyses have repeatedly found many heads removable with small quality cost. - MLP channels. Remove columns of the up-projection and the matching rows of the down-projection. The intermediate dimension falls, and everything stays dense.
- Whole layers (depth pruning). Drop entire transformer blocks. Blunt, and effective: the residual stream means a removed block is an identity, and adjacent layers in deep models are often near-redundant.
- Hidden dimension (width pruning). Reduce the model dimension itself, which touches every matrix in the network.
The result needs no special kernel, no special format and no special hardware. It is a smaller model, and every existing tool serves it. The standard recipe is prune then heal: remove structure, then fine-tune the survivor with distillation from the original as the teacher, which recovers most of the loss for a small fraction of the original training cost. NVIDIA’s Minitron work in 2024 is a published worked example, pruning and distilling Llama-3.1 8B down to 4B.
Deciding what to remove
“Remove the smallest weights” is the obvious criterion and it is measurably the wrong one for large language models, because of outlier features. A small number of hidden dimensions in these models carry activations orders of magnitude larger than the rest. A modest weight multiplying one of those contributes far more to the output than a large weight multiplying a typical one, so magnitude alone ranks the wrong things.
The one-shot methods that work correct for this, and both were published in 2023:
| Method | Description |
|---|---|
| Wanda | Score each weight by |weight| times the norm of its corresponding input activation, measured on a small calibration set, and prune per output row. No gradients, no retraining, one forward pass. The simplicity is the point: it is a one-line change from magnitude pruning that accounts for the thing magnitude pruning ignores. |
| SparseGPT | Treat each layer as a reconstruction problem: choose which weights to drop and update the survivors to compensate, column by column, using second-order information from the calibration activations. More expensive than Wanda and generally better, particularly at the constrained 2:4 pattern where the selection is forced and compensation matters most. |
| Taylor / activation criteria | For structured pruning, rank whole heads, channels or layers by an estimate of how much the loss would rise if that unit were zeroed — a first-order Taylor estimate, or accumulated activation statistics. This is what decides which layers get dropped in depth pruning. |
Both one-shot methods need calibration data, typically a few hundred sequences, and this is the footgun worth naming. The pruning decision is fitted to whatever those sequences look like. Calibrate on generic web text and prune a model you are going to use for code, and you have removed the weights that mattered for the task you did not sample. If you prune, calibrate on data that resembles your workload, and evaluate on it afterwards rather than on a general benchmark.
The calculation that decides it
Token generation is memory-bound. Each token requires reading the entire set of weights from memory, and the arithmetic per weight is tiny, so the ceiling is set by bandwidth:
7B parameters at fp16 = 14 GB of weights read per token
memory bandwidth = 2,000 GB/s (order of magnitude for a
current data-centre accelerator)
time per token >= 14 / 2,000 = 0.007 s -> about 143 tokens/s ceiling
Structured pruning to 3.5B params -> 7 GB
0.0035 s -> about 286 tokens/s
50% unstructured, stored dense -> still 14 GB
no change whatsoever
50% at 2:4 with a supported kernel -> about 7 GB of weight bytes
roughly 2x, if the kernel path holds end to end
That table is the page. The question is never “how many weights are zero”; it is “how many bytes does the accelerator read per token, and can it skip the multiply”. If pruning does not change that number, it has done nothing but lower the quality.
Batched serving changes the picture, because with a large batch the weights are read once and used for many sequences, which moves the workload toward compute-bound. Sparsity that only saves bytes helps less there, and sparsity that saves arithmetic helps more. Single-stream latency and high-throughput batch serving are different problems and they favour different techniques.
Against quantisation and distillation
Pruning does not compete in a vacuum. It competes with two techniques that are usually easier.
Quantisation takes the same 14 GB to roughly 3.5 GB at 4 bits — a 4x reduction in exactly the number that matters — with kernels that every current inference stack already supports, no retraining in the common post-training case, and quality loss that is well characterised. Quantisation is where to start, and choosing the level is a better use of an afternoon than a pruning campaign.
Distillation trains a genuinely smaller model to imitate a larger one, and often ends up ahead of a pruned model of the same size because the small architecture was designed rather than carved. The two combine: prune to get a starting point, distil to recover, which is the recipe cited above.
The trade, stated plainly: unstructured pruning buys the largest quality-per-removed-weight and cashes in as nothing on current hardware. Structured pruning buys a real speed-up on any hardware and costs more quality per parameter removed, because you are forced to remove things you would rather keep. Where you land depends entirely on whether the thing you can exploit is a pattern or a number.
One practical footnote: what a provider serves is often not the checkpoint you would download. Quantisation level and pruning are serving decisions made per provider, and they change both cost and output. Multigrid lists what each provider serves for a given model, which is worth checking before attributing a quality difference to the model rather than to the deployment.
Top comments (0)