Parameter-efficient fine-tuning is a family of methods that freeze a pretrained model and train a small number of new weights alongside it. LoRA is the member everyone uses, and the reason it beat the others is a structural property rather than a quality one: its update can be added into the frozen weights and disappear.
The problem the family solves
Full fine-tuning of a 7B model updates all 7 billion parameters. In 16-bit that is 14 GB of weights, 14 GB of gradients and, with an Adam optimiser keeping two moments per parameter in 32-bit, another 56 GB of optimiser state — before activations. And the output is a new 14 GB checkpoint per task.
Ten tasks means ten checkpoints, ten deployments and ten copies of the same base model in memory. Every method here exists to make the per-task artefact small and the base model shared.
Bottleneck adapters, and their cost
The original formulation (Houlsby and colleagues, 2019) inserts a small module after each sub-layer:
h' = h + W_up @ nonlinearity(W_down @ h)
W_down : (r, D) project down to a small rank r
W_up : (D, r) project back up
plus a residual, so an adapter initialised near zero is a no-op
params per adapter = 2*D*r + r + D
D = 4,096, r = 64 -> 2*4,096*64 = 524,288, about 0.5M
2 adapters per layer * 32 layers * 0.5M = 33.6M
33.6M / 7,000M = 0.48% of the base model
Half a per cent of the parameters, trained, with the rest frozen. That works, and it was a genuine advance. The problem is the first character of the formula: h’ = h + f(h) is a new computation that must happen between two existing ones. It adds sequential depth to every layer at inference, and on a latency-sensitive decode step where each layer is already memory-bound, two extra small matmuls per layer with their own kernel launches is a measurable tax on every token, forever.
Why LoRA won: parallel, not sequential
LoRA changes where the module sits. Instead of a block after the projection, it is a low-rank correction to the projection itself:
Adapter (sequential): y = W @ x then y = y + W_up @ act(W_down @ y)
LoRA (parallel): y = W @ x + (alpha/r) * B @ (A @ x)
A : (r, D) B : (D, r) both trained, W frozen
Because both branches are linear in x, they can be merged:
W_merged = W + (alpha/r) * B @ A
...and after merging there is NO adapter at inference. Zero extra
operations, zero extra latency, an ordinary weight matrix.
That is the whole reason LoRA displaced the rest. The mathematics of what it can express is not obviously better — a rank-r additive update is a different restriction from a rank-r bottleneck with a nonlinearity, not a strictly larger one. But one of them costs nothing to serve and the other costs something on every token of every request.
Sizing it: at rank 16 on a 4,096-dimensional model, one adapted matrix costs 2 * 4,096 * 16 = 131,072 parameters, or 262 KB at fp16. Applied to the query and value projections across 32 layers that is about 8.4M parameters, roughly 17 MB — a file you can attach to an email, against a 14 GB checkpoint.
The merge has one consequence worth knowing: a merged LoRA is no longer swappable. Serving many fine-tunes from one base means keeping them unmerged and applying them as a separate branch per request, which reintroduces exactly the sequential cost merging removed. That is the trade multi-adapter serving makes deliberately: a small per-token cost in exchange for hundreds of tunings on one resident base.
Choosing the rank and the target modules
Two decisions dominate, and they interact in a way that is easy to get wrong.
Which matrices to adapt
The original work adapted only the query and value projections. Common practice now adapts every linear layer, including the MLP, and the parameter arithmetic explains why that is a real decision rather than a detail. Take a 7B model: dimension 4,096, MLP intermediate 11,008, 32 layers, rank 16.
Query and value only:
2 matrices * 2 * 4,096 * 16 = 262,144 per layer
* 32 layers = 8.4M parameters (about 17 MB at fp16)
Every linear projection:
q,k,v,o 4 * 2*4,096*16 = 524,288
gate, up 2 * (4,096+11,008)*16 = 483,328
down 1 * (11,008+4,096)*16 = 241,664
---------
1,249,280 per layer
* 32 layers = 40M parameters (about 80 MB)
Nearly five times the trainable parameters, and the MLP matrices are where most of a transformer’s knowledge and behaviour live, so the broader target set usually does better on tasks that need more than a change of tone. It is still under 0.6 per cent of the base model.
Rank, and the alpha trap
Parameters scale linearly in r, so the cost side is simple. Typical values run from 4 to 64, with higher ranks for tasks that are further from what the base model already does. The subtlety is the scaling factor:
update = (alpha / r) * B @ A
Raise r from 16 to 64 and keep alpha fixed:
capacity goes up 4x
the scale on the update goes DOWN 4x
...two changes at once, from one edit. This is why the
"set alpha = 2r" convention exists: it holds the scale constant
so that changing r changes only capacity.
A great many inconclusive rank sweeps are this effect rather than a finding about rank.
One more detail worth knowing because it explains why LoRA training is stable from step one: A is initialised randomly and B is initialised to zero, so the product is exactly zero at the start and the adapted model is bit-for-bit the base model. There is no initialisation shock to warm up through.
The other members, sized
| Method | Description |
|---|---|
| Prefix / prompt tuning | Learn k virtual key-value vectors prepended at every layer. Costs k * 2 * layers * d parameters — and, unlike the others, costs context: those k positions sit in the KV cache for the life of the request and shorten the usable window. |
| IA3 | Learn one multiplicative scale per channel for keys, values and the MLP activation. Three vectors of length d per layer — about 0.4M parameters on a 7B model, roughly 0.006 per cent. Also mergeable, since scaling a matrix’s rows is still a matrix. |
| BitFit | Train only the bias terms and freeze every weight. Around 0.1 per cent of parameters. Remarkable that it works at all; not competitive when the task needs a real behavioural change. |
| QLoRA | LoRA on a base model quantised to 4 bits, with gradients flowing through the frozen quantised weights. The point is memory: it is what puts fine-tuning a large model on one consumer GPU. |
| Mixture of adapters | Several adapters plus a router that picks per token or per request — the same structural idea as mixture of experts, applied to fine-tuning rather than to capacity. |
Hypernetworks, and why they stay small
A hypernetwork is a network that emits the weights of another network from a task embedding: W_task = H(e_task). The appeal is obvious — generate a specialised model for an unseen task without training on it.
The output dimension is what stops it. To emit one 4,096 by 4,096 matrix the hypernetwork needs 16.8 million outputs, and its final layer alone would dwarf the model it is generating. So every practical hypernetwork emits something compressed: per-layer scales, per-channel shifts, or the low-rank factors A and B of a LoRA.
Which puts it back in this family, one level up. A hypernetwork that emits LoRA factors is a learned function from task descriptions to adapters — the modularity is real, but the module is still a low-rank update, and the constraint on what it can express is exactly the same.
What none of them can do
- Add knowledge. These methods reshape behaviour, not content. Teaching a model your product catalogue is a retrieval problem, and no adapter size fixes that; the frozen weights are where the knowledge lives and they are frozen.
- Exceed the rank. A rank-16 LoRA can only add a rank-16 correction to each matrix. That is an enormous amount of behavioural change and a hard ceiling on how far the function can move. Where full fine-tuning wins, this is usually why.
- Compose reliably. Adding two LoRAs trained separately gives their summed update, and there is nothing that makes the sum do both jobs. Adapter merging works sometimes and interferes other times, and no published method makes it dependable.
- Escape the base model’s failures. Tokeniser limits, context length, refusal behaviour baked into the base and architectural constraints all survive intact.
The trade: train 0.1 to 1 per cent of the parameters, ship a 17 MB artefact instead of a 14 GB one, and keep one base model resident for hundreds of tasks. Pay with a hard cap on how far the model can move, no ability to add facts, unreliable composition, and — for everything except the mergeable methods — a small latency tax on every token.
Top comments (0)