LoRA bolts two skinny matrices onto a frozen one: W ← W + (α/r)·B·A, with B starting at zero so the adapter is a literal no-op at step 0. Then you pick r, and everyone has a story about how to pick it.
This page is built so that r has a correct answer you can write down. The downstream model is the base model plus a known change ΔW* of exact rank ρ, so "rank r is enough" is an algebraic fact rather than a hope. The held-out score is a mean KL from the true conditional — an exact expectation over labels, exactly 0 for the teacher, above a conditional-entropy floor of 1.1479 nats.
Sweep it yourself: https://dev48.infy.uk/dl/day73-lora-low-rank-adaptation.html
First, the adapter that provably exists
At r ≥ ρ a zero-excess adapter exists. So the page does not argue about it — it writes it down:
| value | |
|---|---|
| excess of the constructed adapter | 1.2e-17 |
| weight gap | 9.6e-15 |
That is the SVD's own rounding, not a search. Then it trains one: r = ρ = 4, 200 Adam steps, landing at 0.0414. None of that residual is the rank ceiling, because the ceiling is provably not binding.
The rank that wins is never ρ
| intrinsic rank ρ | 1 | 2 | 4 | 8 | 12 |
|---|---|---|---|---|---|
| best r, over four paired seeds | 1 | 2 | 2 | 3 | 2 |
It never tracks ρ. It beats the largest rank in 20 of 20 paired runs. And a rank that provably cannot express ΔW* beats one that provably can in 107 of 136 paired runs — 78.7%.
Scored as a ranker over 90 trained configurations, rank gets AUC 0.488 against a coin's 0.500, and 0.433 inside a single task-and-rate cell — below the coin exactly where it is asked a question it can see, because within a cell more rank is reliably slightly worse. Ten steps of training scores 0.753 on the same task.
It is regularisation, and the data control says so
| training examples | 250 | 16,000 |
|---|---|---|
| penalty for r = 12 vs the best rank | 0.0740 | 0.0102 |
The penalty shrinks with data, which is what a regularisation effect does. And even at 16,000 examples the best rank is 2, not 4.
The one thing r controls exactly
Trainable parameters: 2rd. Break-even against fine-tuning the matrix outright is r* = mn/(m+n), exactly:
| matrix | 12×12 | 4096×4096 | 4096×11008 |
|---|---|---|---|
| r* | 6 | 2048 | 2985.2 |
So on this page's 12×12, r ≥ 7 costs more trainable parameters than the matrix it is adapting while staying strictly less expressive. That is why the saving is real at scale and absent here — and rank-2 LoRA at 48 parameters still beats fine-tuning the whole 144-parameter matrix, 0.0338 against 0.0390.
481 verifier asserts, 179 in-page checks, 0 failures.
Top comments (0)