Low-Rank Adaptation: Why Freezing Most Weights Still Fine-Tunes a Large Model

Full fine-tuning of a large language model is a strange bargain. You take a model trained on trillions of tokens, then you update every single one of its billions of weights to fit a task that might have only a few thousand examples. It works, but it is expensive, it risks forgetting what the model already knew, and it leaves you with a separate full copy of the model for every task. Low-Rank Adaptation, usually called LoRA, asks a sharper question: how much of that update do you actually need?

The answer, surprisingly often, is very little - provided the little you keep has the right shape.

The observation: updates live in a low-dimensional subspace

A pretrained weight matrix, call it $W_0$ , already contains most of what the model knows. Fine-tuning produces a new matrix $W = W_0 + \Delta W$ . The LoRA hypothesis is that $\Delta W$ does not need full rank. Even though $W_0$ itself is high rank, the change needed to adapt it to a new task has low intrinsic rank. In other words, adaptation is a small nudge in a few important directions, not a rewrite of everything the model knows.

This is not just a guess about efficiency. It connects to a broader finding in the field: large pretrained models appear to have low intrinsic dimensionality for downstream tasks. A model with billions of parameters can often be adapted by optimising only a few thousand well-chosen degrees of freedom and still reach nearly the same performance as full fine-tuning. LoRA turns that observation into a concrete parametrisation.

The LoRA parametrisation

Consider a dense layer with weight matrix $W_0 \in \mathbb{R}^{d \times k}$ , mapping a k-dimensional input to a d-dimensional output. Instead of learning $\Delta W$ directly, LoRA writes it as a product of two thin matrices:

$$\Delta W = B A$$

where $A \in \mathbb{R}^{r \times k}$ and $B \in \mathbb{R}^{d \times r}$ , and r is the rank. The forward pass becomes:

$$h = W_0 x + B A x$$

During training, $W_0$ is frozen. Only $A$ and $B$ receive gradients. At the start of training, $A$ is initialised randomly (for example, from a small Gaussian) and $B$ is initialised to zero, so $\Delta W = 0$ and the model behaves exactly like the pretrained model before any learning happens. That zero start matters: it means LoRA never degrades the model at step zero, unlike a randomly initialised adapter inserted into the network.

The parameter saving is easy to count. Full fine-tuning learns $d k$ numbers. LoRA learns $r (d + k)$ . For a layer with $d = k = 4096$ and rank $r = 16$ , that is about 131,000 trainable parameters instead of about 16.7 million, a reduction of more than 100 times for that layer. Across a whole model, a LoRA adapter is typically tens of megabytes, not gigabytes. You can keep one frozen base model and swap in a different small adapter for each task, each user, or each style.

One more detail matters in practice: scaling. LoRA scales the low-rank branch by $\alpha / r$ , where $\alpha$ is a constant you choose:

$$h = W_0 x + \frac{\alpha}{r} B A x$$

This scaling decouples the effective learning rate of the adapter from the rank. If you raise r, you do not automatically raise the magnitude of the update. In practice, $\alpha$ is often set to 16 or 32, and r to 8, 16, or 32, then tuned from there. Tuning r and $\alpha$ separately, rather than treating them as one knob, avoids a common confusion where increasing rank appears to hurt simply because the update grew larger.

Where to apply it

LoRA is most often applied to the attention weights of a transformer: the query, key, value, and output projection matrices. That choice is not arbitrary. Attention projections are where the model decides what to attend to and how to mix information across positions, so a small change there can redirect behaviour substantially. Applying LoRA to all four projections usually beats applying it to query and value alone, though the gap narrows as rank increases.

A useful mental model: the rank r controls how many independent directions of change you allow. Rank 1 lets the update move along a single direction in weight space. Rank 16 allows 16. For many tasks, performance rises quickly with r at first, then flattens. If your task needs a genuinely new capability, rather than a redirection of an existing one, the flattening point arrives later, and you may need a higher rank or a different method entirely. That flattening curve is itself diagnostic: it tells you how much of the task was already latent in the pretrained model.

LoRA is also commonly applied to feed-forward layers in some setups, and to embeddings far less often. The reason is practical rather than principled: attention and feed-forward matrices dominate the parameter count, so freezing them and adapting them cheaply is where the saving lives. Adapting a small embedding matrix buys little.

A minimal implementation sketch

The code below strips LoRA to its structure: a frozen linear layer plus a trainable low-rank branch. It is a sketch, not a production library, but it shows exactly where the trainable parameters live.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, base_linear, r=16, alpha=32):
        super().__init__()
        self.base = base_linear
        # Freeze the pretrained weights: no gradient flows here.
        for param in self.base.parameters():
            param.requires_grad = False
        d, k = base_linear.weight.shape
        self.A = nn.Parameter(torch.randn(r, k) * 0.02)
        self.B = nn.Parameter(torch.zeros(d, r))
        self.scaling = alpha / r

    def forward(self, x):
        base_out = self.base(x)
        lora_out = (x @ self.A.T @ self.B.T) * self.scaling
        return base_out + lora_out

Two things are worth noticing in this sketch. First, the base weights are frozen by setting requires_grad = False, so the optimiser never touches them, and the memory used by optimiser state (for example, Adam moments) is only spent on A and B. Second, the adapter branch is a pure addition. That property is what makes merging possible, as the next section shows.

Merging: the adapter can disappear at inference time

Because LoRA is additive, you can fold the trained adapter back into the base weights after training:

$$W = W_0 + \frac{\alpha}{r} B A$$

After merging, inference costs exactly the same as the original model. There is no extra layer, no extra latency, and no adapter to load at serving time. This is a real advantage over adapter methods that insert new modules into the network and pay for them on every forward pass forever.

Merging has a trade-off, though. Once merged, the adapter is no longer swappable. If you need to serve many tasks from one base model and switch between them quickly, keep the adapters separate and add the low-rank branch at inference time, accepting the small extra computation. If you serve one task, merge and forget the adapter ever existed. Many teams do both: keep unmerged adapters in training and evaluation, merge only the final deployed copy.

Why it works as well as it does

Three reasons explain LoRA's success, and they are worth separating because they fail in different ways.

First, pretrained models are heavily over-parametrised for any single downstream task. The knowledge needed for the task is a small subset of what the model knows, so the update that extracts it need not be large in rank.

Second, gradient descent on a low-rank parametrisation is not the same as gradient descent on the full matrix followed by truncation. The factorisation $B A$ imposes structure throughout training, which acts as a form of regularisation. It constrains the search to updates that are globally low rank, which tends to preserve the pretrained model's general behaviour better than an unconstrained update that can move in every direction at once.

Third, freezing $W_0$ prevents catastrophic forgetting by construction in the weights themselves. The base knowledge is still there, untouched. What changes is a small correction on top of it. That does not guarantee the model's outputs stay close to the pretrained model's outputs, since a small weight change can still have a large effect through many layers, but it removes the most direct route to forgetting: overwriting the weights that encoded the original knowledge.

Limitations and common pitfalls

LoRA is not a universal replacement for full fine-tuning. Its limits are instructive.

• New knowledge is hard to inject. If the task requires facts or skills that are absent from pretraining, a low-rank update may not have enough capacity to add them. Retrieval or continued pretraining may be needed instead of, or alongside, LoRA.
• Rank is not free. Very high rank erodes the memory and storage savings, and at some point full fine-tuning or a different parametrisation becomes simpler. If you find yourself pushing r into the hundreds, question whether LoRA is still the right tool.
• Which matrices you adapt matters more than you expect. Adapting only one projection, or adapting with too low a rank, can look like a failure of LoRA when it is really a failure of the configuration. Check the rank curve before concluding the method does not work for your task.
• Merged adapters cannot be unmerged cleanly if you lose the original. Keep the base weights and the adapter separately in version control. A merged model is a derived artefact, not a replacement for its ingredients.
• Training hyperparameters do not transfer unchanged. LoRA often needs a higher learning rate than full fine-tuning, because far fewer parameters carry the whole update. Copying a full fine-tuning recipe verbatim and concluding LoRA underperforms is a common, avoidable mistake.
• Evaluation must check forgetting, not just task accuracy. A LoRA adapter that aces the new task while badly damaging general capability has not succeeded. Evaluate on held-out general tasks as well as the target task.

A related pitfall is treating the adapter as a black box. Because the adapter is small, you can actually inspect it: look at the singular values of $B A$ , compare adapters trained for related tasks, and check whether two adapters interfere when combined. That inspectability is one of the underused benefits of keeping the update low rank and explicit.

When to choose LoRA

Choose LoRA when you need to adapt a large model cheaply, serve many task-specific variants from one base model, or fine-tune on hardware that cannot hold optimiser state for the full model. It is the default first attempt for most supervised fine-tuning of large pretrained models today, and that default is well earned.

Choose something else when the adaptation requires large-scale new knowledge, when you have the compute for full fine-tuning and need its last few percent of performance, or when the base model itself is small enough that the savings do not matter. For a small model, full fine-tuning is simple, well understood, and cheap. LoRA's machinery pays off most when the base model is large.

A practical rule: start with rank 16, $\alpha = 32$ , applied to all attention projections, with a learning rate higher than you would use for full fine-tuning. Measure the task metric and a general capability metric. Then vary rank, not everything at once. If rank 8 and rank 32 give nearly the same result, your task was already mostly inside the pretrained model, and the smaller adapter is the honest choice.

Takeaway

Low-Rank Adaptation works because fine-tuning is usually not about teaching a model something entirely new. It is about redirecting what the model already knows, and redirection does not need many degrees of freedom. By writing the update as $B A$ , freezing the base weights, and scaling by $\alpha / r$ , LoRA makes that redirection cheap, swappable, and mergeable, at a fraction of the cost of full fine-tuning.

The deeper lesson generalises beyond LoRA. When a model is large and pretrained, the interesting question is rarely how many parameters you can afford to update. It is how few directions of change the task actually needs, and whether you can find them. LoRA is one clean answer to that question, and its success is a reminder that capacity and adaptability are not the same thing.

---

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial