LoRA & DoRA: The Math, Memory, and Trade-offs

Every engineer who has ever tried to fine-tune a modern 27B or 70B parameter model knows the rude awakening of GPU memory arithmetic.

You look at the raw model weights and think: “27 billion parameters stored in 16-bit brain floats is only 54 gigabytes. That easily fits on an 80 GB NVIDIA H100, right?” Then you hit run. The moment AdamW initializes, your VRAM explodes. Between forward activations, backward gradients, and FP32 optimizer moment tensors, that innocent 54 GB model suddenly demands over 324 gigabytes of VRAM before you have even finished your first training step.

That is why Parameter-Efficient Fine-Tuning (PEFT) is an economic necessity. LoRA freezes the monster model and trains two tiny additive matrices beside it. DoRA takes it a step further: it decouples the weight matrix into directional updates and explicit magnitude scales.

Start here

Count the trainable numbers in a tiny layer, see what DoRA changes geometrically, and build a precise memory estimate that includes the frozen base model.

• Parameter: One learned number in the model, such as an entry in a weight matrix.
• Matrix / vector: A rectangular table of numbers / a list of coordinates. A matrix multiplication maps one vector to another.
• Rank: The maximum number of independent directions the LoRA bottleneck correction can use.
• Gradient / optimizer state: A suggested parameter change / extra tensors the training optimizer keeps to decide updates.

A large model may need only a small trainable adapter to learn a new task. That does not make the frozen model disappear from memory: it removes most of the gradients and optimizer states associated with updating that model. Keeping those two ideas separate makes the memory savings much easier to calculate.

Below, we walk through the exact matrix algebra, explain why standard LoRA couples magnitude and direction, break down the VRAM math so you can budget your cluster without nasty surprises, and look at what published benchmarks actually prove.

1. LoRA: a small branch beside a frozen matrix

Let a linear layer map an input with k coordinates to an output with d coordinates. Its frozen weight matrix W₀ therefore has d rows and k columns. LoRA adds a trainable branch that first reduces the input to r coordinates, then expands it back to d:

$$\begin{aligned}W_0&\in\mathbb{R}^{d\times k}\\A&\in\mathbb{R}^{r\times k},\quad B\in\mathbb{R}^{d\times r}\end{aligned}$$

$$\begin{aligned}s&=\frac{\alpha}{r}\\W&=W_0+sBA\\y&=W_0x+sB(Ax)\end{aligned}$$

Read the branch from the input: A takes k numbers to r; B takes r numbers to d. With this column-vector convention, the product is BA. Here x is the input vector, y the output, W₀ the frozen matrix, A and B the trainable matrices, and s the adapter’s scaling factor. The update’s rank is at most r. This constrains the update, while W₀ itself can remain high rank. The α/r factor is the original LoRA scaling convention; it does not guarantee equal training behavior across ranks. LoRA, Hu et al.

For the small 8 × 6 layer in the figure below, rank 2 needs an A matrix with 2 × 6 = 12 entries and a B matrix with 8 × 2 = 16 entries: 28 trainable numbers instead of 48. At rank 4, the adapter needs 56 entries, so it no longer saves parameters in this tiny example. Small rank relative to layer dimensions is what makes the method economical.

With the usual random-A, zero-B initialization, the branch contributes zero before training. The layer starts with its original output. Initialization schemes can vary by implementation.

$$N_{\mathrm{full}}=dk,\qquad N_{\mathrm{LoRA}}=r(d+k)$$

For a 4,096 × 4,096 matrix at rank 16, that is 16,777,216 dense parameters versus 131,072 adapter parameters: 128× fewer trainable parameters for this layer. Across a model, sum over the layers you actually adapt:

$$N_{\mathrm{LoRA,total}}=\sum_{\ell\in\mathcal{T}}r_\ell(d_\ell+k_\ell)$$

Here, T is the set of targeted layers. Trainable biases, embeddings, or output heads must be counted separately. A model’s total parameter count and adapter rank alone are not enough to infer “0.1% trainable.”

LoRA trains B (8 × r) and A (r × 6) instead of the 48 frozen entries of W₀. The saving holds only while r stays small: at rank 4 the adapter is larger than the layer.

2. DoRA: separate direction from scale

Consider one column of a weight matrix as an arrow. For example, (1, 0) points right with length 1. Adding (0, 1) gives (1, 1), which points diagonally and has length √2. Adding a vector can change both its length and direction. DoRA makes the final scale an explicit parameter: first normalize the updated column, then multiply it by a learned scalar. DoRA, Liu et al., Equation 5.

$$\begin{aligned}V'&=W_0+sBA\\w'_j&=m_j\frac{v'_j}{\lVert v'_j\rVert_2}\end{aligned}$$

The subscript j selects a column. The vector v′ is its candidate direction; m is its separately learned length scale. Dividing (1, 1) by √2 makes it unit length; multiplying by m = 2 sets its length to 2 without changing where it points. Its denominator is that column’s Euclidean norm, not one Frobenius norm for the entire matrix. In the matrix shorthand below, division and multiplication apply independently to each column. The symbol ⊙ means multiplying corresponding entries:

$$\begin{aligned}W'&=m\odot\frac{V'}{\lVert V'\rVert_c}\\(\lVert V'\rVert_c)_j&=\sqrt{\sum_{i=1}^{d}(V'_{ij})^2}\end{aligned}$$

This uses the paper’s column convention, with m shaped 1 × k. Some libraries store or normalize weights along a different axis; check the implementation’s tensor layout before translating the formula into code. The division assumes a nonzero column norm.

$$\begin{aligned}m_{j,0}&=\lVert w_{0,j}\rVert_2,\quad B_0=0\\W'_{\mathrm{init}}&=W_0\\N_{\mathrm{DoRA}}&=r(d+k)+k\end{aligned}$$

In the 4,096 × 4,096 example, DoRA adds 4,096 scales, bringing the total to 135,168 trainable parameters. The low-rank branch controls the candidate directions; normalization and the separate scales determine the final columns.

Rotation does not have to increase length. An additive update can send (1, 0) to (0, 1) without changing its norm. It can also shrink or grow the vector. DoRA gives scale its own parameter; it does not fix a mathematical impossibility in LoRA. And because DoRA rescales the base columns, its final merged update need not have rank r.

One weight column as an arrow. LoRA's additive update moves length and angle together; DoRA normalizes the updated column and gives its length a separate learned scale m. An additive update can also rotate without changing length.

3. Dimensions, geometry, and memory at a glance

The two figures above count the entries of a small layer as the rank grows and follow one weight column through LoRA and DoRA. The figure below compares memory for full fine-tuning, LoRA, and DoRA on one scale; the next section spells out the assumptions behind every number.

State storage for an illustrative 27B model on one scale. LoRA keeps the 54 GB frozen base but drops almost all gradients and optimizer state; activations and buffers are not included.

The original post has an interactive version of all three views.

4. Memory math needs explicit assumptions

Consider an illustrative 27-billion-parameter model. Use decimal GB (10⁹ bytes), BF16 (2-byte) weights and gradients, and two FP32 (4-byte each) Adam moment tensors. Adam’s moments summarize recent gradients and their squared values. Assume everything is resident on the device and initially exclude a separate FP32 master copy of trainable weights. These are accounting assumptions, not measured peak VRAM for a particular Qwen checkpoint or training library.

$$\begin{aligned}M_{\mathrm{full}}&=(2+2+8)P\ \mathrm{bytes}\\&=54+54+216\ \mathrm{GB}\\&=324\ \mathrm{GB}\end{aligned}$$

An FP32 master copy adds 4 bytes per trainable parameter: another 108 GB here, bringing the state subtotal to 432 GB. FP32 gradients would add a further 54 GB relative to BF16 gradients. Actual optimizer-state dtypes depend on the training stack. These are total allocations, not one contiguous 324 GB tensor.

For LoRA, retain the frozen base weights and count weights, gradients, and optimizer states only for q trainable adapter parameters:

$$\begin{aligned}M_{\mathrm{LoRA}}&=2P+(2+2+8)q\ \mathrm{bytes}\\q&=32\times10^6\\M_{\mathrm{LoRA}}&=54+0.384=54.384\ \mathrm{GB}\end{aligned}$$

Illustrative state accounting · no FP32 master copy · decimal GB

Allocation	Full fine-tuning	LoRA, q = 32M
Base weights	54	54
Adapter weights	0	0.064
Gradients	54	0.064
Adam moments	216	0.256
State subtotal	324	54.384

The 32M adapter count is an assumption, not a consequence of choosing rank 16 or 32. The memory figure’s DoRA example adds an explicitly assumed 0.2M scales, for a subtotal of 54.3864 GB. Replace these counts with your actual trainable tensors when budgeting.

Activations still matter. Frozen layers still participate in the forward pass, and gradients must reach adapters. Sequence length, batch size, checkpointing, temporary buffers, and allocator overhead can dominate the remaining memory. Sharding or offloading can reduce device residency. A subtotal below a GPU’s capacity does not establish that a training run fits.

What changes with QLoRA?

QLoRA combines trainable adapters with a quantized frozen base. Its original design uses NF4, double quantization, and paged optimizers. QLoRA, Dettmers et al. If every base parameter were stored at exactly four bits, the raw payload would be:

$$M_{\mathrm{4bit,raw}}=P\frac{4}{8}\ \mathrm{bytes}=13.5\ \mathrm{GB}$$

Quantization metadata, tensors retained at higher precision, adapters, activations, and dequantization buffers come on top. Four-bit storage also does not mean every arithmetic operation runs in four-bit precision.

5. What the published results support

The DoRA paper reports these LLaMA-7B averages across eight commonsense reasoning tasks. The values below come from arXiv v6, Table 1. They are not measurements from our 27B memory example.

LLaMA-7B · eight-task commonsense average · authors’ reported results

Method	Trainable parameters	Average accuracy
LoRA	0.83%	74.7%
DoRA, half rank	0.43%	77.5%
DoRA	0.84%	78.4%

These results support an improvement over LoRA in this setting. They do not establish universal parity with full fine-tuning; this comparison has no full-fine-tuning row. The paper also adjusts learning rates, so these are method configurations rather than an experiment changing only one algebraic operation.

6. Training and serving have different costs

At inference, a merged adapter can be represented by one dense weight matrix:

$$\begin{aligned}W_{\mathrm{LoRA,merged}}&=W_0+sBA\\W_{\mathrm{DoRA,merged}}&=m\odot\frac{W_0+sBA}{\lVert W_0+sBA\rVert_c}\end{aligned}$$

Merging removes the separate adapter operations from that linear layer. It does not imply that every serving engine accepts an unmerged DoRA adapter. Quantized merging may require dequantization and requantization; check output agreement in the precision you intend to serve.

During training, DoRA’s normalization adds work. Implementations can detach the norm from the backward graph, so the forward formula alone is not a complete description of the gradient computation. Review the actual implementation and benchmark quality, peak memory, and time to a target score before selecting a default. Hugging Face PEFT reference.

---

Originally published at g-ftech.com.

Comments

[Dev Support]

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‍​