"LoRA trains about 1% of the model" is the usual one-line explanation. It's true, but it hides the parts that decide whether a run fits on your machine and how long it takes: which layers get adapters, what the rank does to the count, what precision the frozen weights sit in, and how many optimizer steps a small dataset really gives you.
Here are those numbers worked out for one concrete run: Qwen2.5-1.5B-Instruct, rank 16, on a 16GB Apple Silicon Mac. The settings are the defaults of the local trainer in the osFoundry desktop app, which is a plain transformers + PEFT + TRL script, so the arithmetic carries over to any similar setup. (The app's LoRA fine-tuning page shows the UI around it.)
One adapter is two thin matrices
For a weight matrix W of shape d_out × d_in, LoRA freezes W and learns two matrices, A (r × d_in) and B (d_out × r). The layer's output becomes W·x + (α/r)·B·A·x. B starts at zero, so training begins from exactly the base model.
The trainable count for that layer is r × (d_in + d_out). It grows linearly with the rank and with the layer's width, not with the layer's area.
Where the adapters go
The trainer attaches adapters to all seven linear projections in each transformer block:
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"]
Qwen2.5-1.5B has 28 blocks, a hidden size of 1536, an MLP size of 8960, and grouped-query attention with 2 KV heads of 128 dimensions, so k and v project down to 256. Per block, at r = 16:
| Layer | d_in → d_out | Trainable |
|---|---|---|
| q_proj | 1536 → 1536 | 49,152 |
| k_proj | 1536 → 256 | 28,672 |
| v_proj | 1536 → 256 | 28,672 |
| o_proj | 1536 → 1536 | 49,152 |
| gate_proj | 1536 → 8960 | 167,936 |
| up_proj | 1536 → 8960 | 167,936 |
| down_proj | 8960 → 1536 | 167,936 |
That's 659,456 per block, and 18,464,768 over 28 blocks: 18.5M trainable against 1.54B total, or 1.2%.
Two things stand out. The MLP projections hold three quarters of the adapter weights, so leaving them out (as some older recipes do, adapting only q and v) cuts the count to about 2.2M, an eighth of the capacity. And because GQA makes k and v narrow, their adapters are cheap.
Most of the memory is the frozen weights
On Apple's GPU the trainer loads the model in float32:
torch_dtype = torch.float16 if device == "cuda" else torch.float32
Half precision on MPS tends to produce NaN losses, so 32-bit is the safe choice there. That makes the frozen base 1.54B × 4 bytes ≈ 6.2 GB.
The trainable part is small by comparison. 18.5M weights in fp32 is 74 MB, its gradients another 74 MB, and AdamW keeps two moment estimates per weight, another 148 MB. About 0.3 GB in all. The rest of the budget goes to activations, which scale with batch size and sequence length (the trainer caps sequences at 2048 tokens; short chat replies are far below that).
With about 12 GB of a 16 GB Mac usable by the GPU, 1.5B at fp32 leaves room. A 7B base at fp32 would be about 30 GB before training starts, which is why on a Mac the base model stays small, and why QLoRA (4-bit frozen weights) exists, though the bitsandbytes 4-bit path needs an NVIDIA GPU, so on a Mac it is plain LoRA.
A small dataset is fewer updates than you think
The trainer uses batch size 4 with 4 gradient-accumulation steps, so each optimizer update sees 16 examples. With 200 pairs:
- 200 ÷ 4 = 50 forward/backward batches per epoch
- 50 ÷ 4 ≈ 13 optimizer updates per epoch
- 3 epochs ≈ 39 updates in total, the first 10 of them a learning-rate warmup
Thirty-nine updates at a learning rate of 2e-4 with a linear decay is enough to shift tone, not enough to teach new knowledge. That matches what people see in practice: a couple of hundred replies change how a model writes, not what it knows.
What comes out
After training, only the adapter is saved: A and B for each of the 196 adapted layers, tens of megabytes. For inference the app converts it to a GGUF LoRA file and llama.cpp applies it on top of a quantized copy of the base, here qwen2.5-1.5b-instruct at Q4_K_M. The base stays shared, so several adapters (one per tone or task) cost a few dozen MB each.
Note the precision mismatch: the adapter was trained against fp32 base weights and runs against 4-bit ones. For a small style adapter the difference is usually not noticeable, but it's one reason a merged, re-quantized model can behave slightly differently from the same adapter applied at runtime.
Top comments (0)