Full fine-tune vs LoRA vs QLoRA on one L40S: Qwen2.5-1.5B, memory, speed and held-out loss, measured

"Use LoRA, it needs far less memory" and "use QLoRA if LoRA doesn't fit" are the standard advice for fine-tuning on one GPU. I wanted to see what each one costs on a small model, where the weights are not the problem, so I rented an L40S (48 GB) on Runcrate today (2026-10-28) and fine-tuned Qwen2.5-1.5B-Instruct three ways on the same data for the same number of steps. Two machines, both deleted when the results were on disk.

Short version: full fine-tuning needed 36.9 GiB at micro-batch 4 and ran out of memory at micro-batch 8. LoRA needed 14.5 GiB, trained 1.4x faster and reached a slightly lower held-out loss. QLoRA was slower than both and used more memory than LoRA, not less.

Setup

• GPU: one L40S (48 GB, 44.4 GiB usable), Kansas City, US, $1.14 per hour on demand at launch time.
• Model: Qwen/Qwen2.5-1.5B-Instruct (1.54 billion parameters).
• Data: yahma/alpaca-cleaned, shuffled with a fixed seed: 4,000 examples for training (753,486 tokens), the next 300 held out. Qwen chat template, loss on the answer tokens only, at most 512 tokens per example.
• Training: one epoch, 250 optimizer steps of 16 examples each, cosine schedule with 10 warm-up steps, gradient clipping at 1.0, AdamW, no gradient checkpointing. A plain PyTorch loop, the same for all three:
  • full: fp32 master weights, bf16 autocast, learning rate 1e-5;
  • LoRA: bf16 base, rank 16, alpha 32, dropout 0.05 on all attention and MLP projections (18.5 M trainable parameters, 1.2% of the model), learning rate 2e-4;
  • QLoRA: the same LoRA on a 4-bit NF4 base (double quantization, bf16 compute, bitsandbytes 0.50.2), prepared with peft's prepare_model_for_kbit_training.
• Two runs: micro-batch 8 with 2 accumulation steps first. Full fine-tuning ran out of memory there, so a second machine ran all three at micro-batch 4 with 4 accumulation steps (still 16 per step).
• Measured: training wall time (excluding model load and evaluation), tokens per second, peak memory from torch.cuda.max_memory_allocated, loss on the 300 held-out examples before and after.
• Software: torch 2.8.0+cu128, transformers 4.57.1, peft 0.17.1, accelerate 1.15.0, Python 3.12.

The commands

Launch (the instance_type_id comes from GET /api/v1/instances/types):

curl -s https://www.runcrate.ai/api/v1/instances \
  -H "Authorization: Bearer $RUNCRATE_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"ft-l40s","instance_type_id":"<id from /instances/types>",
       "ssh_key_id":"<your key id>","max_runtime_minutes":60}'

On the machine:

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.12 /opt/env && . /opt/env/bin/activate
uv pip install torch 'transformers==4.57.1' 'peft==0.17.1' bitsandbytes accelerate --torch-backend=cu128
for CFG in full lora qlora; do MB=4 ACC=4 python ft_bench.py $CFG > results/${CFG}_mb4.json; done

The parts of ft_bench.py that differ between the three:

if CFG == "full":
    model = AutoModelForCausalLM.from_pretrained(M, torch_dtype=torch.float32).cuda()
    params, lr = list(model.parameters()), 1e-5
else:
    if CFG == "qlora":
        q = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                               bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
        model = AutoModelForCausalLM.from_pretrained(M, quantization_config=q, device_map={"": 0})
        model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=False)
    else:
        model = AutoModelForCausalLM.from_pretrained(M, torch_dtype=torch.bfloat16).cuda()
    model = get_peft_model(model, LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
    params, lr = [p for p in model.parameters() if p.requires_grad], 2e-4
opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
# training step: with torch.autocast("cuda", dtype=torch.bfloat16): loss = model(...).loss

Then DELETE /api/v1/instances/<id> and a check that the instance list is empty.

Results

Method	Micro-batch x accumulation	Trainable	Train time	Tokens/s	Peak memory (PyTorch)	Held-out loss	GPU cost of training
full	4 x 4	1,543.7 M	210 s	3,595	36.9 GiB	1.103 to 1.065	$0.066
LoRA	4 x 4	18.5 M	152 s	4,970	14.5 GiB	1.102 to 1.058	$0.048
QLoRA	4 x 4	18.5 M	215 s	3,507	16.5 GiB	1.144 to 1.090	$0.068
full	8 x 2	1,543.7 M	out of memory
LoRA	8 x 2	18.5 M	161 s	4,680	25.8 GiB	1.102 to 1.056	$0.051
QLoRA	8 x 2	18.5 M	237 s	3,184	30.7 GiB	1.144 to 1.089	$0.075

The cost column is training time only, at $1.14 per hour. The two machines were billed $0.18 and $0.24 in total, including boot, installs, downloads, evaluation and the run that failed.

All three models gave nearly the same answers to three fixed prompts afterwards. For "Rewrite this sentence in the passive voice: The cat chased the mouse." all three answered "The mouse was chased by the cat."

What surprised me

1. Full fine-tuning of a 1.5B model does not fit in 48 GB at micro-batch 8, without gradient checkpointing. fp32 weights, fp32 gradients and the two AdamW states are 16 bytes per parameter, about 23 GiB before a single activation. At micro-batch 4 it peaked at 36.9 GiB. The OOM came in the backward pass with 39.6 GiB already allocated.
2. QLoRA saved nothing here. On a 1.5B model the bf16 weights are only about 3 GiB, so quantizing them to 4 bits can save at most about 2 GiB, and the activations at these batch sizes are much larger than that. QLoRA's peak was 2 GiB higher than LoRA's at micro-batch 4 and 4.9 GiB higher at micro-batch 8. peft's prepare_model_for_kbit_training casts every layer that is not quantized to fp32, including Qwen's 151,936-token embedding, and the 4-bit layers are dequantized on every forward and backward pass; my guess is that these two account for the extra memory and for QLoRA being the slowest of the three, but I did not profile it.
3. Quantizing the base model cost almost as much loss as the fine-tune removed. QLoRA's held-out loss before training was 1.144, against 1.102 for the bf16 model, a 0.042 penalty. Training removed 0.054, about as much as LoRA removed (0.044), so QLoRA finished at 1.090, worse than where LoRA and full fine-tuning finished.
4. LoRA matched full fine-tuning on held-out loss (1.058 against 1.065) with 1.2% of the parameters trainable, while training 1.4x faster and using 40% of the memory. On this data and at these learning rates, full fine-tuning bought nothing.
5. Halving the micro-batch made LoRA faster, not slower (152 s against 161 s) and cut its peak memory from 25.8 to 14.5 GiB. The examples vary in length and each micro-batch is padded to its longest one, so smaller micro-batches probably waste less compute on padding.

What to take from it

• For a model this size on a 48 GB card, plain LoRA in bf16 is the default to beat: least memory, fastest, and no worse on held-out loss than full fine-tuning here.
• QLoRA is a tool for models whose bf16 weights don't fit. Below a few billion parameters, the activations dominate, and the 4-bit base costs speed and starting quality for no memory back.
• If you need full fine-tuning, micro-batch and gradient checkpointing decide whether it fits; check the peak memory on a few steps before you rent a bigger GPU.

Caveats

• One run per setting, one seed, one GPU type, one day. Treat differences under about 10% as noise, and the loss differences between full and LoRA as a tie.
• One epoch on 4,000 Alpaca examples, starting from an instruct model that already follows instructions, so the loss moves only a little. A harder task or more data could favour full fine-tuning.
• The learning rates (1e-5 for full, 2e-4 for LoRA) are common defaults, not tuned. Full fine-tuning at a higher rate might do better.
• No gradient checkpointing, no fused or 8-bit optimizers, no FlashAttention 2 (PyTorch SDPA). Each would change the memory numbers.
• Price is the on-demand price at launch time and moves daily.

Reproduce it

1. Create an API key and add an SSH key (POST /api/v1/ssh-keys).
2. Pick an L40S from GET /api/v1/instances/types and launch with the curl above, max_runtime_minutes set as a safety net.
3. Run the block above, copy results/ back, delete the instance.

If you want to run it yourself: runcrate.ai

Written by the Runcrate team, with AI assistance. The numbers come from the run described above; raw logs are kept with the post.