DEV Community

Phil IPS
Phil IPS

Posted on

Fine-Tuning LLMs on Consumer Hardware: LoRA Guide 2024

#llm #finetuning #lora #tutorial

Fine-Tuning LLMs on Consumer Hardware: LoRA Guide 2024

Train custom models on your GPU using LoRA/QLoRA — no cloud credits needed.

Introduction

Fine-tuning used to require A100s and thousands of dollars. With LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA), you can fine-tune 7B-70B models on a single 24GB GPU (RTX 3090/4090) or even 12GB with quantization.

What is LoRA?

LoRA freezes the base model and trains only small adapter matrices:

Base Model (frozen) + LoRA Adapters (trainable) = Fine-tuned Model
Enter fullscreen mode Exit fullscreen mode

Benefits:

  • 100-1000x fewer trainable parameters
  • Fast training (minutes to hours)
  • Small checkpoint files (MBs not GBs)
  • Mergeable back into base model

Hardware Requirements

Model Size Method VRAM Needed GPU Example
7B LoRA (fp16) ~16 GB RTX 3090/4090
7B QLoRA (4-bit) ~8 GB RTX 3060 12GB
13B QLoRA (4-bit) ~12 GB RTX 3080 12GB
33B QLoRA (4-bit) ~20 GB RTX 3090/4090
70B QLoRA (4-bit) ~40 GB Dual 3090/4090

Setup Environment 

# Create environment
conda create -n lora python=3.10 -y
conda activate lora

# Install dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers peft trl accelerate bitsandbytes datasets wandb
Enter fullscreen mode Exit fullscreen mode

Prepare Dataset 

// data/train.jsonl
{"text": "### Instruction:\nWrite a Python function to reverse a string\n### Response:\ndef reverse_string(s):\n    return s[::-1]"}
{"text": "### Instruction:\nExplain what a decorator does in Python\n### Response:\nA decorator is a function that takes another function and extends its behavior..."}
Enter fullscreen mode Exit fullscreen mode

Format: Alpaca/ShareGPT style with ### Instruction: and ### Response: markers.

Training Script (QLoRA) 

# train_lora.py
import os
import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# Config
MODEL_ID = "mistralai/Mistral-7B-v0.1"  # or "meta-llama/Llama-2-7b-hf"
OUTPUT_DIR = "./mistral-7b-code-lora"
DATA_PATH = "data/train.jsonl"

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=64,                    # Rank (higher = more capacity)
    lora_alpha=16,           # Scaling factor
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Trainable params: ~0.5% of total

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# Dataset
dataset = load_dataset("json", data_files=DATA_PATH, split="train")

# Training arguments
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=10,
    max_steps=100,           # Adjust for your dataset
    learning_rate=2e-4,
    fp16=True,
    logging_steps=5,
    save_strategy="steps",
    save_steps=50,
    report_to="wandb",       # or "none"
    optim="paged_adamw_8bit"
)

# Trainer
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
    args=training_args,
    packing=False
)

# Train
trainer.train()

# Save
trainer.model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)
print(f"Model saved to {OUTPUT_DIR}")
Enter fullscreen mode Exit fullscreen mode

Run Training 

# Single GPU
python train_lora.py

# Multi-GPU (if available)
accelerate launch --multi_gpu train_lora.py
Enter fullscreen mode Exit fullscreen mode

Merge & Export 

# merge_lora.py
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

BASE_MODEL = "mistralai/Mistral-7B-v0.1"
LORA_PATH = "./mistral-7b-code-lora"
OUTPUT_PATH = "./mistral-7b-code-merged"

# Load base
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load & merge LoRA
model = PeftModel.from_pretrained(model, LORA_PATH)
model = model.merge_and_unload()

# Save merged
model.save_pretrained(OUTPUT_PATH)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_pretrained(OUTPUT_PATH)
Enter fullscreen mode Exit fullscreen mode

Convert to GGUF (llama.cpp) 

# Install llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert
python convert-hf-to-gguf.py ../mistral-7b-code-merged \
  --outfile ../mistral-7b-code.Q4_K_M.gguf \
  --outtype q4_k_m
Enter fullscreen mode Exit fullscreen mode

Inference with Ollama 

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./mistral-7b-code.Q4_K_M.gguf
TEMPLATE "{{ .System }}\n\n{{ .Prompt }}"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

# Create model
ollama create mistral-code -f Modelfile

# Test
ollama run mistral-code "Write a Python decorator for timing functions"
Enter fullscreen mode Exit fullscreen mode

Cost Breakdown (RTX 3090 24GB)

Item Cost
Electricity (4 hrs @ 350W) ~$0.20
GPU depreciation ~$1.00
Total per run ~$1.20

vs. Cloud (A100 40GB): $1.50-3.00/hr → $6-12 for same training.

Tips for Success

  1. Dataset quality > quantity — 500 clean examples beat 5000 noisy ones
  2. Start small — 100 steps, check loss, then scale
  3. Monitor VRAM — Reduce batch size if OOM
  4. Use gradient checkpointing — model.gradient_checkpointing_enable()
  5. Eval regularly — Keep a validation set

Common Issues

Error Fix
CUDA OOM Reduce batch size, increase grad accumulation, use 4-bit
Loss not decreasing Increase learning rate, check data format
NaN loss Lower learning rate, add gradient clipping
Slow training Enable flash attention, use packing=True

Conclusion

LoRA/QLoRA democratized LLM fine-tuning. With a consumer GPU, you can create domain-specialized models for coding, writing, analysis — whatever your use case. Start with 7B QLoRA, iterate on data, and deploy locally via Ollama or llama.cpp.

Next: Try DPO/ORPO for preference alignment, or merge multiple LoRAs for multi-task models.

Part of the LLM Fine-Tuning series on Dev.to

Top comments (0)