DEV Community

Cover image for QLoRA hyperparameters that actually matter (rank, alpha, learning rate, epochs)
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

QLoRA hyperparameters that actually matter (rank, alpha, learning rate, epochs)

#qlora #hyperparameters #finetuning #llm

The hyperparameter page in most guides lists twenty knobs and explains none. After many runs building and testing FineTune Studio on small models, here is the short list of settings that changed outcomes, with defaults that work for 1B–8B models on a single GPU.

Rank and alpha

Rank (r) sets the capacity of the adapter; alpha scales its contribution. For behaviour and format tasks, r = 8 to 16 is plenty. Raise it to 32–64 for tasks that need more change, such as a new output language or dense domain knowledge. A common, sensible pairing is alpha = 2 × r. Doubling rank doubles adapter size but rarely doubles quality.

Target modules

Attaching adapters only to attention projections is the classic setup. Including the MLP layers as well generally improves results at modest memory cost, and is my default for QLoRA on small models where memory is already handled by quantisation.

Learning rate

The single most sensitive knob. Adapters tolerate higher rates than full fine-tuning; 1e-4 to 2e-4 with a short warmup and cosine or linear decay is a reliable range. If the loss curve drops off a cliff then climbs, the rate is too high. If it barely moves over an epoch, too low.

Epochs versus dataset size

  • A few hundred examples: 3–5 epochs, watch for the eval loss turning upward.
  • A few thousand examples: 1–3 epochs is usually enough.
  • Tens of thousands: one epoch, and consider whether you need all of it.

Batch size and gradient accumulation

On a small GPU, per-device batch of 1–4 with gradient accumulation to an effective batch of 16–32 is the normal pattern. Accumulation trades time for memory; it does not change the maths. Sequence length is the hidden memory hog: cap it at what your data needs, not at the model's maximum.

Settings people over-tune

Dropout on adapters, weight decay, scheduler exotica. They matter at the margins. Reading the loss curve — see how to read a loss curve during fine-tuning — matters far more than any of them.

Start with the defaults above, change one thing at a time, and let a held-out evaluation, not the training loss, tell you what improved.

About Pranjul Rathour

Pranjul Rathour presenting evaluation criteria — feasibility, innovation, practicality, problem solving — on a projector screen
Walking a room through evaluation criteria

Pranjul Rathour presenting BrandHive on a projector screen
Presenting BrandHive

Pranjul Rathour in a shirt and tie holding a microphone in front of a career-opportunities slide
A career session for students

Pranjul Rathour presenting with a microphone in front of a slide reading 'Now what's the conclusion?'
Presenting to a room

Pranjul Rathour on stage presenting a requirements-gathering and user-flow slide
Requirements gathering, on stage

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-06.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)