How I Fine-Tuned a 7B LLM with LoRA and Unsloth

How I Fine-Tuned a 7B LLM with LoRA and Unsloth

When I was working on Khatta-ka-LLM, two technologies became especially important:

LoRA + Unsloth

If you've heard these names but aren't sure what they actually do, here's a simple explanation.

What is LoRA?

LoRA stands for Low-Rank Adaptation.

Instead of updating all the parameters of a large language model during fine-tuning, LoRA keeps the original model weights frozen and trains a much smaller set of additional parameters.

Think of it like this:

Full fine-tuning

Entire model
↓
Update billions of parameters

LoRA

Original model
↓
Keep the original weights
↓
Train small adapter matrices

The result is a much more parameter-efficient fine-tuning process.

This is especially useful when you don't have access to expensive GPU clusters.

Where does Unsloth come in?

Unsloth is a framework designed to make LLM fine-tuning more efficient.

It focuses on reducing memory usage and improving training speed, making techniques such as LoRA and QLoRA much more practical.

For my Khattak Pashto experiment, my workflow was essentially:

Qwen2.5-7B
↓
LoRA fine-tuning
↓
Unsloth optimization
↓
Custom Khattak Pashto dataset
↓
Fine-tuned model

The goal wasn't to teach the model language from scratch.

The base model already had broad language knowledge.

Instead, I wanted to adapt that existing knowledge toward a specific regional dialect by exposing it to carefully prepared examples.

What happened during training?

For my experiment, the training loss went from:

3.44 → 0.22

That doesn't mean the model became perfect.

Training loss alone isn't enough to prove that a model performs well in real-world usage. Proper evaluation and testing are still important.

But it was a great demonstration of what parameter-efficient fine-tuning could achieve for this experiment.

Why LoRA is interesting

The important idea behind LoRA is that you don't necessarily need to modify every parameter of a large model to adapt it to a new task or domain.

You can keep the original model and learn a relatively small set of trainable parameters.

That makes experimentation significantly more accessible.

For example, instead of thinking:

«"I need a massive GPU cluster to fine-tune an LLM."»

You can start thinking:

«"What is the smallest efficient fine-tuning approach that can solve my problem?"»

That's a very different way of approaching LLM engineering.

The bigger lesson

You don't always need massive infrastructure to experiment with LLMs.

Understanding concepts such as:

• Parameter-Efficient Fine-Tuning (PEFT)
• LoRA
• QLoRA
• Quantization
• Efficient training frameworks

can make LLM experimentation much more accessible.

That's one of the things I enjoyed most about building Khatta-ka-LLM.

The process was simple:

Build
↓
Experiment
↓
Measure
↓
Improve

If you're learning LLM fine-tuning, I'd recommend starting by understanding what LoRA is actually changing inside the model.

Once that clicks, the rest of the workflow becomes much easier to understand.
Want to see the full project?

I documented the complete Khatta-ka-LLM experiment, including the dataset, training approach, model architecture, and results, in a detailed case study on my website.

Read the full case study:
https://www.mzubair.online/blog/khattak-llm-case-study

What's next?

I can break down either:

LoRA vs QLoRA

or

How LoRA works under the hood mathematically.

Which one would you like to see next?