DEV Community: Suman Nath

If a 270M Model Already Worked, Why Did I Fine-Tune a 7B One?

Suman Nath — Sun, 21 Jun 2026 12:23:55 +0000

Over three posts I built three fine-tuned models for the same banking-intent task — full fine-tuning a 270M model, LoRA on 1.5B, QLoRA on 7B. They all landed around the same accuracy.

Which raises an honest, slightly uncomfortable question: if a 270M model on my laptop already worked, why reach for a 7B model at all?

The answer most "bigger is better" content skips

For this task — you wouldn't. A good engineer picks the smallest model that clears the bar, not the biggest one available. The small model is cheaper to serve, runs in milliseconds, and you fully own it. Choosing the 7B here would be over-engineering.

Reaching for a bigger model isn't a flex. It's a response to a requirement the small one can't meet. Here are the four cases where small stops being enough:

1. The task is genuinely hard

Banking77 is easy — 77 fixed labels, short clean queries. Small models saturate it. But ask for reasoning ("which of these three issues is the primary one?"), open-ended generation (write the reply, don't just classify), or real nuance, and there's a capability floor that more parameters buy. No amount of fine-tuning gives a 270M model abilities it doesn't have.

2. You have little data

I had ~10,000 labeled examples — plenty for a small model. With 50, a small model can't learn the task, but a 7B model already "knows" banking concepts from pretraining and only needs a nudge. Bigger models need less task data because they bring more prior knowledge.

3. You need one model for many tasks

This is the quiet superpower of LoRA/QLoRA. A single frozen 7B base can host dozens of swappable adapters — intent classifier, reply writer, summarizer, sentiment — all from one ~5GB footprint in memory. The 270M is single-purpose. This is why companies serve hundreds of fine-tunes from one base model.

4. Accuracy compounds at scale

93% means 7 in 100 queries misrouted. At 10M queries/month, that's 700,000 mistakes. If each costs a support escalation, the 2–3 points a bigger model buys can pay for itself many times over. At small scale, nobody notices. At large scale, it's the whole budget.

So why did I build all three?

Not to beat the small model — they tied. And that tie is the lesson: on an easy task, the technique barely matters.

I built them to learn the techniques, so that when I hit a task where small isn't enough, I can fine-tune a 7B model on a 16GB card without flinching. It's like learning to change a tire in your driveway on a sunny day. The driveway didn't need it — but now I can do it on the highway, in the rain.

The bug no model size could fix

One thread ran through all three projects. Every model — 270M, 1.5B, 7B — confused the same two intents: card_arrival and card_delivery_estimate. Three model scales, three techniques, the same mistake in every confusion matrix.

That's not a capacity problem you can buy your way out of. "Where's my card?" and "when will my card arrive?" genuinely overlap — the ambiguity is in the labels themselves, not the model. Three models agreeing on a "mistake" is a strong signal the data, not the model, is the limit.

Sometimes the answer isn't a bigger model. It's better data.

That might be the most useful thing I learned across the whole series.

The takeaway, as a decision

Is the small model good enough for the actual requirement?
   YES → ship it. Bigger is wasted cost, latency, and complexity.
   NO  → why not?
          capability ceiling? → bigger base model
          too little data?    → bigger base + LoRA (needs less data)
          many tasks at once? → one big base + swappable adapters

Match the model to the requirement. That instinct is worth more than any of the fine-tuning mechanics in Parts 1–3.

📓 All three notebooks on Kaggle: https://www.kaggle.com/work/collections/18659493

Thanks for reading the series. If it was useful, a reaction or a comment helps it reach the next person debugging their first OOM.

QLoRA: Fine-Tuning a 7B Model on a 16GB GPU (It Shrank to 5.4GB in Front of Me)

Suman Nath — Sun, 21 Jun 2026 12:20:53 +0000

In Part 2, LoRA let me fine-tune a 1.5B model by freezing it and training tiny adapters. But the frozen base still sat in memory in 16-bit (~3GB). Now I wanted to go to Qwen2.5-7B — and hit a wall that LoRA alone doesn't solve.

The problem

A 7B model is ~15GB in 16-bit precision. A free-tier T4 GPU has 16GB. It would barely load, with no room left to actually train.

The QLoRA insight

QLoRA asks the question that naturally follows from LoRA: the base is frozen and only ever read — so why store it in full precision?

So you quantize the frozen base to 4-bit (NF4, a format tuned for how neural-net weights are distributed) and run the LoRA adapters on top in normal precision. The base shrinks dramatically; the trainable part stays small and precise.

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # NormalFloat4
    bnb_4bit_use_double_quant=True,        # quantize the quant constants too
    bnb_4bit_compute_dtype=torch.float16,  # dequantize to fp16 for the matmuls
)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, quantization_config=bnb_config, device_map="auto")

Each flag earns its place:

load_in_4bit — store frozen weights in 4 bits instead of 16.
nf4 — a 4-bit type matched to the bell-curve distribution of neural-net weights (better than plain int4).
double_quant — quantize the quantization constants too, for a bit more savings.
compute_dtype — dequantize to fp16 for the actual matmuls, so storage is 4-bit but compute stays precise.

The moment it clicked

One line of output:

loaded in 4-bit. footprint: 5.44 GB

I downloaded 15.2GB of weights and they sat in memory as 5.44GB. A model that couldn't be loaded for full fine-tuning was now training on a single consumer GPU — with room to spare. (The download is still 15GB; bitsandbytes quantizes on the fly during load.)

The QLoRA-standard recipe

Two more pieces beyond Part 2's LoRA setup: prepare the quantized model for training, and target all linear layers (the QLoRA paper found this matters), with a paged 8-bit optimizer:

from peft import prepare_model_for_kbit_training
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)
# ... attach LoRA to every linear layer ...
TrainingArguments(optim="paged_adamw_8bit", gradient_checkpointing=True, ...)

It's slow — and that's fine

A 7B forward pass through 4-bit weights with gradient checkpointing is heavy: ~1 hour for one epoch on a T4, ~3 examples/second. But QLoRA isn't about speed — it's about fit. The model runs at all, on hardware that couldn't otherwise hold it. That's the entire point.

⚠️ Hardware note: bitsandbytes 4-bit is CUDA-first. It does not run on Apple MPS, and AMD/ROCm support exists but is less mature. Run this one on an NVIDIA GPU (Kaggle/Colab T4 works).

Result

QLoRA accuracy: 92.848% (4-bit base was 16.000%)
macro-F1: 0.928

It roughly tied the smaller models from Parts 1 and 2.

And the card_arrival vs card_delivery_estimate confusion that haunted both smaller models? [Say what happened at 7B — did it finally fix it, or hit the same wall?] Either way, it sets up the question I tackle in Part 4: if the 270M model already worked, why did I build any of this?

📓 Full runnable notebook on Kaggle: https://www.kaggle.com/code/sumannath88/03-qlora-qwen2-5-7b

Built with PyTorch + Transformers + PEFT + bitsandbytes. Questions or corrections welcome in the comments.

LoRA: I Trained <1% of a 1.5B Model and Matched a Full Fine-Tune

Suman Nath — Sun, 21 Jun 2026 12:17:16 +0000

In Part 1 I fully fine-tuned a 270M model — updating every weight. That's fine for a tiny model. It gets painful as models grow, because full fine-tuning needs gradients and optimizer state for every parameter (~4× the model size in memory).

So: what do you do when the model is too big to comfortably fine-tune all of?

The idea behind LoRA

LoRA (Low-Rank Adaptation) rests on one observation: the change fine-tuning makes to a weight matrix is "low rank" — it lives in a small subspace. You don't need to learn the full update ΔW; you can learn it as the product of two skinny matrices, B·A:

output = W·x  +  (B·A)·x
         ↑frozen    ↑trainable (tiny)

For a single 1536×1536 layer at rank 16, that's about 49,000 trainable numbers instead of ~2.4 million. And you freeze the entire base model — only the adapters train. B starts at zero, so at step 0 the model behaves exactly like the original and training nudges it from there.

The config

from peft import LoraConfig, get_peft_model, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                 # rank — adapter capacity
    lora_alpha=32,        # scaling; effective scale = alpha / r
    lora_dropout=0.05,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# -> trainable params are ~1% of the model. The other 99% is frozen.

I ran this on Qwen2.5-1.5B-Instruct — 5× bigger than the Gemma model from Part 1. Same Banking77 task. Then the GPU fought back.

Wall #1: `ValueError: Attempting to unscale FP16 gradients`

I'd loaded the model in fp16 to save memory. Wrong move: the optimizer needs fp32 master weights; mixed precision is applied at train time by the trainer, not baked into the loaded weights.

# load weights in fp32; let the Trainer's AMP do fp16 during training
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.float32)
# and set fp16=True in TrainingArguments (on CUDA) for the mixed-precision part

Wall #2: CUDA out of memory at batch size 64

Adapter training still holds activations and optimizer state. Fix: smaller batch + gradient accumulation (keeps the effective batch) + gradient checkpointing (recompute activations in the backward pass):

per_device_train_batch_size=16,
gradient_accumulation_steps=2,     # effective batch 32, lower peak memory
gradient_checkpointing=True,       # ~30% more compute, big memory savings

Wall #3: my laptop and a cloud GPU showed the same speed

This one was sneaky. My Mac (MPS) and a Kaggle T4 reported nearly identical it/s. How is a datacenter GPU no faster than a laptop?

It wasn't. The Kaggle session had 2 GPUs running data-parallel — each step processed 2× the data, so the total step count halved (626 vs 1250) while it/s stayed flat. The fix isn't code, it's how you read the number: compare examples/second, never iterations/second. Once I did, the GPU was clearly ~3× faster.

Result

~96% accuracy again — a frozen 1.5B model + a few-MB adapter matched the fully-fine-tuned 270M model from Part 1, with a saved artifact roughly 1000× smaller.

And that card_arrival vs card_delivery_estimate confusion from Part 1? Still there. Bigger model, different technique, identical mistake. (We resolve that mystery in Part 4.)

What's next

Part 3: I fit a 7-billion-parameter model onto a 16GB GPU that can't even load it normally. That's QLoRA.

📓 Full runnable notebook on Kaggle: https://www.kaggle.com/code/sumannath88/02-lora-qwen2-5-1-5b

Built with PyTorch + Hugging Face Transformers + PEFT. Questions or corrections welcome in the comments.

I Fine-Tuned a 270M Model on My Laptop (Full Fine-Tuning, From Scratch)

Suman Nath — Sun, 21 Jun 2026 12:08:45 +0000

I wanted to actually understand fine-tuning — not run a tutorial and nod along. So I gave myself a constraint: same task, three techniques, smallest model to largest. Full fine-tuning, then LoRA, then QLoRA. Hold the task fixed and the only variable is the method.

This first post is full fine-tuning — the most powerful and most expensive option: update every weight in the model.

The task

Banking77: ~13,000 real bank customer-support messages, 77 intents like card_arrival, lost_or_stolen_card, exchange_rate. The model reads a message and names the intent.

The model: deliberately tiny

I picked Gemma 3, 270M parameters — small enough to fully fine-tune on a laptop (Apple Silicon / MPS). That's intentional: full fine-tuning stores gradients and optimizer state for every parameter, roughly 4× the model's size in memory. I wanted to feel that, not read about it.

One design decision: generate the label, don't classify it

The obvious approach is to bolt a 77-way classification head onto the model. I didn't. Instead I had the model generate the intent as text — literally output card_arrival. Why? Because that's the same shape as instruction-tuning, so the later LoRA/QLoRA projects build naturally on this one.

The key detail is masking the loss so the model is graded only on the label tokens, not the prompt:

# build "prompt + label", but set prompt tokens to -100 so the loss ignores them
prompt_ids = tokenizer(prompt, add_special_tokens=False)["input_ids"]
target_ids = tokenizer(" " + label_name + tokenizer.eos_token,
                       add_special_tokens=False)["input_ids"]
input_ids = prompt_ids + target_ids
labels    = [-100] * len(prompt_ids) + target_ids   # only the label is graded

If you skip that masking, the model spends its capacity learning to reproduce the prompt instead of the answer.

The thing that surprised me: full FT is fragile

Because you're updating all the pretrained weights, a too-high learning rate shreds the model's existing knowledge. I used 5e-5 and it trained cleanly. Bumping to 2e-4 destabilized it. The training config is otherwise unremarkable — and that's the point:

TrainingArguments(
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=5e-5,            # small, on purpose
    lr_scheduler_type="cosine",
    bf16=False, fp16=False,        # fp32 on MPS for stability
)

(The later projects freeze the base, which is exactly why they can tolerate a much higher learning rate — there's no fragile pretrained knowledge to wreck.)

Result

~96% on the common intents. A near-perfect diagonal confusion matrix. A 270M model, fully fine-tuned on a laptop, nailing the task.

The one persistent slip: it confused card_arrival with card_delivery_estimate. Keep that in mind — it shows up in every project in this series, and the reason why is the punchline of Part 4.

What's next

In Part 2, I take a model 5× bigger and train less than 1% of it — and get the same accuracy. That's LoRA.

📓 Full runnable notebook on Kaggle: https://www.kaggle.com/code/sumannath88/01-full-finetune-gemma270m

Built with PyTorch + Hugging Face Transformers. Questions or corrections welcome in the comments.

DEV Community: Suman Nath

If a 270M Model Already Worked, Why Did I Fine-Tune a 7B One?

The answer most "bigger is better" content skips

1. The task is genuinely hard

2. You have little data

3. You need one model for many tasks

4. Accuracy compounds at scale

So why did I build all three?

The bug no model size could fix

The takeaway, as a decision

QLoRA: Fine-Tuning a 7B Model on a 16GB GPU (It Shrank to 5.4GB in Front of Me)

The problem

The QLoRA insight

The moment it clicked

The QLoRA-standard recipe

It's slow — and that's fine

Result

LoRA: I Trained <1% of a 1.5B Model and Matched a Full Fine-Tune

The idea behind LoRA

The config

Wall #1: ValueError: Attempting to unscale FP16 gradients

Wall #2: CUDA out of memory at batch size 64

Wall #3: my laptop and a cloud GPU showed the same speed

Result

What's next

I Fine-Tuned a 270M Model on My Laptop (Full Fine-Tuning, From Scratch)

The task

The model: deliberately tiny

One design decision: generate the label, don't classify it

The thing that surprised me: full FT is fragile

Result

What's next

Wall #1: `ValueError: Attempting to unscale FP16 gradients`