Turning Qwen2.5-0.5B into a JSON API with SFT — 23% 100% on a Free T4

I Tried Post-Training and All I Got Was "systemsystem"

I followed the TRL tutorial to fine-tune a small model with SFT (supervised fine-tuning). Training ran to completion. But no matter what I asked the trained model, the answer was:

systemsystemsystemsystemsystem...

Was the model broken? Was my code wrong? I couldn't even tell which — if you've tried SFT, maybe this sounds familiar.

It turned out I hit that exact symptom twice, from two unrelated causes : once because training genuinely diverged, and once because training had succeeded perfectly and my verification code was wrong. Telling those apart is most of what this article is about.

This is the field log of getting from there to turning Qwen2.5-0.5B into a model that answers every question with nothing but JSON.

Item	Result
Model	Qwen2.5-0.5B base
Method	SFT (implemented with LoRA)
Framework	TRL + PEFT
GPU	Colab T4 ×1 (free tier)
Training time	26 minutes (1,538 s)
Valid-JSON rate	23% → 100%
Valid JSON and correct category	0% → 96%
Training data	8,705 single-turn samples (no_robots)
Evaluation	30 held-out questions

A note on the title: SFT and LoRA are not alternatives to each other. SFT is the objective — train on demonstrations of the behavior you want. LoRA is a parameterization — store the weight update in a small low-rank adapter instead of touching all the weights. Everything below is caused by SFT; LoRA is just where the result is kept. (Why LoRA and not full fine-tuning is itself Trap 4.)

This is part 1 of a post-training series; part 2 covers RM+PPO / DPO.

Setup: Why a 0.5B Base Model

Item	Value
Model	Qwen/Qwen2.5-0.5B (base — not -Instruct)
Data	HuggingFaceH4/no_robots (train 9,500 / test 500; messages + category columns)
Libraries	TRL 1.10.0, transformers 5.13.1, peft 0.19.1, datasets 5.0.1, accelerate 1.14.0, torch 2.11.0+cu128
Environment	Colab T4 ×1, pinned to a single GPU

Three reasons for this configuration:

1. Without a base model there is no "before." -Instruct models have already been through SFT, so they can't show you what SFT changes.
2. GGUF and MLX are inference-only formats — you can't train them. You need safetensors.
3. Full SFT of a 30B-class model needs ~480 GB of VRAM , while a 0.5B model still exposes the same operational failure modes — and a free T4 shows you all of them.

Those library versions matter more than usual here: two of the seven traps below are pure version drift , and they will look different (or disappear) on other versions.

Seven Traps, in Order

Chronological. Each one is error → cause → fix.

Trap 1: KeyError: 'completion'

KeyError: 'completion'

Died the moment training started. Cause: no_robots carries a prompt column alongside messages, and TRL misdetected the dataset as prompt-completion format. One-line fix:

train = ds["train"].select_columns(["messages"])

Trap 2: Output Collapses to "systemsystemsystem…"

The one from the intro — the real one. Cause: full fine-tuning directly on fp16-loaded weights diverged. The T4 has no native bf16, so fp16 is the tempting choice, but updating fp16 weights directly is numerically fragile; the model collapses onto its most frequent token.

Fix: load in fp32 and let mixed precision be handled by the trainer, which keeps an fp32 master copy.

model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32)
cfg = SFTConfig(fp16=True, ...)

Trap 3: RuntimeError — Tensors on cuda:1 and cuda:0

RuntimeError: tensors ... cuda:1 different from cuda:0

I tried Kaggle as well as Colab. Kaggle gives you two T4s, HF Trainer silently wraps the model in nn.DataParallel, and it dies on a device mismatch. A 0.5B model doesn't need two GPUs. Fix — but it only takes effect before import torch, which means a kernel restart if you've already imported:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch

Trap 4: Full Fine-Tuning Collapses Again — Why I Switched to LoRA

Even with Trap 2 fixed, full fine-tuning of this small model stayed extremely sensitive to learning rate and precision, and it collapsed again. That was the turning point: I dropped full FT for LoRA (r=16, alpha=32, lr=2e-4). With the base weights frozen and only 8.8M adapter parameters training (1.75% of the model), the instability went away and never came back.

Caveat I want to be honest about: I never got a clean full-FT run to compare against. So "LoRA is more stable here" is what I observed, not a controlled A/B.

Trap 5: Forgot model.eval() — a Fake Collapse

The one that made me go pale. After switching to LoRA, generation said systemsystem again.

Training had actually succeeded. I had forgotten model.eval(), so greedy decoding ran with dropout still active. I isolated it by varying two things independently:

training	use_cache	output
True	False	A systemsystemsystem…
True	True	A systemsystemsystem…
False	False	A cache is a data structure that stores data…
False	True	A cache is a data structure that stores data…

Only training mattered. Meanwhile the training logs were healthy the whole time — entropy 2.63 → 2.41 (a real collapse drives it toward zero), grad_norm 1.17 → 0.45, token accuracy 0.469 → 0.517.

Diagnose collapse from the training logs, not from the symptom. The symptom is ambiguous; entropy and grad_norm are not.

was_training = model.training
model.eval()
try:
    ...generate...
finally:
    model.train(was_training)

Trap 6: torch.cuda.is_bf16_supported() Returns True on a T4

Recent PyTorch changed this function's default to including_emulation=True, so a T4 answers "yes" and your code happily selects an emulated bf16 that is slower with no benefit:

capability: (7, 5)
is_bf16_supported(): True
is_bf16_supported(including_emulation=False): False

Check the hardware generation instead — it's stable across versions:

use_bf16 = torch.cuda.get_device_capability(0)[0] >= 8 # Ampere or later

Trap 7: Two Version-Drift Errors (BatchEncoding, torchao)

apply_chat_template() returns a BatchEncoding, not a tensor.

AttributeError: ... 'inputs.shape'

Pass return_dict=True, call generate(**enc), and take the prompt length from enc["input_ids"].shape[1].

Colab's preinstalled torchao is too old for recent transformers.

ImportError: Found an incompatible version of torchao. Found version 0.10.0,
but only versions above 0.16.0 are supported

This fires at import time, before any training. If you're not quantizing, remove it; if you are, upgrade it. Either way restart the session afterwards , or the preinstalled version stays loaded:

pip uninstall -y torchao # or: pip install -U torchao

What the trap list actually shows

I originally wanted to claim "most of these were verification-side problems." Lining them up, that isn't true. Two of the seven — Trap 5, and the BatchEncoding half of Trap 7 — were cases where training had genuinely succeeded and only my verification code was stale. The rest failed loudly and early: Traps 1 and the torchao half of 7 died before a single training step, and Traps 2/3/4 were real training failures.

The honest lesson is narrower but still worth having: when SFT "fails," check whether it failed at all. Two of my seven traps cost me hours on a model that was already working.

Experiment 1: What the Format Change Actually Looks Like

One epoch of SFT on no_robots, LoRA, 33 minutes on a T4. Same prompt, before and after — that comparison matters, because on some prompts the base is already better than you'd assume.

"Give me 3 tips for running a 10k race."

BEFORE (base):

Sure, here are three tips for running a 10k race:

1. Start early: It's important to start running early in the morning or early in
   the day to allow your body to adjust to the increased intensity of the race...
2. Listen to your body: Pay attention to...

AFTER (SFT):

1. Start early. If you start too late, you will not be able to run the full 10km.
   Start early and finish early.
2. Don't run too fast. You should be able to run the full 10km in 45 minutes or less...
3. Don't run too long...

Note what did not happen: the base already produced a numbered list. What SFT removed was the preamble (Sure, here are three tips for running a 10k race:) — the model now answers directly, in the style of the training data. That's a real change, and a smaller one than "it learned to make lists."

"What is the capital of France?" — this one is sharper:

BEFORE (base):

The capital of France is Paris. navigationOptions
navigationOptions
You are a helpful assistant.icode
icode
What is the capital of France?icode ...

AFTER (SFT):

The capital of France is Paris. It is the largest city in Europe and the
third-largest city in the world. It is also the seat of the French government
and the seat of the French parliament. It is also the seat of... (repeats)

The base answers correctly and then falls out of the assistant role entirely , echoing chat-template fragments. After SFT it stays in role from start to finish — and states two confident falsehoods (Paris is neither the largest city in Europe nor the third largest in the world). Form improved; factual reliability did not, and arguably got worse-looking because the errors are now delivered fluently.

"Explain what a KV cache is." — both loop, before and after. no_robots contains no KV-cache knowledge, and one epoch of SFT on it adds none.

The part I got wrong at first: SFT made stopping worse

My first read was "the leftover repetition is just a 0.5B + greedy-decoding artifact, not an SFT problem." I measured it, and that's only half right.

I generated 256 tokens for 25 held-out prompts and recorded two things separately: whether the model emitted the terminator <|im_end|>, and whether generation stopped at all.

| Model | Decoding | Stopped | Emitted <|im_end|> | distinct-3 |
| --- | --- | --- | --- | --- |
| base | greedy | 20% | 0% | 0.588 |
| base | greedy + repetition_penalty 1.1 | 40% | 0% | 0.781 |
| base | sampling (T=0.7, top_p=0.9) | 24% | 0% | 0.680 |
| SFT | greedy | 0% | 0% | 0.337 |
| SFT | greedy + repetition_penalty 1.1 | 0% | 0% | 0.700 |
| SFT | sampling | 0% | 0% | 0.534 |

Two separate phenomena, and they behave differently:

• Repetition is decode-fixable. repetition_penalty moves distinct-3 from 0.337 to 0.700. My original claim holds here.
• Termination is not , and SFT made it strictly worse: the base stopped 20–40% of the time; after SFT, never — under every decoding setting I tried.

Why? I looked at the terminal position of reference answers under teacher forcing:

| At the position whose target is <|im_end|> | base |
| --- | --- |
| P(<|im_end|>) | ~0 (below 1e-5) |
| Rank of <|im_end|> | 118,800 out of ~152,000 |
| P(<|endoftext|>) | 0.1006 |
| argmax was <|endoftext|> | 7 / 30 cases |

The base model's terminator is <|endoftext|>, and it barely knows <|im_end|> exists — despite shipping a chat template that uses it. Shipping a chat template and having been trained on it are different things, which I had conflated.

The training data ends every example with <|im_end|> and never contains <|endoftext|>. So SFT pushes down the only terminator the base knew, while failing to lift the new one from rank ~118,800 to rank 1 in one epoch of an 8.8M-parameter adapter. It didn't fail to teach stopping so much as break the stopping the model already had. That is my current explanation, and it's a hypothesis: I confirmed the base-side numbers but lost the VM before measuring the post-SFT side.

Two things I ruled out along the way: truncation at max_length=1024 affects only 1.7% of examples (163/9,500; median length 241 tokens), and training with assistant_only_loss=True — which concentrates the loss on assistant tokens, where the terminator lives — did not help (still 0% termination, and slightly more repetitive).

Experiment 2: Quantifying Success — a 0.5B JSON API

If SFT is good at teaching form, pick a task where form is measurable. Goal: make the model answer every question with nothing but {"category": "...", "answer": "..."}.

Labels reuse no_robots' existing category column, so additional data creation was zero. Restricted to single-turn conversations: train 8,705 / test 446, evaluated on the first 30 test questions.

Metric	BEFORE (base)	AFTER (SFT, 1 epoch)
Valid JSON extractable	23% (7/30)	100% (30/30)
Valid JSON and correct category	0% (0/30)	96% (29/30)

Measured run: 545 steps / 26 minutes (1,538 s) / train_loss 1.857 / mean_token_accuracy 0.653 / 2.10M tokens, on one T4.

How these metrics are defined , because it changes what they mean:

• Valid JSON extractable = the first {...} found anywhere in the output parses and has both keys. It is deliberately lenient: preambles and trailing text don't disqualify a response. So 23% is not"23% of outputs were pure JSON."
• The second row is a compound metric: a response only scores if it both parses and gets the category right. The base's 0% is therefore mostly a JSON failure, not a classification failure — 23 of its 30 answers never became scoreable. Don't read it as "the base cannot classify at all"; a base asked to emit only a bare category label would surely do better than zero.

With that in mind, the BEFORE 23% is the interesting number. Here's what one of those seven looked like:

Sure! Here's an example response for you:

{
"category": "Closed QA",
"answer": "I'm not sure what genre to check out."
}

Small talk, then a JSON object inside a markdown code fence , with a category that's wrong. The base can imitate the shape it was shown in the system prompt; it cannot execute the task (classify the request, then fill the structure), and it cannot suppress the surrounding chatter. My lenient parser counted this as a success, which is exactly why the definition needs stating.

The other two sampled BEFORE responses didn't produce JSON at all:

I'm sorry, but as an AI language model, I don't have personal experiences or
emotions like humans do. However, here is some information about what people...


Write an essay about your experience as a student in the United States.
orda
Write a short story that includes one or more of these words: love, friendship...

The second is the base model doing what base models do — continuing the text rather than answering it.

After SFT:

{"category": "Brainstorm", "answer": "1. Take an umbrella - It will keep you dry while
walking around London. 2. Bring your own water bottle - ..."}

No preamble, no fence, 100% parseable, 96% correct category. What SFT taught was not "the JSON shape" — the base could already mimic that — but executing the task in that shape.

Sample-size caveat : 30/30 on n=30 has a 95% confidence interval of roughly [88%, 100%]. I ran this once, greedy, on a fixed slice of the test set. Treat "100%" as "no failures in 30," not as a guarantee.

The Final Working Code

# Pin to one GPU — must run BEFORE importing torch (Trap 3)
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import torch

# Data: keep only the messages column (Trap 1)
train = ds["train"].select_columns(["messages"])

# Load in fp32 on a T4; mixed precision is the trainer's job (Trap 2)
use_bf16 = torch.cuda.get_device_capability(0)[0] >= 8 # Trap 6
model = AutoModelForCausalLM.from_pretrained(
    MODEL, dtype=torch.bfloat16 if use_bf16 else torch.float32)

from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

cfg = SFTConfig(per_device_train_batch_size=4, gradient_accumulation_steps=4,
                num_train_epochs=1, learning_rate=2e-4, lr_scheduler_type="cosine",
                max_length=1024, packing=False, # T4 has no FA2
                fp16=not use_bf16, bf16=use_bf16, report_to="none")
peft_cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                      target_modules=["q_proj","k_proj","v_proj","o_proj",
                                      "gate_proj","up_proj","down_proj"],
                      task_type="CAUSAL_LM")
trainer = SFTTrainer(model=model, args=cfg, train_dataset=train, peft_config=peft_cfg)
trainer.train()

# Verification (Traps 5 and 7)
m = trainer.model
m.eval() # Trap 5 — do not skip
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_dict=True, return_tensors="pt").to(m.device) # Trap 7
with torch.no_grad():
    out = m.generate(**enc, max_new_tokens=160, do_sample=False,
                     repetition_penalty=1.15, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

packing=False because the T4 doesn't support Flash Attention 2, and without it packing risks cross-sample contamination.

Reproduction Details

Everything needed to rerun Experiment 2:

• Model : Qwen/Qwen2.5-0.5B (base; bf16 checkpoint, loaded as fp32)
• Dataset : HuggingFaceH4/no_robots, single-turn conversations only; category labels reuse the dataset's own annotation
• Target construction : assistant turn is whitespace-normalized and truncated to the first 60 words (with "..." appended), then wrapped as {"category": ..., "answer": ...}. This also teaches brevity, and you will not reproduce these numbers without it.
• Splits : train 8,705 / test 446; evaluation on the first 30 test questions (not a random sample)
• GPU : Colab T4 ×1, pinned via CUDA_VISIBLE_DEVICES=0 before import torch
• Training : 1 epoch, batch 4 × grad-accum 4, lr 2e-4, cosine, warmup_steps=30, max_length=1024, packing=False, fp16 mixed precision
• LoRA : r=16, alpha=32, dropout=0.05, targets q/k/v/o/gate/up/down_proj (8.8M trainable / 502.8M total = 1.75%)
• Generation (eval): model.eval(), max_new_tokens=160, greedy (do_sample=False), repetition_penalty=1.15
• Measured run : 545 steps, 1,538 s, train_loss 1.857, mean_token_accuracy 0.653, 2.10M tokens
• Versions : torch 2.11.0+cu128, transformers 5.13.1, TRL 1.10.0, peft 0.19.1, datasets 5.0.1, accelerate 1.14.0

One reproducibility note in SFT's favour: I ran the Experiment 1 notebook on TRL 1.9.2 and again on 1.10.0 eight days apart. Loss at step 500 was 2.3167 and 2.3170 — identical to four decimals. Not everything drifts.

Discussion: SFT Isn't "Useless" — It Has a Different Job

Put the two experiments side by side:

What you want	Did SFT deliver it here?
Output structure (JSON / tool-call shape)	✅ 23% → 100%
Task execution in that structure (classify, then fill)	✅ 0% → 96%
Response format and register (direct answers, no preamble)	✅ visible in every sample
Staying in the assistant role	✅ base fell out of role; SFT model didn't
New factual knowledge	❌ no evidence of any
Knowing when to stop	❌ worse than the base

Much of "I did SFT and it didn't get smarter" is expecting knowledge from a technique whose demonstrated strength is behavior. Behavior injection got me 23%→100% with 0.5B parameters, 26 minutes, and zero additional data.

But I'd resist the tidy version of this story. SFT didn't just fail to add knowledge — on one axis it actively removed a capability the base had, because my training distribution never contained the token the base was using to stop. That's a general shape worth remembering: fine-tuning moves probability mass, and mass has to come from somewhere. If your data never demonstrates a behavior, one epoch of SFT is a good way to suppress it.

Zooming out: published post-training pipelines for chat models (InstructGPT is the canonical writeup) put SFT at the foundation for exactly this reason, and the Structured Outputs / tool-calling features in commercial APIs solve the same class of problem Experiment 2 demonstrates — getting a model to execute a task in a fixed output shape.

Summary

• SFT taught behavior, not knowledge. Same-prompt before/after shows format and role adherence changing while factual quality does not.
• Quantified : valid JSON 23%→100%, valid-JSON-and-correct-category 0%→96% (0.5B, LoRA, 1 epoch, one free T4, 26 minutes, n=30).
• SFT and LoRA operate at different levels. SFT is the objective; LoRA is where the update is stored. The 100% is SFT's doing.
• Two of seven traps were "training succeeded, my verification was stale." Diagnose collapse from entropy / grad_norm / token accuracy — never from the symptom, which is identical in both cases.
• Full FT of this small model kept collapsing; LoRA removed the instability — observed, not A/B tested.
• SFT broke termination. Base stopped 20–40% of the time; after SFT, 0% under every decoding setting. Repetition is decode-fixable; termination isn't.

Next: SFT's next limitation — it can't teach "A is preferable to B" — with RM+PPO and DPO.

References

• InstructGPT — arXiv:2203.02155
• FLAN — arXiv:2109.01652
• The RLHF Book — Nathan Lambert
• TRL documentation
• TPU v6e-1 vs A100 80GB Single-Chip Showdown (Qwen3-8B on vLLM) — the inference-side measurements for the same Qwen family