Distillation didn't help our small model until we took the labels away

Distil — Mon, 02 Nov 2026 17:38:35 +0000

Disclosure: this post was written by AI, and AI tools helped write the book it comes from. The experiment was run end to end on a laptop, and every number below is real output. The code is MIT licensed.

We distilled a BERT-base sentiment model (110M parameters, 92.55% on SST-2) into BERT-Mini (11M parameters) on a laptop CPU with no GPU. We expected the usual story: the student learns from the teacher's soft targets and beats a student trained on labels alone.

That's not what happened.

Round 1: 10,000 labels, no gain

Both students saw the same 10,000 labeled SST-2 sentences. One trained on the labels alone. The other trained on the standard blend of the teacher's softened probabilities and the labels (T = 2, α = 0.5):

L = α · T² · KL(p_T ‖ q_T) + (1 − α) · CE(y, q_1)

With seed 42, the distilled student came out 0.57 points ahead. That's five sentences out of 872. So we reran both students with two more seeds:

Student	Seed 42	Seed 1	Seed 2	Mean ± std
Labels only	82.00%	83.26%	82.57%	82.61 ± 0.63
Distilled	82.57%	82.00%	83.03%	82.53 ± 0.52

The single best run in the table is a baseline. Seed noise moved the baseline by 11 sentences, which is more than twice the "gain" we saw at first. With three seeds, the honest result is no measurable difference.

The reason showed up once we looked at the soft targets. At T = 2 the teacher put 99.75% of its probability on its chosen class, on average. SST-2 has two classes, and the teacher had been fine-tuned on this exact training split, so it was nearly certain about every sentence. It also agreed with the human label 98.9% of the time. On almost every sentence the "dark knowledge" just repeated the label. Raising T to 4 and 8 softened the targets but didn't improve accuracy either.

Round 2: 1,000 labels plus 9,000 unlabeled sentences

Hinton, Vinyals and Dean pointed out that the transfer set doesn't need labels. The teacher can supply the targets. So we asked a more practical question: if you can only afford to label 1,000 sentences, how much does the teacher help?

The trick in PyTorch is small. Hide the labels with -100, which is cross-entropy's default ignore_index, and give each sentence its own α. Labeled sentences get 0.5, and unlabeled ones get 1.0, which means they learn from the teacher only:

labelled = train[:1000]
unlabelled = [(sentence, -100) for sentence, _ in train[1000:]]
alpha = torch.tensor([0.5] * len(labelled) + [1.0] * len(unlabelled))

We also ran a control, A′, that trains on the same 1,000 labels for as many optimizer steps as the distilled run. Without it you can't tell "had a teacher" apart from "trained longer".

Student	Mean ± std (3 seeds)
A: 1,000 labels	73.85 ± 1.89
A′: 1,000 labels, equal steps	73.78 ± 1.44
B: 1,000 labels + 9,000 teacher-labeled	82.34 ± 0.50

That's +8.5 points. B beat A on every seed, and the worst B run was more than five points above the best A run. B also landed within 0.3 points of the student trained on all 10,000 human labels. Here, a teacher plus unlabeled text did nearly the same job as 9,000 more labels.

What we took from it

Run a labels-only baseline, and run seeds. One seed made distillation look like it helped when it didn't.
Check how soft your teacher's targets really are. If its top probability is around 0.99, the soft targets carry little more than the labels do.
Distillation earns its keep when labels are scarce and unlabeled data is cheap. That's the case it was designed for.
The student is about 10x smaller and ran 16–19x faster per sentence on the CPU, but that comes from picking a small model, not from distillation.

The full walkthrough is a book, Small Model, Big Teacher. It covers the loss, why T² is there, choosing a teacher whose license allows this, the training loop line by line, and the results above. Chapters 1–2 are free at distil.md.

DEV Community: Distil

Distillation didn't help our small model until we took the labels away

Round 1: 10,000 labels, no gain

Round 2: 1,000 labels plus 9,000 unlabeled sentences

What we took from it