Your Distilled Model Performs Worse Than the Teacher

You set up a distillation pipeline. The teacher is a strong frontier model. The student is smaller, cheaper to run. You train, evaluate, and the student scores lower than the teacher on every metric. This is the most common failure in distillation, and it is not obvious why it happens.

What You Will Learn

• Why naive distillation produces a weaker student
• The two knobs that actually matter: temperature and data composition
• A working training loop with the critical details surfaced
• When distillation is the wrong move entirely

The Failure Mode

Standard distillation trains a student to match the teacher's softmax outputs directly. The problem: without temperature scaling, the teacher's probability distribution is overconfident. The student sees near-one-hot targets and learns to replicate that confidence, which amplifies the teacher's mistakes.

A quote from the TechCrunch piece captures the policy push: Garry Tan wants US open-weight labs to distill frontier models so smaller versions stay capable. The engineering reality is that distillation without the right setup produces a model that is both less capable and more confidently wrong.

Temperature Scaling

Temperature softens the teacher's output distribution before the student trains on it. A higher temperature flattens the probabilities, letting the student learn from the teacher's relative preferences between incorrect classes, not just the top guess.

import torch
import torch.nn as nn
import torch.nn.functional as F

def distilled_loss(student_logits, teacher_logits, temperature=4.0, alpha=0.5):
    soft_student = F.log_softmax(student_logits / temperature, dim=-1)
    soft_teacher = F.softmax(teacher_logits / temperature, dim=-1)
    kd_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean') * (temperature ** 2)
    hard_loss = F.cross_entropy(student_logits, labels)
    return alpha * kd_loss + (1 - alpha) * hard_loss

The temperature ** 2 scaling preserves gradient magnitude as temperature grows. Without it, higher temperature flattens gradients and slows learning. The alpha parameter balances distillation loss against the standard supervised loss on hard labels.

Data Composition Is the Hidden Lever

Most practitioners use the teacher's outputs on the training set as distillation data. This creates a feedback loop: the student learns the teacher's errors on data the teacher already saw.

The fix is to mix three sources:

• Teacher outputs on held-out data the teacher never trained on
• Real labeled examples where labels are available
• Synthetic or augmented examples that cover edge cases the teacher handles poorly

def build_distill_dataset(teacher, real_dataset, augment_fn, holdout_loader):
    teacher_outputs = []
    for batch in holdout_loader:
        with torch.no_grad():
            logits = teacher(batch['input'])
        teacher_outputs.append({'input': batch['input'], 'logits': logits})

    synthetic = [augment_fn(x) for x in real_dataset if model_confidence(teacher, x) < 0.7]
    return teacher_outputs + real_dataset + synthetic

The confidence threshold filters out examples the teacher already handles well. You want the student practicing on the boundary cases.

When Distillation Is the Wrong Move

Distillation assumes the teacher's outputs contain usable signal. That breaks down when:

• The teacher is a reasoning model that chains through steps you cannot observe
• The task requires tool use or external knowledge the student cannot access
• The teacher was trained on data the student will never see at inference time

In those cases, fine-tuning the student directly on task-specific data often outperforms distillation.

Tradeoffs at a Glance

Approach	Best When	Watch Out For
Full distillation	Teacher outputs are soft and informative	Overconfident teacher collapses student
Fine-tuning only	Task data is abundant and clean	Student caps out at teacher capability
Hybrid (distill + fine-tune)	You have both teacher outputs and labels	Hyperparameter tuning gets complex

Key Takeaways

• Temperature scaling is not optional; it is the mechanism that makes distillation work
• Mix teacher outputs with real labels and hard examples, not teacher outputs alone
• Monitor the student's confidence calibration, not just accuracy
• Distillation fails silently: the student looks reasonable but loses edge cases
• When the teacher chains reasoning internally, distillation loses the signal you need

Source

Garry Tan wants US open-weight AI labs to 'distill' frontier models, too — I added the practical distillation failure modes, the temperature-scaled loss function, and the data composition strategy that most tutorials omit.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY