DEV Community

Cover image for P2-MOPD Uses Learnable Expert Task-Vector Priors to Speed Multi-Teacher Distillation
aimodels-fyi
aimodels-fyi

Posted on Originally published at aimodels.fyi

P2-MOPD Uses Learnable Expert Task-Vector Priors to Speed Multi-Teacher Distillation

This is a Plain English Papers summary of a research paper called P2-MOPD Uses Learnable Expert Task-Vector Priors to Speed Multi-Teacher Distillation. If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter.

P2-MOPD targets uneven recovery in multi-teacher distillation

P2-MOPD combines trainable student weights with learnable coefficients on frozen expert task vectors, then optimizes both under the multi-teacher on-policy distillation (MOPD) loss. On Qwen3-1.7B, it reaches 94.7% average recovery after 100 steps, compared with 80.9% for MOPD initialized from the same uniform task-arithmetic merge.

The setup addresses a specific weakness of MOPD: a student can improve overall while leaving one expert domain far behind or eroding capability already present in its initialization. The experiments use four domains—medicine, law, finance, and instruction following—with Qwen3-1.7B and Qwen3-4B; the fifth domain in the prior protocol is omitted because its user simulator requires a paid commercial LLM API.

Task arithmetic offers a training-free merge, but its fixed linear combination does not guarantee strong behavior: the paper reports that a uniform merge recovers less than half the gap between the base model and teachers. P2-MOPD treats expert parameters as optimization priors while retaining unrestricted trainable student weights, so the model can move beyond the experts’ parameter span.

How learnable parameter priors change the update...


Continue reading the full paper summary on AIModels.fyi →

Top comments (0)