This is a Plain English Papers summary of a research paper called P2-MOPD Uses Learnable Expert Task-Vector Priors to Speed Multi-Teacher Distillation. If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter.
P2-MOPD targets uneven recovery in multi-teacher distillation
P2-MOPD combines trainable student weights with learnable coefficients on frozen expert task vectors, then optimizes both under the multi-teacher on-policy distillation (MOPD) loss. On Qwen3-1.7B, it reaches 94.7% average recovery after 100 steps, compared with 80.9% for MOPD initialized from the same uniform task-arithmetic merge.
The setup addresses a specific weakness of MOPD: a student can improve overall while leaving one expert domain far behind or eroding capability already present in its initialization. The experiments use four domains—medicine, law, finance, and instruction following—with Qwen3-1.7B and Qwen3-4B; the fifth domain in the prior protocol is omitted because its user simulator requires a paid commercial LLM API.
Task arithmetic offers a training-free merge, but its fixed linear combination does not guarantee strong behavior: the paper reports that a uniform merge recovers less than half the gap between the base model and teachers. P2-MOPD treats expert parameters as optimization priors while retaining unrestricted trainable student weights, so the model can move beyond the experts’ parameter span.
Top comments (0)