Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
The usual way to improve an LLM is to make the model itself better.
Train it on more data. Run more reinforcement learning. Increase the context. Increase the parameter count.
But there is another possibility:
Build several models that become very good at different things, then teach a single student to absorb those capabilities.
That sounds like ordinary knowledge distillation.
The interesting part is what happens when the teachers are specialists, and the student is allowed to make its own mistakes while learning from them.
This is the idea behind Multi-Teacher On-Policy Distillation (MOPD), described in Xiaomi's 2026 MiMo-V2-Flash technical report.
The basic pattern is:
+-------------------+
| Math Teacher |
+---------+---------+
|
+---------v---------+
| |
| Student |----> final model
| |
+---------^---------+
|
+---------+---------+
| |
+---------+---------+ +-----+---------+
| Coding Teacher | | Agent Teacher |
+------------------+ +---------------+
The student generates its own trajectory.
A domain-appropriate teacher looks at those exact tokens and provides dense feedback.
Over many iterations, the student becomes a single model that contains capabilities that previously lived in separate experts.
MiMo-V2-Flash uses this as the third stage of its post-training pipeline, after SFT and domain-specific RL. Its report describes specialist teachers for coding, search, tool use, mathematics, reasoning, and safety. (arXiv)
1. The problem: capability specialization creates a new integration problem
Imagine you have five excellent models:
Teacher A -> mathematics
Teacher B -> coding
Teacher C -> web search
Teacher D -> tool use
Teacher E -> general reasoning
Each model has undergone its own optimization process.
The math teacher has seen enormous amounts of reward pressure around mathematical reasoning.
The coding teacher has spent its RL budget on writing and executing code.
The search teacher has learned how to explore a web environment.
Eventually, you have several models that are locally excellent but globally fragmented.
How do you get one production model with all five capabilities?
There are several obvious approaches.
Parameter merging
Average or otherwise combine the weights.
The problem is that neural networks do not give you a clean "math capability vector" and "coding capability vector" that can simply be added together.
Sequential training
Train on mathematics, then coding, then search.
Now you risk the familiar continual-learning problem: the later optimization changes parameters that were useful for earlier capabilities.
Offline distillation
Ask each teacher to generate a large dataset, then fine-tune the student on those examples.
This works, but the student mostly learns from trajectories that the teachers considered worth generating.
The important question is:
What happens when the student behaves differently from the teacher?
That question leads directly to on-policy distillation.
2. From Hinton's distillation to learning from your own mistakes
The history is useful here.
In 2015, Geoffrey Hinton, Oriol Vinyals, and Jeff Dean described knowledge distillation as a way to compress an ensemble of models into a single deployable model.
The motivation was practical. An ensemble could be more accurate, but serving many large networks for every prediction was expensive. Their idea was to have the large ensemble teach one smaller model. (arXiv)
The basic idea was:
large ensemble
|
v
soft predictions
|
v
small student
For autoregressive language models, however, another problem appears.
Suppose the teacher produces:
The function is convex because its second derivative is positive...
and the student produces:
The function is convex because its derivative is increasing...
Those trajectories have different prefixes.
The probability distribution the student sees at token 20 depends on what the student generated at tokens 1-19.
So training on teacher-generated text gives you one distribution of states, while inference gives you another.
In 2024, Rishabh Agarwal and colleagues at Google DeepMind formalized this issue in Generalized Knowledge Distillation. Their on-policy approach has the student generate its own outputs, then asks the teacher to evaluate those student-generated trajectories. The point is to train on the states the student will actually visit. (ML Anthology)
That distinction is easiest to see like this:
OFF-POLICY
teacher -> generates trajectory -> student learns trajectory
ON-POLICY
student -> generates trajectory
|
v
teacher evaluates it
|
v
student learns from it
For an autoregressive model, that difference is substantial.
The teacher is no longer saying:
"Here is the answer I would have written."
It is effectively saying:
"Given the exact state you reached, here is how I would evaluate the next decision."
That is much closer to interactive coaching.
3. MOPD adds multiple specialist teachers
MOPD takes the on-policy idea and turns it into a multi-expert system.
The MiMo-V2-Flash pipeline is roughly:
Stage 1
Base model
|
v
SFT student
|
v
Stage 2
+-------------------------------+
| |
| math RL -> math teacher|
| coding RL -> code teacher|
| search RL -> search teacher
| tool-use RL -> agent teacher
| reasoning RL -> reasoning teacher
| |
+-------------------------------+
|
v
Stage 3
MOPD
|
v
single general-purpose student
A prompt is associated with an appropriate domain teacher.
For example:
"Prove this number theory result"
|
v
math teacher
"Fix this failing Python test"
|
v
coding teacher
"Find the answer using web search"
|
v
search teacher
The critical part is that the student still generates the response.
Suppose the student is trying to solve a coding problem.
It generates:
Step 1: inspect repository
Step 2: open foo.py
Step 3: modify function
Step 4: run tests
Step 5: ...
The coding teacher evaluates the probabilities it would assign to those student-generated tokens.
The teacher does not necessarily need to generate a competing four-thousand-token answer.
Instead, it supplies token-level information about the student's trajectory.
This makes MOPD very different from simply asking five teachers for five answers and throwing all of them into an SFT dataset.
The MiMo report explicitly describes MOPD as an on-policy RL process rather than parameter merging or static expert-generated datasets. (arXiv)
4. The math: turn teacher probabilities into a reward
Here is the core idea without getting buried in notation.
Let:
pi_student(y_t | x, y_<t)
be the probability the student assigns to the token it actually generated.
Let:
pi_teacher(y_t | x, y_<t)
be the probability the specialist teacher assigns to that same token, given the same prefix.
Now define the token-level signal:
A_t = log( pi_teacher / pi_student )
Everything is evaluated at the actual student token and its actual prefix.
This has a very intuitive interpretation.
Case 1: teacher likes the token more than the student does
Suppose:
student probability = 0.10
teacher probability = 0.50
Then:
A_t = log(0.50 / 0.10)
= log(5)
≈ +1.61
That is a positive learning signal.
The student should increase the probability of making this decision in similar states.
Case 2: teacher dislikes the token
Suppose:
student probability = 0.40
teacher probability = 0.05
Then:
A_t = log(0.05 / 0.40)
= log(0.125)
≈ -2.08
Now the signal is negative.
The student should reduce the probability of making that choice.
Case 3: they agree
Suppose:
student probability = 0.20
teacher probability = 0.20
Then:
A_t = log(1) = 0
There is no corrective pressure.
So the teacher is providing a dense, token-level advantage signal.
The underlying objective can be understood as minimizing the reverse KL divergence:
D_KL(student || teacher)
with the expectation taken over tokens sampled from the student.
The MiMo formulation then turns this into a policy-gradient-style surrogate objective. It also uses importance sampling when the training policy and sampling policy differ, and discards tokens whose probability ratios fall outside a specified range. (arXiv)
That last engineering detail matters.
Your rollout worker may use a slightly different inference configuration from the policy being updated.
If:
mu = sampling policy
pi = current training policy
then a correction factor of roughly
pi(y_t) / mu(y_t)
is needed.
But enormous ratios make the estimator unstable, so MOPD clips the usable region and drops sufficiently discrepant tokens. (arXiv)
This is one reason MOPD should be thought of as an RL system with a specialized reward, rather than simply "cross-entropy against a teacher."
5. Why this can combine capabilities without generating an enormous dataset
There is an important computational distinction.
Imagine a student generates 4,000 tokens.
With ordinary teacher-generated distillation, the teacher might need to autoregressively generate a 4,000-token answer.
That is 4,000 sequential decoding steps.
With on-policy token-level distillation, the student already generated the trajectory.
The teacher can evaluate the student trajectory with a forward pass over the sequence and produce logits for the relevant positions.
Conceptually:
Teacher generation:
t1 -> t2 -> t3 -> t4 -> ... -> t4000
^ ^ ^ ^
serial decoding
Teacher evaluation:
[t1 t2 t3 t4 ... t4000]
|
v
logits for all
causal positions
The exact implementation still has substantial transformer compute, especially with long contexts, but the workload is much more amenable to batching than autoregressive teacher generation.
That changes the economics.
Suppose:
1,000,000 training prompts
4,000 student tokens / prompt
1 selected teacher / prompt
Then the teacher processes approximately:
1,000,000 * 4,000
= 4 billion teacher-scored tokens
The important variable is selected teachers per trajectory, not the number of teachers in your library.
Seven specialist teachers do not automatically mean seven teacher evaluations for every token.
You can instead have:
7 teachers
|
routing
|
1 teacher per trajectory
The fixed cost is building seven good teachers.
The variable cost is scoring student trajectories with the relevant teacher.
That is an interesting economic trade:
Specialist RL
|
v
expensive capability acquisition
|
v
reusable teacher
|
v
many student models
|
v
amortized capability transfer
A company building multiple downstream models could reuse the same expert pool.
That is where multi-teacher distillation starts looking less like a training trick and more like an organizational architecture for model development.
6. The hard part is the infrastructure, not the equation
It is tempting to read the MOPD equation and conclude that implementation is just:
loss = log(student_prob / teacher_prob)
It is not.
A production implementation has at least these components:
+----------------------+
| Prompt / environment |
+----------+-----------+
|
v
+----------------+
| Student rollout|
+--------+-------+
|
v
+-----------------------+
| Select domain teacher |
+-----------+-----------+
|
v
+-----------------------+
| Teacher forward pass |
| on student trajectory |
+-----------+-----------+
|
v
+-----------------------+
| Token-level advantage |
+-----------+-----------+
|
+---------+---------+
| |
v v
MOPD signal ORM signal
| |
+---------+---------+
|
v
policy update
MiMo-V2-Flash is a useful concrete case because its authors describe the surrounding machinery.
Their RL/MOPD infrastructure uses SGLang for inference and Megatron-LM for training, with FP8 training and inference. They also describe rollout-routing replay, data scheduling, partial rollouts, prefix caching, and load balancing. (arXiv)
For coding agents, the scale gets even more concrete.
The report describes training across roughly 120,000 interactive environments and a Kubernetes-based environment setup involving more than 10,000 concurrent pods. (arXiv)
That tells you something important about MOPD:
the loss function is the easy part.
The difficult parts are:
rollout throughput
teacher serving
policy/rollout synchronization
importance sampling
long trajectories
GPU utilization
reward latency
environment orchestration
This is also why the distinction between an ordinary SFT pipeline and an on-policy pipeline matters operationally.
In SFT:
dataset -> GPU -> gradient update
In MOPD:
prompt
-> rollout
-> environment interaction
-> teacher inference
-> reward computation
-> advantage construction
-> policy update
-> next rollout
Your training system has become a distributed control loop.
7. What MOPD changes about LLM post-training
The deeper idea is not "use several teachers."
It is:
separate capability acquisition from capability integration.
A specialist can be optimized very aggressively for one domain.
Then a general student can learn from many such specialists while remaining the entity that generates its own trajectories.
That gives you a useful architecture:
specialist RL
|
+-----------------+------------------+
| | |
v v v
math expert coding expert agent expert
| | |
+-----------------+------------------+
|
v
on-policy student
|
v
unified model
There is also an interesting feedback loop.
MiMo's authors describe a process in which a distilled student can itself become the starting point for another round of specialized RL, producing stronger teachers for a later student. (arXiv)
So you can imagine:
Student v1
|
v
specialized RL
|
v
Teachers v2
|
v
MOPD
|
v
Student v2
|
v
specialized RL
|
...
That starts to resemble an internal capability-production system.
But the benchmark results are also a useful warning against treating this as automatic capability addition.
In the MiMo experiments, MOPD improved or matched the strongest teacher on several listed benchmarks, including AIME 2025, HMMT, LiveCodeBench, and tau²-Bench. But there were also regressions: for example, BrowseComp fell 6.3 points relative to the listed best teacher, and Arena-Hard Creative Writing fell 3.9 points. SWE-Bench Verified was 73.4 after MOPD versus 74.2 for the best teacher in that comparison. (arXiv)
So the realistic interpretation is:
MOPD provides a mechanism for capability transfer.
It does not guarantee perfect capability preservation.
And that is probably the most useful way for developers to think about it.
Knowledge distillation began with the problem of making ensembles cheaper.
On-policy distillation added the idea of teaching students on states they actually visit.
MOPD pushes that one step further:
many specialized policies
|
v
student-generated trajectories
|
v
token-level expert feedback
|
v
one deployable policy
The interesting engineering question is therefore no longer just:
"How do I make a bigger model?"
It becomes:
"How cheaply can I create specialized intelligence, and how efficiently can I transfer it into one model?"
That is a very different post-training economics.
What do you think is the harder scaling problem for the next generation of LLMs: training stronger specialist teachers, or building the infrastructure to distill them into a single general model?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)