Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
There is a simple question at the heart of modern LLM training:
Where did the data come from?
If you train a model on answers generated by itself, you are doing something fundamentally different from training it on answers generated by an older model, a human, or a static dataset.
That difference is called on-policy vs. off-policy learning.
It sounds like reinforcement-learning terminology from the 1990s. It is. But it has become one of the most important ideas for understanding what is happening in modern LLM post-training: RLHF, PPO, rejection sampling, preference optimization, self-play, synthetic data, verifiable rewards, and increasingly, models that generate their own training trajectories.
The interesting part is that the distinction is not really about "online" versus "offline" data.
It is about a much more precise question:
Is the model learning from behavior produced by the policy it is currently trying to improve?
Once you see that distinction, a surprising number of LLM training techniques fall into place.
1. Imagine teaching a programmer
Suppose you are training an LLM to write Python.
You give it:
Write a function that returns the longest increasing subsequence.
The model produces:
def lis(a):
...
An evaluator gives it a score.
Now imagine two training systems.
System A: on-policy
The current model generates the solution.
You evaluate that solution.
You update the model based on the result.
Then the new model generates another solution.
The loop is:
current model
|
v
generate solution
|
v
evaluate solution
|
v
update model
|
v
new model
|
+----> generate again
The data continuously moves with the model.
System B: off-policy
Instead, you have 10 million solutions sitting in a dataset.
Some were written by humans.
Some came from GPT-4.
Some came from an older checkpoint.
Some came from a specialized coding model.
You train your current model on those examples.
The behavior that produced the data is not necessarily the behavior of the model you are currently training.
That is off-policy learning.
The distinction matters because the model's mistakes determine what it gets to learn from.
If the current model is terrible at recursion, an on-policy system will naturally generate lots of terrible recursive solutions. An off-policy dataset might contain excellent recursive solutions that the current model would never have generated.
That sounds like an obvious advantage for off-policy learning.
And sometimes it is.
But there is a catch.
2. The RL concept is older than LLMs
The terminology comes from reinforcement learning.
Chris Watkins' Q-learning work in the late 1980s and early 1990s gave one of the classic examples of off-policy learning. Q-learning can learn about an optimal policy while the agent is actually behaving according to another policy.
The conceptual trick is powerful:
The policy generating the experience does not have to be the policy being learned.
Contrast that with policy-gradient methods, where you typically generate trajectories using the current policy and then use those trajectories to estimate how changing that policy would affect expected reward.
This distinction became especially important as reinforcement learning moved from toy environments to expensive neural-network systems.
By the time John Schulman and colleagues introduced PPO in 2017, the engineering problem was familiar:
Generate experience with a policy, then update the policy without moving it so far that the experience becomes useless.
PPO explicitly alternates between collecting samples from the current policy and optimizing on those samples. It permits multiple optimization epochs over the collected data while constraining how far the updated policy moves from the policy that generated the samples.
That constraint is not cosmetic.
It is the central operational problem.
Suppose your model generated:
"The answer is 42."
You update the model ten times using that batch.
After those ten updates, your model may have changed substantially.
The sample was generated by policy P_old.
You are now optimizing policy P_new.
The more different P_new becomes from P_old, the less directly the old sample tells you about P_new.
That is the basic tension behind on-policy RL.
3. Why LLMs make this unusually expensive
In a game like Atari, generating another million actions may be relatively cheap.
For an LLM, generating another million trajectories can mean running a giant transformer for billions of tokens.
And the reward might require an expensive evaluator.
Consider a deliberately simple calculation.
Suppose:
- model inference costs $2 per million generated tokens
- you generate 100 million tokens per training iteration
- you perform 100 iterations
Generation alone costs roughly:
100M tokens x 100
= 10B generated tokens
10B / 1M x $2
= $20,000
That is a toy number, but the scaling relationship is real.
At frontier-model scale, the expensive resource is often not the gradient update.
It is producing useful experience.
This is why off-policy learning is so attractive.
If you have already paid to generate 10 billion tokens, you would very much like to reuse them.
And you would like to reuse them more than once.
That is the economic argument for off-policy training:
Experience is an asset. Don't throw it away after one gradient update.
This is also why replay buffers became such an important idea in classical RL.
But LLMs introduce an even more interesting problem: the "environment" is often other models, humans, tools, or verifiers rather than a game simulator.
4. InstructGPT shows the transition
One of the clearest historical examples is OpenAI's 2022 InstructGPT work.
The basic pipeline was:
GPT-3
|
v
human demonstrations
|
v
SFT model
|
v
generate multiple answers
|
v
human preference rankings
|
v
reward model
|
v
PPO
|
v
InstructGPT
The important part for our discussion is the final stage.
The reward model evaluates outputs generated by the policy, and PPO uses those interactions to improve the policy.
That is much closer to the classic on-policy RL loop than ordinary supervised fine-tuning.
And it produced a remarkable result.
OpenAI reported that evaluators preferred outputs from the 1.3B-parameter InstructGPT model over the 175B-parameter GPT-3 model on their instruction-following evaluation.
In other words, changing how the model learned from experience could matter more than making the model roughly 100x larger.
This was an important moment in LLM history because it demonstrated that post-training was not merely polishing a pretrained model.
It could substantially change what the model optimized for.
And PPO's on-policy nature was part of that story.
5. So why not always use on-policy learning?
Because on-policy learning has a brutal property:
The data expires quickly.
Imagine training version 1 of your model.
It generates:
Prompt: Prove that sqrt(2) is irrational.
Answer: ...
Reward: 0.8
You update the model.
Now you have version 2.
Why should version 2 be restricted to learning from version 1's trajectories?
It might be able to solve the problem much better.
Conversely, version 1 may have generated a brilliant proof that version 2 will almost never discover again.
This creates a strange asymmetry.
On-policy learning gives you highly relevant data:
data ~= current behavior
but potentially wastes enormous amounts of useful historical data.
Off-policy learning gives you reusable historical data:
data != necessarily current behavior
but now you have to deal with the distribution mismatch.
At a high level:
On-policy:
fresh + relevant
expensive + disposable
Off-policy:
reusable + diverse
potentially stale + mismatched
This is one reason modern LLM training increasingly looks like a hybrid system rather than a pure on-policy or pure off-policy system.
You want the freshness of on-policy data and the economics of off-policy data.
6. The math: what exactly goes wrong?
Let the model be a policy:
pi_theta(y | x)
This means:
Given prompt x, what probability does the model with parameters theta assign to answer y?
Suppose the model gets reward R(x, y).
The objective is conceptually:
J(theta) = E[R(x, y)]
where y is sampled from the model itself.
For a policy-gradient method, a basic gradient estimator looks like:
grad J(theta)
~= E[ grad log pi_theta(y | x) * R ]
The important thing is that y was sampled from:
pi_theta
Now suppose we have old data generated by another policy:
pi_old
but we want to optimize:
pi_theta
The expectation is now being taken under the wrong distribution.
One classical solution is importance sampling.
Conceptually:
E_pi_theta[f(y)]
=
E_pi_old[
pi_theta(y|x) / pi_old(y|x) * f(y)
]
The ratio
pi_theta(y|x) / pi_old(y|x)
corrects for the fact that the data came from the old policy.
This looks elegant.
It can also become horrible.
Suppose a sequence has probability:
pi_old(y|x) = 1e-6
under the old model but
pi_theta(y|x) = 1e-3
under the new model.
The importance weight is:
1e-3 / 1e-6 = 1000
One example can suddenly have 1,000 times the influence of another.
With long autoregressive sequences, probability ratios can become extremely volatile because token-level ratios multiply across the sequence.
That creates a classic RL engineering problem:
How much old data can we safely reuse before the correction becomes statistically ugly?
PPO takes a pragmatic route.
Instead of allowing arbitrary policy movement, it clips the probability ratio:
r(theta) =
pi_theta(y|x) / pi_old(y|x)
and uses an objective that effectively says:
Improve the policy, but don't benefit too much from moving far away from the policy that generated this experience.
This is one reason PPO became so popular: it turns a theoretically nasty distribution-shift problem into something engineers can actually operate.
7. The LLM world is now blurring the boundary
Here is where things get interesting.
People often casually say:
"DPO is off-policy RL."
That is useful shorthand, but technically it is better to be precise.
DPO, introduced by Rafael Rafailov and colleagues in 2023, takes preference pairs such as:
prompt
chosen answer
rejected answer
and directly optimizes the language model using those preferences.
There is no PPO rollout loop.
No reward-model inference is required during optimization.
No requirement that the current model generate every training example.
So operationally it looks much more like learning from a fixed preference dataset.
This is one reason DPO was attractive: it removed much of the machinery associated with RLHF while retaining a principled connection to the underlying reward-maximization problem.
But now consider what happens if we repeatedly generate preference data using the current model:
Model v1
|
v
generate candidates
|
v
judge / verifier
|
v
preference dataset
|
v
train Model v2
|
v
generate candidates
|
v
judge / verifier
|
v
train Model v3
You have built something that is neither simply "offline training" nor simply classical on-policy PPO.
You have a data-generation loop whose policy evolves over time.
That distinction is becoming increasingly important for reasoning models.
A model might generate thousands of candidate proofs, programs, mathematical solutions, tool-use trajectories, or chains of actions. A verifier selects successful ones. Those successful trajectories become training data.
The resulting system has an economic structure very different from ordinary SFT:
compute
-> generate attempts
-> evaluate attempts
-> retain valuable experience
-> train
-> generate better attempts
-> repeat
The bottleneck can move from gradient computation to experience generation and evaluation.
That is the deeper reason on-policy versus off-policy matters for LLM developers.
It is not merely a taxonomy of RL algorithms.
It is a question about how efficiently you turn inference compute into learning signal.
8. The practical rule for LLM engineers
A useful mental model is:
| Training setup | Where examples come from | Policy relationship |
|---|---|---|
| Pretraining | Internet/books/code | Not RL policy data |
| SFT | Humans / curated datasets | Usually off-policy |
| DPO | Preference dataset | Usually offline / off-policy |
| PPO RLHF | Current policy rollouts | On-policy-ish |
| Rejection sampling | Current/older model + verifier | Can be hybrid |
| Self-play | Evolving model(s) | Often strongly on-policy |
| Replay-buffer RL | Historical model rollouts | Off-policy |
| Synthetic-data fine-tuning | Other/older models | Off-policy |
| Iterative self-training | Previous checkpoints | Moving between policies |
The most important engineering questions therefore become:
1. Who generated this data?
Not just "is it synthetic?"
A dataset generated by your current checkpoint is very different from one generated by a model six generations ago.
2. How far away is the behavior policy?
If your current model assigns very different probabilities to the training trajectories, you have distribution shift.
3. How expensive is experience?
If generation is cheap, throwing data away may be fine.
If generation requires a giant reasoning model plus a verifier, replay becomes much more attractive.
4. How reliable is the reward?
On-policy learning can repeatedly exploit a flawed reward function.
The model gets better at finding whatever the evaluator rewards, rather than what you actually wanted.
5. Is diversity valuable?
Off-policy datasets can contain behaviors that the current model would never discover.
This can be extraordinarily valuable in reasoning and coding.
Imagine your model has a 0.01% probability of discovering a particular algorithm.
An on-policy system might need an enormous number of rollouts to find it.
An external expert, stronger model, or historical checkpoint might already have produced it.
In that situation, insisting on on-policy data is throwing away information.
The interesting design space is therefore not:
"Should I use on-policy or off-policy training?"
It is:
Which experiences should be generated by the current policy, which should be harvested from other policies, and how aggressively should each kind be reused?
That is a much more useful question.
And it points toward a future where the training system itself looks increasingly like an experience-management system:
+------------------+
| Current model |
+--------+---------+
|
generate
|
v
+-------------+
| Evaluator |
+------+------+
|
+---------+---------+
| |
v v
fresh experience replay buffer
| |
+---------+---------+
|
v
policy update
|
v
new model
The winning systems may not be the ones that are purely on-policy or purely off-policy.
They may be the ones that are best at deciding when fresh experience is worth paying for and when old experience is still valuable.
That is ultimately an economics problem disguised as a reinforcement-learning problem.
And for LLMs, the economics are unusually stark: every trajectory is potentially an expensive experiment, and every successful trajectory is potentially reusable training capital.
9. The takeaway
The simplest way to remember the distinction is:
On-policy learning learns from what the model currently does. Off-policy learning learns from what some behavior did.
On-policy training gives you data that is tightly matched to the policy being optimized, but generating that data can be enormously expensive.
Off-policy training lets you reuse experience, mix data from different models, and learn from behavior the current model might never discover. But now you inherit the statistical problem of distribution mismatch.
Classical RL encountered this decades ago with Q-learning, actor-critic methods, and policy-gradient algorithms. LLMs have simply made the underlying tradeoff vastly more expensive and more consequential.
InstructGPT demonstrated the power of policy optimization for language models. PPO provided a practical mechanism for repeatedly improving a policy from its own rollouts. DPO subsequently showed how much of the preference-learning problem could be reformulated as direct optimization on a fixed dataset.
The next step is arguably more interesting.
Once models can generate enormous amounts of candidate reasoning, code, proofs, tool trajectories, and other experiences—and automated evaluators can decide which experiences are valuable—the central training question becomes:
How should an LLM decide which of its experiences to learn from, how many times to reuse them, and when to spend compute generating new ones?
That sounds less like traditional fine-tuning and more like building a learning system with a memory.
Do you think future frontier-model training will converge toward predominantly on-policy learning, or will replaying and intelligently curating experience become the bigger competitive advantage?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (1)
On-policy learning uses the current model's own outputs, fresh but expensive and disposable. Off-policy reuses historical data, cheap but distribution-mismatched. Modern LLM training is becoming a hybrid experience-management economics problem.