Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Suppose you have a good base language model.
It can write code, summarize documents, answer questions, and explain things. But you want it to behave more like your product:
- follow instructions more reliably
- prefer concise answers
- refuse certain requests
- write in a particular style
- produce better code
- optimize for a human notion of "good"
A natural question is: should you use RLHF, PPO, or DPO?
There is a terminology trap here.
RLHF is a training paradigm. PPO is an optimization algorithm. DPO is a preference-optimization objective that can replace the reward-model-plus-RL part of the traditional RLHF pipeline.
That distinction matters because it changes what infrastructure you need, what data you collect, how expensive training becomes, and what kinds of behavior you can realistically optimize.
The short intuition is:
DPO says: "Here are two answers. Make the preferred one more likely."
PPO-based RLHF says: "Here's a learned reward function. Change the model to maximize it, while staying reasonably close to the old model."
That sounds like a small implementation detail.
It isn't.
1. Where These Methods Came From
The story starts before today's instruction-tuned LLMs.
In 2017, Paul Christiano, Jan Leike and colleagues demonstrated that a reinforcement-learning agent could learn complicated behavior from human comparisons rather than a hand-written reward function. Humans did not have to say "the robot gets +17.3 reward for doing X"; they simply compared two behaviors and indicated which was better.
This became one of the foundations for what we now call reinforcement learning from human feedback, or RLHF.
OpenAI later applied the basic idea directly to language. In 2020, researchers trained a reward model from human preferences over summaries and then optimized a language model against that reward using PPO. The experiment used about one million sampled episodes, and the RL stage included a KL penalty to keep the resulting policy from drifting too far from the supervised model.
Then came InstructGPT in 2022.
The recipe was approximately:
pretrained GPT
|
v
supervised fine-tuning
|
v
human preference comparisons
|
v
reward model
|
v
PPO
|
v
aligned policy
One memorable result from that work was that a 1.3B-parameter InstructGPT model was preferred by human evaluators over the original 175B GPT-3 model on their prompt distribution.
In other words, post-training could matter more than simply making the model larger.
Then in 2023, Rafael Rafailov and colleagues introduced Direct Preference Optimization (DPO).
Their central observation was mathematically neat:
If you make some assumptions about how preferences arise, you can solve the KL-regularized RLHF objective analytically and rewrite it as a supervised classification-style loss.
So instead of:
preference data
|
v
reward model
|
v
RL sampler
|
v
PPO
|
v
policy
you can do:
preference data
|
v
DPO loss
|
v
policy
That removes a substantial amount of machinery.
2. First Get the Mental Model Right
Imagine you give a model this prompt:
Explain TCP congestion control to a backend engineer.
It generates:
Answer A
TCP congestion control manages transmission rate to prevent network congestion. It uses mechanisms including slow start, congestion avoidance, retransmission detection, and window adjustment...
Answer B
TCP congestion control is basically TCP asking: "How fast can I send packets without angering the network?" It starts cautiously, increases its sending rate, and backs off when packet loss indicates congestion...
A human evaluator chooses B.
You can record that as:
x = prompt
y+ = preferred answer
y- = rejected answer
Now ask:
What exactly should training do with this information?
DPO takes the direct route:
increase P(y+ | x)
decrease P(y- | x)
Traditional RLHF introduces an intermediate concept:
human preference
|
v
reward model
"How good is this response?"
|
v
RL optimization
"Change the policy so it generates
responses with higher reward."
PPO is then the mechanism used to perform that policy optimization.
This gives us the most useful terminology map:
| Term | What it is | Main job |
|---|---|---|
| RLHF | Training paradigm | Learn behavior from preferences |
| Reward model | Learned scoring function | Approximate human preference |
| PPO | RL optimizer | Improve policy against reward |
| DPO | Preference-learning objective | Directly train policy from comparisons |
So comparing "DPO vs RLHF" is slightly like comparing "cross-entropy vs machine learning."
They operate at different abstraction levels.
In practice, though, people often say "PPO vs DPO" because PPO-based RLHF is the major alternative to DPO.
3. What PPO-Based RLHF Actually Optimizes
Let's look at the mathematics without getting lost in reinforcement-learning notation.
Suppose the language model is our policy:
pi_theta(y | x)
where:
-
xis the prompt -
yis the generated response -
thetaare the model parameters
We train a reward model:
r_phi(x, y)
which tries to predict what humans prefer.
A typical preference pair is:
(x, y+, y-)
where y+ is preferred over y-.
A common model for the preference probability is the Bradley-Terry formulation:
P(y+ preferred to y- | x)
= sigmoid(r(x,y+) - r(x,y-))
So if:
r(x,y+) = 3.0
r(x,y-) = 1.0
the model predicts that y+ should be preferred with probability:
sigmoid(2)
≈ 0.88
The reward model is trained to reproduce these preferences.
Now we have a problem.
If we simply tell the LLM:
maximize reward
it can exploit flaws in the reward model.
For example, imagine the reward model has accidentally learned:
longer answers tend to be better.
The policy may respond by producing increasingly bloated answers.
This is one reason the RL objective normally contains a KL penalty:
objective
=
expected reward
-
beta * KL(policy || reference_policy)
The first term says:
Become better according to the reward model.
The second says:
Don't move too far away from the original behavior.
That second term is extremely important.
Without it, the optimizer can discover bizarre high-reward regions that humans never intended.
PPO then performs policy-gradient updates while limiting how dramatically the policy changes in each optimization step. PPO was originally introduced in 2017 as a simpler alternative to more complicated trust-region policy methods.
At a high level, PPO asks:
Did this sampled response get a good advantage?
If yes:
increase its probability somewhat
If no:
decrease its probability somewhat
But:
don't change probabilities too aggressively
That "somewhat" is the important part.
PPO's clipped objective prevents the new policy from moving too far relative to the policy that generated the samples.
This is useful because language models are enormous policies with a particularly awkward action space:
token 1
-> token 2
-> token 3
-> ...
One completion might contain hundreds of token-level decisions.
RL therefore has a serious engineering problem:
you must repeatedly generate model outputs, score them, compute policy-gradient signals, and update the model.
That is much more operationally complicated than ordinary supervised fine-tuning.
4. What DPO Changes
DPO starts from the same preference data:
(prompt, preferred_response, rejected_response)
but asks a different mathematical question.
Consider the KL-regularized objective:
maximize
E[r(x,y)]
-
beta * KL(pi(y|x) || pi_ref(y|x))
For a fixed prompt, the optimal policy has the form:
pi*(y|x)
∝
pi_ref(y|x) * exp(r(x,y) / beta)
In words:
Start with the reference model's distribution, then increase the probability of high-reward responses.
Rearranging that relationship gives an implicit reward:
r(x,y)
=
beta * log(pi(y|x) / pi_ref(y|x))
+ constant(x)
Now consider two answers to the same prompt.
The prompt-dependent constant cancels:
r(x,y+) - r(x,y-)
=
beta * [
log(pi(y+|x) / pi_ref(y+|x))
-
log(pi(y-|x) / pi_ref(y-|x))
]
So we can directly construct a preference objective from the policy and the reference model.
The resulting DPO loss is approximately:
L_DPO
=
- log sigmoid(
beta * (
log(pi(y+|x) / pi_ref(y+|x))
-
log(pi(y-|x) / pi_ref(y-|x))
)
)
The intuition is simpler than the notation:
Push the preferred response's probability upward relative to the reference model, while pushing the rejected response downward.
The reference model acts as the anchor.
Suppose, for a particular pair, we have:
pi_ref(y+) = 0.001
pi_ref(y-) = 0.002
pi(y+) = 0.004
pi(y-) = 0.001
Then:
preferred ratio
= 0.004 / 0.001
= 4
rejected ratio
= 0.001 / 0.002
= 0.5
The model has moved toward the preferred response by a factor of four while making the rejected response half as likely relative to its reference probability.
That's the central DPO idea.
No separate reward model.
No rollout-and-update RL loop.
No PPO.
No value function.
No explicit policy-gradient infrastructure.
DPO therefore looks much more like ordinary supervised fine-tuning from an engineering perspective.
The original DPO paper reported competitive results with PPO-based RLHF while requiring a substantially simpler training procedure.
5. So When Would You Actually Use Each?
Use DPO when your problem is "the model usually knows how to do this; I just want it to prefer behavior A over behavior B."
This is the natural DPO regime.
Examples:
concise answer > verbose answer
correct code > plausible-looking code
company style > generic style
safe refusal > unsafe completion
use citations > don't use citations
structured JSON > free-form prose
You already have preference comparisons, and you want to nudge the model toward the preferred distribution.
DPO is particularly attractive when:
- preference data is available offline
- you want a relatively simple training pipeline
- you don't need an elaborate online reward optimization loop
- your objective can be expressed reasonably well as pairwise preferences
For many LLM post-training projects, this is the first method I would try.
Use PPO-based RLHF when the reward itself is central.
PPO becomes more interesting when you have a meaningful scalar reward that you want the model to optimize through exploration.
Imagine a coding agent.
You don't merely have:
response A > response B
You can potentially execute the generated code.
Then you might have:
compile succeeds +1
tests pass +5
tests fail 0
security violation -10
latency penalty -2
Now your training signal is not merely a static preference dataset.
It is an environment.
The model generates something.
You evaluate it.
The result changes the reward.
The model generates something else.
This is much closer to classical reinforcement learning.
PPO becomes more compelling when the training loop looks like:
prompt
|
v
generate trajectory
|
v
interact with environment
|
v
receive reward
|
v
update policy
|
v
generate again
That is fundamentally different from ordinary DPO training on a fixed dataset.
Use "RLHF" when you are describing the entire preference-training system.
This is useful organizationally.
A production RLHF pipeline might be:
1. collect prompts
2. generate candidate responses
3. obtain human preferences
4. train reward model
5. generate fresh samples
6. score samples
7. optimize policy with PPO
8. evaluate policy
9. repeat
Calling the whole thing "PPO" loses the data-collection and reward-model pieces.
Calling the whole thing "DPO" is also wrong because DPO is one particular preference-optimization method.
6. The Economics: Where the Complexity Actually Goes
The most interesting difference between DPO and PPO is not the loss function.
It is the operations bill.
Consider a simplified example.
Suppose you have:
10 million preference comparisons
and each response averages:
500 tokens
For DPO, the core training data is already there:
10M pairs
x
2 responses
x
500 tokens
≈ 10 billion response tokens
That is substantial, but the training structure is familiar:
dataset
-> minibatch
-> forward pass
-> loss
-> backward pass
-> update
PPO introduces additional work.
You need to repeatedly:
sample responses
evaluate rewards
run policy/value computations
calculate advantages
perform PPO updates
manage reference-policy comparisons
repeat
And because the policy is changing during training, the data-generation and optimization stages are coupled.
This creates an important systems effect:
DPO can spend most of its compute in predictable offline training. PPO spends a meaningful fraction of its complexity on the training loop itself.
For a large model, that difference affects:
- GPU utilization
- inference capacity
- rollout throughput
- memory requirements
- checkpointing
- debugging
- hyperparameter tuning
- reproducibility
- engineering headcount
There is also a less obvious cost.
Suppose your reward model has a subtle bug.
With DPO, the defect is largely in the preference dataset.
With PPO-based RLHF, the defect can interact with an optimizer that is actively searching for ways to exploit the reward.
That makes reward hacking a systems problem, not merely a data-quality problem.
A toy example:
reward_model:
helpfulness = good
verbosity = slightly good
PPO discovers:
"If I write 8,000 words,
the reward goes up."
The optimizer is doing exactly what you asked.
Your reward function was the problem.
This is why reward-model validation can be as important as the RL algorithm itself.
7. The Developer's Decision Tree
For a real LLM project, I would think about it this way.
Start with SFT
First ask:
Can supervised examples solve the problem?
If yes, start there.
A demonstration dataset of:
prompt -> ideal response
is often the cleanest signal.
Then ask whether the desired behavior is easier to express as preferences
If humans can easily answer:
Which response is better?
then DPO is a strong candidate.
prompt
|
+--> response A
|
+--> response B
human:
B is better
You can accumulate many such comparisons and train directly.
Then ask whether you actually have an environment or meaningful scalar reward
If the answer is yes:
generate
->
execute
->
observe outcome
->
reward
->
generate again
then RL becomes much more interesting.
At this point PPO or another modern policy-optimization method may be justified.
A useful heuristic
Think of the three choices this way:
SFT
"Show the model what good looks like."
DPO
"Show the model which of two behaviors is better."
PPO-based RLHF
"Give the model a reward function and let optimization
search for better behavior."
That leads to a practical matrix:
| Situation | SFT | DPO | PPO-based RLHF |
|---|---|---|---|
| Have ideal demonstrations | Excellent | Useful | Usually unnecessary |
| Have pairwise preferences | Possible but indirect | Excellent | Excellent |
| Have scalar reward | Weak fit | Possible | Excellent |
| Have interactive environment | Weak fit | Weak fit | Excellent |
| Want simplest pipeline | Excellent | Excellent | Poor |
| Need exploration | No | Limited | Yes |
| Reward model is important | No | No explicit RM | Yes |
| Need online optimization | No | Usually no | Yes |
| Easy to iterate with offline data | Excellent | Excellent | Harder |
| High infrastructure complexity | Low | Low | High |
8. One More Important Point: DPO Is Not "RLHF Without RL" in Every Sense
It is tempting to describe DPO as:
"RLHF, except simpler."
That is useful as a first approximation, but mathematically it is more precise to say:
DPO derives a direct preference-learning objective from a particular KL-regularized RLHF formulation.
That means DPO inherits assumptions from the underlying formulation.
For example, the preference data matters enormously.
Suppose your dataset contains:
y+ = answer with useful reasoning
y- = answer with obvious nonsense
DPO has learned something useful.
But suppose the pairs are:
y+ = slightly more concise
y- = slightly more detailed
while your real product objective is factual correctness.
The model may become better at concision without becoming more truthful.
DPO does not magically solve preference specification.
It makes the optimization simpler.
The same principle applies to PPO.
PPO can optimize almost anything you can turn into a usable reward.
That is both its strength and its danger.
A weak reward becomes an optimization target.
A weak preference dataset becomes a training signal.
Neither optimizer knows what you really meant.
This is why the most important engineering artifact in alignment work is often not the optimizer configuration.
It is the evaluation suite.
Before asking:
"Should I use DPO or PPO?"
ask:
"How will I know that the model got better?"
That question determines almost everything downstream.
9. Final Takeaway
The modern LLM post-training stack is easier to understand once you stop treating DPO, PPO, and RLHF as competing names for the same thing.
The conceptual hierarchy is:
Preference-based alignment
|
+-----------+-----------+
| |
DPO RLHF-style pipeline
|
+-------+-------+
| |
reward model RL
|
PPO
DPO is attractive because it turns a complicated RLHF optimization problem into something much closer to ordinary supervised learning.
PPO is attractive because it gives you an actual policy-optimization mechanism that can consume a learned reward and, more importantly, can fit naturally into interactive environments.
RLHF is the broader architecture: collect preference information, build a reward signal, and optimize behavior against it.
So for a developer building an LLM today, the pragmatic order is often:
SFT
|
v
Do I have good preference pairs?
|
yes
|
v
DPO
|
v
Do I actually need online reward optimization,
exploration, or environment interaction?
|
yes
|
v
RL / PPO-style training
The interesting part is that the algorithm is only half the problem.
The other half is deciding what "better" means—and constructing data or environments that make that definition measurable.
For your own LLM projects, where do you think the boundary lies: would you rather start with the simplicity of DPO, or accept PPO's complexity when the reward signal gives you something you cannot express cleanly as preference pairs?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)