DPO, IPO, KTO, ORPO: What Changes When You Teach an LLM What "Good" Means?

Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

---

There is a slightly strange thing about modern LLM training.

We spent years making language models better by giving them more text.

Then we discovered that a 1.3B model trained with human feedback could be preferred to a 175B GPT-3 model on instruction-following tasks. The problem was no longer simply "does the model know enough?" It was increasingly:

Given several things the model could say, can we teach it which one we actually want?

That observation led to RLHF, and eventually to a remarkably productive line of work: preference optimization.

DPO, IPO, KTO and ORPO can look like an alphabet soup of slightly different loss functions. They are easier to understand if you see them as answers to different engineering questions:

• Do I need a separate reward model?
• Do I need reinforcement learning?
• Do I need pairs of answers, or are thumbs-up/thumbs-down labels enough?
• Do I need a reference model?
• What happens when the model learns a preference too aggressively?
• Can I combine ordinary SFT and preference learning into one training run?

This article builds the story from the engineering intuition down to the mathematics.

1. First: why does next-token prediction not give us the model we want?

Suppose you train a model on:

"The capital of France is"

and it learns to predict:

"Paris"

Excellent.

But now give it:

"Explain quantum mechanics to a five-year-old."

There isn't one objectively correct next token. There are thousands of plausible continuations.

The pretraining objective is approximately:

maximize  log P(y | x)

where x is the context and y is the observed continuation.

But what we often care about is closer to:

maximize  human_quality(x, y)

Those are not the same objective.

This distinction became painfully concrete in early RLHF work.

In OpenAI's 2020 summarization work, researchers found that a 1.3B parameter model trained using human feedback could outperform a much larger supervised model. They also discovered something amusing and instructive: human labelers preferred longer summaries, so the RL-trained model learned to converge toward the maximum permitted summary length. In other words, the model was doing exactly what the feedback system incentivized.

By 2022, this had become the basic InstructGPT recipe:

pretraining
    |
    v
supervised fine-tuning
    |
    v
collect human preferences
    |
    v
train reward model
    |
    v
PPO / RL
    |
    v
aligned model

InstructGPT produced a particularly memorable result: human evaluators preferred outputs from a 1.3B parameter InstructGPT model over those from the original 175B GPT-3, despite the enormous difference in parameter count.

That was the important conceptual transition.

Scaling gives you capabilities. Preference optimization changes which capabilities get expressed.

2. The RLHF machine is powerful — and annoyingly complicated

Let's make the traditional setup concrete.

Suppose the prompt is:

Write a concise explanation of TCP congestion control.

The model produces two candidates:

A: TCP increases its sending rate until packets start getting dropped...
B: TCP is a protocol used for communication between computers...

A human prefers A.

Do this many thousands of times.

You now have data of the form:

(prompt, preferred_answer, rejected_answer)

The classical RLHF pipeline converts this into a reward-learning problem.

Step 1: train a reward model

Train:

R(x, y) -> scalar reward

so that:

R(x, preferred) > R(x, rejected)

Step 2: optimize the language model against that reward

Now the policy tries to maximize:

E[R(x, y)]

while usually being constrained not to wander too far from the original SFT model:

maximize

E[R(x, y)] - beta * KL(policy || reference)

The KL term is important.

Without it, imagine giving the model a reward for "being helpful." It may discover bizarre ways of maximizing whatever the reward model happens to measure.

The KL penalty says:

"Improve according to the reward, but don't completely reinvent yourself."

This worked remarkably well.

It was also operationally expensive.

You have a policy, a reference model, a reward model, rollouts, and an RL optimizer. PPO introduces its own collection of engineering knobs.

The original OpenAI summarization work gives a useful historical sense of the cost: their 6.7B summarization model required roughly 320 GPU-days for RL fine-tuning.

This is the environment in which DPO appeared.

3. DPO's big trick: turn RL into ordinary classification

In 2023, Rafael Rafailov and collaborators noticed something mathematically beautiful.

If your RLHF objective has the usual KL regularization, you can derive the optimal policy in closed form.

The result gives an implicit relationship between the reward and the policy:

r(x, y)
  =
beta * log( pi(y|x) / pi_ref(y|x) )
  + constant(x)

The annoying unknown constant(x) disappears if we compare two answers.

For a preferred answer yw and rejected answer yl:

r(x, yw) - r(x, yl)

=
beta * [
    log pi(yw|x) / pi_ref(yw|x)
  - log pi(yl|x) / pi_ref(yl|x)
]

Now suppose human preferences follow a Bradley-Terry model:

P(yw preferred to yl)
    =
sigma( r(x,yw) - r(x,yl) )

Substitute the policy expression into it.

You get a loss that looks like ordinary binary classification:

L_DPO
=
- log sigma(
    beta * (
        log pi(yw|x) / pi_ref(yw|x)
      - log pi(yl|x) / pi_ref(yl|x)
    )
  )

That's DPO.

The crucial engineering consequence is:

You no longer need to train an explicit reward model or run PPO.

You have turned:

preference data
      |
      v
reward model
      |
      v
RL

into:

preference data
      |
      v
cross-entropy-like loss
      |
      v
gradient descent

Rafailov et al. showed that DPO could achieve results comparable to or better than PPO-based RLHF on several alignment tasks while being substantially simpler to train.

For an engineer, this is the real contribution of DPO.

It wasn't merely "another alignment loss."

It changed the operations model of preference training.

4. But DPO has an uncomfortable property: it can keep pushing forever

Here's where IPO becomes interesting.

Consider the simplified DPO situation.

The model has already learned:

preferred answer: 0.90
rejected answer: 0.01

You might think:

"Great. The model has learned the preference. We're done."

But look at the DPO loss.

The model is rewarded for increasing the preference margin further.

Conceptually:

small margin  -> large gradient
large margin  -> smaller gradient
huge margin   -> tiny gradient

But the optimum can still lie at an effectively infinite margin.

That creates a theoretical problem under sufficiently deterministic/separable preference data: the implicit reward gap can continue growing instead of settling at a finite value.

This is precisely the issue analyzed by Azar and collaborators in their general theoretical treatment of learning from human preferences. Their analysis shows that DPO can over-train in settings where the preference data are effectively deterministic, while their Identity Preference Optimization (IPO) construction has much stronger resistance to this behavior.

IPO changes one important thing

Define:

m =
    log pi(yw|x) / pi_ref(yw|x)
  - log pi(yl|x) / pi_ref(yl|x)

DPO essentially says:

"Make m larger."

IPO says:

"Make m approach a particular finite target."

The simplified IPO objective is:

L_IPO
=
(
    m - 1/(2*beta)
)^2

So instead of an endless hill:

        /
       /
      /
-----/

you get a bowl:

       \     /
        \   /
         \_/

The distinction is subtle but important.

A numerical example

Suppose:

beta = 0.1

Then the target margin is approximately:

1 / (2 * 0.1) = 5

If your current margin is:

m = 2

then:

loss = (2 - 5)^2 = 9

If the model gets to:

m = 5

then:

loss = 0

If it keeps going to:

m = 10

then:

loss = (10 - 5)^2 = 25

The model is no longer rewarded for becoming arbitrarily confident.

It is being asked to hit a target.

That is the essential intuition behind IPO.

5. KTO asks a different question: what if my data isn't paired?

Here is a practical problem that is easy to underestimate.

DPO wants:

prompt
  -> answer A
  -> answer B

human says:
A > B

But that's not necessarily how production feedback arrives.

Imagine you operate a coding assistant.

You have:

User: "Fix this SQL query."

Response: [model output]

User clicks 👍

or:

User clicks 👎

You have a label.

But you don't necessarily have a competing answer.

And generating a matched alternative and asking someone to compare the two costs additional annotation effort.

This motivated Kahneman-Tversky Optimization, or KTO.

KTO was introduced by Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky and Douwe Kiela in 2024. The paper connects preference optimization to prospect theory, the behavioral-economic framework associated with Daniel Kahneman and Amos Tversky.

The key data structure changes from:

(x, yw, yl)

to:

(x, y, desirable?)

For example:

("Explain TCP", response_1, good)
("Explain TCP", response_2, bad)
("Explain TLS", response_3, good)
("Explain TLS", response_4, good)

No pairing is required.

Why "Kahneman-Tversky"?

The important psychological idea is that humans do not evaluate outcomes as simple absolute utilities.

We evaluate them relative to a reference point.

And losses often hurt more than equivalent gains feel good.

KTO incorporates this sort of asymmetric utility into the optimization objective.

You don't need to memorize the entire derivation to understand the engineering idea:

DPO:
    "Make A more likely than B."

KTO:
    "Make this particular answer more desirable
     relative to a reference point."

That seemingly small change has a major data-engineering consequence.

You can use naturally occurring binary feedback.

The KTO paper reports competitive performance from 1B through 30B parameter models despite using only binary desirable/undesirable signals rather than pairwise preferences.

For a production system, that can matter more than the elegance of the loss function.

6. ORPO takes the opposite engineering shortcut: get rid of the reference model

There is another cost hiding inside DPO and KTO.

They typically need a reference policy:

pi_ref

Why?

Because the optimization is partly about:

"Move toward the preferred response, but measure how much you have moved relative to the starting/reference model."

That means another model has to participate in the computation.

ORPO, introduced by Jiwoo Hong, Noah Lee and James Thorne, asks:

Can we combine SFT and preference optimization into a single objective without a separate reference model?

The answer is yes.

The basic ORPO idea is:

L_ORPO
=
L_SFT
+
lambda * L_preference

The preference component works with an odds ratio between the preferred and rejected responses.

Define the probability of a response under the model as:

p_w = pi(yw | x)
p_l = pi(yl | x)

The odds for a response are:

odds(y) = p(y) / (1 - p(y))

ORPO compares the odds of the chosen and rejected completions and adds a penalty that encourages the chosen response to dominate the rejected one.

The important thing isn't memorizing the exact implementation.

It's the architecture:

DPO:

SFT model
   |
   +----> reference model
   |
   +----> preference optimization

versus:

ORPO:

one model
   |
   +----> SFT loss
   |
   +----> preference/odds loss

So ORPO is attractive when your primary engineering objective is:

"Can I do this in one ordinary fine-tuning stage?"

The original ORPO paper reports experiments from 125M to 7B parameters and showed strong results on models including Phi-2, Llama-2 and Mistral. It also reports strong results from training on UltraFeedback alone.

There is an important philosophical difference here.

IPO is primarily a theoretical correction to preference optimization.

KTO is primarily a change in the feedback interface.

ORPO is primarily an architectural/operational simplification.

7. What should an engineer actually choose?

Here's the useful mental model.

Method	Data	Reference model?	Separate RM?	Main idea
RLHF/PPO	Pairwise preferences	Usually	Yes	Explicit reward + RL
DPO	Chosen/rejected pairs	Yes	No	Convert RLHF into classification
IPO	Chosen/rejected pairs	Yes	No	Give the preference margin a finite target
KTO	Good/bad examples	Yes	No	Optimize reference-dependent human utility
ORPO	Chosen/rejected pairs	No	No	Combine SFT and preference learning

This gives a surprisingly simple decision tree.

You have clean pairwise preferences?

Start with DPO.

It is the baseline against which most of the family makes sense.

You worry about over-training or deterministic preference data?

Try IPO.

Its defining idea is that "more preference margin" should not automatically mean "better forever."

Your real feedback is thumbs-up/thumbs-down?

KTO becomes interesting.

The ability to consume unpaired binary feedback can be worth more than marginal differences between preference losses.

You want the simplest single-stage fine-tuning pipeline?

Consider ORPO.

You trade the reference-model machinery for a combined SFT + preference objective.

The economics are more interesting than the loss functions

Suppose you are building a production coding assistant.

You need 1 million useful preference signals.

There are at least two ways to get them.

Pairwise labeling

For each prompt:

generate A
generate B
human compares A vs B

You have:

1 annotation = 1 preference pair

Binary feedback

Instead:

generate A
user clicks 👍 / 👎

Now every production interaction can potentially become training data.

The value of KTO is therefore not simply:

"Its loss function is clever."

It is:

It potentially changes the marginal cost and volume of preference data.

If a human comparison costs $0.05, then:

1,000,000 comparisons * $0.05
= $50,000

That is before considering generation, quality control, adjudication, and infrastructure.

If an application already generates millions of responses and users naturally provide binary feedback, the economics can look completely different.

This is why I would not choose an alignment algorithm from benchmark tables alone.

The data-generation mechanism is part of the algorithm.

One deeper way to understand the whole family

There is a useful abstraction hiding underneath all these papers.

Almost every preference optimizer is making choices along three axes:

1. What information do I get?

pairwise:
    A > B

binary:
    A = good

scalar:
    A = 4.2 / 5

2. What notion of human preference do I assume?

DPO effectively uses a Bradley-Terry/logistic model.

IPO changes the way preference information is mapped into the optimization target.

KTO introduces a prospect-theoretic utility model.

3. How strongly do I constrain the new policy?

You can think of the reference model as saying:

"Improve, but stay near here."

The parameter controlling that tradeoff is economically similar to a regularization price.

A small beta means:

preference signal is powerful
deviation is relatively cheap

A large beta means:

preference signal is weaker
deviation from the reference is expensive

This gives a useful engineering interpretation of the hyperparameter:

beta is not just a mysterious knob. It controls how much behavioral change you are willing to buy with your preference data.

And that immediately suggests an operational reality:

The optimal setting depends on how trustworthy your preference data are.

If your labels are extremely noisy, aggressive optimization is dangerous.

If your labels are highly informative and the base model is already good, weak optimization may simply fail to move the model enough.

A tiny worked example

Suppose we have:

Prompt:
"Explain why TCP uses a congestion window."

Preferred:
"TCP limits the amount of unacknowledged data in flight
using a congestion window, adapting it based on signals
such as packet loss or increasing delay."

Rejected:
"TCP uses encryption to prevent network congestion."

Our model initially assigns:

P(preferred) = 0.02
P(rejected)  = 0.01

The preference ratio is:

0.02 / 0.01 = 2

Good, but not very decisive.

After preference optimization we might get:

P(preferred) = 0.10
P(rejected)  = 0.001

Now:

0.10 / 0.001 = 100

The model has learned something useful.

But there is a trap.

If we continue optimizing the same preference pair, we might get:

P(preferred) = 0.50
P(rejected)  = 0.000001

The model is now extraordinarily confident.

That confidence may be justified.

Or it may simply mean that the model memorized the local training distinction.

This is the fundamental tension behind the DPO -> IPO progression:

learn the preference
        |
        v
how much should we keep pushing?
        |
        v
what does "enough" mean?

IPO's answer is unusually explicit:

There is a target preference margin.
Stop treating infinite confidence as an improvement.

What is worth remembering

You do not need to memorize five alignment algorithms.

Remember this progression:

RLHF
 |
 | remove reward model + PPO
 v
DPO
 |
 +--> control over-training
 |        |
 |        v
 |       IPO
 |
 +--> remove paired-data requirement
 |        |
 |        v
 |       KTO
 |
 +--> remove reference model + combine SFT
          |
          v
         ORPO

The deeper story is not about acronym proliferation.

It is about progressively removing assumptions and infrastructure from the original RLHF recipe.

RLHF says:

Learn what humans want, then use RL to optimize for it.

DPO says:

We can derive the reward implicitly; just optimize the preference data directly.

IPO says:

Be careful: preference optimization can keep increasing confidence after it has already learned the distinction.

KTO says:

Maybe humans don't need to compare two answers at all.

ORPO says:

Maybe we don't need a separate preference stage or reference model either.

That is a very recognizable pattern in machine learning.

A complicated training pipeline appears first.

Then researchers discover a mathematical identity that removes one component.

Then someone discovers a better objective.

Then someone notices the data doesn't naturally arrive in the assumed format.

Then someone optimizes away another model.

The eventual winner is often not the method with the prettiest equation.

It is the one whose assumptions match the data and whose operational costs match the product.

If you were building a real LLM product today, which constraint would you optimize around first:

GPU cost, annotation cost, preference-data quality, or training stability?

That choice may tell you more about whether you should use DPO, IPO, KTO or ORPO than any leaderboard ever will.

---

*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

HexmosTech / git-lrc

Free, Micro AI Code Reviews That Run on Git Commit

| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |

git-lrc

Free, Micro AI Code Reviews That Run on Commit

 

     

---

GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.

In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen

At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…

View on GitHub