DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

DPO: aligning a model from preference pairs with one loss — no reward model, no RL

#llm #ai #machinelearning #beginners

RLHF aligns a language model in three stages: fine-tune it, train a separate reward model on human preference pairs, then run PPO to chase that reward while a KL penalty keeps the policy near a reference. It works, but it is heavy machinery — three models in memory, an unstable reinforcement-learning loop, and a proxy reward the policy can learn to hack. DPO (Direct Preference Optimization) proved you don't need the middle two. I built a demo that trains a tiny DPO policy live in the browser, and the striking thing is how little is left once the reward model and the RL loop are gone.

The insight: the RLHF optimum is closed-form

The RLHF objective — maximise reward minus a β-weighted KL to the reference — has a known closed-form solution. The optimal policy is the reference re-weighted by the reward:

π*(y|x) ∝ π_ref(y|x) · exp(r(x,y) / β)
Enter fullscreen mode Exit fullscreen mode

DPO simply rearranges this to solve for the reward in terms of the policy:

r(x,y) = β · ( log π(y|x) − log π_ref(y|x) ) + const
Enter fullscreen mode Exit fullscreen mode

That algebra is the entire trick. The reward can be read off the policy itself — so no reward model is ever built.

The implicit reward

That rearrangement gives the implicit reward: how much more probability the trainable policy puts on an answer than a frozen reference does, scaled by β.

def implicit_reward(x, y):
    return beta * (policy.logprob(y|x) - ref.logprob(y|x))
Enter fullscreen mode Exit fullscreen mode

The additive constant cancels whenever you compare two answers to the same prompt — which is exactly what a preference pair is — so you never have to know it.

Reuse Bradley–Terry, but on the policy

RLHF trained its reward model with the Bradley–Terry loss, −log σ(r_chosen − r_rejected). DPO plugs the implicit reward into that identical loss. Because the implicit reward is the policy's own log-prob gap to the reference, minimizing this loss trains the policy directly — the reward model has been algebraically dissolved into it:

def dpo_loss(x, y_chosen, y_rejected):
    d_chosen   = policy.logprob(y_chosen  |x) - ref.logprob(y_chosen  |x)
    d_rejected = policy.logprob(y_rejected|x) - ref.logprob(y_rejected|x)
    margin = beta * (d_chosen - d_rejected)      # r(chosen) - r(rejected)
    return -log_sigmoid(margin)                  # push chosen above rejected
Enter fullscreen mode Exit fullscreen mode

It is a binary classification loss: "is the chosen answer preferred to the rejected one?"

β is the KL leash, folded into the loss

In RLHF the KL penalty was a separate term subtracted during PPO. In DPO the same regularization lives inside the loss through β, which scales the log-prob gap Δ into the reward margin m = β·Δ:

  • Large β — a small gap already makes a big margin, so the loss saturates early and the policy stays glued to the reference (small drift).
  • Small β — only a large gap satisfies the loss, so the policy strays far and prefers chosen answers hard, at the risk of degenerating.

Same reference-anchoring as RLHF's KL term, but no explicit KL term and no reference sampling during a step. In the demo, sliding β from 0.05 to 1.0 visibly changes how far the policy drifts from the reference for the same fit.

Training is plain supervised learning

Because the loss depends only on log-probs of given answers, there is no sampling from the policy, no reward-model forward pass, no advantage estimate, no clipped RL update. You iterate over preference pairs in mini-batches and take a normal gradient step:

for batch in dataloader(prefs):
    loss = mean(dpo_loss(x, yc, yr) for (x, yc, yr) in batch)
    loss.backward()          # one backward pass
    optimizer.step()         # one gradient step — that's it
    optimizer.zero_grad()
Enter fullscreen mode Exit fullscreen mode

Compare that to RLHF's rollout → score → PPO-update dance. It is stable, cheap, and reproducible.

What DPO gives up

It is not magic. DPO optimizes on a fixed set of preferences (offline), so it cannot explore new answers the way online PPO can. With small β it tends to push the chosen and rejected log-probs apart — sometimes lowering both — so over-training can hurt fluency even as the margin grows. And it still needs a good SFT model and clean preference data. Choose β well, though, and it matches or beats PPO-RLHF for a fraction of the effort, which is why DPO and its cousins (IPO, KTO, ORPO) align most open models today.

Set β, press train, and watch the loss fall while the chosen−rejected margin grows — every gradient step is real code:
https://dev48v.infy.uk/ai/days/day35-dpo.html

Top comments (0)