Direct Preference Optimization: Why You Can Align a Model Without a Reward Model

Reinforcement learning from human feedback, usually shortened to RLHF, made modern chat models usable. It also made them complicated. The standard pipeline trains a reward model on human preferences, then optimises a language model against that reward model with a reinforcement learning algorithm, while trying to stop the policy from drifting too far from its starting point. Each stage can fail in its own way, and the failures interact.

Direct Preference Optimization, or DPO, asks a sharper question. If the reward model is only ever used to tell the policy which response is better, do we actually need to learn it as a separate function? DPO shows that, under the standard RLHF objective, the optimal policy already implies a reward function. You can therefore optimise the policy directly on preference data, with a simple classification-style loss. No reward model, no reinforcement learning loop, and no separate value function.

That does not make alignment easy. It makes one part of it cleaner, and it exposes the remaining difficulties more clearly: preference data is noisy, the implied reward can be gamed, and a policy trained only on pairwise comparisons can still drift in directions the data never constrains.

The standard RLHF pipeline, in brief

Start with a supervised fine-tuned model, call it the reference policy $\pi_{\text{ref}}$ . Human annotators compare pairs of responses to the same prompt, choosing a preferred response $y_w$ and a rejected response $y_l$ . A reward model $r_\phi$ is trained to score the preferred response higher, typically with the Bradley-Terry model:

$$p(y_w \succ y_l \mid x) = \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))$$

where $\sigma$ is the logistic sigmoid and $x$ is the prompt. Then a policy $\pi_\theta$ is optimised to maximise expected reward, minus a penalty for moving away from the reference policy:

$$\max_{\pi_\theta} \mathbb{E}{x, y \sim \pi\theta} [r_\phi(x, y)] - \beta \, \mathbb{D}{\text{KL}}[\pi\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x)]$$

The KL term, weighted by $\beta$ , is doing a lot of work. Without it, the policy would exploit quirks of the reward model, producing text that scores highly but reads badly. With it, optimisation is constrained, but it is still reinforcement learning on sequences: high variance, sensitive to hyperparameters, and expensive to stabilise.

The key observation: the optimal policy implies a reward

DPO starts from a classical result. For a fixed reward function, the RLHF objective above has a closed-form optimal policy:

$$\pi^*(y \mid x) = \frac{1}{Z(x)} \, \pi_{\text{ref}}(y \mid x) \exp\left(\frac{1}{\beta} r(x, y)\right)$$

where $Z(x)$ is a normalising constant that depends only on the prompt, not on the particular response. Rearranging this equation gives the reward in terms of the policy:

$$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$$

This is the move that matters. The reward is, up to a prompt-dependent constant, just the log ratio between the optimal policy and the reference policy. When we plug this expression into the Bradley-Terry preference model, the constant $\beta \log Z(x)$ cancels, because it is the same for the preferred and rejected responses to the same prompt. What remains is a preference probability written entirely in terms of policies:

$$p(y_w \succ y_l \mid x) = \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$$

There is no reward model left in the formula. Maximising the log-likelihood of the observed preferences under this model is DPO. The loss on one preference pair is:

$$\mathcal{L}{\text{DPO}} = -\log \sigma\left(\beta \log \frac{\pi\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)$$

In words: increase the relative log-probability of the preferred response, decrease that of the rejected response, and let $\beta$ control how strongly the policy is allowed to move from the reference.

What the gradient actually does

It helps to look at the gradient, because it explains both why DPO works and why it can go wrong. Define the implicit reward of a response under the current policy as:

$$\hat{r}\theta(x, y) = \beta \log \frac{\pi\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$$

Then the gradient of the DPO loss is a weighted difference of the gradients of the log-probabilities of the two responses, where the weight is the model's current error on the pair:

$$\nabla_\theta \mathcal{L}{\text{DPO}} = -\beta \, \sigma(\hat{r}\theta(x, y_l) - \hat{r}\theta(x, y_w)) \left[ \nabla\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \right]$$

When the model already ranks the pair correctly by a large margin, the sigmoid weight is near zero and the update is small. When it ranks the pair incorrectly, or only weakly, the weight is near one and the update is large. This is the same qualitative behaviour as logistic regression, applied to whole sequences instead of single labels.

There is a subtlety hidden in that simplicity. The loss only cares about the difference between the two implicit rewards. It does not directly constrain the absolute log-probability of either response. In particular, the policy can increase the margin by decreasing the probability of the rejected response a lot, while barely increasing, or even decreasing, the probability of the preferred response. In practice, DPO-trained models often show a decline in the log-probability of both responses over training, with the rejected one falling faster. That is mathematically consistent with the loss, but it is not the same as making the model more likely to produce good responses. It is one reason later methods add an extra term that anchors the preferred response, or regularise the policy more explicitly.

A minimal PyTorch sketch

The implementation is strikingly short once you have log-probabilities. Suppose policy_logps and ref_logps are the summed log-probabilities of each response under the policy being trained and under the frozen reference policy, for a batch of preferred and rejected pairs:

import torch
import torch.nn.functional as F

def dpo_loss(policy_chosen, policy_rejected,
             ref_chosen, ref_rejected, beta=0.1):
    # Implicit rewards: how far each response has moved from the reference.
    chosen_reward = beta * (policy_chosen - ref_chosen)
    rejected_reward = beta * (policy_rejected - ref_rejected)
    # Logistic loss on the reward margin.
    logits = chosen_reward - rejected_reward
    return -F.logsigmoid(logits).mean()

Two practical details matter more than the code suggests. First, log-probabilities must be summed over the response tokens only, not over the prompt, otherwise longer responses dominate the loss for the wrong reason. Second, the reference log-probabilities are usually precomputed once and frozen, since the reference policy never changes. Everything else is standard supervised-style training: batches of pairs, Adam, gradient clipping, and a held-out set of preferences to check that accuracy on unseen pairs is actually improving.

Choosing beta, and what it really controls

The parameter $\beta$ is often described as a KL penalty strength, and that is correct, but its effect in DPO is easier to see through the implicit reward. A small $\beta$ means a given change in log-probability ratio counts as a large reward change, so the policy can move a long way from the reference to fit the preferences. A large $\beta$ damps that movement and keeps the policy close to the reference.

Typical values in published work lie around 0.1 to 0.5, but the right value depends on how much you trust your preference data relative to your reference model. If the data is clean and the reference model is weak, a smaller $\beta$ lets the policy learn more. If the data is noisy or narrow, a larger $\beta$ is safer, because DPO has no other mechanism to stop it from overfitting to annotator quirks. Tuning $\beta$ on a held-out preference set, and checking the KL divergence from the reference on a separate prompt set, is more informative than tuning it on training loss alone.

Limitations and common pitfalls

DPO removes the reward model, but it does not remove the problems that made RLHF hard. Four deserve attention.

First, preference data is not ground truth. Annotators disagree, instructions are ambiguous, and pairwise choices hide trade-offs: a response can be more helpful but less accurate, or safer but less useful. DPO will fit whatever pattern the data contains, including length bias. If annotators systematically prefer longer answers, DPO learns that longer is better, unless the data or the loss corrects for it.

Second, DPO is off-policy in a specific sense. It trains on fixed pairs collected beforehand, not on samples from the current policy. As the policy moves, the distribution of responses it would actually produce shifts away from the distribution in the training pairs. The implicit reward is only well constrained near the data. Far from it, the policy can drift in ways that look fine under the loss but produce odd outputs in deployment. Iterative DPO, where new pairs are collected from the current policy and the process repeats, is one response to this, at the cost of reintroducing a data-collection loop.

Third, the loss can be satisfied by suppressing the rejected response rather than improving the preferred one, as noted above. Watching the absolute log-probabilities of chosen and rejected responses during training, not just the accuracy or the loss, catches this early. If both are falling steeply, the model is learning to avoid the rejected style more than it is learning to produce the preferred style.

Fourth, DPO inherits the Bradley-Terry assumption that preferences come from a single scalar reward with logistic noise. Real preferences are often intransitive, context-dependent, or multi-dimensional. When that assumption fails badly, no amount of tuning $\beta$ will fix it, and the right move is to improve the data or the preference model, not to train longer.

When to use DPO, and when not to

DPO is a good default when you have a solid supervised fine-tuned model, a clean set of preference pairs, and limited appetite for running a reinforcement learning pipeline. It is stable, it reuses ordinary training infrastructure, and it often matches RLHF performance on standard benchmarks with far less engineering.

It is a weaker choice when preferences need to be combined with other signals, such as verifiable correctness on maths or code, where a reward function can encode checks that pairwise human labels cannot. It is also weaker when you need online exploration, where the policy must try new behaviours and learn from their outcomes rather than from fixed comparisons. In those settings, an explicit reward model, or a method that goes beyond pairwise preferences, may be worth the extra complexity.

The lasting lesson of DPO is not that reward models are useless. It is that the RLHF objective already defines a tight link between policy and reward, and that link can be used directly. Once you see the policy as an implicit reward model, many later alignment methods become easier to read: they are, in different ways, attempts to keep that implicit reward honest when the preference data runs out.

Takeaway

Direct Preference Optimization replaces two stages of RLHF, reward modelling and reinforcement learning, with one supervised-style loss on preference pairs. The trick is the identity that writes the reward as a log ratio between the optimal policy and the reference policy, so the normalising constant cancels and only policy log-probabilities remain. The result is simpler and more stable, but it is still only as good as its preference data, and it needs monitoring for drift, length bias, and the gap between raising a margin and actually improving the preferred response.

---

Connect with me on LinkedIn: linkedin.com/in/umeshtharukaofficial