DPO Likelihood Displacement: When Preferred Answers Get Rarer

You run Direct Preference Optimization on a clean set of preference pairs. The reward margin climbs, the loss falls, the eval curve looks textbook. Then you sample from the tuned model and the chosen responses — the exact completions you rewarded — come out less often than before training. Not the rejected ones. The preferred ones. Their log-probability went down.

This is DPO likelihood displacement, and it is not a bug in your data loader. It is baked into the DPO objective. If you fine-tune alignment or preference models with DPO and you have never plotted log π(y_chosen) across training steps, you are probably shipping it.

TL;DR

• DPO optimizes a margin, not an absolute. The loss only cares about logπ(chosen) − logπ(rejected). The model can grow that gap by pushing the rejected response down faster than it drags the chosen response down with it.
• Chosen and rejected share tokens, so the gradient that suppresses the rejected response leaks onto the chosen one. When the two completions are similar, log π(y_chosen) can fall for hundreds of steps while the margin still improves.
• The failure mode is invisible in the loss curve. Reward margin and accuracy look healthy; probability mass silently drains toward off-distribution tokens (often the empty/EOS or a third unrelated response).
• Fixes that work: add an explicit SFT/NLL term on the chosen response (rpo_alpha in TRL), use a DPO-positive style anchor that penalizes chosen log-prob dropping below the reference, raise beta, and curate pairs so chosen and rejected are not near-duplicates.

What is DPO likelihood displacement?

Likelihood displacement is when DPO training decreases the probability the model assigns to the preferred (chosen) response, even though DPO's whole point is to make preferred responses more likely. The probability mass does not move to the chosen response — it moves away from both the chosen and the rejected, toward whatever else the model can put in that region of output space.

Start from the objective. DPO's per-pair loss is:

L = -log σ( β · [ (log π_θ(y_w|x) − log π_ref(y_w|x))
                − (log π_θ(y_l|x) − log π_ref(y_l|x)) ] )

y_w is the chosen (winner), y_l the rejected (loser), π_ref the frozen reference (your SFT checkpoint), β the temperature that controls how far you drift from the reference.

Look at what is inside the sigmoid: a difference of log-ratios. The loss is minimized when the margin

Δ = (log π_θ(y_w) − log π_ref(y_w)) − (log π_θ(y_l) − log π_ref(y_l))

is large and positive. Nothing in that expression pins log π_θ(y_w) to a floor. A model that halves the chosen probability and quarters the rejected probability has increased the margin. The loss is happy. Your users are not.

Why does the chosen response's probability go down?

Because the gradient that suppresses the rejected response also suppresses everything the rejected response has in common with the chosen one — and preference pairs usually overlap heavily.

Write the DPO gradient. With d = β·Δ the margin inside the sigmoid, the gradient of the loss is proportional to:

∇L ∝ − β · σ(−d) · [ ∇log π_θ(y_w) − ∇log π_θ(y_l) ]

The scalar σ(−d) is just an adaptive weight — large when the model is still wrong, shrinking as the margin grows. The direction is the bracket: push up the chosen, push down the rejected. In isolation that is exactly what you want.

The problem is that ∇log π_θ(y_w) and ∇log π_θ(y_l) are not orthogonal. If y_w and y_l are "Paris is the capital of France." versus "Paris is the capital of Germany.", they share almost every token and route through almost the same hidden states. The −∇log π_θ(y_l) term pushes down the shared prefix — "Paris is the capital of" — and that prefix is also the chosen response's prefix. The suppression bleeds across.

When the two responses are close in the model's representation space, the negative gradient on the loser dominates the positive gradient on the winner over the shared tokens, and the net effect on log π_θ(y_w) is negative. Recent analysis formalizes this: the displacement is governed by how aligned the chosen and rejected hidden representations are. High similarity → strong displacement. This is why "harder" preference pairs — the ones where chosen and rejected differ by a word — are the most dangerous, not the most informative.

Where does the mass go? Not to the chosen. It leaks to whatever token sits geometrically between suppressing y_l and not fully committing to y_w — frequently EOS (the model learns to say less), or a third response entirely. In alignment settings this is how "unintentional unalignment" happens: you train to prefer a safe refusal over an unsafe answer, and DPO quietly raises the probability of a different unsafe answer you never put in the data.

How do I detect likelihood displacement?

Stop trusting the loss and reward-margin curves — they rise even while displacement is happening. Log the absolute quantities:

• log π_θ(y_w) (mean chosen logprob) per step. This is the canary. If it trends down while reward margin trends up, you have displacement.
• log π_θ(y_w) − log π_ref(y_w) — the chosen response's drift from the reference. It should stay near zero or positive; a steady decline is the failure.
• The gap between reward margin improvement and chosen-logprob change. Healthy training moves both up.

Concretely, in TRL the trainer already emits logps/chosen and logps/rejected. Watch them directly:

# In your logging callback / wandb panel, plot BOTH — not just the margin.
# margin = logps/chosen - logps/rejected   (this looks fine even when broken)
# The real signal:
#   logps/chosen  should NOT be steadily decreasing
#   rewards/chosen = beta * (logps/chosen - ref_logps/chosen) should stay >= ~0

If logps/chosen slides down for more than a warmup's worth of steps, you are displacing.

How do I stop DPO from lowering the preferred probability?

Anchor the absolute chosen probability instead of only the margin. Four levers, roughly in order of impact:

1. Add an SFT/NLL term on the chosen response. This is the most reliable fix and it is one argument in TRL. Setting rpo_alpha adds a weighted negative-log-likelihood loss on the chosen completion, so the objective becomes "win the margin and keep assigning high probability to the winner":

from trl import DPOConfig, DPOTrainer

config = DPOConfig(
    beta=0.1,            # raise toward 0.3–0.5 if displacement persists
    rpo_alpha=1.0,       # NLL anchor on the chosen response; 0 = pure DPO
    learning_rate=5e-7,  # DPO wants a much smaller LR than SFT
    loss_type="sigmoid", # vanilla DPO; see dpop below
    max_length=1024,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
)

trainer = DPOTrainer(
    model=policy_model,
    ref_model=reference_model,   # your frozen SFT checkpoint
    args=config,
    train_dataset=pref_dataset,  # columns: prompt, chosen, rejected
    processing_class=tokenizer,
)
trainer.train()

The NLL term costs almost nothing and removes the degenerate solution where both logprobs collapse. The tradeoff: too large an rpo_alpha and you are basically doing SFT-on-chosen with a preference nudge, which weakens the contrast you were trying to learn. Start at 1.0, tune down if the margin stops separating.

2. Use a DPO-positive (DPOP) style penalty. DPOP adds a hinge term that fires when log π_θ(y_w) drops below log π_ref(y_w), directly punishing the model for making the chosen response rarer than the reference already made it. It targets displacement more surgically than a blanket NLL term and is a good choice when your chosen/rejected pairs are genuinely near-duplicates (math, code, single-token edits).

3. Raise beta. A larger beta keeps the policy closer to the reference, which limits how far any logprob — including the chosen's — can wander. It is a blunt instrument: too high and the model barely learns the preference at all. But if you are seeing wild displacement at beta=0.1, 0.3 is worth a run.

4. Curate pairs by dissimilarity. Displacement scales with how similar chosen and rejected are in representation space. Filter or down-weight pairs where the two responses differ by only a token or two, or where their embeddings are near-identical. Counterintuitively, the razor-thin pairs that feel most informative are the ones most likely to drain probability from the answer you want. If you must keep hard pairs, pair them with the NLL anchor above.

Does this mean DPO is broken?

No. DPO is still a clean, RL-free way to fold preferences into a policy, and for well-separated pairs with a modest beta it behaves. The trap is treating the loss and reward margin as sufficient monitoring. They are necessary and misleading: both improve while the model gets worse at producing the thing you rewarded. IPO, DPOP, and the rpo_alpha NLL anchor all exist because the plain sigmoid objective under-constrains the absolute probabilities. Pick one before you train, not after you notice your aligned model refusing everything.

Bottom line

Why does DPO push down the probability of your preferred answer? Because the DPO loss optimizes only the margin between chosen and rejected log-ratios, and it can widen that margin by suppressing the rejected response faster than the correlated chosen response follows it down — so probability mass drains off both and lands on off-distribution tokens like EOS or an unrelated completion. This DPO likelihood displacement is worst when chosen and rejected responses are similar, it is invisible in the loss and reward-margin curves, and you catch it only by plotting the absolute chosen log-probability. Fix it by anchoring that absolute probability: add an NLL term on the chosen response (rpo_alpha in TRL), apply a DPO-positive penalty, raise beta, and drop near-duplicate preference pairs.