DPO Likelihood Displacement: Why Chosen Responses Get Rarer

You run DPO on a clean preference set. The margin goes up, the win rate on your pairwise eval goes up, and then you scroll back through the training logs: logps/chosen started at -142 and ended at -231. The response you were rewarding is now less likely than it was before training. Not relatively — absolutely.

This is DPO likelihood displacement, and it is not a bug in your data loader. It is a direct consequence of what the DPO objective optimizes, and it is the mechanism behind the most annoying DPO failure mode in practice: you train on "helpful and safe" versus "harmful," and you ship a model that refuses everything.

TL;DR

DPO only constrains the *margin* between chosen and rejected log-probs. Both can fall together and the loss still decreases — nothing in the objective anchors log π(y_w) upward.
The exact condition: log π(y_w) decreases when ⟨∇log π(y_w), ∇log π(y_l)⟩ > ‖∇log π(y_w)‖² — i.e. when the rejected response's gradient is aligned with the chosen one's and larger. Similar-looking pairs are the danger zone.
Displaced probability mass doesn't vanish. It flows to whatever is geometrically adjacent — usually a short refusal, a hedge, or an empty answer, none of which appear in your dataset.
Detect it by logging logps/chosen in absolute terms (not just the margin) plus a fixed "escape probe" string, and watch whether the probe's log-prob rises.
Fix it with an NLL anchor on chosen (rpo_alpha in TRL), by filtering high-similarity pairs (CHES), and by making sure your reference model actually assigns decent probability to the chosen responses.

What is DPO likelihood displacement?

Likelihood displacement is when DPO training reduces the absolute log-probability of the preferred response y_w while still increasing the preference margin, pushing the freed probability mass onto completions that were never in the training data.

The margin looks great. rewards/margins climbs monotonically. rewards/chosen — which in TRL is β(log π_θ(y_w) − log π_ref(y_w)) — goes negative and stays there. That negative number is the whole story: the policy assigns less mass to the chosen response than the reference did.

Why does DPO push down the probability of the chosen response?

Because the DPO loss is a function of a difference, and a difference is invariant to shifting both terms down.

L_DPO = −log σ( β·[log π_θ(y_w|x) − log π_ref(y_w|x)]
              − β·[log π_θ(y_l|x) − log π_ref(y_l|x)] )

∇_θ L_DPO = −β · σ(−β·Δ) · [ ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ]

The update direction is g_w − g_l with equal weight on both terms. Lowering log π(y_l) by 3 nats and lowering log π(y_w) by 1 nat is a perfectly good descent step. The reference model appears only inside Δ as a constant offset; it shifts when the sigmoid saturates, not which direction the gradient points.

Now do the first-order analysis. Under gradient flow, the rate of change of the chosen log-prob is the inner product of its own gradient with the update direction:

d/dt log π_θ(y_w) ∝ ⟨ g_w , g_w − g_l ⟩ = ‖g_w‖² − ⟨g_w, g_l⟩

So log π(y_w) decreases exactly when ⟨g_w, g_l⟩ > ‖g_w‖². Two things drive that: the gradients must point in similar directions, and ‖g_l‖ must be large relative to ‖g_w‖. Both are common. Preference pairs are usually near-duplicates — same prompt, same topic, often the same opening 40 tokens — so g_w and g_l are highly aligned. And if the rejected response is already low-probability under the policy, its gradient norm is large. You get a step that suppresses y_l hard and drags y_w down with it.

Look at the last layer to see why alignment is so high. For unembedding matrix W, the gradient of a token's log-prob is an outer product (e_token − p) hᵀ, where h is the final hidden state. The inner product of two such gradients factorizes into a token-geometry term times a hidden-state similarity term. When chosen and rejected differ only by near-synonyms — "Never" vs "No", "I'd suggest" vs "You should" — the token vectors are close and the hidden states are nearly identical. The suppression signal on the rejected token bleeds straight onto the chosen token through the shared softmax normalizer.

Where does the displaced probability mass go?

Softmax conserves mass. If you push down the rejected continuation and the chosen one comes down with it, that mass lands on whatever the model considers the next-nearest neighbor in output space — and the objective has no term describing that region at all.

In practice the recipient is almost always a short, generic, high-prior string: a refusal, "I don't have enough information," a truncated one-liner. These sit at high probability under the base model and are geometrically far from both members of a typical preference pair, so they absorb displaced mass without ever contributing to the loss.

Why does DPO on safety data cause over-refusal?

Because safety preference sets are the worst case for the condition above. The chosen response is typically a nuanced, careful, non-refusing answer; the rejected one is a harmful answer to the same prompt. They share the prompt, the topic, and often the framing — maximum gradient alignment. The rejected response is far off-policy after SFT, so it has a large gradient norm.

Result: DPO suppresses the harmful answer, drags the careful answer down with it, and the mass lands on a blanket refusal that was never labeled as preferred. Razin et al.'s work on unintentional unalignment documents this in both directions — the same mechanism can also move mass toward unsafe outputs when the pair geometry flips. The lesson generalizes past safety: any dataset where chosen and rejected are minimally-edited variants of each other is a displacement machine.

How do I detect likelihood displacement in a training run?

Log absolute log-probs, not just the margin, and add a probe string that isn't in your data. Three columns tell you almost everything:

`logps/chosen`	`logps/rejected`	Reading
↑	↓	Healthy. This is what you want.
↓ slightly	↓↓ steeply	Displacement. Margin is a lie.
↓↓	↓↓↓	Severe. Check probe mass immediately.
↑	↑	Reference/policy mismatch — verify `π_ref` is your actual SFT checkpoint.

The probe is the part people skip. Pick a string you never want to see more of — a canned refusal — and track its log-prob under the policy on a fixed set of benign prompts:

from transformers import TrainerCallback
import torch

PROBE = "I'm sorry, but I can't help with that."

class EscapeMassProbe(TrainerCallback):
    def __init__(self, tok, prompts, every=50):
        self.tok, self.prompts, self.every = tok, prompts, every

    @torch.no_grad()
    def on_step_end(self, args, state, control, model=None, **kw):
        if state.global_step % self.every:
            return
        total = 0.0
        for p in self.prompts:
            ids = self.tok(p + PROBE, return_tensors="pt").to(model.device)
            n_probe = len(self.tok(PROBE).input_ids)
            logits = model(**ids).logits[:, :-1].log_softmax(-1)
            tgt = ids.input_ids[:, 1:]
            lp = logits.gather(-1, tgt.unsqueeze(-1)).squeeze(-1)
            total += lp[:, -n_probe:].sum().item()   # log p(PROBE | prompt)
        state.log_history.append(
            {"step": state.global_step, "probe/refusal_logp": total / len(self.prompts)}
        )

If probe/refusal_logp climbs while logps/chosen falls, you are watching mass move in real time. That number correlates with over-refusal on your held-out benign set far earlier than any win-rate eval will show it.

How do I fix DPO likelihood displacement?

Four interventions, roughly in order of effort-to-payoff.

1. Anchor the chosen response with an NLL term. This is the single highest-value change. Adding λ · −log π_θ(y_w|x) to the loss gives the optimizer an explicit reason to keep log π(y_w) high instead of only maximizing the margin. TRL exposes it as rpo_alpha; Llama 3's post-training used the same idea, and CPO/RPO variants formalize it.

from trl import DPOConfig, DPOTrainer

cfg = DPOConfig(
    beta=0.1,
    rpo_alpha=1.0,            # NLL anchor on chosen — the fix that matters most
    loss_type="sigmoid",
    learning_rate=5e-7,       # DPO wants an order of magnitude less than SFT
    max_length=2048,
    max_prompt_length=1024,
    logging_steps=10,
)

trainer = DPOTrainer(
    model=policy, ref_model=sft_checkpoint,   # must be the SFT model, not the base
    args=cfg, train_dataset=pairs,
    processing_class=tokenizer,
    callbacks=[EscapeMassProbe(tokenizer, benign_prompts)],
)

2. Filter high-similarity pairs before training. The gradient condition says similar pairs are the problem, so measure similarity in the space that actually drives the gradient — final hidden states, not text. CHES (centered hidden embedding similarity) is a cheap proxy for ⟨g_w, g_l⟩:

@torch.no_grad()
def ches(model, tok, prompt, y_w, y_l):
    def resp_embed(y):
        ids = tok(prompt + y, return_tensors="pt").to(model.device)
        n = len(tok(y).input_ids)
        h = model(**ids, output_hidden_states=True).hidden_states[-1][0]
        return h[-n:].sum(0)                     # sum over response tokens
    hw, hl = resp_embed(y_w), resp_embed(y_l)
    mu = (hw + hl) / 2                           # center within the pair
    hw, hl = hw - mu, hl - mu
    return torch.cosine_similarity(hw, hl, dim=0).item()

pairs = [p for p in pairs if ches(ref, tok, p["prompt"], p["chosen"], p["rejected"]) < 0.75]

Dropping the top-similarity slice — the pairs whose chosen and rejected are minimal edits of each other — removes most of the displacement pressure while keeping the pairs that carry real preference signal. Text-level edit distance is a weaker but zero-cost first pass.

3. Check your reference model. If chosen responses were distilled from a stronger model, log π_ref(y_w) is already low, ‖g_w‖ is large in a direction the policy can't cheaply follow, and the optimizer takes the easy route of crushing y_l. Run one SFT epoch on the chosen responses first and use that as π_ref. This is why "SFT on chosen, then DPO" is the standard recipe and not a ritual.

4. Raise β, carefully. Larger β saturates the sigmoid sooner, so the effective step shrinks once the margin is achieved, limiting how far both log-probs can drift. It also slows learning. Treat it as a damper, not a cure — β from 0.1 to 0.3 is a reasonable sweep before you conclude the data is the problem.

Does this happen with PPO or GRPO too?

Not the same way. Policy-gradient methods increase log π(y) directly for sampled completions with positive advantage — there's no paired difference term that can be satisfied by lowering both sides. Sampling is on-policy, so the mass that gets redistributed lands near the current policy rather than on some distant refusal mode, and the explicit KL penalty against the reference bounds total drift. You trade the displacement failure mode for reward hacking and a much heavier training loop, which is a real trade, not a free win.

The short answer

DPO likelihood displacement happens because the DPO loss constrains only the difference between chosen and rejected log-probabilities, never their absolute level — so gradient descent is free to satisfy the objective by pushing both down, which it does whenever the rejected response's gradient is aligned with and larger than the chosen response's (⟨g_w, g_l⟩ > ‖g_w‖²), the common case for near-duplicate preference pairs. The vacated probability mass moves to whatever is geometrically adjacent and high-prior, typically a generic refusal that never appears in your training set — which is why safety-focused DPO runs so often produce over-refusing models with excellent margin curves. Log logps/chosen in absolute terms alongside a refusal probe, add an NLL anchor on the chosen response (rpo_alpha=1.0), filter out high-CHES pairs, and make sure your reference model is the SFT checkpoint that already fits the chosen responses.