DPO Likelihood Displacement: Why Chosen Logprobs Fall Too

Your DPO run looks perfect. rewards/accuracies climbs to 0.95, rewards/margins widens every step, loss falls smoothly. Then you sample from the checkpoint and it is worse than the SFT model it started from — terser, blander, occasionally answering a question nobody asked. You pull the raw numbers and find it: logps/chosen has been falling the entire time. Not just the rejected responses. The preferred ones too.

This is DPO likelihood displacement, and it is not a bug in your training loop. It is the objective working exactly as written.

TL;DR

• DPO only constrains the difference between chosen and rejected log-probs. Driving log π(y_w|x) down while driving log π(y_l|x) down harder is a perfectly valid way to minimize the loss.
• The displaced probability mass does not flow to the chosen response. It flows to a third, unmodeled response — which on safety data can mean a refusal is replaced by compliance.
• Severity is predicted by how similar your chosen and rejected responses are. Near-duplicate pairs (edit-style preference data) are the worst offenders.
• rewards/accuracies and rewards/margins cannot detect this. You must log the absolute logps/chosen and its delta from the reference model.
• Fixes that work: an NLL anchor term on the chosen response (rpo_alpha in TRL), a DPOP-style hinge against the reference, filtering near-duplicate pairs, higher β, and one epoch at a low LR.

What is DPO likelihood displacement?

DPO likelihood displacement is the phenomenon where preference training reduces the log-probability of the chosen response, not just the rejected one — so probability mass leaves both branches of the pair and lands somewhere you never supervised.

Look at what the loss actually asks for:

import torch.nn.functional as F

# seq_logp() = sum of token log-probs over the completion only (prompt masked out)
pi_logratios  = pol_chosen - pol_rejected
ref_logratios = ref_chosen - ref_rejected
logits = pi_logratios - ref_logratios
loss = -F.logsigmoid(beta * logits).mean()

logits is a scalar difference. The optimizer is free to satisfy it by raising pol_chosen, by lowering pol_rejected, or — most cheaply — by lowering both with the rejected one falling faster. Nothing in the objective pins the absolute likelihood of y_w. The reference terms ref_chosen and ref_rejected are constants under the gradient; they shift the origin, they do not anchor the policy.

The gradient makes the coupling explicit:

∇loss ∝ -β · σ(-β · logits) · [ ∇log π(y_w|x) - ∇log π(y_l|x) ]

One scalar weight, σ(-β · logits), multiplies the difference of two gradients. When y_w and y_l share most of their tokens, those two gradient vectors are nearly parallel and cancel almost everywhere. What survives is a small residual concentrated on the few positions where the sequences diverge — plus whatever second-order damage the negative term does through shared parameters.

Why does pushing down the rejected response drag the chosen one with it?

Because probability lives on a simplex and both responses read out of the same unembedding matrix. There is no "delete mass from y_l" operation. There is only "reallocate mass across the vocabulary at each position," and the model does that using directions in hidden space that y_w and y_l largely share.

Two effects stack:

Gradient entanglement. For a pair that differs by one clause out of twenty, ~95% of the token positions are identical. The positive and negative gradients at those positions are the same vector with opposite signs. They cancel in the first-order term but not in their downstream effect on shared hidden states. Push down a token whose unembedding vector is close to a chosen token's, and you push down the chosen token too. Recent work formalizes this with a centered hidden embedding similarity (CHES) score computed on the two completions — high similarity between chosen and rejected embeddings predicts severe displacement.

The squeezing effect. Applying a negative gradient to a sequence does not spread its mass uniformly over the rest of the vocabulary. It concentrates the freed mass onto whatever token was already the argmax. So unlearning a rejected response sharpens the distribution around the model's existing mode — which is frequently neither y_w nor y_l.

That is why the mass goes to a third response. This is the genuinely dangerous part. If your pair is (chosen: "I can't help with that, but here's a safe alternative", rejected: "I can't help with that."), the two share a long prefix. Displacement can move mass off both refusals and onto a compliant continuation the training data never contained. Researchers have demonstrated exactly this failure — safety-oriented preference sets that measurably reduce refusal rates after DPO.

How do I detect DPO likelihood displacement in a training run?

Log the absolute chosen log-prob, not just the reward margin. The standard TRL metrics are all differences, and differences are precisely what displacement hides in.

def dpo_step(policy, ref, batch, beta=0.1):
    pol_c, pol_r = seq_logp(policy, batch.chosen), seq_logp(policy, batch.rejected)
    with torch.no_grad():
        ref_c, ref_r = seq_logp(ref, batch.chosen), seq_logp(ref, batch.rejected)

    logits = (pol_c - pol_r) - (ref_c - ref_r)
    loss = -F.logsigmoid(beta * logits).mean()

    metrics = {
        # what TRL gives you by default — all differences
        "rewards/chosen":    (beta * (pol_c - ref_c)).mean(),
        "rewards/rejected":  (beta * (pol_r - ref_r)).mean(),
        "rewards/margins":   (beta * logits).mean(),
        "rewards/accuracies": (logits > 0).float().mean(),

        # the two that actually catch displacement
        "logps/chosen_abs":   pol_c.mean(),                    # must not free-fall
        "logps/chosen_delta": (pol_c - ref_c).mean(),          # must not go deeply negative
        "logps/chosen_per_tok": (pol_c / batch.chosen_len).mean(),
    }
    return loss, metrics

Read them together:

rewards/margins	logps/chosen_delta	Diagnosis
up	~0 or up	Healthy. Chosen is being reinforced.
up	steadily down	Displacement. You are only unlearning.
up	sharply down + KL spike	Displacement plus reward hacking. Stop the run.
flat	flat	β too high or LR too low. Nothing is learning.

A useful hard check: after training, compute log π(y_w|x) - log π_ref(y_w|x) on a held-out slice of your own preference set. If the median is negative, your "aligned" model assigns less probability to the responses you told it to prefer.

Which preference pairs cause the worst displacement?

Pairs where the rejected response is an edited version of the chosen one. This is the default output shape of an LLM-judge pipeline: you ask Claude Opus 4.x or GPT-5.x to critique and revise a response, and you keep (revision, original) as a preference pair. The two differ by a sentence. Embedding similarity is near 1. That is the maximum-displacement regime.

Filter them. A cheap proxy in the spirit of CHES, using last-layer hidden states over the completion tokens:

@torch.no_grad()
def pair_similarity(model, tok, prompt, chosen, rejected):
    h_c = completion_hidden(model, tok, prompt, chosen)    # [T_c, d], last layer
    h_r = completion_hidden(model, tok, prompt, rejected)  # [T_r, d]
    v_c, v_r = h_c.sum(0), h_r.sum(0)                      # length-sensitive on purpose
    return F.cosine_similarity(v_c, v_r, dim=0).item()

# drop the top-similarity tail before training
scored = [(pair_similarity(ref_model, tok, *p), p) for p in pairs]
scored.sort(key=lambda x: x[0])
keep = [p for s, p in scored[: int(0.9 * len(scored))]]

Dropping the most-similar decile costs you almost no signal — near-identical pairs carry little preference information anyway — and removes most of the entanglement. Run this with the reference model, before training, once; it is one forward pass per completion.

The other high-risk pattern: pairs where both responses are wrong and one is marginally less wrong. There is no mass to move toward, so all the gradient does is squeeze.

How do I fix DPO likelihood displacement?

Anchor the chosen response's absolute likelihood. Two loss terms do this, and they compose with everything else:

# 1) RPO-style NLL anchor — a length-normalized SFT term on the chosen response.
#    TRL exposes this as DPOConfig(rpo_alpha=1.0).
loss = -F.logsigmoid(beta * logits).mean() \
       + rpo_alpha * (-(pol_c / batch.chosen_len)).mean()

# 2) DPOP-style hinge — penalize only when the policy falls below the reference
#    on the chosen response. Zero cost when nothing is wrong.
loss = -F.logsigmoid(beta * logits).mean() \
       + lambda_dpop * torch.clamp(ref_c - pol_c, min=0).mean()

The NLL anchor is the safer default and is what large-scale post-training recipes converged on. The DPOP hinge is more surgical: it is inactive as long as pol_c ≥ ref_c, so it does not fight the preference signal on healthy batches.

Beyond the loss:

• Raise β. β is the inverse KL penalty strength. At β=0.1 the policy can wander far from the reference; 0.3–0.5 keeps it close, at the cost of a smaller achievable margin. If displacement is your failure mode, trade margin for stability.
• Drop the LR and train one epoch. Full-parameter DPO wants something in the 5e-7 to 1e-6 range. DPO overfits preference data fast; a second epoch mostly deepens displacement.
• Mask shared prefixes. Token-level DPO variants apply the negative gradient only where the sequences diverge. If you are writing the loss yourself, computing the divergence point and zeroing the rejected-side gradient before it is nearly free.
• Do not rely on rewards/accuracies for early stopping. It will keep improving while the model degrades. Checkpoint on an external eval or on logps/chosen_delta.

When is a falling chosen log-prob actually fine?

Some decline is expected and harmless. Sequence log-probs are sums over tokens, so a model that gets slightly more confident on 5% of positions and slightly less on 95% can show a modest per-token drop while sampling better. What matters is the rate and shape: a shallow decline that plateaus is normal regularization; a monotonic free-fall that tracks the margin curve step-for-step is displacement. Watch logps/chosen_per_tok rather than the raw sum, since the raw sum also moves with completion length.

The short answer

Chosen log-probs fall during DPO because the objective is defined purely on the difference between chosen and rejected likelihoods — lowering both, with the rejected falling faster, minimizes the loss just as well as raising the chosen one. Because y_w and y_l are usually near-duplicates that share hidden-state directions and an unembedding matrix, the negative gradient on the rejected response drags the chosen one down with it, and the freed probability mass concentrates on the model's pre-existing mode — a third response you never supervised. Detect it by logging absolute logps/chosen alongside the reward margins; fix it with an NLL anchor on the chosen response (rpo_alpha) or a DPOP hinge, a higher β, a lower learning rate, and by filtering out the near-identical preference pairs your LLM-judge pipeline generates by default.