GRPO Zero-Variance Groups: Why Half Your Rollouts Do Nothing

Your GRPO run looks healthy. Mean reward is climbing, KL to the reference is small, loss is drifting down. Then you plot gradient norm and it has been falling for 400 steps. You check the rollout buffer: on 58% of prompts, all 8 sampled completions got the same reward. Those prompts contributed exactly zero policy gradient — you paid full generation cost for them and got nothing back.

That is the GRPO zero-variance group problem, and it is the single most common reason an RL post-training run plateaus while every dashboard says it is fine.

TL;DR

• GRPO computes advantages within a group of G sampled completions: A_i = (r_i - mean(r)) / std(r). If every completion in the group gets the same reward, the numerator is zero for all of them, so the group produces no policy gradient at all.
• With binary correctness rewards, the fraction of zero-variance groups is roughly p^G + (1-p)^G per prompt. At 90% pass rate with G=8, ~43% of your rollout compute is dead weight — and it gets worse as the model improves.
• Dividing by the group std amplifies rare successes: one correct out of 16 gets an advantage near 3.75, while a balanced 8/16 group gets ~0.97. Lucky guesses and reward hacks on hard prompts get ~4x the gradient weight of clean signal.
• When beta > 0, zero-variance groups are worse than useless: the policy-gradient term is zero but the KL term is not, so those samples pull the policy back toward the reference — including on prompts it has fully solved.
• Fixes, in order of impact: dynamic sampling (DAPO — resample until the batch is full of nonzero-variance groups), drop the std division (Dr. GRPO), set beta = 0, and track the effective group rate as a first-class metric.

What exactly is a GRPO zero-variance group?

A GRPO zero-variance group is a set of G completions sampled for one prompt where every completion receives an identical reward, making all within-group advantages zero. GRPO has no value network — the group mean is the baseline. When the group is degenerate, the baseline equals every sample, and the advantage vanishes.

The objective, stripped to its essentials:

A_i = (r_i - mean(r_1..r_G)) / (std(r_1..r_G) + eps)

J = E_q,{o_i} [ (1/G) Σ_i (1/|o_i|) Σ_t min(ρ_i,t · A_i,
                                            clip(ρ_i,t, 1-ε, 1+ε) · A_i) ]
    - β · D_KL(π_θ || π_ref)

where ρ_i,t is the token-level importance ratio. Every token's gradient is scaled by the scalar A_i. Set A_i = 0 and the whole trajectory drops out of the update regardless of how many tokens it has.

This is not a bug in an implementation. It is structural: a leave-nothing-out group baseline cannot produce signal from a group with no spread. With a learned critic (PPO), an all-correct group still teaches the value head "this state is high value," which sharpens future advantages. GRPO throws that information away.

Why does the dead-rollout fraction get worse as the model improves?

Because zero-variance requires unanimity, and unanimity gets more likely as the policy gets confident. Model per-prompt pass rate as p, and pretend the G samples are iid Bernoulli. The chance a group is degenerate is:

P(zero-variance) = p^G + (1-p)^G

pass rate p	G=4	G=8	G=16
0.5	0.13	0.01	~0
0.8	0.41	0.17	0.03
0.9	0.66	0.43	0.19
0.95	0.81	0.66	0.44

Two things to read off this. First, early in training on a hard set, p ≈ 0 and nearly everything is a zero-variance all-wrong group — that is the cold-start stall people paper over with SFT warmup. Second, and less obvious, the end of training has the same failure. As your model solves more of the training set, the effective batch shrinks toward zero and the run flatlines. The reward curve keeps looking great because it is measuring the thing that is killing you.

The iid assumption is optimistic, too. Rollouts for one prompt share a prefix distribution and a difficulty level, so they are positively correlated — real degenerate rates run above this table, not below.

Raising G helps, but sublinearly and at linear cost: going 8 → 16 at p=0.9 takes you from 43% waste to 19% waste for 2x the generation bill.

Why does dividing by the group std distort hard prompts?

Because std is small exactly when successes are rare, and dividing by a small number blows up the advantage of the one sample that succeeded. With binary rewards and G=16, using the unbiased std (torch.std default, ddof=1):

• 1 correct / 15 wrong: mean = 0.0625, std ≈ 0.25 → A_correct ≈ 3.75, A_wrong ≈ -0.25
• 8 correct / 8 wrong: mean = 0.5, std ≈ 0.516 → A_correct ≈ 0.97, A_wrong ≈ -0.97

The lone success on a hard prompt gets nearly 4x the gradient weight of a success on a balanced prompt. That is backwards from what you want. A single success out of 16 is the trajectory most likely to be a lucky guess, a format exploit, or a verifier false positive — and GRPO's normalization hands it the largest update in the batch. This is the "difficulty bias" that Dr. GRPO (Understanding R1-Zero-Like Training) removes by dropping the std division entirely and keeping only the mean-centering.

If you have ever watched a run suddenly adopt a weird answer format that happens to satisfy your regex extractor, look at the std-normalized advantages on your near-zero-pass-rate prompts.

Why are zero-variance groups worse than useless when beta > 0?

Because the KL penalty does not care that the advantage is zero. In the standard formulation the KL term is a separate additive penalty (usually the k3 estimator, exp(logp_ref - logp) - (logp_ref - logp) - 1), applied per token on every sampled trajectory. So a group where the policy got everything right still contributes a gradient — one that pulls the policy back toward the reference model on a prompt it has mastered.

You are paying compute to unlearn. This is a real, if slow, drag: the only signal from your easiest, most-solved prompts is regularization toward a weaker checkpoint. DAPO drops the KL term entirely for long-CoT reasoning RL for exactly this reason — the policy is supposed to move far from the reference, and the penalty mostly buys you a slower move to the same place.

If you keep beta > 0 for stability, at minimum mask the KL on degenerate groups, or filter them out before the loss (which does both jobs at once).

How do I fix GRPO zero-variance groups?

Filter them out and keep sampling until the batch is full of groups that actually carry signal. This is DAPO's dynamic sampling, and it is a ~20 line change:

import torch

def group_advantages(rewards, scale_rewards=False, eps=1e-4):
    """rewards: (num_prompts, G) -> advantages (num_prompts, G)"""
    mean = rewards.mean(dim=1, keepdim=True)
    adv = rewards - mean
    if scale_rewards:                       # classic GRPO
        adv = adv / (rewards.std(dim=1, keepdim=True) + eps)
    return adv                              # scale_rewards=False -> Dr. GRPO

def collect_batch(prompts, rollout_fn, reward_fn, target_groups, G,
                  max_oversample_rounds=4):
    kept_prompts, kept_completions, kept_rewards = [], [], []
    seen = dropped = 0
    for _ in range(max_oversample_rounds):
        if len(kept_rewards) >= target_groups:
            break
        batch = next_prompts(prompts, target_groups)      # your sampler
        comps = rollout_fn(batch, n=G)                    # (B, G) completions
        r = reward_fn(batch, comps)                       # (B, G) float tensor
        alive = r.std(dim=1) > 1e-6                       # nonzero-variance mask
        seen += len(batch); dropped += int((~alive).sum())
        for i in alive.nonzero().flatten().tolist():
            kept_prompts.append(batch[i])
            kept_completions.append(comps[i])
            kept_rewards.append(r[i])
    log({"effective_group_rate": 1 - dropped / max(seen, 1),
         "rollout_amplification": seen / max(len(kept_rewards), 1)})
    return (kept_prompts, kept_completions,
            torch.stack(kept_rewards[:target_groups]))

Two things this code makes visible that a framework flag hides:

rollout_amplification is your real cost multiplier. At a 40% degenerate rate you generate 1.67x the tokens per optimizer step. That is the honest price of dynamic sampling, and it climbs over training. Budget for it, and cap max_oversample_rounds so a run that has genuinely solved its training set fails loudly instead of spinning.

Dropping a prompt is a curriculum decision. A prompt that returns 0/G today is not the same as one returning G/G. The all-correct ones can be retired permanently; the all-wrong ones should be kept in a hard buffer and retried later, because they are the ones that will produce signal after the policy improves. Deleting both with one mask throws away your curriculum.

In TRL, the equivalents are config flags rather than hand-rolled loops — check names against your installed version, since this API has moved fast:

from trl import GRPOConfig

cfg = GRPOConfig(
    num_generations=16,        # bigger G -> fewer degenerate groups, linear cost
    scale_rewards=False,       # Dr. GRPO: drop the std division
    loss_type="dr_grpo",       # token-level, no per-sequence length normalization
    beta=0.0,                  # no KL pull toward the reference
    epsilon=0.2,               # lower clip
    epsilon_high=0.28,         # "clip-higher": more room for low-prob tokens
)

Does response-length normalization interact with this?

Yes, and it compounds. The 1/|o_i| term in vanilla GRPO averages the loss over each response's own token count, so a 4000-token wrong answer gets a much smaller per-token penalty than a 200-token wrong answer. The policy learns that length dilutes punishment, and you get the classic runaway-CoT drift. Token-level loss (sum over all tokens in the batch, divide by total tokens) removes that asymmetry.

Why it compounds with degenerate groups: dynamic sampling preferentially keeps ambiguous prompts — the ones where some rollouts succeed and some fail. Those are also where length inflation pays off most. Fixing the group filtering without fixing the length normalization concentrates your gradient on the exact prompts where the length exploit is most rewarded.

What should I put on the dashboard?

Three metrics, tracked per step:

1. Effective group rate — fraction of sampled groups with nonzero reward variance. This is your true batch size. If it drops below ~0.5, everything else you are reading is misleading.
2. Reward std histogram, not just the mean. Bimodal collapse to {0, high} is the shape that precedes a plateau.
3. Max |advantage| per batch. Spikes above ~3 with std normalization on mean a single rare success is steering the update — go read that completion before trusting the step.

None of this applies to API-only models: you cannot run GRPO against Claude Opus 4.x or GPT-5.x. It applies the moment you post-train an open-weight model with a verifier, which is now most reasoning work.

So why do half your rollouts do nothing in GRPO?

Because GRPO's advantage is a within-group z-score and its baseline is the group mean, so any group whose completions all receive the same reward produces an advantage of exactly zero for every sample — no policy gradient, full generation cost. With binary rewards the degenerate fraction is about p^G + (1-p)^G, which means both a too-hard training set and a too-easy one starve the update, and the "too easy" end arrives precisely as the model gets good. Fix it by filtering zero-variance groups and oversampling to refill the batch (dynamic sampling), dropping the std division so rare lucky successes stop getting 4x weight, setting beta = 0 so degenerate groups cannot contribute pure KL pull, and putting effective group rate on the dashboard next to mean reward — because mean reward will keep rising for hundreds of steps after learning has already stopped.