GRPO Explained: Why DeepSeek Dropped the Critic in RLHF

Train a reasoning model with PPO and half your GPU memory goes to a network that never ships: the critic. It's the same size as the policy, it's notoriously hard to fit, and its only job is to guess how good a half-finished answer will turn out. GRPO — Group Relative Policy Optimization — deletes it. Instead of learning a value function, it samples several answers to the same prompt and uses their average reward as the baseline. That single swap is most of why DeepSeek could scale RL on math and code the way it did.

TL;DR

• GRPO replaces PPO's learned value network with a baseline computed from a group of sampled completions per prompt — no critic, roughly half the training memory.
• The advantage for a completion is its reward minus the group mean, divided by the group std. Every token in that completion gets the same advantage (outcome supervision).
• The KL penalty to the reference model is added directly to the loss as an unbiased positive estimator, not folded into the reward as in classic RLHF.
• It shines with verifiable rewards (RLVR) — math answers, unit tests — where a rule can score a completion 0/1 without a reward model.
• Known failure modes: groups where all rewards are equal produce zero gradient, and the length/std normalization introduce measurable biases that later work (Dr. GRPO) strips out.

Why does PPO need a critic in the first place?

Policy-gradient methods multiply the log-prob of each action by an advantage: how much better this action was than average. You need a baseline for "average," or the gradient has brutal variance. PPO estimates that baseline with a value network V(s) trained via GAE (generalized advantage estimation), so it can assign credit token-by-token to a partially generated sequence.

That works, but the cost is real. The critic is typically initialized from the same backbone as the policy, so you're holding two large models in memory, running two forward/backward passes, and tuning a second optimizer. In LLM RL the value target is also weak — you get one scalar reward at the end of a long generation, and the critic has to back-propagate that through hundreds of tokens of intermediate reasoning it can't really evaluate. In practice the critic is the flakiest part of the pipeline.

What does GRPO do instead?

GRPO throws out the value network and builds the baseline empirically. For each prompt q, sample a group of G completions from the current policy. Score each one to get rewards r_1 … r_G. The baseline is just the group mean, and the advantage is the standardized reward:

A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)

Every token in completion i receives the same advantage A_i. This is outcome supervision: the model isn't told which token was good, only that the whole answer beat or missed the group average. With verifiable tasks (was the final answer correct? did the tests pass?) that's often all the signal you have anyway, and it turns out to be enough.

The surrogate objective keeps PPO's clipped importance ratio so you can do multiple gradient steps per batch of samples:

ρ_{i,t} = π_θ(o_{i,t} | q, o_{i,<t}) / π_θ_old(o_{i,t} | q, o_{i,<t})

J = (1/G) Σ_i (1/|o_i|) Σ_t [ min( ρ_{i,t} A_i,
                                   clip(ρ_{i,t}, 1-ε, 1+ε) A_i ) ]
      - β · D_KL(π_θ ‖ π_ref)

Two things differ from vanilla PPO. There's no V(s) anywhere. And the KL term sits in the loss, not the reward.

How does GRPO handle the KL divergence?

Classic RLHF shapes the reward: r = r_model − β·log(π_θ/π_ref) per token, then runs PPO on that. GRPO pulls the KL out of the reward and adds it to the loss as an explicit penalty against the frozen reference (usually the SFT model). It uses the low-variance, always-positive k3 estimator:

D_KL ≈ π_ref(o|q) / π_θ(o|q) − log( π_ref(o|q) / π_θ(o|q) ) − 1

That expression is ≥ 0 for every sample and unbiased in expectation, which keeps the penalty well-behaved token to token. Keeping KL out of the reward also means your reward stays interpretable — if you're using a 0/1 correctness reward, it stays 0/1, and the reference anchoring is a separate, tunable force.

What does the core computation actually look like?

Here's the part that replaces the entire critic — group-normalized advantages and the clipped loss, in PyTorch-ish pseudocode:

import torch
import torch.nn.functional as F

def grpo_loss(logp, logp_old, logp_ref, rewards, group_size,
              clip_eps=0.2, beta=0.04):
    # logp*, shape [B*G, T]: per-token log-probs of the sampled tokens
    # rewards, shape [B*G]:  one scalar per completion (e.g. 1.0 if correct)
    B = rewards.shape[0] // group_size
    r = rewards.view(B, group_size)

    # --- the baseline that used to be a whole value network ---
    adv = (r - r.mean(dim=1, keepdim=True)) / (r.std(dim=1, keepdim=True) + 1e-6)
    adv = adv.view(-1, 1)                       # broadcast to every token

    ratio = torch.exp(logp - logp_old)          # importance ratio ρ
    unclipped = ratio * adv
    clipped   = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * adv
    policy_loss = -torch.min(unclipped, clipped)

    # k3 unbiased, non-negative KL to the reference policy
    log_r = logp_ref - logp
    kl = torch.exp(log_r) - log_r - 1.0

    per_token = policy_loss + beta * kl
    # length-normalize within each completion, then average
    return per_token.mean()

Note what's missing: no value_head, no GAE, no value-loss term, no second optimizer. The baseline is three lines of tensor arithmetic. If a group's rewards are all identical, r.std is ~0 and adv collapses to zero — that group contributes no gradient. This is the first failure mode to watch.

When is GRPO the right choice — and when not?

GRPO earns its keep when you can sample many completions cheaply and score them with a rule. Math with a checkable final answer, code with a test suite, format-constrained tasks — this is the RLVR (RL with verifiable rewards) sweet spot. No reward model to train, no critic to babysit, and the group baseline is exactly as good as your reward is honest.

It's a worse fit when:

• Rewards are dense or need per-step credit. Outcome-only advantages give every token the same push. For long-horizon agentic tasks where an early tool call mattered and a late one didn't, a value function (or process reward model) carries information GRPO discards.
• You can't afford G samples per prompt. GRPO's variance drops as the group grows; tiny groups (G=2–4) are noisy. You're trading critic compute for extra rollouts. On long generations, those rollouts dominate wall-clock.
• Reward saturates. As the policy improves, more groups come back all-correct or all-wrong. Those produce zero advantage, so effective batch size silently shrinks and training stalls. Curriculum or difficulty filtering helps keep groups mixed.

What biases does GRPO's normalization introduce?

Two, and both come from the innocuous-looking normalizers. The (1/|o_i|) length term means a wrong short answer is penalized harder per token than a wrong long one, which nudges the model toward longer completions when it's failing — a documented driver of response-length inflation during RL. And dividing by std(r) makes easy prompts (low variance) contribute disproportionately large advantages, biasing optimization toward questions the model already mostly solves.

The "Understanding R1-Zero-like Training" work (a.k.a. Dr. GRPO) argues both terms are unnecessary artifacts: drop the length normalization and the std division, keep only A_i = r_i − mean(r), and you remove the length and difficulty biases while matching or beating quality. If you're rolling your own GRPO and seeing answers balloon in length, that's the first knob to touch:

# Dr. GRPO variant: mean-centered only, no length/std normalization
adv = (r - r.mean(dim=1, keepdim=True)).view(-1, 1)
# ... and sum per-token losses instead of length-averaging

GRPO vs DPO vs PPO — how do they line up?

DPO skips RL entirely: it optimizes a closed-form preference loss on pairs of (chosen, rejected) responses, no sampling, no reward model at inference-time collection. It's cheap and stable but offline — it only ever sees the pairs you hand it, and it has its own pathologies like likelihood displacement. PPO is online, uses a reward model and a critic, and gives the finest-grained credit assignment at the highest engineering cost. GRPO sits between: online like PPO (it samples fresh completions and reacts to the current policy), but critic-free like nothing else, and it needs a scorer rather than a full reward model when your task is verifiable. For reasoning workloads where correctness is checkable, that's the combination that scaled.

The direct answer

Why did DeepSeek drop the critic in RLHF? Because in verifiable-reward RL the critic is the most expensive and least reliable component, and GRPO shows you don't need it: sampling a group of completions per prompt and standardizing their rewards gives a baseline that's good enough to compute advantages directly. That halves training memory, removes the flakiest part of the PPO loop, and — with a rule-based reward for math or code — turns RL fine-tuning into something you can actually scale. The costs are real too: outcome-only credit, zero-gradient groups when rewards saturate, and length/std biases you may want to normalize away. Reach for GRPO when completions are cheap to sample and rewards are cheap to verify; reach for PPO when you genuinely need per-token value estimates, and DPO when you only have offline preference pairs.