DEV Community

Cover image for How GRPO Actually Works: The Math, The Zero-Critic Trick, and Why Reasoning Models Don't Need PPO
Syed Anzar
Syed Anzar

Posted on

How GRPO Actually Works: The Math, The Zero-Critic Trick, and Why Reasoning Models Don't Need PPO

How GRPO Actually Works: The Math, The Zero-Critic Trick, and Why Reasoning Models Don't Need PPO

When the weights for DeepSeek-R1 dropped, the machine learning world noticed two things immediately. First, an open-weights model could match OpenAI's o1 on reasoning benchmarks without tens of thousands of expensive human demonstrations. Second, the training pipeline completely bypassed the most resource-heavy component of traditional Reinforcement Learning from Human Feedback (RLHF): the critic network.

The algorithm powering that shift is Group Relative Policy Optimization (GRPO), first introduced by Shao et al. in the DeepSeekMath paper and later used as the engine for R1.

If you have ever implemented standard Proximal Policy Optimization (PPO) for LLMs, you know the memory footprint is painful. You usually keep four full models in GPU memory simultaneously: the Actor, the Critic (Value Model), the Reference Model, and the Reward Model.

GRPO throws the Critic model in the trash. It drops the parameter overhead of the value function to zero, stabilizes training on outcome-based rewards, and relies on deterministic rule verification instead of noisy reward models.

Here is how the math, the memory layout, and the policy updates actually work under the hood.


1. The PPO Memory Bottleneck: Why Critic Networks Hurt

To understand why GRPO exists, look at what traditional PPO requires when aligning an autoregressive language model.

In standard Actor-Critic PPO:

  1. The Actor ($\pi_\theta$): The model generating tokens. It receives gradients and gets updated.
  2. The Critic ($V_\phi$): A second neural network, usually the exact same size as the Actor, trained to predict the expected cumulative reward from any given state (token prefix).
  3. The Reference Model ($\pi_{ref}$): A frozen copy of the initial supervised fine-tuned (SFT) model used to compute a Kullback-Leibler (KL) divergence penalty so the policy does not drift into gibberish.
  4. The Reward Model ($R_\psi$): A model trained on human preference pairs that evaluates the final response.
Standard PPO Setup (4 Large Models in VRAM):
+-------------------------------------------------------------+
| GPU Memory                                                  |
|  [Actor Model pi_theta]       <-- Trainable (Weights + Opt) |
|  [Critic Model V_phi]         <-- Trainable (Weights + Opt) |
|  [Reference Model pi_ref]     <-- Frozen                    |
|  [Reward Model R_psi]         <-- Frozen                    |
+-------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

If you are fine-tuning a 32-billion parameter model, keeping the Actor, its optimizer states (AdamW takes 8 bytes per parameter in FP32), the Critic, and its optimizer states in memory easily exceeds 160 GB of VRAM before you even allocate space for the KV cache and activation checkpoints.

Beyond the memory penalty, training an LLM critic has a fundamental theoretical problem: reward sparsity.

When solving a competition math problem or writing a Python function, the reward comes at the very end. The answer is either correct (1.0) or incorrect (0.0). In PPO, the critic must learn a value $V(s_t)$ for every single intermediate token step $t$. Training a massive neural network to predict token-level state values from a single terminal scalar reward leads to noisy gradient updates, high variance, and frequent policy collapse.


2. The GRPO Insight: The Group Is Your Baseline

In reinforcement learning, the advantage function $A(s, a) = Q(s, a) - V(s)$ measures how much better a specific action is compared to the average action you could have taken from state $s$.

In PPO, $V(s)$ is approximated by the Critic model.

GRPO replaces the Critic model with a simple Monte Carlo sample across a group of completions for the same prompt.

Instead of asking a separate neural network "How good is this prompt on average?", GRPO samples $G$ distinct outputs from the current policy for the same question $q$:

$${o_1, o_2, \dots, o_G} \sim \pi_{\theta_{old}}(\cdot \mid q)$$

Each completion $o_i$ is evaluated by a reward function to produce a scalar score $r_i$.

Once you have the group of rewards $\mathbf{r} = {r_1, r_2, \dots, r_G}$, GRPO calculates the empirical mean and standard deviation of the group:

$$\bar{r} = \frac{1}{G} \sum_{j=1}^{G} r_j$$

$$\sigma_r = \sqrt{\frac{1}{G} \sum_{j=1}^{G} (r_j - \bar{r})^2}$$

The advantage $\hat{A}_{i,t}$ for completion $i$ is simply the standardized score of that completion relative to its peers:

$$\hat{A}_{i,t} = \tilde{r}_i = \frac{r_i - \bar{r}}{\sigma_r + \epsilon}$$

Every token $t$ in completion $i$ receives the exact same scalar advantage $\hat{A}_i$.

Prompt: "Solve: What is the sum of angles in a convex pentagon?"
Sample G=4 outputs:

Output 1: ... (5-2)*180 = 540 degrees.      Reward: 1.0
Output 2: ... 5*180 = 900 degrees.          Reward: 0.0
Output 3: ... 540 degrees.                  Reward: 1.0
Output 4: ... 360 degrees.                  Reward: 0.0

Mean = 0.5, Std = 0.5
Advantage (Outputs 1 & 3): (1.0 - 0.5) / 0.5 = +1.0  (Tokens reinforced)
Advantage (Outputs 2 & 4): (0.0 - 0.5) / 0.5 = -1.0  (Tokens suppressed)
Enter fullscreen mode Exit fullscreen mode

Look at what happens here:

  • If a completion is better than the group average, its advantage is positive. The policy gradient will increase the likelihood of producing those tokens.
  • If a completion is worse than the group average, its advantage is negative. The policy gradient will penalize those tokens.
  • If all completions in the group produce the exact same reward (for example, all fail with 0.0 or all succeed with 1.0), the numerator $r_i - \bar{r}$ evaluates to 0. The advantage is zero, and no destructive parameter update occurs on that batch.

The Critic network is gone. The parameter overhead drops to zero.


3. PPO vs GRPO: Architectural Comparison

Dimension Standard PPO Group Relative Policy Optimization (GRPO)
Value Estimation Learned neural network $V_\phi(s)$ Empirical group mean $\frac{1}{G}\sum r_j$
Critic Parameters Equal to Actor size (e.g. 7B to 70B) 0 parameters
VRAM Footprint Actor + Critic + Ref + Reward Actor + Ref (or quantized Ref)
Advantage Assignment Token-level Generalized Advantage Estimation (GAE) Group-normalized scalar assigned to all tokens
Reward Sparsity Handling Poor (Critic struggles to learn token values) Robust (Direct terminal reward comparison)
Failure Modes Value drift, critic divergence, OOM crashes Zero-variance batches if group size $G$ is too small

4. The Complete Mathematical Objective

The full GRPO loss function looks intimidating in research papers, but it decomposes into three clean parts.

For a question distribution $P(Q)$ and group size $G$, the objective maximized by GRPO is:

$$\mathcal{J}{GRPO}(\theta) = \mathbb{E}{\substack{q \sim P(Q), \ {o_i}{i=1}^G \sim \pi{\theta_{old}}}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min \left( \rho_{i,t} \hat{A}i, \; \text{clip}(\rho{i,t}, 1-\varepsilon, 1+\varepsilon) \hat{A}i \right) - \beta \mathbb{D}{KL}\left[\pi_\theta \parallel \pi_{ref}\right] \right) \right]$$

Let us unpack the three components:

Component A: The Per-Token Probability Ratio ($\rho_{i,t}$)

$$\rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})} = \exp\left( \log \pi_\theta(o_{i,t}) - \log \pi_{\theta_{old}}(o_{i,t}) \right)$$

This is standard importance sampling. Because sampling $G$ sequences takes significant time, you want to perform multiple gradient steps (or minibatch updates) on the collected rollouts. The ratio $\rho_{i,t}$ tracks how much the active policy $\pi_\theta$ has shifted away from the rollout policy $\pi_{\theta_{old}}$ that originally generated the tokens.

Component B: The Clipped Surrogate

$$\min \left( \rho_{i,t} \hat{A}i, \; \text{clip}(\rho{i,t}, 1-\varepsilon, 1+\varepsilon) \hat{A}_i \right)$$

Just like in standard PPO, the clipping parameter $\varepsilon$ (typically set to 0.1 or 0.2) prevents a single gradient step from drastically altering the probability of a token. If the advantage is positive and the ratio exceeds $1 + \varepsilon$, the gradient is clipped. If the advantage is negative and the ratio falls below $1 - \varepsilon$, the gradient is clipped.

Component C: The Schulman KL Regularization

In traditional PPO, the KL divergence penalty is often subtracted directly from the reward before computing advantage values:

$$r't = r_t - \beta \log \frac{\pi\theta(a_t \mid s_t)}{\pi_{ref}(a_t \mid s_t)}$$

In GRPO, the reward is kept pure, and the KL divergence penalty is added directly into the loss function.

DeepSeek uses the unbiased, low-variance estimator developed by John Schulman:

$$\mathbb{D}{KL}\left[\pi\theta \parallel \pi_{ref}\right] = \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - \log \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - 1$$

Why use this specific formula?

Notice the algebraic property: let $x = \frac{\pi_{ref}}{\pi_\theta}$. The term is $f(x) = x - \log(x) - 1$.
For all $x > 0$, $f(x) \ge 0$, with $f(1) = 0$.

Standard sample-based estimators like $\log(\pi_\theta / \pi_{ref})$ can produce negative sample values that introduce noise into gradient updates. The Schulman estimator is strictly non-negative at every sample point and has provably lower variance.


5. Why Rule-Based Verifiers Beat Neural Reward Models

The second critical innovation behind DeepSeek-R1's reasoning success was not just GRPO itself, but what feeds rewards into GRPO.

In conventional RLHF (used to make chatbots friendly and conversational), rewards come from a Neural Reward Model. That reward model is another neural network trained on human ranking data (Response A is better than Response B).

Neural reward models suffer from Goodhart's Law: When a measure becomes a target, it ceases to be a good measure.

When an LLM trains against a neural reward model for thousands of steps, it finds adversarial exploit paths. It learns that writing five paragraphs of apologetic, formatting-heavy, sycophantic text scores higher than giving a concise answer, even if the underlying code or math is wrong.

For DeepSeek-R1-Zero, the team discarded neural reward models completely for reasoning tasks and used deterministic rule-based verifiers:

$$R_{rule} = R_{accuracy} + R_{format}$$

+-------------------------------------------------------------+
| Prompt: Math / Coding Task                                  |
+-------------------------------------------------------------+
                              |
                     [Current Policy]
                              |
                 Generates G=16 completions
                              |
                              v
+-------------------------------------------------------------+
| Rule-Based Verifiers (Zero Neural Inference)                |
|                                                             |
| 1. Accuracy Verifier:                                       |
|    - Code: Run inside sandbox with `pytest`                 |
|    - Math: Extract `\boxed{...}`, compare via SymPy         |
|    Score: 1.0 if correct, 0.0 if failed                     |
|                                                             |
| 2. Format Verifier:                                         |
|    - Must start with `<think>` and end with `</think>`      |
|    - Final answer enclosed in specified tags                |
|    Score: 1.0 if formatted properly, 0.0 if malformed       |
+-------------------------------------------------------------+
                              |
               Exact binary reward: 0.0 or 1.0
Enter fullscreen mode Exit fullscreen mode

A Python unit test suite does not care how polite your model is. Either solution(5) returns 120, or it fails. A SymPy algebraic solver does not care about persuasive prose. Either the expression simplifies to the ground truth, or it gets zero reward.

Because the reward is objective and deterministic, the model cannot hack the verifier by being verbose. The only way to increase expected reward is to actually get the right answer.


6. How Reasoning Emerges Organically: The "Aha Moment"

One of the most fascinating findings in the DeepSeek-R1 paper was that pure reinforcement learning (R1-Zero) discovered self-reflection without any human prompting.

During training, the model started generating phrases like:

"Wait, let me double check that..."
"Let's re-evaluate from step 2..."
"Wait, this contradicts the earlier condition, let me try a different substitution..."

Why does this happen without supervised examples?

Consider the exploration dynamics in a group of $G=16$ completions:

  1. Prompt: A complex geometry problem.
  2. 15 completions attempt a direct calculation, make a sign error on step 3, and fail. Reward = 0.0.
  3. 1 completion randomly generates a token sequence that pauses to check its work: "Wait, let me verify if the triangle inequality holds." It catches the sign error, corrects the calculation, and outputs the right answer. Reward = 1.0.

Now compute the advantages:

  • Mean reward = $1/16 = 0.0625$.
  • The 15 failing outputs get slightly negative advantage: $-0.26$.
  • The single successful output gets a huge positive advantage: $+3.87$.

During the gradient update, every token in that winning trajectory receives strong positive reinforcement. The weights that generated the backtracking and verification tokens are boosted. Over thousands of optimization steps, the model internalizes self-correction as the most reliable policy to maximize its reward on difficult tasks.


7. Minimal Implementation in PyTorch

Here is a clean, readable implementation showing how group advantage normalization, clipped surrogate loss, and the Schulman KL penalty are computed in PyTorch:

import torch
import torch.nn.functional as F

def compute_group_advantages(rewards: torch.Tensor, eps: float = 1e-4) -> torch.Tensor:
    """
    Computes group-relative advantages across G completions for each prompt.

    Args:
        rewards: Tensor of shape (batch_size, group_size)
        eps: Small stabilizer to prevent division by zero
    Returns:
        advantages: Tensor of shape (batch_size, group_size)
    """
    # Calculate empirical mean and std across the group dimension (dim=-1)
    mean = rewards.mean(dim=-1, keepdim=True)
    std = rewards.std(dim=-1, keepdim=True)

    # Standardize rewards within each prompt's group
    advantages = (rewards - mean) / (std + eps)

    # Edge case: If all completions have identical reward, advantage is zero
    zero_variance_mask = (std < 1e-6)
    advantages = torch.where(zero_variance_mask, torch.zeros_like(advantages), advantages)

    return advantages


def grpo_loss(
    log_probs: torch.Tensor,        # (B * G, seq_len) - Current policy log pi_theta
    old_log_probs: torch.Tensor,    # (B * G, seq_len) - Rollout policy log pi_old
    ref_log_probs: torch.Tensor,    # (B * G, seq_len) - Frozen reference model
    advantages: torch.Tensor,       # (B * G, 1)       - Group advantage for each sequence
    attention_mask: torch.Tensor,   # (B * G, seq_len) - 1 for valid tokens, 0 for pad
    clip_eps: float = 0.2,
    beta_kl: float = 0.04
) -> torch.Tensor:
    """
    Computes token-level GRPO loss with clipped surrogate objective and Schulman KL.
    """
    # 1. Compute per-token importance ratio: exp(log_pi - log_pi_old)
    ratio = torch.exp(log_probs - old_log_probs)

    # 2. Clipped surrogate loss
    surr1 = ratio * advantages
    surr2 = torch.clamp(ratio, 1.0 - clip_eps, 1.0 + clip_eps) * advantages
    policy_loss = -torch.min(surr1, surr2)

    # 3. John Schulman's unbiased KL estimator: exp(log_ref - log_pi) - (log_ref - log_pi) - 1
    log_ratio_ref = ref_log_probs - log_probs
    kl_divergence = torch.exp(log_ratio_ref) - log_ratio_ref - 1.0

    # 4. Total per-token loss
    total_token_loss = policy_loss + beta_kl * kl_divergence

    # 5. Mask padding tokens and average over valid tokens
    masked_loss = total_token_loss * attention_mask
    loss_per_sequence = masked_loss.sum(dim=-1) / attention_mask.sum(dim=-1).clamp(min=1.0)

    return loss_per_sequence.mean()
Enter fullscreen mode Exit fullscreen mode

8. Common Pitfalls When Implementing GRPO

If you attempt to train or fine-tune an open-source model using GRPO (via frameworks like Hugging Face TRL or Unsloth), watch out for these subtle traps:

1. Group Size ($G$) Set Too Low

In standard PPO, mini-batches can be small because the Critic model provides a continuous value baseline. In GRPO, the baseline is estimated strictly from your group samples. If you set $G=2$ or $G=4$, the empirical variance of your baseline is enormous. A single lucky guess completely skews the group mean. In practice, $G \ge 8$ or $G=16$ is needed for stable gradient signals.

2. Format Reward Overweighting

If your format reward (enforcing <think> tags) gives $0.5$ points and your accuracy reward gives $0.5$ points, the model quickly realizes that writing valid XML tags is significantly easier than solving calculus. It will settle into a local optimum where it emits empty <think></think> blocks to harvest the format reward without solving the task. Keep format rewards minimal (or treat them as a prerequisite gate rather than an additive bonus).

3. Length Explosion (Reasoning Bloat)

Because correct answers often correlate with longer, more thorough exploration, the model can develop a bias toward generating massive chains of thought even for trivial queries. Without length regularization or prompt diversity, average generation length can balloon from 500 tokens to 8,000+ tokens, exhausting GPU memory during rollout generation.


The Core Takeaway

Reasoning models do not rely on mysterious black-box magic. The shift represented by GRPO and DeepSeek-R1 comes down to pragmatic systems engineering:

  1. Drop the Critic: Use group sampling to turn relative rank into an advantage signal, slashing VRAM overhead by half.
  2. Drop the Neural Reward Model: Use automated tests, sandboxed execution, and symbolic math checkers to provide deterministic, unhackable feedback.
  3. Let Exploration Work: Given enough test-time tokens and verifiable feedback, reinforcement learning naturally pushes the policy toward backtracking and self-correction.

Understanding GRPO gives you a concrete mental model of how reasoning models train, how test-time compute works in practice, and why the value network was the first thing to go.

Top comments (0)