GDPO: How Decoupled Reward Normalization Fixes Multi-Objective RL for LLMs

Training large language models with reinforcement learning almost always involves juggling multiple reward signals at once. You want the model to be accurate, follow a specific format, stay within a length budget, and avoid unsafe outputs — all simultaneously. The standard approach is to sum those rewards into a single number and feed it into Group Relative Policy Optimization (GRPO). It works, but it quietly throws away information in a way that hurts training. A new paper from NVIDIA researchers introduces GDPO (Group reward-Decoupled Normalization Policy Optimization) to fix this, and the fix turns out to be surprisingly clean.

What GRPO Does — and Where It Breaks Down

GRPO is a popular alternative to PPO for post-training LLMs. Instead of maintaining a separate critic network, it samples a group of responses for each prompt and computes advantages by comparing each response's reward to the group mean. This is memory-efficient and works well when there is a single reward signal.

The trouble starts when you add more rewards. The common workaround is to sum all reward components — say, r_accuracy + r_format + r_length — and treat the total as one number before running the GRPO advantage calculation. This is where reward collapse happens.

Here is the core problem: different combinations of partial rewards can produce the same total. Imagine two binary reward signals. A response that satisfies both rewards (total = 2) and a response that satisfies neither (total = 0) are clearly different. But a response that satisfies only the first reward (total = 1) and one that satisfies only the second (total = 1) are treated as identical by the advantage formula, even though the model should learn different things from each. When you normalize the summed rewards across a group, these distinct situations collapse into the same advantage value, and the training signal loses resolution.

The GDPO paper formalizes this: as the number of reward signals or sampled responses grows, the number of distinct advantage values that naïve GRPO can produce grows much more slowly than it should. The model is effectively learning from a blurrier signal than the data warrants.

The GDPO Fix: Normalize First, Then Sum

GDPO changes the order of operations. Instead of summing rewards and then normalizing, it normalizes each reward dimension independently within the group before summing. The two-step process looks like this:

Step 1 — Reward-decoupled group normalization. For each reward component r_k, compute the group mean and standard deviation across the sampled responses for that prompt. Subtract the mean and divide by the standard deviation to get a normalized advantage for that reward dimension. Do this separately for every reward signal.

Step 2 — Batch-level advantage normalization. Sum the per-reward normalized advantages into a single total advantage. Then apply a second normalization pass across the entire training batch (not just the group for one prompt). This prevents the magnitude of the combined signal from inflating as you add more reward objectives, keeping training numerically stable.

The result is that responses with different reward profiles now receive genuinely different advantage values. In the binary two-reward example, GDPO produces three distinct advantage combinations where naïve GRPO produces only two. At scale — with more rewards and larger group sizes — the gap widens considerably.

Why Batch-Level Normalization Matters

The second step is easy to overlook but important. Standard GRPO normalizes advantages only within the group sampled for a single prompt. When you sum multiple normalized reward signals, the variance of the combined advantage can grow with the number of rewards, making learning rates and clipping thresholds harder to tune. GDPO's batch-level normalization keeps the advantage distribution stable regardless of how many reward objectives you add, which means you do not need to re-tune hyperparameters every time you introduce a new reward signal.

This is a practical benefit for teams iterating on reward design. Adding a new reward component to a GDPO pipeline is less likely to destabilize training than adding it to a naïve GRPO setup.

Benchmark Results

The NVIDIA team tested GDPO using Qwen2.5-Instruct models at 1.5B and 3B parameter scales across three task categories:

Tool calling. Models trained with GDPO showed clear improvements in both tool-calling accuracy and format compliance compared to GRPO baselines. Format compliance is a natural multi-reward problem — the model must simultaneously produce a valid function call structure and select the correct function — so this is where the advantage of decoupled normalization is most direct.

Mathematical reasoning. On benchmarks including MATH, AIME, and AMC, GDPO maintained accuracy while adhering significantly better to length constraints. Length is often added as a secondary reward to discourage verbose reasoning chains; GRPO's reward collapse makes it harder for the model to learn the accuracy-length trade-off cleanly.

Coding. Across Apps, CodeContests, and Codeforces benchmarks, GDPO achieved a better balance between pass rates and bug ratios. Coding tasks often combine a correctness reward (does the code pass tests?) with a style or efficiency reward, making them another natural fit for decoupled normalization.

The paper also reports that GDPO has been applied in the C-MORAL framework for molecular optimization, where competing chemical property rewards (e.g., binding affinity vs. synthesizability) make reward collapse particularly costly.

How This Fits Into the Broader RL-for-LLMs Landscape

GDPO is not the only recent attempt to address GRPO's limitations. AVSPO tackles a related but distinct problem — advantage collapse when all responses in a group receive the same reward — by injecting virtual samples to restore variance. DAPO addresses entropy collapse in long chain-of-thought training. These methods are complementary: GDPO specifically targets the multi-reward normalization problem, while AVSPO and DAPO address single-reward collapse and entropy issues respectively.

What makes GDPO notable is its simplicity. The change is a reordering of normalization steps — no new model components, no additional forward passes, no architectural modifications. It slots into existing GRPO training pipelines with minimal engineering overhead.

Practical Takeaways

If you are training an LLM with multiple reward signals using GRPO, the key question is whether your reward combination step happens before or after normalization. If you are summing first, you are likely losing signal. Switching to per-reward normalization before aggregation is a low-cost change that the GDPO results suggest is worth making.

The batch-level normalization step is also worth adopting even if you only have one reward signal, since it provides more stable advantage magnitudes across training than group-only normalization.

The GDPO paper is concise and the math is accessible — if you are working on post-training pipelines, it is worth reading alongside the original GRPO paper to understand exactly where the information loss occurs and why the fix works.