#nvidia Just Fixed #GRPO! Meet #GDPO: The New Standard for Multi-Reward RL

Dive into a groundbreaking new paper from NVIDIA that identifies a fundamental flaw in Group Relative Policy Optimization (GRPO) when used with multiple rewards. While GRPO has become the de facto training pipeline for aligning LLMs, the researchers found that naively summing rewards causes a "reward collapse," where distinct performance levels are mapped to identical advantage values. This information loss leads to suboptimal training and even early convergence failure. Enter GDPO (Group reward-Decoupled Normalization Policy Optimization). NVIDIA’s new method fixes this by decoupling the normalization of individual rewards before they are aggregated. This simple but effective change preserves the resolution of the training signal, allowing the model to distinguish between "good" and "great" responses across different objectives like accuracy, formatting, and response length. Key Highlights from the Sources: • The Problem: GRPO often collapses 6 distinct reward combinations into just 2 advantage groups. • The Solution: GDPO increases the granularity of the training signal, preserving significantly more distinct advantage groups as rewards or rollouts increase. • Results on Benchmarks: GDPO consistently outperforms GRPO across tool calling, math reasoning (AIME, MATH), and coding tasks. • Real-World Gains: Training DeepSeek-R1-1.5B with GDPO yielded up to 6.3% higher accuracy on AIME while keeping responses more concise. • Stability: GDPO eliminates the training instability seen in GRPO, which often sees correctness scores decline after 400 steps in complex tasks. Whether you're training a reasoning model or working on RLHF, GDPO is a critical update to the RL toolkit. Paper Title: GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization Authors: Shih-Yang Liu, Xin Dong, et al. (NVIDIA)