The 500K Step Wall
Most RL tutorials show you CartPole with dense rewards every step. Then you try a real problem — say, training a robotic arm to insert a peg — and your PPO agent is still flailing randomly after 500K steps.
The issue isn't your hyperparameters. It's that PPO was designed for dense feedback, and you just gave it a binary "success or fail" signal that fires once every 200 steps. SAC handles this 3x better in sample efficiency, but crashes in other scenarios. Knowing when to pick which algorithm saves you days of wasted training runs.
I've burned GPU hours on both. Here's what actually separates them when rewards are sparse.
Why Sparse Rewards Break PPO's Core Assumptions
PPO relies on advantage estimation — specifically, how much better an action was compared to the baseline value function. The advantage $A_t = \sum_{i=0}^{T-t} \gamma^i r_{t+i} - V(s_t)$ requires frequent reward signals to give meaningful gradients.
Continue reading the full article on TildAlice
Top comments (0)