The γ=0.99 Default Is Destroying Your Long-Horizon Training
Your PPO agent trained perfectly on CartPole, so you bump the episode length from 500 steps to 10,000 and watch the reward curve collapse into noise. The culprit? That innocent-looking gamma=0.99 you copied from every tutorial.
I've seen this pattern break training across MuJoCo locomotion, robotic manipulation, and custom industrial control tasks. The math is brutal: with γ=0.99 and a 10,000-step horizon, your effective horizon is $\frac{1}{1-\gamma} = 100$ steps. Everything beyond that gets exponentially crushed. Your agent literally cannot see the long-term consequences of its actions.
But here's what surprised me — dropping gamma to 0.95 or even 0.9 often makes things worse, not better. The fix requires understanding why high gamma causes numerical instability in the first place, and it's not what most people think.
Why γ=0.99 Explodes on Long Horizons
The return calculation in policy gradient methods accumulates discounted rewards:
Continue reading the full article on TildAlice
Top comments (0)