Introduction Reward engineering is often cited as the most challenging aspect of applied reinforcement learning. While algorithms like PPO and SAC (discussed in Part 4) provide robust training mechanisms, the quality of the learned policy fundamentally depends on how well the reward function captures the desired behavior. A poorly designed reward can lead to reward hacking, where agents exploit loopholes to maximize scores without solving the intended task, or deceptive alignment, where behavior looks correct during training but fails in deployment. This episode explores the art and science of reward function design, covering theoretical foundations like potential-based reward shaping, domain-specific patterns for finance and robotics, common pathologies, and intrinsic motivation techniques. We’ll implement practical examples for both a trading bot and a robotic manipulation task. The Theory of Reward Shaping Potential-Based Reward Shaping Reward shaping modifies the original reward function to accelerate learning without changing the optimal policy. The key insight from Ng, Harada, and Russell (1999) is that adding a potential-based term guarantees policy invariance:
Continue reading the full article on TildAlice
Top comments (0)