Reinforcement Learning Series

I created this series to make that steep learning curve far less daunting for you. My goal is to trace the evolution of RL chronologically—from its roots in early psychology and physical mechanical machines to digital binary systems and modern mathematical breakthroughs. By breaking down complex concepts with clear visual guides, graphics, and real-world analogies, I hope to demystify RL and give back to the community that inspired me. Let’s dive in!

Blog 1: The Psychological Seeds (1898–1949)

This entry explores the Law of Effect (1911), where Edward Thorndike established that actions followed by satisfaction are strengthened. It also covers Ivan Pavlov’s formal definition of reinforcement (1927) and Donald Hebb’s 1949 hypothesis that "neurons that fire together, wire together," laying the groundwork for neural learning.

Blog 2: Cybernetics and Early Machines (1948–1954)

This blog details the first computational investigations, including Alan Turing’s "pleasure-pain system" (1948) and Marvin Minsky’s construction of SNARCs (1954), the first analog neural-network reinforcement calculators. It also highlights Claude Shannon’s 1952 demonstration of "Theseus," a maze-running mouse that used trial and error to "remember" paths.

Blog 3: The Mathematical Framework (1950s–1960s)

This post focuses on Richard Bellman’s development of Dynamic Programming and the Bellman Equation (1957), which provided the formal tools for solving optimal control problems. It also introduces Markov Decision Processes (MDPs) and Ron Howard’s policy iteration method (1960), which are the bedrock of modern RL theory.

Blog 4: Early Heuristics and Checkers (1959–1968)

This entry discusses Arthur Samuel’s celebrated checkers program (1959), which was the first to implement temporal-difference (TD) ideas by using board evaluation differences to modify earlier positions. It also details Donald Michie’s MENACE (1961), a physical "matchbox" engine that learned to play Noughts and Crosses through reinforcement.

Blog 5: The Birth of Learning Automata (1961–1974)

This blog examines the Soviet lineage of RL, specifically Mikhail Tsetlin’s work on "learning automata" (1961), which were finite-state machines that updated action probabilities based on stochastic environmental feedback. It traces how this research was later systematised by Narendra and Thathachar in 1974.

Blog 6: The "Hedonistic" Revival (1972–1980)

This post covers Harry Klopf’s pivotal role in reviving the trial-and-error thread by proposing the "hedonistic neuron" (1972), which argued that neurons act as pleasure-seeking units. It also explores Paul Werbos’s 1974 thesis, which was the first to describe backpropagation in the context of "Adaptive Dynamic Programming".

Blog 7: The Sutton and Barto Foundation (1981–1984)

This blog highlights the formalisation of modern RL, beginning with Sutton and Barto’s 1981 model of classical conditioning. It details the invention of the Actor-Critic architecture (1983) and Sutton’s 1984 dissertation, which provided the first rigorous computational analysis of temporal credit assignment.

Blog 8: The Q-Learning Breakthrough (1988–1994)

This entry documents two watershed moments: Sutton’s 1988 formalisation of TD learning and Chris Watkins’s 1989 introduction of Q-Learning, the first model-free, off-policy algorithm. It also covers the introduction of SARSA (1994) as an on-policy alternative that prioritises training stability.

Blog 9: Superhuman Achievement in Games (1992–1999)

This post details Gerald Tesauro’s TD-Gammon (1992), which achieved grandmaster-level backgammon play using neural networks and self-play. It also covers the options framework (1999) by Sutton, Precup, and Singh, which introduced temporal abstraction to help RL agents solve problems across different timescales.

Blog 10: Theoretical Refinement (2000–2010)

This entry explains the Policy Gradient Theorem (2000), which proved that policy parameters could be optimised using only local information. It also introduces Sham Kakade’s Natural Policy Gradient (2002), a precursor to modern trust-region methods that account for the geometry of parameter space.

Blog 11: The Deep RL Revolution (2013–2015)

This post chronicles the birth of Deep RL with DeepMind’s DQN (2013/2015), which used Experience Replay and Target Networks to master Atari games directly from pixels. It also details DDPG (2015), which extended these successes to continuous action spaces like robotics.

Blog 12: Mastering Strategy and Control (2015–2017)

This entry explores Trust Region Policy Optimization (TRPO) (2015) and its simpler successor, Proximal Policy Optimization (PPO) (2017), which became the industry standard for stable training. It also covers the historic victory of AlphaGo over Lee Sedol in 2016.

Blog 13: Integrated Mastery: Rainbow and MuZero (2017–2019)

This post details the Rainbow agent (2017), which combined seven independent DQN improvements into one "super-agent". It also highlights MuZero (2019), which learned to master games like Go and Chess without even being told the rules of the environment.

Blog 14: Hard Exploration and Generalisation (2020–2022)

This blog focuses on Agent57 (2020), the first agent to surpass the human benchmark on all 57 Atari games by solving "hard exploration" problems. It also covers Gato (2022), a single transformer-based generalist agent capable of playing Atari, captioning images, and controlling robot arms.

Blog 15: The Transformer Pivot and LLM Alignment (2021–2023)

This entry describes the Decision Transformer (2021), which reframed RL as a sequence modeling problem, bypassing traditional bootstrapping. It also covers the rise of Reinforcement Learning from Human Feedback (RLHF) with InstructGPT and the subsequent development of Direct Preference Optimization (DPO).

Blog 16: Reasoning Models and Incentivized Thought (2024–2025)

This blog explores DeepSeek Math and GRPO (2024), which removed the need for a separate critic model to improve mathematical reasoning. It details the breakthrough of DeepSeek-R1 (2024/2025), which proved that pure RL can spontaneously elicit self-correction and reflection in language models.

Blog 17: Massive Depth and the Future of Scaling (2025–2026)

This final entry discusses the 2025 NeurIPS Best Paper on 1,000-layer RL networks, which broke the "shallow network" assumption to unlock 50x performance gains. It reflects on Kimi k1.5 and the role of the 2024 Turing Award in cementing RL as a foundational pillar of the AI revolution.