DEV Community: Daksh Jain

Precision Engineering: Landing a Rocket with Soft Actor-Critic (SAC)

Daksh Jain — Thu, 25 Jun 2026 04:00:00 +0000

In the previous projects in this portfolio, our AI agents interacted with the world using a "D-pad." The warehouse robots could move Up, Down, Left, or Right. The smart grid battery could Charge or Discharge. These are called Discrete Actions.

But the physical world rarely works like a D-pad. It works like a steering wheel.

If you are trying to land a multi-million dollar rocket back onto a landing pad, you cannot just tell the main engine to "Turn On" or "Turn Off." You have to output an exact, continuous value. You might need exactly 42.7% thrust on the main engine, while simultaneously firing the left lateral thruster at 14.2%. If you output too much thrust, the rocket shoots back into the atmosphere. If you output too little, it violently crashes into the concrete.

This is the frontier of Continuous Control.

To solve this, I built the Rocket Lander Simulator. In this project, we transition away from the algorithms we've used so far and introduce one of the most powerful continuous control algorithms in modern robotics: Soft Actor-Critic (SAC).

1. The Curse of the Continuous

Why can't we just use the Deep Q-Networks (DQN) we used to solve our Smart Grid and Logistics projects?

DQN is a discrete algorithm. It calculates the expected future reward for a fixed list of actions, and then picks the highest one. If you wanted to use DQN to land a rocket, you would have to chop the engine throttle into discrete buckets: 10%, 20%, 30%, etc.

But what if the perfect landing requires exactly 15.5% thrust? If you create a bucket for every possible decimal percentage across multiple engines, the number of possible combinations explodes into the millions. The neural network would freeze, unable to compute the math in real time.

We need an algorithm that doesn't choose from a list, but rather generates a highly precise continuous number.

The Physics of the Action Space

To achieve this, the neural network in our simulation outputs an array of two continuous numbers, both bounded between -1.0 and 1.0:

Main Engine: Negative values mean the engine is off. Positive values map smoothly to thrust intensity (e.g., 0.5 = 50% thrust).
Lateral Thrusters: Negative values fire the left thruster (pushing the rocket right). Positive values fire the right thruster.

Because these outputs are mathematically continuous, the AI can make micro-adjustments smaller than a fraction of a percent to perfectly balance the rocket.

2. Generating Precision: The Gaussian Actor

In the Warehouse project, we introduced the Actor-Critic architecture. SAC uses this same dual-brain setup, but the Actor behaves entirely differently.

Instead of outputting probabilities for discrete buttons, the SAC Actor outputs the mathematical parameters of a Gaussian Distribution (a Bell Curve). Specifically, for every engine, it outputs two variables:

The Mean (μ): What the AI thinks the exact perfect throttle percentage is.
The Standard Deviation (σ): How confident the AI is in that guess (the spread of the curve).

Here is exactly what that looks like in PyTorch. The neural network outputs the mean and standard deviation, and we sample an action from that curve:

import torch
from torch.distributions import Normal

# 1. The Neural Network outputs the parameters of the Bell Curve
mean = actor_network(state)
log_std = torch.clamp(actor_log_std, min=-20, max=2)
std = log_std.exp()

# 2. We build the mathematical Bell Curve
distribution = Normal(mean, std)

# 3. We sample a random throttle percentage from the curve
action = distribution.rsample() 

# 4. Squeeze it between -1.0 and 1.0 for the physics engine
action = torch.tanh(action)

When the AI is completely untrained, its Standard Deviation is massive. The code above will wildly fire the engines at random percentages. As the Critic network slowly coaches the Actor, the Standard Deviation mathematically shrinks. The Bell Curve becomes a sharp, narrow spike, and the AI outputs highly precise, deterministic throttle commands.

But landing a rocket is dangerous. We have to prevent the AI from becoming recklessly overconfident.

3. Curing Neural Optimism (The Twin Critics)

Neural Networks suffer from a well-documented psychological flaw: Optimism Bias.

If an untrained AI accidentally fires its engines at 100% and miraculously avoids crashing due to a lucky gust of wind, the Critic network might immediately assume that 100% thrust is a genius move. It vastly overestimates the value of that action. In a physical simulation, optimism leads to catastrophic, vehicle-destroying crashes.

To cure this, SAC introduced a brilliant engineering trick: Twin Critics.

Instead of having one Critic coach the Actor, SAC uses two completely independent Critic networks (Q1 and Q2). When the Actor asks, "How many points will I get if I fire the engine at 42%?", both Critics calculate an answer.

The algorithm mathematically forces the Actor to take the minimum of the two predictions:

# Critic 1 predicts the future reward
q1_value = critic_1(state, action)

# Critic 2 independently predicts the future reward
q2_value = critic_2(state, action)

# The AI assumes the absolute worst-case scenario
target_q_value = torch.min(q1_value, q2_value)

By forcing the AI to assume the absolute worst-case scenario, SAC completely eliminates optimism bias. The AI only attempts a dangerous maneuver if both pessimistic coaches agree that it is completely safe.

4. Maximum Entropy RL: Rewarding Chaos

The most defining feature of Soft Actor-Critic is the word "Soft". This refers to Maximum Entropy Reinforcement Learning.

In standard RL, the goal is simple: Maximize the Reward.
But SAC changes the fundamental equation of AI by adding a new term to the objective function:

Objective = Maximize (Reward + α * Entropy)

Entropy is the mathematical measurement of chaos and randomness. Why on earth would we actively reward a rocket for flying chaotically?

Imagine an AI that finds exactly one perfect, elegant path to the landing pad. It memorizes that path perfectly. But what happens during a real flight if a massive gust of wind blows the rocket 10 feet to the left? Because the AI only memorized one path, it has no idea what to do, panics, and crashes.

By mathematically rewarding Entropy (scaled by a temperature parameter α or alpha), we actively force the AI to not memorize a single path.

# Calculate how 'predictable' the AI's action was
log_prob = distribution.log_prob(action)

# If the action was highly unpredictable (high entropy),
# the log_prob is very low/negative. We subtract it to give a bonus!
actor_loss = (alpha * log_prob) - min_q_value

We force it to discover 1,000 different, slightly messy ways to land the rocket. By forcing the AI to explore the chaos, it builds an incredibly robust, generalized intuition. When that massive gust of wind hits it in the real world, the AI doesn't panic—it has already explored that exact chaotic state during its training and knows exactly how to dynamically recover.

5. Liquid Memory and Sample Efficiency

If you've ever trained a neural network, you know they are incredibly data-hungry. To solve this, SAC is designed to be an Off-Policy algorithm with a massive Replay Buffer.

Instead of throwing away data after every flight, SAC records every single millisecond of telemetry (State, Action, Reward, Next State) into a memory bank that holds 1,000,000 steps. While the rocket is flying, the AI is constantly "daydreaming" about old flights, randomly sampling batches of past mistakes to squeeze every possible drop of mathematical insight out of them. This makes SAC incredibly Sample Efficient.

Polyak Averaging (The Smooth Update)

Because the AI is learning so aggressively from its past, we run into a stability problem. If the Target Critic network updates its weights too abruptly, the AI loses its mind and unlearns how to fly.

To fix this, SAC uses a trick called Polyak Averaging. Instead of replacing the Target network's brain entirely, it updates using an exponential moving average (tau = 0.005).
Every step, the new brain is composed of 99.5% of the old weights, and 0.5% of the newly learned weights. This creates an ultra-smooth, liquid learning curve that prevents the math from oscillating wildly.

🧪 Try It Yourself

To truly appreciate the power of SAC, you have to watch its continuous precision and test its limits. Open the Rocket Lander Simulator and run these engineering tests:

Read the Telemetry: Go to the Mission Control tab and run a baseline flight. Look at the Engine Throttle chart at the bottom. You won't see blocky, ON/OFF steps. You will see incredibly smooth, continuous curves as the AI perfectly modulates the main engine to counter gravity, followed by lightning-fast micro-bursts from the lateral thrusters to correct its angle.
The Wind Turbulence Test: This is where the Entropy training shines. Turn the Wind Power up to 15, and the Turbulence up to 1.5. Run the mission again. Watch the replay GIF. You will see the rocket get violently pushed off-course by invisible forces, but because of its Maximum Entropy training, it dynamically corrects its continuous thrusters in real-time to fight the wind and hit the pad.
The Sim-to-Real Gap (Distribution Shift): Change the gravity from -10.0 (Earth) to -5.0 (The Moon). If the pre-trained AI crashes, you are witnessing Distribution Shift. Because the AI was only trained on Earth, its Bell Curves are perfectly tuned for Earth physics. When the physical rules change, the AI fails. This is exactly why deploying robotics in the real world is so hard—simulators never perfectly match reality.
Fine-Tune the Brain: If the AI failed the Moon test, go to the Training Lab. Run 10,000 timesteps of fine-tuning, and then run the Moon mission again. Watch how quickly the Replay Buffer adapts the weights to the new gravity.

Wrapping Up

Moving from discrete grids to continuous physical simulations is the holy grail of robotics. It requires us to abandon simple lookup tables and embrace algorithms like Soft Actor-Critic—managing infinite action spaces with Gaussian distributions, curing optimism with Twin Critics, building physical robustness by actively rewarding chaos, and stabilizing learning with liquid memory updates.

This is the sixth of 12 interactive RL projects I am building to bridge the gap between academic math and real-world intuition. If this deep dive into continuous control helped clarify how real robots think, I would be incredibly grateful if you checked out the source code and dropped a star on the full repository:

⭐ Reinforcement Learning Portfolio on GitHub

Dash10107 / rl-portfolio

End-to-end reinforcement learning projects — Q-Learning, DQN, PPO, SAC, A2C, IPPO, MBRL, HMM, RLHF, and Multi-Armed Bandits — each deployed as an interactive Gradio app on Hugging Face Spaces.

Reinforcement Learning Portfolio

A collection of 12 end-to-end reinforcement learning projects, each deployed as an interactive web application on Hugging Face Spaces. The projects span the full range of modern RL — from the simplest tabular methods that fit on a single page, to multi-agent coordination, model-based planning, and learning from human feedback.

Every project is built to be understood by someone who is new to RL. Each has its own README explaining the algorithm, the environment, and what you are looking at when you run it.

New to reinforcement learning? Start with these two documents before anything else:

CONCEPTS.md — what RL is, the core vocabulary, and how all 12 algorithms relate to each other
GETTING_STARTED.md — step-by-step guide to running your first project and your first experiment

Key Highlights

⚡ Zero-Install Interactive Demos: Every project is deployed live on Hugging Face Spaces for instant testing.
🎓 Curriculum-Based…

View on GitHub

Let me know in the comments: If you had to write the reward function for a self-driving car, what penalty would you assign to a bumpy brake versus a slow arrival?

The Invisible Hand: Teaching Warehouse Robots Teamwork with Multi-Agent PPO

Daksh Jain — Wed, 24 Jun 2026 03:31:00 +0000

If you look inside a massive Amazon fulfillment center, you will see thousands of small orange robots scurrying across the floor, carrying shelves of inventory. They weave past each other in narrow corridors, rarely colliding, and somehow coordinate to process thousands of orders per hour.

How is this coordination actually programmed?

You might assume there is a massive "Central Brain" computer that calculates the path for every single robot simultaneously. But in computer science, we know that is mathematically impossible. Because of the Curse of Dimensionality, the number of possible state combinations for 1,000 robots is larger than the number of atoms in the universe. A central brain would instantly freeze trying to calculate the math.

Instead, the robots must be decentralized. They must think for themselves. But if every robot is acting selfishly to finish its own task, how do you prevent them from causing massive traffic jams?

To answer this, I built the MARL Warehouse Coordinator. In this simulation, we drop a fleet of AI agents into a grid and force them to learn teamwork using Multi-Agent Reinforcement Learning (MARL).

1. The Nightmare of Non-Stationarity

In the previous projects in this portfolio, we trained a single agent interacting with an environment that had fixed rules.

Multi-Agent RL is fundamentally different, and infinitely harder.

Imagine trying to learn how to play chess, but every time you make a move, your opponent also learns and changes their strategy. What was a "good move" yesterday is suddenly a "terrible move" today. In RL, this is known as the Non-Stationarity Problem.

When multiple robots are training in the same warehouse, the "environment" isn't just the walls and packages. The environment is the other robots. Standard Reinforcement Learning algorithms completely break down in non-stationary environments because the mathematical targets are constantly shifting.

To solve this, we rely on one of the most robust algorithms in modern AI: Proximal Policy Optimization (PPO).

2. Independent PPO (IPPO) and the "Proximal" Trick

The simplest way to build a Multi-Agent system is to treat it like a single-agent system. We give every single robot its own independent brain (Neural Network), and let them all train at the same time. This is called Independent Learning.

But as we discussed, independent learning usually causes the math to explode due to non-stationarity. To keep the training stable, we use PPO.

PPO is the exact same underlying algorithm that OpenAI used to fine-tune ChatGPT. It is famous for a mathematical trick called the Clipped Surrogate Objective (the "Proximal" part of PPO).

In Machine Learning, when a neural network discovers a good action, it adjusts its weights to take that action more often. But sometimes, the math calculates a gradient step that is so massive it completely destroys the network's previously learned knowledge (a phenomenon called Policy Collapse).

PPO prevents this by strictly clipping the update ratio. Here is exactly what the "Proximal" math looks like in PyTorch:

import torch

# 1. Calculate how much the robot's brain has changed since the last update
ratio = torch.exp(new_log_prob - old_log_prob)

# 2. Calculate the raw, unbounded gradient
surrogate_1 = ratio * advantage

# 3. Clip the ratio to enforce a strict speed limit (e.g., epsilon = 0.2 means +/- 20%)
surrogate_2 = torch.clamp(ratio, 1.0 - epsilon, 1.0 + epsilon) * advantage

# 4. The math physically prevents massive, destructive updates
actor_loss = -torch.min(surrogate_1, surrogate_2).mean()

By enforcing this strict "speed limit" on learning (torch.clamp), PPO ensures that the robot's policy only changes in small, stable increments. This incredible stability is exactly what allows multiple independent robots to train in the same warehouse without their math exploding into chaos.

3. The Dual Brain: Actor-Critic Architecture

When you train a robot with PPO, you aren't actually training one Neural Network. You are training two. PPO uses an Actor-Critic Architecture.

The Actor: This is the network that actually drives the robot. It looks at the grid and outputs probabilities: "I am 80% sure I should move Left, and 20% sure I should move Up."
The Critic: This network acts as the coach. It doesn't make decisions. Instead, it looks at the grid and outputs a single number predicting how good the current situation is: "I estimate being in this corridor is worth 5 points."

How do they work together? Through a concept called Advantage.

Let's say the Critic predicts the current state is worth 5 points. The Actor decides to move Left, picks up a package, and the robot actually earns 8 points.

The math calculates the Advantage: 8 actual points - 5 expected points = +3 Advantage.
The Critic tells the Actor: "Wow! Moving Left was 3 points better than I expected! Update your weights to do that more often."

Generalized Advantage Estimation (GAE)

If a robot wanders aimlessly for 10 moves, accidentally nudges a package, and earns 8 points, how does the Critic know which of the 10 moves was the "good" one?

In our code, we solve this temporal credit assignment problem using a highly advanced technique called GAE (Generalized Advantage Estimation).

GAE calculates a mathematical "Error" (Delta) for every single step:
Delta = Immediate Reward + (Gamma * Next Expected Value) - Current Expected Value

It then uses a smoothing parameter (Lambda, or λ) to exponentially decay that error backwards through time. Here is the exact logic we use to calculate the Advantage for any given step t:

# The immediate error of the current move
delta_t = reward_t + (gamma * value_next) - value_current

# Blend it with the exponentially decayed errors of all future moves
advantage_t = delta_t + (gamma * lambda * advantage_next)

By perfectly blending short-term immediate rewards with long-term future predictions, GAE allows the Critic to coach the Actor with incredible precision.

4. Engineering the State: Partial Observability

If a central brain can't compute the whole warehouse, how do the independent robots do it?

We use a trick called Partial Observability (or Local Sensing). Instead of feeding the robot a massive 2D image of the entire 100x100 warehouse, we only give the robot a tiny, 13-dimensional array of numbers:

Its own coordinates.
The coordinates of its current package.
The relative coordinates of its 4 nearest neighbors.

By blinding the robot to the rest of the warehouse, the Neural Network only needs 128 hidden neurons. It trains in seconds instead of days. The robot learns to navigate locally, trusting that its localized decisions will result in global efficiency.

5. Entropy & The Invisible Hand of Shared Rewards

If every robot has an independent, localized brain, how do they learn teamwork? How do they know to yield to each other in narrow corridors instead of fighting for space?

They learn through the "Invisible Hand" of economics—specifically, Shared Rewards.

+2.0 points when anyone delivers a package.
-0.3 points if you collide with another robot.
-0.01 points for every step taken.

To ensure the robots actually discover this teamwork, the overall PPO algorithm balances three massive mathematical forces in a single equation:

# The Final PPO Objective Function
total_loss = actor_loss + (0.5 * critic_loss) - (0.01 * entropy)

Notice that we actively subtract the Entropy term from the loss, mathematically rewarding the AI for being chaotic. If we didn't do this, a robot might find one mediocre path and stubbornly stick to it forever. By rewarding chaos (0.01 * entropy) early on, the robots explore the entire warehouse until they discover the optimal choreography.

They learn to claim different pickup zones, and they instinctively wait at intersections to let other robots pass, because avoiding the -0.3 collision penalty results in a higher net score for their independent brains.

🧪 Try It Yourself

To see this multi-agent choreography in action, open the MARL Warehouse Simulator and run these visual experiments:

The Traffic Jam (Greedy Baseline): Go to the Simulation tab. Select the Greedy strategy. Greedy robots are programmed with a simple heuristic: "Always walk directly toward the package." Click Run Simulation. You will watch them instantly cluster together, blocking each other in narrow corridors and creating massive traffic jams because they have no awareness of their teammates.
The Choreography (IPPO Agent): Switch the strategy to IPPO (trained). Run the simulation again. Watch how the robots smoothly weave around each other. Notice how they naturally spread out to different sectors of the warehouse to avoid getting in each other's way.
Train the Swarm: Go to the Training tab. Set the number of robots to 4, and start the training. Watch the live "Collisions per Episode" chart trend downwards as the Critic networks slowly teach the Actor networks how to navigate the non-stationary chaos of their peers.

Wrapping Up

Multi-Agent Reinforcement Learning (MARL) is the frontier of modern AI. Moving from a single agent in a static world to multiple agents in a dynamic world requires robust algorithms like PPO and clever reward shaping. By letting independent brains learn through shared economics, we can create complex, decentralized teamwork that would be impossible to hard-code.

This is the fifth of 12 interactive RL projects I am building to bridge the gap between academic math and real-world intuition. If this breakdown of PPO and Actor-Critic architecture helped things click for you, I would be incredibly grateful if you checked out the source code and dropped a star on the full repository:

⭐ Reinforcement Learning Portfolio on GitHub

Dash10107 / rl-portfolio

End-to-end reinforcement learning projects — Q-Learning, DQN, PPO, SAC, A2C, IPPO, MBRL, HMM, RLHF, and Multi-Armed Bandits — each deployed as an interactive Gradio app on Hugging Face Spaces.

Reinforcement Learning Portfolio

Every project is built to be understood by someone who is new to RL. Each has its own README explaining the algorithm, the environment, and what you are looking at when you run it.

New to reinforcement learning? Start with these two documents before anything else:

CONCEPTS.md — what RL is, the core vocabulary, and how all 12 algorithms relate to each other
GETTING_STARTED.md — step-by-step guide to running your first project and your first experiment

Key Highlights

⚡ Zero-Install Interactive Demos: Every project is deployed live on Hugging Face Spaces for instant testing.
🎓 Curriculum-Based…

View on GitHub

Let me know in the comments: What other real-world systems (like traffic lights or stock trading) do you think could be optimized using independent multi-agent AI?

Energy Arbitrage: How AI Trades Electricity Using Reinforcement Learning

Daksh Jain — Tue, 23 Jun 2026 03:31:00 +0000

One of the most fascinating challenges in the modern energy sector is that the price of electricity changes every single hour.

At 3:00 AM, when wind turbines are spinning but everyone is asleep, power is incredibly cheap—sometimes dropping to 6 cents per kilowatt-hour. But at 6:00 PM, when the sun goes down and everyone turns on their ovens and air conditioners, the grid undergoes immense stress. Prices can violently spike to 60 cents or even $2.50 per kilowatt-hour during a crisis.

If you own a massive industrial battery system, this volatility presents a massive opportunity known as Energy Arbitrage. You buy electricity when it is cheap, store it, and sell it back to the grid (or use it to power your building) when prices spike.

But there is a catch: You do not know the future.

To explore how Artificial Intelligence solves this, I built the Smart Grid Energy Optimizer. In this interactive simulation, we pit a Deep Reinforcement Learning agent against simple human heuristics and a mathematically perfect algorithm to see how well AI can trade energy under uncertainty.

1. The Heuristic Fallacy

If you asked a software engineer to write a script to manage this battery, they would likely write a simple Rule-Based Heuristic. In code, it looks something like this:

def heuristic_trader(current_price, daily_average_price):
    if current_price < (daily_average_price * 0.75):
        return "CHARGE"
    elif current_price > (daily_average_price * 1.25):
        return "DISCHARGE"
    else:
        return "IDLE"

This works reasonably well, but it is deeply flawed because it lacks nuance. A static rule doesn't know if a small price spike at 2:00 PM is the peak of the day, or if an absolutely massive spike is coming at 6:00 PM. Furthermore, if you add solar panels to the roof of your building, the logic becomes overwhelmingly complex. If the sun is shining, should you use that free solar energy to charge the battery, or should you push it directly into the building to offset demand?

Hard-coding rules for every possible combination of price, time, solar generation, and building load is a nightmare.

2. The Theoretical Ceiling: Dynamic Programming

Before we train an AI to solve this, we need a baseline. How do we know if our AI is actually doing a good job? We need to calculate the mathematical upper bound of the problem.

To do this, we use an elegant computer science technique called Dynamic Programming (DP).

If we assume that we have perfect foresight—meaning we magically know the exact electricity price, solar output, and building load for every hour of the upcoming day—we can use DP to calculate the absolute optimal charging schedule.

The trick to Dynamic Programming is Backward Induction via the Bellman Equation:

V(s) = max_a [ Reward(s, a) + V(next_s) ]

We don't start at 12:00 AM. We start at the very end of the day (Hour 24) and work backwards, calculating the exact value of every possible state:

def solve_dp(env, n_soc_levels):
    # Create a grid of all possible battery charge levels
    soc_grid = np.linspace(0, env.cap, n_soc_levels)
    V = np.zeros((25, n_soc_levels))

    # Work backwards from the end of the day to the beginning
    for t in reversed(range(24)):
        for si, soc in enumerate(soc_grid):
            best_val = -1e9

            # Test every possible action (-3kW to +3kW)
            for action in possible_actions:
                reward, next_soc = simulate_action(action, t, soc)

                # The total value is the immediate reward + the known future value
                total = reward + V[t + 1, get_index(next_soc)]

                if total > best_val:
                    best_val = total

            V[t, si] = best_val

    return V # Returns the absolute maximum possible profit

Because DP explores every possible state starting from the end, it guarantees the perfect schedule.

The Curse of Dimensionality

If DP is mathematically perfect, why don't we use it for everything?

The answer is The Curse of Dimensionality. In our simulation, we only have 50 battery charge levels and 24 hours. The DP solver checks $50 \times 24 \times 7 \text{ actions} = 8,400$ combinations. That takes milliseconds. But imagine managing a factory with 10 independent batteries, 100 machines, and stochastic weather predictions. The number of state combinations explodes into the trillions. DP completely breaks down because the math takes years to compute.

Furthermore, DP requires perfect clairvoyance. In the real world, you cannot predict the exact solar output 12 hours in advance. DP is physically impossible to run in real-time, but it gives us a beautiful "Theoretical Ceiling" to grade our AI against.

3. Acting Under Uncertainty: Deep Q-Networks

In the real world, we must make decisions right now based only on what we currently know. This is where Deep Q-Networks (DQN) shine.

Unlike the DP solver, the DQN agent does not get to see the future. Instead, at every hour, it receives an 8-dimensional observation snapshot of the grid. It passes this snapshot through a neural network, which outputs the "Q-Value" (expected future profit) for 7 discrete actions: [-3kW, -2kW, -1kW, 0kW, +1kW, +2kW, +3kW].

To learn, the DQN essentially tries to approximate the exact same Bellman Equation that the DP solver uses, but without knowing the future. Here is what the core learning loop looks like in PyTorch:

import torch
import torch.nn.functional as F

# 1. Ask the neural network what the current state is worth
current_q_values = q_network(state)
q_value_of_action_taken = current_q_values.gather(1, action)

# 2. Ask the Target Network to predict the value of the NEXT state
with torch.no_grad():
    next_q_values = target_network(next_state)
    max_next_q_value = next_q_values.max(1)[0]

# 3. Calculate the Bellman Target (Immediate Reward + Future Value)
expected_q_value = reward + (gamma * max_next_q_value)

# 4. Update the neural network to minimize the mathematical error
loss = F.mse_loss(q_value_of_action_taken, expected_q_value)

Feature Engineering: The Cyclic Time Trick

One of the most fascinating engineering challenges in RL is how you represent the "State" to the neural network.

For example, how do you tell the AI what time it is? If you feed the network the raw integer hour = 23 (11 PM), the next step will be hour = 0 (Midnight). To a neural network, jumping from 23 to 0 looks like a massive, disruptive mathematical anomaly.

To fix this, we use Cyclic Encoding. We map the 24-hour clock onto a circle using sine and cosine functions:

sin_time = np.sin(hour * np.pi / 12)
cos_time = np.cos(hour * np.pi / 12)

Now, Hour 23 and Hour 0 are mathematically right next to each other on the circle. The network smoothly understands the passage of time without any jarring jumps.

Thermodynamics: Learning the "Spread"

The environment enforces a physical reality: battery systems have a 92% round-trip efficiency. If you put 1kW into the battery, you lose 8% to heat, and only get 0.92kW out.

Because of this efficiency loss, buying at 10 cents and selling at 10.5 cents actually loses money. You have to write zero code telling the AI about thermodynamics. The DQN naturally figures out that it must only execute trades when the price "spread" is wide enough to cover the 8% efficiency tax.

4. Preventing Overfitting via Noise Injection

If you train a neural network on the exact same 24-hour price curve for a million episodes, it doesn't actually become intelligent. It just becomes a clock. It memorizes "charge at step 4, discharge at step 16".

To prevent this, the environment uses Stochastic Training.
During training, every single price is perturbed with 12% Gaussian noise, and the solar generation is multiplied by random "cloud factors".

Because the prices are never the same twice, the AI cannot rely on the clock. It is forced to learn the causal relationship between its inputs (Price Trend, Solar Output, Time to Peak) and the reward. This forces the DQN to learn a robust, generalized trading strategy that can survive the chaos of real-world markets.

5. Measuring the "Intelligence Gap"

In the interactive dashboard, we can run a Benchmark that races three strategies on the exact same day:

The Rule-Based Heuristic (The human attempt)
The DQN Agent (The AI acting under uncertainty)
The DP Solver (The theoretical maximum with perfect foresight)

When you look at the final cumulative revenue chart, the DP Solver is always at the top. The Rule-Based system is usually at the bottom. The DQN agent sits in the middle.

The financial gap between the DQN and the DP Solver is literally the mathematical measurement of how much the AI has left to learn, combined with the unavoidable cost of not knowing the future. It is a stunning visual representation of AI performance.

🧪 Try It Yourself

To truly understand how this works, open up the Smart Grid Simulator and run these experiments:

The Benchmark Race: Go to the Benchmark tab. Select the Summer Peak scenario. Click Run Benchmark. Look at the bar chart comparing the total revenue. How much money did the DQN leave on the table compared to the DP optimal?
The Solar Impact: Go to the Dispatch tab. Run the DQN on the Summer Peak scenario (which has a 5kW solar array). Watch how it behaves at 1:00 PM. Then switch the scenario to No Solar and run it again. You will see the agent completely change its strategy, forced to buy expensive grid power earlier in the day because it no longer has free solar power to rely on.
Train the Brain: Go to the Training Lab. Set the steps to 20,000 and watch the live reward curve climb as the agent learns to ignore the immediate cost of charging the battery in favor of the massive delayed gratification of the 6:00 PM discharge.

Wrapping Up

Energy arbitrage perfectly encapsulates the beauty of Reinforcement Learning. It forces an AI to balance short-term costs against long-term gains in a highly volatile, unpredictable environment. By comparing the AI against the mathematical perfection of Dynamic Programming, we stop guessing if our algorithm is "good" and instead mathematically measure its intuition.

This is the fourth of 12 interactive RL projects I am building to bridge the gap between academic math and real-world intuition. If this deep dive helped you understand how AI manages uncertainty, I would be incredibly grateful if you checked out the source code and dropped a star on the full repository:

⭐ Reinforcement Learning Portfolio on GitHub

Dash10107 / rl-portfolio

End-to-end reinforcement learning projects — Q-Learning, DQN, PPO, SAC, A2C, IPPO, MBRL, HMM, RLHF, and Multi-Armed Bandits — each deployed as an interactive Gradio app on Hugging Face Spaces.

Reinforcement Learning Portfolio

Every project is built to be understood by someone who is new to RL. Each has its own README explaining the algorithm, the environment, and what you are looking at when you run it.

New to reinforcement learning? Start with these two documents before anything else:

CONCEPTS.md — what RL is, the core vocabulary, and how all 12 algorithms relate to each other
GETTING_STARTED.md — step-by-step guide to running your first project and your first experiment

Key Highlights

⚡ Zero-Install Interactive Demos: Every project is deployed live on Hugging Face Spaces for instant testing.
🎓 Curriculum-Based…

View on GitHub

Let me know in the comments: If you had a giant battery in your garage, would you trust an AI to trade electricity for you while you slept?

Planning vs Reacting: Why A* Search Fails in Traffic, and How Deep Q-Networks Fix It

Daksh Jain — Mon, 22 Jun 2026 14:30:00 +0000

Imagine you are a delivery driver navigating a busy city. Your goal is to get from Point A to Point B. If you open a standard navigation app, it will likely use a variation of a classical search algorithm—like Dijkstra's or A* (A-Star)—to draw a line showing the absolute shortest path.

But any experienced driver knows a fundamental truth about city driving: the shortest path is rarely the cheapest path.

In modern logistics, companies aren't just optimizing for distance; they are optimizing for fuel consumption and carbon emissions. Heavy traffic zones multiply a vehicle's carbon output (and fuel cost) by a factor of four. An experienced human driver intuitively learns to take a slightly longer, winding route to completely bypass a congested downtown core.

To teach an AI to develop this exact same intuition, I built a Green Logistics Optimizer. In this interactive simulation, we pit a classical mathematical planner (A*) against a Deep Reinforcement Learning agent (DQN).

In this article, we're going to dive into the underlying theory of why classical planning algorithms eventually hit a wall in the real world, and how Neural Networks solve the problem by learning to react rather than plan.

1. The Flaw of the Perfect Planner (A*)

Let's start with the baseline. The A* algorithm is a masterpiece of computer science. It guarantees finding the optimal path between two points by using a "heuristic"—an educated guess of the remaining distance.

In our Python implementation, A* evaluates the grid by adding the distance traveled so far (g) to the estimated distance to the goal (h). It uses a Priority Queue (a heap) to always explore the most promising path first:

def _astar_path(env: GreenCityEnv):
    # A* minimises step count (distance), completely ignoring carbon.
    start = tuple(env.agent_pos.tolist())
    goal  = tuple(env.goal.tolist())

    # The Heuristic: Manhattan distance to the goal
    def h(p):
        return abs(p[0] - goal[0]) + abs(p[1] - goal[1])

    open_heap = [(h(start), 0, start, [start])]
    visited = set()

    while open_heap:
        _, g, cur, path = heapq.heappop(open_heap)

        if cur == goal: return path
        # ... explore neighbors, add to heap, repeat ...

If you look at this code, you'll notice it perfectly calculates the shortest physical distance.

So, what's the problem?
The problem is that A* is a planner. It requires you to know the entire map, and the exact cost of every street, in advance. If you try to modify A* to include "traffic costs" instead of just distance, it works fine—until the traffic changes.

If a traffic light turns red, or an accident occurs, the weights of your map change. A* has to throw away its entire planned route and recalculate the mathematical tree from scratch. In a massive city grid with millions of nodes and dynamically shifting traffic, constantly recalculating A* becomes computationally paralyzing.

We don't want an AI that mathematically calculates a billion possibilities every time a car brakes. We want an AI that looks at the traffic and just knows what to do.

2. The Shift: From Planning to Reacting

This is where Reinforcement Learning enters the picture. Instead of writing an algorithm that searches a map, we drop an agent into the city and let it drive around millions of times.

We use a Deep Q-Network (DQN).
Unlike A*, a DQN doesn't plan a route from start to finish. Instead, it looks at a snapshot of the current state (its location and the traffic around it) and outputs a "Q-Value" for all possible immediate actions (Up, Down, Left, Right).

The Q-Value represents the expected future reward of taking that action, mathematically defined by the Bellman Optimality Equation:

Q(s, a) = Reward(s, a) + γ * max Q(s_next, a_all)

Because the environment is too massive for a simple lookup table, we use a Neural Network as a mathematical function approximator to estimate these Q-Values. In PyTorch, the "Brain" of the vehicle looks like this:

import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, state_dim, action_dim):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.ReLU(),
            nn.Linear(128, action_dim) # Outputs 4 Q-Values (Up, Down, Left, Right)
        )

    def forward(self, state):
        return self.network(state)

The agent simply passes its current state into this network, gets the four numbers, and picks the highest one. It's a purely reactive system.

def run_dqn(env: GreenCityEnv) -> RouteResult:
    # Load the trained Neural Network
    model = DQN.load("green_dqn_model", env=env)

    def _dqn_action(pos, env):
        # Look at the current state, instantly predict the best move
        action, _ = model.predict(pos, deterministic=True)
        return int(action)

    return _rollout(env, _dqn_action)

Notice there is no while loop searching through a map. There is just a single model.predict() call. Because the neural network has already "compiled" the knowledge of the city into its weights during training, querying it takes milliseconds.

But training a neural network to do this requires overcoming two fascinating theoretical hurdles.

3. Overcoming Catastrophic Forgetting (Experience Replay)

Neural Networks have a fatal flaw when used in Reinforcement Learning: Catastrophic Forgetting.

Imagine our delivery driver gets stuck driving around a heavy congestion zone for 500 consecutive steps. The neural network is constantly updating its weights based on this high-traffic data. Because neural networks generalize, the math that adjusts the weights for the traffic zone will aggressively overwrite the weights that control how the car drives on an open, empty highway. By the time the agent escapes the traffic, it has literally "forgotten" how to drive in an empty street!

The Solution: Experience Replay

To solve this, DQN introduced a brilliant concept called Experience Replay.

Instead of training the neural network on the exact sequence of events as they happen, the agent takes every single step it makes and throws it into a massive database called a Replay Buffer.

When it's time to train, the network doesn't look at what just happened. Instead, it reaches into the Replay Buffer and pulls out a completely randomized mini-batch of past experiences:

import random

class ReplayBuffer:
    def __init__(self, capacity=100000):
        self.memory = []

    def store_memory(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    def dream(self, batch_size=64):
        # Pull a randomized batch of 64 memories to break correlation
        return random.sample(self.memory, batch_size)

It might pull one memory from a traffic jam, one memory from an empty highway, and one memory of reaching the goal. By shuffling its memories (random.sample), the neural network mathematically breaks the correlation between consecutive steps. It is the artificial equivalent of human REM sleep—dreaming and shuffling past experiences to consolidate generalized, permanent knowledge.

4. Chasing a Moving Target (Target Networks)

If you've ever trained a standard image classifier, you know the data has fixed labels. An image of a cat is always a cat. The network predicts "Dog", calculates the error against the fixed label "Cat", and updates.

But in Deep Q-Networks, the network is trying to predict a Q-Value, and the "correct label" (the Target) is generated by the Bellman Equation we discussed earlier.

Do you see the paradox? If we use a single neural network, the Target is being generated by the exact same neural network that we are currently updating! If the network adjusts its weights to increase the value of moving "Up", it accidentally changes the calculated target for the next step too. It is like a dog violently chasing its own tail. The network weights will oscillate wildly and mathematically explode.

The Solution: Dual Networks

To stabilize the math, we maintain two identical Neural Networks in PyTorch:

The Policy Network: Actively driving the car and updating its weights every step.
The Target Network: A completely frozen clone of the Policy Network.

When calculating the "correct label", we ask the frozen Target Network:

# 1. Ask the FROZEN Target Network for the next state's value
with torch.no_grad():
    target_next_q = target_network(next_state).max(1)[0]

# 2. Calculate the perfectly stable target label
target_label = reward + (gamma * target_next_q * (1 - done))

# 3. Train the ACTIVE Policy Network against this frozen label
current_q = policy_network(state).gather(1, action)
loss = F.mse_loss(current_q, target_label)

Because the Target Network is frozen (torch.no_grad()), the mathematical anchor stays perfectly still, allowing the Policy Network to steadily learn. Every few hundred steps, we take the weights from the Policy Network and explicitly copy them over to the Target Network (target_network.load_state_dict(policy_network.state_dict())). This simple trick is the theoretical bedrock that makes Deep RL stable.

5. The Power of the Reward Function

When you put all of this together, something magical happens. The agent's behavior is dictated entirely by a tiny reward function. In our environment:

Diesel Truck: Base Cost = 1.0. Congestion Multiplier = 4.0.
Electric Vehicle (EV): Base Cost = 0.2. Congestion Multiplier = 4.0.

A diesel truck entering congestion suffers a massive -4.0 reward penalty per step. An EV suffers a -0.8 penalty.

When you train the DQN, you don't have to write any complex if/else statements telling the Diesel truck to avoid traffic. The math does it automatically. The Diesel agent will learn to take a massive detour around the city to avoid traffic, while the EV agent might decide it's mathematically cheaper to just cut straight through the congestion because its base emissions are so low.

🧪 Try It Yourself

To truly understand the difference between planning and reacting, you have to see the visual traces. Open up the Green Logistics Simulator and run these experiments:

The Distance vs Carbon Gap: Pick the Downtown Rush (7x7) scenario. Select both A* and DQN. Click Deploy Fleet. You will see the A* path cut straight through the red congestion zone (because it is the shortest path). But look at the DQN path—it curves widely around the red zone to save carbon.
Check the Analytics: Go to the Analytics tab. Look at the Carbon Trace. A* reaches the goal in fewer steps, but its cumulative carbon spikes violently. DQN takes more steps, but its carbon line stays flat and low.
Train Your Own Brain: Go to the Training Lab. Set the slider to 10,000 steps and click Train. You can watch the live reward curve rise as the network populates its Replay Buffer and slowly stabilizes its Target Network.

Wrapping Up

Classical algorithms like A* are beautiful, but they require perfect, static knowledge of the world to plan ahead. By utilizing Deep Q-Networks, Experience Replay, and Target Networks, we can teach AI to simply look at a chaotic, shifting environment and intuitively react.

This is the third of 12 interactive RL projects I am building to bridge the gap between academic math and real-world intuition. If this deep dive helped clarify the theory inside Neural Networks, I would be incredibly grateful if you checked out the source code and dropped a star on the full repository:

⭐ Reinforcement Learning Portfolio on GitHub

Dash10107 / rl-portfolio

End-to-end reinforcement learning projects — Q-Learning, DQN, PPO, SAC, A2C, IPPO, MBRL, HMM, RLHF, and Multi-Armed Bandits — each deployed as an interactive Gradio app on Hugging Face Spaces.

Reinforcement Learning Portfolio

Every project is built to be understood by someone who is new to RL. Each has its own README explaining the algorithm, the environment, and what you are looking at when you run it.

New to reinforcement learning? Start with these two documents before anything else:

CONCEPTS.md — what RL is, the core vocabulary, and how all 12 algorithms relate to each other
GETTING_STARTED.md — step-by-step guide to running your first project and your first experiment

Key Highlights

⚡ Zero-Install Interactive Demos: Every project is deployed live on Hugging Face Spaces for instant testing.
🎓 Curriculum-Based…

View on GitHub

Let me know in the comments: Which feels more "intelligent" to you—a mathematical algorithm that plans perfectly, or a neural network that guesses intuitively?

Building an AI that Solves Mazes from Scratch (Q-Learning, SARSA & Monte Carlo)

Daksh Jain — Sun, 21 Jun 2026 14:30:00 +0000

When I first started studying Reinforcement Learning (RL), I spent weeks drowning in dense academic papers. I was tired of staring at Greek letters ($\gamma$, $\theta$, $\tau$) and abstract equations without actually seeing what they meant. I didn't just want to memorize the Bellman equation; I wanted to touch the knobs, tweak the parameters, and watch the math come to life.

So, I decided to build something highly visual: a maze solver.

The problem is classic but surprisingly profound. If you drop an AI agent into a completely dark maze, it has no map. It doesn't know where the walls are, and it certainly doesn't know where the exit is. It only knows its current location.

To escape, it has to explore, make mistakes, remember what it learned, and gradually build a strategy. In this article, we'll walk through exactly how to build this from scratch using three fundamental algorithms: Q-Learning, SARSA, and Monte Carlo.

If you want to play with the final result while you read, I've hosted the interactive playground here:
👉 Live Interactive Demo: RL Maze Solver

1. Setting the Rules of the World

Before we write any algorithms, we have to define the environment in terms the AI can understand. In RL, this boils down to three things:

State: The agent's current location. We flatten the grid, so if the agent is in cell (row 2, col 3) on a 5x5 grid, its state is just an integer index.
Action: The agent can move Up, Down, Left, or Right.
Reward: This is how we communicate our goal to the AI.
- Step on open floor: -1 (We want it to find the exit fast).
- Bump into a wall: -5 (Painful. Don't do this).
- Reach the goal: +100 (Success!).

The heavy -5 wall penalty is the main signal. It forces the algorithms to actually plan clean routes rather than randomly bouncing off the walls to reach the end.

The Dilemma: Exploration vs. Exploitation (The Restaurant Analogy)

Before the AI can learn, it faces the oldest dilemma in Reinforcement Learning: Exploration vs. Exploitation.

Imagine moving to a new city. On your first night, you find a decent pizza place.

If you Exploit your knowledge, you will eat decent pizza every single night. You will never starve, but you will also never experience anything better.
If you Explore, you might suffer through a terrible salad, but you might also discover a 5-star steakhouse.

To solve this, our AI uses an Epsilon-Greedy strategy. It is the mathematical engine of curiosity.

import numpy as np

def choose_action(state, Q_table, epsilon):
    # EXPLORE: Roll a loaded die
    if np.random.uniform(0, 1) < epsilon:
        return np.random.choice(4) # Pick a completely random direction

    # EXPLOIT: Use the cheat sheet
    else:
        return np.argmax(Q_table[state]) # Pick the mathematically best direction

When training starts, epsilon is 1.0 (100% exploration). The AI stumbles blindly into walls. As it learns, epsilon decays to 0.01. It becomes a master of the maze, only exploring 1% of the time just in case there is a slightly faster shortcut it missed.

2. Q-Learning: The Optimistic Explorer

Q-Learning is arguably the most famous tabular RL algorithm. The "tabular" part means the agent literally maintains a massive cheat sheet—a table with one row for every cell in the maze and one column for every possible action.

Each entry in this table is a Q-value: a numerical score of how good the agent currently believes it is to take a specific action from a specific cell.

The magic of Q-Learning lies in its update rule. After every single step, the agent updates its cheat sheet using the Bellman Equation:

Q(s, a) ← Q(s, a) + α * [ R + γ * max Q(s', a') - Q(s, a) ]

In plain English, the math says: The value of my current move is the immediate reward I just got, plus the discounted value of the absolute best possible move from wherever I ended up next.

Here is the entire training loop in Python. Notice how short the core logic is:

def train_qlearning(env, episodes, alpha, gamma, decay):
    agent = TabularAgent(env.n_states, env.action_space.n, alpha, gamma)

    for _ in range(episodes):
        state, _ = env.reset()

        # Walk through the maze until we find the exit or time out
        for _ in range(env.n_states * 4):
            action = agent.choose_action(state) # Epsilon-greedy
            next_state, reward, done, _, _ = env.step(action)

            # The core Q-Learning update rule
            # Notice the np.max() - it bootstraps from the *best* possible next move
            td_target = reward + gamma * np.max(agent.Q[next_state]) * (1 - done)
            agent.Q[state, action] += alpha * (td_target - agent.Q[state, action])

            state = next_state
            if done: break

        agent.decay_epsilon(decay)

Because Q-Learning uses np.max(agent.Q[next_state]), it is highly optimistic. It updates its knowledge assuming it will always take the perfect path in the future. This is known as being off-policy.

3. SARSA: The Cautious Learner

But what if our agent's current strategy involves a lot of random exploration (which it does, thanks to our epsilon-greedy logic)?

If Q-Learning is exploring randomly, it might stumble into a wall and take a -5 penalty. But when it updates its Q-table, it shrugs it off, pretending it would have taken the optimal path instead.

SARSA (State-Action-Reward-State-Action) takes a different approach. Instead of updating toward the maximum possible future value, it updates toward the value of the action it actually chose next.

Here is the mathematical update rule for SARSA:

Q(s, a) ← Q(s, a) + α * [ R + γ * Q(s', a') - Q(s, a) ]

Notice the subtle difference from Q-Learning? There is no max operator. The code difference is literally one line, but the behavioral difference is massive:

# SARSA's on-policy TD update
# We don't assume the best possible next move. 
# We evaluate the action we are ACTUALLY going to take next (a').
td_target = reward + gamma * agent.Q[next_state, next_action] * (1 - done)

Because SARSA learns the value of the policy it is currently running (random mistakes included), it is much more cautious. If an agent randomly stumbles into walls while navigating a tight corridor, SARSA will learn to lower the value of that corridor and might prefer a wider, safer path. Q-learning will confidently march right next to the wall, assuming it won't make a mistake.

4. First-Visit Monte Carlo: Hindsight is 20/20

Both Q-Learning and SARSA update their cheat sheets after every single step. Monte Carlo methods take a completely different philosophical approach: Wait until the run is over.

Instead of bootstrapping values step-by-step, a Monte Carlo agent plays out a full episode to the very end, storing every (state, action, reward) in memory. Then, it works backward, calculating the actual cumulative return G and updating the values based on real outcomes:

G = 0
# Work backwards from the end of the episode to the start
for state, action, reward in reversed(episode_memory):
    G = reward + (gamma * G)

    # Update the Q-Table toward the actual experienced return (G)
    agent.Q[state, action] += alpha * (G - agent.Q[state, action])

The Pro: It is completely unbiased. It uses real experienced returns rather than mathematical estimates.
The Con: It has incredibly high variance. A single unlucky episode where the agent gets lost can temporarily ruin the values of perfectly good states. Furthermore, because it cannot update until the episode ends, it struggles massively in large mazes where initial episodes might take thousands of steps.

5. Visualizing the "Brain"

When building this project, the most satisfying part wasn't writing the code; it was rendering the State-Value Heatmaps.

If you look at the Q-table directly, it's just a giant matrix of floating-point numbers. But if you take the maximum Q-value for each cell and map it to a color gradient, you can literally watch the AI's brain light up.

During training, the cells right next to the goal turn bright yellow first. Then, slowly, episode by episode, that bright path creeps backward through the maze corridors toward the start line. You are watching the reward signal propagate through the environment. It's a beautiful transition from pure random fumbling to precise mathematical pathfinding.

🧪 Try It Yourself

The best way to understand these trade-offs is to see them in action. I built an interactive dashboard where you can race these algorithms against each other.

Open up the Live Playground and try these three experiments:

The Cautious Agent Test: Run both SARSA and Q-Learning on Medium (9x9) difficulty for 400 episodes. Look at the heatmaps for the cells directly adjacent to the walls. You will see Q-Learning assigns them much higher values than SARSA does!
The Size Limit: Run Q-Learning for 500 episodes on Tiny (5x5) and then on Large (13x13). You'll quickly see why exploration decay parameters are so critical as the state space grows.
The Algorithm Race: Go to the "Algorithm Race" tab, select a Large (13x13) DFS maze, and run all three algorithms simultaneously. Watch how Monte Carlo's convergence curve lags behind the TD methods because it has to wait for those massive initial episodes to finish.

Final Thoughts

The maze solver is a perfect microcosm of Reinforcement Learning. It's small enough to solve on your laptop in seconds, but complex enough to visually demonstrate the core tension in RL: exploration vs. exploitation, and on-policy vs. off-policy learning.

This is just the first of 12 interactive RL projects I'm building to demystify artificial intelligence. If you found this breakdown helpful, the highest compliment you can give is checking out the source code and dropping a star on the GitHub repository:

⭐ Reinforcement Learning Portfolio on GitHub

Dash10107 / rl-portfolio

End-to-end reinforcement learning projects — Q-Learning, DQN, PPO, SAC, A2C, IPPO, MBRL, HMM, RLHF, and Multi-Armed Bandits — each deployed as an interactive Gradio app on Hugging Face Spaces.

Reinforcement Learning Portfolio

Every project is built to be understood by someone who is new to RL. Each has its own README explaining the algorithm, the environment, and what you are looking at when you run it.

New to reinforcement learning? Start with these two documents before anything else:

CONCEPTS.md — what RL is, the core vocabulary, and how all 12 algorithms relate to each other
GETTING_STARTED.md — step-by-step guide to running your first project and your first experiment

Key Highlights

⚡ Zero-Install Interactive Demos: Every project is deployed live on Hugging Face Spaces for instant testing.
🎓 Curriculum-Based…

View on GitHub

Let me know in the comments: Have you ever tried to implement Q-learning? What was the hardest part to wrap your head around?

Beyond A/B Testing: How AI Handles Ad Fatigue and Revenue Optimization

Daksh Jain — Sun, 21 Jun 2026 03:48:23 +0000

If you read any standard tutorial on Multi-Armed Bandits, you will hear the exact same story: A/B testing is inefficient because it wastes 50% of your traffic on a losing variation. Instead, use a Bandit algorithm to dynamically shift traffic to the winner.

They usually introduce three algorithms: Epsilon-Greedy, UCB1, and Thompson Sampling.

But almost all of these tutorials make two fatal, mathematically dangerous assumptions that will completely break your algorithms in the real world:

They assume a click is just a click.
They assume the world never changes.

I built a custom Interactive Bandit Simulator to visualize exactly why these assumptions fail, and how advanced Reinforcement Learning actually handles the chaos of the real world.

The Casino Analogy (Where the name comes from)

Imagine walking into a casino and facing a row of slot machines (known as "One-Armed Bandits"). You know that some machines have a higher payout rate than others, but you don't know which ones.

Do you pull the lever on the machine that paid out $10 on your first try (Exploitation)?
Or do you put coins into the unknown machines just in case one of them is the secret jackpot machine (Exploration)?

This is the Multi-Armed Bandit problem. In digital marketing, the slot machines are your ad banners, and the pulls are your website visitors.

1. The Revenue Trap (CTR vs EV)

Most standard bandit implementations optimize purely for Click-Through Rate (CTR). A conversion equals 1, a failure equals 0.

Imagine you are running a SaaS pricing page with three different "Call to Action" (CTA) buttons.

"Start Free Trial": Gets a massive 15% CTR. (Expected lifetime value: $120)
"Request Demo": Gets a 7% CTR. (Expected value: $180)
"Buy Now": Gets a tiny 3% CTR. (Expected value: $300)

If your algorithm only looks at clicks, it will confidently route 100% of your traffic to the "Free Trial" button. It thinks it's winning, but you are actively bleeding potential revenue.

The Fix: Expected Value (EV)

To fix this, our environment must multiply the conversion by the actual revenue. In our simulator's environment code, the reward function looks like this:

def pull(self, arm_idx: int):
    arm = self.arms[arm_idx]

    # 1. Did they click based on the hidden True CTR?
    converted = bool(self.rng.random() < arm.true_ctr)

    # 2. Multiply by the actual monetary value of that conversion!
    reward = arm.revenue * converted

    return reward, converted

When you run Thompson Sampling on the "SaaS Pricing Page" scenario in the live dashboard, you will watch something incredible happen. Initially, the algorithm gets flooded with clicks for the "Free Trial" banner. But over time, the rare—but massive—$300 payouts from the "Buy Now" button cause its revenue-weighted probability distribution to shift all the way to the right. The AI learns to ignore the high click rate and chases the money.

2. Standard Algorithms and Their Deep Flaws

Let's look at how standard algorithms attempt to solve this exploration-exploitation trade-off, and where they break.

The Naive Explorer: Epsilon-Greedy

It rolls a loaded die. 90% of the time, it exploits the best banner. 10% of the time ($\epsilon$), it picks at random.
The Flaw: It never stops exploring. Even after 100,000 impressions when it is absolutely certain which banner is best, it still wastes 10% of its traffic on losers.

We can fix this mathematically with Decaying Epsilon-Greedy. Instead of a fixed 10%, we calculate epsilon dynamically:

ε = decay / √(t)

def choose(self) -> int:
    # Epsilon shrinks as the square root of time!
    eps = min(1.0, self.decay / max(1, np.sqrt(self.t + 1)))

    if np.random.random() < eps:
        return int(np.random.randint(self.n_arms))
    return int(np.argmax(self.values))

This forces heavy exploration early, but gracefully decays exploration to zero as time approaches infinity.

The Genius of the Logarithm: UCB1

Upper Confidence Bound (UCB1) doesn't use randomness. It mathematically calculates the maximum potential value of a banner using this formula:

UCB = Expected_Reward + c * √( ln(t) / pulls )

def choose(self) -> int:
    t = self.t + 1
    # The UCB bonus formula
    bonus = self.c * np.sqrt(np.log(t) / self.counts)
    return int(np.argmax(self.values + bonus))

Why is np.log(t) brilliant? As time (t) moves forward, the numerator grows. But a logarithm grows incredibly slowly. This guarantees that if a banner hasn't been pulled in a long time (the denominator self.counts stays small), its bonus will eventually creep high enough to force the algorithm to test it again. No banner is ever permanently starved of attention.

3. The Static World Fallacy (Ad Fatigue)

Here is the second, much larger trap. UCB1 and Epsilon-Greedy assume that if a banner has a 10% CTR on Day 1, it will have a 10% CTR on Day 100.

In digital marketing, this is completely false. Users get "Ad Fatigue". A brilliant new banner design will get high clicks for a week, and then slowly decay as users go blind to it. We call this a Non-Stationary Environment.

If you run standard UCB1 in a non-stationary environment, it fails spectacularly. Why? Because UCB1 remembers everything. If Banner A was amazing for the first 10,000 impressions, UCB1 builds an incredibly strong mathematical conviction that Banner A is the best. If Banner A suddenly goes blind and its CTR drops to zero, UCB1 is so weighed down by its historical data that it might take another 10,000 failed impressions before it finally changes its mind.

4. Advanced Solutions: Bayes and Gradients

How do modern AI systems handle shifting trends?

Solution A: The Bayesian Master (Thompson Sampling)

Instead of tracking a single "average", Thompson Sampling tracks a Beta Distribution of belief. It calculates the exact probability of a banner's true success rate using Bayes' Theorem:

P(True_CTR | Data) = Beta(α + clicks, β + ignores)

def choose(self) -> int:
    # Sample from the Beta distribution of each arm
    samples = np.random.beta(self.alpha, self.beta)
    return int(np.argmax(samples))

alpha is the number of successes.
beta is the number of failures.

If you open the Learner Mode in the dashboard, you can watch these Beta Distribution curves physically morph. When a banner is new, the curve is flat and wide (high uncertainty). When it gets clicks, it shifts right and becomes a tight spike.

If the environment drifts (Ad Fatigue), a once-great banner starts accumulating failures. Its beta parameter grows, the curve widens and shifts left, and the AI naturally begins exploring other banners again. It gracefully rides the changing waves.

Solution B: Gradient Bandits (The Deep RL Bridge)

Instead of trying to estimate CTR or Revenue at all, what if the agent just learns a relative preference?

Gradient Bandits maintain a preference score H for each banner. It passes these scores through a Softmax function to convert them into probabilities (just like a neural network). The update rule calculates the gradient of the reward against a rolling baseline:

H_chosen = H_chosen + α * (Reward - Baseline) * (1 - Prob_chosen)

def update(self, arm: int, reward: float):
    probs = self._softmax()

    # Stochastic gradient ascent on expected reward
    self.H -= self.alpha_lr * (reward - self.baseline) * probs
    self.H[arm] += self.alpha_lr * (reward - self.baseline)

    # Update running baseline
    self.baseline += (reward - self.baseline) / (self.t + 1)

If a banner performs better than the historical baseline, its preference gets a boost. If it performs worse, it gets penalized. This relative updating makes it incredibly robust to non-stationary environments. Fun fact: This exact math is the foundational stepping stone to Policy Gradient algorithms like PPO, which are used to train Large Language Models!

🧪 Try It Yourself

Don't just read about it. Open up the Live Dashboard and run these exact experiments to see the AI break and recover:

The Revenue Trap: Go to the Face-Off tab. Select the SaaS Pricing Page scenario. Race Epsilon-Greedy against Thompson Sampling. Watch how Thompson Sampling figures out that the lowest-clicked banner is actually the most profitable.
The Ad Fatigue Test: Go to Advanced Settings and set the CTR Drift to 0.008. Run UCB1 against Gradient Bandits. Watch how UCB1 stubbornly clings to early winners long after they have decayed, while Gradient Bandits smoothly adapt.
Step-by-Step Bayesian Learning: Go to the Learner Mode, pick the E-Commerce Sale scenario, and click "Next Impression" manually. Watch the mathematical confidence curves physically narrow in real time.

Wrapping Up

Bandit algorithms are the secret engine behind almost every digital platform you use today—from Netflix thumbnails to Amazon recommendations. But building them for the real world requires moving beyond simple coin-flips and addressing revenue weighting and non-stationarity.

This is the second of 12 interactive RL projects I am building to bridge the gap between academic math and real-world intuition. If this helped things click for you, I would be incredibly grateful if you checked out the source code and dropped a star on the full repository:

⭐ Reinforcement Learning Portfolio on GitHub

Dash10107 / rl-portfolio

End-to-end reinforcement learning projects — Q-Learning, DQN, PPO, SAC, A2C, IPPO, MBRL, HMM, RLHF, and Multi-Armed Bandits — each deployed as an interactive Gradio app on Hugging Face Spaces.

Reinforcement Learning Portfolio

Every project is built to be understood by someone who is new to RL. Each has its own README explaining the algorithm, the environment, and what you are looking at when you run it.

New to reinforcement learning? Start with these two documents before anything else:

CONCEPTS.md — what RL is, the core vocabulary, and how all 12 algorithms relate to each other
GETTING_STARTED.md — step-by-step guide to running your first project and your first experiment

Key Highlights

⚡ Zero-Install Interactive Demos: Every project is deployed live on Hugging Face Spaces for instant testing.
🎓 Curriculum-Based…

View on GitHub

Let me know in the comments: What's the weirdest A/B test result you've ever seen where the data totally contradicted your intuition?