DEV Community: Daniel Cooper

Optimising linear regression: A guide to applying gradient descent

Daniel Cooper — Sun, 26 Feb 2023 14:48:47 +0000

Linear regression is a statistical technique that helps us understand and predict relationships between variables. It is widely used in data science and machine learning.

At its core, linear regression involves finding the best-fitting line through a set of data points, allowing us to make predictions about future values based on past observations.

Finding the line can be a complex and computationally-intensive task, especially when dealing with large datasets.

Enter gradient descent... an optimisation algorithm that we can use to find the best-fitting line by iteratively adjusting its parameters.

It's worth noting that linear regression is a form of supervised learning where our algorithm is trained on "labelled" data. This simply reflects the fact that, within the available data, the inputs have corresponding outputs or labels.

In this blog, we will look at the maths, specifically the partial derivatives, involved in applying gradient descent to linear regression.

Model

Given a training set, our aim is to model the relationship so that our hypothesis $y^\hat{y}$ is a good predicator for the corresponding value of $y$ .

For simple or univariate linear regression, our model can be expressed as:

y^=θ0+θ1x1 \hat{y} = \theta_0 + \theta_1{x_1}

And for multivariate linear regression:

y^=θ0+θ1x1+θ2x2+...+θjxj \hat{y} = \theta_0 + \theta_1{x_1} + \theta_2{x_2} + ... + \theta_j{x_j}

Loss function

We can measure the accuracy of our hypothesis $y^\hat{y}$ by calculating the mean squared error - including a $12\frac{1}{2}$ to simplify the derivatives that follow later.

Loss,l=12m∑i=1m(y^−y)2 {Loss, l} = \frac{1}{2m}\sum_{i=1}^{m}(\hat{y} - y)^2

Gradient descent

We need to minimise the loss to ensure that $y^\hat{y}$ is a good predicator of $y$ .

Gradient descent is well described elsewhere but the principle is that we adjust the $θ\theta$ parameters until the loss is at the very bottom of the curve.

The adjustment is achieved by repeatedly taking the derivative of the loss function (the tangent line) to infer a movement to the right (gradient is negative) or the left (gradient is positive).

The size of each step is relative to a fixed hyper parameter $α\alpha$ which is called the learning rate. Typical values for $α\alpha$ are 0.1, 0.01 and 0.001.

This approach can be articulated as:

\theta_j = \theta_j - \alpha\dfrac{\mathrm{d}l}{\mathrm{d}\theta_j}

There are three types of gradient descent:

Batch gradient descent: Update the $θ\theta$ parameters using the entire training set; slow but guaranteed to converge to the global minimum
Stochastic gradient descent: Update the $θ\theta$ parameters using each training example; faster but may not converge to the global minimum due to randomness
Mini batch gradient descent: Update the $θ\theta$ parameters using a small batch of training examples; compromise between efficiency and convergence

Gradient descent derivatives

Imagine we have the following training data:

x1	x2	y
5	6	31.7
10	12	63.2
15	18	94.7
20	24	126.2
...	...	...
...	...	...
35	42	220.7

Our model will be:

y^=θ0+θ1x1+θ2x2 \hat{y} = \theta_0 + \theta_1{x_1} + \theta_2{x_2}

Let's say we want to apply stochastic gradient descent.

Here are the steps we need to follow:

Set $α\alpha$ to 0.001 (say) and $θ0\theta_0$ , $θ1\theta_1$ and $θ2\theta_2$ to random values between -1 and 1
Take the first training example and use gradient descent to update $θ0\theta_0$ , $θ1\theta_1$ and $θ2\theta_2$
Repeat step 2 for each of the remaining training examples
Repeat the process from step 2 another 1000 times

Let's look at the expressions and derivatives needed, starting with the loss.

When we consider just one training example at a time, as with stochastic gradient descent, $m = 1$ .

As such, the loss between our hypothesis $y^\hat{y}$ and $y$ can be described as:

Loss,l=12(y^−y)2 {Loss, l} = \frac{1}{2}(\hat{y} - y)^2

Our gradient descent equation is:

\theta_j = \theta_j - \alpha\dfrac{\mathrm{d}l}{\mathrm{d}\theta_j}

To apply it, we need to determine:

\dfrac{\mathrm{d}l}{\mathrm{d}\theta_j}

Using the chain rule, we can say that:

dldθj=dld(y^−y).d(y^−y)dθj \dfrac{\mathrm{d}l}{\mathrm{d}\theta_j} = \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)} . \dfrac{\mathrm{d}(\hat{y} - y)}{\mathrm{d}\theta_j}

One of the linearity rules of derivatives tells us that when differentiating the addition or subtraction of two functions, we can differentiate the functions individually and then add or subtract them.

This means we can say:

dldθj=dld(y^−y).[dy^dθj−dydθj] \dfrac{\mathrm{d}l}{\mathrm{d}\theta_j} = \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)} . \left[ \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_j}- \dfrac{\mathrm{d}y}{\mathrm{d}\theta_j} \right]

Since $y$ does not change with respect to $θ\theta$ :

\dfrac{\mathrm{d}y}{\mathrm{d}\theta_j} = 0

Therefore:

dldθj=dld(y^−y).dy^dθj \dfrac{\mathrm{d}l}{\mathrm{d}\theta_j} = \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)} . \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_j}

Let's now consider the two parts to this derivative, starting with the first:

dld(y^−y) \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)}

We can reduce this down:

dld(y^−y)=2.12(y^−y)=(y^−y) \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)} = 2.\frac{1}{2}(\hat{y} - y) = (\hat{y} - y)

Now let's consider the second part of the derivative:

dy^dθj \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_j}

We can this simplify this for $θ0\theta_0$ , $θ1\theta_1$ and $θ2\theta_2$ as follows:

dy^dθ0=d(θ0+θ1x1+θ2x2)dθ0=1 \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_0} = \dfrac{\mathrm{d}(\theta_0 + \theta_1{x_1} + \theta_2{x_2})}{\mathrm{d}\theta_0} = 1

dy^dθ1=d(θ0+θ1x1+θ2x2)dθ1=x1 \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_1} = \dfrac{\mathrm{d}(\theta_0 + \theta_1{x_1} + \theta_2{x_2})}{\mathrm{d}\theta_1} = x_1

dy^dθ2=d(θ0+θ1x1+θ2x2)dθ2=x2 \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_2} = \dfrac{\mathrm{d}(\theta_0 + \theta_1{x_1} + \theta_2{x_2})}{\mathrm{d}\theta_2} = x_2

Let's now combine what we've worked out.

We found that:

\theta_j = \theta_j - \alpha\dfrac{\mathrm{d}l}{\mathrm{d}\theta_j}

Where:

dldθj=dld(y^−y).dy^dθj \dfrac{\mathrm{d}l}{\mathrm{d}\theta_j} = \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)} . \dfrac{\mathrm{d}\hat{y}}{\mathrm{d}\theta_j}

And:

dld(y^−y)=(y^−y) \dfrac{\mathrm{d}l}{\mathrm{d}(\hat{y} - y)} = (\hat{y} - y)

Therefore we can say:

θ0=θ0−αdldθ0=θ0−α(y^−y) \theta_0 = \theta_0 - \alpha\dfrac{\mathrm{d}l}{\mathrm{d}\theta_0} = \theta_0 - \alpha(\hat{y} - y)

θ1=θ1−αdldθ1=θ1−α(y^−y)x1 \theta_1 = \theta_1 - \alpha\dfrac{\mathrm{d}l}{\mathrm{d}\theta_1} = \theta_1 - \alpha(\hat{y} - y)x_1

θ2=θ2−αdldθ2=θ2−α(y^−y)x2 \theta_2 = \theta_2 - \alpha\dfrac{\mathrm{d}l}{\mathrm{d}\theta_2} = \theta_2 - \alpha(\hat{y} - y)x_2

We now have all of the expressions needed to apply gradient descent. Over 1000 iterations, we might expect to see the total loss per iteration decrease like this.

In this post, we looked at the maths, specifically the partial derivatives, involved in applying gradient descent to linear regression.

For more information, check out Andrew Ng's Stanford CS229 lecture on Linear Regression and Gradient Descent.

Breaking the ice: How to solve OpenAI Frozen Lake with the cross-entropy method

Daniel Cooper — Sat, 04 Feb 2023 13:50:53 +0000

In this post, we will look at how to solve the famous Frozen Lake environment using a reinforcement learning (RL) method known as cross-entropy.

Frozen Lake is an OpenAI Gym environment in which an agent is rewarded for traversing a frozen surface from a start position to a goal position without falling through any perilous holes in the ice.

The environment is extremely simple and makes use of only discrete action and observation spaces, which we can evaluate using the following code:

import gym

env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False)

print(f"Action space: {env.action_space}")
print(f"Observation space: {env.observation_space}")

Action space: Discrete(4)
Observation space: Discrete(16)

The Discrete(4) action space indicates we can control our agent in one of four ways using integers between 0 and 3. The Frozen Lake documentation tells us that these integers correspond to the following actions:

0: LEFT
1: DOWN
2: RIGHT
3: UP

Similarly, the Discrete(16) observation space means there are 16 different states represented by integers between 0 and 15. From the documentation, the 16 states correspond to the agent's position on a map encoded using current_row * number_of_rows + current_column where both the row and column start at 0.

The state when the agent reaches the goal position in the bottom right hand corner of the 4x4 map is therefore calculated as: (3 * 4) + 3 = 15.

The reward structure is super simple: +1 for reaching the end goal and 0 for any other position including holes in the ice. The highest total reward for a successful episode is therefore 1.

The approach

Equipped with this information, we can begin to prepare our cross-entropy solution. We will use the following steps:

Create a neural network
Use the neural network to play a batch of episodes using the states as inputs and the outputs as actions
Set a reward threshold and filter out "bad" episodes that have episode rewards below the threshold
Train the neural network on the remaining "good" episodes using the states as inputs and the actions taken as desired outputs; our objective will be to minimise the loss between the neural network output and the desired output
Repeat from step 2 until the mean episode reward of a batch is good enough

The most important parts of the solution are described below, and a complete copy of the source code is included at the end of this post and in my frozen-lake-rl repository.

Creating the neural network

For step 1, we will create a very simple neural network using PyTorch that has a single hidden layer with ReLu activation and 128 neurons.

Before we get into the code, we should work out how we can feed the state into our neural network; this is required in steps 2 and 4. We saw above that the state is an integer with 16 possible values between 0 and 15. In our case, this is not be an ideal input for our neural network. Why? Because our neural network may interpret larger values as better than smaller values or vice versa. This would be fine if the state represented a variable with natural ordering such as speed or temperature but here it does not; it simply reflects the agent's position.

The remedy is to one hot encode the state integer into a list of 16 float numbers where all of the numbers are zero except for a 1.0 in the index that corresponds to the value of the integer.

With this approach, a state of 2, for example, would be represented by:

[0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

The simplest way to one hot encode the state within our solution is to leverage the built-in support for Gym observation wrappers. The code below will create an environment in the usual way and then apply the one hot encoding wrapper to it.

class DiscreteOneHotWrapper(gym.ObservationWrapper):
    def __init__(self, env):
        super(DiscreteOneHotWrapper, self).__init__(env)
        self.observation_space = gym.spaces.Box(0.0, 1.0, (env.observation_space.n, ), dtype=np.float32)

    def observation(self, observation):
        res = np.copy(self.observation_space.low)
        res[observation] = 1.0
        return res

env = DiscreteOneHotWrapper(gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False))

Our neural network will have 16 inputs to accept the one hot encoded state and 4 outputs. The outputs will represent an unnormalised probability distribution over the four possible actions.

Ordinarily, we would use a softmax activation to normalise the outputs and thus make them add up to one. We don’t here because our CrossEntropyLoss loss function expects the unnormalised, raw values. Instead, we just need to remember to apply softmax as a separate function when using our neural network outside of training.

HIDDEN_SIZE = 128
LEARNING_RATE = 0.001

class NeuralNet(nn.Module):
    def __init__(self, obs_size, hidden_size, n_actions):
        super(NeuralNet, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, n_actions)
        )

    def forward(self, x):
        return self.net(x)

net = NeuralNet(observation_shape, HIDDEN_SIZE, actions_n)
objective = nn.CrossEntropyLoss()
optimiser = optim.Adam(params=net.parameters(), lr=LEARNING_RATE)

To train our neural network and monitor progress, we need a mechanism for recording information about each episode. We will do this by defining two named tuples, Episode and EpisodeStep:

Episode = namedtuple('Episode', ['reward', 'reward_with_discount', 'steps'])
EpisodeStep = namedtuple('EpisodeStep', ['state', 'action'])

Training our neural network

Our solution will be orchestrated by a training loop. The loop will iterate batches of episodes that have been created using a generator function called generate_batches_of_episodes.

Each time we process a batch, we will calculate the mean episode reward and use that value to determine whether the environment has been solved. We will look for a mean episode reward of more than 0.8. This translates to more than 8 in every 10 episodes within a batch reaching the goal position.

BATCH_SIZE = 100
PERCENTILE = 30
GAMMA = 0.9

best_episodes_memory = []

# Generate batches of episodes and iterate a batch at a time
for iteration, batch in enumerate(generate_batches_of_episodes(env, net, BATCH_SIZE, actions_n)):

    mean_episode_reward = float(np.mean(list(map(lambda s: s.reward, batch))))
    mean_episode_reward_with_discount = float(np.mean(list(map(lambda s: s.reward_with_discount, batch))))

    # Check the mean reward within the batch
    if mean_episode_reward > 0.8:
        print("Environment solved!")
        break

Early on, because our neural network is intialised with random weights and biases, the mean episode reward will be very low. To increase the mean, we need to carry out the filtering and training steps of the cross-entropy method defined in steps 3 and 4.

To do this, we will filter out the episodes in a batch with the worst rewards using the filter_batch function. The function will return tensors for the states and actions in the current batch that resulted in the best rewards, and also a list called best_episodes_memory containing Episode objects with high rewards from earlier batches. Using new and recent sources of high performing Episode objects will help to speed up training.

    # Filter the batch and retain the best states and actions for training
    best_episodes_memory, batch_states_t, batch_actions_t, episode_reward_threshold = filter_batch(best_episodes_memory+batch, PERCENTILE)

Next, we will train our neural network using the batch_states_t tensor as the input. The objective will be to minimise the cross entropy loss between our neural network's output and the batch_actions_t tensor. In other words, our neural network will gradually align with recommending the actions that we know lead to good rewards.

    # Skip this iteration if we don't have any data to train on
    if not best_episodes_memory:
        continue

    # Train the neural network
    optimiser.zero_grad()
    action_predictions = net(batch_states_t)
    loss = objective(action_predictions, batch_actions_t)
    loss.backward()
    optimiser.step()

This completes the training loop. We should expect to iterate several times - taking a batch of episodes, filtering it and using it for training - before the mean episode reward is high enough to deem the environment solved.

Generating batches of episodes

Let's drill-down into the two functions that are called by the training loop, starting with generate_batches_of_episodes. The purpose of this function is to carry out episodes and perform actions within each episode, before yielding them to the training loop in a list of Episode objects.

We will use the Numpy random.choice method to choose an action. The expression below will generate a random number between 0 and 3 based on the probabilities from our neural network for each of the four possible actions.

Our use of the random.choice method in this way will provide just enough randomness to balance further exploration of the environment with exploiting the actions we know lead to good rewards.

episode_reward = 0.0
episode_steps = []
batch = []

sm = nn.Softmax(dim=1)

# Reset the environment and capture the current state
state, _ = env.reset()

while True:

    # Use the neural network with random.choice to choose an action
    state_t = torch.FloatTensor([state])
    action_probs_t = sm(net(state_t))
    action_probs = action_probs_t.data.numpy()[0]
    action = np.random.choice(actions_n, p=action_probs)

To understand the operation of the last four lines above, consider that we start with a discrete state of 1. In this scenario, our observation wrapper will one hot encode the state as:

[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]

Before we can pass the state to our neural network, we need to convert it to a tensor. We do this using the torch.FloatTensor method to create state_t:

tensor([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])

Next, we pass state_t into our neural network and then into the softmax function to return a tensor a called action_probs_t containing a normalised probability distribution for the four actions:

tensor([[0.2629, 0.2868, 0.2318, 0.2185]], grad_fn=<SoftmaxBackward0>)

We decode it from a tensor into a NumPy array:

[0.26288426 0.28681505 0.23181237 0.21848825]

And finally, we choose an action using random.choice and the probabilities output from our neural network. In this example, the suggested action is 2 which corresponds with a step that moves our agent to the right.

Still in the generate_batches_of_episodes function, the next part of the code will carry out the step, capture the reward and then add an EpisodeStep object containing the state and action to a list:

    # Apply a step using the chosen action
    next_state, reward, terminated, truncated, _ = env.step(action)

    # Add the reward to the total reward for this episode
    episode_reward += reward

    # Record the state before the action was taken and the action itself
    episode_steps.append(EpisodeStep(state=state, action=action))

When the episode is terminated (goal or hole reached) or truncated (too many steps taken), we will wrap up the episode by adding the list of EpisodeStep objects to an Episode object along with the episode reward and a discounted version of the episode reward. The discount will help to provide variability in the reward and uses GAMMA and the number of episode steps to achieve this.

When enough episodes have been gathered, the function returns the list of Episode objects to the training loop using the yield command. When the training loop later hands back to this function, it will continue executing from the line immediately after the yield:

    # Check if the episode has ended
    if terminated or truncated:

        # Discount the total episode reward to create variability between episodes
        episode_reward_with_discount = episode_reward * (GAMMA ** len(episode_steps))

        # Record the episode
        batch.append(Episode(reward=episode_reward, reward_with_discount=episode_reward_with_discount, steps=episode_steps))

        # Reset vars
        episode_reward = 0.0
        episode_steps = []
        next_state, _ = env.reset()

        if len(batch) == batch_size:

            # Return the batch to the training loop
            yield batch
            batch = []

    state = next_state

Filtering batches of episodes

We now move onto the code for the filter_batch function. The purpose of this function is to remove episodes within a batch that have a reward below a threshold.

When it comes to defining the threshold, we can exploit the variability in the discounted episode reward since it will provide a spread of values rather than just the 1 or 0 we have with the standard episode reward.

In our case, we will use the NumPy percentile method to deduce the q-th percentile from the list of discounted episode rewards, where q is PERCENTILE. We will then assign the result to episode_reward_threshold.

We will then iterate through each episode in a batch and only retain it if its discounted episode reward is above the threshold that we have just deduced.

Note, a batch may also include the best episodes from earlier batches so we will limit the number of filtered episodes that we return to 500 using best_episodes[-500:]:

    # Set a threshold based on the n-th percentile of discounted episode rewards within the batch
    episode_reward_threshold = np.percentile(list(map(lambda s: s.reward_with_discount, batch)), percentile)

    best_episodes = []
    batch_states = []
    batch_actions = []

    for episode in batch:
        if episode.reward_with_discount > episode_reward_threshold:

            # Add the states and actions from a high performing episode
            batch_states.extend(map(lambda step: step.state, episode.steps))
            batch_actions.extend(map(lambda step: step.action, episode.steps))

            best_episodes.append(episode)

    return best_episodes[-500:], torch.FloatTensor(batch_states), torch.LongTensor(batch_actions), episode_reward_threshold

Results

Running the code leads to a solution being found after around 120 episodes:

0:      Loss: 1.3657    Mean ep reward: 0.01    Mean disc ep reward: 0.0048
1:      Loss: 1.3814    Mean ep reward: 0.03    Mean disc ep reward: 0.0088
2:      Loss: 1.3742    Mean ep reward: 0.0     Mean disc ep reward: 0.0
3:      Loss: 1.3726    Mean ep reward: 0.02    Mean disc ep reward: 0.0055
4:      Loss: 1.3654    Mean ep reward: 0.01    Mean disc ep reward: 0.0035
5:      Loss: 1.3597    Mean ep reward: 0.03    Mean disc ep reward: 0.0078
6:      Loss: 1.3542    Mean ep reward: 0.02    Mean disc ep reward: 0.0042
7:      Loss: 1.3504    Mean ep reward: 0.0     Mean disc ep reward: 0.0
8:      Loss: 1.3466    Mean ep reward: 0.0     Mean disc ep reward: 0.0
9:      Loss: 1.3447    Mean ep reward: 0.01    Mean disc ep reward: 0.0021
10:     Loss: 1.3438    Mean ep reward: 0.04    Mean disc ep reward: 0.0116
11:     Loss: 1.3393    Mean ep reward: 0.04    Mean disc ep reward: 0.0122
12:     Loss: 1.3352    Mean ep reward: 0.01    Mean disc ep reward: 0.0028
13:     Loss: 1.3291    Mean ep reward: 0.01    Mean disc ep reward: 0.0023
14:     Loss: 1.323     Mean ep reward: 0.04    Mean disc ep reward: 0.011
...
...
...
111:    Loss: 0.5678    Mean ep reward: 0.62    Mean disc ep reward: 0.3
112:    Loss: 0.4479    Mean ep reward: 0.67    Mean disc ep reward: 0.3268
113:    Loss: 0.3356    Mean ep reward: 0.6     Mean disc ep reward: 0.2909
114:    Loss: 0.3315    Mean ep reward: 0.7     Mean disc ep reward: 0.3418
116:    Loss: 0.4725    Mean ep reward: 0.64    Mean disc ep reward: 0.3193
117:    Loss: 0.3119    Mean ep reward: 0.77    Mean disc ep reward: 0.3836
118:    Loss: 0.3133    Mean ep reward: 0.67    Mean disc ep reward: 0.3338
120:    Loss: 0.4974    Mean ep reward: 0.71    Mean disc ep reward: 0.3516
121:    Loss: 0.3878    Mean ep reward: 0.68    Mean disc ep reward: 0.3379
122:    Loss: 0.3064    Mean ep reward: 0.72    Mean disc ep reward: 0.3603
124:    Loss: 0.4871    Mean ep reward: 0.7     Mean disc ep reward: 0.3469
125:    Loss: 0.3627    Mean ep reward: 0.7     Mean disc ep reward: 0.3479
126:    Loss: 0.2901    Mean ep reward: 0.76    Mean disc ep reward: 0.3791
128:    Loss: 0.4612    Mean ep reward: 0.71    Mean disc ep reward: 0.3554
129:    Loss: 0.2774    Mean ep reward: 0.79    Mean disc ep reward: 0.3982
Environment solved!

Loss by iteration

Mean episode reward by iteration

Mean discounted episode reward

Conclusion

In this post, we worked out how to solve the famous Frozen Lake environment using the cross-entropy.

Interestingly, this approach is not so effective with the non-slippery version (is_slippery=True). The mean episode reward peaks at around 0.5. To solve this scenario and indeed more complex environments, other methods can be used including value iteration, Q-learning and deep Q-networks.

For more information about all of these, I highly recommend Deep Reinforcement Learning Hands-On by Maxim Lapan; parts of the code above were inspired by the book.

Source code

Download from GitHub cross-entropy-rl repository

import random
from collections import namedtuple

import gym
import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim

HIDDEN_SIZE = 128
LEARNING_RATE = 0.001

BATCH_SIZE = 100
PERCENTILE = 30
GAMMA = 0.9

Episode = namedtuple('Episode', ['reward', 'reward_with_discount', 'steps'])
EpisodeStep = namedtuple('EpisodeStep', ['state', 'action'])

class DiscreteOneHotWrapper(gym.ObservationWrapper):
    def __init__(self, env):
        super(DiscreteOneHotWrapper, self).__init__(env)
        self.observation_space = gym.spaces.Box(0.0, 1.0, (env.observation_space.n, ), dtype=np.float32)

    def observation(self, observation):
        res = np.copy(self.observation_space.low)
        res[observation] = 1.0
        return res

class NeuralNet(nn.Module):
    def __init__(self, obs_size, hidden_size, n_actions):
        super(NeuralNet, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_size, hidden_size),
            nn.ReLU(),
            nn.Linear(hidden_size, n_actions)
        )

    def forward(self, x):
        return self.net(x)

def generate_batches_of_episodes(env, net, batch_size, actions_n):

    episode_reward = 0.0
    episode_steps = []
    batch = []

    sm = nn.Softmax(dim=1)

    # Reset the environment and capture the current state
    state, _ = env.reset()

    while True:

        # Use the neural network with random.choice to choose an action
        state_t = torch.FloatTensor([state])
        action_probs_t = sm(net(state_t))
        action_probs = action_probs_t.data.numpy()[0]
        action = np.random.choice(actions_n, p=action_probs)

        # Apply a step using the chosen action
        next_state, reward, terminated, truncated, _ = env.step(action)

        # Add the reward to the total reward for this episode
        episode_reward += reward

        # Record the state before the action was taken and the action itself
        episode_steps.append(EpisodeStep(state=state, action=action))

        # Check if the episode has ended
        if terminated or truncated:

            # Discount the total episode reward to create variability between episodes
            episode_reward_with_discount = episode_reward * (GAMMA ** len(episode_steps))

            # Record the episode
            batch.append(Episode(reward=episode_reward, reward_with_discount=episode_reward_with_discount, steps=episode_steps))

            # Reset vars
            episode_reward = 0.0
            episode_steps = []
            next_state, _ = env.reset()

            if len(batch) == batch_size:

                # Return the batch to the training loop
                yield batch
                batch = []

        state = next_state

def filter_batch(batch, percentile):

    # Set a threshold based on the n-th percentile of discounted episode rewards within the batch
    episode_reward_threshold = np.percentile(list(map(lambda s: s.reward_with_discount, batch)), percentile)

    best_episodes = []
    batch_states = []
    batch_actions = []

    for episode in batch:
        if episode.reward_with_discount > episode_reward_threshold:

            # Add the states and actions from a high performing episode
            batch_states.extend(map(lambda step: step.state, episode.steps))
            batch_actions.extend(map(lambda step: step.action, episode.steps))

            best_episodes.append(episode)

    return best_episodes[-500:], torch.FloatTensor(batch_states), torch.LongTensor(batch_actions), episode_reward_threshold

def render_n_steps(env, net, steps_n):

    sm = nn.Softmax(dim=1) 
    state, _ = env.reset()

    for i in range(steps_n):

        state_t = torch.FloatTensor([state])

        if net is None:
            # Choose a random step
            action = env.action_space.sample()
        else:
            # Choose a step using the (trained) neural network
            action_probs_t = sm(net(state_t))
            action = np.argmax(action_probs_t.data.numpy()[0])

        state, reward, terminated, truncated, _ = env.step(action)

        # Render the step on the display
        env.render()

        if terminated or truncated: state, _ = env.reset()            

if __name__ == "__main__":

    # Render random steps before training
    env = DiscreteOneHotWrapper(gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False, render_mode="human"))
    render_n_steps(env, None, 50)

    # Create the environment
    env = DiscreteOneHotWrapper(gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False))

    # Capture environment information
    observation_shape = env.observation_space.shape[0]
    actions_n = env.action_space.n

    # Create the neural network
    net = NeuralNet(observation_shape, HIDDEN_SIZE, actions_n)
    objective = nn.CrossEntropyLoss()
    optimiser = optim.Adam(params=net.parameters(), lr=LEARNING_RATE)

    best_episodes_memory = []

    # Generate batches of episodes and iterate a batch at a time
    for iteration, batch in enumerate(generate_batches_of_episodes(env, net, BATCH_SIZE, actions_n)):

        mean_episode_reward = float(np.mean(list(map(lambda s: s.reward, batch))))
        mean_episode_reward_with_discount = float(np.mean(list(map(lambda s: s.reward_with_discount, batch))))

        # Check the mean reward within the batch
        if mean_episode_reward > 0.8:
            print("Environment solved!")
            break

        # Filter the batch and retain the best states and actions for training
        best_episodes_memory, batch_states_t, batch_actions_t, episode_reward_threshold = filter_batch(best_episodes_memory+batch, PERCENTILE)

        # Skip this iteration if we don't have any data to train on
        if not best_episodes_memory:
            continue

        # Train the neural network
        optimiser.zero_grad()
        action_predictions = net(batch_states_t)
        loss = objective(action_predictions, batch_actions_t)
        loss.backward()
        optimiser.step()

        # Report performance
        print(f"{iteration}:\tLoss: {round(loss.item(), 4)}\tMean ep reward: {round(mean_episode_reward, 4)}\tMean ep reward with disc: {round(mean_episode_reward_with_discount, 4)}")

    # Render some steps after training
    env = DiscreteOneHotWrapper(gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False, render_mode="human"))
    render_n_steps(env, net, 50)

    # Destroy environment
    env.close()

Neural networks 101: The basics of forward and backward propagation

Daniel Cooper — Sat, 04 Feb 2023 12:42:23 +0000

This post shows you how to construct the forward propagation and backpropagation algorithms for a simple neural network. It may be helpful to you if you've just started to learn about neural networks and want to check your logic regarding the maths.

Consider the neural network below. It has two inputs (x1 and x2), two hidden neurons (h1 and h2), one output neuron (o1) and a series of weights and biases.

We want to use this neural network to predict outputs and have four training examples which show the expected output y for different combinations of x1 and x2.

x1	x2	y
0	0	0
0	1	1
1	0	1
1	1	1

Our neural network starts with a random set of weights between 0 and 1, and biases set to 0. In this initial configuration, as you might expect, our neural network will predict outputs which bear no relation to the expected outputs.

To improve the prediction accuracy, we need to optimise the weights and biases so that they work together to predict outputs that are as close as possible to the expected outputs.

This optimisation process is called training and it involves two stages: forward propagation and backpropagation.

Forward propagation

The forward pass stage moves through the neural network from left to right.

Each neuron is configured to sum its weighted inputs with its bias and to then pass that value z through an activation function $σ\sigma$ to produce an output a.

We can express the sum z and output a for each neuron as:

\begin{align*} z_{h1} &= w_1x_1 + w_3x_2 + b_1 \newline a_{h1} &= \sigma(z_{h1}) \newline\newline z_{h2} &= w_2x_1 + w_4x_2 + b_2 \newline a_{h2} &= \sigma(z_{h2}) \newline\newline z_{o1} &= w_5a_{h1} + w_6a_{h2} + b_3 \newline a_{o1} &= \sigma(z_{o1}) \end{align*}

The activation function $σ\sigma$ is used to transform a continuous input into an output between 0 and 1, with 0 for large negative inputs and 1 for large positive inputs.

Here we'll apply the commonly-used Sigmoid function for activation:

\sigma(z) = \frac{\mathrm{1} }{\mathrm{1} + e^{-z}}

With these equations, we can use the x1 and x2 from each of the training examples to predict an output (ao1).

To measure the prediction accuracy of our neural network across all of the training examples, we can the Mean Squared Error function to find the total loss J:

\displaystyle \text{Total loss, J} = \frac{1}{2m}\sum_{i=1}^{m}(a_{o1} - y)^2

where m is the total number of training examples; in our case, m = 4.

To begin, the total loss J is almost certain to be greater than zero unless we have been particularly fortunate with our random weights and biases.

More likely we will need to optimise the weights and biases and to do that, we use backpropagation.

Backpropagation

Backpropagation moves backwards through the neural network, from right to left.

Using equation from above for the total loss J, if we consider just a single training example with m = 1, we can define a loss L:

\displaystyle \text{Loss, L} = \frac{1}{2}(a_{o1} - y)^2

The loss L is dependent on ao1 which itself is dependent on all of the weights and biases. It follows that optimising these weights and biases will reduce the loss L.

Using an iterative process, we can optimise each weight and bias using the principle of gradient descent and a suitable learning rate $α\alpha$ :

\begin{align*} w_s &= w_s - \alpha\dfrac{\mathrm{d}L}{\mathrm{d}w_s} \text{ where s = 1, 2, ..., 6} \newline\newline b_t &= b_t - \alpha\dfrac{\mathrm{d}L}{\mathrm{d}b_t} \text{ where t = 1, 2, 3} \end{align*}

To use these equations, we first need expressions for the derivative of the loss L with respect to each weight and bias. These expressions are partial derivatives which we can derive using the chain rule.

Before we start, there are two derivatives that we will come in handy. The first is the derivative of the Sigmoid activation function:

\begin{align*} \sigma(z) &= \frac{\mathrm{1} }{\mathrm{1} + e^{-z}} \newline\newline \dfrac{\mathrm{d}\sigma(z)}{\mathrm{d}z} &= \sigma(z)(1-\sigma(z)) \end{align*}

The second is the derivative of the loss L with respect to ao1:

\begin{align*} \displaystyle L &= \frac{1}{2}(a_{o1} - y)^2 \newline\newline \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}} &= (a_{o1} - y) \end{align*}

Let's now start with weight w6 and find the derivative of loss L with respect to w6:

\dfrac{\mathrm{d}L}{\mathrm{d}w_6}

Using the chain rule, we can say that:

\dfrac{\mathrm{d}L}{\mathrm{d}w_6} = \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_6}

Therefore:

\dfrac{\mathrm{d}L}{\mathrm{d}w_6} = (a_{o1} - y).\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_6}

This is progress but we need to go further with:

\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_6}

Using the chain rule again, we can say:

\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_6} = \dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}w_6}

where:

\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}} = \dfrac{\mathrm{d}\sigma(z_{o1})}{\mathrm{d}z_{o1}} = \sigma(z_{o1})(1-\sigma(z_{o1}))

and:

\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}w_6} = \dfrac{\mathrm{d}(w_5a_{h1} + w_6a_{h2} + b_3)}{\mathrm{d}w_6} = a_{h2} = \sigma(z_{h2})

Combining these partial derivatives, we get an expression for w6 that can be computed:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}w_6} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_6} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}w_6} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).\sigma(z_{h2}) \end{align*}

Using the same approach for w5 gives:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}w_5} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_5} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}w_5} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).\sigma(z_{h1}) \end{align*}

For b3:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}b_3} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}b_3} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}b_3} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})) \end{align*}

which takes a shorter form because:

\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}b_3} = \dfrac{\mathrm{d}(w_5a_{h1} + w_6a_{h2} + b_3)}{\mathrm{d}b_3} = 1

Next, we need to repeat the exercise with weights w4, w3, w2, w1 and biases b2 and b1.

Let's start with weight w4 and find the derivative of loss L with respect to w4:

\dfrac{\mathrm{d}L}{\mathrm{d}w_4}

Using the chain rule, we can say that:

\dfrac{\mathrm{d}L}{\mathrm{d}w_4} = \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_4}

Therefore:

\dfrac{\mathrm{d}L}{\mathrm{d}w_4} = (a_{o1} - y).\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_4}

Once again, this is progress but we need to go further with:

\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_4}

Using the chain rule again, we can say:

\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_4} = \dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}w_4}

where:

\begin{align*} \dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h2}} &= \dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h2}} \newline\newline &= \dfrac{\mathrm{d}\sigma(z_{o1})}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}(w_5a_{h1} + w_6a_{h2} + b_3)}{\mathrm{d}a_{h2}} \newline\newline &= \sigma(z_{o1})(1-\sigma(z_{o1})).w_6 \end{align*}

and:

\begin{align*} \dfrac{\mathrm{d}a_{h2}}{\mathrm{d}w_4} &= \dfrac{\mathrm{d}a_{h2}}{\mathrm{d}z_{h2}}.\dfrac{\mathrm{d}z_{h2}}{\mathrm{d}w_4} \newline\newline &= \dfrac{\mathrm{d}\sigma(z_{h2})}{\mathrm{d}z_{h2}}.\dfrac{\mathrm{d}(w_2x_1 + w_4x_2 + b_2)}{\mathrm{d}w_4} \newline\newline &= \sigma(z_{h2})(1-\sigma(z_{h2})).x_2 \end{align*}

Finally for w4:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}w_4} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_4} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}w_4} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}z_{h2}}.\dfrac{\mathrm{d}z_{h2}}{\mathrm{d}w_4} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).w_6.\sigma(z_{h2})(1-\sigma(z_{h2})).x_2 \end{align*}

Using the same approach for w3 gives:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}w_3} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_3} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h1}}.\dfrac{\mathrm{d}a_{h1}}{\mathrm{d}w_3} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h1}}.\dfrac{\mathrm{d}a_{h1}}{\mathrm{d}z_{h1}}.\dfrac{\mathrm{d}z_{h1}}{\mathrm{d}w_3} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).w_5.\sigma(z_{h1})(1-\sigma(z_{h1})).x_2 \end{align*}

For w2:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}w_2} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_2} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}w_2} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}z_{h2}}.\dfrac{\mathrm{d}z_{h2}}{\mathrm{d}w_2} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).w_6.\sigma(z_{h2})(1-\sigma(z_{h2})).x_1 \end{align*}

For w1:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}w_1} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}w_1} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h1}}.\dfrac{\mathrm{d}a_{h1}}{\mathrm{d}w_1} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h1}}.\dfrac{\mathrm{d}a_{h1}}{\mathrm{d}z_{h1}}.\dfrac{\mathrm{d}z_{h1}}{\mathrm{d}w_1} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).w_5.\sigma(z_{h1})(1-\sigma(z_{h1})).x_1 \end{align*}

For b2:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}b_2} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}b_2} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}b_2} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h2}}.\dfrac{\mathrm{d}a_{h2}}{\mathrm{d}z_{h2}}.\dfrac{\mathrm{d}z_{h2}}{\mathrm{d}b_2} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).w_6.\sigma(z_{h2})(1-\sigma(z_{h2})) \end{align*}

which takes a shorter form because:

\dfrac{\mathrm{d}z_{h2}}{\mathrm{d}b_2} = \dfrac{\mathrm{d}(w_2x_1 + w_4x_2 + b_2)}{\mathrm{d}b_2} = 1

For b1:

\begin{align*} \dfrac{\mathrm{d}L}{\mathrm{d}b_1} &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}b_1} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}a_{h1}}.\dfrac{\mathrm{d}a_{h1}}{\mathrm{d}b_1} \newline\newline &= \dfrac{\mathrm{d}L}{\mathrm{d}a_{o1}}.\dfrac{\mathrm{d}a_{o1}}{\mathrm{d}z_{o1}}.\dfrac{\mathrm{d}z_{o1}}{\mathrm{d}a_{h1}}.\dfrac{\mathrm{d}a_{h1}}{\mathrm{d}z_{h1}}.\dfrac{\mathrm{d}z_{h1}}{\mathrm{d}b_1} \newline\newline &= (a_{o1} - y).\sigma(z_{o1})(1-\sigma(z_{o1})).w_5.\sigma(z_{h1})(1-\sigma(z_{h1})) \end{align*}

which also takes a shorter form because:

\dfrac{\mathrm{d}z_{h1}}{\mathrm{d}b_1} = \dfrac{\mathrm{d}(w_1x_1 + w_3x_2 + b_1)}{\mathrm{d}b_1} = 1

Training

With expressions for the derivative of the loss L with respect to each weight and bias, we can now train our neural network using the gradient descent equations that we saw earlier:

\begin{align*} w_s &= w_s - \alpha\dfrac{\mathrm{d}L}{\mathrm{d}w_s} \text{ where s = 1, 2, ..., 6} \newline\newline b_t &= b_t - \alpha\dfrac{\mathrm{d}L}{\mathrm{d}b_t} \text{ where t = 1, 2, 3} \end{align*}

Here's the step-by-step process for training:

Step 1. Take the first training example
- Calculate the current outputs from each neuron using the forward propagation equations
- Update the weights and biases using the gradient descent equations
Step 2. Repeat step 1 with each of the remaining training examples
Step 3. Repeat the entire process another 10,000 epochs

After 10,000 epochs, the total loss J of our neural network should reduce, leading to an improvement in prediction accuracy.

In a perfect scenario, plotting the total loss J against the number of iterations performed will reveal something like this:

With this sort of outcome, our trained neural network should do pretty well at predicting outputs that are very close to the expected outputs. For example:

x1	x2	y	ao1
0	0	0	0.04
0	1	1	0.98
1	0	1	0.98
1	1	1	1.00

If we find the total loss J has not reduced at all or sufficiently after training, further optimisation may be achieved by experimenting with different learning rates $α\alpha$ (e.g. 0.1, 0.01, 0.001). That's another topic in its own right...

Summary

Congratulations if you made it to the end and thanks for reading!

Here's what we covered, some in more detail than others:

Forward propagation
Sigmoid activation function
Concept of loss L and total loss J
Backpropagation using partial derivatives and the chain rule
Training to minimise the total loss J

Please comment below if you found this post useful or if you've spotted an error.

Get started with Docker: The most useful commands you need

Daniel Cooper — Sat, 04 Feb 2023 11:39:07 +0000

This post covers the commands that I found most useful when starting to learn Docker from installation to managing images and containers.

The first thing to do is install the Docker engine on your host platform. In my case, that's Ubuntu Mate 21.10 running on a Raspberry Pi 4 in the corner of my office. Docker provides detailed instructions on how to do this by platform and Linux distribution.

Once Docker is installed, we can start a Docker container from a Docker image. This can be an image you've built yourself or one of the 8.4 million images available on Docker Hub. From Redis to Jenkins to WordPress, there's an image for it on the hub.

In the following example, the docker run command is used to create and start a container from an image of Debian, the Linux distribution. Since the image doesn't already exist locally, Docker automatically checks the hub and tries to download it from there:

$ docker run debian

Unable to find image 'debian:latest' locally
latest: Pulling from library/debian
1c47a4233665: Pull complete
Digest: sha256:4d6ab716de467aad58e91b1b720f0badd7478847ec7a18f66027d0f8a329a43c
Status: Downloaded newer image for debian:latest

The image is downloaded and a container starts but immediately exits as it has nothing to do. You can verify it started and then stopped using the docker ps command with the -a flag to return all containers, not just the running ones:

$ docker ps -a

CONTAINER ID   IMAGE     COMMAND   CREATED          STATUS                      PORTS     NAMES
16223809b3db   debian    "bash"    49 seconds ago   Exited (0) 37 seconds ago             gallant_hawking

To give a new container something to do before it exits, we can include a command like ls to list the contents of the default working directory:

$ docker run debian ls

bin
boot
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Once again, we can verify a new container was started and then stopped after the ls command was executed:

$ docker ps -a

CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS                          PORTS     NAMES
4cd7aec68643   debian    "ls"      38 seconds ago       Exited (0) 36 seconds ago                 laughing_mcnulty
16223809b3db   debian    "bash"    About a minute ago   Exited (0) About a minute ago             gallant_hawking

In some cases, you’ll want to interact with a container and have it start you in a particular working directory. You can achieve this with docker run plus the -it and -w flags:

$ docker run -it -w /home debian

root@5d488c949415:/home#

Since the container was started in interactive mode, use exit to leave it:

root@5d488c949415:/home# exit

Once again, use the docker ps command to list all containers:

$ docker ps -a

CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS                      PORTS     NAMES
5d488c949415   debian    "bash"    About a minute ago   Exited (0) 26 seconds ago             magical_shirley
4cd7aec68643   debian    "ls"      2 minutes ago        Exited (0) 2 minutes ago              laughing_mcnulty
16223809b3db   debian    "bash"    3 minutes ago        Exited (0) 3 minutes ago              gallant_hawking

A stopped container can be restarted using the docker container start command and enough characters of the container ID for Docker to know which container you mean; the -i flag restarts the container in interactive mode:

$ docker container start -i 5d

root@5d488c949415:/home#

Likewise, a running container can be stopped:

$ docker container stop 5d

5d

One thing you’ll notice is that Docker takes a conservative approach to cleaning up unused objects like images and containers. When you stop a container, it is not automatically removed unless you use the --rm flag when starting the container:

$ docker run -it -w /home --rm debian

To manually clean up, you can delete a container:

$ docker container rm 5d

5d

Alternatively, you can use the prune command to delete all stopped containers:

$ docker container prune

WARNING! This will remove all stopped containers.
Are you sure you want to continue? [y/N] y
Deleted Containers:
4cd7aec68643d855b757ba29cf9a2cd9df50853c88c5307e1286c50a45accb75
16223809b3db97013f1939c30213c6dc11144bf8d389522d6c8d90bc66e5416b

Total reclaimed space: 0B

The same sort of garbage collection is needed for images. You can list your images using docker image ls with the -a flag for all:

$ docker image ls -a

REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
debian       latest    bf33d9a9dfb7   3 weeks ago   118MB

To delete an image not associated with the container bf33d9a9dfb7:

$ docker image rm bf

Untagged: debian:latest
Untagged: debian@sha256:4d6ab716de467aad58e91b1b720f0badd7478847ec7a18f66027d0f8a329a43c
Deleted: sha256:bf33d9a9dfb709a9a24218f6ea55bab3965d0029cdc2ce01214ad1a87040e530
Deleted: sha256:478649b375d177fb986d0b96804929861077c6c15da73a400fc900afc9187900

Or you can use docker image prune with -a to remove all unused images:

$ docker image prune -a

WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

So those are the basics which provided me with enough information to move onto more interesting scenarios such as hosting Flask apps in containers. My flask-test repository on GitHub provides a demo app which can be built and hosted within a container in just a few steps:

$ git clone https://github.com/exactful/flask-test.git

Cloning into 'flask-test'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 24 (delta 10), reused 5 (delta 0), pack-reused 0
Receiving objects: 100% (24/24), 4.69 KiB | 1.17 MiB/s, done.
Resolving deltas: 100% (10/10), done.

The Dockerfile contained within the newly-cloned flask-test directory contains all of the information that Docker needs to build an image:

$ cat flask-task/Dockerfile

FROM python:3.8-slim-buster
WORKDIR /usr/src/app
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
#Server will reload itself on file changes if in dev mode
ENV FLASK_ENV=development
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
CMD ["flask", "run"]

The image can be built using docker image build and the -t flag to provide it with a tag of flask-test-image:

$ docker image build -t flask-test-image flask-test

Sending build context to Docker daemon   68.1kB
Step 1/9 : FROM python:3.8-slim-buster
3.8-slim-buster: Pulling from library/python
02b45931a436: Pull complete
1b907b3f2a40: Pull complete
ade9c40131e5: Pull complete
6380b003fa8e: Pull complete
1e31a0f45c23: Pull complete
Digest: sha256:9e3036f6b032794efb662f3c579c4c35d0b678bc793590e3e2e217cb5bf1e11b
Status: Downloaded newer image for python:3.8-slim-buster
 ---> db1e59c73041
Step 2/9 : WORKDIR /usr/src/app
 ---> Running in f9c49a06c8df
Removing intermediate container f9c49a06c8df
 ---> c79c8f1f9c1c
Step 3/9 : ENV FLASK_APP=app.py
 ---> Running in 76801af3fdb2
Removing intermediate container 76801af3fdb2
 ---> 4eaf8967484d
Step 4/9 : ENV FLASK_RUN_HOST=0.0.0.0
 ---> Running in 63fb31ebe1d3
Removing intermediate container 63fb31ebe1d3
 ---> 4ff75116a97d
Step 5/9 : ENV FLASK_ENV=development
 ---> Running in bbf385cb1f05
Removing intermediate container bbf385cb1f05
 ---> 97f2f549a0e5
Step 6/9 : COPY requirements.txt requirements.txt
 ---> cb0cc984e762
Step 7/9 : RUN pip install -r requirements.txt
 ---> Running in d3062b512e5e
Collecting flask
  Downloading Flask-2.0.2-py3-none-any.whl (95 kB)
Collecting itsdangerous>=2.0
  Downloading itsdangerous-2.0.1-py3-none-any.whl (18 kB)
Collecting Jinja2>=3.0
  Downloading Jinja2-3.0.2-py3-none-any.whl (133 kB)
Collecting Werkzeug>=2.0
  Downloading Werkzeug-2.0.2-py3-none-any.whl (288 kB)
Collecting click>=7.1.2
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (27 kB)
Installing collected packages: MarkupSafe, Werkzeug, Jinja2, itsdangerous, click, flask
Successfully installed Jinja2-3.0.2 MarkupSafe-2.0.1 Werkzeug-2.0.2 click-8.0.3 flask-2.0.2 itsdangerous-2.0.1
Removing intermediate container d3062b512e5e
 ---> c0810573e7c5
Step 8/9 : COPY . .
 ---> 91e02222a41b
Step 9/9 : CMD ["flask", "run"]
 ---> Running in df1610a6af6a
Removing intermediate container df1610a6af6a
 ---> 6d7816d54b60
Successfully built 6d7816d54b60
Successfully tagged flask-test-image:latest

The now familiar docker run can be used to create a container; the -d flag runs the container in the background (aka detached mode) and the -p flag maps the container’s port 5000 to the Docker host's port 80:

$ docker run -d -p 80:5000 flask-test-image

ae30c4f1e0871f23c91fb85e33501a00ae5dd699f6d46ddc7b9aad3ef74a9e70

With Docker now listening on port 80, you can browse to the IP address of your Docker host to see the demo app. My Raspberry Pi is connected to my local network on 192.168.4.64 so here's what I see on port 80 at http://192.168.4.64:

I hope this post helps you progress further if you’re just starting out with Docker. From here, I highly recommend creating services using docker stack deploy and persisting container data using volumes.

Virtual environments: How to avoid Python dependency hell with venv and pip freeze

Daniel Cooper — Fri, 03 Feb 2023 22:43:12 +0000

As a web dev in the 2005, dependency management was one of the more challenging aspects of my role at a national media company.

One example that comes to mind was the almost literal pain involved in deploying a DLL update to our sprawling .NET web application across a group of three Microsoft IIS servers.

The update process was simple but achingly manual: take each server in turn, manually drag and drop the new DLL into the server's Global Assembly Cache (GAC) folder, and then add a reference to the version number of the DLL in the server's machine.config.

Even when diligently following these steps, a catastrophic "Server Error in '/' Application" would sometimes occur, bringing the whole web application crashing down followed by panicked calls from our editorial teams.

If we were lucky, depending on how many servers had been patched at that point, the web application would stay up every other page refresh or so. Not an ideal customer experience, even for 2005.

The underlying cause was invariably due to an errant reference to an older version of the DLL in a web.config hidden away in some subfolder within the web application. It was commonly referred to as DLL hell.

Fast forward to 2020 and I was reminded of these past woes when trying to get some old Python code up and running on a new machine.

On running the code, Python complained about umpteen missing packages from NumPy to Beautiful Soup. Dependency hell. Now while it's easy enough to install them manually using pip, you need to know the exact name and ideally the version.

pip install beautifulsoup

ERROR: Could not find a version that satisfies the requirement beautifulsoup
ERROR: No matching distribution found for beautifulsoup

pip install beautifulsoup Huh? Oh.. it's actually called beautifulsoup4. Got it. Ahh, the latest version has broken my code. Sigh. Seem familiar?

Fortunately, there is a better way to handle this scenario, and it involves virtual environments, pip freeze and a file called requirements.txt.

Virtual environments. It's good practice to create and activate a virtual environment in your working folder as soon as you start a new project.

The venv module provides a way to do this. Here's what the docs say:

The venv module provides support for creating lightweight "virtual environments" with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories.

It's easy to create a virtual environment in Linux and macOS:

python -m venv venv
source venv/bin/activate

...although the activate command is platform-specific so check the docs for the Windows syntax.

Now, with a virtual environment in place for your project, the interpreter can see only the packages you've installed in that specific environment. You can see exactly which ones by running pip freeze:

pip freeze

beautifulsoup4==4.10.0
certifi==2021.5.30
charset-normalizer==2.0.5
idna==3.2
requests==2.26.0
soupsieve==2.2.1
urllib3==1.26.6

From there, simply feed the output from pip freeze into a file called requirements.txt.

pip freeze > requirements.txt

This creates a definitive record of your project's required packages and version numbers - in the form of requirements.txt - which you and others can use to install those dependencies in future within a fresh environment. Here's the install command:

pip install -r requirements.txt

Had I followed these steps way back at the start of this particular Python project that was causing me grief, I could have used the requirements.txt to get up and running in no time.

Of course, the specific install command above can be useful in other places too. If you happen to build Docker images, you can use it within your Dockerfile.

In summary... virtual environments, pip freeze and requirements.txt all provide a neat way to manage Python packages.