Neural Networks: Building a "Brain" from Scratch

Introduction:

Welcome to our step-by-step guide on building a neural network from scratch! Neural networks are at the forefront of artificial intelligence and have revolutionized various industries, from healthcare to finance. In this blog post, we'll walk you through the process of creating a simple neural network using Python and NumPy, without relying on any external libraries.

Understanding Neural Networks:

Before diving into the implementation, let's understand what neural networks are. Neural networks are a computational model inspired by the human brain's structure. They consist of interconnected nodes, organized into layers. Each node performs a simple computation and passes its output to the next layer. Neural networks can learn from data and make predictions by adjusting their internal parameters.

Setting Up the Environment:

To start, we'll need Python and NumPy installed on your system. You can install NumPy using pip:

pip install numpy

Building Blocks of the Neural Network:

We'll start by implementing the fundamental building blocks of our neural network. This includes the sigmoid activation function and its derivative, which are essential for introducing non-linearity into the model. We'll also initialize the parameters of the neural network: weights and biases.

def sigmoid(x):
    '''
    The sigmoid function, often used as an activation function
in neural networks,
    especially in binary classification problems. It squashes the input values between 0 and 1,
    facilitating non-linear transformations in the network.
    '''
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    '''
    The derivative of the sigmoid function, which is important for the backpropagation algorithm
in neural networks. It is used to calculate gradients for weights and biases updates,
    enabling the network to learn from the training data.
    '''
    return x * (1 - x)

def initialize_parameters(input_size, hidden_size, output_size):
    '''
    Function to initialize weights & biases for a feedforward neural network with one hidden layer.

    input_size: The number of neurons in the input layer, which corresponds to the number of features in the dataset.
    hidden_size: The number of neurons in the hidden layer.
    output_size: The number of neurons in the output layer. For binary classification, this would typically be 1.
    '''
    np.random.seed(2)
    # Initialize weights between input layer and hidden layer
    weights_input_hidden = np.random.rand(input_size, hidden_size) - 0.5
    # Initialize weights between hidden layer and output layer
    weights_hidden_output = np.random.rand(hidden_size, output_size) - 0.5
    # Initialize bias for the hidden layer to zeros
    bias_hidden = np.zeros((1, hidden_size))
    # Initialize bias for the output layer to zeros
    bias_output = np.zeros((1, output_size))
    return weights_input_hidden, weights_hidden_output, bias_hidden, bias_output

I have also added comments in each line for more clarification.

Forward Propagation:

Forward propagation is the process of computing the output of the neural network given an input. We'll pass the input through the network, apply the activation function, and obtain the final output.

def forward_propagation(X, weights_input_hidden, weights_hidden_output, bias_hidden, bias_output):
    '''
    Function to perform forward propagation through the neural network.

    X: Input data
    weights_input_hidden: Weights between input layer and hidden layer
    weights_hidden_output: Weights between hidden layer and output layer
    bias_hidden: Bias for the hidden layer
    bias_output: Bias for the output layer
    '''
    # Calculate hidden layer input
    hidden_input = np.dot(X, weights_input_hidden) + bias_hidden
    # Apply sigmoid activation function to hidden layer input
    hidden_output = sigmoid(hidden_input)
    # Calculate output layer input
    final_input = np.dot(hidden_output, weights_hidden_output) + bias_output
    # Apply sigmoid activation function to output layer input
    final_output = sigmoid(final_input)
    return hidden_output, final_output

Loss Calculation:

To train the neural network, we need to define a loss function that quantifies the difference between the predicted output and the actual output. We'll use the cross-entropy loss function, which is commonly used for binary classification problems.

def compute_loss(Y, Y_hat):
    '''
    Function to compute the loss between predicted output and actual output.

    Y: Actual output
    Y_hat: Predicted output
    '''
    m = Y.shape[0]
    # Compute cross-entropy loss
    loss = -(1/m) * np.sum(Y * np.log(Y_hat) + (1 - Y) * np.log(1 - Y_hat))
    return loss

Backpropagation:

Backpropagation is the algorithm used to update the parameters of the neural network based on the computed gradients of the loss function with respect to the parameters. It enables the network to learn from the training data and improve its predictions over time.

def backpropagation(X, Y, hidden_output, final_output, weights_hidden_output, weights_input_hidden):
    '''
    Function to perform backpropagation to calculate gradients for weights and biases updates.

    X: Input data
    Y: Actual output
    hidden_output: Output of the hidden layer
    final_output: Predicted output
    weights_hidden_output: Weights between hidden layer and output layer
    weights_input_hidden: Weights between input layer and hidden layer
    '''
    # Compute error at output layer
    error = final_output - Y
    # Compute gradients for weights between hidden and output layers
    d_weights_hidden_output = np.dot(hidden_output.T, error * sigmoid_derivative(final_output))
    # Compute gradients for bias at output layer
    d_bias_output = np.sum(error * sigmoid_derivative(final_output), axis=0, keepdims=True)

    # Compute error at hidden layer
    error_hidden = np.dot(error * sigmoid_derivative(final_output), weights_hidden_output.T)
    # Compute gradients for weights between input and hidden layers
    d_weights_input_hidden = np.dot(X.T, error_hidden * sigmoid_derivative(hidden_output))
    # Compute gradients for bias at hidden layer
    d_bias_hidden = np.sum(error_hidden * sigmoid_derivative(hidden_output), axis=0, keepdims=True)

    return d_weights_input_hidden, d_weights_hidden_output, d_bias_hidden, d_bias_output

Updating Parameters:

After computing the gradients using backpropagation, we'll update the parameters of the neural network using gradient descent.

def update_parameters(parameters, grads, learning_rate=1.0):
    '''
    Function to update weights and biases based on gradients computed during backpropagation.

    parameters: Tuple containing weights and biases
    grads: Tuple containing gradients for weights and biases
    learning_rate: Learning rate for gradient descent
    '''
    weights_input_hidden, weights_hidden_output, bias_hidden, bias_output = parameters
    d_weights_input_hidden, d_weights_hidden_output, d_bias_hidden, d_bias_output = grads

    # Update weights between input and hidden layers
    weights_input_hidden -= learning_rate * d_weights_input_hidden
    # Update weights between hidden and output layers
    weights_hidden_output -= learning_rate * d_weights_hidden_output
    # Update bias at hidden layer
    bias_hidden -= learning_rate * d_bias_hidden
    # Update bias at output layer
    bias_output -= learning_rate * d_bias_output

    return weights_input_hidden, weights_hidden_output, bias_hidden, bias_output

Training the Neural Network:

With all the components in place, we can now train the neural network using a sample dataset. We'll iterate through multiple epochs, updating the parameters at each step to minimize the loss.

def train(X, Y, input_size, hidden_size, output_size, learning_rate, epochs):
    '''
    Function to train the neural network using gradient descent.

    X: Input data
    Y: Actual output
    input_size: Number of features in the input data
    hidden_size: Number of neurons in the hidden layer
    output_size: Number of neurons in the output layer
    learning_rate: Learning rate for gradient descent
    epochs: Number of iterations for training
    '''
    # Initialize parameters
    weights_input_hidden, weights_hidden_output, bias_hidden, bias_output = initialize_parameters(input_size, hidden_size, output_size)

    for epoch in range(epochs):
        # Perform forward propagation
        hidden_output, final_output = forward_propagation(X, weights_input_hidden, weights_hidden_output, bias_hidden, bias_output)
        # Compute loss
        loss = compute_loss(Y, final_output)
        # Perform backpropagation
        grads = backpropagation(X, Y, hidden_output, final_output, weights_hidden_output, weights_input_hidden)
        # Update parameters
        weights_input_hidden, weights_hidden_output, bias_hidden, bias_output = update_parameters((weights_input_hidden, weights_hidden_output, bias_hidden, bias_output), grads, learning_rate)

        # Print loss every 1000 epochs
        if epoch % 1000 == 0:
            print(f"Epoch {epoch}, Loss: {loss}")

    return weights_input_hidden, weights_hidden_output, bias_hidden, bias_output

Conclusion:

Congratulations! You've successfully built a neural network from scratch using Python and NumPy. For Full code please go to this link Neural-network and feel free to experiment with different architectures, datasets, and hyperparameters to deepen your understanding of neural networks.

Happy Coding!👨‍💻