DEV Community: Amit

DevLog D-09toD-11. I can't Imagine the backprop.

Amit — Sun, 17 Mar 2024 04:46:14 +0000

Don't Learn here. I think Everything here is exact opposite of what you do in a real job. I am not a profession AI guy. I am a noob.

Forward pass was simple.
I need to pass the input value of one layer to another layers.

But Backpropagation has multiple steps.
I think backpropagation is when you make your network aware of loss. By Aware I mean updating value based on loss of network. I just realized this today(17-03-2024). So Below is the methods that I wrote to train and get loss of network.

 def train(self,train_X,train_Y):
        # pass the train_X to Network to get loss of network or the error in prediction. 
        self.inputL_out = self.inputL.forward(train_X)
        for l in range(0,len(self.neuronsLayers)):
            if l == 0:
                self.neuronsLayers[l].forward(self.inputL_out)
            else:
                self.neuronsLayers[l].forward(self.neuronsLayers[l-1].f_out)
            if l == len(self.neuronsLayers) -1:
                self.outL_f_out = self.outL.forward(self.neuronsLayers[l].f_out) #self.outL_f_out is output of network.

        # now calculate the loss. 
        # I am calculating two type for losses. 1. loss on actual answer 2. Loss on wrong answer. 
        losses = []
        for i in range(0,self.out_dim):
            if train_Y != i:
                if abs(self.outL_f_out[i]) > 0.6:
                    losses.append(abs(self.outL_f_out[i])-0.6) #prediction for other digits other then actual answer should be less then 0.6
                else:
                    losses.append(0)
            else:
                losses.append(0.99 - abs(self.outL_f_out[train_Y])) #loss for actual answer. 

        print(losses)
        print(self.outL_f_out)

Until today I was stuck on derivative and how to update weights.
After wasting two days(2 hours per day) I realized I need to update weight based on loss.

But still how do I update weight & by how much.
And why my network in predicting negative probability.(wrong choice of activation function I think.)

I am weak at math

Dev Log D-07 to D-09. My Backprop is/will sucking/suck.

Amit — Fri, 15 Mar 2024 03:54:15 +0000

Don't learn here. All code is implementing based on Noob Imagination and poor understanding of Articles.

import mpmath as mpMath 
mpMath.mp.dps = 5
class Neuron:
    def __init__(self,W):
        # Weight is an vector.
        self.W =  W
        self.B = np.random.uniform(-1, 1)
    def forward(self,X):
        self.wX = X.dot(self.W)
        self.f_out = self.wX + self.B
        return self.f_out
    def tanh(self):
        self.out_tanh = mpMath.tanh(self.f_out)
        return self.out_tanh
    def derv(self,w_index,step_v,X):
        u = self.W.copy()
        print(f'Printing old Weight \n{self.W[w_index]}')
        u[w_index] += step_v
        print(f'Printing update weight \n {u[w_index]}')
        _d = (self.f_out - X.dot(u))/step_v
        return _d

neuron = Neuron( np.random.uniform(-1.,1.,size=(784,)).astype(np.float16))

print(neuron.forward(train_data[0]))
print(neuron.derv(2,0.001,train_data[0]))

Slope when step is 0.00001

-707.2305512337132
Printing old Weight 
0.496826171875
Printing update weight 
 0.496826171875
39243.93562717796

As you can see output below when I update weight 0.001 there is 951 slope and If I go too small slope is even large. I don't understand.

-5.366328888592351
Printing old Weight 
-0.47216796875
Printing update weight 
 -0.47119140625
951.3314431337232

Turns out this noob brain of mine forgot to add bais.

 _d = (self.f_out - (X.dot(u)+self.B))/step_v)

so moving on.

Below is the code snippet how I am finding out how my model is predicting.

 def predict(self,X):
        self.inputL_out = self.inputL.forward(X)
        # print(f"Input Layer Output---> \n \t {self.inputL_out}")
        # self.inputL.display_output()
        for l in range(0,len(self.neuronsLayers)):
            if l == 0:
                self.neuronsLayers[l].forward(self.inputL_out)
            else:
                self.neuronsLayers[l].forward(self.neuronsLayers[l-1].f_out)
            print(f'Forward output of layer {l} is :--> {self.neuronsLayers[l].f_out}')
            # self.neuronsLayers[l].display_output1()
            if l == len(self.hidden_layer_dim_arr)-1:
                self.outL_f_out = self.outL.forward(self.neuronsLayers[l].f_out)
                print(f'output of outL \n \t{self.outL_f_out}')
        print("Predicting--->")
        max_index = np.argmax(self.outL_f_out)
        print(f"index of max confidente {max_index} and probability is {self.outL_f_out[max_index]}")

So, My last layer have 10 neurons. each index represent each digit.
like index 2 represent 2.

Now I am stuck on another bug.
I works drev method works for 0.001 but it is 0.0 when I change setup to 0.0001.
I think I should learn basic Numpy and Python. But I will not it is fun doing things this way. Getting stuck on silly bugs.

Dev Log D-05&06. Forward Propagation Implemented.

Amit — Tue, 12 Mar 2024 07:12:08 +0000

I am a noob don't learn from this. But Please point out my mistakes.

Mistake in Previous code I used class variable instead of instance variable.

import mpmath as mpMath 
mpMath.mp.dps = 5
class Neuron:
    def __init__(self,W):
        # Weight is an vector.
        self.W =  W
        self.B = np.random.uniform(-1, 1)
    def forward(self,X):
        self.wX = X.dot(self.W)
        self.f_out = self.wX + self.B
        return self.f_out
    def tanh(self):
        self.out_tanh = mpMath.tanh(self.f_out)
        return self.out_tanh

neuron = Neuron( np.random.uniform(-1.,1.,size=(784,)).astype(np.float16))

class NLayer:
    def __init__(self,input_dimension,number_of_neurons):
        self.neurons = []
        # A Layer have input and output dimensions.
        # I am considering the neurons as output dimensions. 
        # and Input dimension will be the dimension/lenght of X like 784 in mnist data. 
        print(f'creating NLayer with {input_dimension} input dimension and {number_of_neurons} neurons \n')
        self.num_of_n = number_of_neurons
        self.input_length = input_dimension
        for n in range(0,number_of_neurons):
            # Initialze the weights and bias and create neurons
            W =  np.random.uniform(-1.,1.,size=(self.input_length,)).astype(np.float16)
            self.neurons.append(Neuron(W))

    def forward(self,X):
        # print(f'Shape of X {X.shape}')
        for n in range(self.num_of_n):
            self.neurons[n].forward(X)
            self.neurons[n].tanh()

        # print(f'{len(self.neurons)}')
        self.f_out = np.array([float(neuron.out_tanh) for neuron in self.neurons])

        return self.f_out
            # print(self.neurons[n].out_tanh)
    def display_output(self):
        # can be used when output dimension is a perfect square
        grid_size = int(np.sqrt(len(self.f_out)))
        image_matrix = np.reshape(self.f_out, (grid_size, grid_size))
        plt.imshow(image_matrix, cmap='gray')
        plt.colorbar()
        plt.title("Array Visualization")
        plt.show()
    def display_output1(self):
        indices = np.arange(len(self.f_out))
        plt.scatter(indices, self.f_out, marker='o', color='b')
        plt.title("Array Visualization")
        plt.xlabel("Index")
        plt.ylabel("Output Values")
        plt.show()
        print(indices.dtype)


# layer_1 = NLayer(784,784)

class NNetwork:
    # A network have layers that

    def __init__(self,input_dim,hidden_layer_dim_arr):
        self.neuronsLayers = []
        self.input_dim = input_dim
        self.hidden_layer_dim_arr = hidden_layer_dim_arr
        self.inputL = NLayer(input_dim,input_dim)
        for d in range(0,len(hidden_layer_dim_arr)):
            if d == 0:
                self.neuronsLayers.append(NLayer(input_dim,hidden_layer_dim_arr[d]))
            else:
                self.neuronsLayers.append(NLayer(hidden_layer_dim_arr[d-1],hidden_layer_dim_arr[d]))
    def train(self):
        # update weights 
        pass
    def test(self):
        # Don't update weights. check weights.
        pass
    def predict(self,X):
        self.inputL_out = self.inputL.forward(X)
        # print(f'Forward out of Input layers is {self.inputL_out}')
        self.inputL.display_output1()
        for l in range(0,len(self.neuronsLayers)):
            if l == 0:
                self.neuronsLayers[l].forward(self.inputL_out)
            else:
                self.neuronsLayers[l].forward(self.neuronsLayers[l-1].f_out)
            print(f'Forward output of layer {l} is :--> {self.neuronsLayers[l].f_out}')
            self.neuronsLayers[l].display_output1()

    def __str__(self):
        print(f'Input Dimension {self.input_dim}. Hidden layer dim array {self.hidden_layer_dim_arr}')
        return f'Neural Network with {len(self.neuronsLayers)} hidden layer' 
nn1 = NNetwork(input_dim=784,hidden_layer_dim_arr=[10,10])
print(nn1)
nn1.predict(train_data[0])

Output of First layer(Input Layer).
The out-put is look like a number when I display forward out of neurons. The below output is tanh()

Output of 2 Layer

Output of 3 Layer

I think Now I should write my training method & how to update weights.

Dev Log (D-01 to D-04). Implementing neural networks from scratch.

Amit — Sun, 10 Mar 2024 05:35:30 +0000

Disclaimer:-I am a noob.(So don't use this if you are learning something).

Getting MNIST data (found this code in https://github.com/tinygrad/tinygrad).

import numpy as np
import gzip
import requests

def fetch(url):
    response = requests.get(url)
    return response.content

def parse(file):
    return np.frombuffer(gzip.decompress(file), dtype=np.uint8).copy()
# Thank to https://github.com/tinygrad/tinygrad mnist example
BASE_URL = "https://storage.googleapis.com/cvdf-datasets/mnist/"
X_train = parse(fetch(f"{BASE_URL}train-images-idx3-ubyte.gz"))[0x10:].reshape((-1, 28*28)).astype(np.float64)
Y_train = parse(fetch(f"{BASE_URL}train-labels-idx1-ubyte.gz"))[8:].astype(np.int8)
X_test = parse(fetch(f"{BASE_URL}t10k-images-idx3-ubyte.gz"))[0x10:].reshape((-1, 28*28)).astype(np.float64)
Y_test = parse(fetch(f"{BASE_URL}t10k-labels-idx1-ubyte.gz"))[8:].astype(np.int8)

# Print the shapes of the loaded data
print("Shape of X_train:", X_train.shape)
print("Shape of Y_train:", Y_train.shape)
print("Shape of X_test:", X_test.shape)
print("Shape of Y_test:", Y_test.shape)

Implementing NN. getting help from help from https://developer.ibm.com/articles/l-neural/

class Neuron:
    def __init__(self):
        # Weight is an vector.
        self.W =  np.random.uniform(-1.,1.,size=(784,)).astype(np.float64)
    def forward(self,X):
        self.wX = X.dot(self.W)
        return self.wX

    def sigmoid(self):
        if self.wX > 0:   
            z = np.exp(-self.wX)
            return 1/(1+z)
        else:
            z = np.exp(self.wX)
            return z/(1+z)
    def sigmoid2(self):
        return np.exp(self.wX) / (1 +  np.exp(self.wX))

    def tanH(self):
        z = np.exp(2*self.wX)
        return (z-1)/(z+1)

# neuron = Neuron()
# print(neuron.forward(train_data[4]))
# print(neuron.sigmoid())

I am stuck at my activation function bcz dot product of Input and weights is to large.

class NLayer:
    neurons = []
    def __init__(self,number_of_neurons):
        for n in range(0,number_of_neurons):
            self.neurons.append(Neuron())

    def forward(self,X):
        for n in self.neurons:
            n.forward(X)
            print(n.tanH())

layer_1 = NLayer(784)
layer_1.forward(train_data[0])

class NNetwork:
    neuronsLayers = []
    def __init__(self):
        pass
    def train(self):
        # Train and update weights. 
        pass
    def test(self):
        # Don't update weights. check weights.
        pass
    def predict(self):
        # Run Model. 
        pass

I am still stuck at the error when executing the activation function code. OverFollow due to number being too large.