DEV Community

Amit
Amit

Posted on

Dev Log D-07 to D-09. My Backprop is/will sucking/suck.

Don't learn here. All code is implementing based on Noob Imagination and poor understanding of Articles.

import mpmath as mpMath 
mpMath.mp.dps = 5
class Neuron:
    def __init__(self,W):
        # Weight is an vector.
        self.W =  W
        self.B = np.random.uniform(-1, 1)
    def forward(self,X):
        self.wX = X.dot(self.W)
        self.f_out = self.wX + self.B
        return self.f_out
    def tanh(self):
        self.out_tanh = mpMath.tanh(self.f_out)
        return self.out_tanh
    def derv(self,w_index,step_v,X):
        u = self.W.copy()
        print(f'Printing old Weight \n{self.W[w_index]}')
        u[w_index] += step_v
        print(f'Printing update weight \n {u[w_index]}')
        _d = (self.f_out - X.dot(u))/step_v
        return _d

neuron = Neuron( np.random.uniform(-1.,1.,size=(784,)).astype(np.float16))
Enter fullscreen mode Exit fullscreen mode
print(neuron.forward(train_data[0]))
print(neuron.derv(2,0.001,train_data[0]))
Enter fullscreen mode Exit fullscreen mode

Slope when step is 0.00001

-707.2305512337132
Printing old Weight 
0.496826171875
Printing update weight 
 0.496826171875
39243.93562717796
Enter fullscreen mode Exit fullscreen mode

As you can see output below when I update weight 0.001 there is 951 slope and If I go too small slope is even large. I don't understand.

-5.366328888592351
Printing old Weight 
-0.47216796875
Printing update weight 
 -0.47119140625
951.3314431337232
Enter fullscreen mode Exit fullscreen mode

Turns out this noob brain of mine forgot to add bais.

 _d = (self.f_out - (X.dot(u)+self.B))/step_v)
Enter fullscreen mode Exit fullscreen mode

so moving on.

Below is the code snippet how I am finding out how my model is predicting.

 def predict(self,X):
        self.inputL_out = self.inputL.forward(X)
        # print(f"Input Layer Output---> \n \t {self.inputL_out}")
        # self.inputL.display_output()
        for l in range(0,len(self.neuronsLayers)):
            if l == 0:
                self.neuronsLayers[l].forward(self.inputL_out)
            else:
                self.neuronsLayers[l].forward(self.neuronsLayers[l-1].f_out)
            print(f'Forward output of layer {l} is :--> {self.neuronsLayers[l].f_out}')
            # self.neuronsLayers[l].display_output1()
            if l == len(self.hidden_layer_dim_arr)-1:
                self.outL_f_out = self.outL.forward(self.neuronsLayers[l].f_out)
                print(f'output of outL \n \t{self.outL_f_out}')
        print("Predicting--->")
        max_index = np.argmax(self.outL_f_out)
        print(f"index of max confidente {max_index} and probability is {self.outL_f_out[max_index]}")
Enter fullscreen mode Exit fullscreen mode

So, My last layer have 10 neurons. each index represent each digit.
like index 2 represent 2.

Now I am stuck on another bug.
I works drev method works for 0.001 but it is 0.0 when I change setup to 0.0001.
I think I should learn basic Numpy and Python. But I will not it is fun doing things this way. Getting stuck on silly bugs.

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Image of Docusign

🛠️ Bring your solution into Docusign. Reach over 1.6M customers.

Docusign is now extensible. Overcome challenges with disconnected products and inaccessible data by bringing your solutions into Docusign and publishing to 1.6M customers in the App Center.

Learn more