DEV Community: Swayam Singh

Project-1: Cats and Dogs Classification using Logistic Regression

Swayam Singh — Wed, 29 Dec 2021 21:53:52 +0000

Overview

Welcome to the 4^th article of Demystifying Machine Learning series. In this article, we'll be going to make our first project of prediction of cats or dogs from their respective images using Logistic Regression. We are going to use scikit-learn library for this project but I already covered Logistic Regression in great depth in the 3^rd article of this series.

The main focus of this sample project is towards data collection and data preprocessing rather than training a Logistic Regression model. The reason is that I showed how to set different hyperparameters in order to achieve satisfying results in 3^rd article of this series and often times whenever working on a project 80% of the time one gonna spent is on data preprocessing.

The project is going to be a Jupyter Notebook. You can grab that notebook from here. Everything you need from code to explanation is already provided in that notebook. This article will tell you few points about the dataset and some techniques we are going to use.

About Dataset

The dataset we are going to use comes from Kaggle. You can view that dataset on Kaggle from here. This dataset only contains images of cats and dogs separated as train and test already for model training.

In the notebook we'll be training our model from images of cats and dogs inside train folder and validate our model using images inside test folder.

The dataset in total contains 11250 images of cats and 11250 images of dogs in train folder. Similarly, it contains 1250 images of cats and 1250 images of dogs in test folder. There is no labels file so we had to create it by ourselves to let our model understand which image is of a cat and which is of a dog.

How to open?

As mentioned earlier this project is basically a Jupyter notebook and you can get that notebook from Github via this link. You don't need to download the dataset manually because inside the notebook you'll find a function that will download the dataset for you automatically and save it in your current working directory then another function will extract that dataset into a folder.

Now you can either upload this notebook on Google Colab or any other cloud service and run it there or download it on your system and create a virtual environment then run it after installing all the required dependencies. Both ways it's totally gonna work fine without any issue.

Using cloud service can be easier because you don't have to manage virtual environments or install required dependencies but you also need to keep in mind that they provide limited RAM storage so keep a keen eye on your memory management and if it's getting out of your provided range then change the number of samples in training set inside notebook.

Note: All the code sample is done in a notebook with a proper explanation of each cell

Conclusion

That's it for this supplement article for our first project. Now head to the repository for this project and grab the notebook. If you just want to take a look at the notebook and it's not opening on Github then click on this link to explore the entire notebook on nbviewer.

I hope you have learnt something new, for more updates on upcoming articles get connected with me through Twitter and stay tuned for more.

Logistic Regression

Swayam Singh — Sat, 18 Dec 2021 11:36:11 +0000

Overview

Welcome to the 3^rd article of the Demystifying Machine Learning series. In this article, we are going to discuss a supervised classification algorithm Logistic Regression, how it works, why it's important, mathematics behind the scenes, linear and non-linear separation and Regularization.

A good grasp of Linear Regression is needed to understand this algorithm and luckily we already covered it, for reference you can read it from here. Logistic Regression builds the base of Neural Networks, so it gets very important to understand the terms and working of this algorithm.

Logistic Regression is not a regression algorithm, its name does consist of the word "Regression" but it's a classification algorithm.

What is Logistic Regression?

Logistic Regression also known as Perceptron algorithm is a supervised classification algorithm i.e. we teach our hypothesis with categorical labelled data and it predicts the classes (or categories) with some certain probability. The reason this algorithm is called Logistic Regression is maybe that it's working is pretty similar to that of Linear Regression and the term "Logistic" is because we use a Logistic function in this algorithm (we'll going to see it later). The difference is that with Linear Regression we intend to predict the continuous values but with Logistic Regression we want to predict the categorical value. Like whether the student gets enrolled in a university or not, if the picture contains a cat or not, etc.

Here's a representation of how Logistic Regression classifies the data points.

We can see that the blue points are separated from the orange points through a line and we call this line a decision boundary. Basically, with Logistic Regression we separate the classes (or categories) with the help of decision boundaries, they can be linear or non-linear.

Working of Logistic Regression

Let's revise what we learnt in Linear Regression, we initialise the parameters with all 0s, then with the help of Gradient Descent calculate the optimal parameters by reducing the cost function and lastly draw the hypothesis to predict the continuous-valued target.

But here we don't need continuous values, instead, we want to output the probability that lies between [0,1]. So the question arises, how we can get probability or a number between [0,1] from the continuous values of Linear Regression?

Sigmoid function

Sigmoid function is a type of logistic function which takes a real number as input and gives out a real number between [0,1] as output.

So basically we'll generate the continuous values using Linear Regression and convert those continuous values into probability i.e. between [0,1] by passing through sigmoid function.

So in the end our final hypothesis will look like this:

This hypothesis is different from the hypothesis of Linear Regression. Yeah looks fair enough, let me give you a visualisation about overall how Logistic Regression works.

Note:- X0 is basically 1, we will it later why?

So it's very clear from the above representation that the part behind the sigmoid function is very similar to that of Linear Regression. Let's now move ahead and define the cost function for Logistic Regression.

Cost function

Our hypothesis is different from that of Linear Regression, so we need to define a new cost function. We already learnt in our 2nd article about what is cost function is and how to define one for our algorithm. Let's use those concepts and define one for Logistic Regression.

For simplicity let's consider the case of binary classification which means our target value will be either 1(True) or 0(False). For example, the image contains a cat (1, True) or the image does not contain a cat (0, False). This means that our predicted values will also be either 0 or 1.

Let me first show you what is the cost function for Logistic Regression and then we'll try to understand its derivation.

Combining both the conditions and taking their mean for m samples in a dataset:

The equation shown above is of the cost function for Logistic Regression and it looks very different than that of Linear Regression, let's break it down and understand how we came up with the above cost function? Get ready, probability class is about to begin...

There's a negative sign in the original cost function because when training the algorithm we want probabilities to be large but here we are representing it to minimize the cost.

minimise loss => max log probability

Okay, that's a lot of maths, but it's all basics. Focus on the general form in the above equation and that's more than enough to understand how we came up with such a complex looking cost function. Let's see how to calculate the cost for m examples for some datasets:

Gradient Descent

We already covered the working of gradient descent in our 2nd article, you can refer to it for revision. In this section, we'll be looking at formulas of gradients and updating the parameters.

The gradient of the cost is a vector of the same length as θ where the jth parameter (for j=0,1,⋯,n) is defined as follows:

Calculation of gradients from cost function is demonstrated in 2nd Article.

As we can see that the formula for calculating gradients is pretty similar to that of Linear Regression but note that the values for hθ(xⁱ) are different due to the use of sigmoid function.

After calculating gradients we can simultaneously update our parameter θ as :

Great now we have all the ingredients for writing our own Logistic Regression from scratch, Let's get started with it in the next section. Till now have a break for 15 minutes cause you just studied a hell of lot of maths by now.

Code Implementation

In this section, we'll be writing our LogisticRegression class using Python.

Note: You can find all the codes for this article from here. It's highly recommended to follow the Jupyter notebook while going through this section.

Let's begin 🙂

Let me give you a basic overall working of this class. Firstly it'll take your feature and target arrays as input then it'll normalize the features around mean (if you want to) and add an extra column of all 1s to your feature array for the bias term, as we know from Linear Regression that y=wx+b. So this b gets handled by this extra column of 1s with matrix multiplication of features and parameters arrays.

for example:

Then it initializes the parameter array with all 0s after that training loop starts till the epoch count and it calculates the cost and gradient for certain parameters and simultaneously keep updating the parameters with a certain learning rate.

class LogisticRegression:
    def __init__(self) -> None:
        self.X = None
        self.Y = None
        self.parameters = None
        self.cost_history = []
        self.mu = None
        self.sigma = None

    def sigmoid(self, x):
        z = np.array(x)

        g = np.zeros(z.shape)

        g = 1/(1 + np.exp(-z) ) 

        return g

    def calculate_cost(self):
        """
        Returns the cost and gradients.
        parameters: None

        Returns:
            cost : Caculated loss (scalar).
            gradients: array containing the gradients w.r.t each parameter

        """

        m = self.X.shape[0]

        z = np.dot(self.X, self.parameters)
        z = z.reshape(-1)
        z = z.astype(np.float128, copy=False)
        y_hat = self.sigmoid(z)
        cost = -1 * np.mean(self.Y*(np.log(y_hat)) + (1-self.Y)*(np.log(1-y_hat)))

        gradients = np.zeros(self.X.shape[1])

        for n in range(len(self.parameters)):
            temp = np.mean((y_hat-self.Y)*self.X[:,n])
            gradients[n] = temp

        # Vectorized form
        #  gradients = np.dot(self.X.T, error)/m 

        return cost, gradients


    def init_parameters(self):
        """
        Initialize the parameters as array of 0s
        parameters: None

        Returns: None

        """
        self.parameters = np.zeros((self.X.shape[1],1))


    def feature_normalize(self, X):
        """
        Normalize the samples.
        parameters: 
            X : input/feature matrix

        Returns:
            X_norm : Normalized X.

        """
        X_norm = X.copy()
        mu = np.mean(X, axis=0)
        sigma = np.std(X, axis=0)

        self.mu = mu
        self.sigma = sigma

        for n in range(X.shape[1]):
            X_norm[:,n] = (X_norm[:,n] - mu[n]) / sigma[n]
        return X_norm

    def fit(self, x, y, learning_rate=0.01, epochs=500, is_normalize=True, verbose=0):
        """
        Iterates and find the optimal parameters for input dataset
        parameters: 
            x : input/feature matrix
            y : target matrix
            learning_rate: between 0 and 1 (default is 0.01)
            epochs: number of iterations (default is 500)
            is_normalize: boolean, for normalizing features (default is True)
            verbose: iterations after to print cost

        Returns:
            parameters : Array of optimal value of weights.

        """
        self.X = x
        self.Y = y
        self.cost_history = []
        if self.X.ndim == 1: # adding extra dimension, if X is a 1-D array
            self.X = self.X.reshape(-1,1)
            is_normalize = False
        if is_normalize:
            self.X = self.feature_normalize(self.X)
        self.X = np.concatenate([np.ones((self.X.shape[0],1)), self.X], axis=1)
        self.init_parameters()

        for i in range(epochs):
            cost, gradients = self.calculate_cost()
            self.cost_history.append(cost)
            self.parameters -= learning_rate * gradients.reshape(-1,1)

            if verbose:
                if not (i % verbose):
                    print(f"Cost after {i} epochs: {cost}")
        return self.parameters


    def predict(self,x, is_normalize=True, threshold=0.5):
        """
        Returns the predictions after fitting.
        parameters: 
            x : input/feature matrix

        Returns:
            predictions: Array of predicted target values.

        """
        x = np.array(x, dtype=np.float64) # converting list to numpy array
        if x.ndim == 1:
            x = x.reshape(1,-1)
        if is_normalize:
            for n in range(x.shape[1]):
                x[:,n] = (x[:,n] - self.mu[n]) / self.sigma[n]
        x = np.concatenate([np.ones((x.shape[0],1)), x], axis=1)
        return [1 if i > threshold else 0  for i in self.sigmoid(np.dot(x,self.parameters))]

This code looks pretty similar to that of Linear Regression using Gradient Descent. If you are following this series you'll be pretty familiar with this implementation. Still, I like to point out a few methods of this class:-

sigmoid: We added this new method to calculate the sigmoid of the continuous values generated from the linear hypothesis i.e. from θ^TX to get the probabilities.
calculate_cost: We change the definition of this function because our cost function is changed too, it's not confusing if you are well aware of the formulas I gave and the numpy library then it won't be difficult for you to understand.
predict: This function takes the input and returns the array of predicted values 0 and 1. There's an extra parameter threshold which had a default value of 0.5, if the predicted value > 0.5 then it'll predict 1 otherwise 0 for the predicted array. You can change this threshold according to your confidence level.

Trying it out on a dataset

In this sub-section, we will use our class on the dataset to check how it's working.

All the codes and implementations are provided in this jupyter notebook, follow it for better understanding in this section.

For the dataset, we have records of students' marks for some Exam1 and Exam2 and the target column represents whether they get admitted into the university or not. Let's visualize it using a plot:

So what we basically want from Logistic Regression is to tell us whether a certain student with some scores of Exam1 and Exam2 is admitted or not. Let's create an instance of the LogisticRegression class and try it out.

Firstly I'm going to find the optimal parameters for this dataset and I'm going to show you two ways of doing it.

Using our custom class

Using Scipy's optimize module

Sometimes using gradient descent takes a lot of time so for time-saving, I'll show you how you can easily find the parameters by just using scipy.optimize.minimize function bypassing the cost function into it.

Firstly I appended an extra column of 1s for bias term then pass my costFunction, initial_theta (initially 0s) and my X and Y as arguments. It easily calculated the optimal parameters in 0.3 seconds much faster than gradient descent which took about 6.5 seconds.

Note: costFunction is similar to what we have in our class method as calculate_cost, I just put it outside to show you the work of scipy.optimize.minimize function.

Great now let's see how well it's performed by printing out its accuracy on the training set.

Hmmm, around 89%, it seems good although there are a few algorithms that we'll be covering in future that can perform way much better than this. Now let me show you its decision boundary, as we can see that we didn't perform any polynomial transformation (for more refer to article 2) on our input features so the decision boundary is going to be a straight line.

That's so great we just implemented our LogisticRegression class on the student's dataset. Let's move ahead and understand the problem of overfitting in the next section. Till then have a short 5-minute break.

Problem of Overfitting

In order to understand overfitting in Logistic Regression, I'll show you an implementation of this algorithm on another dataset where we need to fit a non-linear decision boundary. Let's visualize our 2nd dataset on the graph:

As we can see, it's not linearly separable data so we need to fit a non-linear decision boundary. If you went through the 2nd article of this series then you probably know how we do this, but in brief, we take our original features and apply polynomial transformations on them, like squaring, cubing or multiplying with each other to obtain new features and then training our algorithm on those new features results in non-linear classification.

In the notebook you'll find a function mapFeature that take individual features as input and return new transformed features.

If you wanna know how it's working then consider referring to the notebook and it's recommended to follow it while reading this article.

After getting the new transformed features and following the exact steps we followed in the above section, you'll be able to print out its decision boundary that will look something like this:

After seeing it, you probably may say that "WOW!, it performed so well almost classified all the training points". Well, it does seem good but in reality, it's worst. Our hypothesis fitted so well on the training set that it loses the generality that means if we provide a new set of points that is not in the training set then our hypothesis will not be able to classify it clearly.

In short, it's necessary to maintain the generality in our hypothesis so that it can perform well on the data it is never seen. Regularization is the way to achieve it. Let's see how to maintain generality using Regularization in the next section.

Regularization

In this section, we'll be discussing how to implement regularization. Overfitting occurs when the algorithm provides heavy parameters to some features according to the training dataset and hyperparameters. This makes those features dominant in the overall hypothesis and lead to a nice fit in the training set but not so good on the samples outside the training set.

The plan is to add the square of parameters by multiplying them with some big number (λ) to the cost function because our algorithms' main motive is to decrease the cost function, so in this way, the algorithm will end up giving the small parameters just to cancel the effect addition of parameters by multiplying with a large number (λ). So our final cost function gets modified to:

Note: We denote the bias term as θ0 and it's not needed to regularized the bias term that's why we are only considering only θ1 to θn parameters.

Since our cost function is changed that's why our formulas for gradients were also get affected. The new formula for the gradient are:

The new formulas for gradients can be easily derived by partially differentiating the new cost function J(θ) w.r.t to some θj.

Calculating of gradients from cost function is demonstrated in 2nd Article.

λ is known as a regularization parameter and it should be greater than 0. Large value of λ leads to underfitting and very small values lead to overfitting, so you need to pick the right one for your dataset through iterating on some sample values.

Implementing Regularization on `LogisticRegression` class

We only need to modify the calculate_cost method because only this method is responsible for calculating both cost and gradients. The modified version is shown below:

class RegLogisticRegression:
    def __init__(self) -> None:
        self.X = None
        self.Y = None
        self.parameters = None
        self.cost_history = []
        self.mu = None
        self.sigma = None

    def sigmoid(self, x):
        z = np.array(x)

        g = np.zeros(z.shape)

        g = 1/(1 + np.exp(-z) ) 

        return g

    def sigmoid_derivative(self, x):
        derivative = self.sigmoid(x) * (1 - self.sigmoid(x))
        return derivative

    def calculate_cost(self, lambda_):
        """
        Returns the cost and gradients.
        parameters: None

        Returns:
            cost : Caculated loss (scalar).
            gradients: array containing the gradients w.r.t each parameter

        """

        m = self.X.shape[0]

        z = np.dot(self.X, self.parameters)
        z = z.reshape(-1)
        z = z.astype(np.float128, copy=False)
        y_hat = self.sigmoid(z)
        cost = -1 * np.mean(self.Y*(np.log(y_hat)) + (1-self.Y)*(np.log(1-y_hat))) + lambda_ * (np.sum(self.parameters[1:]**2))/(2*m)

        gradients = np.zeros(self.X.shape[1])

        for n in range(len(self.parameters)):
            if n == 0:
                temp = np.mean((y_hat-self.Y)*self.X[:,n])
            else:
                temp = np.mean((y_hat-self.Y)*self.X[:,n]) + lambda_*self.parameters[n]/m
            gradients[n] = temp

        # gradients = np.dot(self.X.T, error)/m

        return cost, gradients


    def init_parameters(self):
        """
        Initialize the parameters as array of 0s
        parameters: None

        Returns:None

        """
        self.parameters = np.zeros((self.X.shape[1],1))


    def feature_normalize(self, X):
        """
        Normalize the samples.
        parameters: 
            X : input/feature matrix

        Returns:
            X_norm : Normalized X.

        """
        X_norm = X.copy()
        mu = np.mean(X, axis=0)
        sigma = np.std(X, axis=0)

        self.mu = mu
        self.sigma = sigma

        for n in range(X.shape[1]):
            X_norm[:,n] = (X_norm[:,n] - mu[n]) / sigma[n]
        return X_norm

    def fit(self, x, y, learning_rate=0.01, epochs=500, lambda_=0,is_normalize=True, verbose=0):
        """
        Iterates and find the optimal parameters for input dataset
        parameters: 
            x : input/feature matrix
            y : target matrix
            learning_rate: between 0 and 1 (default is 0.01)
            epochs: number of iterations (default is 500)
            is_normalize: boolean, for normalizing features (default is True)
            verbose: iterations after to print cost

        Returns:
            parameters : Array of optimal value of weights.

        """
        self.X = x
        self.Y = y
        self.cost_history = []
        if self.X.ndim == 1: # adding extra dimension, if X is a 1-D array
            self.X = self.X.reshape(-1,1)
            is_normalize = False
        if is_normalize:
            self.X = self.feature_normalize(self.X)
        self.X = np.concatenate([np.ones((self.X.shape[0],1)), self.X], axis=1)
        self.init_parameters()

        for i in range(epochs):
            cost, gradients = self.calculate_cost(lambda_=lambda_)
            self.cost_history.append(cost)
            self.parameters -= learning_rate * gradients.reshape(-1,1)

            if verbose:
                if not (i % verbose):
                    print(f"Cost after {i} epochs: {cost}")
        return self.parameters


    def predict(self,x, is_normalize=True, threshold=0.5):
        """
        Returns the predictions after fitting.
        parameters: 
            x : input/feature matrix

        Returns:
            predictions : Array of predicted target values.

        """
        x = np.array(x, dtype=np.float64) # converting list to numpy array
        if x.ndim == 1:
            x = x.reshape(1,-1)
        if is_normalize:
            for n in range(x.shape[1]):
                x[:,n] = (x[:,n] - self.mu[n]) / self.sigma[n]
        x = np.concatenate([np.ones((x.shape[0],1)), x], axis=1)
        return [1 if i > threshold else 0  for i in self.sigmoid(np.dot(x,self.parameters))]

Now we have our regularized version of RegLogisticRegression class. Let's address the previous problem of overfitting on polynomial regression by using a set of values for λ to pick the right one.

I can say that λ=1 and λ=10 looks pretty good and they both are able to maintain the generality in hypothesis, the curve is more smooth and less wiggling type. But we can see that as we keep increasing the value if λ the more our hypothesis starts to underfit the data. It basically means that it starts to perform even worst on the training set. Let's visualise the underfitting by plotting cost functions for each λ

We can see that as λ increases cost also increases. So it's advised to select the value for λ carefully according to your custom dataset.

Conclusion

Great work everyone, we successfully learnt and implemented Logistic Regression. Most people don't write their Machine Learning algorithm from scratch instead they use libraries like Scikit-Learn. Scikit-Learn contains wrappers for many Machine Learning algorithms and it's really flexible and easy to use. But it's not harmful to know about the algorithm you're going to use and the best way of doing it is to understand the underlying mathematics and implement it from scratch.

So in the next article, we'll be making a classification project using the Scikit-Learn library and you'll see how easy it is to use for making some really nice creative projects.

I hope you have learnt something new, for more updates on upcoming articles get connected with me through Twitter and stay tuned for more. Till then enjoy your day and keep learning.

Linear Regression using Gradient Descent

Swayam Singh — Fri, 03 Dec 2021 09:27:39 +0000

Overview

This is the second article of Demystifying Machine Learning series, frankly, it is basically the sequel of our previous article where we explained Linear Regression using Normal equation. In this article we'll be exploring another optimizing algorithm known as Gradient Descent, how it works, what is a cost function, mathematics behind gradient descent, Python implementation, Regularization and some extra topics like polynomial regression and using regularized polynomial regression.

How Gradient Descent works (Intuition)

Gradient descent is basically an iterative optimizing algorithm i.e. we can use it to find the minimum of a differential function. Intuitively we can think of a situation where you're standing somewhere on a mountain and you want to go to the foot of that mountain as fast as possible. Since we're in a random position on the mountain, one way is to move along the steepest direction while taking small steps (taking large steps towards steepest direction may get you injured) and we'll see later that taking large steps is also not good for algorithm too.

Now that's something similar we are going to do for optimizing our hypothesis to give the least error. As we learnt in the previous article that we need to find the optimal parameters θ, that helps us to calculate the best possible hyperplane to fit the data. In that article, we used the Normal equation to directly find those parameters but here that's not gonna happen.

We are randomly going to pick the parameters and then calculate the cost function that will tell us how much error those random parameters are giving, then we use gradient descent to find the minimum of that cost function and optimize those random parameters into the optimal ones.

Cost Function

A cost function is basically a continuous and differentiable function that tells how good an algorithm is performing by returning the amount of error as output. The lesser the error, the better the algorithm is doing that's why we randomly generate the parameters and then keep changing them in order to reach the minimum of that cost function.

Now let's define the cost function for Linear regression. First, we need to think that in linear regression how we can calculate the error. Mathamtically error is the difference between the original value and the calculated value. Luckily we can use this definition here. Our original values are the target matrix itself and the calculated values are the predictions from our hypothesis.

We simply subtract the original target value from the predicted target value and take the square of them as the error for a single sample. Ultimately we need to find the squared error for all the samples in the dataset and take their mean as our final cost for a certain hypothesis. Squaring the difference helps in avoiding the condition when the negative and positive errors nullify each other in the final hypothesis's cost.

This error function is also known as Mean Square Error (MSE).

So mathematically let's say we have m number of samples in the dataset then:

It's important to notice that our cost function J(θ) is depend upon the parameters θ because our y(target) and X are fixed, the only varying quantity are the parameters θ and it makes sense because that's how gradient descent will help us in finding the appropriate parameters for a minimum of the cost function.

Mathematics of Gradient Descent

Time to talk Calculus.

Before diving into the algorithm let's first talk about what is a Gradient?

Gradient of a differentiable function is a vector field whose value at a certain point is a vector whose components are the partial derivatives of that function at that same point. Alright so many big words let's break them down and try to understand what it really is?

Mathematically suppose you have a function f(x,y,z) then the gradient at some point will be the vector whose components are going to be the partial derivatives of f(x,y,z) w.r.t to x,y and z at that point.

Property: At a certain point, the gradient vector always points towards the direction of the greatest increase of that function.

Since we need to go in the direction of greatest decrease that's why we follow the direction of negative of the gradient vector.

Gradient vector is always perpendicular to the contour lines of the graph of a function (we'll be dealing Contour graphs later)

Let's visualize the gradient concept using graphs. Say a function f(x,y) as:

If we plot the above graph, it'll look something like this:

If you're aware of vector calculus, then you probably know that Contour plots are very useful for working with 3D curves. A contour plot is basically a 2D graph that is the sliced version of a 3D plot along the z-axis at regular intervals, so if we graph the Contour plot of the above function then it'll look something like:

Now, this graph makes it really clear that gradient always points in the direction of the greatest increase of the function, as we can see that the black arrows represent the direction of the gradient and the red arrow represent the direction where we need to move in our cost function to reach the minimum.

Great now we know that in order to reach the minimum we need to move in the opposite direction of the gradient that is in the -∇f(θ) direction and keep updating our initial random parameters accordingly.

θ is the matirx of all parameters θs

θj is the parameter for j^th feature

J(θ) is the cost function

α is learning rate

Everything seems obvious instead of this symbol α . It's known as learning rate, remember we discussed that we need to take small steps, α makes sure that our algorithm should take small steps for reaching the minimum. The learning rate is always less than 1.

But what if we keep a large learning rate?

As we see in the above figure that our cost function will not able to reach a minimum if we take large learning rates and its results in an increment of loss instead of decreasing it as represented below.

Applying Gradient descent to cost function

In this section, we'll be deriving the formulas for gradients so that we can directly use those formulas in Python implementation. Since we already have our cost function as:

expanding Xⁱ into individual n features as [Xⁱ1, Xⁱ2, Xⁱ3, ....., Xⁱn ] then:

This form will be easier to understand the calculation of gradients, let's compute them for each θj.

so basically we can write the partial derivative of cost function w.r.t to any θj as :

Now we can loop over each θj from 0 to n and update them as :

Note: θ0 represent the bias term

That's great, now we have all the tools we need let's jump straight into the code and implement this algorithm in Python.

Python Implementation

In this section, we'll be using Python and the formulas we derived in the previous section to create a Python class that will be able to perform Linear Regression by using Gradient Descent as an optimizing algorithm to work on a dataset.

Note: All the code files can be found on Github through this link.

And it's highly recommended to follow the notebook along with this section for better understanding.

Before we dive into writing code one important observation is to keep in mind that before using gradient descent, it's always helpful to normalize the features around its mean. The reason is that initially in the dataset we can have many independent features and they can have way different values, like on average number of bedrooms can be 3-4 but the area of the house feature can have way large values. Normalizing makes all the values of different features lie on a comparable range and it also makes it easier for the algorithm to identify the patterns.

class LinearRegression:
    def __init__(self) -> None:
        self.X = None
        self.Y = None
        self.parameters = None
        self.cost_history = []
        self.mu = None
        self.sigma = None

    def calculate_cost(self):
        """
        Returns the cost and gradients.
        parameters: None

        Returns:
            cost : Caculated loss (scalar).
            gradients: array containing the gradients w.r.t each parameter

        """

        m = self.X.shape[0]

        y_hat = np.dot(self.X, self.parameters)
        y_hat = y_hat.reshape(-1)
        error = y_hat - self.Y

        cost = np.dot(error.T, error)/(2*m) # Modified way to calculate cost

        gradients = np.zeros(self.X.shape[1])

        for i in range(self.X.shape[1]):
            gradients[i] = np.mean(error * self.X[:,i])

        return cost, gradients


    def init_parameters(self):
        """
        Initialize the parameters as array of 0s
        parameters: None

        Returns:None

        """
        self.parameters = np.zeros((self.X.shape[1],1))


    def feature_normalize(self, X):
        """
        Normalize the samples.
        parameters: 
            X : input/feature matrix

        Returns:
            X_norm : Normalized X.

        """
        X_norm = X.copy()
        mu = np.mean(X, axis=0)
        sigma = np.std(X, axis=0)

        self.mu = mu
        self.sigma = sigma

        for n in range(X.shape[1]):
            X_norm[:,n] = (X_norm[:,n] - mu[n]) / sigma[n]
        return X_norm

    def fit(self, x, y, learning_rate=0.01, epochs=500, is_normalize=True, verbose=0):
        """
        Iterates and find the optimal parameters for input dataset
        parameters: 
            x : input/feature matrix
            y : target matrix
            learning_rate: between 0 and 1 (default is 0.01)
            epochs: number of iterations (default is 500)
            is_normalize: boolean, for normalizing features (default is True)
            verbose: iterations after to print cost

        Returns:
            parameters : Array of optimal value of weights.

        """
        self.X = x
        self.Y = y
        self.cost_history = []
        if self.X.ndim == 1: # adding extra dimension, if X is a 1-D array
            self.X = self.X.reshape(-1,1)
            is_normalize = False
        if is_normalize:
            self.X = self.feature_normalize(self.X)
        self.X = np.concatenate([np.ones((self.X.shape[0],1)), self.X], axis=1)
        self.init_parameters()

        for i in range(epochs):
            cost, gradients = self.calculate_cost()
            self.cost_history.append(cost)
            self.parameters -= learning_rate * gradients.reshape(-1,1)

            if verbose:
                if not (i % verbose):
                    print(f"Cost after {i} epochs: {cost}")

        return self.parameters


    def predict(self,x, is_normalize=True):
        """
        Returns the predictions after fitting.
        parameters: 
            x : input/feature matrix

        Returns:
            predictions : Array of predicted target values.

        """
        x = np.array(x, dtype=np.float64) # converting list to numpy array
        if x.ndim == 1:
            x = x.reshape(1,-1)
        if is_normalize:
            for n in range(x.shape[1]):
                x[:,n] = (x[:,n] - self.mu[n]) / self.sigma[n]
        x = np.concatenate([np.ones((x.shape[0],1)), x], axis=1)
        return np.dot(x,self.parameters)

The class and its methods are pretty obvious, try to go one by one and you'll understand what each method is doing and how it's connected to others.

Still, I would like to put your focus on 3 main methods:

calculate_cost : This method actually uses the formulas we derived in the previous section to calculate the cost according to certain parameters. If you carefully go through the method you may find a weird thing that initially we mentioned cost as:

but in code we are calculating cost as:

No need to be puzzled, they both are the same thing, the second equation is the vectorized form of the first one. If you're aware of Linear Algebra operations you can prove to yourself that they both are the same equations. We preferred the second one because often vectorized operations are faster and efficient instead of using loops.

fit: This is the method where the actual magic happens. It firstly normalizes the features then add an extra feature of all 1s for the bias term and lastly, it keeps iterating to calculate the cost and gradients then update each parameter simultaneously.

Note: We first normalize the features then we add an extra feature of 1s for bias term because it doesn't make any sense to normalize that extra feature that contains all 1s

predict: This method first normalizes the input then uses the optimal parameters calculated by the fit method to return the predicted target values.

Note: predict method uses the same μ and σ that we calculated during the training loop from the training set to normalize the input.

Great now we have our class, it's time to test it on the datasets.

Testing on datasets

In this sub-section, we'll be using Sklearn's generated dataset for linear regression to see how our Linear Regression class is performing. Let's visulaize it on graph:

Let's create an instance of the LinearRegression class and fit this data on it for 500 epochs to get the optimal parameters for our hypothesis.

Okay, let's see how this hypothesis looks:

It fits nicely, but plotting the cost is a great way to assure that everything is working fine, let's do that. LinearRegression class had a property of cost_history and it stores the cost after each iteration, let's plot it:

We can see that our cost function is always decreasing and it's a good sign to show that our model is working pretty good.

Before moving on to the next section and discussing Regularization, I want to demonstrate how we can also fit a curve instead of a straight line to a dataset, let's see it in the next sub-section.

Polynomial Regression

We basically going to take the generated dataset for linear regression from sklearn and apply some transformation to it to make it non-linear.

Note: For detailed code implementation I recommend you to go through the notebook from here since for the sake of learning I'm only showing a few code cells for verification

So what we did is generate the data from Sklearn's make_regression of 1 feature and a target column then apply the following transformation on it to make that data non-linear

and after applying to the dataset, it looks like:

Looks good we are able to introduce non-linearity but it'll be great if it also contains some noise samples, anyway let's start working on this non-linear dataset.

To make our linear regression predict a non-linear hypothesis we need to create more features (since we have only 1 here) from the features we already have. A popular way to create more features is to perform some polynomial functions on the original features one by one. For this example, we are going to make 6 different features from the original one:

We will be stacking these X1, X2, ..., X6 as features to make our final input/feature matrix X_. Now let's use this X_ matrix to predict the optimal curve.

The process of fitting and predicting is the same as shown in the previous section, or you can also refer to the notebook for better clarity.

It looks great, our algorithm is able to predict a fine non-linear boundary and it fits our training set very precisely. But there's a problem, we can see that our algorithm is performing very well on the training set and it's also possible that it won't work good on the data outside the training set. This is known as Overfitting and it leads to the lack of generality in the hypothesis.

We are going to address this problem using Regularization in the next section.

Regularization

With the help of Regularization, we can prevent the problem of overfitting from our algorithm. Overfitting occurs when the algorithm provides heavy parameters to some features according to the training dataset and hyperparameters. This makes those features dominant in the overall hypothesis and lead to a nice fit in the training set but not so good on the samples outside the training set.

The plan is to add the square of parameters by multiplying them with some big number (λ) to the cost function because our algorithms' main motive is to decrease the cost function so in this way algorithm will end up giving the small parameters just to cancel the effect addition of parameters by multiplying with a large number. So our final cost function gets modified to:

Note: We denote the bias term as θ0 and it's not needed to regularized the bias term that's why we are only considering only θ1 to θn parameters.**

Since our cost function is changed that's why our formulas for gradients were also get affected. The new formula for the gradient are:

λ is known as a regularization parameter and it should be greater than 0. A large value of λ leads to underfitting and very small values lead to overfitting, so you need to pick the right one for your dataset by iterating on some sample values.

Let's implement the Regularization by modifying our LinearRegression class. We only need to modify the calculate_cost method because only this method is responsible for calculating both cost and gradients. The modified version is shown below:

class LinearRegression:
    def __init__(self) -> None:
        self.X = None
        self.Y = None
        self.parameters = None
        self.cost_history = []
        self.mu = None
        self.sigma = None

    def calculate_cost(self, lambda_=0):
        """
        Returns the cost and gradients.
        parameters: 
            lambda_ : value of regularization parameter (default is 0)

        Returns:
            cost : Caculated loss (scalar).
            gradients: array containing the gradients w.r.t each parameter

        """
        m = self.X.shape[0]

        y_hat = np.dot(self.X, self.parameters)
        y_hat = y_hat.reshape(-1)
        error = y_hat - self.Y

        cost = (np.dot(error.T, error) + lambda_*np.sum((self.parameters)**2))/(2*m)

        gradients = np.zeros(self.X.shape[1])

        for i in range(self.X.shape[1]):
            gradients[i] = (np.mean(error * self.X[:,i]) + (lambda_*self.parameters[i])/m)

        return cost, gradients


    def init_parameters(self):
        """
        Initialize the parameters as array of 0s
        parameters: None

        Returns:None

        """
        self.parameters = np.zeros((self.X.shape[1],1))


    def feature_normalize(self):
        """
        Normalize the samples.
        parameters: 
            X : input/feature matrix

        Returns:
            X_norm : Normalized X.

        """
        X_norm = self.X.copy()
        mu = np.mean(self.X, axis=0)
        sigma = np.std(self.X, axis=0)

        self.mu = mu
        self.sigma = sigma

        for n in range(self.X.shape[1]):
            X_norm[:,n] = (X_norm[:,n] - mu[n]) / sigma[n]
        return X_norm

    def fit(self, x, y, learning_rate=0.01, epochs=500, lambda_=0, is_normalize=True, verbose=0):
        """
        Iterates and find the optimal parameters for input dataset
        parameters: 
            x : input/feature matrix
            y : target matrix
            learning_rate: between 0 and 1 (default is 0.01)
            epochs: number of iterations (default is 500)
            is_normalize: boolean, for normalizing features (default is True)
            verbose: iterations after to print cost

        Returns:
            parameters : Array of optimal value of weights.

        """
        self.X = x
        self.Y = y
        self.cost_history = []
        if self.X.ndim == 1: # adding extra dimension, if X is a 1-D array
            self.X = self.X.reshape(-1,1)
            is_normalize = False
        if is_normalize:
            self.X = self.feature_normalize()
        self.X = np.concatenate([np.ones((self.X.shape[0],1)), self.X], axis=1)
        self.init_parameters()

        for i in range(epochs):
            cost, gradients = self.calculate_cost(lambda_=lambda_)
            self.cost_history.append(cost)
            self.parameters -= learning_rate * gradients.reshape(-1,1)

            if verbose:
                if not (i % verbose):
                    print(f"Cost after {i} epochs: {cost}")

        return self.parameters


    def predict(self,x, is_normalize=True):
        """
        Returns the predictions after fitting.
        parameters: 
            x : input/feature matrix

        Returns:
            predictions : Array of predicted target values.

        """
        x = np.array(x, dtype=np.float64) # converting list to numpy array
        if x.ndim == 1:
            x = x.reshape(1,-1)
        if is_normalize:
            for n in range(x.shape[1]):
                x[:,n] = (x[:,n] - self.mu[n]) / self.sigma[n]
        x = np.concatenate([np.ones((x.shape[0],1)), x], axis=1)
        return np.dot(x,self.parameters)

Now we have our regularized version of the LinearRegression class. Let's address the previous problem of overfitting on polynomial regression by using a set of values for λ to pick the right one.

From the plots, I think that λ = 10 and λ = 20 looks good. We can see that as we increase the values of λ, our algorithm starts to perform even worst on the training set and leads to Underfitting. So it gets really important to select the right value of λ for our dataset.

Conclusion

Great work everyone, we successfully learnt and implemented Linear Regression using Gradient Descent. There are a few things that we need to keep in our mind that this optimizing algorithm requires more hyperparameters than the Normal equation that we learnt in the previous article but irrespective of that, gradient descent works efficiently on the larger dataset covering the drawback of the Normal equation method.

In the next article we'll be learning our first supervised classification algorithm known as Logistic Regression and going to understand how to Regularization prevents overfitting there.

I hope you have learnt something new, for more updates on upcoming articles get connected with me through Twitter and stay tuned for more. Till then enjoy your day and keep learning.

Linear Regression using Normal Equation

Swayam Singh — Fri, 26 Nov 2021 19:50:48 +0000

Overview

Linear Regression is the first algorithm in the Demystifying Machine Learning series and in this article we'll be discussing Linear Regression using Normal equation. This article covers what is Linear Regression, how it works, the maths behind the normal equation method, fixing some edge cases, handling overfitting and code implementation.

What is Linear Regression?

Linear Regression in simple terms is fitting the best possible linear hypothesis (a line or a hyperplane) on the data having a linear relationship so that we can predict the new unknown data point with the least possible error. It's not necessary to have a linear relationship in data but having such can lead to approximately close predictions. For a reference take a look at the below representation.

In the above picture, only 1 feature (along the x-axis) and the target (along the y-axis) is displayed just for the sake of simplicity and we can see that the red line is fitting the data very nicely covering most of the variance.

One thing is to be noted that we call it Linear Regression but it's not always fitting a line, we call it hypothesis or hyperplane. If we have N-Dimensional data (data having N number of features) then we can fit a hyperplane of atmost N-Dimensions.

Mathematics behind the scenes

Let's take a very simpler problem and dataset to derive and mimic the algorithm we are going to use in Linear Regression.

Assume we have a dataset in which we have only 1 feature say x and target as y such that x = [1,2,3] and y = [1,2,2] and we are going to fit the best possible line on this dataset.

In the above plot, we can see that the feature is displayed on the x-axis and the target is along y-axis and the blue line is the best possible hypothesis through the points. Now all we need to understand is how to come up with this best possible hypothesis that is the above blue line in our case.

The equation of any line is in the format of y = wx + b where w is the slope and b is the intercept. In machine learning lingo we call w as weight and b as bias. For above line it came out to be w = 0.5 and *b = 0.667 * (Don't worry! we'll see how).

Now ultimately we can say that somehow we need to calculate the weight and bias term/terms (since 'x' is already known to us) for hypothesis and then we could use them to get the line's equation.

Linear Algebra

We are going to use some Linear Algebra concepts for finding the right weight and bias terms for our data.

Let say the weight and bias terms are w and b respectively. So we can write each data point (x, y) as :

we can write the above equations as a system of equations using matrices as Xθ = Y where X is input / feature matrix, θ is matrix for unknowns and Y is the target matrix as:

Great, Now all we need is to solve this system of equations and get the w and b terms.

Wait there's a problem. We can't solve the above system of equations because target matrix Y does not lie in the column space of input matrix X. In simple terms, if we see the previous graph again then we can notice that our data points are not collinear i.e. they don't lie on the line so and that's why we can't find the w and b for the above system of equations.

And if we think for a moment then it sounds right because in Linear Regression we fit a hypothesis to predict the target for some input with the least possible error. We do not intend to predict the exact target.

So what we can do here? We can't solve the above system of equations because Y is not in the column space of X. So instead we can project the Y onto the column space of X. It is exactly equivalent to projecting one vector onto another.

In the above representation, a and b are two vectors and the a1 is the projection of vector a onto b. With this, we can see that now we have the component of vector a that lies in the vector space of b.

We can achieve the component of Y that lies in the column space of X by doing inner product (also known as dot product).

The inner product of two vectors a and b can be found by calculating a^Tb

Now we can re-write our system of equations as:

multiplying both sides by X^T.

Assuming (X^TX) to be invertible

The above equation is known as Normal equation. Now we have the formula to find our matrix θ, let's use it and calculate the w and b.

from the above last equation we have our w = 0.5 and b = 2/3 (0.6667) and we can check from the equation of blue line that our w and b are exactly correct. That's how we can get the weights and bias terms for our perfect hypothesis using the Normal equation.

Python Implementation

Now it's time to get our hands dirty with code implementation and make our algorithm work for real datasets. In this section we're going to create a class NormalLinearRegression to handle all the computations for us and return the optimal values of weights.

Note: All the code files can be found on Github through this link.

And it's highly recommended to follow the notebook along with this section for better understanding.



class NormalLinearRegression:
    def __init__(self) -> None:
        self.X = None
        self.Y = None
        self.theta = None

    def fit(self,x,y):
        """
        Returns the optimal weights.
        parameters: 
            x : input/feature matrix
            y : target matrix

        Returns:
            theta: Array of the optimal value of weights.

        """
        self.X = x

        if self.X.ndim == 1: # adding extra dimension, if X is a 1-D array
            self.X = self.X.reshape(-1,1)

        # adding extra column of 1s for the bias term
        self.X = np.concatenate([np.ones((self.X.shape[0], 1)), self.X], axis=1)

        self.Y = y
        self.theta = np.zeros((self.X.shape[1],1))

        self.theta = self.calculate_theta()
        self.theta = self.theta.reshape(-1,1)

        return self.theta

    def predict(self, x):
        """
        Returns the predicted target.
        parameters: 
            x : test input/feature matrix

        Returns:
            y:  predicted target value.

        """
        x = np.array(x) # converting list to numpy array
        if x.ndim == 1:
            x = x.reshape(1,-1) # adding extra dimension in front
        x = np.concatenate([np.ones((x.shape[0],1)), x], axis=1)
        return np.dot(x,self.theta)

    def calculate_theta(self):
        """
        Calculate the optimal weights.
        parameters: None
        Returns:
            theta_temp: Array containing the calculated value of weights

        """
        y_projection = np.dot(self.X.T, self.Y)
        cov = np.dot(self.X.T, self.X)
        cov_inv = np.linalg.pinv(cov)
        theta_temp = np.dot(cov_inv, y_projection)

        return theta_temp

Code is pretty self-explanatory and I added comments wherever I found necessary but still, I want to point out a few things. We are adding an extra column of 1s to X for the bias term. Since our theta matrix had both weight and bias terms so we just added an extra column of 1s so that matrix multiplication handles the addition of bias term.

We are going to test our hypothesis on two datasets. Our first dataset contains 2 columns in which the first one (only feature) is the population of a city (in 10,000s) and the second column is the profit of a food truck in that city (in $10,000s). A negative value for profit indicates a loss. Let's visualize it on the graph.

Now let's use our NormalLinearRegression class to find the best hypothesis to fit on our data.

Great now let's find the predictions using the params. Class NormalLinearRegression had a method predict that we can use to get the predictions and after that, we can use them to draw the hypothesis as shown below.

Okay, our hypothesis looks pretty nice. Now let's take a dataset that has multiple features, for the sake of graphical representation our next dataset contains a training set of housing prices in Portland, Oregon. The first column is the size of the house (in square feet), the second column is the number of bedrooms, and the third column is the price of the house. So in this dataset, we have 2 features and 1 target. Let's visualize it on the graph.

If you're wondering how I plot them, just visit the repo for this algorithm through this link and you'll find the notebook where all the implementations are already done for you.

Now let's find the weights of the best hypothesis for this dataset.

Awesome, now let's find the predictions and plot the hypothesis for this dataset using the predict method of our class.

Yeah I know, it looks messy but we can uniform it as:

Now the thing to be noted is that it's not a straight line, plotting a 3D graph over a 2D surface may give a feel that it's a line but it's not. It's an N-1dimensional hyperplane and in our case, it's a 2D plane.

Great work people, so far we designed our own Linear Regression algorithm using a Normal equation and tested it on 2 datasets with single and multiple features, you can even try it on your custom dataset to see how it works. So far so good.

BUT still, there's an edge case is left, let's handle it in the next section.

What if (X^TX) is non-invertible?

When we are deriving the Normal equation, we assumed that (X^TX) to be invertible and then we calculated its inverse to find the matrix θ. But what if it's not invertible?

Let's discuss the cases when it cannot be invertible:-

(X^TX) is not a square matrix
The columns or rows of (X^TX) are not independent

The 1^st case is obviously wrong, let's see how?

Suppose the dimensions of X are (m,n) then the dimensions of X^T will be (n,m). So after performing matrix multiplication the dimensions of (X^TX) will be (m,m) and hence it's a square matrix.

But the 2^nd case can be true, let's see how?

Suppose you have a dataset and in which the features are not linearly independent. For example, let's say there's a feature labelled as weight in Kg and another feature labelled as weight in pounds, both the features are linearly dependent i.e we can get one feature by performing some linear transformations on another feature and this can make (X^TX) as non-invertible.

Although in our python implementation we used np.linalg.pinv() function to calculate the inverse and it uses Singular Value Decomposition to return the pseudo-inverse of the matrix if it's non-invertible.

Another way to remove such ambiguity is to identify those features and remove them manually. OR we can use Regularization and make it invertible. Let's see how we can use Regularization to achieve this.

Regularized Normal Equation

In Regularization, we add an extra Identity matrix of dimensions (n+1, n+1) to (X^TX) where n is the number of features and adding extra 1 denotes the extra column of 1s for bias term.

Regularized Normal equation can be written as:

Now let's understand the above equation, look at the 1^st column it has only 0s no 1s because the 1st column in X is the column of 1s for the bias term and we do not regularize that column. Mathematically it can be proven that (X^TX + λM) is always invertible.

λ is called the regularization parameter. You need to set it according to your dataset by choosing from a set of values that should be greater than 0 and select the one which gives the least root mean square error on your training set other than λ = 0.

Let's implement this in code and see how this method works. We only need to change the calculate_theta method of our class which is reponsible for the calculation of (X^TX)^-1X^TY .

The modified calculate_theta method should look something like this:



def calculate_theta(self, lambda_):
        """
        Calculate the optimal weights.
        parameters: None
        Returns:
            theta_temp: Array containing the calculated value of weights

        """
        y_projection = np.dot(self.X.T, self.Y)
        # Creating matrix M (identity matrix with fist element 0)
        M = np.identity(self.X.shape[1])
        M[0,0] = 0

        cov = np.dot(self.X.T, self.X) + lambda_*M # adding lambda_ times M to X.T@X
        cov_inv = np.linalg.pinv(cov)
        theta_temp = np.dot(cov_inv, y_projection)

        return theta_temp

We don't need to change anything else in our class and now let's pick some random values for λ and pick the best one.

Intuitively we can see that plots for λ = 0 and λ = 10 are quite good, just to be sure we stored the root mean squared error in a dictionary errors let's print it out and see which λ got the least error.

We can see that λ = 0 got the least error and λ = 10 is slightly greater than that but just be on the safer side we will pick λ = 10 for our hypothesis.

Conclusion

Great work everyone, we have successfully learnt and implemented our first machine learning algorithm of Linear Regression using a Normal equation on 2 datasets. There are a few things that we need to keep in our mind while using the Normal equation method. This method is not efficient with respect to computation time on large datasets (samples > 10,000) and the reason is the operation of calculating the inverse of a matrix that had a time complexity of around O(n³).

So in that case Gradient Descent method comes very handy and we'll be going to learn that in our next article. Also, we'll be exploring Regularization in great depth too.

I hope you have learnt something new, for more updates on upcoming articles get connected with me through Twitter and stay tuned for more. Till then enjoy your day and keep learning.

Demystifying Machine Learning

Swayam Singh — Fri, 19 Nov 2021 18:49:35 +0000

Overview

Demystifying Machine Learning refers to the series of blogs in which I am going to explain the different algorithms of Machine Learning with behind the scenes mathematics and code implementation from scratch. This is an introductory article for the entire series about what the audience can expect, what are Algorithms we are going to cover and what prerequisites you need to fulfil in order to get the most out of this series.

Prerequisites

There are a few things you need to be comfortable with so that you can get the best out of the articles.

Working knowledge of Python and OOP concepts. It'll be great if you already worked with Numpy before.
Basic mathematical concepts like Linear Algebra, Calculus and probability will be needed.

Don't worry, if you're not a pro in maths. It's going to be a beginner-friendly introduction to all algorithms and mathematics with explaining every concept in brief.

Algorithms to cover

Below is the list of all the algorithms we are going to cover in this series. The main focus of this series is going to be on Supervised and Unsupervised algorithms.

Deep Learning is a subset of Machine Learning, so we will also be dealing with Neural Networks from scratch, but just to keep content consistent, advanced stuff like CNNs, sequence models, etc are kept out of the scope for this series.

Linear Regression :
1. Linear Regression using normal equation and regularization
2. Linear Regression using gradient descent and regularization
Logistic Regression with regularization
Support Vector Machines (SVM)
1. Optimal Margin Classifiers
2. Kernel and SVM
Neural Networks :
1. Shallow Neural Networks
2. Deep Neural Networks
K-Means Clustering
Dimensionality Reduction (P.C.A)
Anomaly Detection

And the best part, we're not just going to understand the algorithm and implement it with Python. In the end or in middle, we'll be going to make our own Python package including classes of the algorithms we implemented and later on publish it on PIP so that you can pip install <package> it whenever you want. (Just for fun 😉)

The best route

Articles are going to be long and comprehensive with loads of mathematical notations and code implementations, so it'll be obvious not to hurry just to finish an article within a specific reading time. Take your time, do google search whenever required, make hand notes and understand what the code is actually doing.

If feeling overwhelmed adopt the policy of "1 algorithm a week". This series is designed in such a way that you get to know your algorithm by asking how it's working and why it's working. A good practice is NOT to skip lines and jump to some random section. If get stuck somewhere or you got any other doubts, feel free to contact me via my socials.

All code implementations will be pushed to a GitHub repository whose link will be shared in the respective algorithm's article, so that you can also contribute to it, if you got something interesting about that code implementation.

Conclusion

This is all to this introductory article. Nothing better than ending your week by learning a new ML algorithm 😁. So make sure to follow me on Twitter for updates on newly published articles. See you in the next week with Linear Regression, till then enjoy and have a nice day 🍁

How to Set Up Firebase Authentication in React from Scratch

Swayam Singh — Fri, 28 May 2021 04:43:18 +0000

Overview:

User authentication is a must if you are building a platform that stores some private data of users like a social media app. At the same time, it’s kind of tricky to implement. In this article, we will be discussing how we can use Firebase to create a fully functional and secure user authentication.

Agenda:

Following is the list of features we will build later on in this article.

Sign Up
Log In
Dashboard
Log Out
Forget Password
Protected Routes

Prerequisites:

Familiar with React environment.
Basic knowledge of Context API.
Basic knowledge of routing in React.

Let’s go with the flow

So firstly, we need to create a React app. Navigate into the Desktop folder inside the terminal and type npx create-react-app <give any name> . Inside the src folder, just keep index.js and App.js, delete the rest — we don’t need them.

Setting up Firebase:

Okay, so now, visit firebase and click on go to console there, you click on Add Project and give it any name you want.

Click on continue and disable google analytics again click on continue . It’ll take some time to process and when it’s done, our Firebase app is now ready.

Now, we need to integrate it with our React web app. Click on web icon. Then, it’ll ask you to enter another name of the project for integration. Don’t worry, it can be any name you want.

Click on the Register app Now, Firebase will give you some keys and other configuration settings so that you can connect your React app with Firebase services.

Configuration file will be different for you. Also, don’t use the above keys. This demo app will be deleted after this article.

We will use them later. Now, let’s set up authentication. Navigate to the home page of your firebase app and click on authentication. Then, click on Get Started. It’ll show you some methods that you can use to create user authentication. In this article, we will be using the Email/Password method. So, click on it. Then hit enable only for the first option and click save.

Great! Now we have a method for authentication and a config file to connect the app. Let’s go to our favourite code editor and start coding!

Danger! Code ahead

First, we are going to create a .env.local file for storing our Firebase configuration details. Putting them openly naked inside your code will make it easy for hackers to access your Firebase app. Inside our react-app, create a .env.local file and store only the keys and values that are inside the variable firebaseConfig, as shown below:

In React, naming of environment variables start with REACT_APP

Great! Now, it’s time to connect it. For that, we are going to use the firebase module. And below is the list of all the modules we’ll be using for this entire project.

react-router-dom for working with different routes.
bootstrap for styling
react-bootstrap for pre-built styled-components
firebase for working with Firebase

So, go ahead and install them all at once using the command:npm i react-router-dom bootstrap react-bootstrap firebase .

Inside react-app, create a file firebase.js for making the connection with Firebase.

If you have been observant, then this file is very similar to the file that Firebase gave us for creating a connection. Yeah, we just used the firebase module instead of an external JavaScript script. We initialised the app with environment variables stored in .env.local file and app.auth() is stored inside auth which will be responsible for calling several methods like login, signup, logout, etc.

Very well. Now, it’s time to set up the **Context API **so that we can just define our authentication methods in one file and access them in the relevant components. Inside the src folder, create another folder with the name **context **and inside it, create a file authContext.js as shown below.

Basic knowledge of Context API is mentioned in the prerequisites. But still, I’m explaining it. Above, you can see the basic structure of the context JavaScript file. Firstly, we create a context using React.createContext() . It gives us two things — Provider and Consumer. For now, we are only concerned with Provider which enables us to pass the value in it and use them in any component.

Component AuthProvider returns the <AuthContext.Provider>; component with a value prop that contains the values we want to pass, {children} refers to the root component which will be wrapped by the Provider In the end, we created a custom hook useAuth() which directly gives you all of the passed values.

Now, let’s start creating our authentication methods inside authContext.js and pass them to Provider . Replace the comment in the above code with the following lines.

You can see that firstly, we create two states for storing currentUser that had info about logged in user and loading for displaying {children} after it is set to false. So, instead of simply rendering {children} inside <AuthContext.Provider>, change it to {!loading && children}.

As I said earlier, all the authentication methods will be stored inside auth from firebase.js. So, we used auth to call different methods and stored them inside respective functions so that when needed, we can call them. useEffect() contains an event handler that continuously listens to the authentication state like when the user logs in and when they sign out. According to that, it sets the currentUser state to logged in user or undefined .

auth.onAuthStateChanged() is an event handler. Whenever that component loads, useEffect() sets that handler for use many times. It may cause a memory leak and make your app slow. For dealing with this, auth.onAuthStateChanged() returns a unsubscribe method that can unsubscribe you from the event handler as the component unloads.

After that, we just pass all the methods and states to values prop inside Provider.

Now, we need to wrap our root component with AuthProvider component. In this app, we’ll be wrapping our App component. So, open index.js and do the following:

Now our <App /> will be {children} for <AuthProvider> . You can see I also imported a bootstrap CSS file for giving style to each of our components. But you can create your own custom style sheet if you want. In this article, we are just focusing on functionality.

Till now, we have been dealing with the functionality setup. So, it’s time to create components for each method. Create a folder inside src folder with the name as components.

Sign-up component:

Firstly, we are going to deal with the sign-up component. So, inside the components folder, create a file with the name signup.js.

We will create a form with three fields — email, password, confirm-password and check if confirm-passowrd matches with password . Only then are we going to call the signup method from authContext.js If the signup succeeded, then we redirect our user to the dashboard component (going to create this later).

As I mentioned earlier in article, our custom hook useAuth() will be used to receive passed values in context and in this component, we are using the signup method created inside authContext.js . Link is used to take the user to the login page if they already had an account and the useHistory hook is used for redirecting the user after successfully registering.

emailRef , passwordRef and passwordConfirmRef are used as references for respective input fields, and later, we destructure the signup method from the useAuth hook.

Now, take a look at the function handleSubmit. It’s an async function because authentication methods from Firebase return a promise So, we are using async / await to handle it. Here, first we are checking if the password and confirmPassword are the same. And then, inside the try / catch block, we are calling the signup function by passing the email and password entered by the user.

That is our functionality for signup So now, inside return, let’s create the form and other UIs. We are using bootstrap cards and form for styling purposes.

Just go line by line. The code is very simple. All we are doing is using bootstrap components and creating the form. In the end, we created a link to the login component if the user already has an account. That’s it. Our signup component is ready.

Login Component:

Create a file inside the component folder and name it login.js.

This component will be pretty much the same as the signup component. The only difference is we are calling the login function instead of signup and we don’t need to have the confirm-password field here. The rest of the code in both the components will be the same.

What we are returning is also very similar to signup Except instead of creating the link to login we ask if they are not registered, then take them to the signup component

The extra thing we are allowing users is an option to reset their password by creating a link to the forgetPassword component (going to create it later).

Forget Password Component:

The way Firebase password reset works is that when the user clicks on it, they send an email to the registered email address with further instructions and a link to reset their password.

The cool thing is again, the code will be pretty similar to the above component, and here we are calling the resetpassword method. Go ahead and create a file with the name forgetpassword.js and take a look below.

As you can see, we are destructuring resetPassword from custom useAuth() hook and had an extra state for storing messages like check your inbox blah blah blah… after successfully calling the resetPassword function.

Nothing new. We are already familiar with this type of code and that’s it — our reset password component is ready.

Dashboard:

For now, our dashboard just shows the email of the currentUser and also contains a logout button for logging out the user. You can add more functionality according to your custom project.

Go ahead and create a file with the name dashboard.js inside the components folder.

The thing to be notice is that here we are destructuring logout and currentUser for handling signing out and showing the email of the logged-in user respectively. As the user successfully logs out, we then redirect him/her to the login component using the useHistory() hook. The Dashboard component is ready.

Setting up Routes for components:

We are done with all of our components. Now let’s set up each of their routes inside App.js using react-router-dom . Open App.js and do the following.

We exported BrowserRouter and Route for setting up routes for each of the different components, and Switch for loading a single component at a time.

Now, if you start the app by running npm start, then you see a blank screen because the home page is empty right now. For seeing different components, go to their respective URLs. For example: http://localhost:3000/signup will take you to:

And as you enter your details and click on the signup button, you’ll be redirected to the dashboard component

Now, one by one, you can check out the other components too. Now we have successfully implemented authentication. You can see the list of registered users in your Firebase console under the authentication section. Something is still missing…

Securing sensitive routes:

Our app is working perfectly and authenticating users very well but still, something is missing, we forget something. Can you guess what?

Well, if you log out the user, and try to access the dashboard boom! You can still access it and that’s not good. We don’t want any guest user to easily access our dashboard. Although it does not contain anything right now, it might not be true in your project’s case. We still need to secure it, so that only registered users can access their dashboard.

The way we can do so is to create another component. Basically a modified version of the Route component from react-router-dom and it’ll check whether someone is logged in or not. If it’s true, then it’ll render the dashboard otherwise, just redirect to the login component.

Create a file with the name privateRoute.js inside the components folder and take look at what it’s going to contain.

Let’s understand what we are doing. As I said, PrivateRoute is a modified version of the Route component which checks for currentUser before setting up any route. Here, first we got hold of the currentUser from useAuth().

Then, we are returning the <Route> component with all the props required for a normal <Route> component like path, exact , etc. Then it checks for currentUser. If true, then it renders the dashboard otherwise, it redirects to login . Here, instead of using useHistory() for redirecting, we’ll be using the <Redirect> component from react-router-dom because we need to render something if currentUser is not logged in.

Now, inside App.js , import this component and replace the code line where you set up the route for the dashboard to the following:

<PrivateRoute exact path=”/dashboard” component={Dashboard} />

We are done. Now if you try to access the dashboard as a guest user, you’ll be redirected to the login component.

Conclusion:

That’s it. We successfully created all the basic features required for user authentication. You can upgrade it in the way you want or instead of using context , you can go for Redux. Firebase has got so many cool features like Firestore , a real-time database, and much more. It really comes in handy while designing big heavy projects. We’ll be discussing more about Firestore in future articles. Till then, stay healthy and keep coding.

If you want the complete code files for your project, grab them from here.

What's next:

If you are still reading, make sure to follow me on Twitter as I share some cool projects and updates there and yeah don't forget I have some exciting stuff coming up every weekend. See Y'all next time and stay safe ^^ 🌻

How to convert PDF file into an Audiobook

Swayam Singh — Sat, 15 May 2021 12:51:02 +0000

Agenda:

Hey welcome back, so today we are going to do something to automate our reading task using Python. We are going to build a GUI program to select pdf files and then play them inside our software, exactly no more eye reading it'll read it for you and all you need to do is to sit back and enjoy.

Prerequisites:

Happy relationship with basic Python and tkinter.

yeah that's it, I'll be explaining the rest 😉.

Analysing:

So now we know what we are going to do so let's break it down into smaller chunks and focus on each of them individually.

First of all we are going to create a window and a dialog box to open the desired pdf file. We also create a text box for displaying the pdf text content and a button play for start playing it as audio.

Modules used:

tkinter (for dealing with GUI)
gTTS (for converting text into speech)
playsound (for playing the audio file)
PyMuPDF (for reading pdf files)

before moving ahead I want to tell you something that in most of the online tutorials you'll find people using PyPDF2 for working with pdf files but the reason we are not using it is because it does not always work, like till the date I'm writing this post if you use PyPDF2 for reading pdf generated by google like from google docs, it's not able to read text from it.

gTTS stands for Google Text To Speech it's a Python library as well as a CLI tool for converting text into speech.

playsound is also a Python library for playing audio files like .mp3 or .wav.

we are using playsound just for playing the audio file that will be created using gTTS, you can use any Python library for that like pydub or use os module to play on native audio player installed on terminal, but I guess this only works on Mac OS X and linux

Let's dive into the code now ✨

Step 1 :

In this step we'll be creating our GUI so open up your favourite code editor and create a file as main.py and import tkinter.

tkinter comes preinstalled with Python so no need to install it from pip.

from tkinter import *
from tkinter import filedialog

# creating main window instance from Tk class defined in tkinter 
window = Tk()
window.title("convert pdf to audiobook")
window.geometry("500x500")   # setting default size of the window  

# creating text box for displaying pdf content
text_box = Text(window, height=30, width=60)
text_box.pack(pady=10)

# creating menu instance from Menu class
menu = Menu(window)
window.config(menu=menu)

# adding `File` tab into menu defined above
file_menu = Menu(menu, tearoff=False)
menu.add_cascade(label="File", menu=file_menu)

# adding drop-downs to `file_menu`
file_menu.add_command(label="Open")
file_menu.add_command(label="clear")
file_menu.add_separator()
file_menu.add_command(label="Exit")

# adding play button for playing audio
play_btn = Button(text="Play")
play_btn.pack(pady=20)

# for keeping window open till we don't close it manually
window.mainloop()

Now if you run it, you'll see something like this,

Step 2 :

In this step we will create function open_pdf this function will create a dialogue box for selecting pdf file and then reading all of it text and showing inside the text box created earlier, then it'll use gTTS for creating audio file of all the text from text box.

import fitz  # fitz is actually PyMuPDF
from gtts import gTTS


def open_pdf():

    # creating dialogue box

    open_file = filedialog.askopenfilename(
        initialdir="/Users/swayam/Downloads/",
        title="Open PDF file",
        filetypes=(
            ("PDF Files", "*.pdf"),
            ("All Files", "*.*")
        )
    )

    if open_file:

        #reading pdf file and creating instance of Document class from fitz
        doc = fitz.Document(open_file)

        # getting total number of pages
        total_pages = doc.page_count

        # looping through all the pages, collecting text from each page and showing it on text box
        for n in range(total_pages):
            page = doc.load_page(n)
            page_content = page.get_textpage()
            content = page_content.extractText()
            text_box.insert(END, content)

        # after whole pdf content is stored then retrieving it from textbox and storing it inside variable 
        text = text_box.get(1.0, END)

        # using gTTS to convert that text into audio and storing it inside file named as audio.mp3
        tts = gTTS(text, lang='en')
        tts.save("audio.mp3")

You need to install gTTS and PyMuPDF, so inside your terminal run pip install PyMuPDF and pip install gTTS for installing them.

As you can see above code is self explanatory but still I want to highlight some points. First look at the line that says text_box.insert(END, content) basically END is defined inside tkinter and it returns the last index that means where is the end of file, similarly 1.0 means the beginning of the text.

So basically when we store the first page data inside text box then starting index == last index == END after that we'll keep inserting text at the end of the previous stored text.

Step 3 :

Now we have the function so it's time to provide each widget it's own functionality like pressing button and clicking on menu really perform something.

Go to the code and add command attribute to all the file_menu drop-downs and play_btn as show below

from playsound import playsound  

file_menu.add_command(label="Open", command=open_pdf)
file_menu.add_command(label="clear", command=lambda: text_box.delete(1.0, END))
file_menu.add_command(label="Exit", command=window.quit)

play_btn = Button(text="Play", command=lambda: playsound("audio.mp3"))

playsound requires pyobjc as dependency for working so you need to install it by pip install pyobjc

Basically function provided in command will execute as you click on the widget. For short function like clear or exit we used lambda functions.

window.quit will close the window and clear is self explanatory. As the audio.mp3 gets saved playsound("audio.mp3") will play it after you click the button.

So if you followed well then in the end your final code will somewhat look like:

from tkinter import *
from tkinter import filedialog
import fitz
from gtts import gTTS
from playsound import playsound 

window = Tk()
window.title("convert pdf to audiobook")
window.geometry("500x500")


def open_pdf():
    open_file = filedialog.askopenfilename(
        initialdir="/Users/swayam/Downloads/",
        title="Open PDF file",
        filetypes=(
            ("PDF Files", "*.pdf"),
            ("All Files", "*.*")
        )
    )

    if open_file:
        doc = fitz.Document(open_file)
        total_pages = doc.page_count
        for n in range(total_pages):
            page = doc.load_page(n)
            page_content = page.get_textpage()
            content = page_content.extractText()
            text_box.insert(END, content)

        text = text_box.get(1.0, END)
        tts = gTTS(text, lang='en')
        tts.save("audio.mp3")


text_box = Text(window, height=30, width=60)
text_box.pack(pady=10)


menu = Menu(window)
window.config(menu=menu)

file_menu = Menu(menu, tearoff=False)
menu.add_cascade(label="File", menu=file_menu)
file_menu.add_command(label="Open", command=open_pdf)
file_menu.add_command(label="clear", command=lambda: text_box.delete(1.0, END))
file_menu.add_separator()
file_menu.add_command(label="Exit", command=window.quit)

play_btn = Button(text="Play", command=lambda: playsound("audio.mp3"))
play_btn.pack(pady=20)

window.mainloop()

Let's test it

Now it's time to run our code and check if everything is working or not, take a sample pdf file with some text and open it.

YAYYYYYY...... 🎉 🥳, WE DID IT GUYS

we just created our own pdf to audio book convertor, now if you want to go some steps further I will recommend you to read gTTS official documentation also if someone wants then you can convert this python script into exe file and share it with your friends so they can have fun too

let's keep converting python scripts to .exe files for next tutorial 😅.

What's next !

How to integrate React frontend with Node backend

Swayam Singh — Sun, 02 May 2021 03:59:55 +0000

Overview :

React framework is great for creating awesome web apps and UIs. But sometimes we need extra functionalities like integrating database or performing authentication. Such stuff needs to be done in backend, you don't want that anyone can see your secret keys or hashing stuff in frontend.

That's why in this article we are going to learn how to connect your React frontend with express backend. Sometimes this task get really overwhelming, you might get stuck with CORS issue, we will be handling all of them in this one article.

Our motive is to create a server that host an API and then we make a GET request from React frontend and display the data on screen.

Prerequisites :

Basic React knowledge and comfortability with environment.
Basic Express knowledge.

Let's get going 👍

First of all create a folder and call it anything, we are naming it as app. This folder will contain all our frontend + backend code.

Now we will start with creating backend first,

Setting up Server

Open the terminal and locate to your app folder. Now inside this folder we will create server.js file. Yeah exactly this file will contain code responsible for server running and API hosting.

now we are going to initialize npm in this folder to handle external packages and dependencies.
In the terminal type npm init -y it will initialized the npm with default values.

As a result you will see a file with name package.json will automatically get created.

From this step we are going to handle rest of the things on your favourite code editor. I am using VSCode in this article.
Open app folder on VSCode.

Now we have to install some packages, these are

express
cors

I hope you are familiar with express it's a widely used module for maintaining backend. Now what is the use of cors library, okay so for this first we need to understand what CORS really is.

CORS is shorthand for Cross-Origin Resource Sharing. It is a mechanism to allow or restrict requested resources on a web server depend on where the HTTP request was initiated.
Whenever we make request to a server we send a header called origin. It contain the information about from where the request is originated and using this header a web server can restrict or allow the resource sharing between client and server.

By default requests from any other origins will be restricted by the browser.

If you want to read more about CORS, here's the link you can refer to More on CORS

Now with the use cors middleware we can allow CORS for all routes or to some specific routes, in this article we will allow for all routes but if you want to read more then refer to cors documentation.

For installing above packages open terminal in your VSCode and type following npm install express cors.

Now all left is to setup our backend server, coding time 🥳

Let's start with creating a file data.js. This file will contain our API data that we are going to host and then we export the API data so that we can use it inside our server.js.

Okay now let's setup our server, open the server.js and follow the below image

As you can see that the code is self explanatory but still I want to highlight a point that is port number on which our server is listening

Take any free port number you want except 3000 why? because port 3000 is used by react frontend and if you use same for your backend then it might gonna crash. Here you can see I used port 5000.

Now let's test if our server is working or not. Open the terminal again and type following node server.js and in the console you can see it'll be printing server is running on port 5000.

Open your browser and go to the following URL http://localhost:5000/api

You can see your API data there in JSON format. For better visualization you can use extensions like JSON viewer pro here's the link for download.

YAYYYY 🎉... Our server is up and running.

Time to move to frontend and start building it.

Setting up React frontend

First of all make a folder client, this will contain our frontend stuff.

Now go inside the client folder and type the following on terminal npx create-react-app my_app

It'll take some time to process and when it done you'll see a folder named my_app created, see below

Now inside the VSCode you can see that client/my_app will contain some pre-build files they all are the initial setup for React frontend. You can modify them as the way you want, but in this article we just modify package.json and App.js to our need.

Now open the package.json from the client/my_app folder on VSCode and add the following property below private: true property.

proxy: "http://localhost:5000"

see below for reference

Now adding this property makes React to use http://localhost:5000 as default URL for making requests.

It's time to setup App.js to make request to our server and render data on client's screen.
Open App.js on VSCode and in the function App delete everything inside the div having class 'App' and do the following.

As you can see the code is self explanatory but still I again want to highlight a point that, just take a look at the fetch("/api"). You can notice we are not providing complete endpoint like http://localhost:5000/api and the reason is we don't need to remember the proxy property we set earlier. All credit goes to it, now we can create as many routes we want inside your server and can access them in React with similar manner.

Now open terminal inside VSCode and select a new zsh or bash whatever you prefer and make sure you are inside themy_app folder.

and when you are in, type following in terminal to start the React frontend server. npm start

ignore all the warnings for now

This command will basically compile your React code and start the server at port 3000. It will also automatically open your web browser and get located to http://localhost:3000 and what you can see is a big *" Hello World " * on screen.

Open the Developers tools inside brower and in console you can see that our API data is logged there successfully.

Now we are sure that our frontend is working properly and data is also fetched without any problem, so it's time to render the data on screen. Open App.js on VSCode and replace the already written code with the highlighted part of code.

As you can notice we just replaced the console logging and Hello World to the other code so that it can set the state value to data array and render it on screen with some styling.

Now just save it and open your browser again to see the final result.

If you want to re-check just modify the data inside data.js from backend and save it, the modified result will also be displayed on your screen.

YAYYYYYY...... 🎉 🥳 our backend and frontend are now perfectly connected, now you can use your backend for integrating database or authentication without any worries of exposing private data on frontend.

What's next !

If you are still reading, make sure to follow me on Twitter and subscribe to my newsletter for more as I have some exciting stuff coming up every weekend. See Y'all next time and stay safe ^^ 🌻

Automating birthday wishes using Python

Swayam Singh — Tue, 27 Apr 2021 13:07:56 +0000

Overview:

Remembering dates is kinda hard but we are programmers, making hard things easier is our only job so instead of we remember dates why not automate this task.
In this article we gonna automate the birthday wishes, yeah exactly our program will check if there's any birthday today and then mail your friend a beautiful wish 😁.

Note: I highly recommend you to remember dates because friends get offended if they got to know about it.

Prerequisites :

comfortable with Python
some basic knowledge about pandas module

Let's get going 🌱

1. Setup:

So first of all before heading to write code, we need to create a csv file to store the information about our dearest friends like their email address, name, birthdate. Name that file as birthdays.csv.

csv stands for " comma separated values " it's basically a type of file in which you store the data separated by commas and the first row denote the heading of each values.
Just like a spreadsheet first row denote headings then below each heading we write it's value separated by commas.

Here's an example of our csv file.

Now we have a file which contains all the required data of our friends, it's time to create some really creative wishes.
We are going to create .txt files which store the wishes for our friends.

Here's the example what we are doing...

Python code will replace [NAME] with the birthday boy's/girl's name.

I recommend you to create multiple files (at least 3) and write different good wishes on them.

rename each of your birthday letters as letter_1.txt,letter_2.txt, etc and save your files inside the folder letters.

Now we have a folder letters that contains our nice wishes and a csv file. It's time to write our Python code.

2. Real hustle begins

Now open your favourite code editor, create a main.py file and start coding...

as mentioned in prerequisites I'm assuming you worked with python in past and comfortable with the syntax.

below is the list of modules that we are going to used in this project

datetime { for finding today's date and match it with the record }
pandas { for managing and filtering data from our csv file }
random { for selecting a random letter from letters }
smtplib { for sending mails to friends }

now let's import all the modules to our main.py file

from datetime import datetime # importing datetime class from datetime module
import pandas
import random
import smtplib

my_email = "email@gmail.com"
passw = "your_password"

since our program will send mails to friends so it also needs mail address of sender.

use your email address, you don't want your friend to receive birthday wishes by someone else's mail.

So moving ahead now we need to get hold of today's date so that we can compare it with data stored in csv file.

today = datetime.now()

here we are calling now() method from datetime class, it'll return today's date time and we are storing it in today variable.

Now we are going to use pandas to read our csv file and convert it into dataframe.

# reading csv file and making it's dataframe

data = pandas.read_csv("birthdays.csv")

# filtering data to check if there's any record that birthdate matches with today's date

bday = data[(data.month == today.month) & (data.day == today.day)]

# storing our friend's name having birthday today and email to separate variables, stays empty otherwise

name = bday["name"].tolist()
email = bday["email"].tolist()

# making a list of all the friends having birthdays today

friends = []

for n in range(len(name)):
    friends.append(
        {
            "name": name[n],
            "email": email[n]
        }
    )

Now it's time to select a random letter from the letters we created to send wishes.
First we gonna check if our friends list is not empty then we loop over each of its items and generate letters for them

# selecting a random integer as letter number from all letters, I assume you have 3.

if not friends:
    print("no birthday")
else:
    for friend in friends:
        num = random.randint(1, 3)
        with open(f"letters/letter_{num}.txt") as letter:
            lines = letter.readlines()
            lines[0].strip()
            lines[0] = lines[0].replace("[NAME]", friend["name"]) # replacing [NAME] with friend's name
            message = "".join(lines)

Now the only part remain is to send a mail with the selected random wish to our friend.
Here's how we gonna do this, in that same loop

# connecting to gmail's service
with smtplib.SMTP("smtp.gmail.com") as connection:
    connection.starttls()

# login with our email and password
    connection.login(user=my_email, password=passw)

# sending mail to friend's email address
    connection.sendmail(from_addr=my_email, to_addrs=friend["email"], msg=f"Subject: HAPPY BIRTHDAY\n\n{message}")
    print(f"message sent to {friend['name']}")

And that's it, if you followed well then in the end your code will look something like this:

import datetime as dt
import pandas
import random
import smtplib

my_email = "your_email@gmail.com"
passw = "your_password"

data = pandas.read_csv("birthdays.csv")
today = dt.datetime.now()
bday = data[(data.month == today.month) & (data.day == today.day)]
name = bday["name"].tolist()
email = bday["email"].tolist()

friends = []

for n in range(len(name)):
    friends.append(
        {
            "name": name[n],
            "email": email[n]
        }
    )

if not friends:
    print("no birthday")
else:
    for friend in friends:
        num = random.randint(1, 3)
        with open(f"letters/letter_{num}.txt") as letter:
            lines = letter.readlines()
            lines[0].strip()
            lines[0] = lines[0].replace("[NAME]", friend["name"])
            message = "".join(lines)

        with smtplib.SMTP("smtp.gmail.com") as connection:
            connection.starttls()
            connection.login(user=my_email, password=passw)
            connection.sendmail(from_addr=my_email, to_addrs=friend["email"], msg=f"Subject: HAPPY BIRTHDAY\n\n{message}")
            print(f"message sent to {friend['name']}")

No not yet, It's time to check if it's working or not.

since in the csv there is no birthdays today so you can see the message in consoleno birthdays today

Now if I change the csv and set any birth date to today's date and save it then after running the program again

Now it states message sent to {whatever name} ,
you can also check the mail to confirm it.

That's all, we did it, YAYYYY 🥳

What's next !

If you are still reading, make sure to follow me and subscribe to my newsletter for more as I have some exciting stuff coming up every weekend. See ya next time! 🌻

How to handle big integers in C ++

Swayam Singh — Sun, 18 Apr 2021 16:05:11 +0000

why we need to understand this topic ?

If we take a look on the number limits of integer data type of C++, you'll find something like:

int : approx 10⁹
long int : approx 10¹²
long long int : approx 10¹⁸

that means we can only store a maximum of 10¹⁸ integer i.e. only a number upto 19 digits. What if we have to deal with numbers greater than 19 digits ?

well using standard data types for such problems will lead to overflow. That's why languages like Python, Java, Ruby, etc. have libraries like Big integers or their variables are able to handle such large numbers.

In this article we'll learn how to store such large numbers in C++ and perform some fundamental operations on them.

Prerequisites

Basic STL (vectors) || you can also use arrays
Functions
Loops

How to store such large integers ?**

As we saw in standard data types of integer we can only store number upto 19 digits so in order to store greater than 19 digits we will use vectors and string for storing and taking input respectively. As we know in vectors / arrays items are stored in contiguous memory locations and we can create a vector upto 10⁷ items, so if we store each digit of our number into each of the memory locations then we will able to store 10⁷ digits. OMG it is much much greater than just 19 digits.

Remember we are storing digits of our number into each memory locations, not entire number into one memory location.

Below is a code sample to explain this

#include<bits/stdc++.h>
using namespace std;
int main()
{
    string num = "123499999999999999999999999999999999";  /* taking input as a string */

    vector <int> number;

    for(int i = 0; i < num.size(); i++)
        number.push_back(num[i] - '0');  /* converting from character to int (ASCII conversion) */

    for(auto i : number)   /* looping on each item in vector */
        cout<<i;
}

/* output: 123456789 */

So now we know how to store such large numbers and that's not the end of the story.
As we saw we need to use a contiguous memory locations to store such large numbers in C++, that's why operating on such numbers gets complicated as compare to handling just simple numbers of int data type.
For doing operations we need to build our own logic like how to add or subtract such numbers.

Let's dive into how to Add such numbers

before going ahead I recommend you to take a pen and a paper and add 12345 + 6789, like we use to do in our childhood, step by step

Algorithm:

take input and store both numbers into two different vectors / arrays.
reverse the vector (because we add from right to left).
initiate a variable to store carry.
store the sizes of vectors as min_len, max_len.
first loop till min_len, do addition of respective digits and store them into another vector (answer).
loop from min_len to max_len and store the sum of rest digits + carry into answer vector.
after addition of all digits if still carry remain then store it into answer vector.
again reverse the answer vector (because we previously reversed our two vectors)
return the answer vector.

here's the code with comments for better understanding of the logic

#include<bits/stdc++.h>
using namespace std;

vector <int> add(vector<int> x, vector<int> y) /* function to add two vectors */
{
    reverse(x.begin(),x.end());  /* reversing since we add from right to left */
    reverse(y.begin(), y.end());

    vector<int> ans; /* vector to store the answer */

    int min_len = min(x.size(), y.size()); /* calculating min_len and max_len from sizes of both vectors */
    int max_len = max(x.size(), y.size());
    int carry = 0;  /* initially carry will be 0 */

    for(int i=0;i<min_len;i++)   /* looping till minimum digits of both the numbers and doing addition */
    {
        int digit_sum = x[i] + y[i] + carry;
        carry = digit_sum / 10;    /* tens digit of digit_sum will be carry */
        ans.push_back(digit_sum % 10);  /* we store the once digit as answer */
    }

    /* handling remaining digits */

    if(x.size() > y.size())
    {
        for(int i=min_len; i < x.size(); i++)
        {
            int digit_sum = x[i] + carry;
            carry = digit_sum / 10;
            ans.push_back(digit_sum % 10);
        }
    }

    if(x.size() < y.size())
    {
        for(int i=min_len; i < y.size(); i++)
        {
            int digit_sum = y[i] + carry;
            carry = digit_sum / 10;
            ans.push_back(digit_sum % 10);
        }
    }

    /* handling remaining carry */

    while(carry)
    {
        ans.push_back(carry % 10);
        carry = carry / 10;
    }

    reverse(ans.begin(), ans.end()); /* reversing the answer vector to get correct answer vector */

    return ans;

}

int main()
{
    string a, b;
    cin >> a >> b;

    vector<int> a2,b2;

    for(int i=0;i<a.size();i++)  /* storing characters as integers into vector */
        a2.push_back(a[i] - '0');

    for(int i=0; i<b.size(); i++)
        b2.push_back(b[i] - '0');

    vector <int> result = add(a2, b2); /* calling and passing both vectors to add */

    for(auto i: result) /* printing output */
        cout<<i;
}

And we are done.

Yeah exactly I also never thought addition can be so complex, but if take a closer look all we did is just fundamental addition that we did in our childhood, but never considered so many cases.

So let's head forwards and understand another important concept, I bet when you started learning C++ you must made a program to calculate factorial of a number, the flaw of that program is that you only calculate factorials upto number 20, on inputting 21 and onwards you'll face overflow.

Now we are going to remake that same program but this time it'll be able to calculate factorials of much bigger numbers.

Computing factorials of large integers**

Idea is same as before, we take a number and store its factorial into vector, but this time instead of calculating addition we'll go for multiplication.

Algorithm:

input the number.
initiate answer vector and put 1 in it. (because multiplying any number with leads to same number )
loop from i=2 to number and each time multiplies the answer vector with i .
reverse the answer vector for correct answer.
print output.

here's the code for above problem

#include<bits/stdc++.h>
using namespace std;


void factorial(vector<int> &v, int x) /* passing with reference so that changes also occur in original vector */
{
    int carry = 0; /* initially carry will be 0 */

    /* doing fundamental way of multiplication */
    for(int i = 0; i < v.size(); i++)
    {
        int digit_product = v[i]*x + carry;
        carry = digit_product / 10;
        v[i] = (digit_product % 10);
    }

    /* handling if carry remains */

    while(carry)
    {
        v.push_back(carry % 10); 
        /* push_back() adds number at the end of vector but in multiplication we put carry in front
        so we need to reverse the vector to get the correct answer */
        carry /= 10;
    }
}

int main()
{
    int n;
    cin >> n;

    vector <int> ans; /* vector to store answer */
    ans.push_back(1);

    for(int i = 2; i <= n; i++) /* looping from 2 to n */
    {
        factorial(ans, i); /* passing vector and i to multiply the number in vector with i */
    }

    reverse(ans.begin(), ans.end()); /* reversing the vector to get the correct answer */

    for(auto i:ans)
        cout<<i;
}

try this code on your own and grasp the concept behind it, I make it as clear as I can with proper comments but you need to visualize the flow using pen and paper, doing the fundamental multiplication of a digits inside vector by a number.

What's next !

If you are ready now and feeling confident with this, I have a homework for you.

Try subtraction of two big integers on your own.

And yeah that was all for this now!

*If you are still reading, make sure to follow me for more as I have some exciting stuff coming up every weekend. See ya next time! *

DEV Community: Swayam Singh

Project-1: Cats and Dogs Classification using Logistic Regression

Overview

About Dataset

How to open?

Conclusion

Logistic Regression

Overview

What is Logistic Regression?

Working of Logistic Regression

Sigmoid function

Cost function

Gradient Descent

Code Implementation

Trying it out on a dataset

Problem of Overfitting

Regularization

Implementing Regularization on LogisticRegression class

Conclusion

Linear Regression using Gradient Descent

Overview

How Gradient Descent works (Intuition)

Cost Function

Mathematics of Gradient Descent

Applying Gradient descent to cost function

Python Implementation

Testing on datasets

Polynomial Regression

Regularization

Conclusion

Linear Regression using Normal Equation

Overview

What is Linear Regression?

Mathematics behind the scenes

Linear Algebra

Python Implementation

What if (XTX) is non-invertible?

Regularized Normal Equation

Conclusion

Demystifying Machine Learning

Overview

Prerequisites

Algorithms to cover

The best route

Conclusion

How to Set Up Firebase Authentication in React from Scratch

Overview:

Agenda:

Prerequisites:

Let’s go with the flow

Setting up Firebase:

Danger! Code ahead

Sign-up component:

Login Component:

Forget Password Component:

Dashboard:

Setting up Routes for components:

Securing sensitive routes:

Conclusion:

What's next:

How to convert PDF file into an Audiobook

Agenda:

Prerequisites:

Analysing:

Modules used:

Let's dive into the code now ✨

Step 1 :

Step 2 :

Step 3 :

Let's test it

What's next !

How to integrate React frontend with Node backend

Overview :

Prerequisites :

Let's get going 👍

Setting up Server

Setting up React frontend

What's next !

Automating birthday wishes using Python

Overview:

Implementing Regularization on `LogisticRegression` class

What if (X^TX) is non-invertible?