DEV Community: atultyagi612

How to deal with Missing Values in Machine Learning

atultyagi612 — Fri, 09 Apr 2021 11:15:48 +0000

When you work on real-world data it is often to have lots of missing values. There is much cause of missing data may be People do not respond to specific questions in the survey, Data entry errors, The individual drops out before sampling or many more reasons. The handling of data is very important during the preprocessing of the dataset because most machine learning algorithms do not support missing values.

In this article, I try to cover many ways to handle missing values in the dataset.

Variable Deletion
Replacing missing values with Mean/Median/Mode
- Impute missing values for continuous variable
- Impute missing values for categorical variable
- Assigning An Unique Category
Predicting The Missing Values
Using Algorithms that Support missing values

1. Variable Deletion

This method commonly used to handle the null values. In this variable Deletion method, we delete a whole column with missing values and it depends on case to case. We use this method only when there are lots of missing values in a particular column than variables. Let's see it with an example

For this example, we are going to use the famous Kaggle dataset house price prediction

data=pd.read_csv('house_price.csv')
total = data.isnull().sum().sort_values(ascending=False)
percent =total/len(data)*100
pd.concat([total,percent], axis=1, keys=['Total','Percent']).head(5)

Here you can clearly see that 4 columns have Null values higher than 80% so it is good to drop those columns from our data.

data=data[data.columns[data.isnull().mean() < 0.80]]

You can also drop specific rows with a higher number of missing values. This may increase the accuracy of your model rather than filling these columns. This is beneficial only when you have a large amount of data.

data.isnull().sum(axis=1).sort_values(ascending=False).head()

Complete removal of data with missing values results in a robust and highly accurate model if it does not have a high weightage. But if the column has a high correlation with the output we recommend not to delete it because it may lead to less accuracy and Loss of a lot of information.

2. Replacing missing values with Mean/Median/Mode

Impute missing values for continuous variable

This is another common technique use to handle missing data. In this method we replace missing values of numerical data like age, a salary of a person with the mean/median of the column, that's why it can prevent the loss of data compared to the previous method.

# The missing values are replaced by the mean value 
data['MSSubClass']=data['MSSubClass'].fillna(data['MSSubClass'].mean())

Impute missing values for categorical variable

The above code is only for numerical values for catagirical values we replacing the missing values with the maximum occurred value in a feature.

# The missing values are replaced by the median value.
data['SaleCondition']=data['SaleCondition'].fillna(data['SaleCondition'].mode()[0])

Assigning An Unique Category

If the number of missing values is very large then it can be replaced with a new category. Let's understand it with an example
For this example, we are going to use the Titanic dataset from Kaggle.

# Read data from CSV
data=pd.read_csv('titanic.csv')
data['Cabin']

# Replace null values with New category 'U'
 data['Cabin']=data['Cabin'].fillna('U')
data['Cabin']

This method is an approximation that may lead to variance in our data but it is better than dropping a whole column/row that's why it Prevent data loss and work better with a small dataset. this method works better on linear data but may cause data leakage.

3. Predicting The Missing Values

In this method, we use the features which have not null values and train a model to predict missing values. This method may give us better accuracy than replacing it with the mean/mode/median if the null values are less. let see it with an example.

In this example, we are going to Kaggle the Stroke dataset and train a RandomForestRegressor model using other available features to predict the missing values in 'bmi' feature.

# Load dataset 
data=pd.read_csv('healthcare-dataset-stroke-data.csv')
# Find Column with null values
data.isnull().sum().sort_values(ascending=False).head(3)

# Predict null values and replace null values with predicted values in data
data1=data[data['bmi'].notnull()]
data2=data[data['bmi'].isnull()]
temp1_Y=data1['bmi']
temp1_X=data1.drop(['bmi'] , axis=1)
temp2_X=data2.drop(['bmi'] , axis=1)
from sklearn.ensemble import RandomForestRegressor
mod=RandomForestRegressor()

mod.fit(temp1_X,temp1_Y)
pred=mod.predict(temp2_X)

k=0
for i in range(len(data['bmi'])):
    if(data['bmi'][i]==0.0):
        data['bmi'][i]=pred[k]
        k=k+1

It gives a better result than earlier methods.

4. Using Algorithms that Support missing values

All the machine learning algorithms don’t support missing values but some may. The k-nearest neighbors (KNN) algorithm is a simple, supervised machine learning algorithm that can be used to solve both classification and regression problems. It works on the principle of a distance measure that's why it can ignore a column when a value is missing. Naive Bayes can also support missing values when making a prediction. But in python, The sklearn implementations of naive Bayes and k-Nearest Neighbors do not support missing values. In these both algorithms you no need to handle missing values in each column as ML algorithms will handle them efficiently

Conclusion

In this article, I discuss many ways to deal with missing values and I try my best to explain these methods but there is no perfect method to handle missing data, all the methods give the best performance on different conditions depend on data. now let it stop here I hope you find this article useful and able to learn new things from it.

Thank you for reading👏

How to deploy Machine Learning model using Flask

atultyagi612 — Tue, 23 Mar 2021 05:53:37 +0000

Creating a machine learning model and doing predictions sounds cool. But, that’s not very useful for anyone if it’s only available on their machine. so in this tutorial, we are going to create a simple machine learning model and deploy the model into production using a Flask REST API. If you don't know much about REST API's please refer to my this article.

When a data machine learning engineer or a data scientist develops a machine learning model using Scikit-Learn, TensorFlow, or using any other tool the main goal is to make it available in production.

Deployment of machine learning models into production means making your models available to the end-users. This article makes you get started with putting your trained machine learning models into production using Flask API.

We are going to use RandomForestClassifier to predict the Quality of Red wine. we only create a simple model and save it into a file using pickle.

Why flask?

It is easy to use
Integrated unit testing support.
Faster than Django.

Before we get started I assume that you know about how to create a machine learning model using scikit-learn.

Dependencies

* scikit-learn
* Pandas
* Numpy
* flask
* pickle

Data

you can get data from this Kaggle dataset that we are going to use in creating the ML model.

Let's go

Make a ML model

first, we are going to create a simple ML model and convert the model which is in the form of a python object into a character stream using pickling.

# Import Dependencies
import pandas as pd
import numpy as np
import pickle
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Import data from csv file
data=pd.read_csv('winequality-red.csv')

# Split data into output and input
x=data.iloc[:,:-1]
y=data.iloc[:,-1:]

# Split data into train and test
x_train,x_test,y_train,y_test=train_test_split(x,np.array(y).flatten(),test_size=0.1,random_state=1)
model =RandomForestClassifier()
model.fit(x_train,y_train)

predic=model.predict(x_test)
accuracy_score(y_test,predic)

# Save model using pickle
pickle.dump(model, open('model.pkl','wb'))

Now a file model.pkl is created in your folder.

Rest API

Now create a restful API using a flask that takes inputs from the user and predicts the output using the ML model. If you don't know how to create a restful API using flask please refer to my this article.
file:app.py

from flask import Flask, jsonify
from flask_restful import Resource, Api, reqparse
import pickle

app=Flask(__name__)
api=Api(app)

data_arg=reqparse.RequestParser()
data_arg.add_argument("id" , type=str)
# load ML model
model=pickle.load(open('model.pkl', 'rb'))
class predict(Resource):
    def __init__(self):
        self.model1 = model
    def post(self):
        # parse data from post request
        args = data_arg.parse_args()
        # convert string into int list
        temp=args.id.strip('][').split(',')
        temp = [float(i) for i in temp]
        # predict output
        out=self.model1.predict([temp])
        # Return prediction
        return jsonify({"message":  int(out)})
api.add_resource(predict, '/')


if __name__ == '__main__':
    app.run(debug=True)

Now our API is ready. now it's time to create a web page to take input from the user.

Front end

file:front.html
HTML page:-

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
</head>

<body>
    <form id="form_id" onsubmit="sub(event)">
        <div class="form-group">
            <h1>Red Wine Quality  prediction</h1>
            <div class="element">
                <label for="inputPassword5">fixed acidity</label>
                <input type="number"  min="4.6" max="16"step="0.00001"class="form-control" value=10.4>
            </div>
            <div class="element">
                <label for="inputPassword5">volatile acidity</label>
                <input type="number"  min="0.12" max="1.6"step="0.00001"class="form-control" value=0.55>
            </div>
            <div class="element">
                <label for="inputPassword5">citric acid</label>
                <input type="number"  min="0" max="1"step="0.0000001"class="form-control" value=0.23>
            </div>
            <div class="element">
                <label for="inputPassword5">residual sugar</label>
                <input type="number"  min="0.9" max="15.5"step="0.00001"class="form-control" value=2.7>
            </div>
            <div class="element">
                <label for="inputPassword5">chlorides</label>
                <input type="number"  min="0.012" max="0.61"step="0.00001"class="form-control" value=0.091>
            </div>
            <div class="element">
                <label for="inputPassword5">free sulfur dioxide</label>
                <input type="number"  min="1" max="72"step="1"class="form-control" value=18>
            </div>
            <div class="element">
                <label for="inputPassword5">total sulfur dioxide</label>
                <input type="number"  min="6" max="278"step="1"class="form-control" value=48>
            </div>
            <div class="element">
                <label for="inputPassword5">density</label>
                <input type="number"  min="0.98" max="10"step="0.00001"class="form-control" value=0.9994>
            </div>
            <div class="element">
                <label for="inputPassword5">pH</label>
                <input type="number"  min="2.74" max="4.01"step="0.000001"class="form-control" value=3.22>
            </div>
            <div class="element">
                <label for="inputPassword5">sulphates</label>
                <input type="number"  min="0.33" max="2"step="0.000001"class="form-control" value=0.64>
            </div>
            <div class="element">
                <label for="inputPassword5">alcohol</label>
                <input type="number"  min="8.4" max="14.9"step="0.000001"class="form-control" value=10.3>
            </div>


        </div>
        <input type="submit">
    </form>
    <p id="demo"></p>
</body>

</html>

Now add styling to the page

<style>
    #form_id{
        display: flex;
    flex-direction: column;
    align-items: center;
    }

    .form-group{
        width: 50%;
        margin-top: 50px;
    }
    #demo{
        text-align: center;
        color: firebrick;
    font-size: larger;
    font-weight: 900;

    }
</style>

Now it's time for the main and the last part and javascript functions that take data from the HTML form and send it to the restful API using XMLHttpRequest.

 <script>
        let out=""
        function loadDoc(data) {
            var xhttp = new XMLHttpRequest();
            xhttp.onreadystatechange = function () {
                console.log(this.responseText)
                if (this.readyState == 4 && this.status == 200) {
                    out=JSON.parse(this.responseText).message
                    document.getElementById("demo").innerHTML = ` Result => ${out}`;
                }
            };
            xhttp.open("POST", "http://127.0.0.1:5000/.", true);
            xhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
            xhttp.send(`id=${data}`);
        }
        var myForm=""
        let ouput_data=[]
        function sub(e){
            ouput_data=[]
            e.preventDefault();
            console.log("Done")
            myForm = document.getElementById('form_id');
            for(let i=0;i<11;i++){
                ouput_data.push(myForm.elements[i].value)
            }
            senddata(ouput_data)
        }
        function senddata(dat){
            let temp="["+dat.toString()+"]"
            loadDoc(temp)

        }
    </script>

Result

Now run you run your rest API app.py.

Next, run yourfront.html using your IDE for mine its visual studio code.

Now fill the form and press the Submit button at the end of the form. The result is:-

You may also get an error in your console like:-Access to XMLHttpRequest at 'http://127.0.0.1:5000/' from origin 'null' has been blocked by CORS policy: The 'Access-Control-Allow-Origin'
Use this chrome extension this will resolve the issue.

Conclusion

This article shows a very simple way to deploy machine learning models. I used Random Forest Classifier to predict the Quality of Red Wine. One can use the knowledge gained in this blog to make some cool models and take them into production so that others can appreciate their work. I hope you find this article useful and able to learn new things from it.

Thank you for reading👏

Build a basic Todo app with Nodejs+Mongodb

atultyagi612 — Mon, 22 Mar 2021 07:00:01 +0000

In this tutorial, we will be going to build a simple to-do app with Nodejs using Express, MongoDB(to store Todo's in the database), and passport(for user authentication).

In this App, users will able to create delete, and mark Done their todo's.

Before we get start

We assume that you have a certain knowledge of Nodejs.
Good knowledge of Javascript.

Let's start

First, you need to set up your google authentication for login into the app. Please refer to my this blog and develop a simple google authentication app first.

I assume that you have completed your google authentication app. We use this app to get our to-do's according to the user's email id.

Other Dependencies

There are no additional dependencies required for this. we use only those dependencies we used in creating google authentication.

Todo Model

Now create a todo.js model for storing todo's into the database.
file:models/todo.js



const mongoose=require('mongoose');
const Todoschema=new mongoose.Schema({
    todo:{
        type:String,
        required:true,
    },
    email_:{
        type:String,
        required: true,
          },
    done:{
        type:String,
        required: true,
    }

});

module.exports=new mongoose.model("Todo",Todoschema);

Out to-do model is ready.

Update app.js file

Now update your app.js file. just add your new route we are going to create after that.
Add this code after the other two routes.



app.use(require("./routes/todo"))

Now our app.js file is ready.

Todo route

Now it's time for the main part. we are going to create our todo.js route which helps us to get, delete, mark done user's todo's from the database according to their email id.



const router=require('express').Router()
const Todo_model=require('../models/todo')

Create method to get all the user todo's



router.get('/add/todo',(req,res)=>{
    const {todo}=req.body;
    const newTodo=new Todo_model({todo,email_:req.user.email,done:"0"})
    if(todo==""){
        res.redirect('/')
    }
    newTodo.save()
    .then(()=>{
        console.log("done")
        res.redirect('/')
    })
    .catch(err=>console.log(err))
})

Method for deleting the todo.



.get("/delete/todo/:_id",(req,res)=>{
    const {_id}=req.params;
    Todo_model.deleteOne({_id})
    .then(()=>{
        console.log("deleted")
        res.redirect('/')
    })
    .catch((err)=>console.log(err));
})

Method to mark todo as done.



.get("/update/todo/:_id",(req,res)=>{
    const {_id}=req.params;
    const info=Todo_model.find();
    console.log(info)
    Todo_model.updateOne({_id}, { done:"1"})
    .then(()=>{
        console.log("deleted")
        res.redirect('/')
    })
    .catch((err)=>console.log(err));
});

module.exports=router;

Our to-do router is ready for export.👍

Now it's time to make our login page more interactive.

Update login.ejs

Now change your login.ejs file. its same as the previous google authentication once but just add few lines and a logo of our app.
file"views/login.ejs



<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.0-2/css/all.min.css"
        integrity="sha256-46r060N2LrChLLb5zowXQ72/iKKNiw/lAmygmHExk/o=" crossorigin="anonymous" />
    <link rel="stylesheet" href="/css/style.css">
    <title>Login</title>
</head>

<body>
    <div class="container login-container">
        <div class="card" style="margin-top:100px;">
            <div class="card-content">
                <h3 style="text-align: center;"><i class="fas fa-book"></i>&nbsp;To-do app</h3>
                <div class="section">
                    <p class="lead" style="text-align: center;">Lorem ipsum dolor sit amet consectetur adipisicing elit. Quod blanditiis hic
                        laudantium, quidem deleniti impedit.</p>
                </div>
                <div class="divider"></div>
                <div class="section" style="text-align: center;">
                    <a href="/auth/google" class="btn red darken-1">
                        <i class="fab fa-google left"></i> Log In With Google
                    </a>
                </div>
            </div>
        </div>
    </div>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
</body>

</html>

Todo main page

Now replace your previous views/index.ejs file with this one.
In this file, we just add some elements to show our todos and add button for deletion and marked as done.
file:views/index.ejs



<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Todo app</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/css/bootstrap.min.css" rel="stylesheet"
        integrity="sha384-giJF6kkoqNQ00vy+HMDP7azOuL0xtbfIcaT9wjKHr8RbDVddVHyTfAAsrekwKmP1" crossorigin="anonymous">
    <style>
        .hov:hover {
            background-color: rgb(201, 193, 209);
        }

        .basicstyle {
            cursor: pointer;
        }

        .logo {
            border-radius: 64px;
            height: 38px;
        }
        .libasic {
            display: flex;
            justify-content: space-between;
            font-family: sans-serif;
            font-size:1.2rem;
        }

    </style>
</head>

<body>

    <!-- As a link -->
    <nav class="navbar navbar-light bg-light">
        <div class="container-fluid">
            <a class="navbar-brand" href="/"><img class="logo" src=<%=userinfo.image %> alt=""> &nbsp; <%=
                    userinfo.firstName %>'s To-do</a>
            <a class="navbar-brand btn btn-danger btn-small" style="color: white;" href="/auth/logout">Logout</a>
        </div>
    </nav>


    <div class="container mt-5 ">
        <form action="/add/todo" class="d-flex" method="POST">
            <input type="text" name="todo" class="form-control" placeholder="Enter Todo">
            <input type="submit" value="Add" class="btn btn-md btn-primary mx-3">
        </form>

        <ul class="list-group mt-5 ">
            <% todo.forEach(todos=>{ %>

                <li class="list-group-item hov libasic" 
                <% if(todos.done==1) 
                { %>style=" background-color: #accec6 ; text-decoration: line-through;"
                <% } %> >
                    <%= todos.todo %>

                    <span>

                        <% if(todos.done==0) 
                        { %> <a href="/update/todo/<%=todos._id%>" class="btn btn-success btn-small" style="margin-right:0px">Done</a>
                        <% } %>

                        <a href="/delete/todo/<%=todos._id%>" class="btn btn-danger btn-small">Delete</a>
                    </span>
                </li>

                <% }) %>
        </ul>

    </div>

    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/js/bootstrap.bundle.min.js"
        integrity="sha384-ygbV9kiqUc6oa4msXn9868pTtWMgiQaeYH7/t7LECLbyPA2x65Kgf80OJFdroafW"
        crossorigin="anonymous"></script>
    <script src="ejs.min.js"></script>
</body>

</html>

Now our app is ready ✨🎉
Test this app Run the app by the command npm run dev then open your browser and visit:localhost:3000
You can take a demo of this app on this.

GitHub repo.

atultyagi612 / Todo-App

A Todo app using nodejs and google oauth

Conclusion

In this article, we create a simple Todo APP with Nodejs. we use the Express, EJS template, and MongoDB to store data and also use Passport for google authentication. Users can use this app to store their todos delete them and mark them as Done. I hope you find this article useful and able to learn new things from it. Now it's your time to try and build.

Thank you for reading👏

How to Build Simple RESTful APIs with Python and Flask

atultyagi612 — Sat, 20 Mar 2021 16:50:40 +0000

What is REST API?

A REST API (also known as RESTful API) is an application programming interface (API or web API) that conforms to the constraints of REST architectural style and allows for interaction with RESTful web services. REST stands for representational state transfer and was created by computer scientist Roy Fielding.more.

What is Flask?

Flask is a web framework. This means flask provides you with tools, libraries, and technologies that allow you to build a web application. This web application can be some web pages, a blog, a wiki, or go as big as a web-based calendar application or a commercial website. Flask is part of the categories of micro-framework. Micro-framework is normally a framework with little to no dependencies on external libraries.

Why use Flask for rest API?

Flask is able to achieve generally faster performance than Django, generally due to the fact that Flask is much more minimal in design. Flask also has NoSQL support. Flask also has Flask-RESTful an extension for Flask that provides additional support for building REST APIs.Flask-Restful is a lightweight abstraction that works with the existing ORM/libraries. Flask-RESTful encourages best practices with minimal setup. And also take less time to master.

Before you get started

Make sure you have Python installed on your system.
Postman Installed on your system. Postman helps to test your Rest API.

Flask Installation

Write the below command in your cmd for the Installation of Flask using pip.reference

pip install Flask

Install Flask-RESTful

Flask-RESTful is an extension for Flask that adds support for quickly building REST APIs more. the command for the installation

pip install flask-restful

Pandas Installation

We use pandas to CRUD data in CSV file.Write the below command in your cmd for the Installation of Pandas using pip.reference

pip install pandas

Lets Start ✨

First, we have to Import the dependencies.

from flask import Flask, jsonify 
from flask_restful import Resource, Api , reqparse
import pandas as pd

Now, let’s understand the working of the code line-by-line:

from flask import Flask, jsonify -> Import Flask class
jsonify -> jsonify is a function in Flask's flask. json module. jsonify serializes data to JavaScript Object Notation (JSON) format, wraps it in a Response object with the application/json mimetype.
from flask_restful import Resource, Api , reqparse -> import Flask-RESTful. it is an extension for Flask .
Resource->It is a base class that can define the routing for one or more HTTP methods for a given URL.
reqparse->Enables adding and parsing of multiple arguments in the context of a single request. Help
import pandas as pd -> Importing Pandas library as a name pd.

Now it's time for the main entry point for the application. Create an instance of the class.

app=Flask(__name__)
api=Api(app)

now add reqparse to our code. it Enables adding and parsing of multiple arguments in the context of a single request.
there is only 4 field(in CSV) in csv file so only add 4 arguments to be parse after initializing reqparse and also add help which return a description id if the argument is not valid.

data_arg=reqparse.RequestParser()
data_arg.add_argument("ID" , type=int ,help="Enter ID")
data_arg.add_argument("Name" , type=str ,help="Enter Name")
data_arg.add_argument("Language" , type=str ,help="Enter Language")
data_arg.add_argument("Age" , type=int ,help="Enter Age")

Create classes

Now create a class read_Delete which read and delete data from a CSV file and return Response. In this class we define function get This means that any GET Request on the url endpoint hit this function and delete function will hit by Delete request.

class read_Delete(Resource):
    def __init__(self):
        # read csv file
        self.data = pd.read_csv('data.csv')
    # GET request on the url will hit this function
    def get(self,ID):
        # find data from csv based on user input
        data_fount=self.data.loc[self.data['ID'] == ID].to_json(orient="records")
        # return data found in csv
        return jsonify({'message': data_fount})
    # Delete request on the url will hit this function
    def delete(self,ID):
        if ((self.data['ID'] == ID).any()):
            # Id it present delete data from csv
            self.data = self.data.drop(self.data["ID"].loc[self.data["ID"] == ID].index)
            self.data.to_csv("data.csv", index=False)
            return jsonify({"message": 'Deleted successfully'})
        else:
            return jsonify({"message": 'Not Present'})

Now create a class Create_Update which read data from a CSV file and perform update and create on data. In this class we define functions post , put means any POST PUT Request on the URL endpoint will be hitting their respective function.

class Create_Update(Resource):
    def __init__(self):
        # read data from csv
        self.data = pd.read_csv('data.csv')

    # POST request on the url will hit this function
    def post(self):
        # data parser to parse data from url
        args = data_arg.parse_args()
        # if ID is already present
        if((self.data['ID']==args.ID).any()):
            return jsonify({"message": 'ID already exist'})
        else:
            # Save data to csv
            self.data= self.data.append(args, ignore_index=True)
            self.data.to_csv("data.csv", index=False)
            return jsonify({"message": 'Done'})

    # PUT request on the url will hit this function
    def put(self):
        args = data_arg.parse_args()
        if ((self.data['ID'] == args.ID).any()):
            # if ID already present Update it
            self.data=self.data.drop(self.data["ID"].loc[self.data["ID"] == args.ID].index)
            self.data = self.data.append(args, ignore_index=True)
            self.data.to_csv("data.csv", index=False)
            return jsonify({"message": 'Updated successfully'})
        else:
            # If ID not present Save that data to csv
            self.data = self.data.append(args, ignore_index=True)
            self.data.to_csv("data.csv", index=False)
            return jsonify({"message": 'successfully Created'})

adding the defined resources along with their corresponding URLs

api.add_resource(read_Delete, '/<int:ID>')
api.add_resource(Create_Update,'/')

main function for calling the app

if __name__ == '__main__':
    app.run(debug=True)

🎉🎉 Our API is ready
Now, Visit the URL http://127.0.0.1:5000/ at your browser, you API is hosted at that URL.

Now its testing time

Let's run our code and open the Postman to be able to test the endpoints.

GET request

for this we have to pass the link with id at the end:

It works fine.

POST request

Let's try to input the data pass the link with ID and parameters.

PUT request

Let's try to update the data, set the PUT method to pass the link with input data for the update.

DELETE request

The last thing to do is the DELETE method, select it and pass the link with ID at the end.

Everything works fine cheer up.

Conclusion

In this article, we create a simple restful API with python. we use the Flask framework and Flask_RESTful library an extension for flask which makes it fast and easy. Our API allows us to CRUD data. I hope you will find this tutorial helpful and useful. Now it's your time to try and build.

Easy Google OAuth2 Authentication in Nodejs

atultyagi612 — Wed, 17 Mar 2021 12:09:56 +0000

IMAGE CREDITS
In this project, we simply use the passport google Strategy.
Passport is authentication middleware for Node.js. Extremely flexible and modular, a Passport can be unobtrusively dropped into any Express-based web application. A comprehensive set of strategies support authentication using a username and password, Facebook, Twitter, and more. reference
Before we get start assume that you have a good knowledge of JavaScript and Nodejs.
so without any further delay let's start 👍

Google credentials

First, we have to get Google's credentials.
To get credentials 'if don’t already have them ' go to Google developer Console

1)create a new project
2)Select the project and click credentials and then select OAuth client ID
3)Now Select Web Application in application type.
4)Input your app name or whatever else you like, in Authorized JavaScript origins add this linehttp://localhost:3000 and in Authorized redirect URIs field add this line http://localhost:5000/auth/google/callback and click to create.
5)Now copy your Google client ID and Google client secret
Help

Lets Initialize the New Project

To initialize the new project you just need to create a new folder "App name" and open a folder in visual studio (or any other IDE ) code or any other IDE and run the below code in the command line

 npm init

Just fill in the project name and any other detail or just skip. After the package.json file is generated.

Structure of the project

As with the reference of the above image create folders and files leave node_modules package-lock and package-json as they generate automatically.

Install Dependencies

These are the Dependencies we need to install for our project.

express
ejs
connect-mongo
dotenv
express-session
mongoose
passport
passport-google-oauth20

Install Dependencies by writing the below code in your terminal

npm i ejs connect-mongo dotenv express-session mongoose passport passport-google-oauth20

Setup App for a run

To start the server automatically we just need to install Nodemon which restart the server automatically when any change is detected

npm i -D nodemon

Setup application for developer run and normal run. Just change the Script section with the below code in package.json.

"scripts": {
    "start": "node app.js",
    "dev": "nodemon app.js"
  },

Start a local server

To start our app for testing/developer just simply type the following command in the command line:

npm run dev

The main work starts from there

You need to just put your google client id and secret in this file. And also the MongoDB URI like(mongodb://localhost:27017/) if you are hosting MongoDB from your system. if you are using Mongodb Atlas it like(mongodb+srv://XXXX:XXXX@cluster0.obaan.mongodb.net/{DBNAME}?retryWrites=true&w=majority)
file:config/config.env

PORT = 3000
MONGO_URI=mongodb+srv://XXXX:XXXX@cluster0.obaan.mongodb.net/{DBNAME}?retryWrites=true&w=majority
GOOGLE_CLIENT_ID = XXXXXXXXXX
GOOGLE_CLIENT_SECRET = XXXXXXXXXXXXXXXX

In my case we use Mongodb Atlas . you can refer this for getting mongodb atlas URI . and refer this for Google client id and secret if any problem occur .

Application

Its time code our app.js file this is the main file and it will sit in the root of our website.
In this file we have to setup our server.

file:app.js

Import all the necessary modules.

const express = require('express');
const mongoose=require('mongoose');
const dotenv = require('dotenv')
const passport = require('passport')
const session = require('express-session')
const MongoStore = require('connect-mongo')(session)
require('./config/passport')(passport)

Connect to mongodb and set express template.

var app=express();
const PORT = process.env.PORT||3000;
dotenv.config({ path: './config/config.env' })

mongoose.connect(process.env.MONGO_URI,{
    useNewUrlParser:true,
    useUnifiedTopology: true
})
app.use(express.static('public'))
app.set('view engine','ejs');

Initialize middleware and setup database for storing sessions.

app.use(express.urlencoded({extended:true}))
app.use(
    session({
      secret: 'keyboard cat',
      resave: false,
      saveUninitialized: false,
      store: new MongoStore({ mongooseConnection: mongoose.connection }),
    })
  )
  // Passport middleware
app.use(passport.initialize())
app.use(passport.session())

last part import routes

app.use(require("./routes/index"))
app.use('/auth', require('./routes/auth'))

app.listen(PORT,console.log(`listening at ${PORT}`))

Now our app.js file is ready🎉🎉

Routes

Now its time to code our routes
we are to code 2 routes files one auth.js for authentication and another one index.js for redirecting between pages
Let's code out the auth.js file.
file:auth.js

//Importing required modules 
const express = require('express')
const passport = require('passport')
const router = express.Router()

send to google to do the authentication.
In scopes, profile gets us their basic information including their name and email gets their emails.


router.get('/google', passport.authenticate('google', { scope: ['profile','email'] }))

Callback after google has authenticated the user.

router.get(
  '/google/callback',
  passport.authenticate('google', { failureRedirect: '/' }),
  (req, res) => {
    res.redirect('/log')
  }
)

For logout

router.get('/logout', (req, res) => {
  req.logout()
  res.redirect('/')
})

module.exports = router

Now our auth.js file is ready🎉🎉

Before creating the `index.js` file we have to create our middleware to ensure that the user is authenticated or not.

file:middleware/auth.js

module.exports = {
  // if user is authenticated the redirected to next page else redirect to login page
  ensureAuth: function (req, res, next) {
    if (req.isAuthenticated()) {
      return next()
    } else {
      res.redirect('/')
    }
  },
  // if user is authenticated and going to login page then redirected to home page if not authenticated redirected to login page  .
  ensureGuest: function (req, res, next) {
    if (!req.isAuthenticated()) {
      return next();
    } else {
      res.redirect('/log');
    }
  },
}

Now our middleware is ready let's code our next router index.js.
file:routes/index.js


const router = require('express').Router()
//importing middleware
const { ensureAuth, ensureGuest } = require('../middleware/auth')

router.get('/', ensureGuest ,(req, res) => {
    res.render('login')
  })

router.get("/log",ensureAuth, async(req,res)=>{
    res.render('index',{userinfo:req.user})
})
module.exports=router;

Configure Passport's Google startegy

file:config/passport.js

// import all the things we need  
const GoogleStrategy = require('passport-google-oauth20').Strategy
const mongoose = require('mongoose')
const User = require('../models/User')

module.exports = function (passport) {
  passport.use(
    new GoogleStrategy(
      {
        clientID: process.env.GOOGLE_CLIENT_ID,
        clientSecret: process.env.GOOGLE_CLIENT_SECRET,
        callbackURL: '/auth/google/callback',
      },
      async (accessToken, refreshToken, profile, done) => {
        //get the user data from google 
        const newUser = {
          googleId: profile.id,
          displayName: profile.displayName,
          firstName: profile.name.givenName,
          lastName: profile.name.familyName,
          image: profile.photos[0].value,
          email: profile.emails[0].value
        }

        try {
          //find the user in our database 
          let user = await User.findOne({ googleId: profile.id })

          if (user) {
            //If user present in our database.
            done(null, user)
          } else {
            // if user is not preset in our database save user data to database.
            user = await User.create(newUser)
            done(null, user)
          }
        } catch (err) {
          console.error(err)
        }
      }
    )
  )

  // used to serialize the user for the session
  passport.serializeUser((user, done) => {
    done(null, user.id)
  })

  // used to deserialize the user
  passport.deserializeUser((id, done) => {
    User.findById(id, (err, user) => done(err, user))
  })
}

User model

Now it's time to create our database model to user data in the database.
file:models/User.js

const mongoose = require('mongoose')

const UserSchema = new mongoose.Schema({
  googleId: {
    type: String,
    required: true,
  },
  displayName: {
    type: String,
    required: true,
  },
  firstName: {
    type: String,
    required: true,
  },
  lastName: {
    type: String,
    required: true,
  },
  image: {
    type: String,
  },
  email:{
type:String,
required: true,
  },
  createdAt: {
    type: Date,
    default: Date.now,
  },
})

module.exports = mongoose.model('User', UserSchema)

Good news at that time all the routes, models, and middlewares are ready the only things is ready is our HTML(EJS) Pages.

Login and main pages

Now its time to create our login page using bootstrap.
file:views/login.ejs

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/css/materialize.min.css">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.0-2/css/all.min.css"
        integrity="sha256-46r060N2LrChLLb5zowXQ72/iKKNiw/lAmygmHExk/o=" crossorigin="anonymous" />
    <link rel="stylesheet" href="/css/style.css">
    <title>Login</title>
</head>

<body>
    <div class="container login-container">
        <div class="card" style="margin-top:100px;">
            <div class="card-content">
                <div class="section" style="text-align: center;">
                    <a href="/auth/google" class="btn red darken-1">
                        <i class="fab fa-google left"></i> Log In With Google
                    </a>
                </div>
            </div>
        </div>
    </div>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/materialize/1.0.0/js/materialize.min.js"></script>
</body>

</html>

Lets Create Main page which appear after user login.
file:views/index.ejs

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Done</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/css/bootstrap.min.css" rel="stylesheet"
        integrity="sha384-giJF6kkoqNQ00vy+HMDP7azOuL0xtbfIcaT9wjKHr8RbDVddVHyTfAAsrekwKmP1" crossorigin="anonymous">
</head>

<body>

    <!-- As a link -->
    <nav class="navbar navbar-light bg-light">
        <div class="container-fluid">
            <a class="navbar-brand" href="/"><img class="logo" src=<%=userinfo.image %> alt=""> &nbsp; <%=
                    userinfo.firstName %></a>
            <a class="navbar-brand btn btn-danger btn-small" style="color: white;" href="/auth/logout">Logout</a>
        </div>
    </nav>




    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta1/dist/js/bootstrap.bundle.min.js"
        integrity="sha384-ygbV9kiqUc6oa4msXn9868pTtWMgiQaeYH7/t7LECLbyPA2x65Kgf80OJFdroafW"
        crossorigin="anonymous"></script>
    <script src="ejs.min.js"></script>
</body>

</html>

Preview

🎉🎉

Our Google Authentication App is ready.

Now it's your time to use that amazing Middleware passport.js Good Luck 😎🎶

Live Preview

Here is the demo. I use the above code in my project Todo app
Live Preview.
Want to build a to-do app? refer to my this article.

GitHub repo.

atultyagi612 / Google-Authentication-nodejs

Build a news app with Nodejs , Express , EJS and NewsApi

atultyagi612 — Wed, 17 Mar 2021 06:36:56 +0000

Today we are going to develop a simple News app with the help of NodeJS , Express , EJS and bootstrap.

There are going to be 2 main features in this website search and display news articles And we are going to use Newapi for getting news articles.

Lets Start

Initialize the New Project

To initialize the new project you just need to create a new folder "News App" and open folder in visual studio code or any other IDE and run the below code in command line

 npm init

This takes only few seconds and also asks few question about your project like project name , description etc. after that initialization is over and a file name "Package.json" is generated .

Structure of project

As with the reference of the above image create folders and files leave node_modules package-lock and package-json as they generate automatically .

Install Dependencies

These are the Dependencies we need to install for our project.

express
ejs
body-parser
axios
math
moment

For installing these Dependencies you just need to write the below code in your terminal

npm install express ejs body-parser axios math moment

Setup App for run

To start the server automatically we just need to install Nodemon which restart server automatically when any change is detected

npm install -D nodemon

Setup application for developer run and normal run. Just change the Script section with the below code in package.json.

"scripts": {
    "start": "node app.js",
    "dev": "nodemon app.js"
  },

Start developer local server

To start our app for testing/developer just simply type the following command in the command line:

npm run dev

Application

Lets code our app.js file this is the main file and it will sit in the root of our website.
In this file we have to setup our server .

file:-> app.js

const express = require('express')
const app=express()
const port = process.env.PORT||3000;
const bodyParser = require('body-parser');
const moment = require('moment')
app.locals.moment = moment;

// template engine  
app.use(express.static('public'))
app.set('view engine','ejs')

app.use(bodyParser.urlencoded({ extended: true }));
app.use('/',require('./routes/news'))

app.set('views','./views')

app.listen(port,()=> console.log("started"))

Routes

Lets build a extremal routes
External routing is a way of structuring your code so that it stays nice and organized by taking the route implementations outside of the main server file and moving them into a separate router file.

First you need your NewsApi API key for getting data , Go to NewsApi site and get your Api key then and simply replace YOUR_API in the url with your Api key in the below code
file:-> routes/news.js

const express = require('express')
const axios = require('axios')
const newsr=express.Router()
const moment = require('moment')
const math = require('math')


newsr.get('/',async(req,res)=>{
    try {
        var url = 'http://newsapi.org/v2/top-headlines?' +
          'country=in&' +
          'apiKey={YOUR_API}';

        const news_get =await axios.get(url)
        res.render('news',{articles:news_get.data.articles})




    } catch (error) {
        if(error.response){
            console.log(error)
        }

    }
})

newsr.post('/search',async(req,res)=>{
    const search=req.body.search
    // console.log(req.body.search)

    try {
        var url = `http://newsapi.org/v2/everything?q=${search}&apiKey={YOUR_API}`

        const news_get =await axios.get(url)
        res.render('news',{articles:news_get.data.articles})





    } catch (error) {
        if(error.response){
            console.log(error)
        }

    }
})


module.exports=newsr

Views

file:-> views/news.ejs

<!DOCTYPE html>
<html lang="en">


<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>News</title>
    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css"
        integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
        <link rel="preconnect" href="https://fonts.gstatic.com">
        <link href="https://fonts.googleapis.com/css2?family=PT+Sans:ital@1&display=swap" rel="stylesheet">
        <link rel="stylesheet" href="/css/style.css">
        <style>
            .form-control{
                border-color: red;
            }
        </style>
</head>

<body>

    <nav class="navbar navbar-expand-lg navbar-light bg-light navbar-fixed-top fixed-top">
        <a class="navbar-brand" href="/" style="font-weight: 700;
        font-size: 24px;"> <span style="color: red;" >X</span>-news</a>
        <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarSupportedContent"
            aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
            <span class="navbar-toggler-icon"></span>
        </button>

        <div class="collapse navbar-collapse searc-bar" id="navbarSupportedContent">

            <form class="form-inline my-2 my-lg-0" action="/search" method="POST">
                <input class="form-control mr-sm-2"  name="search" type="search" placeholder="Search" style="width: 522px;"
                    aria-label="Search">
                <button class="btn btn-outline-danger my-2 my-sm-0" type="submit">Search</button>
            </form>
        </div>
    </nav>





    <div class="news-container ">
        <div class="news ">

            <% articles.forEach(function(article,index){ %>

                <% if ((typeof article.url=='object') || (typeof article.title=='object') || (typeof article.urlToImage=='object') || (typeof article.content=='object')){ %>
                    <% } else{ %>
                        <a href="<%- article.url %>" target="_blank" class="news-box Hover-effect">

                            <img src="<%- article.urlToImage %>" alt="Image">
                            <h3>
                                <%- article.title %>
                            </h3>


                            <p>
                                <%- article.content.replace(/<[^>]+>/g, '') %>
                            </p>
                            <br>
                            <p style="margin-bottom: 0px; position: absolute; right: 0; bottom: 0; font-family: ui-monospace;">Updated: <span style="color:red;" ><%- new Date(article.publishedAt.slice(0,10)).toDateString() %>    <%- moment.utc(article.publishedAt).local().format('hh : mm A') %> </span></p>


                        </a>
                        <% } %>
                        <% }) %>
        </div>
    </div>
    <script src="moment.js"></script>
<script src="moment-timezone-with-data.js"></script>
    <script src="https://code.jquery.com/jquery-3.2.1.slim.min.js" integrity="sha384-KJ3o2DKtIkvYIK3UENzmM7KCkRr/rE9/Qpg6aAZGJwFDMVNA/GpGFF93hXpG5KkN" crossorigin="anonymous"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.12.9/umd/popper.min.js" integrity="sha384-ApNbgh9B+Y1QKtv3Rn7W3mgPxhU9K/ScQsAP7hUibX39j7fakFPskvXusvfa0b4Q" crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js" integrity="sha384-JZR6Spejh4U02d8jOt6vLEHfe/JQGiRRSQQxSfFWpi1MquVdAyjUar5+76PVCmYl" crossorigin="anonymous"></script>
</body>

</html>

Style

body{
    margin: 0;
    font-family: sans-serif;
    background-color: rgba(245, 238, 238, 0.89);
}
nav{
    margin-bottom: 40px;
}
img{
    max-width: 100%;
    transition: all 0.7s ;
}

.Hover-effect:hover{
    transform:  translateY(-3px) scale(1.01);

}
.Hover-effect:hover img{
    transform:  translateY(-5px) scale(1.02);
    box-shadow: 10px 9px 17px -10px rgba(0,0,0,0.28);
}
h3{
    font-size: 1.55rem;
}
h2{
    font-size: 1.55rem;
}
.news-container{
    margin-bottom: 69px;
    padding: 0 1rem;
    padding-top: 92px;
}
.news{
    display: grid;
    grid-template-columns: repeat(auto-fill,minmax(300px,1fr));
    grid-gap: 2rem;

}
.news-box{
    transition: all 0.3s ;
    text-decoration: none;
    color: rgb(29, 25, 25);

    background-color: white;
    padding: 20px;
}

.news-box:hover{
    text-decoration: none;
    color: rgb(29, 25, 25);;
    box-shadow: 0 3px 3px rgba(0,0,0,0.16), 0 3px 3px rgba(0,0,0,0.23);
}


.searc-bar{
    justify-content: center;
}

/* #effect ********************** */

.Hover-effect {
    position: relative;
}
:root{
    --underline_color: red;
    --underline-hight:0.175rem;
}
.Hover-effect::after {
    content: "";
    position: absolute;
    width: 100%;
    height: var(--underline-hight);
    background-color: var(--underline_color);
    left: 0;
    bottom: 0;
    transform: scale(0, 1);
    transition: transform 0.3s ease;
}



.Hover-effect:hover::after {
    transform: scale(1, 1);
}

DEV Community: atultyagi612

How to deal with Missing Values in Machine Learning

1. Variable Deletion

2. Replacing missing values with Mean/Median/Mode

Impute missing values for continuous variable

Impute missing values for categorical variable

Assigning An Unique Category

3. Predicting The Missing Values

4. Using Algorithms that Support missing values

Conclusion

How to deploy Machine Learning model using Flask

Why flask?

Dependencies

Data

Let's go

Make a ML model

Rest API

Front end

Result

Conclusion

Build a basic Todo app with Nodejs+Mongodb

Before we get start

Let's start

Other Dependencies

Todo Model

Update app.js file

Todo route

Update login.ejs

Todo main page

GitHub repo.

atultyagi612 / Todo-App

A Todo app using nodejs and google oauth

Conclusion

How to Build Simple RESTful APIs with Python and Flask

What is REST API?

What is Flask?

Why use Flask for rest API?

Before you get started

Flask Installation

Install Flask-RESTful

Pandas Installation

Lets Start ✨

Create classes

Now its testing time

GET request

POST request

PUT request

DELETE request

Conclusion

Easy Google OAuth2 Authentication in Nodejs

Google credentials

Lets Initialize the New Project

Structure of the project

Install Dependencies

Setup App for a run

Start a local server

The main work starts from there

Application

Routes

Before creating the index.js file we have to create our middleware to ensure that the user is authenticated or not.

Configure Passport's Google startegy

User model

Good news at that time all the routes, models, and middlewares are ready the only things is ready is our HTML(EJS) Pages.

Login and main pages

Preview

🎉🎉

Our Google Authentication App is ready.

Live Preview

GitHub repo.

atultyagi612 / Google-Authentication-nodejs

Build a news app with Nodejs , Express , EJS and NewsApi

Lets Start

Initialize the New Project

Structure of project

Install Dependencies

Setup App for run

Start developer local server

Application

Routes

Views

Before creating the `index.js` file we have to create our middleware to ensure that the user is authenticated or not.