DEV Community: Anirban

Correlation is not the causation!!

Anirban — Sun, 17 Jan 2021 08:38:28 +0000

Many times we hear that "the correlation is not the causation" or "correlation does not imply causation". But what do they mean? Let's understand this phrase using some statistics and examples because most of the time these terms are mostly misunderstood and often used interchangeably.

What are even these things?

Correlation is a statistical technique that tells us how strong a pair of variables are linearly related. If we measure this relationship, we will get a number between -1 to 1. The sign of the number denotes how one will change if the other variable changes. If we get a number 0, that means there is no correlation between those two variables.

Example: Correlation between ice cream sales and sunglass sold.

The sales of ice cream increases as the sales of sunglass increases.

Causation adds a cause to the relationship. If two variables are correlated then it says any change in one variable will cause change in the other variable. That means these two variables are in cause and effect relationship. One event will help to occur the other event.

Example: When the wind blows faster then a windmill produces more power.
Here, the faster wind is the cause to rotate the windmill faster and the effect of this event is more power.

So, now we understand what correlation and causation are, let's understand why "Correlation does not imply causation!" with the famous example.

"Ice cream sales are correlated with homicides in New York"

This hypothesis says that as the sales of ice cream rise and fall, the number of homicides also rise and fall. But does ice cream consumption causing the death of people?

The answer is "No". If two things are correlated that doesn't mean that one will cause another.

When two independent things are tied together, these can be either be bound by causality or correlation. But in most cases, it is just a coincidence. Correlation is something we think of when we cannot see under the hood. The less information we have, the more we are forced to see correlations. Similarly, if we have more information then we will be able to see the actual causal relationships.

Why correlation is not the causation in this example?

Generally, we collect some samples to test our hypothesis. What if our sample is not large enough and a little too biased? What if there is a hidden factor that we did not record in our sample dataset? These things can influence the causation and correlation of two variables. Here in this example, weather is the hidden factor.

Let's consider a sunny summer day. It is a lovely day to go outside and people are enjoying the beach. As the day is hot, people are buying ice creams also people are exposed to accidents. The weather is causing a rise in the ice cream sale and homicides.

So, there is no causal relationship between ice cream sales and the homicide rate. Sunny weather is the hidden factor that is bringing both factors together. Also, ice cream sale and the homicide rate both have a causal relationship with the weather.

Don't jump to a conclusion too fast!

When we see a correlation is present between two variables, do not jump to a conclusion too early. Always take time to explore if any underlying influencing factor is present or not. If such a factor is present then verify if they are correlated or not and then conclude. You can consider the below questions before concluding.

Do we have enough samples?
Did we consider the seasonality?

Hope this post will clear your doubts!
My Twitter.

Thanks for reading!!

How to setup a cloud deployable machine learning model using Flask 🚀

Anirban — Fri, 08 Jan 2021 16:35:03 +0000

A well trained machine learnig model is not useful untill we are able to use it to predict on the future data. These kind of predictive models are used to solve several problems. So, we are going to learn how to use a trained machine learning model to predict on future data which will be feed into the model through an API.

In order to use a maching learning model through a callable API, we need to have a machine learning model in the first place. In this post we are not concentrating to create a model from the scratch (I will write another post on how to create a basic artificial neural network model using tensorflow) instead, we are going to use a model trained using tensorflow and keras libraries on the 'Iris' (flower) dataset. It is very famous dataset and we know that it has 3 classes (setosa, versicolor, virginica) present on the target variable.

Another thing we need is the Flask library to create the API. So. let's install flask usnig pip.
pip install flask

After installing flask, let's open sublime text3/ any text editor and create a python file. I am going to name this file as my_app.py. Then I am going to test if the environment is properly setup by creating a basic flask application with only one '/' route.

from flask import Flask

app = Flask(__name__)

@app.route("/")
def index():
    return '<h1>FLASK APP IS RUNNING!</h1>'

if __name__ == '__main__':
    app.run()

Now, let's open the command prompt in the current working directory where the my_app.py file is located and run the program using python my_app.py command. We should be able to see in the command prompt like below.

If we open http://127.0.0.1:5000 in the browser, then we should be able to see the app is running. If you see like below then we are set to proceed.

Now, we are going to follow below steps to create an API using Flask and call it through python script.

First, we will load the model using tensorflow.keras load_model() method.
Then, we will load the fitted Scaler object (i saved as pickle file) using joblib load() method to transform the new incoming data.
Create a function to be able to take a model object, a scaler object and the new data to predict.
Create an API endpoint using Flask .
Test the API using postman.
Call the API using Python script and get the prediction.

Load the model

We are going to import load_model from tensorflow.keras.models as the model is build using tensorflow. The model can be saved by any method.

from tensorflow.keras.models import load_model

Then we will use below code to load the model in my variable called flower_model. Here, I saved my model as my_model.h5 in the current working directory.

flower_model = load_model("my_model.h5")

Load the scaler object

I saved my fitted MinMaxScaler object in a pickle file in this current working directory as my_scaler.pkl. So, I am going to use joblib to load this scaler object in my variable called flower_scaler. Before that, we need to import the joblib module.

import joblib
flower_scaler = joblib.load("my_scaler.pkl")

Creating a function that return prediction

We are going to create a function that can take a model, a scaler and a data. The data is going to be in JSON format. So, we are going to read each field and will save into variables. This way we can create a list with similar shape of the training features. We have 4 features in this case. These are sepal_length, sepal_width,petal_length,petal_width. Then we will transform the data using scaler object and will use the model to predict the classes. The function code is given below.

def return_prediction(model, scaler, data):
    s_len = data["sepal_length"]    
    s_wid = data["sepal_width"]    
    p_len = data["petal_length"]    
    p_wid = data["petal_width"]

    classes = np.array(['setosa', 'versicolor', 'virginica'])

    flower = [[s_len, s_wid, p_len, p_wid]]

    flower = scaler.transform(flower)

    class_ind = model.predict_classes(flower)[0]

    return classes[class_ind]

We need to import numpy as np in the top in order to convert the list labels (['setosa', 'versicolor', 'virginica']) into numpy array

Creating API endpoint using Flask

I am going to create a route like /api/flower which can respond to a HTTP Post method. I am going to use request module to extract the JSON data posted by a client. Then I am going to use return_prediction() function to predict on this new data and return a JSON object using jsonify.

@app.route("/api/flower", methods=['POST'])
def flower_prediction():
    content = request.json
    result = return_prediction(flower_model, flower_scaler, content)
    return jsonify(result)

We need to import request and jsonify modules from flask library

from flask import request, jsonify

Testing the API using Postman

Now, I am going to run the program using below command in command prompt from the working directory. python my_app.py. This will launch the server and we are ready to test the API.

Now, let's open Postman and issue a post request to the endpoint http://127.0.0.1:5000/api/flower. We will create a body of the request in JSON format and will send the request.

{
    "sepal_length":5.1,
    "sepal_width":3.5,
    "petal_length":1.4,
    "petal_width":0.2
}

We can see the status code is 200 OK and the API has returned it's prediction as 'setosa' based on the input data.

Calling the API using Python script

Next, we are going to write a python script to call the API. We need to import request library in order to do that.

import requets

Then we will create a dictionary object to pass through the API.

flower_example = {
    "sepal_length":5.1,
    "sepal_width":3.5,
    "petal_length":1.4,
    "petal_width":0.2
}

Finally, we will call the API using post method and print the result.

result = requests.post("http://127.0.0.1:5000/api/flower", json=flower_example)
print(result)

That's all for today. We will talk about how to deploy this model into cloud in my next post.
My Twitter for any suggestion
Have a great one!

JavaScript Promise

Anirban — Thu, 09 Jul 2020 21:18:26 +0000

A promise in JavaScript is an object that may produce a single value any time in the future. It may give us either a resolved value or a reason that it is not resolved (rejected). A promise can be in one of the following three states.

Fulfilled
Rejected
Pending

We can create a promise like below. The constructor function of the promise takes a function that has parameters that either resolves or rejects.

const promise = new Promise((resolve, reject) => {
    if (true) {
        resolve("It worked");
    } else {
        reject("Error, it broke");
    }
});

Here, we can check the returned value from the promise by chaining with .then like below.

promise.then((result) => console.log(result));

This should log "It worked" in the console as the promise was resolved.

We can catch the rejected value comes out of the promise by chaining with .catch like below.

const promise = new Promise((resolve, reject) => {
    if (false) {
        resolve("It worked");
    } else {
        reject("Error, it broke");
    }
});

promise
.then((result) => console.log(result))
.catch((err) => console.log(err));

This code should write "Error, it broke" in the console.

Also, we can get the resolved results from the multiple promises using Promise.all() method.

const promise1 = new Promise((resolve, reject) => {
    setTimeout(resolve, 100, "Promise1");
});

const promise2 = new Promise((resolve, reject) => {
    setTimeout(resolve, 1000, "Promise2");
});

const promise3 = new Promise((resolve, reject) => {
    setTimeout(resolve, 5000, "Promise3");
});

Promise.all([promise1, promise2, promise3]).then((results) => {
    console.log(results);
});

Here, we will get an array as an output in the console after 5 seconds. The output will look like [ 'Promise1', 'Promise2', 'Promise3' ].

That's all for today. If you like this post and want to talk more about it, please feel free to DM me here
Have a great one!