Hey Everyone! This side your favourite guy, Lucky Jha. So today I came up with something more interesting which came out of my knowledge related to machine learning.
You guys already know my curiosity regarding and about Machine Learning. It's like whenever I open up anything related to machine learning I give my 110% to it. Even today I was starting up with my machine learning project. I started web scraping and all to gather data.
And soon I found the CSV dataset on kaggle. And the data was about the customers who churned and not churned after making a contract with IBM, I don't know whether it was official or unofficial dataset released by IBM. But what it taught me today is way more important than anything.
So I was basically validating my data using my data validation library developed by me called Veris. I was doing validation, soon I ran a command to check whether for the null column to check whether how many rows are null?? There was a column "TotalCharge" which was null for 11 rows.
So a minute after watching the data, I decided to drop those rows with null column but soon realised after 10 minutes of data investigation, I hovered over all the columns and found out some patterns between tenure, contract, and churn.
I realised a pattern which was showing that person with tenure "0", has totalCharge = 0, So I realised that the person hasn't even charged even once. He hasn't paid his first month bill that's why his or her totalCharge is null not even 0.
So Data Investigation is a very important step in the process of data validation, you should be able to answer all the questions related to your data. You should know which column is responsible for model training, which column effects model training, which doesn't. Data Investigation is important in every term.
Its more important before data preprocessing, Exploratory Data Analysis. For data validation, data investigation is very important. You should be able to answer every question about the data like you own the business.
So to conclude the entire blog in one line summary, I'd say one thing only, investigate the data before you drop the null rows it might help you to train the model for missing values.
Will Keep sharing my day-to-day journey and learnings. I love writing in public. Hope you love it like your own son, brother, friend, boyfriend.
Top comments (0)