Data Preprocessing Techniques for ML Models

Data preprocessing is the act of carrying out certain actions or steps on a dataset before it is used for machine learning or other tasks. Data preprocessing involves cleaning, formatting or transforming data in order to improve its quality or ensure that it is suitable for its main purpose (in this case, training a model). A clean and high-quality dataset enhances the machine learning model's performance.

Common issues with low-quality data include:

• Missing values
• Inconsistent formats
• Duplicate values
• Irrelevant features

In this article, I will show you some of the common data preprocessing techniques to prepare datasets for use in training models. You will need basic knowledge of Python and how to use Python libraries and frameworks.

Requirements:
The following are required to be able to get the best out of this guide

• Python 3.12
• Jupyter Notebook or your favourite notebook
• Numpy
• Pandas
• Scipy
• Scikit learn
• Melbourne Housing Dataset

You can also check out the output of each code in these Jupyter notebooks on Github.

Setup

If you haven't installed Python already, you can download it from the Python website and follow the instructions to install it.

Once Python has been installed, install the required libraries

pip install numpy scipy pandas scikit-learn

Install Jupyter Notebook.

pip install notebook

After installation, start Jupyter Notebook with the following command

jupyter notebook

This will launch Jupyter Notebook in your default web browser. If it doesn't, check the terminal for a link you can manually paste into your browser.

Open a new notebook from the File menu, import the required libraries and run the cell

import numpy as np
import pandas as pd
import scipy
import sklearn

Go to the Melbourne Housing Dataset site and download the dataset. Load the dataset into the notebook using the following code. You can copy the file path on your computer to paste in the read_csv function. You can also put the csv file in the same folder as the notebook and import the file as seen below.

data = pd.read_csv(r"melb_data.csv")

# View the first 5 columns of the dataset
data.head()

Split the data into training and validation sets

from sklearn.model_selection import train_test_split

# Set the target
y = data['Price']

# Firstly drop categorical data types
melb_features = data.drop(['Price'], axis=1) #drop the target column

X = melb_features.select_dtypes(exclude=['object'])

# Divide data into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(X, y, train_size=0.8, test_size=0.2, random_state=0)

You have to split data into training and validation sets in order to prevent data leakage. As a result, whatever preprocessing technique you carry out on the training features set is the same as the one you carry out on the validation features set.

Now the dataset is ready to be processed!

Data Cleaning

Handling missing values
Missing values in a dataset are like holes in a fabric that are supposed to be used to sew a dress. It spoils the dress before it is even made.

There are 3 ways to handle missing values in a dataset.

1. Drop the rows or columns with empty cells

#Drop the rows with empty cells from the original dataframe
data.dropna(inplace=True)

#Drop the columns with empty cells
#firstly, get the names of columns with empty cells
cols_with_empty_cells = [col for col in X_train.columns if X_train[col].isnull().any()]

#secondly, drop the columns with the empty cells
removed_X_train_cols = X_train.drop(cols_with_empty_cells, axis=1)
removed_X_valid_cols = X_valid.drop(cols_with_empty_cells, axis=1)

The issue with this method is that you may lose valuable information that you are to train your model with. Unless most values in the dropped rows or columns are missing, there is no need to drop either rows or columns with empty cells.

1. Impute values in the empty cells You can impute or fill in the empty cells with the mean, median or mode of the data in that particular column. SimpleImputer from Scikit learn will be used to impute values in the empty cells

from sklearn.impute import SimpleImputer

# Impute values
imputer = SimpleImputer()
imputed_X_train_values = pd.DataFrame(imputer.fit_transform(X_train))
imputed_X_valid_values = pd.DataFrame(imputer.transform(X_valid))

# Imputation removed column names so we put them back
imputed_X_train_values.columns = X_train.columns
imputed_X_valid_values.columns = X_valid.columns

# See the new columns and their values
X_train.head()

1. Impute and notify How this works is that you impute values in the empty cells but you also create a column that indicates that the cell was initially empty.

# Make new columns indicating what will be imputed
# The column will have booleans as values
for col in cols_with_empty_cells:
    X_train[col + '_was_missing'] = X_train[col].isnull()
    X_valid[col + '_was_missing'] = X_valid[col].isnull()

# Impute values
imputer = SimpleImputer()
imputed_X_train_values = pd.DataFrame(imputer.fit_transform(X_train))
imputed_X_valid_values = pd.DataFrame(imputer.transform(X_valid))

# Imputation removed column names so we put them back
imputed_X_train_values.columns = X_train.columns
imputed_X_valid_values.columns = X_valid.columns

# See the new columns and their values
X_train.head() 

Duplicate removal
Duplicate cells mean repeated data and it affects model accuracy. The only way to deal with them is to drop them.

# Check for the number of duplicate rows in the dataset
X_train.duplicated().sum()

# Drop the duplicate rows
X_train.drop_duplicates(inplace=True)
X_valid.drop_duplicates(inplace=True)

Dealing with outliers
Outliers are values that are significantly different from the other values in the dataset. They can be unusually high or low compared to other data values. They can arise due to entry errors or they could genuinely be outliers.

It is important to deal with outliers or else they will lead to inaccurate data analysis or models. One method to detect outliers is by calculating z-scores.

The way it works is that the z-score is used to check if a data point is 3 points or more away from the mean value. This calculation is done for every data point. If the z-score for a data point equals 3 or a higher value, the data point is an outlier.

from scipy import stats
import pandas as pd

# Calculate Z-scores for each row in the training and validation datasets
X_train_zscores = stats.zscore(X_train, axis=1)
X_valid_zscores = stats.zscore(X_valid, axis=1)

# Define the threshold for outlier detection
threshold = 3

# Identify rows in X_train and X_valid with values above the threshold (meaning they're outliers)
outliers_train = np.abs(X_train_zscores) > threshold
outliers_valid = np.abs(X_valid_zscores) > threshold

# Remove rows identified as outliers from X_train and X_valid (~ means NOT)
X_train_no_outliers = X_train[~outliers_train]
X_valid_no_outliers = X_valid[~outliers_valid]

# Display the results
print("Original X_train shape:", X_train.shape)
print("X_train shape after removing outliers:", X_train_no_outliers.shape)

print("Original X_valid shape:", X_valid.shape)
print("X_valid shape after removing outliers:", X_valid_no_outliers.shape)

Data Transformation

Normalization
You normalize features so they can be described as a normal distribution.

A normal distribution (also known as the Gaussian distribution) is a statistical distribution where there are roughly equal distances or distributions above and below the mean. The graph of the data points of a normally distributed data form a bell curve.

The point of normalizing data is if the machine learning algorithm you want to use assumes that the data is normally distributed. An example is the Gaussian Naive Bayes model.

from sklearn.preprocessing import MinMaxScaler

# Initialize the MinMaxScaler
scaler = MinMaxScaler()

# Fit the scaler on the training data and transform it
X_train_normalized = scaler.fit_transform(X_train)

# Transform the validation data using the same scaler
X_valid_normalized = scaler.transform(X_valid)

# Convert the normalized data back into DataFrames to keep column names
X_train_normalized = pd.DataFrame(X_train_normalized, columns=X_train.columns, index=X_train.index)
X_valid_normalized = pd.DataFrame(X_valid_normalized, columns=X_valid.columns, index=X_valid.index)

# Display the results
print("First few rows of normalized X_train:")
print(X_train_normalized.head())

print("First few rows of normalized X_valid:")
print(X_valid_normalized.head())

Standardization
Standardization transforms the features of a dataset to have a mean of 0 and a standard deviation of 1. This process scales each feature so that it has similar ranges across the data. This ensures that each feature contributes equally to model training.

You use standardization when:

• The features in your data are on different scales or units.
• The machine learning model you want to use is based on distance or gradient-based optimizations (e.g., linear regression, logistic regression, K-means clustering).

You use StandardScaler() from the sklearn library to standardize features.

from sklearn.preprocessing import StandardScaler

# Initialize the StandardScaler
scaler = StandardScaler()

# Fit the scaler on the training and validation data and transform them
X_train_standardized = scaler.fit_transform(X_train)
X_valid_standardized = scaler.transform(X_valid)

# Convert the standardized data back into DataFrames to keep column names
X_train_standardized = pd.DataFrame(X_train_standardized, columns=X_train.columns, index=X_train.index)
X_valid_standardized = pd.DataFrame(X_valid_standardized, columns=X_valid.columns, index=X_valid.index)

# Display the results
print("First few rows of standardized X_train:")
print(X_train_standardized.head())

print("First few rows of standardized X_valid:")
print(X_valid_standardized.head())

Conclusion

Data preprocessing is not just a preliminary stage. It is part of the process of building accurate machine learning models. It can also be tweaked to fit the needs of the dataset you are working with.

Like with most activities, practice makes perfect. As you continue to preprocess data, your skills will improve as well as your models.

I would love to read your thoughts on this 👇