DEV Community

Cover image for Handling Outliers 101: Why the IQR Method is Your Go-To Tool
allan-pg
allan-pg

Posted on

Handling Outliers 101: Why the IQR Method is Your Go-To Tool

Introduction

Before uncovering any insights from real-world data, it is important to scrutinize your data to ensure that data is consistent and free from errors. However, Data can contain errors and some values may appear to differ from other values and these values are known as outliers. Outliers negatively impact data analysis leading to wrong insights which lead to poor decision making by stake holders. Therefore, dealing with outliers is a critical step in the data preprocessing stage in data science. In this article, we will asses IQR method of handling outliers.

Outliers

Outliers are data points that differ significantly from the majority of the data points in a dataset. They are values that fall outside the expected or usual range of values for a particular variable. outliers occur due to various reason for example, error during data entry, sampling errors. In machine learning outliers can cause your models to make incorrect predictions thus causing inaccurate predictions.

Detecting outliers in a dataset using Jupyter notebook

  • Import python libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
plt.style.use('ggplot')
Enter fullscreen mode Exit fullscreen mode
  • Load your csv file using pandas
df_house_price = pd.read_csv(r'C:\Users\Admin\Desktop\csv files\housePrice.csv')
Enter fullscreen mode Exit fullscreen mode
  • Check the first five rows of house prices data set to have a glimpse of your datafrane
df_house_price.head()
Enter fullscreen mode Exit fullscreen mode

First Five rows of your dataframe

  • Check for outliers in the price column by use of a box plot
sns.boxplot(df_house_price['Price'])
plt.title('Box plot showing outliers in prices')
plt.show()
Enter fullscreen mode Exit fullscreen mode

Box plot

  • From the box plot visualization the price column has outlier values
  • Now we have to come up with ways to handle these outlier values to ensure better decision making and ensure machine learning models make the correct prediction

IQR Method of handling outlier values

  • IQR method means interquartile range measures the spread of the middle half of your data. It is the range for the middle 50% of your sample.

Steps for removing outliers using interquartile range

  • Calculate the first quartile (Q1) which is 25% of the data and the third quartile (Q3) which is 75% of the data.
Q1 = df_house_price['Price'].quantile(0.25)
Q3 = df_house_price['Price'].quantile(0.75)
Enter fullscreen mode Exit fullscreen mode
  • compute the interquartile range
IQR = Q3 - Q1
Enter fullscreen mode Exit fullscreen mode
  • Determine the outlier boundaries.
lower_bound = Q1 - 1.5 * IQR
Enter fullscreen mode Exit fullscreen mode

Lower Bound

  • Lower bound means any value below -5454375000.0 is an outlier
upper_bound = Q3 + 1.5 * IQR
Enter fullscreen mode Exit fullscreen mode

upper bound

  • Upper bound means any value above 12872625000.0 is an outlier

  • Remove outlier values in the price column

filt = (df_house_price['Price'] >= lower_bound) & (df_house_price['Price'] <= upper_bound)

df = df_house_price[filt]
df.head()
Enter fullscreen mode Exit fullscreen mode

clean dataframe

  • Box plot After removing outliers
sns.boxplot(df['Price'])
plt.title('Box plot after removing outliers')
plt.show()
Enter fullscreen mode Exit fullscreen mode

Box plot without outliers

Different methods of handling outlier values

  • Z-Score method
  • Percentile Capping (Winsorizing)
  • Trimming (Truncation)
  • Imputation
  • Clustering-Based Methods e.g DBSCAN

Conclusion

IQR method is simple and robust to outliers and does not depend on the normality assumption. The disadvantage is that it can only handle univariate data, and that it can remove valid data points if the data is skewed or has heavy tails.

Thank you
follow me on linked in and on github for more.

Top comments (0)