Introduction
Data cleaning is where most of a data analyst's time actually goes as most real-world datasets are rarely perfect. They can contain missing values, duplicates, incorrect values, inconsistent text, unusual numbers, and other problems that can affect the results of an analysis.
In this article, I will demonstrate how Pandas, one of Python's most commonly used data analysis libraries, can be used to clean a real-world-style housing dataset.
What is Pandas?
Pandas is a Python library used for working with structured data.
It provides two important data structures:
Series – similar to a single column
DataFrame – similar to a table containing rows and columns
1. Importing Pandas and Loading the Dataset
The first step is tom import pandas (using the standard alias as pd), and the CSV needs to be read into a Data Frame with pd.read_csv().
What does head() do?
head() displays the first five rows of the Data Frame.
This is useful because it gives us a quick look at what the data actually looks like.
2. Understanding the Dataset Before Cleaning
Before changing anything, it is important to understand what we are working with. One needs to get the shape, column types, and a statistical summary; this is what tells you where to look.
3. Checking the Data Types
One of the first things I check when cleaning a dataset is the data type of each column.
We can use:** df.info()**
This gives us information about: Column names, Number of non-null values and Data types.
For example, our dataset contains: int64, float64 and object
4. Checking for Missing Values
Missing values are very common in real-world datasets. We can find them using: df.isnull().sum(). For our dataset, we discover missing values in three columns (service_charge_kes, security_rating and satisfaction_score)
Checking on missing values is very important as failure to identify and handle those values early can severely compromise your entire data analysis pipeline.
5. What Should We Do With Missing Values?
There is no single solution for every missing value.
We need to understand the column before deciding what to do.
For example, for a numeric variable such as security_rating, we could replace missing values with the median:
Why use the median?
The median is often useful when a variable may contain extreme values because it is less affected by outliers than the mean.
However, we should not automatically fill every missing value. For example, a missing service charge could mean that the property genuinely does not have a service charge. The correct treatment depends on the meaning of the missing value.
6. Checking for Duplicate Records
Duplicate records can cause us to count the same property more than once. Pandas makes it easy to check for duplicates: For our dataset, the result is 0, This means there are no completely duplicated rows.
7. Checking for Outliers
Another important part of data cleaning is identifying outliers. An outlier is a value that is unusually far from most other observations. For example, our dataset contains some very high monthly rents. An outlier is not automatically an error. A luxury property in Karen or Lavington could legitimately have a very high rent.
Therefore, instead of immediately deleting these records, we should investigate them.
8. Using the IQR Method to Detect Outliers
One common statistical approach is the Interquartile Range (IQR). This gives us properties whose rents are unusually high compared with the rest of the dataset.
Again, the purpose of this step is investigation, not automatic deletion.
9. Filtering the Data
Pandas also allows us to filter records.
For example, we can find properties in Karen. The & means AND. So both conditions must be true. This kind of filtering is extremely useful when investigating suspicious records.
Conclusion
Data cleaning is one of the most important stages of a data science project because the quality of our analysis depends heavily on the quality of our data. Using the Nairobi Housing Statistics Dataset, we used Pandas to investigate:
Missing values
Duplicate records
Potential outliers
Data validation
One of the biggest lessons I learned from this process is that not every unusual value is an error.
For example, a monthly rent of KSh 5,000,000 may look suspicious, but it should not automatically be deleted. We need to investigate the context before making that decision. Pandas makes the technical part of this process much easier, but effective data cleaning still requires analytical thinking.
Top comments (0)