The first time I opened a dataset, I made a rookie mistake.
I was eager to build charts, train machine learning models, and uncover hidden patterns. Without asking a single question about the data, I jumped straight into analysis.
Within an hour, everything fell apart. Some columns contained missing values. Dates were stored as plain text. Numeric values were mixed with currency symbols. Duplicate records quietly inflated my calculations and one customer somehow appeared to be 247 years old.
The problem wasn’t my code.
The problem was that I skipped the most important conversation every data analyst should have:
What is this data trying to tell me?
That’s exactly what Exploratory Data Analysis, or EDA, is all about.
EDA is the process of investigating your dataset before making assumptions. It helps you understand its structure, discover patterns, identify anomalies, and ask better questions. Think of it as detective work before you write the final report.
In this article, I’ll show you how I approach EDA using Python and Pandas, with practical examples you can start using today.
What Is Exploratory Data Analysis?
Exploratory Data Analysis is the process of summarizing, visualizing and understanding a dataset before building dashboards or machine learning models.
Instead of asking, How do I build a prediction model?”, EDA asks questions like:
How many rows and columns are in this dataset?
Are there missing values?
Which columns are numerical or categorical?
Are there duplicate records?
Do any values look suspicious?
What patterns are already visible?
The answers to these questions often determine whether your analysis succeeds or fails.
1. Step 1: Import Pandas and Load your Data
Every EDA journey starts by loading the data
import pandas as pd
df = pd.read_csv(“sales_data.csv”)
Now the data is loaded, resist the urge to start plotting graphs
first, get to know your datasets.
2. Step 2: Take Your First Look
the quickest way to inspect a dataset is with head().
df.head()
this shows the first five rows.
want to see the last five rows?
df.tail()
Sometimes problems are hiding at the end of a dataset rather than the beginning.
3. Step 3: Understand the Structure
Next check the overall structure.
df.info()
This single command tells you:
Numbers of rows
Number of columns
Data types
Missing values
Memory usage
it's one the most useful functions in Pandas because it immediately reveals potential issues.
4. Step 4: Generate Summary Statistics
Numbers often tell stories that rows cannot.
df.describe()
You'll see statistics
Mean
Median
Standard deviation
Minimum
Maximum
Quartiles
If your average salary is ₦450,000 but the maximum salary is ₦250 million, that’s worth investigating.
Statistics help you spot unusual values long before you build visualizations.
5. Step 5: Check for Missing Values
Missing data is inevitable.
Find it with:
df.isnull().sum()
This shows exactly how many missing values exist in each column.
If one column has 95% missing values, you may decide to remove it entirely.
If another column only has three missing entries, filling them might be a better option.
EDA helps you make informed decisions instead of guessing.
Step 6: Look for Duplicate Records
Duplicate rows can quietly distort your analysis.
Check for them:
df.duplicated().sum()
If duplicates exist, investigate before removing them.
Sometimes duplicate records represent real events.
Other times they’re simply data entry mistakes.
Knowing the difference matters.
Step 7: Explore Individual Columns
Suppose you’re working with sales data.
How many unique product categories exist?
df[“diagnosis”].unique()
How many are there?
df[“diagnosis”].nunique()
What is the frequency of each category?
df[“diagnosis”].value_counts()
This quickly reveals which categories dominate your dataset.
Step 8: Explore Relationships Between Variables
One of the most powerful parts of EDA is discovering relationships.
Suppose you want total transaction amount by type.
aiml.groupby("type")["amount"].sum()
Want Average transaction amount by fraud status:?
aiml.groupby("isFraud")["amount"].mean()
These simple groupings often reveal business insights within seconds.
9. Step 9: Sort Your Data
Need to Identify the largest transactions:
aiml.sort_values(by="amount", ascending=False)
Looking for the lowest-performing branches?
Simply reverse the order.
10. Step 10: Find Correlations
When working with numerical data, correlation can uncover hidden relationships.
df.corr(numeric_only=True)
A correlation close to 1 suggests two variables move together.
A value near -1 suggests they move in opposite directions.
A value close to 0 indicates little or no linear relationship.
Remember, though:
Correlation does not imply causation.
Just because two variables move together doesn’t mean one causes the other.
Sorting helps highlight trends that might otherwise go unnoticed.
Building a Simple EDA Workflow
Whenever I receive a new dataset, I follow the same routine.
Load data
df = pd.read_csv(“data.csv”)
Inspect data
df.head()
Structure
df.info()
Summary statistics
df.describe()
Missing values
df.isnull().sum()
Duplicates
df.duplicated().sum()
Category counts
df[“category”].value_counts()
Group analysis
df.groupby(“region”)[“sales”].mean()
Correlation
df.corr(numeric_only=True)
This entire process usually takes less than fifteen minutes.
Yet it often reveals issues that could have ruined hours of analysis later.
Common Mistakes During EDA
Over the years, I’ve noticed beginners make the same mistakes.
They jump straight into visualization without understanding the data.
They ignore missing values.
They assume every outlier is an error.
They forget to check data types.
They build machine learning models before asking whether the dataset even makes sense.
EDA isn’t about generating pretty charts.
It’s about asking better questions.
Why EDA Matters in the Real World
Imagine you’re working for an e-commerce company.
Sales suddenly drop by 20%.
A dashboard alone won’t explain why.
EDA might reveal that:
One region stopped reporting data.
Product categories were renamed.
Thousands of orders are duplicated.
Customer IDs changed after a system update.
A pricing error affected only one city.
Without exploration, you might spend days solving the wrong problem.
EDA helps you find the real story hidden inside the data.
Final Thoughts
The best data analysts aren’t the ones who know the most algorithms.
They’re the ones who know how to ask the right questions before writing the first model.
Exploratory Data Analysis is more than a technical step. It’s a mindset of curiosity, patience, and investigation.
Every dataset has a story.
Your job isn’t just to analyze it.
Your job is to listen first.
The next time you open a CSV file, don’t rush into building charts or training models. Slow down, explore the data, challenge your assumptions and let the numbers guide you.
That’s how raw data becomes meaningful insight.
Top comments (0)