DEV Community: saud khan

Data Cleaning in Pandas (Handling Missing Data)

saud khan — Fri, 24 Apr 2026 11:38:34 +0000

The Reality of Real‑World Data
Over the past few days, we have been working with perfect, pristine datasets. I built those datasets specifically so we could focus on learning commands like filter() and groupby() without any errors.

However, out in the real world, data is incredibly messy. Humans make typos when entering data, sensors go offline and miss readings, and database migrations often corrupt text. When Pandas encounters an empty cell in a CSV file, it fills it with a special marker called NaN (Not a Number).

If you try to run mathematical operations on a column filled with NaNs, your analysis will either crash or, even worse, return mathematically incorrect results that could lead to terrible business decisions. Today, I am going to teach you how to identify and clean this messy data professionally.

Note: Because our interactive workspace acts just like a real Jupyter Notebook, we only need to load our data in the very first cell. The remaining cells will remember the variables!

Step 1: Diagnosing the Mess (Finding Missing Data)
Before you can clean a house, you need to know where the dirt is. You cannot manually scroll through 500,000 rows looking for empty cells. Instead, we use Pandas diagnostic tools.

The combination of isnull() and sum() is the gold standard for diagnosing missing data. It will instantly tell you exactly how many missing values exist in every single column of your DataFrame.

import pandas as pd
import io

# Let's create a highly realistic, messy dataset
# Notice the blank spaces indicating missing data
messy_csv = """Emp_ID,Name,Department,Salary,Rating
101,Ali,Sales,65000,4.5
102,Sara,IT,,4.8
103,,Marketing,72000,3.9
104,Zoya,IT,88000,
105,Mike,,61000,4.1"""

df = pd.read_csv(io.StringIO(messy_csv))

# Diagnosing missing values
missing_data_report = df.isnull().sum()

print("--- Missing Values Report ---")
print(missing_data_report)

Look at that output. In less than a second, Pandas told us that we are missing 1 Name, 1 Department, 1 Salary, and 1 Rating. Now that we know where the problems are, we can fix them.

Step 2: Dropping Missing Data (The Nuclear Option)
The easiest and fastest way to deal with missing data is simply to delete any row that contains an empty value. In Pandas, we do this using the dropna() function.

However, this is the 'nuclear option'. If you drop rows blindly, you might lose valuable data. For example, if a row is missing a 'Rating' but has the employee's 'Salary', dropping the entire row deletes that perfectly good salary data too. Let's see what happens to our df.

# Dropping any row that contains at least one NaN value
clean_df_dropped = df.dropna()

print("--- Data After dropna() ---")
print(clean_df_dropped)

Learn More

What is LogicStack (LogicStack | Tech Insights & Development Hub (http://logicstack.org))? The End of "Tutorial Hell"

saud khan — Wed, 25 Mar 2026 13:03:22 +0000

Imagine you want to learn Python or SQL to level up your career. You open a popular tutorial, watch the instructor code for 30 minutes, and you feel like a genius. Everything makes perfect sense. But then, you open a blank code editor on your own computer... and you completely freeze. Your mind goes blank.
If this sounds familiar, you are not alone. This is what developers call "Tutorial Hell"—the endless cycle of watching tutorials without actually writing code or building anything yourself.
My name is Muhammad Saud, a Full-Stack Web Developer, and I experienced this exact frustration when I first started coding. I realized that the internet doesn't need another generic coding blog. It needs a place where reading and doing happen at the exact same time.
That is why I built
https://logicstack.org/
Not Just a Blog, A Live Playground
LogicStack is a free, interactive learning platform designed to bridge the gap between passive reading and active coding. Instead of asking you to install complex software, setup environments, or configure databases, LogicStack brings the entire coding environment directly into your web browser.
Here is what makes LogicStack fundamentally different from standard tutorial websites:
1. Zero-Setup Interactive Compilers
Whether you are following my "30 Days of SQL" course or the complete Python Curriculum, you will never just read code. Right beneath every concept, there is a live, fully functional code editor.

For SQL: We use a powerful SQLite WebAssembly engine that runs directly in your browser. You can write advanced queries, use Window Functions (RANK(), OVER()), and execute complex Joins instantly.
For Python: We utilize Pyodide (Web Workers) to run actual Python code. You can test logic, build algorithms, and see the terminal output in milliseconds. 2. The "Read Mode" & "Code Mode" Experience Learning a complex concept requires focus. That is why I engineered a unique toggle system. If you are trying to understand a deep theoretical concept, you can switch to Read Mode—the code editor slides away, and the text expands for a clean, distraction-free reading experience. Ready to test what you learned? Switch back to Code Mode, and your terminal and editor reappear instantly. 3. A Living, Breathing Community Coding by yourself can feel incredibly lonely. To solve this, I recently introduced the Ephemeral Community Feed. Right inside the Python playground, there is a live chat drawer. While you are writing code, you can share your "Aha!" moments, ask questions, or help others in real-time. Don't want to create an account? No problem. The system assigns you a fun, anonymous identity (like Code_Ninja or Python_Hacker). To keep the database clean and create a sense of urgency, all community thoughts automatically vanish after 24 hours! 4. Gamified Progress Tracking Motivation is key to finishing any course. On LogicStack, you don't just read articles; you complete them. You can mark topics as "Completed", track your overall course progress via a dynamic progress bar, and unlock interactive challenges to test your skills. Who is LogicStack For?
Beginners: Who want to start writing Python or SQL immediately without spending hours configuring their computer.
Self-Taught Developers: Who are stuck in tutorial hell and need a place to practice real-world scenarios.
Data Enthusiasts: Who want to master complex database queries and data structures interactively. The Vision Forward LogicStack is completely free to use. My goal isn't to put knowledge behind a paywall; it is to create the most frictionless learning experience possible. If you find value in the platform, you can support the project to get a premium badge next to your name in the community feed, but the core tools will always remain accessible to everyone. Stop just reading about code. Start writing it. Come visit https://logicstack.org/ today, open the interactive editor, and let's build something amazing together.

30 DAYS OF SQL - DAY1

saud khan — Sun, 22 Mar 2026 15:39:14 +0000

Welcome to Day 1 of the 30 Days of SQL series here at LogicStack! Whether your goal is to become a Data Scientist, a Data Analyst, a Backend Developer, or simply someone who wants to make sense of massive amounts of information, you are in the right place.

Every single day, the world generates 2.5 quintillion bytes of data. Companies like Google, Amazon, and Facebook need a way to store, manage, and extract meaning from this ocean of information. This is where SQL comes into play.

What is SQL and Why is it Essential for Data Analysis?

SQL stands for Structured Query Language. Simply put, it is the universal language used to talk to databases. Imagine a database as a massive, highly organized warehouse, and SQL as the forklift operator. You give the operator a set of instructions (a query), and they fetch exactly the box of data you need from the millions of boxes stored inside.

You might be wondering: "Why can't I just use Microsoft Excel?"

Excel is fantastic, but it has a hard limit. Once your spreadsheet hits around 1 million rows, it freezes, crashes, and becomes impossible to work with. Real-world companies deal with tens of millions, sometimes billions, of rows of data. SQL databases are designed to handle this massive scale flawlessly and return answers in milliseconds. As a data analyst, SQL is your primary weapon for cleaning, filtering, and analyzing raw data to find actionable business insights.

The Hierarchy: Databases vs. Tables

Before we write any code, we need to understand how data is structured. It follows a simple hierarchy:

RDBMS (Relational Database Management System): The software that runs everything (e.g., PostgreSQL, MySQL, SQLite).
Database: The main container or "virtual filing cabinet" for a specific project (e.g., "LogicStack_Ecommerce").
Table: The folders inside the cabinet. A table looks exactly like an Excel spreadsheet, with rows and columns.
[Image comparing a database table to an excel spreadsheet]
In a real-world scenario, you would first create a database using a command like CREATE DATABASE logicstack_db;. However, to make learning seamless, our interactive LogicStack SQL Engine automatically spins up a virtual database for you in memory. So, we will jump straight into creating tables!

Step 1: Creating a Table (CREATE TABLE)

To store data, we must first define its structure. We need to tell the database what the table is called and what kind of data each column will hold (Data Types).

Let's create a table named customers to keep track of people who visit our site. It will have three columns:

id: A unique number assigned to every customer. The data type is INTEGER.
name: The customer's full name. The data type is TEXT (or VARCHAR in some systems).
country: Where the customer is from. The data type is TEXT.
SQL LIVE EDITOR

CREATE TABLE customers (
  id INTEGER,
  name TEXT,
  country TEXT
);

Step 2: Adding Data (INSERT INTO)

Once the CREATE TABLE command runs, the database builds the empty structure. Now, we need to populate it with rows of actual data using the INSERT INTO command.

You specify the table name, the columns you want to fill, and then provide the VALUES in the exact same order.

INSERT INTO customers (id, name, country) VALUES (1, 'Ali', 'Pakistan');
INSERT INTO customers (id, name, country) VALUES (2, 'Sara', 'UK');
INSERT INTO customers (id, name, country) VALUES (3, 'John', 'USA');
INSERT INTO customers (id, name, country) VALUES (4, 'Aisha', 'UAE');

complete blog

Need help in Machine Learning

saud khan — Fri, 10 Jan 2025 07:10:04 +0000

Hello everyone,

I am a beginner in machine learning, and I am currently working with the Heart Disease UCI dataset downloaded from Kaggle. Upon exploring the data, I noticed that several columns have missing values, and I believe all these columns are important for the analysis. Here is a summary of the missing values in my dataset:

id: 0 missing values
age: 0 missing values
sex: 0 missing values
dataset: 0 missing values
cp: 0 missing values
trestbps: 59 missing values
chol: 30 missing values
fbs: 90 missing values
restecg: 2 missing values
thalch: 55 missing values
exang: 55 missing values
oldpeak: 62 missing values
slope: 309 missing values
ca: 611 missing values
thal: 486 missing values
num: 0 missing values
Could anyone please guide me on how to handle these missing values effectively, considering all columns are significant? Should I use imputation techniques, or are there better methods for this scenario? Any advice, especially with examples, would be greatly appreciated!

Thank you!