DEV Community

Cover image for Introduction to Pandas
Ismile Hossain
Ismile Hossain

Posted on Edited on

Introduction to Pandas

Welcome! This tutorial will teach you the essential Pandas skills you need to start working with data in Python.

We'll use one small car sales dataset throughout the tutorial. This makes it easier to follow because you'll keep working with the same data as you learn new things.

You don't need to know Pandas already. You only need basic Python knowledge.

By the end, you'll be able to:

  • Understand DataFrames and Series
  • Load and save data
  • Inspect and understand your data
  • Select, filter, and sort rows and columns
  • Clean messy data and handle missing values
  • Create new features
  • Summarize data with groupby and crosstab
  • Create basic visualizations
  • Apply custom transformations

We'll create the car sales dataset ourselves. It will also contain some common problems that you may find in real-world data, such as messy prices, inconsistent text, and missing values.


0. What is Pandas?

Pandas is a Python library that helps us work with table-like data.

Think about a table in Excel:

Name Age City
Alice 25 New York
Bob 30 London

Pandas lets us create, read, change, filter, and analyze tables like this using Python.

Pandas mainly gives us two important things:

  • DataFrame – the entire table
  • Series – one column from the table

A simple way to remember this:

DataFrame = whole table
Series = one column


1. Setup

First, install Pandas if you haven't already.

Run this in your terminal:

pip install pandas
Enter fullscreen mode Exit fullscreen mode

Then import Pandas in your Python script or notebook:

import pandas as pd
Enter fullscreen mode Exit fullscreen mode

Here, pd is just a shorter name for Pandas. Without as pd, we would have to write pandas every time. With as pd, we can simply write:

pd.DataFrame()
Enter fullscreen mode Exit fullscreen mode

instead of:

pandas.DataFrame()
Enter fullscreen mode Exit fullscreen mode

2. A Quick First DataFrame

Before working with our car data, let's create a very small DataFrame.

# Create a simple DataFrame
data = {
    "Name": ["Alice", "Bob", "Charlie", "Diana"],
    "Age": [25, 30, 35, 28],
    "City": ["New York", "London", "Paris", "Tokyo"]
}

df = pd.DataFrame(data)
print(df)
Enter fullscreen mode Exit fullscreen mode

Output:

      Name  Age      City
0    Alice   25  New York
1      Bob   30    London
2  Charlie   35     Paris
3    Diana   28     Tokyo
Enter fullscreen mode Exit fullscreen mode

Let's understand what we're seeing.

There are three columns:

  • Name
  • Age
  • City

There are also numbers on the left:

0
1
2
3
Enter fullscreen mode Exit fullscreen mode

These numbers are called the index. Pandas automatically creates an index for each row. The index helps Pandas identify each row. It is not one of our actual data columns.


3. Loading the Car Sales Dataset

Now let's create the dataset we'll use for the rest of the tutorial.

Our dataset contains these columns:

  • Make – the car manufacturer
  • Model – the car model
  • Year – the manufacturing year
  • Price – the price of the car
  • Odometer (KM) – how many kilometers the car has been driven
  • Doors – number of doors
  • Color – the car's color

Some of the data is intentionally messy. For example, the price contains $ and commas, and Toyota is written in different ways.

data = {
    "Make": ["Toyota", "Honda", "toyota", "Ford", "BMW", "Honda", "TOYOTA", "Ford", "BMW", "Nissan"],
    "Model": ["Corolla", "Civic", "Camry", "Focus", "3 Series", "Accord", "RAV4", "Fiesta", "5 Series", "Altima"],
    "Year": [2015, 2018, 2017, 2016, 2019, 2020, 2014, 2018, 2017, 2016],
    "Price": ["$15,000", "$18,500", "$16,200", "$10,000", "$35,000", "$22,000", "$14,500", "$12,000", "$32,500", "$17,000"],
    "Odometer (KM)": [120000, 60000, 85000, 130000, 45000, 20000, 150000, 90000, 55000, 100000],
    "Doors": [4, 4, 4, 5, 4, 4, 4, 3, 4, 4],
    "Color": ["White", "Black", "Red", "Blue", "Silver", "White", "Red", "Green", "Black", "Blue"]
}

car_sales = pd.DataFrame(data)
print(car_sales)
Enter fullscreen mode Exit fullscreen mode

Output:

      Make     Model  Year    Price  Odometer (KM)  Doors   Color
0   Toyota   Corolla  2015  $15,000        120000      4   White
1    Honda     Civic  2018  $18,500         60000      4   Black
2   toyota     Camry  2017  $16,200         85000      4     Red
3     Ford     Focus  2016  $10,000        130000      5    Blue
4      BMW  3 Series  2019  $35,000         45000      4  Silver
5    Honda    Accord  2020  $22,000         20000      4   White
6   TOYOTA      RAV4  2014  $14,500        150000      4     Red
7     Ford    Fiesta  2018  $12,000         90000      3   Green
8      BMW  5 Series  2017  $32,500         55000      4   Black
9   Nissan    Altima  2016  $17,000        100000      4    Blue
Enter fullscreen mode Exit fullscreen mode

We'll keep using this DataFrame. Later, we'll save it as a CSV file and load it again.


4. Data Structures: Series and DataFrame

Pandas has two main data structures:

  • Series – one column
  • DataFrame – the complete table

Our car_sales variable is a DataFrame. Each column inside car_sales is a Series.

For example:

# Get the 'Make' column as a Series
make_series = car_sales["Make"]

print(type(make_series))
print(make_series)
Enter fullscreen mode Exit fullscreen mode

Output:

<class 'pandas.core.series.Series'>
0     Toyota
1      Honda
2     toyota
3       Ford
4        BMW
5      Honda
6     TOYOTA
7       Ford
8        BMW
9     Nissan
Name: Make, dtype: object
Enter fullscreen mode Exit fullscreen mode

So when we do:

car_sales["Make"]
Enter fullscreen mode Exit fullscreen mode

we get one Series containing the Make column.

A Series has:

  • an index
  • values

The index is on the left, and the actual values are on the right.


5. Import & Export

In real projects, data usually doesn't come directly from a Python dictionary. It might come from a CSV file. Pandas makes it easy to save data to a file and read it back.

Save to CSV

We can save our DataFrame as a CSV file:

car_sales.to_csv("car_sales.csv", index=False)
Enter fullscreen mode Exit fullscreen mode

The index=False means:

Don't save Pandas's index as an extra column in the CSV file.

Read from CSV

Now we can read the CSV file:

car_sales_loaded = pd.read_csv("car_sales.csv")
print(car_sales_loaded.head())
Enter fullscreen mode Exit fullscreen mode

We can also replace our original DataFrame with the loaded data:

car_sales = pd.read_csv("car_sales.csv")
Enter fullscreen mode Exit fullscreen mode

Now car_sales contains the data that came from the CSV file.


6. Understanding Your Data

Before working with data, it's a good idea to inspect it.

You want to know things like:

  • What are the column names?
  • What type of data is in each column?
  • How many rows and columns are there?
  • Are there missing values?
  • What do the numbers look like?

Column names

To see the column names:

print(car_sales.columns)
Enter fullscreen mode Exit fullscreen mode

Output:

Index(['Make', 'Model', 'Year', 'Price', 'Odometer (KM)', 'Doors', 'Color'], dtype='object')
Enter fullscreen mode Exit fullscreen mode

This tells us which columns exist in our DataFrame.


Data types

We can check the type of each column:

print(car_sales.dtypes)
Enter fullscreen mode Exit fullscreen mode

Output:

Make              object
Model             object
Year              int64
Price             object
Odometer (KM)     int64
Doors             int64
Color             object
dtype: object
Enter fullscreen mode Exit fullscreen mode

Notice something interesting:

Price    object
Enter fullscreen mode Exit fullscreen mode

You might expect Price to be a number. But our prices look like:

"$15,000"
Enter fullscreen mode Exit fullscreen mode

Because they contain $ and ,, Pandas treats them as text. We'll fix this later.


Shape

shape tells us how many rows and columns our DataFrame has.

print(car_sales.shape)
Enter fullscreen mode Exit fullscreen mode

Output:

(10, 7)
Enter fullscreen mode Exit fullscreen mode

This means:

10 rows
7 columns
Enter fullscreen mode Exit fullscreen mode

The first number is the number of rows. The second number is the number of columns.


Index

We can look at the DataFrame's index:

print(car_sales.index)
Enter fullscreen mode Exit fullscreen mode

Output:

RangeIndex(start=0, stop=10, step=1)
Enter fullscreen mode Exit fullscreen mode

Our index starts at 0 and goes up to 9.


Summary statistics

We can ask Pandas for some basic statistics about our numeric columns:

print(car_sales.describe())
Enter fullscreen mode Exit fullscreen mode

Output:

               Year  Odometer (KM)     Doors
count    10.000000      10.000000  10.00000
mean   2017.000000   85500.000000   4.00000
std       1.763834   40963.059426   0.47140
min    2014.000000   20000.000000   3.00000
25%    2016.000000   55000.000000   4.00000
50%    2017.000000   87500.000000   4.00000
75%    2018.000000  115000.000000   4.00000
max    2020.000000  150000.000000   5.00000
Enter fullscreen mode Exit fullscreen mode

This gives us a quick summary of the numeric columns.

For example:

  • min = smallest value
  • max = largest value
  • mean = average
  • count = number of values

DataFrame info

Another useful method is info():

car_sales.info()
Enter fullscreen mode Exit fullscreen mode

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 7 columns):
 #   Column         Non-Null Count  Dtype
---  ------         --------------  -----
 0   Make           10 non-null     object
 1   Model          10 non-null     object
 2   Year           10 non-null     int64
 3   Price          10 non-null     object
 4   Odometer (KM)  10 non-null     int64
 5   Doors          10 non-null     int64
 6   Color          10 non-null     object
dtypes: int64(3), object(4)
memory usage: 688.0+ bytes
Enter fullscreen mode Exit fullscreen mode

This gives us a quick overview of the DataFrame.

For example, we can see that every column currently has 10 non-null values. That means there are no missing values yet.


Basic calculations

Pandas can also perform calculations for us. To calculate the average of numeric columns:

print(car_sales.mean(numeric_only=True))
Enter fullscreen mode Exit fullscreen mode

Output:

Year            2017.0
Odometer (KM)   85500.0
Doors               4.0
dtype: float64
Enter fullscreen mode Exit fullscreen mode

We can calculate the total odometer value:

print(car_sales["Odometer (KM)"].sum())
Enter fullscreen mode Exit fullscreen mode

Output:

855000
Enter fullscreen mode Exit fullscreen mode

We can also count how many rows we have:

print(len(car_sales))
Enter fullscreen mode Exit fullscreen mode

Output:

10
Enter fullscreen mode Exit fullscreen mode

7. Viewing Data

Sometimes you don't want to print the entire DataFrame. You may only want to see the first few or last few rows.

head()

car_sales.head()
Enter fullscreen mode Exit fullscreen mode

By default, head() shows the first 5 rows. You can also choose how many rows you want:

car_sales.head(10)
Enter fullscreen mode Exit fullscreen mode

tail()

tail() shows rows from the bottom:

car_sales.tail(3)
Enter fullscreen mode Exit fullscreen mode

This shows the last 3 rows. These methods are useful when you simply want to take a quick look at your data.


8. Selecting Columns

We can select one column from our DataFrame.

# Single column (Series)
car_sales["Make"]
Enter fullscreen mode Exit fullscreen mode

This gives us a Series. If we want multiple columns, we put their names inside another list:

# Multiple columns (DataFrame)
car_sales[["Make", "Price"]]
Enter fullscreen mode Exit fullscreen mode

The important difference is:

car_sales["Make"]
Enter fullscreen mode Exit fullscreen mode

selects one column.

While:

car_sales[["Make", "Price"]]
Enter fullscreen mode Exit fullscreen mode

selects multiple columns.


9. Selecting Rows with .loc and .iloc

Pandas gives us two useful ways to select rows:

  • .iloc – select using the row's integer position
  • .loc – select using the row's index label

A simple way to remember:

iloc = integer location
loc = label/location

With our current default index, they can look very similar.

Using .iloc

The first row is at position 0:

car_sales.iloc[0]
Enter fullscreen mode Exit fullscreen mode

The third row is at position 2:

car_sales.iloc[2]
Enter fullscreen mode Exit fullscreen mode

We can also select the first three rows:

car_sales.iloc[:3]
Enter fullscreen mode Exit fullscreen mode

This means:

start at 0
stop before 3
Enter fullscreen mode Exit fullscreen mode

So we get rows:

0
1
2
Enter fullscreen mode Exit fullscreen mode

Using .loc

We can select the row whose index label is 3:

car_sales.loc[3]
Enter fullscreen mode Exit fullscreen mode

We can also select a range:

car_sales.loc[0:2]
Enter fullscreen mode Exit fullscreen mode

With .loc, the ending value is included. So:

car_sales.loc[0:2]
Enter fullscreen mode Exit fullscreen mode

gives us rows:

0
1
2
Enter fullscreen mode Exit fullscreen mode

Selecting one specific cell

We can select a single value by giving the row and column positions:

car_sales.iloc[1, 0]
Enter fullscreen mode Exit fullscreen mode

This means:

row 1
column 0
Enter fullscreen mode Exit fullscreen mode

The result is:

Honda
Enter fullscreen mode Exit fullscreen mode

We can reach the same cell using the label-based approach:

car_sales.loc[1, "Make"]
Enter fullscreen mode Exit fullscreen mode

Output:

Honda
Enter fullscreen mode Exit fullscreen mode

.loc[1, "Make"] addresses the row by its index label and the column by its name, which is often easier to read than iloc's positional [1, 0].


10. Filtering Rows (Boolean Indexing)

Filtering means:

Show me only the rows that match a condition.

For example, we can ask:

Which cars are made by Toyota?

toyota_cars = car_sales[car_sales["Make"] == "Toyota"]
print(toyota_cars)
Enter fullscreen mode Exit fullscreen mode

Only rows where Make is exactly "Toyota" will be returned. Notice that "toyota" and "TOYOTA" are not returned. That's because Python treats these as different strings. We'll fix this later when we clean our data.

Multiple conditions

We can combine conditions.

For example:

Find cars with more than 100,000 KM and exactly 4 doors.

filtered = car_sales[(car_sales["Odometer (KM)"] > 100000) & (car_sales["Doors"] == 4)]
print(filtered)
Enter fullscreen mode Exit fullscreen mode

Here:

&
Enter fullscreen mode Exit fullscreen mode

means AND.

Both conditions must be true.

isin() for multiple values

Suppose we want cars made by either Toyota or Honda.

We can use isin():

selected = car_sales[car_sales["Make"].isin(["Toyota", "Honda"])]
print(selected)
Enter fullscreen mode Exit fullscreen mode

This checks whether each value exists in the list:

["Toyota", "Honda"]
Enter fullscreen mode Exit fullscreen mode

Remember these operators:

  • & = AND
  • | = OR
  • isin() = value is one of several choices

When using multiple conditions, put each condition inside parentheses.


11. Sorting Data

We can sort our DataFrame by a column.

For example, let's sort cars by year:

# Sort by Year (ascending)
df_sorted = car_sales.sort_values("Year")
print(df_sorted)
Enter fullscreen mode Exit fullscreen mode

By default, sorting is ascending.

That means:

2014
2015
2016
2017
...
Enter fullscreen mode Exit fullscreen mode

If we want the newest cars first, we can sort in descending order:

df_sorted = car_sales.sort_values("Year", ascending=False)
Enter fullscreen mode Exit fullscreen mode

We can also sort by more than one column:

car_sales.sort_values(["Make", "Year"])
Enter fullscreen mode Exit fullscreen mode

This first sorts by Make, and then sorts by Year within each make.


12. Crosstab – Frequency Tables

pd.crosstab() helps us count how often values appear together.

For example:

pd.crosstab(car_sales["Make"], car_sales["Doors"])
Enter fullscreen mode Exit fullscreen mode

Output:

Doors    3  4  5

Make
BMW      0  2  0
Ford     1  1  1
Honda    0  2  0
Nissan   0  1  0
TOYOTA   0  1  0
Toyota   0  1  0
toyota   0  1  0
Enter fullscreen mode Exit fullscreen mode

We can read this as:

How many cars of each make have 3, 4, or 5 doors?

For example:

BMW    0  2  0
Enter fullscreen mode Exit fullscreen mode

means BMW has:

  • 0 cars with 3 doors
  • 2 cars with 4 doors
  • 0 cars with 5 doors

Notice that Toyota appears three times:

TOYOTA
Toyota
toyota
Enter fullscreen mode Exit fullscreen mode

Pandas treats them as different values because their capitalization is different. We'll clean this later.


13. GroupBy – Aggregating by Group

groupby() is useful when we want to divide our data into groups and calculate something for each group.

For example, we can group cars by their manufacturer:

car_sales.groupby("Make").mean(numeric_only=True)
Enter fullscreen mode Exit fullscreen mode

This asks:

For each car manufacturer, what are the average numeric values?

Output:

          Year  Odometer (KM)     Doors
Make
BMW     2018.0       50000.0  4.000000
Ford    2017.0      110000.0  4.000000
Honda   2019.0       40000.0  4.000000
Nissan  2016.0      100000.0  4.000000
TOYOTA  2014.0      150000.0  4.000000
Toyota  2015.0      120000.0  4.000000
toyota  2017.0       85000.0  4.000000
Enter fullscreen mode Exit fullscreen mode

The important idea is:

groupby("Make")
Enter fullscreen mode Exit fullscreen mode

puts rows with the same Make together.

Then:

.mean()
Enter fullscreen mode Exit fullscreen mode

calculates the average for each group.

Again, because Toyota has different capitalization, Pandas creates three separate groups. We'll fix this in the cleaning section.


14. Basic Data Visualization

Pandas can also create simple charts. Pandas uses Matplotlib for its built-in plotting features.

Line plot

We can create a basic line plot of the odometer values:

car_sales["Odometer (KM)"].plot()
Enter fullscreen mode Exit fullscreen mode

This gives us a visual representation of the values.

Histogram

We can also create a histogram:

car_sales["Odometer (KM)"].hist()
Enter fullscreen mode Exit fullscreen mode

A histogram helps us see how the odometer values are distributed. For example, it can help us see whether most cars have lower or higher mileage.

In a plain Python script (as opposed to a Jupyter notebook), Pandas charts won't appear unless you import Matplotlib and call show():

import matplotlib.pyplot as plt

car_sales["Odometer (KM)"].plot()
plt.show()

In notebooks, the chart usually displays automatically after the cell runs.


15. Cleaning Data

Now let's start fixing the messy parts of our dataset. Real-world data is often not perfectly clean. Our dataset has a few problems:

  • Prices contain $ and ,
  • The same manufacturer is written with different capitalization
  • We'll introduce some missing values

Let's fix these one at a time.


15.1 Clean the Price Column

Our prices currently look like this:

$15,000
$18,500
$16,200
Enter fullscreen mode Exit fullscreen mode

These are strings, not numbers. We want them to become:

15000
18500
16200
Enter fullscreen mode Exit fullscreen mode

First, remove the $:

.str.replace("$", "", regex=False)
Enter fullscreen mode Exit fullscreen mode

Then remove the commas:

.str.replace(",", "", regex=False)
Enter fullscreen mode Exit fullscreen mode

Finally, convert the result to a float:

.astype(float)
Enter fullscreen mode Exit fullscreen mode

We can do all of that together:

car_sales["Price"] = (
    car_sales["Price"]
    .astype(str) # Convert to string type first to ensure .str accessor works
    .str.replace("$", "", regex=False)   # remove dollar sign
    .str.replace(",", "", regex=False)   # remove commas
    .astype(float)                       # convert to float
)

print(car_sales["Price"].head())
Enter fullscreen mode Exit fullscreen mode

Output:

0    15000.0
1    18500.0
2    16200.0
3    10000.0
4    35000.0
Name: Price, dtype: float64
Enter fullscreen mode Exit fullscreen mode

Now Price is a numeric column. That means we can perform calculations on it.


15.2 Standardize the Make Column

Earlier, we saw these values:

Toyota
toyota
TOYOTA
Enter fullscreen mode Exit fullscreen mode

They all mean the same thing, but Pandas treats them as different values. We can convert all of them to lowercase:

car_sales["Make"] = car_sales["Make"].str.lower()
print(car_sales["Make"].unique())
Enter fullscreen mode Exit fullscreen mode

Output:

['toyota' 'honda' 'ford' 'bmw' 'nissan']
Enter fullscreen mode Exit fullscreen mode

Now all Toyota values are:

toyota
Enter fullscreen mode Exit fullscreen mode

Our data is more consistent.

Let's see the payoff. Compare this groupby with the one in Section 13 — the three Toyota spellings have collapsed into one group:

car_sales.groupby("Make").mean(numeric_only=True)
Enter fullscreen mode Exit fullscreen mode

Output:

          Year  Odometer (KM)  Doors
Make
bmw     2018.0        50000.0    4.0
ford    2017.0       110000.0    4.0
honda   2019.0        40000.0    4.0
nissan  2016.0       100000.0    4.0
toyota  2015.5       118333.3    4.0
Enter fullscreen mode Exit fullscreen mode

The toyota row now averages the three Toyotas together, which is what we actually want.


16. Handling Missing Data

Real-world datasets often contain missing values. For example, a car might not have its mileage recorded. Let's create some missing values ourselves so we can learn how to handle them. We'll use None. Pandas will treat these as missing values, usually shown as NaN.

car_sales.loc[2, "Odometer (KM)"] = None   # Toyota Camry missing odometer
car_sales.loc[5, "Doors"] = None           # Honda Accord missing doors
car_sales.loc[7, "Price"] = None           # Ford Fiesta missing price
Enter fullscreen mode Exit fullscreen mode

Now let's count the missing values:

car_sales.isna().sum()
Enter fullscreen mode Exit fullscreen mode

Output:

Make             0
Model            0
Year             0
Price            1
Odometer (KM)    1
Doors            1
Color            0
dtype: int64
Enter fullscreen mode Exit fullscreen mode

This tells us:

  • Price has 1 missing value
  • Odometer (KM) has 1 missing value
  • Doors has 1 missing value
  • The other columns have no missing values

Fill vs. drop

Before filling anything, it helps to see what dropna() would do on the real data. It would remove every row that contains any missing value:

# Preview only — this would remove the 3 rows with missing values
# car_sales.dropna()
Enter fullscreen mode Exit fullscreen mode

In our case, that would throw away 3 of our 10 rows, which is a lot. So instead of dropping, we'll fill the missing values with reasonable replacements.

Fill Missing Values

One way to handle missing data is to replace the missing value with something else. We can use a calculated value such as the mean or median.


Fill Odometer with the mean

First, calculate the average odometer value:

mean_odometer = car_sales["Odometer (KM)"].mean()
Enter fullscreen mode Exit fullscreen mode

Then fill the missing value with that average:

car_sales["Odometer (KM)"] = car_sales["Odometer (KM)"].fillna(mean_odometer)
Enter fullscreen mode Exit fullscreen mode

So the missing odometer value is replaced with the average odometer value.


Fill Doors with the median

First, find the median:

median_doors = car_sales["Doors"].median()
Enter fullscreen mode Exit fullscreen mode

Then use it to fill the missing value:

car_sales["Doors"] = car_sales["Doors"].fillna(median_doors)
Enter fullscreen mode Exit fullscreen mode

Fill Price with a constant

We can also choose a specific value. For example, we can replace the missing price with 0:

car_sales["Price"] = car_sales["Price"].fillna(0)
Enter fullscreen mode Exit fullscreen mode

Now check the missing values again:

car_sales.isna().sum()
Enter fullscreen mode Exit fullscreen mode

All values should now be zero.


Drop Missing Values

Another option is to remove rows that contain missing values.

We can use:

car_sales.dropna(inplace=True)
Enter fullscreen mode Exit fullscreen mode

This removes rows containing any NaN. In our example, we've already filled the missing values, so we don't need to use dropna(). In real projects, whether you should fill or remove missing data depends on the situation.


17. Creating New Columns (Feature Engineering)

We can create new columns using existing data. This is sometimes called feature engineering. Let's look at several ways to create a new column.

From a Series

We can create a Series containing the number of seats:

seats = pd.Series([5, 5, 5, 5, 5, 5, 5, 5, 5, 5])
car_sales["Seats"] = seats
Enter fullscreen mode Exit fullscreen mode

Now our DataFrame has a new Seats column. If the Series doesn't have enough values for every row, the missing rows will get NaN.


From a Python list

We can also create a column directly from a Python list:

fuel_economy = [7.5, 9.2, 5.0, 9.6, 8.7, 4.7, 7.6, 8.7, 3.0, 4.5]
car_sales["Fuel per 100KM"] = fuel_economy
Enter fullscreen mode Exit fullscreen mode

Each value in the list is placed into the corresponding row.


From a single value

We can assign the same value to every row:

car_sales["Number of wheels"] = 4
Enter fullscreen mode Exit fullscreen mode

Now every car has:

Number of wheels = 4
Enter fullscreen mode Exit fullscreen mode

From a calculation

We can create a column using values from other columns.

For example:

car_sales["Total fuel used (L)"] = car_sales["Odometer (KM)"] / 100 * car_sales["Fuel per 100KM"]
Enter fullscreen mode Exit fullscreen mode

This calculates the estimated total fuel used. The calculation is performed for every row.


Boolean column

We can also create a column containing True or False:

car_sales["Passed road safety"] = True
Enter fullscreen mode Exit fullscreen mode

Now every row has:

Passed road safety = True
Enter fullscreen mode Exit fullscreen mode

Our DataFrame now contains several new columns.


18. Renaming Columns

Sometimes a column name isn't convenient to use.

We can rename columns with rename().

For example:

car_sales = car_sales.rename(columns={
    "Odometer (KM)": "Odometer_KM",
    "Fuel per 100KM": "Fuel_per_100KM"
})
Enter fullscreen mode Exit fullscreen mode

Now:

Odometer (KM)
Enter fullscreen mode Exit fullscreen mode

becomes:

Odometer_KM
Enter fullscreen mode Exit fullscreen mode

And:

Fuel per 100KM
Enter fullscreen mode Exit fullscreen mode

becomes:

Fuel_per_100KM
Enter fullscreen mode Exit fullscreen mode

19. Removing Columns

We can remove columns using drop().

For example:

car_sales = car_sales.drop("Total fuel used (L)", axis=1)
Enter fullscreen mode Exit fullscreen mode

Here:

axis=1
Enter fullscreen mode Exit fullscreen mode

means we're working with columns. We can remove multiple columns at once:

car_sales = car_sales.drop(["Seats", "Number of wheels"], axis=1)
Enter fullscreen mode Exit fullscreen mode

Now those columns are removed from the DataFrame.


20. Shuffling and Sampling

Sometimes we want to randomly rearrange or select rows.

Shuffle the entire DataFrame

We can randomly change the order of all rows:

car_sales_shuffled = car_sales.sample(frac=1)
print(car_sales_shuffled)
Enter fullscreen mode Exit fullscreen mode

Here:

frac=1
Enter fullscreen mode Exit fullscreen mode

means:

Take 100% of the rows.

But because sample() is random, their order changes.


Take a random subset

We can also take only part of the DataFrame.

For example:

car_sales_sample = car_sales.sample(frac=0.3)
print(car_sales_sample)
Enter fullscreen mode Exit fullscreen mode

frac=0.3 means:

Take 30% of the rows.

We can also tell Pandas exactly how many rows we want:

car_sales_sample = car_sales.sample(n=5)
Enter fullscreen mode Exit fullscreen mode

This gives us 5 random rows.


21. Resetting the Index

When we shuffle or select rows, the original index values stay attached to those rows.

For example, after shuffling, the index might look like:

7
2
9
1
5
...
Enter fullscreen mode Exit fullscreen mode

If we want a clean index again:

0
1
2
3
4
...
Enter fullscreen mode Exit fullscreen mode

we can use reset_index():

car_sales_shuffled.reset_index(drop=True, inplace=True)
print(car_sales_shuffled)
Enter fullscreen mode Exit fullscreen mode

The important part is:

drop=True
Enter fullscreen mode Exit fullscreen mode

This tells Pandas not to keep the old index as a new column. Now the index is:

0, 1, 2, ...
Enter fullscreen mode Exit fullscreen mode

in the new order.


22. Applying a Function to a Column

Sometimes we want to perform the same calculation on every value in a column. We can use .apply().

For example, our odometer values are in kilometers. Let's convert them to miles. We can use:

1 mile ≈ 1.6 km
Enter fullscreen mode Exit fullscreen mode

So we divide kilometers by 1.6.

# 1 mile ≈ 1.6 km
car_sales["Odometer (miles)"] = car_sales["Odometer_KM"].apply(lambda x: x / 1.6)

print(car_sales[["Odometer_KM", "Odometer (miles)"]].head())
Enter fullscreen mode Exit fullscreen mode

Here:

.apply()
Enter fullscreen mode Exit fullscreen mode

runs our function for every value in the column.

This part:

lambda x: x / 1.6
Enter fullscreen mode Exit fullscreen mode

means:

Take each value and divide it by 1.6.

We can also create a normal named function and pass that function to .apply().


23. Putting It All Together: A Full Workflow

Now let's combine many of the things we've learned into one workflow. This workflow reuses the DataFrame from Section 3. If you're running this as a fresh script, run Section 3 first to define car_sales.

import pandas as pd

# Reuse the DataFrame from Section 3 (defined as `car_sales`).
# Work on a copy so the rest of the tutorial's data isn't changed.
df = car_sales.copy()

# 1. Clean Price
df["Price"] = (
    df["Price"]
    .astype(str)
    .str.replace("$", "", regex=False)
    .str.replace(",", "", regex=False)
    .astype(float)
)

# 2. Clean Make
df["Make"] = df["Make"].str.lower()

# 3. Introduce and handle missing values
df.loc[2, "Odometer (KM)"] = None
df["Odometer (KM)"] = df["Odometer (KM)"].fillna(df["Odometer (KM)"].mean())

# 4. Create a new feature
df["Price per 1000 km"] = df["Price"] / (df["Odometer (KM)"] / 1000)

# 5. Filter
filtered = df[df["Price"] < 20000]

# 6. Group summary
summary = filtered.groupby("Make")["Price"].mean()
print(summary)

# 7. Save
df.to_csv("cleaned_car_sales.csv", index=False)
Enter fullscreen mode Exit fullscreen mode

Let's follow what this code does:

Step 1 — Start from a copy

df = car_sales.copy()
Enter fullscreen mode Exit fullscreen mode

We work on a copy of the DataFrame from Section 3 so the workflow is runnable end-to-end without redefining the data.

Step 2 — Clean the price

df["Price"] = ...
Enter fullscreen mode Exit fullscreen mode

We remove $ and commas and convert the values to numbers.

Step 3 — Clean the make

df["Make"] = df["Make"].str.lower()
Enter fullscreen mode Exit fullscreen mode

This makes the manufacturer names consistent.

Step 4 — Handle missing data

We create a missing odometer value and then replace it with the average.

Step 5 — Create a new feature

df["Price per 1000 km"] = ...
Enter fullscreen mode Exit fullscreen mode

We calculate the price per 1,000 kilometers.

Step 6 — Filter the data

filtered = df[df["Price"] < 20000]
Enter fullscreen mode Exit fullscreen mode

Now we only keep cars costing less than $20,000.

Step 7 — Group the data

summary = filtered.groupby("Make")["Price"].mean()
Enter fullscreen mode Exit fullscreen mode

We group the filtered cars by manufacturer and calculate the average price for each manufacturer.

Step 8 — Save the result

df.to_csv("cleaned_car_sales.csv", index=False)
Enter fullscreen mode Exit fullscreen mode

Finally, we save our cleaned DataFrame to a CSV file.

This is a simple example of how several Pandas operations can work together in a real data workflow.


Summary of What You Learned

You learned the following Pandas concepts:

  • Data Structures – Series and DataFrame
  • Import/Export – pd.read_csv(), df.to_csv()
  • Understanding Data – dtypes, columns, shape, index, describe(), info(), mean(), sum(), len()
  • Viewing Data – head(), tail()
  • Selecting Data – column selection, .loc, .iloc
  • Filtering – Boolean indexing, &, |, isin()
  • Sorting – sort_values()
  • Crosstab – pd.crosstab()
  • GroupBy – groupby().mean() etc.
  • Visualization – plot(), hist()
  • Data Cleaning – string replacement, .str.lower()
  • Missing Data – isna(), fillna(), dropna()
  • Feature Engineering – new columns from Series, lists, single values, calculations, booleans
  • Renaming Columns – rename()
  • Removing Columns – drop(axis=1)
  • Shuffling & Sampling – sample(frac=1), sample(frac=0.2)
  • Resetting Index – reset_index(drop=True)
  • Applying Functions – .apply(lambda ...)

About This Article

This article is based on my personal learning notes from my portfolio site, where I document concepts, examples, and lessons as I learn.

You can find the original notes here: Introduction to Pandas.

A living collection of what I learn through building, curiosity, and exploration.


Top comments (0)