DEV Community

Pandas Tutorial With Examples: A Hands-On Guide for Developers

Pandas is one of the most useful Python libraries for working with structured data. Whether you are cleaning a CSV export, analyzing application logs, transforming API responses, or preparing data for a machine-learning model, Pandas helps you move from raw data to useful answers quickly.

This tutorial is designed for developers who prefer building over theory. You will create DataFrames, load files, clean messy values, filter records, aggregate data, join datasets, handle errors, and make your code faster.

Note: The keyword best data science training in bangalore appears naturally in this article for search relevance, but this tutorial is focused on practical, self-directed development skills—not promotion.


What You’ll Build

We will work with a small e-commerce dataset and answer questions such as:

  • Which products generated the most revenue?
  • Which customers placed the most orders?
  • How do we clean missing or malformed data?
  • How do we combine orders with customer information?
  • How can we process large CSV files without exhausting memory?

By the end, you will have a reusable Pandas workflow that applies to product analytics, backend reporting, finance exports, operational dashboards, and data-science projects.


Prerequisites

You need:

  • Python 3.9 or newer
  • Basic Python knowledge: variables, lists, dictionaries, functions
  • A code editor such as VS Code or PyCharm
  • A terminal

Install Pandas:

python -m pip install pandas
Enter fullscreen mode Exit fullscreen mode

For notebook-based exploration, install Jupyter too:

python -m pip install jupyter pandas
Enter fullscreen mode Exit fullscreen mode

Start a notebook:

jupyter notebook
Enter fullscreen mode Exit fullscreen mode

Or create a normal Python file named pandas_tutorial.py.

Verify the installation:

import pandas as pd

print(pd.__version__)
Enter fullscreen mode Exit fullscreen mode

You should see a version number such as 2.x.x.


Project Setup

Create a folder for the project:

mkdir pandas-ecommerce-tutorial
cd pandas-ecommerce-tutorial
Enter fullscreen mode Exit fullscreen mode

A simple project structure might look like this:

pandas-ecommerce-tutorial/
├── data/
│   ├── orders.csv
│   └── customers.csv
├── notebooks/
│   └── analysis.ipynb
├── src/
│   └── analysis.py
├── requirements.txt
└── README.md
Enter fullscreen mode Exit fullscreen mode

Add dependencies to requirements.txt:

pandas
jupyter
Enter fullscreen mode Exit fullscreen mode

Install them:

python -m pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

This structure works nicely in a GitHub repository. Keeping raw data, notebooks, and reusable scripts separate makes the project easier to maintain and review.


Understanding Pandas Basics

Pandas has two primary data structures:

  • Series: A one-dimensional labeled column of data.
  • DataFrame: A two-dimensional table made of rows and columns.

Think of a DataFrame as a spreadsheet, SQL result table, or JSON collection converted into tabular form.

Create a Series

import pandas as pd

prices = pd.Series([499.0, 799.0, 1299.0], name="price")

print(prices)
Enter fullscreen mode Exit fullscreen mode

Output:

0     499.0
1     799.0
2    1299.0
Name: price, dtype: float64
Enter fullscreen mode Exit fullscreen mode

The numbers on the left are the index. Pandas adds an index automatically unless you provide one.

inventory = pd.Series(
    [10, 5, 18],
    index=["Keyboard", "Mouse", "Monitor"],
    name="stock"
)

print(inventory)
Enter fullscreen mode Exit fullscreen mode

Output:

Keyboard    10
Mouse        5
Monitor     18
Name: stock, dtype: int64
Enter fullscreen mode Exit fullscreen mode

Now you can access a value by label:

print(inventory["Mouse"])
Enter fullscreen mode Exit fullscreen mode

Output:

5
Enter fullscreen mode Exit fullscreen mode

Create a DataFrame

Here is a basic order dataset:

import pandas as pd

orders = pd.DataFrame(
    {
        "order_id": [1001, 1002, 1003, 1004, 1005],
        "customer_id": [501, 502, 501, 503, 504],
        "product": ["Keyboard", "Mouse", "Monitor", "Keyboard", "Mouse"],
        "category": ["Accessories", "Accessories", "Displays", "Accessories", "Accessories"],
        "quantity": [1, 2, 1, 1, 3],
        "unit_price": [1200, 500, 15000, 1200, 500],
        "order_date": ["2026-01-05", "2026-01-06", "2026-01-07", "2026-01-08", "2026-01-09"],
    }
)

print(orders)
Enter fullscreen mode Exit fullscreen mode

Every dictionary key becomes a column. Each list represents the values in that column.

Inspect the first few rows:

print(orders.head())
Enter fullscreen mode Exit fullscreen mode

Inspect the final rows:

print(orders.tail(2))
Enter fullscreen mode Exit fullscreen mode

Check dimensions:

print(orders.shape)
Enter fullscreen mode Exit fullscreen mode

Output:

(5, 7)
Enter fullscreen mode Exit fullscreen mode

This means five rows and seven columns.

Check column names:

print(orders.columns)
Enter fullscreen mode Exit fullscreen mode

Check data types and missing values:

orders.info()
Enter fullscreen mode Exit fullscreen mode

This is one of the first commands you should run after loading any unfamiliar dataset.


Loading CSV, JSON, and Excel Data

Most practical Pandas work begins with reading files.

Load a CSV file

Create data/orders.csv:

order_id,customer_id,product,category,quantity,unit_price,order_date
1001,501,Keyboard,Accessories,1,1200,2026-01-05
1002,502,Mouse,Accessories,2,500,2026-01-06
1003,501,Monitor,Displays,1,15000,2026-01-07
1004,503,Keyboard,Accessories,1,1200,2026-01-08
1005,504,Mouse,Accessories,3,500,2026-01-09
Enter fullscreen mode Exit fullscreen mode

Read it:

import pandas as pd

orders = pd.read_csv("data/orders.csv")

print(orders.head())
Enter fullscreen mode Exit fullscreen mode

If your date column should be treated as a date immediately, use parse_dates:

orders = pd.read_csv(
    "data/orders.csv",
    parse_dates=["order_date"]
)

print(orders.dtypes)
Enter fullscreen mode Exit fullscreen mode

This prevents a common problem: dates loaded as plain strings.

Load JSON from an API response

Suppose an API returns this JSON-like Python object:

api_response = [
    {"id": 1, "name": "Asha", "city": "Bengaluru"},
    {"id": 2, "name": "Rahul", "city": "Mumbai"},
    {"id": 3, "name": "Meera", "city": "Chennai"},
]

customers = pd.DataFrame(api_response)

print(customers)
Enter fullscreen mode Exit fullscreen mode

For a JSON file:

customers = pd.read_json("data/customers.json")
Enter fullscreen mode Exit fullscreen mode

Nested JSON needs flattening. Use pd.json_normalize():

nested_data = [
    {
        "order_id": 1001,
        "customer": {
            "name": "Asha",
            "city": "Bengaluru"
        },
        "total": 1200
    }
]

df = pd.json_normalize(nested_data)

print(df)
Enter fullscreen mode Exit fullscreen mode

Expected columns:

order_id  total customer.name customer.city
Enter fullscreen mode Exit fullscreen mode

Load Excel files

Install the Excel engine:

python -m pip install openpyxl
Enter fullscreen mode Exit fullscreen mode

Then read a sheet:

sales = pd.read_excel(
    "data/monthly_sales.xlsx",
    sheet_name="January"
)
Enter fullscreen mode Exit fullscreen mode

For large Excel files, convert them to CSV when possible. CSV files are usually faster to read and easier to process in pipelines.


Explore Data Before Changing It

Before filtering, cleaning, or aggregating a dataset, inspect its structure.

print(orders.head())
print(orders.sample(3, random_state=42))
print(orders.info())
print(orders.describe())
Enter fullscreen mode Exit fullscreen mode

describe() summarizes numeric fields:

print(orders.describe())
Enter fullscreen mode Exit fullscreen mode

Typical output includes:

  • Count
  • Mean
  • Standard deviation
  • Minimum and maximum
  • Quartiles

For categorical fields, use:

print(orders["category"].value_counts())
Enter fullscreen mode Exit fullscreen mode

To include missing values in the count:

print(orders["category"].value_counts(dropna=False))
Enter fullscreen mode Exit fullscreen mode

Check unique values:

print(orders["product"].unique())
Enter fullscreen mode Exit fullscreen mode

Count them:

print(orders["product"].nunique())
Enter fullscreen mode Exit fullscreen mode

These quick checks catch issues early: unexpected product names, inconsistent spelling, duplicate categories, or fields that contain empty strings instead of proper missing values.


Select, Filter, and Sort Rows

Most analysis involves selecting the subset of data that matters.

Select columns

Select one column:

products = orders["product"]

print(products)
Enter fullscreen mode Exit fullscreen mode

Select multiple columns:

order_summary = orders[["order_id", "product", "quantity", "unit_price"]]

print(order_summary)
Enter fullscreen mode Exit fullscreen mode

Filter rows

Find all keyboard orders:

keyboard_orders = orders[orders["product"] == "Keyboard"]

print(keyboard_orders)
Enter fullscreen mode Exit fullscreen mode

Find orders with quantities greater than one:

bulk_orders = orders[orders["quantity"] > 1]

print(bulk_orders)
Enter fullscreen mode Exit fullscreen mode

Use multiple conditions with & and |.

high_value_accessories = orders[
    (orders["category"] == "Accessories") &
    (orders["unit_price"] >= 1000)
]

print(high_value_accessories)
Enter fullscreen mode Exit fullscreen mode

Always wrap each condition in parentheses. This avoids confusing operator-precedence errors.

Filter with several allowed values:

selected_products = orders[
    orders["product"].isin(["Keyboard", "Monitor"])
]

print(selected_products)
Enter fullscreen mode Exit fullscreen mode

Filter missing values:

missing_price = orders[orders["unit_price"].isna()]
Enter fullscreen mode Exit fullscreen mode

Filter non-missing values:

valid_price = orders[orders["unit_price"].notna()]
Enter fullscreen mode Exit fullscreen mode

Use query() for readable filters

For complex filters, query() can be easier to read:

result = orders.query(
    "category == 'Accessories' and quantity >= 2"
)

print(result)
Enter fullscreen mode Exit fullscreen mode

This is especially useful in exploratory notebooks. For production code, normal boolean indexing is often easier to refactor and validate with static tools.

Sort data

Sort by unit price:

sorted_orders = orders.sort_values("unit_price", ascending=False)

print(sorted_orders)
Enter fullscreen mode Exit fullscreen mode

Sort by multiple columns:

sorted_orders = orders.sort_values(
    by=["category", "unit_price"],
    ascending=[True, False]
)
Enter fullscreen mode Exit fullscreen mode

Create Calculated Columns

Raw datasets often do not contain the metric you need. Add it.

Calculate revenue

Revenue is quantity multiplied by unit price:

orders["revenue"] = orders["quantity"] * orders["unit_price"]

print(orders[["product", "quantity", "unit_price", "revenue"]])
Enter fullscreen mode Exit fullscreen mode

Expected result:

    product  quantity  unit_price  revenue
0  Keyboard         1        1200     1200
1     Mouse         2         500     1000
2   Monitor         1       15000    15000
3  Keyboard         1        1200     1200
4     Mouse         3         500     1500
Enter fullscreen mode Exit fullscreen mode

Use vectorized operations

Pandas works best when you apply operations to entire columns.

orders["discounted_price"] = orders["unit_price"] * 0.90
Enter fullscreen mode Exit fullscreen mode

Avoid manual loops:

# Avoid this for normal column calculations
for index, row in orders.iterrows():
    orders.loc[index, "revenue"] = row["quantity"] * row["unit_price"]
Enter fullscreen mode Exit fullscreen mode

The loop works, but it is slow and verbose. The vectorized version is faster and clearer:

orders["revenue"] = orders["quantity"] * orders["unit_price"]
Enter fullscreen mode Exit fullscreen mode

Create conditional columns

Use numpy.select() or np.where() for simple conditions.

import numpy as np

orders["order_size"] = np.where(
    orders["revenue"] >= 5000,
    "High value",
    "Standard"
)

print(orders[["order_id", "revenue", "order_size"]])
Enter fullscreen mode Exit fullscreen mode

For multiple conditions:

conditions = [
    orders["revenue"] >= 10000,
    orders["revenue"] >= 2000,
]

choices = [
    "Enterprise",
    "Medium",
]

orders["customer_segment"] = np.select(
    conditions,
    choices,
    default="Small"
)
Enter fullscreen mode Exit fullscreen mode

Clean Real-World Data

Production data is rarely clean. A CSV might have blank values, extra whitespace, duplicate rows, inconsistent capitalization, or invalid numbers.

Here is a deliberately messy dataset:

messy_orders = pd.DataFrame(
    {
        "order_id": [1001, 1002, 1002, 1003, 1004],
        "product": [" Keyboard ", "mouse", "mouse", None, "MONITOR"],
        "quantity": ["1", "2", "2", "one", None],
        "unit_price": [1200, 500, 500, 15000, None],
        "city": [" Bengaluru", "Mumbai ", "Mumbai ", "Chennai", "Bengaluru"],
    }
)

print(messy_orders)
Enter fullscreen mode Exit fullscreen mode

Find missing values

print(messy_orders.isna().sum())
Enter fullscreen mode Exit fullscreen mode

This gives the count of missing values in every column.

Standardize text fields

Remove whitespace and normalize case:

messy_orders["product"] = (
    messy_orders["product"]
    .str.strip()
    .str.title()
)

messy_orders["city"] = (
    messy_orders["city"]
    .str.strip()
    .str.title()
)
Enter fullscreen mode Exit fullscreen mode

Now " Keyboard " becomes "Keyboard" and "MONITOR" becomes "Monitor".

Convert invalid numeric values safely

messy_orders["quantity"] = pd.to_numeric(
    messy_orders["quantity"],
    errors="coerce"
)
Enter fullscreen mode Exit fullscreen mode

The string "one" cannot become a number, so Pandas converts it to NaN rather than crashing.

Check the result:

print(messy_orders[["quantity"]])
Enter fullscreen mode Exit fullscreen mode

Handle missing values

You have several options.

Remove rows where important data is missing:

clean_orders = messy_orders.dropna(
    subset=["product", "quantity", "unit_price"]
)
Enter fullscreen mode Exit fullscreen mode

Fill missing numeric values:

messy_orders["unit_price"] = messy_orders["unit_price"].fillna(0)
Enter fullscreen mode Exit fullscreen mode

Fill missing text:

messy_orders["city"] = messy_orders["city"].fillna("Unknown")
Enter fullscreen mode Exit fullscreen mode

Use a meaningful business rule. Filling missing prices with zero may be acceptable for a temporary report, but it can distort revenue calculations. In many systems, invalid pricing data should be flagged and excluded instead.

Remove duplicates

deduplicated_orders = messy_orders.drop_duplicates()
Enter fullscreen mode Exit fullscreen mode

Or remove duplicates based on business keys:

deduplicated_orders = messy_orders.drop_duplicates(
    subset=["order_id"]
)
Enter fullscreen mode Exit fullscreen mode

Be careful with this step. Repeated product rows can be valid if an order contains multiple line items. Deduplicate only after understanding what a row represents.


Group, Aggregate, and Analyze

Aggregation is where Pandas starts to feel like SQL.

First, ensure the revenue column exists:

orders["revenue"] = orders["quantity"] * orders["unit_price"]
Enter fullscreen mode Exit fullscreen mode

Revenue by product

revenue_by_product = (
    orders.groupby("product", as_index=False)["revenue"]
    .sum()
    .sort_values("revenue", ascending=False)
)

print(revenue_by_product)
Enter fullscreen mode Exit fullscreen mode

This is similar to:

SELECT product, SUM(revenue)
FROM orders
GROUP BY product
ORDER BY revenue DESC;
Enter fullscreen mode Exit fullscreen mode

Multiple aggregations

product_metrics = (
    orders.groupby("product", as_index=False)
    .agg(
        total_orders=("order_id", "count"),
        total_quantity=("quantity", "sum"),
        total_revenue=("revenue", "sum"),
        average_order_value=("revenue", "mean"),
    )
    .sort_values("total_revenue", ascending=False)
)

print(product_metrics)
Enter fullscreen mode Exit fullscreen mode

Named aggregations make output columns explicit and readable.

Revenue by category

category_metrics = (
    orders.groupby("category", as_index=False)
    .agg(
        revenue=("revenue", "sum"),
        units_sold=("quantity", "sum"),
    )
    .sort_values("revenue", ascending=False)
)

print(category_metrics)
Enter fullscreen mode Exit fullscreen mode

Customer spending

customer_spending = (
    orders.groupby("customer_id", as_index=False)
    .agg(
        order_count=("order_id", "count"),
        total_spend=("revenue", "sum"),
    )
    .sort_values("total_spend", ascending=False)
)

print(customer_spending)
Enter fullscreen mode Exit fullscreen mode

Group data by month

Convert the order date first:

orders["order_date"] = pd.to_datetime(orders["order_date"])
Enter fullscreen mode Exit fullscreen mode

Create a monthly period:

orders["order_month"] = orders["order_date"].dt.to_period("M")
Enter fullscreen mode Exit fullscreen mode

Aggregate:

monthly_revenue = (
    orders.groupby("order_month", as_index=False)["revenue"]
    .sum()
)

print(monthly_revenue)
Enter fullscreen mode Exit fullscreen mode

For time-series reporting, resample() is another powerful option:

daily_revenue = (
    orders.set_index("order_date")
    .resample("D")["revenue"]
    .sum()
)

print(daily_revenue)
Enter fullscreen mode Exit fullscreen mode

Join Multiple DataFrames

In real applications, data is rarely stored in one table. You might have:

  • Orders from a transaction system
  • Customers from a CRM
  • Product metadata from a catalog
  • Support tickets from a help desk

Create customer data:

customers = pd.DataFrame(
    {
        "customer_id": [501, 502, 503, 504, 505],
        "customer_name": ["Asha", "Rahul", "Meera", "Vikram", "Nisha"],
        "city": ["Bengaluru", "Mumbai", "Chennai", "Bengaluru", "Pune"],
        "signup_date": ["2025-12-01", "2025-12-15", "2026-01-01", "2026-01-03", "2026-01-10"],
    }
)
Enter fullscreen mode Exit fullscreen mode

Merge customer information into orders:

orders_with_customers = orders.merge(
    customers,
    on="customer_id",
    how="left"
)

print(orders_with_customers.head())
Enter fullscreen mode Exit fullscreen mode

Choosing a join type

Join type What it keeps Typical use case
inner Only matching records in both DataFrames Analyze orders with known customers
left All rows from the left DataFrame Keep every order, even if customer data is missing
right All rows from the right DataFrame Less common; keep all customer records
outer Every row from both DataFrames Data reconciliation and quality checks

Example: identify orders that did not find a matching customer.

orders_with_customers = orders.merge(
    customers,
    on="customer_id",
    how="left",
    indicator=True
)

unmatched_orders = orders_with_customers[
    orders_with_customers["_merge"] == "left_only"
]

print(unmatched_orders)
Enter fullscreen mode Exit fullscreen mode

This is extremely useful when integrating services or validating ETL pipelines.


A Reusable GitHub-Style Script

Here is a compact script you could place in src/analysis.py.

from pathlib import Path
import pandas as pd


DATA_PATH = Path("data/orders.csv")
OUTPUT_PATH = Path("data/product_revenue.csv")


def load_orders(path: Path) -> pd.DataFrame:
    df = pd.read_csv(path, parse_dates=["order_date"])

    required_columns = {
        "order_id",
        "customer_id",
        "product",
        "category",
        "quantity",
        "unit_price",
        "order_date",
    }

    missing_columns = required_columns - set(df.columns)

    if missing_columns:
        raise ValueError(
            f"Missing required columns: {sorted(missing_columns)}"
        )

    return df


def clean_orders(df: pd.DataFrame) -> pd.DataFrame:
    clean_df = df.copy()

    clean_df["product"] = clean_df["product"].str.strip().str.title()
    clean_df["category"] = clean_df["category"].str.strip().str.title()

    clean_df["quantity"] = pd.to_numeric(
        clean_df["quantity"],
        errors="coerce"
    )

    clean_df["unit_price"] = pd.to_numeric(
        clean_df["unit_price"],
        errors="coerce"
    )

    clean_df = clean_df.dropna(
        subset=["order_id", "product", "quantity", "unit_price"]
    )

    clean_df = clean_df.drop_duplicates(
        subset=["order_id"]
    )

    clean_df = clean_df[
        (clean_df["quantity"] > 0) &
        (clean_df["unit_price"] >= 0)
    ]

    clean_df["revenue"] = (
        clean_df["quantity"] * clean_df["unit_price"]
    )

    return clean_df


def create_product_report(df: pd.DataFrame) -> pd.DataFrame:
    report = (
        df.groupby("product", as_index=False)
        .agg(
            orders=("order_id", "count"),
            units_sold=("quantity", "sum"),
            revenue=("revenue", "sum"),
        )
        .sort_values("revenue", ascending=False)
    )

    return report


def main() -> None:
    orders = load_orders(DATA_PATH)
    clean_orders_df = clean_orders(orders)
    report = create_product_report(clean_orders_df)

    report.to_csv(OUTPUT_PATH, index=False)

    print("Product revenue report created.")
    print(report)


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run it:

python src/analysis.py
Enter fullscreen mode Exit fullscreen mode

This is a good starting point for a GitHub project because it separates loading, validation, cleaning, and reporting into testable functions.

A useful README command section could include:

git clone <your-repository-url>
cd pandas-ecommerce-tutorial
python -m pip install -r requirements.txt
python src/analysis.py
Enter fullscreen mode Exit fullscreen mode

Avoid committing sensitive production data, API keys, database credentials, or customer-identifiable information to GitHub.


Common Errors and Fixes

KeyError: 'column_name'

This means Pandas cannot find the column you requested.

orders["customer"]
Enter fullscreen mode Exit fullscreen mode

If the actual column is customer_id, Pandas raises a KeyError.

Check the columns:

print(orders.columns.tolist())
Enter fullscreen mode Exit fullscreen mode

A common cause is whitespace in CSV headers. Fix it:

orders.columns = orders.columns.str.strip()
Enter fullscreen mode Exit fullscreen mode

You can also normalize names:

orders.columns = (
    orders.columns
    .str.strip()
    .str.lower()
    .str.replace(" ", "_")
)
Enter fullscreen mode Exit fullscreen mode

TypeError: can't multiply sequence by non-int

This typically means one of your numeric columns is actually text.

orders["revenue"] = orders["quantity"] * orders["unit_price"]
Enter fullscreen mode Exit fullscreen mode

Inspect types:

print(orders.dtypes)
Enter fullscreen mode Exit fullscreen mode

Convert safely:

orders["quantity"] = pd.to_numeric(
    orders["quantity"],
    errors="coerce"
)

orders["unit_price"] = pd.to_numeric(
    orders["unit_price"],
    errors="coerce"
)
Enter fullscreen mode Exit fullscreen mode

SettingWithCopyWarning

This happens when Pandas is unsure whether you are modifying a view or a copy.

Problematic pattern:

accessories = orders[orders["category"] == "Accessories"]
accessories["discount"] = 0.10
Enter fullscreen mode Exit fullscreen mode

Better:

accessories = orders[
    orders["category"] == "Accessories"
].copy()

accessories["discount"] = 0.10
Enter fullscreen mode Exit fullscreen mode

Or update the original DataFrame using .loc:

orders.loc[
    orders["category"] == "Accessories",
    "discount"
] = 0.10
Enter fullscreen mode Exit fullscreen mode

Dates Not Behaving Like Dates

If this fails:

orders["order_date"].dt.month
Enter fullscreen mode Exit fullscreen mode

Your date column is probably still an object/string.

Fix it:

orders["order_date"] = pd.to_datetime(
    orders["order_date"],
    errors="coerce"
)
Enter fullscreen mode Exit fullscreen mode

For a known date pattern, specify it:

orders["order_date"] = pd.to_datetime(
    orders["order_date"],
    format="%d-%m-%Y",
    errors="coerce"
)
Enter fullscreen mode Exit fullscreen mode

Merge Creates More Rows Than Expected

This often means the join key appears multiple times in one or both DataFrames. A many-to-many merge can multiply records.

Validate the expected relationship:

orders.merge(
    customers,
    on="customer_id",
    how="left",
    validate="many_to_one"
)
Enter fullscreen mode Exit fullscreen mode

If customer IDs should be unique in customers, this validation will catch duplicates immediately.


Performance Tips for Large Files

Pandas is fast, but loading millions of rows without a plan can consume significant memory.

Read only needed columns

orders = pd.read_csv(
    "data/orders.csv",
    usecols=[
        "order_id",
        "customer_id",
        "product",
        "quantity",
        "unit_price",
        "order_date",
    ]
)
Enter fullscreen mode Exit fullscreen mode

Set data types while loading

orders = pd.read_csv(
    "data/orders.csv",
    dtype={
        "order_id": "int64",
        "customer_id": "int64",
        "product": "category",
        "quantity": "int32",
        "unit_price": "float32",
    },
    parse_dates=["order_date"]
)
Enter fullscreen mode Exit fullscreen mode

The category type is particularly helpful for columns with repeated values, such as city, product category, status, country, or payment method.

orders["category"] = orders["category"].astype("category")
Enter fullscreen mode Exit fullscreen mode

Process large files in chunks

For files too large to fit in memory:

import pandas as pd

total_revenue = 0

for chunk in pd.read_csv(
    "data/orders.csv",
    chunksize=100_000
):
    chunk["quantity"] = pd.to_numeric(
        chunk["quantity"],
        errors="coerce"
    )

    chunk["unit_price"] = pd.to_numeric(
        chunk["unit_price"],
        errors="coerce"
    )

    chunk["revenue"] = (
        chunk["quantity"] * chunk["unit_price"]
    )

    total_revenue += chunk["revenue"].sum()

print(total_revenue)
Enter fullscreen mode Exit fullscreen mode

This pattern is practical for log files, transaction exports, event data, and analytics pipelines.

Prefer vectorized work

Use built-in column operations, groupby(), merge(), where(), and string methods before reaching for apply().

Usually slower:

orders["revenue"] = orders.apply(
    lambda row: row["quantity"] * row["unit_price"],
    axis=1
)
Enter fullscreen mode Exit fullscreen mode

Usually faster:

orders["revenue"] = (
    orders["quantity"] * orders["unit_price"]
)
Enter fullscreen mode Exit fullscreen mode

Save cleaned data in Parquet

CSV is portable, but Parquet is generally smaller, preserves data types, and is faster for repeated analytics workflows.

orders.to_parquet(
    "data/clean_orders.parquet",
    index=False
)
Enter fullscreen mode Exit fullscreen mode

Read it back:

orders = pd.read_parquet(
    "data/clean_orders.parquet"
)
Enter fullscreen mode Exit fullscreen mode

You may need:

python -m pip install pyarrow
Enter fullscreen mode Exit fullscreen mode

Best Practices for Production Work

Pandas notebooks are excellent for exploration. Production data workflows need a little more structure.

  • Copy input DataFrames before making major transformations if functions should not mutate caller-owned data.
  • Validate required columns before processing a file.
  • Convert data types early, especially IDs, dates, numeric amounts, and categories.
  • Treat missing values as a business decision, not merely a technical inconvenience.
  • Use .loc[] for assignment and .copy() after filtering when needed.
  • Keep raw data immutable and write cleaned outputs separately.
  • Use descriptive names such as monthly_revenue, customer_spending, and clean_orders.
  • Add tests for key transformations, especially joins, deduplication rules, and revenue calculations.
  • Log row counts before and after cleaning. A sudden drop can signal a broken source file.
  • Store reusable work in .py modules and reserve notebooks for exploration and communication.

If you are comparing structured courses, tutorials, or the best data science training in bangalore, use this same practical checklist: does the material teach data loading, validation, cleaning, joins, grouping, debugging, performance, and end-to-end project structure? Those are the skills developers use repeatedly.


Practice Exercises

Try these before looking up solutions.

Exercise 1: Find the top customer

Using orders, calculate total revenue per customer and return the customer with the highest spending.

Hint:

orders.groupby("customer_id")["revenue"].sum()
Enter fullscreen mode Exit fullscreen mode

Exercise 2: Identify incomplete records

Create a DataFrame containing only orders where product, quantity, or unit_price is missing.

Hint:

orders[orders[["product", "quantity", "unit_price"]].isna().any(axis=1)]
Enter fullscreen mode Exit fullscreen mode

Exercise 3: Add a discount rule

Create a discount_rate column:

  • 15% if revenue is at least 10,000
  • 10% if revenue is at least 2,000
  • 0% otherwise

Then calculate final_revenue.

Exercise 4: Monthly category report

Generate a report with:

  • Month
  • Category
  • Number of orders
  • Total quantity
  • Total revenue

Hint:

orders["month"] = orders["order_date"].dt.to_period("M")
Enter fullscreen mode Exit fullscreen mode

Then group by ["month", "category"].

Exercise 5: Data quality checks

Write a function that raises a ValueError when:

  • order_id has duplicates
  • Quantity is zero or negative
  • Unit price is negative
  • Order dates cannot be parsed
  • Required columns are missing

This is a valuable exercise because real data engineering and analytics work often involves validating inputs before calculating outputs.


Learning Resources

Once you are comfortable with the examples above, deepen your skills through practice rather than memorizing every Pandas method.

A productive path looks like this:

  1. Load a CSV from a real project or public dataset.
  2. Run head(), info(), describe(), and isna().sum().
  3. Clean one or two meaningful issues.
  4. Write a small report with groupby().
  5. Join a second dataset.
  6. Export the cleaned result.
  7. Turn the notebook logic into a reusable Python script.
  8. Add the project to GitHub with a clear README.

Useful topics to explore next:

  • pivot_table() for spreadsheet-style summaries
  • melt() for reshaping wide data into long data
  • concat() for combining datasets vertically
  • Time-series analysis with resample()
  • Data validation with Pandera or Pydantic
  • SQL and Pandas workflows
  • Visualization with Matplotlib, Seaborn, or Plotly
  • Faster DataFrame tools such as Polars or DuckDB for larger workloads

Final Takeaway

Pandas becomes much easier when you treat it as a workflow:

Load → Inspect → Clean → Transform → Analyze → Validate → Export
Enter fullscreen mode Exit fullscreen mode

Start with small, understandable datasets. Write transformations as readable steps. Prefer vectorized operations over loops. Validate assumptions before merging or aggregating. Most importantly, practice with messy real-world data—not only neat tutorial examples.

That is where Pandas shifts from “a library I know a little” to “a tool I can use to solve problems.”

Top comments (0)