Pandas is one of the most useful Python libraries for working with structured data. Whether you are cleaning a CSV export, analyzing application logs, transforming API responses, or preparing data for a machine-learning model, Pandas helps you move from raw data to useful answers quickly.

This tutorial is designed for developers who prefer building over theory. You will create DataFrames, load files, clean messy values, filter records, aggregate data, join datasets, handle errors, and make your code faster.
Note: The keyword best data science training in bangalore appears naturally in this article for search relevance, but this tutorial is focused on practical, self-directed development skills—not promotion.
What You’ll Build
We will work with a small e-commerce dataset and answer questions such as:
- Which products generated the most revenue?
- Which customers placed the most orders?
- How do we clean missing or malformed data?
- How do we combine orders with customer information?
- How can we process large CSV files without exhausting memory?
By the end, you will have a reusable Pandas workflow that applies to product analytics, backend reporting, finance exports, operational dashboards, and data-science projects.
Prerequisites
You need:
- Python 3.9 or newer
- Basic Python knowledge: variables, lists, dictionaries, functions
- A code editor such as VS Code or PyCharm
- A terminal
Install Pandas:
python -m pip install pandas
For notebook-based exploration, install Jupyter too:
python -m pip install jupyter pandas
Start a notebook:
jupyter notebook
Or create a normal Python file named pandas_tutorial.py.
Verify the installation:
import pandas as pd
print(pd.__version__)
You should see a version number such as 2.x.x.
Project Setup
Create a folder for the project:
mkdir pandas-ecommerce-tutorial
cd pandas-ecommerce-tutorial
A simple project structure might look like this:
pandas-ecommerce-tutorial/
├── data/
│ ├── orders.csv
│ └── customers.csv
├── notebooks/
│ └── analysis.ipynb
├── src/
│ └── analysis.py
├── requirements.txt
└── README.md
Add dependencies to requirements.txt:
pandas
jupyter
Install them:
python -m pip install -r requirements.txt
This structure works nicely in a GitHub repository. Keeping raw data, notebooks, and reusable scripts separate makes the project easier to maintain and review.
Understanding Pandas Basics
Pandas has two primary data structures:
- Series: A one-dimensional labeled column of data.
- DataFrame: A two-dimensional table made of rows and columns.
Think of a DataFrame as a spreadsheet, SQL result table, or JSON collection converted into tabular form.
Create a Series
import pandas as pd
prices = pd.Series([499.0, 799.0, 1299.0], name="price")
print(prices)
Output:
0 499.0
1 799.0
2 1299.0
Name: price, dtype: float64
The numbers on the left are the index. Pandas adds an index automatically unless you provide one.
inventory = pd.Series(
[10, 5, 18],
index=["Keyboard", "Mouse", "Monitor"],
name="stock"
)
print(inventory)
Output:
Keyboard 10
Mouse 5
Monitor 18
Name: stock, dtype: int64
Now you can access a value by label:
print(inventory["Mouse"])
Output:
5
Create a DataFrame
Here is a basic order dataset:
import pandas as pd
orders = pd.DataFrame(
{
"order_id": [1001, 1002, 1003, 1004, 1005],
"customer_id": [501, 502, 501, 503, 504],
"product": ["Keyboard", "Mouse", "Monitor", "Keyboard", "Mouse"],
"category": ["Accessories", "Accessories", "Displays", "Accessories", "Accessories"],
"quantity": [1, 2, 1, 1, 3],
"unit_price": [1200, 500, 15000, 1200, 500],
"order_date": ["2026-01-05", "2026-01-06", "2026-01-07", "2026-01-08", "2026-01-09"],
}
)
print(orders)
Every dictionary key becomes a column. Each list represents the values in that column.
Inspect the first few rows:
print(orders.head())
Inspect the final rows:
print(orders.tail(2))
Check dimensions:
print(orders.shape)
Output:
(5, 7)
This means five rows and seven columns.
Check column names:
print(orders.columns)
Check data types and missing values:
orders.info()
This is one of the first commands you should run after loading any unfamiliar dataset.
Loading CSV, JSON, and Excel Data
Most practical Pandas work begins with reading files.
Load a CSV file
Create data/orders.csv:
order_id,customer_id,product,category,quantity,unit_price,order_date
1001,501,Keyboard,Accessories,1,1200,2026-01-05
1002,502,Mouse,Accessories,2,500,2026-01-06
1003,501,Monitor,Displays,1,15000,2026-01-07
1004,503,Keyboard,Accessories,1,1200,2026-01-08
1005,504,Mouse,Accessories,3,500,2026-01-09
Read it:
import pandas as pd
orders = pd.read_csv("data/orders.csv")
print(orders.head())
If your date column should be treated as a date immediately, use parse_dates:
orders = pd.read_csv(
"data/orders.csv",
parse_dates=["order_date"]
)
print(orders.dtypes)
This prevents a common problem: dates loaded as plain strings.
Load JSON from an API response
Suppose an API returns this JSON-like Python object:
api_response = [
{"id": 1, "name": "Asha", "city": "Bengaluru"},
{"id": 2, "name": "Rahul", "city": "Mumbai"},
{"id": 3, "name": "Meera", "city": "Chennai"},
]
customers = pd.DataFrame(api_response)
print(customers)
For a JSON file:
customers = pd.read_json("data/customers.json")
Nested JSON needs flattening. Use pd.json_normalize():
nested_data = [
{
"order_id": 1001,
"customer": {
"name": "Asha",
"city": "Bengaluru"
},
"total": 1200
}
]
df = pd.json_normalize(nested_data)
print(df)
Expected columns:
order_id total customer.name customer.city
Load Excel files
Install the Excel engine:
python -m pip install openpyxl
Then read a sheet:
sales = pd.read_excel(
"data/monthly_sales.xlsx",
sheet_name="January"
)
For large Excel files, convert them to CSV when possible. CSV files are usually faster to read and easier to process in pipelines.
Explore Data Before Changing It
Before filtering, cleaning, or aggregating a dataset, inspect its structure.
print(orders.head())
print(orders.sample(3, random_state=42))
print(orders.info())
print(orders.describe())
describe() summarizes numeric fields:
print(orders.describe())
Typical output includes:
- Count
- Mean
- Standard deviation
- Minimum and maximum
- Quartiles
For categorical fields, use:
print(orders["category"].value_counts())
To include missing values in the count:
print(orders["category"].value_counts(dropna=False))
Check unique values:
print(orders["product"].unique())
Count them:
print(orders["product"].nunique())
These quick checks catch issues early: unexpected product names, inconsistent spelling, duplicate categories, or fields that contain empty strings instead of proper missing values.
Select, Filter, and Sort Rows
Most analysis involves selecting the subset of data that matters.
Select columns
Select one column:
products = orders["product"]
print(products)
Select multiple columns:
order_summary = orders[["order_id", "product", "quantity", "unit_price"]]
print(order_summary)
Filter rows
Find all keyboard orders:
keyboard_orders = orders[orders["product"] == "Keyboard"]
print(keyboard_orders)
Find orders with quantities greater than one:
bulk_orders = orders[orders["quantity"] > 1]
print(bulk_orders)
Use multiple conditions with & and |.
high_value_accessories = orders[
(orders["category"] == "Accessories") &
(orders["unit_price"] >= 1000)
]
print(high_value_accessories)
Always wrap each condition in parentheses. This avoids confusing operator-precedence errors.
Filter with several allowed values:
selected_products = orders[
orders["product"].isin(["Keyboard", "Monitor"])
]
print(selected_products)
Filter missing values:
missing_price = orders[orders["unit_price"].isna()]
Filter non-missing values:
valid_price = orders[orders["unit_price"].notna()]
Use query() for readable filters
For complex filters, query() can be easier to read:
result = orders.query(
"category == 'Accessories' and quantity >= 2"
)
print(result)
This is especially useful in exploratory notebooks. For production code, normal boolean indexing is often easier to refactor and validate with static tools.
Sort data
Sort by unit price:
sorted_orders = orders.sort_values("unit_price", ascending=False)
print(sorted_orders)
Sort by multiple columns:
sorted_orders = orders.sort_values(
by=["category", "unit_price"],
ascending=[True, False]
)
Create Calculated Columns
Raw datasets often do not contain the metric you need. Add it.
Calculate revenue
Revenue is quantity multiplied by unit price:
orders["revenue"] = orders["quantity"] * orders["unit_price"]
print(orders[["product", "quantity", "unit_price", "revenue"]])
Expected result:
product quantity unit_price revenue
0 Keyboard 1 1200 1200
1 Mouse 2 500 1000
2 Monitor 1 15000 15000
3 Keyboard 1 1200 1200
4 Mouse 3 500 1500
Use vectorized operations
Pandas works best when you apply operations to entire columns.
orders["discounted_price"] = orders["unit_price"] * 0.90
Avoid manual loops:
# Avoid this for normal column calculations
for index, row in orders.iterrows():
orders.loc[index, "revenue"] = row["quantity"] * row["unit_price"]
The loop works, but it is slow and verbose. The vectorized version is faster and clearer:
orders["revenue"] = orders["quantity"] * orders["unit_price"]
Create conditional columns
Use numpy.select() or np.where() for simple conditions.
import numpy as np
orders["order_size"] = np.where(
orders["revenue"] >= 5000,
"High value",
"Standard"
)
print(orders[["order_id", "revenue", "order_size"]])
For multiple conditions:
conditions = [
orders["revenue"] >= 10000,
orders["revenue"] >= 2000,
]
choices = [
"Enterprise",
"Medium",
]
orders["customer_segment"] = np.select(
conditions,
choices,
default="Small"
)
Clean Real-World Data
Production data is rarely clean. A CSV might have blank values, extra whitespace, duplicate rows, inconsistent capitalization, or invalid numbers.
Here is a deliberately messy dataset:
messy_orders = pd.DataFrame(
{
"order_id": [1001, 1002, 1002, 1003, 1004],
"product": [" Keyboard ", "mouse", "mouse", None, "MONITOR"],
"quantity": ["1", "2", "2", "one", None],
"unit_price": [1200, 500, 500, 15000, None],
"city": [" Bengaluru", "Mumbai ", "Mumbai ", "Chennai", "Bengaluru"],
}
)
print(messy_orders)
Find missing values
print(messy_orders.isna().sum())
This gives the count of missing values in every column.
Standardize text fields
Remove whitespace and normalize case:
messy_orders["product"] = (
messy_orders["product"]
.str.strip()
.str.title()
)
messy_orders["city"] = (
messy_orders["city"]
.str.strip()
.str.title()
)
Now " Keyboard " becomes "Keyboard" and "MONITOR" becomes "Monitor".
Convert invalid numeric values safely
messy_orders["quantity"] = pd.to_numeric(
messy_orders["quantity"],
errors="coerce"
)
The string "one" cannot become a number, so Pandas converts it to NaN rather than crashing.
Check the result:
print(messy_orders[["quantity"]])
Handle missing values
You have several options.
Remove rows where important data is missing:
clean_orders = messy_orders.dropna(
subset=["product", "quantity", "unit_price"]
)
Fill missing numeric values:
messy_orders["unit_price"] = messy_orders["unit_price"].fillna(0)
Fill missing text:
messy_orders["city"] = messy_orders["city"].fillna("Unknown")
Use a meaningful business rule. Filling missing prices with zero may be acceptable for a temporary report, but it can distort revenue calculations. In many systems, invalid pricing data should be flagged and excluded instead.
Remove duplicates
deduplicated_orders = messy_orders.drop_duplicates()
Or remove duplicates based on business keys:
deduplicated_orders = messy_orders.drop_duplicates(
subset=["order_id"]
)
Be careful with this step. Repeated product rows can be valid if an order contains multiple line items. Deduplicate only after understanding what a row represents.
Group, Aggregate, and Analyze
Aggregation is where Pandas starts to feel like SQL.
First, ensure the revenue column exists:
orders["revenue"] = orders["quantity"] * orders["unit_price"]
Revenue by product
revenue_by_product = (
orders.groupby("product", as_index=False)["revenue"]
.sum()
.sort_values("revenue", ascending=False)
)
print(revenue_by_product)
This is similar to:
SELECT product, SUM(revenue)
FROM orders
GROUP BY product
ORDER BY revenue DESC;
Multiple aggregations
product_metrics = (
orders.groupby("product", as_index=False)
.agg(
total_orders=("order_id", "count"),
total_quantity=("quantity", "sum"),
total_revenue=("revenue", "sum"),
average_order_value=("revenue", "mean"),
)
.sort_values("total_revenue", ascending=False)
)
print(product_metrics)
Named aggregations make output columns explicit and readable.
Revenue by category
category_metrics = (
orders.groupby("category", as_index=False)
.agg(
revenue=("revenue", "sum"),
units_sold=("quantity", "sum"),
)
.sort_values("revenue", ascending=False)
)
print(category_metrics)
Customer spending
customer_spending = (
orders.groupby("customer_id", as_index=False)
.agg(
order_count=("order_id", "count"),
total_spend=("revenue", "sum"),
)
.sort_values("total_spend", ascending=False)
)
print(customer_spending)
Group data by month
Convert the order date first:
orders["order_date"] = pd.to_datetime(orders["order_date"])
Create a monthly period:
orders["order_month"] = orders["order_date"].dt.to_period("M")
Aggregate:
monthly_revenue = (
orders.groupby("order_month", as_index=False)["revenue"]
.sum()
)
print(monthly_revenue)
For time-series reporting, resample() is another powerful option:
daily_revenue = (
orders.set_index("order_date")
.resample("D")["revenue"]
.sum()
)
print(daily_revenue)
Join Multiple DataFrames
In real applications, data is rarely stored in one table. You might have:
- Orders from a transaction system
- Customers from a CRM
- Product metadata from a catalog
- Support tickets from a help desk
Create customer data:
customers = pd.DataFrame(
{
"customer_id": [501, 502, 503, 504, 505],
"customer_name": ["Asha", "Rahul", "Meera", "Vikram", "Nisha"],
"city": ["Bengaluru", "Mumbai", "Chennai", "Bengaluru", "Pune"],
"signup_date": ["2025-12-01", "2025-12-15", "2026-01-01", "2026-01-03", "2026-01-10"],
}
)
Merge customer information into orders:
orders_with_customers = orders.merge(
customers,
on="customer_id",
how="left"
)
print(orders_with_customers.head())
Choosing a join type
| Join type | What it keeps | Typical use case |
|---|---|---|
inner |
Only matching records in both DataFrames | Analyze orders with known customers |
left |
All rows from the left DataFrame | Keep every order, even if customer data is missing |
right |
All rows from the right DataFrame | Less common; keep all customer records |
outer |
Every row from both DataFrames | Data reconciliation and quality checks |
Example: identify orders that did not find a matching customer.
orders_with_customers = orders.merge(
customers,
on="customer_id",
how="left",
indicator=True
)
unmatched_orders = orders_with_customers[
orders_with_customers["_merge"] == "left_only"
]
print(unmatched_orders)
This is extremely useful when integrating services or validating ETL pipelines.
A Reusable GitHub-Style Script
Here is a compact script you could place in src/analysis.py.
from pathlib import Path
import pandas as pd
DATA_PATH = Path("data/orders.csv")
OUTPUT_PATH = Path("data/product_revenue.csv")
def load_orders(path: Path) -> pd.DataFrame:
df = pd.read_csv(path, parse_dates=["order_date"])
required_columns = {
"order_id",
"customer_id",
"product",
"category",
"quantity",
"unit_price",
"order_date",
}
missing_columns = required_columns - set(df.columns)
if missing_columns:
raise ValueError(
f"Missing required columns: {sorted(missing_columns)}"
)
return df
def clean_orders(df: pd.DataFrame) -> pd.DataFrame:
clean_df = df.copy()
clean_df["product"] = clean_df["product"].str.strip().str.title()
clean_df["category"] = clean_df["category"].str.strip().str.title()
clean_df["quantity"] = pd.to_numeric(
clean_df["quantity"],
errors="coerce"
)
clean_df["unit_price"] = pd.to_numeric(
clean_df["unit_price"],
errors="coerce"
)
clean_df = clean_df.dropna(
subset=["order_id", "product", "quantity", "unit_price"]
)
clean_df = clean_df.drop_duplicates(
subset=["order_id"]
)
clean_df = clean_df[
(clean_df["quantity"] > 0) &
(clean_df["unit_price"] >= 0)
]
clean_df["revenue"] = (
clean_df["quantity"] * clean_df["unit_price"]
)
return clean_df
def create_product_report(df: pd.DataFrame) -> pd.DataFrame:
report = (
df.groupby("product", as_index=False)
.agg(
orders=("order_id", "count"),
units_sold=("quantity", "sum"),
revenue=("revenue", "sum"),
)
.sort_values("revenue", ascending=False)
)
return report
def main() -> None:
orders = load_orders(DATA_PATH)
clean_orders_df = clean_orders(orders)
report = create_product_report(clean_orders_df)
report.to_csv(OUTPUT_PATH, index=False)
print("Product revenue report created.")
print(report)
if __name__ == "__main__":
main()
Run it:
python src/analysis.py
This is a good starting point for a GitHub project because it separates loading, validation, cleaning, and reporting into testable functions.
A useful README command section could include:
git clone <your-repository-url>
cd pandas-ecommerce-tutorial
python -m pip install -r requirements.txt
python src/analysis.py
Avoid committing sensitive production data, API keys, database credentials, or customer-identifiable information to GitHub.
Common Errors and Fixes
KeyError: 'column_name'
This means Pandas cannot find the column you requested.
orders["customer"]
If the actual column is customer_id, Pandas raises a KeyError.
Check the columns:
print(orders.columns.tolist())
A common cause is whitespace in CSV headers. Fix it:
orders.columns = orders.columns.str.strip()
You can also normalize names:
orders.columns = (
orders.columns
.str.strip()
.str.lower()
.str.replace(" ", "_")
)
TypeError: can't multiply sequence by non-int
This typically means one of your numeric columns is actually text.
orders["revenue"] = orders["quantity"] * orders["unit_price"]
Inspect types:
print(orders.dtypes)
Convert safely:
orders["quantity"] = pd.to_numeric(
orders["quantity"],
errors="coerce"
)
orders["unit_price"] = pd.to_numeric(
orders["unit_price"],
errors="coerce"
)
SettingWithCopyWarning
This happens when Pandas is unsure whether you are modifying a view or a copy.
Problematic pattern:
accessories = orders[orders["category"] == "Accessories"]
accessories["discount"] = 0.10
Better:
accessories = orders[
orders["category"] == "Accessories"
].copy()
accessories["discount"] = 0.10
Or update the original DataFrame using .loc:
orders.loc[
orders["category"] == "Accessories",
"discount"
] = 0.10
Dates Not Behaving Like Dates
If this fails:
orders["order_date"].dt.month
Your date column is probably still an object/string.
Fix it:
orders["order_date"] = pd.to_datetime(
orders["order_date"],
errors="coerce"
)
For a known date pattern, specify it:
orders["order_date"] = pd.to_datetime(
orders["order_date"],
format="%d-%m-%Y",
errors="coerce"
)
Merge Creates More Rows Than Expected
This often means the join key appears multiple times in one or both DataFrames. A many-to-many merge can multiply records.
Validate the expected relationship:
orders.merge(
customers,
on="customer_id",
how="left",
validate="many_to_one"
)
If customer IDs should be unique in customers, this validation will catch duplicates immediately.
Performance Tips for Large Files
Pandas is fast, but loading millions of rows without a plan can consume significant memory.
Read only needed columns
orders = pd.read_csv(
"data/orders.csv",
usecols=[
"order_id",
"customer_id",
"product",
"quantity",
"unit_price",
"order_date",
]
)
Set data types while loading
orders = pd.read_csv(
"data/orders.csv",
dtype={
"order_id": "int64",
"customer_id": "int64",
"product": "category",
"quantity": "int32",
"unit_price": "float32",
},
parse_dates=["order_date"]
)
The category type is particularly helpful for columns with repeated values, such as city, product category, status, country, or payment method.
orders["category"] = orders["category"].astype("category")
Process large files in chunks
For files too large to fit in memory:
import pandas as pd
total_revenue = 0
for chunk in pd.read_csv(
"data/orders.csv",
chunksize=100_000
):
chunk["quantity"] = pd.to_numeric(
chunk["quantity"],
errors="coerce"
)
chunk["unit_price"] = pd.to_numeric(
chunk["unit_price"],
errors="coerce"
)
chunk["revenue"] = (
chunk["quantity"] * chunk["unit_price"]
)
total_revenue += chunk["revenue"].sum()
print(total_revenue)
This pattern is practical for log files, transaction exports, event data, and analytics pipelines.
Prefer vectorized work
Use built-in column operations, groupby(), merge(), where(), and string methods before reaching for apply().
Usually slower:
orders["revenue"] = orders.apply(
lambda row: row["quantity"] * row["unit_price"],
axis=1
)
Usually faster:
orders["revenue"] = (
orders["quantity"] * orders["unit_price"]
)
Save cleaned data in Parquet
CSV is portable, but Parquet is generally smaller, preserves data types, and is faster for repeated analytics workflows.
orders.to_parquet(
"data/clean_orders.parquet",
index=False
)
Read it back:
orders = pd.read_parquet(
"data/clean_orders.parquet"
)
You may need:
python -m pip install pyarrow
Best Practices for Production Work
Pandas notebooks are excellent for exploration. Production data workflows need a little more structure.
- Copy input DataFrames before making major transformations if functions should not mutate caller-owned data.
- Validate required columns before processing a file.
- Convert data types early, especially IDs, dates, numeric amounts, and categories.
- Treat missing values as a business decision, not merely a technical inconvenience.
- Use
.loc[]for assignment and.copy()after filtering when needed. - Keep raw data immutable and write cleaned outputs separately.
- Use descriptive names such as
monthly_revenue,customer_spending, andclean_orders. - Add tests for key transformations, especially joins, deduplication rules, and revenue calculations.
- Log row counts before and after cleaning. A sudden drop can signal a broken source file.
- Store reusable work in
.pymodules and reserve notebooks for exploration and communication.
If you are comparing structured courses, tutorials, or the best data science training in bangalore, use this same practical checklist: does the material teach data loading, validation, cleaning, joins, grouping, debugging, performance, and end-to-end project structure? Those are the skills developers use repeatedly.
Practice Exercises
Try these before looking up solutions.
Exercise 1: Find the top customer
Using orders, calculate total revenue per customer and return the customer with the highest spending.
Hint:
orders.groupby("customer_id")["revenue"].sum()
Exercise 2: Identify incomplete records
Create a DataFrame containing only orders where product, quantity, or unit_price is missing.
Hint:
orders[orders[["product", "quantity", "unit_price"]].isna().any(axis=1)]
Exercise 3: Add a discount rule
Create a discount_rate column:
- 15% if revenue is at least 10,000
- 10% if revenue is at least 2,000
- 0% otherwise
Then calculate final_revenue.
Exercise 4: Monthly category report
Generate a report with:
- Month
- Category
- Number of orders
- Total quantity
- Total revenue
Hint:
orders["month"] = orders["order_date"].dt.to_period("M")
Then group by ["month", "category"].
Exercise 5: Data quality checks
Write a function that raises a ValueError when:
-
order_idhas duplicates - Quantity is zero or negative
- Unit price is negative
- Order dates cannot be parsed
- Required columns are missing
This is a valuable exercise because real data engineering and analytics work often involves validating inputs before calculating outputs.
Learning Resources
Once you are comfortable with the examples above, deepen your skills through practice rather than memorizing every Pandas method.
A productive path looks like this:
- Load a CSV from a real project or public dataset.
- Run
head(),info(),describe(), andisna().sum(). - Clean one or two meaningful issues.
- Write a small report with
groupby(). - Join a second dataset.
- Export the cleaned result.
- Turn the notebook logic into a reusable Python script.
- Add the project to GitHub with a clear README.
Useful topics to explore next:
-
pivot_table()for spreadsheet-style summaries -
melt()for reshaping wide data into long data -
concat()for combining datasets vertically - Time-series analysis with
resample() - Data validation with Pandera or Pydantic
- SQL and Pandas workflows
- Visualization with Matplotlib, Seaborn, or Plotly
- Faster DataFrame tools such as Polars or DuckDB for larger workloads
Final Takeaway
Pandas becomes much easier when you treat it as a workflow:
Load → Inspect → Clean → Transform → Analyze → Validate → Export
Start with small, understandable datasets. Write transformations as readable steps. Prefer vectorized operations over loops. Validate assumptions before merging or aggregating. Most importantly, practice with messy real-world data—not only neat tutorial examples.
That is where Pandas shifts from “a library I know a little” to “a tool I can use to solve problems.”
Top comments (0)