Getting to Know Your Data: An Introduction to Pandas

Once you start working with real-world data like spreadsheets, CSV exports, database then Python's built-in lists and dictionaries quickly become cumbersome. Pandas is the library that fills this gap: it gives you a structure built specifically for tabular data, along with fast, readable tools for exploring it before you do any real analysis. This article introduces pandas and covers the first thing you should do with any new dataset.

What Is Pandas and Why It Exists?

Pandas is an open-source library used for data manipulation, analysis and cleaning.
Pandas consists of:

1. Series - a single column

2. DataFrame - more than one column. A table with labeled rows and columns similar to a spreadsheet.

Series Example

import pandas as pd

student_list = pd.Series(["Amina","Brian","Fatuma","Dennis"])

student_list

Output

0     Amina
1     Brian
2    Fatuma
3    Dennis
dtype: str

DataFrame Example

data = {
    "name": ["Ana", "Sam", "Lee"],
    "age": [29, 34, 41],
    "city": ["Nairobi", "Lagos", "Accra"]
}
results = pd.DataFrame(data)
results

Output

  name  age     city
0  Ana   29  Nairobi
1  Sam   34    Lagos
2  Lee   41    Accra

In practice, you will rarely type data in by hand like this but you will read it from a file.

Reading a CSV File

sales = pd.read_csv(r"C:\Data Science\Python Files\pharmacy_sales.csv")

Reading an Excel File

sales = pd.read_excel(r"C:\Data Science\Python Files\pharmacy_sales.xlsx")

Reading From Database

from sqlalchemy import create_engine

engine = create_engine("postgresql+psycopg2://username:password@localhost:5432/database_name")

# define query and columns to select
query = "SELECT order_id, city, branch, channel FROM public.pharmacy_sales"

# read the database directly into a DataFrame
sales_sql = pd.read_sql(query, con=engine)

# preview the data
sales_sql.head()

• create_engine() - builds a connection object(an 'engine') that pandas and SQLAlchemy use to talk to the database.

• +psycopg2 - the specific driver used to talk to PostgreSQL (psycopg2 needs to be installed separately: pip install psycopg2-binary)

Inspecting a DataFrame

Before filtering, grouping, or calculating anything, it's worth building a habit: look at the data before you trust it. Real datasets almost always have missing values, wrong data types, unexpected ranges, duplicate rows. Catching these early saves you from building analysis on top of bad assumptions. The methods below are the standard first steps for any new DataFrame.

Structural Inspection

• df.info() - provides a summary including index range, column names, counts of non-values, data types and memory usage.

sales.info()

Output

<class 'pandas.DataFrame'>
RangeIndex: 6000 entries, 0 to 5999
Data columns (total 26 columns):
 #   Column                 Non-Null Count  Dtype         
---  ------                 --------------  -----         
 0   order_id               6000 non-null   str           
 1   order_date             6000 non-null   datetime64[us]
 2   order_time             6000 non-null   object        
 3   customer_id            6000 non-null   str           
 4   customer_gender        6000 non-null   str 
 ..  ...............        ...................                    
 25  customer_rating        4454 non-null   float64       
dtypes: bool(1), datetime64[us](1), float64(4), int64(6), object(1), str(13)
memory usage: 1.2+ MB

• df.columns - returns an index array of all column labels.

sales.columns

Output

Index(['order_id', 'order_date', 'order_time', 'customer_id',
       'customer_gender', 'customer_age', 'city', 'branch', 'channel',
       'product_id', 'product_name', 'brand', 'category', 'subcategory',
       'dosage_form', 'requires_prescription', 'list_price_kes',
       'unit_price_kes', 'unit_cost_kes', 'quantity', 'discount_pct',
       'total_amount_kes', 'delivery_fee_kes', 'gross_profit_kes',
       'payment_method', 'customer_rating'],
      dtype='str')

• df.shape - an attribute that returns a tuple representing the dimension (rows, columns).

sales.shape

Output

(6000, 26)

• df.dtypes - lists the data type of each column.

sales.dtypes

Output

order_id                            str
order_date               datetime64[us]
order_time                       object
customer_id                         str
customer_gender                     str
customer_age                      int64
city                                str
requires_prescription              bool
list_price_kes                    int64
unit_price_kes                    int64
unit_cost_kes                   float64
quantity                          int64
discount_pct                      int64
total_amount_kes                float64
delivery_fee_kes                  int64
gross_profit_kes                float64
payment_method                      str
customer_rating                 float64
dtype: object

Preview of The Data

• df.head() - returns the first n rows(defaults to 5).

• df.tail() - returns the last n rows(defaults to 5).

• df.sample() - returns a random selection of n rows.

Statistical Summary

• df.describe() - generate descriptive statistics for numerical columns (mean, standard deviation, min, max and quartiles).

  • df.describe(include='all') - to incorporate categorical summaries like unique values and top frequencies.

• df.describe().T - transposes the resulting table for better readability.

• df[['column_name']].describe() - generates descriptive statistics for a specific column.

• df['column_name'].value_counts() - gives the summary count of each of the categories in the categorical column.
  It helps with exposing inconsistent categories.
  

sales['city'].value_counts()

Output

city
Nairobi    3329
Mombasa    1020
Nakuru      599
Kisumu      582
Eldoret     470
Name: count, dtype: int64

Data Quality Inspection

• df.isna().sum() or df.isnull().sum() - counts the number of missing values per column.

sales.isna().sum()

Output

order_id                    0
order_date                  0
order_time                  0
customer_id                 0
customer_gender             0
customer_age                0
............               ..
customer_rating          1546
dtype: int64

• df.empty - A quick boolean attribute that returns True if the DataFrame contains zero elements.

sales.empty

Output

False

• df.duplicated().sum() - give the number of duplicated entries or rows.

sales.duplicated().sum()

Output

np.int64(0)

Data Indexing & Selection

• df.loc and df.iloc They are both used to select data.
• df.loc - selects data by label i.e index labels and column names (inclusive of the stop bound).

[0:2, ["order_id","order_date"]]

Output

  order_id      order_date     
0  ORD-100000   2026-01-01  
1  ORD-100001   2026-01-01    
2  ORD-100002   2026-01-01   

• df.iloc - selects data by integer position (exclusive of the stop bound).

sales.iloc[0:2,0:2]

Output

  order_id      order_date     
0  ORD-100000   2026-01-01  
1  ORD-100001   2026-01-01

• df['column_name'] - outputs a single column as a series.

Practical Inspection Routine

Here is a simple sequence that covers most of what you need to know about a new dataset before doing anything else with it:

import pandas as pd

df = pd.read_csv('sales_data.csv')

df.shape                 # how big is the dataset?
df.head()                # what does it look like?
df.info()                # types and missing values at a glance
df.describe()            # are the numbers reasonable?
df.isnull().sum()        # exactly where is data missing?
df.duplicated().sum()    # are there repeated rows? 

Conclusion

Pandas earns its popularity by making tabular data feel as natural to work with as a spreadsheet, but with the full power of Python behind it. Before any real analysis, filtering, or visualization, a quick inspection pass; shape, head, info, describe, missing values and duplicates tells you what you're actually working with. This step is easy to skip when you're eager to get to results, but skipping it is exactly how subtle data problems end up baked into an entire analysis without anyone noticing until much later.