DEV Community: Ugonma Ononogbu

Introduction to Pandas

Ugonma Ononogbu — Sat, 22 Jun 2024 19:54:45 +0000

Introduction

If you're reading this article, you probably want to get an understanding of Pandas, so let's get to it.

Pandas - short for “Panel Data” - is a popular open-source programming language widely used for performing data manipulation and analysis. It has in-built functions to efficiently clean, transform, manipulate, visualize, and analyze data.

The Pandas library is an essential tool for Data analysts, Scientists, and Engineers working with structured data in Python.

This article will teach you basic functions you need to know when using Pandas library–its specific uses, and how to install Pandas.

Getting Started with Pandas

Let’s learn how to install Python Pandas Library.

How to install Pandas:

When working with Pandas Library, the first step is to ensure that it is installed in the system using the pip command.

pip install pandas

Requirement already satisfied: pandas in c:\users\userpc\anaconda3\lib\site-packages (2.1.4)Note: you may need to restart the kernel to use updated packages.

Requirement already satisfied: numpy<2,>=1.23.2 in c:\users\userpc\anaconda3\lib\site-packages (from pandas) (1.24.3)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\userpc\anaconda3\lib\site-packages (from pandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in c:\users\userpc\anaconda3\lib\site-packages (from pandas) (2022.7)
Requirement already satisfied: tzdata>=2022.1 in c:\users\userpc\anaconda3\lib\site-packages (from pandas) (2023.4)
Requirement already satisfied: six>=1.5 in c:\users\userpc\anaconda3\lib\site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)

Importing Pandas

To begin working with Pandas, import pandas package as follows:

import pandas as pd

We are importing pandas from anaconda.
The most common and preferred short form for pandas is 'pd'. This shorter name is used because it makes the code shorter and easier to write whenever you need to use a pandas function.

Components of pandas

Pandas provides two data structures for manipulating data. They include:

Series
DataFrame

A Series is essentially a column while a DataFrame is like a multi-dimensional structure or table formed by combining multiple Series together.

Creating Series:

A pandas series is just like a column from an Excel Spreadsheet.

Example:

#create a pandas Series
numbers = pd.Series([1, 2, 3, 4, 5])

# Displaying the Series
numbers

Output:

0    1
1    2
2    3
3    4
4    5
dtype: int64

Note: The codes embedded inside the hashtag symbol(#) is called comments. Comments is used in python to explain what a code is about so that incase another programmer gets to go through your written code, the person can understand what the code is about.

Another example includes:

# Creating a Pandas Series of colors
colors_series = pd.Series(['red', 'blue', 'green'])

# Displaying the Series
colors_series

Output:

0      red
1     blue
2    green
dtype: object

For more reference on how create Pandas Series, refer to this article on Creating Pandas Series.

Creating DataFrame:

A DataFrame is a two-dimensional data structure similar to a table in a spreadsheet. It has rows and columns, where each row represents a record or observation, and each column represents a variable or an attribute.

A DataFrame lets you easily organize, manipulate, and analyze a dataset.

One simple way of creating DataFrames is by using a dictionary.

Here's how:


# Creating a DataFrame with fruits and car types
data = {'Fruits': ['Apple', 'Banana', 'Orange'],'Car Types': ['SUV', 'Sedan', 'Truck']}

df = pd.DataFrame(data)

# Displaying the DataFrame
df

Output:

	Fruits	Car Types
0	Apple	SUV
1	Banana	Sedan
2	Orange	Truck

Importing Datasets

There are various formats in which data can be imported into the working environment(in this case, the working environment is Jupyter notebook). These formats can be: CSV, excel, HTML, JSON, SQL, and many more.

Comma-Seperated Values(CSV) is the most common format. It is imported into the working environment by using the pd.read_csv() function.

Now let's import our real dataset from here at Kaggle

df = pd.read_csv("C:/Users/USERPC/Downloads/house_price.csv")
df

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	waterfront	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country
0	2014-05-02 00:00:00	3.130000e+05	3.0	1.50	1340	7912	1.5	0	0	3	1340	0	1955	2005	18810 Densmore Ave N	Shoreline	WA 98133	USA
1	2014-05-02 00:00:00	2.384000e+06	5.0	2.50	3650	9050	2.0	0	4	5	3370	280	1921	0	709 W Blaine St	Seattle	WA 98119	USA
2	2014-05-02 00:00:00	3.420000e+05	3.0	2.00	1930	11947	1.0	0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	Kent	WA 98042	USA
3	2014-05-02 00:00:00	4.200000e+05	3.0	2.25	2000	8030	1.0	0	0	4	1000	1000	1963	0	857 170th Pl NE	Bellevue	WA 98008	USA
4	2014-05-02 00:00:00	5.500000e+05	4.0	2.50	1940	10500	1.0	0	0	4	1140	800	1976	1992	9105 170th Ave NE	Redmond	WA 98052	USA
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
4595	2014-07-09 00:00:00	3.081667e+05	3.0	1.75	1510	6360	1.0	0	0	4	1510	0	1954	1979	501 N 143rd St	Seattle	WA 98133	USA
4596	2014-07-09 00:00:00	5.343333e+05	3.0	2.50	1460	7573	2.0	0	0	3	1460	0	1983	2009	14855 SE 10th Pl	Bellevue	WA 98007	USA
4597	2014-07-09 00:00:00	4.169042e+05	3.0	2.50	3010	7014	2.0	0	0	3	3010	0	2009	0	759 Ilwaco Pl NE	Renton	WA 98059	USA
4598	2014-07-10 00:00:00	2.034000e+05	4.0	2.00	2090	6630	1.0	0	0	3	1070	1020	1974	0	5148 S Creston St	Seattle	WA 98178	USA
4599	2014-07-10 00:00:00	2.206000e+05	3.0	2.50	1490	8102	2.0	0	0	4	1490	0	1990	0	18717 SE 258th St	Covington	WA 98042	USA

4600 rows × 18 columns

Let's explore some pandas functions:

.head()

A dataset consists of several rows and columns, which is hard to see all at once. So in this case, using the .head() function will return the first 5 rows of the dataset by default.

df.head()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country
0	2014-05-02 00:00:00	313000.0	3.0	1.50	1340	7912	1.5	0	3	1340	0	1955	2005	18810 Densmore Ave N	Shoreline	WA 98133	USA
1	2014-05-02 00:00:00	2384000.0	5.0	2.50	3650	9050	2.0	4	5	3370	280	1921	0	709 W Blaine St	Seattle	WA 98119	USA
2	2014-05-02 00:00:00	342000.0	3.0	2.00	1930	11947	1.0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	Kent	WA 98042	USA
3	2014-05-02 00:00:00	420000.0	3.0	2.25	2000	8030	1.0	0	4	1000	1000	1963	0	857 170th Pl NE	Bellevue	WA 98008	USA
4	2014-05-02 00:00:00	550000.0	4.0	2.50	1940	10500	1.0	0	4	1140	800	1976	1992	9105 170th Ave NE	Redmond	WA 98052	USA

In case you want to see the first 7 or 10 rows of the dataset, you pass in the number of rows you want to see into the bracket like this:

# to return the first 7 rows of the dataset
df.head(7)

# to return the first 10 rows of the dataset
# df.head(10)

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country
0	2014-05-02 00:00:00	313000.0	3.0	1.50	1340	7912	1.5	0	3	1340	0	1955	2005	18810 Densmore Ave N	Shoreline	WA 98133	USA
1	2014-05-02 00:00:00	2384000.0	5.0	2.50	3650	9050	2.0	4	5	3370	280	1921	0	709 W Blaine St	Seattle	WA 98119	USA
2	2014-05-02 00:00:00	342000.0	3.0	2.00	1930	11947	1.0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	Kent	WA 98042	USA
3	2014-05-02 00:00:00	420000.0	3.0	2.25	2000	8030	1.0	0	4	1000	1000	1963	0	857 170th Pl NE	Bellevue	WA 98008	USA
4	2014-05-02 00:00:00	550000.0	4.0	2.50	1940	10500	1.0	0	4	1140	800	1976	1992	9105 170th Ave NE	Redmond	WA 98052	USA
5	2014-05-02 00:00:00	490000.0	2.0	1.00	880	6380	1.0	0	3	880	0	1938	1994	522 NE 88th St	Seattle	WA 98115	USA
6	2014-05-02 00:00:00	335000.0	2.0	2.00	1350	2560	1.0	0	3	1350	0	1976	0	2616 174th Ave NE	Redmond	WA 98052	USA

.tail()

The .tail() function returns the last 5 rows of the dataset by default. You can also pass in the specific number of rows that you want to the function to return as i earlier stated. In this case, you say .tail(8) or .tail(3). Calling these functions will return the last 8 rows of the dataset and the last 3 rows of the dataset. Let's run these codes, shall we?

# to return the last 5 rows of the dataset(by default)
df.tail()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country
4595	2014-07-09 00:00:00	308166.666667	3.0	1.75	1510	6360	1.0	4	1510	0	1954	1979	501 N 143rd St	Seattle	WA 98133	USA
4596	2014-07-09 00:00:00	534333.333333	3.0	2.50	1460	7573	2.0	3	1460	0	1983	2009	14855 SE 10th Pl	Bellevue	WA 98007	USA
4597	2014-07-09 00:00:00	416904.166667	3.0	2.50	3010	7014	2.0	3	3010	0	2009	0	759 Ilwaco Pl NE	Renton	WA 98059	USA
4598	2014-07-10 00:00:00	203400.000000	4.0	2.00	2090	6630	1.0	3	1070	1020	1974	0	5148 S Creston St	Seattle	WA 98178	USA
4599	2014-07-10 00:00:00	220600.000000	3.0	2.50	1490	8102	2.0	4	1490	0	1990	0	18717 SE 258th St	Covington	WA 98042	USA

# to return the last 8 rows of the dataset
df.tail(8)

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country
4592	2014-07-08 00:00:00	252980.000000	4.0	2.50	2530	8169	2.0	3	2530	0	1993	0	37654 18th Pl S	Federal Way	WA 98003	USA
4593	2014-07-08 00:00:00	289373.307692	3.0	2.50	2538	4600	2.0	3	2538	0	2013	1923	5703 Charlotte Ave SE	Auburn	WA 98092	USA
4594	2014-07-09 00:00:00	210614.285714	3.0	2.50	1610	7223	2.0	3	1610	0	1994	0	26306 127th Ave SE	Kent	WA 98030	USA
4595	2014-07-09 00:00:00	308166.666667	3.0	1.75	1510	6360	1.0	4	1510	0	1954	1979	501 N 143rd St	Seattle	WA 98133	USA
4596	2014-07-09 00:00:00	534333.333333	3.0	2.50	1460	7573	2.0	3	1460	0	1983	2009	14855 SE 10th Pl	Bellevue	WA 98007	USA
4597	2014-07-09 00:00:00	416904.166667	3.0	2.50	3010	7014	2.0	3	3010	0	2009	0	759 Ilwaco Pl NE	Renton	WA 98059	USA
4598	2014-07-10 00:00:00	203400.000000	4.0	2.00	2090	6630	1.0	3	1070	1020	1974	0	5148 S Creston St	Seattle	WA 98178	USA
4599	2014-07-10 00:00:00	220600.000000	3.0	2.50	1490	8102	2.0	4	1490	0	1990	0	18717 SE 258th St	Covington	WA 98042	USA

# to return the last 3 rows of the dataset
df.tail(3)

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	condition	sqft_above	sqft_basement	yr_built	street	city	statezip	country
4597	2014-07-09 00:00:00	416904.166667	3.0	2.5	3010	7014	2.0	3	3010	0	2009	759 Ilwaco Pl NE	Renton	WA 98059	USA
4598	2014-07-10 00:00:00	203400.000000	4.0	2.0	2090	6630	1.0	3	1070	1020	1974	5148 S Creston St	Seattle	WA 98178	USA
4599	2014-07-10 00:00:00	220600.000000	3.0	2.5	1490	8102	2.0	4	1490	0	1990	18717 SE 258th St	Covington	WA 98042	USA

.shape

We use the .shape attribute to check how large the dataset is. This function will return the number of rows and column in the dataset.

df.shape

Output:

(4600, 18)

Here we can see that the dataset has 4600 rows and 18 columns.

Note: The index of the dataset always starts with 0 by default and not 1. If the index starts from 1, the index number of the last row will be 4600. So since the index of the dataset starts from 0, the index of the last row will be 4599. You can go through the .tail() to check it.

.info()

The .info() function displays basic information about the dataset including the datatype, columns, non-null values, number of rows and columns and the memory usage.

df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4600 entries, 0 to 4599
Data columns (total 18 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   date           4600 non-null   object 
 1   price          4600 non-null   float64
 2   bedrooms       4600 non-null   float64
 3   bathrooms      4600 non-null   float64
 4   sqft_living    4600 non-null   int64  
 5   sqft_lot       4600 non-null   int64  
 6   floors         4600 non-null   float64
 7   waterfront     4600 non-null   int64  
 8   view           4600 non-null   int64  
 9   condition      4600 non-null   int64  
 10  sqft_above     4600 non-null   int64  
 11  sqft_basement  4600 non-null   int64  
 12  yr_built       4600 non-null   int64  
 13  yr_renovated   4600 non-null   int64  
 14  street         4600 non-null   object 
 15  city           4600 non-null   object 
 16  statezip       4600 non-null   object 
 17  country        4600 non-null   object 
dtypes: float64(4), int64(9), object(5)
memory usage: 647.0+ KB

.describe()

The .describe() function returns the descriptive statistics of the dataframe.

df.describe()

Output:

	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	waterfront	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated
count	4.600000e+03	4600.000000	4600.000000	4600.000000	4.600000e+03	4600.000000	4600.000000	4600.000000	4600.000000	4600.000000	4600.000000	4600.000000	4600.000000
mean	5.519630e+05	3.400870	2.160815	2139.346957	1.485252e+04	1.512065	0.007174	0.240652	3.451739	1827.265435	312.081522	1970.786304	808.608261
std	5.638347e+05	0.908848	0.783781	963.206916	3.588444e+04	0.538288	0.084404	0.778405	0.677230	862.168977	464.137228	29.731848	979.414536
min	0.000000e+00	0.000000	0.000000	370.000000	6.380000e+02	1.000000	0.000000	0.000000	1.000000	370.000000	0.000000	1900.000000	0.000000
25%	3.228750e+05	3.000000	1.750000	1460.000000	5.000750e+03	1.000000	0.000000	0.000000	3.000000	1190.000000	0.000000	1951.000000	0.000000
50%	4.609435e+05	3.000000	2.250000	1980.000000	7.683000e+03	1.500000	0.000000	0.000000	3.000000	1590.000000	0.000000	1976.000000	0.000000
75%	6.549625e+05	4.000000	2.500000	2620.000000	1.100125e+04	2.000000	0.000000	0.000000	4.000000	2300.000000	610.000000	1997.000000	1999.000000
max	2.659000e+07	9.000000	8.000000	13540.000000	1.074218e+06	3.500000	1.000000	4.000000	5.000000	9410.000000	4820.000000	2014.000000	2014.000000

Where;

• count - is the total number of non-missing values

• mean - is the average of the dataframe

• std - is the standard deviation

• min - is the minimum value

• 25% - is the 25th percentile

• 50% - is the 50th percentile

• 75% - is the 75th percentile

• max - is the maximum value

.isnull()

This is an important function used to check for null values in a dataset.

# this function will find null values in the first 5 rows of the dataset
df.isnull().head()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	waterfront	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country
0	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False
1	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False
2	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False
3	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False
4	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False	False

To count the number of null or missing values in the dataset, we do this:

df.isnull().sum()

Output:

date             0
price            0
bedrooms         0
bathrooms        0
sqft_living      0
sqft_lot         0
floors           0
waterfront       0
view             0
condition        0
sqft_above       0
sqft_basement    0
yr_built         0
yr_renovated     0
street           0
city             0
statezip         0
country          0
dtype: int64

We can see that there are no null values because their individual sum is 0.

.columns

This function is used to view the column names.

df.columns

Output:

Index(['date', 'price', 'bedrooms', 'bathrooms', 'sqft_living', 'sqft_lot',
       'floors', 'waterfront', 'view', 'condition', 'sqft_above',
       'sqft_basement', 'yr_built', 'yr_renovated', 'street', 'city',
       'statezip', 'country'],
      dtype='object')

Selecting a column

To select a specfic column from the dataframe, insert the name of the column into square brackets [].

# return the first five values of the 'date' column in the dataframe
df[["date"]].head()

Output:

	date
0	2014-05-02 00:00:00
1	2014-05-02 00:00:00
2	2014-05-02 00:00:00
3	2014-05-02 00:00:00
4	2014-05-02 00:00:00

Note: To select a column, you can decide to call the name of the column to be returned as a dataframe object or a series object. This is usually done by either using double brackets or a single bracket. Hence the reason for using double brackets the example above.

To call a variable or the specific name of a column as a Series object, you pass the name of the column into a single bracket by doing this;

df['date'].head()

Output:

0    2014-05-02 00:00:00
1    2014-05-02 00:00:00
2    2014-05-02 00:00:00
3    2014-05-02 00:00:00
4    2014-05-02 00:00:00
Name: date, dtype: object

Did you notice the difference in the output of the two functions? The first output with double square brackets is a DataFrame object while this example gave an output as Series

Grouping data

Grouping data or columns is done by using the groupby function. Example:

# group the dataset by the number of bedrooms and calculate the average prize
grouped = df.groupby('bedrooms')['price'].mean()
grouped

Output:

bedrooms
0.0    1.195324e+06
1.0    2.740763e+05
2.0    3.916219e+05
3.0    4.886130e+05
4.0    6.351194e+05
5.0    7.701860e+05
6.0    8.173628e+05
7.0    1.049429e+06
8.0    1.155000e+06
9.0    5.999990e+05
Name: price, dtype: float64

Adding Rows and Columns

You can create new rows and columns to your dataset. However, when adding a new column to your dataset, you want to choose a column that can provide additional insights or useful information for analysis.
Let's add a new column called Price per Square Foot.

df['Price_per_Square_Ft'] = df['price'] / df['sqft_living']
df.head()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country	Price_per_Square_Ft
0	2014-05-02 00:00:00	313000.0	3.0	1.50	1340	7912	1.5	0	3	1340	0	1955	2005	18810 Densmore Ave N	Shoreline	WA 98133	USA	233.582090
1	2014-05-02 00:00:00	2384000.0	5.0	2.50	3650	9050	2.0	4	5	3370	280	1921	0	709 W Blaine St	Seattle	WA 98119	USA	653.150685
2	2014-05-02 00:00:00	342000.0	3.0	2.00	1930	11947	1.0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	Kent	WA 98042	USA	177.202073
3	2014-05-02 00:00:00	420000.0	3.0	2.25	2000	8030	1.0	0	4	1000	1000	1963	0	857 170th Pl NE	Bellevue	WA 98008	USA	210.000000
4	2014-05-02 00:00:00	550000.0	4.0	2.50	1940	10500	1.0	0	4	1140	800	1976	1992	9105 170th Ave NE	Redmond	WA 98052	USA	283.505155

See that a new column called 'Price per Square Ft' has been added to the dataset. Let's select the column so that we can see it.

# Selecting the Price_per_Square_Ft column from the dataset
df[['Price_per_Square_Ft']]

Output:

	Price_per_Square_Ft
0	233.582090
1	653.150685
2	177.202073
3	210.000000
4	283.505155
...	...
4595	204.083885
4596	365.981735
4597	138.506368
4598	97.320574
4599	148.053691

4600 rows × 1 columns

df

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	waterfront	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country	Price_per_Square_Ft
0	2014-05-02 00:00:00	3.130000e+05	3.0	1.50	1340	7912	1.5	0	0	3	1340	0	1955	2005	18810 Densmore Ave N	Shoreline	WA 98133	USA	233.582090
1	2014-05-02 00:00:00	2.384000e+06	5.0	2.50	3650	9050	2.0	0	4	5	3370	280	1921	0	709 W Blaine St	Seattle	WA 98119	USA	653.150685
2	2014-05-02 00:00:00	3.420000e+05	3.0	2.00	1930	11947	1.0	0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	Kent	WA 98042	USA	177.202073
3	2014-05-02 00:00:00	4.200000e+05	3.0	2.25	2000	8030	1.0	0	0	4	1000	1000	1963	0	857 170th Pl NE	Bellevue	WA 98008	USA	210.000000
4	2014-05-02 00:00:00	5.500000e+05	4.0	2.50	1940	10500	1.0	0	0	4	1140	800	1976	1992	9105 170th Ave NE	Redmond	WA 98052	USA	283.505155
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
4595	2014-07-09 00:00:00	3.081667e+05	3.0	1.75	1510	6360	1.0	0	0	4	1510	0	1954	1979	501 N 143rd St	Seattle	WA 98133	USA	204.083885
4596	2014-07-09 00:00:00	5.343333e+05	3.0	2.50	1460	7573	2.0	0	0	3	1460	0	1983	2009	14855 SE 10th Pl	Bellevue	WA 98007	USA	365.981735
4597	2014-07-09 00:00:00	4.169042e+05	3.0	2.50	3010	7014	2.0	0	0	3	3010	0	2009	0	759 Ilwaco Pl NE	Renton	WA 98059	USA	138.506368
4598	2014-07-10 00:00:00	2.034000e+05	4.0	2.00	2090	6630	1.0	0	0	3	1070	1020	1974	0	5148 S Creston St	Seattle	WA 98178	USA	97.320574
4599	2014-07-10 00:00:00	2.206000e+05	3.0	2.50	1490	8102	2.0	0	0	4	1490	0	1990	0	18717 SE 258th St	Covington	WA 98042	USA	148.053691

4600 rows × 19 columns

Now to add a new row, do this;

# Create a dictionary with the new row data
new_row_data = {
    'date': '2014-07-10 00:00:00',
    'price': 350000.000000,
    'bedrooms': 3.0,
    'bathrooms': 2.5,
    'sqft_living': 1800,
    'sqft_lot': 8000,
    'floors': 2.0,
    'waterfront': 0,
    'view': 0,
    'condition': 3,
    'sqft_above': 1800,
    'sqft_basement': 0,
    'yr_built': 1990,
    'yr_renovated': 0,
    'street': '1234 New St',
    'city': 'Seattle',
    'statezip': 'WA 98105',
    'country': 'USA'
}

# Calculate the index for the new row
new_row_index = len(df)

# Assign the new row data to the DataFrame using .loc
df.loc[new_row_index] = new_row_data

# Display the updated DataFrame
df.tail()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country	Price_per_Square_Ft
4596	2014-07-09 00:00:00	534333.333333	3.0	2.5	1460	7573	2.0	3	1460	0	1983	2009	14855 SE 10th Pl	Bellevue	WA 98007	USA	365.981735
4597	2014-07-09 00:00:00	416904.166667	3.0	2.5	3010	7014	2.0	3	3010	0	2009	0	759 Ilwaco Pl NE	Renton	WA 98059	USA	138.506368
4598	2014-07-10 00:00:00	203400.000000	4.0	2.0	2090	6630	1.0	3	1070	1020	1974	0	5148 S Creston St	Seattle	WA 98178	USA	97.320574
4599	2014-07-10 00:00:00	220600.000000	3.0	2.5	1490	8102	2.0	4	1490	0	1990	0	18717 SE 258th St	Covington	WA 98042	USA	148.053691
4600	2014-07-10 00:00:00	350000.000000	3.0	2.5	1800	8000	2.0	3	1800	0	1990	0	1234 New St	Seattle	WA 98105	USA	NaN

The first line of code is creating a dictionary called new_row_data where the keys are the column names of your DataFrame and the values are the new data you want to add.

The second line of the code calculates the new row's index as the new row will be added at the end of the DataFrame.

In the third line of code, the .loc function is used to assign the new_row_data to the DataFrame at the calculated index.

And then lastly, we display the DataFrame using the df.tail() function as this will display the last five rows of the Dataframe.

.iloc and .loc

The .iloc() and .loc() functions are used in Pandas to access data in a DataFrame using different types of indexing.

.iloc() is for integer-based indexing - meaning you can specify the position of rows and columns using integer indices, while .loc() is for label-based indexing - meaning you can specify the names(labels) of the rows and columns.

Here's how to use .iloc():

Accessing a specific row based on its integer index.

# Accessing the row at index 0
df.iloc[0]

Output:

date                    2014-05-02 00:00:00
price                              313000.0
bedrooms                                3.0
bathrooms                               1.5
sqft_living                            1340
sqft_lot                               7912
floors                                  1.5
waterfront                                0
view                                      0
condition                                 3
sqft_above                             1340
sqft_basement                             0
yr_built                               1955
yr_renovated                           2005
street                 18810 Densmore Ave N
city                              Shoreline
statezip                           WA 98133
country                                 USA
Price_per_Square_Ft               233.58209
Name: 0, dtype: object

Accessing a specific column within a row:

# Access the 'price' column of the row at index 0
df.iloc[0, 1]

Output:

313000.0

Accessing multiple rows and columns using slices:

# Access rows at index 0 to 2(inclusive) and columns at index 0 to 4
df.iloc[0:3, 0:4]

Output:

	date	price	bedrooms	bathrooms
0	2014-05-02 00:00:00	313000.0	3.0	1.5
1	2014-05-02 00:00:00	2384000.0	5.0	2.5
2	2014-05-02 00:00:00	342000.0	3.0	2.0

Changing a value in a specific cell:

# Change the price in the first row to 350000
df.iloc[0, 1] = 350000
df.iloc[0]

Output:

date                    2014-05-02 00:00:00
price                              350000.0
bedrooms                                3.0
bathrooms                               1.5
sqft_living                            1340
sqft_lot                               7912
floors                                  1.5
waterfront                                0
view                                      0
condition                                 3
sqft_above                             1340
sqft_basement                             0
yr_built                               1955
yr_renovated                           2005
street                 18810 Densmore Ave N
city                              Shoreline
statezip                           WA 98133
country                                 USA
Price_per_Square_Ft               233.58209
Name: 0, dtype: object

Here's how to use .loc():

Accessing a specified row based on its label:

# Access the row with index 4500
df.loc[4500]

Output:

date                   2014-06-17 00:00:00
price                             540000.0
bedrooms                               3.0
bathrooms                             2.75
sqft_living                           2750
sqft_lot                             18029
floors                                 1.0
waterfront                               0
view                                     2
condition                                5
sqft_above                            1810
sqft_basement                          940
yr_built                              1978
yr_renovated                             0
street                    4708 154th Pl SE
city                              Bellevue
statezip                          WA 98006
country                                USA
Price_per_Square_Ft             196.363636
Name: 4500, dtype: object

Accessing a specified column within a row using the name of the column.

# Access the 'price' column of the row with index 5
df.loc[5, 'price']

Output:

490000.0

Accessing multiple rows and columns using label slices or lists.

# Access rows with indices 0 to 3 and columns 'price' and 'bedrooms'
df.loc[0:3, ['price', 'bedrooms']]

Output:

	price	bedrooms
0	350000.0	3.0
1	2384000.0	5.0
2	342000.0	3.0
3	420000.0	3.0

The key difference between .iloc() and .loc() is that, .loc() uses integer indices to access rows and columns, while .loc() uses labels(names).

Dropping Rows and Columns

Here's how to drop a row from your dataset:

# Specify the index of the row you want to drop
row_to_drop = 4

# Drop the column and return a new DataFrame
df_new = df.drop(row_to_drop)

# Display the new dataset
df_new.head()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	city	statezip	country	Price_per_Square_Ft
0	2014-05-02 00:00:00	350000.0	3.0	1.50	1340	7912	1.5	0	3	1340	0	1955	2005	18810 Densmore Ave N	Shoreline	WA 98133	USA	233.582090
1	2014-05-02 00:00:00	2384000.0	5.0	2.50	3650	9050	2.0	4	5	3370	280	1921	0	709 W Blaine St	Seattle	WA 98119	USA	653.150685
2	2014-05-02 00:00:00	342000.0	3.0	2.00	1930	11947	1.0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	Kent	WA 98042	USA	177.202073
3	2014-05-02 00:00:00	420000.0	3.0	2.25	2000	8030	1.0	0	4	1000	1000	1963	0	857 170th Pl NE	Bellevue	WA 98008	USA	210.000000
5	2014-05-02 00:00:00	490000.0	2.0	1.00	880	6380	1.0	0	3	880	0	1938	1994	522 NE 88th St	Seattle	WA 98115	USA	556.818182

You can see that row 4 is no longer in the dataset.

Here's how to drop a column from your dataset:

# Specify the name of the column you want to drop
column_to_drop = 'city'

# Drop the column and return a new Dataframe
df_new = df.drop(columns=[column_to_drop])

#Display the new DataFrame
df_new.head()

Output:

	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	view	condition	sqft_above	sqft_basement	yr_built	yr_renovated	street	statezip	country	Price_per_Square_Ft
0	2014-05-02 00:00:00	350000.0	3.0	1.50	1340	7912	1.5	0	3	1340	0	1955	2005	18810 Densmore Ave N	WA 98133	USA	233.582090
1	2014-05-02 00:00:00	2384000.0	5.0	2.50	3650	9050	2.0	4	5	3370	280	1921	0	709 W Blaine St	WA 98119	USA	653.150685
2	2014-05-02 00:00:00	342000.0	3.0	2.00	1930	11947	1.0	0	4	1930	0	1966	0	26206-26214 143rd Ave SE	WA 98042	USA	177.202073
3	2014-05-02 00:00:00	420000.0	3.0	2.25	2000	8030	1.0	0	4	1000	1000	1963	0	857 170th Pl NE	WA 98008	USA	210.000000
4	2014-05-02 00:00:00	550000.0	4.0	2.50	1940	10500	1.0	0	4	1140	800	1976	1992	9105 170th Ave NE	WA 98052	USA	283.505155

Now, the 'city' column is no longer present in the dataset.

Conclusion

Pandas is a crucial tool for data work in Python. We've demonstrated how Pandas can effectively help manipulate and analyze housing data. Whether you're a beginner or a seasoned professional, Pandas provides a variety of features to make your data analysis smoother.

For more information, check out the official Pandas documentation and online tutorials for advanced topics and features. Happy coding!

Understanding Supervised and Unsupervised Learning: A Beginners guide.

Ugonma Ononogbu — Sat, 15 Jun 2024 06:49:22 +0000

Every day, we interact with machine learning through smart assistants like Siri and Alexa, streaming services like Netflix and Spotify, search engines like Google, and our favorite social media platforms like Tiktok and Instagram. These technologies bring us closer, making our world smarter and more connected.

In this article, you will learn the fundamentals of machine learning--Supervised and unsupervised learning. We’ll discuss their types, real-world applications, advantages and disadvantages, and how they differ.

Machine Learning is a branch of Artificial Intelligence that enables computers to learn from and make predictions or decisions based on a given data without being programmed to do so.
In simpler terms, it is like teaching computers to learn and get better from experience, just like humans, but using lots of data and powerful algorithms.

Machine learning is widely categorized into two main types:

Supervised Learning
Unsupervised Learning

Each one uses different methods to train models depending on the kind of data.

Supervised Learning

In supervised learning, the model learns from a dataset that is labeled. This simply means that the model is taught using examples that have the correct answers. For instance, if you have a set of fruit images with their names labeled on them, the model learns to recognize the fruits from the labeled images. Later, when given new images, it can predict the fruit names based on what it has learned.

Types of Supervised Learning

1.Regression: This is a type of supervised learning algorithm used to predict continuous values.
Examples:

•House price predictions: Predicting the sales price of a house based on features like size, location, and number of bedrooms in the house.
•Forecasting Temperature: By forecasting future weather temperatures based on past weather records, a regression model can forecast the temperature for the next day or week.
•Predicting stock price: by analyzing past stock prices, trading volume, and other financial indicators, a regression model can attempt to predict the future price movements of a stock.

2.Classification: This is a type of supervised learning algorithm used to categorize data. It is like sorting objects into different groups based on their characteristics. For instance, you have a basket of fruits and you want to sort them into groups like apples, bananas, and oranges. The model learns that apples are red and round, while bananas are yellow and elongated, and then proceeds to group them accordingly. Similarly, in email spam detection, the model learns patterns in emails to know whether they are spam, based on the sender and other features of the mail.

Applications of Supervised Learning

• Email Spam Filtering: The supervised learning algorithm is trained on a dataset of emails to identify and classify emails that are spam or non-spam by learning to recognize patterns and features that distinguish the two.

• Speech Recognition: The model is trained on audio recordings to convert spoken language into text. The recordings have their spoken words written down with them. This helps the model learn how people speak and change what they say into written text.

• Customer Churn Predictions: The model can predict which customers are likely to stop using a service by analyzing their past behavior.

• Predictive Maintenance: The models learn from past machines' data to spot signs that the equipment might need fixing soon.

Supervised Learning Algorithms

Supervised learning algorithms teach computers to make predictions or decisions by learning from examples given to them. Here are some common examples:
•Linear Regression
•Logistic Regression
•Decision Trees
•Random Forests
•Support Vector Machines (SVM)
•k-Nearest Neighbors (k-NN)

Advantages of Supervised Learning

•It makes accurate predictions.
•The models use past data to predict what might happen in the future.
•The algorithms are easy to understand and interpret.
•You can easily spot when the model makes mistakes and correct them during the training process.
•The more labeled data you have, the better the model can learn and improve its accuracy.
•The algorithms can learn from large datasets, making them powerful tools for big data analysis.

Disadvantages of Supervised Learning

•Supervised learning requires labeled data.
•Training a supervised learning model can be time-consuming.
•The model can only predict the specific tasks they were trained on.
•If there are errors in the labeled data, the model will learn the errors, causing the model to make inaccurate predictions.
•Some algorithms are complex and difficult to interpret.

Unsupervised Learning

In Unsupervised learning, the model works with data that doesn't have any labels or correct answers. It figures out patterns and groups on its own.

For example, if you give the model a bunch of fruit pictures without telling it which fruit is which, the model will find similarities and differences among the pictures and group the fruits accordingly. It doesn’t know the names, but it can still organize them based on their characteristics.

Types of Unsupervised Learning

• Clustering: Clustering is a type of unsupervised learning that groups data points based on their similarities.
Examples:
-K-Means Clustering
-Hierarchical Clustering
-Independent Component Analysis
-Density-Based Spatial Clustering of Applications with Noise(DBSCAN)

• Dimensionality Reduction: This technique simplifies complex data while keeping important informations. Examples:
-Principal Component Analysis
-Autoencoders

• Association Rule Learning: This type of unsupervised learning finds patterns and relationships between items in data. Examples:
-Apriori Algorithm
-Eclat Algorithm

Applications of Unsupervised Learning

•Customer Segmentation: The algorithm looks at customer data e.g. purchase history, website activity, etc., and groups customers into different categories based on their behaviors and preferences.

•Imagine Compression: The algorithm will identify the most important parts of an image and compress it while retaining important information.

•Recommendation Systems: Unsupervised learning can suggest products, movies, or music based on user behavior.

•Market Basket Analysis: The algorithm analyzes shopping data to find products that are frequently bought together.

Advantages of Unsupervised Learning

•It does not require labeled data.
•It can identify hidden patterns in data.
•It is useful in fraud detection.
•It is useful in exploratory data analysis.

Disadvantages of Unsupervised Learning

•It is hard to determine the accuracy of the model without labels.
•The interpretations are difficult to understand.
•It requires the knowledge of experts to choose the right algorithm and interpret results.

Differences between Supervised and Unsupervised Learning

Aspect	Supervised Learning	Unsupervised Learning
Definition	Involves training a model with labeled data	Involves training a model with unlabeled data
Objective	Makes accurate predictions	Finds hidden patterns or structures
Examples of Algorithms	Algorithms like Linear Regression, Decision Trees, SVM	Algorithms like k-Means Clustering, Hierarchical Clustering
Applications	Used for tasks like spam detection, fraud detection	Used for tasks like customer segmentation, image compression

Conclusion

Supervised and unsupervised learning are important techniques in machine learning, each with its own strengths and weaknesses. While supervised learning needs a lot of labeled data and can sometimes make mistakes, it is very accurate. On the other hand, Unsupervised learning does not need labeled data but its results can be hard to understand.

Knowing when to use each, method helps in solving different types of problems effectively, making the most out of machine learning.