DEV Community: arju10

Pandas for Machine Learning & Deep Learning

arju10 — Tue, 26 Nov 2024 10:28:16 +0000

Pandas

Grouping & Aggregation

Grouping data frame using aggregation functions, including sum and mean, calculated per category.

## Numpy
import pandas as pd

# Create a Data Frame
data = {'Category':['A', 'B', 'C', 'A','B','C','C','A'],
        'Value':[10, 20, 40, 45, 50, 90,100, 200]}

df = pd.DataFrame(data)


# Grouping by 'category' and calculating the sum and mean
grouped = df.groupby('Category').agg({'Value': ['sum', 'mean']})

print("Grouped Data Frame with Aggregation: \n\n", grouped)

Output:

Grouped Data Frame with Aggregation:

Category	Value (sum)	Value (mean)
A	255	85.000000
B	70	35.000000
C	230	76.666667

Here,the data is grouped by Category, and for each category, we calculate:

Sum of the Value column
Mean of the Value column

Explanation of Columns:

Category: The category label for each group.
Value (sum): The total sum of values within each category.
Value (mean): The average value within each category.

Grouping & Aggregation

Grouping data frame using aggregation functions, including sum, max and mean, calculated per category.

import pandas as pd

# Create dataframe
data = {
    'Category':['A','A','B','B'],
    'Values':[10, 20, 30,40]
}
df = pd.DataFrame(data)

# Grouping by 'category' and calculating the sum and mean
grouped_df = df.groupby('Category').agg({'Values': ['sum', 'mean','max']})

print("Grouped Data Frame with with multiple functions \n\n", grouped_df)

Output:

Custom Aggregation Result:

Category	sum	mean	max
A	30	15.0	20
B	70	35.0	40

MutiIndex DataFrame Operations

import pandas as pd

# Create Multi Index Dataframe
arrays = [['A', 'A', 'B', 'B'], [2020, 2021, 2022, 2023]]
index = pd.MultiIndex.from_arrays(arrays, names=('Category', 'Year'))
data = [100, 150, 200, 250]
df = pd.DataFrame(data, index=index, columns=['Sales'])

# Accessing data in a mutiindex Dataframe
sales_in_2020 = df.xs(2020, level='Year')

print("Sales Data in 2020: ", sales_in_2020)

Output

Category	Sales (2020)
A	100

Here, output for sales data in 2020, showing the total sales for each category.

Using Apply with Lambda Functions

import pandas as pd

# Create Dataframe
data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Score':[85,90,95]}
df = pd.DataFrame(data)

# Applying a lambda function to modify scores
df['Adjusted Score'] = df['Score'].apply(lambda x: x+5 if x<90 else x)

print("Dataframe with Adjusted Scores :\n",df)

Output

Name	Score	Adjusted Score
Alice	85	90
Bob	90	90
Charlie	95	95

Here, this data frame includes a list of names, their original scores, and the adjusted scores:

Merging DataFrame with Different Keys

import pandas as pd

# Create two dataframe
df1 = pd.DataFrame({'ID':[1,2,3], 'Name':['Alice', 'Bob', 'Charlie']})
df2 = pd.DataFrame({'Emp_ID':[2,3,4], 'Department':['HR','Finance','IT']})

#Merging with different keys
merged_df = pd.merge(df1, df2, left_on='ID', right_on='Emp_ID', how='outer')

print("Merged Dataframe: \n", merged_df)

Output

ID	Name	Emp_ID	Department
1.0	Alice	NaN	NaN
2.0	Bob	2.0	HR
3.0	Charlie	3.0	Finance
NaN	NaN	4.0	IT

Here, the merged data frame combines two data sources, which may contain missing values

ID: Identifier from the first data source.
Name: Name from the first data source.
Emp_ID: Employee ID from the second data source.
Department: Department from the second data source.
NaN values indicate missing data from one of the sources.

Handling Missing Values with Custom Functions

import pandas as pd
import numpy as np

# Creating dataframe with missing values
data = {'Name':['Alice', 'Bob', np.nan, 'David'], 'Age':[13,np.nan,32,20]}
df = pd.DataFrame(data)

# Filling Missing Values using a custom function
df['Name'].fillna('Unknown', inplace=True)
df['Age'].fillna(df['Age'].median(), inplace=True)

print("Dataframe after handling Missing values: \n", df)

Output

Dataframe after handling Missing values:

Name	Age
Alice	13.0
Bob	20.0
Unknown	32.0
David	20.0

Here,missing values in the Name or Age columns were replaced with a default string ("Unknown") or a specified numeric value.

Handling Missing Values with `ffilll` & `bfill` method

import pandas as pd
import numpy as np

# Creating dataframe with missing values
data = {'Name':['Alice', 'Bob', np.nan, 'David'], 'Age':[13,np.nan,32,20]}
df = pd.DataFrame(data)

# Filling Missing Values using ffill & bfill
df_ffill = df.fillna(method = 'ffill')
df_bfill = df.fillna(method = 'bfill')

print("Dataframe with forward fill: \n", df_ffill)
print("Dataframe with backwark fill: \n", df_bfill)

Output
DataFrame with Forward Fill

Forward fill propagates the last valid non-null value forward until the next non-null value is encountered.

Name	Age
Alice	13.0
Bob	13.0
Bob	32.0
David	20.0

DataFrame with Backward Fill

Backward fill propagates the next valid non-null value backward until the previous non-null value is encountered.

Name	Age
Alice	13.0
Bob	32.0
David	32.0
David	20.0

Pivoting DataFrame

import pandas as pd

# Create dataframe
data = {'Date':['2023-01-01', '2023-01-02','2023-01-01','2023-01-02'],
        'City':['NY', 'NY','LA', 'LA'],
        'Sales':[200,250,300,400]
        }

df = pd.DataFrame(data)

# Pivoting the Dataframe
pivot_df = df.pivot(index='Date', columns='City', values='Sales')

print('Pivoted Dataframe : \n', pivot_df)

Output

Pivoted Dataframe :

Date	LA	NY
2023-01-01	300	200
2023-01-02	400	250

Pivoting DataFrame with multi index columns

import pandas as pd

# Create dataframe
data = {'Date':['2023-01-01', '2023-01-02','2023-01-01','2023-01-02'],
        'City':['NY', 'NY','LA', 'LA'],
        'Type': ['Online', 'Store', 'Online','Store'],
        'Sales':[200,250,300,400]
        }

df = pd.DataFrame(data)

# Pivoting to create a Dataframe with multi-index columns
pivot_df = df.pivot(index='Date', columns=['City','Type'], values='Sales')

print('Pivoted Dataframe with multi-index columns: \n', pivot_df)

Output

Pivoted DataFrame with Multi-Index Columns:

Date	City	NY		LA
	Type	Online	Store	Online	Store
2023-01-01		200.0	NaN	300.0	NaN
2023-01-02		NaN	250.0	NaN	400.0

Melting DataFrame

import pandas as pd

# Create dataframe
data = {'ID':[1,2], 'Math':[90,79], 'Science':[85,88]}
df = pd.DataFrame(data)

# Melting the dataframe
melted_dataframe = pd.melt(df,id_vars=['ID'], value_vars=['Math', 'Science'], var_name='Subject', value_name='Score')

print("Melted Dataframe : \n", melted_dataframe)

Output

Melted DataFrame :

ID	Subject	Score
1	Math	90
2	Math	79
1	Science	85
2	Science	88

Time based indexing and resampling

import pandas as pd

# Create time Series Dataframe
date_range = pd.date_range(start='2023-01-01', periods=10, freq='D')
data = {'Sales':[100, 200, 150, 300, 250, 400, 300, 350, 300, 400]}
df = pd.DataFrame(data, index=date_range)

# Resampling to find weekly sales
weekly_sales = df.resample('W').sum()

print("Weekly Sales : \n", weekly_sales)

Output

Weekly Sales :

Date	Sales
2023-01-01	100
2023-01-08	1950
2023-01-15	700

Resampling with different aggregation

import pandas as pd

# Create time Series Dataframe
date_range = pd.date_range(start='2023-01-01', periods=10, freq='D')
data = {'Sales':[100, 200, 150, 300, 250, 400, 300, 350, 300, 400]}
df = pd.DataFrame(data, index=date_range)

# Resampling to find weekly sales
resampled_df = df.resample('W').agg({'Sales':['sum', 'mean', 'max']})

print("Resampled Dataframe with different aggregation : \n", resampled_df)

Output

Resampled DataFrame with Different Aggregation:

Date	Sales (sum)	Sales (mean)	Sales (max)
2023-01-01	100	100.000000	100
2023-01-08	1950	278.571429	400
2023-01-15	700	350.000000	400

Conditional Filtering with Multiple Conditions

import pandas as pd

# Create dataframe
data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'], 'Score':[85, 80,40,70], 'Passed':[True, False, True, False]}
df = pd.DataFrame(data)

# Filtering with multiple conditions
filtered_df = df[(df['Score'] > 80) & (df['Passed'] == True)]

print("FIltered Dataframe : \n", filtered_df)

Output

Example Filtered DataFrame :

Name	Score	Passed
Alice	85	True

Creating Custom Categorical Data

import pandas as pd

# Create dataframe
data = {'Name':['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85, 70, 95,60]}
df = pd.DataFrame(data)

# Creating a new column with categorical data
df['Performance'] = pd.cut(df['Score'], bins=[0,70,90,100], labels=['Poor', 'Average', 'Excellent'])

print('Dataframe with categorical performance : \n', df)

Output
DataFrame with Categorical Performance :

Name	Score	Performance
Alice	85	Average
Bob	70	Poor
Charlie	95	Excellent
David	60	Poor

Binning Data using `pd.cut`

import pandas as pd

# Create dataframe
data = {'Score': [85, 70, 95,60, 100, 90]}
df = pd.DataFrame(data)

# Binning the scores into categories
df['Grade'] = pd.cut(df['Score'], bins=[0,59,69,79,89,100], labels=['F', 'D', 'C', 'B','A'])

print('Dataframe with binned scores : \n', df)

Output
Dataframe with binned scores:

Score	Grade
85	B
70	C
95	A
60	D
100	A
90	A

Rolling Window Calculations

import pandas as pd

# Create a time series dataframe
data_range = pd.date_range(start='2023-01-01', periods=10, freq='D')
data = {'Sales': [100, 200, 150, 300, 250, 400, 300, 350, 300, 400]}
df = pd.DataFrame(data, index= data_range)

# Calculating a rolling mean with a window of 3 days
df['Rolling Mean'] = df['Sales'].rolling(window=3).mean()

print("Dataframe with rolling mean : \n", df)

Output

DataFrame with Rolling Mean :

Date	Sales	Rolling Mean
2023-01-01	100	NaN
2023-01-02	200	NaN
2023-01-03	150	150.000000
2023-01-04	300	216.666667
2023-01-05	250	233.333333
2023-01-06	400	316.666667
2023-01-07	300	316.666667
2023-01-08	350	350.000000
2023-01-09	300	316.666667
2023-01-10	400	350.000000

Shift and Lagging Data

import pandas as pd

# Create Dataframe
data = {'Date': pd.date_range(start='2023-01-01', periods=5, freq='D'),
        'Temperature': [30, 32, 35,33, 31]}

df = pd.DataFrame(data)

# Shifting data by one to create a lag
df['Prev Day Temp'] = df['Temperature'].shift(1)

print('Dataframe with shifted data: \n', df)

Output
DataFrame with Previous Day Temperature:

Date	Temperature	Prev Day Temp
2023-01-01	30	NaN
2023-01-02	32	30.0
2023-01-03	35	32.0
2023-01-04	33	35.0
2023-01-05	31	33.0

Cumulative sum and product

import pandas as pd

# Create dataframe
data = {'Sales': [100, 200, 150, 300, 250]}
df = pd.DataFrame(data)

# Calculating cumulative sum and product
df['cumulative sum'] = df['Sales'].cumsum()
df['cumulative product'] = df['Sales'].cumprod()

print('DataFrame with cumulative operations:\n', df)

Output
DataFrame with Cumulative Operations:

Sales	Cumulative Sum	Cumulative Product
100	100	100
200	300	20000
150	450	3000000
300	750	900000000
250	1000	225000000000

Merging on multiple columns

import pandas as pd

# Create two dataframe
df1 = pd.DataFrame({'ID':[1,2,3], 'Year':[2020,2021,2023], 'Name':['Alice','Bob','Charlie'] })
df2 = pd.DataFrame({'ID':[2,3,1], 'Year':[2023,2023,2020], 'Score':[88,92,75]})

# Merging on multiple columns
merged_df = pd.merge(df1,df2, on=['ID','Year'], how='inner')

print('Merged Dataframe on multiple columns : \n', merged_df)

Output
Merged DataFrame on Multiple Columns:

ID	Year	Name	Score
1	2020	Alice	75
3	2023	Charlie	92

Handling Outliers

import pandas as pd

# Create dataframe
data = {'Value': [10, 12, 14, 100, 15, 13, 12]}
df = pd.DataFrame(data)

# Indentifying outliers using the IQR method
Q1 = df['Value'].quantile(0.25)
Q3 = df['Value'].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df['Value'] < (Q1 - 1.5 * IQR)) | (df['Value'] > (Q3 + 1.5 * IQR))]

print('Outliers: \n', outliers)

Output
DataFrame with Outliers:

Value
10
15
20
100

Here, the value 100 is an outlier compared to the other values.

Creating a pivot table with multiple aggregation

import pandas as pd

# Create dataframe
data = {'City':['NY', 'LA', 'NY','SF','LA'], 'Year':[2020,2020,2021,2021,2020], 'Sales':[100, 150, 200, 250,300]}
df = pd.DataFrame(data)

# Create a pivot table with multiple aggregation
pivot_table = pd.pivot_table(df, values='Sales', index='City', columns='Year', aggfunc=['sum', 'mean'])

print('Pivot table with multiple aggregation: \n', pivot_table)

Output
Pivot Table with Multiple Aggregation :

City	Year	Sum	Mean
LA	2020	450.0	225.0
NY	2020	100.0	100.0
NY	2021	200.0	200.0
SF	2021	250.0	250.0

Here, a pivot table is created with sum and mean as the aggregation functions.

Creating a pivot table with multiple indexes

import pandas as pd

# Create dataframe
data = {'City':['NY', 'LA', 'NY','SF','LA'], 'Year':[2020,2020,2021,2021,2020], 'Sales':[100, 150, 200, 250,300]}
df = pd.DataFrame(data)

# Create a pivot table with multiple indexes
pivot_table = pd.pivot_table(df, values='Sales', index=['City', 'Year'], aggfunc='sum')

print('Pivot table with multiple indexes: \n', pivot_table)

Output
Pivot Table with Multiple Indexes :

City	Year	Sales
LA	2020	450
NY	2020	100
	2021	200
SF	2021	250

Using `map()` for Value Replacement

import pandas as pd

# Create dataframe
data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Department':['HR', 'IT', 'Finance']}
df = pd.DataFrame(data)

# Mapping department to codes
department_map = {'HR':1, 'IT':2, 'Finance':3}
df['Dept Code'] = df['Department'].map(department_map)

print("Dataframe with mapped values: \n", df)

Output
DataFrame with Mapped Values:

Name	Department	Dept Code
Alice	HR	1
Bob	IT	2
Charlie	Finance	3

Here, the department names are mapped to department codes.

Detecting Duplicates

iimport pandas as pd

# Create a dataframe with duplicate rows
data = {'ID':[1,2,2,3,4,4], 'Value': [10, 20,20,30,40,40]}
df = pd.DataFrame(data)

# Detecting duplicates
duplicates = df[df.duplicated()]

print('Duplicate Row : \n', duplicates)

Output
DataFrame with Duplicate Rows:

ID	Value
2	20
4	40
2	20
4	40

In this example, the rows with ID 2 and 4 are duplicates.

Using `explode()` for list in columns

import pandas as pd

# Create a dataframe with lists in a column
data = {'ID':[1,2], 'Hobbies':[['Reading', 'Swimming'], ['Running', 'Cycling'] ]}
df = pd.DataFrame(data)

# Explodeing the 'Hobbies' Column
exploded_df = df.explode('Hobbies')

print("Dataframe after exploring lists: \n", exploded_df)

Output
DataFrame After Exploring Lists:

ID	Hobbies
1	Reading
1	Swimming
2	Running
2	Cycling

In this example, each hobby in a list associated with an ID is expanded into its own row.

Using `rank()` to rank values

import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85,90,78,92]}
df = pd.DataFrame(data)

# Ranking scores in descending order
df['Rank'] = df['Score'].rank(ascending=False)

print('Dataframe with ranked scores : \n', df)

Output
DataFrame with Ranked Scores:

Name	Score	Rank
Alice	85	3.0
Bob	90	2.0
Charlie	78	4.0
David	92	1.0

In this example, the scores are ranked in descending order, with the highest score receiving the highest rank.

Using `applymap()` for element-wise operations

import pandas as pd

# Create dataframe
data = {'A': [1,2,3], 'B':[4,5,6]}
df = pd.DataFrame(data)

# Applying an operation to each element in the Dataframe
df_transformed = df.applymap(lambda x:x**2)

print("Dataframe with squared values : \n", df_transformed)

Output
DataFrame with Ranked Scores:

Name	Score	Rank
Alice	85	3.0
Bob	90	2.0
Charlie	78	4.0
David	92	1.0

In this example, the scores are ranked in descending order, with the highest score receiving the highest rank.

Using `df.apply()` with axis parameter

import pandas as pd

# Create dataframe
data = {'A': [1,2,3], 'B':[4,5,6]}
df = pd.DataFrame(data)

# Applying a function to rows and columns using the axis parameter
row_sum = df.apply(lambda x: x.sum() , axis=1)
col_sum = df.apply(lambda x: x.sum() , axis=0)

print("Row wise sum : \n", row_sum)
print("Column wise sum : \n", col_sum)

Output
Row-wise Sum:

	Sum
0	5
1	7
2	9

Column-wise Sum:

	Sum
A	6
B	15

Create dummy variables

import pandas as pd

# Create dataframe
data = {'City': ['NY', 'LA', 'SF', 'NY']}
df = pd.DataFrame(data)

# Creating dummy varibales
dummies = pd.get_dummies(df['City'], prefix='City')

print('Dummy varibales dataframe: \n', dummies)

Output
DataFrame with Squared Values :

A	B
1	16
4	25
9	36

In this example, each value in columns A and B has been squared.

Use `query()` for filtering

import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85,90,78,92]}
df = pd.DataFrame(data)

# Use query for filtering
filtered_df = df.query('Score > 80')

print('Filtered dataframe using query : \n', filtered_df)

Output
Filtered DataFrame :

Name	Score
Alice	85
Bob	90
David	92

Here, only rows with scores above a certain threshold are included in the filtered DataFrame.

Adding a new column using `assign()`

import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85,90,78,92]}
df = pd.DataFrame(data)

# Adding a new column using assign
df = df.assign(Grade= lambda x: ['A' if Score >= 90 else 'B' for Score in x['Score']])

print("Dataframe with assigned grade column : \n", df)

Output
DataFrame with Assigned Grade Column :

Name	Score	Grade
Alice	85	B
Bob	90	A
Charlie	78	B
David	92	A

In this example, grades are assigned based on the score values: 'A'for high scores, 'B' for average scores, etc.

Applying a custom function with `groupby`

import pandas as pd

# Create dataframe
data = {
    'Category':['A','A','B','B'],
    'Value':[10, 20, 30,40]
}

df = pd.DataFrame(data)

# Applying a custom function with groupby
grouped_df = df.groupby('Category') ['Value'].apply(lambda x: x.max() - x.min())

print("Custom Aggregation result : \n", grouped_df)

Output
Custom Aggregation Result:

Category	Value
A	10
B	10

In this example, the Value column is aggregated using a custom function (such as summing specific elements) for each category.

Applying functions to grouped data

import pandas as pd

# Create dataframe
data = {
    'Team':['A','A','B','B'],
    'Points':[10, 20, 30,40]
}
df = pd.DataFrame(data)

# Applying a function to grouped data
grouped_df = df.groupby('Team') ['Points'].transform(lambda x: x/x.max() )

print("Dataframe with applied functions to grouped data : \n", grouped_df)

Output
Dataframe with Applied Functions to Grouped Data:

	Points
0	0.50
1	1.00
2	0.75
3	1.00

DataFrame info and memory usage

import pandas as pd

# Create dataframe
data = {
    'A': range(1000),
    'B':range(1000, 2000)
}
df = pd.DataFrame(data)

# Displaying dataframe info and memory usage
df_info = df.info(memory_usage='deep')

print('Dataframe info and Memory usage : \n', df_info)

Output
DataFrame Info and Memory Usage :

#	Column	Non-Null Count	Dtype
0	A	1000 non-null	int64
1	B	1000 non-null	int64

Dtypes: int64 (2 columns)
Memory Usage: 15.8 KB

Filter with `isin()`

import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob', 'Charlie', 'David'], 'City': ['NY','LA','SF','NY']}
df = pd.DataFrame(data)

# filter with isin()
filtered_df = df[df['City'].isin(['NY','SF'])]

print("Filtered Dataframe with isin method: \n", filtered_df)

Output
Filtered Dataframe with isin method:

	Name	City
0	Alice	NY
2	Charlie	SF
3	David	NY

This section demonstrates how to filter rows in a DataFrame based on a list of values using the isin method.

Concatenating Dataframe using `concat()`

import pandas as pd

# Create dataframes
df1 = pd.DataFrame({
    'A':[1,2],
    'B':[3,4]
})

df2 = pd.DataFrame({
    'A': [5,6],
    'B': [7,8]
})

# Concatening Dataframes
concat_df = pd.concat([df1, df2], ignore_index=True)

print("COncatenated DataFrame : \n", concat_df)

Output
Concatenated DataFrame :

	A	B
0	1	3
1	2	4
2	5	7
3	6	8

This section demonstrates how to concatenate two DataFrames vertically.

Sorting a DataFrame by Multiple Columns

import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85,90,78,92], 'Age': [25,37,35,21]}
df = pd.DataFrame(data)

# Sorting by multiple columns
sorted_df = df.sort_values(by=['Score', 'Age'], ascending=[False, True])

print("Sorted Dataframe by multiple columns: \n", sorted_df)

Output
Sorting a DataFrame by Multiple Columns :

Index	Name	Score	Age
3	David	92	21
1	Bob	90	37
0	Alice	85	25
2	Charlie	78	35

Primary Sorting by Score in descending order.
Secondary Sorting by Age in ascending order.

Displaying the styled Dataframe

import pandas as pd

# Create dataframe
data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85, 90, 78, 92]}
df = pd.DataFrame(data)

# Applying styles to highlight scores above 80
styled_df = df.style.map(
    lambda x: 'background-color: yellow' if isinstance(x, int) and x > 80 else ''
)

# Save styled DataFrame to HTML
styled_df.to_html("styled_dataframe.html")

# Displaying the styled Dataframe
print("Styled dataframe with conditional formatting has been saved to 'styled_dataframe.html'.")

Output
"Styled dataframe with conditional formatting has been saved to 'styled_dataframe.html'."

Index	Name	Score
0	Alice	85
1	Bob	90
2	Charlie	78
3	David	92

Note: Cells with scores greater than 80 are highlighted in yellow. To view the styling, check the styled_dataframe.html file in a browser.

Using `explode()` for nested JSON columns

import pandas as pd

# Create a dataframe with nested lists
data = {
    'ID' : [1,2],
    'Hobbies':[['Reading', 'Swimming', 'Gaming'], ['Hiking', 'Drawing']]
}
df = pd.DataFrame(data)

# Exploding the 'Hobbies' column
exploded_df = df.explode('Hobbies')

print("Dataframe after exploding nested json columns:\n", exploded_df)

Output
DataFrame after Exploding Nested JSON Columns

Index	ID	Hobbies
0	1	Reading
0	1	Swimming
0	1	Gaming
1	2	Hiking
1	2	Drawing

Using `pipe()` for method chaining

import pandas as pd

# Create dataframe
data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85, 90, 78, 92]}
df = pd.DataFrame(data)

# Custom function for modifying the dataframe
def add_grade_column(df):
    df['Grade'] = df['Score'].apply(lambda x: 'A' if x >=90 else 'B')
    return df

# Using pipe() for method chaining
df = df.pipe(add_grade_column)

print("Dataframe after using pipe : \n", df)

Output
DataFrame after Using Pipe

Index	Name	Score	Grade
0	Alice	85	B
1	Bob	90	A
2	Charlie	78	B
3	David	92	A

Multi Index Slicing

import pandas as pd

# Create Multi Index Dataframe
arrays = [['A', 'A', 'B', 'B'], [2020, 2021, 2022, 2023]]
index = pd.MultiIndex.from_arrays(arrays, names=('Category', 'Year'))
data = [100, 150, 200, 250]
df = pd.DataFrame(data, index=index, columns=['Sales'])

# Slicing the  mutiindex Dataframe
sliced_df = df.loc['A']

print("Sliced  mutiindex Dataframe \n", sliced_df)

Output
Sliced MultiIndex DataFrame

Year	Sales
2020	100
2021	150

Adding prefix or suffix to column names

import pandas as pd

# Create a dataframe
data = {
    'Math': [90, 80],
    'Science':[85,88]
}
df = pd.DataFrame(data)

# Adding prefix to column names
df_prefixed = df.add_prefix('Grade_')

# Adding suffix to column names
df_suffixed = df.add_suffix('_Score')

print("Dataframe with Prefix :\n", df_prefixed)
print('\n')
print("Dataframe with Suffix :\n", df_suffixed)

Output
DataFrame with Prefix

Index	Grade_Math	Grade_Science
0	90	85
1	80	88

DataFrame with Suffix

Index	Math_Score	Science_Score
0	90	85
1	80	88

Using `at` and `iat` for fast scalar access

import pandas as pd

# Create dataframes
data = {
    'A':[1,2,3],
    'B':[4,5,6]
}
df = pd.DataFrame(data)

# Accessing a single value using at and iat
value_at = df.at[1, 'A']
value_iat = df.iat[1,0]

print('values using at : ' , value_at)
print('values using iat : ' , value_iat)

Output
values using at : 2
values using iat : 2

Renaming index & columns


import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob'], 'Score': [78,92]}
df = pd.DataFrame(data, index=['Row1','Row2'])


# Renaming index & columns
df_renamed = df.rename(index={'Row1':'Student1', 'Row2':'Student2'}, columns={'Score':'Grade'})

print("Renamed Dataframe : \n",df_renamed)

Output
Renamed DataFrame

Index	Name	Grade
Student1	Alice	78
Student2	Bob	92

Use `query()` with variables

import pandas as pd

# Create dataframe
data = {'Name' : ['Alice', 'Bob', 'Charlie', 'David'], 'Score': [85,90,78,92]}
df = pd.DataFrame(data)

# Using query() with varibales
threshold = 80
filtered_df = df.query('Score > @threshold')

print("Filtered Dataframe with query() and varibales: \n", filtered_df)

Output

Filtered DataFrame withquery()and Variables

Index	Name	Score	Age
0	Alice	85	25
1	Bob	90	37
3	David	92	21

Using `groupby` with custom aggregation

import pandas as pd

# Create dataframe
data = {
    'Category':['A','A','B','B'],
    'Value':[10, 20, 30,40]
}

df = pd.DataFrame(data)

# using groupby with custom aggregation
custom_df = df.groupby('Category').agg(max_value = ('Value', 'max'), min_value=('Value', 'min'))

print("Custom Aggregation result : \n", custom_df)

Output
Custom Aggregation Result

Category	max_value	min_value
A	20	10
B	40	30

Pivoting with fill_value

import pandas as pd

# Create dataframe
data = {'Date':['2023-01-01', '2023-01-02','2023-01-01','2023-01-02'],
        'City':['NY', 'NY','LA', 'LA'],
        'Sales':[200,250,300,None]
        }

df = pd.DataFrame(data)

# Pivoting with fill_value
pivot_df = df.pivot(index='Date', columns='City', values='Sales').fillna(0)

print('Pivoted Dataframe with fill_value: \n', pivot_df)

Output

Pivoted Dataframe with fill_value:

Date	LA	NY
2023-01-01	300.0	200.0
2023-01-02	0.0	250.0

Pivoting with fill_value

import pandas as pd

# Create a dataframe with duplicate rows
data = {'ID':[1,2,2,3,4,4], 'Value': [10, 20,20,30,40,40]}
df = pd.DataFrame(data)

# Dropping duplicates rows
unique_df = df.drop_duplicates()

print("Dataframe after dropping duplicates: \n", unique_df)

Output

DataFrame After Dropping Duplicates

Index	ID	Value
0	1	10
1	2	20
3	3	30
4	4	40

Combining Dataframe with Multi-level Indexes

import pandas as pd

# Create Multi Index Dataframe
arrays = [['A', 'A', 'B', 'B'], [2020, 2021, 2022, 2023]]
index = pd.MultiIndex.from_arrays(arrays, names=('Category', 'Year'))
data1 = [100, 150, 200, 250]
df1 = pd.DataFrame(data1, index=index, columns=['Sales'])

data2 = [300, 400, 500, 6000]
df2 = pd.DataFrame(data2, index=index, columns=['Profit'])

# Combining Dataframe
combined_df = pd.concat([df1, df2], axis=1)

print("Combining Dataframe with Multi-level Indexes: \n ", combined_df)

Output

Combining DataFrame with Multi-level Indexes:

Category	Year	Sales	Profit
A	2020	100	300
	2021	150	400
B	2022	200	500
	2023	250	6000

Using `rolling()` with custom functions

import pandas as pd

# Create a time series dataframe
data_range = pd.date_range(start='2023-01-01', periods=10, freq='D')
data = {'Sales': [100, 200, 150, 300, 250, 400, 300, 350, 300, 400]}
df = pd.DataFrame(data, index= data_range)

# Applying a custom function with rolling()
df['Rolling Sum'] = df['Sales'].rolling(window=3).apply(lambda x:x.sum() if x.sum() > 50 else 0)

print("Dataframe with custom rolling sum : \n", df)

Output

DataFrame with Custom Rolling Sum

Date	Sales	Rolling Sum
2023-01-01	100	NaN
2023-01-02	200	NaN
2023-01-03	150	450.0
2023-01-04	300	650.0
2023-01-05	250	700.0
2023-01-06	400	950.0
2023-01-07	300	950.0
2023-01-08	350	1050.0
2023-01-09	300	950.0
2023-01-10	400	1050.0

Setting & Resetting Indexes

import pandas as pd

# Create dataframe
data = {'City':['NY', 'LA', 'NY','SF','LA'], 'Year':[2020,2020,2021,2021,2020], 'Sales':[100, 150, 200, 250,300]}
df = pd.DataFrame(data)

# Setting an index
df.set_index('City', inplace=True)
print("Dataframe after setting index: \n", df)
print('\n')
# Resetting index
df.reset_index(inplace=True)
print("Dataframe after resetting index: \n", df)

Output
DataFrame after Setting Index

City	Year	Sales
NY	2020	100
LA	2020	150
NY	2021	200
SF	2021	250
LA	2020	300

DataFrame after Resetting Index

City	Year	Sales
NY	2020	100
LA	2020	150
NY	2021	200
SF	2021	250
LA	2020	300

Working with time-zone

import pandas as pd

# Create a time series dataframe
data_range = pd.date_range(start='2023-01-01', periods=10, freq='D')
data = {'Sales': [100, 200, 150, 300, 250, 400, 300, 350, 300, 400]}
df = pd.DataFrame(data, index=data_range)

# Localizing the timezone to UTC (or any initial timezone)
df.index = df.index.tz_localize('UTC')

# Converting to a different timezone (US/Eastern)
df = df.tz_convert('US/Eastern')

print("Time series dataframe with time zone: \n", df)

Output
Time series dataframe with time zone:

DateTime	Sales
2022-12-31 19:00:00-05:00	100
2023-01-01 19:00:00-05:00	200
2023-01-02 19:00:00-05:00	150
2023-01-03 19:00:00-05:00	300
2023-01-04 19:00:00-05:00	250
2023-01-05 19:00:00-05:00	400
2023-01-06 19:00:00-05:00	300
2023-01-07 19:00:00-05:00	350
2023-01-08 19:00:00-05:00	300
2023-01-09 19:00:00-05:00	400

Detecting and Replacing Outliers

import pandas as pd

# Create dataframe
data = {'Values': [10,12,14,100,15,13,12]}
df = pd.DataFrame(data)

# Detecting Outliers using the IQR method
Q1 = df['Values'].quantile(0.25)
Q3 = df['Values'].quantile(0.75)

IQR = Q3 - Q1
outliers = (df['Values'] < (Q1-1.5 * IQR)) | (df['Values'] > (Q3 + 1.5 * IQR))

# Replacing outliers with the median
df.loc[outliers, 'Values'] = df['Values'].mean()

print("Dataframe after Replacing outliers: \n", df)

Output
Dataframe after Replacing Outliers:

Index	Values
0	10.000000
1	12.000000
2	14.000000
3	25.142857
4	15.000000
5	13.000000
6	12.000000

Dataframe with evaluated calculation

import pandas as pd

# Create a dataframe
data = {'A':[1,2,3,4], 'B':[5,6,7,8]}
df = pd.DataFrame(data)

# Using eval for calculations
df['C'] = df.eval('A + B *2')

print("Dataframe with evaluated calculation\n",df)

Output
Dataframe with Evaluated Calculation:

	A	B	C
0	1	5	11
1	2	6	14
2	3	7	17
3	4	8	20

Merging indicator for tracing source

import pandas as pd

# Create dataframe
df1 = pd.DataFrame({'ID': [1,2,3,4] ,'Name' : ['Alice', 'Bob', 'Charlie', 'David']})
df2 = pd.DataFrame({'ID': [3,4,5,6] ,'Name' : ['Charlie', 'David','Robin','Michale']})


# Merging with indicator to track the source
merged_df = pd.merge(df1, df2, on='ID', how='outer', indicator=True)

print('Merged dataframe with  indicator: \n', merged_df)

Output
Merged DataFrame with Indicator:

ID	Name_x	Name_y	_merge
1	Alice	NaN	left_only
2	Bob	NaN	left_only
3	Charlie	Charlie	both
4	David	David	both
5	NaN	Robin	right_only
6	NaN	Michale	right_only

Time Series with Business days

import pandas as pd

# Create a time series dataframe
data_range = pd.date_range(start='2023-01-01', periods=10, freq='B')
data = {'Sales': range(10)}
df = pd.DataFrame(data, index= data_range)

print("Time series Dataframe with business days : \n", df)

Output
Time Series DataFrame with Business Days:

Date	Sales
2023-01-02	0
2023-01-03	1
2023-01-04	2
2023-01-05	3
2023-01-06	4
2023-01-09	5
2023-01-10	6
2023-01-11	7
2023-01-12	8
2023-01-13	9

Stacking & Unstacking dataframe

import pandas as pd

# Create dataframe
data = {'City':['NY','LA'],
        '2020':[100,500],
        '2021':[200,450],
        }

df = pd.DataFrame(data)

# Stacking & Unstacking dataframe
stack_df = df.stack()
unstack_df = df.unstack()

print('Stacking dataframe \n', stack_df)
print('Unstacking dataframe \n', unstack_df)

Output
Stacked DataFrame:

Level 0	Level 1	Value
0	City	NY
	2020	100
	2021	200
1	City	LA
	2020	500
	2021	450

Unstacked DataFrame:

City	2020	2021
NY	100	200
LA	500	450

Creating Custom Indexes with Multi-Index

import pandas as pd

# Create a multi-index from tuple
index = pd.MultiIndex.from_tuples([('A',2020), ('A', 2021), ('B', 2021), ('C', 2022)], names=['Category', 'Year'])
data = [100, 150, 200, 250]
df = pd.DataFrame(data, index=index, columns=['Sales'])

print("Dataframe with custom multi-index: \n", df)

Output
Dataframe with Custom Multi-Index:

Category	Year	Sales
A	2020	100
A	2021	150
B	2021	200
C	2022	250

NumPy for Machine Learning & Deep Learning

arju10 — Fri, 08 Nov 2024 03:49:45 +0000

You can follow github.

Numpy For Machine Learning & Deep Learning

Numpy

Numpy is a python library used to work with arrays and stands for Numarical python. It works on linear algebra, fourier transform, and matrices domain.
Numpy is faster than list because numpy provides array object.

Numpy Functions:

Note: To use numpy always import it.

import numpy as np # Here, numpy is imported as np

1. zeros(): It creates a array with zeros. Example:

array_zeros = np.zeros((3,3))
print("Array of Zeros: \n", array_zeros)

Output

Array of Zeros:
[[0. 0. 0.]
[0. 0. 0.]
[0. 0. 0.]]

2. ones(): It creates a array with ones. Example:

array_ones = np.ones((2,2))
print("Array of ones: \n",array_ones)

Output

Array of ones:
[[1. 1.]
[1. 1.]]

3. full(): It creates a array with all elements as 7. Example:

array_full = np.full((2,2),7)
print("Array with all elements as 7 : \n",array_full)

Output

Array with all elements as 7 :
[[7 7]
[7 7]]

4. range(): It creates a array between 0 to 20. Example:

array_range = np.arange(20)
print("Array with range of numbers : ",array_range)

Output

Array with range of numbers :
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]

5. random():
Example:

import numpy as np

# Create array
random_array = np.random.rand(3,3) # Random values in 3 X 3
normal_array = np.random.randn(3,3) # Random values with a normal distribution

print("Randmon Array : \n", random_array)
print("\n")
print("Normal Array : \n", normal_array)

Output

Randmon Array :
[[0.14635308 0.23362121 0.28310351]
[0.23209788 0.67298124 0.84145588]
[0.6829577 0.45439367 0.05747398]]

Normal Array :
[[-0.57461744 -0.09078458 -0.50067585]
[ 0.86439072 -1.44615505 0.25666949]
[ 0.01952512 -1.70556208 -0.80816368]]

6. transpose():
Example:

import numpy as np

# Create 2D array
original_array = np.array([
    [1,2],
    [3,4]
])

# Transpose the array
transposed_array = np.transpose(original_array)

print("Original Array: \n", original_array)
print("Transposed Array: \n", transposed_array)

Output

Original Array:
[[1 2]
[3 4]]
Transposed Array:
[[1 3]
[2 4]]

7. reshape():
Example:

import numpy as np

# Create array
original_array = np.array(
    [1,4,6,7,8,9]
)

# Reshaping to different dimensions
reshaped_array = original_array.reshape((2,3))

print("Original Array : \n", original_array)
print("\n")
print("Reshaping Array : \n", reshaped_array)

Output

Original Array :
[1 4 6 7 8 9]
Reshaping Array :
[[1 4 6]
[7 8 9]]

8. concatenate():
Example:

import numpy as np

# Create an array
array1 = np.array([1,2,3])
array2 = np.array([4,5,6])

# Concatenating arrays
concatenating_array = np.concatenate((array1, array2))

print("Concatenate Array : ", concatenating_array)

Output

Concatenate Array : [1 2 3 4 5 6]

9. array_split():
Example:

import numpy as np

# create an array
array = np.array(
    [1,2,34,4,5,6]
)

# Spliting the array into three parts
split_arrays = np.array_split(array,3)

print("Original Array : ", array)
print("Spiting Array : ", split_arrays)

Output

Original Array : [ 1 2 34 4 5 6]
Spiting Array : [array([1, 2]), array([34, 4]), array([5, 6])]

10. linspace():
Example:

import numpy as np

# Generating a linear space array with evenly spaced values
linear_space = np.linspace(0,10,5)

print("Linear Space : ", linear_space)

Output

Linear Space : [ 0. 2.5 5. 7.5 10. ]

11. meshgrid():
Example:

import numpy as np

# Create arrays
x = np.array([1,2,3])
y = np.array([4,5,6])

# Create a meshgrid
x,y = np.meshgrid(x,y)

print("Meshgrid X : \n",x)
print("\n")
print("Meshgrid Y : \n",y)

Output

Meshgrid X :
[[1 2 3]
[1 2 3]
[1 2 3]]

Meshgrid Y :
[[4 4 4]
[5 5 5]
[6 6 6]]

12. clip():
Example:

import numpy as np

# Create a array
array = np.array([1,2,4,5])

# Clipping values to be between 2 and 4
clipped_array = np.clip(array, 2,4)

print("Original Array",array)
print("Clipped Array",clipped_array)

Output

Original Array [1 2 4 5]
Clipped Array [2 2 4 4]

13. flatten():
Example:

import numpy as np

# Create a 2D array
array_2d = np.array(
    [
        [1,2,3],
        [4,5,6],
        [7,8,9]
    ]
)

# Flattening the array
flattened_array = array_2d.flatten()

print("Flattened Array : ",flattened_array)

Output

Flattened Array : [1 2 3 4 5 6 7 8 9]

Arithmetic Operations

1. add(): It adds minimum two arrays. Both array size should be same.
2. subtract(): It will minus from one array to another array & Both array size should be same.
3. multiply(): It multiplys between minimum two arrays. Both array size should be same.
4. divide(): It divides an array with another array & Both array size should be same.
Examples:

import numpy as np

# Create arrays
array1 = np.array([10,20,30])
array2 = np.array([40,50,60])

# Arithmetic Operations
added = np.add(array1, array2)
subtracted = np.subtract(array1 , array2)
multiplied = np.multiply(array1, array2)
divided = np.divide(array1, array2)

print("Added: ", added)
print("Subtracted :", subtracted)
print("Multiplied: ", multiplied)
print("Divided: ", divided)

Output

Added: [50 70 90]
Subtracted : [-30 -30 -30]
Multiplied: [ 400 1000 1800]
Divided: [0.25 0.4 0.5 ]

5. power(): A power is a number multiplied by itself a certain number of times, represented by an exponent. For example:
10^2 means 10 multiplied by itself, which is (10 X 10 = 100).
6. sqrt(): A square root of a number is a value that, when multiplied by itself, gives the original number. It is represented by the square root symbol (√).

For example:

√121 = 11

In this case, 11 × 11 = 121, so the square root of 121 is 11.

Properties of Square Roots

The square root of a positive number has both a positive and a negative solution (e.g., √25 = ±5), but typically, the positive root is used.
The square root of 0 is 0.
Negative numbers do not have real square roots (they result in imaginary numbers).

Examples :

√144 = 12
√64 = 8
√49 = 7

import numpy as np

# Create array
array = np.array([2,6,4,3,7])

# Power & Square Root
squared = np.power(array, 2)
sqrt = np.sqrt(array)

print("Squared Array :", squared)
print("Square root of array :", sqrt)

Output

Squared Array :[ 4 36 16 9 49]
Square root of array : [1.41421356 2.44948974 2. 1.73205081 2.64575131]

Trigometric Functions

1. sin():
2. cos():
3. tan():

Example:

import numpy as np

# Create array
angles = np.array([0, np.pi/2, np.pi])

# Trigometric functions
sin_array = np.sin(angles)
cos_array = np.cos(angles)
tan_arraY = np.tan(angles)

print("Sins of angles: ", sin_array)
print("Cos of angles : ", angles)
print("Tan of angles : ", angles)

Output

Sins of angles: [0.0000000e+00 1.0000000e+00 1.2246468e-16]
Cos of angles : [0. 1.57079633 3.14159265]
Tan of angles : [0. 1.57079633 3.14159265]

Exponential & Logarithm Function

1. exp():
2. log():

Example:

import numpy as np

# Create an array
array = np.array([10,20,30])

# Exponential & Logarithm functions
exp_array = np.exp(array)
log_array = np.log(array)

print("Exponential of Array: ",exp_array)
print("Logarithm of Array: ",log_array)

Output

Exponential of Array: [2.20264658e+04 4.85165195e+08 1.06864746e+13]

Logarithm of Array: [2.30258509 2.99573227 3.40119738]

Statistical Functions

1. mean():
2. median():
3. std():
4. var():

import numpy as np

# Create array
array = np.array(
    [1,2,3,4,5]
)

# Calculating statistical functions
mean = np.mean(array)
median = np.median(array)
std_dev = np.std(array)
variance = np.var(array)

print("Mean  : ", mean)
print("Median : ", median)
print("Standard deviation : ", std_dev)
print("Variance : ", variance)

Output
Mean : 3.0
Median : 3.0
Standard deviation : 1.4142135623730951
Variance : 2.0

Unique Elements and Their Counts

1. unique(): It finds the unique elements from array. If pass the parameterretuen_counts = True, it will show the number of unique elements from an array.

Example:

import numpy as np

# Create an array
array = np.array(
    [14,1,54,5,125,214,1,2,43,4,2,2,41,14,54,214,1]
)

# Finding Unique Element and their counts
unique_element , counts= np.unique(array, return_counts=True)

print("Unique Elements : ", unique_element)
print("Counts of Unique Elements: ", counts)

Output
Unique Elements : [ 1 2 4 5 14 41 43 54 125 214]
Counts of Unique Elements: [3 3 1 1 2 1 1 2 1 2]

Cumulative Sum and Product

1. cumsum():
2. cumprod():

Example:

import numpy as np

# Create a array
array = np.array([1,2,4,5])

# Calculating cumulative sum and product
cumulative_sum = np.cumsum(array)
cumulative_product = np.cumsum(array)

print("cumulative sum : ", cumulative_sum)
print("cumulative product : ", cumulative_product)

Output

cumulative sum : [ 1 3 7 12]
cumulative product : [ 1 3 7 12]

Broadcasting

Adding a scaler to an array.

Example:

import numpy as np

# create array
array = np.array(
    [1,2,3]
)

# Braodcasting: Adding a scaler to an array
result = array + 10

print("Original Array: ", array)
print("Array after broadcasting : ", result)

Output
Original Array: [1 2 3]
Array after broadcasting : [11 12 13]

Sorting Array `sort()`

1. sort(): It sorts the array by default ascending order.

import numpy as np

# Create array
array = np.array(
    [2,4,5,3,10,7,8]
)

# Sort the array
sorted_array = np.sort(array,)

print("Sorted Arrays: ", sorted_array)

Output
Sorted Arrays: [ 2 3 4 5 7 8 10]

Array Indexing & Slicing: `[row:column]`

Basic Indexing and Slicing

import numpy as np

# Create a array
array = np.array([1,2,4,5])

# Aceesing element by index
first_element = array[0]

# Aceesing element from the end uing negative inidices
last_element = array[-1]
second_last_element = array[-2]

# Slicing the array
slice_array = array[1:4]

# Selecting Specific indices
selected_elements = array[[1,3]]

# Slicing with a step
step_slice = array[::2] # Every Second Element

#Modifying values specific indices
array[[1,3]] = [100, 200]

# Reversing the array using slicing
reversed_array = array[::-1]

print("First Element: ", first_element)
print("Last Element: ", last_element)
print("Second Last Element: ", second_last_element)
print("Sliced Element: ", slice_array)
print("Selected Elements :", selected_elements)
print("Sliced Array with Step 2 :", step_slice)
print("Modified Array : ",array)
print("Reversed Array: ", reversed_array)

Output

First Element: 1
Last Element: 5
Second Last Element: 4
Sliced Element: [2 4 5]
Selected Elements : [2 5]
Sliced Array with Step 2 : [1 4]
Modified Array : [ 1 100 4 200]
Reversed Array: [5 4 2 1]

2D Array indexing & Slicing

import numpy as np

# Create a 2D array
array_2d = np.array(
    [
        [1,2,3],
        [4,5,6],
        [7,8,9]
    ]
)

# Accessing individual elements
element = array_2d[1,2] # Accessing element at row 1, column 2

# Slicing Row
row_slice = array_2d[0, : ] # First row

# Slicing Column
column_slice = array_2d[:, 1] # Second Column

# Using arrays for indexing
rows = np.array([0,1,2]) 
cols = np.array([2,1,0])
indexed_elements = array_2d[rows, cols]

# Multi Dimensional Slicing
sub_array = array_2d[1:, 1:3]

# Selecting specific rows and columns using slices
selected_slice = array_2d[1:, :2]

# Slicing every second element in rows and columns [Advance Slicing with Step]
step_sliced_element = array_2d[::2, ::2]

print("Element at (1,2) : ", element)
print("First Row : ", row_slice)
print("Second Column: ", column_slice)
print("Indexed Elements : ", indexed_elements)
print("Sub Array (Slicing Rows & Columns) : \n", sub_array)
print("Selected Slice : \n", selected_slice)
print("Advanced sliced array with step \n", step_sliced_element)

Output
Element at (1,2) : 6

First Row : [1 2 3]

Second Column: [2 5 8]

Indexed Elements : [3 5 7]
Sub Array (Slicing Rows & Columns) :
[[5 6]
[8 9]]

Selected Slice :
[[4 5]
[7 8]]

Advanced sliced array with step
[[1 3]
[7 9]]

3D Array Indexing & Slicing

import numpy as np

# Create a 3D array
array_3d = np.array(
    [
        [
            [10,20], [30,40]
        ],
        [
            [50,60], [70,80]
        ]
    ]
)

# Accessing elements in a 3D Array
element = array_3d[1,0,1] # Accessing element at [1, 0 , 1]

# Slicing in 3D
slice_3D = array_3d[:, 0, : ] # First row

print("Element at (1,0,1): ", element)
print("\n")
print("Sliced 3D Array : \n", slice_3D)

Output
Element at (1,0,1): 60
Sliced 3D Array :
[[10 20]
[50 60]]

Multi Dimensional Array indexing with mixed types

import numpy as np

# Create 3D Array
array_3D = np.array([
        [[1,2], [3,4]], [[5,6], [7,8]], [[9,10], [11,12]]
]
)

# Mixing slicing with direct indexing
mixed_index = array_3D[1:, 0, 1]

print("Mixed type indexing result: ", mixed_index)

Output
Mixed type indexing result: [ 6 10]

Boolean Indexing

import numpy as np

# Create a array
array = np.array(
    [10,20,30,40,60,80]
)

# Boolean Indexing
greater_than_20 = array[array > 20]

# Boolean Indexing
even_numbers = array[array %2==0]

print("Original : ", array)
print("Elements greater than 20 : ",greater_than_20)
print("Even Numbers : ",even_numbers)

Output
Original : [10 20 30 40 60 80]
Elements greater than 20 : [30 40 60 80]
Elements greater than 20 : [10 20 30 40 60 80]

Fancy Indexing

import numpy as np

# Create a array
array = np.array([1,2,4,5])

# Selecting Specific indices
selected_elements = array[[1,3]]

print("Selected Elements :", selected_elements)

Output
Selected Elements : [2 5]

Combined Boolean and Fancy Indexing

import numpy as np

# Creating an array
array = np.array([10, 15, 20, 25, 30])

# Boolean condition combined with specific indices
result = array[(array > 10) & (array < 30)][[0, 2]]

print('Combined Boolean and Fancy Indexing Result:', result)

Output

Combined Boolean and Fancy Indexing Result: [15 25]

Combined Boolean , Fancy and slice indexing

import numpy as np

# Creating a 2D array
array_2d = np.array([[10, 20, 30], [40, 50, 60], [70, 80, 90]])

# Applying boolean indexing first, then fancy indexing and slicing
result = array_2d[array_2d > 30].reshape(-1, 3)[:, [0, 2]]

print('Combined Indexing Result:', result)

Output

Combined Indexing Result:
[[40 60]
[70 90]]

Conditional Indexing

import numpy as np

# Create a array
array = np.array(
    [1,2,3,4,5,6]
)

# Setting elements that satisfying a condition
array[array % 2 == 0] = -1 # Set even numbers to -1

print("Array after conditional indexing : ",array)

Output
Array after conditional indexing : [ 1 -1 3 -1 5 -1]

Ellipsis Indexing

import numpy as np

# Create a 3D array
array_3D = np.array(
    [[
        [1,2],
        [3,4],
        [5,6],
        [7,8],
        [9,10],
        [11,12]
    ]]
)

# Using Ellipisis to seletect all elements along axes
result = array_3D[..., 1] # Selecting the second column in each 2D Slices

print("Result using Ellipsis (...): \n", result)

Output

Result using Ellipsis (...): [[ 2 4 6 8 10 12]]

Mask Indexing `np.ma`

import numpy as np

# Create array
array = np.array([1,-2,3,-4,5])

# Creating a Mask for positive values
mask = array > 0
# Applyinh mask to get only positive values
positive_values = array[mask]

# Masking elements where values are negative
masked_array = np.ma.masked_where(array < 0, array)


print("Positive Values : ", positive_values)
print("Masked Array with negative values hidden: ", masked_array)

Output

Positive Values : [1 3 5]
Masked Array with negative values hidden: [1 -- 3 -- 5]

Matrix

Indexing using `np.diag()` `np.triu()` `np.tril`

import numpy as np

# Create a square Matrix 
matrix = np.array([
    [1,2,3],
    [4,5,6],
    [7,8,9]
])

# Extracting the main diagnal
diagnoal_elements = np.diag(matrix)

# Extracting the upper triangle (above the main diaginal)
upper_triangle = np.triu(matrix)

# Extracting the lower triangle (below the main diaginal)
lower_triangle = np.tril(matrix)

print("Diagnoal Elements: \n", diagnoal_elements)
print("Upper Triangle of the Matrix: \n", upper_triangle)
print("Lower Triangle of the Matrix: \n", lower_triangle)

Output

Diagnoal Elements: [1 5 9]
Upper Triangle of the Matrix:
[[1 2 3]
[0 5 6]
[0 0 9]]

Lower Triangle of the Matrix:
[[1 0 0]
[4 5 0]
[7 8 9]]

Indexing using `np.wehere()`

import numpy as np

# Create a array
array = np.array(
    [10,20,30,40,60,80]
)

# Using np.where to find indices where elements are greater than 25
indices = np.where(array > 25)

# Replacing values based on condition where elements are greater than 25
result = np.where(array > 25, 0, array)


print("Indices where elements > 25 : ",indices)
print("Values where elements > 25: ",array[indices])
print("Array with Values where elements > 25 replaced with 0: ",result)

Output

Indices where elements > 25 : (array([2, 3, 4, 5]),)
Values where elements > 25: [30 40 60 80]
Array with Values where elements > 25 replaced with 0: [10 20 0 0 0 0]

Indexing using `np.take()`

import numpy as np

# Create a array
array = np.array(
    [10,20,30,40,60,80]
)

# Using np.take to select elements at specific indices
indices = [0,2,4]
result = np.take(array, indices)

print("Selected Elements using np.take", result)

Output

Selected Elements using np.take [10 30 60]

Indexing using advanced `meshgrid()` **

Example:

import numpy as np

# Create arrays
x = np.array([1,2,3])
y = np.array([4,5,6])

# Create a meshgrid
x,y = np.meshgrid(x,y)

indices = np.vstack(
    [x.ravel(), y.ravel()]
)

print("Indices for Meshgrid Advanced Indexing: \n", indices)

Output

Indices for Meshgrid Advanced Indexing:
[[1 2 3 1 2 3 1 2 3]
[4 4 4 5 5 5 6 6 6]]

#### Multiple condition using np.choose() **

Example:

import numpy as np

# Create an array
array = np.array([1, 2, 4, 5])

# Extend choices to include placeholder arrays for out-of-bounds indices
choices = [array * 2, array + 10, array ** 2, array * 0, array * 0, array * 0]

# Use np.choose with the modified choices array
result = np.choose(array, choices)

print("Result using np.choose:", result)

Output

Result using np.choose: [11 4 0 0]

Python Program: How to Sort a List of Numbers

arju10 — Tue, 29 Oct 2024 09:20:11 +0000

List:

A list is a collection of items in python, created using square brackets [] . List can contain different data types, including numbers, strings and other lists. List is mutable ( It can't changed/modified).

Sort:

In list very easily numbers can be sorted.

Prerequisites for Sorting a List of Numbers:

Before starting sort the list, you need to know about List & How to create a list.

Create a List:

In the beginning, We will create a list

# Create a list with Numbers
my_list = [10,80,20,60,30,40,70]

# Print the list
print("List: ", my_list)

Here, 10, 80, 20, 60, 30, 40 and 70 are Numbers or we can call integer type data.
We will store these numbers in my_list variable.
print() is a builtin function in python which used to print or show the result whatever we want.

Multiple Ways to Sort List of Numbers or Integers

There are many ways sort a list, here I talk about some technique which are used for sort a list.

Method 1: Using the builtin sorted() function

sorted() is a builtin methods in python.By default, It sorted the number in ascending order. But If you want to descending the number just put , reverse=True parameter for descending the numbers.

- Ascending order:

# Create a list 
numbers_list = [10,80,20,60,30,40,70]

# Sorting in ascending order using sorted()
sorted_num_ascending = sorted(numbers_list)

# Displaying the ascending sorted numbers
print(sorted_num_ascending)

Output: [10, 20, 30, 40, 60, 70, 80]

- Descending Order:

# Create a list 
numbers_list = [10,80,20,60,30,40,70]

# Sorting in descending order using sorted() and add paremeter reverse = true
sorted_num_descending = sorted(numbers_list,  reverse=True)

# Displaying the descending sorted numbers
print(sorted_num_descending)

Output: [80, 70, 60, 40, 30, 20, 10]

Method 2: Using the builtin sort() function

sort() is a builtin methods in python.By default, It sorted the number in ascending order. But If you want to descending the number just put , reverse=True parameter for descending the numbers.
Note: It is same as sorted() function.but a difference between them;

sort() function change the original list , while sorted() method keep the original list , and store the sorted list in a new variable.

- Ascending order:

# Create a list 
numbers_list = [10,80,20,60,30,40,70]

# Sorting in ascending order using sort() 
numbers_list.sort()

# Displaying the ascending sorted numbers
print(numbers_list)

Output: [10, 20, 30, 40, 60, 70, 80]

- Descending Order:

# Create a list 
numbers_list = [10,80,20,60,30,40,70]

# Sorting in descending order using sort() and add paremeter reverse = True
numbers_list.sort(reverse=True)

# Displaying the descending sorted numbers
print(numbers_list)

Output: [80, 70, 60, 40, 30, 20, 10]

Method 3: Using Lambda function with sorted()

If there is need any complex criteria for sorting, lambda function can be used there.
For example:

# Create a list 
numbers_list =[25, 17, 23, 51, 50]

# Sorting by remainder when divided by 5 using lambda function with sorted()
sorted_num_remainder = sorted(numbers_list, key=lambda x: x % 5)

# Displaying the sorted numbers
print(sorted_num_remainder)

Output: [25, 50, 51, 17, 23]

Method 4: Using slicing technique `[::-1]`

Elements of a list can be reverse using slicing technique.In slicing, there are no direct method to sort a list, but can reverse a list.

# Create a list 
numbers_list = [10,80,20,60,30,40,70]

# Reversing the list using slicing
reversed_number = numbers_list[::-1]

# Displaying the reversed numbers
print(reversed_number)

Output: [70, 40, 30, 60, 20, 80, 10]

Method 5: Using heapq module for large lists

If there is need any deep complex criteria for sorting, heapq can be used there.heapq is heap based algorithm which is very efficient memory rather than other sorting methods.
For example:

import heapq as hq
# Create a list 
numbers_list =[25, 17, 23, 51, 50]

# Sorting in descending order using heapq
hq.heapify(numbers_list)
sorted_num_hq = [hq.heappop(numbers_list) for _ in range(len(numbers_list))]

# Displaying the sorted numbers
print(sorted_num_hq)

Output: [17, 23, 25, 50, 51]

DEV Community: arju10

Pandas for Machine Learning & Deep Learning

Pandas

Grouping & Aggregation

Grouping & Aggregation

MutiIndex DataFrame Operations

Using Apply with Lambda Functions

Merging DataFrame with Different Keys

Handling Missing Values with Custom Functions

Handling Missing Values with ffilll & bfill method

Pivoting DataFrame

Pivoting DataFrame with multi index columns

Melting DataFrame

Time based indexing and resampling

Resampling with different aggregation

Conditional Filtering with Multiple Conditions

Creating Custom Categorical Data

Binning Data using pd.cut

Rolling Window Calculations

Shift and Lagging Data

Cumulative sum and product

Merging on multiple columns

Handling Outliers

Creating a pivot table with multiple aggregation

Creating a pivot table with multiple indexes

Using map() for Value Replacement

Detecting Duplicates

Using explode() for list in columns

Using rank() to rank values

Using applymap() for element-wise operations

Using df.apply() with axis parameter

Create dummy variables

Use query() for filtering

Adding a new column using assign()

Applying a custom function with groupby

Applying functions to grouped data

DataFrame info and memory usage

Filter with isin()

Concatenating Dataframe using concat()

Sorting a DataFrame by Multiple Columns

Displaying the styled Dataframe

Using explode() for nested JSON columns

Using pipe() for method chaining

Multi Index Slicing

Adding prefix or suffix to column names

Using at and iat for fast scalar access

Renaming index & columns

Use query() with variables

Using groupby with custom aggregation

Pivoting with fill_value

Pivoting with fill_value

Combining Dataframe with Multi-level Indexes

Using rolling() with custom functions

Setting & Resetting Indexes

Working with time-zone

Detecting and Replacing Outliers

Dataframe with evaluated calculation

Merging indicator for tracing source

Time Series with Business days

Stacking & Unstacking dataframe

Creating Custom Indexes with Multi-Index

NumPy for Machine Learning & Deep Learning

You can follow github.

Numpy

Numpy Functions:

Arithmetic Operations

Properties of Square Roots

Trigometric Functions

Exponential & Logarithm Function

Statistical Functions

Unique Elements and Their Counts

Cumulative Sum and Product

Broadcasting

Sorting Array sort()

Array Indexing & Slicing: [row:column]

Basic Indexing and Slicing

2D Array indexing & Slicing

3D Array Indexing & Slicing

Multi Dimensional Array indexing with mixed types

Boolean Indexing

Handling Missing Values with `ffilll` & `bfill` method

Binning Data using `pd.cut`

Using `map()` for Value Replacement

Using `explode()` for list in columns

Using `rank()` to rank values

Using `applymap()` for element-wise operations

Using `df.apply()` with axis parameter

Use `query()` for filtering

Adding a new column using `assign()`

Applying a custom function with `groupby`

Filter with `isin()`

Concatenating Dataframe using `concat()`

Using `explode()` for nested JSON columns

Using `pipe()` for method chaining

Using `at` and `iat` for fast scalar access

Use `query()` with variables

Using `groupby` with custom aggregation

Using `rolling()` with custom functions

Sorting Array `sort()`

Array Indexing & Slicing: `[row:column]`

Mask Indexing `np.ma`

Indexing using `np.diag()` `np.triu()` `np.tril`

Indexing using `np.wehere()`

Indexing using `np.take()`

Indexing using advanced `meshgrid()` **

Method 4: Using slicing technique `[::-1]`