DEV Community: Charles De Barros

Back to Basics - Pandas #02

Charles De Barros — Tue, 16 Apr 2024 09:26:57 +0000

The power of the 'locs'

One of the most common tasks we perform with Pandas is data indexing and selection. We do that pretty much daily.

Let's delve into the world of Pandas and explore the differences between loc() and iloc() operators. These two methods are essential for data manipulation in Python, especially when working with DataFrames. Knowing how to apply those two well is the key to filtering a DataFrame efficiently. Did you get it? loc and key? Nevermind! Let's jump in.

Selecting with `loc()` and `iloc()`

1. `loc()` - Label-Based Data Selection

The loc() function is a label-based data selection method. It allows us to select rows or columns based on their labels (i.e., row or column names) but may also be used with a boolean array with the same length as the row axis. Some key points about loc():

We pass the name of the row or column we want to select.
It includes the last element of the range specified.
It can accept boolean data for filtering.
It's useful for selecting data based on specific conditions.

2. `iloc()` - Integer-Based Data Selection

The iloc() function, on the other hand, is an integer-based data selection method. It uses integer positions to access data but may also be used with a boolean array. Here are some aspects of iloc() to keep in mind:

We specify the integer index of the row or column we want to select.
It excludes the endpoint when slicing (similar to the Python slicing convention).
Like loc[], it also accepts boolean data for filtering.
It's ideal for accessing data by position.

Examples

Let's demonstrate these concepts using a sample DataFrame containing information about cars:

import pandas as pd
data = pd.DataFrame({
    'Brand': [
        'Ford', 'Hyundai', 'VW', 'Vauxhall', 'Ford', 
        'Hyundai', 'Renault', 'VW', 'Ford
    '],
    'Year': [
        2012, 2014, 2011, 2015, 2012, 
        2016, 2014, 2018, 2019
    ],
    'Kms Driven': [
        50000, 30000, 60000, 25000, 10000, 
        46000, 31000, 15000, 12000
    ],
    'City': [
        'Manchester', 'London', 'Birmingham', 'London', 'Birmingham', 
        'London', 'Birmingham', 'Liverpool', 'Nottingham'],
    'Mileage': [
        28, 27, 25, 26, 28, 29, 24, 21, 24
    ]
})
# Displaying the DataFrame
data

Displaying the DataFrame above we get:

Brand	Year	Kms Driven	City	Mileage
Maruti	2012	50000	Manchester	28
Hyundai	2014	30000	London	27
VW	2011	60000	Birmingham	25
Vauxhall	2015	25000	London	26
Ford	2012	10000	Birmingham	28
Hyundai	2016	46000	London	29
Renault	2014	31000	Birmingham	24
VW	2018	15000	Liverpool	21
Ford	2019	12000	Nottingham	24

Example 1: Conditional Selection Data

Let's use loc() to find Ford cars with a mileage greater than 25:

display(data.loc[(data.Brand == 'Ford') & (data.Mileage > 25)])

Output:

Brand	Year	Kms Driven	City	Mileage
Ford	2012	50000	Manchester	28
Ford	2012	10000	Birmingham	28

Example 2: Row Selection using ranges

We'll use iloc() to extract rows with indices from 2 to 5 (inclusive):

display(data.iloc[2:6])

Output:

Brand	Year	Kms Driven	City	Mileage
VW	2011	60000	Birmingham	25
Vauxhall	2015	25000	London	26
Ford	2012	10000	Birmingham	28
Hyundai	2016	46000	London	29

Summary

The Pandas loc and iloc are powerful tools for selecting and manipulating data within Pandas DataFrames and Series. Its utility ranges from simple row-and-column selections to more complex operations combined with other Pandas features like groupby. They can be adapted to work with boolean conditions, thereby offering a flexible approach to data manipulation tasks. Mastering loc and iloc will add flexibility to any Data Analyst's toolbox.

Back to Basics - Python #02

Charles De Barros — Wed, 10 Apr 2024 14:08:47 +0000

The Numeric data type

There are eight major data types in Python:

Description	Type
Text	str
Numeric	int, float, complex
Sequence	list, tuple, range
Mapping	dict
Set	set, frozenset
Boolean	bool
Binary	bytes, bytearray, memoryview
None	NoneType

Numeric data types are widely used in mathematical, statistical, data science, and machine learning solutions.

There are three distinct numeric types in Python:

int (integer)
float
Complex

Integers

What is an integer?

An integer is a whole number, positive or negative, with no fractional part, i.e., no decimals, and unlimited length. It includes the 0 (zero).

We can write them down as "...-3, -2, -1, 0, 1, 2, 3..."

Python uses the class int to represent all integer numbers.

When we define a variable num with a value of 5, the variable num does not hold the real integer 5. It stores a reference pointing to where the integer 5 is held in the computer's memory.

num = y
print(type(num))  # returns the variable num's type

The output is:

<class 'int'>

Note: In Python 3, there is effectively no limit to how long an integer value can be. The limit is constrained by the amount of memory your system has.

Integer operations

We can perform all standard mathematical operations using Python's integers, including:

Operation	Operator
Addition	+
Substraction	-
Multiplication	*
Division	/
Exponentiation	**

For example:

num_1 = 10
num_2 = 5

c = num_1 + num_2
print(c)
print(type(c))

c = num_1 - num_2
print(c)
print(type(c))

c = num_1 * num_2
print(c)
print(type(c))

c = num_1 / num_2
print(c)
print(type(c))

c = num_1 ** num_2
print(c)
print(type(c))

The above calculations will return the following:

# Addition
15
<class 'int'>

# Subtraction
5
<class 'int'>

# Multiplication
50
<class 'int'>

# Division
# Note the result of the division is a decimal number. 
# More on that later.
2.0
<class 'float'>

# Exponetiation
100000
<class 'int'>

Floats

In addition to integers, consisting of whole numbers without fractions, Python also offers us the float type.

What is a float number?

A float is a positive or negative number with afractional part, i.e., decimals, and unlimited length. It includes the 0 (zero).

Float numbers precision

Python’s built-in float data type provides up to 15 digits of decimal precision. That is usually enough for common calculations.

100 / 30
3.3333333333333335 
# The floor division above returns a float.
# Note the value is rounded to the nearest 15-digit value available.

The default precision is 6 decimal places. You can set the precision using the round():

For example:

x = 7.78
y = 3.3
result = x * y  # return 25.674
round(result, 2) 
25.67   # returns 'result' round to two decimals

Operations with Floats

Operations with floats follow the same rules as operations with integers. The difference is, for obvious reasons, the presence of decimal points in the numbers used for the calculations.

Order of Operations

As in normal mathematical calculations, Python integers and floats follow a defined order when performing operations. The famous PEMDAS:

PEMBAS

PEMDAS was created to help remember the mathematical order of operations.

PEMBAS Rank and Priority

Rank/Priority	Operation
1	Parentheses (Brackets)
2	Exponents and Roots
3	* Multiplication
4	* Division
5	** Addition
6	** Subtraction

* In PEMDAS Multiplication and Division have equal rank in the order of operations and do not need to be in any specific order. Whichever comes first, from left to right, is performed first.

** The same principle of equal ranking applies to Addition and Subtraction. Whichever comes first, from left to right, is performed first.

Examples:

# Here multiplication happens first then the division
6 * 2 / 4
3.0

# And here the Division takes place first the the multiplication
6 / 2 * 4
12.0

Division and Floor Division

In Python, there are two ways to divide numbers:

The normal division using the / operator;
and the floor division, using the // operator.

Which one we will use will depend on the result we want to achieve.

What is the difference between the two?

The single forward slash operator / is known as float division and always returns a float point value.

We would normally expect the following when performing a float division.

# Two integers
100 / 30
3.3333333333333335

# One float and an integer
100.0 / 30
3.3333333333333335

# Two floats
100.0 / 30.0
3.3333333333333335

The double forward slash operator // returns a floored integer or a floating point value.

The result of a double slash operator will depend on whether we use int or float numbers.

Examples:

# Two integers return an integer. Where it is an exact division, 
# and there is no remainder.
100 // 25
4

# One integer and a float, always return a float.
100 // 25.0
4.0

# Two floats always return a float.
100.0 // 25.0
4.0

One interesting note about this is what happens with negative numbers:

-7.0 // 3
-3.0

This makes sense because the result will be rounded down (i.e., floored), meaning that while we may expect it to be equal to -2.0, rounded down, the value is correctly -3.0.

Complex Numbers

Complex numbers are the numbers that are expressed in the form of a+ib where, a,b are real numbers and ‘i’ is an imaginary number called “iota”. The value of i = (√-1). For example, 2+3i is a complex number, where 2 is a real number (Re) and 3i is an imaginary number (Im).

Examples of complex numbers:

x = 7 + 2i
y = -5 + 1.2i

Conversion between Numeric types

A Numeric type can be converted into another numeric type by typecasting with the intended type.

x = 2.78
int_x = int(x)  # converts a float into an integer

y = 4
float_y = float(y)  # converts an integer into a float

Important notes on conversions:

Converting a float to an integer will truncate the decimal part. In the example above, converting 2.78 into an integer, the result was 2, the number before the decimal point. Turning 2.99 into an int would still return 2.
Converting a large integer to float may lose precision.
Converting an integer to a complex number just adds 0j as the imaginary part.

The use of Numeric types

There are numeric types all over the programming world. Some examples of areas where Numeric types are used but not restricted to:

Finance apps
Games
Scientific apps
Statistics
Graphics
Machine Learning
Web Development
Data Analysis and Data Science
Embedded systems

Summary

Whether using Python for programming, analysis, or web development, amongst other fields, developers rely heavily on Numeric types. Integers, float, complex numbers, calculations, equations, designing; the possibilities are endless.

Understanding how Numeric types work is an essential and useful skill people should have. It becomes even more important if our fields of work involve working with programming languages. Fortunately, Python is a solid language for processing numbers and has a very generous number of libraries available to expand Python's capability even further.

Have a go at exploring Python Numeric types. It will certainly be a nice addition to your programming skills. Pun intended.

Back to Basics - Pandas #01

Charles De Barros — Tue, 09 Apr 2024 11:20:31 +0000

Introduction to Pandas

Pandas Official Documentation.

What is Pandas?

As per the Pandas Official Documentation website:

Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

Pandas simplifies common data analysis tasks:

Load data from numerous types of files and online sources
Fast and efficient handling of large amounts of data
Filtering, sorting, editing and processing of data
Joining and aggregation of datasets
Tools for time series and statistical analysis
Display of data in tables and charts

Pandas data structures: DataFrames & Series

DataFrame (rows and columns)

DataFrame is a 2-dimensional labelled data structure with columns of potentially different types. You can think of it like a spreadsheet, an SQL table, or a dictionary of Series objects. It is generally the most commonly used Pandas object. Like Series, DataFrame accepts many different kinds of input:

Dict of 1D ndarrays, lists, dicts, or Series
2-D numpy.ndarray
Structured or record ndarray
A Series
Another DataFrame

Notes on Data Frames:

2-dimensional labelled data structure made of rows and columns of 'potentially' different types.
Similar principle of a spreadsheet or SQL table. The most commonly used data structure used in Pandas.
Indexing starts from 0 (zero) for both rows and columns.

Series (one-dimensional data)

Series is a one-dimensional labelled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.).

It is a one-dimensional labelled data.
An example of a Series is one column from a Data Frame.
Indexing in a Series starts from 0 (zero).

The basic method to create a Series is to call:

s = pd.Series(data, index=index)

A Series plus another Series equals a Data Frame.

Pandas Common Data Types

Data type	Description
object	Used for strings, or if the column contains a mix of data types
int64	Used for integers ('64' relates to memory usage)
float64	Used for floats, or where the column has both integers and NaN values
bool	Booleans, i.e., `True` or `False`
datatime64 / timedelta	Time-based values

Pandas Missing Values

The NaN marker represents Pandas missing values or NULL values. In most cases, the terms missing and null are interchangeable. Date values use the NaT marker.

Symbol	Description
NaN	Used to indicate missing values in most instances and is supported by the floar datatype.
NaT	Used to indicate missing values where a `date type` object may have been expected.

Exploratory Data Analysis - EDA

What is EDA?

Exploratory data analysis (EDA) is used by data scientists to analyse and investigate data sets and summarise their main characteristics, often employing data visualisation methods.

EDA helps determine how best to manipulate data sources to get the answers needed, making it easier for data scientists to discover patterns, spot anomalies, test a hypothesis, or check assumptions.

EDA is primarily used to see what data can reveal beyond the formal modelling or hypothesis testing task and provides a better understanding of data set variables and their relationship. It can also help determine if the statistical techniques you are considering for data analysis are appropriate.

Basic Pandas Data Frame exploration

Importing Pandas (and NumPy)

The following lines will bring both Pandas and NumPy libraries to the working environment.

import pandas as pd
import numpy as np

Data can be imported from various formats: CVS, spreadsheets, JSON, databases, etc.

The following line will import a CSV (Comma Separated Values) to the working sessions.

df = pd.read_csv(<filepath>)

Note: different parameters can be used with the .read_csv() function. In its simplest form, only the filepath is required.

The "Telco-Customer-Churn.csv" dataset can be found here at Kaggle.

# Will open the 'Telco-Customer-Churn.csv' in the 'data' folder
df = pd.read_csv("data/Telco-Customer-Churn.csv")

Checking Top/Bottom rows

We can use the .head() and the .tail() functions to see the top first 5 rows, and the bottom last 5 rows, in ascending order.

The .head() function returns the first n rows for the object based on position. It is useful for quickly testing if your object has the right type of data in it. The default is 5 rows.

The .tails() function returns the last n rows from the object based on position. It is useful for quickly verifying data, for example, after sorting or appending rows. The default is 5 rows.

# Returns the first 5 top rows in ASC order
df.head()

# Returns the last 5 bottom rows in ASC order
df.tail()

Checking random sample rows

The .sample() will return sample rows at random. It offers an interesting look at the body of the Data Frame. The default is to return only 1 row.

# Returns 10 random row samples.
.sample(10)

Showing the Data Frame Dimensions

The .shape method to display the dataframe dimensions: rows and columns.

# In the 'Telco-Customer-Churn.csv' there are 7043 rows and 21 columns.
df.shape
(7043, 21)

Displaying the Data Frame basic info

The .info() prints information about a DataFrame including the index dtype and columns, non-null values and memory usage.

# Print a concise summary of a DataFrame.
df.info()

Returns:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7043 entries, 0 to 7042
Data columns (total 21 columns):
 #   Column            Non-Null Count  Dtype  
---  ------            --------------  -----  
 0   customerID        7043 non-null   object 
 1   gender            7043 non-null   object 
 2   SeniorCitizen     7043 non-null   int64  
 3   Partner           7043 non-null   object 
 4   Dependents        7043 non-null   object 
 5   tenure            7043 non-null   int64  
 6   PhoneService      7043 non-null   object 
 7   MultipleLines     7043 non-null   object 
 8   InternetService   7043 non-null   object 
 9   OnlineSecurity    7043 non-null   object 
 10  OnlineBackup      7043 non-null   object 
 11  DeviceProtection  7043 non-null   object 
 12  TechSupport       7043 non-null   object 
 13  StreamingTV       7043 non-null   object 
 14  StreamingMovies   7043 non-null   object 
 15  Contract          7043 non-null   object 
 16  PaperlessBilling  7043 non-null   object 
 17  PaymentMethod     7043 non-null   object 
 18  MonthlyCharges    7043 non-null   float64
 19  TotalCharges      7043 non-null   object 
 20  Churn             7043 non-null   object 
dtypes: float64(1), int64(2), object(18)
memory usage: 1.1+ MB

Generating Descriptive Statistics

The .describe() function generates descriptive statistics.

Descriptive statistics is the process of using current and historical data to identify trends and relationships. It includes those that summarize the central tendency, dispersion and shape of a dataset’s distribution, excludingNaN values.

Analyzes both numeric and object series, as well as DataFrame column sets of mixed data types. The output will vary depending on what is provided.

It returns the following:

count: Total number of non-missing values
mean: The mean value
std: The standard deviation
min: The minimum value
25%: The value of the first quartile (25th percentile)
50%: The median value (50th percentile)
75%: The value of the third quartile (75th percentile)
max: The maximum value

df.describe()

Returns:

      SeniorCitizen     tenure  MonthlyCharges
count   7043.000000   7043.000000      7043.000000
mean       0.162147     32.371149        64.761692
std     0.368612     24.559481       30.090047
min     0.000000      0.000000       18.250000
25%     0.000000      9.000000       35.500000
50%     0.000000     29.000000       70.350000
75%     0.000000     55.000000       89.850000
max     1.000000     72.000000      118.750000

Sorting Values

The .sort_values() function sorts by the values along either axis. it allows us to choose a 'column' to sort it by. The default is ASC.

# ascending=False will sort the column values in DESC order.
df.sort_values(by='monthlycharleges', ascending=False)

Summary

As a high-level, data manipulation and analysis library, Pandas is a powerful and versatile tool in any Data Analyst arsenal. It is fast, easy to use, and very comprehensive to analyse and manipulate datasets.

Pandas' realm of functions and methods is vast, no doubt about it. Pandas is a must-have tool data analysts should be acquainted with and use with their daily tasks.

Back to Basics - Python #01

Charles De Barros — Mon, 01 Apr 2024 16:25:16 +0000

Intro

It's March 2024 and I have joined a Level 4 Data Analytics Bootcamp! Hurray! Technically lessons will only start at the beginning of April but I am already very excited about it and thinking of things.

To celebrate my new adventure and start having my brain cells back working, I am writing a series of Back to Basics articles exploring Data Analytics basics. I am starting with Python, the programming language we will use in the boot camp, its components, data exploring libraries (NumPy, Pandas, Matplotlib anyone?), and styling conventions, to name a few.

This series intends to be more of a reference for myself and my bootcamp studies but I hope it can be useful for anyone starting their journey on Data Analysis. It is far from a comprehensive guide; it is indeed an open, live document that will, hopefully, expand with time. Any useful additions and constructive criticism will be immensely appreciated.

In this first edition of Back to Basics I am going to explore one of the very basic Python's building blocks: variables.

Introduction to Python

Python is a high-level, object-oriented, general-purpose programming language. Guido Van Rossum, a Dutch programmer, first conceived it in the late 1980s. It first appeared in February 1991, and its 1.0 version was released in 1994.

Van Rossum thought programming with C, another programming language, was too time-consuming and a programming language should allow programmers to work faster and more efficiently. Also, he primed for readability, and simplicity with bash capabilities.

As of the writing of this article, Python is on version 3.12.2 and it can be downloaded here.

Python's naming curiosity

Contrary to popular belief, Python was not named after a snake species, it was named as a tribute to the British comedy group Monty Python as Python developers aim it to be fun to use.

Variables

What is a variable?

A variable is a container for storing data. A variable is a basic building block of a program.

Python is not a "statically typed" programming language, i.e., we do not need to declare variables types before using them. Python is a "dynamically typed" programming language, i.e., it only checks variables at execution time (runtime). Python is so intelligent it can figure those out without us explicitly having to tell it.

The variable value can be changed during the execution of a program.
A variable does not hold a real value but a reference to it. For example, a variable called 'x' with a value of '10' does not hold the real integer '10' but a reference to where it is located in the computer's memory. It works as a pointer to where the value is stored.

If we assign a different value to 'x', the variable reference location will change and point to the new in-memory location. '10' will still be '10. The integer's value did not change in the computer's memory.

Example:

x = 10  # points to the integer '10' location reference
x = 20  # it now points to the integer '20' location reference

Declaring a variable

A variable is declared (created) when a value is assigned to it for the first time. To declare a variable we need to specify a name and a type.

Note: Python has no command for declaring a variable.

The syntax to declare a variable can vary amongst programming languages. In Python, we use a = (equal symbol) to assign a value to a variable name.

For example:

quantity = 1
price = 2.5
product = 'muffin'
order = [2, 'cakes', 5.95]

The variables's names are 'quantity', 'price', 'product', and 'order'. The variable values are 1, 2.5, 'muffin', and [2, 'cakes', 5.95]. The variable values can be of different types, as explained further below.

Variable Naming Rules

Variables in Python can be of any length, and consist of uppercase and lowercase letters (A-Z, a-z), digits(0-9), and the underscore character '_'.

Note: The first character of a variable name cannot be a digit. Declaring a variable with the name '2x' will raise a Syntax Error.

Example:

2x = 4
  Cell In[2], line 1
    2x = 4
    ^
SyntaxError: invalid decimal literal

Note: Variable names cannot have spaces. Declaring a variable name with space on it will raise a Syntax Error.

Example:

party cake = 'good'
  Cell In[1], line 1
    party cake = 'good'
          ^
SyntaxError: invalid syntax

Case-sensitiviness

Python variables are case-sensitive, i.e., variables named 'cakes', 'Cakes', and 'CAKES', although using the same word, are three different variables. They reference different things.

Example:

cakes = 'yummy'
Cakes = 'delicious'
CAKES = 'more please'

Python Style Guide = PEP 8

Python has a recommended Styling Guide. It is available here.

Python Reserved words

We cannot use certain words as variable names, function names, or any other identifier because they are reserved for exclusive Python use.

As of the writing of this article, the following are Python's reserved words:


False	def	if	raise
None	del	import	return
True	elif	in	try
and	else	is	while
as	except	lambda	with
assert	finally	nonlocal	yield
break	for	not
class	form	or
continue	global	pass

Note: All keyword are in lowercase with the exception of True, False, and None.

Python built-in Data Type

A data type in a programming language refers to an attribute that identifies the nature of a piece of data's value. It tells a computer how to interpret the data value. Some common examples include strings, integers, floats, and booleans.

Understanding Data Types is important as variables can store data types. Different data types can do different things.

The following Data Types are built-in by default in Python.

Description	Type
Text	str
Numeric	int, float, complex
Sequence	list, tuple, range
Mapping	dict
Set	set, frozenset
Boolean	bool
Binary	bytes, bytearray, memoryview
None	NoneType

To find the type of a variable, we can use Python's type() function:

For example:

quantity = 1                # 'quantity' is of type 'int'
price = 2.5                 # 'price' is of type 'float'
product = 'muffin'          # 'product' is of type 'str'
order = [2, 'cakes', 5.95]  # 'order' is of type 'list'

print("quantity", type(quantity))
print("price", type(price))
print("product", type(product))
print("order", type(order))

# Returns
quantity <class 'int'>
price <class 'float'>
product <class 'str'>
order <class 'list'>

Summary

Variables are essential building blocks of any programming language. Understanding how they work and what we can do with them will make the difference between a concise, clearly written codebase and a convoluted, difficult-to-understand, and maintain one.

Consistency in how we create, name and use our variables will help us keep things under control as our codebase grows and becomes more complex.

Treat your codebase with care. You will thank yourself later.

DEV Community: Charles De Barros

Back to Basics - Pandas #02

The power of the 'locs'

Selecting with loc() and iloc()

1. loc() - Label-Based Data Selection

2. iloc() - Integer-Based Data Selection

Examples

Example 1: Conditional Selection Data

Example 2: Row Selection using ranges

Summary

Back to Basics - Python #02

The Numeric data type

Integers

Integer operations

Floats

Float numbers precision

Operations with Floats

Order of Operations

PEMBAS

PEMBAS Rank and Priority

Division and Floor Division

Complex Numbers

Conversion between Numeric types

Important notes on conversions:

The use of Numeric types

Summary

Back to Basics - Pandas #01

Introduction to Pandas

What is Pandas?

Pandas data structures: DataFrames & Series

DataFrame (rows and columns)

Series (one-dimensional data)

Pandas Common Data Types

Pandas Missing Values

Exploratory Data Analysis - EDA

Basic Pandas Data Frame exploration

Importing Pandas (and NumPy)

Checking Top/Bottom rows

Checking random sample rows

Showing the Data Frame Dimensions

Displaying the Data Frame basic info

Generating Descriptive Statistics

Sorting Values

Summary

Back to Basics - Python #01

Intro

Introduction to Python

Python's naming curiosity

Variables

What is a variable?

Declaring a variable

Variable Naming Rules

Case-sensitiviness

Python Style Guide = PEP 8

Python Reserved words

Python built-in Data Type

Summary

Selecting with `loc()` and `iloc()`

1. `loc()` - Label-Based Data Selection

2. `iloc()` - Integer-Based Data Selection