DEV Community: Fernando

Using pipes in Pandas, a travel of flow and data

Fernando — Fri, 08 Dec 2023 19:54:08 +0000

Pandas library in Python is indeed one of the most significant libraries to learn, especially if you want to dedicate yourself to data science. One of the common challenges people face is managing the cleaning and transformation of data, as well as writing code that is not too long and convoluted.

What is imperative programming?

Imperative programming is a style that focuses on the "how" rather than the "what" detailing the precise instructions for Pandas to execute. It involves explicitly defining the steps required to manipulate and transform data, making it more suitable for tasks that involve data manipulation.

import pandas as pd

df = pd.read_csv('dataset.csv')

df.drop_duplicates(inplace=True)
df.dropna(subset=['column1', 'column2'], inplace=True)
try:
    df['date'] = pd.to_datetime(df['date'])
except ValueError:
    print("Invalid dates found, dropping rows...")
    bad_dates = df[pd.to_datetime(df['date'], errors='coerce').index[pd.to_datetime(df['date'], errors='coerce').isnull()]]
    df.drop(bad_dates.index, inplace=True)

df.rename({'old_name': 'new_name'}, axis='columns', inplace=True)

It is a straightforward approach for beginners, but it can become cumbersome and less maintainable for complex data manipulation tasks. As the number of steps and data volume increase, the code can become lengthy and difficult to understand

Programming with a functional approach

Let's explore the steps involved in creating a cleaning and transformation process for a small CSV file. This will allow us to track all the changes we make to the data and ensure that everything is in order. By doing so, we can minimize the risk of errors and ensure that our data is accurate and reliable.

Id,Age,Name,Genre Likes,Height,Weight
12345,20,John Doe,Rock,5'10",150
23456,21,Jane Smith,Pop,5'7",130
34567,22,Alice Johnson,Country,5'9",160
45678,23,Bob Brown,Hip Hop,6'0",180
56789,24,Charlie Wilson,Classical,5'8",140
67890,25,Diana Ross,Jazz,5'11",170
78901,26,Elvis Presley,Blues,5'11",185
89012,27,Mariah Carey,R&B,5'9",175
90123,28,Michael Jackson,Soul,5'10",165
101234,,Madonna,Dance,5'7",145
111235,30,Lady Gaga,Electronic,5'1",120
121236,31,Katy Perry,Indie,5'8",135
131237,32,Taylor Swift,Folk,5'10",155
141238,33,Beyonce,Alternative,5'7",160
151239,34,Pink,Metal,5'7",140
161240,35,Justin Bieber,,5'9",
171241,36,Selena Gomez,Reggae,5'5",130
181242,37,Ed Sheeran,Ska,5'8",160
191243,38,Ariana Grande,Funk,5'0",125
201244,39,Dua Lipa,Disco,5'8",145
211245,40,Billie Eilish,Latin,5'3",115
221246,41,The Weeknd,K-Pop,5'11",175
231247,42,Olivia Rodrigo,J-Pop,5'6",130
241248,43,Imagine Dragons,Vocal,5'11",180
251249,44,Maroon 5,,6'0",190

As you can see, there are missing data in the dataset. Therefore, we will need to clean the data before using it. Here is the code that we will use for the cleaning process.

import pandas as pd

def clean_missed_data(df):
    return df.dropna(inplace=True)

def filter_by_age(df):
    return df[df["Age"] <= 40]


data = pd.read_csv("../data/data.csv")

cleaned_df = data.pipe(clean_missed_data).pipe(filter_by_age)

The given code demonstrates how to use Pandas' pipe() function. It's a nifty trick that lets you chain multiple transformations on a DataFrame without creating extra variables. It's like a flow of functions that flow one after another, simplifying your code.
By following this approach, we avoid cluttering the code with extra variables and assignment statements.

To further illustrate the functionality of the provided code snippets, we will now add two additional functions that convert feet to meters and pounds to kilograms.

def inch_to_meters(x):
    if x is np.NaN:
        return x
    else:
        ft, inc = x.split("'")
        inches = inc.split("\"")[0]
        return ((12 * int(ft)) + int(inches)) * 0.0254 

def convert_feet_to_meters(df):
    df['Height'] = df['Height'].apply(inch_to_meters)
    return df

def convert_pounds_to_kilograms(df):
    df['Weight'] = df['Weight'] * 0.453592
    return df


universal_df  = cleaned_df.pipe(convert_feet_to_meters).pipe(convert_pounds_to_kilograms)

When using the pipe function in Pandas, it is essential to be aware that it modifies the original dataframe. To avoid changing it, you can use a copy in the function

Type coercion or why 6 + True is 7 in Python

Fernando — Sat, 21 Oct 2023 17:40:16 +0000

Type Coercion

In the landscape of programming languages, many of them have a feature called Type coercion. This means that the compiler/VM/interpreter will implicitly convert one object to another type depending on circumstances.
By default, Python does not have this feature. This lack of type coercion in certain situations can lead to unexpected results or behavior that may surprise some developers. Let's investigate these cases and circumstances to learn more about it.

Numeric types:

Fraction type and integer:

from fractions import Fraction

print(Fraction(1,2) + 1)  # Output: Fraction(3, 2)
print(1 + Fraction(1,3))  # Output: Fraction(4, 3)
print(Fraction(1,2).__add__(1))  # Output: Fraction(3, 2)

Python works seamlessly with Fractions and integers. However when it comes to floats and integers:

print(1 + 2.3)    # Output: 3.3
x = 1
print(x.__add__(2.3))   # Output: NotImplemented

Integer doesn't know how to behave with a float,
but float numbers do.

x = 2.3
print(x.__add__(1))

In the above example, adding an integer and a float using the '+' operator works fine in Python because it automatically performs type conversion for us behind the scenes. However when we try to use 'add' method explicitly on an integer object with a float value as argument (i.e., x.add(2.3)), it returns 'NotImplemented', indicating that this operation is not supported by default.

Strings

And what about object string?

a = 'test'
>>> a + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to str

So is it possible to hack this kind of behaviour in python?
Yes, we can create a custom string class that allows us to add integers to it using the __add__ dunder method. This is an example of metaprogramming in Python.
Here's an example code snippet:

class CoolStr(str):
    def __add__(self, x: int):
        return self + str(x)

a = CoolStr('test')
print(a + 1)  # output: 'test1'

In this example, we define a custom string class CoolStr that inherits from the built-in str class. We override the add method to return a new string that is the concatenation of the current string and the converted integer.
Note that we use the str(x) function to convert the integer x to a string, so that we can concatenate it with the original string.
This code will output 'test1', which demonstrates that we have successfully "hacked" the string class to allow integer addition.

However, it's important to note that this is not a recommended way to modify the behavior of a built-in class like str. It's better to use the built-in str.format() method or the + operator with strings and integers, as it's more readable and maintainable.

Boolean

What about the example in the title? Well, as you may know or infer, when a boolean value is cast to an integer, True is converted to 1 and False is converted to 0. When we perform arithmetic operations on a boolean value and an integer, type coercion converts the boolean value to an integer, making the magic happen.

Conclusion

I hope this information was useful for you. In short, Python is a powerful language with workarounds about type coercion.
Finally, there are libraries, like Numpy, which can help handle strong typing in case your work involves strict data type.