DEV Community: Glenn Viroux

Unlocking Python's Design Patterns: Exploring Powerful Solutions with Examples

Glenn Viroux — Fri, 14 Jul 2023 14:46:16 +0000

Design Patterns are invaluable solutions for common problems in software design. While they may not provide ready-made solutions that can be directly copied and pasted into your code, they serve as powerful blueprints that can be customized and tailored to address specific design challenges.

One of the significant advantages of using design patterns is the clarity they bring to your codebase. By leveraging well-established patterns, you effectively communicate to your fellow developers the problem you're addressing and the approach you're taking. It's often said that code is written once but read many times, highlighting the importance of code readability. In the software design process, placing greater emphasis on code readability over ease of writing is essential. Incorporating battle-tested design patterns into your codebase is a highly effective means of enhancing code readability.

In this article, we'll delve into three distinct design patterns and provide detailed examples of how they can be applied in Python. By understanding and harnessing these patterns, you'll be equipped with powerful tools to enhance the structure, maintainability, and clarity of your code.

Decorator

The decorator design pattern enables you to wrap existing objects in special wrapper objects, adding new behavior without modifying the original object.

To illustrate this pattern, imagine yourself on a cold winter day, walking outside in a t-shirt. As you start feeling the chill, you decide to put on a sweater, acting as your first decorator. However, if you're still cold, you might also choose to wrap yourself in a jacket, serving as another decorator. These additional layers "decorate" your basic behavior of wearing just a t-shirt. Importantly, you have the flexibility to remove any garment whenever you no longer need it, without altering your fundamental state.

By employing the decorator pattern, you can dynamically enhance objects with new functionality while keeping the core object unchanged. It offers a flexible and modular approach to extending behavior, much like adding or removing layers of clothing based on your comfort level.

Different Coffee Products, the Wrong Way

Imagine you're tasked with developing a coffee ordering system for a local coffee shop. The system should support various types of coffee with different flavors and toppings. Let's consider the following coffee variations:

Simple coffee
Coffee with milk
Coffee with vanilla topping
Coffee with milk and vanilla topping

Without knowing about the decorator design pattern, you might just implement all different types of coffee with their toppings as a separate variable.

In this approach, we start by defining an interface for all types of coffee. This interface specifies common properties such as the coffee's name, price, list of ingredients, and whether it's vegan-friendly. We also include an order method to allow customers to request a specific coffee and specify the quantity.

from dataclasses import dataclass

@dataclass
class Coffee:
    name: str
    price: float
    ingredients: list[str]
    is_vegan: bool

    def order(self, amount: int) -> str:
        if amount == 1:
            return f"Ordered 1 '{self.name}' for the price of {self.price}€"
        return f"Ordered {amount} times a '{self.name}' for the total price of {self.price * amount:.2f}€"SIMPLE_COFFEE = Coffee(name="Simple Coffee", price=1.0, ingredients=["Coffee"], is_vegan=True)

Next, let’s define our four different types of coffee:

SIMPLE_COFFEE = Coffee(name="Simple Coffee", price=1.0, ingredients=["Coffee"], is_vegan=True)
SIMPLE_COFFEE_WITH_MILK = Coffee(name="Simple Coffee + Milk", price=1.2, ingredients=["Coffee", "Milk"], is_vegan=False)
SIMPLE_COFFEE_WITH_VANILLA = Coffee(name="Simple Coffee + Vanilla", price=1.3, ingredients=["Coffee", "Vanilla"], is_vegan=True)
SIMPLE_COFFEE_WITH_MILK_AND_VANILLA = Coffee(name="Simple Coffee + Milk + Vanilla", price=1.5, ingredients=["Coffee", "Milk", "Vanilla"], is_vegan=False)

Now, a client can place an order as follows:

orders = [
    SIMPLE_COFFEE.order(amount=1),
    SIMPLE_COFFEE_WITH_MILK.order(amount=3),
    SIMPLE_COFFEE_WITH_MILK_AND_VANILLA.order(amount=2)
]
for order in orders:
    print(order)

# Output:
# Ordered 1 'Simple Coffee' for the price of 1.0€
# Ordered 3 times a 'Simple Coffee + Milk' for the total price of 3.60€
# Ordered 2 times a 'Simple Coffee + Milk + Vanilla topping' for the total price of 3.00€

You’ll quickly see how this code will grow exponentially when new types of coffee and more combinations with different types of toppings and flavors. Instead of following this dark spiral of infinite code, let’s explore how decorators can help us out here.

Different Coffee Products, the Right Way

As we discussed earlier, the decorator pattern allows us to wrap a simple base coffee object with additional functionality to create the desired final coffee product. In this improved approach, we'll leverage the power of decorators to add various toppings and flavors to our base coffee.

We'll begin by maintaining our Coffee interface class and our SIMPLE_COFFEE variable as the foundation for all coffee variations. However, instead of defining separate classes for each type of coffee, we'll implement decorators that wrap around our base coffee object.

To get started, let's define a basic interface for our coffee decorators. This interface will ensure consistency among all decorator classes and provide a common set of methods to work with.

from abc import ABC
from dataclasses import dataclass

@dataclass
class BaseCoffeeDecorator(ABC):
    coffee: Coffee

    @property
    @abstractmethod
    def extra_cost(self) -> float:
        raise NotImplementedError

    @property
    @abstractmethod
    def extra_name(self) -> str:
        raise NotImplementedError

    @property
    @abstractmethod
    def extra_ingredients(self) -> list[str]:
        raise NotImplementedError

    def __call__(self) -> Coffee:
        name = f"{self.coffee.name} + {self.extra_name}"
        price = self.coffee.price + self.extra_cost
        ingredients = self.coffee.ingredients + self.extra_ingredients
        is_vegan = self.coffee.is_vegan and not any(
            ingredient in NON_VEGAN_INGREDIENTS for ingredient in self.extra_ingredients
        )
        return replace(self.coffee, name=name, price=price, ingredients=ingredients, is_vegan=is_vegan)

Here, we say how each coffee decorator should define an extra cost, an extra name and possible extra ingredients that will be added to the coffee.

By implementing specific decorator classes, such as MilkDecorator or VanillaDecorator, we can easily add desired toppings or flavors to our coffee. Each decorator class will encapsulate the base coffee object and modify its behavior by adding the desired functionality.

class MilkDecorator(BaseCoffeeDecorator):
    extra_name = "Milk"
    extra_cost = 0.2
        extra_ingredients = ["Milk"]

class VanillaDecorator(BaseCoffeeDecorator):
    extra_name = "Vanilla"
    extra_cost = 0.3
        extra_ingredients = ["Vanilla"]

And our client can place the order like follows:

coffee_with_milk = MilkDecorator(SIMPLE_COFFEE)()
coffee_with_milk_and_vanilla = VanillaDecorator(MilkDecorator(SIMPLE_COFFEE)())()
orders = [
    SIMPLE_COFFEE.order(amount=1),
    coffee_with_milk.order(amount=3),
    coffee_with_milk_and_vanilla.order(amount=2),
]

for order in orders:
    print(order)

# Output:
# Ordered 1 'Simple Coffee' for the price of 1.0€
# Ordered 3 times a 'Simple Coffee + Milk' for the total price of 3.60€
# Ordered 2 times a 'Simple Coffee + Milk + Vanilla' for the total price of 3.00€

The significant advantage of this approach is that you only need to define one decorator class per type of topping or flavor, rather than creating subclasses for every possible combination. Consider the scenario where customers can combine any topping or flavor with any other topping or flavor. In this case, with just 10 different types of toppings, there would be a whopping 1023 possible combinations. It would be impractical and cumbersome to create 1023 variables for each combination.

By adopting this approach, we achieve a more flexible and modular design. New toppings or flavors can be added by creating additional decorator classes, without the need to modify the existing coffee classes. This allows for easy customization and expansion of our coffee offerings.

In summary, by utilizing the decorator pattern, we create a cohesive and extensible coffee ordering system where various combinations of toppings and flavors can be applied to our base coffee, resulting in a delightful and customizable coffee experience for our customers.

Different Coffee Products, Done Right with Python Decorators

Taking our coffee ordering system to the next level, we can leverage the powerful built-in decorator functionality offered by Python itself. With this approach, we can achieve the same functionality as before while embracing the elegance and simplicity of Python decorators.

Instead of creating separate decorator classes like MilkDecorator and VanillaDecorator, we can utilize the @ symbol and apply decorators directly to our coffee functions. This not only streamlines the code but also enhances its readability and maintainability.

Let's dive into an example of how we can achieve the same results using built-in Python decorators:

from typing import Callable
from functools import wraps

def milk_decorator(func: Callable[[], Coffee]) -> Callable[[], Coffee]:
    @wraps(func)
    def wrapper() -> Coffee:
        coffee = func()
        return replace(coffee, name=f"{coffee.name} + Milk", price=coffee.price + 0.2)

    return wrapper

def vanilla_decorator(func: Callable[[], Coffee]) -> Callable[[], Coffee]:
    @wraps(func)
    def wrapper() -> Coffee:
        coffee = func()
        return replace(coffee, name=f"{coffee.name} + Vanilla", price=coffee.price + 0.3)

    return wrapper

@milk_decorator
def make_coffee_with_milk():
    return SIMPLE_COFFEE

@vanilla_decorator
@milk_decorator
def make_coffee_with_milk_and_vanilla():
    return SIMPLE_COFFEE

# Output:
# Ordered 1 'Simple Coffee' for the price of 1.0€
# Ordered 3 times a 'Simple Coffee + Milk' for the total price of 3.60€
# Ordered 2 times a 'Simple Coffee + Milk + Vanilla' for the total price of 3.00€

Notice how here, we achieved the exact same result as before, but instead of using our BaseCoffeeDecorator class, we used python’s built-in decorator functionality by using the wraps function from the functools package (https://docs.python.org/3/library/functools.html#functools.wraps).

Chain of Responsability

The Chain of Responsibility (CoR) design pattern provides a flexible and organized approach to handle sequential operations or requests on an object by passing it through a series of handlers. Each handler in the chain has the ability to perform specific actions or checks on the object and make a decision to either process the request or delegate it to the next handler in line.

Imagine you, as an employee, want to have a second monitor for your work setup. To make this happen, you need to go through a chain of approval. First, you submit a request to your immediate team lead, who evaluates whether it aligns with the department's policies and budget. If your team lead approves, they pass the request to the finance department, which verifies the availability of funds. The finance department might then consult with other relevant departments, such as procurement or IT infrastructure, to ensure the request can be fulfilled. Eventually, the request reaches the final decision-maker, such as the department head or the finance director, who makes the ultimate decision.

In this scenario, each level of approval corresponds to a link in the chain of responsibility. Each person in the chain has a specific responsibility and authority to handle their part of the request. The chain allows for a structured and sequential flow of information and decision-making, ensuring that each level of the organization's hierarchy is involved and has the opportunity to contribute to the final decision.

Performing Checks on Coffee Orders

Let's delve further into our coffee ordering system example and explore additional checks that can be applied. In a real-life scenario, the system could become more intricate than our current implementation. Imagine the coffee place owner specifying that no single coffee should cost more than 10€ to maintain an affordable brand reputation. Additionally, there should be a mechanism in place to double-check the vegan status of a coffee based on its ingredients.

For this section, we’ll define three different types of coffee: a simple coffee, a cappuccino and an expensive cappuccino:

SIMPLE_COFFEE = Coffee(name="Simple Coffee", price=1.0, ingredients=["Coffee"], is_vegan=True)
CAPPUCCINO = Coffee(name="Cappuccino", price=2.0, ingredients=["Coffee", "Milk"], is_vegan=True)
EXPENSIVE_CAPPUCCINO = Coffee(name="Cappuccino", price=12.0, ingredients=["Coffee", "Milk"], is_vegan=False)

Notice how we marked the cappuccino as vegan, while milk is included in its ingredients. Also, the price of the expensive cappuccino is higher than 10€.

Next, we establish a unified interface for all the handlers in our Chain of Responsibility (CoR):

@dataclass
class BaseHandler(ABC):
    next_handler: BaseHandler | None = None

    @abstractmethod
    def __call__(self, coffee: Coffee) -> Coffee:
        raise NotImplementedError

In this code snippet, we define the BaseHandler class as an abstract base class (ABC) using the @dataclass decorator. The class includes a next_handler attribute that represents the next handler in the chain. Each handler in the CoR implements this interface and overrides the __call__ method to define its specific action or check on the coffee object.

If a next_handler is provided when creating a handler instance, it signifies that there is another handler to process the coffee object after the current handler completes its operation. Conversely, if no next handler is provided, the current handler serves as the endpoint of the chain.

This common interface ensures that all handlers adhere to a consistent structure, allowing for seamless chaining and flexibility in adding, removing, or rearranging the handlers as needed.

Now we can go on to define our two handlers that will check for the maximum price of €10, and will verify whether no coffee is marked as vegan incorrectly:

NON_VEGAN_INGREDIENTS = ["Milk"]

@dataclass
class MaximumPriceHandler(BaseHandler):
    def __call__(self, coffee: Coffee) -> Coffee:
        if coffee.price > 10.0:
            raise RuntimeError(f"{coffee.name} costs more than €10?!")

        return coffee if self.next_handler is None else self.next_handler(coffee)

@dataclass
class VeganHandler(BaseHandler):
    def __call__(self, coffee: Coffee) -> Coffee:
        if coffee.is_vegan and any(ingredient in NON_VEGAN_INGREDIENTS for ingredient in coffee.ingredients):
            raise RuntimeError(f"Coffee {coffee.name} is said to be vegan but contains non-vegan ingredients")

        if not coffee.is_vegan and all(ingredient not in NON_VEGAN_INGREDIENTS for ingredient in coffee.ingredients):
            raise RuntimeError(f"Coffee {coffee.name} is not not labelled as vegan when it should be")

        return coffee if self.next_handler is None else self.next_handler(coffee)

Let’s test our handlers with the following code:

handlers = MaximumPriceHandler(VeganHandler())

try:
    cappuccino = handlers(CAPPUCCINO)
except RuntimeError as err:
    print(str(err))

try:
    cappuccino = handlers(EXPENSIVE_CAPPUCCINO)
except RuntimeError as err:
    print(str(err))

# Output:
# Coffee Cappuccino is said to be vegan but contains non-vegan ingredients
# Expensive Cappuccino costs more than €10?!

We observed how the ordering system correctly handles the addition of toppings to a simple coffee order. However, attempting to create a Cappuccino or an ExpensiveCappuccino order results in an exception being raised. This behavior highlights the strict processing logic enforced by the chain of responsibility.

What's noteworthy is how easily we can extend this code by defining additional handlers to perform other specific operations on coffee orders. For instance, let's say you want to offer a 10% discount on takeaway orders. You can effortlessly create a new handler and add it to the chain. This handler would reduce the price of the order by 10% if it's flagged as a takeaway order.

One of the key advantages of the Chain of Responsibility design pattern is its adherence to the Open-Closed principle in software development. The existing handlers remain closed for modification, promoting code stability and reusability. However, the design pattern allows for easy expansion by introducing new handlers to the chain when necessary. This flexibility empowers developers to accommodate changing requirements without disrupting the existing codebase.

Composite / Object Tree

The Composite design pattern comes into play when you're faced with a scenario involving a mixture of simple end objects (leaves) and more complex containers that can hold other containers (branches) or end objects (leaves).

A real-world analogy of the Composite pattern can be found in a file system. Consider a directory structure where folders (branches) can contain other folders or files (leaves), while individual files (leaves) exist independently. This hierarchical arrangement allows you to treat the entire structure as a unified object, regardless of whether you're working with a single file or a complex directory.

Composite Coffee Orders

In our coffee ordering system, we can apply the Composite design pattern to handle both simple individual coffee orders (leaves) and more complex coffee order structures (branches).

Consider a scenario where a single coffee order represents an individual line on the bill, such as "Two simple coffees for a total price of €2." However, we also need to accommodate complex coffee orders that consist of multiple individual orders or even other complete coffee orders. For instance, customers ordering from the terrace might decide to add additional items to their existing processed order.

To manage this complexity, we can utilize the Composite design pattern, which provides a unified interface for both individual coffee orders and composite order structures. This common interface defines essential methods, like calculating the total price of the final order, regardless of whether it's a single coffee or a complex combination.

By employing the Composite pattern, we can streamline the handling of coffee orders, ensure consistent operations across different order types, and enable seamless integration of new functionalities into the coffee ordering system.

from dataclasses import dataclass

@dataclass
class Coffee:
    name: str
    price: float
    ingredients: list[str]
    is_vegan: bool

SIMPLE_COFFEE = Coffee(name="Simple Coffee", price=1.0, ingredients=["Coffee"], is_vegan=True)
CAPPUCCINO = Coffee(name="Cappuccino", price=2.0, ingredients=["Coffee", "Milk"], is_vegan=False)

class CoffeeOrderComponentBase(ABC):
    @property
    @abstractmethod
    def total_price(self) -> float:
        raise NotImplementedError

    @property
    @abstractmethod
    def all_ingredients(self) -> list[str]:
        raise NotImplementedError

    @property
    @abstractmethod
    def is_vegan(self) -> bool:
        raise NotImplementedError

    @property
    @abstractmethod
    def order_lines(self) -> list[str]:
        raise NotImplementedError

Here, we have defined our Coffee data class, which specifies the properties required for each type of base coffee. We have created two instances of base coffees, namely SIMPLE_COFFEE and CAPPUCCINO.

Next, we introduce the CoffeeOrderComponentBase, which serves as the common interface for both leaves (single coffee orders) and complex containers (composite coffee orders). This interface defines the following methods that should be implemented by both types:

total_price: Calculates the total price of the order.
all_ingredients: Retrieves all the ingredients included in the order.
is_vegan: Indicates whether the complete order is vegan or not.
order_lines: Generates a summary of the order as text lines.

Now, let's focus on the implementation of the leaf component, which represents a single coffee order:

@dataclass
class CoffeeOrder(CoffeeOrderComponentBase):
    base_coffee: Coffee
    amount: int

    @property
    def total_price(self) -> float:
        return self.amount * self.base_coffee.price

    @property
    def all_ingredients(self) -> list[str]:
        return self.base_coffee.ingredients

    @property
    def is_vegan(self) -> bool:
        return self.base_coffee.is_vegan

    @property
    def order_lines(self) -> list[str]:
        if self.amount == 1:
            return [f"Ordered 1 '{self.base_coffee.name}' for the price of {self.total_price}€"]
        return [
            f"Ordered {self.amount} times a '{self.base_coffee.name}' for the total price of {self.total_price:.2f}€"
        ]

Now, let's move on to the implementation of the more complex coffee order, which is capable of holding a list of children. This allows for the nesting of both leaves (single coffee orders) and other complex containers.

from dataclasses import field

from more_itertools import flatten

@dataclass
class CompositeCoffeeOrder(CoffeeOrderComponentBase):
    children: list[CoffeeOrderComponentBase] = field(default_factory=list)

    @property
    def total_price(self) -> float:
        return sum(child.total_price for child in self.children)

    @property
    def all_ingredients(self) -> list[str]:
        return list(set(flatten([child.all_ingredients for child in self.children])))

    @property
    def is_vegan(self) -> bool:
        return all(child.is_vegan for child in self.children) or not len(self.children)

    @property
    def order_lines(self) -> list[str]:
        return list(flatten([child.order_lines for child in self.children]))

Now we can represent a complex, composite order like this:

order = CompositeCoffeeOrder(
    children=[
        CoffeeOrder(amount=2, base_coffee=CAPPUCCINO),
        CoffeeOrder(amount=1, base_coffee=SIMPLE_COFFEE),
        CompositeCoffeeOrder(
            children=[CoffeeOrder(amount=3, base_coffee=SIMPLE_COFFEE)]
        ),
    ]
)

for order_line in order.order_lines:
    print(order_line)

print("-" * 40)
print(f"The total price of the order is {order.total_price:.2f}€")
print(f"These are all the ingredients included in this order: {', '.join(order.all_ingredients)}")

# Output:
# Ordered 2 times a 'Cappuccino' for the total price of 4.00€
# Ordered 1 'Simple Coffee' for the price of 1.0€
# Ordered 3 times a 'Simple Coffee' for the total price of 3.00€
# ----------------------------------------
# The total price of the order is 8.00€
# These are all the ingredients included in this order: Milk, Coffee

Combining Different Design Patterns

Now that we have explored the Decorator, Chain of Responsibility, and Composite design patterns individually, let's see how we can bring them all together to build a complex coffee order with different toppings and flavors. This example will demonstrate how we can apply the Chain of Responsibility pattern to perform validation steps on our coffee order while utilizing the Composite pattern to create a structured order hierarchy.

By combining these patterns, we can create a powerful and flexible coffee ordering system that allows customers to customize their orders with various toppings and flavors while ensuring that all validation checks are performed seamlessly.

handlers = MaximumPriceHandler(VeganHandler())
coffee_with_milk_and_vanilla = VanillaDecorator(MilkDecorator(SIMPLE_COFFEE)())()

order = CompositeCoffeeOrder(
    children=[
        CoffeeOrder(amount=2, base_coffee=handlers(CAPPUCCINO)),
        CoffeeOrder(amount=1, base_coffee=handlers(coffee_with_milk_and_vanilla)),
        CompositeCoffeeOrder(
            children=[CoffeeOrder(amount=3, base_coffee=handlers(VanillaDecorator(CAPPUCCINO)()))]
        ),
    ]
)

for order_line in order.order_lines:
    print(order_line)

print("-" * 40)
print(f"The total price of the order is {order.total_price:.2f}€")
print(f"These are all the ingredients included in this order: {', '.join(order.all_ingredients)}")
print(f"This order is {'' if order.is_vegan else 'not'} vegan")

# Output:
# Ordered 2 times a 'Cappuccino' for the total price of 4.00€
# Ordered 1 'Simple Coffee + Milk + Vanilla' for the price of 1.5€
# Ordered 3 times a 'Cappuccino + Vanilla' for the total price of 6.90€
# ----------------------------------------
# The total price of the order is 12.40€
# These are all the ingredients included in this order: Coffee, Vanilla, Milk
# This order is not vegan

First, we define our Chain of Responsibility, which consists of the VeganHandler is responsible for checking if any product is incorrectly labeled as vegan, and the MaximumPriceHandler responsible for verifying that no single coffee exceeds the price of 10€.

Next, we utilize the VanillaDecorator and MilkDecorator to transform a simple coffee into a coffee with milk and vanilla, respectively.

Finally, we employ the CompositeCoffeeOrder to create an order that includes two single coffee orders and another complex order.

When we run the script, we can observe the decorators modifying the names and prices of the different orders. The CompositeCoffeeOrder correctly calculates the total price of the final order. Additionally, we can view the complete list of ingredients and determine whether the entire order is vegan or not.

Conclusion

In conclusion, design patterns play a crucial role in software development, offering solutions to common problems and promoting code reusability, flexibility, and maintainability. In this blog post, we explored three powerful design patterns in Python: Decorator, Chain of Responsibility, and Composite.

The Decorator pattern allowed us to dynamically add new behavior to existing objects, demonstrating its usefulness in extending the functionality of coffee orders with various toppings and flavors. We learned how decorators can be implemented both with custom wrapper classes and Python's built-in decorator functionality, providing flexible options for code organization.

The Chain of Responsibility pattern proved valuable in performing sequential operations on coffee orders, emulating a real-world scenario of approval processes in an organization. By creating a chain of handlers, each responsible for a specific task, we achieved modularity and extensibility while ensuring that requests pass through the chain correctly.

The Composite pattern enabled us to create structured hierarchies of coffee orders, incorporating both simple orders and more complex compositions. By defining a common interface, we achieved consistency in accessing and manipulating orders, regardless of their complexity.

Throughout our examples, we witnessed the power of design patterns in enhancing code readability, maintainability, and extensibility. By adopting these proven solutions, we can improve collaboration among developers and build software systems that are robust, flexible, and adaptable to changing requirements.

You can find all the source code provided in this blog post here: https://github.com/GlennViroux/design-patterns-blog

Feel free to contact me with any questions or comments!

References

https://refactoring.guru/design-patterns/composite

https://refactoring.guru/design-patterns/chain-of-responsibility

https://refactoring.guru/design-patterns/decorator

Mastering MultiIndexes in Pandas: A Powerful Tool for Complex Data Analysis

Glenn Viroux — Sun, 18 Jun 2023 16:49:01 +0000

Pandas is a widely used data manipulation library in Python that offers extensive capabilities for handling various types of data. One of its notable features is the ability to work with MultiIndexes, also known as hierarchical indexes. In this blog post, we will delve into the concept of MultiIndexes and explore how they can be leveraged to tackle complex, multidimensional datasets.

Understanding MultiIndexes: Analyzing Sports Performance Data

A MultiIndex is a pandas data structure that allows indexing and accessing data across multiple dimensions or levels. It enables the creation of hierarchical structures for rows and columns, providing a flexible way to organize and analyze data. To illustrate this, let's consider a scenario where you are a personal trainer or coach monitoring the health parameters of your athletes during their sports activities. You want to track various parameters such as heart rate, running pace, and cadence over a specific time interval.

Synthetic Health Performance Data

To work with this type of data, let's begin by writing Python code that simulates health performance data, specifically heart rates and running cadences:

from __future__ import annotations
from datetime import datetime, timedelta
import numpy as np
import pandas as pd

start = datetime(2023, 6, 8, 14)
end = start + timedelta(hours=1, minutes=40)
timestamps = pd.date_range(start, end, freq=timedelta(minutes=1), inclusive='left')

def get_heart_rate(begin_hr: int, end_hr: int, break_point: int) -> pd.Series[float]:
    noise = np.random.normal(loc=0.0, scale=3, size=100)
    heart_rate = np.concatenate((np.linspace(begin_hr, end_hr, num=break_point), [end_hr] * (100 - break_point))) + noise
    return pd.Series(data=heart_rate, index=timestamps)

def get_cadence(mean_cadence: int) -> pd.Series[float]:
    noise = np.random.normal(loc=0.0, scale=1, size=100)
    cadence = pd.Series(data=[mean_cadence] * 100 + noise, index=timestamps)
    cadence[::3] = np.NAN
    cadence[1::3] = np.NAN
    return cadence.ffill().fillna(mean_cadence)

The code snippet provided showcases the generation of synthetic data for heart rate and cadence during a sports activity. It begins by importing the necessary modules such as datetime, numpy, and pandas.

The duration of the sports activity is defined as 100 minutes, and the pd.date_range function is utilized to generate a series of timestamps at one-minute intervals to cover this period.

The get_heart_rate function generates synthetic heart rate data, assuming a linear increase in heart rate up to a certain level, followed by a constant level for the remainder of the activity. Gaussian noise is introduced to add variability to the heart rate data, making it more realistic.

Similarly, the get_cadence function generates synthetic cadence data, assuming a relatively constant cadence throughout the activity. Gaussian noise is added to create variability in the cadence values, with the noise values being updated every three minutes instead of every minute, reflecting the stability of cadence compared to heart rates.

With the data generation functions in place, it is now possible to create synthetic data for two athletes, Bob and Alice:

bob_hr = get_heart_rate(begin_hr=110, end_hr=160, break_point=20)
alice_hr = get_heart_rate(begin_hr=90, end_hr=140, break_point=50)
bob_cadence = get_cadence(mean_cadence=175)
alice_cadence = get_cadence(mean_cadence=165)

At this point, we have the heart rates and cadences of Bob and Alice. Let's plot them using matplotlib to get some more insight into the data:

from __future__ import annotations
import matplotlib.dates as mdates
import matplotlib.pyplot as plt

date_formatter = mdates.DateFormatter('%H:%M:%S')  # Customize the date format as needed

fig = plt.figure(figsize=(12, 6))
ax = fig.add_subplot(111)
ax.xaxis.set_major_formatter(date_formatter)
ax.plot(bob_hr, color="red", label="Heart Rate Bob", marker=".")
ax.plot(alice_hr, color="red", label="Heart Rate Alice", marker="v")
ax.grid()
ax.legend()
ax.set_ylabel("Heart Rate [BPM]")
ax.set_xlabel("Time")

ax_cadence = ax.twinx()
ax_cadence.plot(bob_cadence, color="purple", label="Cadence Bob", marker=".", alpha=0.5)
ax_cadence.plot(alice_cadence, color="purple", label="Cadence Alice", marker="v", alpha=0.5)
ax_cadence.legend()
ax_cadence.set_ylabel("Cadence [SPM]")
ax_cadence.set_ylim(158, 180)

Great! The initial analysis of the data provides interesting observations. We can easily distinguish the differences between Bob and Alice in terms of their maximum heart rate and the rate at which it increases. Additionally, Bob's cadence appears to be notably higher than Alice's.

Using Dataframes for Scalability

However, as you might have already noticed, the current approach of using separate variables (bob_hr, alice_hr, bob_cadence, and alice_cadence) for each health parameter and athlete is not scalable. In real-world scenarios with a larger number of athletes and health parameters, this approach quickly becomes impractical and cumbersome.

To address this issue, we can leverage the power of pandas by utilizing a pandas DataFrame to represent the data for multiple athletes and health parameters. By organizing the data in a tabular format, we can easily manage and analyze multiple variables simultaneously.

Each row of the DataFrame can correspond to a specific timestamp, and each column can represent a health parameter for a particular athlete. This structure allows for efficient storage and manipulation of multidimensional data.

By using a DataFrame, we can eliminate the need for separate variables and store all the data in a single object. This enhances code clarity, simplifies data handling, and provides a more intuitive representation of the overall dataset.

bob_df = pd.concat([bob_hr.rename("heart_rate"), bob_cadence.rename("cadence")], axis="columns")

This is what the Dataframe for Bob’s health data looks like:

	heart_rate	cadence
2023-06-08 14:00:00	112.359	175
2023-06-08 14:01:00	107.204	175
2023-06-08 14:02:00	116.617	175.513
2023-06-08 14:03:00	121.151	175.513
2023-06-08 14:04:00	123.27	175.513
2023-06-08 14:05:00	120.901	174.995
2023-06-08 14:06:00	130.24	174.995
2023-06-08 14:07:00	131.15	174.995
2023-06-08 14:08:00	131.402	174.669

Introducing Hierarchical Dataframes

The last dataframe looks better already! But now we still have to create a new dataframe for each athlete. This is where pandas MultiIndex can help. Let's take a look at how we can elegantly merge the data of multiple athletes and health parameters into one dataframe:

from itertools import product
bob_df = bob_hr.to_frame("value")
bob_df["athlete"] = "Bob"
bob_df["parameter"] = "heart_rate"

values = {
    "Bob": {
        "heart_rate": bob_hr,
        "cadence": bob_cadence,
    },
    "Alice": {
        "heart_rate": alice_hr,
        "cadence": alice_cadence
    }
}

sub_dataframes: list[pd.DataFrame] = []
for athlete, parameter in product(["Bob", "Alice"], ["heart_rate", "cadence"]):
    sub_df = values[athlete][parameter].to_frame("values")
    sub_df["athlete"] = athlete
    sub_df["parameter"] = parameter
    sub_dataframes.append(sub_df)

df = pd.concat(sub_dataframes).set_index(["athlete", "parameter"], append=True)
df.index = df.index.set_names(["timestamps", "athlete", "parameter"])

This code processes heart rate and cadence data for athletes Bob and Alice. It performs the following steps:

Create a DataFrame for Bob's heart rate data and add metadata columns for athlete and parameter.
Define a dictionary that stores heart rate and cadence data for Bob and Alice.
Generate combinations of athletes and parameters (Bob/Alice and heart_rate/cadence).
For each combination, create a sub-dataframe with the corresponding data and metadata columns.
Concatenate all sub-dataframes into a single dataframe.
Set the index to include levels for timestamps, athlete, and parameter. This is where the actual MultiIndex is created

This is what the hierarchical dataframe df looks like:

	values
(Timestamp('2023-06-08 14:00:00'), 'Bob', 'heart_rate')	112.359
(Timestamp('2023-06-08 14:01:00'), 'Bob', 'heart_rate')	107.204
(Timestamp('2023-06-08 14:02:00'), 'Bob', 'heart_rate')	116.617
(Timestamp('2023-06-08 14:03:00'), 'Bob', 'heart_rate')	121.151
(Timestamp('2023-06-08 14:04:00'), 'Bob', 'heart_rate')	123.27
(Timestamp('2023-06-08 14:05:00'), 'Bob', 'heart_rate')	120.901
(Timestamp('2023-06-08 14:06:00'), 'Bob', 'heart_rate')	130.24
(Timestamp('2023-06-08 14:07:00'), 'Bob', 'heart_rate')	131.15
(Timestamp('2023-06-08 14:08:00'), 'Bob', 'heart_rate')	131.402

At this point, we have got ourselves a single dataframe that holds all information for an arbitrary amount of athletes and health parameters. We can now easily use the .xs method to query the hierarchical dataframe:

df.xs("Bob", level="athlete")  # get all health data for Bob

	values
(Timestamp('2023-06-08 14:00:00'), 'heart_rate')	112.359
(Timestamp('2023-06-08 14:01:00'), 'heart_rate')	107.204
(Timestamp('2023-06-08 14:02:00'), 'heart_rate')	116.617
(Timestamp('2023-06-08 14:03:00'), 'heart_rate')	121.151
(Timestamp('2023-06-08 14:04:00'), 'heart_rate')	123.27

df.xs("heart_rate", level="parameter")  *# get all heart rates*

	values
(Timestamp('2023-06-08 14:00:00'), 'Bob')	112.359
(Timestamp('2023-06-08 14:01:00'), 'Bob')	107.204
(Timestamp('2023-06-08 14:02:00'), 'Bob')	116.617
(Timestamp('2023-06-08 14:03:00'), 'Bob')	121.151
(Timestamp('2023-06-08 14:04:00'), 'Bob')	123.27

df.xs("Bob", level="athlete").xs("heart_rate", level="parameter")  # get heart_rate data for Bob

timestamps	values
2023-06-08 14:00:00	112.359
2023-06-08 14:01:00	107.204
2023-06-08 14:02:00	116.617
2023-06-08 14:03:00	121.151
2023-06-08 14:04:00	123.27

Use Case: Earth Temperature Changes

To demonstrate the power of hierarchical dataframes, let's explore a real-world and complex use case: analyzing the changes in Earth's surface temperatures over the last decades. For this task, we'll utilize a dataset available on Kaggle, which summarizes the Global Surface Temperature Change data distributed by the National Aeronautics and Space Administration Goddard Institute for Space Studies (NASA-GISS).

Inspect and Transform Original Data

Let's begin by reading and inspecting the data. This step is crucial to gain a better understanding of the dataset's structure and contents before delving into the analysis. Here's how we can accomplish that using pandas:

from pathlib import Path

file_path = Path() / "data" / "Environment_Temperature_change_E_All_Data_NOFLAG.csv"
df = pd.read_csv(file_path , encoding='cp1252')
df.describe()

	Area Code	Months Code	Element Code	Y1961	Y1962	Y1963	Y1964	Y1965	Y1966	Y1967	Y1968	Y1969	Y1970	Y1971	Y1972	Y1973	Y1974	Y1975	Y1976	Y1977	Y1978	Y1979	Y1980	Y1981	Y1982	Y1983	Y1984	Y1985	Y1986	Y1987	Y1988	Y1989	Y1990	Y1991	Y1992	Y1993	Y1994	Y1995	Y1996	Y1997	Y1998	Y1999	Y2000	Y2001	Y2002	Y2003	Y2004	Y2005	Y2006	Y2007	Y2008	Y2009	Y2010	Y2011	Y2012	Y2013	Y2014	Y2015	Y2016	Y2017	Y2018	Y2019
count	9656	9656	9656	8287	8322	8294	8252	8281	8364	8347	8345	8326	8308	8303	8323	8394	8374	8280	8209	8257	8327	8290	8283	8276	8237	8205	8259	8216	8268	8284	8273	8257	8239	8158	8354	8315	8373	8409	8439	8309	8370	8324	8342	8241	8312	8390	8415	8424	8503	8534	8475	8419	8435	8437	8350	8427	8377	8361	8348	8366	8349	8365
mean	821.806	7009.88	6674.5	0.402433	0.315527	0.317393	0.269382	0.217839	0.376419	0.263239	0.24487	0.382172	0.365322	0.240934	0.302553	0.427691	0.261849	0.314653	0.221112	0.422978	0.355488	0.442465	0.43827	0.437693	0.404857	0.503748	0.366971	0.365511	0.398096	0.535514	0.546662	0.469231	0.621797	0.499991	0.447798	0.439094	0.611078	0.635836	0.477239	0.617341	0.818264	0.704445	0.674191	0.741673	0.802509	0.769485	0.726237	0.777465	0.791795	0.842554	0.742614	0.814177	0.884504	0.768488	0.78893	0.829647	0.913872	1.01882	1.08149	1.00334	1.01083	1.0946
std	1781.07	6.03825	596.531	0.701567	0.713777	0.853133	0.749216	0.739418	0.73737	0.725421	0.7549	0.725313	0.662412	0.727313	0.765895	0.677769	0.76885	0.723964	0.755587	0.677094	0.662228	0.670377	0.638585	0.70157	0.675693	0.749268	0.700364	0.765491	0.710202	0.756869	0.664038	0.758738	0.724396	0.656754	0.843925	0.81786	0.752511	0.754133	0.72521	0.741812	0.785076	0.723811	0.783754	0.781683	0.840545	0.794839	0.644364	0.720366	0.821489	0.854893	0.852562	0.694464	0.878303	0.750631	0.858586	0.713753	0.815933	0.840189	0.877399	0.8098	0.872199	0.853953
min	1	7001	6078	-4.018	-5.391	-8.483	-7.309	-4.728	-8.147	-6.531	-8.407	-6.784	-5.847	-7.671	-7.722	-4.896	-4.732	-6.169	-4.263	-6.495	-8.228	-6.319	-5.784	-6.591	-4.55	-6.101	-5.437	-8.411	-6.345	-8.75	-8.963	-5.311	-4.03	-4.598	-5.414	-7.389	-5.099	-4.862	-4.027	-4.059	-6.031	-3.035	-3.596	-5.493	-6.17	-6.118	-5.025	-5.171	-4.981	-3.19	-9.334	-3.543	-6.072	-4.854	-5.785	-3.642	-5.367	-4.068	-3.306	-3.584	-2.216	-2.644
25%	78	7005	6078	0.057	-0.033	0.03025	-0.1025	-0.214	0.055	-0.169	-0.164	0.171	0.094	-0.209	-0.028	0.201	-0.184	-0.115	-0.219	0.174	0.091	0.217	0.2455	0.205	0.168	0.268	0.077	0.123	0.164	0.293	0.285	0.186	0.298	0.264	0.18425	0.214	0.289	0.324	0.244	0.305	0.385	0.327	0.3	0.349	0.38	0.37525	0.361	0.373	0.371	0.378	0.337	0.381	0.392	0.365	0.37225	0.4085	0.418	0.437	0.457	0.443	0.434	0.455
50%	153.5	7009	6674.5	0.366	0.333	0.355	0.326	0.303	0.36	0.313	0.312	0.385	0.367	0.305	0.346	0.413	0.305	0.325	0.309	0.388	0.35	0.406	0.424	0.409	0.39	0.468	0.373	0.374	0.38	0.504	0.478	0.404	0.49	0.448	0.419	0.427	0.493	0.539	0.45	0.528	0.715	0.548	0.514	0.596	0.668	0.658	0.597	0.676	0.646	0.667	0.589	0.693	0.771	0.64	0.651	0.719	0.745	0.858	0.949	0.865	0.81	0.939
75%	226.25	7016	7271	0.6765	0.627	0.64775	0.609	0.584	0.66025	0.601	0.595	0.677	0.642	0.5885	0.628	0.709	0.586	0.625	0.586	0.695	0.633	0.69775	0.701	0.709	0.689	0.825	0.666	0.674	0.678	0.854	0.802	0.728	0.8485	0.758	0.777	0.748	0.874	0.905	0.777	0.938	1.188	0.98825	0.937	1.028	1.08425	1.063	0.991	1.104	1.082	1.096	1.0425	1.1345	1.2765	1.091	1.108	1.126	1.19	1.389	1.496	1.36475	1.341	1.508
max	5873	7020	7271	5.771	4.373	4.666	5.233	5.144	5.771	4.768	4.373	4.411	4.373	4.373	9.475	6.304	6.912	6.15	7.689	4.875	5.956	5.483	4.519	6.144	5.411	6.513	9.303	5.948	6.845	4.516	4.982	6.841	9.73	6.11	6.017	5.989	6.477	7.221	7.27	5.637	6.816	7.017	5.836	6.092	8.719	6.171	7.461	7.651	11.331	7.655	8.298	6.415	7.19	6.531	10.826	6.738	11.759	7.59	10.478	7.389	9.228	7.215

From this initial inspection, it becomes evident that the data is organized in a single dataframe, with separate rows for different months and countries. However, the values for different years are spread across several columns in the dataframe, labeled with the prefix 'Y'. This format makes it challenging to read and visualize the data effectively. To address this issue, we will transform the data into a more structured and hierarchical dataframe format, enabling us to query and visualize the data more conveniently.

from dataclasses import dataclass, field
from datetime import date
from pydantic import BaseModel

MONTHS = {
    "January": 1, 
    "February": 2, 
    "March": 3, 
    "April": 4, 
    "May": 5, 
    "June": 6, 
    "July": 7, 
    "August": 8,
  "September": 9, 
    "October": 10,
  "November": 11, 
    "December": 12
}

class GistempDataElement(BaseModel):
    area: str
    timestamp: date
    value: float

@dataclass
class GistempTransformer:
    temperature_changes: list[GistempDataElement] = field(default_factory=list)
    standard_deviations: list[GistempDataElement] = field(default_factory=list)

    def _process_row(self, row) -> None:
        relevant_elements = ["Temperature change", "Standard Deviation"]
        if (element := row["Element"]) not in relevant_elements or (month := MONTHS.get(row["Months"])) is None:
            return None

        for year, value in row.filter(regex="Y.*").items():
            new_element = GistempDataElement(
                timestamp=date(year=int(year.replace("Y", "")), month=month, day=1),
                area=row["Area"],
                value=value
            )
            if element == "Temperature change":
                self.temperature_changes.append(new_element)
            else:
                self.standard_deviations.append(new_element)

    @property
    def df(self) -> pd.DataFrame:
        temp_changes_df = pd.DataFrame.from_records([elem.dict() for elem in self.temperature_changes])
        temp_changes = temp_changes_df.set_index(["timestamp", "area"]).rename(columns={"value": "temp_change"})

        std_deviations_df = pd.DataFrame.from_records([elem.dict() for elem in self.standard_deviations])
        std_deviations = std_deviations_df.set_index(["timestamp", "area"]).rename(columns={"value": "std_deviation"})

        return pd.concat([temp_changes, std_deviations], axis="columns")

    def process(self):
        environment_data = Path() / "data" / "Environment_Temperature_change_E_All_Data_NOFLAG.csv"
        df = pd.read_csv(environment_data, encoding='cp1252')
        df.apply(self._process_row, axis="columns")

This code introduces the GistempTransformer class, which demonstrates the processing of temperature data from a CSV file and the creation of a hierarchical DataFrame containing temperature changes and standard deviations.

The GistempTransformer class, defined as a dataclass, includes two lists, temperature_changes and standard_deviations, to store the processed data elements. The _process_row method is responsible for handling each row of the input DataFrame. It checks for relevant elements, such as "Temperature change" and "Standard Deviation," extracts the month from the "Months" column, and creates instances of the GistempDataElement class. These instances are then appended to the appropriate lists based on the element type.

The df property returns a DataFrame by combining the temperature_changes and standard_deviations lists. This hierarchical DataFrame has a MultiIndex with levels representing the timestamp and area, providing a structured organization of the data.

transformer = GistempTransformer()
transformer.process()
df = transformer.df

	temp_change	std_deviation
(datetime.date(1961, 1, 1), 'Afghanistan')	0.777	1.95
(datetime.date(1962, 1, 1), 'Afghanistan')	0.062	1.95
(datetime.date(1963, 1, 1), 'Afghanistan')	2.744	1.95
(datetime.date(1964, 1, 1), 'Afghanistan')	-5.232	1.95
(datetime.date(1965, 1, 1), 'Afghanistan')	1.868	1.95

Analyzing Climate Data

Now that we have consolidated all the relevant data into a single dataframe, we can proceed with inspecting and visualizing the data. Our focus is on examining the linear regression lines for each area, as they provide insights into the overall trend of temperature changes over the past decades. To facilitate this visualization, we will create a function that plots the temperature changes along with their corresponding regression lines.

def plot_temperature_changes(areas: list[str]) -> None:
    fig = plt.figure(figsize=(12, 6))
    ax1 = fig.add_subplot(211)
    ax2 = fig.add_subplot(212)

    for area in areas:
        df_country = df[df.index.get_level_values("area") == area].reset_index()
        dates = df_country["timestamp"].map(datetime.toordinal)
        gradient, offset = np.polyfit(dates, df_country.temp_change, deg=1)
        ax1.scatter(df_country.timestamp, df_country.temp_change, label=area, s=5)
        ax2.plot(df_country.timestamp, gradient * dates + offset, label=area)

    ax1.grid()
    ax2.grid()
    ax2.legend()
    ax2.set_ylabel("Regression Lines [°C]")
    ax1.set_ylabel("Temperature change [°C]")

In this function, we are using the **get_level_values** method on a pandas MultiIndex to efficiently query the data in our hierarchical Dataframe on different levels. Let's use this function to visualize temperature changes in the different continents:

plot_temperature_changes(["Africa", "Antarctica", "Americas", "Asia", "Europe", "Oceania"])

From this plot, we can draw several key conclusions:

The regression lines for all continents have a positive gradient, indicating a global trend of increasing Earth surface temperatures.
The regression line for Europe is notably steeper compared to other continents, implying that the temperature increase in Europe has been more pronounced. This finding aligns with observations of accelerated warming in Europe compared to other regions.
The specific factors contributing to the higher temperature increase in Europe compared to Antarctica are complex and require detailed scientific research. However, one contributing factor may be the influence of ocean currents. Europe is influenced by warm ocean currents, such as the Gulf Stream, which transport heat from the tropics towards the region. These currents play a role in moderating temperatures and can contribute to the relatively higher warming observed in Europe. In contrast, Antarctica is surrounded by cold ocean currents, and its climate is heavily influenced by the Southern Ocean and the Antarctic Circumpolar Current, which act as barriers to the incursion of warmer waters, thereby limiting the warming effect.

Now, let's focus our analysis on Europe itself by examining temperature changes in different regions within Europe. We can achieve this by creating individual plots for each European region:

plot_temperature_changes(["Southern Europe", "Eastern Europe", "Northern Europe", "Western Europe"])

From the plotted temperature changes in different regions of Europe, we observe that the overall temperature rises across the European continent are quite similar. While there may be slight variations in the steepness of the regression lines between regions, such as Eastern Europe having a slightly steeper line compared to Southern Europe, no significant differences can be observed among the regions.

Ten Countries Most and Less Affected by Climate Change

Now, let's shift our focus to identifying the top 10 countries that have experienced the highest average temperature increase since the year 2000. Here's an example of how we can retrieve the list of countries:

df[df.index.get_level_values(level="timestamp") > date(2000, 1, 1)].groupby("area").mean().sort_values(by="temp_change",ascending=False).head(10)

area	temp_change	std_deviation
Svalbard and Jan Mayen Islands	2.61541	2.48572
Estonia	1.69048	nan
Kuwait	1.6825	1.12843
Belarus	1.66113	nan
Finland	1.65906	2.15634
Slovenia	1.6555	nan
Russian Federation	1.64507	nan
Bahrain	1.64209	0.937431
Eastern Europe	1.62868	0.970377
Austria	1.62721	1.56392

To extract the top 10 countries with the highest average temperature increase since the year 2000, we perform the following steps:

Filter the dataframe to include only rows where the year is greater than or equal to 2000 using df.index.get_level_values(level='timestamp') >= date(2000, 1, 1).
Group the data by the 'Area' (country) using .groupby('area').
Calculate the mean temperature change for each country using .mean().
Select the top 10 countries with the largest mean temperature change using **.sort_values(by="temp_change",ascending=True).head(10)**.

This result aligns with our previous observations, confirming that Europe experienced the highest rise in temperature compared to other continents.

Continuing with our analysis, let's now explore the ten countries that are least affected by the rise in temperature. We can utilize the same method as before to extract this information. Here's an example of how we can retrieve the list of countries:

df[df.index.get_level_values(level="timestamp") > date(2000, 1, 1)].groupby("area").mean().sort_values(by="temp_change",ascending=True).head(10)

area	temp_change	std_deviation
Pitcairn Islands	0.157284	0.713095
Marshall Islands	0.178335	nan
South Georgia and the South Sandwich Islands	0.252101	1.11
Micronesia (Federated States of)	0.291996	nan
Chile	0.297607	0.534071
Wake Island	0.306269	nan
Norfolk Island	0.410659	0.594073
Argentina	0.488159	0.91559
Zimbabwe	0.493519	0.764067
Antarctica	0.527987	1.55841

We observe that the majority of countries in this list are small, remote islands located in the southern hemisphere. This finding further supports our previous conclusions that southern continents, particularly Antarctica, are less affected by climate change compared to other regions.

Temperature Changes during Summer and Winter

Now, let's delve into more complex queries using the hierarchical dataframe. In this specific use case, our focus is on analyzing temperature changes during winters and summers. For the purpose of this analysis, we define winters as the months of December, January, and February, while summers encompass June, July, and August. By leveraging the power of pandas and the hierarchical dataframe, we can easily visualize the temperature changes during these seasons in Europe. Here's an example code snippet to accomplish that:

all_winters = df[df.index.get_level_values(level="timestamp").map(lambda x: x.month in [12, 1, 2])]
all_summers = df[df.index.get_level_values(level="timestamp").map(lambda x: x.month in [6, 7, 8])]
winters_europe = all_winters.xs("Europe", level="area").sort_index()
summers_europe = all_summers.xs("Europe", level="area").sort_index()

fig = plt.figure(figsize=(12, 6))
ax = fig.add_subplot(111)
ax.plot(winters_europe.index, winters_europe.temp_change, label="Winters", marker="o", markersize=4)
ax.plot(summers_europe.index, summers_europe.temp_change, label="Summers", marker='o', markersize=4)
ax.grid()
ax.legend()
ax.set_ylabel("Temperature Change [°C]")

From this figure, we can observe that temperature changes during the winters exhibit greater volatility compared to temperature changes during the summers. To quantify this difference, let's calculate the standard deviation of the temperature changes for both seasons:

pd.concat([winters_europe.std().rename("winters"), summers_europe.std().rename("summers")], axis="columns")

	winters	summers
temp_change	1.82008	0.696666

Conclusion

In conclusion, mastering MultiIndexes in Pandas provides a powerful tool for handling complex data analysis tasks. By leveraging MultiIndexes, users can efficiently organize and analyze multidimensional datasets in a flexible and intuitive manner. The ability to work with hierarchical structures for rows and columns enhances code clarity, simplifies data handling, and enables simultaneous analysis of multiple variables. Whether it's tracking health parameters of athletes or analyzing Earth's temperature changes over time, understanding and utilizing MultiIndexes in Pandas unlocks the full potential of the library for handling complex data scenarios.

You can find all code included in this post here: https://github.com/GlennViroux/pandas-multi-index-blog

How to Create a Code Editor in React and ChakraUI

Glenn Viroux — Fri, 02 Jun 2023 13:11:53 +0000

If you're a software engineer, programmer or data scientist, you're well aware of the important role played by a reliable code editor. Whether you're creating a small script, experimenting with new ideas, or tackling complex projects, a code editor becomes your trusted companion when coding. And when it comes to my own note-taking application, Ballistic, the code editor takes center stage.

In this post, we'll embark on a fascinating journey into the implementation details of a code editor using React and CodeMirror. At the end, you’ll also find all the source code of the work mentioned in the post, so you can follow along where needed.

CodeMirror

As programmers, we embrace the principle of laziness, which means we prefer not to reinvent the wheel when it comes to implementing a fully functioning code editor. Instead, our first instinct is to explore existing solutions that we can leverage in our application. Fortunately, there are several notable options available:

Ace Editor: Ace is a robust code editor written in JavaScript. It offers an array of features, including syntax highlighting, code folding, code completion, and support for multiple cursors.
Monaco Editor: Powering Visual Studio Code, Monaco Editor provides a rich editing experience. Its feature set encompasses IntelliSense, debugging integration, Git integration, and more.
React-CodeMirror: If you're specifically working with React, React-CodeMirror presents an excellent option. Acting as a wrapper component for CodeMirror, it seamlessly integrates CodeMirror's functionality into your React application. This allows you to leverage CodeMirror within your React components conveniently.

In this article, we'll focus on React-CodeMirror for several reasons. Since our development revolves around a React application, this library aligns perfectly with our needs. Additionally, React-CodeMirror is an open-source project licensed under the permissive MIT license. This means it has been thoroughly used and tested by numerous developers, often leading to the resolution of many small issues before we even embark on its implementation. The presence of a vibrant community surrounding an external library is always a crucial factor in our decision-making process, as it can provide valuable support and guidance when integrating the library into our project.

Project Setup

With the project context and our chosen options in mind, let's dive into setting up the development environment. We'll begin by creating a new React app and using Yarn as our dependency manager:

yarn create react-app react-code-editor --template typescript

Once the app is set up, we need to install the necessary CodeMirror packages. Open your terminal and run the following command:

yarn add @uiw/react-codemirror @uiw/codemirror-theme-github @uiw/codemirror-theme-darcula @codemirror/lang-markdown @codemirror/lang-python @codemirror/lang-javascript @codemirror/lang-cpp @codemirror/lang-html @codemirror/lang-json @codemirror/lang-java

To accelerate development and enhance our component implementation, we'll also incorporate Chakra UI, a comprehensive component library:

yarn add @chakra-ui/react @emotion/react @emotion/styled framer-motion

By leveraging Chakra UI, we gain access to a wide range of pre-built components that facilitate the creation of a visually appealing and user-friendly interface.

Creating a Basic Code Editor

Now that we have our React application set up and all the necessary dependencies installed, let's begin building our code editor by editing the src/App.tsx file. Replace the existing code with the following:

import React, {useState} from 'react';
import {Center, ChakraProvider, Divider, Heading, VStack} from "@chakra-ui/react";
import {githubLight} from '@uiw/codemirror-theme-github';
import {python} from "@codemirror/lang-python";
import CodeMirror from '@uiw/react-codemirror';

function App() {
    const [text, setText] = useState("print(\"Hello world!\")");
    return (
        <ChakraProvider>
            <Center h={"100vh"}>
                <VStack boxShadow={'md'} p={4} borderStyle={"solid"} borderWidth={1} rounded={"lg"}>
                    <Heading>Code Editor</Heading>
                    <Divider/>
                    <CodeMirror
                        value={text}
                        onChange={(newValue) => setText(newValue)}
                        theme={githubLight}
                        extensions={[python()]}
                        basicSetup={{autocompletion: true}}
                        minWidth={'500px'}
                        minHeight={'500px'}
                    />
                </VStack>
            </Center>
        </ChakraProvider>

    );
}

export default App;

In this code snippet, we've created a basic version of our code editor using a few Chakra UI components. Specifically, we've used the CodeMirror component from @uiw/react-codemirror. By configuring its properties, we've specified that the code editor should interpret Python code and have a size of 500px by 500px. At this stage, we have a simple code editor that looks like this:

Note how inside the App component, we use the useState hook from React to create a state variable text and a corresponding setter function setText with an initial value of 'print("Hello world!")'.

The component returns a JSX structure that includes Chakra UI components like Center, VStack, Heading, and Divider. The CodeMirror component from @uiw/react-codemirror is used to render the code editor itself.

The CodeMirror component is configured with properties such as value (bound to the text state), onChange (updating the text state with the new value), theme (using the GitHub Light theme), extensions (using the Python language mode), basicSetup (enabling autocompletion), and minWidth and minHeight (setting the dimensions of the code editor).

Adding More Supported Languages

To make our code editor more versatile and support multiple programming languages, we need to add language selection functionality. Let's start by creating an overview of the supported languages and their corresponding CodeMirror language support objects:

import { LanguageSupport } from '@codemirror/language';
import {markdown} from "@codemirror/lang-markdown";
import {javascript} from "@codemirror/lang-javascript";
import {cpp} from "@codemirror/lang-cpp";
import {html} from "@codemirror/lang-html";
import {json} from "@codemirror/lang-json";
import {java} from "@codemirror/lang-java";

const EXTENSIONS: { [key: string]: LanguageSupport[] } = {
    markdown: [markdown()],
    python: [python()],
    javascript: [javascript()],
    typescript: [javascript()],
    cpp: [cpp()],
    'c++': [cpp()],
    html: [html()],
    json: [json()],
    java: [java()],
};

Now, we can enhance the user experience by providing a dropdown menu for language selection, and store the selected language in another state variable:

function App() {
    const [language, setLanguage] = useState("python");
    const [text, setText] = useState("print(\"Hello world!\")");
    return (
        <ChakraProvider>
            <Center h={"100vh"}>
                <VStack boxShadow={'md'} p={4} borderStyle={"solid"} borderWidth={1} rounded={"lg"}>
                    <HStack w={"100%"} justify={"space-between"}>
                        <Heading>Code Editor</Heading>
                        <Menu>
                            <MenuButton as={Button}>
                                {language}
                            </MenuButton>
                            <MenuList>
                                {Object.entries(EXTENSIONS).map(([language, _]) => (
                                    <MenuItem onClick={() => setLanguage(language)}>{language}</MenuItem>
                                ))}

                            </MenuList>
                        </Menu>
                    </HStack>

                    <Divider/>
                    <CodeMirror
                        value={text}
                        onChange={(newValue) => setText(newValue)}
                        theme={githubLight}
                        extensions={[EXTENSIONS[language]]}
                        basicSetup={{autocompletion: true}}
                        minWidth={'500px'}
                        minHeight={'500px'}
                    />
                </VStack>
            </Center>
        </ChakraProvider>

    );
}

With the language selection dropdown menu in place, users can now choose the programming language they want to work with. The code editor will dynamically update based on the selected language. Here's how the updated application should look like:

Now, you have a more versatile code editor that supports multiple programming languages, providing a better user experience and expanding the range of use cases for your application.

Introducing Backend API Calls

In practice, it's common to save the text written by end-users to the backend application of your program. Currently, we are storing the written text in the React state variable text. However, if the web page is refreshed, all changes made by the user will be lost.

Let's pause for a moment and take a closer look at the setText function and its implications. Currently, every keystroke triggers a call to setText. However, if this function also makes an API call to update the database and re-render the updated text in the <App /> component, it could result in an excessive number of API calls. This can negatively impact frontend performance and lead to a poor user experience.

To address this, it's beneficial to separate the API call from the immediate update shown to the user. One effective approach is to debounce the API call, ensuring it's executed after a short delay or when the user has finished typing. This reduces unnecessary API calls and improves performance, resulting in a smoother user experience.

Debounce the API calls

A debounce function is a useful tool that can delay the execution of a function until a certain amount of time has passed without the function being called again. It's particularly valuable in optimizing application performance by reducing unnecessary function calls. Let's introduce a new function called updateBackend, which represents the backend call. Add the following code snippet to your App component:

import { debounce } from 'ts-debounce';

// ...

const updateBackend = (newText: string) => {
    console.log(`Updated backend with: ${newText}`);
};
const updateTextDebounced = useRef(debounce((newText: string) => updateBackend(newText), 1000 * 2));
useEffect(() => {
    updateTextDebounced.current(text);
}, [text]);

Note that we're using an external library like ts-debounce to implement the debounce functionality. However, you can also create your own debounce function if you prefer. In the code, we use the useState and useEffect hooks to handle and debounce text input changes. Additionally, the useRef hook ensures that the debounce function is created only once every two seconds when the text is being updated.

It's important to note that debounce functions have many other use cases as well. Here are a few examples:

Search bar: When a user types a search term into a search bar, we might want to trigger a search function to display relevant results. However, triggering the search function on every keystroke can lead to unnecessary API calls and hinder application performance. By using a debounce function, we can wait for a certain duration after the user stops typing before triggering the search function, reducing API calls and improving overall performance.
Resize events: If we have an element on the page that needs to be resized based on the window size, using a debounce function can delay the resize function until the user has finished resizing the window. This approach prevents the function from being called excessively during the resize event.
Mousemove events: In scenarios where an element on the page needs to change based on the mouse's position, employing a debounce function can delay the update function until the user stops moving the mouse. This approach avoids triggering the function on every mousemove event, improving performance.
Form validation: When a user fills out a form, it's common to validate the inputs and display any errors. By using a debounce function, we can delay the validation function until the user has finished typing, reducing unnecessary validation calls and enhancing performance.

Debounce functions are versatile and can be beneficial in any situation where you need to delay the execution of a function until a specific duration has passed without the function being called again. This approach reduces unnecessary function calls, leading to improved application performance.

The final and complete react component for your code editor now looks like this:

import React, {useEffect, useRef, useState} from 'react';
import {
    Button,
    Center,
    ChakraProvider,
    Divider,
    Heading,
    HStack,
    Menu,
    MenuButton,
    MenuItem,
    MenuList,
    VStack
} from "@chakra-ui/react";
import {githubLight} from '@uiw/codemirror-theme-github';
import {python} from "@codemirror/lang-python";
import CodeMirror from '@uiw/react-codemirror';

import {LanguageSupport} from '@codemirror/language';
import {markdown} from "@codemirror/lang-markdown";
import {javascript} from "@codemirror/lang-javascript";
import {cpp} from "@codemirror/lang-cpp";
import {html} from "@codemirror/lang-html";
import {json} from "@codemirror/lang-json";
import {java} from "@codemirror/lang-java";
import {debounce} from "ts-debounce";

const EXTENSIONS: { [key: string]: LanguageSupport } = {
    markdown: markdown(),
    python: python(),
    javascript: javascript(),
    typescript: javascript(),
    cpp: cpp(),
    'c++': cpp(),
    html: html(),
    json: json(),
    java: java(),
};

function App() {
    const [language, setLanguage] = useState("python");
    const [text, setText] = useState("print(\"Hello world!\")");

    const updateBackend = (newText: string) => {
        console.log(`Updated backend with: ${newText}`);
    };
    const updateTextDebounced = useRef(debounce((newText: string) => updateBackend(newText), 1000 * 2));
    useEffect(() => {
        updateTextDebounced.current(text);
    }, [text]);

    return (
        <ChakraProvider>
            <Center h={"100vh"}>
                <VStack boxShadow={'md'} p={4} borderStyle={"solid"} borderWidth={1} rounded={"lg"}>
                    <HStack w={"100%"} justify={"space-between"}>
                        <Heading>Code Editor</Heading>
                        <Menu>
                            <MenuButton as={Button}>
                                {language}
                            </MenuButton>
                            <MenuList>
                                {Object.entries(EXTENSIONS).map(([language, _]) => (
                                    <MenuItem key={language} onClick={() => setLanguage(language)}>{language}</MenuItem>
                                ))}

                            </MenuList>
                        </Menu>
                    </HStack>

                    <Divider/>
                    <CodeMirror
                        value={text}
                        onChange={(newValue) => setText(newValue)}
                        theme={githubLight}
                        extensions={[EXTENSIONS[language]]}
                        basicSetup={{autocompletion: true}}
                        minWidth={'500px'}
                        minHeight={'500px'}
                    />
                </VStack>
            </Center>
        </ChakraProvider>

    );
}

export default App;

Conclusion

In conclusion, this blog post has guided you through the process of implementing a code editor in React using CodeMirror, a powerful JavaScript-based text editor. By following the step-by-step instructions, you've learned how to create a functional component that can render code or markdown text with syntax highlighting.

One crucial aspect of any text editor is the ability to update the text. However, directly calling an API endpoint on every text change can lead to an excessive number of API calls, impacting the frontend performance and resulting in a suboptimal user experience.

To address this issue, we've introduced the concept of a debounce function. By incorporating a debounce function, we can delay the execution of the API call until a short delay has passed or until the user has finished typing. This optimization reduces unnecessary API calls, improves performance, and ensures a smoother user experience.

The debounce function acts as a valuable tool in scenarios where you need to optimize performance by reducing the number of unnecessary function calls. Whether it's a code editor, a search bar, or any other interactive component, implementing debounce logic can significantly enhance the efficiency of your application.

By applying the knowledge and techniques shared in this blog post, you now have the skills to create a feature-rich code editor with an improved user experience. Remember to explore and adapt the debounce function to other areas of your application where performance optimizations are required.

Happy coding and enjoy building your next text editor with React and CodeMirror!

You can find all the source code used in this blog post here: https://github.com/GlennViroux/react-code-editor

How to Connect a FastAPI Server to PostgreSQL and Deploy on GCP Cloud Run

Glenn Viroux — Fri, 26 May 2023 10:13:23 +0000

In the world of web development, a robust backend is crucial for delivering a high-quality user experience. One popular approach for building backends is by using RESTful APIs, which enable clients to communicate with servers using HTTP requests.

When I was building Ballistic, I chose for FastAPI as the web framework for my backend API in Python, since it had gained significant popularity in the Python community in a very short amount of time. Apart from its performance, powered by the usage of the asyncio library, I was convinced by the core integration of type annotations and the well-built documentation. Take into account that good documentation and a broad community can save you tons of time when developing your backend.

Building a web application involves more than just a web server; it requires a reliable and efficient database to store and manage the application's stateful data. Among the various database options available, PostgreSQL stands out as a popular choice for developers. Its extensive community and ecosystem provide invaluable support, making it easier to solve problems. PostgreSQL's reputation for reliability and stability further solidifies its position as a trusted solution. Additionally, PostgreSQL's compatibility with multiple platforms ensures flexibility in deployment. In this blog post, we will explore how to connect our FastAPI server with a PostgreSQL database. It's worth noting that the guide can be easily adapted for other databases like MySQL or MongoDB with minimal modifications.

When it comes to deploying REST servers, there are many options to choose from. One particularly appealing option is to use cloud providers like Google Cloud Platform (GCP), which offer scalable, stateless services like Cloud Run and App Engine. With a cloud provider like GCP, you won’t have to worry about keeping your servers up 100% of the time and making sure they are available, also with high traffic loads. Your initial investment is also significantly lower, as you’ll just have to pay for what you consume, instead of buying a whole server up-front.

In this blog post, we'll explore how to build a REST server using FastAPI, PostgreSQL, and GCP, and deploy it to a stateless environment like GCP Cloud Run. At the end of the blog post, I’ll also include all the source code to help you follow along the explanations given in the post itself.

Why Stateless REST Servers are Ideal for Cloud Deployments

Before diving into the details of building a REST server with GCP, let's first take a step back and consider why stateless servers are ideal for cloud deployments.

When a server is stateless, it means that it doesn't store any session data or state information between requests. This makes it easier to scale and deploy, since any instance of the server can handle any incoming request. This is in contrast to stateful servers, which need to keep track of session data and ensure that each request is handled by the same server instance.

Stateless servers also offer other benefits, such as improved reliability and fault tolerance. Since each request can be handled by any instance of the server, failures or crashes can be quickly detected and handled by spinning up new instances.

While it's essential for your app to keep track of user data, it's important to remember that this responsibility falls on the database. The database is designed to store and maintain data integrity, while the REST server is responsible for handling individual, independent requests that operate on the data stored in the database. By separating these responsibilities, your app can be more efficient and easier to maintain over time.

GCP Cloud Run vs. App Engine

When it comes to deploying REST servers on GCP, there are two main options to choose from: Cloud Run and App Engine.

Cloud Run is a fully-managed, container-based platform that allows you to run stateless HTTP containers in a serverless environment. This means you don't need to worry about managing infrastructure, and you only pay for the resources you use.

App Engine, on the other hand, is a platform-as-a-service (PaaS) offering that allows you to build and deploy web applications and APIs using a variety of programming languages and frameworks. It's a more fully-featured platform than Cloud Run, but it also requires more configuration and management.

Both Cloud Run and App Engine are good options for deploying REST servers, and the choice largely depends on your specific use case and requirements.

Building the Backend

Setting up a new project is the first step towards building any application. While there are multiple tools available for managing packages and dependencies, I personally recommend using poetry for its simplicity and ease of use. Alternatively, you can also use a plain old virtual environment to set up your project.

Assuming you have installed poetry in your system, you can use the following command to create a new project:

poetry new fastapi-gcp

This will create a new directory named fastapi-gcp with some default files and a basic project structure. Next, navigate to the newly created directory using cd fastapi-gcp and install the fastapi package using poetry as shown below:

poetry add fastapi asyncpg

This will install the fastapi package along with its dependencies, making it available for use in your project.

Building the FastAPI Server

Now, let's dive into the code and start building our FastAPI server. With just a few lines of code, we can create a simple server to handle requests. Create a [main.py](http://main.py) file and add the following code:

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI(title="fastapi-gcp")

@app.get("/", response_class=HTMLResponse)
def healthcheck():
    return "<h1>All good!</h2>"

Here, we are just using a minimal setup to create a server that returns a HTML-ready response when someone opens the server’s root URL in the browser. To run this server on localhost:8080, execute the following command:

uvicorn main:app --host 0.0.0.0 --port 8080

Now that your server is running, make sure to open a browser on the http://localhost:8080 to see the “All good!” message.

Since FastAPI is based on OpenAPI, at this point you can also use the automatically generated docs. There are multiple options, and two are included by default. Try them out by accessing the following URLs:

Setting Up the PostgreSQL DB

Now that our server is up and running, let's spin up our PostgreSQL server as well. The easiest way to accomplish this is by utilizing Docker and the official PostgreSQL image. If you have Docker installed, you can quickly achieve this by running a simple one-liner:

docker run --name postgres-fastapi -e POSTGRES_PASSWORD=postgres-fastapi --rm -p 5432:5432 postgres

Here, we start a container named "postgres-fastapi" with the password "postgres-fastapi," exposing the default PostgreSQL port 5432. If you'd like to delve deeper into this topic, you can find more detailed information in this article: How to Use the PostgreSQL Docker Official Image.

Settings Management

Before connecting our FastAPI server to the PostgreSQL database, it is important to consider how we manage our settings. Storing access credentials to our database in plain text within our Python repository would pose a significant security risk, as it could be easily discovered by hackers or individuals with malicious intent. To address this concern, it is best practice to extract all sensitive credentials to an external environment file, such as an .env file, and then load these values into our FastAPI server. This approach allows for flexibility, as by simply using a different .env file, we can connect to a different database. This proves especially useful when running different versions of the application, such as one in a staging environment and another in the production environment.

Since FastAPI is built on Pydantic, we can leverage the BaseSettings object to facilitate our settings management. To begin, we need to install the additional dependency by executing the following command:

poetry add pydantic[dotenv]

Let’s create a new file named settings.py and add the following code:

from pathlib import Path

from pydantic import BaseSettings

class Settings(BaseSettings):
    DB_NAME: str
    DB_HOST: str
    DB_USER: str
    DB_PASS: str

settings = Settings(_env_file=Path(__file__).parent/ ".env", _env_file_encoding="utf-8")

This code assumes that you have an .env file in the parent directory of this settings.py module that contains your database name (DB_NAME), the host (DB_HOST), the user (DB_USER) and the password (DB_PASS), like this:

DB_NAME=postgres
DB_HOST=localhost
DB_USER=postgres
DB_PASS=postgres-fastapi

Of course, this .env file should be included in your .gitignore file to exclude it from being uploaded to the repository. At this point, you can use your database settings easily anywhere in your python code, like this:

from settings import settings

print(f"My database name is {settings.DB_NAME}")

Connect FastAPI to the PostgreSQL DB

With our FastAPI and PostgreSQL servers up and running, and our settings management in place, we can now proceed to establish a connection between our FastAPI server and the PostgreSQL database.

To do this, we can use the Tortoise-ORM. Begin by installing the package:

poetry add tortoise-orm

Next, create a new file named database.py where we'll define all the details of the database connection. Here's an example:

from fastapi import FastAPI
from tortoise import Tortoise
from tortoise.contrib.fastapi import register_tortoise

from settings import settings

Tortoise.init_models(["models"], "models")

TORTOISE_ORM = {
    "connections": {
        "default": f"postgres://{settings.DB_USER}:{settings.DB_PASS}@{settings.DB_HOST}:5432/{settings.DB_NAME}",
    },
    "apps": {
        "models": {
            "models": ["models", "aerich.models"],
            "default_connection": "default",
        },
    },
    "use_tz": False,
}

def init_db(app: FastAPI) -> None:
    register_tortoise(
        app,
        config=TORTOISE_ORM,
        modules={"models": ["models", "aerich.models"]},
        generate_schemas=True,
        add_exception_handlers=True,
    )

In this [database.py](http://database.py) module, we define the core config of the ORM in the TORTOISE_ORM object, after which we define a function init_db that actually registers the tortoise orm to an existing FastAPI application.

Note also that we're using the tortoise/aerich tool to perform database migrations. Database migrations are a crucial aspect of managing the evolution of a database schema over time. As an application evolves, the database schema often needs to be modified to accommodate new features, fix bugs, or optimize performance. A database migration tool like Aerich provide a systematic approach to apply these changes while preserving existing data. Make sure to install the aerich dependency by running:

poetry add aerich

Let’s now use the init_db function in the [main.py](http://main.py) module where we define the FastAPI application:

from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI(title="fastapi-gcp")
init_db(app)

@app.get("/", response_class=HTMLResponse)
def healthcheck():
    return "<h1>All good!</h2>"

After implementing the init_db function, your FastAPI server has the capability to connect to your database. However, before proceeding, we need to create our first database model and initialize the Tortoise ORM integration by running the Aerich migrations for the first time. Let's create our initial Note model, which will correspond to a notes table in the PostgreSQL database. This table will store a collection of notes with attributes such as filenames, titles, and content.

from tortoise import fields
from tortoise.models import Model

class Note(Model):
    id = fields.IntField(pk=True)
    created_at = fields.DatetimeField(auto_now_add=True)
    updated_at = fields.DatetimeField(auto_now=True)
    filename = fields.CharField(max_length=256)
    title = fields.CharField(max_length=1000)
    content = fields.TextField(default="")

Now, let’s initialize the aerich migrations by running:

aerich init -t database.TORTOISE_ORM
aerich init-db
aerich migrate
aerich upgrade

Next up, we’ll expand our [main.py](http://main.py) server module with two endpoints: one to create a new note in the database, and one to filter existing notes by providing their title:

from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from pydantic import BaseModel

from database import init_db
from models import Note

app = FastAPI(title="fastapi-gcp")
init_db(app)

@app.get("/", response_class=HTMLResponse)
def healthcheck():
    return "<h1>All good!</h2>"

class CreateNotePayload(BaseModel):
    filename: str
    title: str
    content: str

@app.post("/notes")
async def create_note(payload: CreateNotePayload):
    note = await Note.create(**payload.dict())
    return {"message": f"Note created successfully with id {note.id}"}

@app.get("/notes/{title}")
async def get_note_by_title(title: str):
    if not (note := await Note.get_or_none(title=title)):
        raise HTTPException(status_code=404, detail="Note not found")

    return note.id

You can conveniently test the endpoints and the database connection by utilizing the Swagger or Redoc automatic documentation tools. Start by using the POST endpoint to create a new note, and then employ the GET endpoint to retrieve the ID of the recently created note. If this process succeeds, congratulations! You have successfully established a connection between your FastAPI server and the PostgreSQL database.

Deploying to GCP

Now that we have developed our backend and established the connection between our FastAPI server and the database, let's delve into the process of deploying our application to a cloud provider like GCP (Google Cloud Platform).

Deploying a PostgreSQL DB to GCP

Up until this point, we have been working with a local PostgreSQL server to test our application. However, when deploying our application to a cloud provider, it is essential to use a PostgreSQL server that runs in the cloud. Fortunately, you have several options for achieving this.

The most straightforward choice is to utilize the Cloud SQL service provided by GCP. Cloud SQL is a fully-managed database service that handles the PostgreSQL instance for you. Another option is to deploy the PostgreSQL database on a compute engine instance using the same PostgreSQL Docker image we used earlier for our local instance.

While deploying on a compute engine instance offers more flexibility, Cloud SQL holds several advantages as a fully managed solution:

Managed Service: With Cloud SQL, Google takes care of the underlying infrastructure, maintenance, and updates, relieving you of these responsibilities.
Scalability: Cloud SQL provides automatic scalability to handle varying workloads. It allows you to easily adjust resources such as CPU, memory, and storage based on your application's requirements. Scaling can be performed with minimal downtime and without the need to provision additional virtual machines.
Backup and Recovery: Cloud SQL offers automated backups of your databases, ensuring the safety of your data. You can effortlessly restore backups or perform point-in-time recovery to a specific moment in the past. Automated backups can be enabled, and the retention period can be set according to your needs.

It's important to note that these advantages come with a higher cost. The pricing for Cloud SQL depends on various factors, including load, CPU usage time, disk usage, and memory usage. For smaller databases, the costs of using Cloud SQL can be up to 10 times higher than deploying the same database on a compute engine instance. To get an estimate of the prices for different options, you can use the GCP Pricing Calculator.

Regardless of the deployment option you choose, make sure to update the database credentials in the .env file you created earlier to match the correct credentials of your deployed PostgreSQL database.

Containerize the FastAPI Server

To facilitate the deployment of your application on GCP, it is recommended to containerize it using Docker. This involves creating a Dockerfile that installs Poetry, copies the FastAPI server code into the image, and configures the Uvicorn server as the entrypoint. Below is an example of a Dockerfile that accomplishes these tasks:

FROM python:3.11

WORKDIR /app
ENV PYTHONFAULTHANDLER 1
ENV PYTHONUNBUFFERED 1
ENV PIP_NO_CACHE_DIR off
ENV PIP_DISABLE_PIP_VERSION_CHECK on

RUN curl -sSL https://install.python-poetry.org | python3 -
ENV PATH="/root/.local/bin:${PATH}"

COPY poetry.lock .
COPY pyproject.toml .

RUN POETRY_VIRTUALENVS_CREATE=false poetry install --only main --no-interaction --no-ansi

COPY . /app

EXPOSE 8080

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

In this Dockerfile, we start with the official Python base image, specify the working directory as /app, and install Poetry for managing the project's dependencies. Next, we copy the poetry.lock and pyproject.toml files into the container and install the project dependencies using Poetry. Then, we copy the FastAPI server code from the root directory on the host machine to the /app directory inside the container.

We expose port 8080, assuming that the FastAPI server will listen on that port. If your server uses a different port, make sure to update the EXPOSE statement accordingly.

Finally, we set the entrypoint command to run the Uvicorn server with the appropriate parameters. In this example, it runs the app module's main FastAPI instance, binds it to all network interfaces (0.0.0.0), and listens on port 8080.

Feel free to modify this Dockerfile based on your specific requirements, such as adding additional dependencies or environment variables.

Once you have created your Dockerfile, you can build the image using the following command:

docker build -t fastapi-gcp-backend:1.0 .

If the poetry installation step during this build fails, make sure to remove the

packages = [{include = "fastapi_gcp"}]

line from your pyproject.toml file in case you have it. This should fix the problem of poetry installing in an existing project.

Push image to GCP Artifact Registry

When the build has completed successfully, you can push the image to the GCP artifact registry and select it when creating a GCP cloud run configuration to deploy your application with ease.

To push an image to the GCP artifact registry, you should tag it with the appropriate repository name, and push the image after authenticating to a repository. You can learn more about authenticating to a GCP repository here. You should then tag your image like this:

LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE

In our case, this will be:

us-east1-docker.pkg.dev/<your gcp project>/<your repo>/fastapi-gcp-backend

Where you should fill in your GCP project and repository for your project. We can tag and push the image we build earlier like this:

docker tag fastapi-gcp-backend:1.0 us-east1-docker.pkg.dev/<your gcp project>/<your repo>/fastapi-gcp-backend:1.0
docker push us-east1-docker.pkg.dev/<your gcp project>/<your repo>/fastapi-gcp-backend:1.0

Deploy image to GCP Cloud Run

Now that your image has been successfully pushed to the GCP Artifact Registry, you can easily use this image in a Cloud Run configuration. To configure a new Cloud Run service, you can follow these steps using the GCP web console at https://console.cloud.google.com:

Navigate to the Cloud Run product by clicking on the appropriate menu option.
Click on "Create Service" to start the configuration process.
In the "Create Service" page, you will find a section to specify the container image. Choose the option that allows you to select the image from the Artifact Registry.
Select the image you just pushed to the Artifact Registry from the available options.
Configure other settings as per your requirements, such as service name, region, authentication, and scaling options.
Once you have completed the necessary configurations, click on "Create" to create the Cloud Run service.

In the “Authentication” section, make sure to select “Allow unauthenticated invocations” if you are creating a public API:

By following these steps, you will be able to configure a Cloud Run service that uses the container image from the Artifact Registry. This allows you to easily deploy your FastAPI application to a scalable and managed environment provided by GCP.

Please note that the steps may vary slightly depending on any updates to the GCP web console interface. Always refer to the GCP documentation for the most up-to-date instructions.

Conclusion

In conclusion, building a robust backend for your application is essential to provide a high-quality user experience. RESTful APIs provide an efficient way to build backends and deploying them on cloud platforms like Google Cloud Platform (GCP) can provide scalability and statelessness to the servers. In this blog post, we explored the benefits of using stateless REST servers on cloud deployments, discussed the differences between Cloud Run and App Engine on GCP, and learned how to build a REST server using FastAPI and PostgreSQL and deploy it to GCP Cloud Run. We also saw how to use Object-Relational Mapping (ORM) like Tortoise ORM to map database tables to Python objects and provided an easy-to-use API for interacting with the database. By following the steps outlined in this post, you can build and deploy a scalable RESTful API that meets your application requirements.

Possible improvements

There are several ways to optimize this workflow, and it is recommended that you explore these options if the benefits outweigh the additional effort required. Some potential improvements include:

Use Terraform to define GCP resources such as Cloud Run, Compute Engine, and CloudSQL in a declarative manner. This allows for automated deployments and standardized infrastructure provisioning, reducing the likelihood of human error and providing increased visibility.
Consider exploring alternatives to Tortoise-ORM. While the documentation for Tortoise-ORM is comprehensive, a larger community of async ORMs, such as SQLModel, may be better equipped to help with more specific queries.

Source Code

You can find all the source code here: https://github.com/GlennViroux/fastapi-postgres-gcp-blog

Make sure to let me know if you have any comments, questions or suggestions!

Happy coding! :)

The Great Operating System Battle: Why WSL2 is Winning

Glenn Viroux — Fri, 12 May 2023 15:48:12 +0000

Software engineers have a never-ending debate about which operating system is the best for their work. Windows, MacOS, and Linux all have their dedicated user base, with each system offering different features and benefits. However, when developers work in teams with different OS preferences, it can create problems with compatibility issues. In this article, we'll discuss how Microsoft's WSL2 has addressed these issues and how it has become a preferred option over virtual machines for developers.

OS Stereotypes

If you've ever worked in software engineering, you know that operating system preferences can be a hot topic. It's like a turf war between developers, each one waving the flag of their favorite OS. Here’s my take on the three types of people in the workspace:

The Linux guy/girl, always the senior software engineer, who swears by the flexibility and customization of the operating system. They know that Linux is known for its robust security features, making it a hacker's worst nightmare.
The Apple enthusiast, who praises the MacBook as if it were their child. They love the Unix-based macOS and its slick interface, even if it is closed-source.
The non-technical people, who just use Windows because it's what they know. They get excited about all the corporate software like Teams, Outlook, and Office that are readily available.

Compatibility Issues

But what happens when these three worlds collide? That's when the real trouble begins. The Linux guy/girl can't get the microphone to work on the Microsoft Teams app they installed on their Linux distro. The Apple enthusiast tries to use a Linux command on their MacBook and ends up getting a "not found" error message. These compatibility issues can quickly turn a productive day into a frustrating one.

Virtual Machines

A common solution is for developers to use virtual machines on their Windows computers, through software like VirtualBox. Virtual machines were clunky and time-consuming to set up. You had to allocate certain resources like RAM and disk storage for each virtual machine. If you wanted to use two different Linux distributions, that would mean you’d have to assign RAM and disk storage twice, once for each distribution. Starting the virtual machine wasn’t a easy neither, as it came with the overhead of a hypervisor.

WSL1

In 2016, Microsoft introduced WSL1, a response to the growing number of developers using Linux as their primary development environment. WSL1 was a significant step forward for developers who wanted to use Linux binaries on a Windows machine without the need for a virtual machine. It worked by translating Linux system calls into Windows system calls, which made it possible to run Linux binaries on Windows without needing to modify them.

However, WSL1 had some limitations that made it unsuitable for certain types of development tasks. One of the biggest drawbacks of WSL1 was its lack of support for background services and daemons. This meant that developers couldn't run certain types of Linux applications that required these services to be running in the background. For example, if you wanted to run a web server or a database server, you couldn't do so with WSL1.

Another limitation of WSL1 was that it relied on the Windows NT kernel to function. While this was an improvement over the need for a virtual machine, it still meant that WSL1 couldn't offer the same level of compatibility with Linux applications as a true Linux kernel.

Despite these limitations, WSL1 was still a significant improvement over previous options for running Linux on Windows, such as virtual machines. It allowed developers to use Linux binaries on a Windows machine with fewer resources and better performance than a virtual machine would provide. However, there was still room for improvement, which is where WSL2 comes in.

WSL2

WSL2 addressed many of these limitations by using a real Linux kernel instead of a translation layer. This allowed for full system call compatibility, which in turn made it possible for WSL2 to run Docker containers natively. By using a real Linux kernel, WSL2 eliminated the need for a translation layer to map Linux system calls to Windows system calls. This not only improved performance but also ensured better compatibility with Linux applications.

One of the key advantages of WSL2 is that it allows developers to work in a Linux environment while using Windows as their primary operating system. This is particularly useful for those who prefer Windows for their day-to-day work but need to use Linux-specific tools and software for their development tasks.

Installing WSL2

To use WSL2, you need to install a Linux distribution from the Microsoft Store, such as Ubuntu, Debian, or Fedora. If you don’t want to use the Microsoft Store, you can also install WSL2 using PowerShell:

wsl --install

Note that this command will only work if you don’t already have WSL installed. if you run wsl --install and see the WSL help text, try running wsl --list --online to see a list of available distros and run wsl --install -d <DistroName> to install a distro.

Conclusion

For me personally, WSL2 has proven to be a preferred option over virtual machines. It allows for better performance and compatibility with Linux applications, eliminates the overhead of a hypervisor, and requires fewer resources.

While building Ballistic, I’m using WSL2 on a daily basis, as all my development work is done in an Ubuntu distribution on my Windows machine. With the addition of a real Linux kernel, WSL2 is a game-changer for developers who use Windows as their primary development environment.

References

How to deploy Next.js to Firebase

Glenn Viroux — Fri, 05 May 2023 10:50:48 +0000

Next.js is a powerful framework for building web applications, built on top of React.js and Node.js. As per Stackoverflow’s 2022 Developer Survey, React.js is the second most loved web framework, right after Node.js. Next.js provides a great set of features for building production-ready applications, such as server-side rendering and static site generation.

Firebase, on the other hand, is an easy-to-use platform by Google that helps you build mobile and web applications.

If you're developing a web application using these two technologies, you're in good company. However, when I was developing Ballistic, I stumbled upon a number of issues when trying to deploy the application. In this article, we'll explore these issues and walk through a step-by-step guide to successfully deploy your Next.js application on GCP.

Firebase Framework-Aware Hosting for Next.js

Firebase Hosting is a robust and reliable option for hosting websites, particularly when you require rapid and easy deployment. It is a fully-managed hosting service provided by Google, which allows developers to deploy web applications and static content to a global content delivery network (CDN) with a simple command. Firebase Hosting's free tier is particularly appealing for small projects, as it provides a limited amount of storage and data transfer per day at no cost. If your website grows beyond the free tier limits, you can use pay-as-you-go pricing, which allows you to pay only for the resources that you use.

Firebase's framework-aware hosting is a valuable integration that allows for seamless deployment of popular web frameworks such as Angular and Next.js. If you're using Next.js, Firebase will automatically detect your next.config.js file and recognize that you're running a Next.js web application. This results in Firebase translating your Next.js settings into Firebase settings with minimal configuration needed. Additionally, if your app includes dynamic server-side logic, the integration will translate it into Cloud Functions for Firebase. This integration can significantly streamline the deployment process for developers, making it a convenient and efficient option for hosting web applications.

One of the challenges with these "black box integrations" is that when something goes wrong, it can be difficult to troubleshoot. In the case of Next.js and Firebase hosting, an error message such as "Couldn't find a pages directory. Please create one under the project root" can leave developers scratching their heads. When you've verified that the issue isn't caused by your own code, you may be left with no other option than to create an issue on GitHub (if it's an open source project) or contact support and hope for a quick resolution.

That’s why, in this blog post, we’ll explore how to manually translate the Next.js configuration settings to Firebase settings, so we don’t have to rely on the experimental integration of Next.js in Firebase.

Translating Your Next.js App to Firebase

One of the core advantages of Next.js is its support for Server-Side Rendering (SSR), which allows web pages to be pre-rendered on the server, rather than relying on client-side JavaScript to render the page. This translates to a significant improvement in user experience, as the user can view a fully rendered HTML page almost instantly, without waiting for JavaScript or CSS files to load.

By translating your Next.js app to Firebase, you can further enhance its speed and scalability, as Firebase offers a robust and reliable platform for hosting and serving web content. With Firebase Hosting, you can easily deploy and serve your pre-rendered Next.js app to a global audience, providing them with a fast and seamless browsing experience.

To achieve this, we need to divide our Next.js codebase into two parts: the client and the server. The server part will define a Firebase function that handles SSR, while the client part will contain the rest of our Next.js codebase. In the following steps, we'll cover all the details of creating a Next.js project and deploying it to Firebase, from start to finish.

Setting up the Project

Let's begin by setting up our Next.js project by running the following command:

yarn create next-app next-firebase --typescript --eslint --src-dir --no-tailwind

Accept the default interactive options, and you should see a message saying Success! Created next-firebase.

This will give you a fully functional Next.js codebase that you can test quickly by starting the development server:

yarn dev

Then, open your browser and navigate to http://localhost:3000

Creating Server and Client Separation

After successfully setting up your project in the previous step, let's move on to splitting your codebase into a server function responsible for SSR and a client codebase containing all the Next.js code.

Create a client and server folder inside the src directory. Move the original contents of the src directory to the client directory, along with the public folder, next.config.js file, and tsconfig.json file. This will leave the server folder empty for now. We'll populate it with a Firebase function responsible for SSR in the next steps.

Creating the SSR Server Code

To create the function responsible for server-side rendering, we need to install some dependencies first. Run the following command to install them:

yarn add firebase-admin firebase-functionscd 
yarn add **@babel/core @babel/cli @babel/preset-env cross-env rimraf**

After installing the dependencies, create an index.js file inside the server folder, as shown below:

import * as admin from 'firebase-admin';
import * as functions from 'firebase-functions';
import next from 'next';

admin.initializeApp();

const dev = process.env.NODE_ENV !== 'production';
const app = next({
    dev,
    // the absolute directory from the package.json file that initialises this module
    // IE: the absolute path from the root of the Cloud Function
    conf: { distDir: 'dist/client' },
});
const handle = app.getRequestHandler();

export const nextjsServer = functions.https.onRequest((request, response) => {
    // log the page.js file or resource being requested
    console.log('File: ' + request.originalUrl);
    return app.prepare().then(() => handle(request, response));
});

Since we'll be transpiling this code with Babel, we need to instruct Babel on how to compile the code. We'll create a .babelrc file inside the server folder for this purpose:

{
  "presets": [
    [
      "@babel/preset-env",
      {
        "targets": {
          "node": "10.15.3"
        }
      }
    ]
  ]
}

Updating package.json

We need to update the scripts in our package.json to reflect the new folder structure we created:

"scripts": {
    "dev:client": "next src/client",
    "dev:server": "babel src/server --out-dir dist/server --source-maps --watch",
    "dev": "yarn run dev:client & yarn run dev:server",
    "build:client": "next build src/client",
    "build:server": "babel src/server --out-dir dist/server --source-maps",
    "build": "yarn run build:client && yarn run build:server"
}

At this point, you can spin up a local development server that hosts your client side code and serves your SSR server. You can also build a production-ready version of your Next.js app.

Also add the following lines to your package.json, in order to override the main entrypoint to your server:

With these changes, you can spin up a local development server that hosts your client-side code and serves your SSR server. You can also build a production-ready version of your Next.js app.

Additionally, add the following lines to your package.json to override the main entry point to your server:

"main": "dist/server/index.js",
"engines": {
  "node": "16"
},

These lines ensure that when you deploy your app to Firebase, it uses the compiled version of your server code instead of the original source code. They also specify that the app requires Node.js version 16 to run.

Setting up Firebase

To integrate Firebase with our Next.js app, we'll need to create a new Firebase project in the Firebase console. If you already have a project set up, you can skip this step. Once you've created your project, you can add Firebase to your web app by navigating to the "Project Settings" page, clicking on the gear icon next to "Project Overview," and selecting "Add to Web App" on the "General" tab. Here, you can register your new web application.

Next, in the root directory of your project, run the following command to initialize Firebase:

firebase init

In the root directory of your project. During Project Setup, select the following options:

Firebase Hosting and Firebase Functions
use an existing project, and select the firebase project you just created or that you wish to use
What language would you like to use to write Cloud Functions? → JavaScript
Do you want to install dependencies with npm now? → No
Detected an existing Next.js codebase in the current directory, should we use this? → No
Do you want to use a web framework? (experimental) → No
What do you want to use as your public directory? → Public
Configure as a single-page app (rewrite all urls to /index.html)? → No
Set up automatic builds and deploys with GitHub? → No

After running this command, a .firebaserc file and a firebase.json file will be created in your root folder. You'll need to modify the firebase.json file to include the following code:

{
  "hosting": {
    "public": "public",
    "rewrites": [
      {
        "source": "**/**",
        "function": "nextjsServer"
      }
    ]
  },
  "functions": {
        "runtime": "nodejs16"
    "source": ".",
    "ignore": [
      ".firebase/**",
      ".firebaserc",
      "firebase.json",
      "**/node_modules/**",
      "**/public/**",
      "**/cache/**"
    ]
  }
}

The previous process also generates a public folder with a 404.html file and an index.html file. Make sure to delete the index.html file before continuing.

Test your website and deploy

At this point, you can test your web app locally, share your changes with others, and deploy the app to the public. To test your app locally, you can use the serve script from package.json:

yarn serve

After verifying that your web app works fine locally by running the serve script from package.json, you're ready to deploy it to the public. To do this, you need to add the necessary deployment scripts to your package.json file:

"scripts": {
    "predeploy": "rimraf dist/ && yarn run build",
    "deploy": "cross-env NODE_ENV=production firebase deploy --only functions,hosting",
 }

Now, when you're ready to deploy your app, you can simply run yarn deploy to deploy your app to Firebase hosting.

Conclusion

In conclusion, we have covered the entire process of building and deploying a Next.js web app with Firebase Cloud Functions and Firebase Hosting. By following the steps outlined in this post, you should now be able to create and launch your own Next.js app with Firebase.

The full source code for this project is available on GitHub at: https://github.com/GlennViroux/next-firebase-blog.

Thank you for reading, and I hope this article has been informative and helpful for you.

Understanding Software Licenses: Open-Source vs. Commercial

Glenn Viroux — Tue, 28 Mar 2023 19:33:49 +0000

Introduction

Software licensing is about as exciting as watching paint dry, but it's an essential part of software development. In this post, I'll try to spice things up and make software licensing a little more entertaining. I'll explore the two most common software licenses: open-source and commercial licenses. I'll also show you how to use open-source software in your commercial projects without breaking any laws.

Open-Source Software Licenses

Open-source software is like the free samples at Costco: it's free to try, and you can take as much as you want. The source code is open for everyone to see, and you can modify it to your heart's content. There are different types of open-source software licenses, but the most common one is the GNU General Public License (GPL). Under this license, you can access, modify, and distribute the software, but any modified versions must be released under the same license. It's like a chain letter, but for software.

One big advantage of open-source software is that it's usually free. This makes it ideal for startups that prefer lower starting costs and faster project starts. Plus, it's often community-driven, which means that developers from around the world contribute to its development, making it more reliable and secure. It's like having a global team of software developers working for you, who all have an interest in keeping the software up-to-date and bug-free, as they are probably using it themselves as well.

However, there are some downsides to open-source software. While it's free, it often requires technical expertise to implement and maintain. Therefore, businesses need to consider the cost of hiring skilled developers to work with the software. Additionally, open-source software may lack some features that are available in commercial software.

If you want to learn more about open-source software, check out The Open Source Initiative for a comprehensive guide to open-source software licenses.

Commercial Software Licenses

Commercial software is like a fancy restaurant: you have to pay to get in, and the menu is limited. The license usually restricts your ability to modify, distribute, or reverse-engineer the software, just like you’re not really allowed to change any ingredients in the dish the chef of the restaurant is preparing for you. Commercial software is usually developed by software companies who charge you for the right to use it. Examples of commercial software include Microsoft Office and Adobe Photoshop.

One of the main advantages of commercial software is that it's usually backed by a company that provides support and regular updates. This makes it more reliable and secure than open-source software, where support and updates are often community-driven. It's like having a personal IT department at your beck and call.

However, commercial software can be expensive, which may not be ideal for individuals and small businesses. Additionally, the license restrictions can limit the user's ability to modify the software to suit their needs, and while it’s true that in many cases, a higher price also means better value, that’s not always the case. Just like you might just come back home dissapointed after spending a small fortune in a fancy restaurant.

If you want to learn more about commercial software licenses, check out Techopedia for an in-depth explanation of commercial software licenses.

Using Open-Source Software in Commercial Projects

Using open-source software in commercial projects is like using your neighbor's lawnmower to mow your lawn: it's legal as long as they give you permission. Many businesses are hesitant to use open-source software in their commercial projects because they're not sure if it's legal or if it will affect their intellectual property rights. However, it's legal as long as the software is licensed under an open-source license that allows commercial use.

To use open-source software in commercial projects, businesses need to understand the terms and conditions of the license and ensure that they comply with them. For example, some open-source licenses require that the source code of any modified software is made available to the public. It's like having to share your secret recipe with the world.

It's also essential to note that while open-source software is free, it often requires technical expertise to implement and maintain. Therefore, businesses need to consider the cost of hiring skilled developers to work with the software. It's like having to hire a sous chef to help you cook that secret recipe.

Elastic License Controversy

The Elastic license controversy is a recent example of how licensing issues can cause disputes in the open-source community. In 2018, Elastic, the company behind the Elasticsearch and Kibana open-source projects, created a new license called the Elastic License. The license was more restrictive than the Apache 2.0 license that Elasticsearch and Kibana were previously licensed under. The Elastic License added several limitations, including restrictions on cloud service providers such as Amazon Web Services (AWS), who were using Elasticsearch and Kibana in their own services.

The move was met with backlash from the open-source community, with many accusing Elastic of violating the principles of open-source software. In response, AWS forked the Elasticsearch and Kibana projects and cloned them under the Apache 2.0 license, creating the OpenSearch project.

The Elastic License controversy highlights the importance of understanding the legal implications of using open-source software and the need for clear licensing terms. It also shows how licensing issues can cause disputes and lead to the forking of open-source projects.

If you want to learn more about the Elastic License controversy, check out this article from TechCrunch for an in-depth analysis of the issue.

The Case of the faker.js Package

The importance of understanding software licenses was demonstrated in the case of the faker.js package. faker.js is a popular open-source package that generates fake data for testing and development purposes. However, in 2021, the package was removed from the npm registry due to a licensing dispute. The package's creator, Marak Squires, had changed the license from the MIT license to a more restrictive license without the consent of the package's contributors. This move violated the open-source software principles of transparency and collaboration.

The incident highlights the importance of understanding software licenses and the legal implications of using open-source software. Businesses need to ensure that they comply with the terms and conditions of the license when using open-source software to avoid legal disputes.

If you want to learn more about the faker.js package dispute, check out this article from ZDNet for an in-depth analysis of the issue.

Can I use open source software in my commercial project?

Open source software is often distributed under specific licenses that allow users to access and modify the source code. Some open source licenses, such as the Apache License, the MIT License, and the BSD License, are permissive and allow for commercial use and distribution, with few restrictions.

Other open source licenses, such as the GNU General Public License (GPL), require that any software that uses or modifies GPL-licensed code must also be licensed under the GPL and made available to others under the same terms.

Therefore, if you plan to use open source software in your commercial project, it's important to carefully review the license of the software and ensure that you are complying with its terms and conditions. You may need to make your source code available to others, provide attribution to the original authors, and adhere to any specific restrictions or requirements of the license.

Additionally, if you are using open source software that has been modified or integrated into your commercial software, you may need to provide notice to your users about the use of open source components and the licenses that apply to them.

In summary, you can use open source software in your commercial software project, but it's important to understand the license terms and comply with any requirements or restrictions that apply to the software you are using.

Conclusion

In conclusion, software licensing doesn't have to be boring. Open-source and commercial software licenses offer different benefits and drawbacks, and it's up to businesses to decide which type of license is best suited for their needs. While open-source software is free, it often requires technical expertise to maintain it. On the other hand, commercial software is reliable and secure, but it can be expensive. Using open-source software in commercial projects is legal and can be a cost-effective solution for businesses, provided that they comply with the terms and conditions of the license. So, go ahead and give open-source software a try. You might just like it.