Master Decorators and Generators in Python

Introduction

Decorators and generators, two powerful concepts from Python's functional programming, allow for the modification and enhancement of functions and provide a memory-efficient method for working with data sequences. This article will guide you to comprehensively understand decorators and generators, with illustrative examples and practical applications. Before we delve into decorators and generators, let's briefly discuss Python functions, as this knowledge will be beneficial in understanding decorators and generators.

Functions

Consider functions as small, specific task helpers that you can create. They're like mini-programs within your larger program. Instead of repeatedly writing the same code, you can place that code inside a function and assign it a name. Here's an easy analogy: Imagine a function as a recipe in a cookbook. The recipe has a title (like "Chocolate Chip Cookies"), and when you want to bake cookies, you follow the steps in the recipe.
Similarly, in Python, when you want to perform a specific task, you call the function by its name, and it executes that task for you based on the instructions you've provided within the function. So, Python functions, also known as first-class objects, are like reusable instruction sets that make your code more organized and manageable. They accept inputs (like ingredients in a recipe) and can produce outputs (like the tasty cookies resulting from following the recipe). Fuctions make your code more efficient and easier to comprehend. Function usage includes:
Usage of functions:
Defining and invoking your function
Passing arguments and returning values from functions.
Utilizing built-in functions and modules.
Creating lambda functions and high-order functions.
Employing decorators and generators.

Defining and calling your function:

To define a function, use the def keyword, followed by the function name and parentheses. Inside the parentheses, you can optionally specify some parameters the function will accept as input. Then, write the function's body, indented under the def statement. To invoke a function, use its name followed by parentheses, passing any arguments that match the parameters.
For example:

# Defining a function that prints a greeting
def say_hello(name):
    print(f"Hello, {name}!")

# Call the function with different arguments
say_hello("Alice")
#Output:  Alice
say_hello("Bob")
#Output: Bob

Passing arguments and returning values from functions:

Arguments are values passed to a function when it's called. Parameters are variables that receive these arguments within the function definition. You can pass different types of arguments, such as positional, keyword, default, or variable-length arguments, to a function. A function can also return a value to the caller using the return statement.
For example:

# Define a function that takes two numbers and returns their sum
def add_numbers(a, b):
    return a + b

# Call the function with positional arguments
result1 = add_numbers(3, 5)
print(result1)

# Call the function with keyword arguments
result2 = add_numbers(b=7, a=4)
print(result2)

# Call the function with default arguments
def multiply_numbers(a, b=2):
    return a * b

result3 = multiply_numbers(6)
print(result3)

# Call the function with variable-length arguments
def print_names(*names):
    for name in names:
        print(name)

print_names("Alice", "Bob", "Charlie")

Utilizing built-in functions and modules:

Python has numerous built-in functions available for use without defining them yourself. For instance, len() returns an object's length, type() returns an object's type, print() displays an object to standard output, etc.
More built-in functions can be found in the official documentation. Python also offers many modules that provide additional functionality and features. A module is a file that contains pre written Python code like variables, functions, classes, etc. To use a module's functionality, import it using the import statement.
For example:

# Import the math module
import math

# Use some of the functions from the math module
print(math.pi) # The mathematical constant pi
print(math.sqrt(16)) # The square root of 16
print(math.factorial(5)) # The factorial of 5

# Import the random module
import random

# Use some of the functions from the random module
print(random.randint(1, 10)) # A random integer between 1 and 10
print(random.choice(["red", "green", "blue"])) # A random element from a list
print(random.shuffle([1, 2, 3, 4])) # Shuffle a list in place

Creating lambda functions and high-order functions:

Lambda functions are anonymous functions defined using the lambda keyword. They are useful for creating simple functions that can be passed as arguments to other functions or returned as values from other functions. High-order functions take other functions as arguments or return other functions as values. For example:

# Define a lambda function that returns the square of a number
square = lambda x: x ** 2

# Call the lambda function
print(square(5))

# Define a high-order function that takes a function and list as arguments and applies the function to each element of that list
def map_function(function, list):
    result = []
    for element in list:
        result.append(function(element))
    return result

# Call the high-order function with different arguments
print(map_function(square, [1, 2, 3])) # Apply the square lambda function to each element of [1, 2, 3]
print(map_function(lambda x: x + 1, [4, 5, 6])) # Apply an anonymous lambda function that adds 1 to each element of [4, 5, 6]

Now that you have an understanding of what function is. Let's dive into Decorators and Generators.

Decorators

Python decorators let you modify the behaviour and functionality of functions without changing their code. Decorators are like wrappers around functions that allow you to add functionality before the function is called. Decorators are especially useful for implementing cross-cutting concerns like logging, authentication, and caching without making the primary function's code messy.

Syntax and Usage

In Python, decorators are represented by the @ symbol followed by the name of the decorator function. They're placed directly above the function they modify, which means the function is passed as an argument to the decorator.

@decorator_function
def my_function():
    # Function code

Typical Decorator Use Cases

Decorators in Python serve various purposes, such as:

• Logging: You can use decorators to log the input, output, or execution time of a function. For example, you can write a decorator that prints the arguments and returns the value of a function every time it is called. This decorator can help you debug your code or monitor its performance.

import time

# Define a decorator that logs the input, output, and execution time of a function
def log(func):
    # Define a wrapper function that takes any number of arguments
    def wrapper(*args, **kwargs):
        # Get the current time before calling the function
        start = time.time()
        # Call the original function and store the output
        result = func(*args, **kwargs)
        # Get the current time after calling the function
        end = time.time()
        # Calculate the elapsed time
        elapsed = end - start
        # Log the input, output, and execution time
        print(f"Input: {args}, {kwargs}")
        print(f"Output: {result}")
        print(f"Execution time: {elapsed} seconds")
        # Return the output
        return result
    # Return the wrapper function
    return wrapper

# Use the decorator on a function that calculates the sum of the first n natural numbers
@log
def sum_n(n):
    # Initialize the sum to zero
    s = 0
    # Loop from 1 to n
    for i in range(1, n + 1):
        # Add i to the sum
        s += i
    # Return the sum
    return s

# Call the decorated function
sum_n(100)

The Output

Input: (100,), {}
Output: 5050
Execution time: 1.9073486328125e-05 seconds

As you can see, the decorators logs the input, output and execution time of the sum_n function.

• Caching: You can use decorators to cache the results of a function that is expensive to compute or depends on external resources. For example, you can write a decorator that stores the output of a function in a dictionary and returns it if the same input is given again. Caching can improve the efficiency and speed of your program.

# Define a decorator that caches the results of a function
def cache(func):
    # Create an empty dictionary to store the results
    memo = {}
    # Define a wrapper function that takes any number of arguments
    def wrapper(*args, **kwargs):
        # Convert the arguments to a hashable key
        key = (args, tuple(kwargs.items()))
        # Check if the key is already in the dictionary
        if key in memo:
            # Return the cached result
            return memo[key]
        else:
            # Call the original function and store the output
            result = func(*args, **kwargs)
            # Store the output in the dictionary with the key
            memo[key] = result
            # Return the output
            return result
    # Return the wrapper function
    return wrapper

# Use the decorator on a function that calculates the Fibonacci number of a given index
@cache
def fib(n):
    # Base case
    if n == 0 or n == 1:
        return n
    # Recursive case
    else:
        return fib(n - 1) + fib(n - 2)

# Call the decorated function multiple times with different inputs
print(fib(10))
print(fib(20))
print(fib(30))
print(fib(40))

The Output

55
6765
832040
102334155

As you can see, the decorator cache the result of the fib function and returns them if the same input is given again. This improves the efficiency and speed of the program.

• Validation: You can use decorators to check the validity of the input or output of a function. For example, you can write a decorator that raises an exception if the input is not a positive integer or if the output is not a list. This can prevent errors and ensure the correctness of your code.

# Define a decorator that validates the input and output of a function 
def validate(func):
    # Define a wrapper function that takes any number of arguments 
    def wrapper(*args, **kwargs):
        # Check if there is exactly one argument 
        if len(args) != 1 or kwargs:
            raise ValueError("The function expects exactly one argument")
        # Check if the argument is a string 
        if not isinstance(args[0], str):
            raise TypeError("The argument must be a string")
        # Call the original function and store the output 
        result = func(*args, **kwargs)
        # Check if the output is also a string 
        if not isinstance(result, str):
            raise TypeError("The output must also be a string")
        # Return the output 
        return result 
    # Return the wrapper function 
    return wrapper 

# Use the decorator on a function that reverses a string 
@validate 
def reverse(s):
    # Initialize an empty string 
    r = ""
    # Loop from the last character to the first character 
    for i in range(len(s) - 1, -1, -1):
        # Append the character to the reversed string 
        r += s[i]
    # Return the reversed string 
    return r 

# Call the decorated function with a valid input 
print(reverse("hello"))

# Call the decorated function with an invalid input 
print(reverse(123))

The Output

olleh
Traceback (most recent call last):
  File "<stdin>", line 35, in <module>
  File "<stdin>", line 10, in wrapper
TypeError: The argument must be a string

As you can see, the decorator validates the input and output of the reverse function and raises an exception if the input is not a string.

• Authorization: You can use decorators to control the access to a function based on some conditions. For example, you can write a decorator that checks if the user has the required permissions or credentials before executing a function. This decorator can enhance the security and privacy of your program.

# Define a decorator that authorizes the access to a function based on some conditions 
def authorize(func):
    # Define a wrapper function that takes any number of arguments 
    def wrapper(*args, **kwargs):
        # Get the username and password from the arguments 
        user = args[0]
        password = args[1]
        # Checking if the username and password given are correct 
        if user == "admin" and password == "1234":
            # Call the original function and store the output 
            result = func(*args, **kwargs)
            # Return the output 
            return result 
        else:
            # Deny the access and print a message 
            print("Access denied")
            # Return None 
            return None 
    # Return the wrapper function 
    return wrapper 

# Use the decorator on a function that prints a secret message 
@authorize 
def print_secret(user, password):
    # Print the secret message 
    print("The secret message is: Hello world!")
    # Return None 
    return None 

# Call the decorated function with a valid username and password 
print_secret("admin", "1234")

# Call the decorated function with an invalid username and password 
print_secret("user", "4321")

The Output

The secret message is: Hello world!

Access denied

The decorator authorizes access to the print_secret function based on the username and password given. It returns access denied if the wrong details are given.

Creating Custom Decorators

To create a decorator, you define a function that accepts another function as its argument. Within this wrapper function, you can add the desired functionality before and after calling the original function. Here's a simple example of an authorization decorator:

# Define a custom decorator that adds a greeting message
def greet(func):
    # Define a wrapper function that takes any number of arguments
    def wrapper(*args, **kwargs):
        # Print a greeting message before calling the function
        print("Hello, welcome to the program!")
        # Call the original function and store the output
        result = func(*args, **kwargs)
        # Print another greeting message after calling the function
        print("Thank you for using the program!")
        # Return the output
        return result
    # Return the wrapper function
    return wrapper

# Use the custom decorator on any function you want
@greet
def add(a, b):
    # Return the sum of two numbers
    return a + b

# Call the decorated function
add(3, 4)

Output

Hello, welcome to the program!
Thank you for using the program!
7

Applying Multiple Decorators (i.e. Chaining and Nesting Decorators)

Multiple decorators can be applied to a single function by stacking them on top of each other. The order of execution follows a bottom-up pattern, with the innermost decorator being applied first.

• Chained decorators: If you want to apply both the log and cache decorators from the previous examples to a function that calculates the power of a number, you can do so as follows:

# Import the log and cache decorators from the previous examples
from log import log
from cache import cache

# Use both the log and the cache decorators on a function that calculates the power of a number
@log
@cache
def power(x, n):
    # Return x to the power of n
    return x ** n

# Call the decorated function multiple times with different inputs
print(power(2, 10))
print(power(3, 5))
print(power(2, 10))

The output of this code

Input: (2, 10), {}
Output: 1024
Execution time: 1.1920928955078125e-05 seconds
1024
Input: (3, 5), {}
Output: 243
Execution time: 9.5367431640625e-06 seconds
243
Input: (2, 10), {}
Output: 1024
Execution time: 9.5367431640625e-06 seconds
1024

As you can see, the log decorator prints the input, output, and execution time of the power function, and the cache decorator stores and returns the results if the same input is given again. The order of the decorators is important, as the first decorator applied will be the last one executed. In this case, the cache decorator is applied before the log decorator, so the log decorator will not print anything if the result is cached.

• Nested decorators: If you want to create a decorator that takes a parameter and modifies another decorator based on that parameter. For example, you want to create a decorator that limits the number of times a function can be called based on a given limit. You can do something like this:

# Define a decorator that takes a parameter and modifies another decorator based on that parameter
def limit(n):
    # Define another decorator that limits the number of times a function can be called
    def limiter(func):
        # Create a counter to keep track of the number of calls
        count = 0
        # Define a wrapper function that takes any number of arguments
        def wrapper(*args, **kwargs):
            # Use the nonlocal keyword to access and modify the counter variable
            nonlocal count
            # Check if the counter is less than the limit
            if count < n:
                # Increment the counter by one
                count += 1
                # Call the original function and store the output
                result = func(*args, **kwargs)
                # Return the output
                return result 
            else:
                # Print a message that the limit is reached 
                print("The limit is reached")
                # Return None 
                return None 
        # Return the wrapper function 
        return wrapper 
    # Return the modified decorator 
    return limiter 

# Use the limit decorator with a parameter of 3 on a function that prints hello 
@limit(3)
def hello():
    # Print hello 
    print("Hello")

# Call the decorated function multiple times 
hello()
hello()
hello()
hello()

Output of this code

Hello
Hello
Hello
The limit is reached

As you can see, the limit decorator takes a parameter of 3 and modifies another decorator, limiting the number of times the hello function can be called to 3. After that, any further calls will print a message that the limit is reached.

Utilizing Built-in Decorators

Built-in decorators are Python functions and can be used to modify other functions or classes. Some of the most common built-in decorators are:

• @property: This decorator turns a method into an attribute that can be accessed or modified without parentheses. For example, you can use this decorator to create a read-only property that returns the area of a circle:

class Circle:
    def __init__(self, radius):
        self.radius = radius

    @property
    def area(self):
        return 3.14 * self.radius ** 2

c = Circle(5)
print(c.area) # prints 78.5
c.area = 10 # raises AttributeError

• @classmethod: This decorator turns a method into a class method that can be called on the class itself, rather than on an instance. The first argument of a class method is the class itself, usually named cls. For example, you can use this decorator to create an alternative constructor for a class:

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    @classmethod
    def from_birth_year(cls, name, year):
        return cls(name, 2023 - year)

p1 = Person("Alice", 25)
p2 = Person.from_birth_year("Bob", 1998)
print(p1.name, p1.age) # prints Alice 25
print(p2.name, p2.age) # prints Bob 25

• @staticmethod: This decorator turns a method into a static method that can be called on either the class or an instance but does not receive any implicit argument. A static method is typically a utility function that does not depend on the state of the class or instance. For example, you can use this decorator to create a helper function that validates an email address:

class User:
    def __init__(self, name, email):
        self.name = name
        self.email = email

    @staticmethod
    def is_valid_email(email):
        return "@" in email and "." in email

u1 = User("John", "[e-mail]")
u2 = User("Jane", "jane")
print(u1.is_valid_email(u1.email)) # prints True
print(u2.is_valid_email(u2.email)) # prints False
print(User.is_valid_email("[e-mail]")) # prints True

These are some of the built-in decorators in Python.

Pros and Cons of Decorators

1. Benefits: Decorators promote code reusability by separating concerns. They allow you to modularize functionality and apply it consistently across multiple functions.

2. Drawbacks: Overuse of decorators can lead to code complexity. Additionally, some decorators might introduce overhead, impacting performance.

Generators

Generators are an efficient way to create iterators in Python. They offer a memory-friendly approach to working with data sequences, especially when dealing with large or infinite datasets. Generators enable lazy evaluation, producing elements one at a time as needed.

Generator Functions

Generator functions are defined like regular functions, but use the "yield" keyword instead of "return". This suspends the function's state and allows it to resume from where it left off when the next value is requested.

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for num in countdown(5):
    print(num)  

# Outputs: 5, 4, 3, 2, 1

Iterating Over Generators

Generators are iterable; you can use "for" loops to iterate over their values. When there are no more values to yield, a "StopIteration" exception is raised.

Generator Expressions

Generator expressions are just like list comprehensions but generate values on the fly without creating an entire list in memory. They must be in parentheses instead of square brackets.

squared = (i ** 2 for i in range(10))
for num in squared:
    print(num)  

# Outputs: 0, 1, 4, 9, 16, 25, 36, 49, 64, 81

Use Cases of Generators

Generators can be used for various purposes, such as:

• Processing large data sets: Generators can assist in processing large datasets that do not fit into memory, such as log files, web pages, or database records. You can use generators to read and process the data in chunks without loading the entire data set into memory at once. For example, you can write a generator function that reads a file line by line and yields each line as a string:

def read_file(filename):
    # Open the file in read mode
    with open(filename, "r") as file:
        # Loop through each line in the file
        for line in file:
            # Yield the line as a string
            yield line

• Implementing iterators: Generators can help you implement custom iterators that produce a sequence of values when iterated over. You can use generators to create iterators that follow a specific pattern, logic, or algorithm. For example, you can write a generator function that yields the Fibonacci numbers:

def fibonacci(n):
    # Initialize the first two Fibonacci numbers
    a = 0
    b = 1
    # Loop until the nth Fibonacci number
    for i in range(n):
        # Yield the current Fibonacci number
        yield a
        # Update the next two Fibonacci numbers
        a, b = b, a + b

• Creating pipelines: Generators can help you create pipelines that transform and filter data in multiple stages. You can use generators to chain multiple generator functions or expressions together and pass the output of one generator as the input of another. For example, you can write a pipeline that reads a file, splits each line into words, converts each word to lowercase, and removes any punctuation:

import string

# Create a generator that reads a file line by line
lines = read_file("text.txt")

# Create a generator that splits each line into words
words = (word for line in lines for word in line.split())

# Create a generator that converts each word to lowercase
lowercase = (word.lower() for word in words)

# Create a generator that removes any punctuation from each word
no_punctuation = (word.strip(string.punctuation) for word in lowercase)

# Iterate over the final generator and print the words
for word in no_punctuation:
    print(word)

Composing Generators

The "yield from" statement simplifies the composition of generators. It allows one generator to delegate part of its operations to another generator.
For example, suppose you have two generators that yield different types of fruits:

def apples():
    # Yield some apples
    yield "red apple"
    yield "green apple"
    yield "yellow apple"

def oranges():
    # Yield some oranges
    yield "navel orange"
    yield "blood orange"
    yield "mandarin orange"

Suppose you want to create a generator that yields all the fruits from both generators. One way to do this is to use a for loop and yield each value from the other generators:

def fruits():
    # Yield all the fruits from apples()
    for apple in apples():
        yield apple
    # Yield all the fruits from oranges()
    for orange in oranges():
        yield orange

Nonetheless, this can become redundant if you have many generators to delegate to. A more straightforward and refined approach is to employ the yield from statement, which lets you yield all the values from another generator in a single line of code.

def fruits():
    # Yield all the fruits from apples() with one line of code
    yield from apples()
    # Yield all the fruits from oranges() with one line of code
    yield from oranges()

The "yield from" statement yields all the values from another generator and forms a clear two-way connection between them. This means you can also send values or raise exceptions to or from the delegated generator, which will be processed correctly.
For example, suppose you have a generator that calculates the factorial of a number:

def factorial(n):
    # Initialize the result to 1
    result = 1
    # Loop from 1 to n
    for i in range(1, n + 1):
        # Multiply the result by i
        result *= i
        # Yield the intermediate result and receive a new value for n if any
        n = yield result

This generator can receive a new value for n using the send() method. For example:

# Create a generator object
gen = factorial(5)
# Get the first value (1)
print(next(gen)) # Output: 1
# Send a new value for n (3) and get the next value (2)
print(gen.send(3)) # Output: 2 
# Get the next value (6)
print(next(gen)) # Output: 6 
# Get the next value (StopIteration exception)
print(next(gen)) # Output: StopIteration 

Suppose you want to create a generator that delegates to factorial(), but also prints some messages before and after each value. You can use the "yield from" statement to do this:

def print_factorial(n):
    # Print a message before delegating to factorial()
    print(f"Calculating factorial of {n}")
    # Delegate to factorial() and receive its output as result 
    result = yield from factorial(n)
    # Print a message after delegating to factorial()
    print(f"Factorial of {n} is {result}")

This generator will yield all the values from factorial() and pass any values or exceptions sent to or from it. For example:

# Create a generator object 
gen = print_factorial(5) 
# Get the first value (1) 
print(next(gen)) 
# Output: Calculating factorial of 5 
# 1 
# Send a new value for n (4) and get the next value (2) 
print(gen.send(4)) # Output: 2 
# Get the next value (6) 
print(next(gen)) # Output: 6 
# Get the next value (24) 
print(next(gen)) # Output: 24 
# Get the next value (StopIteration exception with result 24) 
print(next(gen)) 
# Output: Factorial of 4 is 24 
# StopIteration(24) 

Advantages and Limitations of Generators

1. Benefits: Generators save memory by generating values as needed. They are ideal for situations where full data preloading is not feasible.

2. Limitations: Generators can only be used once; you cannot iterate through them multiple times. Additionally, their syntax may be confusing for some developers.

Use Cases Examples

Let's examine three practical applications of decorators and generators.

Example 1: Developing a Timing Decorator for Performance Testing

import time

def timing_decorator(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        print(f"Execution time: {end_time - start_time} seconds")
        return result
    return wrapper

@timing_decorator
def long_running_function():
    # Simulate a time-consuming operation
    time.sleep(3)
    return "Operation complete"

Example 2: Designing a Generator for Pagination in Web Applications

def paginate(items, items_per_page):
    start = 0
    while start < len(items):
        yield items[start:start + items_per_page]
        start += items_per_page

items = range(1, 21)
for page in paginate(items, 5):
    print(page)  # Outputs: [1, 2, 3, 4, 5], [6, 7, 8, 9, 10], ...

Example 3: Using a Generator to Parse a Large Log File

def parse_log_file(file_path):
    with open(file_path, 'r') as log_file:
        for line in log_file:
            yield line.strip()

for log_entry in parse_log_file('large_log.log'):
    process_log_entry(log_entry)

Comments

[Rhiannon Hecht]

This was very useful! THANKS!

[Andy DataScientist]

This is Excellent! Very very well written and thoroughly explained. I am happy I have found this article. It clears up a lot of my doubts.
Thanks!!