Bas Steins

Posted on Feb 24, 2022 • Originally published at bas.codes

Understanding Decorators in Python

#python #functional

What are decorators

Decorators are wrappers around Python functions (or classes) that change how these classes work. A decorator abstracts its own functioning as far away as possible. The Decorator notation is designed to be as minimally invasive as possible. A developer can develop his code within his domain as he is used to and only use the decorator to extend the functionality. Because this sounds very abstract, let's look at some examples.

In Python, decorators are used primarily to decorate functions (or methods, respectively). Maybe, one of the most commonly used decorators is the @property decorator:

class Rectangle:
    def __init__(self, a, b):
        self.a = a
        self.b = b

    @property
    def area(self):
        return self.a * self.b

rect = Rectangle(5, 6)
print(rect.area)
# 30

As you see in the last line, you can access the area of our Rectangle like an attribute, i.e., you don't have to call the area method. Instead, when accessing area like an attribute (without the ()) the method is called implicitly because of the @property decorator.

How does it work?

Writing @property in front of a function definition is the equivalent to writing area = property(area). In other words: property is a function that takes another function as an argument and returns a third function. And this is exactly what decorators do.

As a result, decorators change the behaviour of the decorated function.

Writing Custom Decorators

Retry Decorator

With that vague definition, let's write our own decorators to understand how they work.

Let's say we have a function that we want to retry if it fails. We need a function (our decorator) that calls our function once or twice (depending on whether it failed the first time).

According to our initial definition of a decorator, we could write this simple decorator like this:

def retry(func):
    def _wrapper(*args, **kwargs):
        try:
            func(*args, **kwargs)
        except:
            time.sleep(1)
            func(*args, **kwargs)
    return _wrapper

@retry
def might_fail():
    print("might_fail")
    raise Exception

might_fail()

retry is the name of our decorator, which accepts any function as an argument (func). Inside the decorator, a new function (_wrapper) is defined and returned. It can look somewhat unfamiliar to define a function inside another function at first sight. However, this is syntactically perfectly fine and has the advantage that our _wrapper function is just valid inside the namespace of our retry decorator.

Note that in this example, we decorated our function just with @retry. There are no parentheses (()) after the @retry decorator. Thus, when calling our might_fail() function, the retry decorator is called with our function (might_fail) as a first argument.

In total, we handle three functions here:

retry
_wrapper
might_fail

In some cases, we need the decorator to accept arguments. In our case, we could make the number of retries a parameter. However, a decorator must take our decorated function as the first argument. Remember that we did not need to call our decorator when decorating a function with it, i.e. we just wrote @retry as opposed to @retry() in front of our decorated function definition.

The decorator is nothing else than a function (which accepts another function as argument)
The decorator is used by putting it in front of a function definition without calling it

Hence, we could introduce a fourth function which accepts the parameter we want as configuration and returns a function that actually is a decorator (which accepts another function as argument).

Let's try this:

def retry(max_retries):
    def retry_decorator(func):
        def _wrapper(*args, **kwargs):
            for _ in range(max_retries):
                try:
                    func(*args, **kwargs)
                except:
                    time.sleep(1)
        return _wrapper
    return retry_decorator


@retry(2)
def might_fail():
    print("might_fail")
    raise Exception


might_fail()

Tearing that one apart:

On the first level, we have a function called retry.
retry accepts an arbitrary argument (max_retries in our case) and returns a function
retry_decorator is the function returned by retry and is our actual decorator
_wrapper works in the same way as before (it now just obeys the maximum number of retries)

That's for the definition of our decorator.

might_fail is decorated by a function call this time, i.e. @retry(2).
retry(2) cause the function retry to be called and it returns the actual decorator
might_fail is eventually decorated by retry_decorator as this function is the result of the retry(2) call.

Timer Decorator

Here is another example of a useful decorator: Let's create a decorator which measures the runtime of the functions decorated with it.

import functools
import time

def timer(func):
    @functools.wraps(func)
    def _wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        runtime = time.perf_counter() - start
        print(f"{func.__name__} took {runtime:.4f} secs")
        return result
    return _wrapper

@timer
def complex_calculation():
    """Some complex calculation."""
    time.sleep(0.5)
    return 42

print(complex_calculation())

Output:

complex_calculation took 0.5041 secs
42

As we see, the timer decorator executes some code before and after the decorated function and works in the exact same way as in the last example.

`functools.wraps`

You might have noticed that the _wrapper function itself is decorated with @functools.wraps. This does not in any way change the logic or functionality of our timer decorator. You could as well decide to not use functools.wraps.

However, since our @timer decorator could have as well been written as: complex_calculation = timer(complex_calculation), the decorator necessarily changes our complex_calculation function. Insbesondere, it changes some of the magic reflection attributes:

__module__
__name__
__qualname__
__doc__
__annotations__

When using @functools.wraps, these attributes are set back to their originals

Without @functools.wraps

print(complex_calculation.__module__)       # __main__
print(complex_calculation.__name__)         # wrapper_timer
print(complex_calculation.__qualname__)     # timer.<locals>.wrapper_timer
print(complex_calculation.__doc__)          # None
print(complex_calculation.__annotations__)  # {}

With @functools.wraps

print(complex_calculation.__module__)       # __main__#
print(complex_calculation.__name__)         # complex_calculation
print(complex_calculation.__qualname__)     # complex_calculation
print(complex_calculation.__doc__)          # Some complex calculation.
print(complex_calculation.__annotations__)  # {}

Class Decorators

So far, we have just looked at decorators for functions. It's, however, possible to decorate classes, too.

Let's take the timer decorator from the example above.
It's perfectly fine to wrap a class with this decorator like so:

@timer
class MyClass:
    def complex_calculation(self):
        time.sleep(1)
        return 42

my_obj = MyClass()
my_obj.complex_calculation()

The result?

Finished 'MyClass' in 0.0000 secs

So, there is obviously no timing printed for our complex_calculation method. Remember that the @ notation is just the equivalent for writing MyClass = timer(MyClass), i.e., the decorator will get called only when you "call" the class. Calling a class means instantiating it, so the timer is only executed at the line my_obj = MyClass().

Class methods are not automatically decorated when decorating a class. To put it simple, using a normal decorator to decorate a normal class decorates its constructor (__init__ method), only.

However, you can change the behaviour of a class as a whole by using another form of a constructor. However, let's first see if decorators can work the other way around, i.e. whether we can decorate a function with a class. Turns out we can:

class MyDecorator:
    def __init__(self, function):
        self.function = function
        self.counter = 0

    def __call__(self, *args, **kwargs):
        self.function(*args, **kwargs)
        self.counter+=1
        print(f"Called {self.counter} times")


@MyDecorator
def some_function():
    return 42


some_function()
some_function()
some_function()

Output:

Called 1 times
Called 2 times
Called 3 times

The way this works:

__init__ is called when decorating some_function. Again, remember that decorating is just like some_function = MyDecorator(some_function).
__call__ is called when an instance of a class is used, like calling a function. As some_function is now an instance of MyDecorator but we still want to use it as a function, the DoubleUnderscore magic method __call__ is responsible for this.

Decorating a class in Python, on the other hand works by changing the class from the outside (i.e., from the decorator).

Consider this:

def add_calc(target):

    def calc(self):
        return 42

    target.calc = calc
    return target

@add_calc
class MyClass:
    def __init__():
        print("MyClass __init__")

my_obj = MyClass()
print(my_obj.calc())

Output:

MyClass __init__
42

Again, if we recap the definition of a decorator, everything which happens here follows the same logic:

my_obj = MyClass() is calling the decorator first
the add_calc decorator patches the calc method to the class
eventually, the class is instantiated by using the constructor.

You can use decorators to change classes in a way inheritance would do. If this is a good choice or not heavily depends on the architecture of your Python project as a whole. The standard library's dataclass decorator is an excellent example of a sensible usage choosing decorators over inheritance. We'll discuss that in a second.

Using decorators

decorators in Python's standard library

In the following sections, we will get to know a few of the most popular and most useful decorators that are already included in the standard library.

property

As already discussed, the @property decorator is probably one of the most commonly used decorators in Python.
It's purpose is that you can access the result of a method like an attribute. Of course, there is also a counterpart to @property so that you could call a method behind the scenes when performing an assignment operation.

class MyClass:
    def __init__(self, x):
        self.x = x

    @property
    def x_doubled(self):
        return self.x * 2

    @x_doubled.setter
    def x_doubled(self, x_doubled):
        self.x = x_doubled // 2

my_object = MyClass(5) 
print(my_object.x_doubled)  #  10  
print(my_object.x)          #  5  
my_object.x_doubled = 100   #    
print(my_object.x_doubled)  #  100 
print(my_object.x)          #  50

staticmethod

Another familiar decorator is staticmethod. This decorator is used when you want to call a function defined inside a class without instantiating the class:

class C:
    @staticmethod
    def the_static_method(arg1, arg2):
        return 42

print(C.the_static_method())

functools.cache

When you deal with functions that carry on a complex calculation, you might want to cache its result.

You could do something like this:

_cached_result = None
def complex_calculations():
    if _cached_result is None:
        _cached_result = something_complex()
    return _cached_result

Storing a global variable like _cached_result, checking it for None, and putting the actual result into that variable if not present are repetitive tasks. This makes an ideal candidate for a decorator. Luckily, there is a decorator in Python's standard library which does exactly this for us:

from functools import cache

@cache
def complex_calculations():
    return something_complex()

Now, whenever you call complex_calculations(), Python will check for a cached result first before it calls something_complex. If there is a result in the cache, something_complex will not get called twice.

dataclasses

In the section about class decorators we saw that decorators can be used to modify the behaviour of classes in the same way inheritance would change it.

The dataclasses module in the standard library is a good example when using a decorator is preferable over using inheritance. Let's first see how to use dataclasses in action:

from dataclasses import dataclass

@dataclass
class InventoryItem:
    name: str
    unit_price: float
    quantity: int = 0

    def total_cost(self) -> float:
        return self.unit_price * self.quantity


item = InventoryItem(name="", unit_price=12, quantity=100)
print(item.total_cost())    # 1200

On the first sight, the @dataclass decorator only added a constructor for us, so we avoided boiler plate code like this:

...
    def __init__(self, name, unit_price, quantity):
        self.name = name
        self.unit_price = unit_price
        self.quantity = quantity
...

However, if you decide to build a REST-API for your Python project and need to convert your Python objects into JSON strings.

There is a package called dataclasses-json (not in the standard library) which decorates dataclasses and provide the serialisation and deserialisation of objects to JSON strings and vice versa.

Let's see how that looks:

from dataclasses import dataclass
from dataclasses_json import dataclass_json

@dataclass_json
@dataclass
class InventoryItem:
    name: str
    unit_price: float
    quantity: int = 0

    def total_cost(self) -> float:
        return self.unit_price * self.quantity


item = InventoryItem(name="", unit_price=12, quantity=100)

print(item.to_dict())
# {'name': '', 'unit_price': 12, 'quantity': 100}

There are two takeaways here:

decorators can be nested. The order of their appearance is important.
the @dataclass_json decorator added a method called to_dict to our class

Of course, we could have written a mixin class that does the heavy work of implementing a data type safe to_dict method and then let our InventoryItem class inherit from that mixin.

In the present case, however, the decorator only adds a technical functionality (as opposed to an extension within the subject domain). As a result, we can simply switch the decorator on and off without our domain application changing its behaviour. Our "natural" class hierarchy is preserved and no changes need to be made to the actual code. We could also add the dataclasses_json decorator to a project without changing existing method bodies.

In such a case, changing a class with a decorator is much more elegant (because it is more modular) than inheriting or using mixins.