Real case of Python Metaclass application

Metaclass __prepare__()

>>> A tangible example of metaclasses with Python

Why meta?

"In Python everything is an object and every object is an instance of something we call a class, which in turn is an object and therefore an instance of something called a metaclass which in turn is an object.
In python, therefore, everything is an object."
"Someone once said."

Let's check it out.
Taking the int(1) object we can evaluate that it is an instance of the int class.

int(1)
# Output: 1
...
int(1).__class__
# Output: <class 'int'>
...
isinstance(int(1), int)
# Output: True

That is, int is the class of int(1).
Let's iterate, with the same idea, over int. It's supposed to be an object.

isinstance(int, object)
# Output: True

So, which class is it an instance of?

type(int)
# Output: <class 'type'>
isinstance(int, type)
# Output: True

So it seems that type, apart from being a 'callable' that retrieves the type of an object, behaves like the class of something that is a class.

That's right, in the same way that object is the base class of any other class, type is the base metaclass of any other metaclass.

Alternative use of type:
we can use type to dynamically create classes. A quick example:

CustomInteger = type(
   "Dummy",
   (int,),
   {"custom_to_str": lambda self, custom_msg: f"{custom_msg} {str(self)}"}
)
custom_int = CustomInteger(1)
print(custom_int, CustomInteger)
# Prints: 1 <class '__main__.Dummy'>
print(custom_int.custom_to_str(custom_msg="this is my custom int ->"))
# Prints this is my custom int -> 1

Let's notice that type admits as parameters the name of the class, an n-tuple with the bases and a dictionary with the class attributes.
In this case we have used it to dynamically define the custom_to_str method.

Why I should learn meta?

Many times it is not easy to explain what real use cases metaprogramming with Python can have. Nor explain or justify its use in a project that does not require in-depth study.

Maybe the simple pleasure of 'playing' and learning more advanced concepts is enough reason to get interested.
But, if that were not enough, we are going to present a challenge and we will see
how to solve it from a metaprogramming approach.

Goal

Let's imagine that in our project we represent certain logic through a class that has different methods that are executed with a certain flow order.
We may be interested in controlling/monitoring, in a certain way, what is happening. For example, log the invocation of each method. Or for example validate
that the types of the parameters of a function or method correspond to the annotations defined in the signatures. Or for example record the execution time of each method.
For this kind of problem, a pythonic approach is to use decorators for each method. Decorators are wrappers, and they can encapsulate the execution of one function inside another,
which generally modifies, registers, caches, etc...

Statement: Suppose that our class involves a good number of methods, and we don't feel like decorating each of them.
Let's try to define a decoration policy that affects all the methods of the class.

The idea is to avoid this,

class A:
  @my_decorator1
  @my_decorator2
  def method_1(self): ...
  ...
  @my_decorator1
  @my_decorator2
  def method_n(self): ...

and that somehow the body of the function is cleaner without repeating the decorators' declaration.

Approach

Disclaimer: this is not intended to be a complete python metaprogramming tutorial.
Among other things, it is not a trivial question and its study requires patience and time to assimilate.
Based on my experience I have reached a certain point and I have created my own idea but I do not consider myself an expert in this area.

For the interested reader, I share what I believe is the best
material to go into depth about it. This is the Ionel blog.
Although he makes an introduction to metaclasses from scratch, he goes through and explains in quite detail the flow of the methods that participate in the creation, preparation, instantiation and so on,
both at the class, object, and metaclass level. If anyone is interested, I invite you to play with the examples that he proposes since they allow you to follow the creation flows of objects, classes and metaclasses.
Next we are going to look at the diagrams that Ionel exposes since they are quite illustrative to minimally explain the theory behind the construction flow.

Creation of an object

When we instantiate an object the following happens:

1. Metaclass.__call__, triggers the instantiation process and retrieves the class object to execute Class.__new__.
2. Class.__new__, constructs an instance and returns it to Metaclass.__call__ for initialization.
3. Class.__init__, initializes the instance with the instantiation parameters.
4. Once the object is constructed and initialized, the instantiation call is returned from Metaclass.__call__.

Last pain in the ass before programming.

Creation of a class

A class can be instantiated directly as we saw before with type(NAME, BASES, ATTRIBUTES). Although generally
we declare it with class A:.... If declared, Python takes care of its instantiation.

In any case, the idea of instantiating a class is based on filling in attributes and methods defined or passed dynamically.
This process follows this flow.

1. M̀etaclass.__prepare__ is a special method. This is going to be the cornerstone for the solution that we are going to propose. It is a static method: it can be called from outside the metaclass (see diagram).
   1. You must provide a DictLike object. This object is going to be the blank template to create the class from which to 'start doing things'. It is as if it were a painter's blank canvas or a potter's ball of clay.
   2. It does not have to be an empty dictionary although by default it is.
   3. You can customize DictLike methods like __set_item__, __get_item__, etc...
2. Metametaclass.__call__ triggers the recovery of the metaclass.
3. Metaclass.__new__ invokes the instantiation of the class-instance and takes into account the DictLike returned by Metaclass.__prepare__.
4. Metaclass.__init__ is executed and parameters of the class-instance are initialized.
5. The class-instance is released from Metametaclas.__call__

Note: there is an analogy with the previous diagram, removing the __prepare__ part.

Take a Break

If at this point it is confusing to assimilate the concepts, I recommend practicing with Ionel's examples.

Resolution

Below we are going to address a possible solution to the problem, step by step.

Metaclass definition

Import this. We only need those external libraries.

from functools import wraps
from time import sleep
from typing import Self, Generator

import pytest

We first define the DictLike class. Here is the key.

class WrapStackPreparer(dict):

    def __init__(self, array_wraps):
        super().__init__()
        self.array_wraps = array_wraps

    def __setitem__(self, key, value):
        if callable(value):
            for wrap in self.array_wraps:
                value = wrap(value)
        dict.__setitem__(self, key, value)

Let's remember that we want to apply a list of wrappers.

1. In the __init__ we expect a parameter array_wraps that contains each of the wrappers to apply.
2. In __set_item__ we alter the default behavior of the dict class.
   1. This method will be executed every time a class attribute is registered or a method in the class is being 'prepared'.
   2. If it is a class attribute, nothing will happen. On the other hand, for the methods, which are callable, we will wrap them.
   3. Finally we invoke the dict.__setitem__ that will register the already encapsulated method.

Now let's define the metaclass.

class MetaWrap(type):

    @classmethod
    def __prepare__(metacls, name, bases, **kwargs):
        return WrapStackPreparer(**kwargs)

    def __new__(metacls, name, bases, classdict, **kwargs):
        result = type.__new__(metacls, name, bases, classdict)
        return result

Let us note that:

1. We set type as a base.
2. We define __prepare__ as classmethod returning the DictLike and propagating **kwargs so that, as we will see later, we send the array of wraps.
3. We define a standard __new__, without customizing anything, to build the class.

Use cases

Logger

We define a basic logging wrapper.

def logged(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print(f"Calling {func.__name__}")
        result = func(*args, **kwargs)
        print(f"Finished {func.__name__}")
        return result

    return wrapper

Now we can create a class that uses it.

class LoggedClass(metaclass=MetaWrap, array_wraps=[logged]):
    def method1(self):
        sleep(0.1)

    def method2(self):
        sleep(0.15)

    def main(self):
        self.method1()
        self.method2()

To use the metaclass we created we pass metaclass=MetaWrap. And to propagate the list of wraps so that it can reach the __prepare__
we define the parameter array_wraps=[logged] as kwarg.

If we now execute,

logged_cl = LoggedClass()
print("Calling directly")
logged_cl.method1()
logged_cl.method2()
print("Calling from main")
logged_cl.main()

we obtain as output,

Calling directly
Calling method1
Finished method1
Calling method2
Finished method2
Calling from main
Calling main
Calling method1
Finished method1
Calling method2
Finished method2
Finished main

and we check that it describes the execution flow as we expected.

Type validator

We now write a slightly more elaborate wrapper that takes into account the types declared in the signature of a method and encapsulates its execution
to evaluate that the input and output of a method fulfill with the type annotations.
To do this, we inspect the __annotations__ attribute, typical of any callable. The complete wrapper looks like this,

def validate_annotations(func, validate_return=False):
    def validate(val, typ):
        if typ is Self:
            return

        if typ is None:
            typ = typ.__class__
        assert isinstance(val, typ), (
            f"Error {"invoking" if validate_return is False else "returning value from"} {func.__name__}"
            f"\n"
            f"Expected {typ} type, got {val} item of type {val.__class__.__name__}")

    @wraps(func)
    def wrapper(*args, **kwargs):

        annotations = func.__annotations__

        for k, val in kwargs.items():
            typ = annotations.get(k)
            validate(val, typ)

        for i, arg in enumerate(args):
            validate(arg, list(annotations.values())[i])

        result = func(*args, **kwargs)

        validate(result, validate_return=annotations.get("return") is not None)

        return result

    return wrapper

• We have defined in the body of the wrapper the validate function, which throws an exception when some annotation is not met.
• It also offers a minimal readable explanation of why the failure occurred.
• Then the wrapper encapsulates the call and validates both positional arguments, key-value arguments and the type of returned object. > Note: This is a limited validator to exemplify, for a real use case we can go to the validator offered by pydantic, for example.

To test it, let's define the following class with annotated methods.

class ValidatedClass(metaclass=MetaWrap, array_wraps=[validate_annotations]):

    def concat_str(self: Self, s1: str, s2: str) -> str:
        return s1 + s2

    def sum_int(self: Self, gen: Generator) -> int:
        return sum(gen)

    def wrong_return_type(self: Self) -> None:
        return 1

And now, if we execute the following calls,

    validated_cl.concat_str("Hello", "World")
    validated_cl.sum_int(i ** 2 for i in range(10))
    with pytest.raises(AssertionError) as e:
        print("This should fail")
        validated_cl.wrong_return_type()
    print(e)

    with pytest.raises(AssertionError) as e:
        print("This should fail, also")
        validated_cl.concat_str("Hello", 1)
    print(e)

we get the following output.

## Validate benchmark
This should fail
<ExceptionInfo AssertionError("Error returning value from wrong_return_type\nExpected <class 'NoneType'> type, got 1 item of type int") tblen=3>
This should fail, also
<ExceptionInfo AssertionError("Error invoking concat_str\nExpected <class 'str'> type, got 1 item of type int") tblen=3>

• As you can see above, the first two calls are successful, since the parameters values are aligned with the types annotated in the definition of the methods.
• On the other hand, the last two calls raise exceptions. In the first case due to the function being poorly defined with respect to the annotation. In the second case due to misuse in the call.

Putting all together

We can define a class where we require applying both wrappers.

class LogValClass(metaclass=MetaWrap, array_wraps=[validate_annotations, logged]):
    def print_two_strings(self: Self, s1: str, s2: str) -> None:
        print(s1, s2)

And running the following,

 logval_cl = LogValClass()
 logval_cl.print_two_strings("Hello", "World")

we get the output,

Calling print_two_strings

Hello World

Finished print_two_strings

Conclusion

That's all. As we have previously commented, we have proposed a solution to the problem from the perspective of metaclasses.
If you liked it, I invite you to investigate and study by yourself.

If you have any correction or suggestion you can reach out to [e-mail] or [e-mail].

References

Lastly I quote the two main sources that have helped me learn about the subject and invent this example and write the post

• [1] Ionel's blog, as I said, seems to me to be the best resource to learn about metaclasses and deep dive into the subject.
• [2] The Python Morsels project, in general, has helped me a lot to take a step further and do advanced things with python. Pedagogically it seems brutal to me. I highly recommend his blog and tutorials in which he explains fundamental concepts in detail and also offers very well thought out exercises so that you can break your brain and fully understand why things happen. The bad thing is that some content is paid, yes.

From Python Morsels I have learnt what Metaclass.prepare() is and how it works so I'd like to congratulate Trey Hunner for his work.

Code

Complete code

from functools import wraps
from time import sleep
from typing import Self, Generator

import pytest


class WrapStackPreparer(dict):

    def __init__(self, array_wraps):
        super().__init__()
        self.array_wraps = array_wraps

    def __setitem__(self, key, value):
        if callable(value):
            for wrap in self.array_wraps:
                value = wrap(value)
        dict.__setitem__(self, key, value)


# The metaclass
class MetaWrap(type):

    @classmethod
    def __prepare__(metacls, name, bases, **kwargs):
        return WrapStackPreparer(**kwargs)

    def __new__(metacls, name, bases, classdict, **kwargs):
        result = type.__new__(metacls, name, bases, classdict)
        return result


# decorator
def logged(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print(f"Calling {func.__name__}")
        result = func(*args, **kwargs)
        print(f"Finished {func.__name__}")
        return result

    return wrapper


def validate_annotations(func):
    def validate(val, typ, validate_return=False):
        if typ is Self:
            return

        if typ is None:
            typ = typ.__class__
        assert isinstance(val, typ), (
            f"Error {"invoking" if validate_return is False else "returning value from"} {func.__name__}"
            f"\n"
            f"Expected {typ} type, got {val} item of type {val.__class__.__name__}")

    @wraps(func)
    def wrapper(*args, **kwargs):

        annotations = func.__annotations__

        for k, val in kwargs.items():
            typ = annotations.get(k)
            validate(val, typ)

        for i, arg in enumerate(args):
            validate(arg, list(annotations.values())[i])

        result = func(*args, **kwargs)

        validate(result, annotations.get("return"), validate_return=True)

        return result

    return wrapper


class LoggedClass(metaclass=MetaWrap, array_wraps=[logged]):
    def method1(self):
        sleep(0.1)

    def method2(self):
        sleep(0.15)

    def main(self):
        self.method1()
        self.method2()


class ValidatedClass(metaclass=MetaWrap, array_wraps=[validate_annotations]):

    def concat_str(self: Self, s1: str, s2: str) -> str:
        return s1 + s2

    def sum_int(self: Self, gen: Generator) -> int:
        return sum(gen)

    def wrong_return_type(self: Self) -> None:
        return 1


class LogValClass(metaclass=MetaWrap, array_wraps=[validate_annotations, logged]):
    def print_two_strings(self: Self, s1: str, s2: str) -> None:
        print(s1, s2)


if __name__ == '__main__':
    print("\n\n", "## Logger benchmark", sep="")
    logged_cl = LoggedClass()
    print("\n", "### Calling directly", sep="")
    logged_cl.method1()
    logged_cl.method2()
    print("\n", "### Calling from main", sep="")
    logged_cl.main()
    ...
    print("\n\n", "## Validate benchmark", sep="")
    validated_cl = ValidatedClass()

    validated_cl.concat_str("Hello", "World")
    validated_cl.sum_int(i ** 2 for i in range(10))
    with pytest.raises(AssertionError) as e:
        print("This should fail")
        validated_cl.wrong_return_type()
    print(e)

    with pytest.raises(AssertionError) as e:
        print("This should fail, also")
        validated_cl.concat_str("Hello", 1)
    print(e)

    ...
    print("\n\n", "## LogVal benchmark", sep="")
    logval_cl = LogValClass()
    logval_cl.print_two_strings("Hello", "World")