DEV Community: Bikramjeet Singh

Soft Deletes in Django

Bikramjeet Singh — Fri, 02 Jul 2021 19:13:16 +0000

What is Soft Deletion?

Sometimes, you want to delete something from your database without actually deleting it. You want to preserve it, in case you ever decide to restore it in the future, or to use it for some other purpose, such as analytics … but it should still appear as if it were actually deleted, not show up in unexpected places in your application. This is called the 'soft delete' pattern.

Instead of actually deleting a record from your database, which is irreversible, you simply 'mark' it as deleted, usually in another column in your table. Now the challenge remains, how to prevent soft-deleted records from 'leaking' into your application.

In this article, we will learn how to implement soft deletion in Django by taking the example of a simple note-taking app backend.

Soft Delete Model

We will start by creating a base SoftDeleteModel that can be inherited by the rest of our models.

class SoftDeleteModel(models.Model):

    is_deleted = models.BooleanField(default=False)

    def soft_delete(self):
        self.is_deleted = True
        self.save()

    def restore(self):
        self.is_deleted = False
        self.save()

    class Meta:
        abstract = True

Note that we have marked this model as abstract = True. This means that Django will not create a database table for it.

Now, we can create our models as subclasses of SoftDeleteModel to grant them the ability to be soft-deleted. Let's take the example of a Note model.

class Note(SoftDeleteModel):

    user = models.ForeignKey(User, on_delete=models.CASCADE, related_name="notes")
    title = models.CharField(max_length=32)
    content = models.CharField(max_length=255)

We can query our database filtering by is_deleted in order to exclude records that have been soft-deleted.

Note.objects.filter(is_deleted=False)

Trying It Out

Let's try playing around a bit with the code we've written so far. First, open up the Django shell by typing python manage.py shell in your terminal.

Import the models required:

from django.contrib.auth.models import User
from tutorialapp.models import Note

Since each note is foreign-keyed to a user, our first step is to create a User object:

john = User.objects.create_user('john', 'lennon@thebeatles.com', 'johnpassword')

Now we can create a couple of notes:

my_note = Note.objects.create(user=john, title="Strawberry Fields", content="Strawberry Fields Forever")
another_note = Note.objects.create(user=john, title="Here Comes The Sun", content="It's All Right")

You are now ready to soft delete and restore notes:

my_note.soft_delete()
my_note.restore()

You can query for all notes, whether they have been soft deleted or not:

Note.objects.all()

You can also filter only for notes that have not been soft deleted:

Note.objects.filter(is_deleted=False)

Soft Delete Manager

While our code is functionally correct, the disadvantage is that we will have to remember to filter by is_deleted=False each time we write a query.

We can improve upon this behaviour by creating a custom model manager to apply the filter automatically, behind the scenes. If you've used Django in the past, you might be familiar with statements that look like this: MyModel.objects.all(). The .objects part in the statement is the manager. Managers act as the 'bridge' between your Django code and the database. They control the database operations performed on the tables that they 'manage'.

Our new custom manager can be defined as:

class SoftDeleteManager(models.Manager):

    def get_queryset(self):
        return super().get_queryset().filter(is_deleted=False)

We will need to add the new manager to our SoftDeleteModel base class:

class SoftDeleteModel(models.Model):

    is_deleted = models.BooleanField(default=False)
    objects = models.Manager()
    undeleted_objects = SoftDeleteManager()

    def soft_delete(self):
        self.is_deleted = True
        self.save()

    def restore(self):
        self.is_deleted = False
        self.save()

    class Meta:
        abstract = True

Note that, since we have added a custom manager to our class, we are required to explicitly add the default objects manager as well.

Then, we can simply rewrite our query as,

Note.undeleted_objects.all()

to get a QuerySet of undeleted notes.

We can still use

Note.objects.all()

to get the full list of notes, including those that have been soft-deleted.

Handling Foreign Key Relationships

Now, what if you have multiple users, and you want to fetch all the notes belonging to a specific user? The naive approach is to simply write a query filtering against the user:

Note.objects.filter(user=john, is_deleted=False)

However, a more elegant and readable solution is to make use of the reverse relationships Django provides for this purpose.

john.notes.all()

Try soft deleting some of your notes and running this query. Do you notice something unusual about the results?

We find that the resultant QuerySet contains records that we had soft deleted. This is because Django is using the default objects manager to perform the reverse lookup, which, as you may recall, does not filter out soft deleted records.

How can we force Django to use our custom SoftDeleteManager to perform reverse lookups? We can simply replace the default objects manager in our SoftDeleteModel:

class SoftDeleteModel(models.Model):

    is_deleted = models.BooleanField(default=False)
    objects = SoftDeleteManager()
    all_objects = models.Manager()

    def soft_delete(self):
        self.is_deleted = True
        self.save()

    def restore(self):
        self.is_deleted = False
        self.save()

    class Meta:
        abstract = True

Now, the objects manager will automatically filter out soft-deleted objects when querying our database, ensuring they never leak into our application under any circumstances! If we want to, we can still include soft deleted objects in our queries by making use of the all_objects manager.

Note.all_objects.all()

Storing More Information

We've already got a pretty solid soft deletion framework in our Django app, but we can make one final improvement. Knowing whether a record is soft deleted or not is useful, but another piece of information that would be nice to know is when the record was soft deleted. For this, we can add a new a attribute deleted_at to our SoftDeleteModel:

deleted_at = models.DateTimeField(null=True, default=None)

We can also update our soft_delete and restore methods as follows:

def soft_delete(self):
    self.deleted_at = timezone.now()
    self.save()

def restore(self):
    self.deleted_at = None
    self.save()

For undeleted records, the value of deleted_at will be null, while for soft deleted records, it will contain the date and time at which it was deleted.

The addition of the new deleted_at attribute makes our previously created is_deleted attribute redundant, because we can simply perform a null-check on deleted_at to find out whether the record is soft deleted or not.

Our rewritten SoftDeleteModel now looks like this:

class SoftDeleteModel(models.Model):

    deleted_at = models.DateTimeField(null=True, default=None)
    objects = SoftDeleteManager()
    all_objects = models.Manager()

    def soft_delete(self):
        self.deleted_at = timezone.now()
        self.save()

    def restore(self):
        self.deleted_at = None
        self.save()

    class Meta:
        abstract = True

And our rewritten SoftDeleteManager looks like this:

class SoftDeleteManager(models.Manager):

    def get_queryset(self):
        return super().get_queryset().filter(deleted_at__isnull=True)

In addition to our previous capabilities, we can now also see the exact date and time at which our record was soft deleted:

my_note.deleted_at

In Conclusion

Soft deletion is a powerful software pattern that comes with several advantages, including better data preservation and restoration, history tracking, and faster recovery from failures.

At the same time, it should be used with care. Sensitive and personal data including payment-related information should always be hard-deleted. Users should always have the option to have their data permanently deleted, if they wish. Several jurisdictions around the world have information privacy and data protection laws that include the 'right to be forgotten', such as the European Union's GDPR. It might also make sense to periodically delete or archive data that is very old, to avoid eating up excess database storage space.

If you would like to view the complete source code for the example used in this tutorial, it is available on GitHub.

Image by Makalu from Pixabay

References and Further Reading

How to use managers in Django
What Are Soft Deletes, and How Are They Implemented?
Soft Deletion in Django

6 Nifty Python Language Features You Might Not Know About

Bikramjeet Singh — Sun, 07 Feb 2021 17:58:58 +0000

One of the strengths of the Python programming language is that it is both easy for beginners to learn and, at the same time, immensely rewarding for advanced users.

Data classes

Data classes are classes with the sole purpose of acting as containers of data. They usually do not contain any business logic inside them.

Python provides the @dataclass decorator which, when added to a class, automatically generates several useful methods so you don't have to write them out by hand. These include the __init__ method for generating a constructor, __str__ and __repr__ for generating string representations and __eq__ for checking instances of the class for equality.

Defining a data class is simple.

@dataclass()
class Article:
    title: str
    author: str
    published_on: datetime
    views: int
    likes: int

Optionally, you can also set default values for variables.

@dataclass()
class Article:
    title: str = "John Doe"
    author: str = "Unknown"
    published_on: datetime = datetime.now()
    views: int = 0
    likes: int = 0

For the full list of automatically generated methods, you can check out the official documentation.

Any & All

Sometimes you need to have a check on multiple conditions before performing an action, such as in an if statement. Python provides the boolean and and or operators for evaluating such logical expressions. However, when there are a very large number of conditions in play, your statement might begin to look a bit unwieldy.

The any() and all() methods, are simply a shorter and more readable way of evaluating a large number of boolean expressions at once. any() is equivalent to a series of or operations, while all() is equivalent to a series of and operations.

Suppose you are given a list of the marks a certain student scored ...

marks = [67, 85, 48, ]
passing_marks = 33

... and you want to check if the student qualifies to be promoted to the next year. One way to do this would be to simply join each comparison with the and operator:

if marks[0] > passing_marks and marks[1] > passing_marks and marks[2] > passing_marks:
    promoted = True
else:
    promoted = False

You can make this code slightly more readable by using the all() function:

if all([marks[0] > passing_marks, marks[1] > passing_marks, marks[2] > passing_marks]):
    promoted = True
else:
    promoted = False

Finally, you can combine it with a list comprehension to get a sweet, concise solution:

if all([subject_marks > passing_marks for subject_marks in marks]):
    promoted = True
else:
    promoted = False

The capabilities of these two functions can further be extended by combining them with the not operator.

Below is a quick summary of these methods.

any(): Returns True if at least one of the arguments evaluates to True
all(): Returns True if all of the arguments evaluate to True
not any(): Returns True if none of the arguments evaluate to True
not all(): Returns True if at least one of the arguments evaluates to False

Advanced Slice Operator

The slice operator is commonly used to access certain parts of a list or string.

string = "Python is awesome!"
string[3] # 'h'
string[4:7] # 'on '

It has many more advanced usages as well. For example, negative indexes can be used to slice from the end instead of the beginning.

string[-2] # 'e'

You can also specify a 'step' to skip over a certain number of elements while slicing.

string[2:9:2] # 'to s'

The step value can be negative. This causes the list/string to be sliced in reverse.

string[9:2:-2] # ' inh'

A common shortcut to reverse an array or list is to slice it with [::-1].

string[::-1] # '!emosewa si nohtyP'

Argument Unpacking

* and ** are special operators that allow multiple items to be packed into (or unpacked from) a single variable.

You might have seen the words *args and **kwargs present in the definitions of functions. When present in the context of a function definition, the * operator combines multiple arguments into a single tuple, while the ** operator combines multiple keyword arguments into a dictionary.

def product(*args):
    res = 1
    for arg in args:
        res *= arg
    return res

product(3, 4, 5)


def print_dict(**kwargs):
    for key in kwargs:
        print(f"{key}: {kwargs[key]}")

print_dict(firstname="Bikramjeet", lastname="Singh")

On the other had, when present in the context of a function call, they do the opposite - the * operator spreads the contents of a list/tuple into individual arguments, while the ** operator spreads the contents of a dictionary into individual keyword arguments.

list_of_nums = [3, 4, 5]
product(*list_of_nums)


d = {"firstname": "Bikramjeet", "lastname": "Singh"}
print_dict(**d)

In functions that have a large number of parameters, it is often convenient to collect them into a dictionary before passing them to the function.

def my_function(arg1, arg2, arg3, arg4 ... ):
    ...

params = {'arg1': ...}
my_function(**params)

Another use of these operators is to combine lists and dictionaries.

combined_list = [*l1, *l2]
combined_dict = {**d1, **d2}

Functools

Python supports higher order functions - functions that can take and return other functions. The concept of higher order functions is central to several other Python features, such as decorators.

The functools package provides useful helper functions for when you're working with higher order functions. Let's take a look at some of them.

partial

There are cases where you might want to create a 'specialized' version of an existing, more generalized function. This is done by 'freezing' the values of some of the base functions parameters.

Consider a simple function that calculates the nth power of a number:

def pow(base, exp):
    res = 1
    for i in range(exp):
        res *= base
    return res

pow(2, 3) # returns 8

The operation of squaring a number is common enough that it is worth it to create a dedicated function for it, simply so we don't have to pass 2 as an argument each time. However, instead of rewriting our pow function, we can simply reuse it with the functools.partial method.

square = functools.partial(pow, exp=2)
square(4) # returns 16, equivalent to pow(4, 2)

Similarly, we can also create a cube function:

cube = functools.partial(pow, exp=3)
cube(5) # returns 125, equivalent to pow(5, 3)

cached_property

This is a decorator that allows you to cache the return values of potentially expensive methods. For example, database calls tend to be relatively long-running operations, so it's a good idea to cache them if you don't anticipate their values to change very often.

@functools.cached_property
def my_expensive_method():
    ...
    return result

The method is evaluated in full the first time it is called, and its return value cached. The next time the same method is called, its result can simply be fetched from the cache.

Note that the cached_property decorator is only available from Python 3.8 onwards. For lower versions of Python, there are separate packages available.

total_ordering

When you define a custom class, especially one that is a container for numeric data of some sort, it is useful to define comparison methods __eq__ (equals), __gt__ (greater than), __ge__ (greater than or equal to), __lt__ (less than) & __le__ (less than or equal to) to make comparing objects of those classes easier.

class Distance:
    km: int # Kilometers
    m: int # Meters

    def __init__(self, km, m):
        self.km = km
        self.m = m

    def __eq__(self, other):
        return self.km == other.km and self.m == other.m

    def __gt__(self, other):
        if self.km > other.km:
            return True
        elif self.km < other.km:
            return False
        elif self.m > other.m:
            return True
        else:
            return False

However, defining all of these methods can be cumbersome and result in a large amount of boilerplate code, especially since there are 5 of them. Luckily you don't have to! Given the logic of __eq__ and any one of __gt__, __ge__, __lt__ or __le__, all the other comparison methods can be derived.

For example, in the example above, the __le__ method can be derived as

def __le__(self, other):
    return not self > other

This is what the @total_ordering decorator does. When applied onto a class that defines at least the __eq__ and one other comparison method, it derives all the others for you.

@functools.total_ordering
class Distance:
    ...

Dunder/Magic Methods

Dunder (double underscore) methods are special methods that allow you to make use of Python's built-in operators and keywords, such as +, *, len, etc in user-defined classes. This is very useful in making code more concise and readable. You can read more about dunder methods in my previous article, here.

Image by Paul Brennan from Pixabay

A Comprehensive Guide to Django Middleware

Bikramjeet Singh — Mon, 12 Oct 2020 17:39:38 +0000

What is Middleware?

As the name suggests, middleware is basically a mechanism that comes in the middle of the usual request-response cycle, usually to provide some sort of intermediate functionality. A middleware component takes an HTTP request, performs some operation upon it, and then passes it on to the next component in line, which may be either another middleware or the final view.

Image by Jody Boucher

Middleware can also return a response directly instead of forwarding the request further down the chain. For example, Django's CSRF Middleware checks if the request comes from a valid origin (usually the same domain as the server) and immediately returns with a 403 Forbidden response if not. This helps prevent a potentially malicious request from entering the system.

Middleware is always global, that is, it is applied to every request entering the system.

Uses for Middleware

1. Filtering Requests

Middleware can be written to filter out invalid or potentially malicious requests and have them return immediately (usually with an error response), thus blocking them from proceeding further. One contrived example is the CSRF middleware mentioned above, which filters out requests sent from different domains. Another possible example could be an RBAC (Role-Based Access Control) middleware, that prevents users from reading or modifying resources they are not authorised to access. Or, yet another example is geo-blocking middleware that filters out requests from certain geographical locations.

2. Injecting Data into Requests

Middleware can be used to inject additional data into the request that can be used further inside the application. We can take the example of Django's Authentication Middleware, which adds a user object to every valid request. This is a convenient way for the view and other middleware to access details of the logged in user, simply by calling request.user.

3. Performing Logging, Analytics and Other Miscellaneous Tasks

Some middleware don't actually directly modify the request/response at all, but simply make use of the information contained inside them. In other words, these are 'read-only' middleware. For example, imagine an analytics middleware that stores, in a database, the details of all requests entering the system, such as the associated user, the URL, timestamp, etc. This data would later be analysed to identify useful information or trends. Such a middleware would simply read the request content and create/update some related records in the database, then allow it to transparently pass through. Another example would be a usage monitoring middleware that tracks how much of their usage quota a user has exhausted.

Middleware in Django

The structure of a basic middleware in Django looks like this:



class ExampleMiddleware:

    def _init_(self, get_response):
        self.get_response = get_response

    def _call_(self, request):

        # Code that is executed in each request before the view is called

        response = self.get_response(request)

        # Code that is executed in each request after the view is called
        return response

    def process_view(self, request, view_func, view_args, view_kwargs):
        # This code is executed just before the view is called

    def process_exception(self, request, exception):
        # This code is executed if an exception is raised

    def process_template_response(self, request, response):
        # This code is executed if the response contains a render() method
        return response

The first method, __init__, is the constructor for our Python class. It is called only once, at the time the server starts up. Here is where we perform any initializations or other one-time tasks we may want to do.

As you can see from the code, our constructor has one parameter, the get_response function. This function is passed to our middleware by the Django framework, and its purpose is to pass the request object over to the next middleware, and the get the value of the response.

The second method, __call__ allows an object of our class to be called like a function! That is, it turns it into a callable. This is where we put our actual middleware logic. This method is called by the Django framework to invoke our middleware.

The other three are special 'hook' methods that allow you to invoke your middleware under specific conditions. Note that these are optional, and you can implement them only if you require the functionality they provide. Only the first two methods, __init__ and __call__, are required by the Django framework for your middleware to work properly.

Let's take a look at the hook methods in more detail.

The Django Middleware Hook Methods

process_view

This method is called each time Django receives a request and routes it to a view. How is it different from the __call__ method? Well, here we have access to the view function that Django is routing the request to along with any further arguments (args and kwargs) that will be passed to it. For example, path parameters and their values are usually passed in kwargs, so in case our middleware needs access to them, it must implement the process_view method.

The method must either return None, in which case Django will continue processing the request as usual, or an HttpResponse, in which case it will immediately return with that response.

process_exception

This method is called whenever a view raises an exception that isn't caught within the view itself. Hence, process_exception is invoked after the request has reached and returned from the view.

Similar to the above, the. process_exception method must either return None or an HttpResponse.

process_template_response

This method is also invoked after the view has finished executing. It is only called if the resultant response contains a render() method, which indicates a template is being rendered. You can use this method to alter the content of the template, including its context data, if required.

Registering Middleware

Once you have written your custom middleware class, it has to be registered with your Django project in order to add it to the sequence of middleware each request is passed through. To do this, simply add it as an entry in the MIDDLEWARE list in your main settings.py file:



MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    .
    .
    .
    # Add your custom middleware to this list
]

The entry should contain the full path to your middleware class, in string form.

Note that the order in which middleware are present in this list is very important. This is because some middleware might depend upon another to function properly, or might even impede the proper functioning of another. Most security-related middleware tend to reside towards the beginning of the list, so they can catch and filter out potentially harmful requests early. As a rule of thumb, most custom middleware are added to the end of the list. However, this is not always true, especially if, as noted above, your custom middleware is seecurity-related. You must decide where to place your middleware entry on a case-by-case basis.

Learning By Practice

Let's create three example middleware classes that will cover all three use-cases we saw earlier. One will keep a record of the number of requests handled and exceptions raised by the server. The second will detect and inject the user agent information into the request. Finally, the last one will filter out requests from certain unsupported user agents.

💡 Note: we'll only be covering the middleware in this practical section, not the setup of the rest of the Django project. If you're unfamiliar with the basics of Django, the official tutorial is a great place to start.

CountRequestsMiddleware

Our CountRequestsMiddleware looks like this:



class CountRequestsMiddleware:

    def __init__(self, get_response):
        self.get_response = get_response
        self.count_requests = 0
        self.count_exceptions = 0

    def __call__(self, request, *args, **kwargs):
        self.count_requests += 1
        logger.info(f"Handled {self.count_requests} requests so far")
        return self.get_response(request)

    def process_exception(self, request, exception):
        self.count_exceptions += 1
        logger.error(f"Encountered {self.count_exceptions} exceptions so far")

In the constructor, we are initializing two variables, count_requests and count_exceptions. In the __call__ method, which is invoked with each request, we are incrementing the value of count_requests and also logging it.

We have also defined the process_exception method, where we similarly increment and log the value of count_exceptions. We are not concerned with the specific type of exception here, although if we wanted to, we could track each type individually by inspecting the value of the exception argument.

We are not defining the other hook methods, since we don't really require them.

SetUserAgentMiddleware

In HTTP, the user agent information is present in the User-Agent header. We will make use of the user-agents Python package to parse this header string and extract meaningful information from it.



class SetUserAgentMiddleware:

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request, *args, **kwargs):
        request.user_agent = user_agents.parse(request.META["HTTP_USER_AGENT"])
        return self.get_response(request)

Here, we're grabbing the value of the user agent header (request.META["HTTP_USER_AGENT"]) and parsing it using the library. Then, we are setting the result as an attribute of the same request object, so that later middlewares and views can access it.

💡 Note: this middleware is basically a reimplimentation of the middleware in the django-user-agents package. While this is fine for learning purposes, in a real-world application always make use of readily available solutions. Don't reinvent the wheel!

BlockMobileMiddleware

Say we're writing a webapp that is only compatible with desktop browsers for some reason (perhaps it's an online game that requires mouse support). We would need some way to detect and block requests that come from mobile browsers. Since we have already written the SetUserAgentMiddleware middleware, we can take advantage of it and simply write another middleware that checks the value of request.user_agent and returns immediately if it is unsupported. This way, we won't have to complicate our view code with user agent checks, since we can guarantee that any request that reaches our it comes from a supported browser.



class BlockMobileMiddleware:

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request, *args, **kwargs):
        if request.user_agent.is_mobile:
            return HttpResponse("Mobile devices are not supported", status=400)
        return self.get_response(request)

Here, we simply check the value of the boolean is_mobile property and, if it is True, immediately return a 400 (Bad Request) response with an appropriate error message.

Note that this depends upon the request.user_agent attribute which is injected by SetUserAgentMiddleware. Therefore, SetUserAgentMiddleware must always come before BlockMobileMiddleware in the middleware list inside your settings file.

Conclusion

I hope you enjoyed this post and found it informative. Please leave a comment below if you spot an error or have something to add.

The complete source code for the example is available on GitHub here.

If you would like to learn more about Django middleware, I cannot recommend the official documentation enough!

Cover image by MustangJoe from Pixabay

Write composable, reusable Python classes using Mixins

Bikramjeet Singh — Fri, 02 Oct 2020 14:35:29 +0000

An important principle in programming is DRY - Don't Repeat Yourself. It's considered good practice to avoid repeating similar chunks of code in multiple places, and instead define the functionality once - say, in a separate function or class. Then, wherever that functionality is required, you can simply invoke the function or create an instance of the class to do what you need.

Why is DRY important? Because, when you inevitably have to make changes in your code sometime in the future, you want to be able to make those changes at one place, rather than have to hunt through your entire codebase. Not only does this make your life as a developer a lot easier, it also reduces the risk of you missing out on one usage instance and thus creating inconsistent behaviour in your code.

One way to follow the DRY principle in Python is to make use of mixins. Let's see what mixins are and how they help us.

💡 Note: this article assumes you have some understanding of basic OOP (Object Oriented Programming) concepts. If you are not very familiar with OOP yet, don't worry! I've found this article to be very useful in explaining the basics.

Inheritance in Python

On important thing to note is that Python, unlike some other Object-Oriented languages such as Java or C#, permits multiple inheritance. That is, a single class can have multiple parent classes. This is important because, as we'll see very soon, multiple inheritance is the basis for mixins in Python.

What is a Mixin?

Let's say you're baking pizza for your family. The problem is, your entire family is very fussy when it comes to food. Your sister likes her pizza with olives, your father likes it with sausage, your mother likes neither and you like both!

How do you go about cooking 4 different pizzas efficiently? Would you make the toppings for each pizza from scratch, each time?

No! You'd probably first buy all the ingredients you need at once, then grate the cheese, chop up the olives, chop up the sausages and keep them aside in bowls. Then, as you're preparing each pizza, 'mix in' toppings whenever required before slipping it into the oven.

This example leads us to the definition of a mixin: a piece of standalone optional functionality that can be 'mixed in' whenever required. In our example, the toppings (olives and sausages) are both examples of mixins. They are optional - that is, the pizza would taste just fine without them - but their presence acts as a sort of enhancement. The same mixin can be added to multiple pizzas, or multiple mixins can be added to a single pizzas.

How Do I Create a Mixin in Python?

There is no special syntax for creating mixins. Just write a regular Python class.

class OlivesMixin:

    def add_olives(self):
        print("Adding olives!")

Just like any other class, a mixin can extend other classes, if required. It can also be instantiated by itself, but there is usually little reason to do this. Remember, mixins are meant to assist other classes by providing them specific functionality, not exist in isolation.

Now that we have defined our mixin, all we need to do to benefit from it is to make another class inherit it:

class SistersPizza(OlivesMixin, PlainPizza):

    def prepare_pizza(self):
        self.add_olives()

This way, we can make use of the add_olives method anywhere in our class, just as if it was defined inside the class. At the same time, we can take advantage of reusability and use it in any other class we want, just by adding the OlivesMixin mixin!

In its current form, our mixin doesn't provide much utility, because it only consists of a single method that can simply be extracted and written as a standalone function. The real advantage of mixin classes is that, while they cannot contain any internal state of their own, they can manipulate the internal state of the class they are 'mixed' into.

class PlainPizza:

    def __init__(self):
        self.toppings = []


class OlivesMixin:

    def add_olives(self):
        print("Adding olives!")
        self.toppings += ['olives']


class SistersPizza(OlivesMixin, PlainPizza):

    def prepare_pizza(self):
        self.add_olives()

A single class can inherit and make use of multiple mixins. One crucial point to note is that the mixins must always go to the left of the 'base' class, i.e the main parent class should always be the right-most in the list of superclasses.

class MyPizza(OlivesMixin, SausagesMixin, PlainPizza):
    ...

This is important because of Python's MRO (Method Resolution Order), the priority order in which Python searches classes for a method. You can read about the reasoning behind this here.

Let's Take a Practical Example

Our pizza example is well and good for the sake of introduction, but it's not the kind of code we'd ever need to write in the real world. Let's take a more practical example.

The Django web framework makes extensive use of mixins, both in its internal code and the APIs it exposes.

TemplateResponseMixin & ContextMixin

Generally, if you want to render a template in Django, you'd use the TemplateView class.

class MyView(TemplateView):

    template_name = 'my_template.html'

However, this class only works with GET requests. What if you run into a situation where you have to render a template in response to a POST or PUT request? You can return an individual TemplateResponse in each method ...

class MyView(View):

    def post(self, request, *args, **kwargs):
        return TemplateResponse(
            request=request,
            template='my_template.html'
        )

    def put(self, request, *args, **kwargs):
        return TemplateResponse(
            request=request,
            template='my_template.html'
        )

... but this causes some unnecessary duplication of code, as we are specifying the template name twice.

For this purpose we can use Django's TemplateResponseMixin, which provides the render_to_response method.

class MyView(TemplateResponseMixin, View):

    template_name = 'my_template.html'

    def post(self, request, *args, **kwargs):
        return self.render_to_response(context={})

    def put(self, request, *args, **kwargs):
        return self.render_to_response(context={})

The context parameter above, which is currently empty, allows us to pass a dictionary of context variables, which are dynamically inserted into our template. For this purpose, it is also useful to subclass the ContextMixin, containing the get_context_data method.

class MyView(TemplateResponseMixin, ContextMixin, View):

    template_name = 'my_template.html'

    def post(self, request, *args, **kwargs):
        return self.render_to_response(context=self.get_context_data())

    def put(self, request, *args, **kwargs):
        return self.render_to_response(context=self.get_context_data())

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        # Add your context variables here
        return context

TemplateResponseMixin and ContextMixin are almost always found together, since they work well in tandem. You can read more about them here

Conclusion

We've seen how using Python mixins in our classes helps make our code more modular. The resulting code is shorter, simpler to understand and follows DRY best practices. Mixins are used everywhere in the object-oriented Python world, from web development to AI.

I hope you found this article useful. If you spot any errors or would like to offer any suggestions, please feel free to drop a comment below.

Image by joshuemd from Pixabay

References and Further Reading

What is a mixin, and why are they useful?
Using Mixins With Class-Based Views
Multiple inheritance and mixin classes in Python
Djangocon: Demystifying mixins with Django

How to write better Python classes using Magic Methods

Bikramjeet Singh — Sat, 01 Aug 2020 19:40:52 +0000

It is important to make the code we write clear and as easy to read as possible, so anyone unfamiliar with the codebase is able to understand what it does without too much hassle. When dealing with object-oriented Python code, using dunder methods - also known as magic methods - is one useful way to achieve this. They allow our user-defined classes to make use of Python's built-in and primitive constructs - such as +, *, / and other operators, and keywords such as new or len - which are often a lot more intuitive than chains of class methods.

What are dunder methods?

Chances are, if you have programmed in Python for a while, you might have come across strange methods whose names begin and end with pairs of underscores. If you've worked with Python classes, you might even be familiar with one specific method: __init__, which acts as a constructor and is called when a class is initialized.

__init__ is an example of a dunder (double underscore) method, also known as magic methods. They provide a way for us to 'tell' Python how to handle primitive operations, such as addition or equality check, done on our custom classes. For example, when you try to add two instances of a custom class using the '+' operator, the Python interpreter goes inside the definition of your class and looks for an implementation of the __add__ magic method. If it is implemented, it executes the code within, just like with any other 'regular' method or function.

Learning by Example

Let's implement a Time Period class and see how, using Python's Magic Methods, it can be made clearer and more readable. For simplicity, our class will only deal with hours and minutes and will provide a couple of basic functionalities like adding and comparing time periods.

A Basic TimePeriod class

A basic implementation of this class might look something like this:

class TimePeriod:

    def __init__(self, hours=0, minutes=0):
        self.hours = hours
        self.minutes = minutes

    def add(self, other):
        minutes = self.minutes + other.minutes
        hours = self.hours + other.hours

        if minutes >= 60:
            minutes -= 60
            hours += 1

        return TimePeriod(hours, minutes)

    def greater_than(self, other):
        if self.hours > other.hours:
            return True
        elif self.hours < other.hours:
            return False
        elif self.minutes > other.minutes:
            return True
        else:
            return False

Then we can create instances of our class like this:

time_i_sleep = TimePeriod(9, 0)
time_i_work = TimePeriod(0, 30)

And perform operations on them like this:

print(time_i_sleep.greater_than(time_i_work)) # Should print True

There's nothing functionally wrong with this code. It works exactly as intended and has no bugs. But what if we want to perform a more complex operation, like comparing the sum of two time periods with the sum of another two time periods?

time_i_sleep.add(time_i_watch_netflix).greater_than(time_i_work.add(time_i_do_chores))

Hmmm, doesn't look that great anymore, does it? The reader has to dive in and read each word to understand what the code does.

A smart developer might split the two sums into a pair of temporary variables and then make the comparison. This does indeed improve code clarity a bit:

time_spent_unproductively = time_i_sleep.add(time_i_watch_netflix)
time_spent_productively = time_i_work.add(time_i_do_chores)
time_spent_unproductively.greater_than(time_spent_productively)

A better TimePeriod class

But the best solution comes from implementing Pythons magic methods and moving our logic into them. Here, we'll implement the __add__ and __gt__ methods that correspond to the '+' and '>' operators. Let's try that now:

class TimePeriod:

    def __init__(self, hours=0, minutes=0):
        self.hours = hours
        self.minutes = minutes

    def __add__(self, other):
        minutes = self.minutes + other.minutes
        hours = self.hours + other.hours

        if minutes >= 60:
            minutes -= 60
            hours += 1

        return TimePeriod(hours, minutes)

    def __gt__(self, other):
        if self.hours > other.hours:
            return True
        elif self.hours < other.hours:
            return False
        elif self.minutes > other.minutes:
            return True
        else:
            return False

Now, we can rewrite our complex operation as:

(time_i_sleep + time_i_watch_netflix) > (time_i_work + time_i_do_chores)

There we go! Much cleaner and easier on the eyes because we can now make use of well-recognized symbols ike '+' and '>'. Now, anyone who reads our code will, at a single glance, be able to discern exactly what it does.

Adding More Functionality

What if we want to compare two TimePeriod objects and check if they're equal using the '==' operator? We can simply implement the eq method, like below:

def __eq__(self, other):
    return self.hours == other.hours and self.minutes == other.minutes

Python's magic methods aren't restricted to just arithmetic and comparison operations either. One of the most useful such methods, that you might come across quite often, is __str__, which allows you to create an easy-to-read string representation of your class.

def __str__(self):
    return f"{self.hours} hours, {self.minutes} minutes"

You can get this string representation by calling str(time_period), useful in debugging and logging.

If we want, we can even turn our class into a dictionary of sorts!

def __getitem__(self, item):
    if item == 'hours':
        return self.hours
    elif item == 'minutes':
        return self.minutes
    else:
        raise KeyError()

This would allows us to use the dictionary access syntax [] to access the hours of the time period using ['hours'] and the minutes using ['minutes']. In all other cases, it raises a KeyError, indicating the key does not exist.

Other Useful Magic Methods

The full list of Python's dunder methods is available in the Python Language Reference. Here, I've made a list of some of the most interesting ones. Feel free to suggest additions in the comments.

__new__: While the __init__ method (that we are already familiar with) is called when initializing an instance of a class, the __new__ method is called even earlier, when actually creating the instance. You can read more about it here.
__call__: The __call__ method allows instances of our class to be callable, just like methods or functions! Such callable classes are used extensively in Django, such as in middleware.
__len__: This allows you to define an implementation of Python's builtin len() function.
__repr__: This is similar to the __str__ magic method in that it allows you to define a string representation of your class. However, the difference is __str__ is targeted towards end-users and provides a more user-friendly, informal string, __repr__ on the other hand is targeted towards developers and may contain more complex information about the internal state of the class. You can read more about this distinction here.
__setitem__: We have already taken a look at the __getitem__ method. __setitem__ is the other side of the same coin - while the former allows getting a value corresponding to a key, the latter allows setting a value.
__enter__ & __exit__: Credits to Waylon Walker's comment for this one. These two methods, when used together, allow your class to be used as a context manager, a powerful way to ensure important code - especially when dealing with external resources such as files - is always executed.

In Conclusion

We've seen how a few minor tweaks to our code can bring great improvements in terms of clarity and readability, as well as give us access to powerful Python language features. In this post we have, of course, only scratched the surface of the capabilities magic methods can provide us, so I would recommend going through the links below to discover interesting new user-cases and possibilities for them.

If you see any errors in this article or would like to suggest any type of improvements, please feel free to write a comment below.

Image by Gerald Friedrich from Pixabay

References and Further Reading

Python Magic Methods Explained
A Guide to Python's Magic Methods
Magic Methods in Python, by example