DEV Community: Ivan Slavko Matić

Django 4.1: Controlling Events With Signals

Ivan Slavko Matić — Fri, 18 Nov 2022 15:00:21 +0000

Introduction

When we look at web applications, we often see them as well-organized systems, made out of many components that are working together in harmony. There are many ways in which those components interact with each other. For example, some component interactions might be triggered by client action, while some components are triggered internally by containing methods that have nested functions. Django itself has embedded background processes that constantly communicate with multiple parts of its framework to keep everything in sync and working order. In this article, we aim to utilize Django's built-in & custom signals to control events within an application.

Django Signals Overview

In Django Framework, we already have internal component logic sequences that are predefined by the framework itself. For instance, let us consider the 'save()' method from 'django.db.models'. We know that the save() method will be automatically triggered implicitly once we call create() method on a model (if we interpret create() method as a wrapper for the save() method). One method indirectly contacted another. With Django signals, that behaviour can be similarly recreated and customized to promote even more interconnectivity between Django components. With Django signals as our notifier, we can trigger certain events to occur via callback functions.

We can even compare Django signals with Javascript listeners. The concept is similar, we have a listener in javascript waiting for an event to trigger. In parallel, the Django signal consists of a receiver and a sender. We register our callbacks, and those callbacks are standing ready to receive input from senders.

Project Setup

For examples we are about to write and go through, we will be using the project setup 'speedster' from one of the previous articles 'Improving Database Accessibility'. There can be found instructions to get our project up and running. For signals code situating, official Django docs say the following:

"Strictly speaking, signal handling and registration code can live anywhere you like, although it's recommended to avoid the application's root module and its models module to minimize side-effects of importing code."

So to keep things nice and structured, let's create a python package in our Django app 'speedy' just for our signals housing. We will name the folder 'signals' and the folder itself will contain 'init.py'. Now, if we want, this segregation can be extended to other parts such as converting 'models.py' into individual model bundles (their own '.py' files) and storing them in a python package:

speedy/
    migrations/  
    signals/
        __init__.py
        employees_signals.py
    models/
        employees.py
        ...
        __init__.py
    ...
    __init__.py

It all depends on our project structure preferences. We mustn't forget to import files in our python package in our newly created 'init.py':

Python package: signals/employees

from .employee_assignment import *

One last thing (if you decide on the suggested structure above), beware of the circular imports. Order of imports we set in our 'init.py' matters.

Django Built-Ins

From my research, the most common usage of signals is for event controlling in models (or manipulating model instances). We differentiate three groups of signals that are self-explanatory:

pre_save/post_save
pre_delete/post_delete
pre_init/post_init

All of the above sets originate from 'django.db.models.signals'. Less familiar and used (but still useful) sets are m2m_changed and class_prepared. More about them and more can be found in the following reference [1]. With the mentioned sets above, we can create sequences of events between related models and make them reactive. Let's see these built-ins in action.

Example: Employee Assignment Automation & Maintenance

In this example, we will be auto-creating a shift (if it doesn't already exist) for our newly created employee or updating an existing shift with the new employee. One shift can have a maximum of three employees. If you followed the project setup from the link above, then you will notice slight changes to models to accommodate this example.

employees.py:

class Employees(models.Model):
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    address = models.CharField(max_length=200)
    age = models.PositiveSmallIntegerField()
    date_of_birth = models.DateField(auto_now_add=True)

shifts.py:

class Shifts(models.Model):
    employees_shift = models.ManyToManyField(Employees, through='ShiftsEmployees', related_name='shifts')
    wage_bonus = models.DecimalField(max_digits=4, decimal_places=2, default=50.00)
    total_hours = models.IntegerField(default=8)
    shift_date = models.DateField(default=now)

class ShiftsEmployees(models.Model):
    employee = models.ForeignKey(Employees, on_delete=models.CASCADE, null=True)
    shift = models.ForeignKey(Shifts, on_delete=models.CASCADE)

    class Meta:
        unique_together = ('employee', 'shift')

I haven't found a way to directly call the 'unique_together' property for the ManyToManyField in the class meta of our model. So I've made a slight workaround, where I defined a 'through' table for our m2m field. And with direct access to foreign keys, unique_together is added for both keys. We can notice that the model is exactly the same as the default would be - it just has the unique_together property.

Optional (and off-topic): An often missed opportunity for slight optimization arises when we declare our 'through' model. As we added unique_together for our foreign keys, we can also add indexes to those keys. Note that assigning the id of 'through' tables is probably not necessary considering that we probably won't use the id field in lookups. That might come in handy if we ever use those FKs for lookups.

# Optional
indexes = [
    models.Index(fields=['employee', 'shift'])
]

Considering that mentioned Django built-ins are related to model events, we've already done half of our task. All that's left is writing signals logic.

Django's signals detect the event on the model instance, so the pre_save event will occur before the post_save. All the signals related to the sender 'Employees' will be written in employee_assignment.py.

employee_assignment.py:

employee_created():

In the employee_created() signal, there are multiple queries being executed, which at the first glance doesn't seem that much of a big deal. But we should keep the big picture in mind, and what I mean by that is we call these signals every time we call the save() method on our model instance. There are no restrictions on how much logic (and by extension queries) we can stuff in signals - but I believe we shouldn't get carried away. For instance, the client clicks create simple x object, and the 'creating…' logo is spinning for 10–15 seconds. That could be a potential consequence of having multiple 'stuffed' signals. Just something to keep in mind if we are looking after our UX rating closely.

@receiver(post_save, sender=Employees)
def employee_created(sender, instance, created, **kwargs):
    # Check if employee was created
    if created:
        # Find first free shift for newly created employee
        try:
            # Check if shift is under three employees
            # Order by 'shift_date' and get first object
            free_shift = Shifts.objects.annotate(employee_count=Count('employees_shift')) \
                .filter(employee_count__lt=3).order_by('shift_date').first()
        except (Shifts.DoesNotExist, IndexError, Exception,) as e:
            print('free_shift Error: ', e)
            free_shift = None

        # Check if empty shift exists
        if free_shift:
            # Assign employee to empty shift
            free_shift.employees_shift.add(instance.id)
        else:
            # No free shift found.
            # Find the shift with the latest date, and add shift object with: 'latest date' + 1 day
            try:
                try:
                    latest_date = Shifts.objects.order_by('-shift_date').values('shift_date').first()
                    latest_date = latest_date['shift_date'] + datetime.timedelta(days=1)
                except (Shifts.DoesNotExist, IndexError, Exception) as e:
                    print('latest_date Error: ', e)
                    latest_date = None
                # Check if there is at least one object in DB (Fresh DBs)
                if latest_date:
                    new_shift = Shifts.objects.create(shift_date=latest_date)
                    # Save object
                    new_shift.save()
                    # Add M2M relation
                    new_shift.employees_shift.add(instance.id)
                else:
                    # There aren't any Shifts objects in DB, default will input current date
                    new_shift = Shifts.objects.create()
                    new_shift.save()
                    new_shift.employees_shift.add(instance.id)
            except (Shifts.DoesNotExist, Exception) as e:
                print(e)

wipe_expired_shifts():

All shifts before the current date in our database, are considered redundant. Those objects are not reusable unless we have an automated sequence that updates the expired object's data. Now we can debate which is faster, creating an object or updating it - but I digress. To keep things simple, let's go with the option of deleting objects and creating them anew. Doing maintenance through wiping data brings many benefits. Not only are we freeing storage in our DBs, but we are also accelerating query lookups by doing fewer comparisons with data in DB.

@receiver(pre_save, sender=Employees)
def wipe_expired_shifts(sender, **kwargs):
    current_date = datetime.date.today()
    try:
        Shifts.objects.filter(shift_date__lt=current_date).delete()
    except(Shifts.DoesNotExist, Exception) as e:
        print('Wipe redundant shifts Error: ', e)

We are almost done :). The last thing we need to do is import our signals inside our related app's ready() function. Notice, that we are importing a python package and not individual files which if we remember, are already imported in init.py.

class SpeedyConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'speedy'

    def ready(self):
        import speedy.signals

Code can be tested by creating a simple function-based view and calling a .create() method on the Employee model. Like this:

def test_signals(request):
    Employee.objects.create(<<insert_test_data>>)
    return HttpResponse()

This is an alternate way of calling a signal directly to the event. For example, we can use a built-in event post_save and append out a custom function with the model instance:

post_save.connect(my_custom_function, sender=Employees)

Model Methods vs. Built-in Signals

If we think about it, all the logic we put in the callback function of the signal can be easily put in a custom model method or override of the save() method. Both can perform an action over a model instance that is exactly the same and both are triggered when called upon. So why should we bother with signals configuration, when we can put all of the logic in a simple method of a model?

Here are a few cool benefits we get from moving model method logic to callback functions of signals:

Allows unobstructed cooperation between different models - signals are easier to connect to models without editing the models themselves
Reusable apps prefer signals due to easier adaptation to a new environment, in addition to signals being available to all models at all times unlike embedded model methods
Official Django docs state that overridden model methods are not used on delete bulk operations (For more info, check reference [2])
Quality slug & already generated data processing
A great solution for circular import errors

It should also be mentioned, that we do not benefit much from signals if we do not intend to connect multiple models or other components of Django. So if the logic is contained inside a single model - then we just create a model method or override the existing one. Still, no harm is done if we decide to go deep into signals and move everything there. The result will be the same.

Rules of thumb:

If the task is completely related to the model at hand, keep the logic inside the model itself. No need to implicate signals. We override our save() method, do our thing and call super() at the end (pre_save/post_save situation)
If the task involves other models (or in other words, the task is related to more than one model), signals would be perfect for the task

Custom Signals

Even though we can do various 'magic' in our callback functions, the main intention of signals is to act as a notifier. Signals should be interpreted as add-ons to an already existing logic. Events in custom signals work the same way as built-ins. We have pre_event_x and post_event_x situations. Let's prepare for the next example.

We start by creating a custom_signals.py file in our speedy/signals python package. Next, we open the init.py and add a line to import all from custom_signals.py:

from .custom_signals import *

We are all set. Let's start with a simple example of logging events.

Example: App Event Logging

We will start by declaring a simple custom logging function. In custom_signals.py we declare a log_event() function:

def log_event(message):
    # Creates file 'log_event.log' in project root with append mode 'a'
    f = open('log_event.log', 'a')
    # Write current timestamp with user submitted message
    f.write(datetime.now().strftime("%d_%m_%Y_%H_%M_%S") + ' -- ' + message + '\n')
    f.close()

The function creates a file or appends to an existing file, a custom message sent by the user. The created file is situated in the project root. Next up is custom signal setup which comes in three steps: signal declaration, receiver setup and connection. Declaration and receiver we will situate right below our log_event() function, like this:

def log_event(message):
    # Creates file 'log_event.log' in project root with append mode 'a'
    f = open('log_event.log', 'a')
    # Write current timestamp with user submitted message
    f.write(datetime.now().strftime("%d_%m_%Y_%H_%M_%S") + ' -- ' + message + '\n')
    f.close()

# Declare signals
event_signal = django.dispatch.Signal()

@receiver(event_signal)
def view_event_logging(sender, **kwargs):
    print(sender, kwargs['message'], kwargs['request'])
    message = '{} -- {}'.format(sender, kwargs['message'])
    log_event(message)

As we can see, the declaration is quite simple, but on the receiver, we should point out a few things. Unlike built-ins, we don't append the sender parameter, only our above-declared signal. In custom signals, a sender can be anything - which means we can pass (at the very minimum) a simple string that indicates where that signal came from. Now, for the callback function itself, we can send as many parameters as we want. The callback function prints out sent parameters, form a message from them and calls the log_event() function. That should be enough for the start.

Virtually, signals can be sent from anywhere. Let's demonstrate that, by creating a simple view:

def test_logging(request):
    # Empty string
    message = ''
    try:
        # Create an object
        emp_inst = Employees.objects.create(first_name='user1',
                                            last_name='last_user',
                                            address='address',
                                            age=93)
        message += 'Employees object created: {} {}'.format(emp_inst.first_name, emp_inst.last_name)
    except (Employees.DoesNotExist, ValueError) as e:
        print('Error during employees creating occured: ', e)
        # Convert ValueError error to string for successful concatenation
        message += str(e)
        emp_inst = None
        pass

    event_signal.send(sender='View: test_logging', message=message, instance=emp_inst)
    return HttpResponse()

Our view returns an empty HttpResponse(), which once added to urls.py and when accessed - creates an Employee object and sends our signal with a success/error message and newly created instance.

Some example data that is being written to log_event.log, an error and a success:

01_11_2022_01_15_45 -- View: test_logging -- Field 'age' expected a number but got 'age'. -- None
14_11_2022_14_31_12 -- View: test_logging -- Employees object created: user1 last_user -- Employees object (72)

If we look at the code closely - you might wonder and say: "Why not just call log_event() directly into view and remove redundant signals middleman?". And that is good thinking. Signals may seem perfect for some situations and redundant for others. It's very contextual in nature of what we're trying to do. In those moments of doubt, the power of projection is extremely important. If we believe that our written signal above is complete and will not receive further feature upgrades - then we can conclude with certainty, that it's not needed. However, if we decide to introduce additional models, and logic from other Django components that might clash with other imports, elements and segments of code, in that case, having extra accessibility from Django signals will come in handy. Think of it as an alternate route to do the same logic we did before.

Conclusion

Django Signals should not be used in every situation. If the code gets too big and hard to understand, debugging and testing can get very complicated and even messy. And that comes directly from official Django docs (reference [0]), where they state that we should opt for direct code calling instead of having signals as our intermediary between components. Signals are a wonderful alternative if the need arises. We can set it up quickly and write it as any other method as we did before. Its usage is situational. The programming world is ever-evolving, with new client demands, and new challenges to face - having another tool and an extra bit of knowledge, should make us able to cover more situations in an elegant manner.

References

[0] "docs.djangoproject.com", "Signals", https://docs.djangoproject.com/en/4.1/topics/signals/

[1] "docs.djangoproject.com", "Signals (Library Details)", https://docs.djangoproject.com/en/4.1/ref/signals/

[2] "docs.djangoproject.com", "Overriding predefined model methods", https://docs.djangoproject.com/en/dev/topics/db/models/#overriding-predefined-model-methods

[3] "lexev.org", "Django: signal or model method?", http://www.lexev.org/en/2016/django-signal-or-model-method/

[4] "django-advanced-training.readthedocs.io", "Creating and triggering custom signals", https://django-advanced-training.readthedocs.io/en/latest/features/signals/

[5] "codeunderscored.com", "Custom Signals in Django", https://www.codeunderscored.com/custom-signals-in-django/, March 17, 2022, Humphrey

Django 4.1: Caching With Redis

Ivan Slavko Matić — Wed, 05 Oct 2022 18:58:53 +0000

Introduction

Django is one of those frameworks that can do wonders if you invest enough time in them and pay attention to small details. It’s a framework that is rich in features and very customizable. I am happy that you (the reader) share the enthusiasm to get the most out of Django. While reading up on database access improvements, caching in Django caught my eye. So I’ve decided to research how to hook up Redis with Django as an alternative to Memcached.

Setup & Environment

Considering this topic is related to improving database accessibility, I saw it fitting to just continue working on the same project we used there. You can see more about the project setup on the following link:
Improving Database Accessibility

Why use caching?

As mentioned in the referenced article above, caching is a method for speeding up data access for the end user. It eliminates the required processing time that it takes to prepare data and send it to the user. Data is ready ahead of time. We’ve encountered caching whilst using widely known pages like Instagram, Facebook and Google — such fast and dynamic websites, all use caching. And therefore, it’s not a bad idea to utilize caching in our Django projects.

More speed → Faster website → Greater user satisfaction. Caching is a good investment. As your dynamic website grows, so will your need for speed.

Now, it all depends on our needs. For instance, we might not be building a completely dynamic website, but nevertheless, we might have dynamic elements that could benefit from caching. As we know, Django offers to cache per-site, per-view and template fragment caching (and to an extent various interfaces and low-level caching), which work in our favour — allowing us to cache large to small segments of our project.

Some keynotes that should be mentioned. Caching takes either storage/main memory space and may involve miscellaneous background processes. It can vary in cost, depending on where we decide to put your data. It can be stored in RAM, remote database, or local storage (HDDs, SSDs) — all being valid options, with RAM being the fastest and local storage slowest. So how do we choose the right one?

If we are bottlenecked by our hardware, then we use whichever hardware resource that is underutilized and unused, for our caching.

That should narrow down the choice for us. For instance, RAM is usually more precious than storage drives. If we have plenty of everything, then just go with the fastest option. Ok, let’s explore some cool options and see what we have in store for us.

Django’s Options

Caching configuration in Django is usually straightforward and simple. Cool thing is, we can actually use pure-python libraries that are compatible with Django and third-party software. That widens our possibilities. But for this article, we will be exploring Redis only.

Redis

Like Memcached, Redis uses RAM for data storage. It is described as an in-memory database or main memory database system.

Difference between Redis & Memcached

At first glance, it’s hard to notice the difference between Redis and Memcached. They both offer sub-millisecond response times, allow data distribution across multiple nodes via in-memory databases, and offer high scalability. But surprisingly, they differ a lot. Here are some notable differences:

Command-line: Memcached is direct and connects to the server via telnet to execute commands, Redis on the other hand, offers dedicated ‘redis-cli’, which is a dedicated command-line interface.
Disk dumping: Memcached relies on third-party tools to handle disk dumping, while Redis has highly configurable mechanisms like RDB (Redis database file), which also may involve some background processes.
Data Structures: Memcached stores data as key-value pairs of String, while Redis supports other data structures like list, set, hash and others.
Architecture: Redis shows better performance when dealing with smaller datasets while using a single core. Memcached on the other hand is utilizing multi-threaded architecture with multiple cores and shows better performance when dealing with larger datasets. More details on reference [2].

Personal opinion: I avoid declaring winners (even if they are obvious) when comparing approaches, methods, technologies and others. I think every developer learns really quick that clients' requests/demands come in all shapes and sizes. And therefore, I like to keep my options open, everything can be a tool and having alternative options can come in handy if we want to cover more situations. We analyse the situation, project our future needs and choose the software/package accordingly.

Installation

Officially, Redis is supported on Linux and macOS. Windows users can use Redis via VM or WSL (Linux sub-system). Now, considering that I have a Windows version that doesn’t support WSL2, I will be using a Hyper V Virtual Machine solution and setup a Debian distribution (Debian iso version — v11.5) for my Redis. First, we need to get a Debian distribution from their official website (if using an intel/amd processor — get an amd64 version).

Hyper V

Before we get our VM going, we will need to configure our NAT. NAT (network address translation) will handle our network connection to our VM. And we will also need it if we intend to connect our Django project with the Redis port.

Configure NAT:

Open W. Powershell as Administrator
Create NAT virtual network

New-VMSwitch -SwitchName "nat4debian" -SwitchType Internal

Get ifIndex of your newly created NAT

Get-NetAdapter

Configure the gateway

New-NetIPAddress -IPAddress 192.168.0.1 -PrefixLength 24 -InterfaceIndex 88

[Optional Step — Info] Open View Network Connections

nat4debian → Status → Details → IPv4 Address (192.168.0.1) and our IPv4 Subnet Mask (255.255.240.0)

If you need explanations about any of the steps above, you can find them in reference [5]. It’s also possible you might have a default switch (vEthernet) that is already configured — probably comes with Hyper V installation.

Hyper V Manager:

Actions → New → VM → Name: debian11 → Gen1 → Startup Memory: 4096MB (4GB, uncheck dynamic memory) → Connection: nat4debian → vhdx: default value → Install from Image file (.iso): debian-11.5.iso → Finish
Run VM and perform Debian installation

GNU/Linux & Redis Installation:

Keep the installation light — go with ‘Install’ (non-GUI option). Follow the installation steps up to ‘Network Configuration’. Once there, enter your IPv4 Address (192.168.3.232). If you fail the Debian archive mirror country step → you will probably need to double-check the network configuration step.
Software selection: only standard system utilities. Everything else should be off. Remember, we are keeping our installation light and cost-efficient. Any unnecessary background (coming from the Desktop GNOME for instance) is not desirable. If you feel the need for a GUI, then install GNOME.
Once the installation is over, bring up your terminal and run the Redis installation sequence of commands:

# Enter root (sudoer), enter your root password
su - 
# You can get it via curl (add it to apt) then just install via apt
# I will install it with snapcraft
apt update
apt install snapd
# Snap will autoupdate then install redis
snap install redis
# Test if Redis is working correctly
redis-cli ping
# Response is "PONG"

Now, in my case of Redis snap installation — path to redis-cli got mismanaged. If that is also your case, you can do the ‘manual run’ of redis-cli (without the path in your .profile), like this:

cd /snap/redis/current/usr/bin
./redis-cli ping
# You can type 
redis-cli
# To get address and port of the server, if not started then run
./redis-server

Django Implementation

Now our Redis server is up and running, what we need to do next is configure a Django binding to our server. Django supports Redis natively with redis library and also recommends installing the hiredis package. ‘hiredis’ package improves communication between Django and Redis by handling multi-bulk replies with the Reader class which would serve as our parser. To get the most out of our redis package we could also use the optional django-redis package, which offers pluggable clients, serializers, raw access and many other features.

Let's install those packages:

pip install redis
pip install hiredis
pip install django-redis

Once installed, it’s time to handle our settings.py configuration:

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        }
    }
}

Let’s create a simple function-based view to test our connection:

def test_redis(request):
    get_redis_connection("default").flushall()
    return HttpResponse()

If you run this view, you will get an error like this:

redis.exceptions.ConnectionError: Error 10061 connecting to 127.0.0.1:6379. No connection could be made because the target machine actively refused it.

And that is because we need to search our Redis via our NAT network gateway. So we retain our Redis port but access it via a gateway. In Debian, you need to create an IPv4 address from the provided gateway (192.168.0.1) and enter that address into your settings.py:

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://192.168.3.232:6379',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'PASSWORD': 'debian'
        }
    }
}

Now we have access to our Redis server. Last thing we need to do is disable Protected mode from Redis:

./redis-cli
config set protected-mode no

You can also find a config file ‘/etc/redis/’ and set the value ‘no’. We run our test view, and it should return ‘true’, we have Redis server access.

Important: It is fully recommended to use proper authentication when using the Redis service. For testing purposes, I will skip the authentication setup steps. If using the redis.py package, you will most likely have to embed your username and password to access your redis server (as matter of fact, it’s a must). Also, django-redis package cautions us that Redis ACL (Access Control List) requires authentication on attempted access.

Simple Benchmarking

Our setup is done, let's do some simple benchmarking to see some Redis performance benefits in action, but before that, just a quick note from Redis's official page on memory footprint:

1 Million small Keys -> String Value pairs use ~ 85MB of memory.
1 Million Keys -> Hash value, representing an object with 5 fields, use ~ 160 MB of memory.

The environment where we situated our Redis server has the capacity of 4GB RAM, but we haven’t limited/checked how much RAM Redis is actually using. I usually don’t like to let anything ‘run wild’, especially my RAM, so let’s set the maximum memory Redis can use with the following commands:

./redis-cli
config get maxmemory
config set maxmemory 512M
# Response: OK

My recommendation: always start small then increase if needed — RAM is an expensive resource! Following the memory footprint above 512M is a good starting point. The allocated resource can always be easily adjusted.

Let’s fix up a render function view that fetches all objects (and their related objects). Our ‘Stores’ model will come in handy now:

@cache_page(180)
def test_redis(request):
    # get_redis_connection("default").flushall()
    try:
        stores = Stores.objects.select_related('manager').all()
    except (Stores.DoesNotExist, ObjectDoesNotExist) as e:
        print(e)
        stores = None
        pass

    return render(request, "demo_temp.html", {'data': stores})

Now in this view, the cache on the first call is just being set, so we will notice the difference in performance on the second view call when the data is already cached. On our redis-cli we run a check-up to see if there is any data already:

./redis-cli
keys *
# It should respond with '(empty array)'
# If there is any data, empty the cache with
flushall

These are the comparison times received from django-debug-toolbar:

# Average of 3x repeated process to compensate for background processes from IDE
Total: 427.72ms ~ average
# Access after cache has been set
Total: 15.00ms

Works like a charm! We got our desired effect.

Where to use it

Redis caching service is best used remotely and probably on a pure Linux or macOS environment, where dedicated hardware resources are allocated and served for Redis (caching) service use only. Redis service will significantly benefit if no background processes are messing with RAM allocation. But this is a ‘touchy’ subject, again we need to reiterate the sentiment we heard before. It depends. As we can see, we can set up our Redis to work locally on our VM or localhost without a problem. Still, looking from a hardware perspective, we did just put a strain on our local machine, by taking away 4GB of RAM and adding background processes to be run as a result of Hyper V VM (any serious/major project distribution would require even more).

Redis is accessible, customizable and secure. Furthermore, Django from the 3.2 version up to the 4.1 version, worked on introducing more and more async support features, making remote Redis access completely a valid option with minimal performance degradation. Of course, if there is a possibility of using Redis locally (for instance a local network) instead of remotely, I consider that approach to be more performance friendly due to the lack of possible latency issues. Async would definitely do its job properly by utilizing await. Redis is definitely worth the investment of both set-up and hardware sacrifice.

Addendum: Accessing The Cache

Related to this topic, how to access stored data (cache) in Redis is another topic that is equally important. Find more in references [3] and [4].

References

[1] “data-flair.training”, “Django Caching”, https://data-flair.training/blogs/django-caching/,

[2] “baeldung.com”, “Memcached vs Redis”, https://www.baeldung.com/memcached-vs-redis

[3] “docs.djangoproject.com”, “Accessing the cache”, https://docs.djangoproject.com/en/4.1/topics/cache/#accessing-the-cache

[4] “redis.io”, “Install Redis on Windows”, “https://redis.io/docs/getting-started/installation/install-redis-on-windows/”

[5] “learn.microsoft.com”, “Set up a NAT network”, “https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/setup-nat-network”

[6] “redis.io”, “What’s the Redis memory footprint?”, “https://redis.io/docs/getting-started/faq/”

Django 4.1: Improving Database Accessibility

Ivan Slavko Matić — Sun, 18 Sep 2022 20:01:38 +0000

Introduction

Today’s commodity in all applications, written by all manner of different programming languages, is speed. For example, how fast we get our render, send our information, process complex operations, or in general, how fluent is the whole experience while using an application. We’ve developed a certain standard we expect from our applications. So how do we maintain that standard with Django?

There are many factors that decide how fast or how reliable an application is. And those factors get especially noticed as our application grows with each new feature and operation we add-in. After a while, it's fairly common for a developer to try and improve the performance of its application. Django’s strength lies in procuring easily assembled and fast queries from a database, and that is the angle we want to explore today.and that is the angle we want to explore today.

Speed Traps

First, you might wonder, what are ‘Speed Traps’? We’ll that term is meant to explain the following. To sustain a level of reliability in our applications, some elements such as caching, indexing and query optimisation, are not fully automated in Django and are left to the end user to do so. They come at a cost. A trade-in, where we sacrifice memory and storage space (which provides ‘breathing room’ for our applications and provides stability), to procure speed. This is not just a Django quandary, it is in fact, present in game engines, database systems and programming languages, which engineers have to face whilst planning ahead and upgrading. While some performance boosts are straightforward (where performance is added and not transferred or traded), trading memory and storage space for speed are not. As matter of fact, a wrong allocation of resources can hurt the application and slow it down, or even worse - crash it. And as we know, user experience is affected by both speed and reliability. So in short, by saying speed traps, I’m referring to the trade of memory, storage space and speed which can be costly if done wrong.

But here’s a cool thing, with some careful and well-thought-out fine-tuning, we can optimize our application to run at full speed. We can take control of the unknown in our Django application and push it to new heights. Let’s set up some tests and examples, and see how we can use these boosts to our best advantage.

Quick Setup Of Testing Environment

We will be using PostgreSQL 13.6 and Python 3.9 for this project.

Start project

django-admin startproject speedster

Start app

./manage.py startapp speedy

Create venv

./manage.py venv venv

Python packages (use ‘pip install’ or our IDE package manager to install packages)

#Packages
 asgiref==3.5.2
 Django==4.1.1
 psycopg2==2.9.3
 django-debug-toolbar==3.6.0
 sqlparse==0.4.2
 tzdata==2022.2

Note: ‘django-debug-toolbar’ is an optional third-party package. It is used primarily for monitoring and troubleshooting. For full installation details, follow this link https://django-debug-toolbar.readthedocs.io/en/latest/installation.html.

Define our app in settings.py

INSTALLED_APPS = [
  'django.contrib.admin',
  ...
  'speedy'
 ]

Setup our DB, login as superuser in our ‘postgres’ shell (‘postgres’ is superuser in my case)

psql -U postgres

Creating DB and user with privileges

CREATE DATABASE speedster_db;
CREATE USER speedster_user WITH PASSWORD 'speedsterpass';
ALTER ROLE speedster_user SET client_encoding TO 'utf8';
ALTER ROLE speedster_user SET default_transaction_isolation TO 'read committed';
GRANT ALL PRIVILEGES ON DATABASE speedster_db TO speedster_user;
\q

Add our database configuration to our settings.py

DATABASES = {
  'default': {
   'ENGINE': 'django.db.backends.postgresql',
   'NAME': 'speedster_db',
   'USER': 'speedster_user',
   'PASSWORD': 'speedsterpass',
   'HOST': 'localhost',
   'PORT': '5432',
  }
 }

Run migrate for our default apps like ‘admin’, ‘contenttypes’, ‘auth’... (We haven’t any models yet, so ‘makemigrations’ command is not necessary)

./manage.py migrate
./manage.py check   # Can't hurt

Models

We are keeping our models modest and simple. Altogether, I will be filling tables with approx. 10000 objects where each object has a similar size (that does not include the ‘through’ relations table autogenerated from the m2m field).

from django.db import models
from django.db.models import CASCADE


class Employees(models.Model):
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    address = models.CharField(max_length=200)
    age = models.PositiveSmallIntegerField()
    date_of_birth = models.DateField()


class Stores(models.Model):
    manager = models.ForeignKey(Employees, on_delete=CASCADE)
    city = models.CharField(max_length=100)


class Shifts(models.Model):
    name = models.CharField(max_length=100)
    employees_shift = models.ManyToManyField(Employees)
    wage = models.DecimalField(max_digits=5, decimal_places=5)
    hours = models.CharField(max_length=15)

Ok, the initial setup for our project is done. Let’s explore some ways to gain performance boosts.

Indexes

We’ve all encountered long renders or just waiting for a slow API to respond due to a query sifting through large amounts of data in our database (imagine a query fetching 1mil.+ objects from a database that aren’t small integers — that takes a while). Let’s say we want to fetch all employees whose id is over 284:

# Django Query
try:
    employee = Employees.objects.filter(id__gte=284)
    print(connection.queries)
    print(employee.explain())
except Employees.DoesNotExist as e:
    print('Error: ', e)
    employee = None
    pass
# Postgres explain: 
# Seq Scan on speedy_employees  (cost=0.00..228.00 rows=9971 width=50)
# SQL: 
# SELECT "speedy_employees"."id", "speedy_employees"."first_name", "speedy_employees"."last_name", "speedy_employees"."address", "speedy_employees"."age", "speedy_employees"."date_of_birth" FROM "speedy_employees" WHERE "speedy_employees"."age" >= 30

By running the ‘Postgres EXPLAIN’ command on our query, we can see that Postgres did a sequential scan on 9971 rows, meaning it scanner went row by row, comparing values from our filter to our condition. Let’s apply indexes and compare index scan. In Django, we can add indexes with ‘Meta.index’ or with ‘Field.db_index’.

Field.db_index example:

Use this one if your needs are simple and if you’re looking for an ‘inline declaration’

id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False, db_index=True)

Meta.index example:

With this approach, you can create custom name/s for your index/es

class Meta:
    indexes = [
        models.Index(fields=['id'])
    ]

After you declared it, run ‘makemigrations’ and ‘migrate’ to apply changes.

After we run our ‘postgres EXPLAIN’ on our employee query, we can notice the cost has reduced to 203.00. The lower the cost, the fast the query execution is and in this case, we got a faster query as a result. For more information on how the cost is calculated, please check reference [1].

Now, we’ve indexed our field ‘id’, which means that if we filter with parameter ‘id’ in our ORM filter(), exclude() and other methods that involve filtering with the field ‘id’ as our parameter, those rows will be fetched faster.

Important note:

Employees.objects.filter(id__gte=284, age=49)

The ‘id’ field in the example above is indexed, while the ‘age’ field is not, which means you get an index scan on one part of the query and sequential on the other. So plan accordingly. The decision on where to apply indexing, depends on your situation. Django’s recommendation states that indexing is best used on fields that are unique (pk’s, fk’s, unique’s). Lookups will be faster because there is no chance of matching a duplicate value.

My personal rule of thumb is to apply indexing on those fields that are most used in my filter(), order() and other similar methods.

Storage space
Index table size should take at least the same amount of space as the data inside the column. And that is one of the bigger costs of using indexing. Space taken by your indexes scales with the size of your column.

Pros:

There is no need for constant reindexing or maintenance, our model index declaration handles it for us (in the PostgreSQL case — no need for aggressive ‘VACCUM’ usage).
Reduces the cost and speeds up query lookup
Today’s servers and host devices are pretty powerful, so storage space and computing power (at least for now) are bountiful

Cons:

Uses storage space (limited resource, no matter how big)
Trade of query speed versus index maintenance may result in no performance gain (may even reduce it) if utilized incorrectly
Cost of maintenance scales with the number of indexed fields (Even though it’s automated — it still requires the execution of the command to re-add index)

Where and when to use:
Usually, it’s unnecessary to use indexing while in the early development stage of the application. Indexing is something I wouldn’t do proactively but rather reactively — there should be a need for indexing, a need for performance upgrade. And that need often appears in the processing of large quantities of data and long query lookups.

Caching

Alternative to waiting for long renders of complex (or pages with expensive/time-consuming operations) to produce, is caching. The idea behind caching is to predict the future needs of end-users/clients, and prepare data (which can come in many forms) ahead. Prepared data is stored in a cache. Configuring caches is quite simple, the part which requires the most work is deciding where to cache the data. Caching isn’t designed for permanent data storing, but temporary, which means if you get a server crash — data is lost, keep that in mind. Now looking at Django’s support for caching, we can see that it is quite extensive, and you can learn more by visiting reference [4]. Let’s try out one of the caching options Django offers.

Memcached

A really powerful addition to performance but also a very serious speed trap. That is how I would summarize memcaching. Before we try it, let's debunk what Memcached actually does. Quote from official Django documentation:

Memcached runs as a daemon and is allotted a specified amount of RAM. All it does is provide a fast interface for adding, retrieving and deleting data in the cache. All data is stored directly in memory, so there’s no overhead of database or filesystem usage.

An important note to notice is, that Memcached is using RAM. And even more importantly, the RAM being used is from the host itself. That might factor in your calculation and planning if you decide to put Memcached into your host/remote server. Let’s install and try it out.

Memcached is third-party software and requires downloading a release from https://memcached.org/downloads. Note that the release you find will be originally meant for debian/ubuntu. To make it work on Windows, you will have to search the internet a bit for a Windows release (it’s not particularly hard to find). Personally, I recommend installing it on Linux (or Linux VM if on Windows) — support is better and the configuration does not require you to dig through the registry like we are just about to do on Windows.

After you unpacked it, you need to install it and start it. So the daemon will be running in the background.

Navigate to your unzipped folder and run these commands to start the daemon

memcached.exe -d install
memcached.exe -d  start -p 11211 -m 2048 
# -p is already set by default, this is for demonstrating

By default, the Memcached server will be listening on port 11211 and 64 MB memory limit (unless specified otherwise). Both can be changed. Port can stay the same for now, but memory limit we would like to expand. For starters, I recommend setting memory limit to 512 MB. In Windows Run type ‘regedit’ and navigate to:

HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/memcached/ImagePath

On variable ‘ImagePath’ you can edit parameters manually and set memory limit. If you open ‘Services’, you should find ‘memcached’ service running. Ok, now we need our ‘middleman’ python library to communicate with our Memcached server. Django recommends pylibmc and pymemcache. We will be using pymemcache. Let’s run a quick pip install:

pip install pymemcache

The final step is to define our CACHE configuration in our settings.py

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': '127.0.0.1:11211',
    }
}

And our setup is done, now here comes an equally important part — deciding what to cache. Some of the possibilities are to cache a whole site, a view or a template fragment. Let’s do a simple site cache, which is done by updating MIDDLEWARE_CLASSES in settings.py:

MIDDLEWARE_CLASSES += (
   'django.middleware.cache.UpdateCacheMiddleware',
   'django.middleware.common.CommonMiddleware',
   'django.middleware.cache.FetchFromCacheMiddleware',
)

Note: order of the classes mentioned above are strict — update comes before fetch. End now we just have to note which configured cache in our CACHES configuration in settings.py to use and set the timeout for cache usability:

CACHE_MIDDLEWARE_ALIAS = 'default'  # Which alias to use
CACHE_MIDDLEWARE_SECONDS = '600'  # How long to keep cache

Now, if you look closely, we just fell into a speed trap. We used expensive RAM to cache a whole site. Usually, that is very ineffective and performance costly. So instead of caching an entire site, better use for ‘Memcached’ would be something smaller and ‘focused’ such as ‘Per-view cache’. It is always recommended to cache only the most important parts of your application, then expand if needed. Per-view cache is quite simple, you add a decorator to your view and add ‘cache_page()’ in your URL path, like this:

from django.views.decorators.cache import cache_page

@cache_page(60 * 15)
def your_view(request):
    ...

# or

from django.views.decorators.cache import cache_page

urlpatterns = [
    path('object/<int:object_id>/', cache_page(60 * 15)(your_view)),
]

With a little tinkering, this can be instrumental in providing a user with a satisfying experience. For instance, this can be useful for fetching a user’s profile page in repeated succession.

Pros:

Can use RAM, for even greater speeds
Customizable amount of RAM/storage space
Caching has a large variety of choices and customizations
if properly configured, a cached copy can actually save the day, by providing content to the user while the server is crashed

Cons:

Just like indexing, caching can be expensive performance-wise if overused
In some cases, requires third-party software installations (for instance, Redis and Memcached)
Data cached is temporary, when a server or a service is stopped/crashed — data is lost

Where and when to use:
If there is a need to reduce access latency (especially in cases where data is frequently accessed) or if there is a need for alternative data storage. Rule of thumb: find the busiest view with the most query calls in your application and provide cache.

Querysets

Unlike caching and indexing, which are best used later on in development phases or reactively (when there is a performance hit), you can always improve your query writing game and get a performance boost for your application as you write your code!

Iteration as a last resort

This is more view oriented, but it involves queries as well. You probably heard this before, but I will repeat it because it’s very important. Let the queries do the heavy lifting. Query in ORM, already does caching and data assembly, iterating through those objects again to form another data bundle is a delay, and you should avoid that as much as possible.

Instead of rushing ahead, stop and take your time with queries. One quality query in Django is the centrepiece of efficient view.

As matter of fact, you should try to even avoid caching in your queries by using ‘iterator()’. For example, you create a query (which caches results, to provide quicker access) and for the rest of the view, you use that query results only once (let’s say to send data to the template in render). In this case, we are actually creating and carrying cache through our view that won’t be used. Now, if we use the iterator(), we actually eliminate the default cache from our query.

Example:

employee = Employees.objects.filter(id__gte=284).iterator()

By default, the chunk_size parameter the iterator uses is set to 2000 objects. It all depends, on the amount of data you are fetching, it can always be increased. Granted, this may not be a large speed boost, but if you intend on writing many queries, this practice can pay off really quick if used often.

‘select_related()’ and ‘prefetch_related()’

These two methods are getting more noticed by the day. First, let’s explain what these two methods do. ‘select_related’ and ‘prefetch_related’ unlike iterator(), actually add to the query caching. Queryset they perform, fetch related objects of fk and m2m fields, all in one query by using SQL join. It is best shown through example. This is our query:

stores = Stores.objects.filter(id__gte=284)

Now what seems simple and benign to the eye, if you do a related object access via foreign key:

for obj in stores:
    print(obj.id)

you are actually performing an additional query to fetch a related object, which means on top of your cached query, you are performing another query. And that's where prefetch and select related comes in:

Stores.objects.select_related('manager').filter(id__gte=284)
# or 
Shifts.objects.prefetch_related('employees_shift').all()

Imagine fetching 10000 objects and performing a for loop in which you access a related object. That means you will perform ~10000 queries towards your database, and that I think you will agree, is a mistake. That mistake can be avoided by using select_related and prefetch_related, which generate only 1 query and thus prevent all those unnecessary calls to the database.

I heartily recommend checking the references below. I believe you will find quite valuable additional information to bring out your A game in Django.

References

[0] “djangoproject.com”, “Database access optimization”, https://docs.djangoproject.com/en/4.1/topics/db/optimization/

[1] “https://scalegrid.io/”, “PostgreSQL EXPLAIN — What are the Query Costs?”, https://scalegrid.io/blog/postgres-explain-cost/, March 30, 2022

[2] “kaiv.wordpress.com”, “Decreasing the index size on wide columns”, “https://kaiv.wordpress.com/2007/07/23/decreasing-the-index-size-on-wide-columns/”, July 23, 2007

[3] “wiki.postgresql.org”, “Index Maintenance”, “https://wiki.postgresql.org/wiki/Index_Maintenance”, 9 June 2021

[4] “docs.djangoproject.com”, “Django’s cache framework”, “https://docs.djangoproject.com/en/4.1/topics/cache/”

[5] “honeybadger.io”, “Everything You Need to Know About Caching in Django”, “https://www.honeybadger.io/blog/caching-in-django/”, Jun 16, 2022

[6] “tutorialspoint.com, “Django Caching”, “tutorialspoint.com/django/django_caching.htm”

Django 4.1: Where To Apply Async

Ivan Slavko Matić — Thu, 01 Sep 2022 13:03:14 +0000

Introduction

Some exciting changes are coming in the future of Django Framework's ongoing development. One of the notable changes is updated support for asynchronous data access operations and HTTP method handlers. Throughout this article, we will go through some of the tests I've created and compare the execution times of both asynchronous and synchronous processes.

Difference between synchronous and asynchronous in Django

Synchronous programming follows a strict set of sequences (or in other words, it executes sequentially), where operations are executed one at a time in strict order. The following quote from David Bevans at 'mendix.com' provides a great description of synchronous programming:

To illustrate how synchronous programming works, think of a telephone. During a phone call, while one person speaks, the other listens. When the first person finishes, the second tends to respond immediately.

Asynchronous programming allows multiple related operations to run at the same time without waiting for prior tasks to finish. We are describing it as a multithreaded model that implements non-blocking architecture. If a thread is free and can perform an operation - it performs it, while pausing the operation that is being awaited. Another excellent description from David Bevans of asynchronous programming:

Texting is an asynchronous communication method. One person can send a text message and the recipient can respond at their leisure. In the meantime, the sender may do other things while waiting for a response.

That being said, we can discern that async should bring a major performance boost if utilized correctly.

Django at its core works and performs synchronously, and somewhere around version 3.0 (when extensions such as 'twisted', 'channels' and 'asyncio' started to gain traction), the idea that Django can work asynchronously started to manifest.

From my research, best way to decide where to use sync or async, is to discern if the view (and by extension what we want to achieve with that view) is I/O bound or CPU-bound. Async should be utilized when approaching I/O bound issues.

Environment and prerequisites

Before we start testing, let's review the setup and some important notes. First things first, an obvious note (but an important one): results may vary from IDE to IDE and everyone's setup of related/used packages in their project, background processes and third-party software. Therefore, the execution times you see here, may or may not be, similar to the times you achieve, if you decide to recreate these tests.

For these tests, we are using Python 3.9 as our interpreter and PostgreSQL 13.6 as our database system. Installed python packages are as follows:

Django~=4.1 psycopg2~=2.9.3 hypercorn~=0.14.1

psycopg2 acts as our database adapter, and hypercorn will emulate the ASGI test server for our environment.

You might wonder and say:

Yeah, but why use hypercorn, when we already have asgi.py in our project ?

That is true, asgi.py in our project indeed contains an application callable and can be used by any ASGI server in development or production. But it is not used by the development server (which we call with runserver command), and that is where hypercorn comes in. By calling myproject.asgi:application, server is up and ready to be used locally.

Important note: proper way to measure performance and execution time would be through TestCase (or even more feature-rich TransactionTestCase) and testing modules/units in general. But I have some concerns regarding the AsyncRequestFactory used in tests and AsyncClient(). Reading through the documentation, it seems that async behaviour is emulated partly, where async functions are executed, and then the process reverts to synchronous behaviour. Saying that the most authentic/realistic way to test this is by doing a simple view and calling it via URL with hypercorn emulated ASGI server. References [3] and [4] contain more information about 'Testing asynchronous code' and 'Advanced testing topics' - it is possible I missed some crucial context.

Models

For our testing, we will keep our models as light as possible. The fields we use are varied, but the data size of each object inside the models will be similar if not the same even.

from django.db.models import Model, TextField, CharField, DateField, ManyToManyField, ForeignKey, CASCADE
from django.utils.timezone import now


# Create your models here.
class CarParts(Model):
    engine = TextField()


class Manufacturer(Model):
    name = CharField(max_length=150)


class Store(Model):
    location = CharField(max_length=150)


class Car(Model):
    created_at = DateField(default=now)
    store = ManyToManyField(Store, null=True)
    car_parts = ForeignKey(CarParts, on_delete=CASCADE, null=True)
    manufacturer = ForeignKey(Manufacturer, on_delete=CASCADE, null=True)

We will be querying and iterating through 50, 500 and 1500 objects per model, so we can track how it scales with an increased number of data.

Testing

We have two class views extending the base class 'View' from django.views, where each will contain multiple methods. Django does not allow the mixing of sync and async methods - remember that sync implements a blocking approach and non-blocking in async. So one class view will have all async methods and other sync methods.

Tests will be performed in the following order:

Load x number of objects
Perform three consecutive tests (multiple tests to compensate for potential IDE background processes that may cause performance penalties)
Analyse

Test A: CRUD cycle using ORM via get and post methods

Global functions to be called in views (one async and other sync):

async def async_iteration():
    car_obj = Car.objects.all()
    async for i in car_obj:
        a_query = await i.store.afirst()
    return car_obj

def sync_iteration():
    car_obj = Car.objects.all()
    for i in car_obj:
        a_query = i.store.first()
    return car_obj

Our sync class:

class PerformanceTestSync(View):
    def get(self, request):
        e_list = []
        start = time.time()
        for entry in Car.objects.all():
            try:
                a_query = entry.store.first()
                b_query = CarParts.objects.filter(
                    Q(engine__contains='engine') & 
                    Q(car__manufacturer_id__gt=20)).first()
            except (Exception, KeyError) as e:
                print('Error during sync iteration: ', e)
        co = sync_iteration()
        for v in co:
            e_list.append(v)
        end = time.time()
        print('Sync GET time: ', end - start)

    def post(self, request):
        start = time.time()
        engine = request.POST.get('engine')
        name = request.POST.get('name')
        location = request.POST.get('location')
        try:
            car_parts_obj = CarParts.objects.create(engine=engine)
        except (CarParts.DoesNotExist, ObjectDoesNotExist, 
                Exception) as e:
            print('CarParts failed to create: ', e)
            car_parts_obj = None
            pass
        try:
            manufacturer_obj = Manufacturer.objects.create(
                                                 name=name)
        except (Manufacturer.DoesNotExist, ObjectDoesNotExist,
                             Exception) as e:
            print('Manufacturer failed to create: ', e)
            manufacturer_obj = None
            pass
        try:
            store_obj = Store.objects.create(location=location)
        except (Store.DoesNotExist, ObjectDoesNotExist, 
                Exception) as e:
            print('Store failed to create: ', e)
            store_obj = None
            pass

        if car_parts_obj and manufacturer_obj:
            try:
                car_obj = Car.objects.create(
                            car_parts=car_parts_obj, 
                            manufacturer=manufacturer_obj
                          )
                car_obj.store.add(store_obj)

            except (Car.DoesNotExist, ObjectDoesNotExist, 
                    Exception) as e:
                print('Car failed to create: ', e)
                pass

        end = time.time()
        print('Sync POST time: ', end - start)
        return HttpResponse("ok", status=200)

Our async class:

class PerformanceTestAsync(View):
    async def get(self, request):
        e_list = []
        start = time.time()
        async for entry in Car.objects.all():
            try:
                a_query = await entry.store.afirst()
                b_query = await CarParts.objects.filter(
                        Q(engine__contains='engine') & 
                        Q(car__manufacturer_id__gt=20)).afirst()
            except (Exception, KeyError) as e:
                print('Error during async iteration: ', e)
        co = await async_iteration()
        async for v in co:
            e_list.append(v)
        end = time.time()
        print('Async GET time: ', end - start)

        return render(request, 'demo_template.html')

    async def post(self, request):
        start = time.time()
        engine = request.POST.get('engine')
        name = request.POST.get('name')
        location = request.POST.get('location')
        try:
            car_parts_obj = await CarParts.objects.acreate(
                                 engine=engine
                                )
        except (CarParts.DoesNotExist, ObjectDoesNotExist, 
                Exception) as e:
            print('CarParts failed to create: ', e)
            car_parts_obj = None
            pass
        try:
            manufacturer_obj = await Manufacturer.objects.acreate(
                                        name=name
                                      )
        except (Manufacturer.DoesNotExist, ObjectDoesNotExist, 
                Exception) as e:
            print('Manufacturer failed to create: ', e)
            manufacturer_obj = None
            pass

        if car_parts_obj and manufacturer_obj:
            try:
                car_obj = await Car.objects.acreate(
                                  car_parts=car_parts_obj, 
                                  manufacturer=manufacturer_obj
                                )
                if car_obj:
                    try:
                        await car_obj.store.acreate(
                               location=location
                              )
                    except Exception as e:
                        print('Couldnt add store object 
                               relations for m2m: ', e)
            except (Car.DoesNotExist, ObjectDoesNotExist, 
                    Exception) as e:
                print('Car failed to create: ', e)
                pass

        end = time.time()
        print('Async POST time: ', end - start)
        return HttpResponse("ok", status=200)

In the get() method, we are iterating through objects in our Car model and in the loop we are making one simple query and one a bit complex query with Q expressions. All wrapped with the new ORM interface introduced in 4.1. In the end, we call a global function that will iterate, and return queryset whose items will be appended to an empty list. Meanwhile, in the post() method we are creating some objects for our models and updating existing data and in the end, deleting. Data in post is received from the client via request. Deleting could be a method on its own as could updating, but to keep things simple, get and post will do the trick.

You will notice in the async class that I didn't use .add in my m2m field update. That is because (sadly) I couldn't find an async adaptation of it in the documentation, and Django treats it as a sync operation. Therefore, I did a little workaround where I create the 'Store' object directly.

Test results

Method get() for 50 objects (displaying an average of three tests):

async: ~0.39832 s
sync: ~0.29502 s

Method post() for creating single object for all models:

async: ~0.07699 s
sync: ~0.08299 s

Method get() for 500 objects:

async: ~3.67973 s
sync: ~2.39427 s

Method get() for 1500 objects:

async: ~8.79098 s
sync: ~6.26344 s

From this test, we can conclude that this test polygon is definitely CPU-bound, our async is underutilized.

Test B: API communication
For this test, you might need to install python's httpx package. I first tried python's requests package believing it had async but it didn't have sadly, so httpx came in as a wonderful alternative. Also, python's built-in asyncio package will also be needed to gather data from the awaited call to the function. Finally, we are importing shield from asyncio to prevent potential cancellation.

pip install httpx
# Then in your view
import httpx
import asyncio
from asyncio import shield

Using GET request, we will be contacting an external API for exchange rates at apilayer.com.

Note: I was using a free subscription which acts as a trial/demo for testing purposes, which is perfect for this testing. Specifically, I was using API on https://apilayer.com/marketplace/exchangerates_data-api. Additionally, I want to point out that it is pretty easy to use and it even offers template code for multiple programming languages to call its API.

From the offered API set, we are using only one and that is:

GET/timeseries

The targeted API allows us to pull exchange rates from the past (max. period of 365 days), which should result in a lengthy response that should take some time to assemble/send. apilayer.com is really fast and responsive, and for that reason, a whole year of exchange rates is taken for this test.

As before, we are creating a sync version and async version to execute API calls. As before, three consecutive tests will be made, and the average displayed.

API function-based views:

def api_exchange_sync():
    url = "https://api.apilayer.com/exchangerates_data/timeseries"

    payload = {}
    headers = {
        "apikey": "[Insert your API key]"
    }

    r = httpx.get(url, params={"start_date": "2016-01-02", 
                               "end_date": "2017-01-01"},
                               headers=headers)

    return r.json()

async def api_exchange_async():
    url = "https://api.apilayer.com/exchangerates_data/timeseries"

    payload = {}
    headers = {
        "apikey": "[Insert your API key]"
    }

    async with httpx.AsyncClient(timeout=50.0) as client:
        r = await client.get(url, params={"start_date": "2016-01-
                             02", "end_date": "2017-01-01"},
                             headers=headers)

    return r.json()

We set the timeout on AsyncClient to prevent httpx.ReadTimeout error. That can happen if you mash the button for restarting API call :P.

Sync and async function-based views:

def append_data_sync(request):
    start = time.time()
    e_list = []
    data = [api_exchange_sync()]
    for k in data:
        e_list.append(k)
    end = time.time()
    print('read_sync time: ', end - start)
    return HttpResponse()

async def append_data_async(request):
    start = time.time()
    e_list = []
    data = await asyncio.gather(*[api_exchange_async()])
    for d in data:
        e_list.append(d)
    end = time.time()
    print('read_async time: ', end - start)
    return HttpResponse()

Test results

async: ~2.23437 s
sync: ~3.30023 s

As we iterate through lists of received data from API functions, we append them to a list. Future objects from asyncio.gather() proved advantageous and I believe that if used data streaming, we would be even faster. Async won the race in this test.

Conclusion

Async came in handy while we were waiting for API to respond, we managed to receive data faster and we used the time better. Sync excelled when there was a need for internal data processing and communication with DB via ORM. There was no need for waiting, shielding and multithreading hence the extra speed sync got. There is no need to compare which is faster with sync and async - because it heavily depends on the context of what we want to do. It's important to recognize CPU-bound and I/O-bound situations to properly utilize async and sync. We don't need to use async for every little delay we encounter. I believe a pattern of delays (during our development) must be noticed before a decision is made to implement the async approach.

Django 4.1 async update is not a speed update - I rather look at it as an update that allows us to cover better and more situations.

References

mendix.com, https://www.mendix.com/blog/asynchronous-vs-synchronous-programming/, 'Difference between async and sync programming', David Bevans
djangoproject.com, https://docs.djangoproject.com/, 'Async related pages, related 4.1 changes and sites'
djangoproject.com, https://docs.djangoproject.com/en/4.0/topics/testing/tools/
djangoproject.com, https://docs.djangoproject.com/en/4.1/topics/testing/advanced/