Multithreading in Django using threading and Queue

In Django, if the process executed on a request is time-consuming, you may want to keep the process running behind and return a response anyway.

For example, sending a confirmation e-mail when a user registration is completed. If an application that takes 1 second to send an email receives 1,000 accesses per second, following things will be required.

• The application does not lock during network I/O for sending mail.
• The mail sending jobs are run behind and executed in the order in which they are accessed.

I had the opportunity to create an email sending function in Django, so here are my notes.

Confirmation of basic operation

First, we modified the sample code in the queue reference a little to check the process using queue and threading.

import threading
import queue
import time

q = queue.Queue()

def worker():
    while True:
        item = q.get()
        print(f'Working on {item}')
        time.sleep(1)
        print(f'Finished {item}')
        q.task_done()

# turn-on the worker thread
threading.Thread(target=worker, daemon=True).start()

# send thirty task requests to the worker
for item in range(5):
    print("Put item", item)
    q.put(item)
print('All task requests sent\n', end='')

# block until all tasks are done
q.join()
print('All work completed')

First declare q as a Queue instance in a global variable. This queue will be filled with jobs to be executed.

Next, we define the worker method as the job executor, which will retrieve the job, print it, and wait one second. Because of while true, it's always waiting,
When q.get() returns a return value, the underlying process is executed.

Next, start a new thread with threading.Thread(), passing target the method you want to execute. Then add the item as a job to the queue and the worker starts running. q.join() stops further processing until q.task_done() is called for all q.

Outpus

Put item 0
Put item 1
Put item 2
Put item 3
Put item 4
All task requests sent
Working on 0
Finished 0
Working on 1
Finished 1
Working on 2
Finished 2
Working on 3
Finished 3
Working on 4
Finished 4
All work completed

The output shows that all job additions are completed first, and the jobs are executed in the order they were added, indicating that job management and execution are performed asynchronously. It seems that sending emails can also be implemented in this way. Try it.

Asynchronous processing of tasks in Django

First, create a Django API for testing.

$ django-admin startproject mysite
$ cd mysite/
$ python manage.py startapp api

Then add the following code.

mysite/settings.py

...
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'api',
    'rest_framework'
]
...

mysite/urls.py

from django.contrib import admin
from django.urls import path
from api import views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('index/', views.index)
]

api/views.py

from django.http import HttpResponse
# Create your views here.
def index(request):
    return HttpResponse("Hello, world.")

Then, start server

$ python manage.py runserver

Access http://127.0.0.1:8000/index/ and if Hello, world. is displayed, you are ready.

Now let's move on to the creation of the asynchronous mail function.

Create a new file api/mail.py and write the contents as follows.

import threading
import queue
import time

q = queue.Queue()

def worker():
    while True:
        item = q.get()
        print(f'Working on {item}')
        time.sleep(3)  # Do any processing here.
        print(f'Finished {item}')
        q.task_done()

# Five threads are set up to parallelize the process.
for _ in range(5): 
    threading.Thread(target=worker, daemon=True).start()

def add(item):
    q.put(item)

Next, modify views.py as follows.

from django.http import HttpResponse
from api.mail import add as mail_add

# Create your views here.
def index(request):
    item = request.GET.get("param")
    mail_add(item)
    return HttpResponse("Hello, world.")

Restart the server in this state and try sending a series of requests as follows.

curl http://127.0.0.1:8000/index/?param=1
curl http://127.0.0.1:8000/index/?param=2
curl http://127.0.0.1:8000/index/?param=3
curl http://127.0.0.1:8000/index/?param=4
curl http://127.0.0.1:8000/index/?param=5

Outputs

Working on 1
[15/May/2021 12:12:28] "GET /index/?param=1 HTTP/1.1" 200 13
Working on 2
[15/May/2021 12:12:28] "GET /index/?param=2 HTTP/1.1" 200 13
Working on 3
[15/May/2021 12:12:28] "GET /index/?param=3 HTTP/1.1" 200 13
Working on 4
[15/May/2021 12:12:28] "GET /index/?param=4 HTTP/1.1" 200 13
Working on 5
[15/May/2021 12:12:28] "GET /index/?param=5 HTTP/1.1" 200 13
Finished 1
Finished 2
Finished 3
Finished 4
Finished 5

Finished will be output almost simultaneously. If you rewrite the time.sleep(3) part as mail processing, an asynchronous mail sending function is completed.