An Introduction to Asynchronous Tasks and Background Workers in Django

Asynchronous tasks are operations that are carried out independently of the main program's flow. These operations are performed by simple processes called background workers.

If you didn't quite get that, don't worry. It'll make much more sense in a bit.

In this article, we will go through what asynchronous tasks and background workers are, why they're so important, and how to build a project that utilizes them with Django.

Who is this article for?

I welcome anyone interested in learning something new (or from a different perspective) to gain from my writing. However, I wrote this article with a specific set of people in mind, and they are:

• Developers who have seen these terms before but never really got to understand what they meant
• Django developers who want to learn how to use asynchronous tasks and background workers in their projects

Prerequisites

To get the most out of this tutorial, you'll need:

• Basic Django knowledge (You know, project setup, models, views, templates, etc)
• Basic understanding of the request-response cycle

ATTENTION: To get our hands dirty with asynchronous tasks, we're going to be relying on a distributed queuing system called Celery. Now while Celery technically runs on all major operating systems, there is no (longer) official support for Windows. This means some features like multiprocessing won't work out of the box.

If you're on Windows, I strongly urge you to consider using WSL (Windows Subsystem for Linux) for a smoother experience—as I will assume every reader on Windows is, from here onward. Head here for crystal-clear instructions on how to get it set up, then come back once you're done.

Synchronous Tasks vs Asynchronous Tasks

Before we unravel the secrets of asynchronous tasks, let's understand what it means for tasks to be synchronous using a simple example.

Take a look at this view function:

def example_view(request):
    do_something()
    do_another_thing()
    do_one_last_thing()
    return HttpResponse("Thank you for using our services, User!")

In our example_view, each task is processed one after the other, from top to bottom, before displaying a thank you response to our visiting user. That means do_something is executed completely before do_another_thing is processed, and so on. That is, the tasks run synchronously, or in order.

Additionally, Django was originally built on the WSGI (Web Server Gateway Interface) standard. Don't worry if you're not familiar with that term and what it means. All you need to know (for now), is that Django was designed to handle one request at a time. It will only process request_2 after request_1 has been processed completely. In other words, Django is synchronous by default.

Okay. Nothing wrong here, right?

True, but let's take a closer look at the tasks in our view and give them a little more context.

In Django views, certain operations are typically carried out before returning a response to the client. Creating and updating objects using data from the request are a couple of the more common ones, but what about tasks that are less simple and can involve delays or heavy computation—such as sending emails or generating a PDF:

def example_view(request):
    if request.method == "POST":
        # This is our example_view "doing something" i.e creating a new Note object
        new_note_title = request.POST.get('title')
        Note.objects.create(title=new_note_title) 

        # ...and these are its other two tasks renamed to give additional context
        send_email_to_someone_somewhere()
        some_heavy_task_that_takes_30_seconds()

        # Return a response to the client once the tasks are complete
        return HttpResponse("Thank you for using our services, User!")

We now have a clear idea of what happens in our view. A new note is created first, then the other two tasks are processed one after the other.

Here's the problem.

To send an email, your Django application has to communicate with an SMTP server or email service API. To generate a PDF, you'd typically work with a library that converts rendered HTML to a PDF document. The duration of each of these activities can range from a couple of seconds to a few minutes depending on certain factors. Some of these factors such as SMTP latency and the speed of the PDF library are beyond your control.

What this means for us is that Django will be stuck on a request to our example_view until every task within the view has been processed and completed:

def example_view(request):
    ...

    send_email_to_someone_somewhere() # Takes some time 
    some_heavy_task_that_takes_30_seconds() # Takes at least 30 seconds

    # The user probably leaves your web application dissatisfied before even seeing the appreciative text response
    return HttpResponse("Thank you for using our services, User!")

To prevent scenarios where users have to wait for such operations to complete before making another request, we can convert them to asynchronous tasks, that is, they would run independently and out of the main program's flow. These asynchronous tasks are offloaded to be performed by simple processes called background workers.

What are Background Workers?

Background workers are separate processes or programs that run independently from the main application to perform tasks asynchronously. These workers handle long-running, resource-intensive or time-consuming jobs without blocking the main application's responsiveness.

When we offload these sort of tasks to background workers, our web applications can remain fast and responsive to user requests while ensuring that heavy operations are completed reliably, but in the background.

Put simply, our user wouldn't have to wait on a loading screen while the email sends or some heavy task that takes about 30 seconds:

def example_view(request):
    ...

    # Pretend these operations are now being performed by background workers
    send_email_to_someone_somewhere() # but asynchronously
    some_heavy_task_that_takes_30_seconds() # still takes 30 seconds, but asynchronously (doesn't block the request)

    # The user sees the response almost instantaneously
    return HttpResponse("Thank you for using our services, User!")

How to use Asynchronous Tasks and Background Workers in Django

Now that you understand what asynchronous tasks and background workers are, it's time to figure out how to use them by building a small Django project.

Project Overview

We're going to build an app with an "About Me" form for users to fill. The form will contain fields for their name, bio, hobbies, etc. Our app will generate a PDF summary of their inputs and automatically send it to their email.

Project Setup

• Create a new folder that will house our Django project. You can name it whatever you want to:

mkdir aboutme

• Move into the new directory and create a virtual environment:

cd aboutme
python3 -m venv myenv

• Now activate it:

source myenv/bin/activate

• Install Django and create a new project (we'll call it "core") within the current directory:

(myenv) pip install django
(myenv) django-admin startproject core .

Now, your project structure should look something like this:

- aboutme/
  - manage.py
  - core/
    - __init__.py
    - asgi.py
    - settings.py
    - urls.py
    - wsgi.py

For our project to handle asynchronous tasks, we're going to install Celery, a distributed task queue system, and Redis, to serve as its message broker.
To install the core Celery package and extra dependencies needed to use Redis as its message broker:

(myenv) pip install celery[redis]

The last thing we need to install now is redis-server which runs the actual Redis database process locally:

# On Linux (Ubuntu)
sudo apt-get install redis-server
sudo service redis-server start # Start it

# On MacOS
brew install redis-server
brew services start redis # Start it

To test if redis-server is working correctly, open a new terminal and start the Redis CLI:

# Opens the Redis CLI installed with redis-server
redis-cli

Then type:

ping

If everything works correctly, you should be replied with an enthusiastic:

PONG

Now we're set!

Project Development

To get started, create a new Django application:

(myenv) python manage.py startapp myapp

Now add it to INSTALLED_APPS in your settings.py file:

# settings.py

# ...

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'myapp' # Replace with your application's name if different
]

# ...

Don't forget to tell Django where to check for any templates we'll be using:

# settings.py

# ...

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'], # Add this line
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },

# ...

In your app directory, create a templates directory. Within the newly-created templates directory, create another directory and name it whatever your app was named. This is what your folder structure should look like:

- aboutme/
  - manage.py
  - core/
    - __init__.py
    ...
  - myapp/
    - templates/
      - myapp/

Now let's build the index page for our application.
Create an index.html file in the templates/myapp directory, then copy & paste the HTML code displayed below:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>About Me Form</title>
  <style>
    body {
      font-family: Arial, sans-serif;
      background: #f9f9f9;
      margin: 0;
      padding: 0;
      display: flex;
      align-items: center;
      justify-content: center;
      height: 100vh;
    }

    .form-container {
      background: white;
      padding: 2rem;
      border-radius: 10px;
      box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
      width: 100%;
      max-width: 500px;
    }

    h2 {
      text-align: center;
      margin-bottom: 1.5rem;
      color: #333;
    }

    label {
      display: block;
      margin: 0.75rem 0 0.25rem;
      font-weight: bold;
    }

    input, textarea, select {
      width: 100%;
      padding: 0.6rem;
      border: 1px solid #ccc;
      border-radius: 5px;
      font-size: 1rem;
    }

    textarea {
      resize: none;
    }

    button {
      margin-top: 1.5rem;
      width: 100%;
      padding: 0.75rem;
      font-size: 1rem;
      color: white;
      background-color: #007bff;
      border: none;
      border-radius: 5px;
      cursor: pointer;
    }

    button:hover {
      background-color: #0056b3;
    }
  </style>
</head>
<body>
  <div class="form-container">
    <h2>Tell Us About You</h2>

    {% if messages %}
      <ul style="list-style: none; padding: 0;">
        {% for message in messages %}
          <li style="
            margin-bottom: 1rem;
            padding: 1rem;
            border-radius: 5px;
            color: white;
            background-color:
              {% if message.tags == 'success' %} #28a745
              {% elif message.tags == 'error' %} #dc3545
              {% elif message.tags == 'warning' %} #ffc107; color: black;
              {% else %} #007bff {% endif %};
          ">
            {{ message }}
          </li>
        {% endfor %}
      </ul>
    {% endif %}

    <form method="POST">
      {% csrf_token %}

      <label for="email">Email</label>
      <input type="email" name="email" id="email" required>

      <label for="name">Name</label>
      <input type="text" name="name" id="name" required>

      <label for="about">About Me</label>
      <textarea name="about" id="about" rows="4" required></textarea>

      <label for="hobby">Favorite Hobby</label>
      <input type="text" name="hobby" id="hobby" required>

      <label for="color">Favorite Color</label>
      <select name="color" id="color" required>
        <option value="">-- Select a color --</option>
        <option value="Red">Red</option>
        <option value="Blue">Blue</option>
        <option value="Green">Green</option>
        <option value="Yellow">Yellow</option>
        <option value="Purple">Purple</option>
        <option value="Black">Black</option>
        <option value="White">White</option>
      </select>

      <label for="saying">Favorite Saying</label>
      <input type="text" name="saying" id="saying">

      <label for="dream_job">Dream Job</label>
      <input type="text" name="dream_job" id="dream_job">

      <button type="submit">Generate My PDF</button>
    </form>
  </div>
</body>
</html>

Now, to set up the index view.
In your views.py, copy & paste the lines of code displayed below:

# myapp/views.py

from django.shortcuts import render, redirect
from django.contrib import messages


def index(request):
    if request.method == 'POST':
        email = request.POST.get('email')
        name = request.POST.get('name')
        about = request.POST.get('about')
        hobby = request.POST.get('hobby')
        color = request.POST.get('color')
        saying = request.POST.get('saying')
        dream_job = request.POST.get('dream_job')

        data = {
            'email': email,
            'name': name,
            'about': about,
            'hobby': hobby,
            'color': color,
            'saying': saying,
            'dream_job': dream_job
        }

        # Nothing special happens yet. Just printing the form data
        print(data)

        messages.success(request, "Your summary is being generated and will be emailed shortly.")
        return redirect('index')

    return render(request, 'myapp/index.html') 

Let's configure the URL routing. In your Django app's directory, create a urls.py file and set up the route to our index view:

# myapp/urls.py

from django.urls import path
from . import views 

urlpatterns = [
    path('', views.index, name='index'), 
]

Now in your project's core urls.py, include your app's urls:

# core/urls.py

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('myapp.urls')) # Add this line
]

Now run the development server and check out what we've done so far:

(myenv) python manage.py runserver

# ...then visit http://localhost:8000 in your browser

Perfect. We now have the basic backbone of our project: a form that collects data from users and sends it to our view.

It's time to handle actually generating the PDF and sending it to the user's email—asynchronously.

How Does Celery Work?

Celery is a task queue system that allows you to run operations asynchronously in the background, outside the main request-response cycle of your application.

When a task is triggered, it's packaged as a message and sent to a message broker (in our case, Redis)—instead of being executed immediately. This broker acts as a middleman, holding the task until a separate process, known as a Celery worker, retrieves it.

The worker continuously listens for new tasks, picks them up from the queue, and executes them independently.

Setting Up Celery

Before we can write any asynchronous tasks, we need to configure our project to use Celery. We've already installed Celery and its dependencies, we just need to tell our project how to work with it and vice-versa.

In the same directory as your settings.py (that is, the core directory), create a celery.py file and paste the following lines of code:

# core/celery.py

import os
from celery import Celery

# set the default Django settings module
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "core.settings")

app = Celery("core")

# Load task modules from all registered Django app configs.
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print(f"Request: {self.request!r}")

Then in the __init__.py file in the same directory, add the following lines to make sure Celery is loaded when Django starts:

# core/__init__.py

from .celery import app as celery_app

__all__ = ("celery_app",)

Now we need to add Celery-related configurations to our settings.py file:

# core/settings.py

# Use Redis as the broker
CELERY_BROKER_URL = 'redis://localhost:6379/0'

# (Optional) Celery backend to store task results
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

Our Django project is now ready to work with Celery for queueing and processing asynchronous tasks.

If you found these configurations a bit confusing or overwhelming, that's perfectly fine. These are things that stick the more you use them. You are not expected to grasp it immediately or have it all stored in your head. The Celery docs are perfect for making references and going deeper.

What you need to know (for now) is that you will always make such configurations before your projects can use Celery.

Generating PDFs and Sending Emails Asynchronously

We want to define an asynchronous task that generates a PDF and sends it in an email to a user.

First, we need to install a package that handles the conversion of HTML to PDF. For this, we'll use WeasyPrint:

(myenv) pip install weasyprint

Now we need to create the HTML template that will be converted to a PDF with the user's summary. In your templates/myapp directory create a pdf_template.html file and paste the following lines of code:

<!DOCTYPE html>
<html>
<head>
  <meta charset="UTF-8">
  <title>{{ name }}'s Personal Summary</title>
  <style>
    @page {
      margin: 2cm;
    }
    body {
      font-family: 'Georgia', serif;
      background: #f9f9f9;
      color: #2c3e50;
      padding: 2rem;
    }
    .container {
      background: white;
      border: 1px solid #ddd;
      border-radius: 10px;
      padding: 2rem;
      box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
    }
    h1 {
      text-align: center;
      color: #34495e;
      font-size: 28px;
      margin-bottom: 1.5rem;
    }
    .field {
      margin-bottom: 1.2rem;
    }
    .label {
      font-weight: bold;
      color: #555;
    }
    .value {
      margin-top: 0.2rem;
      font-size: 16px;
      line-height: 1.5;
      padding: 0.4rem 0.8rem;
      background: #ecf0f1;
      border-radius: 5px;
    }
    .quote {
      font-style: italic;
      color: #7f8c8d;
    }
    .footer {
      margin-top: 3rem;
      text-align: center;
      font-size: 12px;
      color: #aaa;
    }
  </style>
</head>
<body>
  <div class="container">
    <h1>{{ name }}'s Personal Summary</h1>

    <div class="field">
      <div class="label">About Me:</div>
      <div class="value">{{ about }}</div>
    </div>

    <div class="field">
      <div class="label">Favourite Hobby:</div>
      <div class="value">{{ hobby }}</div>
    </div>

    <div class="field">
      <div class="label">Favorite Color:</div>
      <div class="value">{{ color }}</div>
    </div>

    <div class="field">
      <div class="label">Favorite Saying:</div>
      <div class="value quote">"{{ saying }}"</div>
    </div>

    <div class="field">
      <div class="label">Dream Job:</div>
      <div class="value">{{ dream_job }}</div>
    </div>

    <div class="footer">
      Generated with ❤️ 
    </div>
  </div>
</body>
</html>

Now we need to write the actual task.

To define tasks in your project, create a tasks.py file in your app directory, then paste the following lines of code:

# myapp/tasks.py

from celery import shared_task # Special decorator used to define asynchronous tasks
from django.core.mail import EmailMessage
from django.template.loader import render_to_string
from weasyprint import HTML
import tempfile

@shared_task
def generate_and_send_pdf_summary(data):
    # Render HTML from template using user's data
    html_string = render_to_string('myapp/pdf_template.html', data)

    # Generate PDF using WeasyPrint
    with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as pdf_file:
        HTML(string=html_string).write_pdf(pdf_file.name)
        pdf_file.seek(0)
        pdf_content = pdf_file.read()

    # Send the PDF via email
    email = EmailMessage(
        subject="Your Personal Summary",
        body="Attached is your personalized summary.",
        to=[data['email']],
    )
    email.attach('summary.pdf', pdf_content, 'application/pdf')
    email.send()

So we have an asynchronous task that can generate a PDF and send it to an email. Now we have to actually trigger it in our view.

Go back to your myapp/views.py and call the asynchronous task we just defined:

# myapp/views.py

# Import the task
from .tasks import generate_and_send_pdf_summary

def index(request):
    if request.method == 'POST':
        # ... shortened for brevity

        data = {
            'email': email,
            'name': name,
            'about': about,
            'hobby': hobby,
            'color': color,
            'saying': saying,
            'dream_job': dream_job
        }

        # Add the task to a queue so it runs in the background
        generate_and_send_pdf_summary.delay(data)

       # ...

    return render(request, 'myapp/index.html') 

We're 99% there. One more thing.

Because we want our application to send emails to users, we have to configure our project's email settings.

Create a .env file in your project root directory and paste these lines with their appropriate values:

# aboutme/.env
# ALWAYS USE A .env file to store secrets even if you're not planning on committing to source control immediately (or at all). I think it's safer and good practice

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = ''
EMAIL_PORT = 587
EMAIL_USE_TLS = True
EMAIL_HOST_USER = '[e-mail]'
EMAIL_HOST_PASSWORD = 'your-password'  

Now install the python-dotenv package which helps our Django project load these environmental variables:

(myenv) pip install python-dotenv

Then in the settings.py file, import the load_env function and call it:

# core/settings.py

from dotenv import load_dotenv
import os # You'll need this too
load_dotenv() # Now call it

# ...

Lastly, add the email configurations to your settings:

# core/settings.py

# ...

# EMAIL
EMAIL_BACKEND='django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST=os.getenv('EMAIL_HOST')
EMAIL_PORT=os.getenv('EMAIL_PORT')
EMAIL_USE_TLS=True
EMAIL_HOST_USER=os.getenv('EMAIL_HOST_USER')
EMAIL_HOST_PASSWORD=os.getenv('EMAIL_HOST_PASSWORD')

And we're done!

Running The Project

It's time to run our entire project.

Starting the Redis Server

Open up a new terminal and start the Redis server by typing redis-server (if it's not already running):

> redis-server

If you got something like "Could not create server TCP listening socket *:6379: bind: Address already in use" as a response, that means the Redis server is already running—which is good news for us.

Starting the Celery worker

To start the Celery worker, open up a new terminal and use the following command from the root of your project (where manage.py is):

celery -A core worker --loglevel=info

Now Celery is ready to handle asynchronous tasks from our application.

Trying out the Application

Make sure the Django development server is running, then check out what you've just created at http://localhost:8000.

Once you fill the form and click on the "Generate My PDF" button, you're immediately shown a message informing you that your PDF would be emailed to you shortly.

This may take a couple of seconds, but you don't have to watch the page load or wait for the process to complete. The app remains responsive, and you could even start filling the form, again. Whenever the task running in the background completes, you'll find your "About Me" summary (or summaries, if you eventually filled the form multiple times) in your email.

I hope this has helped demystify the concept of asynchronous tasks and background workers, and that you now feel more confident approaching them in Django.

If you have any questions, tips, or maybe need some help, please do not hesitate to leave a comment.

Thank you for reading!