DEV Community: Tobi-De

Task queues and schedulers in django

Tobi-De — Fri, 19 Jan 2024 20:34:23 +0000

This article has been excerpted from a series of guides crafted for my project, Falco. You can explore the complete set of guides at falco.oluwatobi.dev.

Task queues

Task queues are used to offload tasks to a dedicated worker process when the processing of those tasks does not fit into a traditional request-response cycle. In other words, if you need to do something that might take too long to process and whose result does not need to be shown immediately to the user, you use a queue manager.

Think of it as your Django application having an assistant. When a task is too time-consuming to be handled instantly, it's assigned to the assistant for completion, allowing your app to continue functioning normally and serving your clients. Once the assistant completes the task, it returns the result to you.

Example

A user uploads a large excel file to your web application for processing.
The Django app immediately returns a processing... message to avoid blocking.
The Django project adds the task to a broker (redis, rabbitmq, database, etc.), a service used to store tasks that need processing.
Another process, the worker, retrieves the task from the broker and processes it.
The worker process the task.

There is a final optional step not shown in the diagram to avoid clutter. In this step, through some form of callback mechanism, the worker notifies the Django app that it has completed its work and sends back the result. How this callback mechanism works depends on the tool you choose for the
task queue implementation.

Task scheduling

Schedulers are used to periodically execute tasks. They assist in scheduling tasks to run at specific times or intervals. For instance, if you need to send an email to your users every day at 8:00 AM, a scheduler can be used for this purpose. While this can also be achieved using a cron job, which is a common approach, most task queues also provide a scheduler. If task scheduling is all you need, a simple and straightforward option is to use cron jobs in combination with custom django management commands (or job scheduling from django-extensions). The Django management command would contain the code to, for example, send the email to users, and the crontab would execute the command at
the specified schedule.

Popular packages options

There are many options available in the django third-party
ecosystem, some focus solely on providing a task queue, others are just schedulers and many of them provide both in one package. You can also search for purely python solutions and integrate them into your django project yourself.

I prefer options that do not require additional infrastructure (redis, rabbitmq, etc.) for simple tasks. For instance, solutions that can leverage any existing database setup I have. For more complex tasks, I tend to choose a solution that supports redis as a task broker.

My task queue of choice

My current pick is django-q2. It is a fork of the
original django-q project, which is no longer maintained. Here is an example of using django-q2 to run a task in background:

from django_q.tasks import async_task

def long_running_task(user_id):
        ...

def my_view(request):
        task_id = async_task(long_running_task, user_id=request.user.id)
        ...

The async_task function returns instantly, as it doesn't directly execute the long_running_task function. Instead, it delegates the function to a worker for execution. To initiate this worker process (which also includes a scheduler), you need to open a new terminal and execute the following command:

python manage.py qcluster

Couldn't get much easier than this, right? :)

Package	Repository URL	Task Queue	Scheduler	Requires External Service?
django-q2	https://github.com/GDay/django-q2	Yes	Yes	No
wakaq	https://github.com/wakatime/wakaq	Yes	Yes	Yes
django-pgpubsub	https://github.com/Opus10/django-pgpubsub	Yes	No	No
rocketry	https://github.com/Miksus/rocketry	No	Yes	No
django-dramatiq	https://github.com/Bogdanp/django_dramatiq	Yes	No	Yes
django-rq	https://github.com/rq/django-rq	yes	yes	yes
	https://github.com/dsoftwareinc/django-rq-scheduler
procrastinate	https://github.com/procrastinate-org/procrastinate	Yes	Yes	No
django-chard	https://github.com/drpancake/chard	Yes	No	No
celery	https://github.com/celery/celery	Yes	Yes	Yes

Auto reload in development

If you are using one of these you might want an automatic reload feature when files changes in dev, you can use the hupper python package for that purpose. It watches for file changes in the current directory and restarts the worker process automatically.

hupper -m django_q.cluster

Basic django-q2 configuration

Using the database as broker

Q_CLUSTER = {
    "name": "DjangORM",
    "workers": 4,
    "timeout": 90,
    "retry": 120,
    "queue_limit": 50,
    "bulk": 10,
    "orm": "default",
    "catch_up": False,
}

Using redis as broker

CACHES = {"default": env.cache("REDIS_URL")}

# This configuration assumes that Redis is configured for caching in a similar way as described above.
Q_CLUSTER = {
    'name': 'DJRedis',
    'workers': 4,
    'timeout': 90,
    'django_redis': 'default'
}

Deployment with a task queue

Deploying a Django project that uses a task queue is not as
straightforward, but still relatively simple. At this point, I hope you've understood that running a task queue or task schedulers implies running another process (the worker) in addition to your django server. You can have one process for the task queues and another for the schedulers, but usually, with most packages, you can have both in one process with one command. For example, if you chose django-q2, all you
need to run is:

python manage.py qcluster

This command will enable both the task queue and scheduling
capabilities. If you are running your Django app on a Linux server, the most common option is to have a process manager to run and manage both your Django server and the worker process, or any other processes your Django project needs. The two most popular options are systemd and
supervisord. Systemd is natively available on most Linux distributions, but you need to install Supervisor. In my experience, there are no real advantages of one over the other, so I
would advise just picking one; either will be fine.

Here are some basic configuration examples. Please note that the code provided only concerns the worker process.

Systemd example

[Unit]
Description=Your Django Qcluster Worker

[Service]
WorkingDirectory=/path/to/your/project
ExecStart=/path/to/your/venv/bin/python manage.py qcluster
User=your_username
Group=your_groupname
Restart=always
StandardOutput=append:/var/log/your_project/qcluster.out.log
StandardError=append:/var/log/your_project/qcluster.err.log

[Install]
WantedBy=multi-user.target

Supervisor example

[program:your_project_qcluster]
command=/path/to/your/venv/bin/python manage.py qcluster
directory=/path/to/your/project
user=your_username
group=your_groupname
autostart=true
autorestart=true
stderr_logfile=/var/log/your_project_qcluster.err.log
stdout_logfile=/var/log/your_project_qcluster.out.log

If you are running your project with Docker, the process is the same.
You need to have another Dockerfile in addition to your main one. This
Dockerfile is practically identical, but with the entry command running
the worker process (e.g., python manage.py qcluster) instead of your
Django application server. There is also a simple alternative to run
both the Django process and the worker in a single container. For more
on that, read the guide on running your project in a single
container.

On the other hand, if you are running your project on a platform as a service (PAAS), they usually have a way to declare a worker process. For example, Heroku (and most PAAS that use a Procfile) have a
straightforward way to declare a worker process in the Procfile.

Here is an example of what that looks like with Heroku:

web: gunicorn myproject.wsgi
worker: python manage.py qcluster

The End

In conclusion, this guide aimed to provide enough information for you to
understand and choose a task queue solution for your Django project, and
to grasp its potential impact on your deployment process. For any
questions or feedback, please open a
discussion.

Falco - CRUD generator, project-starter and a set of guides for django devs

Tobi-De — Mon, 25 Dec 2023 16:41:21 +0000

Hi all,

I've been working on a side project for the last few days that I would like to share with you. It is a rewrite of a previous project. I mentioned on the GitHub readme that it was a toolkit, but I'm not really sure how to describe it, so I'll just say what it does. There are two parts to the project: a CLI and a set of guides.

The most noteworthy commands in the CLI are a project starter that generates a Django project setup with Hatch, htmx, and Tailwind, and a CRUD (Create, Read, Update, Delete) command generator that creates CRUD views based on a Django model. It is a similar idea to Neapolitan but with a completely different approach. Instead of a class to inherit from, the CLI generates and inserts code for function-based views directly into your views.py\ file. It also generates some basic html templates with minimal Tailwind styling and registers URLs. The goal is faster prototyping. You write a model, and then you can easily get basic CRUD views to test things out. Updating the code is quite straightforward since it is directly available in your source code files.

The second part is the Guides. You can think of it as the Django official topic guides, but instead of focusing on the components of the framework like forms, models, class-based views, etc., it is more focused on more general web subjects like realtime, task queues, deployment, etc. The idea is to have a place where you'll get general views on how to handle each topic, but from the perspective of a Django developer. Much of it is based on my own experience, but I'll also do some additional research since the idea is to get a general view and explore available options for handling each topic.

All the available CLI commands and guides are listed on the GitHub readme.

The project caters to a target audience ranging from beginners to intermediate developers, leaning more towards the latter. Please note that this project is actively under development. While the CLI is nearing completion, the guides are not yet halfway done.

Finally, thank you for taking the time to read this, and Merry Christmas to all those who celebrate!

https://github.com/tobi-de/falco

The Art of Skimming Through Documentation

Tobi-De — Wed, 20 Dec 2023 02:27:52 +0000

TL;DR: The title is already self-explanatory, but nonetheless: Skimming through well-written documentation is one of the best ways I know to discover new techniques and tools. It is easier and can sometimes be more effective than straining your brain attempting to decode someone else's code for the purpose of learning.

This could have been the perfect opportunity to begin with the classic: This is something I wish I knew as a beginner. However, in my case, I believe I was already doing this even as a beginner. I don't recall where I picked it up, but I knew two things that might have pushed me toward that edge:

There is a lot to understand in this tech stuff.
Understanding any topic requires reading about it first. Reading may not be enough, but it is a prerequisite at least.

I'm sure you've already heard countless times the classic go read the docs advice. Some colleagues might have even shouted that to you. And I can reassure you, they were right. Reading documentation is crucial, but I'm here to advocate for something else, similar yet different and also complementary - skimming through the documentation.

What do I mean by skimming? The basic workflow for most is as follows: use a tool, encounter a problem, Google for a solution (or refer to the documentation if you're more experienced and if it is applicable), and repeat the cycle. This approach works fine and you'll eventually become proficient with that tool after repeated use.
But let's imagine this: the first time you use the tool, you skim through the docs. You speed run it, reading every chapter title and subtitle, and maybe even a few lines of each paragraph (if you have time and the doc isn't too long, you might read the whole thing). The first thing you'll gain from this is a sense of possibilities. You'll get a general overview of what you can do with the tool, and a deeper one than if you had just read the readme or tutorial section. Who knows, perhaps while reading, you'll discover a new approach or pattern, or even a new tool(maybe a dependency of that project). I can give you two examples of when that happened to me:

I gained a better understanding of concurrency in Python while reading the FastAPI docs.
I discovered the PostgreSQL LISTEN/NOTIFY protocol while reading the Procrastinate documentation.

The patterns and tools that I discovered while exploring the deeper parts of tool documentation often bring me value later on.
You can also view skimming through the documentation as a form of preliminary research. Doing so will assist you in acquiring enough background knowledge to make using the tool easier and, as they say, effortlessly connecting the dots.

Small aside, this "A.I." trend is taking away the opportunity for a lot of people to appreciate well-written documentation. Copilot and ChatGPT are great for getting quick answers, but they usually keep you on a surface level of understanding (unless you continuously prompt them for deeper knowledge, which I doubt most people do). These tools are lowering the barrier to entry and increasing productivity for developers, but at the cost of a deeper understanding of the tools they are using. The best developers I know are the one with a deep knowledge in their craft. But who knows, maybe we will all be obsolete in a decade and none of this will matter.

A side benefit of skimming through documentation is increasing your chances for those moments when you think, Haven't I read something on this somewhere? It's unlikely that you'll remember even half of what you skimmed through, but that's not the point. The goal is to potentially discover new techniques and tools and increase your chances for these déjà vu moments when you stumble upon an issue that might be solved by something you already read in the past.

Small Tip: Reading code can be challenging, especially when dealing with large projects written by others. However, even if you can't understand the code itself, take a look at the requirements or dependencies - you might find something valuable there!

I hope you have learned something from this article. What I describe here may not work for you or match your style of learning, and that's fine. I'm just making you aware of this option in case it was not already the case. And if you are already doing this, great buddy! Just know that you are not alone.

A comprehensive guide to multi-timezone support in Django

Tobi-De — Wed, 11 Oct 2023 15:40:36 +0000

TL;DR: To support multiple timezones in your Django project, you need a way to request your users' specific timezones and create a middleware that uses django.timezone.activate(user_tz) to enable a specific timezone for a user globally on your site. This ensures that every use of django.timezone.now() uses the activated timezone. If you prefer reading code directly, see middleware.py and models.py.

Throughout this article, I'll guide you on setting up multi-timezone support in a Django project. This post is aimed at beginners (assuming basic Django knowledge) and intermediates. If what I'm writing seems blatantly obvious to you, you're likely not in one of these categories, so please bear with us.

Introduction

Time is universally a challenging subject to deal with, especially in software engineering. Handling time zones correctly is hard. Luckily for us, when working with Django, a significant portion of the work has already been done.

For the longest time (literally until yesterday), I thought that the piece of code below was enough to have timezones fully managed and working in Django:

# settings.py
TIME_ZONE = "UTC"  # Sometimes, I'd switch this to my primary audience's timezone.
USE_TZ = True

It turns out I was wrong, or at least I was missing the full picture. There's a bit more work left if you want to consider every user's timezone. Let's explore what's left to do.

Note The official Django documentation on timezones is very well written, so I highly suggest you read it. However, this article will provide you with the essentials, at least that's what I hope.

The Basics

First, let's start with the basics. In Python, there are two types of datetime objects: naive and aware. We'll keep it simple. Naive datetime objects hold no timezone information, while aware datetimes hold timezone information.

When you have the USE_TZ setting set to True in your project settings (as shown in the snippet above), Django will ensure that all the DateTime objects you create are timezone-aware. This is, of course, assuming that you use Django's timezone module to create your dates.

from django.utils import timezone

now = timezone.now()

This works if you're not concerned about user-specific timezones, and you're not doing anything fancy with datetime. But what if you are?

The Challenge

Let me paint a picture: I'm building an app where I need to send daily notifications to users at specific times. My project's timezone is set to UTC – the recommended practice by most of the Django community. Here's the twist: I live in a UTC+1 timezone, and I realized my notifications were coming in an hour late. My cousin, who lives in Europe at a different timezone, also uses my app. Changing the default timezone wasn't an option. My cousin and I, along with potentially many other users, needed to navigate the app seamlessly without time feeling off. Scheduling and time management are at the core of my app, and I couldn't afford to mess that up.
So, I decided to dive deep into understanding how Django deals with time zones – something I'd never made a priority before.

After reading and re-reading the Django documentation, things started to click. First, let's grasp the fundamentals and then dive into setting up our timezone magic.

There are two important concepts that Django uses: the "default time zone" and the "current time zone."

The default time zone is the timezone you set in your Django settings via TIME_ZONE.
The current time zone is the timezone used for rendering. It's the one in which your users will browse your site.

The current time zone defaults to the default time zone unless you activate the user's specific timezone using django.utils.timezone.activate.

Does it start to make sense?
Django, by default, doesn't know a user's timezone. It's not typically available in request data, so it sticks with the default timezone. You need to ask your users for their time zone – a simple form does the trick – and then manually activate it. The easiest and probably the best way to do this is through a middleware.

Setting Up Multi-Timezone Support

Now, let's dive into a simple example. We'll create a user Profile model to collect and store each user's timezone. For simplicity's sake, I'll leave out the non-essential parts of the code.

Here's a model for our users' profiles:

import zoneinfo

TIMEZONES_CHOICES = [(tz, tz) for tz in zoneinfo.available_timezones()]

class Profile(TimeStampedModel):
    user = models.OneToOneField(
        "users.User", related_name="profile", on_delete=models.CASCADE
    )
    timezone = models.CharField(
        verbose_name=_("Fuseau horaire"),
        max_length=50,
        default="UTC",
        choices=TIMEZONES_CHOICES,
    )

With the model above, we have a simple CharField to hold the user's timezone and a choices field to render a select using a Django form. You can customize the list of available timezones obtained from the zoneinfo module. For a simpler user experience, you might choose to filter it by continent or display the city as the label.

Tip: If you're displaying a long list of timezones, consider using a select element with a search feature(e.g, tom-select) for a better user experience.

You'll need to create a form and a view to set the timezone value – standard Django stuff.

Now, let's create the middleware to activate the user's timezone:

import zoneinfo

from django.utils import timezone

from .models import Profile


class TimezoneMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        if request.user.is_authenticated:
            try:
                profile = request.user.profile
            except Profile.DoesNotExist:
                timezone.deactivate()
            else:
                timezone.activate(zoneinfo.ZoneInfo(profile.timezone))
        else:
            timezone.deactivate()
        return self.get_response(request)

This middleware activates the user's timezone if they're authenticated and have a profile. In other cases, we call deactivate, which sets the timezone to the default. While this last step is not strictly required, that's how the Django docs suggest doing it, so let's stick with that. To complete the setup, don't forget to register your middleware in your settings within the MIDDLEWARE list.

MIDDLEWARE = [
     ...
    "your_app.middleware.TimezoneMiddleware",
]

With this setup, we've made sure that every call to timezone.now will takes the user's specific timezone into account. When datetime objects are saved to the database, they are automatically converted to UTC. For example, in my case (UTC+1), if I input "6:00 pm" on the UI to record a time, it will be saved in the database as 5:00 pm (though it will still be rendered as "6:00 pm" to me on the frontend).

Quick Tips and bits:

If you need to create a timezone-aware datetime object manually, for example by combining a date and time, here's how you do it:

import datetime as dt

naive_datetime = dt.datetime.combine(my_date, my_time)
aware_datetime = timezone.make_aware(naive_datetime, zoneinfo.ZoneInfo(user_timezone))

The code below will generate a new migration every time you run python manage.py makemigrations:

TIMEZONES_CHOICES = [(tz, tz) for tz in zoneinfo.available_timezones()]

class Profile(models.Model):
  timezone = models.CharField(
          verbose_name=_("Fuseau horaire"),
          max_length=50,
          default="UTC",
          choices=TIMEZONES_CHOICES,
      )

The culprit line is choices=TIMEZONES_CHOICES. A simple fix is to update the migrations file to use the TIMEZONES_CHOICES constant directly:

class Migration(migrations.Migration):
    ...

    operations = [
        ...
        migrations.AddField(
            model_name="profile",
            name="timezone",
            field=models.CharField(
                choices=TIMEZONES_CHOICES,
                default="UTC",
                max_length=50,
            ),
        ),
    ]

Conclusion

There you have it! Your Django app is now fully equipped to support multiple timezones.
I hope this article adds a touch of timezone magic to your Django projects.
Thanks for the read.

Maximizing Productivity: PyCharm and htmx Integration

Tobi-De — Mon, 18 Sep 2023 09:32:41 +0000

TL;DR Learn how to add support for htmx in PyCharm for seamless development using web-types.

If you're not familiar with htmx, check out this fantastic htmx + Django introduction by BugBytes.

Introduction

htmx is my go-to frontend tool for building web applications, and PyCharm from JetBrains is my daily code editor/IDE. Unfortunately, by default, PyCharm doesn't recognize htmx attributes when used in templates, resulting in ugly warning lines 🙁. This article will guide you on how to resolve this issue and improve your development workflow.

The Fix

There's a simple way to add autocompletion and documentation for htmx attributes in JetBrains editors using web-types. Web-types is a JSON-based format that provides IDEs with metadata information about web component libraries like htmx. I stumbled upon this tip via this tweet, so kudos to the author for sharing it!

Step-by-Step Guide

Create a package.json File

Start by creating a package.json file, the central configuration file for Node.js-based applications. Although we're not building a Node.js package, this is the method to reference web-types in your PyCharm project. Ensure you have Node.js installed on your computer. If not, I recommend using nvm to install it. After installing Node.js, you can use npm (included with all Node.js installations) to initialize a Node.js project and generate the package.json file.

   npm init -y

Create htmx.web-types.json File

Create a new file named htmx.web-types.json and copy the htmx web-types source into it:

   touch htmx.web-types.json

Update package.json

Add a new entry to your package.json with "web-types" as the key and the path to your htmx.web-types.json file as the value. Here's an example:

   {
     "web-types": "./htmx.web-types.json"
   }

Enjoy Autocompletion and Documentation

Now, in your HTML templates, type an htmx attribute like hx-get, and you should see autocompletion and documentation directly accessible in your IDE. Cool, right? 😎

Conclusion

With this simple setup, you can enhance your PyCharm experience and work seamlessly with htmx attributes in your Django projects. Happy coding!

Exploring SSE and PostgreSQL for Realtime Communication in Django

Tobi-De — Sat, 16 Sep 2023 14:43:59 +0000

TL;DR: In this article, I explore how we built a relay system to serve real-time notifications to our Django project using Server-Sent Events (SSE) and PostgreSQL LISTEN/NOTIFY. Check out the final project on GitHub here. 🚀

Recently, at my workplace, I was tasked with creating a real-time notification system for one of our Django projects. I opted for Server-Sent Events (SSE). Luckily for me, around that time Django 4.2 introduced async iterators support for StreamingHttpResponse, which seemed ideal. However, as I delved deeper into the project, I decided instead to build a relay service using Starlette which transmits messages received through the PostgreSQL LISTEN/NOTIFY protocol to client browsers connected via SSE.

It all began with an excellent post on Writing a chat application in Django 4.2 using async StreamingHttpResponse, Server-Sent Events, and PostgreSQL LISTEN/NOTIFY. I read the post and successfully applied the concepts. This wasn't my first encounter with the PostgreSQL LISTEN/NOTIFY protocol; I knew that tools like Procrastinate (a python task queue) also used it. This protocol was appealing because it eliminated the need to introduce additional infrastructure like Redis into our setup since we were already using PostgreSQL as our database. At the end, what I did added an extra service to our setup, but I think it was worth it, even just for the idea it sparked.

Update: I identified the issue that was crashing our database, and it had nothing to do with our Daphne setup; it was related to our business logic, entirely on our side. To simplify, we have a multi-tenant application, and some users, mainly administrators, have access to multiple tenants. The problem arose when you were an admin with access to multiple tenants, but no specific tenant was selected. If no tenant was selected, we cannot determine which channel to listen to. As a result, we sent an empty channel name to the server as part of the SSE connection request. Consequently, our server responded to SSE connections with an empty response, causing the browser to try to reconnect each time.
The solution is straightforward. When a tenant is not selected (meaning no channel can be determined), we send a response with a retry interval set to a high value (in our case, 1 year). This effectively prevents the browser from attempting the SSE connection. This is the simplest solution we've found to halt the SSE connection for now.

Initially, the setup worked seamlessly. Notifications were correctly transmitted and received. We switched from using Gunicorn to Daphne to run the project, and everything seemed fine. However, after a while, a colleague noticed that our server was constantly receiving requests on our SSE async view. It turned out that the connection was consistently disrupted, causing the browser to repeatedly attempt reconnection. This ultimately led to our PostgreSQL database sporadically crashing with an error message stating "too many clients."

At this point, we did something that, now that I think of it, makes absolutely no sense. We split our Django server into two instances: one running in WSGI mode with Gunicorn, and the other under Daphne in ASGI mode. It did not resolve the issue, maybe we did that thinking it would mitigate the issue? Who knows. However, it did provide me with an idea – replacing the Django app running under Daphne with something better suited for asynchronous operations.

I ended up creating a relay using Starlette, which transmitted messages received through the PostgreSQL LISTEN/NOTIFY protocol to client browsers connected via SSE (The diagram above). This solution proved to be more stable and reliable than the Django approach. Furthermore, it allowed me to simplify the original Django project by completely removing the ASGI setup and code from the project. I was not very fond of the idea of mixing and matching sync and asynchronous code anyway, so this was a win for me. For a straightforward communication method via SSE in Django, this approach worked pretty well.

I've noticed several repositories on GitHub exploring similar ideas. I'm curious about its long-term viability 🤔, but for now, it works pretty well with our setup. After all, we only need the async stuff for real-time notifications. It was not worth turning our entire project into ASGI just for that. Plus, we can easily use the relay for some other precise and simple stuff in the project like updating charts, etc.

Thanks for reading! I've turned the relay project into a package and a Docker image; you can check it out at https://github.com/Tobi-De/sse_relay_server.

Fuzzy-Couscous - The CLI for a better django development experience

Tobi-De — Mon, 27 Feb 2023 11:17:30 +0000

Hello everyone!

I'm excited to introduce you to fuzzy-couscous, my latest project designed to enhance your django
development experience with Django.

I've spent the last few months working on this command line tool, and I'm excited to share it with you and get your feedback.
The primary feature of fuzzy-couscous is to generate a django project using a project template that I've built, inspired heavily by the popular cookiecutter-django
project template. I simplified their template as much as I could while adding some of my favorite tools like htmx and poetry.

With fuzzy-couscous, you can quickly create a new Django project using the built-in template, which includes support for either Bootstrap or Tailwind.
It also comes with a variety of convenient commands, such as launching multiple processes at once including the Django dev server, a Redis server, a worker process, and more (although I'm currently working on improving this feature),
as well as a command to easily download the latest version of htmx. I have some exciting ideas for potential new commands, including one to generate CRUD code (#27) and another to ensure that projects
created with fuzzy-couscous can easily get updates from the base template (#22).
I'm always looking for new ideas and suggestions, so please feel free to open a GitHub discussion or an issue.

This project is nothing revolutionary, but it's useful for me and I think it could be useful for other django developers out there.
If you're interested, you can find fuzzy-couscous on GitHub at https://github.com/Tobi-De/fuzzy-couscous.
I hope you'll give it a try and let me know what you think. Thanks for your time!

Host your Django project on DigitalOcean

Tobi-De — Mon, 12 Oct 2020 07:12:25 +0000

In this post we are going to see how you can easily deploy your awesome Django project on a linux server( Virtual Private Server a.k.a vps ). In this tutorial I will be using DigitalOcean, a well know cloud provider that offers you a full control on your server, I also choose DigitalOcean because they have a 1-click app Dokku droplet to get you up and running quickly.

What is Dokku ?

Dokku is an herokuish (give similar environment to heroku) platform that help you deploy quite easily any web based application like a Django app. It is an open source and free PaaS built on Docker. If you already own a vps, installing Dokku is very easy, you can follow the installation guide. This is not the only way to deploy your Django project, you can get away with the basic nginx + gunicorn + postgresql manual setup, but I find this option more easy to setup and manage, I don't like the hassle that comes with managing a linux server, I want to focus on building solutions not managing servers and I think Dokku help with that. If you are Interested by other ways to deploy your django applications, I've linked some useful resources at the end of this guide to help you.

Prerequisites

What you need to be able to follow this guide.

A DigitalOcean account, you can create one here (following this link you will get $100 free credits to be able to test the plaform)
A droplet (DigitalOcean term to represent a linux server) with Dokku installed on it, fortunately for us, DigitalOcean got us covered, you can create one with Dokku already installed on it here
A domain name, you can get one for cheap at namecheap or GoDaddy, use the provider of your choice, I'm using namecheap in this guide.
A Django project obviously 😆
git installed on you computer

Setup your VPS

At this step I presume you have already a droplet (or a vps) with linux installed on it, in my case ubuntu 18.04 LTS. Dokku should already be installed also, if you use the DigitalOcean 1-click Dokku droplet app then you are ready to go. After the server is launched you should wait few minutes for dokku to be ready. Meanwhile follow these steps to get your server ready.

Connect to your server with ssh then search for update

If you are using DigitalOcean, go to your dashboard then click on Droplets, select your droplet, then click on your ipv4 address to copy it and run this command

ssh root@your_ip_address

When you were setting up your droplet, you added a root password, you need it here to run sudo commands.

sudo apt update
sudo apt upgrade

add a limited user

adduser your_user_name
adduser your_user_name sudo

Now you can close your session and connect back with

ssh your_user_name@your_ip_address

setup ssh server on local

If you find it painful to type your ip address each time, you can just add an alias to your system or even better, add this code to your local ~/.ssh/config file.

Host a_server_name
    ForwardAgent yes
    Hostname your_ip_address
    Port 22
    ServerAliveInterval 60
    ServerAliveCountMax 60

Now to connect to your server, you can type

ssh your_user_name@a_server_name

create an ssh public key

You will need an ssh key to configure your dokku instance, if you already have one then you can skip this step. If you are on linux, run this command to generate the ssh key. The command will ask for the path to create the ssh key and a passphrase. You can leave the default path for the key directory, and for the passphrase you can leave it blank but you will have to remember it if you set one.

ssh-keygen -b 4096

If you are on windows then you can follow this link to help you create your ssh key.

Setup your domain name

This is very simple, just create an A record with the value of your server ip address, example :

Type : A Record

Host : *.

Value : XXX.XXX.XXX.XX (your ip address)

Configure Dokku

At this step, your Dokku instance should be ready. Visit http://your_ip_address to configure the dokku instance. You should see a Dokku Setup page with a form, the only field you need to worry about is the Public ssh key field. If you generate the key with the command above and leave the path as the default one (it should be something like this ~/.ssh/id_rsa.pub) , then open the id_rsa.pub file with a text editor and copy the content inside the Public ssh Key field, then click finish setup and now your dokku instance is configured.
To test it, connect to your server and type dokku (try with sudo dokku if you have permissions issue), if you see the help menu then you are good to go, if not you can always ask a question in the comment section bellow or check if you miss a step above.

Now that our dokku instance is ready, we can deploy our django project, but before that we need to install some plugins and create the app.

create an App

dokku apps:create your_app_name

Install postgres plugin

This is the command you need to install the postgres plugin to your dokku instance :

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git

Create a new postgres database instance and link it to your app by running the commands below, linking the database to your app will automatically create a DATABASE_URL environment variable that connect your app to your database instance.

sudo dokku postgres:create your_app_name_db
dokku postgres:link your_app_name_db your_app_name

SideNote : if you need a redis instance, just replace "postgres" by "redis". For any other plugins refer to the offcial dokku documentation plugin page. To setup database backups on an aws S3 bucket check here

Add a domain to your app

dokku domains:add your_app_name your_domain
dokku domains:set your_app_name you_domain

you_domain: something like www.mydomain.com

Set environment variables

You need to set the environment variables before pushing your app to dokku, if not the deployment will fail. To set an environment variable, type:

dokku config:set your_app_name env_name=env_value

Example

dokku config:set my_app PYTHONHASHSEED=random
dokku config:set my_app WEB_CONCURRENCY=4
dokku config:set my_app DJANGO_DEBUG=False
dokku config:set my_app DJANGO_SETTINGS_MODULE=config.settings.production
dokku config:set my_app DJANGO_SECRET_KEY="$(openssl rand -base64 64)"
dokku config:set my_app DJANGO_ADMIN_URL="$(openssl rand -base64 4096 | tr -dc 'A-HJ-NP-Za-km-z2-9' | head -c 32)/"

Deploy your app

Create a Procfile

In the root of your project create a file name Procfile and inside that file copy this code

web: gunicorn wsgi_file_path:application
release: python manage.py migrate

In my project the wsgi.py file is in the config folder, a folder in the root of my project, you need to find your wsgi file and replace the wsgi_file_path by your path, usually it is in a folder named base on your project name, In my case the first line give this:

web: gunicorn config.wsgi:application

Requirements file

You also need a requirements.txt or Pipfile so Dokku can install all dependencies when deploying. If you are using a virtual environment (you better be using one) just type (in a terminal) from the root of your project :

pip freeze > requirements.txt

Add the dokku remote repository to your project with git :

git remote add dokku dokku@my_server:your_app_name

You can now deploy your app with :

git push dokku master

when you are done, you can check your deployment setup like this

dokku run your_app_name bash
python manage.py check --deploy

Sidenote : Using dokku clients

Using a dokku client you can create new apps, add plugins easily from your local machine, check the available clients here. When a dokku client is installed and the dokku remote repository is added to your project with git using the above command, you can easily manage your app from your local machine. To set environment variable normally, you should type (assuming you install the dokku-toolbelt) from the root of your project on your local machine :

dt config:set env_name=env_value

But if you have fifteen or more environment variables to set, it will take you decades to set them all. If you want a more efficient way to proceed, you can use this little python script I wrote to set easily and quickly your environment variables on dokku. Read through it and adjust it to your need. The script can be run from the local or the remote machine.

Setup ssl certificate

Install and configure letsencrypt plugin for secure connection through https.

dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
dokku config:set --no-restart --global DOKKU_LETSENCRYPT_EMAIL=hello@youdomain.com

You can use your personal email if you want, it must be a valid email address.

Add a certificate to your app

dokku letsencrypt your_app_name

Setup a cron task to renew ssl certificate

dokku letsencrypt:cron-job --add

In this tutorial we have seen how you can deploy your django project on a vps using dokku. If you need more deployment options, you can check out the ones below.

Deploy your django project with heroku, if you try this option, you will realize that deployment with heroku is more simple, but in the long run heroku is much more expensive, for example the $25 offer from heroku is roughly equivalent to the $5 droplet from digital ocean.
Deploy django using nginx + gunicorn + postgres, manual setup
Deploy django using docker
Deploy django using kubernetes

Obviously this list is not exhaustive, you can check on google for more options.
Thank you for reading this post, I hope it taught you something, feel free to give me your feedback in the comments section below.
This is the first article I have written so I would love to have feedback so that I can improve myself