DEV Community: Kostja Appliku.com

Django + Celery In-depth tutorial for beginners

Kostja Appliku.com — Mon, 14 Oct 2024 12:42:11 +0000

This article was originally posted on Appliku's Blog

Welcome to Django and Celery tutorial.

You will learn what is Celery, why and when to use it and how to set up a Django project with Celery and see a few examples of different Celery tasks.

Git repository

The code from this tutorial is available on GitHub: Celery Tutorial on GitHub

What is Celery?

Celery is a distributed task queue system for Python.

What is a task queue

A task queue is like a to-do list for your app. It stores tasks that need to be executed, along their associated function calls and arguments. Tasks are processed in a specific order, first-in-first-out.

What distributed means?

Multiple workers: Celery can distribute tasks across multiple worker processes. These workers can run on different servers, allowing for parallel processing of tasks.
Scalability: You can scale task processing horizontally by adding more workers on more servers to handle increased workloads without modifying your application.
Fault tolerance: If one worker fails or goes offline, tasks can be redistributed to other available workers.
Load balancing: Celery can distribute tasks evenly across available workers, preventing any single worker from becoming overloaded.
Geographical distribution: Workers can be located in different physical locations or data centers, which can be helpful if workload needs to be performed in different physical locations.

Why and When to Use Celery

While working with Django, most of our code focuses on responding to HTTP requests from browsers and other HTTP clients.

It is important to respond to requests quickly. If a website takes longer than 500ms to respond it can start to annoy users. A few seconds to generate a response is only acceptable in rare situations, such as submitting an e-commerce order.

By default, Gunicorn HTTP server drops the connection if a response takes more than 30 seconds.

While your app might not be complex, there are types of tasks that take long time to complete.

Applications may need to perform complex and time-consuming operations or interact with other services.

For example, sending an email can take anywhere from 50ms to several seconds.

The good news is that some operations can be performed outside the request/response cycle.

Back to the sending an email example, if a user signs up, we can schedule sending an email and proceed with generating and sending the response to the client without the delays for actually sending that email.

One way to achieve this is by using asynchronous features, but that alone doesn't guarantee successful completion of the operation, nor does it provide robust control over retries in case of a failure. Additionally, it would be tied to a specific worker process, if it is terminated, then all in async tasks are lost with it.

This is where task queues, and specifically Celery, come into play. When you need to perform a potentially time-consuming operation, you send a message to a task queue and move on. This message will be later picked up and executed by a Celery worker process. Common choices for the task queue database are Redis or RabbitMQ.

To perform an operation via a task queue, you need to define a Task, which is essentially a function wrapped in a task Python decorator. You will then call it in a special way, which will be explained in this tutorial.

Another use case for Celery is scheduled jobs. If you want your app to perform certain operations at specific intervals, such as every 10 minutes, every hour, midnight, or Saturday mornings, you will need a scheduler. A scheduler is a process that waits for the right moment to send a Celery task to the queue. That's its sole job.

A scheduler does not guarantee the precise timing of execution; it only sends messages to the queue at the correct time. When the execution actually happens depends on your queue workers' setup and the number of tasks in the queue. In some cases, it may not happen at all if no worker is available to process the messages.

You can have multiple queues to separate tasks and prevent clogging. For instance, you might have a separate queue for short-lived but important and time-sensitive tasks like sending emails and notifications. This leaves other potentially long-running tasks in a queue where you are less concerned about their timely completion.

An advanced use of Celery involves having multiple projects communicate asynchronously via different Celery queues. One application can send tasks that only another application reads from, and vice versa. Unless you need an immediate response from the other application (in which case HTTP calls would be necessary), this method of communication is convenient. The other application, often called a service or microservice, does not need to be running all the time or scaled to handle the workload immediately; it will process tasks from the queue as quickly as it can.

Warning

While Celery specifically and task queues in general are great for offloading work from the request/response cycle, they are not a silver bullet for poorly performing code. You cannot merely offload tasks to the queue and expect performance problems to disappear. If you send enough jobs to the task queue, it will eventually take a while for workers to process them. You can scale up the number of workers, but then the database might become a bottleneck, slowing down web requests because background tasks overloaded the database.

I hope this introduction has been useful and provides enough context about what Celery is good for, when to use it, and what it actually does.

Starting a Django project with Celery using Docker

Create a directory for your project, switch to it and initialize a git repository:

mkdir celerytutorial
cd celerytutorial
git init

We will be using Docker for the development of our project. You will not be needing a Python installation outside of Docker.

Open your favorite code editor and create these files:

Dockerfile

The essential part of building a Docker image

FROM python:3.12.5-bullseye
SHELL ["/bin/bash", "--login", "-c"]
ARG USER_ID=1000
ARG GROUP_ID=1000
RUN groupadd -g $USER_ID -o app
RUN useradd -m -u $USER_ID -g $GROUP_ID -o -s /bin/bash app
ENV PIP_NO_CACHE_DIR off
ENV PIP_DISABLE_PIP_VERSION_CHECK on
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1
ENV COLUMNS 80
RUN apt-get update \
 && apt-get install -y --force-yes \
 curl nano python3-pip gettext chrpath libssl-dev libxft-dev \
 libfreetype6 libfreetype6-dev  libfontconfig1 libfontconfig1-dev \
  && rm -rf /var/lib/apt/lists/*
WORKDIR /code/
COPY requirements.txt /code/
RUN pip install wheel
RUN pip install -r requirements.txt
COPY . /code/
USER app

requirements.txt

Needed to specify and manage Python package dependencies for our project

Django==5.1.1
django-environ==0.11.2
gunicorn==22.0.0
psycopg[binary]==3.2.1
whitenoise==6.7.0
Pillow==10.4.0
celery-redbeat==2.2.0
celery[redis]==5.4.0

docker-compose.yml

Needed to define and manage all the services for our project.

x-app: &app
  build: .
  restart: always
  env_file:
    - .env
  volumes:
    - .:/code
  links:
    - db
    - redis
  depends_on:
    - db
    - redis

services:
  redis:
    image: redis:7
    ports:
      - "6379:6379"
  db:
    image: postgres:16
    environment:
      - POSTGRES_USER=celerytutorial
      - POSTGRES_PASSWORD=celerytutorial
      - POSTGRES_DB=celerytutorial
    ports:
      - "5432:5432"
  web:
    <<: *app
    command: python manage.py runserver 0.0.0.0:9000
    ports:
      - "127.0.0.1:9000:9000"
  celery:
    <<: *app
    command: celery -A project.celeryapp:app  worker -Q default -n celerytutorial.%%h --loglevel=INFO --max-memory-per-child=512000 --concurrency=1
  beat:
    <<: *app
    command: celery -A project.celeryapp:app beat -S redbeat.RedBeatScheduler  --loglevel=DEBUG --pidfile /tmp/celerybeat.pid

.env

Create .env and .start.env files with identical content. .start.env will be added to version control, but not .env. .start.env serves as a template for developers to copy and modify locally, ensuring necessary variables are documented without exposing sensitive data.

DATABASE_URL=postgresql://celerytutorial:celerytutorial@db:5432/celerytutorial
REDIS_URL=redis://redis:6379/0
DEBUG=True

.gitignore

Create this file in the root of the project.
This file lists all files and paths within a project that shouldn't be included into version control.
Here is the content of the file you should include for a start, feel free to add paths that you see fit(e.g. temporary files created by your code editor or any other tool)

.vscode/*
.idea/*
node_modules/*
Thumbs.db
Thumbs.db:encryptable
ehthumbs.db
ehthumbs_vista.db
[Dd]esktop.ini
$RECYCLE.BIN/
.DS_Store
.AppleDouble
.LSOverride
__pycache__/
*.py[cod]
*$py.class
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
db.sqlite3
db.sqlite3-journal
*.log
*.mo
*.pot
celerybeat-schedule
celerybeat.pid
env/
.env

.dockerignore

Similar to .gitignore, .dockerignore is a list of files that shouldn't be included into Docker builds.

It is not super important here, but it can help reduce the size of the build image.

In this tutorial we are not using python virtual environments on the host, but let's include it here anyway.

Including the .git folder here also helps reduce the size of the image.

env/
.git/

Pull images with Docker Compose

Go to the terminal and run this command to pull all images for DBs.

Running this command will make Docker download images for services with defined image and not require the build.

This step is optional because they will be pulled anyway on the start of containers.

docker compose pull

The output of successful execution of this command should look like this:

Build Docker image for Django Project

This is an optional step for the first go, because the image will be built on containers start.
But if you change requirements later on, you will need to run this command again.

docker compose build

The output of successful execution of this command should look similar to this:

Start Django Project

Now let's start a shell within our web container.

docker compose run web bash

Now you will see a bash prompt from within a container.
To leave this shell type exit or press CTRL-D.

Start our Django project:

django-admin startproject project .

Initial Django configuration

Open your code editor and let's edit some files.

Edit the project/settings.py file:

from pathlib import Path
import environ
import os

env = environ.Env(
    # set casting, default value
    DEBUG=(bool, False)
)

BASE_DIR = Path(__file__).resolve().parent.parent
# Take environment variables from .env file
environ.Env.read_env(os.path.join(BASE_DIR, ".env"))

SECRET_KEY = env("SECRET_KEY", default="change_me")

DEBUG = env("DEBUG", default=False)

ALLOWED_HOSTS = env.list("ALLOWED_HOSTS", default=["*"])
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'project.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'project.wsgi.application'


# Database
# https://docs.djangoproject.com/en/5.0/ref/settings/#databases

DATABASES = {
    "default": env.db(default="sqlite:///db.sqlite3"),
}
# Password validation
# https://docs.djangoproject.com/en/5.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "handlers": {"console": {"class": "logging.StreamHandler"}},
    "loggers": {"": {"handlers": ["console"], "level": "DEBUG"}},
}

# Internationalization
# https://docs.djangoproject.com/en/5.0/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/5.0/howto/static-files/

STATIC_URL = env.str("STATIC_URL", default="/static/")
STATIC_ROOT = env.str("STATIC_ROOT", default=BASE_DIR / "staticfiles")

WHITENOISE_USE_FINDERS = True
WHITENOISE_AUTOREFRESH = DEBUG


# Default primary key field type
# https://docs.djangoproject.com/en/5.0/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'


MEDIA_ROOT = env("MEDIA_ROOT", default=BASE_DIR / "media")
MEDIA_URL = env("MEDIA_PATH", default="/media/")

REDIS_URL = env("REDIS_URL", default=None)

Create a file project/celeryapp.py with the following content:

import os
from celery import Celery
from kombu import Queue
from .settings import REDIS_URL

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.settings")
app = Celery("project")
app.autodiscover_tasks()
app.conf.broker_url = REDIS_URL
app.conf.result_backend = REDIS_URL
app.conf.accept_content = ["application/json"]
app.conf.task_serializer = "json"
app.conf.result_serializer = "json"
app.conf.task_default_queue = "main"
app.conf.task_create_missing_queues = True
app.conf.task_queues = (Queue("main"),)
app.conf.broker_pool_limit = 1
app.conf.broker_connection_timeout = 30
app.conf.worker_prefetch_multiplier = 1
app.conf.redbeat_redis_url = REDIS_URL

This file configures Celery for your Django project:

Imports necessary modules
Loads environment variables
Initializes Celery app
Configures Celery settings (serialization, queues, timeouts, etc.)

Edit the file project/__init__.py:

This will make import Celery

from .celeryapp import app as celery_app

__all__ = [
    "celery_app",
]

Don't forget to save all files.

Apply migrations

Let's apply Django migrations and make sure our project and database connection works correctly.

In the Docker bash shell run this:

python manage.py migrate

By the way if you closed the shell or you want to run one command in Docker without launching shell you can do it another way:

docker compose run web python manage.py migrate

Start containers

This command will start all services defined in docker-compose.yml. Run this command outside docker compose shell.

docker compose up -d

The output of successful execution of this command should look this way:

Check logs of Django Project with docker compose logs

Now to make sure our services are actually working fine let's see logs.

docker compose logs -f

You will see logs for all services running. To stop following logs press CTRL-C.

To show logs for specific app you can specify process(es) to follow:

docker compose logs -f web celery

Here is the output for docker compose logs -f web:

Important notes:

If you make changes to the .env file, you'll need to restart Docker Compose for the changes to take effect. Do this by pressing CTRL-C to stop the current process(or docker compose stop), then run docker compose up again.
When modifying the docker-compose.yml file, you may need to perform a complete reset. Use docker compose stop to stop and remove all containers, networks created by Docker Compose. Keep in mind docker compose down will also remove all the DB data in the current setup. Then, restart the project with docker compose up -d.
From here it is implied that python commands are executed within a Docker Compose shell.

You can now see it in browser.

Please note, that if you are on Windows + WSL, http://0.0.0.0 address will not work, you should use http://localhost:9000

Start an application

In order to showcase Celery tasks, need an app where they will live, a model and some views.

python manage.py startapp mainapp

Create mainapp/tasks.py and leave it empty for now. That's where our Celery tasks will be defined.

Create a Report model in mainapp/models.py:

from django.db import models
from collections import namedtuple

REPORT_STATUSES = namedtuple("REPORT_STATUSES", "pending running finished error")._make(
    range(4)
)


class Report(models.Model):
    REPORT_STATUS_CHOICES = [
        (REPORT_STATUSES.pending, "Pending"),
        (REPORT_STATUSES.running, "Running"),
        (REPORT_STATUSES.finished, "Finished"),
        (REPORT_STATUSES.error, "Error"),
    ]
    dt_created = models.DateTimeField(auto_now_add=True)
    dt_started = models.DateTimeField(null=True, blank=True)
    dt_finished = models.DateTimeField(null=True, blank=True)
    complexity = models.PositiveIntegerField(default=10)
    status = models.IntegerField(
        choices=REPORT_STATUS_CHOICES, default=REPORT_STATUSES.pending
    )
    result = models.TextField(blank=True, null=True)

    def __str__(self):
        return f"{self.name} - {self.id}"

    class Meta:
        ordering = ["-dt_created"]
        verbose_name = "Report"
        verbose_name_plural = "Reports"

Let's create a page which we will use to trigger our tasks.

Create a view in mainapp/views.py:

from django.shortcuts import render

def index(request):
    return render(request, "mainapp/index.html")

Create a template in mainapp/templates/mainapp/index.html:

<html>
<head>
    <title>Tasks</title>
    <meta charset="UTF-8">
</head>

<body>
    <h1>Tasks</h1>
    <ul>
    </ul>
</body>
</html>

And add this view into URLConf in project/urls.py:

from django.contrib import admin
from django.urls import path
from mainapp.views import index
urlpatterns = [
    path('', index),
    path('admin/', admin.site.urls),
]

Add our application to INSTALLED_APPS in project/settings.py:


INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "mainapp", # new
]

Let's create and apply migrations in Docker:

If you are already in Docker shell:

python manage.py makemigrations
python manage.py migrate

and open the app in browser: https://localhost:9000/

Make the initial commit

git add .
git commit -m'Initial Commit'

Go to GitHub create a new repository.

When you create a repo you will be offered with a list of commands, copy the one that starts with git remote add origin and paste it in your terminal, outside of Docker container shell and run it.

Run git push -u origin master

Refresh the GitHub repository page to see that your code is there. You can also check that none of the unwanted files got added to version control.

Deploying Django Celery app

Now this step you might consider optional, but I insist it is very important.
You should deploy your app sooner rather than later.

By deploying app as early as possible, you will avoid the situation when you finished the whole project, it works on your machine, but doesn't work when deployed.

Deploying your app after you have finished the whole project means the task of debugging would be more time consuming and you might end up having to rewrite larger chunks of code.

By deploying the project early and often, you make it easier to find the cause of the problem in small most recent code changes.

Go to Appliku Dashboard

Add an Ubuntu or a Debian server from a cloud provider of your choice, if you haven't already.

My personal favorite is Hetzner, especially their ARM64 Ampere VPS servers. Their performance is great and pricing is unbeatable.

Create an application from your GitHub repository.

From your application dashboard page go to databases and add a PostgreSQL database and a Redis one.

Next step add processes to run your app. From Application overview page click Add Processes button. Or go to Settings -> Processes.

Click Add Process and 4 processes:

web: gunicorn project.wsgi --log-file -
release: python manage.py migrate
worker:celery -A project.celeryapp:app  worker -Q default -n celerytutorial.%%h --loglevel=INFO --max-memory-per-child=512000 --concurrency=1
beat: celery -A project.celeryapp:app beat -S redbeat.RedBeatScheduler  --loglevel=DEBUG --pidfile /tmp/celerybeat.pid

Keep in mind that the web and release processes' names are special. The process called web will receive HTTP traffic from the Nginx and the release one is a command that is executed after each successful deployment.

The rest of the processes can have any names as long as they consist of letters and numbers, no special characters.

Click on Save and Deploy and the first deployment will start.

After deployment has finished, click on Open App -> Domain name.

Our app has been successfully deployed!

Appliku gives the app a subdomain with your app's name and SSL certificate via Let's Encrypt. You can also add your own custom domain(s) in application settings.

That was easy, right?

Now let's get back to the code. Time to add the first Celery task.

Simple Celery task

Go to mainapp/tasks.py and make the first task.

from celery import shared_task
import logging
import time
logger = logging.getLogger(__name__)


@shared_task(name="dummy_and_slow")
def dummy_and_slow():
    time.sleep(2)
    logger.debug("Dummy and slow task has finished")

This task doesn't do anything useful, it waits for 2 seconds, which will be enough to illustrate the potential problem of a long running code.

Let's call it as a regular function from a view.

Go to mainapp/views.py and add another view.

from django.shortcuts import render
from .tasks import dummy_and_slow # new
import time # new

def index(request):
    return render(request, "mainapp/index.html")

def dummy_and_slow_view(request): # new
    start_time = time.time()
    dummy_and_slow()
    end_time = time.time()
    execution_time = end_time - start_time
    context = {"task_name": "Dummy and slow", "execution_time": round(execution_time, 2)}
    return render(request, "mainapp/generic.html", context=context)

Create a template templates/mainapp/generic.html:

<html>

<head>
    <title>Task results</title>
    <meta charset="UTF-8">
</head>
<body>
    <h1>Task {{task_name}} has finished</h1>
    <h2>Execution time: {{execution_time}}s</h2>
</body>
</html>

and add the view to project/urls.py:

from django.contrib import admin
from django.urls import path
from mainapp.views import index, dummy_and_slow_view # added a view
urlpatterns = [
    path('', index),
    path('dummy_and_slow_view', dummy_and_slow_view, name="dummy_and_slow_view"), # new
    path('admin/', admin.site.urls),
]

And update mainapp/templates/mainapp/index.html to include a link to our view:

<html>
<head>
    <title>Tasks</title>
    <meta charset="UTF-8">
</head>

<body>
    <h1>Tasks</h1>
    <ul>
        <li>
            <a href="{% url "dummy_and_slow_view" %}">Dummy and slow</a>
        </li>
    </ul>
</body>
</html>

Go to our app in the browser http://localhost:9000/

click on the new link, let's see the result:

It took more than 2 seconds for the page to render.
This is a very representative example of how long it might take to send a single email. Imagine, a person clicked a "Sign Up" button and waiting, watching the browser loading indicator to spin. That's suboptimal experience, let's make it better. Let's dispatch a Celery task.

The simplest way to do that is to go to our view and replace a direct function call dummy_and_slow() with calling dummy_and_slow.delay() method.

Now our view should look like this in mainapp/views.py:

def dummy_and_slow_view(request):
    start_time = time.time()
    dummy_and_slow.delay() # <-- this changed
    end_time = time.time()
    execution_time = end_time - start_time
    context = {"task_name": "Dummy and slow", "execution_time": round(execution_time, 2)}
    return render(request, "mainapp/generic.html", context=context)

Don't forget to stop the docker compose.
To do that run docker compose down, then docker compose up -d

Now when you open this URL http://localhost:9000/dummy_and_slow_view you will see this:

Execution time: 0.08s

You see that execution time of the view itself improved when the Celery task was sent, but and the long running operation is not blocking the request/re

You will see that the task was executed from the docker compose logs:

celery-1         | [2024-08-17 10:59:10,154: WARNING/ForkPoolWorker-1] Dummy and slow task has finished
celery-1         | [2024-08-17 10:59:10,156: INFO/ForkPoolWorker-1] Task dummy_and_slow[4c97e4ef-d73b-4623-8eec-b9fb5bea4753] succeeded in 2.002374355099164s: None

Celery API in more details

Celery Task definitions

There are two ways to define a task.

One way is to import the Celery app instance and use @app.task decorator.
Another way is using the @shared_task decorator, which is not tied to a particular instance of Celery.

The key difference lies in how these tasks are registered and how they behave when multiple Celery app instances are involved:

`@app.task` decorator:

Requires importing a Celery app instance
Ties task to specific app instance

from project.celeryapp import app

@app.task
def sometask():
    pass

@shared_task decorator(recommended):

Not tied to specific Celery instance
Automatically discovered by any Celery app

from celery import shared_task

@shared_task
def another_task():
    pass

The "not tied to a particular Celery instance" aspect of @shared_task means that it's more flexible and can be easily shared across different parts of a large project without needing to pass around Celery app instances.

For many simple Celery setups, this distinction may not be noticeable or important.

If you are an author of a library, you have to use the @shared_task decorator, because you can't have your code tied to a particular Celery instance.

So, since at the end of the day @shared_task doesn't have any downsides and usually there is only one Celery instance in a project I suggest to stick to the @shared_task decorator.

Like any python function your task can be with or without arguments:

@shared_task(name="fn1")
def fn1():
    pass

@shared_task(name="fn2")
def fn2(dishes, pancake, tasty=False):
    pass

Three ways to call a Celery task

Using .delay()
Using .apply_async()
Using current_app.send_task()

Sending a Celery task via .delay

This is the most simple way of sending a Celery task.

Instead of calling the function itself directly, you call its method .delay

Without arguments:

fn1.delay()

With arguments:

fn2.delay(1, 'good_one', something_else=true)

Sending a Celery task via .apply_async

This method allows you to pass additional parameters to the task itself as well as arguments to the task function.

Without arguments:

fn1.apply_async()

With positional arguments:

fn2.apply_async(args=(1, 'nice_one', true))

With keyword arguments:

fn2.apply_async(kwargs={"dishes": 1, "pancacke": "best one", "tasty": True})

In addition to arguments of the function itself we can also pass arguments that would affect the task execution.

fn2.apply_async(
    args=(1, 'nice_one', true), # passing arguments as positional
    kwargs={"dishes": 1, "pancacke": "best one", "tasty": True}, # passing keyword arguments
    expires=10, # in seconds, after this many seconds task received a worker will be rejected,
    countdown=5, # in seconds, execute task after N seconds, not before
    eta=datetime.now(timezone.utc) + timedelta(days=1), # datetime, execute task after this date and time
    queue='priority', # queue name where to send task
    ignore_result=False, # boolean, wether or not to store the result of the task in result backend,
    retry=False, # if task has failed, attempt retry or not
)

Those were probably the most important and typical parameters you might want to use while sending tasks.
Please do read the official documentation as there are a lot of caveats and detailed explanations for each of those arguments listed here and some more: Calling tasks

Sending tasks via current_app.send_task

send_task is similar to apply_async, but instead of importing a function, you use the name of the task as a string, the rest is similar to apply_async.

from celery import current_app

current_app.send_task(
    "fn2",
    kwargs={"dishes": 1, "pancacke": "best one", "tasty": True},
    queue="priority",
    expires=30,
)

Since you are not importing the function, but calling a task by name as a string you have to make sure you know the task name and it won't change when you rearrange the code later.

For this reason I recommend specifying the task name in the @shared_task decorator instead of relying on automatic task naming.

This way of calling Celery tasks is also my favorite because it allows to solve the circular import problem when you call a task from a Django model method and the task also imports this model.

The current_app.send_task approach also allows to call tasks that are not defined in the code of the current project.

That's what I mentioned in the intro, about having multiple projects interact with each other via the same celery instance.

You can't import the function, so you have to use send_task method and pass the name of the function as a string. That means you have to come up with some naming convention for tasks, have some prefixes for task names so they don't overlap between projects. For example, give them prefixes with the name of the service: main_ for the main part of the app, and a service that does video compression video_, service that does emails sending email_service_ prefix. This will also help with automatic routing of tasks based on their names.

I recommend having some prefix for the project from day one, because if you need to introduce another prefix later, you won't need to fix all the existing tasks name. Even if you won't need that in the future it is a good habit to have.

One important thing, about .send_task:
There is a Celery setting, called task_always_eager Celery Configuration and Defaults: task_always_eager
If this is True, all tasks will be executed locally by blocking until the task returns.

This setting will not affect tasks sent with .send_task, they will not be executed without a worker processing it.

How to correctly send a Celery task from a view

Usually, when your view is running, all interactions with the database are happening in a transaction.

This means that until the view finishes executing, the transaction is not committed and as such changes to the database won't be visible from the outside including your Celery task.

Django Docs: Database transactions

When you create or change a database object and immediately send a task with its ID, the task might start before the database update is complete.

This can cause errors because the task won't see the recent changes.

You might get an 'ObjectDoesNotExist' error if the task tries to access an object that hasn't been fully saved yet.

Let's see some examples.

# views.py
from celery import current_app

def welcome(request):
    user_object = User.objects.create(**somearguments)
    current_app.send_task("send_welcome_email", kwargs={"user_id": user_object.id})
    return render(request, template_name="mainapp/welcome.html")

# tasks.py
@shared_task(name="send_welcome_email")
def send_welcome_email(user_id):
    user_object = User.objects.get(pk=user_id)
    send_an_email(template='mainapp/email_templates/welcome.html', user_object=user_object)

This code looks fine at first, but the line user_object = User.objects.get(pk=user_id) will cause an ObjectDoesNotExist exception.

It will happen because the transaction will be committed after the view finishes, but our task has already been sent.

To fix that we will use transaction.on_commit to run a function that sends our task after transaction is committed.

# views.py
from celery import current_app
from functools import partial
from django.db import transaction

def welcome(request):
    user_object = User.objects.create(**somearguments)
    transaction.on_commit(partial(current_app.send_task, "send_welcome_email", kwargs={"user_id": user_object.id}))
    return render(request, template_name="mainapp/welcome.html")

Relevant Django Docs: Performing actions after commit

Workers setup

Let's review worker's command line and the arguments.

To run a worker the bare minimum is this command:

celery -A project.celeryapp:app worker

Here are important arguments you might want to use:

-Q: queues to consume tasks from
--concurrency: determines how many tasks a worker can process simultaneously
--loglevel: Determines the verbosity of the logging output. Possible values: DEBUG, INFO, ERROR, WARNING, CRITICAL.
-n, --hostname used to set a custom hostname for the worker. Example: -n celerytutorial.%%h
--max-memory-per-child This option sets a memory limit for each worker child process. When a worker reaches this memory threshold, it will be replaced with a new process after completing its current task.
--max-tasks-per-child limits based on the number of tasks processed.

Example App: Generate reports

Let's explore a common challenge in web applications: generating reports efficiently as your data grows.

Imagine you've built a feature to generate reports in your Django application. Initially, it works well, especially in development with minimal data. However, as you deploy the application and accumulate more data over time, you start to notice some issues:

Report generation time increases significantly.
The process blocks the web server, leading to slow response times.
Complex reports with multiple database queries and data manipulations become problematic.
What was once a simple view function now struggles to handle the increased load. This is where Celery, a distributed task queue, can help.

By moving the report generation process to a Celery task, we can:

Offload the heavy processing from the web server.
Allow the report to generate asynchronously in the background.
Improve the overall responsiveness of our application.
In this example, we'll simulate a time-consuming report generation process using time.sleep(). This simplification allows us to focus on the implementation of Celery tasks without getting bogged down in complex report logic.

Let's dive in and see how we can use Celery to handle our report generation more efficiently.

Go to mainapp/views.py and create 3 views: ReportCreateView, ReportDetailView and ReportListView

from django.views.generic.edit import CreateView
from django.views.generic.detail import DetailView
from django.views.generic.list import ListView
from django.urls import reverse  # Add this line
from .models import Report

class ReportCreateView(CreateView):
    model = Report
    fields = ["complexity", ]
    template_name = "mainapp/report_form.html"
    def get_success_url(self):
        return reverse("report_detail", kwargs={"pk": self.object.pk})

class ReportDetailView(DetailView):
    model = Report
    context_object_name = "report"
    template_name = "mainapp/report_detail.html"

class ReportListView(ListView):
    model = Report
    template_name = "mainapp/report_list.html"
    context_object_name = "reports"
    ordering = ["-dt_created"]
    paginate_by = 10

and our project/urls.py now looks like this when we import our views and add report related URLs:

from django.contrib import admin
from django.urls import path
from mainapp.views import (
    index,
    dummy_and_slow_view,
    ReportCreateView,
    ReportDetailView,
    ReportListView,
)

urlpatterns = [
    path("", index),
    path("dummy_and_slow_view", dummy_and_slow_view, name="dummy_and_slow_view"),
    path("report", ReportListView.as_view(), name="report_list"),
    path("report/create", ReportCreateView.as_view(), name="report_create"),
    path("report/detail/<int:pk>", ReportDetailView.as_view(), name="report_detail"),
]

Make 5 new templates. If you are not familiar with Django templates, I urge you to go read the official documentation: Django Docs: Templates

mainapp/templates/mainapp/report_base.html

<html>
<head>
    <title>{% block title %}{% endblock %}</title>
    <meta charset="UTF-8">
</head>

<body>
    {% block content %}{% endblock %}
</body>
</html>

mainapp/templates/mainapp/report_form.html

{% extends "mainapp/report_base.html" %}

{% block content %}
    <h1>Create a report</h1>
    <form method="post">
        {% csrf_token %}
        {{ form.as_p }}
        <button type="submit">Submit</button>
    </form>
{% endblock %}

mainapp/templates/mainapp/report_list.html

{% extends "mainapp/report_base.html" %}

{% block content %}
    <h1>Report List</h1>
    <a href="{% url "report_create" %}">Create a report</a>
    {% if reports.exists %}
    <ul>
        {% for report in reports %}
            <li>
                <a href="{% url "mainapp:report_detail" report.id %}">{{ report.title }}</a>
            </li>
        {% endfor %}
    </ul>
    {% else %}
        <p>No reports available</p>
    {% endif %}
{% endblock %}

mainapp/templates/mainapp/report_detail.html

{% extends "mainapp/report_base.html" %}

{% block content %}
    <h1>Report of complexity: {{report.complexity}}</h1>
    <p>Created at: {{ report.dt_created }}</p>
    {% if report.dt_started %}
        <p>Started at: {{ report.dt_started }}</p>
    {% endif %}
    {% if report.status == 2 %}
        <p>Report is ready</p>
        <p>Finished at: {{ report.finished_at }}</p>
    {% elif report.status == 0 %}
        <p>Report is waiting in queue</p>
    {% elif report.status == 1 %}
        <p>Report is generating</p>
    {% else %}
        <p>Task has failed to generate</p>
    {% endif %}
    {% if report.status == 0 or report.status == 1 %}
        <p>Refresh page in 5 seconds.</p>
        <script>
            setTimeout(function(){
                window.location.reload(1);
            }, 5000);
        </script>
    {% endif %}
    {% if report.result %}<p>{{ report.result|safe }}</p>{% endif %}
{% endblock %}

Now while the rest of templates are really simple, the last one needs some explaining:

The report is available in the report variable.

Statuses are defined in the nametuple REPORT_STATUSES.

They have indexes: pending is 0, running is 1 finished is 2 error is 3.

Maybe not the most convenient thing to work with in templates, but I wanted to show how you can use namedtuple.

If the report is ready - the result and finished time will be shown.

If the report has failed for whatever reason, it will state the error.

If the report is not yet ready it will be stated, but also a page will refresh for statuses 0 and 1 which is pending and running.

Pending status will be the one when the report is just created.

Running when a worker picked up the task and started working on it

Error is when the task has gracefully failed (meaning, reported the failure and saved that to DB).

Finished, after the task has finished and written the report into the DB.

Lastly, the template for report contents which we will use to generate to string and store in the report model field result:
mainapp/templates/mainapp/report_content.html

<table style="border:1px solid orange">
    <tr>
        <td>
            <h1>The sales report: {{report.complexity}}.</h1>
            <h2>Requested at: {{ report.dt_created }}.</h2>
            <p>It is a very insightful report for sales and marketing team.</p>
        </td>
    </tr>
</table>

Modify our model, we need to add a save method, a generate method and the generate_data method. The whole file mainapp/models.py will look this way:

from django.db import models
from collections import namedtuple
import time
from datetime import datetime
from django.db import transaction
from django.template.loader import render_to_string
from functools import partial
from celery import current_app

REPORT_STATUSES = namedtuple("REPORT_STATUSES", "pending running finished error")._make(
    range(4)
)


class Report(models.Model):
    REPORT_STATUS_CHOICES = [
        (REPORT_STATUSES.pending, "Pending"),
        (REPORT_STATUSES.running, "Running"),
        (REPORT_STATUSES.finished, "Finished"),
        (REPORT_STATUSES.error, "Error"),
    ]
    dt_created = models.DateTimeField(auto_now_add=True)
    dt_started = models.DateTimeField(null=True, blank=True)
    dt_finished = models.DateTimeField(null=True, blank=True)
    complexity = models.PositiveIntegerField(default=10)
    status = models.IntegerField(
        choices=REPORT_STATUS_CHOICES, default=REPORT_STATUSES.pending
    )
    result = models.TextField(blank=True, null=True)

    def __str__(self):
        return f"{self.name} - {self.id}"

    class Meta:
        ordering = ["-dt_created"]
        verbose_name = "Report"
        verbose_name_plural = "Reports"

    def save(self, *args, **kwargs):
        """This method is overridden to start the report generation task when the report is saved."""
        if self.status == REPORT_STATUSES.pending:
            self.dt_started = None
            self.dt_finished = None
        super().save(*args, **kwargs)
        if self.status == REPORT_STATUSES.pending:
            transaction.on_commit(
                partial(
                    current_app.send_task,
                    "mainapp_generate_report",
                    kwargs={"report_id": self.id},
                )
            )

    def generate_data(self):
        """This is where the actual generation of the report data would happen."""
        time.sleep(self.complexity)
        self.result = render_to_string("mainapp/report_content.html", {"report": self})

    def generate(self):
        """This method is called by the Celery task or Django admin action.
        Contains some boilerplate code to reflect of the progress and datetimes."""
        self.status = REPORT_STATUSES.running
        self.dt_started = datetime.now()
        self.save(update_fields=["status", "dt_started"])
        self.generate_data()
        self.status = REPORT_STATUSES.finished
        self.dt_finished = datetime.now()
        self.save(update_fields=["status", "dt_finished", "result"])

Add a Celery task to mainapp/tasks.py:

from .models import Report

@shared_task(name="mainapp_generate_report")
def generate_report(report_id):
    report = Report.objects.get(id=report_id)
    report.generate()

Let's break down what's happening:

save method:

- This is an overridden method of a Django model.
- When a report is saved with a 'pending' status, it resets the start and finish times.
- After saving, if the status is still 'pending', it schedules a Celery task.

transaction.on_commit:

- This ensures the task is only sent after the database transaction is committed.
- It prevents potential issues where the task tries to access a record that hasn't been committed yet.

current_app.send_task:

- This is a Celery method to send a task to the queue.
- The task name is "mainapp_generate_report".
- It passes the report's ID as an argument.

generate_data method:

- This method simulates the actual report generation.
- It uses `time.sleep` to mimic a time-consuming process.
- The report content is rendered using a template.

generate method:

- This is the main method of the Report model called by the Celery task.
- It updates the report status to 'running' and sets the start time.
- Calls `generate_data` to create the report content.
- Updates the status to 'finished' and sets the finish time.

Celery task:

- Has a name defined in the decorator `@shared_task(name="mainapp_generate_report")`
- It retrieves the report object and calls its `generate` method.
- The `generate_report` function is the task that Celery will execute.

When a report is saved with a 'pending' status, it triggers an asynchronous Celery task. This task runs the report generation process in the background, updating the report's status and timestamps along the way.

Let's update our main template to include the link to reports and then see how it works.

Edit mainapp/templates/mainapp/index.html to include a new link

<html>
<head>
    <title>Tasks</title>
    <meta charset="UTF-8">
</head>

<body>
    <h1>Tasks</h1>
    <ul>
        <li>
            <a href="{% url "dummy_and_slow_view" %}">Dummy and slow</a>
        </li>
        <li>
            <a href="{% url "report_list" %}">Reports</a>
        </li>
    </ul>
</body>
</html>

Restart your docker compose if it is running, CTRL-C to stop, docker compose up to start

In another terminal window make sure to run:

docker compose run web python manage.py makemigrations
docker compose run web python manage.py migrate

This creates and applies migrations.

If you are unfamiliar with this topic here is the relevant page in Django Docs: Migrations

Open our app in browser: http://localhost:9000/

Click on the "Reports" link.

Click on the "Create a report" link

The complexity is just a number of seconds the report will take to generate, leave it like that, it should be enough for the demonstration.

Click on the "Submit" button.

The page will say that report is in the queue, because our Celery task hasn't yet been started.

After 5 seconds page refreshes and the report is being generated.

5 seconds later, another refresh, the report is ready. The content of the report is in the orange box.

This is it!

Conclusion

Well done!
You've made it to the end of our Django and Celery tutorial.

Let's recap what we've learned:

We explored what Celery is and why it's such a powerful tool for handling background tasks.
You've discovered how to create tasks that can run independently of your main application.
We've delved into various ways to call these tasks, giving you flexibility in your implementations.
Together, we built an example project that generates time-consuming reports, with an auto-refreshing page that shows a waiting/in-progress screen and displays the final report when it's ready.

This tutorial has equipped you with a solid foundation, but there's so much more to discover.
Celery is an incredibly versatile tool and allows a lot of advanced ways to use it.

Good luck, happy coding and happy deploying!

Django REST Framework and DataTable Tutorial

Kostja Appliku.com — Thu, 09 Sep 2021 04:15:21 +0000

In this tutorial we are going to build API as a data source for a DataTable jQuery plugin.

Introduction

There is this awesome plugin I have used recently for displaying and sorting data https://datatables.net

It is very easy to start using.

You just make an HTML table, add a loop in Django template to display contents of table and then initialize this table with a JavaScript call.

<table id="myDataTable">
... table contents
</table>

<script>
$(document).ready( function () {
    $('#myDataTable').DataTable();
} );
</script>

Sounds fun and easy until you need to display a lot of data.

At the point where there are at least a thousand rows the size of the page will be huge, it will take a very long time and resources (CPU/RAM) to generate that page. Also page load time will be long killing user experience.

Thankfully, there is a solution to this problem.

ServerSide mode for DataTable.

From DataTables documentation:

Server-side processing

There are times when reading data from the DOM is simply too slow or unwieldy, particularly when dealing with many thousands or millions of data rows. To address this DataTables' server-side processing feature provides a method to let all the "heavy lifting" be done by a database engine on the server-side (they are after all highly optimised for exactly this use case!), and then have that information drawn in the user's web-browser. Consequently, you can display tables consisting of millions of rows with ease.

https://datatables.net/manual/server-side

In this article I want to show how to build the Django API with Django REST Framework that can be used as source for Datatables.

For this tutorial I will make a new repository from our Djangitos template and cloning it to my machine.

Go to Djangitos GitHub repository https://github.com/appliku/djangitos

Click button "Use this template"

Give a name to the new repository and click the "Create repository from template" button.

When new repository is ready, copy the path and use it to clone repo on your machine with git clone, in this case git clone git@github.com:appliku/tutorial_jquery_datatable_api.git

Switch to directory of the project with cd tutorial_jquery_datatable_api

Create an .env file with the following contents:

DATABASE_URL=postgresql://djangito:djangito@db/djangito
REDIS_URL=redis://redis/0
DJANGO_SECRET_KEY=123
DJANGO_DEBUG=True

It is needed in order to run our project with docker-compose.

Now you can open your editor or IDE, for pycharm on mac you can type open -a pycharm .

Now let's create an django application where we will put models, views and templates for this tutorial.

docker-compose run web python manage.py startapp datatable

This will create a directory in the root of our project datatable

Let's add the app to INSTALLED_APPS setting, so Django recognizes it.

Open djangito/settings.py and add 'datatable' to PROJECT_APPS.

Create models

I want to make this tutorial complex enough so there is chance to illustrate where can be performance issues and how to solve them and generally have a chance to talk about adjacent topics.

As an example we'll use an imaginary service company that does certain work for clients.

They need to track statuses of their work orders, what should be done and who are their clients.

Open datatable/models.py. Put these models in this file.


from django.db import models

from datatable.tuples import ORDER_STATUSES


class Client(models.Model):
    name = models.CharField(max_length=255)
    phone = models.CharField(max_length=255)
    email = models.EmailField()

    def __str__(self):
        return self.name

    class Meta:
        verbose_name = "Client"
        verbose_name_plural = "Clients"
        ordering = ('name',)


class Order(models.Model):
    STATUS_CHOICES = (
        (ORDER_STATUSES.proposal, 'Proposal'),
        (ORDER_STATUSES.in_progress, 'In Progress'),
        (ORDER_STATUSES.done, 'Done'),
        (ORDER_STATUSES.rejected, 'Rejected'),
    )
    name = models.CharField(max_length=255)
    client = models.ForeignKey(Client, on_delete=models.CASCADE)
    address = models.CharField(max_length=255)
    state = models.CharField(max_length=255)
    zip_code = models.CharField(max_length=10)
    status = models.IntegerField(choices=STATUS_CHOICES, default=ORDER_STATUSES.proposal)
    date_start = models.DateField()
    date_end = models.DateField()

    def __str__(self):
        return self.name

    class Meta:
        verbose_name = "Order"
        verbose_name_plural = "Orders"
        ordering = ('date_end',)


class OrderLine(models.Model):
    order = models.ForeignKey(Order, on_delete=models.CASCADE)
    name = models.CharField(max_length=255, )
    description = models.TextField()
    unit_price = models.DecimalField(max_digits=10, decimal_places=2)
    quantity = models.IntegerField()

    def __str__(self):
        return self.name

    class Meta:
        verbose_name = "Order Line"
        verbose_name_plural = "Order Lines"
        ordering = ('name',)

Edit datatable/admin.py to register our models in Django Admin:

from django.contrib import admin
from datatable.models import Order, OrderLine, Client

admin.site.register(Order)
admin.site.register(OrderLine)
admin.site.register(Client)

For statuses we'll use namedtuple.

Create a file datatable/tuples.py with this code:

from collections import namedtuple

ORDER_STATUSES = namedtuple('ORDER_STATUSES', 'proposal in_progress done rejected')._make(range(4))

Namedtuples are great for preventing errors and also provides code completion in IDE.

Now let's make migrations for these models. Run this command in the root of your project.

docker-compose run web python manage.py makemigrations

That's the output you should expect:

Now let's apply migrations, in order to do that, run the migrate management command:

docker-compose run web python manage.py migrate

Also we need a superuser. Let's create one.

Appliku Djangitos template comes with a simplified way to create superuser, the management command called makesuperuser.

docker-compose run web python manage.py makesuperuser

It will generate a super user with email/username admin@example.com and a random password.

Find the password in the output of this command, we'll need it in a few moments.

src/tutorial_jquery_datatable_api % docker-compose run web python manage.py makesuperuser
Creating tutorial_jquery_datatable_api_web_run ... done
Using selector: EpollSelector
admin user not found, creating one
===================================
A superuser was created with email admin@example.com and password NDTbnmPuyieX
===================================
admin@example.com
src/tutorial_jquery_datatable_api %

Let's start our project with this command:

docker-compose up

When you see this, then our app is running.

web_1       | Watching for file changes with StatReloader
web_1       | Watching for file changes with StatReloader
web_1       | Performing system checks...
web_1       |
web_1       | System check identified no issues (0 silenced).
web_1       | April 30, 2021 - 07:27:51
web_1       | Django version 3.1.6, using settings 'djangito.settings'
web_1       | Starting development server at http://0.0.0.0:8060/
web_1       | Quit the server with CONTROL-C.

Open the app in your browser at http://0.0.0.0:8060/admin/ and log in with admin@example.com and the password that was generated for you my makesuperuser command.

On the admin dashboard you can find our models.

You can go crazy now and create dozen orders with multiple line items so we have data to work with in the next steps.

I recommend creating several different clients so we can test sorting and search features of datatable.

Datatable with server rendered table

For the purpose of illustration of what Datatable can do and comparison later let's first create a page where datatable works with server rendered table.

Create a directory and a file datatable/templates/base.html where we include all the common structure and resources for our views.

<html lang="en">
<head>
    <title>DataTable</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/css/bootstrap.min.css" rel="stylesheet"
          integrity="sha384-eOJMYsd53ii+scO/bJGFsiCZc+5NDVN2yr8+0RDqr0Ql0h+rP48ckxlpbzKgwra6" crossorigin="anonymous">
    <link rel="stylesheet" href="//cdn.datatables.net/1.10.24/css/jquery.dataTables.min.css">
    {% block extra_head %}
    {% endblock %}
</head>

<body>
<div class="container mt-5">
    {% block content %}

    {% endblock %}
</div>
<script src="https://code.jquery.com/jquery-3.6.0.min.js"
        integrity="sha256-/xUj+3OJU5yExlq6GSYGSHk7tPXikynS7ogEvDej/m4=" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/js/bootstrap.bundle.min.js"
        integrity="sha384-JEW9xMcG8R+pH31jmWH6WWP0WintQrMb4s7ZOdauHnUtxwoG2vI5DkLtS3qm9Ekf"
        crossorigin="anonymous"></script>
<script src="//cdn.datatables.net/1.10.24/js/jquery.dataTables.min.js"></script>
{% block extra_js %}
{% endblock %}
</body>
</html>

Now let's make a template for our first view with static datatable. Let's call it datatable_static.html and full path will be datatable/template/datatable_static.html

{% extends "datatable/base.html" %}

{% block content %}
    <table id="myStaticDatatable">
        <thead>
        <tr>
            <th>ID</th>
            <th>ORDER</th>
            <th>CLIENT</th>
            <th>TOTAL</th>
            <th>STATUS</th>
        </tr>
        </thead>
        <tbody>
        {% for order in order_list %}
            <tr>
                <td>{{ order.id }}</td>
                <td>
                    {{ order.name }}
                    <br>
                    {{ order.address }} {{ order.state }} {{ order.zip_code }}
                </td>
                <td>
                    {{ order.client.name }}
                    <br>{{ order.client.phone }}
                    {{ order.client.email }}
                </td>
                <td>{{ order.amount }}</td>
                <td>{{ order.get_status_display }}</td>
            </tr>
        {% endfor %}
        </tbody>
    </table>
{% endblock %}


{% block extra_js %}
    <script>
        $(document).ready(function () {
            $('#myStaticDatatable').DataTable();
        });
    </script>
{% endblock %}

Open the file datatable/views.py, let's create our first view here.

from django.db.models import Sum, F, DecimalField
from django.shortcuts import render

from datatable.models import Order


def datatable_static(request, *args, **kwargs):
    orders_qs = Order.objects.all().select_related('client').annotate(
        amount=Sum(
            F('orderline__unit_price') * F('orderline__quantity'),
            output_field=DecimalField())
    )
    return render(
        request=request,
        template_name="datatable/datatable_static.html",
        context={
            "order_list": orders_qs
        })

Create datatable/urls.py file:

from django.urls import path

from datatable.views import datatable_static

urlpatterns = [
    path('static', datatable_static, name='datatable_static'),
]

Edit project's urls.py: djangito/urls.py. Add a line to include our datatable urls.

path('datatable/', include('datatable.urls')),

Now if we open our page at http://0.0.0.0:8060/datatable/static we'll see our table:

Let's summarise points you should pay attention to:

We made a base template that includes all resources and for our view we made template that extends the base one
We used .annotate() to calculate total amount of order on the database level. If we'd do it on python level it would require fetching all OrderLines and calculating them and it will be a massive performance hit.
Finally, we made an HTML table in our template, filled it with out data using for-loop and made it a datatable.

Now let's make it not static, but server-rendered via API.

Django REST Framework API for Datatable

To make our API we need another View, a line in urls.py and a serializer.

Create datatable/serializers.py.

We will create only one serializer, because we only need a flat object to display in datatable. We could use nested objects with datatable too, but I see no reason to make our code more complex.

from rest_framework import serializers

from datatable.models import Order


class OrderSerializer(serializers.ModelSerializer):
    amount = serializers.DecimalField(max_digits=10, decimal_places=2)
    client_name = serializers.ReadOnlyField(source='client.name')
    client_email = serializers.ReadOnlyField(source='client.email')
    client_phone = serializers.ReadOnlyField(source='client.phone')
    status = serializers.SerializerMethodField()

    class Meta:
        model = Order
        fields = (
            'id', 'name', 'address',
            'state', 'zip_code', 'status',
            'date_start', 'date_end',
            'client_name', 'client_phone', 'client_email', 'amount')

    def get_status(self, obj: Order):
        return obj.get_status_display()

Now add a new class based view to our datatable/views.py



class DataTableAPIView(ListAPIView):
    serializer_class = OrderSerializer

    def get_queryset(self):
        return Order.objects.all().select_related('client').annotate(
        amount=Sum(
            F('orderline__unit_price') * F('orderline__quantity'),
            output_field=DecimalField())
    )

    def filter_for_datatable(self, queryset):
        # filtering
        search_query = self.request.query_params.get('search[value]')
        if search_query:
            queryset = queryset.annotate(
                search=SearchVector(
                    'name',
                    'client__name',
                    'address', 'zip_code')
            ).filter(search=search_query)
        # ordering
        ordering_column = self.request.query_params.get('order[0][column]')
        ordering_direction = self.request.query_params.get('order[0][dir]')
        ordering = None
        if ordering_column == '0':
            ordering = 'id'
        if ordering_column == '1':
            ordering = 'name'
        if ordering and ordering_direction == 'desc':
            ordering = f"-{ordering}"
        if ordering:
            queryset = queryset.order_by(ordering)
        return queryset

    def list(self, request, *args, **kwargs):
        draw = request.query_params.get('draw')
        queryset = self.filter_queryset(self.get_queryset())
        recordsTotal = queryset.count()
        filtered_queryset = self.filter_for_datatable(queryset)
        try:
            start = int(request.query_params.get('start'))
        except ValueError:
            start = 0
        try:
            length = int(request.query_params.get('length'))
        except ValueError:
            length = 10
        end = length + start
        serializer = self.get_serializer(filtered_queryset[start:end], many=True)
        response = {
            'draw': draw,
            'recordsTotal': recordsTotal,
            'recordsFiltered': filtered_queryset.count(),
            'data': serializer.data
        }
        return Response(response)

Add 2 more items to datatable/urls.py:


    path('dynamic', TemplateView.as_view(template_name='datatable/datatable_dynamic.html'), name='datatable_dynamic'),
    path('data', DataTableAPIView.as_view(), name='datatable_data'),

dynamic refers to a generic TemplateView and data refers to our class based view.

Add the template for our dynamic table, datatable/templates/datatable/datatable_dynamic.html:

{% extends "datatable/base.html" %}

{% block content %}
    <table id="myStaticDatatable">
        <thead>
        <tr>
            <th>ID</th>
            <th>ORDER</th>
            <th>CLIENT</th>
            <th>TOTAL</th>
            <th>STATUS</th>
        </tr>
        </thead>
        <tbody>

        </tbody>
    </table>
{% endblock %}


{% block extra_js %}
    <script>
        let data_url = '{% url "datatable_data" %}';
        $(document).ready(function () {
            $('#myStaticDatatable').DataTable({
                'order': [[1, 'desc']],
                'processing': false,
                'serverSide': true,
                'ajax': {
                    url: data_url,
                    dataSrc: 'data'
                },
                columns: [
                    {
                        data: 'id',
                        orderable: true
                    },
                    {
                        data: null,
                        render: function (data, type, row) {
                            return `${row.name}<br>${row.address} ${row.state} ${row.zip_code}`;
                        },
                        orderable: true
                    },
                    {
                        data:null,
                        render: function (data, type, row){
                            return `${row.client_name}<br/>${row.client_phone}<br/>${row.client_email}`
                        },
                        orderable: false
                    },
                    {
                        data: 'amount',
                        orderable: false
                    },
                    {
                        data: 'status',
                        orderable: false
                    }

                ]
            });
        });
    </script>
{% endblock %}

Change from the static table template is that we removed data for-loop, added an URL to our API data_url and initialized the table with more configuration options.

Let's go over initialization of the datatable:

order is default ordering for the table, it will be the second column, descending order.
processing is disabled, I didn't want the "Processing" label to appear while table is loading. It just looks ugly.
serverSide is what makes datatable rely on server to load results according to sorting, filtering, page
ajax is an object that tells where our API resides. ajax.url is the API endpoint URL and data is object in endpoint response JSON that contains actual data
columns defines how to display data in columns from the JSON endpoint returns. The data attribute tells to use a field from JSON for response. render is a function to render the column cell and we use it to build a piece of HTML based on several fields of our JSON, data should be null in this case. orderable when enabled allows user to sort by this column.

Go to the page http://0.0.0.0:8060/datatable/dynamic and see the table that works the same way as before, but it sources data from API.

Full source of the project for this article can be found here: https://github.com/appliku/tutorial_jquery_datatable_api

Django Project Tutorial for beginners: Settings, Docker-Compose, Postgres and Redis

Kostja Appliku.com — Thu, 01 Apr 2021 17:25:34 +0000

This article is brought to you by the first deployment service, dedicated specifically to Python & Django Appliku.com.
Never manage a server again.
Deploy your Django app in 5 minutes.

In this article:

Django Tutorial Source Code
Create the environment for the project
Requirements for Django Project
.gitignore for Django Project
Dockerfile
docker-compose.yml
Explanation about Django and Docker
Django Settings
Django Custom User Model
Procfile for Django Project
Structure of Django Project
Push your Django Application to GitHub
Deploying Django Project
Application Processes Section
Config Variables Section
Heroku Config Vars Sync
Databases

Django Tutorial Source Code

You can find the project source code here: https://github.com/appliku/django_appliku_tutorial

Create the environment for the project

Let's make a directory in our home directory to hold our virtual environments for different projects.

Then we'll create an environment in it, activate it, install Django and then create our new project.

mkdir -p ~/envs
python3 -m venv ~/envs/tutorial
source ~/envs/tutorial/bin/activate
pip install -U pip
pip install Django

I prefer to keep all my code directories in ~/src directory. Let's make one if you don't have it and switch to it.

mkdir -p ~/src
cd ~/src/
django-admin startproject tutorial
cd tutorial

At this stage I usually open my favorite IDE: PyCharm.

open -a pycharm .

Now we need to create files in the root of our project.

requirements.txt will hold all dependencies our project needs.

.gitignore will tell git what files should not be added to the repository.

Requirements for Django Project

Open the create and open requirements.txt and put these lines in the file:

Django==3.1.7
Pillow==7.2.0
gunicorn==20.0.4
requests==2.25.1
django-redis==4.12.1
pytz==2021.1
psycopg2-binary==2.8.6
arrow==1.0.3
djangorestframework==3.12.2
djangorestframework-simplejwt==4.6.0
django-allauth==0.44.0
django-environ==0.4.5
django-storages==1.11.1
django-cors-headers==3.7.0
django-braces==1.14.0
django-extensions==3.1.1
django-post-office==3.5.3
django-crispy-forms==1.11.1
boto3==1.17.22
boto3-stubs==1.17.22.0
django-import-export==2.5.0
honeybadger==0.4.2
django-ses==1.0.3
djangoql==0.14.3
flake8==3.8.4
whitenoise==5.2.0

Couple words about these requirements and versions.

These are the packages I need in almost all of my projects, so I suggest including them in the tutorial project.

You may have question about old Pillow library. I had some issues where more fresh versions were incompatible with Django when uploading images and the only solution I found was downgrading to 7.2.0

.gitignore for Django Project

That's the most vital records for .gitignore file.

env/
venv/
.idea
.env

**/__pycache__/

.DS_Store

My OS is Mac so .DS_Store is the file from Finder that I don't want to get in the repository.

env and venv are typical names for environments that are created inside project directory. We don't have it now, but when you clone project on another machine or other developer will join the project, they would expect such names to be ignored.

.idea is directory that PyCharm creates to store project specific settings.

.env is local environment variables and we never should include it in repository.

__pycache__ is "compiled" bytecode version of your .py files. You interpreter may create it. Having them in repository is bad because they will cause problems when you try to run the app on even slightly different version of Python.

Dockerfile

We will run our project with Docker.

Why?

Because you really want reproducible environment and avoid messing up with host machine.

In order to do that, let's create 2 files.

First, Dockerfile:

FROM python:3.8
ENV PIP_NO_CACHE_DIR off
ENV PIP_DISABLE_PIP_VERSION_CHECK on
ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 0
ENV COLUMNS 80
RUN apt-get update \
 && apt-get install -y --force-yes \
 nano python-pip gettext chrpath libssl-dev libxft-dev \
 libfreetype6 libfreetype6-dev  libfontconfig1 libfontconfig1-dev\
  && rm -rf /var/lib/apt/lists/*
WORKDIR /code/
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/

docker-compose.yml

Second, docker-compose.yml:

version: '3.3'
services:
  redis:
    image: redis
    command: redis-server
    ports:
      - "14000:6379"
  db:
    image: postgres
    environment:
      - POSTGRES_USER=tutorial
      - POSTGRES_PASSWORD=tutorial
      - POSTGRES_DB=tutorial
    ports:
      - "127.0.0.1:21003:5432"
  web:
    build: .
    restart: always
    command: python manage.py runserver 0.0.0.0:8600
    env_file:
      - .env
    ports:
      - "127.0.0.1:8600:8600"
    volumes:
      - .:/code
    links:
      - db
      - redis
    depends_on:
      - db
      - redis

Explanation about Django and Docker

Dockerfile defines the state of the OS and filesystem where your application will be executing. That's oversimplified and completely non-nerdy explanation :)

With instruction from Dockerfile docker builds images.

If Dockerfile is a set of instructions, image is the actual archive with files that can be used to execute apps in containers.

For developer environment we need Django development server to run with a certain command and we need a postgres DB and a Redis instance.

To define it with a code we will use docker-compose.yml file.

docker-compose.yml defines services it will run.

A service in docker-compose.yml is defined primarily by the image it uses, a command to execute, ports to expose, volumes to mount and environment variables to set.

Again, it is a very high-level explanation with my own words and attempt to explain with as simple words as possible.

Let's talk about what we defined in our docker-compose.yml.

We have a postgres service. When it is first initialized it will have a user, password and a database "tutorial".

It will be available for other service on the DNS name db on the port 5432. And it will be available for the host machine on port 21003.

It uses "postgres" image, that it will pull from Docker Hub.

There is no volume for the database defined, so if you kill the DB then you will loose the data.

Next service is for our Django development server.

Instead of image, we specify folder where to build image from Dockerfile, which in this case is current directory (.).

If it fails, we want it to be always restarting.

We specify what command to use to run our dev server.

Environment variables will be taken from .env file that we'll create later.

We expose port 8600 on 127.0.0.1, so it will be accessable only from local machine. Keep in mind that if you want to change port you should also update the port in the command.

The volumes section tells what directories will be mounted from the host machine inside the container. We want the current directory to be mounted on /code where our app is running. See WORKDIR in our Dockerfile.
Since it is the development environment, we want changes to our code reflected in the container so dev server would reload automatically.

links section will make resolution of DNS name db possible inside container. In other words django dev server will be able to connect to db. Same for redis.

depends_on section lists services that must be started before starting the web service. In this case redis and db will be started first, then web will be started.

Last step, let's create the .env file in the root of the project.

DATABASE_URL=postgresql://tutorial:tutorial@db/tutorial
REDIS_URL=redis://redis/0
DJANGO_SECRET_KEY=supersecret123!
DJANGO_DEBUG=True

Here in form of special URLs we pass to our django project credentials for database and redis instance, secret_key Django should use and enable debug mode for Django.

None of it affects our app yet. But it will be very important in a little bit.

We need to test that our Docker image can be built and docker-compose has no errors.

For now, let's just tell it to build our image.

Run this command:

docker-compose build

In case of success, last lines of the output should be roughly like these:

Removing intermediate container 757d0bd934ca
 ---> b4bba357f84c
Step 11/11 : COPY . /code/
 ---> fa5d799d8fc1

Successfully built fa5d799d8fc1
Successfully tagged tutorial_web:latest

Great job!

Time to work on settings.

Django Settings

We want our apps to be scalable, work under a lot of traffic and handle growth.

In order to do that we need to build our app so it allows scaling.

In this case it is important that our app follows rules of The 12-factor app: https://12factor.net

Let's list the key points here:

Codebase – One codebase tracked in revision control, many deploys
Dependencies – Explicitly declare and isolate dependencies
Config – Store config in the environment
Backing services – Treat backing services as attached resources
Build, release, run – Strictly separate build and run stages
Processes – Execute the app as one or more stateless processes
Port binding – Export services via port binding
Concurrency – Scale out via the process model
Disposability – Maximize robustness with fast startup and graceful shutdown
Dev/prod parity – Keep development, staging, and production as similar as possible
Logs – Treat logs as event streams
Admin processes – Run admin/management tasks as one-off processes

I strongly recommend to read all pages on that site.

With that in mind let's open our settings file: tutorial/settings.py

We remove everything from there and start building our own from scratch.

I will explain every code block then put the show the whole file so you can just copy it to your project.

First we import a couple of libraries, set our root path of the project and create an env variable.

from pathlib import Path
import environ
import os

BASE_DIR = Path(__file__).resolve(strict=True).parent.parent
env = environ.Env()

BASE_DIR we need to set proper paths to places in our project.

env will help us properly get configuration from environment variables.

DEBUG = env.bool("DJANGO_DEBUG", False)

DEBUG should be off at all times except local development environment.

# Allowed Hosts Definition
if DEBUG:
    # If Debug is True, allow all.
    ALLOWED_HOSTS = ['*']
else:
    ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS', default=['example.com'])

If DEBUG is True, then we should allow all HOSTS when opening the app.

SECRET_KEY = env('DJANGO_SECRET_KEY')

Secret Key is used to encrypt/sign cookies, passwords, etc. You must keep it safe and out of the version control.

"""
Project Apps Definitions
Django Apps - Django Internal Apps
Third Party Apps - Apps installed via requirements.txt
Project Apps - Project owned / created apps

Installed Apps = Django Apps + Third Part apps + Projects Apps
"""
DJANGO_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.sites',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.redirects',
]

THIRD_PARTY_APPS = [
    'django_extensions',
    'rest_framework',
    'storages',
    'corsheaders',
    'djangoql',
    'post_office',
    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    'allauth.socialaccount.providers.google',
    'crispy_forms',
]

PROJECT_APPS = [
    'usermodel',
]

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + PROJECT_APPS

It is very convenient to differentiate where a certain app comes from, that's why we separate built-in Django apps, third party apps and project's apps.

MIDDLEWARE = [
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.redirects.middleware.RedirectFallbackMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

# Databases
DATABASES = {
    "default": env.db("DATABASE_URL")
}
DATABASES["default"]["ATOMIC_REQUESTS"] = True
DATABASES["default"]["CONN_MAX_AGE"] = env.int("CONN_MAX_AGE", default=60)

Our app will receive DATABASE_URL from environment variables in the form of the URL like postgres://username:password@database-host.com:1234/databasename.


ROOT_URLCONF = 'tutorial.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'tutorial.wsgi.application'

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]
AUTHENTICATION_BACKENDS = [
    'django.contrib.auth.backends.ModelBackend',
]

# User Model Definition
AUTH_USER_MODEL = 'usermodel.User'

For every new Django project, don't forget to create a custom user model, otherwise you effectively will not be able to change it later. We will discuss it a bit later.

TIME_ZONE = 'UTC'
LANGUAGE_CODE = 'en-us'
SITE_ID = 1
USE_I18N = True
USE_L10N = True
USE_TZ = True

These settings are pretty static, but if you want to learn more I recommend to read official docs: https://docs.djangoproject.com/en/3.1/ref/settings/

# Admin URL Definition
ADMIN_URL = env('DJANGO_ADMIN_URL', default='admin/')

Admin, never have admin on the default URL. With DJANGO_ADMIN_URL env variable you will be able to set it different for every environment: production, staging, but leave default for local development.


# Redis Settings
REDIS_URL = env('REDIS_URL', default=None)

if REDIS_URL:
    CACHES = {
        "default": env.cache('REDIS_URL')
    }

REDIS is primarily used for caching end temporary data storage.

In this case we make it optional, and if REDIS_URL is defined then we enable the default cache.

SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

This is how we tell Django to detected that app is working behind SSL proxy.

In our nginx server definition we must set X-Forwarded-Proto header.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler'
        },
    },
    'loggers': {
        '': {  # 'catch all' loggers by referencing it with the empty string
            'handlers': ['console'],
            'level': 'DEBUG',
        },
    },
}

That's a very simple logging, that should output everything to console from all modules with level of DEBUG, which means output everything it can.

# Static And Media Settings
AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME', default=None)
if AWS_STORAGE_BUCKET_NAME:
    AWS_DEFAULT_ACL = None
    AWS_QUERYSTRING_AUTH = False
    AWS_S3_CUSTOM_DOMAIN = env('AWS_S3_CUSTOM_DOMAIN', default=None) or f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=600'}

    # s3 static settings
    STATIC_LOCATION = 'static'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'tutorial.storages.StaticStorage'

    # s3 public media settings
    PUBLIC_MEDIA_LOCATION = 'media'
    MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'tutorial.storages.PublicMediaStorage'
else:
    MIDDLEWARE.insert(2, 'whitenoise.middleware.WhiteNoiseMiddleware')
    STATICFILES_STORAGE = 'whitenoise.storage.CompressedStaticFilesStorage'
    WHITENOISE_USE_FINDERS = True
    STATIC_HOST = env('DJANGO_STATIC_HOST', default='')
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
    STATIC_URL = STATIC_HOST + '/static/'
    if DEBUG:
        WHITENOISE_AUTOREFRESH = True

This is the config for working with static and media files. It would upload static with python manage.py collectstatic --noinput to S3 and all media files will be stored in S3 when uploaded by user or app itself.

If the the environment variable AWS_STORAGE_BUCKET_NAME is not present, this part of config will not be enabled, which should be the case for local development.

If AWS_STORAGE_BUCKET_NAME is not set, Django will use whitenoise to server static files.

DEFAULT_FROM_EMAIL = env('DEFAULT_FROM_EMAIL', default='test@example.com')

Our app probably will send emails at some point, at least password resets. We should specify what will be the default sender. Again, as env var.

Let's get to third party apps configuration.

First of all, we need to know about any error that will happen in production.

We will set our app to report to the error tracking service https://HoneyBadger.io.

Again, it will be only enabled if the env var HONEYBARDGER_API_KEY is set. You can get this env var from the project settings in HoneyBadger.

Now let's configure Celery to run our background tasks.

It is optional. If celery is not installed, or env var CELERY_BROKER_URL is not defined, then it is not enabled.

# Celery Settings
try:
    from kombu import Queue
    from celery import Celery
    CELERY_BROKER_URL = env('CELERY_BROKER_URL', default='amqp://localhost')
    if CELERY_BROKER_URL:
        CELERYD_TASK_SOFT_TIME_LIMIT = 60
        CELERY_ACCEPT_CONTENT = ['application/json']
        CELERY_TASK_SERIALIZER = 'json'
        CELERY_RESULT_SERIALIZER = 'json'
        CELERY_RESULT_BACKEND = env('REDIS_URL', default='redis://localhost:6379/0')
        CELERY_DEFAULT_QUEUE = 'default'
        CELERY_QUEUES = (
            Queue('default'),
        )
        CELERY_CREATE_MISSING_QUEUES = True
except ModuleNotFoundError:
    print("Celery/kombu not installed. Skipping...")

Now let's configure Django-AllAuth in order to have option to register/login via social logins. Let's have Google as the only provider for this tutorial.


# AllAuth Settings
AUTHENTICATION_BACKENDS += [
    # `allauth` specific authentication methods, such as login by e-mail
    'allauth.account.auth_backends.AuthenticationBackend',
]

ACCOUNT_USER_MODEL_USERNAME_FIELD = None
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_USERNAME_REQUIRED = False
ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_UNIQUE_EMAIL = True
ACCOUNT_EMAIL_VERIFICATION = 'mandatory'
ACCOUNT_FORMS = {'signup': 'usermodel.forms.MyCustomSignupForm'}
ACCOUNT_MAX_EMAIL_ADDRESSES = 2
SOCIALACCOUNT_PROVIDERS = {

}
SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID = env('SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID', default=None)
if SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID:
    SOCIALACCOUNT_PROVIDERS['google'] = {
        'SCOPE': [
            'profile',
            'email',
        ],
        'AUTH_PARAMS': {
            'access_type': 'online',
        },
        # For each OAuth based provider, either add a ``SocialApp``
        # (``socialaccount`` app) containing the required client
        # credentials, or list them here:
        'APP': {
            'client_id': SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID,
            'secret': env('SOCIALACCOUNT_PROVIDERS_GOOGLE_SECRET'),
        }
    }

In order for it to work we need to obtain CLIENT_ID and CLIENT_SECRET from Google Cloud services.

As before, I prefer to have it optional. So if env var SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID is present then we add Google to the list of providers.

# Crispy Forms Settings
CRISPY_TEMPLATE_PACK = 'bootstrap4'

The crispy-forms package help our forms look good. In this case with bootstrap4 setting it will render our forms according to bootstrap 4 HTML structure.

And let's wrap up with the email sending part.

I love two libraries that help with sending and managing outgoing emails.

One of them is django-post-office which gives users email templates, scheduling, prioritization and stores outgoing emails, with logs and statuses so you can conveniently debug on apps side if email went out, what exactly was sent and if it wasn't - there will be logs attached to each email.

Second library is django-ses, for sending emails via AWS Simple Email Service(SES).

# Django Post Office Settings
EMAIL_BACKEND = 'post_office.EmailBackend'

POST_OFFICE = {
    'BACKENDS': {
        'default': 'django_ses.SESBackend',
    },
    'DEFAULT_PRIORITY': 'now',
}

# AWS SES Settings
AWS_SES_REGION_NAME = env('AWS_SES_REGION_NAME', default='us-east-1')
AWS_SES_REGION_ENDPOINT = env('AWS_SES_REGION_ENDPOINT', default='email.us-east-1.amazonaws.com')
AWS_SES_CONFIGURATION_SET = env('AWS_SES_CONFIGURATION_SET', default=None)

It's worth some explanation.

When you want to send email via SES, you need to request sending capacity from AWS in a specific region. Before that your account in that region is in email sandbox and can only send emails to yourself, a.k.a. validated email address.

When AWS allows you send email via that region, you must make app aware of it via settings above.

AWS_SES_CONFIGURATION_SET setting is needed if you have configured AWS CloudWatch to track opens, clicks, and so on. Leave it empty if you haven't.

This wraps up working on our tutorial/settings.py and here is the full file for you to copy:

from pathlib import Path
import environ
import os

env = environ.Env()

"""
Project Settings
"""

BASE_DIR = Path(__file__).resolve(strict=True).parent.parent

DEBUG = env.bool('DJANGO_DEBUG', default=False)

# Allowed Hosts Definition
if DEBUG:
    # If Debug is True, allow all.
    ALLOWED_HOSTS = ['*']
else:
    ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS', default=['example.com'])

SECRET_KEY = env('DJANGO_SECRET_KEY')

"""
Project Apps Definitions
Django Apps - Django Internal Apps
Third Party Apps - Apps installed via requirements.txt
Project Apps - Project owned / created apps

Installed Apps = Django Apps + Third Part apps + Projects Apps
"""
DJANGO_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.sites',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.redirects',
]

THIRD_PARTY_APPS = [
    'import_export',
    'django_extensions',
    'rest_framework',
    'storages',
    'corsheaders',
    'djangoql',
    'post_office',
    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    'allauth.socialaccount.providers.google',
    'crispy_forms',
]

PROJECT_APPS = [
    'usermodel',
]

INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + PROJECT_APPS

MIDDLEWARE = [
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    'django.contrib.redirects.middleware.RedirectFallbackMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

# Databases
DATABASES = {
    "default": env.db("DATABASE_URL")
}
DATABASES["default"]["ATOMIC_REQUESTS"] = True
DATABASES["default"]["CONN_MAX_AGE"] = env.int("CONN_MAX_AGE", default=60)

ROOT_URLCONF = 'tutorial.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'tutorial.wsgi.application'

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]
AUTHENTICATION_BACKENDS = [
    'django.contrib.auth.backends.ModelBackend',
]
# User Model Definition
AUTH_USER_MODEL = 'usermodel.User'

TIME_ZONE = 'UTC'
LANGUAGE_CODE = 'en-us'
SITE_ID = 1
USE_I18N = True
USE_L10N = True
USE_TZ = True

# Admin URL Definition
ADMIN_URL = env('DJANGO_ADMIN_URL', default='admin/')

# Redis Settings
REDIS_URL = env('REDIS_URL', default=None)

if REDIS_URL:
    CACHES = {
        "default": env.cache('REDIS_URL')
    }
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler'
        },
    },
    'loggers': {
        '': {  # 'catch all' loggers by referencing it with the empty string
            'handlers': ['console'],
            'level': 'DEBUG',
        },
    },
}


# Static And Media Settings
AWS_STORAGE_BUCKET_NAME = env('AWS_STORAGE_BUCKET_NAME', default=None)
if AWS_STORAGE_BUCKET_NAME:
    AWS_DEFAULT_ACL = None
    AWS_QUERYSTRING_AUTH = False
    AWS_S3_CUSTOM_DOMAIN = env('AWS_S3_CUSTOM_DOMAIN', default=None) or f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
    AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=600'}

    # s3 static settings
    STATIC_LOCATION = 'static'
    STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{STATIC_LOCATION}/'
    STATICFILES_STORAGE = 'tutorial.storages.StaticStorage'

    # s3 public media settings
    PUBLIC_MEDIA_LOCATION = 'media'
    MEDIA_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{PUBLIC_MEDIA_LOCATION}/'
    DEFAULT_FILE_STORAGE = 'tutorial.storages.PublicMediaStorage'

    STATICFILES_DIRS = (
        # os.path.join(BASE_DIR, "static"),
    )
else:
    MIDDLEWARE.insert(2, 'whitenoise.middleware.WhiteNoiseMiddleware')
    STATICFILES_STORAGE = 'whitenoise.storage.CompressedStaticFilesStorage'
    WHITENOISE_USE_FINDERS = True
    STATIC_HOST = env('DJANGO_STATIC_HOST', default='')
    STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
    STATIC_URL = STATIC_HOST + '/static/'
    if DEBUG:
        WHITENOISE_AUTOREFRESH = True
DEFAULT_FROM_EMAIL = env('DEFAULT_FROM_EMAIL', default='test@example.com')

"""
Third Party Settings
"""

# Honeybadger Settings
HONEYBADGER_API_KEY = env('HONEYBADGER_API_KEY', default=None)
if HONEYBADGER_API_KEY:
    MIDDLEWARE = ['honeybadger.contrib.DjangoHoneybadgerMiddleware'] + MIDDLEWARE
    HONEYBADGER = {
        'API_KEY': HONEYBADGER_API_KEY
    }

# Celery Settings
try:
    from kombu import Queue
    from celery import Celery
    CELERY_BROKER_URL = env('CELERY_BROKER_URL', default='amqp://localhost')
    if CELERY_BROKER_URL:
        CELERYD_TASK_SOFT_TIME_LIMIT = 60
        CELERY_ACCEPT_CONTENT = ['application/json']
        CELERY_TASK_SERIALIZER = 'json'
        CELERY_RESULT_SERIALIZER = 'json'
        CELERY_RESULT_BACKEND = env('REDIS_URL', default='redis://localhost:6379/0')
        CELERY_DEFAULT_QUEUE = 'default'
        CELERY_QUEUES = (
            Queue('default'),
        )
        CELERY_CREATE_MISSING_QUEUES = True
except ModuleNotFoundError:
    print("Celery/kombu not installed. Skipping...")

# AllAuth Settings
AUTHENTICATION_BACKENDS += [
    # `allauth` specific authentication methods, such as login by e-mail
    'allauth.account.auth_backends.AuthenticationBackend',
]

ACCOUNT_USER_MODEL_USERNAME_FIELD = None
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_USERNAME_REQUIRED = False
ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_UNIQUE_EMAIL = True
ACCOUNT_EMAIL_VERIFICATION = 'mandatory'
ACCOUNT_FORMS = {'signup': 'usermodel.forms.MyCustomSignupForm'}
ACCOUNT_MAX_EMAIL_ADDRESSES = 2
SOCIALACCOUNT_PROVIDERS = {

}
SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID = env('SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID', default=None)
if SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID:
    SOCIALACCOUNT_PROVIDERS['google'] = {
        'SCOPE': [
            'profile',
            'email',
        ],
        'AUTH_PARAMS': {
            'access_type': 'online',
        },
        # For each OAuth based provider, either add a ``SocialApp``
        # (``socialaccount`` app) containing the required client
        # credentials, or list them here:
        'APP': {
            'client_id': SOCIALACCOUNT_PROVIDERS_GOOGLE_CLIENT_ID,
            'secret': env('SOCIALACCOUNT_PROVIDERS_GOOGLE_SECRET'),
        }
    }
# Crispy Forms Settings
CRISPY_TEMPLATE_PACK = 'bootstrap4'

# Django Post Office Settings
EMAIL_BACKEND = 'post_office.EmailBackend'

POST_OFFICE = {
    'BACKENDS': {
        'default': 'django_ses.SESBackend',
    },
    'DEFAULT_PRIORITY': 'now',
}

# AWS SES Settings

AWS_SES_REGION_NAME = env('AWS_SES_REGION_NAME', default='us-east-1')
AWS_SES_REGION_ENDPOINT = env('AWS_SES_REGION_ENDPOINT', default='email.us-east-1.amazonaws.com')
AWS_SES_CONFIGURATION_SET = env('AWS_SES_CONFIGURATION_SET', default=None)

On more step, create a file next to settings.py call it storages.py

from storages.backends.s3boto3 import S3Boto3Storage


class PublicMediaStorage(S3Boto3Storage):
    location = 'media'
    default_acl = 'public-read'
    file_overwrite = False


class StaticStorage(S3Boto3Storage):
    location = 'static'
    default_acl = 'public-read'

This is needed for our media and static storage to work, as we refer these classes from settings.py.

Django Custom User Model

Every Django project that employes Users and authentication should define a custom user model. Even if you keep it same as the stock one, it will give you ability to modify it later to better fit your project's requirements.

Changing user model mid-project is very tricky task since you will have other models instances refering existing Users and all this you will have to migrate.

With that said, in root of the project create a folder usermodel and an empty file in it __init__.py.

You could achieve something similar by running python manage.py startapp usermodel, but i just wanted to have a chance to talk about what makes a directory a python module.

To be able to import a module or anything from it, a directory needs to have __init__.py in it.

Now create usermodel/models.py.

This we should put in the file:

import uuid

from django.contrib.auth.base_user import AbstractBaseUser
from django.contrib.auth.models import PermissionsMixin
from django.contrib.postgres.fields import CIEmailField
from django.core.mail import send_mail
from django.db import models
from django.utils.translation import gettext_lazy as _
from django.utils import timezone

from usermodel.managers import UserManager


class User(AbstractBaseUser, PermissionsMixin):
    """
    An abstract base class implementing a fully featured User model with
    admin-compliant permissions.
    Username and password are required. Other fields are optional.
    """
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    email = CIEmailField(
        _('Email Address'),
        unique=True,
        error_messages={
            'unique': _("A user with that username already exists."),
        },
    )

    first_name = models.CharField(_('First Name'), max_length=255, blank=True)
    last_name = models.CharField(_('Last Name'), max_length=255, blank=True)

    is_staff = models.BooleanField(
        _('Staff Status'),
        default=False,
        help_text=_('Designates whether the user can log into this admin site.'),
    )

    is_active = models.BooleanField(
        _('Active'),
        default=True,
        help_text=_(
            'Designates whether this user should be treated as active. '
            'Unselect this instead of deleting accounts.'
        ),
    )

    # Audit Values
    is_email_confirmed = models.BooleanField(
        _('Email Confirmed'),
        default=False
    )
    date_joined = models.DateTimeField(
        _('Date Joined'),
        default=timezone.now
    )

    objects = UserManager()

    EMAIL_FIELD = 'email'
    USERNAME_FIELD = 'email'
    REQUIRED_FIELDS = [
        'first_name',
        'last_name'
    ]

    class Meta:
        verbose_name = _('User')
        verbose_name_plural = _('Users')

    def clean(self):
        super().clean()
        self.email = self.__class__.objects.normalize_email(self.email)

    def get_full_name(self):
        """
        Return the first_name plus the last_name, with a space in between.
        """
        return f"{self.first_name} {self.last_name}"

    def get_short_name(self):
        """Return the short name for the user."""
        return self.first_name

    def email_user(self, subject, message, from_email=None, **kwargs):
        """Send an email to this user."""
        send_mail(subject, message, from_email, [self.email], **kwargs)

We use email as login, UUID as a primary key.

General benefits of UUID vs integer primary key:

Nobody can estimate number of users/orders/payments by looking at the freshest ID they got when placing another order or creating any other type of object.
you can't bruteforce your way into seeing other objects, so it adds kind of layer of "security". Even if some links to objects are public, but intended only for those who has the link – they will be hard to access without having exact ID.
and now the other problem that will be nice to have: biggest integer number seems to big a big number, 2,147,483,647. But when you reach it, then you can't add any other object. Integer is the default type of primary key. When you run out of numbers, database will refuse any new records. You might think, okay I will just change the field type to be a bigint, but imagine how much time it will take to apply migration to such huge table of 2 billion records? You can as well take a vacation during that time, that you probably need while building such a big project. 😂 UUID will not have such problem.

Now, in our model we referenced a custom manager.

objects = UserManager()

Let's create a file usermodel/managers.py and put this code in it:

from django.contrib.auth.base_user import BaseUserManager


class UserManager(BaseUserManager):
    use_in_migrations = True

    def _create_user(self, email, password, **extra_fields):
        """
        Create and save a user with the given username, email, and password.
        """
        if not email:
            raise ValueError('The given username must be set')
        email = self.normalize_email(email)
        user = self.model(email=email, **extra_fields)
        user.set_password(password)
        user.save(using=self._db)
        return user

    def create_user(self, email=None, password=None, **extra_fields):
        extra_fields.setdefault('is_staff', False)
        extra_fields.setdefault('is_superuser', False)
        return self._create_user(email, password, **extra_fields)

    def create_superuser(self, email, password, **extra_fields):
        extra_fields.setdefault('is_staff', True)
        extra_fields.setdefault('is_superuser', True)

        if extra_fields.get('is_staff') is not True:
            raise ValueError('Superuser must have is_staff=True.')
        if extra_fields.get('is_superuser') is not True:
            raise ValueError('Superuser must have is_superuser=True.')
        return self._create_user(email, password, **extra_fields)

And usermodel/admin.py:

from django.contrib import admin
from django.utils.translation import gettext_lazy as _

from usermodel.models import User
from django.contrib.auth.admin import UserAdmin as DefaultUserAdmin


@admin.register(User)
class UserAdmin(DefaultUserAdmin):
    fieldsets = (
        (
            None,
            {
                'fields': (
                    'email', 'password'
                )
            }
        ),
        (
            _('Permissions'),
            {
                'fields': (
                    'is_active',
                    'is_staff',
                    'is_superuser',
                    'groups',
                    'user_permissions',
                ),
            }
        ),
        (
            _('Important dates'),
            {
                'fields': (
                    'last_login',
                    'date_joined',
                )
            }
        ),
        (
            _('User data'),
            {
                'fields': (
                    ('is_email_confirmed',),
                )
            }
        ),
    )
    add_fieldsets = (
        (
            None,
            {
                'classes': ('wide',),
                'fields': ('email', 'password1', 'password2'),
            }
        ),
    )
    list_display = ('email', 'first_name', 'last_name', 'is_staff')
    search_fields = ('first_name', 'last_name', 'email')
    ordering = ('email',)

And cherry on top will be a non-interactive management command to create a superuser.

While we have a way to create a super user, it is not convenient for non-interactive cases like release phase.

What I chose to have in all my projects is this script, which checks if there is a superuser in database and if not - creates one with a random password.

Create these directories and files inside usermodel directory:

├── management
│   ├── __init__.py
│   └── commands
│       ├── __init__.py
│       └── makesuperuser.py

__init__.py should be empty, and here is the code for usermodel/management/makesuperuser.py:

from django.contrib.auth import get_user_model
from django.core.management.base import BaseCommand
from django.utils.crypto import get_random_string

User = get_user_model()


class Command(BaseCommand):
    def handle(self, *args, **options):
        try:
            u = None
            if not User.objects.filter(email='admin@example.com').exists() and not User.objects.filter(
                    is_superuser=True).exists():
                print("admin user not found, creating one")
                email = 'admin@example.com'
                new_password = get_random_string()

                u = User.objects.create_superuser(email, new_password)
                print(f"===================================")
                print(f"A superuser was created with email {email} and password {new_password}")
                print(f"===================================")
            else:
                print("admin user found. Skipping super user creation")
            print(u)
        except Exception as e:
            print(f"There was an error: {e}")

Now let's create migrations for our custom user model.

Run this:

docker-compose run web python manage.py makemigrations usermodel

Output of command should be something like this:

Migrations for 'usermodel':
  usermodel/migrations/0001_initial.py

Open this file with initial migration.

On top of the file add import command:

from django.contrib.postgres.operations import CITextExtension

And add CITextExtension(), as the first element of operations list.

The file should look like this:

# Generated by Django 3.1.7 on 2021-03-08 13:10

import django.contrib.postgres.fields.citext
from django.contrib.postgres.operations import CITextExtension
from django.db import migrations, models
import django.utils.timezone
import usermodel.managers
import uuid


class Migration(migrations.Migration):

    initial = True

    dependencies = [
        ('auth', '0012_alter_user_first_name_max_length'),
    ]

    operations = [
        CITextExtension(),
        migrations.CreateModel(
            name='User',
            fields=[
                ('password', models.CharField(max_length=128, verbose_name='password')),
                ('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
                ('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
                ('id', models.UUIDField(default=uuid.uuid4, primary_key=True, serialize=False)),
                ('email', django.contrib.postgres.fields.citext.CIEmailField(error_messages={'unique': 'A user with that username already exists.'}, max_length=254, unique=True, verbose_name='Email Address')),
                ('first_name', models.CharField(blank=True, max_length=255, verbose_name='First Name')),
                ('last_name', models.CharField(blank=True, max_length=255, verbose_name='Last Name')),
                ('is_staff', models.BooleanField(default=False, help_text='Designates whether the user can log into this admin site.', verbose_name='Staff Status')),
                ('is_active', models.BooleanField(default=True, help_text='Designates whether this user should be treated as active. Unselect this instead of deleting accounts.', verbose_name='Active')),
                ('is_email_confirmed', models.BooleanField(default=False, verbose_name='Email Confirmed')),
                ('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='Date Joined')),
                ('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.Group', verbose_name='groups')),
                ('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.Permission', verbose_name='user permissions')),
            ],
            options={
                'verbose_name': 'User',
                'verbose_name_plural': 'Users',
            },
            managers=[
                ('objects', usermodel.managers.UserManager()),
            ],
        ),
    ]

Now you can apply migration:

docker-compose run web python manage.py migrate

The output should look like this:

Using selector: EpollSelector
Operations to perform:
  Apply all migrations: account, admin, auth, contenttypes, post_office, redirects, sessions, sites, socialaccount, usermodel
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying usermodel.0001_initial... OK
  Applying account.0001_initial... OK
  Applying account.0002_email_max_length... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying post_office.0001_initial... OK
  Applying post_office.0002_add_i18n_and_backend_alias... OK
  Applying post_office.0003_longer_subject... OK
  Applying post_office.0004_auto_20160607_0901... OK
  Applying post_office.0005_auto_20170515_0013... OK
  Applying post_office.0006_attachment_mimetype... OK
  Applying post_office.0007_auto_20170731_1342... OK
  Applying post_office.0008_attachment_headers... OK
  Applying post_office.0009_requeued_mode... OK
  Applying post_office.0010_message_id... OK
  Applying post_office.0011_models_help_text... OK
  Applying sites.0001_initial... OK
  Applying redirects.0001_initial... OK
  Applying sessions.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
  Applying socialaccount.0001_initial... OK
  Applying socialaccount.0002_token_max_lengths... OK
  Applying socialaccount.0003_extra_data_default_dict... OK
src/tutorial %

Now run the makesuperuser management command.

docker-compose run web python manage.py makesuperuser

The output will contain user password, copy it somewhere, you will need it to log in to admin panel.

admin user not found, creating one
===================================
A superuser was created with email admin@example.com and password rWKwHw5FK6tw
===================================
admin@example.com
src/tutorial %

Try running this command again.

admin user found. Skipping super user creation
None
src/tutorial %

See, second time no user is created.

Congratulations, we are finished with our custom user model!

Procfile for Django Project

Procfile is the file which tells Appliku Deploy how to run your application.

This file must be located in the root of the project.

There are 3 types of records:

web
release
other

web process will be the one that handles HTTP requests.

release will hold the command that is executed on each release, like applying migrations and other activities.

All other processes have no special meaning. For example, you can put celery worker, scheduler or anything else specific to your app.

Procfile can only have one web process and one release process. You can have as many other type of processes as you need.

For our tutorial this will be the Procfile:

web: gunicorn tutorial.wsgi --log-file -
release: bash release.sh

Create a file in the root of the project release.sh:

#!/bin/bash
python manage.py migrate --noinput
python manage.py makesuperuser

release.sh will get executed on every new release and it will apply migrations and try to create a superuser in our app. As you remember only on the first release the user will be created.

Structure of Django Project

Let's take a look at our project before we start with deployment.

src/tutorial % tree
.
├── Dockerfile
├── Procfile
├── docker-compose.yml
├── manage.py
├── release.sh
├── requirements.txt
├── tutorial
│   ├── __init__.py
│   ├── asgi.py
│   ├── settings.py
│   ├── storages.py
│   ├── urls.py
│   └── wsgi.py
└── usermodel
    ├── __init__.py
    ├── admin.py
    ├── management
    │   ├── __init__.py
    │   └── commands
    │       ├── __init__.py
    │       └── makesuperuser.py
    ├── managers.py
    ├── migrations
    │   ├── 0001_initial.py
    │   ├── __init__.py
    └── models.py

Push your Django Application to GitHub

In order to deploy your app you need it to be pushed to a GitHub repository.

At this point of tutorial it is assumed that you have account on GitHub. It you don't, go to https://github.com and sign up.

Then create a repository, let's call it django_appliku_tutorial.

For purposes of privacy you can make repository private, or you can make it public so your peers or future employers can see what you were up to :)

Just remember: never store any credentials in the code. Also remember: whatever you delete from the code, stays in repository history.

Hit create repository button.

Now you are looking at the empty repository page. Let's use their instructions from the section "…or create a new repository on the command line".

Here is the example they gave me.

You will have to change the path to your repository.

Also since we have files we want to add all files git add .

Go to terminal to the root of the project and write these commands:

echo "# django_appliku_tutorial" > README.md
git init
git add .
git commit -m "first commit"
git branch -M main
git remote add origin git@github.com:appliku/django_appliku_tutorial.git
git push -u origin main

After you pushed the code, the GitHub page of repository should look like this:

Deploying Django Project

Instead of doing old fashined manual deployment, writing a lot of configs and hoping it works, we'll use Appliku Deploy.

What Appliku Deploy does:

Provisions a server in Digital Ocean(or AWS)
Takes the code from GitHub repo,
Builds your app on your server
Deploys your app on your server
Sets up web server (nginx) and issues SSL certificate from Let's Encrypt.

In order to deploy your app you should have accounts in GitHub, Digital Ocean and Appliku Deploy.

In case if you don't have them follow these links and create them:

GitHub.com https://github.com/join
Digital Ocean https://cloud.digitalocean.com/registrations/new
Appliku Deploy https://app.appliku.com/

If you just registered on Appliku Deploy and signed in, make sure to complete onboarding, connecting the account to GitHub and Digital Ocean.

First step is to create a server.

It is done via Appliku Deploy interface. Please note, that you can't re-use an existing server you created manually via Digital Ocean interface. It must be provisioned via Appliku Deploy.

Go to the Servers tab: https://app.appliku.com/servers

Click "Create New Server" button and you will be taken to Provider selection page ( https://app.appliku.com/providers

Select DigitalOcean.

You will be taken to the page where you can select type of Droplet you want to provision ( https://app.appliku.com/new-server-digital-ocean ).

For the purpose of this tutorial we will select cheapest available server type 1gb RAM, 1 CPU, 25GB SSD for $5/month and region: 🇳🇱AMS3.

Click "Create A Server".

After this you will be taken to the server list. You will see your server without any details. It is because Digital Ocean haven't fully provisioned it yet. When server is provisioned IP address and server size will appear. Progress of provisioning is displayed in the rightmost column.

You can click on the server name to see server details. This page is updated regularly to reflect server's current status.

When the "Status" becomes "Active" it means that Digital Ocean finished provisioning server.

At this moment the "Setup" field should say "Started".

It means that Appliku Deploy connected to server and is running setup scripts. It will install software needed to run your apps: Docker, Nginx, certbot and configure them.

It should take 2-3 minutes to complete setup.

You can look at progress by going to "Setup Logs" tab. Please keep in mind that this page is not updated on its own and you will have to refresh the page to see latest records.

Back to server's Overview tab: When the "Setup" field says "Finished" it means you can create an app and deploy it on this server.

If the "Setup" field says "Failed" then you can check on "Setup Logs" tab to try to figure out what happened.

Most popular reason for failure is that cloud provider gave us a bad server, that was unable to reach internet due to networking or disk issues. It is a rare occasion but it happens.

In this case you should click "Manage Server in Digital Ocean Panel", destroy the server and back in Appliku Deploy interface create another server.

If you still see the old server in the list - open the server details and it will make page refreshed with the current status of the server. If it is deleted, status of the server will become "deleted" and server will be gone from the server list. You can now create another server.

Create an application. Go to "Applications" tab and click "New App From GitHub" ( https://app.appliku.com/start ).

You will see the form "Creating a new application".

Fill the Application Name, pick repository(that we created earlier), branch(main) and select the server to deploy to.

After that you can click "Create an Application".

You will find yourself on the page of a newly created application.

Let's go over this page real quick.

First section is build settings.

It says that the base image is Python 3.8, which means that under the hood Appliku Deploy will use python:3.8 Docker image to build the image with your app.

You can specify the "build command" which will be executed as the last statement of our Dockerfile. Keep in mind that we pass all environment variables to the build, so your build command will be able to use it.

If you need to use your own Dockerfile instructions, you can select "Custom Dockerfile" in dropdown and put instructions in text field.

Application Processes Section

In this list you will see all records from the Procfile in your repository except release.

Choose what processes you want to enable. For now we have only web, so switch the toggle to On.

Config Variables Section

In this section you should specify environment variables that will be passed to your application when running as well as at build stage.

Let's create several variables.

DJANGO_SECRET_KEY give this variable some long value like "foisr45r4ufuihsuihiuh3rluhihuihrui4wh4uihu4huiwhui44343423" that nobody will ever guess.

DJANGO_ALLOWED_HOSTS should contain your appname + applikuapp.com. Or any other domain you will later attach to the site. In our case it is djangoapplikututorial.applikuapp.com.

Heroku Config Vars Sync

If you are migrating this application from Heroku, this this feature will come in handy. You need to enter your Heroku API key and application name and it will be continuously pulling any changes in Config Vars from Heroku and update them for your app. Keep in mind that editing config vars in appliku when Sync is enabled makes no sense – they will be overriden on the next sync.

Now we need to create a database and a redis instance.

So instead of deploying application right now we should click on "Continue to Application Overview".

This how application dashboard looks like for a new app:

Go to Databases tabs.

Databases

Click "New Database", select Postgres and select your server.

Click "Create Database"

Your new postgres database should appear in the list.

You can see the type of the database, State and credentials URL.

When the State column is Deployed, it means that your database is ready to accept connections.

We also need a redis instance.

Let's add it the same way.

Click the "New Database" button, choose redis and the same server. Click "Create Database".

Redis instance should appear on the list, just like the postgres one.

Now we can go back to editing our config vars to make use of our new databases.

Go to application's Settings tab.

Click "Reveal Config Vars"

That's what we have there right now:

Postgres and redis credentials were added to their own, instance specific variables.

What we need now is to create DATABASE_URL with the value from DATABASE_72_URL and a REDIS_URL with value from REDIS_73_URL.

The reason why we have to do this manually is because you can have multiple databases of the same type and you are responsible for setting your application's env vars to use.

That's how config vars should look at this stage.

Keep in mind that the number in DATABASE_72_URL and REDIS_73_URL will be different for you.

Now we are all set with config vars, you should go to the Overview tab.

We are ready to start the first deployment.

Click "DEPLOY NOW" button.

Information about the current deployment will appear:

When it is finsihed, status will reflect that, saying it is finished:

Let's click Manage Deployments and see our deployment logs for the generated admin password.

You are at the "Deploys" tab.

First card has the form with deployment settings. You can change the repository branch, server to deploy to and toggle Push to Deploy feature, that starts deployment on push to GitHub repository.

Second card contains history of deployments.

In this example I have two deployments. First one failed for me, because I made a typo in settings file, while writing this tutorial. You can see that it says "Failed Building". This means it obviously failed, but another piece of information – we know at what stage it has failed: Build stage.

You can click "View logs" to find out why it could fail.

Our focus right now should be on the successful deployment.

It says "Finished Cleaning up".

There are several stages of deployments in Appliku Deploy:

New - Deployment was just created, but nothing yet has been done.
Building - your server is pulling code from repository and building the image
Deploying - deploying our image
Releasing - release command is being executed
Cleaning up - your server is cleaning up obsolete docker image layers to free up some disk space.

When you click "View logs" you will see the password somewhere in the end of the logs window.

Let's open our app. In the Application navigation click "Open App". Your site will open in new window.

You should see "Not Found".

It is expected because we don't have any pages defined.

If you get 502 Gateway error, then you forgot to enable the "Web" process. Go back to the "Processes" tab and enable Web worker.

Then on the overview page click "Apply Processes & Env Vars" button. This should apply changes faster than the whole build.

If you see 400 Bad Request error then you didn't spell the domain name correctly in env var DJANGO_ALLOWED_HOSTS.

To fix it go to application "Settings" tab and edit the value for DJANGO_ALLOWED_HOSTS to match the domain.

If there is any other error, try redeploying the app and watching logs for anything error-related. There is also application "Log" tab that can help you figure out what can be the problem.

Now let's go to admin interface of your Django Project.

Add /admin to the end of your webiste.

You will see Django Login form.

Enter admin@example.com as the login. Password is the one generated, that you saw in Deployment logs.

Hit the "Log In" button.

You should see the Django admin panel.

Congratulations!

You have just created your very first Django app and deployed it on Appliku Deploy. All that without learning anything devops: nginx, certificates, etc. All is done for you.

New articles coming soon, you will be able to expand functionality of your Django Project with sending emails, accepting payments and building a proper SaaS product.

Happy deploying!

Stop deployment

Kostja Appliku.com — Sun, 04 Oct 2020 03:46:09 +0000

I am working on Appliku, which is building, and deployment automation SaaS.

Read the story: https://dev.to/kostjapalovic/tired-of-deployments-built-my-own-heroku-47ed.

It is built with Django and heavily uses Celery.

One of the most important parts that I want to improve is the build/deployment process. More specifically being able to manage running build/deploy jobs.

What problems I see right now:

Deployments run as long as they want
There is no way to stop/cancel the deployment
If the app has a push-to-deploy setting On then several deployments will be running at the same time and this can exhaust the server's resources and lead to errors due to concurrent deployments
I want to disable the "Deploy Now" button while there are deployments in the queue
by the way, I want deployments to be in the queue with only one running
If a deployment is stuck or taking a significant amount of time to complete this will effectively block the user from deploying the next version of the code which might have fixes for the underlying problem.

With all said, it boils done one thing right now: I need a way to stop a deployment job.

How Appliku runs the build/deployment process?

There is a Celery task, that connects to the server using the Paramiko library.

The actual code of the function is this:

What I want to do is to somehow pass the signal from the dashboard to stop this. The function will have to kill the command running and close the connection, mark the deployment as canceled.
Also, it must have timeouts in case if any of the stages are stuck for any reason.

I use Redis cache to store the abort flag for deployment.

Now I only need to add the "Abort Deployment" button in the interface of https://appliku.com/

Tired of Deployments, built my own Heroku

Kostja Appliku.com — Sun, 16 Aug 2020 07:58:59 +0000

This article is brought to you by the first deployment service, dedicated specifically to Python & Django Appliku.com.
Never manage a server again.
Deploy your Django app in 5 minutes.

TLDR: I got tired of deploying my Django projects, went to Heroku, and got burnt by their invoices, built my own SaaS to deploy apps to existing clouds.

Story begins with custom software development agency

6 Years ago I started yet another agency where we were building custom solutions for businesses that needed automate time-consuming tasks:

Semi-automated task distribution for a large service company,
Big e-commerce businesses,
Generation of construction documentation
Many smaller projects.

It was an interesting journey.

I loved what we've built. The money we saved. Manual labor we got rid of(well, not everyone was happy about the last part of course).

It lasted for 5 years and I loved most of what we had to do.

But one thing was annoying every single time.

Every now and then I needed to spin up a new project and set up a test environment and then production one.

Deployment was a pain every time

During that time I have experimented with different deployment approaches and tools:

Used Fabfile,
Added CI/CD – first Atlassian Bamboo, then GitLab
Used Supervisord, then Docker,
Bare-metal servers, Digital Ocean, Linode.

At some point, I came up with a number of scripts that used docker-compose and GitLab CI to start deployment on new commits pushed to the repository.

But for each new project, a lot of manual work had to be done, it was human-error prone and took a full day to set everything up:

.gitlab-ci.yml
docker-compose.yml
set up Nginx and let's encrypt
databases
list of small things goes on

I loved it when a new client project came up, but I feared going over that process again and again.

I thought there has to be a better way.

Let's give Heroku a try

At that time we came up with a couple of our own side projects, where we use Heroku.

It was such a pleasant experience.

I was "WOW"-ing on every step.

I literally spent zero time on anything DevOps.

I moved a couple more projects there, that was easy. None of those projects were generating any revenue since it was hobby sites.

Then the time came and I received SMS from the bank about Heroku charging my card. At that moment all excitement went away.

Here comes Heroku Invoice

In 2 months Heroku spendings climbed from $15/mo up to $80/mo. For hobby projects.

I have immediately disabled those sites. One I moved off, as the only one that we were interested in.

This coincided with a drastic change in my personal financial situation. I was completely broke for several months.

Getting back on track, seeking for the idea

When I solved my financial issues, I had time to think about what to do next.

We have finished with custom software development.

What was left after doing that business clearly was the pain from deployments and pain from paying for Heroku.

I wanted to get back to some of our hobby projects.

I had time to make it the right way and enjoy doing it.

I wanted something as pleasant as Heroku, but I didn't want to pay that much for that.

Light bulb

It became clear to me: I need to build my own Heroku.

At that time on IndieHackers, I stumbled upon Hatchbox.io for Ruby and heard about Laravel Forge for PHP.

I treated that as validation for a similar tool but for Python/Django.

I came up with a name for it a month later: Appliku.

I started from bash scripts for building Docker, spun up a Django up to manage them, and iterated on that for 17 months to this moment (Sun, 16th Aug 2020).

What do I have now?

Automated provisioning of servers on both AWS and DigitalOcean
Server setup (Docker, Nginx, Let's Encrypt)
Building apps from the source code on GitHub Repo
An almost as good interface as Heroku
Heroku Config Vars Sync for those who are trying to move off Heroku step by step.

What is under the hood?

Django application is the core of the product.

The frontend is built with Angular based on mdbootstrap.com Angular Package.

Tons of background tasks run by Celery.

When a user requests creation of a new server – Appliku talks via API to Digital Ocean or AWS to provision it.

When Server is provisioned, with help of the Paramiko module Appliku connects to the server over SSH and runs the setup script.

When the setup is complete – you can deploy your app.

App creation consists of naming your app, picking repository and branch, and the server you want to deploy to.

The only requirement for the app is having a Procfile.

I figured that since I was moving off of Heroku, there must be others who do the same. And I liked this idea of having processes in the code itself rather than specifying them in UI.

Procfile is a text file where you specify what processes are needed to be running for your app to operate.

web: python gunicorn wsgi:app
release: python manage.py migrate
celery: celery worker -Q default

"web" tells us that this process should accept HTTP connections.

"release" is what get's executed on every release.

Other processes have no special meaning.

Appliku asks what programming language you use to run the app. Right now supported languages are Python, PYPY, PHP, and Node.

For complex builds, we allow writing your own Dockerfile.

Heroku Sync is a feature that pulls config vars of the app from Heroku API every minute and updates config vars for the related app in Appliku and rebuilds it.

This has proven to be useful for those who moved part of Heroku Dynos to Digital Ocean with the purpose of reducing expenses. Usually, this is done with memory-hungry background workers.

When you hit Deploy.

Appliku makes your server to:

go to your GitHub repo
clone the code
create docker-compose file(based on Procfile and processes you enabled to run)
build the image
start it

Also, it requests Let's Encrypt SSL certificate for appname.applikuapp.com domain or any other custom domain you have added.

By making your server do that we ensure that all your code is only stored on your server. The only piece of information from your code we store is the contents of the Procfile.

After the app is deployed it runs without much of the interaction with our app.

If you push new commits - we run the build/deploy process.

Every day we run certbot to check if any of the certificates are about to expire.

Couple words about Nginx configuration.

Nginx vhost is setup in a way that it respects cache headers from your app.

proxy_cache_path /home/app/_cache/sitename.com levels=1:2 keys_zone=sitename.com:10m max_size=1g inactive=60m use_temp_path=off;
location / {
        proxy_cache sitename.com;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 1;
        proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_background_update on;
        proxy_cache_lock on;
        proxy_cache_bypass $cookie_nocache ;
        add_header X-Cache-Status $upstream_cache_status;

        proxy_pass http://127.0.0.1:8003;
        proxy_read_timeout 600;
        proxy_send_timeout 600;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

This way you can use dummy caching on the application side (like Dummy Cache Backend in Django) in order to generate proper cache headers and the actual caching will happen before the request reaches the backend.

Closing thoughts.

I have built this whole thing alone with the help of my wife that took ownership of most non-engineering tasks.

It is a fun thing to work on. I have learned a lot about Docker, Nginx, Paramiko, Django itself.

I think that this app can help many people besides me to:

ship their idea faster
avoid doing DevOps
solve the frustration for beginners "I built the app, how do I publish it?"

Business-model is simple – you pay Appliku fixed fee for service and pay directly to cloud providers for their services.

There is a free plan on Appliku and AWS gives 1 year for new users.

If you are learning to code or seeking for a (mostly) free and enjoyable way to run your app – I really hope in these 17 months I built a solution that can give you that.

P.S.

I really hope you that hearing about my journey and what I built was interesting and doesn't look like a promotional thing.

It was a really interesting journey, full of excitement, frustration (Hello, AWS Docs!), and the idea of making someone's work easier.

I shared all this here to tell that even big and complex things can be done if you believe in it and if you show up every day (even for 30 minutes) and persistently work on it.

If the app helps someone or the article itself motivates somebody to get up and create something – I will be extremely happy.

Thanks for taking time to read this.

P.P.S.

Nearly forgot about the link to the app: https://appliku.com/

CookieCutter based project generator on the web

Kostja Appliku.com — Mon, 26 Feb 2018 11:44:42 +0000

I have recently started a few Django projects, mostly to play around with some new tech and automate some of my workflows.

Then I thought, that there is cookiecutter project that should automate most part of project creation.

While this is a great and very useful tool, I don't like to answer questions line by line in terminal to generate a project.

I wanted a web interface for that.

In several hours I came up with a solution and it resides here: Generator.KPavlovsky.pro

Python and Python-Django projects are included in available templates. After feeling form with questions (taken from templates' cookiecutter.json) and clicking Generate, you get ZIP archive with your new project.

Wish it will be useful for at least somebody.

Feedback is welcome.

Contact me here or on twitter to provide feedback and/or feature requests.

@all, have a great day!

Setting up tests in GitLab CI for Django project with Docker Engine

Kostja Appliku.com — Thu, 16 Feb 2017 18:51:32 +0000

Hello!

In this article I will describe how we set up GitLab CI to run tests for Django project.

But first a couple of words about what tools we were using.

What we had before GitLab – Atlassian

For a few years we were using Atlassian stack for our development process. We had JIRA, Confluence, Stash(now it is called BitBucket Server) and Bamboo.

At first we were happy with that setup, because all apps had good integration with each other.

We could see all commits related to currently open issues, and we could see what issues and commits were included in build in Bamboo.

It looked wonderful at first, but after some time we noticed that:

JIRA began consuming all our time even with Agile Board
keeping this stack up-to-date was a huge pain. I'd spend whole weekends updating and repairing all four apps, because they'd changed requirements for Mysql settings, and logs in Atlassian products are buried deep in various directories
Every new version of Atlassian apps was introducing numerous bugs and interface changes (which with obvious lack of QA was causing even more bugs)

Plus we only had a 16GB RAM server for all this tools.

So instead of doing our job, we were spending all our time dealing with JIRA.

New developers who were joining our team were frustrated with the JIRA interface.

At the end of last year we thought: “That's enough, we need to replace this enterprise-focused monster with something easier and productive tool.”

At first, we started looking for a replacement for task tracking functionality. We tried many apps and services for that, but all of them, while having some strong features, also had issues or lack of functionality which prevented us from being productive.

Then I tried GitLab and suddenly we not only found the solution for convenient task tracking tool, but we found how to replace the whole Atlassian stack.

What is also is amazing we get everything for free! (Except worker, but i have unused linux box, which now serves as worker).

GitLab has a clean and simple interface, instead of the enterprise-crazy workflows of Jira, issues have labels, there is CI pipelines, Wiki, markdown for issue descriptions, comments and wiki articles, and many other great things.

We have switched to GitLab.com

GitLab projects solved all our problemsâ€Š–â€Šno more maintenance, no pain with productivity-killing interface, all in one solution.

So gitlab project provides git repository hosting, issue tracker, wiki, CI/CD management system. But what we need is CI runner. Runner does actual job, executing building, testing and deployment jobs.

As I already said, I had unused linux box, which now is used for runner.

Runner installation for ubuntu described here:

https://docs.gitlab.com/runner/install/linux-repository.html

We used docker for running our builds.

Migration from Bamboo to GitLab CI/CD

In Bamboo build and deploy plans are setup from GUI.

For gitlab .gitlab-ci.yml must be created in root of git repository.

Before I provide example of this file, I must point out that we use Postgres as a database. Thanks to docker we can require runner to run it as service with credentials from this file.

.gitlab-ci.yml

image: kpavlovsky/backoffice:latest  

services: 
  - postgres:latest

stages: 
  - test  
  - deploy

variables:  
  POSTGRES_DB: dev  
  POSTGRES_USER: dev  
  POSTGRES_PASSWORD:dev  

all_tests:  
  stage: test  
  script: 
  - bash ./tests_gitlab.sh

Line by line:

image: kpavlovsky/backoffice:latest is docker image we use to run container.

This image is based on python:3-onbuild. We moved all long-running pip install and apt-get tasks there. By doing this we achieved two things: 1) each build runs faster, because it doesn't involve packages installation 2) we do not abuse apt and pip repositories by downloading packages 10s or 100s time per day 3) we decrease time of each build so we get results of tests much faster (who wants to wait for 15 minutes after each git push ?).

In services docker images for services are listed. Here we have only postgres.

In variables section sets up postgres database and its credentials. Hostname for postgres will be â€˜postgres'.

The ordering of elements in stages defines the ordering of builds' execution:

Builds of the same stage are run in parallel.
Builds of the next stage are run after the jobs from the previous stage complete successfully.

Job in â€˜test' stage goes first. Jobs in â€˜deploy' stage goes after â€˜test' stage.

all_tests is a job our of pipeline, belonging to â€˜test' stage. script hold all commands that will be issued. We have only one command hereâ€Š–â€Što run tests.

#!/bin/bash  
coverage run --source="app1,app2,app3" manage.py test --noinput --
testrunner="xmlrunner.extra.djangotestrunner.XMLTestRunner" 
app1.tests app2.tests app3.tests  
coverage report --skip-covered

Dockerfile for that separate kpavlovsky/backoffice:latestimage looks this way:

FROM python:3-onbuild  
ENV PYTHONUNBUFFERED 1  
ENV PYTHONDONTWRITEBYTECODE 0  
ENV DJANGO_SETTINGS_MODULE project.settings.docker  
RUN apt-get update && apt-get install -y --force-yes mc nano htop python python-pip netcat gettext && rm -rf /var/lib/apt/lists/*  
RUN mkdir /code  
WORKDIR /code  
COPY requirements.txt /code/  
RUN pip install --upgrade pip  
RUN pip install -r requirements.txt  
CMD ["bash"]

So now if we push code to repo and tests pass, then we see notification in slack and email that pipeline is successful or that pipeline is failed if tests fail.

Deployment

Now that we migrated the build part, we need to deploy our project in case of successful build on â€˜dev' branch to staging.

â€˜Stage' environment is another linux box, without docker, just supervisor and gunicorn.

Deployment process involves ssh-ing to remote box from runner, activating virtualenv, git pulling and running django management commands.

First step is add job in â€˜deploy' stage in our .gitlab-ci.yml

deploy_stage:  
  stage: deploy  
  script: - bash ./deploy_stage_gitlab.sh  
  when: on_success  
  only: - dev</pre>

This job will only be run on â€˜dev' branch and only if â€˜test' stage is successful.

To ssh to â€˜stage' machine we need to transfer ssh keys to runner.

Storing keys in repository is bad practice.

Thanks to gitlab â€˜Variables' we can transfer keys via environment variables, write them to files and then issue fabric command to execute required tasks on â€˜stage' box.

First we need to generate ssh key without passphrase. For this purpose, use ssh-keygen .

Public key must be put in ~/.ssh/authorized_keys on stage server.

Then we put contents of public and private keys to Variables.

After adding variables with keys, Variables screen in gitlab project looks similar to this:

deploy_stage_gitlab.sh looks this way:

#!/usr/bin/env bash
mkdir ~/.ssh/  
echo $STAGE_PRIVATE_KEY > ~/.ssh/id_rsa  
echo $STAGE_PUBLIC_KEY > ~/.ssh/id_rsa.pub  
fab upd_dev

Quick note: use fabric3 in python3.5 environment!

fabric function logs in to remote server, git pull everything, runs migrations and uses supervisorctl to restart process group for this project.

Conclusion

With .gitlab_ci.yml we have ability to change pipeline, so it reflects changes in our code. That was harder to achieve with Bamboo and its build/deploy settings in GUI. Of course, we could let some bash-script to do all tasks and it would be changed from commit to commit. In this case it is impossible to add/remove/change stages, etc.

Also Bamboo's guy build/deploy setup requires much more time to set up, I didn't find a way to clone it from project to project, which is very easy to do with .gitlab_ci.yml.

From now, we don't need a separate large server for Atlassian tools, we don't spend weekends updating them. And the best part: we can focus on doing real jobs, developing and delivering OUR apps, instead of wasting time on Atlassian's.

Happy developing!

If you have any ideas about gitlab CI, ideas how to improve described workflow – post comments, i will be very happy to hear and discuss them!

DEV Community: Kostja Appliku.com

Django + Celery In-depth tutorial for beginners

Git repository

What is Celery?

What is a task queue

What distributed means?

Why and When to Use Celery

Warning

Starting a Django project with Celery using Docker

Dockerfile

requirements.txt

docker-compose.yml

.env

.gitignore

.dockerignore

Pull images with Docker Compose

Build Docker image for Django Project

Start Django Project

Initial Django configuration

Apply migrations

Start containers

Check logs of Django Project with docker compose logs

Start an application

Deploying Django Celery app

Simple Celery task

Celery API in more details

Celery Task definitions

@app.task decorator:

@shared_task decorator(recommended):

Three ways to call a Celery task

Sending a Celery task via .delay

Sending a Celery task via .apply_async

Sending tasks via current_app.send_task

How to correctly send a Celery task from a view

Workers setup

Example App: Generate reports

Conclusion

Django REST Framework and DataTable Tutorial

Introduction

Create models

Datatable with server rendered table

Django REST Framework API for Datatable

Django Project Tutorial for beginners: Settings, Docker-Compose, Postgres and Redis

Django Tutorial Source Code

Create the environment for the project

Requirements for Django Project

.gitignore for Django Project

Dockerfile

docker-compose.yml

Explanation about Django and Docker

Django Settings

Django Custom User Model

Procfile for Django Project

Structure of Django Project

Push your Django Application to GitHub

Deploying Django Project

Application Processes Section

Config Variables Section

Heroku Config Vars Sync

Databases

Stop deployment

Tired of Deployments, built my own Heroku

Story begins with custom software development agency

Deployment was a pain every time

Let's give Heroku a try

Here comes Heroku Invoice

Getting back on track, seeking for the idea

Light bulb

What do I have now?

What is under the hood?

When you hit Deploy.

Couple words about Nginx configuration.

Closing thoughts.

P.S.

P.P.S.

CookieCutter based project generator on the web

Setting up tests in GitLab CI for Django project with Docker Engine

What we had before GitLab – Atlassian

We have switched to GitLab.com

Migration from Bamboo to GitLab CI/CD

`@app.task` decorator: