How to configure Django Q with Supervisor on AL2 Beanstalk

Working on the fintech project backend I decided to use Django Q package for scheduling and executing different tasks in the background. Our CD-CI flow was based on Beanstalk new Amazon Linux 2 (AL2) machine. The trick is that Beanstalk configuration files have to include all instructions to run your code from scratch on a brand new virtual machine. Why? That's how the AWS Beanstalk works: it manages the server for you and requires precise instructions on what to install/configure in case it has to run an update or recreate/scale the virtual servers from scratch. I hoped to google some decent bash script for AL2, but there was none. So I had to dive into DevOps thing again and write my own.

Postdeploy hooks to get Django-Q cluster running on Beanstalk's AL2

Note #1: This guide is for Amazon Linux 2 EC2 instances which were rolled out for Beanstalk python platform in April 2020. If you're looking for a way to set up Django Q cluster on an older version (Amazon Linux), here is the working gist for it.

Note #2: If you're not familiar with Django Q cluster (celery based task manager) I strongly recommend reading the docs. Besides Beanstalk environment Django Q requires some message broker to schedule the tasks, so you'll need to set up a separate Redis server or Amazon SQS and connect it to your security group.

I assume you have your project configured to be deployed to Beanstalk. AWS Beanstalk allows you to place a custom number of bash scripts with .sh file extension into the following folder .platform/hooks/postdeploy in your root project directory. For convenience, I split it into 4 steps. Here's the full gist.

01_set_env.sh Grab the environmental variables and enable amazon-linux-extras

	#!/bin/bash
	#Create a copy of the environment variable file.
	sudo cp /opt/elasticbeanstalk/deployment/env /opt/elasticbeanstalk/deployment/custom_env_var
	#Set permissions to the custom_env_var file so this file can be accessed by any user on the instance. You can restrict permissions as per your requirements.
	sudo chmod 644 /opt/elasticbeanstalk/deployment/custom_env_var
	#Remove duplicate files upon deployment.
	sudo rm -f /opt/elasticbeanstalk/deployment/*.bak
	#Install/enable extra packages
	sudo amazon-linux-extras install epel -y
	#Install supervisor
	sudo yum install supervisor -y >/dev/null 2>&1
	if [ $? -ne 1 ]; then # Exit on any any error except 'nothing to do'
	exit 0
	fi

view raw 01_set_env.sh hosted with ❤ by GitHub

AL2 provides access to environmental variables we set in the Beanstalk dashboard through a specific file. Here we're grabbing them for further use. AL2 doesn't include supervisor. That's why we're also telling Beanstalk to enable extra extensions for us.

02_django_migrate.sh Just Django routine commands

	#!/bin/bash
	source /var/app/venv/*/bin/activate && {
	# collect static
	python manage.py collectstatic --noinput;
	# log which migrations have already been applied
	python manage.py showmigrations;
	# migrate user model prior to other models if you have Custom User Model < NB
	#python manage.py migrate users --noinput;
	# migrate the rest
	python manage.py migrate --noinput;
	# create superuser
	python manage.py createsu;
	}

view raw 02_django_migrate.sh hosted with ❤ by GitHub

This doesn't have much to do with Django Q cluster, but as part of Django app deployment process these steps are necessary.

03_django_q.sh Django Q supervisor configuration

	#!/usr/bin/env bash
	# Author: Valentine Solonechnyi <[e-mail]>
	# Based on instructions by Rémi Héneault
	# https://gist.github.com/codeSamuraii/0e11ce6d585b3290b15a9ad163b9aa06
	# Django Q supervisor configuration and setup for Amazon Linux 2
	mkdir -p /var/log/djangoq/ /var/run/djangoq/
	# Get django environment variables
	# grep '^PYTHONPATH\|^PATH' no filtering of env variables
	# djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g'`
	# fix from @matiszz
	djangoqenv=`cat /opt/elasticbeanstalk/deployment/custom_env_var | tr '\n' ',' | sed 's/=/="/'g | sed 's/,/",/g' | sed 's/="="/=="/'g | sed 's/""/"/'g`
	djangoqenv=${djangoqenv%?}
	# Create djangoq configuraiton script
	djangoqconf="[program:django-q]
	command=bash -c 'source /var/app/venv/*/bin/activate && python manage.py qcluster'
	directory=/var/app/current
	user=nobody
	numprocs=1
	stdout_logfile=/var/log/djangoq/worker.log
	stderr_logfile=/var/log/djangoq/worker.log
	autostart=true
	autorestart=true
	startsecs=10
	; Need to wait for currently executing tasks to finish at shutdown.
	stopwaitsecs = 600
	; When resorting to send SIGKILL to the program to terminate it
	; send SIGKILL to its whole process group instead,
	; taking care of its children as well.
	killasgroup=true
	stopasgroup=true
	environment=$djangoqenv
	"
	# Create the djangoq supervisord conf script
	echo "$djangoqconf" | sudo tee /etc/supervisord.d/djangoq.conf
	# Add configuration script to supervisord conf (if not there already)
	if ! grep -Fxq "files = supervisord.d/*.conf" /etc/supervisord.conf
	then
	sed -i "s/*.ini/*.conf/g" /etc/supervisord.conf
	fi
	#Launch supervisord process
	sudo supervisord -c /etc/supervisord.conf
	# Reread the supervisord config
	sudo supervisorctl reread
	# Update supervisord in cache without restarting all services
	sudo supervisorctl update
	# Start/Restart djangoqd through supervisord.
	sudo supervisorctl -c /etc/supervisord.conf restart django-q

view raw 03_django_q.sh hosted with ❤ by GitHub

Now we got Django up and running, but we still need to launch Django Q cluster and make sure it will be alive even if the process crashes. Here we're preparing a configuration block which tells the supervisor (Linux process control system) where to find and how to launch Django Q cluster.

04_supervisor_init.sh Supervisor daemon configuration

	#!/usr/bin/env bash
	#Supervisor init config
	# Add a new configuration of restart Supervisor
	sudo cp /var/app/current/supervisord.sample /etc/init.d/supervisord
	#Add execute authority
	sudo chmod +x /etc/init.d/supervisord
	#Add the configuration into system
	sudo chkconfig --add supervisord
	#Switch on the configuration and start
	sudo chkconfig supervisord on
	sudo service supervisord start

view raw 04_supervisor_init.sh hosted with ❤ by GitHub

The last but not least: we need to take care about the supervisor itself and add config for its daemon. Thus, we make sure that supervisor will be loaded after EC2 instance reboot. This is the last bash script file, but pay attention to this line: sudo cp /var/app/current/supervisord.sample /etc/init.d/supervisord

/var/app/current/ - that's the default path to your project. In my root directory I placed a supervisord sample config file (by Dan MacKinlay). You can place it wherever you want, just don't forget to change the path in the bash script to it.

That's all, you're good to go. This guide doesn't cover full CI-CD flow with Django app and AWS, but I might be covering it in the next article. Stay tuned!

This article was originally published in my blog post