Django Background Tasks

charliemday — Thu, 05 Oct 2023 20:03:31 +0000

Background tasks are processes that run in the background so your server is ready to take further requests.

Some common use cases for background tasks:

Web Scraping
Heavy data processing
Image processing
Multiple 3rd party API calls
Building Slack-bots (Slack requires a 3 second response)

Basically, if the task takes a long time to complete (e.g. >30 seconds) you probably want to offload it to a background task.

If you have your Django project set up it's dead easy to get a background tasks up and running.

Step 1: Getting Started

We want to install Django Q via:



pip install django-q

Django Q allows us to set up some queues to run in the background. Be sure to run the migrations as well to generate the job models (which you'll see in your admin panel)



python manage.py migrate

It requires a broker though...

Step 2: Setting up Redis

We'll use redis. This is basically the communication channel between your django instance (normal django) and your django worker (this is your background task/worker).

You can check if your redis server is running by running:



redis-cli ping

If not you'll need to follow the redis instructions to set up on your system.

Step 3: Setting up the Cluster

The standard django-q configuration will work for us locally but with a couple of changes so we can run this in production as well. Add the following to your base settings.py file:



REDIS_HOST = os.environ.get("REDIS_HOST", "localhost")
REDIS_PORT = os.environ.get("REDIS_PORT", 6379)
REDIS_DB = os.environ.get("REDIS_DB", 0)
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", None)

Q_CLUSTER = {
    "name": "hello-django",
    "workers": 8,
    "recycle": 500,
    "timeout": 60,
    "compress": True,
    "cpu_affinity": 1,
    "save_limit": 250,
    "queue_limit": 500,
    "label": "Django Q",
    "redis": {
        "host": REDIS_HOST,
        "port": REDIS_PORT,
        "db": REDIS_DB,
        "password": REDIS_PASSWORD,
    },
}

You can play around with the number of workers - the default is 8 and these are the number of instances that can run in parallel with each other.

We can now actually start running the cluster. Have 2 terminal windows and in 1 have your standard python manage.py runserver running and in the other run:



python manage.py qcluster

You now have your cluster running.

Step 4: Creating our Task

Create a standard API to make a simple POST request and let's add the task:



from django_q.tasks import async_task
from rest_framework.views import APIView
import time

...

class SimpleTask(APIView):

    def task(self, s = 5):
        print("⏳ Running task")
        # We could call some long running code here
        time.sleep(s)
        print("⌛ Task finished")

    def post(self, request):
        # We can run the task in the background
        async_task(self.task, s=5)
        return Response({
            "message": "Task started"
        })

We use async_task to wrap the task. The code will essentially see this and send it to the redis server (which will send it to the cluster) and continue to the response.

You can place anything in the task and the response will still be returned in the same amount of time.

Step 5: Deploying to Production

Having this run in a safe local environment is one thing. Deploying it to a production environment is another.

We'll be using Railway (other than them I'd recommend Fly.io or Render - the principles are the same).

We want to add a railway.json file in our Django App with the following:



{
  "$schema": "https://railway.app/railway.schema.json",
  "build": {
    "builder": "NIXPACKS"
  },
  "deploy": {
    "restartPolicyType": "ON_FAILURE",
    "restartPolicyMaxRetries": 10
  }
}

In Railway you want to create 1 Redis database, 2 Django Apps. In the first Django App you want the Deploy (Start Command) to be:



python manage.py migrate && gunicorn NAMEOFYOURAPP.wsgi

And in the second:



python manage.py migrate && python manage.py qcluster

You'll want to update the environment variables for both applications to match the REDIS environment variables (see Step 2), for this I recommend using Railway's shared variables. The end result looks something like:

Think of this mirroring your terminals (the standard Django instance and the new Django qcluster instance)

To verify this is all working, make a request to your SimpleTask (you can find the URL if you click on the api app) and take a look at the Observability tab and you should see your background task running 🎉

My Stack for Rapid Prototyping ⚡

charliemday — Tue, 26 Sep 2023 07:29:24 +0000

As a solo developer one of the advantages you have over large incumbents and companies is your ability to rapidly build and test in a real-world environment.

There are no layers of bureaucracy with single developers (or small groups of developers). You can deploy straight to prod because of the most important factor - you have so much more to gain than to lose.

With larger companies, this is reversed. So I would lean into this advantage and you can do that by rapidly prototyping applications. Here's my stack of tools that I feel have helped me speed up development - nothing fancy, and all (mostly) free!

Frontend

For landing pages I use Astrobuilt

For the actual app interface I use a NextJS Boilerplate that I've used on previous projects (I don't bother with latest versions because the user doesn't care)

Deploying the app and landing page I use Vercel as it has a free tier to deploy straight from GitHub and site analytics which is a win

Backend

Depending on complexity I got for either Supabase (If I have a basic CRUD app)or for more complicated backends I go with a trusty Django Boilerplate with authentication APIs already setup and a PostgreSQL database.

Deploying the backend I use Railway as the interface is really clean and it's significantly cheaper than and (and in my opinion just easier to use than Heroku)

That's it! Using this I can get a landing page up within 30 minutes and an MVP inside 2 weeks (which is my general target). Would be interesting to hear other people's stacks for rapid prototyping!

DEV Community: charliemday