DEV Community: Zach

Start Charging Customers with Django and DjStripe

Zach — Mon, 22 Apr 2024 19:36:45 +0000

We're going to use the Dj-Stripe package so that we can start collecting payments. This should take about 10 minutes to set up.

This guide will include:

Setting up products and prices in the Stripe dashboard.
Building an e-commerce store with a shopping cart feature.
Using dj-stripe to sync product and price changes from Stripe dashboard to database.

Here's a video of the e-commerce store in action:

Previewing the project

Want to see a live version of the app? You can create a copy of this project now and try it out.

Run the online demo

Setting up the Django app

Install packages and create the Django application.

django-admin startproject ecommerce .
python3 manage.py startapp core

Add core and other apps to the INSTALLED_APPS list.

# settings.py
INSTALLED_APPS = [
    ...,
    "mathfilters",
    "djstripe",
    "core",
]

Installing the requirements

Create a requirements.txt file if you don't have one already.

Django==5.0.2
psycopg2==2.9.9
tzdata==2023.4
phonenumbers==8.13.35
django-phonenumber-field==7.3.0
django-mathfilters==1.0.0
dj-stripe==2.8.4

Adding the templates

Create a directory named templates within the core app.
Create a file named base.html within the templates directory.

<!DOCTYPE html>
<html>
  <head> 
    <script src="https://cdn.tailwindcss.com"></script>
    <script src="https://js.stripe.com/v3/"></script>
  </head>
  <body>
    {% block content %}
    {% endblock %}
  </body>
</html>

Create a file named shopping.html for displaying the main e-commerce products page.

{% extends "core/base.html" %}
{% load mathfilters %}
{% block content %}
{% if messages %}
    {% for message in messages %}
        <div class="message max-w-7xl mx-auto px-4 sm:px-6 lg:px-8">
            <div class="mt-4 p-4 text-sm text-white rounded-md {% if message.tags == 'success' %}bg-green-500{% elif message.tags == 'error' %}bg-red-500{% elif message.tags == 'warning' %}bg-yellow-500{% else %}bg-blue-500{% endif %}">
                {{ message }}
            </div>
        </div>
    {% endfor %}
{% endif %}
<div class="max-w-7xl mx-auto p-4 sm:p-6 lg:p-8">
  {% include 'core/partials/_cart.html' with cart=cart %}
  <div class="grid grid-cols-1 sm:grid-cols-2 md:grid-cols-3 gap-6">
    {% for product in products %}
    <div class="bg-white p-6 shadow-lg flex flex-col justify-between space-y-6">
      <div>
        <img src="{{ product.images|first }}" width="300" height="300" class="w-full object-cover rounded-md mb-4">
        <div class="flex flex-col justify-between flex-grow">
          <div>
            <h3 class="text-lg font-semibold mb-2">{{ product.name }}</h3>
            <p class="text-gray-600" style="height: 8em; overflow: scroll;">{{ product.description | default:"No description available." }}</p>
          </div>
          <p class="text-lg font-semibold text-gray-900 mt-auto">${{ product.prices.all.0.unit_amount|div:100 }}</p>
        </div>
      </div>
      <form action="{% url 'add_to_cart' %}" method="post">
        {% csrf_token %}
        <input type="hidden" name="price_id" value="{{ product.prices.all.0.id }}">
        <input type="hidden" name="quantity" value="1">
        <button type="submit" class="w-full bg-blue-500 hover:bg-blue-600 text-white py-2 px-4 rounded transition duration-300">Add to Cart 🛒</button>
      </form>
    </div>
    {% endfor %}
  </div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function() {
    const messages = document.querySelectorAll('.message');
    setTimeout(function() {
        messages.forEach((msg) => msg.style.display = 'none');
    }, 3000);
});
</script>
{% endblock %}

Create a file named checkout.html to display a summary of what is in the user cart prior to checking out with Stripe.

{% extends "core/base.html" %}
{% load mathfilters %}

{% block content %}
<div class="max-w-4xl mx-auto px-4 sm:px-6 lg:px-8">
    <div class="mt-10 mb-6">
        <h2 class="text-2xl font-bold leading-7 text-gray-900 sm:text-3xl sm:truncate">Shopping Cart</h2>
    </div>
    <div class="flex flex-col">
        <div class="-my-2 overflow-x-auto sm:-mx-6 lg:-mx-8">
            <div class="py-2 align-middle inline-block min-w-full sm:px-6 lg:px-8">
                <div class="shadow overflow-hidden border-b border-gray-200 sm:rounded-lg">
                    <table class="min-w-full divide-y divide-gray-200">
                        <thead class="bg-gray-50">
                            <tr>
                                <th scope="col" class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Product</th>
                                <th scope="col" class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Price</th>
                                <th scope="col" class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Quantity</th>
                                <th scope="col" class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Total</th>
                            </tr>
                        </thead>
                        <tbody class="bg-white divide-y divide-gray-200">
                            {% for item in cart.items.all %}
                            <tr>
                                <td class="px-6 py-4 whitespace-nowrap">
                                    <div class="flex items-center">
                                        <div class="flex-shrink-0 h-10 w-10">
                                            <img class="h-10 w-10 rounded-full object-cover" src="{{ item.product.images|first }}" alt="Image of {{ item.product.name }}" width="300" height="300">
                                        </div>
                                        <div class="ml-4">
                                            <div class="text-sm font-medium text-gray-900">{{ item.product.name }}</div>
                                            <div class="text-sm text-gray-500">{{ item.product.description|truncatechars:50 }}</div>
                                        </div>
                                    </div>
                                </td>
                                <td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">${{ item.unit_price }}</td>
                                <td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">{{ item.quantity }}</td>
                                <td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">${{ item.total_price }}</td>
                            </tr>
                            {% endfor %}
                        </tbody>
                    </table>
                </div>
            </div>
        </div>
    </div>
    <div class="flex justify-between items-center mt-6">
        <div class="text-lg font-medium text-gray-900">Total: ${{ total }}</div>
        <form action="{% url 'checkout' %}" method="post">
          {% csrf_token %}
          <button class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded transition duration-300 ease-in-out">
            Checkout
          </button>
        </form>
    </div>
</div>
{% endblock %}

Create a file named checkout_success.html to use when a user is redirected back from Stripe after a successful payment.

{% extends "core/base.html" %}
{% block content %}
    <div class="bg-gradient-to-br from-emerald-50 to-sky-100 min-h-screen flex flex-col items-center justify-center px-4 py-16">
        <div class="bg-white rounded-lg shadow-lg p-8 max-w-md mx-auto">
            <div class="flex justify-center mb-6">
                <img alt="A 3D rendering of a cheerful golden trophy cup with confetti bursting around it, set against a vibrant blue background"
                     class="w-24 h-24 rounded-full object-cover object-center"
                     src="https://circumeo.io/static/images/checkout.jpg" />
            </div>
            <h2 class="text-3xl font-bold text-gray-800 text-center mb-4">Victory! Your Order is Complete! 🏆</h2>
            <p class="text-gray-600 text-center mb-8">
                We're thrilled to confirm your order is successful! An email confirmation containing all the juicy details is winging its way to your inbox. Your total was ${{ total }}.
            </p>
            <div class="flex justify-center">
                <a class="inline-flex items-center px-6 py-3 bg-blue-600 hover:bg-blue-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-blue-500 text-white font-medium rounded-md shadow-sm transition duration-150 ease-in-out"
                   href="{% url 'shopping' %}">
                    <span class="mr-2">Continue Shopping</span>
                    <svg class="h-5 w-5"
                         fill="none"
                         stroke="currentColor"
                         viewbox="0 0 24 24"
                         xmlns="http://www.w3.org/2000/svg">
                        <path d="M16 11V7a4 4 0 00-8 0v4M5 9h14l1 12H4L5 9z" stroke-linecap="round" stroke-linejoin="round" stroke-width="2">
                        </path>
                    </svg>
                </a>
            </div>
        </div>
    </div>
{% endblock %}

Similarly, we'll need the checkout_canceled.html template to display something when a user cancels their checkout on the Stripe page.

{% extends "core/base.html" %}
{% block content %}
    <div class="bg-gradient-to-br from-red-50 to-yellow-100 min-h-screen flex flex-col items-center justify-center px-4 py-16">
        <div class="bg-white rounded-lg shadow-lg p-8 max-w-md mx-auto">
            <div class="flex justify-center mb-6">
                <img alt="A 3D rendering of a shopping cart tipped on its side, with its contents spilling out, set against a light gray background"
                     class="w-24 h-24 rounded-full object-cover object-center"
                     src="https://placehold.co/150" />
            </div>
            <h2 class="text-3xl font-bold text-gray-800 text-center mb-4">Order Cancelled 🛒</h2>
            <p class="text-gray-600 text-center mb-8">
                No worries, your order has been cancelled. You haven't been charged. Feel free to browse around and come back anytime!
            </p>
            <div class="flex justify-center">
                <a class="inline-flex items-center px-6 py-3 bg-blue-600 hover:bg-blue-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-blue-500 text-white font-medium rounded-md shadow-sm transition duration-150 ease-in-out"
                   href="{% url 'shopping' %}">
                    <span class="mr-2">Continue Shopping</span>
                    <svg class="h-5 w-5"
                         fill="none"
                         stroke="currentColor"
                         viewbox="0 0 24 24"
                         xmlns="http://www.w3.org/2000/svg">
                        <path d="M16 11V7a4 4 0 00-8 0v4M5 9h14l1 12H4L5 9z" stroke-linecap="round" stroke-linejoin="round" stroke-width="2">
                        </path>
                    </svg>
                </a>
            </div>
        </div>
    </div>
{% endblock %}

Create a partials directory within the existing core/templates directory structure.
Create a file named _cart.html within partials and enter the following.

{% load cart_tags %}
<div class="flex justify-end items-center p-4">
    <a href="{% url 'checkout' %}" class="{% if not cart.items.all %}pointer-events-none{% endif %}">
      <button class="flex items-center bg-blue-500 hover:bg-blue-600 text-white font-semibold py-2 px-4 rounded transition duration-300 ease-in-out {% if not cart.items.all %}opacity-50 cursor-not-allowed{% endif %}">
        <span class="text-lg">🛒</span>
        <span class="ml-2">Your Cart</span>
        {% if cart and cart.items.all %}
        <span class="ml-2 bg-red-600 text-white font-bold py-1 px-2 rounded-full text-xs">
          {{ cart|items_total }} Items
        </span>
        {% else %}
        <span class="ml-2 bg-red-600 text-white font-bold py-1 px-2 rounded-full text-xs">0 Items</span>
        {% endif %}
      </button>
    </a>
</div>

Adding the views

Copy and paste the following into views.py within the core directory.

import stripe

from django.shortcuts import render, redirect, get_object_or_404
from django.views.decorators.http import require_POST
from django.contrib import messages
from django.utils import timezone
from django.urls import reverse
from django.conf import settings

from djstripe.enums import APIKeyType
from djstripe.models import APIKey, Price, Product
from core.models import Cart

# Set the Stripe API key from settings
stripe.api_key = settings.STRIPE_TEST_SECRET_KEY

def shopping_view(request):
    """
    Displays a list of products available for purchase and the user's active shopping cart.
    """
    products = Product.objects.all()
    cart = None

    cart_id = request.session.get("cart_id")
    if cart_id:
        cart = Cart.objects.get(id=cart_id, is_active=True)
    return render(request, "core/shopping.html", {"cart": cart, "products": products})

def checkout_view(request):
    """
    Handles the checkout process, creating a Stripe checkout session for payment
    and directing the user to the payment page.
    """
    cart_id = request.session["cart_id"]
    cart = get_object_or_404(Cart, id=cart_id, is_active=True)

    if request.method == "POST":
        success_url = request.build_absolute_uri(reverse("checkout_success"))
        cancel_url = request.build_absolute_uri(reverse("checkout_canceled"))

        checkout_session = stripe.checkout.Session.create(
            success_url=success_url,
            cancel_url=cancel_url,
            mode="payment",
            line_items=[
                {"price": item.price.id, "quantity": item.quantity}
                for item in cart.items.all()
            ],
            payment_method_types=["card"],
        )

        return redirect("checkout_redirect", session_id=checkout_session["id"])

    total = sum(item.total_price() for item in cart.items.all())
    return render(request, "core/checkout.html", {"cart": cart, "total": total})

@require_POST
def add_to_cart_view(request):
    """
    Adds a product to the user's shopping cart from a POST request with product and quantity details.
    """
    price_id = request.POST.get("price_id")
    quantity = int(request.POST.get("quantity", 1))

    price = get_object_or_404(Price, id=price_id)

    cart_id = request.session.get("cart_id")
    if cart_id:
        cart = Cart.objects.get(id=cart_id, is_active=True)
    else:
        cart = Cart.objects.create(created_at=timezone.now())
        request.session["cart_id"] = cart.id

    cart.add_product(price.product, price, quantity)

    messages.success(request, "Item added successfully to shopping cart!")
    return redirect("shopping")

def checkout_redirect_view(request, session_id):
    """
    Redirects the user to the Stripe checkout session for payment, passing the necessary session details.
    """
    public_keys = APIKey.objects.filter(type=APIKeyType.publishable)[:1]
    return render(
        request,
        "core/checkout_redirect.html",
        {
            "stripe_public_key": public_keys.get().secret,
            "checkout_session_id": session_id,
        },
    )

def checkout_success_view(request):
    """
    Displays a success message and order summary after a successful checkout, clearing the cart.
    """
    cart_id = request.session.get("cart_id")
    if cart_id:
        cart = Cart.objects.get(id=cart_id, is_active=True)
        total = sum(item.total_price() for item in cart.items.all())
        cart.delete()
        request.session.pop("cart_id")
    else:
        total = 0
    return render(request, "core/checkout_success.html", {"total": total})

def checkout_canceled_view(request):
    """
    Handles the scenario where a checkout is canceled, clearing the cart and redirecting the user.
    """
    cart_id = request.session.get("cart_id")
    if cart_id:
        cart = Cart.objects.get(id=cart_id, is_active=True)
        cart.delete()
        request.session.pop("cart_id")
    return render(request, "core/checkout_canceled.html")

Updating URLs

Create urls.py in the core directory.

from django.urls import include, path
from core.views import (
    shopping_view,
    checkout_redirect_view,
    checkout_view,
    checkout_success_view,
    checkout_canceled_view,
    add_to_cart_view,
)

urlpatterns = [
    path("", shopping_view, name="shopping"),
    path(
        "checkout/redirect/<str:session_id>",
        checkout_redirect_view,
        name="checkout_redirect",
    ),
    path("checkout", checkout_view, name="checkout"),
    path("checkout/success/", checkout_success_view, name="checkout_success"),
    path("checkout/canceled/", checkout_canceled_view, name="checkout_canceled"),
    path("add-to-cart", add_to_cart_view, name="add_to_cart"),
    path("stripe/", include("djstripe.urls", namespace="djstripe")),
]

Update the existing urls.py within the project ecommerce directory.

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]

Adding the database models

Overwrite the existing models.py with the following:

from decimal import Decimal
from django.db import models

from djstripe.models import Product, Price


class Cart(models.Model):
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)
    is_active = models.BooleanField(default=True)

    def add_product(self, product, price, quantity=1):
        cart_item, created = CartItem.objects.get_or_create(
            cart=self,
            product=product,
            price=price,
            defaults={"quantity": quantity, "product": product, "price": price},
        )

        if not created:
            cart_item.quantity += quantity
            cart_item.save()


class CartItem(models.Model):
    cart = models.ForeignKey(Cart, on_delete=models.CASCADE, related_name="items")
    product = models.ForeignKey(Product, on_delete=models.CASCADE)
    price = models.ForeignKey(Price, on_delete=models.CASCADE)
    quantity = models.PositiveIntegerField(default=1)

    def unit_price(self):
        return Decimal(self.price.unit_amount) / Decimal(100)

    def total_price(self):
        dollar_amount = Decimal(self.price.unit_amount) / Decimal(100)
        return dollar_amount * self.quantity

Setting up Products and Prices

Head to Stripe and register if you haven't already. We can use the Stripe API in Test Mode to build the e-commerce app. You can add a bank account and get verified later when you're ready to start collecting real payments.

Once you're registered, go to the Product Catalog to start adding products and prices.

We only need a few products to test the code.

The following snippet is a list of test products that I used when making the e-commerce app.

products = [
            {
                "name": "UltraView HD 4K LED Smart TV - 55 Inches",
                "description": "Experience cinematic quality with our UltraView HD 4K Smart TV. Featuring a 55-inch LED display, HDR10+ support, and smart connectivity to stream your favorite shows and apps directly. With voice control and a sleek, bezel-less design, this TV is the perfect centerpiece for your home entertainment system.",
                "price": "450.99",
                "image_url": "https://circumeo.io/static/images/hdtv.jpg"
            },
            {
                "name": "EchoBeat Wireless Noise Cancelling Headphones",
                "description": "Dive into an ocean of sound with EchoBeat Wireless Headphones. Equipped with advanced noise cancelling technology, long-lasting battery life, and seamless Bluetooth connectivity. Enjoy high-fidelity audio, comfortable ear cups, and a lightweight design for hours of musical bliss.",
                "price": "129.99",
                "image_url": "https://circumeo.io/static/images/headphones.jpg"
            },
            {
                "name": "PhotonBeam Mini Projector - Portable HD",
                "description": "Bring the big screen anywhere with the PhotonBeam Mini Projector. This compact, portable HD projector offers vibrant visuals and incredible clarity. Easy to set up and compatible with multiple devices, it's ideal for home theaters, outdoor movies, or business presentations.",
                "price": "199.99",
                "image_url": "https://circumeo.io/static/images/projector.jpg"
            },
            {
                "name": "VirtualWave Immersive VR Headset",
                "description": "Step into new worlds with the VirtualWave VR Headset. Featuring cutting-edge technology for immersive virtual reality experiences, comfortable fit, and easy integration with your smartphone or PC. Perfect for gaming, educational content, or exploring virtual landscapes.",
                "price": "299.99",
                "image_url": "https://circumeo.io/static/images/vr_headset.jpg"
            },
            {
                "name": "QuantumLeap Smart Fitness Watch",
                "description": "Track your health and fitness goals with the QuantumLeap Smart Fitness Watch. This device offers real-time monitoring of your heart rate, steps, and sleep patterns. Features GPS tracking, water resistance, and a durable touchscreen. Syncs seamlessly with your mobile devices to keep you informed and motivated throughout the day.",
                "price": "249.99",
                "image_url": "https://circumeo.io/static/images/smartwatch.jpg"
            },
            {
                "name": "PulseMax Portable Bluetooth Speaker",
                "description": "Bring your music to life with the PulseMax Portable Bluetooth Speaker. Offers high-quality sound with deep bass and crystal-clear highs. It's waterproof and dustproof, making it perfect for outdoor adventures. Features a long-lasting battery and a compact design for easy portability.",
                "price": "89.99",
                "image_url": "https://circumeo.io/static/images/bluetooth_speaker.jpg"
            },
            {
                "name": "Starlight 360 Home Security Camera",
                "description": "Keep your home secure with the Starlight 360 Home Security Camera. Features 360-degree video surveillance with night vision, motion detection, and remote viewing via a secure app. Weather-resistant and easy to install, providing peace of mind for any homeowner.",
                "price": "159.99",
                "image_url": "https://circumeo.io/static/images/security_camera.jpg"
            },
            {
                "name": "GlideTech Electric Scooter",
                "description": "Zip around town on the GlideTech Electric Scooter. It's eco-friendly, featuring a powerful electric motor that reaches speeds up to 20 mph. Comes with a foldable design, LED headlights, and a digital display dashboard. Ideal for quick commutes and fun rides.",
                "price": "329.99",
                "image_url": "https://circumeo.io/static/images/scooter.jpg"
            }
        ]

Stripe API Keys and Webhook Secret

The dj-stripe package uses the API keys and webhook secret to enable sync between Stripe and our Django database. This means that we'll make any product or price updates in Stripe, and through web-hooks, Stripe will notify our app of the changes.

This video shows the process of setting up the Django admin dashboard, registering the Stripe API keys, and creating a webhook for dj-stripe to use.

Once the API keys and web-hook are established, you can trigger a manual sync of data from Stripe into the Django database.

python3 manage.py djstripe_sync_models

The sync will take a moment. Once the sync is complete, all products and prices should be copied from Stripe.

You're ready to start charging customers!

Customers can now add items to their cart and checkout.

You can also make changes from the Stripe dashboard and see them reflected within the Django app. The dj-stripe package handles the web-hooks from Stripe and updates the database.

Other ways to implement payments

If you'd like to learn more about Django and Stripe, Tom over at PhotonDesigner has also written a great article on the subject.

Add Stripe subscriptions to Django in 7 minutes 💵

Handling Big Data with Django and AgGrid

Zach — Wed, 10 Apr 2024 19:53:44 +0000

We're going to build an app that can handle filtering and sorting large amounts of data with AgGrid.

Sending all of your data to the browser is a quick way to get started with AgGrid, but what if you have a million rows? That's when server-side filtering and sorting come to the rescue.

This guide includes:

using AgGrid in infinite scroll mode so that the server is responsible for paginating, filtering, and sorting
implementing filtering, sorting, and pagination for AgGrid on top of a Django ListView class
creating a Django management command to populate test data

But first, here's a quick peek at what we're building today:

Previewing the project

Want to see a live version of the app? You can create a copy of this project now and try it out.

Create a fork of the AgGridDemo project on Circumeo.

Setting up the Django app

Install packages and create the Django application.

pip install --upgrade django faker
django-admin startproject bigdata .
python3 manage.py startapp core

Add core to the INSTALLED_APPS list.

# settings.py
INSTALLED_APPS = [
  "core",
    ...
]

Adding the templates

Create a directory named templates within the core app.
Create a file named index.html within the templates directory.

<!DOCTYPE html>
<html>
  <head>
    <script src="https://unpkg.com/ag-grid/dist/ag-grid.min.noStyle.js"></script>
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-grid.css">
    <link rel="stylesheet" href="https://unpkg.com/ag-grid/dist/styles/ag-theme-balham.css">
    <style>
      #container {
        max-width: 1250px;
        width: 100%;
        height: 100%;
        margin: 0px auto;
        margin-top: 30px;
      }
    </style>
  </head>
  <body>
    <div id="container">
      <div id="data-grid" style="height: 600px; width: 100%" class="ag-theme-balham"></div>
    </div>
    <script type="text/javascript" charset="utf-8">
      const numberFilterParams = {
        filterParams: {
          filterOptions: ["equals", "notEqual", "lessThan", "greaterThan"],
          suppressAndOrCondition: true,
        },
      };

      var columnDefs = [
        { headerName: "Name", field: "name", filter: "agTextColumnFilter" },
        {
          headerName: "Description",
          field: "description",
          filter: "agTextColumnFilter",
        },
        { headerName: "Category", field: "category", filter: "agTextColumnFilter" },
        {
          headerName: "Price",
          field: "price",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Stock Quantity",
          field: "stock_quantity",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
        {
          headerName: "Rating",
          field: "rating",
          filter: "agNumberColumnFilter",
          ...numberFilterParams,
        },
      ];

      var gridOptions = {
        columnDefs: columnDefs,
        defaultColDef: {
          filterParams: {
            suppressAndOrCondition: true,
          },
        },
        enableServerSideSorting: true,
        enableServerSideFilter: true,
        rowModelType: "infinite",
        cacheBlockSize: 100,
        maxBlocksInCache: 10,
      };

      var dataSource = {
        getRows: function (params) {
          var filtering = encodeURIComponent(JSON.stringify(params.filterModel));
          var sorting = encodeURIComponent(JSON.stringify(params.sortModel));

          var startRow = params.startRow;
          var endRow = params.endRow;

          var url = `/products?startRow=${startRow}&endRow=${endRow}&filter=${filtering}&sort=${sorting}`;

          fetch(url)
            .then((response) => response.json())
            .then((data) => {
            params.successCallback(data.rows, data.totalRows);
          })
            .catch((err) => {
            params.failCallback();
          });
        },
      };

      var gridDiv = document.querySelector("#data-grid");

      new agGrid.Grid(gridDiv, gridOptions);
      gridOptions.api.setDatasource(dataSource);
      gridOptions.api.sizeColumnsToFit();
    </script>
  </body>
</html>

Adding the views

Remove the existing views.py file.
Create a new directory named views within the core folder.
Create a new file named index.py within the views directory.
Copy and paste the following into index.py within the views directory.

from django.shortcuts import render

def index_view(request):
    return render(request, "core/index.html")

Create another file named products.py within the same directory.
Copy and paste the following into the products.py file.

import json

from django.views.generic.list import ListView
from django.http import JsonResponse
from django.db.models import Q

from core.models import Product


class ProductListView(ListView):
    model = Product

    def get_queryset(self):
        """
        Convert AgGrid filter and sort objects into a Django query.
        An example filter:
            {"category": {"type": "contains", "filter": "electronics"}}
        """
        queryset = super().get_queryset()

        filter_params = self.request.GET.get("filter", None)
        if filter_params:
            filters = json.loads(filter_params)
            q_objects = Q()

            for key, filter_info in filters.items():
                filter_type = filter_info.get("type")
                filter_value = filter_info.get("filter")

                if filter_type == "contains":
                    lookup = f"{key}__icontains"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "equals":
                    lookup = f"{key}__exact"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "notEqual":
                    lookup = f"{key}__exact"
                    q_objects &= ~Q(**{lookup: filter_value})
                elif filter_type == "greaterThan":
                    lookup = f"{key}__gt"
                    q_objects &= Q(**{lookup: filter_value})
                elif filter_type == "lessThan":
                    lookup = f"{key}__lt"
                    q_objects &= Q(**{lookup: filter_value})

            queryset = queryset.filter(q_objects)

        sort_params = self.request.GET.get("sort", None)
        if sort_params:
            sort_objects = json.loads(sort_params)
            sort_fields = []
            for sort_object in sort_objects:
                col_id = sort_object["colId"]
                sort_order = sort_object["sort"]
                if sort_order == "asc":
                    sort_fields.append(col_id)
                elif sort_order == "desc":
                    sort_fields.append(f"-{col_id}")

            if sort_fields:
                queryset = queryset.order_by(*sort_fields)
        return queryset

    def get(self, request, *args, **kwargs):
        start_row = int(request.GET.get("startRow", 0))
        end_row = int(request.GET.get("endRow", 100))

        queryset = self.get_queryset()

        total_rows = queryset.count()
        queryset = queryset[start_row:end_row]

        products = list(
            queryset.values(
                "name", "description", "category", "price", "stock_quantity", "rating"
            )
        )

        return JsonResponse({"rows": products, "totalRows": total_rows})

Updating URLs

Create urls.py in the core directory.

from django.urls import path

from core.views.index import index_view
from core.views.products import ProductListView

urlpatterns = [
    path("", index_view, name="index"),
    path("products", ProductListView.as_view(), name="products"),
]

Update the existing urls.py within the project bigdata directory.

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]

Adding the database models

Overwrite the existing models.py with the following:

from django.db import models

class Product(models.Model):
    name = models.CharField(max_length=255)
    description = models.TextField(blank=True)
    category = models.CharField(max_length=255)
    price = models.DecimalField(max_digits=10, decimal_places=2)
    stock_quantity = models.IntegerField()
    rating = models.FloatField(blank=True, null=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

Adding a Django Management Command for Test Data

Create the directory structure management/commands within the core folder.
Open a file named populate_products.py within the new directory and enter the following.

from faker import Faker
import random

from django.core.management.base import BaseCommand
from core.models import Product

class Command(BaseCommand):
    def handle(self, *args, **kwargs):
        faker = Faker()

        categories = [
            'Electronics',
            'Books',
            'Clothing',
            'Home & Garden',
            'Toys & Games',
            'Sports & Outdoors',
            'Health & Beauty',
            'Automotive',
            'Groceries',
            'Pet Supplies'
        ]

        for _ in range(1000):
            Product.objects.create(
                name=faker.text(max_nb_chars=20).capitalize(),
                description=faker.text(),
                category=random.choice(categories),
                price=round(random.uniform(5.0, 500.0), 2),
                stock_quantity=random.randint(0, 100),
                rating=round(random.uniform(1.0, 5.0), 1)
            )

        self.stdout.write(self.style.SUCCESS('Successfully populated the database with products.'))

Open a shell session in order to run the Django management command.

python3 manage.py populate_products

Up and Running with Django and AgGrid

You're now ready to handle millions of rows without crashing browser tabs! Add database indexes where needed, and you can scale this approach pretty far.

Build a Live Search Feature with Django using HTMX and PostgreSQL FTS (in 10 minutes)

Zach — Sat, 06 Apr 2024 11:46:41 +0000

We're going to build a simple app that uses advanced PostgreSQL full-text search features, with results that update as the user enters their query.

This includes:

using HTMX to fetch results as the user types (without extra JavaScript)
allowing the user to enter search engine style queries (i.e. "python or django")
adding a Django management command to quickly insert test data

Here's an image of the working product:

Previewing the project

Want to see a live version of the app? You can create a copy of this project now and try it out.

Copy the FullTextSearchApp Project

Setting up the Django app

Install packages and create the Django application.

pip install --upgrade django faker
django-admin startproject eventsearch .
python3 manage.py startapp core

Add core to the INSTALLED_APPS list.

# settings.py
INSTALLED_APPS = [
  "core",
    ...
]

Adding the templates

Create a directory named templates within the core app.
Create a file named search.html within the templates directory.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <script src="https://unpkg.com/htmx.org"></script>
        <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css"
              rel="stylesheet"
              integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC"
              crossorigin="anonymous">
        <link href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.10.0/font/bootstrap-icons.css"
              rel="stylesheet">
        <style>
      html,
      body {
        height: 100%;  
      }

      #content {
        display: flex;
        flex-direction: column;
        height: 100%;
      }

      #events-content {
        background-color: rgb(246, 247, 248);
        flex: 1;
      }

      #events-content-inner {
        max-width: 600px;
        width: 100%;
      }

      #event-search-input {
        border-top-left-radius: 8px;
        border-bottom-left-radius: 8px;
        border: 1px solid rgb(217, 217, 217);
        border-right: none;
        max-width: 600px;
        outline: none;
      }

      #search-submit-btn {
        background-color: rgb(246, 88, 88);
        border-top-right-radius: 8px;
        border-bottom-right-radius: 8px;
        border: none;
        color: #fff;
        width: 42px;
      }

      .search-result {
        border-bottom: 1px solid #ccc;
      }

      .search-result-inner {
         display: flex;
         gap: 1rem;
      }

      .event-thumbnail {
         border-radius: 8px;
         width: 160px;
         height: 100px;
      }

      .event-description {
         font-size: 13px;  
      }

       .event-date {
         color: rgb(124, 111, 80);
         font-weight: 500;
       }
        </style>
    </head>
    <body>
        <div id="content">
            <form method="post">
                {% csrf_token %}
                <div id="event-search" class="d-flex justify-content-center p-3">
                    <div class="d-flex justify-content-center w-50">
                        <input id="event-search-input"
                               name="q"
                               class="w-100 p-2"
                               type="text"
                               value="{{ query }}"
                               placeholder="Search events"
                               hx-post="/"
                               hx-select="#events-content-inner"
                               hx-swap="outerHTML"
                               hx-trigger="input changed delay:500ms, search"
                               hx-target="#events-content-inner" />
                        <button id="search-submit-btn" type="submit">
                            <i class="bi bi-search"></i>
                        </button>
                    </div>
                </div>
            </form>
            <div id="events-content" class="d-flex justify-content-center">
                <div id="events-content-inner" class="p-4">
                    {% for event in events %}
                        <div class="search-result py-4">
                            <div class="search-result-inner">
                                <div>
                                    {% if event.image_url %}<img class="event-thumbnail" src="{{ event.image_url }}" />{% endif %}
                                </div>
                                <div>
                                    <div class="event-date">{{ event.event_date|date:"l, F d" }} {{ event.event_time }}</div>
                                    <div>{{ event.name }}</div>
                                    <div class="event-description mt-2">{{ event.description }}</div>
                                </div>
                            </div>
                        </div>
                    {% empty %}
                    <div class="mx-auto text-center py-4">
                      <p>No events found.  Have you run the management command yet?</p>  
                      <p>Click Shell in the IDE and then Connect.</p>
                      <pre class="mt-5">python3 manage.py populate_events</pre>
                    </div>
                    {% endfor %}
                </div>
            </div>
        </div>
    </body>
</html>

Adding the views

Copy and paste the following into views.py within the core directory.

from django.contrib.postgres.search import SearchQuery
from django.shortcuts import render
from core.models import Event


def search(request):
    events, q = [], ""

    if request.method == "POST":
        q = request.POST.get("q")
        if q:
            events = Event.objects.filter(
                description_vector=SearchQuery(q, search_type="websearch")
            )

    if not events:
        events = Event.objects.all()

    return render(request, "core/search.html", {"query": q, "events": events})

Updating URLs

Create urls.py in the core directory.

from django.urls import path
from .views import search

urlpatterns = [
    path("", search, name="search")
]

Update the existing urls.py within the project eventsearch directory.

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]

Adding the database models

Overwrite the existing models.py with the following:

from psycopg2.extensions import register_adapter, AsIs

from django.db import models
from django.db.models import F
from django.contrib.auth.models import User

from django.contrib.postgres.search import SearchVectorField
from django.contrib.postgres.indexes import GinIndex

class PostgresDefaultValueType:
    pass

# Wrap the built-in SearchVectorField so that we pass DEFAULT when value is None.
# Otherwise, the Django ORM will try to update the `description_vector` field, which
# PostgreSQL won't allow because it is a GENERATED column.

# Credit for this approach belongs here:
# https://code.djangoproject.com/ticket/21454#comment:28
class DelegatedSearchVectorField(SearchVectorField):
    def get_prep_value(self, value):
        return value or PostgresDefaultValueType()

register_adapter(PostgresDefaultValueType, lambda _: AsIs('DEFAULT'))

class Event(models.Model):
    name = models.CharField(max_length=100)
    description = models.TextField()
    description_vector = DelegatedSearchVectorField()

    image_url = models.URLField(max_length=1024, blank=True, null=True)
    event_date = models.DateField()
    event_time = models.TimeField()

Adding a Manual Migration for the Search Vector Field

Run the migration command to generate a new empty migration.

python3 manage.py makemigrations core --empty

Populate the empty migration with the following.

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('core', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            """
            ALTER TABLE core_event ADD COLUMN description_vector tsvector GENERATED ALWAYS AS (
                setweight(to_tsvector('english', coalesce(name, '')), 'A') ||
                setweight(to_tsvector('english', coalesce(description, '')), 'B')
            ) STORED;
            CREATE INDEX IF NOT EXISTS description_vector_gin ON core_event USING GIN(description_vector);
            """,
            reverse_sql=
            """
            ALTER TABLE core_event DROP COLUMN description_vector;
            """,
        ),
    ]

The migration will create an automatically updated tsvector by using a GENERATED column. Django 5.0 supports the GeneratedField type, but the SQL expression is too complex to handle that way, so we'll use a manual migration to handle it without the ORM.

Adding a Django Management Command for Test Data

Create the directory structure management/commands within the core folder.
Open a file named populate_events.py within the new directory and enter the following.

from datetime import datetime, timedelta
import random

from django.core.management.base import BaseCommand
from django.contrib.auth.models import User
from django.utils.timezone import make_aware
from faker import Faker

from core.models import Event

class Command(BaseCommand):
    def handle(self, *args, **options):
        self.stdout.write(self.style.SUCCESS('Starting database population...'))

        faker = Faker()

        for i in range(10):
            first_name = faker.first_name()
            last_name = faker.last_name()
            User.objects.create_user(username=f'user{i}',
                                     email=f'user{i}@example.com',
                                     password='password',
                                     first_name=first_name,
                                     last_name=last_name)

        event_names = [
            'Django for Beginners',
            'Python Data Science Meetup',
            'React Native Workshop',
            'Machine Learning Basics',
            'Introduction to IoT',
            'Blockchain 101',
            'Advanced Django Techniques',
            'Full-stack Development with Python and JavaScript',
            'Mobile App Development Trends',
            'Cybersecurity Fundamentals',
        ]

        descriptions = [
            'Learn Django through practical examples, from setting up your environment to building and deploying a web application.',
            'Explore data science techniques using Python, including statistical analysis, machine learning algorithms, and data visualization.',
            'Build your first React Native app by learning the basics of this cross-platform framework and creating a simple application.',
            'An introduction to machine learning concepts, covering foundational principles, algorithms, and real-world applications.',
            'Getting started with the Internet of Things (IoT): Learn how to connect devices and collect data for analysis and insights.',
            'Understanding blockchain technology, from the basics of decentralized networks to smart contracts and cryptocurrency.',
            'Deep dive into Django features, exploring advanced functionalities that can help in building robust and scalable web applications.',
            'Learn full-stack development from front to back, covering everything from basic HTML/CSS to backend programming and databases.',
            'What\'s new in mobile app development: Explore the latest trends, tools, and technologies shaping the future of mobile applications.',
            'Protect your digital assets with cybersecurity basics, learning about threats, safeguards, and best practices for online security.',
        ]

        image_urls = [
          'https://circumeo.io/static/images/server-racks.png',
          'https://circumeo.io/static/images/typewriter.jpg',
          'https://circumeo.io/static/images/library.jpg',
          'https://circumeo.io/static/images/landscape.jpg',
          'https://circumeo.io/static/images/laptop.jpg',
          'https://circumeo.io/static/images/latte.jpg',
        ]

        for name, description in zip(event_names, descriptions):
            image_url = random.choice(image_urls)

            event_date = datetime.now() + timedelta(days=random.randint(1, 90))
            event_time = datetime.now().time()

            Event.objects.create(name=name, description=description, image_url=image_url, event_date=make_aware(event_date), event_time=event_time)

        self.stdout.write(self.style.SUCCESS('Database population completed successfully!'))

Open a shell session in order to run the Django management command.

python3 manage.py populate_events

Up and Running with Full Text Search

That's it! You now have an app up and running that incorporates powerful full-text search capabilities. Not to mention, the search is dynamic as well, without the need for additional JavaScript, thanks to HTMX.

Monitoring Celery in Production

Zach — Mon, 22 Jan 2024 13:19:21 +0000

How can you determine if your Celery queues are healthy or overwhelmed with tasks?

Anything from sudden surges of user traffic to abuse of your app can drag down your Celery cluster. Knowing when this is happening is half the battle.

Our goal in this series is to build automated monitoring tools for Celery so that you don't have to find out about problems from your users.

Monitoring Queue Wait Times

There are many valuable metrics to monitor such as queue length, processing time, failed task count, etc.

But we're going to start with what I think is the most effective metric to monitor: queue wait times.

Queue wait time is simply how long a task spends waiting in the queue before a worker accepts it and begins processing.

Queue length is important, but harder to judge. What constitutes too many tasks in the queue? For one service, having 10 items in the queue might be too many, while 100,000 is perfectly normal for another service.

It's also easy to have short bursts where the queue grows, but as long as the workers process them quickly enough, is it really an issue? This can lead to false alarms.

Queue wait times, on the other hand, are a true reflection of your user's experience.

Let's say you have a Django app that auto-tunes videos of people singing. You also implemented a helpful progress bar. A user uploads a video of themselves singing a Beyonce song, but the progress bar doesn't budge for several minutes. That user gives up, and tells all their friends that the app doesn't work.

What happened? Tasks may have still been processing, but the queue wait time was too high.

Setting Up the Example Django App

Let's build a very simple Django app to demonstrate how to monitor Celery queue wait times.

We'll use Docker to make it simpler to set up Celery and Redis.

Create the Django project.

django-admin startproject djautotune
cd djautotune
python3 manage.py startapp core

Update the djautotune/settings.py file to add the core application. We'll also add the Celery related settings while we're editing the file.

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "core",
]

REDIS_HOST = os.environ["REDIS_HOST"]
REDIS_PORT = os.environ["REDIS_PORT"]

CELERY_METRICS_TTL = int(os.environ.get("CELERY_METRICS_TTL", 300))

CELERY_IMPORTS = ("core.tasks",)
CELERY_BROKER_URL = os.environ["CELERY_BROKER_URL"]
CELERY_RESULT_BACKEND = os.environ["CELERY_RESULT_BACKEND"]
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"

Adding the Celery Worker Entrypoint

Create a new file named celery.py in the same directory as the settings.py file. This file acts as the entrypoint for the Celery worker.

import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djautotune.settings")

app = Celery("djautotune")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Implementing the Background Task Logic

Now create the core/tasks.py file to contain our task logic.

Since actually auto-tuning videos is beyond the scope of this tutorial, we'll add some dummy code that simulates the workload.

import random
import time

from celery import shared_task

@shared_task
def autotune():
    short_duration = random.uniform(0.5, 2)  # Short duration for most tasks
    medium_duration = random.uniform(2, 5)  # Medium duration for 10% of tasks
    long_duration = random.uniform(5, 10)  # Long duration for 1% of tasks (P99)

    probability = random.random()

    if probability < 0.90:
        time_to_sleep = short_duration
    elif probability < 0.99:
        time_to_sleep = medium_duration
    else:
        time_to_sleep = long_duration

    time.sleep(time_to_sleep)

Creating a View to Invoke the Background Task

We'll need a way to invoke this new autotune Celery task. Open core/views.py to add the following HTTP endpoint.

from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from .tasks import autotune

@csrf_exempt
def enqueue_autotune_task(request):
    if request.method == "POST":
        result = autotune.delay()
        return JsonResponse(
            {"task_id": result.task_id, "status": "Task submitted successfully!"},
            status=200,
        )
    else:
        return JsonResponse({"error": "Only POST method is allowed."}, status=405)

Now, as usual, you'll need to add the new route to your Django router config.

To start, add the following in the core/urls.py file.

from django.urls import path
from . import views

urlpatterns = [
    path("autotune/", views.enqueue_autotune_task, name="enqueue_autotune_task"),
]

Finally, update the main urls.py file to include the routes from the core app.

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("core.urls")),
]

Setting Up the Docker Development Environment

Create the requirements.txt file.

Django==4.2.4
redis==4.5.5
celery==5.2.7

Create the Dockerfile for the Django app in the root project directory.

FROM python:3.11

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

WORKDIR /usr/src/app

COPY requirements.txt /usr/src/app/

RUN pip install --no-cache-dir -r requirements.txt

COPY . /usr/src/app/

RUN mkdir -p /usr/src/app/

Now create the docker-compose.yml to manage the containers.

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ${PWD}:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - CELERY_METRICS_TTL=300
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    ports:
      - "8000:8000"
    depends_on:
      - redis

  celery:
    build:
      context: .
      dockerfile: Dockerfile
    command: celery -A djautotune worker --loglevel=info
    volumes:
      - ${PWD}:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
      - CELERY_METRICS_TTL=300
      - REDIS_HOST=redis
      - REDIS_PORT=6379
    depends_on:
      - redis

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

Starting Up and Testing the App

You should be ready to start the app now with the docker-compose up command.

If all goes well, you can enqueue a task with the following curl command.

curl -X POST http://localhost:8000/autotune/

Using the docker logs command on the Celery container, you should be able to see a success message like this for the task.

Task core.tasks.autotune[9cdfcb7d-1f65-49cc-aa33-20c694157e61] succeeded in 1.9952244879677892s: None

Adding the Celery Monitoring Logic

We have a functioning background task, but no monitoring to speak of.

To add monitoring, we'll create a Django view that returns the longest queue wait time in the past 30 seconds. The window of time can be easily adjusted, but it's important that we use a sliding window of some length.

We'll use a Redis sorted set with an expiration to create the sliding window of queue wait times.

Marking New Tasks with a Start Timestamp

The first step is to mark each new task with a timestamp.

This timestamp represents the moment the task was created and published, but before any worker has picked up the task.

Open core/apps.py and add the following.

import time
from django.apps import AppConfig
from celery.signals import before_task_publish

def before_task_publish_handler(sender=None, headers=None, body=None, **kwargs):
    headers["__metadata__"] = {"publish_timestamp": time.time()}

class CoreConfig(AppConfig):
    default_auto_field = "django.db.models.BigAutoField"
    name = "core"

    def ready(self):
        before_task_publish.connect(before_task_publish_handler)

Recording Queue Wait Time in Redis

Once a worker has picked up the task, we need to calculate the elapsed time and store it in Redis.

Since we're using the task_prerun signal, this represents the moment the task was picked up, and does not include actual processing time.

Open djautotune/celery.py and add the following.

import time
import os
import redis

from celery import Celery
from celery.signals import task_prerun
from django.conf import settings

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djautotune.settings")

app = Celery("djautotune")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

redis_client = redis.Redis(host=settings.REDIS_HOST, port=settings.REDIS_PORT, db=0)

@task_prerun.connect
def task_prerun_handler(task_id, task, *args, **kwargs):
    metadata = getattr(task.request, "__metadata__", {})
    publish_timestamp = metadata.get("publish_timestamp")

    if publish_timestamp:
        elapsed_time = time.time() - publish_timestamp

        redis_client.zadd("task_elapsed_times", {task_id: elapsed_time})
        redis_client.expire("task_elapsed_times", settings.CELERY_METRICS_TTL, nx=True)

The zadd call adds the queue wait time to the sorted set. The set is sorted by the queue wait time, so it's efficient to request the greatest value, which will be important when we add a view to ask for this value.

Because the expire call is passed nx=True, only the initial call sets the expiration. In other words, even though expire is called for each task, the expiration is not reset. Eventually, the key task_elapsed_times will expire, and an entirely new sorted set is created with a new expiration.

Requesting the Longest Queue Wait Time

For the final piece of the puzzle, let's create a view to return the longest queue wait in the current time window (as defined by the CELERY_METRICS_TTL setting).

Open core/views.py and add the following view code.

import redis

from django.conf import settings
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from .tasks import autotune

def max_queue_wait_time(request):
    redis_client = redis.Redis(host=settings.REDIS_HOST, port=settings.REDIS_PORT, db=0)

    max_time_data = redis_client.zrevrange("task_elapsed_times", 0, 0, withscores=True)
    max_elapsed_time = max_time_data[0][1] if max_time_data else 0

    return JsonResponse({"max_elapsed_time": max_elapsed_time})

Setting Up Alerting for Celery

We'll use Cronitor to set up alerting so that we receive a notification when queue wait times become too high.

To do this, Cronitor will make requests to our max_queue_wait_time endpoint and parse the JSON response.

This means that Cronitor must have an endpoint that it can reach. Normally, we can't do that when developing on a personal machine. For this tutorial, however, we can use ngrok to establish a tunnel to our local Django application for testing purposes.

Using Ngrok to Establish a Tunnel

Head over to ngrok and sign up for a free account.

Download the CLI tool, enter your token, and open a tunnel with the following command.

ngrok http 8000

Copy the domain that is listed next to the Forwarding column.

Adding the Celery Check in Cronitor

Add a new Check with details similar to the following. Remember to add your own Ngrok domain.

Take note of the ngrok-skip-browser-warning header and be sure to add that.

Under the assertions section, set up the following so that Cronitor understands when to trigger an alert.

This means that Cronitor will fire an alert when max_elapsed_time exceeds 10 seconds. You can of course adjust this time to whatever metric suits your application.

Here's an example of what happened when I added a huge number of tasks so that my Celery queue became overwhelmed.

Further Celery Monitoring Enhancements

You've learned how to monitor queue wait time, a metric that directly reflects what your users are experiencing.

This is a great start, but there's a lot more to monitoring your service comprehensively.

As a sneak peek, in the next article we'll learn how to monitor more Celery stats, such as failed tasks and retries. We'll also dive into building a custom Celery monitoring dashboard using Bokeh.

You can see the entire djautotune sample project on GitHub if you'd like to copy some of the code as a template.

Draw Together: Building a Multiplayer Sketch Game with Daphne, NextJS & OpenAI!

Zach — Sun, 14 Jan 2024 14:26:07 +0000

Let's use the OpenAI API to build a multiplayer sketching game!

In this guide, we'll build a Django and NextJS app that allows two players to connect and share a sketch. OpenAI will supply the drawing prompts that we'll give to the players.

If you're more of a visual learner, I've also made a video where I give an overview of the tutorial.

Django App Setup

Creating the Django project.

django-admin startproject djdoodle
cd djdoodle/
python3 manage.py startapp core

Updating the djdoodle/settings.py file to add the core application.

INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "core",
]

Creating the requirements.txt file.

Django==4.1.7
daphne==4.0.0
channels==4.0.0
channels-redis==4.1.0
websockets==11.0.3
openai==1.6.1

Creating the Docker file for the Django app.

FROM python:3.11

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

WORKDIR /usr/src/app

COPY requirements.txt /usr/src/app/

RUN pip install --no-cache-dir -r requirements.txt

COPY . /usr/src/app/

RUN mkdir -p /usr/src/app/

Now we'll create the initial docker-compose.yml to manage the containers. We'll eventually add NextJS, but let's start with Django.

version: "3.7"

services:
  daphne:
    build:
      context: .
    command: daphne -b 0.0.0.0 -p 8000 djdoodle.asgi:application
    ports:
      - "8000:8000"
    volumes:
      - $PWD:/usr/src/app

The build and context keys tell Compose that it can find our Dockerfile in the current directory.

Building the Django app image.

docker-compose build

Running the initial version as a test.

docker-compose up -d
docker ps

You should see output similar to this.

CONTAINER ID   IMAGE              COMMAND                  CREATED         STATUS        PORTS                                       NAMES
555ef0c2ad40   djdoodle-daphne   "daphne -b 0.0.0.0 -…"   2 seconds ago   Up 1 second   0.0.0.0:8000->8000/tcp, :::8000->8000/tcp   djdoodle-daphne-1

NextJS App Setup

Inside the base project directory, create the NextJS application.

Name the app djdoodle-ui and select JavaScript as the language when prompted.

npx create-next-app@latest

Creating the Dockerfile for the NextJS app. The file should be under the djdoodle-ui directory.

FROM node:18-alpine

WORKDIR /usr/src/app

COPY package.json ./

RUN npm install

COPY . .

CMD ["npm", "run", "dev"]

Now we update the docker-compose.yml file to include the NextJS app.

version: "3.7"

services:
  daphne:
    build:
      context: .
    command: daphne -b 0.0.0.0 -p 8000 djdoodle.asgi:application
    ports:
      - "8000:8000"
    volumes:
      - $PWD:/usr/src/app

  nextjs:
    build:
      context: ./djdoodle-ui
    ports:
      - "3000:3000"
    volumes:
      - $PWD/djdoodle-ui:/usr/src/app

Build and start the new NextJS image. Run the following in the project base directory.

docker-compose build
docker-compose up -d

You should now be able to access Django at http://localhost:8000 and NextJS at http://localhost:3000 with your browser.

Integrating Django and NextJS

Next we need to allow NextJS and Django to communicate.

We'll use NGINX to present the entire application on one network port. NGINX will proxy requests to Django or NextJS based on the URL path.

Open nginx.conf and add the following.

server {
    listen 80;

    location /api {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://daphne:8000;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_pass http://nextjs:3000;
    }
}

Now update the docker-compose.yml file.

version: "3.7"

services:
  daphne:
    build:
      context: .
    command: daphne -b 0.0.0.0 -p 8000 djdoodle.asgi:application
    volumes:
      - $PWD:/usr/src/app

  nextjs:
    build:
      context: ./djdoodle-ui
    volumes:
      - $PWD/djdoodle-ui:/usr/src/app

  nginx:
    image: nginx:latest
    ports:
      - "8000:80"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - $PWD:/usr/src/app

Now you should be able to visit http://localhost:8000 and http://localhost:8000/api to see NextJS and Django respectively.

Adding Django Channels

The Django Channels package, which handles websockets, uses Redis to store connection and message data.

We need to update settings.py to tell django-channels where to find Redis.

The ASGI_APPLICATION variable must also be declared. We'll add the djdoodle/asgi.py file in a moment.

WSGI_APPLICATION = "djdoodle.wsgi.application"                                            
ASGI_APPLICATION = "djdoodle.asgi.application"                                            

CHANNEL_LAYERS = {                                                                         
    "default": {                                                                           
        "BACKEND": "channels_redis.core.RedisChannelLayer",                                
        "CONFIG": {                                                                        
            "hosts": [(os.environ["REDIS_HOST"], 6379)],                                   
            "capacity": 1500,                                                              
        },                                                                                 
    },                                                                                     
}

Django Channels uses Redis to store message data.

We'll update docker-compose.yml to include a Redis image and add an environment variable to the daphne container.

version: "3.7"

services:
  daphne:
    build:
      context: .
    command: daphne -b 0.0.0.0 -p 8000 djdoodle.asgi:application
    environment:
      - REDIS_HOST=redis
    volumes:
      - $PWD:/usr/src/app
    depends_on:
      - redis

  nextjs:
    build:
      context: ./djdoodle-ui
    volumes:
      - $PWD/djdoodle-ui:/usr/src/app

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

  nginx:
    image: nginx:latest
    ports:
      - "8000:80"
    volumes:
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - $PWD:/usr/src/app

Implementing the Game Server Logic

Create a new directory to contain the async server logic.

mkdir core/ws_consumers

Add the following to the core/ws_consumers/draw.py module.

import json
import asyncio

from django.conf import settings
from openai import OpenAI

from channels.generic.websocket import AsyncWebsocketConsumer

client = OpenAI(api_key=settings.OPENAI_API_KEY)

# An initial message to send to OpenAI that explains the concept of the
# game and the kind of responses we want.
system_prompt = """
There is a game in which two players both work to complete a simple drawing
according to a series of prompts that build on one another.

Each player receives a series of distinct prompts.  The prompts should be
related in that they both apply to the same drawing, but the prompts
should be designed so that they apply to different parts of the drawing.

For instance, one player might receive a series of prompts around drawing
the sky, while the other player is drawing the ground.  This is only an example.
Please generate your own theme.

Please generate two series of prompts in JSON format, one for each player.

The JSON should have a player1 key and a player2 key.  The value of each key
should be an array of strings.
"""


class DrawConsumer(AsyncWebsocketConsumer):
    # The game prompts are fetched from OpenAI when the first player connects.
    # We need to avoid re-fetching when the second player connects, and also
    # make sure both players are sent their prompts at the same time.
    # The first player to connect will acquire the fetch_lock while the second
    # player will block on fetch_lock until the prompts are ready.
    fetch_lock = asyncio.Lock()

    prompts_fetched = False

    countdown_time = 30
    countdown_active = False

    # Track the connection count as an easy way to know which set of prompts
    # to send when a new connection is established.
    connection_count = 0

    player1_prompts = []
    player2_prompts = []

    async def start_countdown(self):
        """
        Starts a countdown timer and sends an update of how many seconds
        remain to each connected Websocket client.

        The timer will restart when it reaches zero.
        """
        if not DrawConsumer.countdown_active:
            DrawConsumer.countdown_active = True
            while DrawConsumer.countdown_time > 0:
                await asyncio.sleep(1)
                await self.channel_layer.group_send(
                    "draw_group",
                    {
                        "type": "countdown_message",
                        "countdown": DrawConsumer.countdown_time,
                    },
                )
                DrawConsumer.countdown_time -= 1

                if DrawConsumer.countdown_time == 0:
                    DrawConsumer.countdown_time = 30

    async def fetch_prompts(self):
        """
        Fetch prompts from the OpenAI API.

        The first caller will trigger the API request while subsequent calls
        block on the fetch_lock, eventually returning immediately because the
        prompts_fetched bool has been set to True.
        """
        async with DrawConsumer.fetch_lock:
            if not DrawConsumer.prompts_fetched:
                DrawConsumer.prompts_fetched = True

                completion = client.chat.completions.create(
                    model="gpt-4-1106-preview",
                    response_format={"type": "json_object"},
                    messages=[
                        {
                            "role": "user",
                            "content": system_prompt,
                        }
                    ],
                )

                data = json.loads(completion.choices[0].message.content)

                DrawConsumer.player1_prompts = data["player1"]
                DrawConsumer.player2_prompts = data["player2"]

    async def connect(self):
        """
        Set up Websocket connection and make OpenAI request if this is the
        first connection.  Distributes prompts to players.
        """
        DrawConsumer.connection_count += 1

        await self.channel_layer.group_add("draw_group", self.channel_name)
        await self.fetch_prompts()

        await self.accept()

        prompts = [DrawConsumer.player1_prompts, DrawConsumer.player2_prompts]

        await self.send(
            text_data=json.dumps(
                {
                    "type": "prompts",
                    "prompts": prompts[DrawConsumer.connection_count % 2],
                }
            )
        )

        asyncio.create_task(self.start_countdown())

    async def disconnect(self, close_code):
        await self.channel_layer.group_discard("draw_group", self.channel_name)

    async def receive(self, text_data):
        """
        Broadcast messages received from one player back to themselves and to
        the other player in the game.
        """
        data = json.loads(text_data)
        await self.channel_layer.group_send(
            "draw_group", {"type": "draw_message", "message": data}
        )

    async def draw_message(self, event):
        message = event["message"]
        await self.send(text_data=json.dumps(message))

    async def countdown_message(self, event):
        countdown = event["countdown"]
        message = {"type": "countdown", "countdown": countdown}
        await self.send(text_data=json.dumps(message))

Now we'll wire up the routing for the DrawConsumer class. Open core/ws_routing.py and add the following.

from django.urls import path
from core.ws_consumers.draw import DrawConsumer

ws_urlpatterns = [
    path("api/ws/draw/", DrawConsumer.as_asgi()),
]

Update the djdoodle/asgi.py file so Daphne is aware of our DrawConsumer endpoint.

import os
from django.core.asgi import get_asgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "djdoodle.settings")

application = get_asgi_application()

from channels.routing import ProtocolTypeRouter, URLRouter
from core.ws_routing import ws_urlpatterns

application = ProtocolTypeRouter(
    {
        "websocket": URLRouter(ws_urlpatterns),
    }
)

For the above to work, you'll need to add your OpenAI API key to the settings.py file.

OPENAI_API_KEY = "your key here"

Implementing the NextJS UI

To get started, open djdoodle-ui/app/drawing.js and add the following.

This code will tie all the components together and manage communication with Django over the websocket.

"use client";
import React, { useState, useEffect, useRef } from "react";
import CanvasComponent from "./canvas";
import CountdownDisplay from "./countdown";
import ColorPicker from "./picker";
import Prompt from "./prompt";
import styles from "./drawing.module.css";

const startCountdown = 30;

const DrawingApp = () => {
  const [drawData, setDrawData] = useState([]);
  const [isPromptVisible, setIsPromptVisible] = useState(false);
  const [countdown, setCountdown] = useState(startCountdown);
  const [prompts, setPrompts] = useState([]);
  const [promptMessage, setPromptMessage] = useState("");
  const [color, setColor] = useState("#00ff00");
  const ws = useRef(null);

  useEffect(() => {
    // Show the prompt popup with the next prompt once
    // the timer rolls back around to 30 seconds.
    if (countdown == 30 && prompts.length) {
      setPromptMessage(prompts.shift());
      setIsPromptVisible(true);
    }
  }, [prompts, countdown]);

  useEffect(() => {
    if (!ws.current) {
      ws.current = new WebSocket(
        `ws://${window.location.hostname}:8000/api/ws/draw/`,
      );

      ws.current.onmessage = (event) => {
        const data = JSON.parse(event.data);

        if (data.type === "prompts") {
          setPrompts(data.prompts);
        } else if (data.type === "countdown") {
          setCountdown(data.countdown);
        } else {
          setDrawData((prevDrawData) => [...prevDrawData, data]);
        }
      };
    }

    return () => {
      if (ws.current) {
        ws.current.close();
        ws.current = null;
      }
    };
  }, []);

  const handleMouseEvent = (eventType, position) => {
    if (ws.current && ws.current.readyState === WebSocket.OPEN) {
      ws.current.send(JSON.stringify(position));
    }
  };

  return (
    <div>
      <CanvasComponent
        drawData={drawData}
        color={color}
        onMouseEvent={handleMouseEvent}
      />
      <div className={styles.picker}>
        <ColorPicker
          color={color}
          onColorChange={(newColor) => setColor(newColor)}
        />
      </div>
      <div className={styles.countdown}>
        <CountdownDisplay countdown={countdown} />
      </div>
      <div className={styles.prompt}>
        <Prompt
          message={promptMessage}
          isVisible={isPromptVisible}
          onDismiss={() => setIsPromptVisible(false)}
        />
      </div>
    </div>
  );
};

export default DrawingApp;

The child components include the HTML5 canvas, a countdown timer, a popup for the drawing prompt and a color picker.

Open djdoodle-ui/app/canvas.js and enter the following for the canvas component.

"use client";
import React, { useState, useRef, useEffect } from "react";

const CanvasComponent = ({ drawData, color, onMouseEvent }) => {
  const [isMouseDown, setIsMouseDown] = useState(false);
  const canvasRef = useRef(null);

  const drawBackground = (ctx, width, height) => {
    ctx.fillStyle = "black";
    ctx.fillRect(0, 0, width, height);
  };

  const drawCircle = (ctx, x, y, color) => {
    ctx.fillStyle = color;
    ctx.beginPath();
    ctx.arc(x, y, 12, 0, 2 * Math.PI);
    ctx.fill();
  };

  const handleMouseEvent = (event) => {
    if (isMouseDown) {
      const canvas = canvasRef.current;
      const rect = canvas.getBoundingClientRect();
      const position = {
        x: event.clientX - rect.left,
        y: event.clientY - rect.top,
      };
      onMouseEvent(event.type, { color: color, ...position });
    }

    if (event.type === "mousedown") {
      setIsMouseDown(true);
    } else if (event.type === "mouseup") {
      setIsMouseDown(false);
    }
  };

  useEffect(() => {
    const canvas = canvasRef.current;
    const context = canvas.getContext("2d");
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    drawBackground(context, canvas.width, canvas.height);

    drawData.forEach(({ x, y, color }) => {
      drawCircle(context, x, y, color);
    });
  }, [drawData]);

  useEffect(() => {
    const canvas = canvasRef.current;
    canvas.addEventListener("mousemove", handleMouseEvent);
    canvas.addEventListener("mousedown", handleMouseEvent);
    canvas.addEventListener("mouseup", handleMouseEvent);

    return () => {
      canvas.removeEventListener("mousemove", handleMouseEvent);
      canvas.removeEventListener("mousedown", handleMouseEvent);
      canvas.removeEventListener("mouseup", handleMouseEvent);
    };
  }, [isMouseDown]);

  return <canvas ref={canvasRef} />;
};

export default CanvasComponent;

Now we'll add the countdown timer in the djdoodle-ui/app/countdown.js file.

"use client";
import React from "react";
import styles from "./countdown.module.css";

const CountdownDisplay = ({ countdown }) => {
  return <div className={styles.countdown}>{countdown}</div>;
};

export default CountdownDisplay;

Open and update djdoodle-ui/app/prompt.js to create the Prompt component.

import React, { useState } from "react";
import styles from "./prompt.module.css";

const Prompt = ({ message, isVisible, onDismiss }) => {
  if (!isVisible) return null;

  return (
    <div className={styles.prompt}>
      <p>{message}</p>
      <button onClick={onDismiss}>OK</button>
    </div>
  );
};

export default Prompt;

Finally, we'll add the following to the djdoodle-ui/app/picker.js file to create the color picker component.

import React from "react";

const ColorPicker = ({ color, onColorChange }) => {
  const handleColorChange = (event) => {
    onColorChange(event.target.value);
  };

  return (
    <div>
      <input
        type="color"
        id="color-picker"
        name="color"
        value={color}
        onChange={handleColorChange}
      />
    </div>
  );
};

export default ColorPicker;

Next Steps

This has hopefully been a fun way to explore Daphne and websockets!

There are many ways to improve and build upon this simple example:

Add a game lobby where players are matched up
Add more drawing tools for the players
Improve the websocket communication to be more efficient
Make the app ready for a production deployment

You can view the full code for the app on GitHub.

While most of the code was shared here, a few parts such as CSS were omitted to make this guide shorter.

Django Websockets

Zach — Wed, 06 Sep 2023 22:12:24 +0000

While many tutorials demonstrate how to implement WebSockets in a Django app through chat applications, we'll take a different route in this guide.

We'll lean into the real-time capabilities of WebSockets to create a simple multiplayer game. For this tutorial, we'll use React on the frontend and leverage Django Channels and Daphne on the backend to handle the WebSocket connections.

1. Django Websockets in Action
2. Creating the Django Project
3. Setting Up the Django Template
4. Django Channels and Daphne
- 4.1 Routing and WebsocketConsumer
- 4.2 Using NGINX and Daphne in Production
- 4.3 Implementing Game Logic in the Daphne Consumer
5. Setting Up the React Game Component
6. Load Testing the Daphne Server

Django Websockets in Action

Before we dive into the tutorial, I'd like to share a live demo of the multiplayer game!

You can find the live demo here.

While simple, this game demonstrates that Django Channels and Daphne can support highly interactive real-time applications. Continue reading to see how you can implement something similar.

Creating the Django Project

In this section you will set up the Django project and verify that all prerequisites are in place.

Begin by using the django-admin tool to create a new Django project.

django-admin startproject multiplayer_channels_demo

Now create a new application within the multiplayer_channels_demo directory.

cd hero_builder
python3 manage.py startapp channels_demo

Run the initial migrations for the SQLite development database.

python3 manage.py migrate

Start the development server in order to verify that everything is working so far.

python3 manage.py runserver

You should be able to visit http://localhost:8000 now. Django will serve a placeholder page.

Setting Up the Django Template

Even though our focus is on Websockets, we still need a Django template to serve the initial page. That first page will contain JavaScript that opens a Websocket connection.

If you haven't already, you'll need to create a template directory for the channels_demo app.

mkdir -p channels_demo/templates/channels_demo

Now we'll create the index.html template inside the new templates directory. The template contains a custom element named multiplayer-channels-demo that will create the HTML5 canvas and communicate with Django.

The component is built with React but has been converted into a Web Component so that it can be dropped into the template with minimal dependencies. The React runtime is loaded from a CDN, as the custom element doesn't include those dependencies.

We'll go into greater depth on the React component later in the tutorial.

<!DOCTYPE html>
{% load static %}
<html>
  <head>
    <meta charset="utf-8" />
    <link rel="stylesheet" href="{% static 'channels_demo/css/index.css' %}"/>

    <script crossorigin src="https://unpkg.com/react@18/umd/react.production.min.js"></script>
    <script crossorigin src="https://unpkg.com/react-dom@18/umd/react-dom.production.min.js"></script>

    <script src="{% static 'channels_demo/js/main.js' %}"></script>

    <title>Multiplayer Demo</title>
  </head>
  <body>
    <multiplayer-channels-demo></multiplayer-channels-demo>
  </body>
</html>

With the template created, we need routing and a view function. Open views.py in the channels_demo directory and insert the following.

from django.shortcuts import render

def index(request):
    return render(request, "channels_demo/index.html")

We also need a urls.py file for the app. Create one under channels_demo and update it with the following.

from django.urls import path
from . import views

urlpatterns = [
    path("", views.index, name="index"),
]

Lastly, the urls.py file under multiplayer_channels_demo needs to be updated to include the new routes.

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("multiplayer/", include("channels_demo.urls")),
    path("admin/", admin.site.urls),
]

Django Channels and Daphne

So far we have a vanilla Django project. Next we'll add Django Channels and Daphne to our application.

We'll use a requirements.txt file to manage the project dependencies. Create a file named requirements.txt within your top-level project folder and place the following in it.

Django==4.1.7
daphne==4.0.0
channels==4.0.0

Use pip to install the dependencies before moving further with the tutorial.

pip install -r requirements.txt

Next we need to update settings.py to include Daphne and the demo app in the INSTALLED_APPS list. Note that Daphne must be listed first in order to properly extend the runserver management command.

INSTALLED_APPS = [ 
    "daphne",
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "channels_demo",
]

One more addition is needed in settings.py before we can move on. Daphne requires an entrypoint that satisfies the Asynchronous Server Gateway Interface (ASGI) in order to serve our Django application. Luckily, new Django projects come standard with an asgi.py file that we can use for this, but we need to tell Daphne how to find it.

ASGI_APPLICATION = "multiplayer_channels_demo.asgi.application"

Lastly, we need a minimal setup inside the asgi.py file to get started. This minimal setup won't have any Websocket functionality initially, but will allow Daphne to serve regular Django views. Open the file and write the following.

import os

from channels.routing import ProtocolTypeRouter
from django.core.asgi import get_asgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "multiplayer_channels_demo.settings")

application = ProtocolTypeRouter(
    {
        "http": get_asgi_application(),
    }
)

You should be able to launch the development server again, but this time Daphne will act as the server.

python3 manage.py runserver

The output should look similar to the first time, but if you look closely, you should see that Daphne is listed as the server.

 Performing system checks...

System check identified no issues (0 silenced).
September 03, 2023 - 15:56:03
Django version 4.1.7, using settings 'multiplayer_channels_demo.settings'
Starting ASGI/Daphne version 4.0.0 development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Routing and WebsocketConsumer

Now that the essentials are in place, we'll add a Websocket consumer and the associated routing. In Django Channels terminology, a consumer is much like a normal Django view. Likewise, routing for consumers is normally done separately from the urls.py file that we typically see in every Django project. Open a new file under the channels_demo directory named routing.py and insert the following.

from django.urls import re_path
from . import consumers

websocket_urlpatterns = [
    re_path(r"ws/game/$", consumers.MultiplayerConsumer.as_asgi()),
]

Next we'll define the simplest possible WebsocketConsumer class that we can use to get started. Open consumers.py in the same directory as the routing.py file and enter the following.

import json
import uuid

from channels.generic.websocket import WebsocketConsumer
from channels.layers import get_channel_layer
from asgiref.sync import async_to_sync

class MultiplayerConsumer(WebsocketConsumer):
    game_group_name = "game_group"
    players = {}

    def connect(self):
        self.player_id = str(uuid.uuid4())
        self.accept()

        async_to_sync(self.channel_layer.group_add)(
            self.game_group_name, self.channel_name
        )

        self.send(
            text_data=json.dumps({"type": "playerId", "playerId": self.player_id})
        )

    def disconnect(self, close_code):
        async_to_sync(self.channel_layer.group_discard)(
            self.game_group_name, self.channel_name
        )

    def receive(self, text_data):
        text_data_json = json.loads(text_data)

This is a very basic consumer, but it illustrates the important methods. There is a dedicated MultiplayerConsumer instance per connection. The connect method is called on initial connection, disconnect when the connection is broken, and receive whenever a message is received from the websocket. The send method communicates directly with the connected client. It's used in the code above to give new players their own unique player ID.

Before we can connect at this endpoint, however, we need to update asgi.py to handle routing websocket connections. Up to this point, Channels has only been set up to handle normal views. Open the asgi.py file and make the following update.

import os

from channels.routing import ProtocolTypeRouter, URLRouter
from django.core.asgi import get_asgi_application

from channels_demo.routing import websocket_urlpatterns

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "multiplayer_channels_demo.settings")

application = ProtocolTypeRouter(
    {
        "http": get_asgi_application(),
        "websocket": URLRouter(websocket_urlpatterns),
    }
)

The URLRouter class is responsible for delegating requests to the correct consumer.

One more tweak is needed before this basic setup will function. The code in the consumers.py file makes use of Channel layers by referring to the self.channel_layer variable that's inherited from the WebsocketConsumer class. The channel_layer allows our code to organize connections into groups and broadcast messages to those groups.

Before we can do any of that, however, Django Channels requires that we define a backend. The Channels backend is responsible for storing connection info. If you have a single server and you don't care about losing connections on a restart, then the InMemoryChannelLayer might be sufficient. For multiple servers and persistence, however, you may want to use the RedisChannelLayer instead.

Add the following to settings.py to make use of the InMemoryChannelLayer for now.

CHANNEL_LAYERS = { 
    "default": {
        "BACKEND": "channels.layers.InMemoryChannelLayer"
    }   
}

Using NGINX and Daphne in Production

Now we'll take a moment to make this setup ready for production. Daphne is production ready, but it typically isn't used to serve static files or provide SSL termination.

We can use Docker to create a consistent environment for both development and production. The first step is to create a Dockerfile to represent the application and its dependencies.

FROM python:3.8

ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

WORKDIR /usr/src/app

COPY requirements.txt /usr/src/app/

RUN pip install --no-cache-dir -r requirements.txt

COPY . /usr/src/app/

RUN mkdir -p /usr/src/app/

CMD ["daphne", "multiplayer_channels_demo.asgi:application", "-u", "/usr/src/app/daphne.sock"]

The Daphne server will read and write from daphne.sock instead of listening on a port. NGINX will forward requests to the socket file. Open a file named nginx.conf and insert the following.

worker_processes 1;

events {
    worker_connections 1024;
}

http {
    include mime.types;

    server {
        listen 8443 ssl http2;

        ssl_certificate /etc/letsencrypt/live/your_domain_name/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/your_domain_name/privkey.pem;
        ssl_prefer_server_ciphers off;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";

            proxy_pass http://unix:/usr/src/app/daphne.sock;
        }

        location /static {
            alias /usr/src/app/staticfiles;
        }
    }
}

Be sure to update the your_domain_name placeholder with your SSL certificate path. You can use LetsEncrypt to obtain a certificate quickly and easily. If you don't wish to use SSL, it's easy to strip that out and listen for plain HTTP connections. This setup forwards all requests other than those starting with /static to the Daphne server.

Now we'll create a file named docker-compose.yml to tie all this together. We'll have an app container and an NGINX container. The containers will share a volume so that both have access to the same daphne.sock file.

version: "3.7"

services:
  app:
    build:
      context: .
    restart: unless-stopped
    command: daphne multiplayer_channels_demo.asgi:application -u /usr/src/app/daphne.sock
    volumes:
      - $PWD:/usr/src/app

  nginx:
    image: nginx:latest
    restart: always
    ports:
      - "8443:8443"
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - $PWD:/usr/src/app
    depends_on:
      - app

With the above in place and Docker Compose installed, you should be able to issue the docker-compose up -d command to bring up both containers in the background.

Implementing Game Logic in the Daphne Consumer

We have a containerized Django and Daphne application, but it doesn't do much other than accept connections at this point.

Now we'll fill out the game logic in the MultiplayerConsumer class that we created earlier. The logic consists of a few important concepts:

New connections receive a unique player ID.
The client sends mouse down and mouse release events to Django.
The client sends the angle between the mouse cursor and player sprite to Django every 50ms.
The game server performs simple physics updates on a 50ms interval.
Django broadcasts the position and facing angle of all sprites to all connections every 50ms.

In a real game, the client would most likely also do its own physics calculations, with updates from the game server taking precedence. In this way, the game can appear to keep functioning even with lag. When the connection is restored, then the state of the game can be reconciled with the server.

For this simple example, however, all physics calculations are done within the MultiplayerConsumer class. In fact, the majority of the code is dedicated to simulation. The websocket related code is still relatively simple.

import json
import uuid
import asyncio
import math

from channels.generic.websocket import AsyncWebsocketConsumer
from asgiref.sync import async_to_sync

class MultiplayerConsumer(AsyncWebsocketConsumer):
    MAX_SPEED = 5
    THRUST = 0.2

    game_group_name = "game_group"
    players = {}

    update_lock = asyncio.Lock()

    async def connect(self):
        self.player_id = str(uuid.uuid4())
        await self.accept()

        await self.channel_layer.group_add(
            self.game_group_name, self.channel_name
        )

        await self.send(
            text_data=json.dumps({"type": "playerId", "playerId": self.player_id})
        )

        async with self.update_lock:
            self.players[self.player_id] = {
                "id": self.player_id,
                "x": 500,
                "y": 500,
                "facing": 0,
                "dx": 0,
                "dy": 0,
                "thrusting": False,
            }

        if len(self.players) == 1:
            asyncio.create_task(self.game_loop())

    async def disconnect(self, close_code):
        async with self.update_lock:
            if self.player_id in self.players:
                del self.players[self.player_id]

        await self.channel_layer.group_discard(
            self.game_group_name, self.channel_name
        )

    async def receive(self, text_data):
        text_data_json = json.loads(text_data)
        message_type = text_data_json.get("type", "")

        player_id = text_data_json["playerId"]

        player = self.players.get(player_id, None)
        if not player:
            return

        if message_type == "mouseDown":
            player["thrusting"] = True
        elif message_type == "mouseUp":
            player["thrusting"] = False
        elif message_type == "facing":
            player["facing"] = text_data_json["facing"]

    async def state_update(self, event):
        await self.send(
            text_data=json.dumps(
                {
                    "type": "stateUpdate",
                    "objects": event["objects"],
                }
            )
        )

    async def game_loop(self):
        while len(self.players) > 0:
            async with self.update_lock:
                for player in self.players.values():
                    if player["thrusting"]:
                        dx = self.THRUST * math.cos(player["facing"])
                        dy = self.THRUST * math.sin(player["facing"])
                        player["dx"] += dx
                        player["dy"] += dy

                        speed = math.sqrt(player["dx"] ** 2 + player["dy"] ** 2)
                        if speed > self.MAX_SPEED:
                            ratio = self.MAX_SPEED / speed
                            player["dx"] *= ratio
                            player["dy"] *= ratio

                    player["x"] += player["dx"]
                    player["y"] += player["dy"]

            await self.channel_layer.group_send(
                self.game_group_name,
                {"type": "state_update", "objects": list(self.players.values())},
            )
            await asyncio.sleep(0.05)

Setting Up the React Game Component

You might recall from earlier that the Django template contained a multiplayer-channels-demo element. Behind the scenes, this element is a React component that has been converted into a Web Component.

I covered this technique in an earlier post on progressively enhancing Django templates with Web Components.

In short, we'll build a normal React component and use R2WC to perform the conversion.

The following is the CanvasComponent that handles all client side logic, including communication with the Django backend.

import React, { useRef, useEffect, useState } from "react";

const calculateAngle = (playerX, playerY, mouseX, mouseY) => {
  const dx = mouseX - playerX;
  const dy = mouseY - playerY;
  return Math.atan2(dy, dx);
};

const CanvasComponent = () => {
  const canvasRef = useRef(null);
  const [containerSize, setContainerSize] = useState({ width: 0, height: 0 });
  const [gameObjects, setGameObjects] = useState([]);
  const [playerId, setPlayerId] = useState(null);
  const gameObjectsRef = useRef(gameObjects);
  const playerIdRef = useRef(playerId);
  const requestRef = useRef(null);
  const moveSpriteRef = useRef(null);
  const idleSpriteRef = useRef(null);
  const wsRef = useRef(null);

  const coordinateWidth = 1000;
  const coordinateHeight = 1000;

  // Load the idle and move sprites that are used to render the players.
  useEffect(() => {
    const idleImage = new Image();
    idleImage.onload = () => (idleSpriteRef.current = idleImage);
    idleImage.src = "/static/channels_demo/images/sprite-idle.png";

    const moveImage = new Image();
    moveImage.onload = () => (moveSpriteRef.current = moveImage);
    moveImage.src = "/static/channels_demo/images/sprite-move.png";
  }, []);

  // Resize the canvas to fit the width and height of the parent
  // container.  The "logical" canvas is always 1000/1000
  // pixels, so a transform is needed when the physical canvas
  // size does not match.
  const resizeCanvas = () => {
    const canvas = canvasRef.current;
    const ctx = canvas.getContext("2d");
    const boundingRect = canvas.parentNode.getBoundingClientRect();

    const pixelRatio = window.devicePixelRatio || 1;

    canvas.width = boundingRect.width;
    canvas.height = boundingRect.height;

    ctx.setTransform(
      (canvas.width / coordinateWidth) * pixelRatio,
      0,
      0,
      (canvas.height / coordinateHeight) * pixelRatio,
      0,
      0
    );
  };

  // Render each player sprite.  Use context save/restore to apply transforms
  // for that sprite specifically without affecting anything else.
  const drawGameObject = (ctx, obj) => {
    if (!moveSpriteRef.current) return;

    const spriteWidth = 187;
    const spriteHeight = 94;

    ctx.save();
    ctx.translate(obj.x, obj.y);
    ctx.rotate(obj.facing);

    const sprite = obj.thrusting
      ? moveSpriteRef.current
      : idleSpriteRef.current;

    ctx.drawImage(
      sprite,
      0,
      0,
      spriteWidth,
      spriteHeight,
      -spriteWidth / 4,
      -spriteHeight / 4,
      spriteWidth / 3,
      spriteHeight / 3
    );
    ctx.restore();
  };

  // Server messages are either game state updates or the initial
  // assignment of a unique playerId for this client.
  const handleWebSocketMessage = (event) => {
    const messageData = JSON.parse(event.data);
    if (messageData.type === "stateUpdate") {
      setGameObjects(messageData.objects);
    } else if (messageData.type === "playerId") {
      setPlayerId(messageData.playerId);
    }
  };

  // Notify game server of mouse down.
  const handleMouseDown = (event) => {
    if (event.button !== 0 || !playerIdRef.current) return;
    wsRef.current.send(
      JSON.stringify({ type: "mouseDown", playerId: playerIdRef.current })
    );
  };

  // Notify game server of mouse release.
  const handleMouseUp = (event) => {
    if (event.button !== 0 || !playerIdRef.current) return;
    wsRef.current.send(
      JSON.stringify({ type: "mouseUp", playerId: playerIdRef.current })
    );
  };

  // Callback for requestAnimationFrame. Clears the canvas and renders the
  // black background before rendering each individual player sprite.
  const animate = (time) => {
    requestRef.current = requestAnimationFrame(animate);

    const canvas = canvasRef.current;
    const ctx = canvas.getContext("2d");

    ctx.clearRect(0, 0, coordinateWidth, coordinateHeight);

    ctx.fillStyle = "black";
    ctx.fillRect(0, 0, coordinateWidth, coordinateHeight);

    gameObjectsRef.current.forEach((obj) => drawGameObject(ctx, obj));
  };

  // Refresh the gameObjectsRef when gameObjects is updated.  This is necessary
  // because the animate callback triggers the "stale closure" problem.
  useEffect(() => {
    gameObjectsRef.current = gameObjects;
  }, [gameObjects]);

  // Refresh the playerIdRef when playerId is updated.  This is necessary
  // because the animate callback triggers the "stale closure" problem.
  useEffect(() => {
    playerIdRef.current = playerId;
  }, [playerId]);

  // Sets up an interval that calculates the angle between the mouse cursor
  // and the player sprite.  Sends that angle to the game server on a 50ms
  // interval.
  useEffect(() => {
    let mousePosition = { x: 0, y: 0 };

    const updateMousePosition = (event) => {
      const canvas = canvasRef.current;
      const boundingRect = canvas.getBoundingClientRect();
      const scaleX = canvas.width / boundingRect.width;
      const scaleY = canvas.height / boundingRect.height;

      mousePosition.x = (event.clientX - boundingRect.left) * scaleX;
      mousePosition.y = (event.clientY - boundingRect.top) * scaleY;
    };

    window.addEventListener("mousemove", updateMousePosition);

    const intervalId = setInterval(() => {
      const playerId = playerIdRef.current;
      const gameObjects = gameObjectsRef.current;

      if (!playerId) return;
      const playerObj = gameObjects.find((obj) => obj.id === playerId);

      if (playerObj) {
        const facing = calculateAngle(
          playerObj.x,
          playerObj.y,
          mousePosition.x * (coordinateWidth / canvasRef.current.width),
          mousePosition.y * (coordinateHeight / canvasRef.current.height)
        );

        wsRef.current.send(
          JSON.stringify({
            type: "facing",
            playerId,
            facing,
          })
        );
      }
    }, 50);

    return () => {
      clearInterval(intervalId);
      window.removeEventListener("mousemove", updateMousePosition);
    };
  }, []);

  // Entrypoint.  Establish websocket, register input callbacks, and start
  // the animation frame cycle.
  useEffect(() => {
    const canvas = canvasRef.current;

    wsRef.current = new WebSocket("ws://localhost:8000/ws/game/");
    wsRef.current.onmessage = handleWebSocketMessage;

    canvas.addEventListener("mousedown", handleMouseDown);
    canvas.addEventListener("mouseup", handleMouseUp);

    requestRef.current = requestAnimationFrame(animate);
    return () => {
      cancelAnimationFrame(requestRef.current);
      canvas.removeEventListener("mousedown", handleMouseDown);
      canvas.removeEventListener("mouseup", handleMouseUp);
      wsRef.current.close();
    };
  }, []);

  // Call resizeCanvas on initial load and also whenever browser is resized.
  useEffect(() => {
    resizeCanvas();
    window.addEventListener("resize", resizeCanvas);

    return () => {
      window.removeEventListener("resize", resizeCanvas);
    };
  }, [containerSize]);

  return &lt;canvas ref={canvasRef} /&gt;;
};

export default CanvasComponent;

Load Testing the Daphne Server

We have a working game server and client at this point, but how many clients can the server host?

I decided to write a script to load test a single Daphne server behind NGINX, on a host with 1 vCPU. The load test script simulates multiple client connections, in that it connects and then sends updates on a 50ms interval, just like a real client. The script begins with one connection and adds an additional connection every second, until it has run for 90 seconds.

from collections import defaultdict

import asyncio
import json
import math
import time
import websockets

time_since_last_update = defaultdict(list)

async def send_facing_updates(websocket, player_id):
    while True:
        current_time = time.time()
        new_facing = math.sin(current_time)

        facing_msg = {"type": "facing", "playerId": player_id, "facing": new_facing}

        await websocket.send(json.dumps(facing_msg))
        await asyncio.sleep(0.05)


async def receive_updates(websocket, client_id, player_id):
    last_update_time = time.time()
    while True:
        message = await websocket.recv()
        data = json.loads(message)

        current_time = time.time()
        delta_time = current_time - last_update_time
        last_update_time = current_time

        time_since_last_update[client_id].append(delta_time)


async def game_client(client_id):
    uri = "ws://localhost:8000/ws/game/"
    async with websockets.connect(uri) as websocket:
        init_msg = await websocket.recv()
        init_data = json.loads(init_msg)
        player_id = init_data.get("playerId", None)

        if player_id is None:
            return

        send_task = send_facing_updates(websocket, player_id)
        receive_task = receive_updates(websocket, client_id, player_id)
        await asyncio.gather(send_task, receive_task)


async def main():
    client_id = 0
    tasks = []
    start_time = time.time()
    while True:
        if time.time() - start_time > 90:
            for task in tasks:
                task.cancel()
            break

        task = asyncio.create_task(game_client(client_id))
        tasks.append(task)

        client_id += 1
        await asyncio.sleep(1)

    await asyncio.gather(*tasks, return_exceptions=True)

    print(json.dumps(time_since_last_update))

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main())

Once the script has completed, it outputs JSON containing the time elapsed between each server update, for every connection. The time elapsed should be close to 50ms as long as the server is keeping up. As the load increases, however, the elapsed time begins to exceed 50ms.

Here's a graph of how that test turned out.

The further to the right on the X axis, the more connections have been opened.

We start with 1 connection and end with 90 connections. Aside from a few lag spikes, we stay mostly at 50ms between updates, until the script has opened 50 or so connections. At that point, the lag would be noticeable, with updates taking 250ms or more. At the 80 connection mark, lag increases dramatically, with some updates taking well over a full second. Our game would be pretty unplayable at that point.

Conclusion

I hope this tutorial has given you some insight into what you can achieve with Django, Django Channels, and Daphne! Many tutorials focus on use cases like chat apps, and while there's nothing wrong that, it's interesting to push the limits and see just how real-time you can get with Django.

Using React to Progressively Enhance a Django App

Zach — Thu, 17 Aug 2023 10:42:55 +0000

Introduction

In this tutorial, you will create a Hero Builder application using Django and React. Hero Builder will allow users to build a character and distribute points across attributes such as strength, dexterity, and intelligence.

Django is a high-level, open-source web framework written in Python that encourages rapid development and clean, pragmatic design. Released in 2005, it follows the "batteries-included" philosophy, providing developers with a rich set of tools and utilities out of the box, including an ORM (Object-Relational Mapping), an admin interface, and form handling, to name a few. Django emphasizes the DRY (Don't Repeat Yourself) principle and reusability of components, allowing developers to build robust, scalable, and maintainable web applications with less code.

React, often referred to as React.js, is an open-source JavaScript library developed by Facebook for building user interfaces or UI components. Introduced in 2013, it allows developers to construct complex, interactive UIs using a component-based architecture. Each component in React manages its own state and renders it to the DOM, making it easier to develop and reason about modular pieces of an application. Over the years, React has become a foundational tool in modern web development, powering many of today’s most popular websites and applications.

In this application, the Django framework will serve the frontend, while React will take over specific parts of the page that require dynamic interactions.

Prerequisites

To build this application, you will need to complete the following:

Install Node.js and NPM
Install Python3

Setting up the Django Project

In this section you will set up the Django project and verify that all prerequisites are in place.

Begin by using the django-admin tool to create a new Django project.

django-admin startproject hero_builder

Now create a new application within the hero_builder directory.

cd hero_builder
python3 manage.py startapp hero

Run the initial migrations for the SQLite development database.

python3 manage.py migrate

Start the development server in order to verify that everything is working so far.

python3 manage.py runserver

You should be able to visit http://localhost:8000 now. Django will serve a placeholder page.

Registering the Hero Application

You should now register the hero app that was created in earlier steps. Open the hero_builder/settings.py file in your editor and add the hero app to the INSTALLED_APPS list.

INSTALLED_APPS = [ 
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "hero",
]

Save your changes to the settings.py file.

Defining the Character Model

The hero app will have a Character model to store the distribution of points across attributes. Open hero/models.py and add the following.

from django.db import models

class Character(models.Model):
    strength = models.PositiveIntegerField(default=20)
    dexterity = models.PositiveIntegerField(default=20)
    health = models.PositiveIntegerField(default=20)
    intelligence = models.PositiveIntegerField(default=20)
    charisma = models.PositiveIntegerField(default=20)

    name = models.CharField(max_length=200)

    def __str__(self):
        return self.name

Now you will add and run the migration necessary to create the Character model.

python manage.py makemigrations hero

To run the migration:

python manage.py migrate hero

Setting up the Django Forms and Views

The Hero app views will use a simple Django form class to validate character data. Open hero/forms.py and add the following.

from django import forms

class CharacterForm(forms.Form):
    strength = forms.IntegerField(min_value=0, initial=20)
    dexterity = forms.IntegerField(min_value=0, initial=20)
    health = forms.IntegerField(min_value=0, initial=20)
    intelligence = forms.IntegerField(min_value=0, initial=20)
    charisma = forms.IntegerField(min_value=0, initial=20)

    def clean(self):
        cleaned_data = super().clean()
        total = (
            cleaned_data.get("strength", 0)
            + cleaned_data.get("dexterity", 0)
            + cleaned_data.get("health", 0)
            + cleaned_data.get("intelligence", 0)
            + cleaned_data.get("charisma", 0)
        )

        if total > 100:
            raise forms.ValidationError("The total attributes must not exceed 100.")
        return cleaned_data

The form class will validate that each attribute is a positive number and that the total does not exceed 100 points.

Next you'll use this form class as part of the character view, where the browser will make a form POST request to create characters. Open the hero/views.py file and add the following.

from django.shortcuts import render, redirect, get_object_or_404
from .models import Character
from .forms import CharacterForm

def home(request):
    form = CharacterForm()
    return render(request, "hero/home.html", {"form": form})

def character(request, character_id=None):
    if request.method == "POST":
        form = CharacterForm(request.POST)
        if form.is_valid():
            character = Character(
                strength=form.cleaned_data["strength"],
                dexterity=form.cleaned_data["dexterity"],
                health=form.cleaned_data["health"],
                intelligence=form.cleaned_data["intelligence"],
                charisma=form.cleaned_data["charisma"],
            )
            character.save()
            return redirect("character_with_id", character_id=character.pk)
    else:
        character = get_object_or_404(Character, id=character_id)

    return render(request, "hero/character.html", {"character": character})

The home view renders the character creation page. The character page will render the attributes of the created character in a read-only table during a GET request, and will create a new character during a POST request.

Defining the Django Templates

Now you will define the Django templates that are used in the Hero app.

The base.html template will act as a foundation, so that all the common script and style imports happen in one place.

Open hero/templates/hero/base.html and add the following.

<!DOCTYPE html>
<html lang="en">
{% load static %}
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Hero Builder</title>

    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-EVSTQN3/azprG1Anm3QDgpJLIm9Nao0Yz1ztcQTwFspd3yD65VohhpuuCOmLASjC" crossorigin="anonymous">
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.2/dist/js/bootstrap.bundle.min.js" integrity="sha384-MrcW6ZMFYlzcLA8Nl+NtUVF0sA7MsXsP1UyJoMp4YLEuNSfAP+JcXn/tWtIaxVXM" crossorigin="anonymous"></script>

    <script crossorigin src="https://unpkg.com/react@18/umd/react.production.min.js"></script>
    <script crossorigin src="https://unpkg.com/react-dom@18/umd/react-dom.production.min.js"></script>

    {% block extra_js %}{% endblock extra_js %}
    {% block extra_css %}{% endblock extra_css %}
</head>
<body>
    {% if messages %}
        {% for message in messages %}
            <div class="alert alert-danger alert-bottom alert-dismissible fade show" role="alert">
                {{ message }}
                <button type="button" class="btn-close" data-bs-dismiss="alert" aria-label="Close"></button>
            </div>
        {% endfor %}
    {% endif %}
    <nav class="navbar navbar-expand-lg">
        <div class="container-fluid">
            Hero Builder
        </div>
    </nav>

    <div>
        {% block content %}{% endblock content %}
    </div>
</body>
</html>

This template loads the minified production source code for the React and ReactDOM libraries. This will be important later when you include the React web component that you are going to build in the next few steps.

Now you will write the home.html template where the React component representing the hero editor is contained.

{% extends "hero/base.html" %}
{% load static %}

{% block extra_js %}
<script src="{% static 'hero/js/main.js' %}"></script>
{% endblock %}

{% block extra_css %}
<link href="{% static 'hero/css/main.css' %}" rel="stylesheet">
{% endblock %}

{% block content %}
<div class="d-flex justify-content-center align-items-center vh-100">
    <div class="w-75 p-4 mb-4 border">
        <hero-attribute-editor></hero-attribute-editor>
        <form method="POST" action="{% url 'character' %}">
            {% csrf_token %}

            {{ form.strength.as_hidden }}
            {{ form.dexterity.as_hidden }}
            {{ form.health.as_hidden }}
            {{ form.intelligence.as_hidden }}
            {{ form.charisma.as_hidden }}

            <button type="submit" class="btn btn-primary">Submit</button>
        </form>
    </div>
</div>
{% endblock %}

If you were to view this page rendered in the browser right now, the hero-attribute-editor tag would not work. You'll need to create this web component with React in the next few steps. The main.js and main.css files will contain the React build output for the hero editor tag.

And finally, you will create the character.html template. This template simply allows the user to view the attributes of the hero that they have created. Open hero/templates/hero/character.html and place the following there.

{% extends "hero/base.html" %}
{% load static %}

{% block content %}
<div class="d-flex justify-content-center align-items-center vh-100">
    <div class="w-75 p-4 mb-4 border">
        <table class="table">
            <thead>
                <tr>
                    <th>Attribute</th>
                    <th>Value</th>
                </tr>
            </thead>
            <tbody>
                <tr>
                    <td>Strength</td>
                    <td>{{ character.strength }}</td>
                </tr>
                <tr>
                    <td>Dexterity</td>
                    <td>{{ character.dexterity }}</td>
                </tr>
                <tr>
                    <td>Health</td>
                    <td>{{ character.health }}</td>
                </tr>
                <tr>
                    <td>Intelligence</td>
                    <td>{{ character.intelligence }}</td>
                </tr>
                <tr>
                    <td>Charisma</td>
                    <td>{{ character.charisma }}</td>
                </tr>
            </tbody>
        </table>
    </div>
</div>
{% endblock %}

Setting up the React Project

In this section you'll set up the React project and create a web component for use in the Django template.

First create the directory that you'll use to contain the project.

mkdir hero-attribute-editor

Now create a package.json to list the project dependencies and build commands.

{
  "name": "hero-attribute-editor",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "webpack"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@r2wc/react-to-web-component": "^2.0.2",
    "rc-slider": "^10.2.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "devDependencies": {
    "@babel/core": "^7.22.10",
    "@babel/preset-env": "^7.22.10",
    "@babel/preset-react": "^7.22.5",
    "babel-loader": "^9.1.3",
    "css-loader": "^6.8.1",
    "style-loader": "^3.3.3",
    "webpack-cli": "^5.1.4"
  }
}

The most important dependency here is the r2wc/react-to-web-component library. This library acts as the bridge between a React component and the Web Component API. The concept of React props, for example, requires special handling to make possible when working in the web component medium.

Next create the index.js that is the entrypoint defined in the package.json file. The index.js file should contain the following.

import React from 'react';

import r2wc from '@r2wc/react-to-web-component';
import AttributeEditor from './components/AttributeEditor';

const wcAttributeEditor = r2wc(AttributeEditor, { props: {} });

customElements.define("hero-attribute-editor", wcAttributeEditor);

The r2wc library takes the AttributeEditor component and wraps it to create a Web Component compatible object. You then register that web component with the browser using the customElements function. The customElements function is part of the Web Components API.

The next file implements the AttributeEditor component using React. The component renders a series of sliders representing each character attribute. Each attribute can have up to 100 points assigned to it, but there are only 100 total points available. When all points are used, any additional points assigned to an attribute will be automatically deducted from other attributes.

The component will issue a custom event on every attribute change. We'll see later how those events are intercepted in the Django template.

Place the following in the components/AttributeEditor.jsx file.

import React, { useState } from "react";

import Slider from "rc-slider";
import "rc-slider/assets/index.css";

import "./AttributeEditor.css";

const TOTAL_POINTS = 100;

function AttributeEditor() {
  const [attributes, setAttributes] = useState({
    Strength: 20,
    Dexterity: 20,
    Health: 20,
    Intelligence: 20,
    Charisma: 20,
  });

  const remainingPoints =
    TOTAL_POINTS - Object.values(attributes).reduce((a, b) => a + b, 0);

  const handleSliderChange = (attribute, value) => {
    setAttributes((prev) => {
      const newAttributes = { ...prev, [attribute]: value };
      let totalUsed = Object.values(newAttributes).reduce((a, b) => a + b, 0);

      if (totalUsed > TOTAL_POINTS) {
        let excess = totalUsed - TOTAL_POINTS;
        let attributesToAdjust = Object.keys(newAttributes).filter(
          (key) => key !== attribute
        );

        while (excess > 0) {
          for (let key of attributesToAdjust) {
            if (newAttributes[key] > 0 && excess > 0) {
              newAttributes[key] -= 1;
              excess -= 1;
            }
          }
        }
      }

      window.dispatchEvent(
        new CustomEvent("heroAttributesChange", {
          detail: newAttributes,
          bubbles: true,
          composed: true,
        })
      );

      return newAttributes;
    });
  };

  return (
    <div>
      <div className="total">Total Points Left: {remainingPoints}</div>
      {Object.entries(attributes).map(([attribute, value]) => (
        <div className="attribute" key={attribute}>
          <label htmlFor={attribute}>{attribute}</label>
          <Slider
            value={value}
            min={0}
            max={100}
            onChange={(val) => handleSliderChange(attribute, val)}
          />
        </div>
      ))}
    </div>
  );
}

export default AttributeEditor;

The AttributeEditor contains a few references to CSS classes. Create a new file at components/AttributeEditor.css and insert the following.

.attribute {
  margin: 65px 0;
}

.attribute label {
  display: block;
  margin-bottom: 10px;
}

.total {
  font-weight: bold;
  text-align: center;
  margin-bottom: 30px;
}

Babel requires a config file named .babelrc, where you'll define preset-react as one of the loaded presets. The React preset loads a collection of plugins that are capable of transforming JSX into JavaScript for the final build output.

{
  "presets": [
    "@babel/preset-env", 
    ["@babel/preset-react", {"runtime": "automatic"}]
  ]
}

Finally, you'll create a webpack.config.js file to instruct Webpack how to build the project.

The config defines index.js as the build entrypoint, and sets up Webpack to compile any JSX or CSS files that it finds. The config also strips the React runtime out of the compiled output, as the Django template will use a script tag to download the React runtime from a CDN.

const path = require('path');

module.exports = {
  entry: './index.js',
  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: 'index.js',
    libraryTarget: 'umd'
  },
  module: {
    rules: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        use: 'babel-loader'
      },
      {
        test: /\.css$/,
        use: ['style-loader', 'css-loader']
      }
    ]
  },
  resolve: {
    extensions: ['.js', '.jsx']
  },
  externals: {
    'react': 'React',
    'react-dom': 'ReactDOM'
  }
};

You should now be able to build the React project. You can do so using an npm command.

npm run build

This should result in a dist directory that contains an index.js build file. The entire React component and its CSS are compiled into this file.

Now you should copy this output file into the appropriate Django static directory.

cp dist/index.js ../hero_builder/hero/static/hero/js/main.js

Start the Django dev server if it isn't already running and visit http://localhost:8000 again to view the app.

Intercepting Custom Events in the Django Template

The final piece of the puzzle is gathering the data from the hero-attribute-editor and POSTing it to Django.

Django doesn't know anything about the web component, so you'll need to construct a hidden form to POST the data. You'll also use a small amount of JavaScript to update the hidden form inputs whenever the custom event is received.

Open the hero_builder/hero/templates/hero/home.html template and update it with the following.

{% extends "hero/base.html" %}
{% load static %}

{% block extra_js %}
<script src="{% static 'hero/js/main.js' %}"></script>
{% endblock %}

{% block extra_css %}
<link href="{% static 'hero/css/main.css' %}" rel="stylesheet">
{% endblock %}

{% block content %}
<div class="d-flex justify-content-center align-items-center vh-100">
    <div class="w-75 p-4 mb-4 border">
        <hero-attribute-editor></hero-attribute-editor>
        <form method="POST" action="{% url 'character' %}">
            {% csrf_token %}

            {{ form.strength.as_hidden }}
            {{ form.dexterity.as_hidden }}
            {{ form.health.as_hidden }}
            {{ form.intelligence.as_hidden }}
            {{ form.charisma.as_hidden }}

            <button type="submit" class="btn btn-primary">Submit</button>
        </form>
    </div>
</div>
<script>
const attributes = [
    "Strength",
    "Dexterity",
    "Health",
    "Intelligence",
    "Charisma"
];

window.addEventListener("heroAttributesChange", (e) => {
    for (const attribute of attributes) {
        const lowercasedAttribute = attribute.toLowerCase();
        const value = e.detail[attribute];

        const inputElement = document.getElementById(`id_${lowercasedAttribute}`);
        if (inputElement) {
            inputElement.value = value;
        }
    }
});
</script>
{% endblock %}

The final script tag in the template will listen for heroAttributesChange events from the web component and update the hidden form. On form submission, Django will handle the form as it would any other form.

GitHub Repositories

You can find the code for the Django application on GitHub here. The code for the hero attribute editor component can also be found on GitHub.

Conclusion

You should now have a working example of including a React-based web component in a Django application. This approach still uses template rendering with Django, but allows for progressive enhancement where the requirements demand more interactivity.

HTML to PDF Using Python and Headless Chrome

Zach — Sun, 13 Aug 2023 20:35:42 +0000

Converting an HTML document to PDF doesn't have to be difficult. In fact, it's pretty easy if you're building that functionality into a Python app.

Invoking Headless Chrome to Print to PDF

By far the easiest method for printing HTML to PDF programmatically is to use a headless Chrome browser.

If you haven't played with headless Chrome before, it's basically the Chrome you're used to, but launched without a GUI. Typically the browser is controlled programmatically. This is commonly done to perform automated testing, or to scrape websites. In our case, we'll instruct Chrome to print a document to PDF and then immediately exit.

There are other tools for HTML conversion, such as wkhtmltopdf, but in my experience I've found Chrome to be the easiest to work with.

Because Chrome will do the heavy lifting, we only need to worry about invoking it properly, and making sure files are where the browser expects them. The following is a simple utility function for invoking the Chrome browser in headless mode.

import subprocess
import tempfile

def html_to_pdf(html_file_path):
    pdf_file = tempfile.NamedTemporaryFile(delete=False, suffix=".pdf")
    pdf_file.close()

    command = [
        "google-chrome-stable",
        "--headless",
        "--no-sandbox",
        "--disable-gpu",
        "--print-to-pdf=" + pdf_file.name,
        html_file_path,
     ]

     process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
     process.communicate()

     return pdf_file_path

Dedicated CSS for the Printed Version

Making sure your content fits onto the PDF and isn't clipped off is one potential stumbling block.

We can target CSS specifically for the printed page using the @page rule. You might have to experiment here, but the size property can be adjusted to control the page width and height.

Here is an example of using the @page rule.

@page {
    size: 13in 6.5in;
}

How content wraps from one page to the next is also an issue unique to the printed PDF version.

As an example, if you have a list of products and you'd like to make sure each page has a single product, you can use the break-before property to accomplish this.

.product {
    break-before: page
}

You can also do the opposite if you want to ensure that some content is never broken across pages. You might have a block quote, for example, that should always be displayed on one page. The break-inside property controls this behavior.

.blockquote {
    break-inside: avoid;
}

Bonus Points: Dockerizing the Process

What we've done so far works fine for a one-off script. But what if we want to incorporate this into a deployed application? The biggest problem is ensuring that Chrome is available in the application environment.

Assuming we're working with Docker, we can write a Dockerfile that will take care of installing Chrome and its dependencies.

FROM python:3.9

ENV DEBIAN_FRONTEND=noninteractive

# Install Chrome and its dependencies
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        wget \
        ca-certificates \
        apt-transport-https \
        gnupg \
    && wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] https://dl.google.com/linux/chrome/deb/ stable main" > /etc/apt/sources.list.d/google.list \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
        google-chrome-stable \
    # Cleanup
    && apt-get purge --autoremove -y wget \
    && rm -rf /var/lib/apt/lists/* \
    && rm -rf /etc/apt/sources.list.d/google.list

You can combine this Dockerfile with whatever extra dependencies you have, such as Python packages installed through a requirements.txt file. We explicitly leave out the CMD argument so that the container can be launched with whatever command is necessary. Chrome will be available within the container, and the html_to_pdf function should work properly.

If you're incorporating this conversion into a Django app, you might want to consider doing the conversion as part of a Celery task.

One or two page documents can be converted very quickly, but long documents may cause too much burden for your web server and are better off-loaded to a Celery worker.

Working with Multiple Celery Queues in Django

Zach — Thu, 10 Aug 2023 12:08:09 +0000

Celery is an asynchronous task queue system based on distributed message passing. It allows developers to execute background jobs concurrently, separate from the primary application flow, thus ensuring tasks like sending emails, generating reports, or processing data do not block or delay a user's request.

By offloading these heavyweight tasks to Celery, web frameworks like Django can quickly respond to the user and allow the task to run in the background, ensuring that users are not kept waiting.

While employing a single queue might suffice for smaller applications, as your project grows and tasks become more varied, you may find the need to prioritize certain tasks or run them on specialized workers. In this article we'll dive into how to use multiple queues with Django, both during local development and when deploying to production.

1. Integrating Celery with Django
2. Local Development with Django and Celery
- 2.1 Using Docker Compose to Manage the Containers
- 2.2 Implementing a Simple Celery Task
- 2.3 Using Watchdog for Celery Worker Hot Reload
- 2.4 Adding Multiple Queues in Local Development
3. Production Deployment with Django and Celery
- 3.1 Using a GitHub Actions Workflow to Trigger Ansible
- 3.2 Using Systemd to Manage Celery Workers
- 3.3 Deploying Celery with Ansible
4. Multiple Celery Queues Example App

Integrating Celery with Django

We first need to set up our Django project to work with Celery before adding multiple queues.

The first step to adding Celery to Django is to make sure the necessary dependencies are installed. We could install them on the command line, but instead we'll use a requirements.txt file to make the process repeatable.

Django==4.1.7
celery==5.2.7
redis==4.5.5
gunicorn==20.1.0

Now create a file named celery.py in the same directory containing your settings.py file. This file will act as the entry point for each Celery worker. Config settings are loaded here and the worker is told where to find task definitions. The celery.py file can also optionally contain scheduled tasks configuration.

import os
from datetime import timedelta

from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")

app = Celery("myapp")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

app.conf.beat_schedule = {
    "run-every-10-seconds": {
        "task": "myapp.tasks.foo",
        "schedule": timedelta(seconds=10),
    },
    "run-every-1800-seconds": {
        "task": "myapp.tasks.bar",
        "schedule": timedelta(seconds=1800),
    },
}

Finally, we need to make a small update to settings.py to include the CELERY_BROKER_URL variable. The broker URL will contain a connection string that points to a Redis instance. This Redis instance will be responsible for managing the queues that contain the pending task data.

CELERY_BROKER_URL = os.environ.get("CELERY_BROKER_URL")

Using an environment variable will allow the local and production deployments to work without code changes.

With that final tweak our Django project should be almost ready to work with Celery. Next we'll see how to containerize this setup and run it locally using Docker.

Local Development with Django and Celery

We can use Docker and Compose together to make local development with Django and Celery easier. This method also makes it easier to collaborate on the project with other developers.

The first step is to containerize the Django and Celery worker processes. To do this, we'll write a Dockerfile that contains the instructions to create an image with the necessary dependencies.

FROM python:3.9-slim-buster

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

RUN apt-get update && apt-get install -y \
    libpq-dev \
    && apt-get -y clean

WORKDIR /usr/src/app

COPY ./requirements.txt /usr/src/app

RUN pip install --upgrade pip && pip install -r requirements.txt

COPY . /usr/src/app/

If you're familiar with the Dockerfile format, you may notice there is no CMD argument at the end. This is intentional, as we'll use docker-compose to specify the command later. This will allow one Dockerfile to work for both the Django and Celery containers.

Using Docker Compose to Manage the Containers

This is a simple docker-compose.yml file that will configure a Django dev server, Celery worker, and Redis instance. We use the dev server here because we'd like to take advantage of the hot reload feature. The production deployment will use Gunicorn to serve the application.

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: django_dev
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ${PWD}:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
    ports:
      - "8000:8000"
    depends_on:
      - redis

  celery:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: celery_worker
    command: celery -A myapp worker --loglevel=info
    volumes:
      - ${PWD}:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
    depends_on:
      - redis

  redis:
    image: redis:latest
    container_name: redis
    ports:
      - "6379:6379"

Starting and running this collection of containers requires just one command, which will also build images and download any remote images, such as for the Redis container.

docker-compose up -d

Implementing a Simple Celery Task

Now that we have all the containers up and running, we should take a moment to implement a proof-of-concept task.

We'll add a new file named tasks.py to the multiple_queues_app/ directory and implement a simple async task. The task will compute the Nth Fibonacci number, albeit in a very inefficient recursive fashion.

from celery import shared_task

@shared_task
def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n - 1) + fibonacci(n - 2)

Now we need a view that we can call to invoke this task and process it in the background. Add the following code in the multiple_queues_app/views.py file.

from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from .tasks import fibonacci

@csrf_exempt
def calculate_fibonacci(request):
    if request.method == "POST":
        try:
            N = int(request.POST.get("N"))
            result = fibonacci.delay(N)
            return JsonResponse(
                {"task_id": result.task_id, "status": "Task submitted successfully!"},
                status=200,
            )
        except ValueError:
            return JsonResponse({"error": "Invalid N value provided."}, status=400)
    else:
        return JsonResponse({"error": "Only POST method is allowed."}, status=405)

Of course, we're almost there, but we need to add a urls.py to the app, and update the main project urls.py as well before this view will respond.

from django.urls import path
from . import views

urlpatterns = [
    path("fibonacci/", views.calculate_fibonacci, name="calculate_fibonacci"),
]

from django.contrib import admin
from django.urls import include, path

urlpatterns = [
    path("admin/", admin.site.urls),
    path("", include("multiple_queues_app.urls")),
]

With the view and task in place, we can now give this all a try. First issue a docker-compose restart so that the Celery worker code is updated. Now we can use any tool of our choice to test the view, but I'll use curl to make a request.

curl -X POST -d "N=25" http://localhost:8000/fibonacci/

Watching the logs of the Celery worker, we should see the task be enqueued and take about 3 seconds to execute.

Using Watchdog for Celery Worker Hot Reload

You may have noticed that we had to restart the Celery worker manually before we could try out the new task. Manually restarting the worker is going to become cumbersome very quickly. Ideally, we could have functionality similar to hot reload in the Django dev server.

Luckily, the watchdog utility provides an easy way to restart a process when a directory changes. Let's modify the docker-compose.yml file to use the watchmedo script of the watchdog utility, in the command section for Celery.

  celery:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: celery_worker
    command: watchmedo auto-restart --directory=multiple_queues_app --pattern=*.py --recursive -- celery -A myapp worker --loglevel=info
    volumes:
      - ${PWD}:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
    depends_on:
      - redis

Before updating the containers, however, we first need to add a requirements-dev.txt file and modify the Dockerfile to install from it.

FROM python:3.9-slim-buster

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

RUN apt-get update && apt-get install -y \
    libpq-dev \
    && apt-get -y clean

WORKDIR /usr/src/app

COPY ./requirements.txt /usr/src/app
COPY ./requirements-dev.txt /usr/src/app

RUN pip install --upgrade pip && pip install -r requirements.txt && pip install -r requirements-dev.txt

COPY . /usr/src/app/

With the containers rebuilt and restarted, you should now be able to modify Python files and see the Celery worker restart automatically.

Adding Multiple Queues in Local Development

How can we update this local development setup to include multiple Celery queues?

The process is as simple as adding another service to the docker-compose.yml file. The problem here, however, is that your Compose file can start to become long and full of duplication. Instead of copying the entire Celery service block over and over, we can use some templating magic to keep things DRY.

Below is the updated docker-compose.yml with a new Celery worker for a specific queue added.

version: '3.8'

x-worker-opts: &worker-opts
  build:
    context: .
    dockerfile: Dockerfile
  volumes:
    - ${PWD}:/usr/src/app
  environment:
    - CELERY_BROKER_URL=redis://redis:6379/0
  depends_on:
    - redis

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: django_dev
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - ${PWD}:/usr/src/app
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
    ports:
      - "8000:8000"
    depends_on:
      - redis

  fibonacci-worker:
    command: tools/start_celery.sh -Q fibonacci --concurrency=1
    <<: *worker-opts

  prime-worker:
    command: tools/start_celery.sh -Q prime --concurrency=1
    <<: *worker-opts

  redis:
    image: redis:latest
    container_name: redis
    ports:
      - "6379:6379"

We also need to tweak the Python task code as well. Previously, all tasks were sent to a single worker that handled everything. Now that we're dividing work into separate queues, we need to specify which queue should handle a task.

from celery import shared_task

@shared_task(queue="fibonacci")
def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n - 1) + fibonacci(n - 2)

@shared_task(queue="prime")
def nth_prime(n):
    count = 0
    num = 1

    while count < n:
        num += 1
        if num < 2:
            continue
        if any(num % i == 0 for i in range(2, int(num**0.5) + 1)):
            continue
        count += 1

    return num

Production Deployment with Django and Celery

There are many, many ways to deploy Celery workers to a production environment, but in this article I'll focus on a relatively simple technique. For this article we'll go through the process of deploying with a combination of GitHub Actions and Ansible.

Ansible is a reliable and time tested framework for applying configuration to remote machines over SSH. Users define playbooks that configure applications and services on remote machines. Ansible takes these playbooks and checks that the target machine matches the desired configuration, installing and updating whatever is necessary to bring the machine up to date.

You might ask why not use Docker and Compose for the production server as well? That can be a legitimate choice, but a tool like Ansible makes it possible to deploy your application across multiple machines. Doing the same with Docker most likely means using a tool like Kubernetes or Nomad, which increases the complexity level significantly.

Because Ansible works through SSH, we first need to grant the tool access to the target machine.

In a real production scenario, you would probably want to use a virtual machine image that has a specific Ansible user already provisioned. For testing purposes, I've simply started a Digital Ocean droplet with a root user. We'll generate a new SSH key pair locally and then copy the public key to the target machine. The private key will reside on GitHub as a secret.

ssh-keygen -f ansible-key
ssh-copy-id -i ansible-key.pub root@24.199.126.163

You should of course change the IP address to that of a machine you control. The ssh-copy-id program assumes that your local user can already log in with a valid SSH key or password. The point here is to set up a newly generated key-pair that only Ansible will use.

Now that the public key has been uploaded to the target machine, we'll store the private key on GitHub so that Ansible can log in while running inside an Actions pipeline. From the command above, we named the key-pair as ansible-key and ansible-key.pub -- we need to upload the private (i.e. non .pub) key to GitHub.

Using a GitHub Actions Workflow to Trigger Ansible

Before diving into the Ansible itself, we'll need something to trigger and execute the playbook. GitHub Actions are a convenient tool for this purpose. For this simple tutorial, we'll run the Ansible playbook on every push to the main branch. More robust setups could use the branch name to determine whether to deploy to a staging environment or to production.

For now, however, here is a simple deploy.yml that initiates the Ansible playbook. It also demonstrates how to pass GitHub secrets into Ansible.

name: Deploy

on:
  push:
    branches:
      - main

jobs:
  update_celery:
    runs-on: ubuntu-latest
    environment: production
    container: willhallonline/ansible:alpine
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Set up SSH key
        working-directory: ansible
        env:
          ANSIBLE_SSH_PRIVATE_KEY: ${{ secrets.ANSIBLE_SSH_PRIVATE_KEY }}
        run: |
          echo "$ANSIBLE_SSH_PRIVATE_KEY" > id_rsa
          chmod 600 id_rsa

      - name: Run Ansible playbook
        working-directory: ansible
        run: ansible-playbook -i inventory.ini playbook.yml --tags "app,redis,celery" --extra-vars "redis_password=${{ secrets.REDIS_PASSWORD }}"

Using Systemd to Manage Celery Workers

Within our Ansible playbook we'll have a few tasks responsible for setting up Celery workers and keeping the code updated. I find it useful to create an ansible directory to store all of this configuration.

Here's an overview of the directory structure, to give an idea of what's involved with Ansible in our project.

ansible
├── ansible.cfg
├── inventory.ini
├── playbook.yml
└── roles
    ├── app
    │   ├── tasks
    │   │   └── main.yml
    │   └── templates
    │       ├── gunicorn.conf.py.j2
    │       ├── gunicorn-run-dir.service.j2
    │       └── gunicorn.service.j2
    ├── celery
    │   ├── tasks
    │   │   └── main.yml
    │   └── templates
    │       ├── celery-beat.service.j2
    │       └── celery.service.j2
    └── redis
        └── tasks
            └── main.yml

There are two primary tasks, one of which does the initial setup and installation of Redis, and another to configure the Celery workers. Since this approach only uses Docker for local development, we'll use the systemd service to orchestrate Celery. If you've never used systemd, don't worry, it won't be too complicated. Using systemd is an easy way to ensure that Celery is brought back up if the VM should restart or if the worker should die.

We'll have a separate systemd service for each worker, and one for the celery-beat instance that's used for scheduled tasks. I won't show both config files, as they are very similar, but below is the systemd unit file for the Celery worker.

[Unit]
Description=Celery Worker {{ item.name }} for Multiple Queues App
After=network.target

[Service]
Environment="CELERY_BROKER_URL=redis://:{{ redis_password }}@localhost:6379/0"
Environment="PYTHONUNBUFFERED=1"
Type=simple
User=root
Group=root
WorkingDirectory={{ app_path }}
ExecStart={{ app_path }}/.venv/bin/celery -A multiple_queues_app worker -Q {{ item.name }} --loglevel=info
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=celery-{{ item.name }}

[Install]
WantedBy=multi-user.target

You'll notice that the template contains Jinja tags. This will allow us to write a unit file for each separate Celery queue, keeping messy duplication to a minimum as we expand the number of queues.

Defining a new Celery queue is done at the playbook level.

- hosts: production
  become: yes 
  vars:
    app_repo: git@github.com:zchtodd/multiple_queues_app.git
    app_path: /opt/multiple_queues_app
    venv_path: /opt/multiple_queues_app/.venv
    gunicorn_config_path: /etc/systemd/system/gunicorn.service
    run_dir_maker_config_path: /etc/systemd/system/gunicorn-run-dir.service
    celery_workers:
      - name: fibonacci

To create a new type of queue, you can add the name to the celery_workers list and Ansible will take care of writing the config files via the templating system.

Deploying Celery with Ansible

The next step is to use Ansible to create the Celery worker services. This process is fairly straightforward and just involves writing out the systemd config files that we saw earlier. A separate Ansible role handles cloning the app code and starting the web server process. You can find all of the details in the project GitHub repository.

- name: Create Celery logs directories
  ansible.builtin.file:
    path: "/var/log/celery/{{ item.name }}"
    state: directory
  loop: "{{ celery_workers }}"

- name: Create Celery Beat Systemd service file
  ansible.builtin.template:
    src: "{{ playbook_dir }}/roles/celery/templates/celery-beat.service.j2"
    dest: "/etc/systemd/system/celery-beat.service"
    mode: "0644"

- name: Create Celery Systemd service files
  ansible.builtin.template:
    src: "{{ playbook_dir }}/roles/celery/templates/celery.service.j2"
    dest: "/etc/systemd/system/celery-{{ item.name }}.service"
    mode: "0644"
  loop: "{{ celery_workers }}"

- name: Reload Systemd configuration
  ansible.builtin.systemd:
    daemon_reload: yes

- name: Enable and start Celery services
  ansible.builtin.systemd:
    name: "celery-{{ item.name }}"
    state: started
    enabled: yes
  loop: "{{ celery_workers }}"

- name: Enable and start Celery beat
  ansible.builtin.systemd:
    name: "celery-beat"
    state: started
    enabled: yes

- name: Restart Celery service
  ansible.builtin.systemd:
    name: "celery-{{ item.name }}"
    state: restarted
    daemon_reload: yes
  loop: "{{ celery_workers }}"

- name: Restart Celery beat
  ansible.builtin.systemd:
    name: "celery-beat"
    state: restarted
    daemon_reload: yes

Once the Celery workers have been set up, you can use the journalctl and systemctl programs to monitor and manage the processes.

To tail the logs of the Fibonacci worker, for example, you would execute the following command.

journalctl -u celery-fibonacci.service -f

Multiple Celery Queues Example App

This tutorial doesn't show all of the code, so if you'd like to see the full implementation, the example app is hosted on GitHub.

The full app should hopefully be a complete template that you can expand on.

Happy hacking!

An Opinionated Guide to DRF OAuth

Zach — Thu, 03 Aug 2023 13:07:50 +0000

In this tutorial I'll demonstrate an opinionated approach to building social authentication into a Django app using REST Framework. I say opinionated because it avoids using any social auth app, relying only on django.contrib.auth and the standalone requests-oauthlib module. After experimenting with social auth apps, I decided I preferred this approach because it felt distinctly less magical and easier to extend.

Assuming you're sold on this idea, how does it work then?

Getting Started

Dive in straightaway with creating a new Django project and app.


django-admin startproject drf_auth_tutorial
cd drf_auth_tutorial
python manage.py startapp tutorial

We have very few dependencies for this tutorial. In fact, we just need DRF and requests-oauthlib installed. We'll also use django-sslserver to enable SSL during local development. Our requirements.txt file should look like below.


django-sslserver==0.22
djangorestframework==3.14.0
requests-oauthlib==1.3.1

Next we need to add rest_framework and our app to the INSTALLED_APPS list. We'll also need rest_framework.authtoken to allow issuing authentication tokens once a successful signup or login has happened. You'll also want to add sslserver for local development, because the OAuth2 flow will require a secure connection, even during local testing.


INSTALLED_APPS = [
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    "sslserver",
    "rest_framework",
    "rest_framework.authtoken",
    "tutorial.apps.TutorialConfig",
]

For this tutorial we'll set up OAuth2 for both Google and GitHub. Before going any further, we'll need credentials from both before we can start testing.

To set up Google first go to the Google Cloud console and create a project. Once the project is created, proceed to the Credentials page and click Create Credentials. You should choose OAuth Client ID and then choose Web Application as the credential type. Here we'll configure what domains and URLs are allowed. The allowed redirect URLs must match the routes created in urls.py exactly, or Google will display an error message to your users.

We'll go through a similar process for setting up GitHub credentials. For GitHub, go to Settings and then to Developer Settings (bottom left of the page), and then select OAuth Apps. Configure your OAuth app similarly to what's shown below. You can put whatever you'd like in the Homepage URL field.

For both Google and GitHub you should have a client ID and secret. These values should go into a .env file that is not checked into source control. To pull these values into Django, we'll update settings.py to retrieve them from the environment. Add the following to the settings file.


GOOGLE_CLIENT_ID = os.environ["GOOGLE_CLIENT_ID"]
GOOGLE_CLIENT_SECRET = os.environ["GOOGLE_CLIENT_SECRET"]
GOOGLE_TOKEN_URL = "https://www.googleapis.com/oauth2/v4/token"
GOOGLE_AUTH_URL = "https://accounts.google.com/o/oauth2/v2/auth"

GITHUB_CLIENT_ID = os.environ["GITHUB_CLIENT_ID"]
GITHUB_CLIENT_SECRET = os.environ["GITHUB_CLIENT_SECRET"]
GITHUB_TOKEN_URL = "https://github.com/login/oauth/access_token"
GITHUB_AUTH_URL = "https://github.com/login/oauth/authorize"

Here's an overview of the project structure before we dive into specific files.


├── db.sqlite3
├── dev.env
├── drf_auth_tutorial
│   ├── asgi.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── manage.py
├── requirements.txt
└── tutorial
    ├── admin.py
    ├── apps.py
    ├── __init__.py
    ├── models.py
    ├── tests.py
    ├── urls.py
    ├── utils.py
    └── views
        ├── github_oauth_login_callback.py
        ├── github_oauth_login.py
        ├── github_oauth_signup_callback.py
        ├── github_oauth_signup.py
        ├── google_oauth_login_callback.py
        ├── google_oauth_login.py
        ├── google_oauth_signup_callback.py
        ├── google_oauth_signup.py
        └── __init__.py

3 directories, 25 files

Each OAuth2 provider has four Views associated with them, with one pair of files implementing login and the other the signup flow. As we look into each of these files, we'll see that they contain only a small amount of code. This is because most of the logic is reusable and centralized in the tutorial/utils.py file.

We'll start off with looking at the Google signup flow.


from django.shortcuts import redirect
from django.urls import reverse

from rest_framework.views import APIView
from tutorial.utils import google_setup

class GoogleOAuth2SignUpView(APIView):
    def get(self, request):
        # The redirect_uri should match the settings shown on the GCP OAuth config page.
        # The call to build_absolute_uri returns the full URL including domain.
        redirect_uri = request.build_absolute_uri(reverse("google_signup_callback"))
        return redirect(google_setup(redirect_uri))

When the user visits the /signup/google/ URL they are automatically redirected to the Google hosted auth page, where they'll select which account they want to use with your application.

Google will then send the user back to the redirect_uri passed to the google_setup function in the above view. The URL will also include an OAuth code that'll be used to obtain an auth token in the next step. The google_setup function belongs to the shared utils.py file that I mentioned earlier.

The utils.py file contains all of the code related to using the requests-oauthlib library. I've centralized the code there because the process of logging in and signing up are largely the same.

Here I'll share the google_setup and google_callback methods that are used in both the login and signup process.


from django.conf import settings
from requests_oauthlib import OAuth2Session


def google_setup(redirect_uri: str):
    session = OAuth2Session(
        settings.GOOGLE_CLIENT_ID,
        redirect_uri=redirect_uri,
        scope=[
            "openid",
            "https://www.googleapis.com/auth/userinfo.email",
            "https://www.googleapis.com/auth/userinfo.profile",
        ],  
    )   

    authorization_url, _ = session.authorization_url(
        settings.GOOGLE_AUTH_URL, access_type="offline", prompt="select_account"
    )   

    return authorization_url


def google_callback(redirect_uri: str, auth_uri: str):
    session = OAuth2Session(
        settings.GOOGLE_CLIENT_ID,
        redirect_uri=redirect_uri,
        scope=[
            "openid",
            "https://www.googleapis.com/auth/userinfo.email",
            "https://www.googleapis.com/auth/userinfo.profile",
        ],  
    )   

    session.fetch_token(
        settings.GOOGLE_TOKEN_URL,
        client_secret=settings.GOOGLE_CLIENT_SECRET,
        authorization_response=auth_uri,
    )   

    user_data = session.get("https://www.googleapis.com/oauth2/v1/userinfo").json()
    return user_data

Once the user has selected which account they want to use, Google will redirect their browser to the URL we built in the GoogleOAuth2SignUpView class. This URL must match one of the authorized redirect URLs set on the GCP config page. Google will send extra query parameters along with the request, including an OAuth code.

This OAuth code is embedded in what I've called the auth_uri in the class seen below, which handles the redirected request.

from django.contrib.auth.models import User

from django.http import JsonResponse
from django.urls import reverse

from rest_framework.authtoken.models import Token
from rest_framework.views import APIView

from tutorial.utils import google_callback
from tutorial.models import UserProfile

class GoogleOAuth2SignUpCallbackView(APIView):
    def get(self, request):
        redirect_uri = request.build_absolute_uri(reverse("google_signup_callback"))
        auth_uri = request.build_absolute_uri()

        user_data = google_callback(redirect_uri, auth_uri)

        # Use get_or_create since an existing user may end up signing in
        # through the sign up route.
        user, _ = User.objects.get_or_create(
            username=user_data["email"],
            defaults={"first_name": user_data["given_name"]},
        )

        # Populate the extended user data stored in UserProfile.
        UserProfile.objects.get_or_create(
            user=user, defaults={"google_id": user_data["id"]}
        )

        # Create the auth token for the frontend to use.
        token, _ = Token.objects.get_or_create(user=user)

        # Here we assume that once we are logged in we should send
        # a token to the frontend that a framework like React or Angular
        # can use to authenticate further requests.
        return JsonResponse({"token": token.key})

Note that we've extended the built-in Django User model with our own model named UserProfile that stores social auth info. In this tutorial app, every User will have an associated UserProfile in a one-to-one relationship. The UserProfile is used to store the google_id and github_id values, and at least one of them must be populated.

It's also worth noting that if an existing user tries to go through sign-up again, they are simply logged back in. You could of course choose to do something different in this case, like raise an error and inform the user that they are already signed up.

from django.core.exceptions import ValidationError
from django.contrib.auth.models import User
from django.db import models

class UserProfile(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE)
    google_id = models.CharField(max_length=255, unique=True, null=True)
    github_id = models.CharField(max_length=255, unique=True, null=True)

    def clean(self):
        if self.google_id is None and self.github_id is None:
            raise ValidationError("One of google_id or github_id must be set.")

Now that we've seen how the sign-up process with Google works, I'll show how little the login process actually differs.

from django.shortcuts import redirect
from django.urls import reverse

from rest_framework.views import APIView
from tutorial.utils import google_setup

class GoogleOAuth2LoginView(APIView):
    def get(self, request):
        # The redirect_uri should match the settings shown on the GCP OAuth config page.
        # The call to build_absolute_uri returns the full URL including domain.
        redirect_uri = request.build_absolute_uri(reverse("google_login_callback"))
        return redirect(google_setup(redirect_uri))

If this looks remarkably familiar to you, it's because it is! The only difference here from the GoogleOAuth2SignUpView class is the redirect URI that is used. Instead of getting sent back to the google_signup_callback we are sending the user to the google_login_callback URL.

We need a separate view for the login callback because the logic is of course slightly different. Namely, if there is no existing user associated with the email, we should return an error.

from django.contrib.auth.models import User

from django.http import JsonResponse
from django.urls import reverse

from rest_framework.authtoken.models import Token
from rest_framework.views import APIView

from tutorial.utils import google_callback

class GoogleOAuth2LoginCallbackView(APIView):
    def get(self, request):
        redirect_uri = request.build_absolute_uri(reverse("google_login_callback"))
        auth_uri = request.build_absolute_uri()

        user_data = google_callback(redirect_uri, auth_uri)

        try:
            user = User.objects.get(username=user_data["email"])
        except User.DoesNotExist:
            return JsonResponse(
                {"error": "User does not exist. Please sign up first."}, status=400
            )

        # Create the auth token for the frontend to use.
        token, _ = Token.objects.get_or_create(user=user)

        # Here we assume that once we are logged in we should send
        # a token to the frontend that a framework like React or Angular
        # can use to authenticate further requests.
        return JsonResponse({"token": token.key})

Notice that here we're reusing the google_callback again, so that the logic for requesting user details is not repeated.

To tie it all together, I'll show the urls.py for the tutorial app below.

from django.urls import path

from .views.google_oauth_signup import GoogleOAuth2SignUpView
from .views.google_oauth_signup_callback import GoogleOAuth2SignUpCallbackView
from .views.google_oauth_login import GoogleOAuth2LoginView
from .views.google_oauth_login_callback import GoogleOAuth2LoginCallbackView

from .views.github_oauth_signup import GitHubOAuth2SignUpView
from .views.github_oauth_signup_callback import GitHubOAuth2SignUpCallbackView
from .views.github_oauth_login import GitHubOAuth2LoginView
from .views.github_oauth_login_callback import GitHubOAuth2LoginCallbackView

urlpatterns = [
    path("signup/google/", GoogleOAuth2SignUpView.as_view(), name="google_signup"),
    path(
        "google/callback/signup",
        GoogleOAuth2SignUpCallbackView.as_view(),
        name="google_signup_callback",
    ),
    path("login/google/", GoogleOAuth2LoginView.as_view(), name="google_login"),
    path(
        "google/callback/login",
        GoogleOAuth2LoginCallbackView.as_view(),
        name="google_login_callback",
    ),
    path("signup/github/", GitHubOAuth2SignUpView.as_view(), name="github_signup"),
    path(
        "github/callback/signup",
        GitHubOAuth2SignUpCallbackView.as_view(),
        name="github_signup_callback",
    ),
    path("login/github/", GitHubOAuth2LoginView.as_view(), name="github_login"),
    path(
        "github/callback/login/",
        GitHubOAuth2LoginCallbackView.as_view(),
        name="github_login_callback",
    ),
]

I hope this has been a helpful tutorial for OAuth with Django REST Framework! I'm not showing the GitHub views here because they are very similar to the Google process, but if you'd like to see all of the code, you can always visit the GitHub repo for the tutorial project.

Django and React with Live Reload

Zach — Thu, 27 Jul 2023 11:33:12 +0000

In this post I'll show how to integrate Django and React in both your development workflow and production deployment.

What is the main problem we're trying to solve? Ideally, we can work with React as usual. That is, we can make changes and have the React app reload itself. It would also be nice if we don't have to make big changes when going to production.

So how do we accomplish these goals?

We'll serve the React build output from Django using the standard static files system. No extra apps or plugins required. To do that, we'll need to customize our React configuration, so that the build output is placed in our Django static directory instead of the usual location.

Before diving into the React config, it would be helpful to get an overview of the Django project structure.

├── app.env
├── backend
│   ├── asgi.py
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── db.sqlite3
├── docker-compose.yml
├── Dockerfile
├── frontend
│   ├── admin.py
│   ├── apps.py
│   ├── __init__.py
│   ├── models.py
│   ├── static
│   │   └── frontend
│   ├── templates
│   │   └── frontend
│   │       └── index.html
│   ├── tests.py
│   ├── urls.py
│   └── views.py
├── manage.py
├── nginx.conf
├── requirements.txt
├── sockets
│   └── gunicorn.sock
└── staticfiles

The React app is up a directory. Updated build outputs are moved into frontend/static/frontend when we do an npm run start in the React app.

Customizing React Build Config

Using create-react-app is probably the most popular way of starting a React project. Part of the appeal is that CRA hides the complexity of Webpack and gives you a good setup that will work in most circumstances. Unfortunately, that paradigm causes a bit of a dilemma. You face a daunting task if you want to customize anything. The standard advice on changing the setup involves npm run eject which will reveal all of the previously hidden Webpack configuration.

The resulting Webpack configuration is over 700 lines long!

Luckily, it doesn't have to be that hard. The CRACO project makes it possible to override specific settings without needing to run the eject command. That means we can have a small override config instead of taking on responsibility for the entire massive Webpack config.

This is the craco.config.js that relocates build outputs to the Django static directory.

const MiniCssExtractPlugin = require('mini-css-extract-plugin');
const path = require('path');

module.exports = {
  webpack: {
    configure: (webpackConfig, { env, paths }) => { 
      paths.appBuild = webpackConfig.output.path = path.resolve("../backend/frontend/static/frontend");
      webpackConfig.output.filename = "main.js";
      webpackConfig.output.publicPath = "/static/frontend/"
      webpackConfig.plugins = webpackConfig.plugins.map(plugin => {
        if (plugin.constructor.name === 'MiniCssExtractPlugin') {
          return new MiniCssExtractPlugin({
            filename: "style.css",
          });
        }

        return plugin;
      });

      return webpackConfig;
    }
  },
  devServer: {
    devMiddleware: {
      writeToDisk: true
    }
  },
};

Making React and Django Routing Play Nice

We need to serve index.html from Django and also make sure our frontend routes don't conflict with the backend.

The index.html will load React build outputs using the standard Django static files mechanism.

<!DOCTYPE html>
<html>
  {% load static %}
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="stylesheet" href="{% static 'frontend/style.css' %}" />
    <title>Django and React</title>
  </head>
  <body>
    <div id="root" class="content"></div>
    <script
      type="text/javascript"
      src="{% static 'frontend/main.js' %}"
    ></script>
  </body>
</html>

Now we need to set up a view to serve the index.html file. We also need to make sure that requests to unmatched paths end up serving the same index.html file, so that React routing will work properly.

from django.urls import path, re_path
from .views import index_view

urlpatterns = [path("", index_view), re_path(r"^.*$", index_view)]

The index_view itself is very simple and just returns the rendered index.html template.

from django.shortcuts import render

def index_view(request):
    return render(request, "frontend/index.html", context=None)

Importantly, when we include the frontend app URLs in the main urls.py file, we include it last so that any Django routes take precedence.

from django.contrib import admin
from django.urls import path, include

urlpatterns = [path("admin/", admin.site.urls), path("", include("frontend.urls"))]

Going to Production

I've set up a docker-compose.yml to mirror a production setup to prove that this also works in that context.

I'm using Nginx and Gunicorn for a mock production setup. The compose file defines Django and Nginx as the services.

version: "3.7"

services:
  nginx:
    image: nginx:1.14.2
    restart: unless-stopped
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./staticfiles:/usr/src/app/staticfiles
      - ./sockets:/usr/src/app/sockets
    ports:
      - "80:80"

  app:
    build:
      context: .
    restart: unless-stopped
    volumes:
      - $PWD:/usr/src/app
      - ./sockets:/usr/src/app/sockets
    env_file:
      - app.env

The Django app and Nginx are connected with a Unix domain socket. The Nginx config is very simple, and serves static assets out of the staticfiles directory for any URL that matches the /static/ pattern. All other requests go to the app container.

events {}

http {
    sendfile on;

    server {
        listen 80;

        location / {
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_pass http://unix:/usr/src/app/sockets/gunicorn.sock;
        }

        location /static/ {
            include /etc/nginx/mime.types;
            alias /usr/src/app/staticfiles/;
            expires 30d;
            access_log off;
        }
    }
}

This setup is fairly simple and convenient. The only downside is that Django is forced to serve index.html any time the browser reloads. Alternately, you could have Nginx serve the index.html file for unknown routes, but then routing to Django would have to happen only at a specific prefix, such as URLs beginning with /api/.

I don't think Django having to route for React paths is a big issue. Maybe it is if you're serving a huge amount of traffic? The other potential issue is that if your Django app is broken, so is your React routing (on page reloads).