DEV Community: Dani Hodovic

GraphQL-like features in Django Rest Framework

Dani Hodovic — Thu, 23 May 2024 19:49:31 +0000

I was recently working for a client that had built their backend built with GraphQL and Graphene. They were asking me how to streamline building features in Django because their lead engineers had no prior Django experience.

Having worked with a number of REST and specifically DRF backends I have to admit that I don't like GraphQL. Or perhaps I don't like how it's implemented in Django land. It feels clunkly and doesn't seem to fit the design philosophy of Django.

I don't like how GraphQL breaks from traditional HTTP patterns. HTTP as a protocol has stood the test of time across a myriad of changes in web technology. From Java servlets, to Rails, microservices and single page apps. If I had to make a bet I'd say HTTP will still be here 10 years from now
as a consequence I don't like how GraphQL breaks traditional scaling and caching patterns that we've built on top of HTTP and REST
REST has a knowledge and talent pool that's much deeper to draw from which has been standardized across many languages and frameworks. It makes Stackoverflow and ChatGPT driven development easier.
lastly and importantly I have no experience in GraphQL. I don't like re-learning things that work and generally only learn new things if they multiply productivity many times over.

In particular I don't like Graphene and Django-Graphene because:

it's not batteries included. Basic features like auth and caching are scattered across a myriad of repositories most of which are unmaintained.
it doesn't automatically generate schema and methods, like DRF serializers and viewsets do
lacks boilerplate for CRUD, making me spend a substantial amount of time on code that should be automatic or generated
it swallows errors, making debugging a mess
doesn't provide built in docs like DRF and DRF Spectacular do
doesn't provide built-in auth like DRF does
doesn't integrate with django-debug-toolbar

On the other hand frontend engineers tend to like GraphQL because it allows them to fetch data across many tables in a single query. This is great. GraphQL is also touted as a performance tool for low-bandwidth devices, but I'm skeptical if this is worth dumping REST for.

Could we combine the benefits of DRF and steal some inspiration from GraphQL...?

drf-flex-fields to the rescue!

Robert Singer has written a drf-flex-fields, a package that emulates GraphQL like query features in REST. It allows you to:

ask for what fields to you want to see, e.g users/?fields=id,first_name
remove the fields you want to omit, e.g users/?omit=id,first_name
return related resources in the same request, e.g users/?expand=organization.owner.roles

The last feature, returning related resources in a nested graph, is probably the most useful and requested feature by frontend engineers. They typically don't want to issue multiple requests and await all responses to return, either because of complexity or because of bandwidth limitations (mobile devices).

Below is an example of an app with a user model and a many-to-many relation to an organization model (a user could be a member of many organizations). I've created an example repository to demo the implementation, but below is a snippet.

from django.contrib.auth import get_user_model
from django.db import models

User = get_user_model()

class Organization(models.Model):
    name = models.CharField(max_length=50)
    users = models.ManyToManyField(to=User)

Let's request a list of users and return the related organization_set model:

curl http://localhost:8000/api/v1/users/?expand=organization_set

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json

{
    "count": 1,
    "results": [
        {
            "id": 1,
            "name": "Dani Hodovic",
            "created_at": "2024-04-22T18:20:46.263286Z",
            "organization_set": [
                {
                    "id": 1,
                    "name": "My Great Organization",
                    "created_by": 1
                }
            ]
        }
    ]
}

I could go one step further and expand the nested created_by property of the organization which refers to the parent model.

curl http://localhost:8000/api/v1/users/?expand=organization_set.created_by

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json

{
    "count": 1,
    "results": [
        {
            "id": 1,
            "name": "Dani Hodovic",
            "created_at": "2024-04-22T18:20:46.263286Z",
            "organization_set": [
                {
                    "id": 1,
                    "name": "My Great Organization",
                    "created_by": {
                        "id": 1,
                        "name": "Dani Hodovic",
                        "created_at": "2024-04-22T18:20:46.263286Z",
                    },
                }
            ]
        }
    ]
}

The magic works by extending your serializer class from rest_flex_fields.FlexFieldsModelSerializer and setting the meta property expandable_fields.

from rest_flex_fields import FlexFieldsModelSerializer

class OrganizationSerializer(FlexFieldsModelSerializer):
    class Meta:
        model = Organization
        fields = ["name", "users"]
        expandable_fields = {"users": (UserSerializer, {"many": True})}

The FlexFieldsModelSerializer class takes care of the rest!

I recommend curious frontend engineers to use the star (*) value to discover what fields can be expanded. This will attempt to expand any nested payload that inherits from FlexFieldsModelSerializer.

curl http://localhost:8000/api/v1/users/?expand=*

Be careful however, the nested expansion performs no limiting or pagination of the results, you could be returning a gigantic array if you're scanning a related table.

The Django settings for the project allow you to change the query parameters used and limit the maximum expansion depth.

Magically discovering expandable_fields

I realized that we could make the expansion magic more interesting. Given that the Django model metadata exposes the relations to other models I guessed that we could discover all of the expandable_fields in the graph, no matter the distance (level of nesting) in the model hierarchy. At the bottom of the page is a "magic" snippet that extends the FlexFieldsModelSerializer further by automatically setting the expandable_fields property to all related fields.

The result is that we can inherit from the MagicFlexFieldsModelSerializer and all related properties will be expandable by default. We don't have to statically specify expandable_fields.

from myapp.api.serializers.utils import MagicFlexFieldsModelSerializer

class UserSerializer(MagicFlexFieldsModelSerializer):
    class Meta:
        model = User
        fields = [
            "id",
            "name",
        ]

This is great in case you're running a private API that only your frontend engineers have access to, or if you're early stage and favor development speed over stability. It could be a little dangerous if you're exposing a large public API because it would allow attackers to flood your servers with traffic.

Below is the source code for the MagicFlexFieldsModelSerializer:

import difflib
import importlib
import logging
from functools import lru_cache

from django.db.models import ForeignKey, ManyToManyField
from rest_flex_fields.serializers import FlexFieldsModelSerializer


SERIALIZERS_DIR = "myapp.api.serializers"


class MagicFlexFieldsModelSerializer(FlexFieldsModelSerializer):
    """
    Magically expands all related fields by inspecting related fields for a model.
    """

    def __init__(self, *args, **kwargs):
        self.expandable_fields = discover_expandable_fields(self.Meta.model)
        super().__init__(*args, **kwargs)


def discover_expandable_fields(model):
    """
    Magically assigns `expandable_fields` from all related model fields.
    Discovers the serializer class based on the related model name. Assigns
    expandable_fields:

    The end result looks like this:
    ```


    {
        'created_by': backend.api.serializers.user_serializer.UserSerializer,
        'workspace': backend.api.serializers.workspace_serializer.WorkspaceSerializer,
        'users': (backend.api.serializers.user_serializer.UserSerializer, {'many': True}),
        'archived_by': (backend.api.serializers.user_serializer.UserSerializer, {'many': True}),
        'notified': (backend.api.serializers.user_serializer.UserSerializer, {'many': True}),
        'conversation_messages': (backend.api.serializers.message_serializer.MessageSerializer, {'many': True})
    }


    ```

    See the example at: https://github.com/rsinger86/drf-flex-fields?tab=readme-ov-file#quick-start
    """
    expandable_fields = {}
    serializer_classes = find_serializer_classes()
    for field in model._meta.get_fields():
        if isinstance(field, (ForeignKey, ManyToManyField)):
            related_model_name = field.related_model.__name__
            serializer_cls = find_closest_matching_serializer_by_name(
                related_model_name, serializer_classes
            )
            if field.related_model != serializer_cls.Meta.model:
                logging.debug(
                    f"Unable to find the correct serializer class for: {field.related_model}"
                )
                continue

            if isinstance(field, ForeignKey):
                expandable_fields[field.name] = serializer_cls
            else:
                expandable_fields[field.name] = (
                    serializer_cls,
                    {"many": True},
                )

    for field in model._meta.related_objects:
        related_model_name = field.related_model.__name__
        serializer_cls = find_closest_matching_serializer_by_name(
            related_model_name, serializer_classes
        )
        if field.related_model != serializer_cls.Meta.model:
            logging.debug(
                f"Unable to find the correct serializer class for: {field.related_model}"
            )
            continue

        accessor_name = field.get_accessor_name()
        expandable_fields[accessor_name] = (serializer_cls, {"many": True})

    return expandable_fields


def find_closest_matching_serializer_by_name(input_str, serializer_classes):
    matcher = difflib.SequenceMatcher(None, input_str, str(serializer_classes[0]))
    closest_match = serializer_classes[0]
    max_similarity = matcher.ratio()
    for cls in serializer_classes[1:]:
        matcher = difflib.SequenceMatcher(None, input_str, str(cls))
        similarity = matcher.ratio()
        if similarity > max_similarity:
            closest_match = cls
            max_similarity = similarity

    return closest_match


@lru_cache(maxsize=None)
def find_serializer_classes():
    """
    Technically you could get this from the DRF router object via the registered views.
    """
    init_module = importlib.import_module(SERIALIZERS_DIR)
    serializer_classes = [
        getattr(init_module, name)
        for name in dir(init_module)
        if callable(getattr(init_module, name)) and "Serializer" in name
    ]

    return serializer_classes

Color Django shell by development environment

Dani Hodovic — Mon, 15 May 2023 14:14:41 +0000

I often perform operational tasks such as modifying data or debugging failures using the Django shell. In fact, anything I can't do through the admin or through management commands I use the shell for. The package django-extensions provides a shell on steroids that I much prefer over the built in shell. It allows you to run a custom REPL such as IPython or Ptpython and has autocomplete, command history, automatic model class imports and a myriad of other dev tools. shell_plus is for a Django programmer what a scalpel is for a surgeon.

Most Django sites that have receive some amount of traffic have multiple development environments. Naturally there is always a production environment and usually a staging and (local) development environment. In the past I've fat-fingered and corrupted production data while thinking I was working on the staging environment. To avoid the fiasco I've built tooling to reduce the probability of future fuck-ups.

Using the shell and some Python configuration we can get visual indicators of what environment we're in. I like to mark the production environment in some tone of red to indicate danger and staging in blue to indicate that careless hacking is fine.

Local development

Staging

Production

Installing shell_plus and ptpython

To use a programmable shell we need the following ingredients:

django-extensions/django-extensions which contains the management command shell_plus and many other great tools
prompt-toolkit/ptpython a better Python REPL which will power the shell_plus experience. You could use other REPLs such as IPython or BPython, but I prefer ptpython.

Enable django_extensions by adding it to your INSTALLED_APPS

INSTALLED_APPS = (
    ...
    "django_extensions",
    ...
)

By default shell_plus will detect and use an enhanced REPL if one is installed.

Customizing ptpython

Ptpython reads from a configuration file where we can specify custom keybindings and the REPL colorscheme. The default configuration file can be copied from prompt-toolkit/ptpython/blob/master/examples/ptpython_config/config.py.

At the bottom of the configure method we'll add code that detects the current environment and changes the REPL title and colors depending the environment. I use Sentry in nearly all of my projects and filter issues based on the environment. SENTRY_ENVIRONMENT is injected by the deployment tool and we can use that variable to paint the shell distinctly for each environment. I like black for local development, blue for staging and red for production. Additionally we'll make the ptpython toolbar use bold font for for staging and production.

"""
Configuration example for ``ptpython``.
Copy this file to $XDG_CONFIG_HOME/ptpython/config.py
On Linux, this is: ~/.config/ptpython/config.py
"""
import os

from prompt_toolkit.styles import Style

__all__ = ["configure"]


def configure(repl):
    # Default configuration above...

    # Read the environment variable from what we set for Sentry
    environment = os.environ.get("SENTRY_ENVIRONMENT", "local")

    if "prod" in environment:
        custom_colorscheme = {"status-toolbar": "bg:#C70039 #000000 bold"}
        title = "PRODUCTION"
    elif "staging" in environment:
        custom_colorscheme = {"status-toolbar": "bg:#005ca4 #000000 bold"}
        title = "STAGING"
    else:
        custom_colorscheme = {"status-toolbar": "#000000"}
        title = "DEV"

    repl.install_ui_colorscheme("custom", Style.from_dict(custom_colorscheme))
    repl.use_ui_colorscheme("custom")
    repl.title = title

All that's left to do is to start the shell

./manage.py shell_plus

If you're trying out different colors and want to reflect the changes in real-time you can set the variable for the command and copy the configuration file to reflect the changes.

cp ptpython_config.py ~/.config/ptpython/config.py && SENTRY_ENVIRONMENT=production ./manage.py shell_plus

Configuring ptpython for Docker

FROM python:3.10.8

# Install your dependencies here

COPY ptpython_config.py /root/.config/ptpython/config.py

COPY . /app/

CMD ["gunicorn", "-b", "0.0.0.0:80", "config.wsgi"]

Django.WTF - discover trending Django packages

Dani Hodovic — Sun, 22 May 2022 08:52:08 +0000

https://django.wtf

I've often used djangopackages.org to explore app I could re-use in my projects. While the site is great, I thought we could build a modern variant with focus on package curation. I've been overwhelmed with the number of categories djangopackages.org offers and the poor categorization. It also includes packages aren't maintained or updated anymore.

Together with a friend, I've built a new site based on the same concept - index popular Django apps, categorize, rank and display the most important information for each package. The site contains fewer categories than djangopackages.org and we curate and categorize projects ourselves.

Since we were scraping Github for package info, we figured we could scrape contributors to Django projects as well ;). The site contains the most prolific contributors to Django projects as well who is trending right now.

If you have any ideas for features we could add to the site, let us know :)

Comparing results between Airflow runs

Dani Hodovic — Mon, 10 Jan 2022 13:36:26 +0000

The Airbnb engineering team built Airflow to automate workflows related to data engineering problems.

Given how generic the tool is, it can be used for right about anything that needs to be run on a regular schedule. Think of it as cron on steroids, but with better interoperability with the outside world (integrations with third party providers) and a better debugging experience (built in UI, metadata and logs).

I use it to automate a various tasks, for example:

I track stock and BTC prices and notify when the price fluctuates above a certain percentage
I send myself a notification when one of my Github repositories gains a star
I monitor the air pollution in the city during winter

Airflow doesn't automatically store results between DAG runs (DAGs are a fancy word for Python scripts). It stores logs, success status and other metadata, but doesn't provide a trivial way to store custom data between script executions.

Airflow does store the dag run metadata in serial fashion and the metadata contains execution date. We can use it to construct a key ($DAG_ID__$DATE) which points to custom data. I use Redis in the examples below, but any key value store will do.

To get historical DAG runs we use the Airflow internals:

def get_previous_dagrun():
    from airflow.models.dagrun import DagRun, DagRunState
    # Use the context to dynamically find the dag name. Otherwise you can just
    # use the name of the dag as it's displayed in the Airflow UI.
    ctx = get_current_context()
    dag_id = ctx["dag"].dag_id
    # DagRun.find returns dag metadata ordered by execution date. The last
    # entry corresponds to the last dag run
    dag_runs = DagRun.find(dag_id=dag_id, state=DagRunState.SUCCESS)
    if len(dag_runs) >= 1:
        return dag_runs[-1]

To construct a key based on the dag name and runtime:

def create_key(run_time):
    return f"airflow:github_stars:{run_time}"

last_run = get_previous_dagrun()
last_run_id = last_run.run_id if last_run else None
old_key = create_key(last_run_id)
# old_key = '"airflow:github_stars:scheduled__2022-01-01T08:00:00+00:00"'

Below is a full example of a script I use to check new Github stars for my repositories. It uses the Github API and Redis to store the number of stars for each day. If a repository gains a star a notification is sent to my Pushbullet (mobile) device.

import logging
from datetime import datetime

from airflow.decorators import dag, task
from airflow.operators.python import get_current_context
from airflow.providers.redis.hooks.redis import RedisHook


@dag(
    schedule_interval="0 8 * * *",
    start_date=datetime(2021, 1, 1),
    catchup=False,
)
def github_stars_notifier():
    import requests
    from pushbullet import Pushbullet

    from airflow.models import Variable

    http = requests.Session()

    def read_most_popular_repos():
        res = http.get(
            "https://api.github.com/users/danihodovic/repos?per_page=100&sort=pushed"
        )
        res.raise_for_status()
        by_stars = sorted(res.json(), key=lambda d: d["stargazers_count"], reverse=True)
        return by_stars

    def notify_pushbullet(repo_name, delta, stargazers_count):
        pb_token = Variable.get("pushbullet_token")
        verb = "gained" if delta > 0 else "lost"
        message = (
            f"{repo_name} {verb} {delta} stars ★. It now has {stargazers_count} stars"
        )
        pb = Pushbullet(pb_token)
        pb.push_note("Airflow notification ★", message)

    def create_key(repo_name, run_time):
        return f"airflow:github_stars:{repo_name}:{run_time}"

    def get_previous_dagrun():
        from airflow.models.dagrun import DagRun, DagRunState
        ctx = get_current_context()
        dag_runs = DagRun.find(dag_id=ctx["dag"].dag_id, state=DagRunState.SUCCESS)
        if len(dag_runs) >= 1:
            return dag_runs[-1]

    @task()
    def check_for_new_stars():
        ctx = get_current_context()

        redis = RedisHook("redis_github_stars_dag").get_conn()
        by_stars = read_most_popular_repos()[0:10]
        for repo in by_stars:
            repo_name = repo["full_name"]
            last_run = get_previous_dagrun()
            last_run_id = last_run.run_id if last_run else None
            old_key = create_key(repo_name, last_run_id)
            stargazers_count = repo["stargazers_count"]
            cached_value_str = redis.get(old_key)

            if cached_value_str:
                cached_value = int(cached_value_str.decode())
                delta = stargazers_count - cached_value
                logging.info(
                    f"{repo_name=} {cached_value=} {stargazers_count=} {delta=}"
                )
                if delta != 0:
                    notify_pushbullet(repo_name, delta, stargazers_count)

            # Update the new value
            new_key = create_key(repo_name, ctx["run_id"])
            # Expire the key to avoid flooding the Redis DB
            redis.set(new_key, stargazers_count, keepttl=604800)

    check_for_new_stars()


instance = github_stars_notifier()

Monitoring Django apps

Dani Hodovic — Thu, 06 May 2021 19:16:19 +0000

Every once in a while people reach out to me for advice on how to monitor their Django sites. How do you know when your site breaks? How do you fix things before your users notice?

My answer: it depends.

In this post I'll summarize the different monitoring methods I've found and used for Django projects. Some methods are language and platform agnostic - they apply if you are writing Rails or Node code, while others are specific to Django.

Crowdsourced monitoring

This is the simplest form of monitoring and the starter pack for every web developer. Simply wait until an angry user to tweets you that your site is offline and fix the bugs as the complaints come in.

Pros: cheap because it requires no work.

Cons: humiliating and costly. You might lose customers. It's bad publicity. If your boss finds out, he might shout at you.

Crowdsourced monitoring is best suited for hobby projects and startups, where uptime isn't important because your app isn't making money. Instead of building monitoring tools you can invest time in building features. Move fast and break things.

Built in Django error reports

Django is packaged with a lot of clever stuff and one of the them is built in error reports. When a user triggers a server error, Django automatically emails the site admins and includes the traceback of the error and the request context.

In order to enable error reporting you need to set the admins that will receive the report.

Every user in the ADMINS setting in settings.py will receive an email:

ADMINS = [('Jose', 'jose@devops.es'), ('Adin', 'adin@fix-the-site.se')]

You also need to configure an email server. Given how difficult it is to configure your own server I usually use third-party API's for this. Mailgun and Sendgrid are cheap and easy to integrate.

I use django-anymail as the email backend. It hooks into Django and allows you to easily swap in and swap out different email providers. That way you're provider agnostic and can avoid vendor lock-in. Did Sendgrid hike their prices? Change three lines of code and use Amazon SES.

The full documentation on email reporting is here.

Since most people use Slack as a means of communication you can use the django-slack package to send errors reports to Slack instead of email.

Alright, so what do I think about Django's built in error reports?

Pros:

useful for tracking down individual errors and bugs
suitable for all sizes of companies

Cons: It doesn't actually work if your app has crashed completely. Django
needs to be running to send the error reports. In that case you'll still get
angry tweets from users.

Healthchecks: are we alive?

OK, so we probably need something outside of our app to check that our app is indeed alive. In DevOps lingo this is called a "healthcheck". In simple terms: an external program sends a request to your app asking it if it's alive and your app either responds OK, ERROR or doesn't return a response at all. If the program receives an ERROR or gets no response, it triggers an alert (phone call, sms, email) telling you that you need to wake up and fix things.

The simplest way to implement this is to configure a health check at your root url https://mycoolapp.com. If the root url is accessible by our health checking robot it's probably accessible by other users too.

Smarter health checking

A more sophisticated approach is to add a dedicated url for health checks. When a request is sent to that url - check all external services your app depends on. This could be a check to your database, cache or microservice.

The hard job has already been done by Kristian Oellegaard who built a health checking app for Django.

Install django-health-check at at the /health/ url and configure it like so:

urlpatterns = [
    # ...
    url(r'^ht/', include('health_check.urls')),
]

INSTALLED_APPS = [
    # ...
    'health_check',                             # required
    'health_check.db',                          # stock Django health checkers
    'health_check.cache',
    'health_check.contrib.celery',              # requires celery
]

Now you need a component which sends requests to your app on a regular interval and reports if your app is unresponsive. There are two ways to go about this: deploy a tool on your own servers or use a third party provider.

Because I'm lazy and clever I outsource this job to a third party provider. If you decide to host a tool yourself, you'll have a recursive problem: who monitors your monitoring tool? Who watches the watchmen?

Luckily for us, developers loves building tools for other developers. So much, that most of them are free. Here is a list of common health check providers I use:

If you want something self-hosted and you're using Prometheus the Blackbox exporter is the way to go.

StatusCake can be configured to send Slack reports when a site goes offline:

Pros: Suitable for all project sizes. It doesn't matter if you have a blog or if you're Amazon.com. It's good to know when you're site is down. It's simple to integrate and cheap (usually free) to deploy.

Cons: A health check is basic and won't tell you detailed information about how your app is performing. It's usually a binary response of online or offline. You won't know if every 6th request is too slow or if users are trigger errors on urls other than the health check url.

Specialized monitoring tools

Alright so I usually recommend this category if you're already making money. The cost of your app being offline has to outweigh the time invested in learning and deploying specialized monitoring tools.

Most of these tools have a steep learning curve, but pay dividends when you need powerful and detailed introspection. They're useful if you want to answer questions like:

what percentage of our HTTP requests result in 5xx server errors?
what is the mean time our servers take to respond to a request?
what is the slowest middleware in our stack?
what model / database table is updated the most?
is our database or our application layer the bottleneck for scaling?

There are both open source and closed source SaaS monitoring solutions. It's a question of in-house deployment and maintainance cost vs. the cost of a subscription. I lean towards open-source tools because I'm more experienced with them.

I like Prometheus and Grafana which take time to learn, but are incredibly powerful. The open source community around these tools is strong and there are plugins available for everything from Django to Bitcoin prices.

So how does it work?

Prometheus operates in a pull-based model, where it pulls metrics from applications. Your Django app collects metrics in memory and exposes it over HTTP at the endpoint /metrics. Prometheus polls this endpoint on a regular interval and stores the exported data in it's database. Grafana is used to visualize fancy graphs of the aggregated metrics.

Prometheus stores data in a custom format and provides it's own metrics SDK's. Uriel Corfa has done the heavy lifting for us by writing an app which hooks into the Django middleware, ORM and cache layer and automatically exposes metrics generalized metrics. All you need to do is to install and configure django-prometheus.

Out of the box it provides metrics for requests & responses, cache and database operations. You can also add custom metrics to track anything you think is important, such as user signups and new monthly subscriptions. For example:

# settings.py
PROMETHEUS_USER_SIGNED_UP = Counter(
    "django_users_signed_up",
    "The number of users that signed up in total.",
)

# views.py
from django.conf import settings
from django.shortcuts import render

def sign_up_view(request):
    settings.PROMETHEUS_USER_SIGNED_UP.inc()
    return render(request, 'welcome.html')

Prometheus comes with it's own query language PromQL. We could graph the user signups in Prometheus with the following query to find out how many users signed up in the last 24 hours:

increase(django_users_signed_up_total[24h])

Pros: Gives you fine-grained data on how your app is performing. Covers almost all use cases except individual bug reports / stacktraces.

Cons: It takes time to learn, deploy and maintain a complex monitoring stack. If you're working on a hobby project or a small startups you're probably wasting your time deploying something like this.

Monitoring Celery tasks

Most medium to large scale Django projects I've worked on use Celery for processing background tasks. Usually we offload tasks that are

too slow to complete during a HTTP request / response cycle. For example: generating invoices or reading data from slow third party API's.
tasks that are scheduled to run at a later time. For example: payment reminder emails.

Since Celery tasks are executed outside of the Django context we need to monitor Celery separately from Django. I use Prometheus extensively, so much so that I wrote my own exporter for Celery metrics.

I like to keep an eye on the following metrics in a dashboard:

the total tasks executed in the last 24h
the most common tasks
the tasks that fail the most (% success rate)

I send automated alerts on the following metrics:

when more than N tasks fail in the last 10 minutes. N depends on how critical a task is. Sending an invoice = very critical, so alert me whenever this happens. Retrieve third party API data = not very critical.
when all Celery workers are dead

The exporter comes with automatically generated alerts that I've used in production for a few client sites. You can find them under the mixin subdirectory.

Advanced usage of Python requests - timeouts, retries, hooks

Dani Hodovic — Fri, 28 Feb 2020 14:18:47 +0000

The Python HTTP library requests is probably my favourite HTTP utility in all the languages I program in. It's simple, intuitive and ubiquitous in the Python community. Most of the programs that interface with HTTP use either requests or urllib3 from the standard library.

While it's easy to immediately be productive with requests because of the simple API, the library also offers extensibility for advanced use cases. If you're writing an API-heavy client or a web scraper you'll probably need tolerance for network failures, helpful debugging traces and syntactic sugar.

Below is a summary of features I've found useful in requests when writing web scraping tools or programs that extensively use JSON API's.

Request hooks
Setting base URLs
Setting default timeouts
Retry on failure
Debugging HTTP requests
Testing and mocking requests
Mimicking browser behaviors

Request hooks

Often when using a third party API you want to verify that the returned response is indeed valid. Requests offers the shorthand helper raise_for_status() which asserts that the response HTTP status code is not a 4xx or a 5xx, i.e that the request didn't result in a client or a server error.

For example

response = requests.get('https://api.github.com/user/repos?page=1')
# Assert that there were no errors
response.raise_for_status()

This can get repetitive if you need to raise_for_status() for each call. Luckily the requests library offers a 'hooks' interface where you can attach callbacks
on certain parts of the request process.

We can use hooks to ensure raise_for_status() is called for each response object.

# Create a custom requests object, modifying the global module throws an error
http = requests.Session()

assert_status_hook = lambda response, *args, **kwargs: response.raise_for_status()
http.hooks["response"] = [assert_status_hook]

http.get("https://api.github.com/user/repos?page=1")

> HTTPError: 401 Client Error: Unauthorized for url: https://api.github.com/user/repos?page=1

Setting base URLs

Suppose you are only using one API hosted at api.org. You'll end up repeating the protocol and domain for every http call:

requests.get('https://api.org/list/')
requests.get('https://api.org/list/3/item')

You can save yourself some typing by using BaseUrlSession. This allows you to specify the base url for the HTTP client and to only specify the resource path at the time of the request.

from requests_toolbelt import sessions
http = sessions.BaseUrlSession(base_url="https://api.org")
http.get("/list")
http.get("/list/item")

Note that the requests toolbelt isn't included in the default requests installation, so you'll have to install it separately.

Setting default timeouts

The requests documentation recommends that you set timeouts on all production code. If you forget to set timeouts a misbehaving server may cause your application to hang, especially considering most Python code is synchronous.

requests.get('https://github.com/', timeout=0.001)

However this becomes repetitive and may cause future table flips when you realize someone has forgot to set a timeout and halted the program in production.

Using Transport Adapters we can set a default timeout for all HTTP calls. This ensures that a sensible timeout is set even if the developer forgets to add the timeout=1 parameter to his individual call, but allows for overrides on a per-call basis.

Below is an example of a custom Transport Adapter with default timeouts, inspired by this Github comment. We override the constructor to provide a default timeout when constructing the http client and the send() method to ensure that the default timeout is used if a timeout argument isn't provided.

from requests.adapters import HTTPAdapter

DEFAULT_TIMEOUT = 5 # seconds

class TimeoutHTTPAdapter(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        self.timeout = DEFAULT_TIMEOUT
        if "timeout" in kwargs:
            self.timeout = kwargs["timeout"]
            del kwargs["timeout"]
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):
        timeout = kwargs.get("timeout")
        if timeout is None:
            kwargs["timeout"] = self.timeout
        return super().send(request, **kwargs)

We can use it like so:

import requests

http = requests.Session()

# Mount it for both http and https usage
adapter = TimeoutHTTPAdapter(timeout=2.5)
http.mount("https://", adapter)
http.mount("http://", adapter)

# Use the default 2.5s timeout
response = http.get("https://api.twilio.com/")

# Override the timeout as usual for specific requests
response = http.get("https://api.twilio.com/", timeout=10)

Retry on failure

Network connections are lossy, congested and servers fail. If we want to build a truly robust program we need to account for failures and have a retry strategy.

Add a retry strategy to your HTTP client is straightforward. We create a HTTPAdapter and pass our strategy to the adapter.

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

retry_strategy = Retry(
    total=3,
    status_forcelist=[429, 500, 502, 503, 504],
    method_whitelist=["HEAD", "GET", "OPTIONS"]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
http = requests.Session()
http.mount("https://", adapter)
http.mount("http://", adapter)

response = http.get("https://en.wikipedia.org/w/api.php")

The default Retry class offers sane defaults, but is highly configurable so here is a rundown of the most common parameters I use.

The parameters below include the default parameters the requests library uses.

total=10

The total number of retry attempts to make. If the number of failed requests or redirects exceeds this number the client will throw the
urllib3.exceptions.MaxRetryError exception. I vary this parameter based on the API I'm working with, but I usually set it to lower than 10, usually 3 retries is enough.

status_forcelist=[413, 429, 503]

The HTTP response codes to retry on. You likely want to retry on the common server errors (500, 502, 503, 504) because servers and reverse proxies don't always adhere to the HTTP spec. Always retry on 429 rate limit exceeded because the urllib library should by default incrementally backoff on failed requests.

method_whitelist=["HEAD", "GET", "PUT", "DELETE", "OPTIONS", "TRACE"]

The HTTP methods to retry on. By default this includes all HTTP methods except POST because POST can result in a new insert. Modify this parameter to include POST because most API's I deal with don't return an error code and perform an insert in the same call. And if they do, you should probably issue a bug report.

backoff_factor=0

This is an interesting one. It allows you to change how long the processes will sleep between failed requests. The algorithm is as follows:

{backoff factor} * (2 ** ({number of total retries} - 1))

For example, if the backoff factor is set to:

1 second the successive sleeps will be 0.5, 1, 2, 4, 8, 16, 32, 64, 128, 256.
2 seconds - 1, 2, 4, 8, 16, 32, 64, 128, 256, 512
10 seconds - 5, 10, 20, 40, 80, 160, 320, 640, 1280, 2560

The value is exponentially increasing which is a sane default implementation for retry strategies.

This value is by default 0, meaning no exponential backoff will be set and retries will immediately execute. Make sure to set this to 1 in to avoid hammering
your servers!.

The full documentation on the retry module is here.

Combining timeouts and retries

Since the HTTPAdapter is comparable we can combine retries and timeouts like so:

retries = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
http.mount("https://", TimeoutHTTPAdapter(max_retries=retries))

Debugging HTTP requests

Sometimes requests fail and you can't figure out why. Logging the request and response might give you insight to the failure. There are two ways to do this - either by using the built in debug logging settings or by using request hooks.

Printing HTTP headers

Changing the logging debug level greater than 0 will log the response HTTP headers. This is the simplest option, but it doesn't allow you to see the HTTP request or the response body. It's useful if you're dealing with an API that returns a large body payload that is not suitable for logging or contains binary content.

Any value that is greater than 0 will enable debug logging.

import requests
import http

http.client.HTTPConnection.debuglevel = 1

requests.get("https://www.google.com/")

# Output
send: b'GET / HTTP/1.1\r\nHost: www.google.com\r\nUser-Agent: python-requests/2.22.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Fri, 28 Feb 2020 12:13:26 GMT
header: Expires: -1
header: Cache-Control: private, max-age=0

Printing everything

If you want to log the entire HTTP lifecycle, including both the textual representation of the request and response you can use request hooks and the dump utils from requests_toolbelt.

I prefer this option any time I'm dealing with a REST based API that doesn't return very large responses.

import requests
from requests_toolbelt.utils import dump

def logging_hook(response, *args, **kwargs):
    data = dump.dump_all(response)
    print(data.decode('utf-8'))

http = requests.Session()
http.hooks["response"] = [logging_hook]

http.get("https://api.openaq.org/v1/cities", params={"country": "BA"})

# Output
< GET /v1/cities?country=BA HTTP/1.1
< Host: api.openaq.org

> HTTP/1.1 200 OK
> Content-Type: application/json; charset=utf-8
> Transfer-Encoding: chunked
> Connection: keep-alive
>
{
   "meta":{
      "name":"openaq-api",
      "license":"CC BY 4.0",
      "website":"https://docs.openaq.org/",
      "page":1,
      "limit":100,
      "found":1
   },
   "results":[
      {
         "country":"BA",
         "name":"Goražde",
         "city":"Goražde",
         "count":70797,
         "locations":1
      }
   ]
}

See https://toolbelt.readthedocs.io/en/latest/dumputils.html

Testing and mocking requests

Using third-party API's introduce a pain point in development - they're difficult to unit test. The engineers at Sentry have alleviated some of this pain by writing a library to mock requests during development.

Instead of sending the HTTP response to the server getsentry/responses intercepts the HTTP request and returns a pre-defined response you've added during tests.

It's better demonstrated with an example.

import unittest
import requests
import responses


class TestAPI(unittest.TestCase):
    @responses.activate  # intercept HTTP calls within this method
    def test_simple(self):
        response_data = {
                "id": "ch_1GH8so2eZvKYlo2CSMeAfRqt",
                "object": "charge",
                "customer": {"id": "cu_1GGwoc2eZvKYlo2CL2m31GRn", "object": "customer"},
            }
        # mock the Stripe API
        responses.add(
            responses.GET,
            "https://api.stripe.com/v1/charges",
            json=response_data,
        )

        response = requests.get("https://api.stripe.com/v1/charges")
        self.assertEqual(response.json(), response_data)

If a HTTP request that doesn't match the mocked responses is made, a ConnectionError is thrown.

class TestAPI(unittest.TestCase):
    @responses.activate
    def test_simple(self):
        responses.add(responses.GET, "https://api.stripe.com/v1/charges")
        response = requests.get("https://invalid-request.com")

Output

requests.exceptions.ConnectionError: Connection refused by Responses - the call doesn't match any registered mock.

Request:
- GET https://invalid-request.com/

Available matches:
- GET https://api.stripe.com/v1/charges

Mimicking browser behaviors

If you've written enough web scraper code, you'll notice how certain websites return different HTML depending on if you're using a browser or accessing the site programmatically. Sometimes this is an anti-scraping measure, but usually servers engage in User-Agent sniffing to find out what content best fits the device (e.g desktop or mobile).

If you want to return the same content as the browser displays you can override the User-Agent header requests sets with something Firefox or Chrome would send.

import requests
http = requests.Session()
http.headers.update({
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0"
})

Testing Cloudflare workers

Dani Hodovic — Tue, 22 Oct 2019 18:56:03 +0000

I like Cloudflare workers. They allow you to execute arbitrary Javascript as close to your users as possible. For simple use cases your scripts probably don't need to be tested, but once they grow in size and complexity you probably want to be confident that they don't break production.

Something I dislike about the serverless trend is that most platforms are hard to test and debug. Testing is poorly documented and examples are far and few between.

I wrote this blog post after implementing a non-trivial Cloudflare worker script for a client. The script used the Caching API, communicated with a backend server and manipulated HTTP headers. It required correctness tests before we rolled it out in production. I dug through the Cloudflare documentation and related blog posts to find testing examples, but there wasn't enough material out there so I figured I'd consolidate the knowledge I have on testing Cloudflare workers.

The starting point for testing workers is in the Cloudflare documentation. However I struggled get the examples to work and it didn't cover common Node.js testing techniques such as stubbing, mocking and running debuggers. This post will cover common testing techniques and scenarios I use to test Cloudflare workers.

Introducing Cloudworker
Setup
Testing
- Unit testing
- Testing with an upstream server
- Using Nock
- Running a Cloudworker server
- Spying and mocking with Sinon.js
Tips & tricks
- Debugging with NDB
- Debugging 500 errors
- Enabling the cache API
- Examples

Introducing Cloudworker

The engineering team at Dollar Shave Club developed the Cloudworker project to provide a development and testing platform for Cloudflare workers. It's a simulated execution environment, very similar to the environment Cloudflare uses in production.

Cloudworker strives to be as similar to the Cloudflare Worker runtime as possible. A script should behave the same when executed by Cloudworker and when run within Cloudflare Workers. Please file an issue for scenarios in which Cloudworker behaves differently. As behavior differences are found, this package will be updated to match the Cloudflare Worker runtime. This may result in breakage if scripts depended on those behavior differences.

Cloudworker allows you to both run a local server to run worker scripts and to run the environments programmatically, such as in Mocha test suites. We'll cover programmatic use of Cloudworker in this post.

Setup

To run through these examples you'll need a few libraries:

Mocha.js as the test runner
Chai.js for assertions
Cloudworker to run a Cloudflare worker environment
Sinon.js for spying and mocking
Express.js or Nock for a HTTP backend
Axios for HTTP testing
NDB for debugging

Testing

Unit testing requests and responses

Cloudworker allows us to unit test requests and responses by injecting requests to the worker script and observing the generated response.

Suppose we are building a simple worker script that returns a JSON response. Cloudflare workers implement the Service worker API. In practise this means that they receive a 'fetch' event containing a request and respond to the request by returning a response.

You have probably dealt with the request and response interfaces before, because they are the same objects as the ones used in the web Fetch API.

Here is our example worker:

async function handleRequest(event) {
  const body = {message: 'Hello mocha!'};
  return new Response(JSON.stringify(body), { status: 200 })
}

// eslint-disable-next-line no-restricted-globals
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event));
});

Now let's write a unit test which injects a request and runs it through the worker to return a response. The test looks like a standard mocha test. The trick is that we'll use the Cloudworker library to simulate a worker execution environment.

When creating the Cloudworker object we read our worker file and pass it as a string to the Cloudworker constructor. It then spawns a separate VM where the worker runs and exposes methods to dispatch requests to the worker.

The dispatch method takes a request and returns a promise containing the response. Using Mocha and Chai we can assert that the properties of the response object, such as HTTP body and status code are correct.

Example unit test:

const fs = require('fs');
const path = require('path');
const Cloudworker = require('@dollarshaveclub/cloudworker');
const { expect } = require('chai');

const workerScript = fs.readFileSync(path.resolve(__dirname, '../worker.js'), 'utf8');

describe('worker unit test', function () {
  this.timeout(60000);
  let worker;

  beforeEach(() => {
    // Create the simulated execution environment
    worker = new Cloudworker(workerScript);
  });

  it('tests requests and responses', async () => {
    const request = new Cloudworker.Request('https://mysite.com/api')
    // Pass the request through Cloudworker to simulate script exectuion
    const response = await worker.dispatch(request);
    const body = await response.json();
    expect(response.status).to.eql(200);
    expect(body).to.eql({message: 'Hello mocha!'});
  });
});

HTTP integration testing with an upstream server

Cloudflare workers usually communicate with an upstream backend. It could be a Rails, Django or Express server. We want to test the interaction between the worker script and the upstream backend. Since the worker uses HTTP to interact with the backend in production, we can create a simple HTTP API for testing purposes.

If you've written your backend in Node.js you could technically import it as a library and use it in tests. If not you can use Express or Nock to build a testing API that returns identical responses to your production backend.

async function handleRequest(event) {
  const { request } = event;
  // Fetch the response from the backend
  const response = await fetch(request);
  // The response is originally immutable, so we have to clone it in order to
  // set the cache headers.
  // https://community.cloudflare.com/t/how-can-we-remove-cookies-from-request-to-avoid-being-sent-to-origin/35239/2
  const newResponse = new Response(response.body, response);
  newResponse.headers.set('my-header', 'some token');
  return newResponse;
}

// eslint-disable-next-line no-restricted-globals
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event));
});

The worker will issue actual HTTP requests to the backend server running on localhost. In this case the worker adds an additional HTTP header to the response before returning it to the client. We will test that the new header is indeed added to the response.

In our tests we'll create an Express testing server. The testing server will listen on localhost and respond to the worker over HTTP.

const fs = require('fs');
const path = require('path');
const Cloudworker = require('@dollarshaveclub/cloudworker');
const { expect } = require('chai');
const express = require('express');

const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8');

function createApp() {
  const app = express();
  app.get('/', function (req, res) {
    res.json({message: 'Hello from express!'});
  });
  return app;
}

describe('worker unit test', function () {
  this.timeout(60000);
  let serverAddress;
  let worker;

  beforeEach(() => {
    const upstream = createApp().listen();
    serverAddress = `http://localhost:${upstream.address().port}`
    worker = new Cloudworker(workerScript);
  });

  it('tests requests and responses', async () => {
    const req = new Cloudworker.Request(serverAddress);
    const res = await worker.dispatch(req);
    const body = await res.json();
    expect(res.headers.get('my-header')).to.eql('some token');
    expect(body).to.eql({message: 'Hello from express!'});
  });
});

Using Nock for HTTP mocking

It's also possible to use github.com/nock/nock to simplify the creation of the testing server.

const fs = require('fs');
const path = require('path');
const Cloudworker = require('@dollarshaveclub/cloudworker');
const { expect } = require('chai');
const nock = require('nock');

const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8');

describe('upstream server test', function () {
  this.timeout(60000);
  let worker;

  beforeEach(() => {
    worker = new Cloudworker(workerScript);
  });

  it('uses Nock upstream server', async () => {
    const url = 'http://my-api.test';
    nock(url)
      .get('/')
      .reply(200, {message: 'Hello from Nock!'});

    const request = new Cloudworker.Request(url);
    const response = await worker.dispatch(request);
    const body = await response.json();
    expect(body).to.eql({message: 'Hello from Nock!'});
  });
});

Running a Cloudworker server

So far we've used the Cloudworker dispatch() method to inject Request objects and return Response objects. But what if we want to send real HTTP requests to the worker just as we would in production? Luckily Cloudworker can run as a standalone HTTP server on localhost, similar to how Cloudflare runs the worker in production. The .listen() method starts a HTTP server that binds to a random port on localhost.

We can now use our favourite HTTP client library such as axios or request to send HTTP requests to the Cloudworker.

const fs = require('fs');
const path = require('path');
const Cloudworker = require('@dollarshaveclub/cloudworker');
const { expect } = require('chai');
const axios = require('axios');

const workerScript = fs.readFileSync(path.resolve(__dirname, '../simple-worker.js'), 'utf8');

describe('http client test', function () {
  this.timeout(60000);
  let serverAddress;

  beforeEach(() => {
    const worker = new Cloudworker(workerScript);
    const server = worker.listen();
    serverAddress = `http://localhost:${server.address().port}`
  });

  it('uses axios', async () => {
    const response = await axios.get(serverAddress);
    expect(response.status).to.eql(200);
    expect(response.data).to.eql({message: 'Hello mocha!'});
  });
});

Spying and mocking with Sinon.js

Sometimes you want to test the internals of your worker script. For example, if your worker uses caching you may want to test that it returns cached responses rather than making upstream HTTP requests.

For this reason we may want to spy on the fetch HTTP client the worker uses. This allows us to assert on all the method calls made with fetch.

The Cloudworker library was written to simulate the production execution at Cloudflare at closely as possible and therefore each worker runs in a separate VM sandbox. This makes mocking difficult because we can't access or modify variables defined inside of the worker. The library instead allows you to create bindings at creation time to replace Service worker primitives with our own mock variants.

We can replace the built-in fetch HTTP client with a mock version to check how many times and with what arguments fetch was called.

worker = new Cloudworker(workerScript, {
  bindings: {
    fetch: fetchMock
  }
});

Here we'll use Sinon.js to mock the fetch API inside of the worker and assert it was called once with the expected request.

const fs = require('fs');
const path = require('path');
const Cloudworker = require('@dollarshaveclub/cloudworker');
const fetch = require('@dollarshaveclub/node-fetch');
const sinon = require('sinon');
const nock = require('nock');

const workerScript = fs.readFileSync(path.resolve(__dirname, '../upstream-worker.js'), 'utf8');

describe('unit test', function () {
  this.timeout(60000);
  let worker;
  let fetchMock;

  beforeEach(() => {
    fetchMock = sinon.fake(fetch);
    worker = new Cloudworker(workerScript, {
        // Inject our mocked fetch into the worker script
        bindings: {
          fetch: fetchMock
        }
      }
    );
  });

  it('uses Sinon.js spies to assert calls', async () => {
    const url = 'http://my-api.test';
    nock(url)
      .get('/')
      .reply(200, {message: 'Hello from Nock!'});

    const request = new Cloudworker.Request(url)
    await worker.dispatch(request);

    const expected = new Cloudworker.Request(url);
    sinon.assert.calledWith(fetchMock, expected);
  });
});

Tips & tricks

Debugging with NDB

NDB is a Node.js debugger that runs in Chrome. It allows us to stop the execution of a worker script and inspect the call stack or the scope variables.

The caveat when running a debugger in Cloudworker is that breakpoints within NDB do not work because the worker script is eval()ed at runtime. However we can workaround this by adding debugger; statements to the source code.

For example:

async function handleRequest(event) {
  const body = {message: 'Hello mocha!'};
  debugger;
  return new Response(JSON.stringify(body), { status: 200 })
}

// eslint-disable-next-line no-restricted-globals
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event));
});

Running ndb allows us to stop and inspect a worker during execution.

ndb mocha test/unit.test.js

Debugging 500 errors

By default Cloudworker will return a 500 error as a repose anytime things go wrong. This is frustrating to debug as you have to guess where the error happened and inject a debugger statement.

I like to add a helper function to automatically debug exceptions that arise in the worker process. In production the debugging code isn't executed, but in tests and development it is.

The helper function wraps the main handler and catches any errors that happen.

async function handleRequest(event) {
  const body = {message: 'Hello mocha!'};
  return new Response(JSON.stringify(body), { status: 200 })
}

// A wrapper function which only debugs errors the DEBUG_ERRORS variable is set
async function handle(event) {
  // If we're in production the DEBUG_ERRORS variable will not be set
  if (typeof DEBUG_ERRORS === 'undefined' || !DEBUG_ERRORS) {
    return handleRequest(event);
  }

  // Debug crashes in test and development
  try {
    const res =  await handleRequest(event);
    return res;
  } catch(err) {
    console.log(err);
    debugger;
  }
}

// eslint-disable-next-line no-restricted-globals
addEventListener('fetch', event => {
  event.respondWith(handle(event));
});

We use Cloudworker bindings to inject the variable DEBUG_ERRORS into the global state of the worker.

const fs = require('fs');
const path = require('path');
const Cloudworker = require('@dollarshaveclub/cloudworker');

const workerScript = fs.readFileSync(path.resolve(__dirname, '../worker-debug-errors.js'), 'utf8');

describe('unit test', function () {
  this.timeout(60000);
  let worker;

  beforeEach(() => {
    worker = new Cloudworker(workerScript, {
        bindings: {
          // Add a global variable that enables error debugging
          DEBUG_ERRORS: true,
        }
      }
    );
  });

  it('uses worker.dispatch() to test requests', async () => {
    const req = new Cloudworker.Request('https://mysite.com/api')
    await worker.dispatch(req);
  });
});

Running mocha with ndb enabled will now ensure the debugger stops anytime unexpected errors happen.

ndb mocha test/

Enabling the Cache API

By default Cloudworker does not enable the caching, presumably because it's still an experimental feature. To test caching you need to manually enable it when creating the Cloudworker object.

const worker = new Cloudworker(script, {
  enableCache: true
});

Examples

The example project can be found on Github.

SEO friendly urls in Django

Dani Hodovic — Thu, 15 Aug 2019 12:19:52 +0000

Examples sites built with Django usually use primary keys as the URL identifier for a resource. While this is the simplest and cleanest approach to resource urls, it's not optimized for SEO.

It's debatable on whether or not keywords in urls matter for Google, but it's obvious that most of the major players have adopted this technique. Below are five examples of popular sites that all use keywords in urls to improve their SEO.

Stackoverflow - https://stackoverflow.com/questions/8318911/why-does-html-think-chucknorris-is-a-color
Wikipedia - https://en.wikipedia.org/wiki/Python_(programming_language)
Amazon.com - https://www.amazon.com/Django-Unchained-Jamie-Foxx/dp/B00BJS7104/
Reddit - https://www.reddit.com/r/soccer/comments/c22vuf/referee_trolls_player/
Dev.to - https://dev.to/danihodovic/optimizing-postgres-full-text-search-with-django-42hg

If you poke around at the top 100 sites as ranked by Alexa you will notice that most sites with public facing content use some sort of keywords approach in their urls.

Besides being a possible SEO optimization strategy it also improves the chances that a user will click on your link over other links.. Both Chrome and Firefox will match and highlight parts of urls that match the keywords if the user has visited the site previously.

Show me the code

Django doesn't have views that use keywords in urls out of the box, but it's trivial to add. I'll go through a basic example with job postings that we use at findwork.dev.

Suppose we have a model for job postings that have a company name and a role.

# models.py
from django.db import models

class JobPosting(models.Model):
    company_name = models.CharField(max_length=50)
    role = models.CharField(max_length=50)

We have a view which shows each job by primary key in the url:

# views.py

from django.http import HttpResponse
from django.shortcuts import get_object_or_404

from .models import JobPosting
def job_posting_view(request, pk):
    job = get_object_or_404(JobPosting, pk=pk)
    html = f"""
    <html>
    <body>
    <h1>{job.role} at {job.company_name}</h1>
    </body>
    </html>
    """
    return HttpResponse(html)

In urls.py:

# urls.py

from django.urls import path
from web import views

urlpatterns = [
    path("<int:pk>", views.job_posting_view, name="job-posting"),
]

The urls we now use identify a resource by primary key. We want to add keywords to the url, but still include primary keys identify the resource by ID to ensure we don't break old urls and because it ensures uniqueness. Otherwise two job postings with the same role and company_name may conflict with each other.

The solution is straightforward. We'll generate keywords from the company_name and role and append them to the end of the url. Using Django's slugify to on the two fields we eliminate any whitespace or other characters that are not permitted in urls. Then we match the <slug> path converter to the slugified company_name + role. It's better explained with code:

urls.py

from django.urls import path
from web import views

urlpatterns = [
    # Match urls based on ID
    path("<int:pk>", views.job_posting_view, name="job-posting-id"),
    # Match urls based on ID + keywords
    path("<int:pk>/<slug:slug>", views.job_posting_view, name="job-posting-keywords"),
]

views.py

from django.http import HttpResponse
from django.shortcuts import get_object_or_404

from .models import JobPosting
def job_posting_view(request, pk, slug):
    job = get_object_or_404(JobPosting, pk=pk)
    html = f"""
    <html>
    <body>
    <h1>{job.role} at {job.company_name}</h1>
    </body>
    </html>
    """
    return HttpResponse(html)

That's the simplest solution I've come up with. However, it suffers one serious drawback for SEO. We now have two different urls that identify a single resource. Search engines will at random choose one version of to display in searches and it will attribute backlinks and link equity to one of the urls at random. This also applies to other sites that link to you. Your content becomes diluted with regards to "SEO points" and it impacts your page rank.

To circumvent this we need to ensure that there exists only one way to identify a resource. Since we want search engines to index the keyword-based url we want that to be considered the single source of truth.

In short:

we are going to use two urls - one identifiable by primary key and one by primary-key + url
the primary-key based url will redirect to the keyword based url to ensure the search engines attribute all the link juice to our keyword based url.

Django has the get_absolute_url method which it uses internally to simplify linking to a specific model. We'll add that to ensure the keyword based url is the definitive url for the resource. Whenever Django internally uses the reverse helper method Django will in turn call get_absolute_url and resolve it to our keyword based url.

models.py

from django.db import models
from django.urls import reverse
from django.utils.text import slugify


class JobPosting(models.Model):
    company_name = models.CharField(max_length=50)
    role = models.CharField(max_length=50)

    def get_absolute_url(self):
        # Slugify the combination of role and company_name as these may contain
        # whitespace or other characters that are not permitted in urls.
        slug = slugify(f"{self.role}-at-{self.company_name}")
        return reverse("job-posting", kwargs={"pk": self.id, "slug": slug})

views.py

def job_posting_view(request, pk, slug):
    job = get_object_or_404(JobPosting, pk=pk)

    # Redirect with a 301 in case someone uses the ID based url. This ensures
    # that the canonical url is the keyword based one and will attribute
    # backlinks and search rankings to our keyword based url.
    if request.path != job.get_absolute_url():
        return redirect(job, permanent=True)

    html = f"""
    <html>
    <body>
    <h1>{job.role} at {job.company_name}</h1>
    </body>
    </html>
    """
    return HttpResponse(html)

Class based views

For a class-based solution we need to override the get method to perform the redirect if the url doesn't contain keywords.

from django.views.generic.detail import DetailView

class JobsDetailView(DetailView):
    model = JobPosting

    def get(self, request, *args, **kwargs):
        self.object = self.get_object()

        if self.request.path != self.object.get_absolute_url():
            return redirect(self.object, permanent=True)

        return super().get(self, request, args, kwargs)

The full code example can be found on Github.

Resources:

Adding charts to Django admin

Dani Hodovic — Fri, 09 Aug 2019 15:51:48 +0000

Introduction

Django offers a functional admin UI out of the box with CRUD interface for db management. This covers most use cases for basic content and user management systems. However, it doesn't have exploratory views that show summaries or historical trends which is something you'd expect from an admin dashboard.

Luckily, the django admin app is extensible and with a few tweaks we can add interactive Javascript charts to the admin.

Problem

I wanted to get a graph overview of the email subscribers over time on findwork.dev. Is the site growing or stagnating in terms of email subscribers? How many subscribers did we have last month? What week did we gain most subscribers? Are all subscribers verifying their emails?

Using exploratory charts we can get a historical overview to how our site is performing.

I initially explored the land of ready-made Django admin apps and dashboards. The requirements were that it included charting abilities, was well documented and looked good. While all of the apps I experimented with looked better that the default admin in terms of styling, they were either lacking in documentation or were unmaintained.

xadmin - no documentation in English
django-jet - unmaintained since the core team is working on a SaaS alternative
django-grapinelli - no charting ability

That's when an idea sprung to mind: why not extend the default admin app?

Extending django-admin

The django admin app is composed ModelAdmin classes. These represent the visual view of your model in the admin interface. By default a ModelAdmin class comes with 5 default views:

ChangeList - a list view of a model collection
Add - a view that allows you to add a new model instance
Change - a view for updating a model instance
Delete - a view for confirming deletion of a model instance
History - history of actions taken on model instance

The ChangeList view is the default admin view when you want to see a particular model. I wanted to add a chart here so that any time I open the EmailSubscribers page I'd be presented with the added subscribers over time.

Suppose we have a model of email subscribers like so:

# web/models.py
from django.db import models

class EmailSubscriber(models.Model):
    email = models.EmailField()
    created_at = models.DateTimeField()

In order to present the Email Subscribers in the admin app we need to create a class which extends from django.contrib.admin.ModelAdmin.

A basic ModelAdmin would look something like this:

# web/admin.py
from django.contrib import admin
from .models import EmailSubscriber

@admin.register(EmailSubscriber)
class EmailSubscriberAdmin(admin.ModelAdmin):
    list_display = ("id", "email", "created_at") # display these table columns in the list view
    ordering = ("-created_at",)                  # sort by most recent subscriber

Let's add some subscribers so that we have an initial data set:

$ ./manage.py shell
Python 3.7.3 (default, Apr  9 2019, 04:56:51)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
from web.models import EmailSubscriber
from django.utils import timezone
from datetime import timedelta
import random
for i in range(0, 100):
    EmailSubscriber.objects.create(email=f"user_{i}@email.com", created_at=timezone.now() - timedelta(days=random.randint(0, 100)))
...
<EmailSubscriber: EmailSubscriber object (1)>
<EmailSubscriber: EmailSubscriber object (2)>
<EmailSubscriber: EmailSubscriber object (3)>
...

If we enter the ChangeList view we'll see that we've added 100 new subscribers with random creation times http://localhost:8000/admin/web/emailsubscriber/.

Suppose we want to add a chart which summarizes the number of subscribers in a bar chart over time. We want to place it above the list of subscribers so that's it's visible as soon as you enter the site.

The red area below outlines where I want to place the chart visually.

If we create a new file we can force django-admin to load our template instead the default template. Let's create an empty file in

web/templates/admin/web/emailsubscriber/change_list.html.

The naming scheme when overriding admin templates is

{{app}}/templates/admin/{{app}}/{{model}}/change_list.html.

The default ChangeList view is extensible and has multiple blocks that can be overridden to suit your needs. When inspecting the default admin template we can see that it contains blocks that can be overridden. We need to override the content block to can change what's rendered before the model table.

Let's extend the default ChangeList view and add custom text:

# web/templates/admin/web/emailsubscriber/change_list.html

{% extends "admin/change_list.html" %}
{% load static %}
{% block content %}

<h1>Custom message!</h1>

<!-- Render the rest of the ChangeList view by calling block.super -->
{{ block.super }}
{% endblock %}

Cool, we've now managed to customize the admin UI. Let's go further and add a Javascript chart using Chart.js. We'll need to override the extrahead block to add script and style elements to load Chart.js in the header.

The Chart.js code is based on their demo bar chart found here. I've slightly modified it to read time series data on the X axis.

# web/templates/admin/web/emailsubscriber/change_list.html

{% extends "admin/change_list.html" %}
{% load static %}

<!-- Override extrahead to add Chart.js -->
{% block extrahead %}
{{ block.super }}
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.min.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.bundle.min.js"></script>
<script>
document.addEventListener('DOMContentLoaded', () => {
  const ctx = document.getElementById('myChart').getContext('2d');

  // Sample data
  const chartData = [
    {"date": "2019-08-08T00:00:00Z", "y": 3},
    {"date": "2019-08-07T00:00:00Z", "y": 10},
    {"date": "2019-08-06T00:00:00Z", "y": 15},
    {"date": "2019-08-05T00:00:00Z", "y": 4},
    {"date": "2019-08-03T00:00:00Z", "y": 2},
    {"date": "2019-08-04T00:00:00Z", "y": 11},
    {"date": "2019-08-02T00:00:00Z", "y": 3},
    {"date": "2019-08-01T00:00:00Z", "y": 2},
  ];

  // Parse the dates to JS
  chartData.forEach((d) => {
    d.x = new Date(d.date);
  });

  // Render the chart
  const chart = new Chart(ctx, {
    type: 'bar',
    data: {
      datasets: [
        {
          label: 'new subscribers',
          data: chartData,
          backgroundColor: 'rgba(220,20,20,0.5)',
        },
      ],
    },
    options: {
      responsive: true,
      scales: {
        xAxes: [
          {
            type: 'time',
            time: {
              unit: 'day',
              round: 'day',
              displayFormats: {
                day: 'MMM D',
              },
            },
          },
        ],
        yAxes: [
          {
            ticks: {
              beginAtZero: true,
            },
          },
        ],
      },
    },
  });
});
</script>
{% endblock %}

{% block content %}
<!-- Render our chart -->
<div style="width: 80%;">
  <canvas style="margin-bottom: 30px; width: 60%; height: 50%;" id="myChart"></canvas>
</div>
<!-- Render the rest of the ChangeList view -->
{{ block.super }}
{% endblock %}

Voilà, we've now rendered a Chart.js chart into the django admin. The only problem is that the data is hardcoded and not derived from our backend.

Injecting chart data into admin template

The ModelAdmin class has a method called changelist_view. This method is responsible for rendering the ChangeList page. By overriding this method we can inject chart data into the template context.

The code below roughly does this:

aggregates the total number of new subscribers on a daily interval
encodes the Django QuerySet to JSON
adds the data to the template context
calls the super() method to render the page

# django_admin_chart_js/web/admin.py
import json

from django.contrib import admin
from django.core.serializers.json import DjangoJSONEncoder
from django.db.models import Count
from django.db.models.functions import TruncDay

from .models import EmailSubscriber


@admin.register(EmailSubscriber)
class EmailSubscriberAdmin(admin.ModelAdmin):
    list_display = ("id", "email", "created_at")
    ordering = ("-created_at",)

    def changelist_view(self, request, extra_context=None):
        # Aggregate new subscribers per day
        chart_data = (
            EmailSubscriber.objects.annotate(date=TruncDay("created_at"))
            .values("date")
            .annotate(y=Count("id"))
            .order_by("-date")
        )

        # Serialize and attach the chart data to the template context
        as_json = json.dumps(list(chart_data), cls=DjangoJSONEncoder)
        extra_context = extra_context or {"chart_data": as_json}

        # Call the superclass changelist_view to render the page
        return super().changelist_view(request, extra_context=extra_context)

The data should now technically be added to the template context, but we now have to use it in our chart instead of the hardcoded data.

Replace the hardcoded data in the chartData variable with data from our backend:

// django_admin_chart_js/web/templates/admin/web/emailsubscriber/change_list.html
const chartData = {{ chart_data | safe }};

Reload the page to see our beautiful chart.

Dynamically loading data using JS

In the example above we're injecting initial chart data directly into the html template. We could more interactive and fetch data after the initial page load. For this we need to:

add a new endpoint to our model admin which returns JSON data
add JS logic to make AJAX calls on a button click and to re-render the chart

Adding a new endpoint requires us to override the get_urls() method over the modeladmin and to inject our own endpoint url.

It's important to note that your custom urls should go before the default urls. The default ones are permissive and will match anything so the request will never go through to our custom method.

Our python code should now look something like this:

# web/admin.py
import json

from django.contrib import admin
from django.core.serializers.json import DjangoJSONEncoder
from django.db.models import Count
from django.db.models.functions import TruncDay
from django.http import JsonResponse
from django.urls import path

from .models import EmailSubscriber

@admin.register(EmailSubscriber)
class EmailSubscriberAdmin(admin.ModelAdmin):
    list_display = ("id", "email", "created_at")
    ordering = ("-created_at",)

    ...

    def get_urls(self):
        urls = super().get_urls()
        extra_urls = [
            path("chart_data/", self.admin_site.admin_view(self.chart_data_endpoint))
        ]
        # NOTE! Our custom urls have to go before the default urls, because they
        # default ones match anything.
        return extra_urls + urls

    # JSON endpoint for generating chart data that is used for dynamic loading
    # via JS.
    def chart_data_endpoint(self, request):
        chart_data = self.chart_data()
        return JsonResponse(list(chart_data), safe=False)

    def chart_data(self):
        return (
            EmailSubscriber.objects.annotate(date=TruncDay("created_at"))
            .values("date")
            .annotate(y=Count("id"))
            .order_by("-date")
        )

We also need to add Javascript logic to reload chart data on button click and re-render the chart. Add the following lines below the declaration of the chart variable:

  // django_admin_chart_js/web/templates/admin/web/emailsubscriber/change_list.html

  const chart = new Chart...
  ...

  // Reload chart data from the backend on button click
  const btn = document.querySelector('#reload');
  btn.addEventListener('click', async() => {
    const res = await fetch("/admin/web/emailsubscriber/chart_data/");
    const json = await res.json();
    json.forEach((d) => {
      d.x = new Date(d.date);
    });
    chart.data.datasets[0].data = json;
    chart.update();
  });

Add a html button below to chart:

{% block content %}
<!-- Render our chart -->
<div style="width: 80%;">
  <canvas style="margin-bottom: 30px; width: 60%; height: 50%;" id="myChart"></canvas>
</div>

<button id="reload" style="margin: 1rem 0">Reload chart data</button>
<!-- Render the rest of the ChangeList view -->
{{ block.super }}
{% endblock %}

Chart.js comes with different visualizations out of the box. It's easy to get going with basic charts and offers customization in case you need it.

The Chart.js docs are here and the Django admin docs are here.

The full sample code can be found on Github.

Optimizing Postgres full text search in Django

Dani Hodovic — Wed, 12 Jun 2019 11:05:46 +0000

Postgres provides great search capability out of the box. For the majority of Django apps there is no need to run and maintain an ElasticSearch cluster unless you need the advanced features ElasticSearch offers. Django integrates nicely with the Postgres search through the built in Postgres module.

For small data sets the default configuration performs well, however when your data grows in size the default search configuration becomes painfully slow and we need to enable certain optimizations to keep our queries fast.

This page will walk you through setting up Django and Postgres, indexing sample data and performing and optimizing full text search.

The examples go through a Django + Postgres setup, however the advice is generally applicable to any programming language or framework as long as it uses Postgres.

If you're already a Django veteran you can skip the first steps and jump right into the optimization.

Project setup
Creating models and indexing sample data
Optimizing search
- Specialized search column and gin indexes
- Postgres triggers
Measuring the performance improvement
Drawbacks
Conclusion

Project setup

Create the directories and setup the Django project.

mkdir django_postgres
cd django_postgres
python -m venv venv
source venv/bin/activate
pip install django
django-admin startproject full_text_search
cd full_text_search
./manage.py startapp web

Now we'll need to install 3 dependencies:

psycopg2: the Postgres client library for Python
wikipedia: a client library to retrieve Wikipedia articles
django-extensions: to simplify debugging of SQL queries

pip install psycopg2 wikipedia django-extensions

We also need to run Postgres locally. I'll use a dockerized version of Postgres here since it's easier to set up, but feel free to install a Postgres binary if you'd like.

Open full_text_search/docker-compose.yml

---
version: '2.4'
services:
  postgres:
    image: postgres:11-alpine
    ports:
      - '5432:5432'
    environment:
      # Set the Postgres environment variables for bootstrapping the default
      # database and user.
      POSTGRES_DB: "my_db"
      POSTGRES_USER: "me"
      POSTGRES_PASSWORD: "password"

The project structure should now look like the output below. We'll ignore the venv directory as that is packed with files and irrelevant for now.

$ tree -I venv
.
└── full_text_search
    ├── docker-compose.yml
    ├── full_text_search
    │   ├── __init__.py
    │   ├── __pycache__
    │   │   ├── __init__.cpython-37.pyc
    │   │   └── settings.cpython-37.pyc
    │   ├── settings.py
    │   ├── urls.py
    │   └── wsgi.py
    ├── manage.py
    └── web
        ├── admin.py
        ├── apps.py
        ├── __init__.py
        ├── migrations
        │   └── __init__.py
        ├── models.py
        ├── tests.py
        └── views.py

5 directories, 15 files

We will modify the default database settings to use Postgres instead of SQLite. In settings.py change the DATABASES attribute:

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "my_db",
        "USER": "me",
        "PASSWORD": "password",
        "HOST": "localhost",
        "PORT": "5432",
        "OPTIONS": {"connect_timeout": 2},
    }
}

We will also modify our INSTALLED_APPS to include a few apps:

django.contrib.postgres the Postgres module for Django which is required for full text search
django_extensions to print SQL logs when executing queries in Python
our web app

Open full_text_search/settings.py and modify:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Added apps below
    'django.contrib.postgres',
    'django_extensions',
    'web',
]

Start Postgres and Django.

cd full_text_search
docker-compose up -d
./manage.py runserver

If we open our browser and enter http://localhost:8000 we should see a successful installation.

Creating models and indexing sample data

Suppose we have a model which represents a Wikipedia page. For simplicity we'll use two fields: title and content.

Open full_text_search/web/models.py

from django.db import models

class Page(models.Model):
    title = models.CharField(max_length=100, unique=True)
    content = models.TextField()

Now run the migrations to create the model.

cd full_text_search
./manage.py makemigrations && ./manage.py migrate
No changes detected
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  # truncated the other output for brievity

We'll use a script to index random wikipedia articles and save the contents to Postgres.

Edit web/index_wikipedia.py

import logging
import wikipedia
from .models import Page

logger = logging.getLogger("django")

def index_wikipedia(num_pages):
    for _ in range(0, num_pages):
        p = wikipedia.random()
        try:
            wiki_page = wikipedia.page(p)
            Page.objects.update_or_create(title=wiki_page.title, defaults={
                "content": wiki_page.content
            })
        except Exception:
            logger.exception("Failed to index %s", p)

Now let's run our script to index Wikipedia. There will be errors when running the script, but don't worry about those as long as we manage to store a few hundred articles. The script will take a while to run so grab a cup of coffee and return in a few minutes.

./manage.py shell_plus

>>> from web.index_wikipedia import index_wikipedia
>>> index_wikipedia(200)

#
## A bunch of errors will be show here, ignore them.
#

>>> Page.objects.count()
183

Optimizing the search

Now suppose we want to allow users to perform a full text search on the content. We'll interactively query our dataset to test the full text search. Open a Django shell session:

$ ./manage.py shell_plus --print-sql

>>> Page.objects.filter(content__search='football')
SELECT t.oid,
       typarray
  FROM pg_type t
  JOIN pg_namespace ns
    ON typnamespace = ns.oid
 WHERE typname = 'hstore'

Execution time: 0.001440s [Database: default]

SELECT typarray
  FROM pg_type
 WHERE typname = 'citext'

Execution time: 0.000260s [Database: default]

SELECT "web_page"."id",
       "web_page"."title",
       "web_page"."content"
  FROM "web_page"
 WHERE to_tsvector(COALESCE("web_page"."content", '')) @@ (plainto_tsquery('football')) = true
 LIMIT 21

Execution time: 0.222619s [Database: default]

<QuerySet [<Page: Page object (2)>, <Page: Page object (7)>...]>

Django performs two preparatory queries and finally executes our search query. Looking at the last query we can at first glance see that the execution was ~315ms for the query execution and serialization alone. That's it far too slow when we want to keep our page load speeds in the double digits in milliseconds.

Let's take a closer look at why this query is performing so slowly. Open a second terminal where we'll use the excellent Postgres query analyzer. Copy the query from above and run EXPLAIN ANALYZE:

$ ./manage.py dbshell
psql (10.8 (Ubuntu 10.8-0ubuntu0.18.10.1), server 11.2)

Type "help" for help.

my_db=# explain analyze SELECT "web_page"."id",
my_db-#        "web_page"."title",
my_db-#        "web_page"."content"
my_db-#   FROM "web_page"
my_db-#  WHERE to_tsvector(COALESCE("web_page"."content", '')) @@ (plainto_tsquery('football')) = true
my_db-#  LIMIT 21
my_db-# ;
                                                  QUERY PLAN
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.00..106.71 rows=1 width=643) (actual time=5.001..220.212 rows=18 loops=1)
   ->  Seq Scan on web_page  (cost=0.00..106.71 rows=1 width=643) (actual time=4.999..220.206 rows=18 loops=1)
         Filter: (to_tsvector(COALESCE(content, ''::text)) @@ plainto_tsquery('football'::text))
         Rows Removed by Filter: 165
 Planning Time: 3.336 ms
 Execution Time: 220.292 ms
(6 rows)

We can see that although the planning time is quite fast (~3ms) the execution time is very slow at ~220ms.

->  Seq Scan on web_page  (cost=0.00..106.71 rows=1 width=643) (actual time=4.999..220.206 rows=18 loops=1)

We can note that the query performs a sequential scan over the entire table in order to find matching records. We can likely optimize the query by using an index.

Filter: (to_tsvector(COALESCE(content, ''::text)) @@ plainto_tsquery('football'::text))

Additionally the query normalizes the content column from text to a tsvector using to_tsvector in order to perform the full text search.

WHERE to_tsvector(COALESCE("web_page"."content", '')) @@ (plainto_tsquery('football')) = true

The tsvector type is a tokenized version of our text which normalizes the search column (more on tokenization here). Postgres needs to perform this normalization for every row and each row contains an entire Wikipedia page. This is both a CPU intensive and slow operation.

Specialized search column and gin indexes

In order avoid on-the-fly casting of text to tsvectors we'll create a specialized column which is used only for search. The column should be populated on inserts or updates. When querying is performed we'll avoid the performance penalty of casting types.

Since we can now have a tsvector type we are also able to add a gin index to speed up the query. The gin index ensures that the search will be performed with a indexed scan instead of a sequential scan over all records.

Open our web/models.py file and make modifications to the Page model.

from django.db import models
from django.contrib.postgres.search import SearchVectorField
from django.contrib.postgres.indexes import GinIndex

class Page(models.Model):
    title = models.CharField(max_length=100, unique=True)
    content = models.TextField()

    # New modifications. A field and an index
    content_search = SearchVectorField(null=True)

    class Meta:
        indexes = [GinIndex(fields=["content_search"])]

Run the migrations

Migrations for 'web':
  web/migrations/0002_auto_20190525_0647.py
    - Add field content_search to page
    - Create index web_page_content_505071_gin on field(s) content_search of model page
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, web
Running migrations:
  Applying web.0002_auto_20190525_0647... OK

Postgres triggers

Theoretically our problem is now solved. We have a Gin indexed column which should perform well when we search on it, but by doing so we have introduced another problem: the optimized content_search column needs to be kept manually in sync and updated whenever the content column updates.

Luckily for us Postgres provides an additional feature which solves this problem, namely triggers. Triggers are Postgres functions that fire when a specific action is performed on a row. We will create a trigger that populates content_search whenever a content row is created or updated. That way Postgres will keep the two columns in sync without us having to write any Python code.

In order to add a trigger we need to craft a manual Django migration. This will add the trigger function and update all of our Pages rows to ensure the trigger is fired and content_search is populated at migration time for our existing records. If you have an extremely large data set you might not want to do this in production.

Add a new migration in web/migrations/0003_create_text_search_trigger.py. Make sure to modify the previous migration in dependencies because the previously autogenerated migration is likely different for you.

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        # NOTE: The previous migration probably looks different for you, so
        # modify this.
        ('web', '0002_auto_20190524_0957'),
    ]

    migration = '''
        CREATE TRIGGER content_search_update BEFORE INSERT OR UPDATE
        ON web_page FOR EACH ROW EXECUTE FUNCTION
        tsvector_update_trigger(content_search, 'pg_catalog.english', content);

        -- Force triggers to run and populate the text_search column.
        UPDATE web_page set ID = ID;
    '''

    reverse_migration = '''
        DROP TRIGGER content_search ON web_page;
    '''

    operations = [
        migrations.RunSQL(migration, reverse_migration)
    ]

Run the migrations

$ ./manage.py makemigrations && ./manage.py migrate
No changes detected
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, web
Running migrations:
  Applying web.0003_create_text_search_trigger... OK

Measuring the performance improvement

Finally on to the fun part, let's verify that are query performs faster than before. Open a Django shell again, but when filtering the rows use the indexed content_search column rather than the normal content column.

./manage.py shell_plus --print-sql

>>> from django.contrib.postgres.search import SearchQuery
>>> Page.objects.filter(content_search=SearchQuery('football', config='english'))
SELECT t.oid,
       typarray
  FROM pg_type t
  JOIN pg_namespace ns
    ON typnamespace = ns.oid
 WHERE typname = 'hstore'

Execution time: 0.000829s [Database: default]

SELECT typarray
  FROM pg_type
 WHERE typname = 'citext'

Execution time: 0.000310s [Database: default]

SELECT "web_page"."id",
       "web_page"."title",
       "web_page"."content",
       "web_page"."content_search"
  FROM "web_page"
 WHERE "web_page"."content_search" @@ (plainto_tsquery('english'::regconfig, 'football')) = true
 LIMIT 21

Execution time: 0.001359s [Database: default]

<QuerySet []>

Query execution time down from 0.220s to 0.001s!

Let's analyze the query again to see how Postgres executes it. Copy the query from above and run it through EXPLAIN ANALYZE.

./manage.py dbshell

my_db=# explain analyze SELECT "web_page"."id",
my_db-#        "web_page"."title",
my_db-#        "web_page"."content",
my_db-#        "web_page"."content_search"
my_db-#   FROM "web_page"
my_db-#  WHERE "web_page"."content_search" @@ (plainto_tsquery('english'::regconfig, 'football')) = true
my_db-#  LIMIT 21
my_db-# ;
                                                                QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=8.01..12.02 rows=1 width=675) (actual time=0.022..0.022 rows=0 loops=1)
   ->  Bitmap Heap Scan on web_page  (cost=8.01..12.02 rows=1 width=675) (actual time=0.020..0.020 rows=0 loops=1)
         Recheck Cond: (content_search @@ '''football'''::tsquery)
         ->  Bitmap Index Scan on web_page_content_505071_gin  (cost=0.00..8.01 rows=1 width=0) (actual time=0.017..0.017 rows=0 loops=1)
               Index Cond: (content_search @@ '''football'''::tsquery)
 Planning Time: 3.061 ms
 Execution Time: 0.165 ms
(7 rows)

Here are the interesting bits:

->  Bitmap Index Scan on web_page_content_505071_gin  (cost=0.00..8.01 rows=1 width=0) (actual time=0.017..0.017 rows=0 loops=1)

Instead of a sequential scan Postgres uses an index on the content_search column.

Index Cond: (content_search @@ '''football'''::tsquery)

We also no longer perform an expensive to_tsquery operation for each row and instead use the content_search column as is.

Drawbacks

Unfortunately there are tradeoffs when using this optimization technique.

Because we're maintaining another column of our text for the sole purpose of search speed our table size takes up significantly more space. Additionally the gin index on the content_search column takes up space on it's own.
Since the search column is updated on every UPDATE or INSERT it also slows down writes to the database.

If you're constrained by memory and disk or need quick writes this technique may not be suitable for your use case. However I suspect that the majority of CRUD apps out there are OK with sacrificing disk and write speed for lightning fast search.

Conclusion

Postgres offers excellent full text search capability, but it's a little slow out of the box. In order to speed up text searches we add a secondary column of type tsvector which is a search-optimized version of our text.

We add a Gin index on the search column to ensure Postgres performs an index scan rather than a sequential scan. This reduces the query execution time by an order of magnitude.

In order to keep the text column and the search column in sync we use a Postgres trigger which populates the search column on any modifications to our text column.

The full code example can be found at Github.

Caching with Cloudflare workers

Dani Hodovic — Tue, 11 Jun 2019 12:57:02 +0000

tl;dr: Cloudflare allows you to vary their cache responses by arbitrary headers, such as Cookies or User-Agent. This requires you to have an enterprise account (~$5000). You could either pay $5000 or solve the problem with $5 and some Javascript code.

Intro

Cloudflare offers caching on their global CDN. For sites that rarely update Cloudflare handles most of the traffic for you without ever reaching your origin server. Once a user visits a specific page, Cloudflare keeps the page response in the cache and serves it for the next visitor. This reduces the load on your servers while also improving the page performance since it's served closer to the user via one of Cloudflare's POPs.

While this is great in theory, it's challenging to implement for any site that requires page customization per visitor (read: most sites). For example: we may want to serve different cache responses based on the cookie (unique caching per visitor) or on the User-Agent (unique caching per device type - mobile / tablet / desktop).

Luckily, Cloudflare allows you to vary the cache by HTTP headers:

Accept-Encoding - caches each resource by the encoding of the payload.
Cookie - allows the cache to be unique per cookie. This is useful if the cache is unique per user or session.
User-Agent - caching per User-Agent ensures that the page is cached differently per device. E.g mobile clients may receive one version of the page which desktop clients may receive another.

The catch? Only Accept-Encoding is freely available, the other two headers require you to upgrade to an enterprise plan. It's rumored that this costs around $5000. Even if you were to upgrade to an enterprise plan you still wouldn't be able to cache by other HTTP headers.

Why do I need cache variation?

It's best to explain why caching by various headers is useful by a practical example.

At findwork.dev we deliver different versions of a page based on the User-Agent. For mobile versions we omit certain parts of the page which don't fit on small screen sizes and only include them for desktop clients. We do this by checking for the User-Agent header and rendering the page differently with django-user-agent.

Here is a fictional example which renders buttons with different sizes depending on if the user is on a mobile device or a desktop device.

{% if request.user_agent.is_mobile %}
<a class="btn btn-sm">Company info</a>
{% else %}
<a class="btn btn-lg">Company info</a>
{% endif %}

Recently we enabled a Cloudflare page rule to cache everything (including the html). Unfortunately we noticed some problems right away: if one user visited a page on a mobile device Cloudflare would cache the mobile version of the page. When another user would access the same page on a desktop Cloudflare would serve the mobile optimized version because that's the page in the cache. This obviously resulted in very ugly looking pages.

Cloudflare workers to the rescue

We briefly discussed moving our CDN layer to AWS and Cloudfront (which allows for arbitrary cache variation headers). However moving our entire infrastructure to AWS just to circumvent caching is impractical.

Cloudflare recently launched Cloudflare workers. Cloudflare workers are Javascript snippets that run on the Cloudflare infrastructure. The workers can interface with various parts of the Cloudflare infrastructure, including the caching API. This meant that we could write arbitrary code to customize how Cloudflare should cache and deliver our content.

The Cloudflare docs state:

Unlike the browser Cache API, Cloudflare Workers do not support the
ignoreSearch or ignoreVary options on match(). You can accomplish this
behavior by removing query strings or HTTP headers at put() time.

So by default the HTTP Vary header is ignored in the cache key. A workaround is instead to add query parameter which distinguishes the response by device.

a request for a mobile version of findwork.dev could be cached under https://findwork.dev?version=mobile
a desktop version could be cached under https://findwork.dev?version=desktop.

Here's a code snippet we came up with which solved our problem:

async function run(event) {
  const { request } = event;

  const cache = caches.default;

  // Read the user agent of the request
  const ua = request.headers.get('user-agent');
  let uaValue;

  if (ua.match(/mobile/i)) {
    uaValue = 'mobile';
  } else {
    uaValue = 'desktop';
  }

  // Construct a new response object which distinguishes the cache key by device
  // type.
  const url = new URL(request.url);
  url.searchParams.set('ua', uaValue);
  const newRequest = new Request(url, request);

  let response = await cache.match(newRequest);
  if (!response) {
    // Use the original request object when fetching the response from the
    // server to avoid passing on the query parameters to our backend.
    response = await fetch(request);

    // Store the cached response with our extended query parameters.
    event.waitUntil(cache.put(newRequest, response.clone()));
  }

  return response;
}

window.addEventListener('fetch', (event) => {
  event.respondWith(run(event));
});

It's worth noting that this not only allows you to vary the cache by HTTP headers. You could get creative and vary the cache by the contents of the body if you wanted.

Don't enable caching in the Cloudflare UI if you're using workers. It may mess with the workers and cause inconsistencies. Either use the workers for caching or use UI / page rules.