DEV Community

Cover image for Zero-Downtime Django Deploys on One Small Droplet
Vicente G. Reyes
Vicente G. Reyes

Posted on Originally published at vicentereyes.org AI-assisted

Zero-Downtime Django Deploys on One Small Droplet

#django #docker #devops

GiggleGigs is a job board I run: Django + DRF on the backend, React + Vite on the front (on Vercel), Celery for background work, all in Docker Compose on a single 2 GB DigitalOcean droplet, with Traefik in front and Cloudflare in front of that.

In the last post I moved image builds off the droplet and into GitHub Actions. That stopped deploys from freezing the server. It didn't stop them from taking the site down.

Why 99.99% means deploys can't cause any downtime

99.99% uptime allows about 4.3 minutes of downtime a month.

I measured a normal deploy with a loop that requested the API every 5 seconds: about two minutes of no available server from Traefik. I deploy several times a day. At that rate, a single morning of deploys would use up the month's budget.

So "faster deploys" wasn't the goal. The goal was deploys with no downtime at all, which left the 4.3 minutes for things I can't control.

Measure first

Everything below is based on one simple measurement: a loop on my laptop that requested the API's health endpoint every 5 seconds while a deploy ran, plus a request to a site that's always up (google.com) on each tick as a control:

while true; do
  echo "$(date -u +%T) $(curl -s -o /dev/null -m 5 -w '%{http_code}' https://api.example.com/healthz/) $(curl -s -o /dev/null -m 5 -w '%{http_code}' https://www.google.com/)"
  sleep 5
done >> uptime.log
Enter fullscreen mode Exit fullscreen mode

The control column matters. My own connection dropped twice during the day, and without the control those drops would have looked like site outages.

Why a deploy went down

The old deploy ran docker compose up -d, which:

  1. stops the only Django container,
  2. starts a new one,
  3. whose start script runs collectstatic before gunicorn starts listening.

Traefik had nothing to send requests to until step 3 finished. That's two minutes.

Step 1: Collect static files when the image is built

collectstatic doesn't need a database or real secrets. It only needs the settings module to import. So it now runs in the Dockerfile, which runs in CI, with placeholder env vars that exist only for that RUN step:

# Collect static files at build time so a new container starts
# serving in seconds. Production settings need these env vars to import,
# but collectstatic never uses them, so placeholders are enough.
RUN DJANGO_SETTINGS_MODULE="config.settings.production" \
  DJANGO_SECRET_KEY="build-only" \
  DATABASE_URL="postgres://build:build@localhost:5432/build" \
  # ...the rest of the required settings...
  python manage.py collectstatic --noinput
Enter fullscreen mode Exit fullscreen mode

The start script went from about 25 lines to one:

exec /usr/local/bin/gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app \
  --timeout 90 --graceful-timeout 90 --no-control-socket
Enter fullscreen mode Exit fullscreen mode

A new container now passes its healthcheck in 13–20 seconds on my laptop and about 43 seconds on the droplet, instead of about two minutes.

One surprise: I also tried running django-compressor's compress at build time, and it failed with No 'compress' template tags found. None of my templates use it. It must have been switched off in production, or the old start script would have crashed. Moving work into the build is a good way to find steps that never did anything.

Step 2: A healthcheck, and a second container

Then a rolling deploy: start a second Django container on the new image, wait until it's healthy, and only then stop the old one. The new container gets a Docker healthcheck on the existing /healthz/ endpoint:

django:
  image: ghcr.io/you/app-django:${IMAGE_TAG:-latest}
  command: /start
  healthcheck:
    test:
      - CMD
      - python
      - -c
      - "import urllib.request; urllib.request.urlopen(urllib.request.Request('http://localhost:5000/healthz/', headers={'Host': 'api.example.com'}), timeout=3)"
    interval: 5s
    timeout: 5s
    retries: 3
    start_period: 60s
  stop_grace_period: 90s
Enter fullscreen mode Exit fullscreen mode

The image has no curl, so the check uses Python. The Host header has to be one of ALLOWED_HOSTS, and /healthz/ has to be in SECURE_REDIRECT_EXEMPT, or the check gets a 301 and fails.

The deploy script uses Compose's --scale and --no-recreate flags, which most people never touch:

old_container=$(docker compose ps --quiet django)
docker compose run --rm django python manage.py migrate
docker compose up --detach --no-deps --no-recreate --scale django=2 django
new_container=$(docker compose ps --quiet django | grep --invert-match "${old_container}")
# ...poll `docker inspect --format '{{.State.Health.Status}}'` until healthy...
docker stop --timeout 90 "${old_container}"
docker rm "${old_container}"
docker compose up --detach --no-deps --no-recreate --scale django=1 django
Enter fullscreen mode Exit fullscreen mode

If the new container never becomes healthy, the script removes it and exits non-zero. The old container keeps serving, and the deploy shows as failed in GitHub Actions.

I rehearsed it locally: the production image behind Traefik, 4 parallel request loops, two deploys in a row. 5,624 requests, all 200. A deliberately broken image was rejected while the old container kept serving. I shipped it.

The first live deploy still dropped requests

Production disagreed:

Time (UTC) API Control
08:16:33 → 08:17:12 503 ×3, 000 ×1 200
08:17:30 → 08:17:46 521 ×4 200

The 521s were expected. That's Cloudflare reporting that Traefik was restarting for its own config change, once. The 503s were not expected. They happened while the new Django container was starting, which was exactly the part the rolling deploy was supposed to handle.

Reproducing it locally

The local rehearsal had passed because my laptop is fast, and a new container there is ready almost as soon as it starts. The droplet is slow. Compose can limit a container's CPU, so I ran the rehearsal again with Django capped to 0.3 CPU:

django:
  cpus: "0.3"
Enter fullscreen mode Exit fullscreen mode

That reproduced it right away:

Time Result
16:37:25–36 4 requests hung for 10 seconds and timed out
16:37:47 35 instant 503s

Two bugs, one cause

My Traefik config routed to Django through a single URL:

services:
  django:
    loadBalancer:
      servers:
        - url: http://django:5000
      healthCheck:
        path: /healthz/
        interval: '5s'
        timeout: '3s'
Enter fullscreen mode Exit fullscreen mode

django is a Docker DNS name. With two containers running, it resolves to both, including the one that's still booting. Gunicorn accepts connections before its worker is ready, so:

  1. Requests that reached the new container waited in its queue and hung.
  2. Traefik's health probe also reached the new container sometimes, timed out, and marked the whole django server down, old container included. That's no available server, the 503s.

The Docker healthcheck I'd added only told the deploy script when the new container was ready. Traefik never saw it.

Step 3: Route only to healthy containers

Traefik's Docker provider solves this directly: it skips containers whose Docker healthcheck is still starting or unhealthy. So Django is now discovered through labels instead of a hardcoded URL:

django:
  labels:
    - traefik.enable=true
    - traefik.http.services.django.loadbalancer.server.port=5000

traefik:
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
Enter fullscreen mode Exit fullscreen mode 
# traefik.yml
http:
  routers:
    web-secure-router:
      rule: 'Host(`api.example.com`)'
      service: django@docker   # from the labels above
providers:
  file:
    filename: /etc/traefik/traefik.yml
  docker:
    exposedByDefault: false
Enter fullscreen mode Exit fullscreen mode

The router, TLS and middlewares stay in the file provider. Only the service moved.

A word on that socket mount: :ro makes the socket file read-only, but it doesn't limit the Docker API, so Traefik can still do anything Docker can. If Traefik were ever compromised, so is the host. On a single box I own, I accepted that. A Docker socket proxy that only allows container listings is the stricter option.

With the Docker provider, the 503s and the startup hangs were gone. In the slow-CPU rehearsal, 6 failures were left, all at the moment the old container stopped.

Step 4: Drain before you stop

docker stop sends SIGTERM, and gunicorn stops accepting connections. But Traefik still sees the old container as healthy until it exits, so for a few seconds it keeps sending requests to a server that's refusing them.

The fix is to take the container out of rotation before sending SIGTERM. Traefik already follows the healthcheck, so the healthcheck becomes the off switch:

healthcheck:
  test:
    - CMD
    - python
    - -c
    - "import os, sys, urllib.request; os.path.exists('/tmp/draining') and sys.exit(1); urllib.request.urlopen(...)"
Enter fullscreen mode Exit fullscreen mode

And in the deploy script:

sleep 5                                         # let Traefik pick up the new container
docker exec "${old_container}" touch /tmp/draining
# ...poll until Docker reports it unhealthy...
sleep 5                                         # let Traefik drop it
docker stop --timeout 90 "${old_container}"     # gunicorn finishes in-flight requests
Enter fullscreen mode Exit fullscreen mode

By the time gunicorn gets SIGTERM, nothing new is being sent to it, and the 90-second grace period lets requests already in flight finish. My gunicorn --timeout is 90 seconds because some requests wait on an LLM.

The rehearsal, with the CPU capped

Setup Requests Failed
DNS routing + Traefik health probe 1,639 39 (35 × 503, 4 × 10 s hangs)
Docker provider 2,847 6 (all when the old container stopped)
Docker provider + drain, 5 rollouts in a row 12,957 0

A gunicorn 26 surprise

While reading container logs I found this on every start:

[ERROR] Control server error: [Errno 13] Permission denied: '/nonexistent'
Enter fullscreen mode Exit fullscreen mode

Gunicorn 26 added a control socket, which defaults to a path under $HOME. cookiecutter-django's django system user doesn't have a home directory. Nothing broke, but --no-control-socket turns it off.

The transitional deploy

The first deploy of this change was still messy. It had to be: the Traefik that was running was the old one, with DNS routing, and the old Django container had no drain logic.

Time (UTC) What happened
09:44:57–09:45:10 503s during startup (old Traefik, old routing)
09:45:45 → 09:47:23 Drain step waited out its limit; the old container's healthcheck had no drain check
09:47:26–09:47:38 521/525 while Traefik restarted on its new config

If you change how traffic is routed, the deploy that ships the change runs on the old routing. Plan for one bad deploy, and don't measure that one.

Result

The next deploy was the first to run entirely on the final setup. It was an ordinary backend change: a query fix for the Browse Jobs page.

Time (UTC) Step
09:55:41 Deploy job starts on the droplet (pull, small image builds, migrate)
09:56:36 New Django container starts next to the old one
09:57:24 New container healthy; old one starts draining
~09:58:14 Old container stopped; Celery and Flower restart
09:58:34 Deploy job done

And the uptime loop:

Window API probes Failed Control
Deploy job, 09:55:41 → 09:58:34 44 0 all 200
Before, during and after, 09:54 → 10:00 64 0 all 200

Zero failed probes, and no Traefik restart, because nothing in its config changed.

For comparison, the morning's deploy on the original setup had about two minutes of no available server, and the transitional deploy had 8 failed probes across three bursts.

To be clear about what that measures: one probe every 5 seconds can't see a blip shorter than 5 seconds. The rehearsal fills that gap. There, two request loops ran continuously against Django capped to 0.3 CPU, and 12,957 requests across five rollouts had zero failures. Together, the two give me confidence the rollout itself no longer drops requests.

The deploy job went from 1m 47s to 2m 53s. The extra minute is mostly waiting for the new container to become healthy and for the old one to drain. That's an easy trade for no downtime.

What 99.99% still needs

Zero-downtime deploys take the biggest planned cause of downtime out of the budget. Several things are still left:

  • An independent monitor. My curl loop was fine for watching a deploy, but to claim 99.99%, you need something that checks every minute, all month, from outside your own network.
  • Backward-compatible migrations. The old container keeps serving while the new migrations run. Adding a column is fine. Renaming or dropping one needs two deploys: first ship code that stops using it, then remove it.
  • One host is still one host. A hypervisor reboot, a kernel update or running out of memory will take the site down, and rolling deploys can't help with that. On a single droplet you can avoid causing downtime yourself, but you can't guarantee against it.

Takeaways

  • Measure with a control. Otherwise your own Wi-Fi shows up as site downtime.
  • Rehearse on hardware as slow as production. A container that starts quickly will hide every timing bug. cpus: "0.3" found mine in one run.
  • Routing to a DNS name isn't the same as routing to healthy containers. Docker DNS includes containers that are still starting.
  • A Docker healthcheck only helps if your proxy reads it. Traefik's Docker provider does; a hardcoded URL doesn't.
  • Drain before SIGTERM. Take the old container out of rotation first, then stop it.
  • The deploy that changes routing runs on the old routing. Expect one rough deploy.
  • Move work to build time. It makes containers start faster and shows you which steps never did anything.

GiggleGigs is a job board for gig and contract work: gigglegigs.com.

Top comments (3)

Collapse
 
rushdeploy profile image
Rishabh Mishra •

The drain-before-stop step is the part I would keep. A healthcheck that only tells the deploy script the new container is ready does not help if Traefik still sends traffic to the old one until it stops.

Collapse
 
lycore profile image
Lycore Development •

The control request to google.com in your uptime loop is a smart detail; most people skip that and blame the server for their own network blips. One thing I'd look at on a single droplet is whether the new container's healthcheck verifies the DB connection and not just that gunicorn is listening, since Traefik will route traffic as soon as it reports healthy. Also curious how you handle migrations that drop or rename columns while the old container is still serving.

Collapse
 
rushdeploy profile image
Rishabh Mishra •

Good point. A healthcheck that only checks that gunicorn is listening can pass while the database is still unreachable, so I would add a simple database query to it.
For migrations, I would keep them backward compatible during the rollout, adding columns first and removing or renaming them only in a later deploy.
That way the old container can keep serving until the new one is fully healthy.