GiggleGigs is a job board I run: Django + DRF on the backend, React + Vite on the front (on Vercel), Celery for background work, all in Docker Compose on a single 2 GB DigitalOcean droplet, with Traefik in front and Cloudflare in front of that.
In the last post I moved image builds off the droplet and into GitHub Actions. That stopped deploys from freezing the server. It didn't stop them from taking the site down.
Why 99.99% means deploys can't cause any downtime
99.99% uptime allows about 4.3 minutes of downtime a month.
I measured a normal deploy with a loop that requested the API every 5 seconds: about two minutes of no available server from Traefik. I deploy several times a day. At that rate, a single morning of deploys would use up the month's budget.
So "faster deploys" wasn't the goal. The goal was deploys with no downtime at all, which left the 4.3 minutes for things I can't control.
Measure first
Everything below is based on one simple measurement: a loop on my laptop that requested the API's health endpoint every 5 seconds while a deploy ran, plus a request to a site that's always up (google.com) on each tick as a control:
while true; do
echo "$(date -u +%T) $(curl -s -o /dev/null -m 5 -w '%{http_code}' https://api.example.com/healthz/) $(curl -s -o /dev/null -m 5 -w '%{http_code}' https://www.google.com/)"
sleep 5
done >> uptime.log
The control column matters. My own connection dropped twice during the day, and without the control those drops would have looked like site outages.
Why a deploy went down
The old deploy ran docker compose up -d, which:
- stops the only Django container,
- starts a new one,
- whose start script runs
collectstaticbefore gunicorn starts listening.
Traefik had nothing to send requests to until step 3 finished. That's two minutes.
Step 1: Collect static files when the image is built
collectstatic doesn't need a database or real secrets. It only needs the settings module to import. So it now runs in the Dockerfile, which runs in CI, with placeholder env vars that exist only for that RUN step:
# Collect static files at build time so a new container starts
# serving in seconds. Production settings need these env vars to import,
# but collectstatic never uses them, so placeholders are enough.
RUN DJANGO_SETTINGS_MODULE="config.settings.production" \
DJANGO_SECRET_KEY="build-only" \
DATABASE_URL="postgres://build:build@localhost:5432/build" \
# ...the rest of the required settings...
python manage.py collectstatic --noinput
The start script went from about 25 lines to one:
exec /usr/local/bin/gunicorn config.wsgi --bind 0.0.0.0:5000 --chdir=/app \
--timeout 90 --graceful-timeout 90 --no-control-socket
A new container now passes its healthcheck in 13–20 seconds on my laptop and about 43 seconds on the droplet, instead of about two minutes.
One surprise: I also tried running django-compressor's compress at build time, and it failed with No 'compress' template tags found. None of my templates use it. It must have been switched off in production, or the old start script would have crashed. Moving work into the build is a good way to find steps that never did anything.
Step 2: A healthcheck, and a second container
Then a rolling deploy: start a second Django container on the new image, wait until it's healthy, and only then stop the old one. The new container gets a Docker healthcheck on the existing /healthz/ endpoint:
django:
image: ghcr.io/you/app-django:${IMAGE_TAG:-latest}
command: /start
healthcheck:
test:
- CMD
- python
- -c
- "import urllib.request; urllib.request.urlopen(urllib.request.Request('http://localhost:5000/healthz/', headers={'Host': 'api.example.com'}), timeout=3)"
interval: 5s
timeout: 5s
retries: 3
start_period: 60s
stop_grace_period: 90s
The image has no curl, so the check uses Python. The Host header has to be one of ALLOWED_HOSTS, and /healthz/ has to be in SECURE_REDIRECT_EXEMPT, or the check gets a 301 and fails.
The deploy script uses Compose's --scale and --no-recreate flags, which most people never touch:
old_container=$(docker compose ps --quiet django)
docker compose run --rm django python manage.py migrate
docker compose up --detach --no-deps --no-recreate --scale django=2 django
new_container=$(docker compose ps --quiet django | grep --invert-match "${old_container}")
# ...poll `docker inspect --format '{{.State.Health.Status}}'` until healthy...
docker stop --timeout 90 "${old_container}"
docker rm "${old_container}"
docker compose up --detach --no-deps --no-recreate --scale django=1 django
If the new container never becomes healthy, the script removes it and exits non-zero. The old container keeps serving, and the deploy shows as failed in GitHub Actions.
I rehearsed it locally: the production image behind Traefik, 4 parallel request loops, two deploys in a row. 5,624 requests, all 200. A deliberately broken image was rejected while the old container kept serving. I shipped it.
The first live deploy still dropped requests
Production disagreed:
| Time (UTC) | API | Control |
|---|---|---|
| 08:16:33 → 08:17:12 |
503 ×3, 000 ×1 |
200 |
| 08:17:30 → 08:17:46 |
521 ×4 |
200 |
The 521s were expected. That's Cloudflare reporting that Traefik was restarting for its own config change, once. The 503s were not expected. They happened while the new Django container was starting, which was exactly the part the rolling deploy was supposed to handle.
Reproducing it locally
The local rehearsal had passed because my laptop is fast, and a new container there is ready almost as soon as it starts. The droplet is slow. Compose can limit a container's CPU, so I ran the rehearsal again with Django capped to 0.3 CPU:
django:
cpus: "0.3"
That reproduced it right away:
| Time | Result |
|---|---|
| 16:37:25–36 | 4 requests hung for 10 seconds and timed out |
| 16:37:47 | 35 instant 503s |
Two bugs, one cause
My Traefik config routed to Django through a single URL:
services:
django:
loadBalancer:
servers:
- url: http://django:5000
healthCheck:
path: /healthz/
interval: '5s'
timeout: '3s'
django is a Docker DNS name. With two containers running, it resolves to both, including the one that's still booting. Gunicorn accepts connections before its worker is ready, so:
- Requests that reached the new container waited in its queue and hung.
- Traefik's health probe also reached the new container sometimes, timed out, and marked the whole
djangoserver down, old container included. That'sno available server, the 503s.
The Docker healthcheck I'd added only told the deploy script when the new container was ready. Traefik never saw it.
Step 3: Route only to healthy containers
Traefik's Docker provider solves this directly: it skips containers whose Docker healthcheck is still starting or unhealthy. So Django is now discovered through labels instead of a hardcoded URL:
django:
labels:
- traefik.enable=true
- traefik.http.services.django.loadbalancer.server.port=5000
traefik:
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
# traefik.yml
http:
routers:
web-secure-router:
rule: 'Host(`api.example.com`)'
service: django@docker # from the labels above
providers:
file:
filename: /etc/traefik/traefik.yml
docker:
exposedByDefault: false
The router, TLS and middlewares stay in the file provider. Only the service moved.
A word on that socket mount: :ro makes the socket file read-only, but it doesn't limit the Docker API, so Traefik can still do anything Docker can. If Traefik were ever compromised, so is the host. On a single box I own, I accepted that. A Docker socket proxy that only allows container listings is the stricter option.
With the Docker provider, the 503s and the startup hangs were gone. In the slow-CPU rehearsal, 6 failures were left, all at the moment the old container stopped.
Step 4: Drain before you stop
docker stop sends SIGTERM, and gunicorn stops accepting connections. But Traefik still sees the old container as healthy until it exits, so for a few seconds it keeps sending requests to a server that's refusing them.
The fix is to take the container out of rotation before sending SIGTERM. Traefik already follows the healthcheck, so the healthcheck becomes the off switch:
healthcheck:
test:
- CMD
- python
- -c
- "import os, sys, urllib.request; os.path.exists('/tmp/draining') and sys.exit(1); urllib.request.urlopen(...)"
And in the deploy script:
sleep 5 # let Traefik pick up the new container
docker exec "${old_container}" touch /tmp/draining
# ...poll until Docker reports it unhealthy...
sleep 5 # let Traefik drop it
docker stop --timeout 90 "${old_container}" # gunicorn finishes in-flight requests
By the time gunicorn gets SIGTERM, nothing new is being sent to it, and the 90-second grace period lets requests already in flight finish. My gunicorn --timeout is 90 seconds because some requests wait on an LLM.
The rehearsal, with the CPU capped
| Setup | Requests | Failed |
|---|---|---|
| DNS routing + Traefik health probe | 1,639 | 39 (35 × 503, 4 × 10 s hangs) |
| Docker provider | 2,847 | 6 (all when the old container stopped) |
| Docker provider + drain, 5 rollouts in a row | 12,957 | 0 |
A gunicorn 26 surprise
While reading container logs I found this on every start:
[ERROR] Control server error: [Errno 13] Permission denied: '/nonexistent'
Gunicorn 26 added a control socket, which defaults to a path under $HOME. cookiecutter-django's django system user doesn't have a home directory. Nothing broke, but --no-control-socket turns it off.
The transitional deploy
The first deploy of this change was still messy. It had to be: the Traefik that was running was the old one, with DNS routing, and the old Django container had no drain logic.
| Time (UTC) | What happened |
|---|---|
| 09:44:57–09:45:10 | 503s during startup (old Traefik, old routing) |
| 09:45:45 → 09:47:23 | Drain step waited out its limit; the old container's healthcheck had no drain check |
| 09:47:26–09:47:38 | 521/525 while Traefik restarted on its new config |
If you change how traffic is routed, the deploy that ships the change runs on the old routing. Plan for one bad deploy, and don't measure that one.
Result
The next deploy was the first to run entirely on the final setup. It was an ordinary backend change: a query fix for the Browse Jobs page.
| Time (UTC) | Step |
|---|---|
| 09:55:41 | Deploy job starts on the droplet (pull, small image builds, migrate) |
| 09:56:36 | New Django container starts next to the old one |
| 09:57:24 | New container healthy; old one starts draining |
| ~09:58:14 | Old container stopped; Celery and Flower restart |
| 09:58:34 | Deploy job done |
And the uptime loop:
| Window | API probes | Failed | Control |
|---|---|---|---|
| Deploy job, 09:55:41 → 09:58:34 | 44 | 0 | all 200 |
| Before, during and after, 09:54 → 10:00 | 64 | 0 | all 200 |
Zero failed probes, and no Traefik restart, because nothing in its config changed.
For comparison, the morning's deploy on the original setup had about two minutes of no available server, and the transitional deploy had 8 failed probes across three bursts.
To be clear about what that measures: one probe every 5 seconds can't see a blip shorter than 5 seconds. The rehearsal fills that gap. There, two request loops ran continuously against Django capped to 0.3 CPU, and 12,957 requests across five rollouts had zero failures. Together, the two give me confidence the rollout itself no longer drops requests.
The deploy job went from 1m 47s to 2m 53s. The extra minute is mostly waiting for the new container to become healthy and for the old one to drain. That's an easy trade for no downtime.
What 99.99% still needs
Zero-downtime deploys take the biggest planned cause of downtime out of the budget. Several things are still left:
- An independent monitor. My curl loop was fine for watching a deploy, but to claim 99.99%, you need something that checks every minute, all month, from outside your own network.
- Backward-compatible migrations. The old container keeps serving while the new migrations run. Adding a column is fine. Renaming or dropping one needs two deploys: first ship code that stops using it, then remove it.
- One host is still one host. A hypervisor reboot, a kernel update or running out of memory will take the site down, and rolling deploys can't help with that. On a single droplet you can avoid causing downtime yourself, but you can't guarantee against it.
Takeaways
- Measure with a control. Otherwise your own Wi-Fi shows up as site downtime.
-
Rehearse on hardware as slow as production. A container that starts quickly will hide every timing bug.
cpus: "0.3"found mine in one run. - Routing to a DNS name isn't the same as routing to healthy containers. Docker DNS includes containers that are still starting.
- A Docker healthcheck only helps if your proxy reads it. Traefik's Docker provider does; a hardcoded URL doesn't.
- Drain before SIGTERM. Take the old container out of rotation first, then stop it.
- The deploy that changes routing runs on the old routing. Expect one rough deploy.
- Move work to build time. It makes containers start faster and shows you which steps never did anything.
GiggleGigs is a job board for gig and contract work: gigglegigs.com.
Top comments (20)
The one-droplet constraint is what makes this interesting. We did something similar with a second port and an nginx switch, and the flakiest part was in-flight requests during the restart. A short grace period and keeping the old worker alive for a few seconds fixed it.
Same pattern, different tools. Your nginx port switch and Traefik picking up healthy containers both solve the cutover. Like you said, the in-flight part is what bites. What fixed it for me was draining first: fail the old container’s healthcheck so Traefik stops routing to it, then send SIGTERM. Did you add a delay before the switch, or just keep the old worker alive afterward?
The critical bottleneck for single-droplet zero-downtime deploys isn't reverse proxy cutover; it's peak memory contention during the worker handoff. Spinning up overlapping application instances—whether via dual Gunicorn processes or parallel containers—on a resource-constrained box routinely trips the Linux OOM killer under real traffic. Decoupling backward-compatible schema migrations from the deployment phase and utilizing systemd socket passing or staggered worker reloads keeps your footprint strictly bounded without dropping in-flight connections.
Bohdan Hordii // bohdanhordii.dev
Fair point, and on 2 GB it’s the real constraint. The overlap is about a minute per deploy with two Django containers. I didn’t include peak memory numbers in the post, and I should have. That’s going in the next round of measurements. Celery and Flower restart after the old container is gone, so the Django overlap is the only one. Gunicorn’s in-place reload (USR2/HUP) is the lower-memory alternative. The trade-off is giving up the “new image proves itself healthy before taking traffic” property, and that check is what caught the broken image in rehearsal. Expand/contract migrations are covered at the end.
The grace period for in-flight requests is the detail people miss. How long did you end up using?
90 seconds, matching gunicorn’s --timeout, because some requests wait on an LLM. In practice the drain step does most of the work. By the time SIGTERM arrives, Traefik has stopped sending new requests, so the grace period only has to cover what’s already in flight. That’s usually a few seconds. The 90 is a ceiling, not a typical wait.
The locking section is the one I keep re-reading.
The control request to google.com in your uptime loop is a smart detail; most people skip that and blame the server for their own network blips. One thing I'd look at on a single droplet is whether the new container's healthcheck verifies the DB connection and not just that gunicorn is listening, since Traefik will route traffic as soon as it reports healthy. Also curious how you handle migrations that drop or rename columns while the old container is still serving.
Good catch on the control. My own Wi-Fi dropped twice that day. On the DB check, there’s a trade-off on a single box: Traefik routes on this healthcheck, so if it queries the DB, a DB blip marks every container unhealthy at once. You’d get 503s either way, but it changes what the check means. The middle ground is a DB check in the deploy script’s readiness gate, and a lighter check for routing. Migrations: expand/contract, covered in the “What 99.99% still needs” section. Renames and drops take two deploys.
Good point. A healthcheck that only checks that gunicorn is listening can pass while the database is still unreachable, so I would add a simple database query to it.
For migrations, I would keep them backward compatible during the rollout, adding columns first and removing or renaming them only in a later deploy.
That way the old container can keep serving until the new one is fully healthy.
Thanks Rishabh, agreed on both points. For the healthcheck I'd keep the query as cheap as SELECT 1 with a short timeout, so a slow database fails the check quickly instead of hanging it. And splitting schema changes expand/contract, adding first and dropping or renaming in a later deploy, means old and new code both work during the switch.
SELECT 1 is the right call — it proves the connection without touching real tables.
If the check passes but the app still serves stale data, a second probe on the read path catches that split.
Thanks for sharing the detail, that distinction is worth keeping in the checklist.
+1 on SELECT 1 with a short timeout. The one caveat is the one above: if the proxy routes on that same check, a slow DB pulls every container out of rotation together. A separate readiness probe and routing probe keeps the two jobs apart.
Good point, separating the two probes makes the distinction clearer. Keeping the routing probe light, so a slow database does not pull healthy containers out of rotation, is the part I would keep.
Exactly. Each probe answers one question: the routing probe asks “can this container take requests?”, and the readiness gate asks “is this deploy safe to cut over?” Thanks for the back-and-forth.
That framing is a good one to keep. If the routing probe stays light, a failing readiness check cannot pull a healthy container out of rotation. Thanks again for the slow-CPU example.
Agreed. It took a slow-CPU rehearsal to see it: the Docker provider alone still left 6 failures, all at the moment the old container stopped. Drain-first got that to zero across 12,957 requests.
The drain-before-stop step is the part I would keep. A healthcheck that only tells the deploy script the new container is ready does not help if Traefik still sends traffic to the old one until it stops.
Cool
Some comments may only be visible to logged-in visitors. Sign in to view all comments.