How I deploy, back up and watch a Django app on two servers

On 18 October I upgraded the server that runs ExtraTime, my football quiz site, from Ubuntu 24.04 to 26.04. The upgrade took 20 minutes. The site came back five hours later. Going through the logs to work out why made me write down the whole setup, so here it is: what runs where, how busy it is, and how deploys, backups and monitoring work on two small servers.

What runs where

ExtraTime is a Django API, a Next.js frontend, and a lot of background jobs. The frontend is on Vercel in Frankfurt, with Cloudflare caching its HTML. Everything else is on two virtual servers at netcup.

	App box	Monitoring box
Size	6 vCPU, 8 GB RAM	4 vCPU, 8 GB RAM
Disk used	28 GB of 251 GB	41 GB of 251 GB
OS	Ubuntu 26.04	Ubuntu 24.04
Runs	the API, WebSockets, Celery, Postgres, Redis	Prometheus, Grafana, Alertmanager, n8n, HealthReporter, backups

On the app box, pm2 runs eight Python processes:

Process	Count	Memory each
gunicorn (Django)	2	~170 MB
Daphne (WebSockets for multiplayer rooms)	2	~310 MB
Celery workers: default, notifications, heavy	3	~195 MB
Celery Beat	1	~200 MB

Next to them: PostgreSQL 18 with a 2.6 GB database, Redis using 16 MB of its 512 MB limit, nginx in front, and a Discord bot and three Prometheus exporters in Docker. Until this month it also ran a mail server, which I replaced with a hosted one. The two boxes talk over WireGuard. Grafana has no public URL at all; I reach it through an SSH tunnel.

How busy it is

From the last two weeks of nginx logs on the API:

• 61,574 to 142,410 requests a day
• 617 to 982 distinct IP addresses a day, which isn't the same as users but gives the scale
• median response time 37 ms (yesterday, 127,304 requests)

From 30 days of Prometheus on the app box:

Metric	Value
CPU, 30-day average	6.6%
CPU, median hour	5.2%
CPU, busiest normal hour	79% (3 October, not traced yet)
Memory available, median hour	2.3 GB
Memory available, lowest	0.63 GB (27 October)
Outbound traffic	100 GB in 30 days

CPU is close to idle. Memory isn't. Postgres holds about 1 GB and the Python processes about 2 GB between them. As of today the box has a 4 GB swap file as a safety net.

Deploys: one folder per release

Deploys used to be git reset --hard in place, and a running process could lazily import a module from the new code while the rest of it was still the old code. Now every deploy is its own folder:

/var/www/html/
  repo/              git repository, every release is a worktree of it
  releases/<sha>/    one folder per deploy, last 5 kept
  shared/            .env, media, staticfiles, logs, symlinked into each release
  venvs/<hash>/      one venv per requirements.txt content
  current -> releases/<sha>

A deploy:

1. dumps the database
2. creates the release worktree and links the shared files
3. builds a venv only if requirements.txt changed
4. runs migrations and collectstatic from the new release while the old one serves
5. switches current with ln -sfn and mv -T, which is atomic
6. restarts the two gunicorns and the two Daphnes one at a time, so one of each is always up
7. runs a smoke test that makes real requests, and switches current back if one returns a 500

Two things I learned the hard way. pm2 never re-reads an app's script or cwd on restart, so every app starts through a small wrapper that resolves the current release when it starts. And processes have to run from the real path, not through the symlink, or Python will import the next release's files the moment current moves.

It still isn't perfect. Last night's deploy produced 775 WebSocket 502s in one minute, on the notification and achievement sockets. The frontend reconnects on its own, but it's on the list.

A cron watchdog probes each gunicorn every minute. After three failed probes it saves stack traces with py-spy, the slowest Postgres queries and Redis's slow log, restarts only that instance, and alerts on Discord.

Backups are pulled, not pushed

Postgres is dumped nightly at 02:00 UTC and before every deploy. At 03:00 the monitoring box pulls the dumps. The app box holds no credentials for anything that stores backups, so if someone took over the app box, they couldn't delete them.

The monitoring box has two SSH keys on the app box, one per directory, and each is locked down in authorized_keys like this:

command="rrsync -ro /var/backups/extratime/daily",restrict,from="<monitoring box's WireGuard address>" ssh-ed25519 AAAA...

That key can read one folder, from one address, and do nothing else. The pull script also doesn't trust the app box's file names when it decides what to keep, so planted files can't push real backups out of the retention window. Every dump is encrypted with age to a public key; the private key is on no server. The monitoring box currently holds 4.3 GB of dumps (14 daily, 8 weekly, 6 monthly) and 1.9 GB of media.

HealthReporter

Grafana shows numbers. It doesn't tell me whether something I fixed two weeks ago is actually fixed. So I built HealthReporter, which writes a report on the backend every Monday morning.

It's a FastAPI service with SQLite in Docker and a Next.js front end, running on the monitoring box behind Cloudflare Access. The collection step is plain code: Prometheus queries, read-only SQL, the watchdog's captures, log tails, all over SSH with a key that can only run read-only commands on the app box. Claude Code then turns the collected data into a structured report with findings, likely causes and recommendations.

What makes it useful is the reconciliation. Each report takes the previous report's findings, looks up the pull requests and issues that were meant to fix them, and checks the metrics to see whether the number actually moved. A merged PR isn't a fix until the metric agrees. Reports are kept as revisions, so I can diff one week against the next.

The five hours

Back to 18 October. The release upgrade ran from 12:03 to the reboot at 12:25. After the reboot, pm2's log filled up with lines like this:

2026-10-18T12:25:34: PM2 log: App [extratime-websocket-prod-2:3] exited with code [126] via signal [SIGINT]

Every Python process exited the moment it started, and pm2 restarted it straight away. By 17:14, when everything was back, there were about 1.9 million of those exits:

Hour	Exits
12:00	259,190
13:00	357,503
14:00	370,953
15:00	415,395
16:00	416,869
17:00	89,900

Exit code 126 means the shell found the program but couldn't execute it. My reading is that the virtualenvs' Python was the one 24.04 shipped, and 26.04 replaced it with Python 3.14, so every script's interpreter line pointed at something that was gone. nginx logged 7,604 502s that day, and Prometheus showed the CPU at 60%, all of it spent restarting processes.

What I got wrong

I ran a release upgrade on the only app server, on a Sunday, without a snapshot, and expected the app to come back after the reboot.

I first read the 60% CPU on 18 October as the busiest day of the month. It was the day the site was down.

And I'd been watching the wrong resource. CPU has been idle all along; memory got down to 0.63 GB.

Caveats

Traffic numbers are from the API's nginx logs only; the frontend's traffic is on Vercel and Cloudflare. Distinct IPs aren't users. The explanation for exit code 126 is my reading of the logs, since the virtualenvs from that day have been replaced and I can't inspect them.

What I'd take from it

• Snapshot before an OS upgrade, even on a side project.
• An exit loop looks like load. Alert on processes that restart far more often than you deploy, not only on CPU.
• Release folders and a symlink switch are cheap, and they end a whole class of half-deployed bugs.
• Pull backups to another box with a read-only key, rather than letting the server you're protecting push them.
• Size the box by its memory, not its CPU.