DEV Community

Ajit Kumar
Ajit Kumar

Posted on Edited on

Moving a production API to Hetzner, part 3: migrating data and cutting over

#devops #postgres #docker #dns

The new server is hardened and running Docker (part 2).
Now we move the application. The goal is a few minutes of downtime, zero lost
data, and a rollback plan we never have to improvise.

To keep the commands concrete, this part uses a small example application. Swap
in your own names; the method doesn't change.

0. The example application

notes-api is a small REST API: users sign in and create, read and delete
notes. It's the same shape as most small production services:

Piece What it does State it holds
server the API, a container image built by CI and pushed to ghcr.io/<you>/notes-api none
postgres PostgreSQL 16: users, notes, schema migrations everything that matters
valkey Valkey (the open-source Redis fork): response cache and rate-limit counters short-lived only
caddy reverse proxy; gets and renews the TLS certificate automatically a certificate it can re-fetch

Users sign in with a token signed by a secret in .env.prod (JWT_SECRET), and
a nightly backup job writes a compressed pg_dump to an S3-compatible bucket.

The repository looks like this:

notes-api/
├── docker/
│   ├── docker-compose.prod.yml
│   └── Caddyfile
├── scripts/
│   └── backup.sh
└── .env.prod.example      # the real .env.prod is never committed
Enter fullscreen mode Exit fullscreen mode

The production compose file, trimmed to what matters for a migration:

# docker/docker-compose.prod.yml
services:
  postgres:
    image: postgres:16-alpine
    container_name: notes_postgres
    environment:
      POSTGRES_USER: notes
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: notes
    volumes: [pgdata:/var/lib/postgresql/data]
    healthcheck: { test: ["CMD", "pg_isready", "-U", "notes"], interval: 5s }
    restart: unless-stopped

  valkey:
    image: valkey/valkey:9-alpine
    container_name: notes_valkey
    command: valkey-server --requirepass ${VALKEY_PASSWORD}
    restart: unless-stopped

  server:
    image: ${APP_IMAGE:-ghcr.io/<you>/notes-api:latest}
    container_name: notes_server
    env_file: ../.env.prod
    depends_on:
      postgres: { condition: service_healthy }
    restart: unless-stopped

  caddy:
    image: caddy:2-alpine
    container_name: notes_caddy
    ports: ["80:80", "443:443", "443:443/udp"]   # the ONLY published ports
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
    restart: unless-stopped

volumes: { pgdata: {}, caddy_data: {} }
Enter fullscreen mode Exit fullscreen mode 
# docker/Caddyfile
api.example.com {
    reverse_proxy server:8080
}
Enter fullscreen mode Exit fullscreen mode

Three details in there matter later:

  • Only Caddy publishes ports. Postgres and Valkey are reachable only on Docker's internal network.
  • image: ${APP_IMAGE:-…:latest}: the image comes from a variable with a default, so a deploy can pin an exact build without editing the file.
  • The API runs its database migrations on startup. That decides the order in which we start things.

Placeholders, as in parts 1 and 2: the old server is old-box (203.0.113.10),
the new one is new-box (198.51.100.20), and the API is api.example.com.

1. Decide what moves, and what doesn't

The most useful migration question is what state lives where.

State Where Moves? Why
Users, notes, migration history Postgres volume yes the only durable state on the box
Secrets (.env.prod) a file next to the compose file yes, byte-identical DB and Valkey passwords, JWT_SECRET, storage and API keys
Cache, rate-limit counters Valkey no rebuilt on demand; counters resetting to zero is harmless
TLS certificate Caddy's volume no Caddy fetches a fresh one in seconds
Backups S3-compatible bucket no already off the box; the new server writes to the same bucket
Code container image in a registry pulled never built on the server (section 3)

Not moving Valkey is worth a moment's thought, because it's tempting to copy
everything "to be safe". Ask of each kind of key: what happens if it's gone?
Cache entries are refetched from Postgres. Rate-limit counters start again from
zero. Nobody is signed out, because sign-in tokens are verified with
JWT_SECRET, which travels in .env.prod, not in Valkey. If your app keeps
sessions in Redis or Valkey, that changes the answer: either copy it or accept
that everyone signs in again.

What moves

2. Copying between two servers through the laptop

The trick

The old and new servers can't log in to each other, and they shouldn't be able
to: that would mean a private key for one sitting on the other. But the laptop can
reach both (part 2 set up old-box and new-box in ~/.ssh/config). So we
connect the two SSH sessions with a pipe. First clone the repository on the
new box, then copy the secrets file across:

ssh new-box 'git clone https://github.com/<you>/notes-api ~/notes-api'
ssh old-box 'cat ~/notes-api/.env.prod' | ssh new-box 'umask 077 && cat > ~/notes-api/.env.prod'
Enter fullscreen mode Exit fullscreen mode

Reading the second line left to right:

  1. ssh old-box 'cat …' runs cat on the old server. Its output, the file, comes back over the first encrypted SSH connection as that command's stdout.
  2. | is a pipe on the laptop. It connects the first command's stdout to the second command's stdin, in memory. Nothing is written to the laptop's disk, and nothing is printed to its terminal.
  3. ssh new-box '… cat > …' runs cat on the new server, whose stdin is the second encrypted connection. It writes the bytes to the destination file.
  4. umask 077 is set first, so the file is created with mode 600 (owner-only). That's different from fixing the permissions afterwards, which leaves a moment when the secrets are world-readable.

ssh pipe between servers

It's one command, with both servers' credentials staying on the laptop. You
don't have to log in to the old box, download, log in to the new box, upload,
and then remember to delete the local copy of your production secrets.

Alternatives, and why not:

  • scp -3 old-box:f new-box:f does the same routing through the laptop. Fine too; the pipe is easier to combine with umask and checks.
  • Agent forwarding (ssh -A old-box, then scp from there to the new box) works, but while you're connected, anyone with root on the old box can use your agent to authenticate as you anywhere. Avoid it on servers you're about to decommission.

Verify it arrived intact 

ssh old-box 'sha256sum ~/notes-api/.env.prod'; ssh new-box 'sha256sum ~/notes-api/.env.prod; stat -c "%a %U" ~/notes-api/.env.prod'
Enter fullscreen mode Exit fullscreen mode 
9f2c41d7…b83e0  /home/ubuntu/notes-api/.env.prod
9f2c41d7…b83e0  /home/ubuntu/notes-api/.env.prod
600 ubuntu
Enter fullscreen mode Exit fullscreen mode

A cryptographic hash is a fingerprint: change one byte and it changes
completely. Matching hashes prove the files are identical, and you never had to
display the secrets to compare them. One wrong character in JWT_SECRET would
otherwise sign every user out at cutover.

Copy only .env.prod. Leave behind any development .env or stray .env.prod.bak
on the old box. With no plain .env present, a command that forgets
--env-file .env.prod fails loudly instead of quietly running with development
settings.

3. Pull the image, never build on the box 

ssh old-box 'docker inspect notes_server --format "{{.Config.Image}}"'
# ghcr.io/<you>/notes-api:3f9c2ab
ssh new-box 'docker pull ghcr.io/<you>/notes-api:3f9c2ab'
Enter fullscreen mode Exit fullscreen mode

The repository gives us the compose file and scripts. The application itself
comes as a prebuilt image, and the first command asks the old box exactly
which tag it's running. Pull that one, not latest. Two reasons:

  1. Memory. Compiling a release build (Rust, Go with CGO, a big TypeScript app) can need several GB of RAM. On a 4 GB box with Postgres running, that's an out-of-memory kill waiting to happen. CI builds the image; the server only downloads it.
  2. Same bits. Running the old box's exact build means the migration changes one thing at a time: the host. If something breaks after cutover, it's the move, not a code change. Upgrades come afterwards, separately.

4. Start the data services only 

ssh new-box 'cd ~/notes-api && docker compose -f docker/docker-compose.prod.yml --env-file .env.prod up -d postgres valkey && docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'
Enter fullscreen mode Exit fullscreen mode 
NAMES            STATUS                   PORTS
notes_postgres   Up 2 minutes (healthy)   5432/tcp
notes_valkey     Up 2 minutes             6379/tcp
Enter fullscreen mode Exit fullscreen mode

Read the PORTS column carefully. 5432/tcp means "exposed inside Docker's
network"; 0.0.0.0:5432->5432/tcp would mean "published to the world", which
is the Docker-bypasses-ufw problem from parts 1 and 2.

Two services are deliberately not started yet:

  • The API server. It runs migrations on startup, so it would create an empty schema, and the restore in section 7 would then collide with it.
  • Caddy. It would immediately try to get a certificate for api.example.com, which still points at the old server. The challenge would fail, and repeated failures run into Let's Encrypt's rate limits.

5. Rehearse the restore before you need it

A backup you have never restored is a hope, not a backup.

The nightly job on the old box uploads notes-YYYYMMDD.sql.gz to a bucket.
Restoring the newest one into a scratch database on the new box, before
cutover, proves three things at once: the storage credentials work from the new
IP (some providers filter tokens by source address), the restore path works on
this machine, and the backups are actually good.

# list the bucket (the AWS CLI works with any S3-compatible store)
ssh new-box 'set -a; . ~/notes-api/.env.prod; set +a;
  docker run --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY amazon/aws-cli \
    --endpoint-url "$S3_ENDPOINT" s3 ls "s3://$BACKUP_BUCKET/" | tail -3'
Enter fullscreen mode Exit fullscreen mode 
2026-10-27 03:02:11      41208 notes-20261027.sql.gz
2026-10-28 03:02:09      41377 notes-20261028.sql.gz
2026-10-29 03:02:14      41503 notes-20261029.sql.gz
Enter fullscreen mode Exit fullscreen mode

set -a; . file; set +a loads .env.prod into the shell as environment
variables without printing them, so the credentials reach the container
through -e NAME and never appear on a command line, where ps could show them.

ssh new-box 'set -a; . ~/notes-api/.env.prod; set +a;
  docker run --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY amazon/aws-cli \
    --endpoint-url "$S3_ENDPOINT" s3 cp "s3://$BACKUP_BUCKET/notes-20261029.sql.gz" - > ~/rehearsal.sql.gz
  docker exec notes_postgres createdb -U notes restore_test
  zcat ~/rehearsal.sql.gz | docker exec -i notes_postgres psql -U notes -d restore_test -q -v ON_ERROR_STOP=1
  docker exec notes_postgres psql -U notes -d restore_test -tAc "select count(*) from users; select count(*) from notes"
  docker exec notes_postgres dropdb -U notes restore_test; rm ~/rehearsal.sql.gz'
Enter fullscreen mode Exit fullscreen mode 
57
1342
Enter fullscreen mode Exit fullscreen mode

Download, restore into a throwaway database, count something you know is there,
drop it. ON_ERROR_STOP=1 makes psql stop at the first error instead of
skipping it and carrying on. Non-zero counts mean the backup is real.

A bug a rehearsal like this can turn up

When I ran this rehearsal against my real service, the newest backup in the
bucket was 26 days old. The job had run every night and failed every night,
and no alert had reached anyone. Translated to this example, the log said:

dump ok: notes-20261029.sql.gz (43503 bytes)
gzip: stdout: Broken pipe
ERROR: dump does not contain table notes — wrong database?
Enter fullscreen mode Exit fullscreen mode

The script sanity-checked each dump before uploading it:

set -o pipefail
zcat "$OUT" | grep -q "CREATE TABLE public.notes" || die "dump does not contain table notes"
Enter fullscreen mode Exit fullscreen mode

Here's what happens:

  1. grep -q exits as soon as it finds the first match. That's the point of -q.
  2. zcat is still writing the rest of the file into the pipe. Writing to a pipe whose reader has gone away gets the writer killed with SIGPIPE, hence Broken pipe.
  3. pipefail (part 2: "fail if any stage fails") sees zcat die and fails the whole pipeline.
  4. The script reports "no table notes" because it found it.

It worked for months because it's a race. A Linux pipe buffers 64 KB. While
the decompressed dump was small, zcat had usually written all of it before
grep found its match and quit, so nothing was left to write and no SIGPIPE
came. Once the database grew, zcat still had data to write when grep exited,
and it lost the race every night.

The fix is to let grep read to the end and throw its output away:

zcat "$OUT" | grep "CREATE TABLE public.notes" >/dev/null || die "…"
Enter fullscreen mode Exit fullscreen mode

Lessons: grep -q and head inside a pipefail pipeline are a classic trap;
a check can fail because the thing it checks for is there; and backup
alerting needs its own test. Section 7's checklist triggers the failure alert
on purpose.

6. DNS: lower the TTL a day ahead

How DNS caching works

When a client looks up api.example.com, the answer comes with a TTL (time to
live) in seconds. Every resolver along the way (your ISP's, 1.1.1.1, 8.8.8.8,
the OS cache) may reuse that answer until the TTL expires. With a TTL of 300, after
you change the record, some clients keep going to the old IP for up to 5 minutes.
With a TTL of a day, for up to a day.

So the TTL has to be lowered in advance, by at least the old TTL's
duration, because resolvers that cached the record before the change still hold
it with the old expiry:

TTL timeline

The day before, set the api record's TTL to 60 seconds in your DNS provider,
then confirm what resolvers see:

dig api.example.com @1.1.1.1 +noall +answer
# api.example.com.  60  IN  A  203.0.113.10
#                   ^^ remaining TTL in seconds
Enter fullscreen mode Exit fullscreen mode

DNS-only, not proxied

Many DNS providers can also proxy traffic through their CDN (Cloudflare calls it
the "orange cloud"). The API record stays DNS-only: Caddy gets its own
certificate with the ACME HTTP-01 challenge, and that needs Let's Encrypt to
reach this server on port 80 directly:

ACME HTTP-01

That's why port 80 is open in both firewalls, and why Caddy only starts after
DNS points here.

7. The cutover

Pick a quiet hour and keep three terminals open: old-box, new-box, and the
laptop. Before starting, check two things on the new box: no reboot is pending
(ls /var/run/reboot-required), and the TTL from section 6 has been 60 seconds
for at least the old TTL's duration.

The sequence is designed so that only one database accepts writes at any
moment:

Cutover sequence

1. Freeze writes. Stop only the API on the old box. From this moment, never
start it again (section 8 explains why):

ssh old-box 'cd ~/notes-api && docker compose -f docker/docker-compose.prod.yml --env-file .env.prod stop server'
Enter fullscreen mode Exit fullscreen mode

Caddy keeps running and answers 502 Bad Gateway. Clients see an error and retry
later. If you receive webhooks (payments, GitHub, anything), the sender treats a
non-2xx response as a failed delivery and retries with backoff, typically for
hours or days, so no event is lost. A 502 is the cheapest possible maintenance
page.

2. Dump straight into the new box:

ssh old-box 'docker exec notes_postgres pg_dump -U notes -Fc notes' | ssh new-box 'umask 077 && cat > ~/cutover.dump && ls -l ~/cutover.dump'
Enter fullscreen mode Exit fullscreen mode

-Fc is Postgres's custom format: compressed, and readable by pg_restore,
which can restore objects selectively and in dependency order. A plain SQL dump
works too, but this is smaller and more flexible. It's the same laptop-pipe
pattern as section 2. Keep cutover.dump afterwards: it's an exact snapshot of
the moment of the move.

3. Restore and compare:

ssh new-box 'cd ~/notes-api && docker compose -f docker/docker-compose.prod.yml --env-file .env.prod exec -T postgres pg_restore -U notes -d notes --no-owner --exit-on-error < ~/cutover.dump && echo RESTORED'
Enter fullscreen mode Exit fullscreen mode
  • exec -T: no TTY, so the dump can be fed on stdin.
  • --no-owner: don't try to recreate the original object owners; everything belongs to the connecting user.
  • --exit-on-error: stop at the first error rather than restoring half a database and reporting success.

Then run the same query on both boxes and require identical output:

Q="select 'users='||(select count(*) from users)||' notes='||(select count(*) from notes)||' migration='||(select max(version) from schema_migrations)"
ssh old-box "docker exec notes_postgres psql -U notes -d notes -tAc \"$Q\""
ssh new-box "docker exec notes_postgres psql -U notes -d notes -tAc \"$Q\""
Enter fullscreen mode Exit fullscreen mode 
users=57 notes=1342 migration=20261012093000
users=57 notes=1342 migration=20261012093000
Enter fullscreen mode Exit fullscreen mode

Count your important tables plus the highest applied migration. The table that
records migrations depends on your framework: schema_migrations in Rails and
golang-migrate, django_migrations, flyway_schema_history,
_sqlx_migrations. Counts don't prove every byte matches, but with writes
frozen, any difference means something went wrong. If they differ, stop and roll
back (section 8).

4. Start the API, pinned to the same build, and check it from inside the
container, before any traffic arrives. Setting APP_IMAGE on the command line
fills the ${APP_IMAGE:-…} variable in the compose file:

ssh new-box 'cd ~/notes-api && APP_IMAGE=ghcr.io/<you>/notes-api:3f9c2ab docker compose -f docker/docker-compose.prod.yml --env-file .env.prod up -d server && sleep 10 && docker exec notes_server wget -qO- http://localhost:8080/health'
Enter fullscreen mode Exit fullscreen mode 
{"status":"ok","build":"3f9c2ab","database":"ok","cache":"ok"}
Enter fullscreen mode Exit fullscreen mode

The migrations it runs at startup find the schema already at the latest version
and do nothing. That's exactly what we want.

5. Point DNS at the new IP. In your DNS provider, change the api A record
to 198.51.100.20, DNS-only. Then watch two public resolvers agree:

dig +short api.example.com @1.1.1.1; dig +short api.example.com @8.8.8.8
Enter fullscreen mode Exit fullscreen mode 
198.51.100.20
198.51.100.20
Enter fullscreen mode Exit fullscreen mode

6. Start Caddy and wait for the certificate:

ssh new-box 'cd ~/notes-api && APP_IMAGE=ghcr.io/<you>/notes-api:3f9c2ab docker compose -f docker/docker-compose.prod.yml --env-file .env.prod up -d && docker logs -f notes_caddy 2>&1 | grep -m1 -iE "certificate obtained|error"'
Enter fullscreen mode Exit fullscreen mode 
{"level":"info","msg":"certificate obtained successfully","identifier":"api.example.com"}
Enter fullscreen mode Exit fullscreen mode

That line ends the downtime. For a database this size, steps 1 to 6 take under
ten minutes.

Verification checklist

Run each check from the laptop, against the public name, so the whole path
(DNS, firewall, Caddy, API, database) is exercised:

curl -s https://api.example.com/health
curl -sI https://api.example.com/health | grep -i strict-transport-security
Enter fullscreen mode Exit fullscreen mode
  • [ ] /health reports "build":"3f9c2ab": the new box is answering, on the same build.
  • [ ] The Strict-Transport-Security header is present. In this app it's only sent in production mode, so it proves the right env file is loaded.
  • [ ] Sign in with an existing account. You don't have to sign in again afterwards, because JWT_SECRET moved intact.
  • [ ] A write and a read: create a note, fetch it, delete it. This is the first write the new database accepts. 
ID=$(curl -s -X POST https://api.example.com/notes -H "Authorization: Bearer $TOKEN" -d '{"text":"migration check"}' | jq -r .id)
curl -s https://api.example.com/notes/$ID -H "Authorization: Bearer $TOKEN"
curl -s -X DELETE https://api.example.com/notes/$ID -H "Authorization: Bearer $TOKEN"
Enter fullscreen mode Exit fullscreen mode
  • [ ] Webhooks: in each sender's dashboard, deliveries that failed during the freeze should now show retries answered with 200. Replay any that don't.
  • [ ] Backups move too: install the nightly backup job on the new box, disable it on the old one (two servers writing dumps into one bucket is confusing at best), run it once by hand, and check the new dump appears in the bucket. Then make it fail on purpose (for example with a wrong bucket name) and confirm the alert actually reaches you.

8. Rollback, and the one rule

Rollback is easy until the new box accepts its first write, and dangerous
after:

When How to roll back
Before DNS changed (counts differ, API won't start…) start server on the old box. Nothing was written anywhere else
After the new box accepted writes stop the new API, copy its database back (the same pipe, reversed), restore it on the old box, then start the old API and point DNS back

⛔ The rule: once the old API is stopped, never start it again on its own.
Starting it while the new box is live gives you two databases accepting writes,
which is called split-brain. Clients with the old IP still cached write to one
server, everyone else to the other, and there's no automatic merge. Whichever box
you keep, the other one's writes are lost.

Split-brain

9. Decommission, slowly

Keep the old server stopped but intact for two weeks. It's the last full copy
from before the move, and it's your fallback if something only shows up after a
few days. Then:

  1. Take one final dump and archive it.
  2. On the old box: docker compose -f docker/docker-compose.prod.yml down -v (the -v removes the volumes, meaning the data), and shred the env files (shred -u .env.prod).
  3. Rotate every secret that lived on it: storage tokens, API keys, webhook secrets. The disk is going back to a provider you no longer control. The same values now live in the new box's .env.prod, so each rotation means updating that file and restarting the server. Rotating JWT_SECRET signs every user out; do it deliberately, with a heads-up.
  4. Remove the old server's address from any allowlists, and its block from ~/.ssh/config.
  5. Delete the server in the old provider's console, and make sure its backups or snapshots go with it.

What the whole series boils down to

  1. Measure, then size. 4 GB with swap was plenty.
  2. Keep the IP (auto delete off) and firewall outside the VM, because Docker ignores ufw.
  3. Never risk a lockout: two passes, test in between, and negative tests at the end.
  4. Know what state lives where and move only the durable parts.
  5. Copy server-to-server through the laptop with an SSH pipe, and verify with a hash.
  6. Rehearse the restore. It's how you find out whether your backups are real.
  7. Freeze writes, move once, change DNS last, and never let two databases accept writes.

If this helped, or you've hit a different trap moving servers, tell me in the
comments.

Top comments (0)