Ajit Kumar

Posted on Oct 30 Edited on Oct 30

Moving a production API to Hetzner, part 1: choosing and creating the server

#devops #linux #hetzner #selfhosting

I recently moved a small production API from one VPS provider to Hetzner Cloud.
The stack is ordinary: a Rust web server, PostgreSQL, Valkey (the open-source
Redis fork), and Caddy as the TLS-terminating reverse proxy, all under Docker
Compose. Encrypted blobs live in S3-compatible object storage, which doesn't
move at all.

These posts are both the record of what I did and a tutorial. Each step comes
with the command, what it does, and why it works that way, because
most migration guides give the commands and skip the reasons, and the reasons
are what you need when something goes wrong. Plenty did go wrong, and I've left
those parts in.

Throughout, the addresses are placeholders from the documentation ranges reserved for examples (RFC 5737 and RFC 3849):

Placeholder	Means
`203.0.113.10`	the old server
`198.51.100.20`	the new Hetzner server
`2001:db8:1234:5678::/64`	the new server's IPv6 range
`api.example.com`	the API's domain
`old-box`, `new-box`	SSH aliases for the two servers (set up in part 2)

What we are moving

Only Caddy publishes ports to the outside world. Everything else talks over
Docker's private network. That detail comes back several times, because it
decides what the firewall has to allow and what the migration has to copy.

1. Picking a server type

The three families

When you click Add Server, the Type step splits the catalogue into
Shared Resources (two families) and Dedicated Resources (one):

Family	Series	CPU	Hetzner's own pitch	Who it's for
Cost-Optimized (shared)	CX (x86) and CAX (Arm64)	shared Intel/AMD, older hardware generations	"cost-efficient… with limited availability"	small production services, side projects. Cheapest per GB of RAM
Regular Performance (shared)	CPX	shared AMD, newer hardware generations	"best price/performance"	the same jobs with faster, more consistent CPU
General Purpose (dedicated)	CCX	dedicated AMD vCPUs, latest hardware	"predictable performance, critical production"	sustained CPU load: builds, busy databases

Here is each family as the console showed it in Helsinki on the day I
created the server (October 2026, USD per month, excluding VAT):

Cost-Optimized. CX23 is 2 vCPU, 4 GB, 40 GB for $6.49. Note that every row is greyed out: at that moment none of them could be ordered in Helsinki. The
summary on the right still says CPX22 because a greyed row can't be selected.

Regular Performance. CPX22 has the same 2 vCPU and 4 GB as CX23 (with 80 GB of disk) for $22.99, more than three times the price.

General Purpose. Dedicated cores start at CCX13 (2 vCPU, 8 GB) for $50.49.

The panel on the right shows one cost that's easy to miss: a public IPv4 is billed on top of the server, $0.60/month.

Cost-Optimized also offers Arm64 (Ampere), which is often the best value of all. I couldn't use it: our container image is built for amd64 only.

Check your image's architecture before choosing Arm. docker manifest inspect <image> lists the platforms it was built for.

"Shared" means your virtual CPU is a time slice of a physical core that other
customers' VMs use too. Most of the time you get the whole slice. When a
neighbour is busy you may get less, which is called CPU steal (shown as st
in top). For an API that idles most of the day that's fine; for a CI runner,
it isn't. Dedicated vCPUs are reserved for you, which is what the large price
jump buys.

How big does it have to be?

Measure before you buy. On the old server, docker stats --no-stream and
free -h showed the entire stack using about 1 GB. On the fresh Hetzner box,
Ubuntu itself sat at about 500 MB used after boot.

Option (Helsinki, October 2026)	vCPU / RAM / disk	Monthly + IPv4	Verdict
CPX02 (Regular Performance)	1 / 1 GB / 20 GB	$7.49 + $0.60	too small: Postgres, Valkey, the server and Caddy together would spill into swap
CPX12 (Regular Performance)	1 / 2 GB / 40 GB	$13.49 + $0.60	tight on RAM, and double CX23's price
CX23 (Cost-Optimized)	2 / 4 GB / 40 GB	$6.49 + $0.60	chosen: 4× headroom on RAM, the cheapest option that fits
CPX22 (Regular Performance)	2 / 4 GB / 80 GB	$22.99 + $0.60	the fallback if CX23 stays unavailable
CCX13 (General Purpose)	2 dedicated / 8 GB / 80 GB	$50.49 + $0.60	far more than an idle API needs

Prices change; take today's from the console, not from this table.

"Limited availability" is real

Cost-Optimized types carry a limited availability label, and it means it. When
I first configured the server, CX23 in Helsinki was available. A few minutes later,
when I came back to click Create, the whole CX row set was greyed out: not available (the first screenshot above)..

Hetzner allocates these types from a pool of hosts per location. When a
location's pool is full, the type disappears there until capacity frees up.
Your options:

Change the location. Nuremberg (nbg1) and Falkenstein (fsn1) in Germany and Helsinki (hel1) in Finland are all in the eu-central network zone, with similar latency to most of Europe. The type is often in stock at one of the others.
Wait and retry. Capacity comes and goes during the day. My CX23 in Helsinki came back later.
Pick the next family up (CPX), if you can't wait and the price is acceptable.

A consequence I didn't expect: once you hold a scarce server type, deleting the
server to fix a mistake is risky. You may not get another one. That shaped
part 2, where I fixed a setup mistake on the running server instead of
recreating it.

Location and image

Location: as close as possible to your users and to your object storage. My storage is a global-anycast S3 provider, so any EU location was fine.
Image: Ubuntu 26.04 LTS. "LTS" means five years of security updates, which is what you want on a server you'd rather not rebuild. 24.04 LTS would have been equally good.

2. SSH keys: how you'll get in

What a key pair is

An SSH key pair is two halves of one mathematical object:

the private key (~/.ssh/hetzner_ed25519) stays on your laptop and never leaves it;
the public key (~/.ssh/hetzner_ed25519.pub) goes on every server you want to log in to, in ~/.ssh/authorized_keys.

The server never learns your private key. At login it asks your SSH client to
sign a value that is unique to this connection. Your client signs it with the
private key, and the server checks the signature against the public key it has
on file. A valid signature proves you hold the private key without the key ever
crossing the network.

Notice there are two key pairs in play:

Key	Belongs to	Proves	Stored where
host key	the server	"I really am 198.51.100.20, not an impostor"	server: `/etc/ssh/ssh_host_*`; your laptop remembers it in `~/.ssh/known_hosts`
user key	you	"I'm allowed to log in as `ubuntu`"	laptop: private key; server: `authorized_keys`

The first time you connect, SSH shows the host key's fingerprint and asks
whether to trust it. This is called TOFU: trust on first use. If the host key
later changes, for example because you rebuilt the server, SSH refuses to connect
and warns about a possible man-in-the-middle. That's correct behaviour. After a
deliberate rebuild, remove the old entry with ssh-keygen -R <host>.

Creating the key

ssh-keygen -t ed25519 -f ~/.ssh/hetzner_ed25519 -C hetzner-api

-t ed25519: a modern elliptic-curve key. It's short (one line), fast, and has no weak-parameter choices to get wrong, unlike older RSA keys.
-f: the file name. A separate key per provider means losing or rotating one doesn't touch the others.
-C: a comment, just a label so you can recognise the key in authorized_keys later.

It asks for a passphrase. Use one: it encrypts the private key on disk, and
ssh-agent remembers it for your session, so you type it once.

Print the public half and paste it into Hetzner → Security → SSH keys → Add
SSH key:

cat ~/.ssh/hetzner_ed25519.pub

When you pick this key on the create screen, Hetzner writes it into the new
server's /root/.ssh/authorized_keys, and, as a side effect, does not set a
root password at all. Remember that; it matters in part 2.

3. Primary IPs: an address that outlives the server

What a Primary IP is

In Hetzner Cloud, a server's public address is a separate resource called a
Primary IP. It's created along with the server and attached to it, but it has
a life of its own:

IPv4: one address, billed separately (about $0.60/month), because the world has run out of IPv4.
IPv6: a whole /64 network, free. That's 2⁶⁴ addresses; the server uses the first one by convention (2001:db8:1234:5678::1).

Why "auto delete" matters

Each Primary IP has an auto delete switch. On, the address is released when
the server is deleted, and you'll probably never get it back. Off, the address
stays in your project, unattached, ready to assign to a replacement server.

Why you want it off for production:

DNS points at it. A new address means a DNS change, and DNS changes take a while to reach everyone (part 3 explains TTLs).
Other services may allowlist it. Object-storage tokens, database firewalls and payment-provider webhook settings sometimes filter by source IP.
It's your escape hatch. If the server breaks beyond repair, you can delete it and create a new one with the same address.

Turn it off in IPs → (the IPv4) → ⋯. If the menu offers "Enable Auto
Delete", it's already off.

I didn't add an IPv6 (AAAA) DNS record. The old server had none, so the
migration changes as little as possible. That can come later.

4. The Cloud Firewall: the perimeter outside your VM

Two places a firewall can sit

A Hetzner Cloud Firewall filters traffic in Hetzner's network, before it
reaches your VM. A host firewall like ufw filters it inside the VM.

The difference matters because of Docker. When a container publishes a port
(ports: ["443:443"]), Docker writes its own NAT rules that send the packet to
the container before it reaches the chain ufw controls. So ufw deny 443
does not block a published Docker port, and plenty of people have exposed a
database to the internet this way while ufw status said everything was closed.

The Cloud Firewall can't be bypassed from inside the VM, so it is the real
perimeter. ufw (set up in part 2) is a second layer for anything that isn't
Docker.

Stateful filtering, in one paragraph

Both firewalls are stateful: they track connections. You write rules for
new incoming connections only. Replies to connections your server opened (to
download packages, call an API, fetch a certificate) are recognised as part of
an existing connection and let back in automatically. That's why the rule set
below has no outbound rules at all: with no outbound rules, Hetzner allows all
outbound traffic, and the server needs outbound access to apt, the container
registry, object storage, the email provider and Let's Encrypt.

The rules

Create it before the server (Firewalls → Create Firewall) so the server is
born protected:

Protocol	Port	Source	Why
TCP	2222	Any IPv4, Any IPv6	SSH, on a non-standard port
TCP	22	Any	SSH, for networks that block outbound 2222
TCP	80	Any	Let's Encrypt's HTTP-01 challenge, and the HTTP→HTTPS redirect
TCP	443	Any	the API
UDP	443	Any	HTTP/3 (QUIC runs over UDP)
ICMP	—	Any	ping, and path-MTU discovery (blocking all ICMP can silently break large packets)

Not on the list: 5432 (Postgres) and 6379 (Valkey). They're never published, and
if a misconfiguration ever did publish them, this firewall would still drop the
traffic.

Why two SSH ports? Port 22 gets a constant stream of automated password-guessing
bots. Moving SSH to 2222 isn't security, since a port scan finds it in seconds,
but it cuts that noise to almost nothing and keeps the logs readable. Key-only
login (part 2) is the actual security. I kept 22 as well, because some networks
block outbound connections to unusual ports. As part 2 shows, my home network
turned out to do the reverse.

5. cloud-init: configuring the server before you first log in

What it is

cloud-init is a program in every major cloud image that runs once, on first
boot. It reads "user data" the provider hands to the VM, which is the text you
paste into the Cloud config box on the create screen, and applies it:
creating users, writing files, installing packages, running commands.

The appeal is that the server is configured before anyone logs in. You never
have a window where it's reachable with default settings, and the configuration
is a document you can review and reuse.

#cloud-config
users:
  - name: ubuntu
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAA...your-public-key... hetzner-api
disable_root: true
ssh_pwauth: false
package_update: true
package_upgrade: true
packages: [ufw, unattended-upgrades]
write_files:
  - path: /etc/ssh/sshd_config.d/10-hardening.conf
    content: |
      Port 22
      Port 2222
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      PermitRootLogin no
      PubkeyAuthentication yes
  - path: /etc/docker/daemon.json
    content: |
      {"log-driver": "json-file", "log-opts": {"max-size": "10m", "max-file": "3"}}
runcmd:
  - ufw default deny incoming
  - ufw default allow outgoing
  - ufw allow 22/tcp
  - ufw allow 2222/tcp
  - ufw allow 80/tcp
  - ufw allow 443/tcp
  - ufw allow 443/udp
  - ufw --force enable
  - systemctl daemon-reload
  - systemctl restart ssh.socket || true
  - systemctl restart ssh || true
  - fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
  - echo '/swapfile none swap sw 0 0' >> /etc/fstab

Every line here is explained in part 2, where we apply the same settings by hand.

The order cloud-init works in

One thing that will surprise you if you SSH in too early: cloud-init runs its
stages in order, and package_upgrade comes before runcmd. On a fresh image
the upgrade can take a few minutes, and during that time port 2222 isn't open yet.
cloud-init status --wait blocks until everything has finished.

What actually happened

I created the server with the type, location, image, SSH key and firewall all
set, and forgot to paste the cloud config. The box came up as a stock image:
only root, only port 22, no ubuntu user, no ufw, no swap. My first
ssh -p 2222 got Connection refused.

"Refused" is useful information. It means the packet reached the machine (the
Cloud Firewall allowed it) and the kernel answered "nothing is listening here".
A firewall drop would have shown up as a timeout instead. Part 2 goes into
that difference.

Since CX23 was close to sold out, I didn't delete and recreate. I applied the same
configuration by hand, as root over port 22. That's part 2.

6. Create

The final create screen, for reference:

Field	Value
Location	Helsinki (or wherever the type is in stock)
Image	Ubuntu 26.04 LTS
Type	Shared vCPU → Cost-Optimized → x86 → CX23
Networking	Public IPv4 ✅, Public IPv6 ✅
SSH keys	`hetzner-api`
Firewalls	the firewall from section 4
Backups	off (the database is backed up to object storage; see part 3)
Cloud config	the YAML above. Don't forget it
Name	something meaningful. The default `ubuntu-4gb-hel1-1` tells you nothing in six months

Then turn off auto delete on the new IPv4 (section 3), and check that the
firewall shows "Fully applied" with one server.

What we have so far

a CX23 with 4 GB RAM for about $7/month including IPv4;
an IPv4 address that survives the server;
a perimeter firewall that allows only SSH, HTTP(S) and ICMP;
a stock Ubuntu, reachable as root on port 22 only, because the cloud config was never applied.

In part 2 we fix that by hand: a non-root user, key-only SSH on two ports,
ufw, swap, automatic security updates and Docker. Along the way: how to configure
two servers in ~/.ssh/config, how to make changes over SSH without locking
yourself out, and how to tell from a port test whether a firewall or a closed
port is in your way.

DEV Community