DEV Community: Minor Keith

Self-host NetBird on Ubuntu 26.04 using a DuckDNS name

Minor Keith — Thu, 24 Sep 2026 13:28:20 +0000

I wanted one thing at home reachable from an ordinary browser, over HTTPS, without opening a port on the router and without the traffic going through someone else's edge. Cloudflare Tunnel and Tailscale Funnel both do the first part, and both route you through their network to do it. NetBird can do it on a VPS you own: the server that runs your WireGuard mesh now has a reverse proxy built in, so the VPS takes the HTTPS request and hands it down the tunnel to the machine at home.

NetBird's docs start with "point your domain and a wildcard record at the server", and that is the part that stops people who never bought a domain. This post does it with a free DuckDNS name, and I ran it on two fresh Ubuntu 26.04 servers: one as the NetBird server, and one standing in for the machine at home, with nothing open but SSH.

The one thing to get straight: the VPS takes every public connection, and the machine at home only ever connects outward. There are three pieces. The NetBird server on the VPS is the control plane (users, peers, keys) and the relay. The peer at home dials out to it, so nothing listens at home. The NetBird Proxy, also on the VPS, answers https://app.<your name> with its own certificate for that name and forwards the request over WireGuard to the peer. Traefik sits in front of both and passes TLS for the proxy's names straight through, which is why NetBird's docs say Traefik is the only front proxy that works for this.

TL;DR. Get a DuckDNS name and point it at the VPS. Open TCP 80 and 443 and UDP 3478 on the VPS. Install Docker and jq, then run NetBird's getting started script with NETBIRD_ENABLE_PROXY=true. Create the admin at https://<name>.duckdns.org/setup. On the machine at home, install the NetBird client and netbird up --management-url https://<name>.duckdns.org --setup-key <key>. Bind the app to 0.0.0.0 or the NetBird address (not 127.0.0.1), add a reverse proxy service for app.<name>.duckdns.org pointing at that peer and port, and open it. DuckDNS answers for every name under yours, which is the wildcard record the docs ask for.

What I tested on

Two Hetzner cx23 servers (2 vCPU, 4GB) with Ubuntu 26.04, on 24 September 2026. The cloud firewall on the "home" box only let SSH in.
Docker Engine 29.8.1 and Compose v5.5.1 on the NetBird server, installed from Docker's repo as in Install Docker on Ubuntu 26.04.
NetBird's server images as the script pulled them on the day (netbirdio/netbird-server:latest, netbirdio/dashboard:latest, netbirdio/reverse-proxy:latest, and traefik:v3.6); 0.79.0, released on 18 September, was the latest NetBird release, and the client on the peer reported 0.79.0.
The DuckDNS name pe-netbird.duckdns.org.

NetBird's docs label the reverse proxy as beta. Everything below worked on this version; I have not tested how a service survives a NetBird upgrade.

1. A free name that covers every subdomain

The proxy gives every service its own hostname under a base name (app.pe-netbird.duckdns.org), so the base name needs every subdomain under it to resolve to the VPS. With a real domain that is a wildcard DNS record. With DuckDNS it is already there: after I pointed the name at the server, every name under it answered with the same address.

$ dig +short pe-netbird.duckdns.org
135.181.38.45
$ dig +short app.pe-netbird.duckdns.org @ns1.duckdns.org
135.181.38.45

(Before that, x.y.pe-netbird.duckdns.org, two levels down, resolved too.)

The other reason DuckDNS works here: duckdns.org is on the Public Suffix List (I checked the list), so per Let's Encrypt's rate limit docs its limits count against your name, not against every DuckDNS user put together. That matters, because this setup asks for new certificates (two, as it turns out) for every service name.

Sign in at duckdns.org with one of the accounts it offers, create a name, and copy the token it shows you. Keep the token in a file rather than in your shell history:

mkdir -p ~/.config/duckdns
install -m 600 /dev/null ~/.config/duckdns/token
nano ~/.config/duckdns/token          # paste the token, save
curl "https://www.duckdns.org/update?domains=pe-netbird&token=$(tr -d '\n' < ~/.config/duckdns/token)&ip=203.0.113.10"

Use your own name and your VPS address. The update URL answers OK, and the new address showed up in dig within seconds.

2. Open the ports on the VPS

The script's closing message lists what it needs:

Open ports:
  - 443/tcp   (HTTPS - all NetBird services)
  - 80/tcp    (HTTP - redirects to HTTPS)
  - 3478/udp   (STUN - required for NAT traversal)
  - 51820/udp (WIREGUARD - (optional) for P2P proxy connections)

I opened TCP 80 and 443 and UDP 3478 in the provider's firewall and left 51820 closed, which matters later (section 5). The machine at home needs nothing opened at all: every connection it makes is outbound.

If the VPS also runs ufw, remember that Docker publishes ports underneath it (the ufw post calls this "Docker punches through"). The compose file the script writes publishes 51820/udp from the proxy container, so on a VPS with only ufw that port is open whatever ufw says. Here the provider's firewall is what kept it closed.

3. Install the server with the proxy switched on

The script needs Docker with the Compose plugin, curl and jq already installed. A fresh 26.04 image has curl but not jq:

sudo apt install jq

NetBird's docs give curl ... | bash. I downloaded it first so I could read it (it is about 1,800 lines), and I ran it unattended, because every prompt has an environment variable:

sudo mkdir -p /opt/netbird && cd /opt/netbird
sudo curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started.sh -o getting-started.sh
sudo NETBIRD_DOMAIN=pe-netbird.duckdns.org \
  NETBIRD_LETSENCRYPT_EMAIL=you@example.com \
  NETBIRD_ENABLE_PROXY=true \
  NETBIRD_ENABLE_CROWDSEC=false \
  NETBIRD_NON_INTERACTIVE=true \
  bash getting-started.sh

Run interactively, it asks the same things in turn: the domain, which reverse proxy (0 is the built in Traefik), an email for Let's Encrypt (required; unattended without it, the script stops with NETBIRD_LETSENCRYPT_EMAIL is required for a non-interactive install.), then Enable proxy? [y/N] and Enable CrowdSec? [y/N]. It finished with:

NETBIRD SETUP COMPLETE
...
NetBird Proxy:
  The proxy service is enabled and running.
  Any domain NOT matching pe-netbird.duckdns.org will be passed through to the proxy.
  The proxy handles its own TLS certificates via ACME TLS-ALPN-01 challenge.
  Point your proxy domain to this server's domain address like in the examples below:

  *.pe-netbird.duckdns.org    CNAME    pe-netbird.duckdns.org

That last line is the wildcard record DuckDNS already gives you. Four containers were running: netbird-server, netbird-dashboard, netbird-proxy and netbird-traefik. The configuration is in /opt/netbird (docker-compose.yml, config.yaml, dashboard.env, proxy.env, traefik-dynamic.yaml), so back that folder up with the Docker volumes it names.

The screenshots come from a clean run on two fresh boxes after the post was written, so addresses, names and timings differ a little from the text. The apt install jq step is left out of this one; it only printed apt's restart notice.

The dashboard certificate was a real one on the first try, CN=pe-netbird.duckdns.org from Let's Encrypt. And it is small: docker stats put the four containers at 70MiB, 22MiB, 21MiB and 20MiB. The docs ask for 1 CPU and 2GB, and that is generous.

4. Create the admin

Open https://pe-netbird.duckdns.org/setup and create the first user. That page only works while no users exist, which you can check: curl https://pe-netbird.duckdns.org/api/instance said {"setup_required":true} before and {"setup_required":false} after. Logins are handled by an embedded Dex server, so there is nothing else to set up for users.

I had no browser on the lab, so I did this with the call the setup page makes, POST /api/setup with an email, password and name, and logged in afterwards with a script. In a browser it is one form. Once in, you need a setup key for the machine at home: create a reusable one in the dashboard (I used the API, POST /api/setup-keys).

5. Join the machine at home

On the machine that runs the service, install the client and point it at your server:

curl -fsSL https://pkgs.netbird.io/install.sh | sh
sudo netbird up --management-url https://pe-netbird.duckdns.org --setup-key <your setup key>
netbird status

Management: Connected
Signal: Connected
Relays: 2/2 Available
FQDN: nb-peer.netbird.selfhosted
NetBird IP: 100.66.194.33/16

It got the address 100.66.194.33 on a new wt0 interface. On the peer, the only TCP port listening on a public address was still SSH. NetBird does open its own UDP sockets (WireGuard on 51820 among them), but the peer's cloud firewall let nothing but SSH in and everything below still worked, because the connections that matter are the ones the peer makes outward.

Once a service exists (next section), the proxy shows up on the peer as another NetBird peer, and netbird status -d says how they are connected:

proxy-...netbird.selfhosted:
  NetBird IP: 100.66.196.27
  Status: Connected
  Connection type: Relayed
  Relay server address: rels://pe-netbird.duckdns.org:443

Relayed is because I left UDP 51820 closed on the server, so the two could not talk WireGuard directly and went through the server's relay over 443 instead. It worked and responses came back in about half a second. Open 51820 if you want direct connections; I did not test that.

6. Publish one service

The app has to be reachable on the peer's NetBird address, because that is where the proxy connects. I used a plain web server with one page, bound to that address (use the NetBird IP your netbird status shows, without the /16):

mkdir -p ~/app
echo "hello from nb-peer, a box with only port 22 open" > ~/app/index.html
python3 -m http.server 8080 --bind 100.66.194.33 --directory ~/app

Binding to 0.0.0.0 works too. Binding to 127.0.0.1 does not (section 9).

Then add the service: in the dashboard it lives under the reverse proxy section (per the docs), and I used the API, POST /api/reverse-proxies/services, with the domain app.pe-netbird.duckdns.org, the peer as the target, protocol http and port 8080. The server filled in the peer's NetBird address as the target host by itself.

The service came back pending, and its status recorded the certificate as issued six seconds after I created it. From my laptop:

$ curl -v https://app.pe-netbird.duckdns.org/
*  subject: CN=app.pe-netbird.duckdns.org
*  issuer: C=US; O=Let's Encrypt; CN=YE1
< HTTP/2 200
hello from nb-peer, a box with only port 22 open

A real certificate for that exact name, served from a box whose cloud firewall only allows SSH in. My first request loop saw the first 200 about a minute after creating the service, so give it a moment before assuming something is wrong.

On the clean run the service is called home. rather than app., for the reason in the next paragraph.

Rebuilding? Mind the certificate limit. The proxy asks Let's Encrypt for two certificates per name, an ECDSA one and an RSA one (the certificate volume held app.pe-netbird.duckdns.org and app.pe-netbird.duckdns.org+rsa). Let's Encrypt allows five certificates for the exact same name per week, so the third time I built this setup with app. in one week, only the RSA one came back. The proxy logged the refusal, and my laptop, which asks for ECDSA, got a TLS error (tlsv1 alert internal error, with acme/autocert: missing certificate in the proxy log) while a client limited to RSA still got a 200. A new name (home.) worked straight away. If you are testing, keep the Docker volumes between rebuilds, or change the name.

The proxy passes the visitor's address through: its log showed my own public IP as client_ip, not Traefik's.

7. Put a password or PIN in front of it

A service with no authentication is open to the internet, which is the point, but most homelab dashboards should not be. The proxy can put a password, a PIN, SSO through your NetBird login, and a few other options in front (the docs list them). I tested the password and the PIN.

I set a password on the service (the auth.password_auth field when you update it through the API; the dashboard has the same options, per the docs). After that, an anonymous request got a 401 and a small "NetBird Service" login page. The page posts the password back to the same URL:

$ curl -s -o /dev/null -w '%{http_code}\n' -F password=wrong https://app.pe-netbird.duckdns.org/
401
$ curl -s -D - -o /dev/null -F password=<the right one> https://app.pe-netbird.duckdns.org/
HTTP/2 303
location: /
set-cookie: nb_session=...; Path=/; Max-Age=86400; HttpOnly; Secure; SameSite=Lax

With that cookie the app answered. Max-Age=86400 means a login lasts a day. The password is checked by the proxy on the VPS, before anything reaches the machine at home, which is the reason to use it rather than a login in the app itself.

8. netbird expose, for something quick

For a quick share there is a command on the peer itself:

netbird expose --with-pin 123456 8080

The first time, it failed with peer expose is not enabled for this account. Turning it on is an account setting (Peer Expose, per the docs), and the server refused it until I also chose a group allowed to use it: peer expose requires at least one group. I picked All. After that:

Service exposed successfully!
  Name:     908osflzfrae
  URL:      https://908osflzfrae.pe-netbird.duckdns.org
  ...
Press Ctrl+C to stop exposing.

Without the PIN that URL gave 401, with it a 303 and then the page. When the command stopped, the name stopped answering. The name is random each time (--with-name-prefix sets the start of it), so every run asks Let's Encrypt for new certificates. On a DuckDNS name those count against Let's Encrypt's limits for your name (the weekly limit on new certificates per registered name is in their docs; I did not hit that one), which is fine for the odd share. Run it in a loop and you use up that limit, and new share names stop getting certificates until the week rolls over.

9. What 502, 401 and a timeout mean

I broke it on purpose, from the outside with curl, with the password from section 7 taken off again (with it still on, every row below is just a 401 until you log in):

What I did on the peer	What the browser gets
stopped the app	`502`, and the proxy log says `status=502 title="Service Unavailable"`
ran the app on `127.0.0.1:8080` only	the same `502`
ran it on `0.0.0.0:8080`	`200`
`netbird down`	no answer at all: `curl` gave up after 20 seconds
`netbird up` again	`200`, but the first request took 14 seconds

In these tests, a stopped app and an app listening only on loopback both gave 502, a disconnected peer gave a timeout, and 401 was only ever my own password or PIN doing its job. So on a 502, check where the app is listening; on a timeout, check netbird status on the peer. The loopback case is the one that catches people, because the app works fine from the machine itself.

10. Next to Cloudflare Tunnel and Tailscale Funnel

I have not tested either of those for this post, so this is only the difference in shape, from their docs. Cloudflare Tunnel ends your visitors' HTTPS on Cloudflare's network. Tailscale Funnel relays the encrypted connection through Tailscale's servers and ends TLS on your own machine, with a certificate for its ts.net name. Both need an account with the vendor, and the public name is theirs or on their DNS. NetBird's proxy ends TLS on your VPS, and the only third parties are DuckDNS for the name and Let's Encrypt for the certificates. The price is that you run and patch the VPS, and the feature is still marked beta. If you already use Tailscale for your own access, you do not need any of this for yourself; this is for letting other people in.

Gotchas I hit

A fresh Ubuntu 26.04 image has no jq. Install it before the getting started script, which needs it.
Unattended mode does not make the Let's Encrypt email optional: set NETBIRD_LETSENCRYPT_EMAIL.
Creating the admin needs the /setup page in a browser (or the POST /api/setup it calls). It only works while there are no users.
An app bound to 127.0.0.1 gives a 502. Bind to 0.0.0.0 or the NetBird address.
With UDP 51820 closed, the proxy reached the peer through the relay on 443. It still worked.
The first request after the peer reconnects can take many seconds.
netbird expose is off until you enable Peer Expose and give it a group, and every run makes a new random hostname and new certificates.
Each service name uses two certificates, ECDSA and RSA. Build with the same name three times in a week and one of them is refused (429 rateLimited ... (5) already issued for this exact set of identifiers); for me it was ECDSA, which showed up as a TLS error on my laptop.

Quick reference

Job	Command or setting
Point DuckDNS at the VPS	`curl "https://www.duckdns.org/update?domains=<name>&token=$(tr -d '\n' < ~/.config/duckdns/token)&ip=<vps ip>"`
VPS ports	TCP 80, 443; UDP 3478 (51820 optional)
Prerequisites	Docker with Compose, `curl`, `sudo apt install jq`
Install with the proxy	`NETBIRD_DOMAIN=<name>.duckdns.org NETBIRD_LETSENCRYPT_EMAIL=<email> NETBIRD_ENABLE_PROXY=true NETBIRD_NON_INTERACTIVE=true bash getting-started.sh`
First user	`https://<name>.duckdns.org/setup`
Join a peer	`curl -fsSL https://pkgs.netbird.io/install.sh \
Check it	{% raw %}`netbird status`, `netbird status -d`
App binding	`0.0.0.0` or the NetBird IP, not `127.0.0.1`
Quick share	`netbird expose --with-pin <6 digits> <port>`

If the public URL gives a 502, look at where the app is listening before you look at NetBird: an app that answers on localhost can still be unreachable through the tunnel.

Originally published at peculiarengineer.com.

Install Portainer CE on Ubuntu 26.04 (without exposing my Docker host)

Minor Keith — Thu, 24 Sep 2026 13:25:55 +0000

The install command in Portainer's docs is one line, and it works. It also publishes the Portainer UI on port 9443 on every address the server has, and Docker writes its own firewall rules for that, so ufw never gets a say. On a box where ufw said only SSH was allowed in, a client standing in for the outside world could still load the Portainer login page. That page is the front door to a container that holds the Docker socket, which means whoever gets through it as an admin is root on the host.

So this is the Portainer install I would actually put on a server: Compose instead of a long docker run, the UI bound to 127.0.0.1, and an SSH tunnel to reach it. It also covers the parts the older guides predate or skip. Since 2.43, Portainer will not let you create the admin account until you give it a token from its logs, and it still locks itself after five minutes if you do not. Stack files end up somewhere docker compose ls points at but you cannot open. And updating, backing up and removing it each have one small trap.

The one thing to get straight: Portainer is not a dashboard sitting beside Docker. It is a web login in front of /var/run/docker.sock. I proved that on the lab box by asking Portainer's API, and nothing else, to start an Alpine container with the host's / mounted and print the first line of /etc/shadow. It printed root:!*:17478:0:99999:7:::. Treat the Portainer login like an SSH login, and keep it off the open internet the same way.

TL;DR. Install Docker from Docker's repo. docker volume create portainer_data, then run Portainer from a Compose file that publishes only 127.0.0.1:9443:9443 and mounts portainer_data as an external volume. Reach it with ssh -N -L 9443:127.0.0.1:9443 you@server and open https://localhost:9443. Within five minutes, create the admin with the setup_token from docker logs portainer; if it times out, docker restart portainer and use the new token. Update with docker compose pull && docker compose up -d. Back up by stopping Portainer and tarring the volume.

What I tested on

A Hetzner cx23 running Ubuntu 26.04.1 LTS, kernel 7.0.0-30. The cloud firewall only let port 22 in.
Docker Engine 29.8.1 and Docker Compose v5.5.1 from Docker's apt repo.
Portainer CE 2.45.1 LTS, which is what the lts tag pulled on 24 September 2026, plus 2.45.0 for the update test.
ufw 0.36.2, enabled with only OpenSSH allowed.

I did every step that normally happens in the browser (creating the admin, adding the environment, deploying a stack, the backup download) through Portainer's HTTP API with curl, because the box had no browser on it. So what I tested is the server's responses; I have not seen the browser flow in 2.45.1, and where a step names a menu, that is from Portainer's docs.

1. Install Docker first

Portainer runs as a container, so Docker comes first. Use Docker's own apt repo, not Ubuntu's docker.io; Install Docker on Ubuntu 26.04 has the exact steps and is what I ran on this box. The commands below assume your user is in the docker group (or put sudo in front). Check it answers:

docker version
docker compose version

2. What 9443, 8000 and 9000 are

The docs command publishes two ports and the image exposes three. The container logs say what each one is:

starting HTTPS server | bind_address=:9443
starting HTTP server | bind_address=:9000
server: Reverse tunnelling enabled
server: Listening on http://0.0.0.0:8000

9443 is the web UI and the API over HTTPS, with a certificate Portainer generates for itself on first start. This is the only one you need.
8000 is a tunnel server for Portainer's Edge agents, which per Portainer's docs is how it manages remote Docker hosts that dial in to it. If Portainer is only managing the machine it runs on, nothing ever connects to 8000. A plain curl to it returns 404. Leave it unpublished.
9000 is the same UI over plain HTTP. It is listening inside the container in 2.45.1 (a curl to the container's address on 9000 returned the status JSON with a 200), and without -p 9000:9000 it is reachable from the host only on the container's bridge address (the section 3 install's 172.17.0.2:9000), not from other machines. Do not publish it. Your password would cross the network in the clear.

3. Why the stock command gets past ufw

This is the command from Portainer's docs:

docker volume create portainer_data
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data portainer/portainer-ce:lts

-p 9443:9443 with no address means every address. docker ps shows it:

0.0.0.0:8000->8000/tcp, [::]:8000->8000/tcp, 0.0.0.0:9443->9443/tcp, [::]:9443->9443/tcp, 9000/tcp

With ufw on and only SSH allowed, I tested from a separate network namespace wired to the host, which stands in for a machine outside (the real internet could not reach the box past Hetzner's firewall). A plain Python web server on port 8081 on the host timed out, which is ufw doing its job. Portainer on 9443 answered 200. The reason is in iptables (the first rule is in the nat table, sudo iptables -t nat -S DOCKER; the second in the filter table, sudo iptables -S DOCKER):

-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9443 -j DNAT --to-destination 172.17.0.2:9443
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 9443 -j ACCEPT

Published ports are rewritten to the container's address before ufw's input rules ever look at them, and Docker adds its own accept rule on the forwarding path. The ufw post calls this out as "Docker punches through", and Portainer is the worst container to learn it with.

The screenshots come from a clean run on a fresh box after the post was written, so tokens, IDs, timestamps and sizes differ from the text. The docker volume create and the stock docker run are left out of this one because the image download fills the screen.

In fairness to Portainer, a stranger who finds a fresh install can no longer just create the admin account (section 6 covers the token). But the login page is still there for anyone to try passwords against, on an app that is root on your box.

4. Install Portainer with Compose on 127.0.0.1

Put the address in front of the port and Docker only listens there. I keep it in a Compose file so the flags are written down and the update in section 8 is two commands.

If you ran the docs command from section 3, take that container away first. The volume stays, and Compose cannot start a second container called portainer while the first one exists (it fails with Conflict. The container name "/portainer" is already in use):

docker stop portainer
docker rm portainer

Then create the volume by hand (it already exists if you ran section 3, which is fine), so its name is exactly portainer_data:

docker volume create portainer_data
sudo mkdir -p /opt/portainer
sudo nano /opt/portainer/compose.yaml

services:
  portainer:
    image: portainer/portainer-ce:lts
    container_name: portainer
    restart: always
    ports:
      - "127.0.0.1:9443:9443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data

volumes:
  portainer_data:
    external: true

cd /opt/portainer
docker compose up -d

Two details in there are deliberate. Only 9443 is published, and only on loopback. And external: true tells Compose to use the existing portainer_data volume instead of making its own. Leave that out and Compose creates portainer_portainer_data, named after the project folder. I did exactly that on the lab box while switching from the docker run install: docker volume ls showed both volumes, Portainer came up empty and printed a fresh setup token, and my admin password got Invalid credentials. The old data was untouched, just ignored. With external: true the same file brought back the admin and the stack.

I checked the listening side on the equivalent docker run -p 127.0.0.1:9443:9443 install on the same volume, which publishes exactly what this file does:

$ docker ps --format '{{.Names}} {{.Ports}}'
portainer 8000/tcp, 9000/tcp, 127.0.0.1:9443->9443/tcp
$ sudo ss -tlnp | grep 9443
LISTEN 0      4096       127.0.0.1:9443      0.0.0.0:*    users:(("docker-proxy",pid=4028,fd=8))

From the outside namespace, 9443 now timed out. From the host itself, curl -k https://127.0.0.1:9443/ returned 200.

If you prefer docker run, the same thing is -p 127.0.0.1:9443:9443 and no -p 8000:8000.

5. Reach it over an SSH tunnel

From your own machine:

ssh -N -L 9443:127.0.0.1:9443 you@your-server

Leave that running and open https://localhost:9443. SSH carries the connection to the server and hands it to 127.0.0.1:9443 there, so Portainer is exactly as reachable as your SSH is. From my Mac, through the tunnel:

$ curl -sk https://localhost:9443/api/system/status
{"Version":"2.45.1","InstanceID":"1b75100e-d5f7-4444-8c93-0117a4534e8b"}

Expect a certificate warning in the browser (I only reached it with curl -k). It is one Portainer made for itself on first start, with an empty subject, valid for five years, and it lives in the volume under /data/certs (it survived the container being recreated). If you want a real certificate, put Portainer behind a reverse proxy such as Caddy, but keep that proxy private. If it accepts connections from the internet, the Portainer login is back on the internet, even though Portainer itself still listens only on loopback.

For everyday access I would use Tailscale so Portainer is reachable from your tailnet and nowhere else. I did not test that route for this post, only the SSH tunnel.

6. First login: the setup token and the five minute window

Older guides say: open the page and create an admin. On 2.43 and later there is a step before that. The startup logs print a token. This is from docker logs portainer on the first start:

setup_token=7416bc9610a1a2fc79c6021969798e9660fd8e0ed67dd5b836e391ea8fdc2614

Paste it into the setup screen, or send it in the X-Setup-Token header.
Start with --no-setup-token to disable.

To pull out just the newest token (which matters after a restart, below):

docker logs portainer 2>&1 | grep -o "setup_token=[0-9a-f]*" | tail -1

Without it, creating the admin fails. Over the API that is a 403:

{"message":"Invalid or missing setup token. Provide the X-Setup-Token header with the token printed in the server logs at startup.","details":"Invalid or missing setup token"}

The release notes for 2.43.0 describe it as a one time token that is required to create the first administrator or to restore a backup on a new instance. It exists for exactly the problem in section 3: someone finding your fresh install before you do. I have not tried --no-setup-token and see no reason to.

With the token, the admin was created. Two more things from that step:

The password has a minimum length. short came back Password does not meet the requirements. A 16 character one was accepted.
After creating the admin through the API, there is no environment yet: GET /api/endpoints returned [], and I added the local Docker socket as an environment called local myself. Per Portainer's docs, the web setup flow does that step for you (Get Started). I did not see it in a browser.

The five minute window is still real. I started Portainer at 02:10:54, did nothing, and at 02:15 the log said:

the Portainer instance timed out for security purposes, to re-enable your Portainer instance, you will need to restart Portainer

The container does not stop. docker ps still said Up 5 minutes, which is what makes this confusing. The front page redirects to /timeout.html, and the API says Administrator initialization timeout. The fix is:

docker restart portainer
docker logs portainer 2>&1 | grep -o "setup_token=[0-9a-f]*" | tail -1

After the restart the log had a second, different token, and the new one is the one I used. Get the tunnel open before you start Portainer, then take the newest token from the logs, not the one you copied earlier.

Once an admin exists, restarts and recreated containers do not print a token.

7. Stacks, and where the Compose file actually lives

A stack in Portainer is a Compose project that Portainer deployed. In the UI it is Stacks, then Add stack, then the web editor; I deployed this one through the API:

services:
  whoami:
    image: traefik/whoami:v1.11
    ports:
      - "127.0.0.1:8080:80"
    restart: unless-stopped

It is a real Compose project, so the host sees it. Here it is next to one I started by hand from /opt/hello:

$ docker compose ls
NAME                STATUS              CONFIG FILES
hello               running(1)          /opt/hello/compose.yaml
whoami              running(1)          /data/compose/1/docker-compose.yml

/data/compose/1/docker-compose.yml is a path inside the Portainer container. On the host, ls /data/compose/1 says No such file or directory. The file is really here:

/var/lib/docker/volumes/portainer_data/_data/compose/1/docker-compose.yml

docker compose -p whoami ps still works from the host, because Compose finds projects by label. But the file to edit is the copy in Portainer, and if you change the stack from the command line Portainer's copy stops describing what is running.

On the clean run Portainer itself was already a Compose project, so it shows up in that list too.

The other direction: Portainer's stack list only holds stacks Portainer created. The hello project did not appear in /api/stacks, although its container did show up with its hello project label. I did not check how the web UI presents a project like that.

My rule of thumb from this: pick one owner per project. Things I deploy from a Git repo or a folder on the server stay plain Compose files, and Portainer is for looking at them, reading logs and restarting. If a stack was born in Portainer, edit it in Portainer.

8. Update Portainer without losing anything

Everything Portainer knows lives in the volume: the database with users, environments and settings, the stack files, the certificate, and the Edge tunnel key. The container is disposable. Updating means a new container on the same volume.

With the Compose file from section 4 and the lts tag:

cd /opt/portainer
docker compose pull
docker compose up -d

To see a real version change, I ran a second instance pinned to 2.45.0 on its own volume, added a second user and a stack, then changed the tag to 2.45.1 and ran the same two commands. The log showed the migration:

migrating database from version 2.45.0 to 2.45.1
db migrated to 2.45.1

The status endpoint reported 2.45.1 with the same instance ID, the second user could log in, the stack was still listed, and the stack's container went from Up 11 seconds to Up 19 seconds, so updating Portainer did not restart the things it manages. The old image stays on disk, 201MB each here, until you remove it. On a pinned tag, remove it by name (docker image rm portainer/portainer-ce:2.45.0); a plain docker image prune reclaimed 0B here, because an image that still has a tag is not dangling. On lts, the pull moves the tag to the new image, so the old one is left untagged and docker image prune is the command that removes it (Docker's rule for dangling images; lts did not move during my test).

On the clean run the test instance was a plain docker run container: I removed it and started it again on lts rather than editing a Compose file, and the migration went the same way.

Portainer also copied its database to /data/backups/portainer.db.bak before migrating. I saw that when I tried going back from 2.45.1 to 2.45.0, which worked for that one patch release. One patch release going back cleanly does not prove the next update can be undone. Take the full volume backup in section 9 first, so if changing the image back fails you can restore Portainer's state; Portainer's docs say the same.

If you installed with the docs docker run command, the update is docker stop portainer, docker rm portainer, docker pull portainer/portainer-ce:lts, and the same docker run again. Removing the container does not touch the volume.

If Portainer comes up after a Docker update and says The environment named local is unreachable, look at your Docker version before you blame Portainer. On a separate lab box, Docker 29.0.0, 29.1.5 and 29.2.0 reported a minimum API version of 1.44, which is the version floor the GitHub reports of that message point at. I did not see the message on a Portainer screen myself. From 29.3.0 through 29.8.1 the minimum was back to 1.40, and Portainer 2.33.3 worked on 29.8.1. This box ran Docker 29.8.1, which reports minimum version 1.40, and 2.45.1 saw the local environment fine. So if docker version shows a 29.0 to 29.2 engine, updating Docker is the first thing to try.

9. Back up and restore portainer_data

The dependable backup is the volume itself, taken while Portainer is stopped so the database is not mid write:

sudo mkdir -p /root/backups
docker stop portainer
docker run --rm -v portainer_data:/data:ro -v /root/backups:/backup alpine:3.22 \
  tar czf /backup/portainer_data-$(date +%F).tar.gz -C /data .
docker start portainer

That produced a file of 9 to 16KB across my runs, holding portainer.db, compose/1/docker-compose.yml, certs/, chisel/private-key.pem and the portainer.key pair. The stack containers kept running while Portainer was stopped. Anyone logged in gets logged out, though: the session token from before the stop came back Invalid JWT token.

To prove it restores, I removed the container and deleted the volume, then ran this (use your own backup's file name; the screenshot run did it the same day with $(date +%F)):

docker volume create portainer_data
docker run --rm -v portainer_data:/data -v /root/backups:/backup alpine:3.22 \
  tar xzf /backup/portainer_data-2026-09-24.tar.gz -C /data
cd /opt/portainer
docker compose up -d

Use the Compose file to start it again, not a docker run line: a container started by docker run is not part of the Compose project, so a later docker compose down would leave it running and docker volume rm would fail with volume is in use. No setup token in the logs, the admin logged in, the environment and the stack were back, and the stack's Compose file came back identical.

Portainer also has its own backup under Settings, then Back up Portainer, which downloads a .tar.gz (the API call takes an optional password for it). Through the API that file held the same things plus a JSON export of the database. Per the docs you restore it on a brand new Portainer from the setup screen. I did it through the API, and that restore needs the setup token as well: without it I got the same 403, with it the restore returned 200 and the user and stack were there. That should put a restore on the same five minute clock, though I did not time it.

Neither backup includes the data your containers keep in their own volumes. The Portainer backup is Portainer's memory, not your apps. Back those up separately; restic to object storage is how I would do it.

10. Remove Portainer, and what stays behind

cd /opt/portainer
docker compose down

That removed the Portainer container and its network. Everything else stayed:

every stack Portainer deployed, still running;
the portainer_data volume;
the Portainer images;
docker compose ls still listing each stack with a /data/compose/... path that now exists nowhere you can reach.

Removing Portainer does not remove what it deployed, which is usually what you want. To turn a stack into a plain Compose project, copy its file out of the volume before you delete the volume, take it down by project name, and bring it back up from the copy:

sudo cp /var/lib/docker/volumes/portainer_data/_data/compose/1/docker-compose.yml ~/whoami-compose.yml
sudo chown $USER: ~/whoami-compose.yml
cat ~/whoami-compose.yml
docker compose -p whoami down
docker compose -p whoami -f ~/whoami-compose.yml up -d

The chown and the copy into your home matter: if your user cannot read the copied file (a copy under /root was unreadable in one walkthrough), the down works and the up fails with permission denied, leaving the stack stopped. After that, docker compose ls showed the project running from the copied file, an ordinary Compose project with a file you can open. Put the copy wherever you keep Compose files. The directory numbers are Portainer's stack IDs, so check the file is the one you think it is before you copy it.

That run was as root, so ~ is /root there, and I skipped the cat check because I had just deployed that stack myself.

To remove everything, take the stacks down first (docker compose -p <name> down), then:

docker volume rm portainer_data
docker image rm portainer/portainer-ce:lts

Gotchas I hit

The docs command is reachable past ufw: -p 9443:9443 and -p 8000:8000 listen on every address and Docker's rules skip ufw. Bind to 127.0.0.1, and do not publish 8000 unless you use Edge agents.
An admin, or any user given access to the local environment, can start a container with / mounted, so Portainer is root on the host. A plain user with no environment access got 403 Permission denied to access environment. Guard the admin login like SSH.
On 2.43 and later the first admin needs the setup_token from docker logs portainer. Guides written before that do not mention it.
The five minute lockout does not stop the container. docker ps says it is up while every page redirects to /timeout.html; docker restart portainer fixes it and prints a new token.
A plain volumes: portainer_data: in Compose becomes portainer_portainer_data, and Portainer looks freshly installed. Use external: true.
Stopping Portainer for a backup invalidated my session. Expect to log in again after any restart.
docker compose ls shows stack files at /data/compose/N/..., a path inside the container; the real file is under /var/lib/docker/volumes/portainer_data/_data/compose/N/.
Both backup methods cover Portainer's own state, not your apps. Their volumes need their own backup.

Quick reference

Job	Command
Create the volume	`docker volume create portainer_data`
Start (Compose)	`cd /opt/portainer && docker compose up -d`
Port line to use	`"127.0.0.1:9443:9443"`
Tunnel from your machine	`ssh -N -L 9443:127.0.0.1:9443 you@server`, then `https://localhost:9443`
Setup token	`docker logs portainer 2>&1 \
Timed out before setup	{% raw %}`docker restart portainer`, then the new token
Version	`curl -sk https://127.0.0.1:9443/api/system/status`
Update	`docker compose pull && docker compose up -d`
Back up	stop, `tar czf` the volume through an `alpine` container, start
Stack files on the host	`/var/lib/docker/volumes/portainer_data/_data/compose/`
Remove	`docker compose down`, then stacks, then `docker volume rm portainer_data`

Portainer is a login in front of the Docker socket. Keep it on loopback, reach it through a tunnel, and back up the volume before you touch it.

Originally published at peculiarengineer.com.

Rocky Linux 10 for Ubuntu people

Minor Keith — Wed, 23 Sep 2026 20:13:14 +0000

Most of the boxes I look after run Ubuntu. The RHEL family (Red Hat, and its free rebuilds Rocky and Alma) is what most of the enterprise world runs, and sooner or later someone hands you one. The Debian post was the easy version of this comparison, because Debian and Ubuntu share apt and most of /etc. This is the hard version. Sit down at a Rocky Linux 10 prompt with Ubuntu habits and apt, dpkg, ufw, netplan and add-apt-repository are all command not found.

I built it the same way. One fresh Rocky Linux 10 server, one fresh AlmaLinux 10 server and one fresh Ubuntu 26.04 server from the same provider on the same day, the same script run on all three, then a second round of hands on checks on the Rocky box: a user, a firewall, a port change, a reverse proxy, Docker, and a reboot. Rocky and Alma came back near identical, so the text says "Rocky" and calls out the one functional place Alma differs. Where something is a property of the cloud image rather than the distribution, I say so, and this time that includes the biggest finding in the post.

TL;DR. systemd, journalctl, sshd_config.d, cloud-init, chrony and rsyslog all carry over. Packaging is the same ideas under different names: dnf instead of apt, .repo files instead of .sources, EPEL instead of PPAs, wheel instead of sudo, and no EXTERNALLY-MANAGED marker, so pip install as root just works. Networking is NetworkManager and nmcli, not netplan. The firewall is firewalld, and it is not on this cloud image. The trap is SELinux: this image ships it permissive, so an SSH port change and an nginx proxy_pass both work, then fail on the first restart or request after you set enforcing, with Bind to port 2222 on 0.0.0.0 failed: Permission denied and (13: Permission denied) while connecting to upstream. The fixes are semanage port and setsebool, and both are below.

What is the same

Start here, because it is more of the list than the reputation suggests. On both boxes:

systemd 25x, systemctl, journalctl, hostnamectl, timedatectl. Every systemd and journal command I ran for this post behaved the same on both, so the systemd units and timers post and the journalctl post should carry over; the one path difference is under logs below.
cloud-init, which on both images creates only root, locks root's password, leaves the admin group empty (wheel on Rocky, sudo on Ubuntu) and drops root ALL=(ALL) NOPASSWD:ALL in /etc/sudoers.d/90-cloud-init-users. That is the image, not the distribution.
sshd with Include /etc/ssh/sshd_config.d/*.conf, root allowed by key only, and your own settings going in a numbered drop-in. Rocky ships three drop-ins already (40-redhat-crypto-policies.conf, 50-redhat.conf, 50-cloud-init.conf); Ubuntu's directory is empty.
chrony for time, on both. chronyc tracking works and timedatectl timesync-status errors out on both, because neither runs timesyncd. Rocky synced to the provider's NTP server, Ubuntu's chrony to Canonical's pool.
rsyslog, running on both, so both have text logs beside the journal. The file names differ, covered below.
bash 5 as the login shell, vim and nano installed, ss, lvm, parted, growpart, mkfs.xfs, mkfs.ext4, mdadm, cryptsetup.
su - to root from a normal user fails on both cloud images with Authentication failure, because root's password is locked. I tried it on Rocky here and on Ubuntu in the Debian post's lab, on the same image.

If your day is systemctl, journalctl and editing files under /etc, you will not notice which one you are on. The rest of this post is the remainder: packaging, networking, the firewall and SELinux, which is to say everything Ubuntu made easy.

The versions

RHEL 10 shipped in May 2025 and its clones followed within weeks. Ubuntu 26.04 is eleven months younger and, like Debian 13, the older release shows it:

	Rocky Linux 10.2 (Red Quartz)	AlmaLinux 10.2 (Lavender Lion)	Ubuntu 26.04.1 (resolute)
Kernel	6.12 (`el10_2` build)	same	7.0 (`-generic`)
systemd	257	257	259
Python	3.12.14	3.12.14	3.14.4
Package tool	dnf 4.20.0	dnf 4.20.0	apt 3.2.0
bash	5.2	5.2	5.3
OpenSSH	9.9p1	9.9p1	10.2p1
sudo	original sudo 1.9.17p2	same	sudo-rs 0.2.13
coreutils	GNU 9.5	GNU 9.5	uutils 0.8.0 (`df` still GNU 9.7)
Packages installed, fresh image	443	443	585
First release (10.0 / 26.04)	11 June 2025	27 May 2025	April 2026
Support until	May 2030 active, May 2035 security	same	May 2031, then Ubuntu Pro to 2036

The screenshots in this post are from a second pair of fresh boxes run the same night, and both of the provider's images had moved in between: the second Rocky box booted as 10.1 and was brought to 10.2 with dnf upgrade before this shot (that story is under updates below), and the second Ubuntu box has kernel build -30 where the first had -31. Build suffixes in the shots differ from the table for that reason; the versions do not.

The release and support dates are from the projects' own pages, not the lab. Everything else in the table is what the boxes printed. Two rows matter more than they look. The kernel: RHEL keeps one kernel base for a whole major release and backports fixes into it, so Rocky 10 will still say 6.12 in 2030 (that is Red Hat's stated policy, not something the box can show). And the support row: ten years, with the last five security only, against Ubuntu's five plus a paid tail. That is what the family is for, and nothing in this post argues with it.

One row that is not in the table because it is about your CPU, not the box: RHEL 10 and Rocky 10 require an x86-64-v3 processor, roughly Intel Haswell (2013) or newer, per their release notes. AlmaLinux 10 also builds for v2, which is the one reason to pick Alma over Rocky on a specific box. The lab CPUs (AMD EPYC Rome) are v3, so I could not test the refusal.

Users: wheel, not sudo

The admin group is wheel, and /etc/sudoers on Rocky already has %wheel ALL=(ALL) ALL. So a new admin user is:

sudo useradd -m -G wheel keith
sudo passwd keith
sudo mkdir -m 700 /home/keith/.ssh
sudo cp /root/.ssh/authorized_keys /home/keith/.ssh/
sudo chown -R keith:keith /home/keith/.ssh

and an existing user joins with sudo usermod -aG wheel keith. That is the create a sudo user post with sudo changed to wheel, plus the key copy, which matters more here than on Ubuntu: this image sets PasswordAuthentication no, so a user with only a password cannot log in over SSH at all. I logged in as keith with the copied key to check. Two details. Rocky's login.defs has CREATE_HOME yes, so -m is not strictly needed, but it is harmless and I keep typing it. And the %wheel line asks for a password; for passwordless sudo, drop a file in /etc/sudoers.d/ as you would on Ubuntu, which I did for keith to run the checks below.

Ubuntu's group is sudo with %sudo ALL=(ALL:ALL) ALL. Same shape, different name, and neither box has the other's group: usermod -aG sudo keith on Rocky fails with group 'sudo' does not exist, and there is no wheel group on Ubuntu.

Packaging: dnf, repos, EPEL and history

Here is the translation. Everything with output quoted below was typed on the box.

The commands. dnf install, dnf remove, dnf upgrade, dnf search, dnf info. dnf check-update lists pending updates and exits 100 when there are any (36 on the day), which is useful in scripts and alarming the first time. apt is not found, dpkg is not found. yum and dnf are both symlinks to the same dnf-3, so old guides still work.

The repos. /etc/yum.repos.d/*.repo, INI style, with one or more repos per file. On the day, Rocky enables baseos, appstream and extras. crb (CodeReady Builder, headers and build dependencies) is disabled on Rocky and enabled on Alma, which is the one functional difference I found between the two. Turn it on with dnf config-manager --set-enabled crb.

EPEL is the PPA replacement, sort of. Half the things an Ubuntu person installs by reflex are not in Rocky's repos at all. htop and fail2ban both answered No matching Packages to list on the fresh box, and pipx, uv, python3.13 and plain netcat turned out to live in the same place, which you reach with:

sudo dnf install epel-release

epel-release comes from the extras repo (dnf info epel-release says so), so that one command works on a fresh box, and after it dnf repolist shows a fourth repo, epel (a fifth if you enabled crb above). It is one repo run by the Fedora project, not a thousand personal ones. There is no add-apt-repository. For a vendor's own repo you download a .repo file into /etc/yum.repos.d/ or use dnf config-manager --add-repo <url>, which is what the Docker step below does.

Package names. The names you know are the Debian names, and dnf does not translate:

You type	Rocky has
`apache2`	`httpd`
`dnsutils`	`bind-utils`
`netcat`	`nmap-ncat` (or `netcat` from EPEL)
`build-essential`	`dnf group install "Development Tools"`
`python3-venv`	nothing to install, `python3 -m venv` works out of the box
`gdisk`	`gdisk`, but not on the image, so `sgdisk` is missing until you install it

When you do not know the name, dnf provides "*/dig" finds the package that owns a file, which is apt-file without the separate database.

History and undo. Here Rocky is closer to Ubuntu 26.04 than Debian is. dnf history list shows every transaction, dnf history info last shows what it changed, and dnf history undo last reverses it. I installed htop from EPEL, undid it, and htop was gone. This is the feature the apt history post is about, and dnf had it first.

Two settings in /etc/dnf/dnf.conf worth knowing about. best=True is the setting that makes dnf refuse an upgrade rather than quietly pick a lower version when the newest one has a dependency problem, where apt's habit is to hold the package back and carry on. installonly_limit=3 is the setting that keeps three kernels installed and removes the oldest when a fourth arrives, which is the kernel half of apt's autoremove, done for you. I read both from the file; a fresh box has one kernel and nothing to hold back, so neither behaviour was exercised.

Missing commands are just missing. Type tree on a fresh Rocky box and bash says command not found and nothing else. Ubuntu's command-not-found hook, which in an interactive shell tells you which package to install (the Debian post shows it), has no equivalent on this image.

Networking: NetworkManager and nmcli

No /etc/netplan, no netplan command, no /etc/network/interfaces.d, and on Rocky 10 no /etc/sysconfig/network-scripts either, which is where older RHEL guides will send you. The stack is:

	Rocky 10 (this image)	Ubuntu 26.04
Config	`/etc/NetworkManager/system-connections/*.nmconnection` (keyfiles)	`/etc/netplan/*.yaml`
Brought up by	`NetworkManager`	`systemd-networkd`
Tool	`nmcli`	`netplan`
DNS	NetworkManager writes `/etc/resolv.conf` directly	`systemd-resolved`, stub at `127.0.0.53`
`resolvectl`	binary present, answers `The name is not activatable`	works

cloud-init wrote one connection, cloud-init eth0, and nmcli con show lists it. A static address is two commands. The values below are placeholders; put in the address, prefix and gateway the box has now (nmcli dev show eth0 prints them), because con up applies the change immediately and the wrong gateway ends your SSH session:

sudo nmcli con mod "cloud-init eth0" ipv4.method manual \
  ipv4.addresses 10.0.0.5/24 ipv4.gateway 10.0.0.1 ipv4.dns "1.1.1.1 1.0.0.1"
sudo nmcli con up "cloud-init eth0"

I ran it on the Alma box with that box's current address and gateway, over SSH, and kept the session. nmcli con mod edits the keyfile (an [ipv4] block with address1=, gateway= and dns=) and con up applies it, so together they are netplan apply. /etc/resolv.conf was rewritten with the new servers on the way. The muscle memory that fails first is resolvectl status and ls /etc/netplan.

Firewall: firewalld, once you install it

Neither image starts with any firewall rules. Both have nft and iptables installed and an empty ruleset. From there the Ubuntu image has ufw installed (ufw status says inactive until you ufw enable), and the Rocky image has no front end at all: firewall-cmd not found, firewalld not installed. That surprised me, because firewalld is the RHEL firewall and I have never seen an installer build without it. This cloud image does not ship it, and dnf history says why: transaction 2 in the image's own build log is remove firewalld, run by whoever built the image. Somebody took it out on purpose.

sudo dnf install firewalld
sudo systemctl enable --now firewalld
sudo firewall-cmd --list-all

Out of the box the default zone is public, eth0 lands in it, and it allows ssh, dhcpv6-client and cockpit (port 9090, whether or not cockpit is installed). Opening things is --add-service or --add-port, and the flag every Ubuntu person forgets is --permanent, because a rule added without it lives in the running config only and goes away on the next reload or reboot (that is what the flag is for; I only tested the permanent path):

sudo firewall-cmd --add-service=http --permanent
sudo firewall-cmd --add-port=2222/tcp --permanent
sudo firewall-cmd --reload

Underneath it is nftables, table inet firewalld, 292 non-blank lines of ruleset for that one zone. ufw allow 22 and firewall-cmd --add-service=ssh --permanent both end up in the kernel's nftables, by different routes and in different tables. The ufw post has no direct equivalent here, and I have not written the firewalld one yet.

SELinux: permissive on this image, and why that is a trap

Ubuntu has AppArmor, loaded with 182 profiles on this image, and most server setups never trip over it. Rocky has SELinux, and most do. Except that on this cloud image you will not, for a while, and that is the finding the post is really about.

$ getenforce
Permissive
$ grep ^SELINUX= /etc/selinux/config
SELINUX=permissive

Permissive means SELinux is loaded, files and processes carry their labels, denials are written to the audit log, and nothing is enforced. Both the Rocky 10 and the AlmaLinux 10 image on Hetzner Cloud shipped that way on the day I ran this (22 September 2026). The installer's default is enforcing; I did not run an installer or Red Hat's own cloud image for this post, so check getenforce on yours rather than assuming either way. A surprising amount of "Rocky is different" on this box is really "this image turned the different part off."

Why it is a trap: everything you set up works. You change the SSH port, it listens. You put nginx in front of an app, it proxies. You get comfortable, or a hardening guide tells you to set SELINUX=enforcing and reboot, and two services you tested are now broken, with no change to their config. The next two sections reproduce both, and the switch between the two states is one command, which I ran between the permissive and enforcing halves of each:

sudo setenforce 1     # enforcing until reboot; getenforce confirms

To make it stick, set SELINUX=enforcing in /etc/selinux/config, which I did at the end, and rebooted to check that both fixes below survived it.

The tools you need, and their packages: getenforce, setenforce, getsebool, setsebool and restorecon are on the image. semanage, the one you need for ports and file contexts, is not; it lives in policycoreutils-python-utils. ausearch -m avc -ts recent reads the denials. That command is worth running on a permissive box before you switch, because every operation you have already exercised that would be blocked is in there; anything you have not exercised yet is not.

SSH: Bind to port 2222 failed: Permission denied

The Debian post found that changing the SSH port on Ubuntu needs daemon-reload and a restart of ssh.socket, because sshd is socket activated there. Rocky runs sshd.service the old way (sshd.socket exists and is disabled), so that trap is gone. SELinux supplies a new one: the label trap replaces the socket trap.

I added /etc/ssh/sshd_config.d/10-port.conf with Port 22 and Port 2222, checked it with sshd -t, and restarted:

Permissive:  0.0.0.0:22 and 0.0.0.0:2222 both listening after `systemctl restart sshd`
Enforcing:   only 0.0.0.0:22 listening, and in the journal:
             sshd: error: Bind to port 2222 on 0.0.0.0 failed: Permission denied.
             (and a second line for ::, one per address family)

Even in permissive mode the audit log already had the denial: comm="sshd" src=2222 ... tcontext=system_u:object_r:unreserved_port_t:s0. Port 2222 is an unreserved port and only ports labelled ssh_port_t are ones sshd is allowed to bind. The fix is one label:

sudo dnf install policycoreutils-python-utils
sudo semanage port -a -t ssh_port_t -p tcp 2222
sudo systemctl restart sshd

semanage port -l | grep ssh_port_t then shows tcp 2222, 22, and sshd binds both under enforcing. Add the firewalld rule from the previous section, and test the new port from a second session before you drop Port 22. Those are the three steps the RHEL port change guides give, and now you know which layer each one is for: semanage for SELinux, the drop-in for sshd, firewall-cmd for the packet filter.

One more sshd difference: this cloud image sets PasswordAuthentication no in 50-cloud-init.conf. The Ubuntu image leaves it yes. If you create a user on Rocky and cannot log in with a password, that is why, and it is the right default.

nginx: (13: Permission denied) while connecting to upstream

The second thing enforcing breaks is the most common thing people put on a server. nginx from AppStream, a stand in app on port 3000, and the smallest reverse proxy there is:

sudo setenforce 0     # the SSH section left the box enforcing; start this one permissive
sudo dnf install nginx
sudo mkdir -p /srv/app && echo hello-from-upstream | sudo tee /srv/app/index.html
(cd /srv/app && sudo setsid python3 -m http.server 3000 >/dev/null 2>&1 &)
echo 'server { listen 8080; location / { proxy_pass http://127.0.0.1:3000; } }' \
  | sudo tee /etc/nginx/conf.d/app.conf
sudo nginx -t && sudo systemctl enable --now nginx
curl 127.0.0.1:8080

Under permissive that last line prints hello-from-upstream. Under enforcing it returns 502. The error log says why, and it is not what it looks like:

connect() to 127.0.0.1:3000 failed (13: Permission denied) while connecting to upstream

That is not a file permission and no chmod will touch it. nginx runs in the httpd_t SELinux domain, and the policy does not let that domain connect out to port 3000 unless a boolean says so. The audit log has it as denied { name_connect } ... comm="nginx" dest=3000 scontext=system_u:system_r:httpd_t:s0. The switch is a boolean, and -P makes it survive a reboot:

getsebool httpd_can_network_connect          # --> off
sudo setsebool -P httpd_can_network_connect on
curl 127.0.0.1:8080                          # hello-from-upstream

On Ubuntu this configuration works first time, which is where the nginx reverse proxy post was run. On a permissive Rocky box it also works first time; I set it up under permissive, got hello-from-upstream, ran setenforce 1, and the next request was the 502. That post's recipe needs one extra step here that is invisible until enforcing is on, and if you only remember one boolean from SELinux, this is the one.

Logs, updates and time

Logs. rsyslog runs on both, but writes different files. Rocky has /var/log/messages and /var/log/secure; Ubuntu has /var/log/syslog and /var/log/auth.log. The grep sshd /var/log/auth.log reflex becomes grep sshd /var/log/secure, or journalctl -u sshd (the unit is sshd, not ssh). The journal was 16M on Rocky after boot against 8M on Ubuntu, and there is no /etc/systemd/journald.conf on Rocky at all: the stock file lives at /usr/lib/systemd/journald.conf, and your overrides go in /etc/systemd/journald.conf.d/, which is exactly what the journald retention post tells you to do anyway.

Updates. Ubuntu ships unattended-upgrades switched on. Rocky ships nothing. The equivalent is dnf-automatic, and it is two decisions, not one:

sudo dnf install dnf-automatic
sudo systemctl enable --now dnf-automatic.timer

The package ships four timers. dnf-automatic.timer reads /etc/dnf/automatic.conf, where the defaults are download_updates = yes and apply_updates = no, so out of the box it downloads updates and installs nothing. Either set apply_updates = yes in that file, or enable dnf-automatic-install.timer instead, whose service runs dnf-automatic --installupdates and installs regardless of the setting. There is no security only mode by default; upgrade_type = security in the same file gives you one. After updates, dnf needs-restarting -r says whether a reboot is due and dnf needs-restarting lists processes running old libraries (empty on a fresh box, which is what I got). There is no needrestart, so nothing prompts you after an upgrade the way Ubuntu's does.

Point releases. The second Rocky box in the screenshot run booted as 10.1, because the provider's image had not been rebuilt since January, and dnf upgrade took it to 10.2 in one go: 258 packages, a new kernel, reboot. That is the whole point release process, and it is the same command as a Tuesday's patching. It is also why that screenshot shows 10.1 before the upgrade.

Time. chrony on both, as noted. The only difference is which servers, and that is the image.

Python, Docker and disks: do the other posts hold

Python. Rocky 10 has Python 3.12, no pip, and no EXTERNALLY-MANAGED marker. That last part changes the pip post completely: after dnf install python3-pip, python3 -m pip install cowsay as root installed into /usr/local/lib/python3.12/site-packages with only the usual "running as root" warning. (Red Hat patches pip to keep root installs out of /usr/lib, where rpm's files live, which is a gentler fence than Debian's marker.) The lockout that post is about does not exist here. Use a venv anyway; python3 -m venv works with no extra package and the venv has pip inside it, where the same command on the Ubuntu box fails until python3.14-venv is installed. pipx, uv and python3.13 are all in EPEL.

Docker. Not in Rocky's repos; podman is, and is the distribution's answer. If you want Docker, Docker's own RHEL repo works on Rocky 10 unchanged:

sudo dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo
sudo dnf install docker-ce docker-ce-cli containerd.io
sudo systemctl enable --now docker
sudo docker run --rm hello-world

That gave Docker 29.8.1 and Hello from Docker!, with SELinux enforcing and firewalld running. Those four commands are the Ubuntu Docker post's install steps with the repo URL and dnf swapped in; I did not rerun the rest of that post here. podman was not installed on the image either, despite being the house container tool.

Disks. parted, partprobe, growpart, LVM, xfsprogs and e2fsprogs are all present. sgdisk is not, so the add a disk post needs dnf install gdisk first, or parted for the partitioning step. One image note: the root filesystem on this Rocky image is ext4, where the Rocky installer's default (not tested here) is XFS. If you are reading a RHEL guide that assumes xfs_growfs, check findmnt -no FSTYPE / first.

Rocky or Alma

The two boxes differed in the .repo file names, the vendor string in some package versions (.rocky.0.1 against .alma.1 on systemd and cloud-init, identical on the kernel), CRB enabled by default on Alma and disabled on Rocky, and nothing else my script could find. Same kernel build, same systemd, same package count, same cloud-init behaviour, same permissive SELinux. By their own descriptions, Rocky aims to be bug for bug identical to RHEL, and Alma aims for ABI compatibility and allows itself fixes and additions, of which the x86-64-v2 build is the visible one. For a homelab or a small fleet, pick by which community you would rather ask questions in. For an older CPU, Alma.

Which one I would pick

For a box that has to behave like RHEL, because the software vendor tests against it or the rest of the estate runs it, Rocky (or RHEL itself, if a checklist says the word), and I would set SELinux to enforcing on day one, before anything is installed, so every denial shows up while I am still watching. Ten years of patches is a real thing. Starting from a 6.12 kernel is the price.

For everything else I run, Ubuntu or Debian, because the guides assume it and I would rather have this year's kernel. The one habit that genuinely needs relearning on Rocky is not dnf, which you will pick up in an afternoon. It is treating SELinux as part of every service you deploy, and this image lets you skip that until it hurts.

The box at the end of the run, for the record: enforcing written to the config file, a reboot, and then sshd on both ports, firewalld running, and the proxy answering once the stand in app was started again, which the screenshot does not show. A bare setsid process is not a unit, so nothing brings it back after a reboot.

Gotchas I hit

SELinux is permissive on this cloud image, on Rocky and Alma both. Everything works until you set enforcing, then the SSH port and the reverse proxy break with no config change. Run ausearch -m avc -ts recent before you switch to see what will.
Bind to port 2222 on 0.0.0.0 failed: Permission denied from sshd, with an unreserved_port_t denial in ausearch, means the port has no ssh_port_t label. semanage port -a -t ssh_port_t -p tcp 2222, and semanage needs policycoreutils-python-utils.
(13: Permission denied) while connecting to upstream from nginx is not a file permission. Check ausearch for a name_connect denial from comm="nginx"; if it is there, it is httpd_can_network_connect, off by default. setsebool -P httpd_can_network_connect on.
firewalld is not on the cloud image. No rules are active either. dnf install firewalld, systemctl enable --now firewalld, and remember --permanent on every rule.
The pip lockout does not exist. No EXTERNALLY-MANAGED marker on Rocky 10, so pip install as root writes into the system site-packages. Nothing stops you. Use a venv.
dnf install htop fails on a fresh box. So does fail2ban, and pipx, uv and python3.13 are in the same repo. dnf install epel-release first.
The admin group is wheel. usermod -aG sudo keith fails on Rocky with group 'sudo' does not exist. -G wheel.
dnf check-update exits 100 when updates are available. Not an error.
Password logins are off on this image (50-cloud-init.conf). Create the user, then add a key.
dnf-automatic installs nothing by default. apply_updates = no. Set it, or enable the -install timer.
No /etc/systemd/journald.conf on Rocky. Drop-ins under journald.conf.d/ still work.
sgdisk is not installed. dnf install gdisk.
Docker is not in the repos, and podman is not on the image. Docker's RHEL repo works on Rocky 10.

Quick reference: Ubuntu habit to Rocky equivalent

Ubuntu habit	On this Rocky Linux 10 image
`sudo apt install x`	`sudo dnf install x`
`sudo apt update && sudo apt upgrade`	`sudo dnf upgrade` (dnf refreshes its own metadata when it is stale)
`apt search x` / `apt show x`	`dnf search x` / `dnf info x`
`apt-file search /bin/dig`	`dnf provides "*/dig"`
`sudo apt history-undo <ID>`	`sudo dnf history undo <ID>`, IDs from `dnf history list`
`sudo add-apt-repository ppa:x/y`	`sudo dnf install epel-release`, or a `.repo` file in `/etc/yum.repos.d/`, or `dnf config-manager --add-repo <url>`
`apt install build-essential`	`dnf group install "Development Tools"`
`sudo usermod -aG sudo keith`	`sudo usermod -aG wheel keith`
`sudo netplan apply`	`nmcli con mod "<name>" ...` then `nmcli con up "<name>"`; keyfiles in `/etc/NetworkManager/system-connections/`
`resolvectl status`	`cat /etc/resolv.conf`, NetworkManager writes it; `nmcli -g IP4.DNS dev show eth0`
`sudo ufw allow 22`	`sudo dnf install firewalld && sudo systemctl enable --now firewalld` first, then `sudo firewall-cmd --add-service=ssh --permanent && sudo firewall-cmd --reload`
`sudo ufw status`	`sudo firewall-cmd --list-all`
`aa-status`	`getenforce`, `sestatus`; denials in `ausearch -m avc -ts recent`
Change `Port` then `systemctl restart ssh`	`semanage port -a -t ssh_port_t -p tcp <port>` first (`semanage` is in `policycoreutils-python-utils`), open the port in firewalld, then `systemctl restart sshd` (no socket to restart)
`grep sshd /var/log/auth.log`	`grep sshd /var/log/secure` or `journalctl -u sshd`
`tail -f /var/log/syslog`	`tail -f /var/log/messages`
`unattended-upgrades` (on by default)	`dnf install dnf-automatic`, set `apply_updates = yes`, `systemctl enable --now dnf-automatic.timer`
`sudo needrestart -r l`	`dnf needs-restarting -r`, and `dnf needs-restarting` for the service list
`apt install python3-venv python3-pip`	`dnf install python3-pip`; venv already works; no `EXTERNALLY-MANAGED`
`apt install docker.io`	`dnf install podman`, or Docker's RHEL repo for `docker-ce`
`sudo sgdisk ...`	`dnf install gdisk` first, or use `parted`
`snap install x`	No `snap` command on this image. EPEL, the project's repo, or a container.
`sudo do-release-upgrade`	No equivalent command. Point releases (10.2 to 10.3) arrive through `dnf upgrade`; a new major version is a reinstall, or a migration tool such as Alma's ELevate, following that release's notes. Not tested here.

[ same systemd, same journal, and getenforce before you trust anything you just set up ]

Originally published at peculiarengineer.com.

Prometheus node exporter on Ubuntu 26.04, and it scrapes over the private network

Minor Keith — Mon, 21 Sep 2026 13:37:39 +0000

Every server I look after eventually gets the same question asked of it: how full is the disk, is it about to run out of memory, and does it need a reboot that nobody has got round to. Node exporter is the standard answer. It is one binary that exposes a few thousand numbers about the machine on port 9100, and Prometheus pulls them every fifteen seconds and keeps them. Ubuntu 26.04 has both in the archive, which is where this post starts and where most guides do not.

Most node exporter guides download a tarball from GitHub, make a user, and write a unit file. That works, and section 7 does it, but on 26.04 apt install prometheus-node-exporter gives you a unit, a dedicated user, a config file for the flags, and a set of extra collectors that report pending apt upgrades and whether the box wants a reboot. The reason for the post is the two things the package does that I did not want (it drags in twenty one other packages by default, and it listens on every interface), the two failure messages that look alike from the Prometheus side and mean different things, and the alert rules that turn "I have metrics" into "I get told".

The one thing to get straight: Prometheus scrapes. The exporter never sends anything anywhere. It sits there answering GET /metrics, and whoever can reach port 9100 can read every mount point, every network interface, the memory and load figures and, with the collectors package, the list of pending updates on the box. So the exporter's listen address and the firewall rule in front of it are part of the install, not an afterthought.

TL;DR. On the server: sudo apt update && sudo apt install prometheus-node-exporter (add --no-install-recommends on a cloud VM where you do not want the hardware collectors, and lose the apt and reboot metrics with them), set ARGS="--web.listen-address=10.20.1.10:9100" in /etc/default/prometheus-node-exporter to bind it to the private interface, sudo ufw allow from <prometheus ip> to any port 9100 proto tcp. On the Prometheus box: sudo apt install prometheus, add a job_name with the server's private address to /etc/prometheus/prometheus.yml, promtool check config, sudo systemctl reload prometheus, and confirm with up in the query API. Keep the recommends on real hardware; the apt_upgrades_pending and node_reboot_required metrics come from that package.

Prerequisites

Two Ubuntu 26.04 machines that can reach each other on a private network: the server you want metrics from, and the one that will run Prometheus. Mine are two cloud VMs with a private network between them, 10.20.1.10 for the server and 10.20.1.20 for Prometheus. A tailnet works the same way; use the 100.x addresses.
A sudo user on both.
Nothing else. Grafana is a separate afternoon and this post stops at Prometheus answering queries.

1. Install the exporter from the archive

One decision before the command. The package recommends a collectors package that brings twenty one more packages and the apt and reboot metrics with them; section 2 shows both sides. I installed the default first, because the rest of this post uses those metrics, and section 2 tells you how to go the small route instead.

sudo apt update
sudo apt install prometheus-node-exporter
prometheus-node-exporter --version

On 26.04 this is node_exporter, version 1.10.2, from universe, and the package does the parts the GitHub tarball leaves to you: a prometheus system user, a unit called prometheus-node-exporter.service that is enabled and running when apt returns, and /etc/default/prometheus-node-exporter with an ARGS="" line where every flag goes. The unit is short and worth reading once:

systemctl cat prometheus-node-exporter

[Service]
Restart=on-failure
User=prometheus
EnvironmentFile=/etc/default/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter $ARGS
ExecReload=/bin/kill -HUP $MAINPID

Check it is answering:

curl -s http://127.0.0.1:9100/metrics | grep -cE '^[a-z]'
curl -s http://127.0.0.1:9100/metrics | grep -E '^node_load1 |^node_memory_MemAvailable_bytes'

Just under two thousand metric lines on my box. node_load1, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes{mountpoint="/"} are the three you will query first.

The screenshots in this post come from a second pair of fresh 26.04 boxes, so timestamps and the odd number differ from the prose.

2. What the collectors package adds, and what it costs

prometheus-node-exporter recommends prometheus-node-exporter-collectors, and apt installs recommends by default. On my fresh VM that one recommendation turned a one package install into twenty two: ipmitool, smartmontools, nvme-cli, freeipmi, openipmi, moreutils, jq, the SNMP libraries and a Python Prometheus client. What you get for it is a set of systemd timers that run collector scripts (every fifteen minutes for the three that run on this VM) and drop their output into /var/lib/prometheus/node-exporter/ as .prom files, which the exporter's textfile collector picks up on the next scrape:

systemctl list-timers --all | grep prometheus
ls /var/lib/prometheus/node-exporter/

prometheus-node-exporter-apt.timer
prometheus-node-exporter-nvme.timer
prometheus-node-exporter-smartmon.timer
prometheus-node-exporter-ipmitool-sensor.timer     (inactive, no IPMI here)
prometheus-node-exporter-mellanox-hca-temp.timer   (inactive)

apt.prom  nvme.prom  smartmon.prom

The one that earns its keep on any Ubuntu box is apt.prom:

curl -s http://127.0.0.1:9100/metrics | grep -E '^(apt_|node_reboot_required)'

apt_autoremove_pending 0
apt_package_cache_timestamp_seconds 1.7899594604788072e+09
apt_upgrades_pending{arch="amd64",origin="Ubuntu:resolute/resolute-security,Ubuntu:resolute/resolute-updates"} 70
apt_upgrades_pending{arch="all",origin="Ubuntu:resolute/resolute-security,Ubuntu:resolute/resolute-updates"} 12
apt_upgrades_pending{arch="amd64",origin="Ubuntu:resolute/resolute-updates"} 8
apt_upgrades_pending{arch="all",origin="Ubuntu:resolute/resolute-updates"} 3
node_reboot_required 0

Ninety three pending upgrades on an image that was a few minutes old, eighty two of them from the security pocket, and a node_reboot_required gauge that goes to 1 on the apt collector's next run after /run/reboot-required appears, so within about fifteen minutes. Those two metrics are the reason I would keep the collectors package on real hardware, where smartmon.prom and nvme.prom also mean something.

On these VMs there is no SMART data and no IPMI, so the trade is ipmitool and friends for one useful script. If you decide that before installing, the smaller install is one flag:

sudo apt install --no-install-recommends prometheus-node-exporter

That installs exactly one package. No timers, an empty textfile directory, the same exporter. If you already did the default install in section 1, the flag does nothing on its own (apt reports nothing new to install); purge prometheus-node-exporter prometheus-node-exporter-collectors, run apt autoremove, and install again with the flag. If you want apt_upgrades_pending without the rest, the script is apt_info.py from the community collector repo; it needs python3-apt and python3-prometheus-client, a timer of your own, and section 4 shows where its output goes.

3. Bind it to the private interface and fence it with ufw

Out of the box the exporter listens on *:9100, every address the machine has. On a server with a public IP that is the whole machine's inventory on an unauthenticated port. Two layers fix it. First, tell the exporter to listen only on the private address:

sudo nano /etc/default/prometheus-node-exporter

ARGS="--web.listen-address=10.20.1.10:9100"

sudo systemctl restart prometheus-node-exporter
sleep 1; ss -tlnp | grep 9100

The socket line now shows 10.20.1.10:9100 instead of *:9100. The sleep is there because the exporter takes a moment to bind after the restart; without it the grep can come back empty and send you looking for a problem that does not exist.

curl http://127.0.0.1:9100/metrics now fails with connection refused. That is correct; the exporter is no longer on loopback. Use the private address for local checks from here on.

Second, allow only the Prometheus box through the firewall. If ufw is not on yet, let SSH in before enabling it:

sudo ufw allow OpenSSH
sudo ufw allow from 10.20.1.20 to any port 9100 proto tcp
sudo ufw enable
sudo ufw status

Belt and braces. If someone later changes ARGS back, or the private interface picks up a second address, the firewall still only admits one client. If your two machines talk over a tailnet instead, bind to the 100.x address and allow from the Prometheus node's 100.x address; nothing else changes.

4. A metric of your own with the textfile collector

The textfile collector is on by default in the Ubuntu package, pointed at /var/lib/prometheus/node-exporter/. Any *.prom file in there, in the exposition format, is served alongside the built in metrics. This is how a deploy script reports when it last ran:

printf '# HELP deploy_timestamp_seconds Unix time of the last deploy.\n# TYPE deploy_timestamp_seconds gauge\ndeploy_timestamp_seconds %s\n' "$(date +%s)" \
  | sudo tee /var/lib/prometheus/node-exporter/deploy.prom.tmp > /dev/null
sudo mv /var/lib/prometheus/node-exporter/deploy.prom.tmp /var/lib/prometheus/node-exporter/deploy.prom
curl -s http://10.20.1.10:9100/metrics | grep -E '^deploy_timestamp_seconds|^node_textfile_scrape_error'

deploy_timestamp_seconds 1.789959553e+09
node_textfile_scrape_error 0

Write to a temporary name and mv into place. The exporter can read the directory at any moment. I dropped a file of garbage in there to see what happens: node_textfile_scrape_error went to 1, the journal got failed to parse textfile data, that file's metrics were skipped, and every other file was still served. A half written file that happens to still parse is worse: it is served as if complete, with no error. Hence the rename. The mv is atomic on the same filesystem, so the exporter sees either the old file or the whole new one.

5. Prometheus from the archive, scraping it

On the second machine:

sudo apt update
sudo apt install prometheus
prometheus --version

That is Prometheus 2.53.5 from universe. Upstream is on 3.x, and 2.53 was upstream's 2.x long term support line, whose upstream support ended in July 2025; Ubuntu maintains the package on its own from here. For a home or small fleet server that is fine: the scrape config, the query language and the alert rules in this post are identical on both. If you want 3.x, the upstream tarball installs the same way as section 7 does for the exporter. Two things the package does that you should know about: it recommends prometheus-node-exporter, so the Prometheus box gets its own exporter and a job_name: node for localhost:9100 already in the config, and it ships /etc/default/prometheus with ARGS="" for flags such as --storage.tsdb.retention.time.

Before touching the config, confirm the network path works from this side:

curl -s --max-time 5 -o /dev/null -w '%{http_code}\n' http://10.20.1.10:9100/metrics

A 200 here means the bind address is right and the firewall lets this box through. Anything else, fix that first; Prometheus will only tell you the same thing more slowly. Then add the job to the end of /etc/prometheus/prometheus.yml:

  - job_name: 'nodex'
    static_configs:
      - targets: ['10.20.1.10:9100']
        labels:
          instance: 'nodex'

Setting instance yourself is optional. Without it, Prometheus uses the target address as the instance label, and every alert reads 10.20.1.10:9100 needs a reboot. With it, alerts and queries say the machine's name. Check and reload:

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

The package's unit has an ExecReload that sends HUP, so reload re reads the config without dropping the database. Give it a scrape interval and ask:

curl -s 'http://127.0.0.1:9090/api/v1/query?query=up'

Prometheus has a web UI on port 9090 that is easier to read, but the API is what works from a headless box over SSH, and the shape is simple: data.result[].metric is the label set and data.result[].value[1] is the number. up is 1 for every target Prometheus could scrape on its last attempt and 0 for every one it could not. Mine showed three targets at 1: Prometheus itself, its local exporter, and nodex. For anything longer than up, a small shell function saves quoting the URL by hand:

q() { curl -s http://127.0.0.1:9090/api/v1/query --data-urlencode "query=$1" \
  | python3 -c "import sys,json; [print(r['metric'].get('instance'), r['value'][1]) for r in json.load(sys.stdin)['data']['result']]"; }
q 'node_load1'

That prints one line per series, instance then value. From there, the numbers you came for, as expression and value:

node_load1{instance="nodex"}                                   0.23
node_filesystem_avail_bytes{mountpoint="/",instance="nodex"}
  / node_filesystem_size_bytes{mountpoint="/",instance="nodex"} * 100   91.4
deploy_timestamp_seconds{instance="nodex"}                     1789959553

The textfile metric from section 4 shows up like any other, which is the point of it.

The rules directory from section 6 was already in place when this shot was taken, which is why promtool counts a rule file here; on a first run it prints SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax.

6. Three alert rules worth having

Metrics nobody looks at are a database. I put these three rules in /etc/prometheus/rules/node.yml on the Prometheus VM (the directory does not exist after the install, so sudo mkdir -p /etc/prometheus/rules first):

groups:
  - name: node
    rules:
      - alert: RebootRequired
        expr: node_reboot_required == 1
        for: 1h
        labels: { severity: info }
        annotations: { summary: "{{ $labels.instance }} needs a reboot" }
      - alert: RootDiskLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 15m
        labels: { severity: warning }
        annotations: { summary: "{{ $labels.instance }} root filesystem under 10% free" }
      - alert: PendingSecurityUpdates
        expr: sum by (instance) (apt_upgrades_pending{origin=~".*security.*"}) > 0
        for: 1d
        labels: { severity: info }
        annotations: { summary: "{{ $labels.instance }} has security updates pending" }

Point the config at the directory by replacing the empty rule_files: line near the top of prometheus.yml, then check both files and reload:

rule_files:
  - /etc/prometheus/rules/*.yml

promtool check rules /etc/prometheus/rules/node.yml
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
curl -s http://127.0.0.1:9090/api/v1/rules

promtool check rules reported SUCCESS: 3 rules found, and within fifteen seconds of the reload, once the first evaluation had run, the rules API listed all three as health: ok (before that they show unknown). Within a minute PendingSecurityUpdates was pending for both of my boxes, which is what images with eighty two security updates waiting should produce, and it will fire after the for: 1d if nobody runs apt upgrade. RebootRequired and PendingSecurityUpdates both come from the apt collector, so they need section 2's collectors package (or apt_info.py) on each target. RootDiskLow uses the built in filesystem collector and works everywhere.

Where the alerts go (email, a chat webhook, a pager) is Alertmanager's job, which is prometheus-alertmanager in the same archive and a post of its own. Until then the /api/v1/alerts endpoint and the Alerts page on 9090 show what is pending and firing.

7. The upstream binary, when the package is too old

Back on the target for this section. The archive's 1.10.2 is from late 2025; upstream was at 1.12.1 when I wrote this. If you need a collector or a fix from a newer release, the upstream binary is a single file and the unit is the same shape as the packaged one:

V=1.12.1
curl -sLO https://github.com/prometheus/node_exporter/releases/download/v$V/node_exporter-$V.linux-amd64.tar.gz
tar xzf node_exporter-$V.linux-amd64.tar.gz
sudo install -m 0755 node_exporter-$V.linux-amd64/node_exporter /usr/local/bin/node_exporter
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node exporter (upstream binary)
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=10.20.1.10:9100 --collector.textfile.directory=/var/lib/prometheus/node-exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl disable --now prometheus-node-exporter
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sleep 1; curl -s http://10.20.1.10:9100/metrics | grep ^node_exporter_build_info

Both want port 9100, so the packaged one has to be stopped and disabled first, and --collector.textfile.directory has to be passed explicitly; the upstream binary has no default for it. On my box node_exporter_build_info switched from version="1.10.2" to version="1.12.1" and Prometheus did not notice anything but the label. The cost is that apt no longer upgrades it; that is now a curl you run yourself.

8. Reading a down target

The targets endpoint on the Prometheus box says why a scrape failed, and the two messages you will see most often look similar and are not. I broke the target on purpose both ways, from the target's own shell, and read the result from the Prometheus box:

curl -s http://127.0.0.1:9090/api/v1/targets | python3 -c \
  "import sys,json; [print(t['labels']['job'], t['health'], t['lastError']) for t in json.load(sys.stdin)['data']['activeTargets']]"

First, sudo systemctl stop prometheus-node-exporter on the target (or node_exporter if you switched in section 7). Within twenty seconds Prometheus reported:

nodex down Get "http://10.20.1.10:9100/metrics": dial tcp 10.20.1.10:9100: connect: connection refused

connection refused almost always means the packet arrived and nothing was listening: the exporter is stopped, crashed, or bound to a different address than the one in the job. (A firewall that rejects rather than drops produces it too.) Check the exporter's unit status and ss -tlnp | grep 9100 on the target.

Then I started it again and deleted the ufw rule instead: sudo ufw delete allow from 10.20.1.20 to any port 9100 proto tcp on the target. One thing to know before you try this: Prometheus keeps the HTTP connection to a target open between scrapes, and ufw does not cut an established connection, so up can stay at 1 for a while after the rule is gone. Restarting the exporter forces a new connection, and the next scrape reported:

nodex down Get "http://10.20.1.10:9100/metrics": context deadline exceeded

context deadline exceeded means the scrape hit its timeout, ten seconds by default, without an answer: a firewall dropping packets, a wrong IP, a route that does not exist, or an exporter that is stalled. The exporter can be perfectly healthy. Check the ufw rule on the target and curl --max-time 5 the address from the Prometheus box (without the limit, a dropped packet leaves curl sitting there for a couple of minutes). Put the rule back with the same ufw allow from section 3 and the target was up on the next scrape.

up going to 0 is the same for both, which is why it is worth having the lastError text to hand before deciding which machine to log into.

Gotchas I hit

apt install prometheus-node-exporter installed twenty two packages. --no-install-recommends gets you one, at the price of the apt and reboot metrics, and it only helps if you pass it on the first install.
The exporter listens on every interface by default. --web.listen-address in /etc/default/prometheus-node-exporter plus a ufw allow from rule. After that, curl 127.0.0.1:9100 is refused on purpose.
Deleting the ufw rule produced context deadline exceeded while the exporter was still running. That message is about the path, not the process. And it only shows up once the scrape opens a new connection; the established one keeps working until something closes it.
Two exporters cannot bind the same address and port; the second one fails with address already in use. Disable the packaged unit before enabling an upstream one on the same address.
The upstream binary has no default textfile directory. Pass --collector.textfile.directory or the .prom files silently stop appearing.
Write .prom files to a temp name and mv them, or node_textfile_scrape_error will occasionally be 1 and the metrics in that file will be missing for a scrape.
The archive's Prometheus is 2.53, a major version behind upstream. Everything in this post works on both, but read the upstream docs with that in mind.

Quick reference

# target
sudo apt update
sudo apt install prometheus-node-exporter          # add --no-install-recommends to skip the collectors package and everything it pulls in
sudo nano /etc/default/prometheus-node-exporter    # ARGS="--web.listen-address=10.20.1.10:9100"
sudo systemctl restart prometheus-node-exporter    # node_exporter instead, if you moved to the upstream unit in section 7
sudo ufw allow from 10.20.1.20 to any port 9100 proto tcp
curl -s http://10.20.1.10:9100/metrics | grep -E '^node_load1 |^node_reboot_required'

# prometheus box
sudo apt update && sudo apt install prometheus
sudo nano /etc/prometheus/prometheus.yml           # add the job under scrape_configs, rule_files: - /etc/prometheus/rules/*.yml
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/node.yml
sudo systemctl reload prometheus
curl -s 'http://127.0.0.1:9090/api/v1/query?query=up'
curl -s http://127.0.0.1:9090/api/v1/targets      # lastError says which machine to look at
curl -s --max-time 5 http://10.20.1.10:9100/metrics | head -n 3   # from here, is the path open
curl -s http://127.0.0.1:9090/api/v1/alerts

Ninety three pending updates on a box that was five minutes old. The exporter did not fix that, but at least now something is counting.

Originally published at peculiarengineer.com.

How To - Nginx reverse proxy on Ubuntu 26.04 without Docker

Minor Keith — Mon, 21 Sep 2026 13:35:36 +0000

The app is running. It listens on 127.0.0.1:8080, it has a systemd unit, and it works fine over an SSH tunnel. Now it needs a hostname, a certificate, and to stop being reachable only by yourself. The Caddy post covers this for containers. This is the version for a plain Ubuntu 26.04 box where the app is a process and the proxy is the nginx that has been in the archive forever.

Nginx needs a dozen lines for this. The reason it gets a post is what happened on my test box between the dozen lines and the working site: the default site kept answering just after a reload, the app saw 127.0.0.1 as every client's address until the forwarding headers went in, a websocket endpoint came back 502 until two more headers went in, and the HTTP/2 syntax every older guide still shows made nginx 1.28 print a deprecation warning. Each of those is one line once you know it, and none of them is in the nginx welcome page.

In this setup nginx is the only thing facing the internet. It accepts the connection on 80 and 443, and opens a second connection from itself to 127.0.0.1:8080. Your app sees nginx as the client. Every proxy_set_header line in this post exists to pass along something the app would otherwise never learn about the real request.

TL;DR. sudo apt install nginx, write /etc/nginx/sites-available/app.conf with a server_name and a location / that does proxy_pass http://127.0.0.1:8080; plus the Host, X-Real-IP, X-Forwarded-For and X-Forwarded-Proto headers, symlink it into sites-enabled, remove the default symlink, sudo nginx -t && sudo systemctl reload nginx. For HTTPS, sudo apt install certbot python3-certbot-nginx and sudo certbot --nginx -d app.example.com; on 26.04 that is certbot 4.0 from the archive and no snap is involved. Then add http2 on; next to the listen 443 ssl; line certbot wrote.

Prerequisites

Ubuntu 26.04 with a sudo user. Everything here ran on a fresh 26.04.1 server image.
An app already listening on a loopback port. Mine is a small Python HTTP server on 127.0.0.1:8080 that prints back the headers it receives, run from a systemd unit, because that makes section 4 visible. Yours is whatever you are fronting. If it listens on 0.0.0.0, change that to 127.0.0.1 first, or the proxy is decorative.
A DNS A record for the hostname pointing at the box, and ports 80 and 443 reachable from the internet. If a cloud firewall sits in front of the server, open both there too. The HTTP challenge certbot uses in section 6 needs 80 reachable from Let's Encrypt.

1. Install nginx and open the firewall

sudo apt update
sudo apt install nginx
nginx -v

On 26.04 that is nginx/1.28.3 (Ubuntu), from main, started and enabled by the package. It comes with HTTP/2, HTTP/3 and the real IP module compiled in, so nothing from a PPA is needed for anything in this post. (The stream module is a separate libnginx-mod-stream package; this post does not need it.) Browse to the box's IP and you get the "Welcome to nginx!" page, served by /etc/nginx/sites-enabled/default. That is the default site I remove in section 3.

The package also registers ufw application profiles:

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

Nginx Full is 80 and 443. There are also Nginx HTTP, Nginx HTTPS, and Nginx QUIC, which is 80 and 443 on TCP plus 443 on UDP. If ufw was already on, just add the second rule. If you have never touched ufw on this box, the ufw basics post explains why OpenSSH goes in before enable.

The screenshots in this post are from a second fresh 26.04 box, so the hostname in them is that box's reverse DNS name where the prose says app.example.com.

2. The site file

Nginx on Ubuntu reads /etc/nginx/nginx.conf, which includes every file in /etc/nginx/sites-enabled/. The convention is one file per site in sites-available and a symlink in sites-enabled to turn it on. Create the site with sudo nano /etc/nginx/sites-available/app.conf:

# /etc/nginx/sites-available/app.conf
server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable it, test the whole configuration, and reload:

sudo ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

nginx -t parses everything nginx would load and refuses if any file is broken, which is why it goes before every reload. A broken config with a plain restart leaves you with no web server at all. With a reload, nginx keeps serving the old config until the new one parses, so the worst case is the change not taking.

Now curl http://app.example.com/. Mine came back with the welcome page on the first try and the app on the second, and the next section is about why. Read it even if you got the app first time, because it ends with removing the default site, and every edit from here on gets the same sudo nginx -t && sudo systemctl reload nginx before you test it.

3. Why the default site keeps winning

Nginx picks a server block by matching the request's Host header against every server_name. When nothing matches, it uses the block marked default_server, and if none is marked, the first one it loaded. sites-enabled/default has both listen lines marked default_server, so any request for a name nginx does not recognise lands there:

curl -s -H 'Host: nothere.example.com' http://127.0.0.1/ | grep -o '<title>.*</title>'

<title>Welcome to nginx!</title>

That is expected and harmless. What bit me was the request for my real hostname also landing on the welcome page, and the cause was not nginx at all. It was me running curl in the same second as the reload. A reload is graceful: the master process parses the new config, starts new workers, and tells the old ones to finish their connections and exit. systemctl reload nginx returns as soon as the signal is sent, not when the swap is done, and on my box a request in that first second was still answered by an old worker running the old config. One second later the same request hit the app. If your first curl after a reload shows the previous config, run it again before you touch anything.

Then decide what should answer for hostnames you did not configure. I remove the default site, so that an unknown name falls through to the first block, which is the app:

sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

If you would rather unknown names get nothing, keep a catch all block with return 444;, which closes the connection without a response. Either is fine. Leaving the welcome page up is the choice that looks unfinished to anyone who scans your IP.

4. The headers your app actually needs

Take the four proxy_set_header lines out, test and reload, and look at what the app receives. Mine prints them:

Host: 127.0.0.1:8080
X-Forwarded-For: None
X-Forwarded-Proto: None
X-Real-IP: None

Without the headers, the app thinks its own name is 127.0.0.1:8080, so any absolute link it builds from Host points at loopback. (Nginx rewrites the Location header of a redirect back to the public name by default; it does nothing about links inside page bodies.) The app sees every visitor as 127.0.0.1, so rate limits and audit logs are wrong. And it has no way to know the request arrived over HTTPS once section 6 is done, which is where secure cookie and redirect loop problems come from. Put the four lines back and the same request shows, with my hostname and address in place of the placeholders:

Host: app.example.com
X-Forwarded-For: 203.0.113.7
X-Forwarded-Proto: http
X-Real-IP: 203.0.113.7

$proxy_add_x_forwarded_for appends the connecting address to whatever X-Forwarded-For already came in, so with a CDN in front the chain ends with the CDN's address and the app has to know how many hops to trust. X-Real-IP is always the address nginx saw. Which header your app trusts is the app's business. Web frameworks have a setting for "I am behind a proxy, believe these headers", and it is off by default because a client can send X-Forwarded-For: 198.51.100.66 itself; on a test box that forged value arrived at the app as 198.51.100.66, <the client address>. Trust the headers only on requests that came from 127.0.0.1, and only as many hops of the chain as you actually have.

5. Websockets

Anything that opens a websocket (a web terminal, a chat, most dashboards that update without a refresh) needs three more lines, and the failure without them is worth seeing once. My websocket app is a small echo server on 127.0.0.1:8081 (Python's websockets package, python3-websockets in the archive, and a client from the same package). I gave it a location /ws/ with proxy_pass http://127.0.0.1:8081/; and the Host header and nothing else. The client got server rejected WebSocket connection: HTTP 502 and nginx logged upstream prematurely closed connection while reading response header from upstream. The app was fine. Nginx had spoken HTTP/1.0 to it and dropped the Upgrade header, so the app closed a handshake it never saw. (If your websocket lives on the same port as the rest of the app, a bare location / sends the request to the app as an ordinary HTTP request and you get whatever the app answers with, a 200 or a 400, and the same fix applies.)

Two things fix it. A map outside the server block that turns the Connection header into upgrade only when the client asked for one, and a location for the websocket path that speaks HTTP/1.1 and passes both headers through:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /ws/ {
        proxy_pass http://127.0.0.1:8081/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
    }
}

The map has to sit at the http level, which in a sites-available file means above server, not inside it. Test, reload, and my echo server answered echo: ping through the proxy. If the app serves the websocket on the same port and path as everything else, put the three extra lines in location / instead of a second location; they do no harm to ordinary requests.

The trailing slash on proxy_pass http://127.0.0.1:8081/ matters. With it, nginx strips the /ws/ prefix and the app sees /. Without it, the app would receive /ws/ and would need a route for that path. Either way, be deliberate.

6. HTTPS with certbot from the archive

Certbot's own instructions still send Ubuntu users to the snap. On 26.04 the archive has certbot 4.0.0 and its nginx plugin in universe, current enough that I see no reason to add snapd for it:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d app.example.com

Certbot answers the HTTP challenge through the running nginx on port 80, gets the certificate, and rewrites app.conf: it turns your existing server block into the HTTPS one, with listen 443 ssl and the ssl_certificate lines pointing into /etc/letsencrypt/live/app.example.com/, and adds a separate port 80 block that redirects to it. Renewal is a systemd timer the package installs, certbot.timer, which runs at midnight and noon with a random delay of up to twelve hours; systemctl list-timers certbot.timer shows the next run, and sudo certbot renew --dry-run proves it can renew without waiting sixty days to find out.

I could not run this step on the box used for this post, because its provider firewall only opens SSH and Let's Encrypt has to reach port 80. The commands above are certbot's documented flow, and the package versions are what 26.04 installs. Everything below in this section I did run with a self signed certificate standing in for the real one, which you can do too if your 80 is closed for now:

sudo openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj /CN=app.example.com \
  -keyout /etc/ssl/private/app.key -out /etc/ssl/certs/app.crt

# a second server block in app.conf, the shape certbot produces
server {
    listen 443 ssl;
    http2 on;
    server_name app.example.com;
    ssl_certificate /etc/ssl/certs/app.crt;
    ssl_certificate_key /etc/ssl/private/app.key;
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Test with curl -k, since the certificate is self signed, and swap the two ssl_ lines for certbot's when the port opens.

Certbot writes listen 443 ssl; and stops there. To turn HTTP/2 on, add one line inside that block:

listen 443 ssl;
http2 on;

Do not write listen 443 ssl http2;. Older configs and most guides still have that syntax. On nginx 1.28 it parses, but nginx -t prints:

[warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead

With http2 on; in place, curl --http2 -I https://app.example.com/ comes back as HTTP/2 200 (add -k for the self signed stand in). Also add X-Forwarded-Proto $scheme to the 443 block if certbot's rewrite did not carry it over. In that block $scheme is https, which is what tells your app to issue secure cookies.

7. Reading the errors: 502, 413, and a reload that has not landed yet

502 Bad Gateway means nginx accepted the request and could not get an answer from the app. I stopped the app (sudo systemctl stop app, mine is a unit called app, then curl -i http://app.example.com/) and got exactly this, with the cause in the log:

connect() failed (111: Connection refused) while connecting to upstream

The line to read is the most recent one in /var/log/nginx/error.log that matches your request. Connection refused is the app not listening (crashed, wrong port, bound to a different address). upstream prematurely closed connection is the app closing mid request, which is the websocket case from section 5, or an app that crashed on that specific request. A 504 after sixty seconds with upstream timed out in the log is proxy_read_timeout, whose default is 60s. It is the wait between two reads from the app, not a total budget, so a slow first byte is what trips it. Raise it in the location if the app legitimately takes longer.

413 Request Entity Too Large is nginx refusing an upload before the app ever sees it. The default client_max_body_size is 1m. A 2 MB POST to my proxy (head -c 2000000 /dev/zero > 2mb.bin; curl -i -X POST --data-binary @2mb.bin http://app.example.com/) got the 413 page from nginx with nothing in the app's log. Set client_max_body_size 100m; (or whatever fits) in the server or location block.

The reload that has not landed is section 3: a request in the same second as systemctl reload nginx can still hit an old worker. Wait a second and try again before debugging a config that is actually fine.

Gotchas I hit

The default site answered for my hostname after a reload. It was the graceful reload still swapping workers, not a config problem. Run the curl twice.
Websockets came back 502 with upstream prematurely closed connection in the log. The map block plus proxy_http_version 1.1, Upgrade and Connection in the location fixed it.
nginx -t warned the "listen ... http2" directive is deprecated. Copying a 22.04 era config onto 26.04 does this. Use listen 443 ssl; and http2 on; on its own line.
A 2 MB upload got 413. client_max_body_size defaults to 1m and the app never sees the request.
Without proxy_set_header Host $host my test app received 127.0.0.1:8080 as the Host header. Without X-Forwarded-For it had no idea who the visitor was.
The map for websockets goes above the server block, not inside it. Inside, nginx -t fails with "map" directive is not allowed here.
Certbot 4.0 comes from apt on 26.04. If you already have certbot from snap on the box, pick one installation and remove the other, so only one of them manages renewal.

Quick reference

# on a fresh box, in this order
# install and firewall
sudo apt install nginx
sudo ufw allow 'Nginx Full'

# site
sudo nano /etc/nginx/sites-available/app.conf
sudo ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

# https
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d app.example.com
sudo certbot renew --dry-run

# when it breaks
sudo tail -n 20 /var/log/nginx/error.log
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/   # is the app even there

# app.conf: the map goes above the server block, and this location replaces the one in section 2
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# the location block, complete
location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    client_max_body_size 100m;    # uploads; default is 1m
    proxy_read_timeout 300s;      # only if the app has slow responses; default is 60s
}

The app still only listens on loopback. It just has a front door now.

Originally published at peculiarengineer.com.

Guide to DNS on Ubuntu 26.04

Minor Keith — Fri, 04 Sep 2026 02:36:06 +0000

For years my understanding of DNS on a Linux box was one file. /etc/resolv.conf has a nameserver line, that is where lookups go, done. Then at some point I opened it on an Ubuntu server and it said nameserver 127.0.0.53, which is a DNS server I never installed, running on an address I never configured, and I closed the file and got on with my day.

I have since written a Netplan post and three Tailscale posts that each wave at that address for a paragraph and move on. This is me finally sitting down to understand it. I took a fresh Ubuntu 26.04 server and followed one name lookup from the application all the way down to the server that answered, and then broke it in the ways I have seen it broken. Everything below is what the box printed.

TL;DR. Two layers. First the C library reads /etc/nsswitch.conf, which says hosts: files dns: check /etc/hosts, then ask DNS. "Ask DNS" means the nameserver in /etc/resolv.conf, which is a symlink to a file systemd-resolved writes, and it says 127.0.0.53. That is systemd-resolved itself, a local caching resolver, and it forwards to the real upstream servers it learned per network interface from DHCP or Netplan. resolvectl status shows those servers, resolvectl query shows which one answered and whether it came from cache. dig skips the first layer and talks to the stub directly, which is why dig and ping can disagree. Change upstreams in Netplan, not by editing resolv.conf, which systemd-resolved rewrites. Split DNS is one server per domain per interface, and it is how Tailscale's MagicDNS works.

The one idea worth holding onto

There are two layers, and most confusion about DNS on Linux comes from not knowing which one you are looking at.

The first layer is not DNS at all. It is the C library's name lookup, the code linked into every program on the box that turns a name into an address, and it consults a list of sources in order. On Ubuntu that list is /etc/hosts first, then DNS. Every normal program, ping, curl, ssh, your Python script, goes through this layer, and a name that /etc/hosts answers never becomes a DNS query.

The second layer is the resolver. On Ubuntu 26.04 that is systemd-resolved, listening on 127.0.0.53. It caches, it knows which upstream servers belong to which network interface, and it forwards your question to the right one. dig, host and nslookup skip the first layer entirely and talk to this one.

Hold onto that and the rest is detail. When something resolves in the browser but not in dig, or in dig but not in ping, the two layers disagree, and Section 5 is about exactly that.

Everything below was run as root on Ubuntu 26.04.1 with systemd 259. The box is a Hetzner server, which matters only because it ends up with four DNS servers from two different sources, and that turns out to be one too many for one of the files.

1. Start where the applications start

grep '^hosts:' /etc/nsswitch.conf

hosts:          files dns

That is the entire first layer on a 26.04 server. files is /etc/hosts. dns is the resolver library, which reads /etc/resolv.conf. Desktop installs usually carry an extra mdns4_minimal entry in there for .local names. The server image does not.

The tool for asking this layer a question directly, without a network round trip you did not ask for, is getent:

getent hosts example.com

2606:4700:10::6814:179a example.com
2606:4700:10::ac42:93f3 example.com

getent hosts does exactly what ping does to turn a name into an address, and nothing else. When a program cannot resolve a name, this is the first command to run, because it answers "is it the program or is it the box".

2. resolv.conf is a symlink, and there are two of them

ls -la /etc/resolv.conf
grep -v '^#' /etc/resolv.conf

lrwxrwxrwx 1 root root 39 Apr 20 18:07 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search .

So /etc/resolv.conf is not a file you own. It points at a file systemd-resolved regenerates, and that file names one server: the stub on 127.0.0.53. A stub resolver is one that owns no DNS data of its own. It takes questions from local programs and forwards them to servers that do. The options line turns on EDNS0, the extension that lets answers be bigger than the original 512 byte limit, and trust-ad, which lets the library trust a DNSSEC flag the stub sets. Both can stay unread until Section 11. The comment block at the top of it, which I cut here, says all of this in so many words, and ends with the line that matters: run resolvectl status to see the real upstream servers.

There is a second generated file next to it, and knowing it exists solves a whole category of problems later:

grep -v '^#' /run/systemd/resolve/resolv.conf

nameserver 2a01:4ff:ff00::add:2
nameserver 2a01:4ff:ff00::add:1
nameserver 185.12.64.1
# Too many DNS servers configured, the following entries may be ignored.
nameserver 185.12.64.2
search .

Same format, but listing the actual upstream servers instead of the stub. This is the file you point at when a program cannot use 127.0.0.53, and Docker is the usual example, in Section 10. That warning comment is real and worth knowing about: the classic C library resolver only reads the first three nameserver lines, so systemd-resolved writes the fourth one with a note that it will probably be ignored. The stub has no such limit, which is one small reason to leave the symlink pointing where it is.

3. Who is 127.0.0.53

ss -ulnp | grep ':53 '

UNCONN 0 0    127.0.0.54:53     0.0.0.0:*  users:(("systemd-resolve",pid=485,fd=18))
UNCONN 0 0 127.0.0.53%lo:53     0.0.0.0:*  users:(("systemd-resolve",pid=485,fd=16))

systemd-resolved, bound to loopback only, so nothing outside the box can ask it anything. It also listens on 127.0.0.54, which the documentation describes as a second stub with different forwarding rules. Nothing in resolv.conf points at it and I did not test it. .53 is the one this post is about.

Now the command the comment told you to run:

resolvectl status

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2a01:4ff:ff00::add:2
       DNS Servers: 2a01:4ff:ff00::add:2 2a01:4ff:ff00::add:1 185.12.64.1
                    185.12.64.2
     Default Route: yes

Read it top to bottom. Global is settings that apply everywhere, and on a fresh box there is nothing in it except a list of protocols that are all switched off, which is the correct default for a server. Then one block per network interface. eth0 knows four servers, is currently using the first, and has Default Route: yes, meaning any name that no other interface claims goes here. Section 7 is about interfaces that claim names.

Where did those four servers come from? Not from a file you edited. The two IPv4 addresses arrived with the DHCPv4 lease, and the cloud image's Netplan file supplies the IPv6 pair statically:

grep -A3 nameservers /etc/netplan/50-cloud-init.yaml

      nameservers:
        addresses:
        - 2a01:4ff:ff00::add:2
        - 2a01:4ff:ff00::add:1

That is the shape to remember. Per interface, DNS servers come in from whatever configured the interface, DHCP or Netplan, systemd-networkd hands them to systemd-resolved, and systemd-resolved writes the two resolv.conf files to match. You never edit the files. You edit what feeds them.

4. Watch a lookup happen

resolvectl query asks the stub the same way an application would, and then tells you what it did:

resolvectl flush-caches
resolvectl query example.com

example.com: 2606:4700:10::ac42:93f3                        -- link: eth0
             2606:4700:10::6814:179a                        -- link: eth0
             104.20.23.154                                  -- link: eth0
             172.66.147.243                                 -- link: eth0

-- Information acquired via protocol DNS in 5.9ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

Which interface it went out of, how long it took, and where the answer came from. Ask again:

-- Information acquired via protocol DNS in 1.0ms.
-- Data from: cache

That cache is the main practical thing the stub gives you over a bare resolv.conf pointing at a remote server. Every repeated lookup on the box, and a busy box does thousands, is answered locally. You can see what is in it:

resolvectl show-cache

Scope protocol=dns ifindex=2 ifname=eth0 DNSSEC=no DNSOverTLS=no
example.com IN A 172.66.147.243
example.com IN A 104.20.23.154
example.com IN AAAA 2606:4700:10::6814:179a
example.com IN AAAA 2606:4700:10::ac42:93f3

and how it is doing overall with resolvectl statistics, which counts transactions, cache hits and misses, and timeouts. Mine showed 10 hits to 30 misses a few minutes into a fresh boot, which is what you expect before the cache warms up.

The tool I wish I had known about earlier is resolvectl monitor. It prints every question the stub receives and every answer it gives, live. Leave it running in one terminal and do anything in another:

→ Q: peculiarengineer.com IN AAAA
← S: success
← A: peculiarengineer.com IN AAAA 2606:4700:3031::6815:467a
← A: peculiarengineer.com IN AAAA 2606:4700:3036::ac43:df66

When you are not sure whether a program is even asking DNS, or what name it is asking for, this answers it in one line.

dig is the same lookup seen from the other side. It reads resolv.conf, so by default it asks the stub too, and it shows you the raw answer including the TTL, the lifetime in seconds the upstream stamped on the answer, counting down as the cache ages:

dig example.com +noall +answer

example.com.        155 IN  A   104.20.23.154

dig @1.1.1.1 example.com skips the stub and asks a server of your choosing, which is how you find out whether a wrong answer is coming from your box or from upstream. dig is preinstalled on 26.04 server, which it was not always.

5. Why dig and ping disagree

Put a name in /etc/hosts and ask each layer about it:

echo "10.99.99.9 mybox" >> /etc/hosts
getent hosts mybox
dig mybox +noall +answer
resolvectl query mybox

10.99.99.9      mybox
mybox.          0   IN  A   10.99.99.9
mybox: 10.99.99.9
-- Data from: synthetic

I expected dig to fail here, because dig skips /etc/hosts. It did not, because on Ubuntu the stub it asks is systemd-resolved, and systemd-resolved reads /etc/hosts itself and serves those entries with a TTL of zero. resolvectl is honest about it: Data from: synthetic, meaning the resolver made the answer up from local knowledge and never sent a packet. So on 26.04 the two layers agree about /etc/hosts. The disagreement is elsewhere.

Ask for a .local name:

resolvectl query printer.local

printer.local: resolve call failed: No appropriate name servers or networks for name found

.local is reserved for multicast DNS, and mDNS is off on a server (-mDNS in the status output). systemd-resolved refuses to send a .local name to a real DNS server, by design, so it fails instantly with that message. A desktop with mdns4_minimal in nsswitch.conf would have answered from the first layer without DNS being involved, which is why the printer resolves on your laptop and not on the server.

And the box's own name:

getent hosts "$(hostname)"

127.0.1.1       dns-post dns-post

That is an /etc/hosts line the installer wrote, pointing the hostname at 127.0.1.1, the second loopback address. It is why ping $(hostname) works on a box with no DNS at all, and why a service that binds to "the hostname" sometimes ends up listening only on loopback.

6. Changing the upstream, and what survives a reboot

There are two ways, and the difference is what happens tomorrow.

The quick way is resolvectl, and it takes effect immediately:

resolvectl dns eth0 1.1.1.1 9.9.9.9
resolvectl dns eth0

Link 2 (eth0): 1.1.1.1 9.9.9.9

I wanted to know exactly how temporary that is, so I tried to knock it out. netplan apply did not touch it. networkctl reconfigure eth0 did not touch it either, which surprised me. A reboot did: the box came back with Hetzner's four servers and no memory of mine. So a resolvectl dns change lasts until the next boot, and no less, which makes it fine for "try this server for an hour" and wrong for anything else.

The permanent way is Netplan. Add a file, and let it feed the chain from the top:

# /etc/netplan/60-dns.yaml
network:
  version: 2
  ethernets:
    eth0:
      nameservers:
        addresses: [1.1.1.1, 9.9.9.9]

chmod 600 /etc/netplan/60-dns.yaml
netplan apply
resolvectl status eth0 | grep -i 'DNS Server'

Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1 9.9.9.9

And now look at the two files from Section 2. /etc/resolv.conf still says nameserver 127.0.0.53, unchanged, because the stub did not move. /run/systemd/resolve/resolv.conf now lists 1.1.1.1 and 9.9.9.9, because the upstreams did. That is the whole model working as intended: you changed the input, both outputs followed, and no program on the box noticed anything.

Netplan merges every file in /etc/netplan/ by filename order, so a 60- file layers over the cloud image's 50-cloud-init.yaml without you editing a file that cloud-init might regenerate. Note what the merge did to the list, though. eth0 went from four servers to exactly two, so the nameservers list is replaced, not appended, and the IPv6 pair from the cloud image is gone. If you want them, put them in your file too. The Netplan post goes into that ordering.

7. Split DNS: a different server for one domain

This is the part that made the design click for me, because it is the thing a single resolv.conf fundamentally cannot do.

The situation: your normal DNS is fine for the internet, but names under corp.example only exist on a private server. In the old world you either pointed the whole box at the private server and hoped it forwarded everything else, or you gave up. With per interface servers and routing domains, you attach the private server to an interface and tell systemd-resolved which names belong to it. A routing domain is written with a leading ~: ~corp.example means "send questions about this domain to this interface's servers", and ~. means "send everything else here", which is what Default Route: yes in Section 3 was showing you.

To try it without a corporate network I ran a tiny DNS server on the box that invents answers for one domain. dnsmasq will do that in four lines of config. Installing it produced the first lesson for free:

dnsmasq: failed to create listening socket for port 53: Address already in use

The stub already has port 53 on loopback, and dnsmasq's default is to bind everything. Pin it to a different loopback address and give it the fake zone:

# /etc/dnsmasq.d/corp.conf
listen-address=127.0.0.2
bind-interfaces
port=53
no-resolv
address=/corp.example/10.7.7.7
address=/git.corp.example/10.7.7.10

Then an interface to hang it on. A dummy interface is fine for the demonstration, a VPN interface is what you would have in real life:

ip link add corp0 type dummy && ip link set corp0 up
ip addr add 10.7.7.1/24 dev corp0
resolvectl dns corp0 127.0.0.2
resolvectl domain corp0 '~corp.example'
resolvectl status corp0

Link 3 (corp0)
    Current Scopes: DNS
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 127.0.0.2
        DNS Domain: ~corp.example
     Default Route: no

Default Route: no. This interface's server only ever sees names under corp.example. Now query one of each:

resolvectl query git.corp.example
resolvectl query example.com

git.corp.example: 10.7.7.10
-- Information acquired via protocol DNS in 3.1ms.

example.com: 104.20.23.154 ...                              -- link: eth0
-- Information acquired via protocol DNS in 18.1ms.

The private name went to 127.0.0.2 on corp0. The public name went out eth0 as before. Neither server saw the other's traffic. And /etc/resolv.conf still says nameserver 127.0.0.53 and nothing else, so every program on the box gets this behaviour without knowing it exists. dig git.corp.example returns 10.7.7.10 too, through the stub.

resolvectl domain

Link 2 (eth0): ~.
Link 3 (corp0): ~corp.example

That two line table is the routing policy for every name on the machine. If you have ever run resolvectl status on a box with Tailscale and wondered what the tailscale0 block with its own DNS server and a ~ domain was for, this is it. MagicDNS is exactly this shape: Tailscale attaches its resolver to its interface, claims the tailnet's domain, and leaves ~. alone.

8. Search domains and single labels

A search domain is what turns git into git.corp.example so you can type the short name. It is per interface, like everything else here, and it is wired into both layers. Using the corp0 interface from Section 7, with the domain written without the ~ this time:

resolvectl domain corp0 corp.example
grep '^search' /etc/resolv.conf
getent hosts git
resolvectl query git

search corp.example
10.7.7.10       git.corp.example
git: 10.7.7.10
     (git.corp.example)

systemd-resolved wrote the domain into the search line of resolv.conf, so the C library appends it, and it also applies it itself when asked directly, which is why resolvectl query git shows the name it actually looked up in brackets. dig needs to be told: dig +search git works, plain dig git gets status: REFUSED from the stub.

That refusal is deliberate and it will bite you once. systemd-resolved does not send single label names, a bare git or mybox with no dots, to a DNS server at all unless a search domain completes them. Without one you get:

mybox: resolve call failed: No appropriate name servers or networks for name found

The setting behind it is ResolveUnicastSingleLabel=no in resolved.conf, and the default is right: a single label leaking to the internet is how a typo becomes a query to somebody else's server.

The ~ form from Section 7 is the other kind. ~corp.example routes queries for the domain to this interface's servers and nothing more: it does not complete short names. Drop the ~ and you get both routing and completion, which is what this section just used.

9. DNS over TLS, and the setting that takes the box offline

Everything so far went over plain UDP. systemd-resolved can encrypt the leg to the upstream, and the configuration lives in a drop-in, a small file under resolved.conf.d that overrides the main resolved.conf without editing it:

# /etc/systemd/resolved.conf.d/dot.conf
[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com
DNSOverTLS=yes

systemctl restart systemd-resolved
resolvectl query example.com | tail -2
ss -tnp | grep ':853'

-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes
ESTAB 0 0 178.105.172.189:53968 1.1.1.1:853 users:(("systemd-resolve",pid=2934,fd=24))

An open TLS connection to port 853, and resolvectl reports the encrypted transport. The #hostname after each address is the certificate name to verify, and you want it there.

Here is the trap, and I walked into it on purpose. DNSOverTLS=yes is strict mode, and it applies to every server on every interface, including the ones DHCP handed you. I removed the DNS= line so the only servers left were Hetzner's, which do not speak TLS, and asked for a name:

(nothing. the twenty second timeout I had wrapped it in gave up, exit 124)

DNS on the box was simply gone. resolvectl status eth0 showed +DNSOverTLS against servers that would never answer on 853. So strict mode is only correct when you have also set the upstreams to servers you know support it, in the same drop-in. DNSOverTLS=opportunistic is the softer setting, it tries TLS and falls back to plain, and against the same Hetzner servers it worked immediately, unencrypted, with no connection to 853 at all. Opportunistic is honest about what it gives you, which is not much. Either set strict with known servers, or leave it off.

10. Three ways it gets broken

Someone replaces the symlink. The most common one, usually from a guide written for a distribution without systemd-resolved, or a VPN client from years ago. I did it the way they do it:

cp --remove-destination /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now /etc/resolv.conf is a regular file listing the upstreams directly. Everything still resolves, so nobody notices. What you lost: resolvectl statistics shows a cache size of zero and stays there, resolvectl monitor sees nothing when programs look names up, split DNS and search domains stop applying to anything except resolvectl itself, and the next DHCP change never reaches the file. Put it back:

ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Docker containers. A container cannot use 127.0.0.53, because inside the container loopback is the container's own. So Docker does not copy /etc/resolv.conf; it copies the other file, the one with the real upstreams:

docker run --rm alpine:3.20 cat /etc/resolv.conf

nameserver 1.1.1.1
nameserver 9.9.9.9
search .

Those are the servers from Section 6, taken from /run/systemd/resolve/resolv.conf, with no cache and no split DNS. On a user defined network Docker instead gives the container nameserver 127.0.0.11, its own embedded resolver, which forwards to the same upstreams and adds container name resolution on top. Either way, a private name that resolves on the host through a routing domain will not resolve inside a container unless you pass --dns or configure the daemon.

Something else wants port 53. dnsmasq did it in Section 7, Pi-hole does it, so does any local DNS server installed with defaults. The failure is Address already in use and the service does not start. Either bind the new server somewhere other than 127.0.0.53, as I did, or if the new server is meant to replace the stub entirely, stop and disable systemd-resolved and manage resolv.conf yourself. Do one or the other. Running both on the same address is not a state the box can be in.

11. Debugging it

The commands, in the order I now reach for them:

getent hosts NAME
resolvectl query NAME
resolvectl status
resolvectl monitor
dig @UPSTREAM NAME

getent for what a program sees, query for which link and source answered, status for who owns the default route, monitor when you are not sure a program is asking at all, and dig at an upstream to take the box out of the picture.

When those do not explain it, turn the resolver's own logging up for a minute:

resolvectl log-level debug
resolvectl query example.com >/dev/null
journalctl -u systemd-resolved --since -1min | grep example.com
resolvectl log-level info

Cache miss for example.com IN AAAA
Firing regular transaction 19300 for <example.com IN AAAA> scope dns on eth0/* (validate=yes).
Cache miss for example.com IN A

That tells you the exact scope and interface a query was sent on, which is the question split DNS problems come down to. And resolvectl show-server-state shows what the resolver has learned about each upstream, such as whether it supports EDNS0 and how many UDP and TCP attempts have failed against it. A server with Failed UDP attempts climbing is a server on the way out, and resolvectl reset-server-features makes systemd-resolved forget and re-probe.

A name that does not exist looks like this, and exits 1, which is useful in scripts:

does-not-exist.peculiarengineer.com: Name 'does-not-exist.peculiarengineer.com' not found

Gotchas worth knowing

/etc/resolv.conf is output, not input. Edit what feeds it: Netplan for permanent, resolvectl dns for the rest of today. If it is a regular file instead of a symlink, somebody broke it, and ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf fixes it.

A resolvectl dns change survives netplan apply and networkctl reconfigure and dies at reboot. Longer lived than I assumed and shorter than you need.

Four DHCP servers is one too many for the classic resolver. Only the stub reads all four. Anything reading /run/systemd/resolve/resolv.conf directly, Docker included, ignores the fourth, and the file says so in a comment.

Single label names never leave the box. No search domain, no query. The error is No appropriate name servers or networks for name found, and it is not a network problem.

.local fails on a server on purpose. mDNS is off and systemd-resolved will not send .local to unicast DNS. Turn on mDNS or use a real name.

DNSOverTLS=yes with DHCP servers is an outage. Strict mode applies to every server on every link. Set DNS= to known TLS servers in the same drop-in, or use opportunistic and accept that it will mostly be plaintext.

dig sees /etc/hosts on Ubuntu. Because the stub it talks to reads the file. Data from: synthetic in resolvectl query is how you tell.

Docker gets the upstreams, not the stub. No cache, no split DNS, no search domains inside containers unless you configure them. Private names that work on the host will not work in a container by default.

Port 53 is taken. By the stub, on 127.0.0.53 and 127.0.0.54. A local DNS server has to bind elsewhere or replace systemd-resolved outright.

Quick reference

Task	Command
Resolve a name the way programs do	`getent hosts NAME`
Resolve with details: link, cache, timing	`resolvectl query NAME`
Servers and routing per interface	`resolvectl status`
Just the servers	`resolvectl dns`
Just the domains	`resolvectl domain`
Watch every query live	`resolvectl monitor`
What is cached	`resolvectl show-cache`
Hit and miss counts	`resolvectl statistics`
Empty the cache	`resolvectl flush-caches`
Change servers until reboot	`resolvectl dns eth0 1.1.1.1 9.9.9.9`
Change servers permanently	`nameservers:` in a `/etc/netplan/*.yaml`, then `netplan apply`
Add a search domain until reboot	`resolvectl domain eth0 corp.example`
Route one domain to one link	`resolvectl domain IFACE '~corp.example'` plus `resolvectl dns IFACE SERVER`
Ask an upstream directly	`dig @1.1.1.1 NAME`
Per upstream health	`resolvectl show-server-state`
Verbose resolver log	`resolvectl log-level debug`, then `journalctl -u systemd-resolved -f`
Restore the symlink	`ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf`
The upstream list for things that cannot use the stub	`/run/systemd/resolve/resolv.conf`

The next time /etc/resolv.conf says 127.0.0.53, that is not a mystery, it is a table of contents. resolvectl status is the book. [ 127.0.0.53 ✓ ]

Originally published at peculiarengineer.com.

Systemd units and timers on Ubuntu 26.04, helpful guide

Minor Keith — Fri, 04 Sep 2026 01:49:06 +0000

I have now written "run it from a systemd timer" in three posts, pointed at a timer each time, and never once explained what one is. The restic post has a timer in it that I asked you to copy on faith. This is the post those should have linked to.

It is written for someone who has never opened a unit file. If you have used cron, you already know the job: run this script at that time. What is different is everything around the job. Where the output goes, what happens when the box was off at three in the morning, what happens when the script fails, and why it fires a few seconds late. I built one job on a fresh Ubuntu 26.04 server and then spent an evening measuring those things instead of assuming them.

TL;DR. Two files in /etc/systemd/system/. job.service says what to run (Type=oneshot, ExecStart=/absolute/path). job.timer says when (OnCalendar=), and it starts the service with the same name. systemctl daemon-reload after every edit, then systemctl enable --now job.timer, the timer, never the service. Add Persistent=true so a run missed while the box was off happens at boot, and AccuracySec=1s if "03:30" has to mean 03:30. Test with systemctl start job.service instead of waiting. Your script runs with no shell, no HOME, a five entry PATH, and every % and $ has a meaning to systemd before your script ever sees it.

The one idea worth holding onto

systemd manages things called units. A service unit is a thing to run. A timer unit is a clock that starts some other unit when it goes off. That is the whole model, and cron has no equivalent to it: in cron the schedule and the command live on one line, and the moment the command runs, cron forgets about it.

Here they are two files with the same name and different extensions. backup.service is what. backup.timer is when. The timer activates the service by name, and because the service is a real unit in its own right, you can start it by hand, read its last exit code, and see its output, all without the timer being involved. Most of what makes timers better than cron falls out of that split.

Everything below was run on Ubuntu 26.04.1 with systemd 259, as root unless I say otherwise. The box is on UTC, which matters in Section 5.

What is already running on your box

You do not have to write a timer to see one. A fresh 26.04 server ships with eighteen:

systemctl list-timers --all

NEXT                                LEFT LAST PASSED UNIT                           ACTIVATES
Thu 2026-09-03 22:50:00 UTC     2min 57s -         - sysstat-collect.timer          sysstat-collect.service
Thu 2026-09-03 23:01:21 UTC        14min -         - systemd-tmpfiles-clean.timer   systemd-tmpfiles-clean.service
Fri 2026-09-04 00:00:00 UTC     1h 12min -         - dpkg-db-backup.timer           dpkg-db-backup.service
Fri 2026-09-04 00:12:59 UTC     1h 25min -         - logrotate.timer                logrotate.service
Fri 2026-09-04 06:41:12 UTC           7h -         - apt-daily-upgrade.timer        apt-daily-upgrade.service
Fri 2026-09-04 10:09:56 UTC          11h -         - apt-daily.timer                apt-daily.service
Mon 2026-09-07 01:02:03 UTC       3 days -         - fstrim.timer                   fstrim.service
-                                      - -         - snapd.snap-repair.timer        snapd.snap-repair.service
...
18 timers listed.

Log rotation, the apt update check, filesystem trimming, the man database: on Ubuntu none of these are cron jobs any more. Each is a timer starting a service. The - rows at the bottom are timers that are loaded but will never fire, often because a condition in the unit is false on this machine.

The fastest way to learn the format is to read one Ubuntu wrote:

systemctl cat fstrim.timer

# /usr/lib/systemd/system/fstrim.timer
[Unit]
Description=Discard unused filesystem blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container
ConditionPathExists=!/etc/initrd-release

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min

[Install]
WantedBy=timers.target

Every line in that [Timer] block gets its own section below, because Ubuntu's choices there are good ones and it is worth knowing why. The two Condition lines are why this timer does nothing inside a container: a condition that is false makes the unit load and then quietly never run. Note the path too. Ubuntu's units live in /usr/lib/systemd/system/. Yours go in /etc/systemd/system/, and if both places hold a file with the same name, /etc wins. Do not edit anything under /usr/lib; Section 14 shows how to change a vendor timer without touching it.

1. The job

Start with something that prints what it can see, because what it can see is the surprising part.

cat > /usr/local/bin/hello-timer.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
echo "hello from $(hostname) at $(date '+%F %T'), pid $$, user $(id -un)"
echo "PATH is: $PATH"
echo "HOME is: ${HOME:-unset}"
EOF
chmod +x /usr/local/bin/hello-timer.sh
/usr/local/bin/hello-timer.sh

hello from systemd-timers-post at 2026-09-03 22:47:04, pid 1741, user root
PATH is: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
HOME is: /root

Remember those last two lines. They are about to change.

2. The service unit

# /etc/systemd/system/hello.service
[Unit]
Description=Say hello (demo job)

[Service]
Type=oneshot
ExecStart=/usr/local/bin/hello-timer.sh

Three things to notice. Type=oneshot tells systemd this is a job that runs and exits, not a daemon that stays up, so "started successfully" means "exited zero" rather than "the process is alive". ExecStart is the command. And there is no [Install] section, on purpose: this unit is started by the timer, not at boot, so it has no place to be enabled into. Section 4 shows what happens when you try.

Tell systemd the file exists, run it once by hand, and look:

systemctl daemon-reload
systemctl start hello.service
systemctl status hello.service

○ hello.service - Say hello (demo job)
     Loaded: loaded (/etc/systemd/system/hello.service; static)
     Active: inactive (dead)

Sep 03 22:47:05 systemd-timers-post systemd[1]: Starting hello.service - Say hello (demo job)...
Sep 03 22:47:05 systemd-timers-post hello-timer.sh[1873]: hello from systemd-timers-post at 2026-09-03 22:47:05, pid 1873, user root
Sep 03 22:47:05 systemd-timers-post hello-timer.sh[1873]: PATH is: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin
Sep 03 22:47:05 systemd-timers-post hello-timer.sh[1873]: HOME is: unset
Sep 03 22:47:05 systemd-timers-post systemd[1]: hello.service: Deactivated successfully.
Sep 03 22:47:05 systemd-timers-post systemd[1]: Finished hello.service - Say hello (demo job).

inactive (dead) is the correct resting state for a oneshot. It ran, it finished, it is not running now. The script's output went straight into the journal with no redirection on your part, which is the first thing cron never gave you. The same lines are there whenever you want them:

journalctl -u hello.service -o short-iso

And there are the two changed lines. PATH has five entries instead of nine, and HOME is not set at all. Your script did not run in your shell. It ran in systemd's idea of a clean environment, and Section 10 is about how far that goes.

One thing I expected to be a gotcha and was not. Every guide says ExecStart needs an absolute path. I tried ExecStart=hello-timer.sh and it ran, because systemd has looked bare command names up in its own PATH for years now. Write the absolute path anyway. It costs nothing and it means the unit does not depend on where PATH points.

3. daemon-reload, and why your edit did nothing

systemd reads unit files once and keeps them in memory. Edit the file on disk and nothing happens until you say daemon-reload. Everyone gets bitten by this exactly once, so here is what it looks like. I changed the Description= line and then asked:

systemctl status hello.service

Warning: The unit file, source configuration file or drop-ins of hello.service changed on disk. Run 'systemctl daemon-reload' to reload units.
○ hello.service - Say hello (demo job)

The warning is loud, but only if you run status. systemctl start prints it too and then starts the old definition regardless:

systemctl start hello.service
systemctl show hello.service -p Description

Warning: The unit file, source configuration file or drop-ins of hello.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Description=Say hello (demo job)

Still the old text. After systemctl daemon-reload it reads Description=Say hello (edited on disk) and the warning goes away. The habit to build is that daemon-reload is part of saving the file, not a separate step you do when something looks wrong.

4. The timer unit

# /etc/systemd/system/hello.timer
[Unit]
Description=Run hello every minute (demo)

[Timer]
OnCalendar=*-*-* *:*:00
Unit=hello.service

[Install]
WantedBy=timers.target

OnCalendar is the schedule, here every minute on the minute, which is a terrible schedule for anything real and a very good one for watching it work. Unit= names what to start, and you can leave it out when the service has the same name as the timer, which it does here and should in general. The [Install] section is what makes enable mean something: timers.target is the unit that starts all timers at boot, and enabling this timer hooks it in there.

This is also the moment to see what enabling the service instead would have done, because it is the most common first mistake:

systemctl enable hello.service

The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.
...

Correct, and that is exactly why the service has no [Install] section. If it had one with WantedBy=multi-user.target, enabling it would make the job run once at every boot, which for a backup or a cleanup is a surprise nobody wants. Enable the timer:

systemctl daemon-reload
systemctl enable --now hello.timer
systemctl list-timers hello.timer

Created symlink '/etc/systemd/system/timers.target.wants/hello.timer' → '/etc/systemd/system/hello.timer'.
NEXT                        LEFT LAST PASSED UNIT        ACTIVATES
Thu 2026-09-03 22:48:00 UTC  51s -         - hello.timer hello.service

--now starts it as well as enabling it, and list-timers is the command you will run more than any other in this post. systemctl status hello.timer shows the same in prose, with a Trigger: line and a Triggers: ● hello.service line that ties the two files together.

Wait two minutes and read the service's journal, and the job is running itself:

2026-09-03T22:48:27+00:00 systemd-timers-post hello-timer.sh[2109]: hello from systemd-timers-post at 2026-09-03 22:48:27, pid 2109, user root

Look at that timestamp though. The timer was for 22:48:00 and the job ran at 22:48:27. Section 6.

5. Reading and writing OnCalendar

The format is DayOfWeek Year-Month-Day Hour:Minute:Second, every part optional, * for any, .. for ranges, , for lists, / for steps. You do not have to remember that, because systemd-analyze calendar will tell you what any expression means and when it next fires, and it is the tool to reach for before you put anything in a unit file:

systemd-analyze calendar "Mon..Fri 09:00"

  Original form: Mon..Fri 09:00
Normalized form: Mon..Fri *-*-* 09:00:00
    Next elapse: Fri 2026-09-04 09:00:00 UTC
       From now: 10h left

Here is a table I built by feeding it the expressions I actually use, so I stop working them out from the man page:

You write	systemd reads it as	Meaning
`daily`	`--* 00:00:00`	midnight every day
`03:30`	`--* 03:30:00`	03:30 every day
`hourly`	`--* *:00:00`	top of every hour
`*:0/15`	`--* *:00/15:00`	every 15 minutes
`weekly`	`Mon --* 00:00:00`	Monday midnight
`Mon..Fri 09:00`	`Mon..Fri --* 09:00:00`	weekday mornings
`Mon,Wed,Fri 18:00`	`Mon,Wed,Fri --* 18:00:00`	three evenings a week
`--01 00:00:00`	`--01 00:00:00`	first of the month
`quarterly`	`*-01,04,07,10-01 00:00:00`	first of Jan, Apr, Jul, Oct
`2026-12-25 08:00`	`2026-12-25 08:00:00`	once, on that date

Two things worth knowing. --iterations=3 shows the next three firings instead of one, which is how you catch that weekly means Monday and not "seven days from now". And a real typo is rejected rather than guessed at:

systemd-analyze calendar "every day at 3"

Failed to parse calendar specification 'every day at 3': Invalid argument

That same rejection is what you get if the typo makes it into a timer unit, except there it is quieter. systemd-analyze verify /etc/systemd/system/bad.timer prints Failed to parse calendar specification, ignoring and Timer unit lacks value setting. Refusing, and systemctl start fails with bad unit file setting. Run verify on anything new before you enable it. One thing it forgives: I typed Mon-Fri 09:00 with a single hyphen, and it silently normalised it to Mon..Fri.

Timezones. OnCalendar is read in the box's local timezone, which on a fresh server is almost always UTC. Check with timedatectl. If the box is on local time and you want a job in UTC anyway, append it: *-*-* 03:30:00 UTC is valid and systemd-analyze calendar shows the UTC in the normalised form.

6. Why it fired late

The 22:48:00 job ran at 22:48:27. The next one, scheduled for 22:49:00, ran at 22:49:18. Nothing is wrong. Timers have a setting called AccuracySec, and its default is one minute:

systemctl show hello.timer -p AccuracyUSec

AccuracyUSec=1min

Which means: fire at any point in the minute after the scheduled time. systemd does this so it can batch timers together and wake the machine up less often, and for a nightly backup a few seconds of drift is nothing. For anything where the exact minute matters, say a job that has to land before another system polls at :05, set it explicitly. I ran the same minute timer twice, once with the default and once with AccuracySec=1s, and logged the moment the job started:

default:           fired at 22:50:03.959   fired at 22:51:08.954
AccuracySec=1s:    fired at 22:52:00.147   fired at 22:53:00.147

With the default I saw the job land anywhere from four to twenty seven seconds after the minute. With AccuracySec=1s it landed 150 milliseconds after, both times. Ubuntu's own fstrim.timer goes the other way and sets AccuracySec=1h, because nobody cares which hour of Monday the disks get trimmed. Pick the one that matches how much you care.

7. The box was off at 03:30

This is the single best reason to use a timer over cron for anything on a machine that is ever switched off or rebooted. Cron skips a missed run and says nothing. A timer with Persistent=true remembers.

# /etc/systemd/system/daily.timer
[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

The mechanism is a stamp file. Every time the timer fires it touches /var/lib/systemd/timers/stamp-daily.timer, and when the timer is next activated, systemd compares that stamp to the schedule. If a run should have happened in between, it runs the job immediately.

Two things I wanted to know before trusting it. First, does a brand new persistent timer fire the moment you enable it, on the theory that it has "missed" every run since the beginning of time? No. On first enable it created the stamp and waited for midnight like any other timer. Second, does it really catch up? I made it look as if the last run was two days ago and started the timer:

systemctl stop daily.timer
touch -d "2 days ago" /var/lib/systemd/timers/stamp-daily.timer
systemctl start daily.timer
journalctl -u daily.service -o short-iso --since "-1min"

2026-09-03T23:00:43+00:00 systemd-timers-post sh[4482]: daily job ran at 23:00:43

Immediately. The same test with the Persistent=true line removed did nothing, and list-timers showed it calmly waiting for midnight.

Then I did it properly. I set the timer for 23:10, powered the server off at 23:07, and turned it back on through the Hetzner API at 23:11:48, so the slot passed with the machine dark:

uptime -s            2026-09-03 23:12:01
2026-09-03T23:12:11+00:00 systemd-timers-post systemd[1]: Starting daily.service...
2026-09-03T23:12:11+00:00 systemd-timers-post sh[964]: daily job ran at 23:12:11

Ten seconds after boot, the missed run happened. A cron job at 23:10 would have been silently skipped and you would have found out when you went looking for last night's backup. Every timer Ubuntu ships that does real work, fstrim, apt-daily, logrotate, has Persistent=true. Yours should too.

8. Timers that count from boot, and jitter

OnCalendar is wall clock time. There is a second family that counts durations instead, and they are useful for "a while after boot" and "every so often" without caring what time it is:

[Timer]
OnBootSec=15min
OnUnitActiveSec=1h

OnBootSec=15min fires fifteen minutes after the machine came up. OnUnitActiveSec=1h fires an hour after the service last ran. Together they mean "first run fifteen minutes after boot, then hourly", which is a better shape for a health check than hourly, because it does not pile onto whatever else runs at the top of the hour. list-timers showed mine for 23:01:21 on a box that booted at 22:46:21, which is the arithmetic you would expect.

RandomizedDelaySec spreads a run out over a window. Ubuntu's apt-daily.timer uses RandomizedDelaySec=12h so that every Ubuntu machine on earth does not hit the mirrors at the same second. I put RandomizedDelaySec=30min on a 03:30 timer and restarted it three times:

Fri 2026-09-04 03:49:13 UTC
Fri 2026-09-04 03:44:33 UTC
Fri 2026-09-04 03:44:47 UTC

A fresh point in the window each time. This is why the restic post has RandomizedDelaySec=30m on its backup: three homelab boxes all backing up to the same bucket at exactly 03:30 is a small self inflicted thundering herd.

9. When the job fails

A script that exits nonzero fails the service, and systemd is not shy about it:

systemctl start flaky.service
systemctl status flaky.service

Job for flaky.service failed because the control process exited with error code.
See "systemctl status flaky.service" and "journalctl -xeu flaky.service" for details.

× flaky.service - A job that fails
     Active: failed (Result: exit-code) since Thu 2026-09-03 22:54:41 UTC; 20ms ago
    Process: 3305 ExecStart=/bin/sh -c echo "about to fail"; exit 3 (code=exited, status=3)

The × and failed stick until you clear them, and systemctl list-units --failed lists every unit on the box in that state, which is a decent morning check on any server. systemctl reset-failed flaky.service clears it. One thing that will confuse you the first time: the journal line reads status=3/NOTIMPLEMENTED. systemd keeps a table of names for exit codes, inherited from the old LSB init script convention where 3 meant "unimplemented feature", and it prints the name next to any code it recognises. Your script exited 3 and that is all it means.

Nothing emails you, and that is the honest gap compared to cron's MAILTO. The building block systemd gives you instead is OnFailure=, which starts another unit when this one fails. Here is the smallest useful version. It uses a template unit, which is worth a sentence because the filename looks odd: a unit named something@.service is a reusable definition you start with an argument after the @, and inside the file %i stands for that argument. So notify-failed@flaky.service is the template run with flaky.service as its argument. The alert goes into the journal at error priority, where anything you already use to watch logs will see it:

# /etc/systemd/system/flaky.service
[Unit]
Description=A job that fails
OnFailure=notify-failed@%n.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "about to fail"; exit 3'

# /etc/systemd/system/notify-failed@.service
[Unit]
Description=Log a failure for %i

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "ALERT: %i failed on $(hostname), see: journalctl -u %i" | systemd-cat -t failure-alert -p err'

%n in the first file expands to the failing unit's own name, which becomes the argument, and %i in the template picks it up, so one notify-failed@.service serves every job on the box. On the systemd-cat line, -t tags the message so you can find it and -p err sets the priority. The result:

journalctl -t failure-alert -o short-iso

2026-09-03T22:54:41+00:00 systemd-timers-post failure-alert[3311]: ALERT: flaky.service failed on systemd-timers-post, see: journalctl -u flaky.service

Swap the echo | systemd-cat for a curl to whatever pings your phone and you have cron's MAILTO, but per job and without a mail server.

10. Your script is not running in your shell

Section 2 showed HOME unset and a short PATH. It goes further than that, and every item below is something I have seen in a real unit file that did not do what its author thought.

There is no shell

ExecStart is split into words by systemd and executed directly. Pipes, redirects and && are just arguments:

ExecStart=/bin/echo hello | /usr/bin/tr a-z A-Z

hello | /usr/bin/tr a-z A-Z

That printed the pipe. If you need shell syntax, ask for a shell: ExecStart=/bin/sh -c 'echo hello | tr a-z A-Z' printed HELLO. Better still, put anything longer than one command in a script file and point ExecStart at the script, which is what Section 1 did.

No tilde, no globs

~ stays a literal ~ and * stays a literal *, for the same reason: those are shell features.

`$` means something to systemd first

systemd does its own variable expansion on ExecStart before your command runs, and it has rules that are almost, but not quite, the shell's. I tested every spelling against Environment=NAME=keith:

ExecStart=/bin/echo bare: $NAME          ->  bare: keith
ExecStart=/bin/echo braces: ${NAME}      ->  braces: keith
ExecStart=/bin/echo quoted: "$NAME"      ->  quoted: keith
ExecStart=/bin/echo doubled: $$NAME      ->  doubled: $NAME
ExecStart=/bin/sh -c 'echo shell: $NAME' ->  shell: keith

$$ is how you get a literal dollar through to your command. Where this bites is shell constructs that systemd half understands: I wrote ${#SECRET_TOKEN} inside an sh -c to print a length, systemd saw ${...}, looked for a variable literally named #SECRET_TOKEN, and substituted nothing. The output was silently empty. Plain $SECRET_TOKEN inside the quotes made it through to the shell and worked.

`%` means something to systemd too

Percent signs in unit files are specifiers: %n is the unit name, %H the hostname, %u the user, %T the temp directory. So this:

ExecStart=/bin/sh -c 'echo "daily job ran at $(date +%T)"'

logs daily job ran at /tmp, because %T became /tmp before date ever saw it. I did this in my own test unit and stared at it for a minute. Double it, %%T, and date gets its %T back.

Set the environment on purpose

The useful knobs, together:

[Service]
Type=oneshot
User=keith
WorkingDirectory=/home/keith
EnvironmentFile=/etc/hello.env
ExecStart=/bin/sh -c 'echo "user=$(id -un) pwd=$(pwd) token=$SECRET_TOKEN"'

user=keith pwd=/home/keith token=abc123

User= runs the job as someone other than root, which most jobs should. WorkingDirectory= matters because the default is /. EnvironmentFile= reads KEY=value lines from a file, which is where a token or a password belongs, in a root owned file with mode 600, and not in the unit file that systemctl cat will happily print to anyone. The restic post's Environment=HOME=/root line exists precisely because of the HOME is: unset result from Section 2, restic wanted a home directory for its cache, and a system service does not get one for free.

11. When the job takes longer than the interval

What if the timer fires again while the previous run is still going? I gave a service a hundred second sleep and a timer that fires every minute:

2026-09-03T22:55:00+00:00 slow start
2026-09-03T22:56:40+00:00 slow end
2026-09-03T22:56:40+00:00 slow start

No second copy started at 22:56:00. Starting a service that is already running does nothing, and what my log shows is the 22:56 trigger waiting behind the running job and firing the instant it finished, at 22:56:40. You get at most one instance at a time, which is exactly right for a backup. If you want a fixed gap between runs instead of a fixed clock time, OnUnitActiveSec from Section 8 counts from the last start, so a slow run simply pushes the next one back.

12. Testing without waiting until 03:30

The service is a normal unit, so the way to test a nightly job is to run it now:

systemctl start backup.service
journalctl -u backup.service -n 20

That runs the exact command, as the exact user, with the exact environment the timer will use. It is a much better test than running the script from your shell, because your shell has a HOME and a full PATH and Section 10 happens.

For a one off "run this in twenty seconds" or "at 03:30 tonight only" there is systemd-run, which creates a temporary timer and service without any files:

systemd-run --on-active=20s --unit=oneoff /bin/sh -c 'echo "oneoff fired at $(date +%T)"'
systemctl list-timers oneoff.timer

Running timer as unit: oneoff.timer
Will run service as unit: oneoff.service
NEXT                        LEFT LAST PASSED UNIT         ACTIVATES
Thu 2026-09-03 22:58:29 UTC  19s -         - oneoff.timer oneoff.service

--on-calendar="*-*-* 03:30:00" gives you the calendar flavour. The transient timer fires, the output lands in journalctl -u oneoff.service, and then the units disappear on their own. It is at, without installing at.

13. Timers for a normal user

Everything so far went in /etc/systemd/system/ as root. A normal user gets their own systemd, with its own units in ~/.config/systemd/user/, driven by systemctl --user. Same file format, no sudo, and the jobs run as you. The catch is what happens when you log out.

I created note.timer and note.service as user keith, logged in over SSH, enabled the timer, and logged out:

systemctl --user daemon-reload
systemctl --user enable --now note.timer
systemctl --user list-timers note.timer

Created symlink '/home/keith/.config/systemd/user/timers.target.wants/note.timer' → '/home/keith/.config/systemd/user/note.timer'.
Thu 2026-09-03 23:05:00 UTC 46s - - note.timer note.service

Seventy five seconds later, from root:

systemctl is-active user@1000.service
journalctl _UID=1000 --since "-2min" | grep "user timer"

inactive

Nothing ran. A user's systemd instance is user@<uid>.service, and by default it starts when the user logs in and stops when their last session ends, taking every user timer down with it. The fix is one command, run once, by root or by the user themselves:

loginctl enable-linger keith

Lingering keeps user@1000.service up with no session at all. Same timer, same enable, log out, and this time:

2026-09-03T23:06:00+00:00 systemd-timers-post sh[2375]: user timer ran as keith at 23:06:00

The user reads their own runs with journalctl --user -u note.service, and root can see them with journalctl _UID=1000. One trap for anyone administering this from root: sudo -iu keith systemctl --user ... does not work. It fails with Failed to connect to user scope bus, because sudo does not set up the session environment the user manager needs. Log in as the user properly. The error message suggests --machine=keith@.host --user as a way for root to reach the user's manager directly, which I have not tried.

14. Changing one of Ubuntu's own timers

You will eventually want apt-daily.timer to run at 04:00 instead of somewhere in a twelve hour window, or fstrim on a Sunday. Do not edit the file under /usr/lib, the next package update will overwrite it. Add an override:

systemctl edit apt-daily.timer

That opens an editor on a new file, /etc/systemd/system/apt-daily.timer.d/override.conf, which systemd merges over the original. systemd calls that file a drop-in, and it is the word in the warning from Section 3. I tested this from a session with no terminal, where systemctl edit refuses to run, so I wrote the same file by hand and ran daemon-reload, which is all edit does for you. Here is the one that pins the apt check to 04:00:

[Timer]
OnCalendar=
OnCalendar=*-*-* 04:00:00
RandomizedDelaySec=0

The blank OnCalendar= line is not a typo and it is the part everyone misses. OnCalendar is a list setting, and a drop-in that supplies another one adds a time rather than replacing it. I checked: without the blank line, systemctl show apt-daily.timer -p TimersCalendar still listed the original 06,18:00 alongside my 04:00, so apt would have run three times a day. An empty assignment clears the list first.

systemctl cat apt-daily.timer now shows both files, original and override, which is how you can always tell what is actually in effect. systemctl revert apt-daily.timer deletes the override and puts things back.

Gotchas worth knowing

Edits do nothing until daemon-reload. start prints a warning and runs the old definition anyway. Make the reload part of saving the file.

Enable the timer, not the service. A service with no [Install] section refuses to be enabled, which is the right outcome. A service you gave WantedBy=multi-user.target runs at every boot on top of its schedule.

It fires up to a minute late by default. AccuracySec=1min. Set AccuracySec=1s when the minute matters. Measured drift on the default was between four and twenty seven seconds.

Without Persistent=true, a run missed during downtime is gone. With it, the job runs within seconds of the next boot. It does not fire on first enable, only for runs it can prove were missed.

% and $ belong to systemd. %T became /tmp in my date format and ${#VAR} became nothing. Double them, %% and $$, to pass them through.

No shell, no HOME, five entry PATH, working directory /. Wrap shell syntax in /bin/sh -c, set Environment=HOME= if a tool wants one, and set WorkingDirectory= if the script assumes one.

status=3/NOTIMPLEMENTED just means exit code 3. The word is systemd's name for that code from the LSB table and has nothing to do with your script.

A drop-in OnCalendar= adds a schedule. Put an empty OnCalendar= above the new one to replace instead.

User timers die at logout unless loginctl enable-linger <user> has been run once. And sudo -iu user systemctl --user cannot reach the user manager at all.

A typo in OnCalendar stops the timer from loading and says so quietly. systemd-analyze verify on the file, or systemd-analyze calendar on the expression, before you enable.

Quick reference

Task	Command
Pick up an edited unit file	`systemctl daemon-reload`
Turn a timer on, now and at boot	`systemctl enable --now job.timer`
See every timer and when it next fires	`systemctl list-timers --all`
Last run and next run of one timer	`systemctl status job.timer`
Run the job right now, timer or not	`systemctl start job.service`
Read the job's output	`journalctl -u job.service -o short-iso`
Follow it live	`journalctl -u job.service -f`
Since last boot only	`journalctl -u job.service -b`
What does this schedule mean	`systemd-analyze calendar "Mon..Fri 09:00" --iterations=3`
Check a unit file before enabling	`systemd-analyze verify /etc/systemd/system/job.timer`
Show the file, plus any overrides	`systemctl cat job.timer`
Every unit currently in a failed state	`systemctl list-units --failed`
Clear a failed state	`systemctl reset-failed job.service`
One off run in twenty seconds	`systemd-run --on-active=20s --unit=oneoff /path/to/cmd`
Override a vendor timer	`systemctl edit apt-daily.timer`
Undo the override	`systemctl revert apt-daily.timer`
User timers survive logout	`loginctl enable-linger <user>`
Where the missed run stamps live	`/var/lib/systemd/timers/stamp-*.timer`

The pair of files I now start every new job from, with everything above baked in:

# /etc/systemd/system/job.service
[Unit]
Description=What this job does

[Service]
Type=oneshot
User=someone
WorkingDirectory=/where/it/expects/to/be
EnvironmentFile=/etc/job.env
ExecStart=/usr/local/bin/job.sh

# /etc/systemd/system/job.timer
[Unit]
Description=When it runs

[Timer]
OnCalendar=*-*-* 03:30:00
Persistent=true
RandomizedDelaySec=10min

[Install]
WantedBy=timers.target

daemon-reload, enable --now job.timer, start job.service once to prove it works, and then leave it alone. That last step is the whole point. [ timer armed ]

Originally published at peculiarengineer.com.

A disposable Hetzner lab with OpenTofu: real Ubuntu box, run a script, destroy it

Minor Keith — Mon, 24 Aug 2026 15:03:03 +0000

Most of what I write here is Ubuntu, and I write it on a Mac. That gap is a problem. A Docker container is not a machine: it shares my kernel, it has no real systemd to fight with, it cannot reboot, and do-release-upgrade inside one is meaningless. A local VM is closer but it lives forever, drifts, and quietly becomes a machine I maintain instead of a machine I test on.

What I actually want is a real Ubuntu server that exists for twenty minutes and then does not. Hetzner bills by the hour and hands you a booted box in about sixty seconds, so the cloud part is easy. The part worth writing down is everything around it: where the state goes when you are creating and destroying the same thing forty times, how you make sure a forgotten VM does not bill you all month, and the three ways this tool could quietly cost me money, open a box to the internet, or leave something behind it no longer knows about. All three were in my own code. I found one by reading it and the other two by trying to build the box for the next post, which failed. All three are fixed now, and each one is in here because the fix is less interesting than the reason it was wrong.

TL;DR. One hcloud_server and one hcloud_firewall in OpenTofu, with the firewall locked to your public IP /32. Keep state in S3 at labs/<lab_id>/state.tfstate, one object per lab, so concurrent labs never touch the same file. Wrap it in a Python script that does up, down, ssh, status, and sweep. Push the work into a runbook, a plain Python file that gets copied to the box and run as root, so the test is a reproducible artifact instead of shell history. Set a short TTL and sweep from cron, because nothing cloud side enforces it.

The shape of it

The commands you will actually use are up, down, ssh, status, and sweep, and the whole thing is a Python file plus five .tofu files.

lab up --image ubuntu-26.04 --ttl 1 --runbook runbooks/check.py
lab ssh -c 'journalctl -n 50 --no-pager'
lab down

up creates one server and one firewall, then prints the SSH command and a destroy deadline. Hand it a runbook and it also waits for SSH, copies the script to the box, and runs it as root while streaming the output back. down destroys it. sweep destroys anything past its deadline. That is the entire product.

The design decision that matters is that a lab is one server, not a fleet. The moment you let it grow a second node you need inventory, ordering, and a graph, and you have written a worse Ansible. One box, born and killed, is a small enough problem to keep honest.

Prerequisites

OpenTofu 1.10 or newer, which is where S3 state locking arrives. I am on 1.12.6. Terraform works too, since this uses nothing OpenTofu specific.
Python 3.10 or newer and uv.
A Hetzner Cloud project and an API token with read and write, from Security, then API tokens.
Your SSH public key uploaded under Security, then SSH keys. Note the name you gave it.
An S3 compatible bucket. I use Hetzner Object Storage in the same region, so state and logs sit next to the servers.

1. The OpenTofu is the small part

Two resources. That is it.

# main.tofu
data "hcloud_ssh_key" "lab" {
  name = var.ssh_key_name
}

resource "hcloud_firewall" "lab" {
  name = "${var.lab_id}-fw"

  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "22"
    source_ips = [var.allowed_ssh_cidr]
  }

  rule {
    direction  = "in"
    protocol   = "icmp"
    source_ips = ["0.0.0.0/0", "::/0"]
  }
}

resource "hcloud_server" "lab" {
  name         = var.lab_id
  server_type  = var.server_type
  image        = var.image
  location     = var.location
  ssh_keys     = [data.hcloud_ssh_key.lab.id]
  firewall_ids = [hcloud_firewall.lab.id]

  labels = {
    lab        = "true"
    lab_id     = var.lab_id
    ttl_hours  = var.ttl_hours
    created_at = var.created_at
  }
}

One more file does quiet work alongside this one. outputs.tofu exports server_ip, server_id, and lab_id, and the Python wrapper reads the first two back with tofu output -json right after apply. Copy the resources, skip the outputs, and up dies immediately after building your server, which is the worst moment for it to die.

Two things in the resources are load bearing and easy to skip past.

Only port 22 and ICMP are open. Nothing else, inbound, ever. If a runbook installs Nginx you will not reach it from your browser, and that is deliberate: a box that exists for twenty minutes with a service exposed to the internet is how you end up in someone's scan results. When I need to see a web service I forward it over the SSH session I already have:

ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    -N -L 8080:localhost:80 root@<ip>

The labels are the safety net. lab = "true", plus the TTL and creation time, live on the server in Hetzner's own API. Local files get deleted, laptops get reimaged, and a .lab/ directory is one rm -rf away from being gone. The labels stay attached to the running server, so no matter what happens on my machine I can always ask Hetzner "what did I leave on?" and get a real answer:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
  'https://api.hetzner.cloud/v1/servers?label_selector=lab%3Dtrue' \
  | python3 -c 'import json,sys; [print(s["name"], s["public_net"]["ipv4"]["ip"], s["labels"]) for s in json.load(sys.stdin)["servers"]]'

Put that somewhere you will find it. It is the command that answers "am I being billed for something I forgot about," and it does not care whether your local state survived.

2. One state object per lab

The backend is declared empty and filled in at init time:

# backend.tofu
terraform {
  backend "s3" {}
}

Then every command passes the key for that specific lab:

tofu init -input=false -reconfigure \
  -backend-config="bucket=$S3_BUCKET" \
  -backend-config="key=labs/$LAB_ID/state.tfstate" \
  -backend-config="region=$S3_REGION" \
  -backend-config="endpoint=$S3_ENDPOINT" \
  -backend-config="access_key=$AWS_ACCESS_KEY_ID" \
  -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY" \
  -backend-config="skip_credentials_validation=true" \
  -backend-config="skip_metadata_api_check=true" \
  -backend-config="skip_region_validation=true" \
  -backend-config="skip_requesting_account_id=true" \
  -backend-config="use_lockfile=true" \
  -backend-config="use_path_style=true"

The four skip_ flags are not optional and they are the thing that sends people to the search bar. The S3 backend assumes AWS. Point it at Hetzner Object Storage, MinIO, or Backblaze and it will try to validate credentials against AWS endpoints, look for an EC2 metadata service that is not there, reject nbg1 as a region name, and call STS to work out an account ID. Every one of those fails against a non AWS provider. Skip them all.

use_path_style=true matters for the same reason. Hetzner Object Storage wants https://nbg1.your-objectstorage.com/bucket/key, not https://bucket.nbg1.your-objectstorage.com/key. Get it wrong and you get a DNS failure that looks nothing like a configuration mistake.

Why one object per lab instead of one workspace or one big state? Because labs are independent and I want to run two at once without them knowing about each other. Separate keys means separate state files, so a 24.04 box and a 26.04 box can be up simultaneously and neither can corrupt the other. It also means a lab's entire existence is one deletable prefix in a bucket.

The credentials failure here is worth calling out because it does not announce itself. Bad S3 keys surface as an opaque tofu init error with no mention of S3. So the first thing the tool does when you complain is check them directly:

lab test-s3
# Endpoint: https://nbg1.your-objectstorage.com
# Region:   nbg1
# Bucket:   minor-lab-state
# Access:   ABC123...
#
# OK: bucket 'minor-lab-state' is reachable.

If that passes and init still fails, the problem is genuinely OpenTofu. If it fails, you have your answer in two seconds instead of twenty minutes.

3. The wrapper

The Python file is the whole user interface. up, down, ssh, logs, status, sweep, plus test-s3, test-discord, and server-types. It shells out to tofu, holds the flags so I never type a backend config by hand, and writes a small metadata file per lab.

Three details in up earned their place.

It waits for SSH properly, when there is a runbook. Hetzner reports the server as running before cloud-init has finished, so connecting immediately gets you a refused connection. The wrapper polls port 22 every two seconds for up to two minutes, with the progress prints trimmed out of the quote below. A bare lab up with no runbook skips this entirely. It applies, prints, and exits without ever probing port 22, so the address it hands you may not answer for another half minute.

def _wait_for_ssh(ip: str, timeout: int = 120) -> None:
    deadline = _now() + timedelta(seconds=timeout)
    while _now() < deadline:
        try:
            with socket.create_connection((ip, 22), timeout=5):
                return
        except OSError:
            pass
        time.sleep(2)
    sys.exit(1)

It saves metadata before it does anything risky. The IP, the server ID, the TTL, and the destroy deadline are written to disk the moment the server exists, before the runbook runs. If the runbook explodes, or I hit Ctrl C, the box is still tracked and still destroyable. Saving that file after the interesting part would mean any failure leaves an untracked, billed server.

It fails closed. If the runbook exits non zero, up destroys the server and firewall, wipes the local metadata, and still exits with the runbook's code. That is the correct default for a tool that spends money, but it has a consequence you need to internalize: the evidence dies with the box. To debug a failure you have to decide beforehand and pass --keep-on-failure. It took two fixes to make that heading true, and both are in the third footgun below.

The log survives a failed runbook, because the upload to S3 happens before the destroy. After an automatic teardown, lab logs fails because the local directory is gone, but the full output including stderr is sitting at s3://<bucket>/labs/<lab_id>/runbook.log. Read it from there.

One hole in that, which I would rather state than have you discover. The upload only gets skipped when the SCP itself fails, because that raises and goes straight to the exception handler, which destroys the box without uploading. A connection that dies partway through a running script is fine: the read loop just ends, up uploads whatever made it into the log, and you get a truncated but real record. So the log survives a script that fails and a session that drops. It does not survive a runbook that never landed on the box.

4. Runbooks, not shell history

A runbook is a Python file that gets copied to /tmp/runbook.py and executed as root. Output is merged, streamed live to my terminal, written locally, and uploaded to S3.

#!/usr/bin/env python3
import subprocess, sys

def run(cmd):
    print(f"$ {' '.join(cmd)}")
    return subprocess.run(cmd, check=True)

def main() -> int:
    run(["apt-get", "update", "-qq"])
    run(["apt-get", "install", "-y", "-qq", "nginx"])
    return 0

if __name__ == "__main__":
    sys.exit(main())

Python rather than bash for one reason: exit codes. A bash script without set -euo pipefail cheerfully reports success after every command in it failed, and I have been burned by a green test that proved nothing. subprocess.run(..., check=True) raises, the script exits non zero, and the tool tears the lab down and tells me. Failure is loud by default instead of by remembering to configure it.

The real payoff is that the runbook is a file. When I write an Ubuntu guide, the script that proves it works can ship next to the post. A reader can run the same file on their own box and get the same output rather than trusting that I ran the commands in the order I wrote them.

One thing bites on the first run. apt-get fails with Could not get lock /var/lib/dpkg/lock-frontend because cloud-init is still installing packages when SSH comes up. Waiting for port 22 is not the same as waiting for the machine to be finished. Either wait for cloud-init or retry:

run(["cloud-init", "status", "--wait"])

5. Teardown is the actual feature

Creating servers is easy. Not paying for them is the part that needs design.

Every lab gets a destroy_after timestamp from --ttl, in hours. sweep walks the labs and destroys anything past its deadline, which goes in cron:

0 * * * * cd ~/projects/hetzner-terraform-dev-lab && uv run ./lab sweep >> /tmp/lab-sweep.log 2>&1

uv run is not decoration there. The script exits at import time if boto3 and python-dotenv are missing, and system Python has neither, so a bare ./lab sweep from cron prints a dependency error into the log every hour and sweeps nothing. Silently, in the one place whose entire job is to stop you being billed for a machine you forgot.

Be honest about what this is: the TTL is advisory. Nothing on Hetzner's side enforces it. If my laptop is asleep, cron does not run, and the box bills merrily. A short TTL plus prompt lab down is the real story, and sweep is a backstop for the times I forget.

sweep also destroys any expired lab, including one somebody else on the machine created. Mine is a single user Mac so that is theoretical, but it is why the command is not wired into up.

The failure I care most about is the one where teardown itself fails. The script prints Manual cleanup may be required, then re-raises, so you get a traceback with it. That is the correct behavior, and it is also the exact moment to go run that label selector query from earlier.

Three footguns I found in my own tool

Writing this post is what surfaced all three. The first I found by reading the code, and the other two found me, in the middle of trying to spin up the box for the next post.

The public IP lookup fails open

up locks SSH to your current public IP by default. It gets that IP from ipify:

def _public_ip() -> str:
    try:
        with urllib.request.urlopen("https://api.ipify.org", timeout=10) as resp:
            return resp.read().decode().strip()
    except Exception:
        return ""

and then:

ip = _public_ip()
allowed_ssh_cidr = f"{ip}/32" if ip else "0.0.0.0/0"

Read that second block again. If ipify is slow, blocked, rate limiting me, or I am on a captive portal, _public_ip() swallows the exception and returns an empty string, and the default silently becomes root SSH open to the entire internet. No warning, no prompt, nothing in the output that looks different from a normal run. The one line in up's output that would tell you is ssh cidr: 0.0.0.0/0, sitting in a block of five lines that all look like routine startup noise.

It fails closed now:

ip = _public_ip()
if not ip:
    print("Could not determine your public IP address.")
    print("Refusing to default the SSH rule to 0.0.0.0/0.")
    print("Pass --allow-ssh <cidr> explicitly if that is really what you want.")
    sys.exit(1)
allowed_ssh_cidr = f"{ip}/32"

And when you do ask for it deliberately, it says so out loud instead of hiding in the startup noise:

WARNING: SSH will be open to the entire internet (0.0.0.0/0).

Wide open is a legitimate thing to want sometimes, from a CI runner with no stable egress address for instance. It is not a legitimate thing to arrive at by accident because a third party HTTP request timed out. Make the human type it.

The same default was sitting in variables.tofu as default = "0.0.0.0/0", which is the belt to match the braces. That default is gone too, so the variable is now required and a missing value is an error rather than an open port.

Hetzner will price a server type it will not sell you

I picked cpx12 when I set this up because the name looked like the small one. Later I built a server-types command so I would stop guessing, ran it, and it told me cpx11 was two cores instead of one and less than half the price. Obvious upgrade. I changed the flag, ran it, and got this:

Error: Server Type "cpx11" is unavailable in "nbg1" and can no longer be ordered

  with hcloud_server.lab,
  on main.tofu line 22, in resource "hcloud_server" "lab":

Here is the part worth the post. Hetzner's /v1/server_types endpoint still returns cpx11 with a current hourly price for nbg1 and deprecation: null. Nothing in that response suggests you cannot have one. The entire cpx*1 generation is retired, and the only endpoint that admits it is /v1/datacenters, which carries a server_types.available list of numeric IDs per datacenter:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
  'https://api.hetzner.cloud/v1/datacenters' \
  | python3 -c 'import json,sys; [print(d["name"], sorted(d["server_types"]["available"])) for d in json.load(sys.stdin)["datacenters"]]'

Prices and deprecation flags describe the catalog. Only that list describes the shop.

Which made my own command a liar. lab server-types --location nbg1 printed the header Available server types in nbg1: and then listed cpx11, cpx21, cpx31, cpx41, cpx51, cx43, and cx53, none of which can be ordered there. It filtered on whether a price exists for the location, which is not the same question. A command whose whole purpose is to stop me guessing sent me straight into a failed apply.

It cross references the datacenter list now, and only prints what you can actually have:

Server types you can actually order in nbg1:
NAME          CPU   RAM   DISK  ORDERABLE   PRICE/MO
cax11           2    4GB    40GB        yes       6.99
cpx12           1    2GB    40GB        yes      13.49
cx23            2    4GB    40GB        yes       6.49
...                                                       (17 rows in total)

Pass --all to include types that are priced here but cannot be ordered.

--all keeps the rest visible with an honest column, which is where cpx11 now shows up as NO.

Among the types you can actually order in nbg1, the original default still looks bad:

Type	Cores	RAM	Hourly	Orderable in nbg1
`cx23`	2	4 GB	€0.0104	yes
`cax11` (arm)	2	4 GB	€0.0112	yes
`cpx12`	1	2 GB	€0.0216	yes
`cpx11`	2	2 GB	€0.0096	no

cx23 is double the cores and double the RAM of the old default, for less than half the hourly. It is the default now. The arm box is worth a look too, since Hetzner publishes ubuntu-26.04 for arm as well as x86.

The teardown handler has two blind spots

The cpx11 failure exposed a third one immediately. When tofu apply fails, the run leaves behind a lab-upgrade-2604-fw firewall with nothing attached to it, and lab status shows an empty table. Local metadata is written after apply succeeds, so from the tool's point of view the lab was never born, and lab down has nothing to destroy.

The cause is one line of control flow. In cmd_up, the apply happens before the try block that owns the cleanup:

_tofu_init(lab_id)
_tofu_apply(lab_id, args, allowed_ssh_cidr, created_at_label)
vm_created = True

try:
    ...
except Exception:
    # tears down the server and the firewall

Every failure after that point was handled properly. The failure of that line was not handled at all, and it is the most likely one, because apply is where quota limits, retired server types, unavailable images, and API hiccups all land.

The apply lives inside the try now. The subtle part is the flag, which has to be set before the apply rather than after it, because a failed apply may already have built the firewall. Set it after and you have written the same bug in a new place, which I did on the first attempt:

resources_touched = False
try:
    _tofu_init(lab_id)
    # Set before the apply, not after: a failed apply can still have created
    # the firewall, and that is exactly the case that used to leak.
    resources_touched = True
    _tofu_apply(lab_id, args, allowed_ssh_cidr, created_at_label)

Running the same cpx11 failure against the fixed version now ends the way it should:

Error: Server Type "cpx11" is unavailable in "nbg1" and can no longer be ordered

Lab setup failed: Command '['tofu', 'apply', ...]' returned non-zero exit status 1.
Tearing down lab to avoid leaving resources up...
...
Destroy complete! Resources: 1 destroyed.
Lab torn down.

There is a second blind spot in the same handler and it is quieter. _wait_for_ssh gives up after two minutes by calling sys.exit(1). That raises SystemExit, and SystemExit does not inherit from Exception:

>>> issubclass(SystemExit, Exception)
False

So it went straight past except Exception and no teardown ran. A box that boots but never opens port 22, which is what a bad image or a firewall mistake looks like, left a running server behind. This one was gentler than the apply case, because metadata was already saved, so lab status saw the lab and lab down worked. It still was not the behavior the tool advertised. The handler catches BaseException now, which is the one place that spelling is worth reaching for.

A dangling firewall costs nothing, so this is clutter rather than a bill. It matters because it breaks the promise the rest of the tool makes: that anything it created, it can also destroy. Here is the audit, which needs no local state and is the thing to reach for when the tool says it knows nothing:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" 'https://api.hetzner.cloud/v1/firewalls' \
  | python3 -c 'import json,sys; [print(f["id"], f["name"], len(f["applied_to"])) for f in json.load(sys.stdin)["firewalls"]]'

Anything with zero attachments and a lab name is mine, and it is safe to delete.

Gotchas worth knowing

tofu init fails with something that never mentions S3. Bad object storage credentials surface as an opaque init error. Run lab test-s3 first, always.

InvalidAccessKeyId when the keys look right. The S3 credentials belong to a different Hetzner project than the bucket. They are scoped per project and the console does not make that obvious.

SSH stops working mid lab. The firewall is pinned to the public IP you had when you ran up. Move from home WiFi to a phone hotspot and you are locked out of your own box. Recreate it, or widen the CIDR.

There was no state locking at all, and that is worse than a stuck lock. I assumed there was, and I was wrong, which is why I am spelling it out. The S3 backend only locks if you ask it to, and this backend config did not. On OpenTofu 1.10 and later that means adding use_lockfile=true, and on older setups it meant a DynamoDB table. With neither, two commands against the same lab do not queue and do not hang. They both write, and the loser silently corrupts the state file. Separate labs are genuinely safe because they have separate state objects. The same lab twice at once was not, and nothing warned you. One flag fixes it, and it is in the backend config now:

-backend-config="use_lockfile=true"

Deleting .lab/ orphans a running server. Destroy is driven from the local metadata file. Wipe the directory and the tool no longer knows the box exists, while Hetzner still does, and still bills for it. This is why the labels matter.

lab logs fails after a failed runbook. Expected. The teardown removed the local directory. The log is in S3.

Discord webhooks fail quietly. Post to a webhook URL and you get a 204 with no body whether or not anything sensible happened. Add ?wait=true and Discord returns the created message or a real error, which turns "did that work?" into a question with an answer.

Quick reference

lab status                                     # what exists, * marks current
lab up --ttl 1                                 # bare box, eligible for sweep after 1h
lab up --image ubuntu-26.04 --ttl 2 --runbook runbooks/x.py
lab up --ttl 1 --runbook r.py --keep-on-failure  # keep the box to debug
lab ssh                                        # interactive shell
lab ssh -c 'cloud-init status --wait'          # one shot command
lab logs --show                                # local runbook log
lab down                                       # destroy current lab
lab down lab-20260821-143000                   # destroy a specific one
lab sweep                                      # destroy everything expired
lab server-types --location nbg1               # only what you can actually order
lab test-s3                                    # check credentials first

And the one that does not need the tool at all, for when local state is gone and you need the truth:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
  'https://api.hetzner.cloud/v1/servers?label_selector=lab%3Dtrue'

The first thing I pointed this at was a 24.04 box and do-release-upgrade, which is a test I was never going to run on a machine I cared about. That one is written up in upgrading Ubuntu 24.04 to 26.04, and two of the three footguns above surfaced while I was building the box for it.

[ lab down ✓ ]

Originally published at peculiarengineer.com.

Backing up a homelab with restic

Minor Keith — Mon, 24 Aug 2026 13:30:49 +0000

I have written a lot here about running things at home and almost nothing about not losing them, which is a strange gap for someone who keeps telling you to self host your own Git server. So I sat down to work out how backups should actually be done, rather than how I had been vaguely assuming they were done.

The tool part turned out to be the easy half. The half that surprised me is that the obvious approach, point a backup tool at your data directory and let it run every night, produces a broken restore most of the time if anything in there is a database. I have numbers on that below, and they changed the shape of this post while I was writing it.

The whole thing is built around one idea. A backup nobody has restored is a rumor. So the restore comes first here, and everything else exists to make that restore possible.

TL;DR. Use restic, pointed at object storage you already pay for. The repository password is the backup: lose it and the data is gone with no recovery path, so store it somewhere separate from the machine. Never back up live database files. Dump them first, VACUUM INTO for SQLite and pg_dump for Postgres, then back up the dumps. Do not use SQLite's .backup on a busy database: it either fails instantly leaving a zero byte file or spins forever. Run it from a systemd timer, not cron. forget drops snapshots from the index and reclaims no space at all, prune is what actually frees it, and you need both. Then test a restore onto a machine that has never seen the data.

Why restic

One static binary, no daemon, no database of its own. It encrypts on your machine before anything leaves it, deduplicates across snapshots so the tenth backup of a 200 GB directory is not 2 TB, and it speaks S3 natively so it can point straight at object storage without a helper.

The alternatives, briefly, because I did look. Borg is excellent and has been around longer, but it has no native object storage support, so you end up mounting your bucket with rclone and now your backup depends on a FUSE mount behaving. Kopia is genuinely good and slightly nicer to use, just younger with less production mileage. rsync is not a backup, it is a file copy: no history, no integrity checking, and it will happily replicate your corruption over the good copy.

For a homelab where the destination is a bucket, restic is the boring correct answer.

Before you start

Everything here is on Ubuntu 26.04, and everything from here runs as root, which is also how the timer will run it. Drop into a root shell with sudo -i and the commands below are exactly what you type.

Install the two tools the post assumes, because neither is on a stock server:

apt update
apt install -y restic sqlite3
restic version

sqlite3 is the command line client, and you need it even if the application that owns the database bundles its own copy. If you are backing up Postgres in a container, pg_dump runs inside that container so there is nothing extra to install for it.

Make the directory the dumps land in now, because several commands below write into it:

mkdir -p /srv/dumps

The password is the backup

Before anything else, because restic init says it and people scroll past:

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

That is not boilerplate. There is no recovery, no support email, no key escrow. Encrypted with a password you do not have is identical to deleted.

So the password goes somewhere that is not the machine being backed up. A password manager, a piece of paper, another machine. Backing up your password manager's vault into the same restic repository, which I nearly did while setting this up, is a circle with no way in.

On the box it lives in a file only root can read, so it stays out of your shell history and out of ps:

install -m 600 /dev/null /root/.restic-password
read -rs -p 'passphrase: ' P && printf '%s' "$P" > /root/.restic-password && unset P

read -rs rather than putting the passphrase on the command line, because a command line ends up in .bash_history and in ps while it runs, which are the two places this file exists to keep it out of.

1. Create the repository

I am using Hetzner Object Storage because I already have a bucket there. Any S3 compatible storage works the same way.

Credentials and repository location go in an environment file, root readable only:

install -m 600 /dev/null /etc/restic.env

# /etc/restic.env
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export RESTIC_REPOSITORY="s3:https://nbg1.your-objectstorage.com/your-bucket/homelab"
export RESTIC_PASSWORD_FILE="/root/.restic-password"

The repository URL is worth reading carefully. It is s3: then the full endpoint URL including https, then the bucket, then a prefix. That prefix matters: it lets one bucket hold several unrelated repositories without them colliding.

Then initialize it once:

set -a; . /etc/restic.env; set +a
restic init

✅ You should see the password warning quoted above, followed by a line naming the repository it made:

created restic repository 461094b21b at s3:https://nbg1.your-objectstorage.com/your-bucket/homelab

2. Decide what to back up

The instinct is to back up everything. Resist it, because a backup you cannot afford to run daily is a backup you will turn off.

What actually needs to be in there:

Application data. Docker volumes and bind mounts, whatever /srv or /opt holds for you.
Database dumps, which is section 4 and the reason this post exists.
/etc. Small, and it is every decision you made about this machine. Restoring config from memory at 2 a.m. is how you discover you had forgotten three of them.
The list of what was installed. dpkg --get-selections into a file.

What does not need to be in there: package caches, anything under /proc, /sys or /tmp, and the backup repository itself if you keep a local one. Docker images do not need backing up either, because they come back from a registry.

Docker volumes are a trap here, and worth being precise about. I use bind mounts, so my container data sits under /srv and gets picked up by the paths above. If you use named volumes instead, which is the more common pattern, that data lives in /var/lib/docker/volumes and nothing above touches it. Either add that path to the backup or move to bind mounts. Skipping /var/lib/docker wholesale, which is the usual advice and which I very nearly wrote here, would leave you with a backup containing none of your actual data.

3. The part that will quietly break you

Here is the experiment that changed this post.

I set up a box with two databases being written to continuously: one in WAL mode, one in the older rollback journal mode, which is still SQLite's default. Which mode your applications use is worth checking rather than assuming, with sqlite3 yourapp.db 'PRAGMA journal_mode;', because as you will see it changes the odds a lot. Then I backed up their files live with restic, restored them, and asked SQLite whether the restored copy was intact.

The first run came back clean. integrity_check: ok. If I had stopped there, this section would have said live backups are fine.

So I ran it ten more times, and then twice more on different machines with different write patterns. Here is the third run, which is the one I would have least expected:

run  wal          journal
  1  malformed    malformed
  2  malformed    ok
  3  malformed    malformed
  4  malformed    malformed
  5  malformed    malformed
 ...
 10  malformed    malformed

WAL failures:     10/10
JOURNAL failures: 9/10
TOTAL:            19/20

malformed there is SQLite's own verdict on the restored file:

Error: stepping, database disk image is malformed (11)

Three runs, three answers:

Run	Rollback journal	WAL	Total
First machine, single row inserts	8 of 10 corrupt	3 of 10	12 of 20
Second machine, moderate load	9 of 10 corrupt	0 of 10	9 of 20
Third machine, batched transactions	9 of 10 corrupt	10 of 10	19 of 20

That spread is the finding, not any one number. Rollback journal mode is reliably terrible, around nine times in ten every time I tried. WAL mode ranged from never failing to failing every single time, depending on nothing more than how the writes were shaped. On the run where WAL never broke I could easily have concluded WAL mode is safe, published that, and been wrong.

So the useful statement is not a percentage. It is that a live file copy of a SQLite database restores to a corrupt file often, unpredictably, and in a way you cannot measure once and trust. It works just enough of the time to convince you.

Postgres behaved differently. Five of five live data directory restores came back working, every one crash recovering on startup:

LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 0/1917A20
LOG:  database system is ready to accept connections

Which makes sense once you say it out loud: a file level copy of a running Postgres looks exactly like someone pulled the power out, and surviving that is a thing Postgres is specifically built to do. I could not break it.

I am still going to tell you not to do it, for two reasons that have nothing to do with superstition. Upstream does not sanction copying a running cluster, because a busy one can be captured mid page write in a way a small test database never will be. And the dump of that same database was 19.6 MB against a 225 MB live data directory. Eleven times smaller, portable between versions, and blessed by the people who wrote it.

4. Dump the databases first

The correct approach costs almost nothing, which removes the last excuse.

SQLite. The obvious command is .backup, SQLite's own online backup, and it is what I recommended here until I tested it properly:

time sqlite3 /srv/app/data/app.db ".backup /srv/dumps/app.db"

real    0m0.089s

Eighty nine milliseconds, passes integrity_check, lovely. On an idle database.

Point it at a database that is actually being written to and it stops being lovely, because the SQLite backup API restarts from page zero every time an external writer commits. I hit two different failure modes on two different machines with the same command.

On one, it gave up instantly and left a file behind:

Error: database is locked
real    0m0.10s

-rw-r--r-- 1 root root 0 Aug 23 00:02 b1.db

Zero bytes. A script that does not check the exit code backs that up and reports success.

On the other, it never finished at all. Under systemd with a 45 second cap:

hang.service: start operation timed out. Terminating.
hang.service: Consumed 44.743s CPU time over 45.016s wall clock time

Forty five seconds of pegged CPU producing nothing, on a database of a few tens of megabytes. Without that timeout it spins for as long as writes keep arriving, and Type=oneshot means systemd waits patiently forever while your backups quietly stop happening.

Use VACUUM INTO instead. It takes one read transaction and writes a clean copy, without restarting when someone else commits:

time sqlite3 /srv/app/data/app.db "VACUUM INTO '/srv/dumps/app.db';"

Same box, same writer hammering it, same moment:

real    0m0.41s

integrity: ok
rows:      60412

Four tenths of a second, and a file that opens. It needs SQLite 3.27 or newer, which means anything from 2019 onwards, so on 26.04 it is simply there. The output file must not already exist, which is why the script in the next section writes to a temporary name and moves it into place.

Postgres. Dump in the custom format so you can restore selectively later:

docker exec app-pg-1 pg_dump -U postgres -d labdb -Fc -f /tmp/labdb.dump
docker cp app-pg-1:/tmp/labdb.dump /srv/dumps/labdb.dump

Then back up /srv/dumps, not the live directories.

5. The backup script

Everything above, in the order it has to happen: dump first, then back up the dumps.

Save this as /usr/local/bin/backup.sh and make it executable, which the next section will fail on if you forget:

The container name and database name below are mine. Change them, or delete those two lines if you have no Postgres.

#!/bin/bash
# /usr/local/bin/backup.sh
set -uo pipefail
set -a; . /etc/restic.env; set +a

mkdir -p /srv/dumps
failed=0

# Databases first, while everything is still running.
rm -f /srv/dumps/app.db.tmp
sqlite3 /srv/app/data/app.db "VACUUM INTO '/srv/dumps/app.db.tmp';" \
  && mv /srv/dumps/app.db.tmp /srv/dumps/app.db || failed=1

docker exec app-pg-1 pg_dump -U postgres -d labdb -Fc -f /tmp/labdb.dump \
  && docker cp app-pg-1:/tmp/labdb.dump /srv/dumps/labdb.dump || failed=1

# What was installed, so a rebuild is not archaeology.
dpkg --get-selections > /srv/dumps/packages.txt

restic backup /srv/dumps /srv/app /etc \
  --exclude /srv/app/data \
  --tag nightly || failed=1

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune || failed=1

exit $failed

Two decisions in there are deliberate and both came from watching this fail.

No -e. The obvious version starts set -euo pipefail, and I wrote it that way first. The problem is that set -e makes a single failed dump abort the entire script, so a stopped Postgres container means restic never runs and nothing gets backed up that night, including /etc and every file that had nothing to do with Postgres. The version above records the failure, backs up everything it still can, and exits non zero so the timer reports it. A partial backup beats no backup.

The dump goes to a temporary name and gets moved into place. If the dump dies halfway, mv never happens and yesterday's good copy stays where it is, rather than being replaced by a truncated file that restic will faithfully preserve forever. The rm -f before it matters too: VACUUM INTO refuses to write to a file that already exists, so a run killed between the vacuum and the move would otherwise fail every night afterwards until somebody deleted the leftover by hand.

Both restic calls are checked. This is the one I nearly shipped without. Wrapping only the dumps means a wrong password, an unreachable bucket, or a repository locked by a prune leaves failed at zero, the unit green, and no backup taken. Restic's exit codes are specific enough to rely on:

clean run                     exit 0
some files could not be read  exit 0   (snapshot still created)
wrong password                exit 12
repository does not exist     exit 10

A file vanishing mid backup, which happens constantly with sockets and pid files, does not fail the run. A broken destination does.

chmod +x /usr/local/bin/backup.sh

The --exclude /srv/app/data is the point of section 3: the live database directory is deliberately not in the backup, because the good copy is already in /srv/dumps.

6. Run it from a systemd timer

Cron works. A timer is better here because it gives you systemctl status on the last run, journal output kept with the unit, and Persistent=true, which runs a missed job after a machine was off rather than skipping the night.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly restic backup
Wants=network-online.target
After=network-online.target docker.service

[Service]
Type=oneshot
TimeoutStartSec=30min
Environment=HOME=/root
ExecStart=/usr/local/bin/backup.sh

# /etc/systemd/system/backup.timer
[Unit]
Description=Run the restic backup nightly

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now backup.timer
systemctl list-timers backup.timer

RandomizedDelaySec matters more than it looks. Everyone picks a time on the hour, and object storage providers notice.

TimeoutStartSec is the backstop for section 4. A oneshot service with no timeout will sit in activating indefinitely if something inside it wedges, and you find out weeks later. With it set, systemd kills the run and marks the unit failed, which is a state you can actually alert on.

Run it once by hand before trusting the schedule:

systemctl start backup.service
journalctl -u backup.service -n 30 --no-pager

7. Retention, and why forget is not enough

restic forget removes snapshots from the index. restic prune removes the data those snapshots referenced. They are separate operations, and running only the first is a very popular way to have a retention policy that does nothing.

Watch what each one does. restic stats --mode raw-data reports the size of the repository itself, which works against object storage where du does not:

restic snapshots --compact | tail -1     # 16 snapshots
restic stats --mode raw-data             # 98 MiB

restic forget --keep-last 3
restic snapshots --compact | tail -1     # 3 snapshots
restic stats --mode raw-data             # 98 MiB

restic prune
restic stats --mode raw-data             # 26 MiB

Sixteen snapshots became three and not one byte was freed. The space came back only after prune. So either pass --prune to forget, as the script above does, or run prune on its own schedule.

One thing that confused me the first time: --keep-last 3 keeps three snapshots per group, and restic groups by host and paths by default. Back up two different path sets and you keep three of each, so the count you land on may be higher than the number you typed.

8. The restore, which is the whole point

Everything so far is preparation. This is the only part that proves anything, and it is the part nobody does.

Do it on a machine that has never seen this data, with nothing but the repository URL and the password. That is the actual disaster scenario: the original box is gone.

That machine needs restic installed, plus sqlite3 and Docker if you want to check the dumps the way I do below. A throwaway cloud box is ideal, because you want somewhere with genuinely none of your data on it, and you want to destroy it afterwards.

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export RESTIC_REPOSITORY="s3:https://nbg1.your-objectstorage.com/your-bucket/homelab"
export RESTIC_PASSWORD="..."

restic snapshots
restic restore latest --target /bare --include /srv/dumps

Then check what came back, rather than admiring the file listing:

sqlite3 /bare/srv/dumps/app.db 'PRAGMA integrity_check;'

ok

and put the Postgres dump into a fresh server to prove it loads:

docker run -d --name pgr -e POSTGRES_PASSWORD=x postgres:17

# Wait for it. `docker run -d` returns long before Postgres accepts connections,
# and running createdb immediately gives you a confusing socket error.
until docker exec pgr pg_isready -U postgres -q; do sleep 1; done

docker cp /bare/srv/dumps/labdb.dump pgr:/tmp/d.dump
docker exec pgr createdb -U postgres restored
docker exec pgr pg_restore -U postgres -d restored /tmp/d.dump
docker exec pgr psql -U postgres -d restored -tAc 'select count(*) from t;'

Your number will differ, obviously. That it is a number at all is the entire point of the post. Not that the backup ran, that the data came back and could be queried on a machine that had never seen it.

Pulling a single file out of an older snapshot is the other restore worth practicing, because it is the one you will actually use:

restic snapshots                     # copy an ID from the first column
restic restore <id> --target /tmp/oops --include /etc/fstab

And confirm the repository itself is readable, which is a different question from whether the last backup succeeded:

restic check

check snapshots, trees and blobs
no errors were found

Put restic check on a monthly timer. A repository that cannot be read is not a backup either, and you want to find that out on a Tuesday rather than during a rebuild.

Gotchas I hit

A single successful restore proves nothing. My first live database restore came back clean. Across three machines the same test then failed between nine and nineteen times out of twenty. Test more than once, and test the thing you actually care about.

forget frees no space. Covered above, and worth repeating because a retention policy that never prunes looks like it is working right up until the storage bill.

Two backups at once do not collide. I expected a repository lock error and got two clean snapshots instead, because backups take a non exclusive lock. prune is the one that takes an exclusive lock, so the collision you can actually hit is a manual backup landing while the timer is pruning.

--unsafe-allow-remove-all refuses to run bare. Trying to empty a repository gives you Fatal: --unsafe-allow-remove-all is not allowed unless a snapshot filter option is specified, so it needs something like --host or --tag alongside it.

Restic reports a warning and still exits zero when a file vanishes mid backup, which happens constantly with sockets and pid files. at least one source file could not be read is worth reading, not worth alerting on.

Under systemd, restic complains about a cache it cannot find. unable to open cache: unable to locate cache directory: neither $XDG_CACHE_HOME nor $HOME are defined, on every run. It is harmless, it just means no local cache and so slower metadata operations. Set Environment=HOME=/root in the service unit to quiet it and get the cache back.

backup.service has no [Install] section, on purpose. It is started by the timer, so systemctl enable backup.service is not a thing you want. Enable backup.timer instead. systemctl status backup.service still works for reading the last run.

That shrinkage on the first snapshot is compression, not dedup. 2.7 MiB of files stored as 874 KiB, with no previous snapshot to deduplicate against. Restic creates version 2 repositories now and compresses by default. Dedup is real too, it just needs a second snapshot before it has anything to work with.

Quick reference

set -a; . /etc/restic.env; set +a      # load credentials into the shell

restic init                            # once, at the start
restic backup /srv/dumps /etc --tag nightly
restic snapshots                       # what have I got
restic snapshots --compact             # the same, readable

restic restore latest --target /bare                        # everything
restic restore <id> --target /tmp/x --include /etc/hosts    # one file
restic check                                                # is the repo sound

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

rm -f /srv/dumps/app.db.tmp                                 # VACUUM INTO wants a free path
sqlite3 live.db "VACUUM INTO '/srv/dumps/app.db.tmp';"      # not cp, and not .backup
docker exec pg pg_dump -U postgres -d mydb -Fc -f /tmp/d.dump && docker cp pg:/tmp/d.dump /srv/dumps/

I went into this assuming the hard part was picking a tool. The hard part was finding out that the obvious way to back up a database works often enough to fool you. Go and restore something.

[ restored ✓ ]

Originally published at peculiarengineer.com.

Running Kimi Code on Claude Code

Minor Keith — Thu, 13 Aug 2026 20:40:23 +0000

I pay for Claude and I pay for Kimi Code, and I want both available from the same terminal without choosing one in the morning. Kimi publishes a guide for pointing Claude Code at their endpoint. It tells you to export a handful of environment variables in your shell profile, which works, and which also routes every Claude Code session on the machine to Kimi from then on. Your Anthropic subscription sits there unused while you wonder why Opus got worse.

The fix is boring: a second settings file and a shell function that loads it. Ten minutes. The part worth writing down is that the headline model ID in Kimi's guide is rejected by Kimi's own endpoint, and it's rejected with a 401 that says your credentials are bad. I lost time auditing a key that was never the problem. Then I found a second version of the same lie living in ~/.claude.json, and that one is better disguised.

TL;DR. Put the Kimi config in ~/.claude/kimi-settings.json rather than ~/.claude/settings.json, then add a claude-kimi() function that runs claude --settings "$HOME/.claude/kimi-settings.json" "$@". Base URL is https://api.kimi.com/coding/ with the key in ANTHROPIC_API_KEY. The model is k3, not k3[1m], whatever the docs say. Set CLAUDE_CODE_MAX_CONTEXT_TOKENS to 1048576 because a bare k3 gives Claude Code no window to infer. If interactive mode 401s while curl and claude -p both work, your key is in the rejected list in ~/.claude.json and no amount of key rotation will fix it.

Kimi has two API surfaces and only one of them is yours

This is the distinction that decides whether anything else in this post works, so get it straight first.

A Kimi Code subscription authenticates against https://api.kimi.com/coding/, using ANTHROPIC_API_KEY, which is sent as an x-api-key header. Model IDs are short: k3, k3-256k, kimi-for-coding.

The Moonshot platform is a different thing. It lives at https://api.moonshot.ai/anthropic, uses ANTHROPIC_AUTH_TOKEN as a bearer token, and wants model IDs with a kimi- prefix. That is the surface you pay for by the token. It's also what most of the search results describe.

They are different products. Your subscription key will not authenticate against the platform host, and the failure is a flat 401 that tells you nothing about why. If you follow the popular guide with a subscription key, this is where you stop.

1. Get a key

Go to kimi.com/code/console and create one.

Two things worth knowing before you click. You get at most five keys, and each is shown exactly once, at creation. There is no reveal button later, only delete and make a new one. Copy it somewhere before you close the dialog.

2. Write the settings file

Create it empty and lock it down before the key goes anywhere near it, so it's never briefly readable by anything else on the machine:

touch ~/.claude/kimi-settings.json
chmod 600 ~/.claude/kimi-settings.json

Then paste this in, replacing the placeholder:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.kimi.com/coding/",
    "ANTHROPIC_API_KEY": "PASTE_YOUR_KIMI_CODE_KEY_HERE",

    "ANTHROPIC_MODEL": "k3",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "k3",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "k3",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "k3",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "kimi-for-coding",
    "CLAUDE_CODE_SUBAGENT_MODEL": "k3",

    "ANTHROPIC_DEFAULT_OPUS_MODEL_NAME": "Kimi K3",
    "ANTHROPIC_DEFAULT_OPUS_MODEL_DESCRIPTION": "Kimi Code K3 (1M context)",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL_NAME": "Kimi K2.7 Code",

    "CLAUDE_CODE_MAX_CONTEXT_TOKENS": "1048576",
    "CLAUDE_CODE_AUTO_COMPACT_WINDOW": "950000",
    "CLAUDE_CODE_EFFORT_LEVEL": "high"
  }
}

Check the permissions again after you save. Some editors write a temp file and rename it into place, which lands at your umask and quietly undoes the chmod you just ran:

ls -l ~/.claude/kimi-settings.json   # want -rw-------

There are six model slots, not one, and that's deliberate. Claude Code uses the haiku slot for background chores like naming your conversation, and pointing those at K3 spends your quota on titles. kimi-for-coding is ungated on every tier, so it's the safe floor for throwaway work.

The two window numbers are the part people get wrong. CLAUDE_CODE_AUTO_COMPACT_WINDOW is a trigger threshold, not a statement of how much context you have, so it needs headroom underneath the real window. Kimi's guide sets it to 1048576, which is out of range anyway: Claude Code accepts 100000 to 1000000. It also has to be a plain integer. Write 500k and it parses as 500 and clamps to the floor, and you'll be wondering why compaction fires roughly immediately.

3. The model ID in the official guide does not work

Kimi's own Claude Code guide tells you to set ANTHROPIC_MODEL to k3[1m]. Their endpoint rejects it. I sent each ID at api.kimi.com/coding/ with curl, and here's what came back:

Model ID	HTTP	What comes back
`k3`	200	Works. Main conversation model
`k3-256k`	200	Works. Smaller window variant
`kimi-for-coding`	200	Works. Ungated on all tiers
`k3[1m]`	401	`Your model id does not exist, recognized as other:k3[1m]. Please set model id as k3.`

Read that 401 again. It's an authentication_error, and the message body is about a model ID. Nothing in the status code suggests you should go looking at your model configuration, so you go and audit your key instead, and your key is fine. It cost me more time than anything else in this setup.

Use the bare k3. It is natively a 1M context model and the suffix was never what unlocked that.

There is a consequence though, and it's why CLAUDE_CODE_MAX_CONTEXT_TOKENS is in the config above rather than being decorative. With a [1m] style ID, Claude Code can read the window size out of the name. A bare k3 tells it nothing, so it falls back to a conservative guess and starts compacting your session far earlier than it needs to. Declaring the real number fixes that.

If you're on k3-256k instead, change ANTHROPIC_MODEL, ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL, ANTHROPIC_DEFAULT_FABLE_MODEL and CLAUDE_CODE_SUBAGENT_MODEL to k3-256k, leave the haiku slot alone, and set CLAUDE_CODE_MAX_CONTEXT_TOKENS to 262144 with the compact window at 240000. I'm listing all five by name on purpose. Forget CLAUDE_CODE_SUBAGENT_MODEL and your subagents keep running on the 1M model while you believe you moved everything down.

4. Add the launcher

cat >> ~/.zshrc <<'EOF'

# Claude Code against the Kimi Code subscription.
# Plain `claude` is unaffected and keeps the claude.ai subscription.
claude-kimi() {
  command claude --settings "$HOME/.claude/kimi-settings.json" "$@"
}
EOF

exec zsh

Use $HOME rather than ~ so the path resolves however the function gets called. The --settings flag layers that file on top of your normal user config, and because ~/.claude/settings.json has no env block on my machine, there's nothing for it to collide with. Check yours before you assume the same:

python3 -c "import json,os;print('env' in json.load(open(os.path.expanduser('~/.claude/settings.json'))))"

If that prints True, whatever is in there will fight with the Kimi config, and you should move it out first.

5. Test the endpoint before you test Claude Code

Confirm the key, host and model at the HTTP layer while there's only one thing that can be wrong. Then any failure in the next step is a Claude Code problem, and you already know it isn't credentials.

read -rs "KEY?Kimi Code key: "; echo

curl -sS https://api.kimi.com/coding/v1/messages \
  -H "x-api-key: $KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"k3","max_tokens":16,
       "messages":[{"role":"user","content":"Say OK."}]}'

You want JSON with a content array in it. read -rs keeps the key out of your shell history, and note the header is x-api-key rather than a bearer token, which is the practical tell that you're on the subscription surface and not the platform one. Run unset KEY when you're finished with it.

Two notes on this. If you get a 404 on everything, your base URL includes the full path and Claude Code is appending v1/messages to something that already ends in it. Set the base, keep the trailing slash, nothing more.

And k3 needs the Moderato tier or above. If it comes back with a 401 naming the model and linking an upgrade page, that's tier gating rather than a bad ID, and you want every slot on kimi-for-coding instead. Both failures arrive as authentication_error, so the status code can't tell them apart and the message body is the only thing that can.

A curl probe here is fine, by the way. Kimi's terms require tools to identify themselves honestly, and curl does. Forging a User-Agent to make one client look like another is the thing they take issue with.

6. Confirm both lanes

In one tab:

claude-kimi
# then, inside the session:
/status

/status should show https://api.kimi.com/coding/ and a Kimi model. Send it a real message so you've exercised the round trip rather than just the startup path.

Then open a separate tab and run plain claude, and check /status there too. This is the whole point of the exercise, so actually do it. Verifying that the isolation held is more important after a Claude Code upgrade than it is today: model variable precedence changed in v2.1.195, and the next release that touches settings layering is not going to send you a note.

Expect one prompt on the first interactive claude-kimi, asking whether to use the detected API key. Say yes.

The rejected key list, which is the worst one

Answer no to that prompt, deliberately or with a stray keypress on a dialog you weren't expecting, and Claude Code writes the key's last 20 characters into a rejected list in ~/.claude.json. From then on, interactive sessions refuse it and tell you this:

Please run /login · API Error: 401

Here is why this one is genuinely nasty. None of the obvious diagnostics consult that list, so every one of them clears the key. I had curl returning 200, claude -p "hello" returning 200, a config file that was correct, an endpoint that was up, and a key that was valid. Only interactive mode failed, and it failed with a message pointing at my credentials.

Do not run /login here. The error asks you to, and it's wrong, and running it inside the Kimi lane is the one action in this whole setup that can disturb the Anthropic credential you were trying to protect.

Check for it instead:

python3 -c "
import json,os
d=json.load(open(os.path.expanduser('~/.claude.json')))
print(json.dumps(d.get('customApiKeyResponses',{}),indent=2))"

If your key's tail is sitting under rejected, move it across:

python3 - <<'PY'
import json, os, shutil
p = os.path.expanduser('~/.claude.json')
shutil.copy(p, p + '.bak-apikey')
d = json.load(open(p))
key = json.load(open(os.path.expanduser(
    '~/.claude/kimi-settings.json')))['env']['ANTHROPIC_API_KEY']
tail = key[-20:]
r = d.setdefault('customApiKeyResponses', {'approved': [], 'rejected': []})
r['rejected'] = [x for x in r.get('rejected', []) if x != tail]
if tail not in r.setdefault('approved', []):
    r['approved'].append(tail)
json.dump(d, open(p, 'w'), indent=2)
print('approved:', r['approved'])
PY

Close every Claude Code session before you run that, since ~/.claude.json is live state and you don't want two writers. Then start fresh, because the list is read at launch. If the prompt appears again, accept it this time.

While we're on things read at launch: settings are too. Editing kimi-settings.json does nothing to a session that's already open, and a session started before your last edit will keep sending the old model ID and keep 401ing on every turn, with an error that once again blames the key. To find stale ones:

ps -eo pid,lstart,command | grep kimi-settings
stat -f '%Sm' ~/.claude/kimi-settings.json

Compare the two and kill anything older than the file.

Living with the request quota

This is the part that changes how you work, and it isn't the setup. Your plan is metered in requests per rolling five hour window, somewhere around 300 to 1,200 depending on tier, with up to 30 running at once. Tokens are not the constraint. Round trips are.

Claude Code burns through that faster than a chat client does, because one instruction can fan out into a lot of requests. Subagents are the big one, since each agent runs its own request loop and several can run concurrently, so a single "go investigate this" can cost dozens. Workflows and ultracode fan out hard enough to approach the 30 concurrent ceiling on their own, which is reason enough to keep them off this lane. Raising the effort level means more tool calls per turn, and every tool call is a request. Background chores are small but constant, which is exactly why the haiku slot points at kimi-for-coding rather than K3.

One quirk on effort. Kimi Code collapses Claude Code's five effort levels onto K3's three, so medium and xhigh land in the same place as their neighbors. Tune between low, high and max and don't expect the in between settings to do anything.

Gotchas worth knowing

Plain claude also goes to Kimi. The variables reached your global environment. Run env | grep ANTHROPIC, and check you didn't paste the config into ~/.claude/settings.json out of habit.

A warning that claude.ai connectors are disabled. Expected, and actually a good sign. An API key outranks your claude.ai login for that session, which is the isolation working. The other tab is unaffected.

WebFetch says it's temporarily unavailable. The endpoint doesn't implement it. Paste the content in, or use an MCP scraping server. WebSearch still works on K3.

/model shows unhelpful names. That's what the _NAME and _DESCRIPTION keys in the config are for. You can also set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 and let the picker populate itself from the endpoint's model list.

Timestamps in the console request log are UTC+8, and a request that failed on a bad model ID logs the model as -. A row with a dash in the model column is a rejected ID, not a credential fault, whatever the status column says.

400 invalid thinking: only type=enabled is allowed. The K2.7 Code models used to reject requests that arrived without thinking enabled. I retested on 12 August 2026 and it no longer reproduces, with kimi-for-coding returning 200 to a request carrying no thinking block at all. I'm leaving it here because my retest used a minimal raw HTTP body rather than Claude Code's real request shape, and I'd rather you recognize it than rediscover it. If you do hit it, stay on K3.

Quick reference

Variable	Value	What it does
`ANTHROPIC_BASE_URL`	`https://api.kimi.com/coding/`	Where requests go. Base only, keep the trailing slash
`ANTHROPIC_API_KEY`	your Kimi Code key	Sent as `x-api-key`. Outranks the claude.ai login for this session
`ANTHROPIC_MODEL`	`k3`	Main conversation. No `[1m]` suffix
`ANTHROPIC_DEFAULT_OPUS_MODEL`	`k3`	What the `opus` alias resolves to
`ANTHROPIC_DEFAULT_SONNET_MODEL`	`k3`	What the `sonnet` alias resolves to
`ANTHROPIC_DEFAULT_FABLE_MODEL`	`k3`	What the `fable` alias resolves to
`ANTHROPIC_DEFAULT_HAIKU_MODEL`	`kimi-for-coding`	Background chores. Ungated on every tier
`CLAUDE_CODE_SUBAGENT_MODEL`	`k3`	Subagents and workflow agents. Your main quota burn
`CLAUDE_CODE_MAX_CONTEXT_TOKENS`	`1048576`	Required. A bare `k3` gives Claude Code nothing to infer from
`CLAUDE_CODE_AUTO_COMPACT_WINDOW`	`950000`	Compaction trigger. `100000` to `1000000`, plain integer
`CLAUDE_CODE_EFFORT_LEVEL`	`high`	Kimi collapses five levels onto three

Backing it out

rm ~/.claude/kimi-settings.json
# then delete the claude-kimi() block from ~/.zshrc
exec zsh

Your claude.ai login was never touched, so claude keeps working the whole time. To pause Kimi without uninstalling anything, just stop typing claude-kimi.

One honest caveat on the cleanup. If you accepted that API key prompt, the last 20 characters of your key are still in ~/.claude.json under customApiKeyResponses, and if you ran the repair script there's a ~/.claude.json.bak-apikey next to it. Neither is dangerous, both are residue, and you'll want to clear them out if you're rotating the key or handing the machine on.

The one thing to remember

Every failure in this setup arrives dressed as an authentication error. A rejected model ID, a model above your tier, and a key you once declined all surface as 401, and two of them will send you to the console to rotate a credential that was never broken. Read the message body, not the status code. That's the habit worth keeping.

[ two lanes, one terminal ]

Originally published at peculiarengineer.com.

Kubernetes, Part Three: GitOps with Argo CD, let Git drive your cluster

Minor Keith — Wed, 05 Aug 2026 02:58:58 +0000

In Part One I deployed nginx on a single k3s node and killed a Pod to watch a replacement appear. In Part Two I pulled the config and the passwords out of the image. Both parts had the same weak spot, and it's the one nobody mentions when they teach you kubectl: I was still standing at a terminal typing kubectl apply at a cluster.

That works fine until it doesn't. A month later nobody can tell you what's actually running, or who changed it, or what the cluster looked like before someone "just quickly fixed" something at 11pm. The YAML on your laptop and the YAML in the cluster quietly drift apart, and the only way to find out is to go and look.

Argo CD fixes that by taking the terminal away from you. You put your manifests in Git, you tell Argo CD where that repo is, and from then on the repo is the truth. Change the cluster by hand and Argo CD changes it back. I ran the whole thing on the same 4GB box that already hosts my Forgejo instance, and I'm writing it down because six separate things bit me and I'd rather not rediscover any of them.

TL;DR Put your manifests in a Git repo first. Install Argo CD with kubectl apply --server-side (plain apply fails on one oversized CRD). Create an Application object that names three things: which repo, which folder, where it goes. Sync it once by hand and watch OutOfSync become Synced. Then set syncPolicy.automated with selfHeal: true and prune: true, and try to fight it: scale a Deployment, delete a Service, swap an image. It puts all of them back within fifteen seconds. Deploy by pushing to Git, roll back with git revert. And remember that Synced means "matches Git", not "works".

The one idea worth holding onto

Part One's idea was that you declare the state you want and Kubernetes makes reality match it. Part Three is the same sentence moved one level out:

You declare the state you want in Git. Argo CD makes the cluster match it.

Kubernetes watches its own objects. Argo CD watches your repo. That's the entire product, and every feature below is a variation on it.

What you get for free is the part I didn't expect. Because every change to the cluster is now a commit, Git becomes your audit log, your rollback mechanism, and your review gate without you building any of those things. "Who scaled this to five replicas and why" stops being an unanswerable question.

Before you start

You need a working cluster and a kubectl that reaches it without sudo. That's Part One, Step 0, the KUBECONFIG fix that catches everyone. You also need a GitHub account, and about forty minutes.

Mine is a single k3s node, v1.36.2, on a 4GB Hetzner box in Falkenstein that was already running Forgejo in Docker. Argo CD cost me about 500MB across seven Pods, taking the box from 2.3GB free to 1.8GB free. It fits on a small machine comfortably. Nothing got OOM killed.

One warning that matters if your box has a public IP like mine does. Argo CD ships with an admin account and a bootstrap password, and Step 6 is where I make sure the web UI is reachable from my devices and from nowhere else. Don't skip it and don't reorder it.

Step 1: the repo comes first

This ordering is the whole philosophy in miniature, so do it in this order even though it feels backwards: the repo exists before Argo CD does.

Create a new public repo. Mine is peculiarengineer-gitops, and it holds exactly two files under apps/hello/.

Save the first as apps/hello/deployment.yaml:

# A Deployment says "keep N copies of this container running, forever."
# Argo CD's job is to make sure this file and the cluster always agree.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  replicas: 2 # change this number, push, and watch Argo CD notice
  selector:
    matchLabels:
      app: hello # which Pods this Deployment owns
  template:
    metadata:
      labels:
        app: hello # must match the selector above, or nothing happens
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            # Small box, so be explicit about what this is allowed to eat.
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi

And the second as apps/hello/service.yaml:

# A Service gives the Pods above one stable address inside the cluster.
# ClusterIP means "reachable from inside the cluster only", so nothing is
# published to the internet by this file.
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: ClusterIP
  selector:
    app: hello # sends traffic to any Pod carrying this label
  ports:
    - port: 80
      targetPort: 80

Nothing new here. This is Part One's Deployment and Service with resource limits added, because a 4GB box deserves them. Commit and push both.

💡 What happened: You wrote down what you want running without running any of it. From here on, editing these files is how you change the cluster. That's the habit the rest of the post builds.

Step 2: install Argo CD

Argo CD installs into the cluster as ordinary Kubernetes objects in its own namespace. No operator, no Helm required, no packages on the host.

▶ Do:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

❌ And that fails. Most of it applies, then right at the end:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

This one confused me for a minute because the error blames the CRD, and the CRD is fine. kubectl apply stashes a copy of the entire manifest it just applied into a last-applied-configuration annotation so it can compute diffs later. The ApplicationSet CRD is bigger than the 256KB limit Kubernetes puts on annotations, so the copy can't be stored.

Server side apply doesn't use that annotation at all. It hands the whole document to the API server and lets it track ownership properly.

▶ Do:

kubectl apply -n argocd --server-side -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

✅ You should see three CRDs land:

customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io serverside-applied

Now wait for the Pods. Seven of them come up, and on my box that took about forty seconds:

kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

💡 What happened: Argo CD is now a workload in your cluster like any other, watching for Application objects that don't exist yet. Worth noticing that it installed by declaring a pile of YAML, which is the same trick it's about to do on your behalf.

Step 3: the CLI and the admin password

Grab the CLI on the node. This is the Linux x86 build, so if you'd rather run it from a Mac or an ARM box, take the matching binary from the releases page.

▶ Do:

sudo curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo chmod +x /usr/local/bin/argocd
argocd version --client --short

✅ You should see a version line and nothing else:

argocd: v3.4.5+564b949

The installer generates a random admin password and leaves it in a Secret. Reading it is a callback to Part Two, base64 and all:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ; echo

Same reminder as Part Two: base64 is encoding, not encryption. You just decoded a password with a pipe. Step 12 replaces this one properly.

Step 4: the Application, and the moment it clicks

This is the object the whole product hangs off, and it's smaller than you'd think. An Application answers three questions and nothing else. Which repo. Which folder inside it. Where the result goes.

You can create it with argocd app create, but write the YAML instead. Seeing the object is what makes the idea stick. Save it as hello-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd # Applications live in the argocd namespace
spec:
  project: default
  source:
    repoURL: https://github.com/peculiarengineer-mk/peculiarengineer-gitops.git
    targetRevision: main # which branch to track
    path: apps/hello # which folder in the repo to apply
  destination:
    server: https://kubernetes.default.svc # this same cluster
    namespace: default # where the manifests land
  # No syncPolicy yet. This first one syncs only when we tell it to.

Swap in your own repo URL. Leaving syncPolicy out is deliberate, and it's the best decision in this whole tutorial, because it lets you see the next bit.

▶ Do:

kubectl apply -f hello-app.yaml
sleep 5
kubectl get application hello -n argocd \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status

✅ You should see two words that are the entire pitch:

NAME    SYNC        HEALTH
hello   OutOfSync   Missing

💡 What happened, and this is the paragraph I'd underline. Nobody asked it to do that. In the five seconds since you created the object, Argo CD cloned your repo, read apps/hello, compared both files against the live cluster, and reported the gap: OutOfSync means Git and the cluster disagree, Missing means those objects don't exist yet. It will keep doing that comparison forever, whether or not you ever sync anything.

The sync is almost an afterthought. The diff is what you're actually buying.

Step 5: the first sync

The argocd CLI has two modes and they behave differently, which caused me the second gotcha. --core talks straight to the Kubernetes API with no Argo CD server in the middle. It's the quickest way to work from the node.

▶ Do:

argocd app sync hello --core

❌ And it fails with a message that is actively misleading:

{"level":"fatal","msg":"configmap \"argocd-cm\" not found"}

Go and look and argocd-cm is sitting right there in the argocd namespace. The problem is that core mode has no server session telling it where Argo CD lives, so it reads the namespace from your kubeconfig context. On k3s that context has no namespace set, so it looks in default and finds nothing.

Setting ARGOCD_NAMESPACE does not help. I tried, twice.

The obvious fix is kubectl config set-context --current --namespace=argocd, and that's the third gotcha, because it poisons every kubectl command you run on that box afterwards. Suddenly kubectl get pods shows you Argo CD's internals instead of your app, forever, and you will not remember why. Use a separate kubeconfig for the argocd CLI instead.

▶ Do:

cp ~/.kube/config ~/.kube/argocd-core.yaml
KUBECONFIG=~/.kube/argocd-core.yaml kubectl config set-context --current --namespace=argocd
KUBECONFIG=~/.kube/argocd-core.yaml argocd app sync hello --core

That copies the kubeconfig Part One had you set up, so you're not going back to the root owned /etc/rancher/k3s/k3s.yaml that Step 0 told you to stop touching.

✅ You should see:

Sync Status:        Synced to main (d1395f1)
Phase:              Succeeded
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME   STATUS  HEALTH       MESSAGE
       Service     default    hello  Synced  Healthy      service/hello created
apps   Deployment  default    hello  Synced  Progressing  deployment.apps/hello created

Check what landed:

kubectl get deploy,pods -l app=hello -n default

deployment.apps/hello   2/2   2   2   22s
pod/hello-5d6b7fbfc4-f6kxf   1/1   Running   0   23s
pod/hello-5d6b7fbfc4-p4x64   1/1   Running   0   23s

That KUBECONFIG= prefix is needed on every argocd command from here on, and typing it each time gets old fast. Set an alias for the rest of the session:

alias argocd='KUBECONFIG=~/.kube/argocd-core.yaml argocd'

Every argocd command below assumes you did that. If one of them ever comes back with configmap "argocd-cm" not found, you're in a new shell and the alias is gone.

💡 What happened: You never ran kubectl apply on those manifests. Argo CD read GitHub and created both objects itself. That's a small thing on two files and a very large thing on two hundred.

Step 6: reach the UI without publishing it to the internet

The CLI does everything, but the web UI is genuinely good and the resource tree is the bit that makes Kubernetes legible to people who don't live in it.

Here's the trap. A k3s NodePort binds to 0.0.0.0, which on a box with a public IP means the open internet. Argo CD has a known admin username and a bootstrap password. So the firewall goes on first and the service gets exposed second. If you get that order backwards you spend a few minutes with an unauthenticated door open, and that's plenty.

My box wasn't on my tailnet at all. Only the Forgejo container was, through its sidecar. So the host joins properly:

curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --hostname=k3s-1 --accept-dns=false --accept-routes=false

Both of those flags are deliberate. --accept-dns=false stops Tailscale rewriting /etc/resolv.conf, and --accept-routes=false stops it touching the routing table. On a box already running k3s and Docker I want Tailscale to add an interface and change nothing else.

Now the firewall, and here's the fourth gotcha, which is the one that can genuinely take your services down. Ubuntu ships this in /etc/default/ufw:

DEFAULT_FORWARD_POLICY="DROP"

Enable ufw with that set and you break pod to pod networking in k3s and Docker's bridge. On my box that would have taken Forgejo down as collateral. Fix it before you enable anything:

sed -i 's/^DEFAULT_FORWARD_POLICY=.*/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw

Before you turn the firewall on, arm a dead man switch. Enabling a firewall over SSH on a box whose only access is SSH is how people lose servers, and a Hetzner rescue console at midnight is a bad time.

▶ Do:

setsid nohup bash -c "sleep 300; ufw --force disable" >/dev/null 2>&1 < /dev/null &
echo $! > /root/ufw-deadman.pid

That undoes the firewall in five minutes unless you cancel it. Now the rules. SSH, the tailnet, and the k3s internal networks:

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow in on tailscale0
ufw allow 41641/udp        # tailscale wireguard
ufw allow in on cni0       # k3s pods
ufw allow in on flannel.1
ufw allow in on docker0
ufw allow from 10.42.0.0/16   # k3s pod cidr
ufw allow from 10.43.0.0/16   # k3s service cidr
ufw --force enable

Open a second SSH session and confirm you can still get in. Check your cluster and anything else on the box is still healthy. Then cancel the timer using the PID file:

kill $(cat /root/ufw-deadman.pid)

Use the PID file, not pkill -f "sleep 300". I tried the pkill version and it killed my own shell, because the command line running that pkill contains the string sleep 300 and therefore matches itself. Exit code 255, dropped connection, and a few seconds of wondering whether I'd just locked myself out.

Now expose the UI:

kubectl -n argocd patch svc argocd-server -p '{"spec":{"type":"NodePort","ports":[
  {"name":"http","port":80,"targetPort":8080,"protocol":"TCP"},
  {"name":"https","port":443,"targetPort":8080,"protocol":"TCP","nodePort":30443}]}}'

If you go looking for a listening socket you won't find one, and that threw me for a second:

ss -tlnp | grep 30443    # returns nothing, and that's correct

kube-proxy implements NodePorts with iptables rules, not a bound socket. The test that actually matters is reaching it from two places:

https://100.87.x.y:30443  ->  HTTP 200     # over the tailnet
https://203.0.113.88:30443  ->  HTTP 000     # public IP, blocked

💡 What happened: The UI is reachable from every device on my tailnet, including the iPad, and invisible from everywhere else. The cert is self signed so your browser will complain once. Log in as admin with the password from Step 3, click into the hello app, and you get the resource tree: Application, Deployment, ReplicaSet, Pods, each with a health dot. That view is why people keep the UI around.

Step 7: break the cluster on purpose

Now the interesting part. Do the exact thing GitOps is supposed to prevent.

▶ Do:

kubectl scale deployment hello -n default --replicas=5
sleep 15
kubectl get application hello -n argocd \
  -o custom-columns=SYNC:.status.sync.status,HEALTH:.status.health.status

✅ You should see:

SYNC        HEALTH
OutOfSync   Healthy

Five Pods running, and Argo CD spotted it within seconds and did nothing at all.

That's not a bug, it's the lesson. Detecting drift and correcting drift are separate features, and right now you only have the first one. Seeing Argo CD notice and deliberately sit on its hands is what makes the next step land.

Healthy sitting next to OutOfSync is worth its own thought too. Health and sync are different axes. Health means the five Pods are fine. Sync means this matches Git. An app can be in perfect health and completely wrong.

Ask what's wrong and it tells you exactly:

argocd app diff hello --core

===== apps/Deployment default/hello ======
120c120
<   replicas: 5
---
>   replicas: 2

< is the live cluster, > is Git. Four lines that answer "how does reality differ from what I said I wanted". You can run that against anything, at any time, and get a real answer.

Step 8: turn on self-heal and try to win

Three settings. selfHeal corrects drift, prune deletes objects whose files vanish from the repo, and automated means it acts without being asked.

▶ Do:

kubectl -n argocd patch application hello --type=merge -p '{
  "spec": {"syncPolicy": {"automated": {"prune": true, "selfHeal": true}}}
}'

Then watch, without running any sync command:

 5s  OutOfSync Healthy   replicas=5  pods=5
10s  Synced Healthy      replicas=2  pods=2

Five Pods became two on their own. I changed a policy, not the cluster, and Argo CD dragged reality back to what the repo says. For my money this is the moment the whole idea clicks, more than the first sync was.

Now try to win. I made three attempts and lost all three inside fifteen seconds:

What I did	What happened
`kubectl scale --replicas=4`	back to 2
`kubectl delete svc hello`	Service recreated
`kubectl set image nginx=nginx:1.25`	back to `nginx:1.27`

💡 What happened, plus the fifth gotcha. Look closely at that recreated Service and it has a different ClusterIP and an age of fifteen seconds. Argo CD did not restore the object you deleted. It created a new one that matches the file. Anything that cached the old address is now talking to nothing. Self-heal keeps your cluster matching your repo, and that is not the same as a backup. Don't let it talk you out of having real backups.

Step 9: deploy by pushing to Git

The whole point. Edit the repo, not the cluster.

▶ Do change replicas: 2 to replicas: 3 in apps/hello/deployment.yaml, then:

git commit -am "Scale hello to 3 replicas"
git push

Now touch nothing and watch:

  0s  revision=d1395f1  replicas=2
...
124s  revision=d1395f1  replicas=2
136s  revision=e6c57cb  replicas=3

136 seconds. That isn't Argo CD being slow, it's the default polling interval. With no webhook configured it asks GitHub "anything new?" every three minutes. In production you point a repo webhook at it and deployments become instant. While you're writing a tutorial and don't want to wait, force it:

argocd app get hello --hard-refresh --core

💡 What happened: Check the Pod ages afterwards and you get 28m, 28m, 2s. It scaled up. It did not redeploy. Argo CD works out the difference between the repo and the cluster and applies only that, so a one line change doesn't churn your running workload.

Step 10: delete a file, delete the object

prune is the setting that decides whether removing a file removes the thing it described. I tested it with something disposable.

Add apps/hello/temp-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-temp
data:
  note: 'delete this file from git and prune should remove the object'

Commit, push, hard refresh, and the ConfigMap appears in the cluster. Now git rm the file, commit, push, hard refresh again:

kubectl get configmap hello-temp -n default

Error from server (NotFound): configmaps "hello-temp" not found

💡 What happened: Deleting a file deleted the object. Worth knowing that prune: false is the default, and with it off Argo CD will happily add and update things but never remove them. That sounds safer and mostly it is, but you end up with orphaned objects in your cluster that no file in any repo describes and nobody remembers creating. Turn prune on early, while your cluster is small enough that a mistake is obvious.

Step 11: ship something broken, then roll it back

This is the most important section in the post, so I deliberately shipped a release that could not possibly work. In deployment.yaml:

image: nginx:9.9.9-doesnotexist

Commit, push, refresh, wait thirty seconds:

=== app health ===
Synced   Progressing

=== pods ===
hello-5d6b7fbfc4-f6kxf 1/1 Running       32m
hello-5d6b7fbfc4-p4x64 1/1 Running       32m
hello-5d6b7fbfc4-vzq7x 1/1 Running       6m
hello-6d96c85587-lgxml 0/1 ErrImagePull  31s

=== is the site still up? ===
HTTP 200

Read that first line again. Synced, and completely broken.

Synced means the cluster matches Git. It does not mean the app works. I put something wrong in the repo, so Argo CD faithfully made the cluster wrong. GitOps guarantees fidelity to your repo, not correctness, and if you remember one sentence from this post make it that one. Your repo is now the thing that needs code review, because it's the thing that ships.

The site stayed up for a reason worth knowing: Kubernetes' rolling update will not kill a healthy Pod until its replacement reports Ready, so three good Pods kept serving while the fourth failed to pull. It also sits in Progressing for a full ten minutes before it admits to being Degraded, which is progressDeadlineSeconds defaulting to 600 rather than Argo CD dragging its feet. I stared at Progressing for a while assuming something was stuck.

Now the fix, and notice what the fix is:

git revert --no-edit <bad-sha>
git push

Synced   Healthy
image: nginx:1.27
HTTP 200

Pod ages afterwards: 32m, 32m, 6m. Not one Pod died. The bad release never took hold, the old Pods served throughout, and recovery was a normal commit.

💡 What happened: Your rollback procedure is the same command as every other change. No special runbook, no "quick fix" applied straight to prod that nobody writes down. The revert is right there in the log next to the mistake it undoes.

Small thing that cost me a minute: git revert -q is not a flag. It quietly prints the usage text and reverts nothing, and if you're not reading closely you'll push an empty change and wonder why nothing recovered.

Step 12: rotate that admin password

The bootstrap password from Step 3 is meant to be temporary. The documented command is argocd account update-password, and here's the sixth gotcha:

$ argocd account update-password --account admin ... --core
failed to get issue time: unable to extract token claims

Core mode has no server session, so there's no token to authenticate a password change. You either log in through the API server properly, or you write the hash yourself:

NEW="pick-something-better-than-this"
HASH=$(argocd account bcrypt --password "$NEW")
kubectl -n argocd patch secret argocd-secret \
  -p "{\"stringData\":{\"admin.password\":\"$HASH\",\"admin.passwordMtime\":\"$(date +%FT%T%Z)\"}}"
kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd delete secret argocd-initial-admin-secret

Confirm both directions. The old password should be refused:

$ argocd login <host>:30443 --username admin --password "<old>" --insecure --grpc-web
Invalid username or password

$ argocd login <host>:30443 --username admin --password "<new>" --insecure --grpc-web
'admin:login' logged in successfully

Worth understanding that those two CLI modes really are different tools wearing the same name. --core skips the server and talks to the Kubernetes API, which is fast and needs no login but can't do anything that depends on a session. --server <host> --grpc-web goes through argocd-server with a real login, which is what you want for anything touching accounts, tokens, or RBAC.

Gotchas I hit

Plain kubectl apply fails on the install manifest. The ApplicationSet CRD exceeds the 256KB annotation limit that apply needs for its last-applied-configuration copy. Use --server-side. The error blames the CRD, which sends you looking in the wrong place.
argocd --core reads its namespace from the kubeconfig context. Not from ARGOCD_NAMESPACE, which does nothing. On k3s the context has no namespace, so you get configmap "argocd-cm" not found while the ConfigMap is plainly sitting in the argocd namespace.
Fixing that by repointing your main kubeconfig poisons everything else. kubectl config set-context --current --namespace=argocd means every later kubectl get pods on that box shows Argo CD internals. Keep a separate kubeconfig for the argocd CLI.
DEFAULT_FORWARD_POLICY="DROP" breaks k3s and Docker when you enable ufw. Set it to ACCEPT in /etc/default/ufw first, and allow the pod and service CIDRs. Skip this and you take down pod networking and every container on the box at the same time.
Self-heal is not a backup. Delete a Service and Argo CD creates a new one matching the file, with a new ClusterIP. It restores the description, not the object. Anything holding the old address is now pointing at nothing.
argocd account update-password does not work in core mode. No session, no token, no password change. Patch the bcrypt hash into argocd-secret instead.
Synced does not mean working. It means the cluster matches Git. Put a broken image tag in the repo and you get a proudly Synced broken app.
Don't cancel a background timer with pkill -f "sleep 300". The shell running that command contains the string, matches itself, and kills your session. Use a PID file.

Quick command reference

Goal	Command
Install Argo CD	`kubectl apply -n argocd --server-side -f <install.yaml>`
Read bootstrap password	`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" \
App status	{% raw %}`kubectl get application <app> -n argocd`
See Git versus cluster	`argocd app diff <app> --core`
Sync now	`argocd app sync <app> --core`
Skip the 3 minute poll	`argocd app get <app> --hard-refresh --core`
List apps through the server	`argocd app list --server <host>:30443 --insecure --grpc-web`
Turn on self-heal and prune	patch `spec.syncPolicy.automated` with `selfHeal` and `prune`
Roll back	`git revert --no-edit <sha> && git push`

Where this series goes next

You now have a cluster that argues with you, which is the correct behaviour. Change something by hand and it changes it back. Deploy by pushing a commit. Roll back with git revert and lose nothing.

There's one loose thread I left deliberately, and it's the obvious next part. That hello-app.yaml from Step 4 is still sitting on the node, applied by hand with kubectl. The thing driving all my GitOps is itself not under GitOps, which is a slightly embarrassing place to stop. The fix has a good name, app of apps, and it answers the question of who deploys the deployer.

After that: repo webhooks to kill the three minute poll, private repo authentication so this can point at my own Forgejo rather than GitHub, and Helm and Kustomize as sources instead of plain YAML. The Kubernetes series hub tracks the lot, and Labels, Volumes, health probes and Ingress are all still on the list.

Go and scale something by hand, then watch it change back while you're still looking at it. That's the bit that made it real for me. [ synced ]

Originally published at peculiarengineer.com.

Self-host Forgejo on your tailnet with Docker Compose and Tailscale

Minor Keith — Thu, 30 Jul 2026 02:53:00 +0000

You want your own Git host. Maybe you're getting off GitHub, maybe you just want somewhere private to keep the repos that shouldn't be on someone else's servers. So you go looking, and every guide hands you the same shape: a public DNS record, ports 80 and 443 open to the entire internet, a reverse proxy, a certificate, and a login page that anyone on earth can now knock on.

For a private code host, that's a strange trade. Nothing about "my repos, for me and three collaborators" requires a public address. If the only people who should reach it are people you already trust, put it on your tailnet and the whole category of internet facing problems stops existing. No open ports, and no login page getting scanned at three in the morning.

The Compose file for this is short. What makes it worth writing down is the sidecar pattern it uses, which is not obvious the first time, and a handful of settings that decide whether it works at all. I built this from nothing on a fresh Ubuntu 26.04 box to check it, and the settings that went wrong were not the ones I expected, so those get their own section at the end.

TL;DR. Run Tailscale as its own container and give Forgejo network_mode: service:ts-forgejo so Forgejo has no published ports and its own tailnet identity. Point TS_SERVE_CONFIG at a serve JSON file and you get real HTTPS on forgejo.your-tailnet.ts.net with no port 80 and no HTTP-01 challenge. Persist /var/lib/tailscale or every restart creates a brand new node. Tag the device so its node key never expires. Do not set START_SSH_SERVER, because the image already runs sshd on port 22.

Prerequisites

A box with Docker installed. Mine runs Ubuntu 26.04, but nothing here is version specific.
A Tailscale account and a tailnet you can already log into. If you're starting cold, install Tailscale on the host first so you have something to test from.
Admin access to the Tailscale console, because two of the steps happen there and not on the box.
MagicDNS turned on. It's on by default for new tailnets, and without it you get an IP instead of a name.

You do not need a domain, a DNS record, a certificate, or a single open port in your firewall. That's the point.

The one thing to get straight

Forgejo is not sitting behind a reverse proxy here. Forgejo is the Tailscale node.

That distinction is the whole post. In the usual setup you'd run Caddy or Nginx on the host, publish ports, and forward traffic to a container. Here, the Tailscale container owns a network namespace, and Forgejo is placed inside that same namespace with network_mode: service:ts-forgejo. The two containers share one network stack the way two processes on one machine do.

Once that clicks, the rest follows on its own:

Forgejo publishes no ports to the host, so docker ps shows nothing listening and the host firewall has nothing to do.
The machine appears in your tailnet as its own device, with its own name and its own ACL rules, separate from the host it happens to run on.
Port 22 inside that namespace belongs to Forgejo, and the host's real sshd is somewhere else entirely as far as the network stack is concerned. The port collision that makes the public version of this setup annoying never happens.
HTTPS arrives without opening port 80. The certificate still comes from Let's Encrypt, but through a DNS-01 challenge that Tailscale completes for you by publishing the _acme-challenge TXT record under ts.net for your node. Nothing has to be reachable from the public internet for validation to pass.

The result is a real, publicly trusted certificate on a machine with no public presence at all. Your browser sees a normal padlock. Nothing is exposed.

1. Turn on HTTPS for your tailnet

Do this before anything else. Skipping it does not stop the stack coming up, which is exactly why it wastes your time later.

In the Tailscale admin console, go to DNS, and under HTTPS Certificates click Enable HTTPS. Note the tailnet name it shows you, something like tail1234.ts.net. Every device in your tailnet gets a name under it, so your Forgejo node will end up at forgejo.tail1234.ts.net.

If you skip this, the sidecar still starts and still says it's running. It logs a line telling you HTTPS is not enabled and links the docs, which is fair enough, but tailscale serve status just answers No serve config with no reason attached. The command that gives you a straight answer is:

docker compose exec ts-forgejo tailscale cert forgejo.tail1234.ts.net

With HTTPS off you get your Tailscale account does not support getting TLS certs. With it on you get two files written and you can move on. I use that as the check before touching anything else, because every other symptom of this is ambiguous.

2. Make a tagged auth key

The container needs a key to join the tailnet unattended, and the way expiry works here catches people out.

Auth keys cap out at 90 days, but that expiry only stops new devices from joining. A node that already registered keeps working until its own node key expires, and that defaults to 180 days. So a plain reusable key gets you roughly six months before your Git host quietly drops off the tailnet on a day you weren't touching anything.

The fix is a tag. Key expiry is disabled by default for tagged devices, because they're owned by the tailnet rather than by a user.

First define the tag in your ACL file, under Access controls:

{
  "tagOwners": {
    "tag:container": ["autogroup:admin"]
  }
}

Then go to Settings, Keys, and generate an auth key with Reusable on and the tag tag:container applied. Copy it somewhere safe now, because the console shows it exactly once.

Put it in a .env file next to your Compose file:

# .env
TS_AUTHKEY=tskey-auth-xxxxxxxxxxxx

And keep that file out of Git:

echo ".env" >> .gitignore

3. The Compose file

Two services. The first is the network, the second is the application.

services:
  ts-forgejo:
    image: tailscale/tailscale:latest
    container_name: ts-forgejo
    hostname: forgejo                      # becomes forgejo.your-tailnet.ts.net
    environment:
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_EXTRA_ARGS=--advertise-tags=tag:container
      - TS_STATE_DIR=/var/lib/tailscale    # persisted, see the volume below
      - TS_SERVE_CONFIG=/config/serve.json
      - TS_USERSPACE=false                 # use the kernel networking path
    volumes:
      - ts-forgejo-state:/var/lib/tailscale
      - ./ts-config:/config
    devices:
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped

  forgejo:
    image: codeberg.org/forgejo/forgejo:15
    container_name: forgejo
    network_mode: service:ts-forgejo       # the whole trick, one line
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__ROOT_URL=https://forgejo.tail1234.ts.net/
      - FORGEJO__server__SSH_DOMAIN=forgejo.tail1234.ts.net
      - FORGEJO__server__SSH_PORT=22
      - FORGEJO__service__DISABLE_REGISTRATION=true
    volumes:
      - forgejo-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - ts-forgejo
    restart: unless-stopped

volumes:
  ts-forgejo-state:
  forgejo-data:

A few of those lines are doing more work than they look like they are.

network_mode: service:ts-forgejo is the pattern. Notice what is absent: there is no ports: block anywhere in this file. Forgejo listens on 3000 and 22 inside the shared namespace, reachable over the tailnet and nowhere else.

TS_USERSPACE=false matters more than it looks. The Tailscale image defaults to userspace networking, and in that mode the TUN device and the net_admin capability sit there unused. Turning it off puts the container on the kernel networking path those lines exist for, which is what you want when something in the namespace needs to answer on a real port.

hostname: forgejo is what decides the name. Tailscale registers the container under it, so this is the value that ends up in your URL. Pick it before first boot, because renaming a node later means fixing ROOT_URL too.

TS_STATE_DIR with a real volume behind it is the difference between one node and a hundred. More on that in the gotchas below.

FORGEJO__server__ROOT_URL uses Forgejo's double underscore convention, where FORGEJO__section__KEY maps to a key in app.ini. Get this value right before you ever start the container. Forgejo bakes it into the clone URLs it shows you, the links in its emails, and its webhook targets. Wrong here means every clone command your users copy points at the wrong place, and it looks fine on screen right up until someone tries it.

There is also something deliberately missing. You will see START_SSH_SERVER=true in a lot of Forgejo examples, and it does not belong here. The rootful image runs OpenSSH on port 22 by itself, unconditionally. Setting that variable starts Forgejo's own Go SSH server as well, and now two SSH servers want the same port in the same namespace. Leave it out and the image does the right thing on its own. That variable is for the rootless image, which is a different setup with SSH on 2222.

DISABLE_REGISTRATION=true from the first boot means nobody gets to sign up while you're still setting things up. You'll create your admin account through the installer instead.

4. The serve config

Create ts-config/serve.json next to your Compose file:

{
  "TCP": {
    "443": {
      "HTTPS": true
    }
  },
  "Web": {
    "${TS_CERT_DOMAIN}:443": {
      "Handlers": {
        "/": {
          "Proxy": "http://127.0.0.1:3000"
        }
      }
    }
  },
  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": false
  }
}

${TS_CERT_DOMAIN} is substituted by the Tailscale container at startup with the node's real name, so you don't have to hardcode your tailnet name in a second place.

The proxy target is 127.0.0.1:3000, and that is only correct because of the shared namespace. Forgejo really is on localhost from Tailscale's point of view. If you ever pull these two apart into separate networks, this line breaks first.

AllowFunnel is set to false on purpose. It's the switch that would put this node on the public internet, and having it present and off is better than having it absent, because you can see what the answer currently is.

Bring it up:

docker compose up -d
docker compose logs -f ts-forgejo

Watch for the node registering and the certificate being issued. Then check it from another machine on your tailnet:

tailscale status | grep forgejo
curl -I https://forgejo.tail1234.ts.net/

A 200 and a valid certificate means the hard part is done.

5. First boot and locking the installer

Open https://forgejo.tail1234.ts.net/ in a browser on any device in your tailnet and you get Forgejo's setup page.

Two things matter here. Leave the database as SQLite unless you have a reason not to. For a personal or small team instance it's genuinely fine, it's one less container to run and one less thing to back up, and you can migrate later if the instance grows into something that needs Postgres. And check that the URL fields on the form match the ROOT_URL you set, because the installer will happily write different values into app.ini and leave you with two sources of truth.

Create your admin account on that same form. Do not skip it and do it later.

The reason for the urgency is smaller here than on a public box, but it still applies: until the installer is completed, whoever reaches that page can complete it and become the administrator. On a tailnet that's limited to devices you've already authorised, which is a much shorter list than "the internet". It's still a list. Finish the form.

Once you're in, confirm the install lock landed:

docker compose exec forgejo grep INSTALL_LOCK /data/gitea/conf/app.ini
# INSTALL_LOCK = true

6. Cloning over SSH

This is the part that would have been a whole section of fighting in the public version, and here it's almost nothing.

git clone git@forgejo.tail1234.ts.net:yourname/yourrepo.git

Port 22, no custom port, no -p 2222 to remember, no conflict with the host's own sshd. The image's SSH server owns port 22 inside its namespace, and the host's sshd owns port 22 on the host, and they never meet.

Add your public key in the Forgejo UI under Settings, SSH / GPG Keys, the same as you would anywhere else.

One thing to check: the clone URL Forgejo displays comes from SSH_DOMAIN and SSH_PORT. If those disagree with reality, the button copies a command that fails, and the error the user sees is a connection timeout that tells them nothing about why.

Who can actually reach it

Right now, every device in your tailnet can reach it. That's fine for a tailnet of one person and three laptops. It stops being fine the moment you add a contractor's machine or a server that runs someone else's code.

ACLs fix that. This is a fragment to merge into your existing policy, not a whole policy file. If you paste it over the top of everything, you delete the default rule that lets your tailnet talk to itself, and every other connection you have stops working at the same moment.

{
  "groups": {
    "group:devs": ["you@example.com", "collaborator@example.com"]
  },
  "tagOwners": {
    "tag:container": ["autogroup:admin"]
  },
  "acls": [
    // ... your existing rules stay here ...
    {
      "action": "accept",
      "src": ["group:devs"],
      "dst": ["tag:container:443,22"]
    }
  ]
}

If you already worked through the ACL section of Tailscale for private networking, this is the same machinery pointed at a container instead of a server.

When you want it public after all

Sometimes you need to hand someone a link without adding them to your tailnet. Tailscale Funnel does that, and it's less of a one liner than it looks.

Two things have to be true first. Your tailnet policy needs the funnel node attribute, which for a tagged container node you will be adding by hand:

"nodeAttrs": [
  {
    "target": ["tag:container"],
    "attr": ["funnel"]
  }
]

Then flip the switch in serve.json and restart, which is the durable way to do it since TS_SERVE_CONFIG reapplies that file every time the container starts:

  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": true
  }

Restart with docker compose restart and no service name after you edit that file. Restarting only the Tailscale container is what leaves Forgejo answering 502 behind a network stack that moved out from under it, which is the gotcha below and the one that wastes the most time.

Funnel only works on ports 443, 8443, and 10000, so 443 is the one you want anyway.

The node attribute is not optional, and skipping it is the worst kind of failure. I flipped AllowFunnel to true without it and everything told me it had worked. tailscale funnel status printed Funnel on with the URL under it, tailscale serve status agreed, and the logs said nothing at all. The name simply never appeared in public DNS, so from outside the tailnet it did not resolve, let alone serve. The node did not have the capability and no part of the tooling mentioned it.

If you turn Funnel on, verify it from something that is not on your tailnet. A phone with WiFi off is enough:

dig +short @1.1.1.1 forgejo.tail1234.ts.net

An empty answer means Funnel is not really on, whatever the CLI told you.

Once it's on, you have inherited every problem this post was avoiding. Registration lockdown and rate limiting become your concern again. It's a good escape hatch and a bad default. Set it back to false when you're done.

The honest limitation: if the people you collaborate with will not join your tailnet, this setup is not for you. Tailnet only means tailnet only. Everyone who touches these repos needs Tailscale on their machine and a place in your ACLs. For a solo developer or a small team that already uses Tailscale, that cost is zero. For an open source project taking drive by contributions, it's a wall.

Gotchas I hit

No state volume, so every restart mints a new node. This is the big one, and I watched it happen. If TS_STATE_DIR has no volume behind it, the container loses its identity, registers again as a fresh device, and Tailscale appends a suffix to keep the name unique. Mine came back as forgejo-1 on a new address while the old forgejo sat there marked offline.

What makes it nasty is how healthy the result looks. The new node got its own certificate within seconds and served Forgejo on 443 without complaint. But ROOT_URL and SSH_DOMAIN live in app.ini inside the data volume, so they still held the old name, and the API cheerfully handed out clone URLs pointing at a node nothing is listening on. The server is fine. Every clone command it gives your users times out. Persist /var/lib/tailscale.

What losing that state does to you depends on your auth key. With a reusable key you get the duplicate node above. With a single use key the container cannot register at all, fails with invalid key: API key ... not valid, and never comes up. Two completely different mornings from the same missing volume.

Auth failures loop instead of stopping. Every registration failure I hit, whether a tag the policy did not allow or a spent key, left the container restarting rather than exiting. With restart: unless-stopped that continues forever. docker compose ps shows restarting, not an error, so if you run up -d and walk away you come back to something that has been failing quietly for however long you were gone. The reason is only ever in docker compose logs.

An untagged node drops off at six months, not three. The auth key expiring at 90 days is the number everyone quotes, but that only blocks new registrations. The node itself runs until its node key hits the 180 day default, which is a much worse way to find out, because by then you've forgotten the setup entirely. Tag the device and key expiry is off.

The tag has to exist before you advertise it. tag:container is my example name, not a default. If it is not in tagOwners in your policy the node refuses to join with requested tags [tag:container] are invalid or not permitted, which is at least an honest error. Use whatever tag your tailnet already has if you have one.

Tailscale SSH swallows Git over SSH. If you add --ssh to TS_EXTRA_ARGS, tailscaled intercepts inbound tailnet connections to port 22 before they ever reach the SSH server in the container. There's no bind conflict and nothing looks broken in the logs. Your clones just stop working. Leave Tailscale SSH off on this node and use it on your other machines.

Restarting the sidecar alone breaks Forgejo in a way that looks like something else. This one cost me the most time, because I did it to myself repeatedly while testing and kept diagnosing the wrong thing. docker compose restart ts-forgejo gives Forgejo a network stack that has moved out from under it. Forgejo keeps running, docker compose ps says both are Up, and the tailnet name still resolves.

What you get is a broken instance that points nowhere useful. The web side answers 502, because tailscaled terminates TLS perfectly well and then cannot reach 127.0.0.1:3000 any more. Git over SSH gives you Connection refused, or just hangs with no banner. Neither symptom points at the container you actually restarted.

The fix is docker compose restart with no service name, so both come back together. I checked this twice: sidecar alone gives 502 and a refused clone, both together gives 200 and a clean clone. Any time you touch the Tailscale container, take Forgejo with it.

ROOT_URL left over from an earlier attempt. If you tried this once with a different hostname, changing the environment variable is not always enough, because the installer wrote the old value into app.ini and that file lives in the volume. Check /data/gitea/conf/app.ini and fix it there, or start from a clean volume.

Quick reference

Setting	What it controls	What breaks without it
`network_mode: service:ts-forgejo`	Puts Forgejo in the sidecar's network namespace	Forgejo is not on the tailnet at all
`TS_USERSPACE=false`	Kernel networking instead of userspace	The TUN device and `net_admin` do nothing
`hostname` on the sidecar	The node name, and so the URL	Random or wrong `ts.net` name
`TS_STATE_DIR` plus a volume	Node identity across restarts	New device on every restart, name drifts
`TS_EXTRA_ARGS=--advertise-tags`	Marks the node as tailnet owned	Node key expires at 180 days, node drops off
`TS_SERVE_CONFIG`	HTTPS and the proxy to port 3000	No TLS, nothing served on 443
`FORGEJO__server__ROOT_URL`	Clone URLs, emails, webhooks	Everything points somewhere wrong
`FORGEJO__server__SSH_DOMAIN`	The SSH clone URL shown in the UI	Copy button hands out a command that times out
no `START_SSH_SERVER`	Lets the image's own sshd own port 22	Two SSH servers want the same port

Useful commands:

docker compose logs -f ts-forgejo                      # node registration and cert issuance
docker compose exec ts-forgejo tailscale status        # is it on the tailnet
docker compose exec ts-forgejo tailscale serve status   # what is being served on 443

# backup, written somewhere writable and then copied out
docker compose exec -u 1000 -w /tmp forgejo \
  forgejo dump -c /data/gitea/conf/app.ini

# the archive is named forgejo-dump-<timestamp>.zip, so read the name back
DUMP=$(docker compose exec -T forgejo sh -c 'ls -1 /tmp/forgejo-dump-*.zip | tail -1')
docker compose cp "forgejo:${DUMP%$'\r'}" ./

That timestamp is the part that bites. forgejo dump prints the filename it wrote and then you are on your own, so a copy command with a fixed name in it fails every time and you find out when you need the backup.

[ no open ports · real certs · git@ over the tailnet ]

Originally published at peculiarengineer.com.