DEV Community: Minor Keith

Prometheus node exporter on Ubuntu 26.04, and it scrapes over the private network

Minor Keith — Mon, 21 Sep 2026 13:37:39 +0000

Every server I look after eventually gets the same question asked of it: how full is the disk, is it about to run out of memory, and does it need a reboot that nobody has got round to. Node exporter is the standard answer. It is one binary that exposes a few thousand numbers about the machine on port 9100, and Prometheus pulls them every fifteen seconds and keeps them. Ubuntu 26.04 has both in the archive, which is where this post starts and where most guides do not.

Most node exporter guides download a tarball from GitHub, make a user, and write a unit file. That works, and section 7 does it, but on 26.04 apt install prometheus-node-exporter gives you a unit, a dedicated user, a config file for the flags, and a set of extra collectors that report pending apt upgrades and whether the box wants a reboot. The reason for the post is the two things the package does that I did not want (it drags in twenty one other packages by default, and it listens on every interface), the two failure messages that look alike from the Prometheus side and mean different things, and the alert rules that turn "I have metrics" into "I get told".

The one thing to get straight: Prometheus scrapes. The exporter never sends anything anywhere. It sits there answering GET /metrics, and whoever can reach port 9100 can read every mount point, every network interface, the memory and load figures and, with the collectors package, the list of pending updates on the box. So the exporter's listen address and the firewall rule in front of it are part of the install, not an afterthought.

TL;DR. On the server: sudo apt update && sudo apt install prometheus-node-exporter (add --no-install-recommends on a cloud VM where you do not want the hardware collectors, and lose the apt and reboot metrics with them), set ARGS="--web.listen-address=10.20.1.10:9100" in /etc/default/prometheus-node-exporter to bind it to the private interface, sudo ufw allow from <prometheus ip> to any port 9100 proto tcp. On the Prometheus box: sudo apt install prometheus, add a job_name with the server's private address to /etc/prometheus/prometheus.yml, promtool check config, sudo systemctl reload prometheus, and confirm with up in the query API. Keep the recommends on real hardware; the apt_upgrades_pending and node_reboot_required metrics come from that package.

Prerequisites

Two Ubuntu 26.04 machines that can reach each other on a private network: the server you want metrics from, and the one that will run Prometheus. Mine are two cloud VMs with a private network between them, 10.20.1.10 for the server and 10.20.1.20 for Prometheus. A tailnet works the same way; use the 100.x addresses.
A sudo user on both.
Nothing else. Grafana is a separate afternoon and this post stops at Prometheus answering queries.

1. Install the exporter from the archive

One decision before the command. The package recommends a collectors package that brings twenty one more packages and the apt and reboot metrics with them; section 2 shows both sides. I installed the default first, because the rest of this post uses those metrics, and section 2 tells you how to go the small route instead.

sudo apt update
sudo apt install prometheus-node-exporter
prometheus-node-exporter --version

On 26.04 this is node_exporter, version 1.10.2, from universe, and the package does the parts the GitHub tarball leaves to you: a prometheus system user, a unit called prometheus-node-exporter.service that is enabled and running when apt returns, and /etc/default/prometheus-node-exporter with an ARGS="" line where every flag goes. The unit is short and worth reading once:

systemctl cat prometheus-node-exporter

[Service]
Restart=on-failure
User=prometheus
EnvironmentFile=/etc/default/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter $ARGS
ExecReload=/bin/kill -HUP $MAINPID

Check it is answering:

curl -s http://127.0.0.1:9100/metrics | grep -cE '^[a-z]'
curl -s http://127.0.0.1:9100/metrics | grep -E '^node_load1 |^node_memory_MemAvailable_bytes'

Just under two thousand metric lines on my box. node_load1, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes{mountpoint="/"} are the three you will query first.

The screenshots in this post come from a second pair of fresh 26.04 boxes, so timestamps and the odd number differ from the prose.

2. What the collectors package adds, and what it costs

prometheus-node-exporter recommends prometheus-node-exporter-collectors, and apt installs recommends by default. On my fresh VM that one recommendation turned a one package install into twenty two: ipmitool, smartmontools, nvme-cli, freeipmi, openipmi, moreutils, jq, the SNMP libraries and a Python Prometheus client. What you get for it is a set of systemd timers that run collector scripts (every fifteen minutes for the three that run on this VM) and drop their output into /var/lib/prometheus/node-exporter/ as .prom files, which the exporter's textfile collector picks up on the next scrape:

systemctl list-timers --all | grep prometheus
ls /var/lib/prometheus/node-exporter/

prometheus-node-exporter-apt.timer
prometheus-node-exporter-nvme.timer
prometheus-node-exporter-smartmon.timer
prometheus-node-exporter-ipmitool-sensor.timer     (inactive, no IPMI here)
prometheus-node-exporter-mellanox-hca-temp.timer   (inactive)

apt.prom  nvme.prom  smartmon.prom

The one that earns its keep on any Ubuntu box is apt.prom:

curl -s http://127.0.0.1:9100/metrics | grep -E '^(apt_|node_reboot_required)'

apt_autoremove_pending 0
apt_package_cache_timestamp_seconds 1.7899594604788072e+09
apt_upgrades_pending{arch="amd64",origin="Ubuntu:resolute/resolute-security,Ubuntu:resolute/resolute-updates"} 70
apt_upgrades_pending{arch="all",origin="Ubuntu:resolute/resolute-security,Ubuntu:resolute/resolute-updates"} 12
apt_upgrades_pending{arch="amd64",origin="Ubuntu:resolute/resolute-updates"} 8
apt_upgrades_pending{arch="all",origin="Ubuntu:resolute/resolute-updates"} 3
node_reboot_required 0

Ninety three pending upgrades on an image that was a few minutes old, eighty two of them from the security pocket, and a node_reboot_required gauge that goes to 1 on the apt collector's next run after /run/reboot-required appears, so within about fifteen minutes. Those two metrics are the reason I would keep the collectors package on real hardware, where smartmon.prom and nvme.prom also mean something.

On these VMs there is no SMART data and no IPMI, so the trade is ipmitool and friends for one useful script. If you decide that before installing, the smaller install is one flag:

sudo apt install --no-install-recommends prometheus-node-exporter

That installs exactly one package. No timers, an empty textfile directory, the same exporter. If you already did the default install in section 1, the flag does nothing on its own (apt reports nothing new to install); purge prometheus-node-exporter prometheus-node-exporter-collectors, run apt autoremove, and install again with the flag. If you want apt_upgrades_pending without the rest, the script is apt_info.py from the community collector repo; it needs python3-apt and python3-prometheus-client, a timer of your own, and section 4 shows where its output goes.

3. Bind it to the private interface and fence it with ufw

Out of the box the exporter listens on *:9100, every address the machine has. On a server with a public IP that is the whole machine's inventory on an unauthenticated port. Two layers fix it. First, tell the exporter to listen only on the private address:

sudo nano /etc/default/prometheus-node-exporter

ARGS="--web.listen-address=10.20.1.10:9100"

sudo systemctl restart prometheus-node-exporter
sleep 1; ss -tlnp | grep 9100

The socket line now shows 10.20.1.10:9100 instead of *:9100. The sleep is there because the exporter takes a moment to bind after the restart; without it the grep can come back empty and send you looking for a problem that does not exist.

curl http://127.0.0.1:9100/metrics now fails with connection refused. That is correct; the exporter is no longer on loopback. Use the private address for local checks from here on.

Second, allow only the Prometheus box through the firewall. If ufw is not on yet, let SSH in before enabling it:

sudo ufw allow OpenSSH
sudo ufw allow from 10.20.1.20 to any port 9100 proto tcp
sudo ufw enable
sudo ufw status

Belt and braces. If someone later changes ARGS back, or the private interface picks up a second address, the firewall still only admits one client. If your two machines talk over a tailnet instead, bind to the 100.x address and allow from the Prometheus node's 100.x address; nothing else changes.

4. A metric of your own with the textfile collector

The textfile collector is on by default in the Ubuntu package, pointed at /var/lib/prometheus/node-exporter/. Any *.prom file in there, in the exposition format, is served alongside the built in metrics. This is how a deploy script reports when it last ran:

printf '# HELP deploy_timestamp_seconds Unix time of the last deploy.\n# TYPE deploy_timestamp_seconds gauge\ndeploy_timestamp_seconds %s\n' "$(date +%s)" \
  | sudo tee /var/lib/prometheus/node-exporter/deploy.prom.tmp > /dev/null
sudo mv /var/lib/prometheus/node-exporter/deploy.prom.tmp /var/lib/prometheus/node-exporter/deploy.prom
curl -s http://10.20.1.10:9100/metrics | grep -E '^deploy_timestamp_seconds|^node_textfile_scrape_error'

deploy_timestamp_seconds 1.789959553e+09
node_textfile_scrape_error 0

Write to a temporary name and mv into place. The exporter can read the directory at any moment. I dropped a file of garbage in there to see what happens: node_textfile_scrape_error went to 1, the journal got failed to parse textfile data, that file's metrics were skipped, and every other file was still served. A half written file that happens to still parse is worse: it is served as if complete, with no error. Hence the rename. The mv is atomic on the same filesystem, so the exporter sees either the old file or the whole new one.

5. Prometheus from the archive, scraping it

On the second machine:

sudo apt update
sudo apt install prometheus
prometheus --version

That is Prometheus 2.53.5 from universe. Upstream is on 3.x, and 2.53 was upstream's 2.x long term support line, whose upstream support ended in July 2025; Ubuntu maintains the package on its own from here. For a home or small fleet server that is fine: the scrape config, the query language and the alert rules in this post are identical on both. If you want 3.x, the upstream tarball installs the same way as section 7 does for the exporter. Two things the package does that you should know about: it recommends prometheus-node-exporter, so the Prometheus box gets its own exporter and a job_name: node for localhost:9100 already in the config, and it ships /etc/default/prometheus with ARGS="" for flags such as --storage.tsdb.retention.time.

Before touching the config, confirm the network path works from this side:

curl -s --max-time 5 -o /dev/null -w '%{http_code}\n' http://10.20.1.10:9100/metrics

A 200 here means the bind address is right and the firewall lets this box through. Anything else, fix that first; Prometheus will only tell you the same thing more slowly. Then add the job to the end of /etc/prometheus/prometheus.yml:

  - job_name: 'nodex'
    static_configs:
      - targets: ['10.20.1.10:9100']
        labels:
          instance: 'nodex'

Setting instance yourself is optional. Without it, Prometheus uses the target address as the instance label, and every alert reads 10.20.1.10:9100 needs a reboot. With it, alerts and queries say the machine's name. Check and reload:

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus

The package's unit has an ExecReload that sends HUP, so reload re reads the config without dropping the database. Give it a scrape interval and ask:

curl -s 'http://127.0.0.1:9090/api/v1/query?query=up'

Prometheus has a web UI on port 9090 that is easier to read, but the API is what works from a headless box over SSH, and the shape is simple: data.result[].metric is the label set and data.result[].value[1] is the number. up is 1 for every target Prometheus could scrape on its last attempt and 0 for every one it could not. Mine showed three targets at 1: Prometheus itself, its local exporter, and nodex. For anything longer than up, a small shell function saves quoting the URL by hand:

q() { curl -s http://127.0.0.1:9090/api/v1/query --data-urlencode "query=$1" \
  | python3 -c "import sys,json; [print(r['metric'].get('instance'), r['value'][1]) for r in json.load(sys.stdin)['data']['result']]"; }
q 'node_load1'

That prints one line per series, instance then value. From there, the numbers you came for, as expression and value:

node_load1{instance="nodex"}                                   0.23
node_filesystem_avail_bytes{mountpoint="/",instance="nodex"}
  / node_filesystem_size_bytes{mountpoint="/",instance="nodex"} * 100   91.4
deploy_timestamp_seconds{instance="nodex"}                     1789959553

The textfile metric from section 4 shows up like any other, which is the point of it.

The rules directory from section 6 was already in place when this shot was taken, which is why promtool counts a rule file here; on a first run it prints SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax.

6. Three alert rules worth having

Metrics nobody looks at are a database. I put these three rules in /etc/prometheus/rules/node.yml on the Prometheus VM (the directory does not exist after the install, so sudo mkdir -p /etc/prometheus/rules first):

groups:
  - name: node
    rules:
      - alert: RebootRequired
        expr: node_reboot_required == 1
        for: 1h
        labels: { severity: info }
        annotations: { summary: "{{ $labels.instance }} needs a reboot" }
      - alert: RootDiskLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 15m
        labels: { severity: warning }
        annotations: { summary: "{{ $labels.instance }} root filesystem under 10% free" }
      - alert: PendingSecurityUpdates
        expr: sum by (instance) (apt_upgrades_pending{origin=~".*security.*"}) > 0
        for: 1d
        labels: { severity: info }
        annotations: { summary: "{{ $labels.instance }} has security updates pending" }

Point the config at the directory by replacing the empty rule_files: line near the top of prometheus.yml, then check both files and reload:

rule_files:
  - /etc/prometheus/rules/*.yml

promtool check rules /etc/prometheus/rules/node.yml
promtool check config /etc/prometheus/prometheus.yml
sudo systemctl reload prometheus
curl -s http://127.0.0.1:9090/api/v1/rules

promtool check rules reported SUCCESS: 3 rules found, and within fifteen seconds of the reload, once the first evaluation had run, the rules API listed all three as health: ok (before that they show unknown). Within a minute PendingSecurityUpdates was pending for both of my boxes, which is what images with eighty two security updates waiting should produce, and it will fire after the for: 1d if nobody runs apt upgrade. RebootRequired and PendingSecurityUpdates both come from the apt collector, so they need section 2's collectors package (or apt_info.py) on each target. RootDiskLow uses the built in filesystem collector and works everywhere.

Where the alerts go (email, a chat webhook, a pager) is Alertmanager's job, which is prometheus-alertmanager in the same archive and a post of its own. Until then the /api/v1/alerts endpoint and the Alerts page on 9090 show what is pending and firing.

7. The upstream binary, when the package is too old

Back on the target for this section. The archive's 1.10.2 is from late 2025; upstream was at 1.12.1 when I wrote this. If you need a collector or a fix from a newer release, the upstream binary is a single file and the unit is the same shape as the packaged one:

V=1.12.1
curl -sLO https://github.com/prometheus/node_exporter/releases/download/v$V/node_exporter-$V.linux-amd64.tar.gz
tar xzf node_exporter-$V.linux-amd64.tar.gz
sudo install -m 0755 node_exporter-$V.linux-amd64/node_exporter /usr/local/bin/node_exporter
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus node exporter (upstream binary)
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter --web.listen-address=10.20.1.10:9100 --collector.textfile.directory=/var/lib/prometheus/node-exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl disable --now prometheus-node-exporter
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sleep 1; curl -s http://10.20.1.10:9100/metrics | grep ^node_exporter_build_info

Both want port 9100, so the packaged one has to be stopped and disabled first, and --collector.textfile.directory has to be passed explicitly; the upstream binary has no default for it. On my box node_exporter_build_info switched from version="1.10.2" to version="1.12.1" and Prometheus did not notice anything but the label. The cost is that apt no longer upgrades it; that is now a curl you run yourself.

8. Reading a down target

The targets endpoint on the Prometheus box says why a scrape failed, and the two messages you will see most often look similar and are not. I broke the target on purpose both ways, from the target's own shell, and read the result from the Prometheus box:

curl -s http://127.0.0.1:9090/api/v1/targets | python3 -c \
  "import sys,json; [print(t['labels']['job'], t['health'], t['lastError']) for t in json.load(sys.stdin)['data']['activeTargets']]"

First, sudo systemctl stop prometheus-node-exporter on the target (or node_exporter if you switched in section 7). Within twenty seconds Prometheus reported:

nodex down Get "http://10.20.1.10:9100/metrics": dial tcp 10.20.1.10:9100: connect: connection refused

connection refused almost always means the packet arrived and nothing was listening: the exporter is stopped, crashed, or bound to a different address than the one in the job. (A firewall that rejects rather than drops produces it too.) Check the exporter's unit status and ss -tlnp | grep 9100 on the target.

Then I started it again and deleted the ufw rule instead: sudo ufw delete allow from 10.20.1.20 to any port 9100 proto tcp on the target. One thing to know before you try this: Prometheus keeps the HTTP connection to a target open between scrapes, and ufw does not cut an established connection, so up can stay at 1 for a while after the rule is gone. Restarting the exporter forces a new connection, and the next scrape reported:

nodex down Get "http://10.20.1.10:9100/metrics": context deadline exceeded

context deadline exceeded means the scrape hit its timeout, ten seconds by default, without an answer: a firewall dropping packets, a wrong IP, a route that does not exist, or an exporter that is stalled. The exporter can be perfectly healthy. Check the ufw rule on the target and curl --max-time 5 the address from the Prometheus box (without the limit, a dropped packet leaves curl sitting there for a couple of minutes). Put the rule back with the same ufw allow from section 3 and the target was up on the next scrape.

up going to 0 is the same for both, which is why it is worth having the lastError text to hand before deciding which machine to log into.

Gotchas I hit

apt install prometheus-node-exporter installed twenty two packages. --no-install-recommends gets you one, at the price of the apt and reboot metrics, and it only helps if you pass it on the first install.
The exporter listens on every interface by default. --web.listen-address in /etc/default/prometheus-node-exporter plus a ufw allow from rule. After that, curl 127.0.0.1:9100 is refused on purpose.
Deleting the ufw rule produced context deadline exceeded while the exporter was still running. That message is about the path, not the process. And it only shows up once the scrape opens a new connection; the established one keeps working until something closes it.
Two exporters cannot bind the same address and port; the second one fails with address already in use. Disable the packaged unit before enabling an upstream one on the same address.
The upstream binary has no default textfile directory. Pass --collector.textfile.directory or the .prom files silently stop appearing.
Write .prom files to a temp name and mv them, or node_textfile_scrape_error will occasionally be 1 and the metrics in that file will be missing for a scrape.
The archive's Prometheus is 2.53, a major version behind upstream. Everything in this post works on both, but read the upstream docs with that in mind.

Quick reference

# target
sudo apt update
sudo apt install prometheus-node-exporter          # add --no-install-recommends to skip the collectors package and everything it pulls in
sudo nano /etc/default/prometheus-node-exporter    # ARGS="--web.listen-address=10.20.1.10:9100"
sudo systemctl restart prometheus-node-exporter    # node_exporter instead, if you moved to the upstream unit in section 7
sudo ufw allow from 10.20.1.20 to any port 9100 proto tcp
curl -s http://10.20.1.10:9100/metrics | grep -E '^node_load1 |^node_reboot_required'

# prometheus box
sudo apt update && sudo apt install prometheus
sudo nano /etc/prometheus/prometheus.yml           # add the job under scrape_configs, rule_files: - /etc/prometheus/rules/*.yml
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/node.yml
sudo systemctl reload prometheus
curl -s 'http://127.0.0.1:9090/api/v1/query?query=up'
curl -s http://127.0.0.1:9090/api/v1/targets      # lastError says which machine to look at
curl -s --max-time 5 http://10.20.1.10:9100/metrics | head -n 3   # from here, is the path open
curl -s http://127.0.0.1:9090/api/v1/alerts

Ninety three pending updates on a box that was five minutes old. The exporter did not fix that, but at least now something is counting.

Originally published at peculiarengineer.com.

How To - Nginx reverse proxy on Ubuntu 26.04 without Docker

Minor Keith — Mon, 21 Sep 2026 13:35:36 +0000

The app is running. It listens on 127.0.0.1:8080, it has a systemd unit, and it works fine over an SSH tunnel. Now it needs a hostname, a certificate, and to stop being reachable only by yourself. The Caddy post covers this for containers. This is the version for a plain Ubuntu 26.04 box where the app is a process and the proxy is the nginx that has been in the archive forever.

Nginx needs a dozen lines for this. The reason it gets a post is what happened on my test box between the dozen lines and the working site: the default site kept answering just after a reload, the app saw 127.0.0.1 as every client's address until the forwarding headers went in, a websocket endpoint came back 502 until two more headers went in, and the HTTP/2 syntax every older guide still shows made nginx 1.28 print a deprecation warning. Each of those is one line once you know it, and none of them is in the nginx welcome page.

In this setup nginx is the only thing facing the internet. It accepts the connection on 80 and 443, and opens a second connection from itself to 127.0.0.1:8080. Your app sees nginx as the client. Every proxy_set_header line in this post exists to pass along something the app would otherwise never learn about the real request.

TL;DR. sudo apt install nginx, write /etc/nginx/sites-available/app.conf with a server_name and a location / that does proxy_pass http://127.0.0.1:8080; plus the Host, X-Real-IP, X-Forwarded-For and X-Forwarded-Proto headers, symlink it into sites-enabled, remove the default symlink, sudo nginx -t && sudo systemctl reload nginx. For HTTPS, sudo apt install certbot python3-certbot-nginx and sudo certbot --nginx -d app.example.com; on 26.04 that is certbot 4.0 from the archive and no snap is involved. Then add http2 on; next to the listen 443 ssl; line certbot wrote.

Prerequisites

Ubuntu 26.04 with a sudo user. Everything here ran on a fresh 26.04.1 server image.
An app already listening on a loopback port. Mine is a small Python HTTP server on 127.0.0.1:8080 that prints back the headers it receives, run from a systemd unit, because that makes section 4 visible. Yours is whatever you are fronting. If it listens on 0.0.0.0, change that to 127.0.0.1 first, or the proxy is decorative.
A DNS A record for the hostname pointing at the box, and ports 80 and 443 reachable from the internet. If a cloud firewall sits in front of the server, open both there too. The HTTP challenge certbot uses in section 6 needs 80 reachable from Let's Encrypt.

1. Install nginx and open the firewall

sudo apt update
sudo apt install nginx
nginx -v

On 26.04 that is nginx/1.28.3 (Ubuntu), from main, started and enabled by the package. It comes with HTTP/2, HTTP/3 and the real IP module compiled in, so nothing from a PPA is needed for anything in this post. (The stream module is a separate libnginx-mod-stream package; this post does not need it.) Browse to the box's IP and you get the "Welcome to nginx!" page, served by /etc/nginx/sites-enabled/default. That is the default site I remove in section 3.

The package also registers ufw application profiles:

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

Nginx Full is 80 and 443. There are also Nginx HTTP, Nginx HTTPS, and Nginx QUIC, which is 80 and 443 on TCP plus 443 on UDP. If ufw was already on, just add the second rule. If you have never touched ufw on this box, the ufw basics post explains why OpenSSH goes in before enable.

The screenshots in this post are from a second fresh 26.04 box, so the hostname in them is that box's reverse DNS name where the prose says app.example.com.

2. The site file

Nginx on Ubuntu reads /etc/nginx/nginx.conf, which includes every file in /etc/nginx/sites-enabled/. The convention is one file per site in sites-available and a symlink in sites-enabled to turn it on. Create the site with sudo nano /etc/nginx/sites-available/app.conf:

# /etc/nginx/sites-available/app.conf
server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Enable it, test the whole configuration, and reload:

sudo ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

nginx -t parses everything nginx would load and refuses if any file is broken, which is why it goes before every reload. A broken config with a plain restart leaves you with no web server at all. With a reload, nginx keeps serving the old config until the new one parses, so the worst case is the change not taking.

Now curl http://app.example.com/. Mine came back with the welcome page on the first try and the app on the second, and the next section is about why. Read it even if you got the app first time, because it ends with removing the default site, and every edit from here on gets the same sudo nginx -t && sudo systemctl reload nginx before you test it.

3. Why the default site keeps winning

Nginx picks a server block by matching the request's Host header against every server_name. When nothing matches, it uses the block marked default_server, and if none is marked, the first one it loaded. sites-enabled/default has both listen lines marked default_server, so any request for a name nginx does not recognise lands there:

curl -s -H 'Host: nothere.example.com' http://127.0.0.1/ | grep -o '<title>.*</title>'

<title>Welcome to nginx!</title>

That is expected and harmless. What bit me was the request for my real hostname also landing on the welcome page, and the cause was not nginx at all. It was me running curl in the same second as the reload. A reload is graceful: the master process parses the new config, starts new workers, and tells the old ones to finish their connections and exit. systemctl reload nginx returns as soon as the signal is sent, not when the swap is done, and on my box a request in that first second was still answered by an old worker running the old config. One second later the same request hit the app. If your first curl after a reload shows the previous config, run it again before you touch anything.

Then decide what should answer for hostnames you did not configure. I remove the default site, so that an unknown name falls through to the first block, which is the app:

sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

If you would rather unknown names get nothing, keep a catch all block with return 444;, which closes the connection without a response. Either is fine. Leaving the welcome page up is the choice that looks unfinished to anyone who scans your IP.

4. The headers your app actually needs

Take the four proxy_set_header lines out, test and reload, and look at what the app receives. Mine prints them:

Host: 127.0.0.1:8080
X-Forwarded-For: None
X-Forwarded-Proto: None
X-Real-IP: None

Without the headers, the app thinks its own name is 127.0.0.1:8080, so any absolute link it builds from Host points at loopback. (Nginx rewrites the Location header of a redirect back to the public name by default; it does nothing about links inside page bodies.) The app sees every visitor as 127.0.0.1, so rate limits and audit logs are wrong. And it has no way to know the request arrived over HTTPS once section 6 is done, which is where secure cookie and redirect loop problems come from. Put the four lines back and the same request shows, with my hostname and address in place of the placeholders:

Host: app.example.com
X-Forwarded-For: 203.0.113.7
X-Forwarded-Proto: http
X-Real-IP: 203.0.113.7

$proxy_add_x_forwarded_for appends the connecting address to whatever X-Forwarded-For already came in, so with a CDN in front the chain ends with the CDN's address and the app has to know how many hops to trust. X-Real-IP is always the address nginx saw. Which header your app trusts is the app's business. Web frameworks have a setting for "I am behind a proxy, believe these headers", and it is off by default because a client can send X-Forwarded-For: 198.51.100.66 itself; on a test box that forged value arrived at the app as 198.51.100.66, <the client address>. Trust the headers only on requests that came from 127.0.0.1, and only as many hops of the chain as you actually have.

5. Websockets

Anything that opens a websocket (a web terminal, a chat, most dashboards that update without a refresh) needs three more lines, and the failure without them is worth seeing once. My websocket app is a small echo server on 127.0.0.1:8081 (Python's websockets package, python3-websockets in the archive, and a client from the same package). I gave it a location /ws/ with proxy_pass http://127.0.0.1:8081/; and the Host header and nothing else. The client got server rejected WebSocket connection: HTTP 502 and nginx logged upstream prematurely closed connection while reading response header from upstream. The app was fine. Nginx had spoken HTTP/1.0 to it and dropped the Upgrade header, so the app closed a handshake it never saw. (If your websocket lives on the same port as the rest of the app, a bare location / sends the request to the app as an ordinary HTTP request and you get whatever the app answers with, a 200 or a 400, and the same fix applies.)

Two things fix it. A map outside the server block that turns the Connection header into upgrade only when the client asked for one, and a location for the websocket path that speaks HTTP/1.1 and passes both headers through:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    listen [::]:80;
    server_name app.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /ws/ {
        proxy_pass http://127.0.0.1:8081/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
    }
}

The map has to sit at the http level, which in a sites-available file means above server, not inside it. Test, reload, and my echo server answered echo: ping through the proxy. If the app serves the websocket on the same port and path as everything else, put the three extra lines in location / instead of a second location; they do no harm to ordinary requests.

The trailing slash on proxy_pass http://127.0.0.1:8081/ matters. With it, nginx strips the /ws/ prefix and the app sees /. Without it, the app would receive /ws/ and would need a route for that path. Either way, be deliberate.

6. HTTPS with certbot from the archive

Certbot's own instructions still send Ubuntu users to the snap. On 26.04 the archive has certbot 4.0.0 and its nginx plugin in universe, current enough that I see no reason to add snapd for it:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d app.example.com

Certbot answers the HTTP challenge through the running nginx on port 80, gets the certificate, and rewrites app.conf: it turns your existing server block into the HTTPS one, with listen 443 ssl and the ssl_certificate lines pointing into /etc/letsencrypt/live/app.example.com/, and adds a separate port 80 block that redirects to it. Renewal is a systemd timer the package installs, certbot.timer, which runs at midnight and noon with a random delay of up to twelve hours; systemctl list-timers certbot.timer shows the next run, and sudo certbot renew --dry-run proves it can renew without waiting sixty days to find out.

I could not run this step on the box used for this post, because its provider firewall only opens SSH and Let's Encrypt has to reach port 80. The commands above are certbot's documented flow, and the package versions are what 26.04 installs. Everything below in this section I did run with a self signed certificate standing in for the real one, which you can do too if your 80 is closed for now:

sudo openssl req -x509 -newkey rsa:2048 -nodes -days 30 -subj /CN=app.example.com \
  -keyout /etc/ssl/private/app.key -out /etc/ssl/certs/app.crt

# a second server block in app.conf, the shape certbot produces
server {
    listen 443 ssl;
    http2 on;
    server_name app.example.com;
    ssl_certificate /etc/ssl/certs/app.crt;
    ssl_certificate_key /etc/ssl/private/app.key;
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Test with curl -k, since the certificate is self signed, and swap the two ssl_ lines for certbot's when the port opens.

Certbot writes listen 443 ssl; and stops there. To turn HTTP/2 on, add one line inside that block:

listen 443 ssl;
http2 on;

Do not write listen 443 ssl http2;. Older configs and most guides still have that syntax. On nginx 1.28 it parses, but nginx -t prints:

[warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead

With http2 on; in place, curl --http2 -I https://app.example.com/ comes back as HTTP/2 200 (add -k for the self signed stand in). Also add X-Forwarded-Proto $scheme to the 443 block if certbot's rewrite did not carry it over. In that block $scheme is https, which is what tells your app to issue secure cookies.

7. Reading the errors: 502, 413, and a reload that has not landed yet

502 Bad Gateway means nginx accepted the request and could not get an answer from the app. I stopped the app (sudo systemctl stop app, mine is a unit called app, then curl -i http://app.example.com/) and got exactly this, with the cause in the log:

connect() failed (111: Connection refused) while connecting to upstream

The line to read is the most recent one in /var/log/nginx/error.log that matches your request. Connection refused is the app not listening (crashed, wrong port, bound to a different address). upstream prematurely closed connection is the app closing mid request, which is the websocket case from section 5, or an app that crashed on that specific request. A 504 after sixty seconds with upstream timed out in the log is proxy_read_timeout, whose default is 60s. It is the wait between two reads from the app, not a total budget, so a slow first byte is what trips it. Raise it in the location if the app legitimately takes longer.

413 Request Entity Too Large is nginx refusing an upload before the app ever sees it. The default client_max_body_size is 1m. A 2 MB POST to my proxy (head -c 2000000 /dev/zero > 2mb.bin; curl -i -X POST --data-binary @2mb.bin http://app.example.com/) got the 413 page from nginx with nothing in the app's log. Set client_max_body_size 100m; (or whatever fits) in the server or location block.

The reload that has not landed is section 3: a request in the same second as systemctl reload nginx can still hit an old worker. Wait a second and try again before debugging a config that is actually fine.

Gotchas I hit

The default site answered for my hostname after a reload. It was the graceful reload still swapping workers, not a config problem. Run the curl twice.
Websockets came back 502 with upstream prematurely closed connection in the log. The map block plus proxy_http_version 1.1, Upgrade and Connection in the location fixed it.
nginx -t warned the "listen ... http2" directive is deprecated. Copying a 22.04 era config onto 26.04 does this. Use listen 443 ssl; and http2 on; on its own line.
A 2 MB upload got 413. client_max_body_size defaults to 1m and the app never sees the request.
Without proxy_set_header Host $host my test app received 127.0.0.1:8080 as the Host header. Without X-Forwarded-For it had no idea who the visitor was.
The map for websockets goes above the server block, not inside it. Inside, nginx -t fails with "map" directive is not allowed here.
Certbot 4.0 comes from apt on 26.04. If you already have certbot from snap on the box, pick one installation and remove the other, so only one of them manages renewal.

Quick reference

# on a fresh box, in this order
# install and firewall
sudo apt install nginx
sudo ufw allow 'Nginx Full'

# site
sudo nano /etc/nginx/sites-available/app.conf
sudo ln -s /etc/nginx/sites-available/app.conf /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t && sudo systemctl reload nginx

# https
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d app.example.com
sudo certbot renew --dry-run

# when it breaks
sudo tail -n 20 /var/log/nginx/error.log
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/   # is the app even there

# app.conf: the map goes above the server block, and this location replaces the one in section 2
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# the location block, complete
location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    client_max_body_size 100m;    # uploads; default is 1m
    proxy_read_timeout 300s;      # only if the app has slow responses; default is 60s
}

The app still only listens on loopback. It just has a front door now.

Originally published at peculiarengineer.com.

Guide to DNS on Ubuntu 26.04

Minor Keith — Fri, 04 Sep 2026 02:36:06 +0000

For years my understanding of DNS on a Linux box was one file. /etc/resolv.conf has a nameserver line, that is where lookups go, done. Then at some point I opened it on an Ubuntu server and it said nameserver 127.0.0.53, which is a DNS server I never installed, running on an address I never configured, and I closed the file and got on with my day.

I have since written a Netplan post and three Tailscale posts that each wave at that address for a paragraph and move on. This is me finally sitting down to understand it. I took a fresh Ubuntu 26.04 server and followed one name lookup from the application all the way down to the server that answered, and then broke it in the ways I have seen it broken. Everything below is what the box printed.

TL;DR. Two layers. First the C library reads /etc/nsswitch.conf, which says hosts: files dns: check /etc/hosts, then ask DNS. "Ask DNS" means the nameserver in /etc/resolv.conf, which is a symlink to a file systemd-resolved writes, and it says 127.0.0.53. That is systemd-resolved itself, a local caching resolver, and it forwards to the real upstream servers it learned per network interface from DHCP or Netplan. resolvectl status shows those servers, resolvectl query shows which one answered and whether it came from cache. dig skips the first layer and talks to the stub directly, which is why dig and ping can disagree. Change upstreams in Netplan, not by editing resolv.conf, which systemd-resolved rewrites. Split DNS is one server per domain per interface, and it is how Tailscale's MagicDNS works.

The one idea worth holding onto

There are two layers, and most confusion about DNS on Linux comes from not knowing which one you are looking at.

The first layer is not DNS at all. It is the C library's name lookup, the code linked into every program on the box that turns a name into an address, and it consults a list of sources in order. On Ubuntu that list is /etc/hosts first, then DNS. Every normal program, ping, curl, ssh, your Python script, goes through this layer, and a name that /etc/hosts answers never becomes a DNS query.

The second layer is the resolver. On Ubuntu 26.04 that is systemd-resolved, listening on 127.0.0.53. It caches, it knows which upstream servers belong to which network interface, and it forwards your question to the right one. dig, host and nslookup skip the first layer entirely and talk to this one.

Hold onto that and the rest is detail. When something resolves in the browser but not in dig, or in dig but not in ping, the two layers disagree, and Section 5 is about exactly that.

Everything below was run as root on Ubuntu 26.04.1 with systemd 259. The box is a Hetzner server, which matters only because it ends up with four DNS servers from two different sources, and that turns out to be one too many for one of the files.

1. Start where the applications start

grep '^hosts:' /etc/nsswitch.conf

hosts:          files dns

That is the entire first layer on a 26.04 server. files is /etc/hosts. dns is the resolver library, which reads /etc/resolv.conf. Desktop installs usually carry an extra mdns4_minimal entry in there for .local names. The server image does not.

The tool for asking this layer a question directly, without a network round trip you did not ask for, is getent:

getent hosts example.com

2606:4700:10::6814:179a example.com
2606:4700:10::ac42:93f3 example.com

getent hosts does exactly what ping does to turn a name into an address, and nothing else. When a program cannot resolve a name, this is the first command to run, because it answers "is it the program or is it the box".

2. resolv.conf is a symlink, and there are two of them

ls -la /etc/resolv.conf
grep -v '^#' /etc/resolv.conf

lrwxrwxrwx 1 root root 39 Apr 20 18:07 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search .

So /etc/resolv.conf is not a file you own. It points at a file systemd-resolved regenerates, and that file names one server: the stub on 127.0.0.53. A stub resolver is one that owns no DNS data of its own. It takes questions from local programs and forwards them to servers that do. The options line turns on EDNS0, the extension that lets answers be bigger than the original 512 byte limit, and trust-ad, which lets the library trust a DNSSEC flag the stub sets. Both can stay unread until Section 11. The comment block at the top of it, which I cut here, says all of this in so many words, and ends with the line that matters: run resolvectl status to see the real upstream servers.

There is a second generated file next to it, and knowing it exists solves a whole category of problems later:

grep -v '^#' /run/systemd/resolve/resolv.conf

nameserver 2a01:4ff:ff00::add:2
nameserver 2a01:4ff:ff00::add:1
nameserver 185.12.64.1
# Too many DNS servers configured, the following entries may be ignored.
nameserver 185.12.64.2
search .

Same format, but listing the actual upstream servers instead of the stub. This is the file you point at when a program cannot use 127.0.0.53, and Docker is the usual example, in Section 10. That warning comment is real and worth knowing about: the classic C library resolver only reads the first three nameserver lines, so systemd-resolved writes the fourth one with a note that it will probably be ignored. The stub has no such limit, which is one small reason to leave the symlink pointing where it is.

3. Who is 127.0.0.53

ss -ulnp | grep ':53 '

UNCONN 0 0    127.0.0.54:53     0.0.0.0:*  users:(("systemd-resolve",pid=485,fd=18))
UNCONN 0 0 127.0.0.53%lo:53     0.0.0.0:*  users:(("systemd-resolve",pid=485,fd=16))

systemd-resolved, bound to loopback only, so nothing outside the box can ask it anything. It also listens on 127.0.0.54, which the documentation describes as a second stub with different forwarding rules. Nothing in resolv.conf points at it and I did not test it. .53 is the one this post is about.

Now the command the comment told you to run:

resolvectl status

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 2a01:4ff:ff00::add:2
       DNS Servers: 2a01:4ff:ff00::add:2 2a01:4ff:ff00::add:1 185.12.64.1
                    185.12.64.2
     Default Route: yes

Read it top to bottom. Global is settings that apply everywhere, and on a fresh box there is nothing in it except a list of protocols that are all switched off, which is the correct default for a server. Then one block per network interface. eth0 knows four servers, is currently using the first, and has Default Route: yes, meaning any name that no other interface claims goes here. Section 7 is about interfaces that claim names.

Where did those four servers come from? Not from a file you edited. The two IPv4 addresses arrived with the DHCPv4 lease, and the cloud image's Netplan file supplies the IPv6 pair statically:

grep -A3 nameservers /etc/netplan/50-cloud-init.yaml

      nameservers:
        addresses:
        - 2a01:4ff:ff00::add:2
        - 2a01:4ff:ff00::add:1

That is the shape to remember. Per interface, DNS servers come in from whatever configured the interface, DHCP or Netplan, systemd-networkd hands them to systemd-resolved, and systemd-resolved writes the two resolv.conf files to match. You never edit the files. You edit what feeds them.

4. Watch a lookup happen

resolvectl query asks the stub the same way an application would, and then tells you what it did:

resolvectl flush-caches
resolvectl query example.com

example.com: 2606:4700:10::ac42:93f3                        -- link: eth0
             2606:4700:10::6814:179a                        -- link: eth0
             104.20.23.154                                  -- link: eth0
             172.66.147.243                                 -- link: eth0

-- Information acquired via protocol DNS in 5.9ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

Which interface it went out of, how long it took, and where the answer came from. Ask again:

-- Information acquired via protocol DNS in 1.0ms.
-- Data from: cache

That cache is the main practical thing the stub gives you over a bare resolv.conf pointing at a remote server. Every repeated lookup on the box, and a busy box does thousands, is answered locally. You can see what is in it:

resolvectl show-cache

Scope protocol=dns ifindex=2 ifname=eth0 DNSSEC=no DNSOverTLS=no
example.com IN A 172.66.147.243
example.com IN A 104.20.23.154
example.com IN AAAA 2606:4700:10::6814:179a
example.com IN AAAA 2606:4700:10::ac42:93f3

and how it is doing overall with resolvectl statistics, which counts transactions, cache hits and misses, and timeouts. Mine showed 10 hits to 30 misses a few minutes into a fresh boot, which is what you expect before the cache warms up.

The tool I wish I had known about earlier is resolvectl monitor. It prints every question the stub receives and every answer it gives, live. Leave it running in one terminal and do anything in another:

→ Q: peculiarengineer.com IN AAAA
← S: success
← A: peculiarengineer.com IN AAAA 2606:4700:3031::6815:467a
← A: peculiarengineer.com IN AAAA 2606:4700:3036::ac43:df66

When you are not sure whether a program is even asking DNS, or what name it is asking for, this answers it in one line.

dig is the same lookup seen from the other side. It reads resolv.conf, so by default it asks the stub too, and it shows you the raw answer including the TTL, the lifetime in seconds the upstream stamped on the answer, counting down as the cache ages:

dig example.com +noall +answer

example.com.        155 IN  A   104.20.23.154

dig @1.1.1.1 example.com skips the stub and asks a server of your choosing, which is how you find out whether a wrong answer is coming from your box or from upstream. dig is preinstalled on 26.04 server, which it was not always.

5. Why dig and ping disagree

Put a name in /etc/hosts and ask each layer about it:

echo "10.99.99.9 mybox" >> /etc/hosts
getent hosts mybox
dig mybox +noall +answer
resolvectl query mybox

10.99.99.9      mybox
mybox.          0   IN  A   10.99.99.9
mybox: 10.99.99.9
-- Data from: synthetic

I expected dig to fail here, because dig skips /etc/hosts. It did not, because on Ubuntu the stub it asks is systemd-resolved, and systemd-resolved reads /etc/hosts itself and serves those entries with a TTL of zero. resolvectl is honest about it: Data from: synthetic, meaning the resolver made the answer up from local knowledge and never sent a packet. So on 26.04 the two layers agree about /etc/hosts. The disagreement is elsewhere.

Ask for a .local name:

resolvectl query printer.local

printer.local: resolve call failed: No appropriate name servers or networks for name found

.local is reserved for multicast DNS, and mDNS is off on a server (-mDNS in the status output). systemd-resolved refuses to send a .local name to a real DNS server, by design, so it fails instantly with that message. A desktop with mdns4_minimal in nsswitch.conf would have answered from the first layer without DNS being involved, which is why the printer resolves on your laptop and not on the server.

And the box's own name:

getent hosts "$(hostname)"

127.0.1.1       dns-post dns-post

That is an /etc/hosts line the installer wrote, pointing the hostname at 127.0.1.1, the second loopback address. It is why ping $(hostname) works on a box with no DNS at all, and why a service that binds to "the hostname" sometimes ends up listening only on loopback.

6. Changing the upstream, and what survives a reboot

There are two ways, and the difference is what happens tomorrow.

The quick way is resolvectl, and it takes effect immediately:

resolvectl dns eth0 1.1.1.1 9.9.9.9
resolvectl dns eth0

Link 2 (eth0): 1.1.1.1 9.9.9.9

I wanted to know exactly how temporary that is, so I tried to knock it out. netplan apply did not touch it. networkctl reconfigure eth0 did not touch it either, which surprised me. A reboot did: the box came back with Hetzner's four servers and no memory of mine. So a resolvectl dns change lasts until the next boot, and no less, which makes it fine for "try this server for an hour" and wrong for anything else.

The permanent way is Netplan. Add a file, and let it feed the chain from the top:

# /etc/netplan/60-dns.yaml
network:
  version: 2
  ethernets:
    eth0:
      nameservers:
        addresses: [1.1.1.1, 9.9.9.9]

chmod 600 /etc/netplan/60-dns.yaml
netplan apply
resolvectl status eth0 | grep -i 'DNS Server'

Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1 9.9.9.9

And now look at the two files from Section 2. /etc/resolv.conf still says nameserver 127.0.0.53, unchanged, because the stub did not move. /run/systemd/resolve/resolv.conf now lists 1.1.1.1 and 9.9.9.9, because the upstreams did. That is the whole model working as intended: you changed the input, both outputs followed, and no program on the box noticed anything.

Netplan merges every file in /etc/netplan/ by filename order, so a 60- file layers over the cloud image's 50-cloud-init.yaml without you editing a file that cloud-init might regenerate. Note what the merge did to the list, though. eth0 went from four servers to exactly two, so the nameservers list is replaced, not appended, and the IPv6 pair from the cloud image is gone. If you want them, put them in your file too. The Netplan post goes into that ordering.

7. Split DNS: a different server for one domain

This is the part that made the design click for me, because it is the thing a single resolv.conf fundamentally cannot do.

The situation: your normal DNS is fine for the internet, but names under corp.example only exist on a private server. In the old world you either pointed the whole box at the private server and hoped it forwarded everything else, or you gave up. With per interface servers and routing domains, you attach the private server to an interface and tell systemd-resolved which names belong to it. A routing domain is written with a leading ~: ~corp.example means "send questions about this domain to this interface's servers", and ~. means "send everything else here", which is what Default Route: yes in Section 3 was showing you.

To try it without a corporate network I ran a tiny DNS server on the box that invents answers for one domain. dnsmasq will do that in four lines of config. Installing it produced the first lesson for free:

dnsmasq: failed to create listening socket for port 53: Address already in use

The stub already has port 53 on loopback, and dnsmasq's default is to bind everything. Pin it to a different loopback address and give it the fake zone:

# /etc/dnsmasq.d/corp.conf
listen-address=127.0.0.2
bind-interfaces
port=53
no-resolv
address=/corp.example/10.7.7.7
address=/git.corp.example/10.7.7.10

Then an interface to hang it on. A dummy interface is fine for the demonstration, a VPN interface is what you would have in real life:

ip link add corp0 type dummy && ip link set corp0 up
ip addr add 10.7.7.1/24 dev corp0
resolvectl dns corp0 127.0.0.2
resolvectl domain corp0 '~corp.example'
resolvectl status corp0

Link 3 (corp0)
    Current Scopes: DNS
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
       DNS Servers: 127.0.0.2
        DNS Domain: ~corp.example
     Default Route: no

Default Route: no. This interface's server only ever sees names under corp.example. Now query one of each:

resolvectl query git.corp.example
resolvectl query example.com

git.corp.example: 10.7.7.10
-- Information acquired via protocol DNS in 3.1ms.

example.com: 104.20.23.154 ...                              -- link: eth0
-- Information acquired via protocol DNS in 18.1ms.

The private name went to 127.0.0.2 on corp0. The public name went out eth0 as before. Neither server saw the other's traffic. And /etc/resolv.conf still says nameserver 127.0.0.53 and nothing else, so every program on the box gets this behaviour without knowing it exists. dig git.corp.example returns 10.7.7.10 too, through the stub.

resolvectl domain

Link 2 (eth0): ~.
Link 3 (corp0): ~corp.example

That two line table is the routing policy for every name on the machine. If you have ever run resolvectl status on a box with Tailscale and wondered what the tailscale0 block with its own DNS server and a ~ domain was for, this is it. MagicDNS is exactly this shape: Tailscale attaches its resolver to its interface, claims the tailnet's domain, and leaves ~. alone.

8. Search domains and single labels

A search domain is what turns git into git.corp.example so you can type the short name. It is per interface, like everything else here, and it is wired into both layers. Using the corp0 interface from Section 7, with the domain written without the ~ this time:

resolvectl domain corp0 corp.example
grep '^search' /etc/resolv.conf
getent hosts git
resolvectl query git

search corp.example
10.7.7.10       git.corp.example
git: 10.7.7.10
     (git.corp.example)

systemd-resolved wrote the domain into the search line of resolv.conf, so the C library appends it, and it also applies it itself when asked directly, which is why resolvectl query git shows the name it actually looked up in brackets. dig needs to be told: dig +search git works, plain dig git gets status: REFUSED from the stub.

That refusal is deliberate and it will bite you once. systemd-resolved does not send single label names, a bare git or mybox with no dots, to a DNS server at all unless a search domain completes them. Without one you get:

mybox: resolve call failed: No appropriate name servers or networks for name found

The setting behind it is ResolveUnicastSingleLabel=no in resolved.conf, and the default is right: a single label leaking to the internet is how a typo becomes a query to somebody else's server.

The ~ form from Section 7 is the other kind. ~corp.example routes queries for the domain to this interface's servers and nothing more: it does not complete short names. Drop the ~ and you get both routing and completion, which is what this section just used.

9. DNS over TLS, and the setting that takes the box offline

Everything so far went over plain UDP. systemd-resolved can encrypt the leg to the upstream, and the configuration lives in a drop-in, a small file under resolved.conf.d that overrides the main resolved.conf without editing it:

# /etc/systemd/resolved.conf.d/dot.conf
[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com
DNSOverTLS=yes

systemctl restart systemd-resolved
resolvectl query example.com | tail -2
ss -tnp | grep ':853'

-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes
ESTAB 0 0 178.105.172.189:53968 1.1.1.1:853 users:(("systemd-resolve",pid=2934,fd=24))

An open TLS connection to port 853, and resolvectl reports the encrypted transport. The #hostname after each address is the certificate name to verify, and you want it there.

Here is the trap, and I walked into it on purpose. DNSOverTLS=yes is strict mode, and it applies to every server on every interface, including the ones DHCP handed you. I removed the DNS= line so the only servers left were Hetzner's, which do not speak TLS, and asked for a name:

(nothing. the twenty second timeout I had wrapped it in gave up, exit 124)

DNS on the box was simply gone. resolvectl status eth0 showed +DNSOverTLS against servers that would never answer on 853. So strict mode is only correct when you have also set the upstreams to servers you know support it, in the same drop-in. DNSOverTLS=opportunistic is the softer setting, it tries TLS and falls back to plain, and against the same Hetzner servers it worked immediately, unencrypted, with no connection to 853 at all. Opportunistic is honest about what it gives you, which is not much. Either set strict with known servers, or leave it off.

10. Three ways it gets broken

Someone replaces the symlink. The most common one, usually from a guide written for a distribution without systemd-resolved, or a VPN client from years ago. I did it the way they do it:

cp --remove-destination /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now /etc/resolv.conf is a regular file listing the upstreams directly. Everything still resolves, so nobody notices. What you lost: resolvectl statistics shows a cache size of zero and stays there, resolvectl monitor sees nothing when programs look names up, split DNS and search domains stop applying to anything except resolvectl itself, and the next DHCP change never reaches the file. Put it back:

ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Docker containers. A container cannot use 127.0.0.53, because inside the container loopback is the container's own. So Docker does not copy /etc/resolv.conf; it copies the other file, the one with the real upstreams:

docker run --rm alpine:3.20 cat /etc/resolv.conf

nameserver 1.1.1.1
nameserver 9.9.9.9
search .

Those are the servers from Section 6, taken from /run/systemd/resolve/resolv.conf, with no cache and no split DNS. On a user defined network Docker instead gives the container nameserver 127.0.0.11, its own embedded resolver, which forwards to the same upstreams and adds container name resolution on top. Either way, a private name that resolves on the host through a routing domain will not resolve inside a container unless you pass --dns or configure the daemon.

Something else wants port 53. dnsmasq did it in Section 7, Pi-hole does it, so does any local DNS server installed with defaults. The failure is Address already in use and the service does not start. Either bind the new server somewhere other than 127.0.0.53, as I did, or if the new server is meant to replace the stub entirely, stop and disable systemd-resolved and manage resolv.conf yourself. Do one or the other. Running both on the same address is not a state the box can be in.

11. Debugging it

The commands, in the order I now reach for them:

getent hosts NAME
resolvectl query NAME
resolvectl status
resolvectl monitor
dig @UPSTREAM NAME

getent for what a program sees, query for which link and source answered, status for who owns the default route, monitor when you are not sure a program is asking at all, and dig at an upstream to take the box out of the picture.

When those do not explain it, turn the resolver's own logging up for a minute:

resolvectl log-level debug
resolvectl query example.com >/dev/null
journalctl -u systemd-resolved --since -1min | grep example.com
resolvectl log-level info

Cache miss for example.com IN AAAA
Firing regular transaction 19300 for <example.com IN AAAA> scope dns on eth0/* (validate=yes).
Cache miss for example.com IN A

That tells you the exact scope and interface a query was sent on, which is the question split DNS problems come down to. And resolvectl show-server-state shows what the resolver has learned about each upstream, such as whether it supports EDNS0 and how many UDP and TCP attempts have failed against it. A server with Failed UDP attempts climbing is a server on the way out, and resolvectl reset-server-features makes systemd-resolved forget and re-probe.

A name that does not exist looks like this, and exits 1, which is useful in scripts:

does-not-exist.peculiarengineer.com: Name 'does-not-exist.peculiarengineer.com' not found

Gotchas worth knowing

/etc/resolv.conf is output, not input. Edit what feeds it: Netplan for permanent, resolvectl dns for the rest of today. If it is a regular file instead of a symlink, somebody broke it, and ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf fixes it.

A resolvectl dns change survives netplan apply and networkctl reconfigure and dies at reboot. Longer lived than I assumed and shorter than you need.

Four DHCP servers is one too many for the classic resolver. Only the stub reads all four. Anything reading /run/systemd/resolve/resolv.conf directly, Docker included, ignores the fourth, and the file says so in a comment.

Single label names never leave the box. No search domain, no query. The error is No appropriate name servers or networks for name found, and it is not a network problem.

.local fails on a server on purpose. mDNS is off and systemd-resolved will not send .local to unicast DNS. Turn on mDNS or use a real name.

DNSOverTLS=yes with DHCP servers is an outage. Strict mode applies to every server on every link. Set DNS= to known TLS servers in the same drop-in, or use opportunistic and accept that it will mostly be plaintext.

dig sees /etc/hosts on Ubuntu. Because the stub it talks to reads the file. Data from: synthetic in resolvectl query is how you tell.

Docker gets the upstreams, not the stub. No cache, no split DNS, no search domains inside containers unless you configure them. Private names that work on the host will not work in a container by default.

Port 53 is taken. By the stub, on 127.0.0.53 and 127.0.0.54. A local DNS server has to bind elsewhere or replace systemd-resolved outright.

Quick reference

Task	Command
Resolve a name the way programs do	`getent hosts NAME`
Resolve with details: link, cache, timing	`resolvectl query NAME`
Servers and routing per interface	`resolvectl status`
Just the servers	`resolvectl dns`
Just the domains	`resolvectl domain`
Watch every query live	`resolvectl monitor`
What is cached	`resolvectl show-cache`
Hit and miss counts	`resolvectl statistics`
Empty the cache	`resolvectl flush-caches`
Change servers until reboot	`resolvectl dns eth0 1.1.1.1 9.9.9.9`
Change servers permanently	`nameservers:` in a `/etc/netplan/*.yaml`, then `netplan apply`
Add a search domain until reboot	`resolvectl domain eth0 corp.example`
Route one domain to one link	`resolvectl domain IFACE '~corp.example'` plus `resolvectl dns IFACE SERVER`
Ask an upstream directly	`dig @1.1.1.1 NAME`
Per upstream health	`resolvectl show-server-state`
Verbose resolver log	`resolvectl log-level debug`, then `journalctl -u systemd-resolved -f`
Restore the symlink	`ln -sf ../run/systemd/resolve/stub-resolv.conf /etc/resolv.conf`
The upstream list for things that cannot use the stub	`/run/systemd/resolve/resolv.conf`

The next time /etc/resolv.conf says 127.0.0.53, that is not a mystery, it is a table of contents. resolvectl status is the book. [ 127.0.0.53 ✓ ]

Originally published at peculiarengineer.com.

Systemd units and timers on Ubuntu 26.04, helpful guide

Minor Keith — Fri, 04 Sep 2026 01:49:06 +0000

I have now written "run it from a systemd timer" in three posts, pointed at a timer each time, and never once explained what one is. The restic post has a timer in it that I asked you to copy on faith. This is the post those should have linked to.

It is written for someone who has never opened a unit file. If you have used cron, you already know the job: run this script at that time. What is different is everything around the job. Where the output goes, what happens when the box was off at three in the morning, what happens when the script fails, and why it fires a few seconds late. I built one job on a fresh Ubuntu 26.04 server and then spent an evening measuring those things instead of assuming them.

TL;DR. Two files in /etc/systemd/system/. job.service says what to run (Type=oneshot, ExecStart=/absolute/path). job.timer says when (OnCalendar=), and it starts the service with the same name. systemctl daemon-reload after every edit, then systemctl enable --now job.timer, the timer, never the service. Add Persistent=true so a run missed while the box was off happens at boot, and AccuracySec=1s if "03:30" has to mean 03:30. Test with systemctl start job.service instead of waiting. Your script runs with no shell, no HOME, a five entry PATH, and every % and $ has a meaning to systemd before your script ever sees it.

The one idea worth holding onto

systemd manages things called units. A service unit is a thing to run. A timer unit is a clock that starts some other unit when it goes off. That is the whole model, and cron has no equivalent to it: in cron the schedule and the command live on one line, and the moment the command runs, cron forgets about it.

Here they are two files with the same name and different extensions. backup.service is what. backup.timer is when. The timer activates the service by name, and because the service is a real unit in its own right, you can start it by hand, read its last exit code, and see its output, all without the timer being involved. Most of what makes timers better than cron falls out of that split.

Everything below was run on Ubuntu 26.04.1 with systemd 259, as root unless I say otherwise. The box is on UTC, which matters in Section 5.

What is already running on your box

You do not have to write a timer to see one. A fresh 26.04 server ships with eighteen:

systemctl list-timers --all

NEXT                                LEFT LAST PASSED UNIT                           ACTIVATES
Thu 2026-09-03 22:50:00 UTC     2min 57s -         - sysstat-collect.timer          sysstat-collect.service
Thu 2026-09-03 23:01:21 UTC        14min -         - systemd-tmpfiles-clean.timer   systemd-tmpfiles-clean.service
Fri 2026-09-04 00:00:00 UTC     1h 12min -         - dpkg-db-backup.timer           dpkg-db-backup.service
Fri 2026-09-04 00:12:59 UTC     1h 25min -         - logrotate.timer                logrotate.service
Fri 2026-09-04 06:41:12 UTC           7h -         - apt-daily-upgrade.timer        apt-daily-upgrade.service
Fri 2026-09-04 10:09:56 UTC          11h -         - apt-daily.timer                apt-daily.service
Mon 2026-09-07 01:02:03 UTC       3 days -         - fstrim.timer                   fstrim.service
-                                      - -         - snapd.snap-repair.timer        snapd.snap-repair.service
...
18 timers listed.

Log rotation, the apt update check, filesystem trimming, the man database: on Ubuntu none of these are cron jobs any more. Each is a timer starting a service. The - rows at the bottom are timers that are loaded but will never fire, often because a condition in the unit is false on this machine.

The fastest way to learn the format is to read one Ubuntu wrote:

systemctl cat fstrim.timer

# /usr/lib/systemd/system/fstrim.timer
[Unit]
Description=Discard unused filesystem blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container
ConditionPathExists=!/etc/initrd-release

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
RandomizedDelaySec=100min

[Install]
WantedBy=timers.target

Every line in that [Timer] block gets its own section below, because Ubuntu's choices there are good ones and it is worth knowing why. The two Condition lines are why this timer does nothing inside a container: a condition that is false makes the unit load and then quietly never run. Note the path too. Ubuntu's units live in /usr/lib/systemd/system/. Yours go in /etc/systemd/system/, and if both places hold a file with the same name, /etc wins. Do not edit anything under /usr/lib; Section 14 shows how to change a vendor timer without touching it.

1. The job

Start with something that prints what it can see, because what it can see is the surprising part.

cat > /usr/local/bin/hello-timer.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
echo "hello from $(hostname) at $(date '+%F %T'), pid $$, user $(id -un)"
echo "PATH is: $PATH"
echo "HOME is: ${HOME:-unset}"
EOF
chmod +x /usr/local/bin/hello-timer.sh
/usr/local/bin/hello-timer.sh

hello from systemd-timers-post at 2026-09-03 22:47:04, pid 1741, user root
PATH is: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
HOME is: /root

Remember those last two lines. They are about to change.

2. The service unit

# /etc/systemd/system/hello.service
[Unit]
Description=Say hello (demo job)

[Service]
Type=oneshot
ExecStart=/usr/local/bin/hello-timer.sh

Three things to notice. Type=oneshot tells systemd this is a job that runs and exits, not a daemon that stays up, so "started successfully" means "exited zero" rather than "the process is alive". ExecStart is the command. And there is no [Install] section, on purpose: this unit is started by the timer, not at boot, so it has no place to be enabled into. Section 4 shows what happens when you try.

Tell systemd the file exists, run it once by hand, and look:

systemctl daemon-reload
systemctl start hello.service
systemctl status hello.service

○ hello.service - Say hello (demo job)
     Loaded: loaded (/etc/systemd/system/hello.service; static)
     Active: inactive (dead)

Sep 03 22:47:05 systemd-timers-post systemd[1]: Starting hello.service - Say hello (demo job)...
Sep 03 22:47:05 systemd-timers-post hello-timer.sh[1873]: hello from systemd-timers-post at 2026-09-03 22:47:05, pid 1873, user root
Sep 03 22:47:05 systemd-timers-post hello-timer.sh[1873]: PATH is: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin
Sep 03 22:47:05 systemd-timers-post hello-timer.sh[1873]: HOME is: unset
Sep 03 22:47:05 systemd-timers-post systemd[1]: hello.service: Deactivated successfully.
Sep 03 22:47:05 systemd-timers-post systemd[1]: Finished hello.service - Say hello (demo job).

inactive (dead) is the correct resting state for a oneshot. It ran, it finished, it is not running now. The script's output went straight into the journal with no redirection on your part, which is the first thing cron never gave you. The same lines are there whenever you want them:

journalctl -u hello.service -o short-iso

And there are the two changed lines. PATH has five entries instead of nine, and HOME is not set at all. Your script did not run in your shell. It ran in systemd's idea of a clean environment, and Section 10 is about how far that goes.

One thing I expected to be a gotcha and was not. Every guide says ExecStart needs an absolute path. I tried ExecStart=hello-timer.sh and it ran, because systemd has looked bare command names up in its own PATH for years now. Write the absolute path anyway. It costs nothing and it means the unit does not depend on where PATH points.

3. daemon-reload, and why your edit did nothing

systemd reads unit files once and keeps them in memory. Edit the file on disk and nothing happens until you say daemon-reload. Everyone gets bitten by this exactly once, so here is what it looks like. I changed the Description= line and then asked:

systemctl status hello.service

Warning: The unit file, source configuration file or drop-ins of hello.service changed on disk. Run 'systemctl daemon-reload' to reload units.
○ hello.service - Say hello (demo job)

The warning is loud, but only if you run status. systemctl start prints it too and then starts the old definition regardless:

systemctl start hello.service
systemctl show hello.service -p Description

Warning: The unit file, source configuration file or drop-ins of hello.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Description=Say hello (demo job)

Still the old text. After systemctl daemon-reload it reads Description=Say hello (edited on disk) and the warning goes away. The habit to build is that daemon-reload is part of saving the file, not a separate step you do when something looks wrong.

4. The timer unit

# /etc/systemd/system/hello.timer
[Unit]
Description=Run hello every minute (demo)

[Timer]
OnCalendar=*-*-* *:*:00
Unit=hello.service

[Install]
WantedBy=timers.target

OnCalendar is the schedule, here every minute on the minute, which is a terrible schedule for anything real and a very good one for watching it work. Unit= names what to start, and you can leave it out when the service has the same name as the timer, which it does here and should in general. The [Install] section is what makes enable mean something: timers.target is the unit that starts all timers at boot, and enabling this timer hooks it in there.

This is also the moment to see what enabling the service instead would have done, because it is the most common first mistake:

systemctl enable hello.service

The unit files have no installation config (WantedBy=, RequiredBy=, UpheldBy=,
Also=, or Alias= settings in the [Install] section, and DefaultInstance= for
template units). This means they are not meant to be enabled or disabled using systemctl.
...

Correct, and that is exactly why the service has no [Install] section. If it had one with WantedBy=multi-user.target, enabling it would make the job run once at every boot, which for a backup or a cleanup is a surprise nobody wants. Enable the timer:

systemctl daemon-reload
systemctl enable --now hello.timer
systemctl list-timers hello.timer

Created symlink '/etc/systemd/system/timers.target.wants/hello.timer' → '/etc/systemd/system/hello.timer'.
NEXT                        LEFT LAST PASSED UNIT        ACTIVATES
Thu 2026-09-03 22:48:00 UTC  51s -         - hello.timer hello.service

--now starts it as well as enabling it, and list-timers is the command you will run more than any other in this post. systemctl status hello.timer shows the same in prose, with a Trigger: line and a Triggers: ● hello.service line that ties the two files together.

Wait two minutes and read the service's journal, and the job is running itself:

2026-09-03T22:48:27+00:00 systemd-timers-post hello-timer.sh[2109]: hello from systemd-timers-post at 2026-09-03 22:48:27, pid 2109, user root

Look at that timestamp though. The timer was for 22:48:00 and the job ran at 22:48:27. Section 6.

5. Reading and writing OnCalendar

The format is DayOfWeek Year-Month-Day Hour:Minute:Second, every part optional, * for any, .. for ranges, , for lists, / for steps. You do not have to remember that, because systemd-analyze calendar will tell you what any expression means and when it next fires, and it is the tool to reach for before you put anything in a unit file:

systemd-analyze calendar "Mon..Fri 09:00"

  Original form: Mon..Fri 09:00
Normalized form: Mon..Fri *-*-* 09:00:00
    Next elapse: Fri 2026-09-04 09:00:00 UTC
       From now: 10h left

Here is a table I built by feeding it the expressions I actually use, so I stop working them out from the man page:

You write	systemd reads it as	Meaning
`daily`	`--* 00:00:00`	midnight every day
`03:30`	`--* 03:30:00`	03:30 every day
`hourly`	`--* *:00:00`	top of every hour
`*:0/15`	`--* *:00/15:00`	every 15 minutes
`weekly`	`Mon --* 00:00:00`	Monday midnight
`Mon..Fri 09:00`	`Mon..Fri --* 09:00:00`	weekday mornings
`Mon,Wed,Fri 18:00`	`Mon,Wed,Fri --* 18:00:00`	three evenings a week
`--01 00:00:00`	`--01 00:00:00`	first of the month
`quarterly`	`*-01,04,07,10-01 00:00:00`	first of Jan, Apr, Jul, Oct
`2026-12-25 08:00`	`2026-12-25 08:00:00`	once, on that date

Two things worth knowing. --iterations=3 shows the next three firings instead of one, which is how you catch that weekly means Monday and not "seven days from now". And a real typo is rejected rather than guessed at:

systemd-analyze calendar "every day at 3"

Failed to parse calendar specification 'every day at 3': Invalid argument

That same rejection is what you get if the typo makes it into a timer unit, except there it is quieter. systemd-analyze verify /etc/systemd/system/bad.timer prints Failed to parse calendar specification, ignoring and Timer unit lacks value setting. Refusing, and systemctl start fails with bad unit file setting. Run verify on anything new before you enable it. One thing it forgives: I typed Mon-Fri 09:00 with a single hyphen, and it silently normalised it to Mon..Fri.

Timezones. OnCalendar is read in the box's local timezone, which on a fresh server is almost always UTC. Check with timedatectl. If the box is on local time and you want a job in UTC anyway, append it: *-*-* 03:30:00 UTC is valid and systemd-analyze calendar shows the UTC in the normalised form.

6. Why it fired late

The 22:48:00 job ran at 22:48:27. The next one, scheduled for 22:49:00, ran at 22:49:18. Nothing is wrong. Timers have a setting called AccuracySec, and its default is one minute:

systemctl show hello.timer -p AccuracyUSec

AccuracyUSec=1min

Which means: fire at any point in the minute after the scheduled time. systemd does this so it can batch timers together and wake the machine up less often, and for a nightly backup a few seconds of drift is nothing. For anything where the exact minute matters, say a job that has to land before another system polls at :05, set it explicitly. I ran the same minute timer twice, once with the default and once with AccuracySec=1s, and logged the moment the job started:

default:           fired at 22:50:03.959   fired at 22:51:08.954
AccuracySec=1s:    fired at 22:52:00.147   fired at 22:53:00.147

With the default I saw the job land anywhere from four to twenty seven seconds after the minute. With AccuracySec=1s it landed 150 milliseconds after, both times. Ubuntu's own fstrim.timer goes the other way and sets AccuracySec=1h, because nobody cares which hour of Monday the disks get trimmed. Pick the one that matches how much you care.

7. The box was off at 03:30

This is the single best reason to use a timer over cron for anything on a machine that is ever switched off or rebooted. Cron skips a missed run and says nothing. A timer with Persistent=true remembers.

# /etc/systemd/system/daily.timer
[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

The mechanism is a stamp file. Every time the timer fires it touches /var/lib/systemd/timers/stamp-daily.timer, and when the timer is next activated, systemd compares that stamp to the schedule. If a run should have happened in between, it runs the job immediately.

Two things I wanted to know before trusting it. First, does a brand new persistent timer fire the moment you enable it, on the theory that it has "missed" every run since the beginning of time? No. On first enable it created the stamp and waited for midnight like any other timer. Second, does it really catch up? I made it look as if the last run was two days ago and started the timer:

systemctl stop daily.timer
touch -d "2 days ago" /var/lib/systemd/timers/stamp-daily.timer
systemctl start daily.timer
journalctl -u daily.service -o short-iso --since "-1min"

2026-09-03T23:00:43+00:00 systemd-timers-post sh[4482]: daily job ran at 23:00:43

Immediately. The same test with the Persistent=true line removed did nothing, and list-timers showed it calmly waiting for midnight.

Then I did it properly. I set the timer for 23:10, powered the server off at 23:07, and turned it back on through the Hetzner API at 23:11:48, so the slot passed with the machine dark:

uptime -s            2026-09-03 23:12:01
2026-09-03T23:12:11+00:00 systemd-timers-post systemd[1]: Starting daily.service...
2026-09-03T23:12:11+00:00 systemd-timers-post sh[964]: daily job ran at 23:12:11

Ten seconds after boot, the missed run happened. A cron job at 23:10 would have been silently skipped and you would have found out when you went looking for last night's backup. Every timer Ubuntu ships that does real work, fstrim, apt-daily, logrotate, has Persistent=true. Yours should too.

8. Timers that count from boot, and jitter

OnCalendar is wall clock time. There is a second family that counts durations instead, and they are useful for "a while after boot" and "every so often" without caring what time it is:

[Timer]
OnBootSec=15min
OnUnitActiveSec=1h

OnBootSec=15min fires fifteen minutes after the machine came up. OnUnitActiveSec=1h fires an hour after the service last ran. Together they mean "first run fifteen minutes after boot, then hourly", which is a better shape for a health check than hourly, because it does not pile onto whatever else runs at the top of the hour. list-timers showed mine for 23:01:21 on a box that booted at 22:46:21, which is the arithmetic you would expect.

RandomizedDelaySec spreads a run out over a window. Ubuntu's apt-daily.timer uses RandomizedDelaySec=12h so that every Ubuntu machine on earth does not hit the mirrors at the same second. I put RandomizedDelaySec=30min on a 03:30 timer and restarted it three times:

Fri 2026-09-04 03:49:13 UTC
Fri 2026-09-04 03:44:33 UTC
Fri 2026-09-04 03:44:47 UTC

A fresh point in the window each time. This is why the restic post has RandomizedDelaySec=30m on its backup: three homelab boxes all backing up to the same bucket at exactly 03:30 is a small self inflicted thundering herd.

9. When the job fails

A script that exits nonzero fails the service, and systemd is not shy about it:

systemctl start flaky.service
systemctl status flaky.service

Job for flaky.service failed because the control process exited with error code.
See "systemctl status flaky.service" and "journalctl -xeu flaky.service" for details.

× flaky.service - A job that fails
     Active: failed (Result: exit-code) since Thu 2026-09-03 22:54:41 UTC; 20ms ago
    Process: 3305 ExecStart=/bin/sh -c echo "about to fail"; exit 3 (code=exited, status=3)

The × and failed stick until you clear them, and systemctl list-units --failed lists every unit on the box in that state, which is a decent morning check on any server. systemctl reset-failed flaky.service clears it. One thing that will confuse you the first time: the journal line reads status=3/NOTIMPLEMENTED. systemd keeps a table of names for exit codes, inherited from the old LSB init script convention where 3 meant "unimplemented feature", and it prints the name next to any code it recognises. Your script exited 3 and that is all it means.

Nothing emails you, and that is the honest gap compared to cron's MAILTO. The building block systemd gives you instead is OnFailure=, which starts another unit when this one fails. Here is the smallest useful version. It uses a template unit, which is worth a sentence because the filename looks odd: a unit named something@.service is a reusable definition you start with an argument after the @, and inside the file %i stands for that argument. So notify-failed@flaky.service is the template run with flaky.service as its argument. The alert goes into the journal at error priority, where anything you already use to watch logs will see it:

# /etc/systemd/system/flaky.service
[Unit]
Description=A job that fails
OnFailure=notify-failed@%n.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "about to fail"; exit 3'

# /etc/systemd/system/notify-failed@.service
[Unit]
Description=Log a failure for %i

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo "ALERT: %i failed on $(hostname), see: journalctl -u %i" | systemd-cat -t failure-alert -p err'

%n in the first file expands to the failing unit's own name, which becomes the argument, and %i in the template picks it up, so one notify-failed@.service serves every job on the box. On the systemd-cat line, -t tags the message so you can find it and -p err sets the priority. The result:

journalctl -t failure-alert -o short-iso

2026-09-03T22:54:41+00:00 systemd-timers-post failure-alert[3311]: ALERT: flaky.service failed on systemd-timers-post, see: journalctl -u flaky.service

Swap the echo | systemd-cat for a curl to whatever pings your phone and you have cron's MAILTO, but per job and without a mail server.

10. Your script is not running in your shell

Section 2 showed HOME unset and a short PATH. It goes further than that, and every item below is something I have seen in a real unit file that did not do what its author thought.

There is no shell

ExecStart is split into words by systemd and executed directly. Pipes, redirects and && are just arguments:

ExecStart=/bin/echo hello | /usr/bin/tr a-z A-Z

hello | /usr/bin/tr a-z A-Z

That printed the pipe. If you need shell syntax, ask for a shell: ExecStart=/bin/sh -c 'echo hello | tr a-z A-Z' printed HELLO. Better still, put anything longer than one command in a script file and point ExecStart at the script, which is what Section 1 did.

No tilde, no globs

~ stays a literal ~ and * stays a literal *, for the same reason: those are shell features.

`$` means something to systemd first

systemd does its own variable expansion on ExecStart before your command runs, and it has rules that are almost, but not quite, the shell's. I tested every spelling against Environment=NAME=keith:

ExecStart=/bin/echo bare: $NAME          ->  bare: keith
ExecStart=/bin/echo braces: ${NAME}      ->  braces: keith
ExecStart=/bin/echo quoted: "$NAME"      ->  quoted: keith
ExecStart=/bin/echo doubled: $$NAME      ->  doubled: $NAME
ExecStart=/bin/sh -c 'echo shell: $NAME' ->  shell: keith

$$ is how you get a literal dollar through to your command. Where this bites is shell constructs that systemd half understands: I wrote ${#SECRET_TOKEN} inside an sh -c to print a length, systemd saw ${...}, looked for a variable literally named #SECRET_TOKEN, and substituted nothing. The output was silently empty. Plain $SECRET_TOKEN inside the quotes made it through to the shell and worked.

`%` means something to systemd too

Percent signs in unit files are specifiers: %n is the unit name, %H the hostname, %u the user, %T the temp directory. So this:

ExecStart=/bin/sh -c 'echo "daily job ran at $(date +%T)"'

logs daily job ran at /tmp, because %T became /tmp before date ever saw it. I did this in my own test unit and stared at it for a minute. Double it, %%T, and date gets its %T back.

Set the environment on purpose

The useful knobs, together:

[Service]
Type=oneshot
User=keith
WorkingDirectory=/home/keith
EnvironmentFile=/etc/hello.env
ExecStart=/bin/sh -c 'echo "user=$(id -un) pwd=$(pwd) token=$SECRET_TOKEN"'

user=keith pwd=/home/keith token=abc123

User= runs the job as someone other than root, which most jobs should. WorkingDirectory= matters because the default is /. EnvironmentFile= reads KEY=value lines from a file, which is where a token or a password belongs, in a root owned file with mode 600, and not in the unit file that systemctl cat will happily print to anyone. The restic post's Environment=HOME=/root line exists precisely because of the HOME is: unset result from Section 2, restic wanted a home directory for its cache, and a system service does not get one for free.

11. When the job takes longer than the interval

What if the timer fires again while the previous run is still going? I gave a service a hundred second sleep and a timer that fires every minute:

2026-09-03T22:55:00+00:00 slow start
2026-09-03T22:56:40+00:00 slow end
2026-09-03T22:56:40+00:00 slow start

No second copy started at 22:56:00. Starting a service that is already running does nothing, and what my log shows is the 22:56 trigger waiting behind the running job and firing the instant it finished, at 22:56:40. You get at most one instance at a time, which is exactly right for a backup. If you want a fixed gap between runs instead of a fixed clock time, OnUnitActiveSec from Section 8 counts from the last start, so a slow run simply pushes the next one back.

12. Testing without waiting until 03:30

The service is a normal unit, so the way to test a nightly job is to run it now:

systemctl start backup.service
journalctl -u backup.service -n 20

That runs the exact command, as the exact user, with the exact environment the timer will use. It is a much better test than running the script from your shell, because your shell has a HOME and a full PATH and Section 10 happens.

For a one off "run this in twenty seconds" or "at 03:30 tonight only" there is systemd-run, which creates a temporary timer and service without any files:

systemd-run --on-active=20s --unit=oneoff /bin/sh -c 'echo "oneoff fired at $(date +%T)"'
systemctl list-timers oneoff.timer

Running timer as unit: oneoff.timer
Will run service as unit: oneoff.service
NEXT                        LEFT LAST PASSED UNIT         ACTIVATES
Thu 2026-09-03 22:58:29 UTC  19s -         - oneoff.timer oneoff.service

--on-calendar="*-*-* 03:30:00" gives you the calendar flavour. The transient timer fires, the output lands in journalctl -u oneoff.service, and then the units disappear on their own. It is at, without installing at.

13. Timers for a normal user

Everything so far went in /etc/systemd/system/ as root. A normal user gets their own systemd, with its own units in ~/.config/systemd/user/, driven by systemctl --user. Same file format, no sudo, and the jobs run as you. The catch is what happens when you log out.

I created note.timer and note.service as user keith, logged in over SSH, enabled the timer, and logged out:

systemctl --user daemon-reload
systemctl --user enable --now note.timer
systemctl --user list-timers note.timer

Created symlink '/home/keith/.config/systemd/user/timers.target.wants/note.timer' → '/home/keith/.config/systemd/user/note.timer'.
Thu 2026-09-03 23:05:00 UTC 46s - - note.timer note.service

Seventy five seconds later, from root:

systemctl is-active user@1000.service
journalctl _UID=1000 --since "-2min" | grep "user timer"

inactive

Nothing ran. A user's systemd instance is user@<uid>.service, and by default it starts when the user logs in and stops when their last session ends, taking every user timer down with it. The fix is one command, run once, by root or by the user themselves:

loginctl enable-linger keith

Lingering keeps user@1000.service up with no session at all. Same timer, same enable, log out, and this time:

2026-09-03T23:06:00+00:00 systemd-timers-post sh[2375]: user timer ran as keith at 23:06:00

The user reads their own runs with journalctl --user -u note.service, and root can see them with journalctl _UID=1000. One trap for anyone administering this from root: sudo -iu keith systemctl --user ... does not work. It fails with Failed to connect to user scope bus, because sudo does not set up the session environment the user manager needs. Log in as the user properly. The error message suggests --machine=keith@.host --user as a way for root to reach the user's manager directly, which I have not tried.

14. Changing one of Ubuntu's own timers

You will eventually want apt-daily.timer to run at 04:00 instead of somewhere in a twelve hour window, or fstrim on a Sunday. Do not edit the file under /usr/lib, the next package update will overwrite it. Add an override:

systemctl edit apt-daily.timer

That opens an editor on a new file, /etc/systemd/system/apt-daily.timer.d/override.conf, which systemd merges over the original. systemd calls that file a drop-in, and it is the word in the warning from Section 3. I tested this from a session with no terminal, where systemctl edit refuses to run, so I wrote the same file by hand and ran daemon-reload, which is all edit does for you. Here is the one that pins the apt check to 04:00:

[Timer]
OnCalendar=
OnCalendar=*-*-* 04:00:00
RandomizedDelaySec=0

The blank OnCalendar= line is not a typo and it is the part everyone misses. OnCalendar is a list setting, and a drop-in that supplies another one adds a time rather than replacing it. I checked: without the blank line, systemctl show apt-daily.timer -p TimersCalendar still listed the original 06,18:00 alongside my 04:00, so apt would have run three times a day. An empty assignment clears the list first.

systemctl cat apt-daily.timer now shows both files, original and override, which is how you can always tell what is actually in effect. systemctl revert apt-daily.timer deletes the override and puts things back.

Gotchas worth knowing

Edits do nothing until daemon-reload. start prints a warning and runs the old definition anyway. Make the reload part of saving the file.

Enable the timer, not the service. A service with no [Install] section refuses to be enabled, which is the right outcome. A service you gave WantedBy=multi-user.target runs at every boot on top of its schedule.

It fires up to a minute late by default. AccuracySec=1min. Set AccuracySec=1s when the minute matters. Measured drift on the default was between four and twenty seven seconds.

Without Persistent=true, a run missed during downtime is gone. With it, the job runs within seconds of the next boot. It does not fire on first enable, only for runs it can prove were missed.

% and $ belong to systemd. %T became /tmp in my date format and ${#VAR} became nothing. Double them, %% and $$, to pass them through.

No shell, no HOME, five entry PATH, working directory /. Wrap shell syntax in /bin/sh -c, set Environment=HOME= if a tool wants one, and set WorkingDirectory= if the script assumes one.

status=3/NOTIMPLEMENTED just means exit code 3. The word is systemd's name for that code from the LSB table and has nothing to do with your script.

A drop-in OnCalendar= adds a schedule. Put an empty OnCalendar= above the new one to replace instead.

User timers die at logout unless loginctl enable-linger <user> has been run once. And sudo -iu user systemctl --user cannot reach the user manager at all.

A typo in OnCalendar stops the timer from loading and says so quietly. systemd-analyze verify on the file, or systemd-analyze calendar on the expression, before you enable.

Quick reference

Task	Command
Pick up an edited unit file	`systemctl daemon-reload`
Turn a timer on, now and at boot	`systemctl enable --now job.timer`
See every timer and when it next fires	`systemctl list-timers --all`
Last run and next run of one timer	`systemctl status job.timer`
Run the job right now, timer or not	`systemctl start job.service`
Read the job's output	`journalctl -u job.service -o short-iso`
Follow it live	`journalctl -u job.service -f`
Since last boot only	`journalctl -u job.service -b`
What does this schedule mean	`systemd-analyze calendar "Mon..Fri 09:00" --iterations=3`
Check a unit file before enabling	`systemd-analyze verify /etc/systemd/system/job.timer`
Show the file, plus any overrides	`systemctl cat job.timer`
Every unit currently in a failed state	`systemctl list-units --failed`
Clear a failed state	`systemctl reset-failed job.service`
One off run in twenty seconds	`systemd-run --on-active=20s --unit=oneoff /path/to/cmd`
Override a vendor timer	`systemctl edit apt-daily.timer`
Undo the override	`systemctl revert apt-daily.timer`
User timers survive logout	`loginctl enable-linger <user>`
Where the missed run stamps live	`/var/lib/systemd/timers/stamp-*.timer`

The pair of files I now start every new job from, with everything above baked in:

# /etc/systemd/system/job.service
[Unit]
Description=What this job does

[Service]
Type=oneshot
User=someone
WorkingDirectory=/where/it/expects/to/be
EnvironmentFile=/etc/job.env
ExecStart=/usr/local/bin/job.sh

# /etc/systemd/system/job.timer
[Unit]
Description=When it runs

[Timer]
OnCalendar=*-*-* 03:30:00
Persistent=true
RandomizedDelaySec=10min

[Install]
WantedBy=timers.target

daemon-reload, enable --now job.timer, start job.service once to prove it works, and then leave it alone. That last step is the whole point. [ timer armed ]

Originally published at peculiarengineer.com.

A disposable Hetzner lab with OpenTofu: real Ubuntu box, run a script, destroy it

Minor Keith — Mon, 24 Aug 2026 15:03:03 +0000

Most of what I write here is Ubuntu, and I write it on a Mac. That gap is a problem. A Docker container is not a machine: it shares my kernel, it has no real systemd to fight with, it cannot reboot, and do-release-upgrade inside one is meaningless. A local VM is closer but it lives forever, drifts, and quietly becomes a machine I maintain instead of a machine I test on.

What I actually want is a real Ubuntu server that exists for twenty minutes and then does not. Hetzner bills by the hour and hands you a booted box in about sixty seconds, so the cloud part is easy. The part worth writing down is everything around it: where the state goes when you are creating and destroying the same thing forty times, how you make sure a forgotten VM does not bill you all month, and the three ways this tool could quietly cost me money, open a box to the internet, or leave something behind it no longer knows about. All three were in my own code. I found one by reading it and the other two by trying to build the box for the next post, which failed. All three are fixed now, and each one is in here because the fix is less interesting than the reason it was wrong.

TL;DR. One hcloud_server and one hcloud_firewall in OpenTofu, with the firewall locked to your public IP /32. Keep state in S3 at labs/<lab_id>/state.tfstate, one object per lab, so concurrent labs never touch the same file. Wrap it in a Python script that does up, down, ssh, status, and sweep. Push the work into a runbook, a plain Python file that gets copied to the box and run as root, so the test is a reproducible artifact instead of shell history. Set a short TTL and sweep from cron, because nothing cloud side enforces it.

The shape of it

The commands you will actually use are up, down, ssh, status, and sweep, and the whole thing is a Python file plus five .tofu files.

lab up --image ubuntu-26.04 --ttl 1 --runbook runbooks/check.py
lab ssh -c 'journalctl -n 50 --no-pager'
lab down

up creates one server and one firewall, then prints the SSH command and a destroy deadline. Hand it a runbook and it also waits for SSH, copies the script to the box, and runs it as root while streaming the output back. down destroys it. sweep destroys anything past its deadline. That is the entire product.

The design decision that matters is that a lab is one server, not a fleet. The moment you let it grow a second node you need inventory, ordering, and a graph, and you have written a worse Ansible. One box, born and killed, is a small enough problem to keep honest.

Prerequisites

OpenTofu 1.10 or newer, which is where S3 state locking arrives. I am on 1.12.6. Terraform works too, since this uses nothing OpenTofu specific.
Python 3.10 or newer and uv.
A Hetzner Cloud project and an API token with read and write, from Security, then API tokens.
Your SSH public key uploaded under Security, then SSH keys. Note the name you gave it.
An S3 compatible bucket. I use Hetzner Object Storage in the same region, so state and logs sit next to the servers.

1. The OpenTofu is the small part

Two resources. That is it.

# main.tofu
data "hcloud_ssh_key" "lab" {
  name = var.ssh_key_name
}

resource "hcloud_firewall" "lab" {
  name = "${var.lab_id}-fw"

  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "22"
    source_ips = [var.allowed_ssh_cidr]
  }

  rule {
    direction  = "in"
    protocol   = "icmp"
    source_ips = ["0.0.0.0/0", "::/0"]
  }
}

resource "hcloud_server" "lab" {
  name         = var.lab_id
  server_type  = var.server_type
  image        = var.image
  location     = var.location
  ssh_keys     = [data.hcloud_ssh_key.lab.id]
  firewall_ids = [hcloud_firewall.lab.id]

  labels = {
    lab        = "true"
    lab_id     = var.lab_id
    ttl_hours  = var.ttl_hours
    created_at = var.created_at
  }
}

One more file does quiet work alongside this one. outputs.tofu exports server_ip, server_id, and lab_id, and the Python wrapper reads the first two back with tofu output -json right after apply. Copy the resources, skip the outputs, and up dies immediately after building your server, which is the worst moment for it to die.

Two things in the resources are load bearing and easy to skip past.

Only port 22 and ICMP are open. Nothing else, inbound, ever. If a runbook installs Nginx you will not reach it from your browser, and that is deliberate: a box that exists for twenty minutes with a service exposed to the internet is how you end up in someone's scan results. When I need to see a web service I forward it over the SSH session I already have:

ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    -N -L 8080:localhost:80 root@<ip>

The labels are the safety net. lab = "true", plus the TTL and creation time, live on the server in Hetzner's own API. Local files get deleted, laptops get reimaged, and a .lab/ directory is one rm -rf away from being gone. The labels stay attached to the running server, so no matter what happens on my machine I can always ask Hetzner "what did I leave on?" and get a real answer:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
  'https://api.hetzner.cloud/v1/servers?label_selector=lab%3Dtrue' \
  | python3 -c 'import json,sys; [print(s["name"], s["public_net"]["ipv4"]["ip"], s["labels"]) for s in json.load(sys.stdin)["servers"]]'

Put that somewhere you will find it. It is the command that answers "am I being billed for something I forgot about," and it does not care whether your local state survived.

2. One state object per lab

The backend is declared empty and filled in at init time:

# backend.tofu
terraform {
  backend "s3" {}
}

Then every command passes the key for that specific lab:

tofu init -input=false -reconfigure \
  -backend-config="bucket=$S3_BUCKET" \
  -backend-config="key=labs/$LAB_ID/state.tfstate" \
  -backend-config="region=$S3_REGION" \
  -backend-config="endpoint=$S3_ENDPOINT" \
  -backend-config="access_key=$AWS_ACCESS_KEY_ID" \
  -backend-config="secret_key=$AWS_SECRET_ACCESS_KEY" \
  -backend-config="skip_credentials_validation=true" \
  -backend-config="skip_metadata_api_check=true" \
  -backend-config="skip_region_validation=true" \
  -backend-config="skip_requesting_account_id=true" \
  -backend-config="use_lockfile=true" \
  -backend-config="use_path_style=true"

The four skip_ flags are not optional and they are the thing that sends people to the search bar. The S3 backend assumes AWS. Point it at Hetzner Object Storage, MinIO, or Backblaze and it will try to validate credentials against AWS endpoints, look for an EC2 metadata service that is not there, reject nbg1 as a region name, and call STS to work out an account ID. Every one of those fails against a non AWS provider. Skip them all.

use_path_style=true matters for the same reason. Hetzner Object Storage wants https://nbg1.your-objectstorage.com/bucket/key, not https://bucket.nbg1.your-objectstorage.com/key. Get it wrong and you get a DNS failure that looks nothing like a configuration mistake.

Why one object per lab instead of one workspace or one big state? Because labs are independent and I want to run two at once without them knowing about each other. Separate keys means separate state files, so a 24.04 box and a 26.04 box can be up simultaneously and neither can corrupt the other. It also means a lab's entire existence is one deletable prefix in a bucket.

The credentials failure here is worth calling out because it does not announce itself. Bad S3 keys surface as an opaque tofu init error with no mention of S3. So the first thing the tool does when you complain is check them directly:

lab test-s3
# Endpoint: https://nbg1.your-objectstorage.com
# Region:   nbg1
# Bucket:   minor-lab-state
# Access:   ABC123...
#
# OK: bucket 'minor-lab-state' is reachable.

If that passes and init still fails, the problem is genuinely OpenTofu. If it fails, you have your answer in two seconds instead of twenty minutes.

3. The wrapper

The Python file is the whole user interface. up, down, ssh, logs, status, sweep, plus test-s3, test-discord, and server-types. It shells out to tofu, holds the flags so I never type a backend config by hand, and writes a small metadata file per lab.

Three details in up earned their place.

It waits for SSH properly, when there is a runbook. Hetzner reports the server as running before cloud-init has finished, so connecting immediately gets you a refused connection. The wrapper polls port 22 every two seconds for up to two minutes, with the progress prints trimmed out of the quote below. A bare lab up with no runbook skips this entirely. It applies, prints, and exits without ever probing port 22, so the address it hands you may not answer for another half minute.

def _wait_for_ssh(ip: str, timeout: int = 120) -> None:
    deadline = _now() + timedelta(seconds=timeout)
    while _now() < deadline:
        try:
            with socket.create_connection((ip, 22), timeout=5):
                return
        except OSError:
            pass
        time.sleep(2)
    sys.exit(1)

It saves metadata before it does anything risky. The IP, the server ID, the TTL, and the destroy deadline are written to disk the moment the server exists, before the runbook runs. If the runbook explodes, or I hit Ctrl C, the box is still tracked and still destroyable. Saving that file after the interesting part would mean any failure leaves an untracked, billed server.

It fails closed. If the runbook exits non zero, up destroys the server and firewall, wipes the local metadata, and still exits with the runbook's code. That is the correct default for a tool that spends money, but it has a consequence you need to internalize: the evidence dies with the box. To debug a failure you have to decide beforehand and pass --keep-on-failure. It took two fixes to make that heading true, and both are in the third footgun below.

The log survives a failed runbook, because the upload to S3 happens before the destroy. After an automatic teardown, lab logs fails because the local directory is gone, but the full output including stderr is sitting at s3://<bucket>/labs/<lab_id>/runbook.log. Read it from there.

One hole in that, which I would rather state than have you discover. The upload only gets skipped when the SCP itself fails, because that raises and goes straight to the exception handler, which destroys the box without uploading. A connection that dies partway through a running script is fine: the read loop just ends, up uploads whatever made it into the log, and you get a truncated but real record. So the log survives a script that fails and a session that drops. It does not survive a runbook that never landed on the box.

4. Runbooks, not shell history

A runbook is a Python file that gets copied to /tmp/runbook.py and executed as root. Output is merged, streamed live to my terminal, written locally, and uploaded to S3.

#!/usr/bin/env python3
import subprocess, sys

def run(cmd):
    print(f"$ {' '.join(cmd)}")
    return subprocess.run(cmd, check=True)

def main() -> int:
    run(["apt-get", "update", "-qq"])
    run(["apt-get", "install", "-y", "-qq", "nginx"])
    return 0

if __name__ == "__main__":
    sys.exit(main())

Python rather than bash for one reason: exit codes. A bash script without set -euo pipefail cheerfully reports success after every command in it failed, and I have been burned by a green test that proved nothing. subprocess.run(..., check=True) raises, the script exits non zero, and the tool tears the lab down and tells me. Failure is loud by default instead of by remembering to configure it.

The real payoff is that the runbook is a file. When I write an Ubuntu guide, the script that proves it works can ship next to the post. A reader can run the same file on their own box and get the same output rather than trusting that I ran the commands in the order I wrote them.

One thing bites on the first run. apt-get fails with Could not get lock /var/lib/dpkg/lock-frontend because cloud-init is still installing packages when SSH comes up. Waiting for port 22 is not the same as waiting for the machine to be finished. Either wait for cloud-init or retry:

run(["cloud-init", "status", "--wait"])

5. Teardown is the actual feature

Creating servers is easy. Not paying for them is the part that needs design.

Every lab gets a destroy_after timestamp from --ttl, in hours. sweep walks the labs and destroys anything past its deadline, which goes in cron:

0 * * * * cd ~/projects/hetzner-terraform-dev-lab && uv run ./lab sweep >> /tmp/lab-sweep.log 2>&1

uv run is not decoration there. The script exits at import time if boto3 and python-dotenv are missing, and system Python has neither, so a bare ./lab sweep from cron prints a dependency error into the log every hour and sweeps nothing. Silently, in the one place whose entire job is to stop you being billed for a machine you forgot.

Be honest about what this is: the TTL is advisory. Nothing on Hetzner's side enforces it. If my laptop is asleep, cron does not run, and the box bills merrily. A short TTL plus prompt lab down is the real story, and sweep is a backstop for the times I forget.

sweep also destroys any expired lab, including one somebody else on the machine created. Mine is a single user Mac so that is theoretical, but it is why the command is not wired into up.

The failure I care most about is the one where teardown itself fails. The script prints Manual cleanup may be required, then re-raises, so you get a traceback with it. That is the correct behavior, and it is also the exact moment to go run that label selector query from earlier.

Three footguns I found in my own tool

Writing this post is what surfaced all three. The first I found by reading the code, and the other two found me, in the middle of trying to spin up the box for the next post.

The public IP lookup fails open

up locks SSH to your current public IP by default. It gets that IP from ipify:

def _public_ip() -> str:
    try:
        with urllib.request.urlopen("https://api.ipify.org", timeout=10) as resp:
            return resp.read().decode().strip()
    except Exception:
        return ""

and then:

ip = _public_ip()
allowed_ssh_cidr = f"{ip}/32" if ip else "0.0.0.0/0"

Read that second block again. If ipify is slow, blocked, rate limiting me, or I am on a captive portal, _public_ip() swallows the exception and returns an empty string, and the default silently becomes root SSH open to the entire internet. No warning, no prompt, nothing in the output that looks different from a normal run. The one line in up's output that would tell you is ssh cidr: 0.0.0.0/0, sitting in a block of five lines that all look like routine startup noise.

It fails closed now:

ip = _public_ip()
if not ip:
    print("Could not determine your public IP address.")
    print("Refusing to default the SSH rule to 0.0.0.0/0.")
    print("Pass --allow-ssh <cidr> explicitly if that is really what you want.")
    sys.exit(1)
allowed_ssh_cidr = f"{ip}/32"

And when you do ask for it deliberately, it says so out loud instead of hiding in the startup noise:

WARNING: SSH will be open to the entire internet (0.0.0.0/0).

Wide open is a legitimate thing to want sometimes, from a CI runner with no stable egress address for instance. It is not a legitimate thing to arrive at by accident because a third party HTTP request timed out. Make the human type it.

The same default was sitting in variables.tofu as default = "0.0.0.0/0", which is the belt to match the braces. That default is gone too, so the variable is now required and a missing value is an error rather than an open port.

Hetzner will price a server type it will not sell you

I picked cpx12 when I set this up because the name looked like the small one. Later I built a server-types command so I would stop guessing, ran it, and it told me cpx11 was two cores instead of one and less than half the price. Obvious upgrade. I changed the flag, ran it, and got this:

Error: Server Type "cpx11" is unavailable in "nbg1" and can no longer be ordered

  with hcloud_server.lab,
  on main.tofu line 22, in resource "hcloud_server" "lab":

Here is the part worth the post. Hetzner's /v1/server_types endpoint still returns cpx11 with a current hourly price for nbg1 and deprecation: null. Nothing in that response suggests you cannot have one. The entire cpx*1 generation is retired, and the only endpoint that admits it is /v1/datacenters, which carries a server_types.available list of numeric IDs per datacenter:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
  'https://api.hetzner.cloud/v1/datacenters' \
  | python3 -c 'import json,sys; [print(d["name"], sorted(d["server_types"]["available"])) for d in json.load(sys.stdin)["datacenters"]]'

Prices and deprecation flags describe the catalog. Only that list describes the shop.

Which made my own command a liar. lab server-types --location nbg1 printed the header Available server types in nbg1: and then listed cpx11, cpx21, cpx31, cpx41, cpx51, cx43, and cx53, none of which can be ordered there. It filtered on whether a price exists for the location, which is not the same question. A command whose whole purpose is to stop me guessing sent me straight into a failed apply.

It cross references the datacenter list now, and only prints what you can actually have:

Server types you can actually order in nbg1:
NAME          CPU   RAM   DISK  ORDERABLE   PRICE/MO
cax11           2    4GB    40GB        yes       6.99
cpx12           1    2GB    40GB        yes      13.49
cx23            2    4GB    40GB        yes       6.49
...                                                       (17 rows in total)

Pass --all to include types that are priced here but cannot be ordered.

--all keeps the rest visible with an honest column, which is where cpx11 now shows up as NO.

Among the types you can actually order in nbg1, the original default still looks bad:

Type	Cores	RAM	Hourly	Orderable in nbg1
`cx23`	2	4 GB	€0.0104	yes
`cax11` (arm)	2	4 GB	€0.0112	yes
`cpx12`	1	2 GB	€0.0216	yes
`cpx11`	2	2 GB	€0.0096	no

cx23 is double the cores and double the RAM of the old default, for less than half the hourly. It is the default now. The arm box is worth a look too, since Hetzner publishes ubuntu-26.04 for arm as well as x86.

The teardown handler has two blind spots

The cpx11 failure exposed a third one immediately. When tofu apply fails, the run leaves behind a lab-upgrade-2604-fw firewall with nothing attached to it, and lab status shows an empty table. Local metadata is written after apply succeeds, so from the tool's point of view the lab was never born, and lab down has nothing to destroy.

The cause is one line of control flow. In cmd_up, the apply happens before the try block that owns the cleanup:

_tofu_init(lab_id)
_tofu_apply(lab_id, args, allowed_ssh_cidr, created_at_label)
vm_created = True

try:
    ...
except Exception:
    # tears down the server and the firewall

Every failure after that point was handled properly. The failure of that line was not handled at all, and it is the most likely one, because apply is where quota limits, retired server types, unavailable images, and API hiccups all land.

The apply lives inside the try now. The subtle part is the flag, which has to be set before the apply rather than after it, because a failed apply may already have built the firewall. Set it after and you have written the same bug in a new place, which I did on the first attempt:

resources_touched = False
try:
    _tofu_init(lab_id)
    # Set before the apply, not after: a failed apply can still have created
    # the firewall, and that is exactly the case that used to leak.
    resources_touched = True
    _tofu_apply(lab_id, args, allowed_ssh_cidr, created_at_label)

Running the same cpx11 failure against the fixed version now ends the way it should:

Error: Server Type "cpx11" is unavailable in "nbg1" and can no longer be ordered

Lab setup failed: Command '['tofu', 'apply', ...]' returned non-zero exit status 1.
Tearing down lab to avoid leaving resources up...
...
Destroy complete! Resources: 1 destroyed.
Lab torn down.

There is a second blind spot in the same handler and it is quieter. _wait_for_ssh gives up after two minutes by calling sys.exit(1). That raises SystemExit, and SystemExit does not inherit from Exception:

>>> issubclass(SystemExit, Exception)
False

So it went straight past except Exception and no teardown ran. A box that boots but never opens port 22, which is what a bad image or a firewall mistake looks like, left a running server behind. This one was gentler than the apply case, because metadata was already saved, so lab status saw the lab and lab down worked. It still was not the behavior the tool advertised. The handler catches BaseException now, which is the one place that spelling is worth reaching for.

A dangling firewall costs nothing, so this is clutter rather than a bill. It matters because it breaks the promise the rest of the tool makes: that anything it created, it can also destroy. Here is the audit, which needs no local state and is the thing to reach for when the tool says it knows nothing:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" 'https://api.hetzner.cloud/v1/firewalls' \
  | python3 -c 'import json,sys; [print(f["id"], f["name"], len(f["applied_to"])) for f in json.load(sys.stdin)["firewalls"]]'

Anything with zero attachments and a lab name is mine, and it is safe to delete.

Gotchas worth knowing

tofu init fails with something that never mentions S3. Bad object storage credentials surface as an opaque init error. Run lab test-s3 first, always.

InvalidAccessKeyId when the keys look right. The S3 credentials belong to a different Hetzner project than the bucket. They are scoped per project and the console does not make that obvious.

SSH stops working mid lab. The firewall is pinned to the public IP you had when you ran up. Move from home WiFi to a phone hotspot and you are locked out of your own box. Recreate it, or widen the CIDR.

There was no state locking at all, and that is worse than a stuck lock. I assumed there was, and I was wrong, which is why I am spelling it out. The S3 backend only locks if you ask it to, and this backend config did not. On OpenTofu 1.10 and later that means adding use_lockfile=true, and on older setups it meant a DynamoDB table. With neither, two commands against the same lab do not queue and do not hang. They both write, and the loser silently corrupts the state file. Separate labs are genuinely safe because they have separate state objects. The same lab twice at once was not, and nothing warned you. One flag fixes it, and it is in the backend config now:

-backend-config="use_lockfile=true"

Deleting .lab/ orphans a running server. Destroy is driven from the local metadata file. Wipe the directory and the tool no longer knows the box exists, while Hetzner still does, and still bills for it. This is why the labels matter.

lab logs fails after a failed runbook. Expected. The teardown removed the local directory. The log is in S3.

Discord webhooks fail quietly. Post to a webhook URL and you get a 204 with no body whether or not anything sensible happened. Add ?wait=true and Discord returns the created message or a real error, which turns "did that work?" into a question with an answer.

Quick reference

lab status                                     # what exists, * marks current
lab up --ttl 1                                 # bare box, eligible for sweep after 1h
lab up --image ubuntu-26.04 --ttl 2 --runbook runbooks/x.py
lab up --ttl 1 --runbook r.py --keep-on-failure  # keep the box to debug
lab ssh                                        # interactive shell
lab ssh -c 'cloud-init status --wait'          # one shot command
lab logs --show                                # local runbook log
lab down                                       # destroy current lab
lab down lab-20260821-143000                   # destroy a specific one
lab sweep                                      # destroy everything expired
lab server-types --location nbg1               # only what you can actually order
lab test-s3                                    # check credentials first

And the one that does not need the tool at all, for when local state is gone and you need the truth:

curl -s -H "Authorization: Bearer $HCLOUD_TOKEN" \
  'https://api.hetzner.cloud/v1/servers?label_selector=lab%3Dtrue'

The first thing I pointed this at was a 24.04 box and do-release-upgrade, which is a test I was never going to run on a machine I cared about. That one is written up in upgrading Ubuntu 24.04 to 26.04, and two of the three footguns above surfaced while I was building the box for it.

[ lab down ✓ ]

Originally published at peculiarengineer.com.

Backing up a homelab with restic

Minor Keith — Mon, 24 Aug 2026 13:30:49 +0000

I have written a lot here about running things at home and almost nothing about not losing them, which is a strange gap for someone who keeps telling you to self host your own Git server. So I sat down to work out how backups should actually be done, rather than how I had been vaguely assuming they were done.

The tool part turned out to be the easy half. The half that surprised me is that the obvious approach, point a backup tool at your data directory and let it run every night, produces a broken restore most of the time if anything in there is a database. I have numbers on that below, and they changed the shape of this post while I was writing it.

The whole thing is built around one idea. A backup nobody has restored is a rumor. So the restore comes first here, and everything else exists to make that restore possible.

TL;DR. Use restic, pointed at object storage you already pay for. The repository password is the backup: lose it and the data is gone with no recovery path, so store it somewhere separate from the machine. Never back up live database files. Dump them first, VACUUM INTO for SQLite and pg_dump for Postgres, then back up the dumps. Do not use SQLite's .backup on a busy database: it either fails instantly leaving a zero byte file or spins forever. Run it from a systemd timer, not cron. forget drops snapshots from the index and reclaims no space at all, prune is what actually frees it, and you need both. Then test a restore onto a machine that has never seen the data.

Why restic

One static binary, no daemon, no database of its own. It encrypts on your machine before anything leaves it, deduplicates across snapshots so the tenth backup of a 200 GB directory is not 2 TB, and it speaks S3 natively so it can point straight at object storage without a helper.

The alternatives, briefly, because I did look. Borg is excellent and has been around longer, but it has no native object storage support, so you end up mounting your bucket with rclone and now your backup depends on a FUSE mount behaving. Kopia is genuinely good and slightly nicer to use, just younger with less production mileage. rsync is not a backup, it is a file copy: no history, no integrity checking, and it will happily replicate your corruption over the good copy.

For a homelab where the destination is a bucket, restic is the boring correct answer.

Before you start

Everything here is on Ubuntu 26.04, and everything from here runs as root, which is also how the timer will run it. Drop into a root shell with sudo -i and the commands below are exactly what you type.

Install the two tools the post assumes, because neither is on a stock server:

apt update
apt install -y restic sqlite3
restic version

sqlite3 is the command line client, and you need it even if the application that owns the database bundles its own copy. If you are backing up Postgres in a container, pg_dump runs inside that container so there is nothing extra to install for it.

Make the directory the dumps land in now, because several commands below write into it:

mkdir -p /srv/dumps

The password is the backup

Before anything else, because restic init says it and people scroll past:

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

That is not boilerplate. There is no recovery, no support email, no key escrow. Encrypted with a password you do not have is identical to deleted.

So the password goes somewhere that is not the machine being backed up. A password manager, a piece of paper, another machine. Backing up your password manager's vault into the same restic repository, which I nearly did while setting this up, is a circle with no way in.

On the box it lives in a file only root can read, so it stays out of your shell history and out of ps:

install -m 600 /dev/null /root/.restic-password
read -rs -p 'passphrase: ' P && printf '%s' "$P" > /root/.restic-password && unset P

read -rs rather than putting the passphrase on the command line, because a command line ends up in .bash_history and in ps while it runs, which are the two places this file exists to keep it out of.

1. Create the repository

I am using Hetzner Object Storage because I already have a bucket there. Any S3 compatible storage works the same way.

Credentials and repository location go in an environment file, root readable only:

install -m 600 /dev/null /etc/restic.env

# /etc/restic.env
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export RESTIC_REPOSITORY="s3:https://nbg1.your-objectstorage.com/your-bucket/homelab"
export RESTIC_PASSWORD_FILE="/root/.restic-password"

The repository URL is worth reading carefully. It is s3: then the full endpoint URL including https, then the bucket, then a prefix. That prefix matters: it lets one bucket hold several unrelated repositories without them colliding.

Then initialize it once:

set -a; . /etc/restic.env; set +a
restic init

✅ You should see the password warning quoted above, followed by a line naming the repository it made:

created restic repository 461094b21b at s3:https://nbg1.your-objectstorage.com/your-bucket/homelab

2. Decide what to back up

The instinct is to back up everything. Resist it, because a backup you cannot afford to run daily is a backup you will turn off.

What actually needs to be in there:

Application data. Docker volumes and bind mounts, whatever /srv or /opt holds for you.
Database dumps, which is section 4 and the reason this post exists.
/etc. Small, and it is every decision you made about this machine. Restoring config from memory at 2 a.m. is how you discover you had forgotten three of them.
The list of what was installed. dpkg --get-selections into a file.

What does not need to be in there: package caches, anything under /proc, /sys or /tmp, and the backup repository itself if you keep a local one. Docker images do not need backing up either, because they come back from a registry.

Docker volumes are a trap here, and worth being precise about. I use bind mounts, so my container data sits under /srv and gets picked up by the paths above. If you use named volumes instead, which is the more common pattern, that data lives in /var/lib/docker/volumes and nothing above touches it. Either add that path to the backup or move to bind mounts. Skipping /var/lib/docker wholesale, which is the usual advice and which I very nearly wrote here, would leave you with a backup containing none of your actual data.

3. The part that will quietly break you

Here is the experiment that changed this post.

I set up a box with two databases being written to continuously: one in WAL mode, one in the older rollback journal mode, which is still SQLite's default. Which mode your applications use is worth checking rather than assuming, with sqlite3 yourapp.db 'PRAGMA journal_mode;', because as you will see it changes the odds a lot. Then I backed up their files live with restic, restored them, and asked SQLite whether the restored copy was intact.

The first run came back clean. integrity_check: ok. If I had stopped there, this section would have said live backups are fine.

So I ran it ten more times, and then twice more on different machines with different write patterns. Here is the third run, which is the one I would have least expected:

run  wal          journal
  1  malformed    malformed
  2  malformed    ok
  3  malformed    malformed
  4  malformed    malformed
  5  malformed    malformed
 ...
 10  malformed    malformed

WAL failures:     10/10
JOURNAL failures: 9/10
TOTAL:            19/20

malformed there is SQLite's own verdict on the restored file:

Error: stepping, database disk image is malformed (11)

Three runs, three answers:

Run	Rollback journal	WAL	Total
First machine, single row inserts	8 of 10 corrupt	3 of 10	12 of 20
Second machine, moderate load	9 of 10 corrupt	0 of 10	9 of 20
Third machine, batched transactions	9 of 10 corrupt	10 of 10	19 of 20

That spread is the finding, not any one number. Rollback journal mode is reliably terrible, around nine times in ten every time I tried. WAL mode ranged from never failing to failing every single time, depending on nothing more than how the writes were shaped. On the run where WAL never broke I could easily have concluded WAL mode is safe, published that, and been wrong.

So the useful statement is not a percentage. It is that a live file copy of a SQLite database restores to a corrupt file often, unpredictably, and in a way you cannot measure once and trust. It works just enough of the time to convince you.

Postgres behaved differently. Five of five live data directory restores came back working, every one crash recovering on startup:

LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 0/1917A20
LOG:  database system is ready to accept connections

Which makes sense once you say it out loud: a file level copy of a running Postgres looks exactly like someone pulled the power out, and surviving that is a thing Postgres is specifically built to do. I could not break it.

I am still going to tell you not to do it, for two reasons that have nothing to do with superstition. Upstream does not sanction copying a running cluster, because a busy one can be captured mid page write in a way a small test database never will be. And the dump of that same database was 19.6 MB against a 225 MB live data directory. Eleven times smaller, portable between versions, and blessed by the people who wrote it.

4. Dump the databases first

The correct approach costs almost nothing, which removes the last excuse.

SQLite. The obvious command is .backup, SQLite's own online backup, and it is what I recommended here until I tested it properly:

time sqlite3 /srv/app/data/app.db ".backup /srv/dumps/app.db"

real    0m0.089s

Eighty nine milliseconds, passes integrity_check, lovely. On an idle database.

Point it at a database that is actually being written to and it stops being lovely, because the SQLite backup API restarts from page zero every time an external writer commits. I hit two different failure modes on two different machines with the same command.

On one, it gave up instantly and left a file behind:

Error: database is locked
real    0m0.10s

-rw-r--r-- 1 root root 0 Aug 23 00:02 b1.db

Zero bytes. A script that does not check the exit code backs that up and reports success.

On the other, it never finished at all. Under systemd with a 45 second cap:

hang.service: start operation timed out. Terminating.
hang.service: Consumed 44.743s CPU time over 45.016s wall clock time

Forty five seconds of pegged CPU producing nothing, on a database of a few tens of megabytes. Without that timeout it spins for as long as writes keep arriving, and Type=oneshot means systemd waits patiently forever while your backups quietly stop happening.

Use VACUUM INTO instead. It takes one read transaction and writes a clean copy, without restarting when someone else commits:

time sqlite3 /srv/app/data/app.db "VACUUM INTO '/srv/dumps/app.db';"

Same box, same writer hammering it, same moment:

real    0m0.41s

integrity: ok
rows:      60412

Four tenths of a second, and a file that opens. It needs SQLite 3.27 or newer, which means anything from 2019 onwards, so on 26.04 it is simply there. The output file must not already exist, which is why the script in the next section writes to a temporary name and moves it into place.

Postgres. Dump in the custom format so you can restore selectively later:

docker exec app-pg-1 pg_dump -U postgres -d labdb -Fc -f /tmp/labdb.dump
docker cp app-pg-1:/tmp/labdb.dump /srv/dumps/labdb.dump

Then back up /srv/dumps, not the live directories.

5. The backup script

Everything above, in the order it has to happen: dump first, then back up the dumps.

Save this as /usr/local/bin/backup.sh and make it executable, which the next section will fail on if you forget:

The container name and database name below are mine. Change them, or delete those two lines if you have no Postgres.

#!/bin/bash
# /usr/local/bin/backup.sh
set -uo pipefail
set -a; . /etc/restic.env; set +a

mkdir -p /srv/dumps
failed=0

# Databases first, while everything is still running.
rm -f /srv/dumps/app.db.tmp
sqlite3 /srv/app/data/app.db "VACUUM INTO '/srv/dumps/app.db.tmp';" \
  && mv /srv/dumps/app.db.tmp /srv/dumps/app.db || failed=1

docker exec app-pg-1 pg_dump -U postgres -d labdb -Fc -f /tmp/labdb.dump \
  && docker cp app-pg-1:/tmp/labdb.dump /srv/dumps/labdb.dump || failed=1

# What was installed, so a rebuild is not archaeology.
dpkg --get-selections > /srv/dumps/packages.txt

restic backup /srv/dumps /srv/app /etc \
  --exclude /srv/app/data \
  --tag nightly || failed=1

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune || failed=1

exit $failed

Two decisions in there are deliberate and both came from watching this fail.

No -e. The obvious version starts set -euo pipefail, and I wrote it that way first. The problem is that set -e makes a single failed dump abort the entire script, so a stopped Postgres container means restic never runs and nothing gets backed up that night, including /etc and every file that had nothing to do with Postgres. The version above records the failure, backs up everything it still can, and exits non zero so the timer reports it. A partial backup beats no backup.

The dump goes to a temporary name and gets moved into place. If the dump dies halfway, mv never happens and yesterday's good copy stays where it is, rather than being replaced by a truncated file that restic will faithfully preserve forever. The rm -f before it matters too: VACUUM INTO refuses to write to a file that already exists, so a run killed between the vacuum and the move would otherwise fail every night afterwards until somebody deleted the leftover by hand.

Both restic calls are checked. This is the one I nearly shipped without. Wrapping only the dumps means a wrong password, an unreachable bucket, or a repository locked by a prune leaves failed at zero, the unit green, and no backup taken. Restic's exit codes are specific enough to rely on:

clean run                     exit 0
some files could not be read  exit 0   (snapshot still created)
wrong password                exit 12
repository does not exist     exit 10

A file vanishing mid backup, which happens constantly with sockets and pid files, does not fail the run. A broken destination does.

chmod +x /usr/local/bin/backup.sh

The --exclude /srv/app/data is the point of section 3: the live database directory is deliberately not in the backup, because the good copy is already in /srv/dumps.

6. Run it from a systemd timer

Cron works. A timer is better here because it gives you systemctl status on the last run, journal output kept with the unit, and Persistent=true, which runs a missed job after a machine was off rather than skipping the night.

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly restic backup
Wants=network-online.target
After=network-online.target docker.service

[Service]
Type=oneshot
TimeoutStartSec=30min
Environment=HOME=/root
ExecStart=/usr/local/bin/backup.sh

# /etc/systemd/system/backup.timer
[Unit]
Description=Run the restic backup nightly

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now backup.timer
systemctl list-timers backup.timer

RandomizedDelaySec matters more than it looks. Everyone picks a time on the hour, and object storage providers notice.

TimeoutStartSec is the backstop for section 4. A oneshot service with no timeout will sit in activating indefinitely if something inside it wedges, and you find out weeks later. With it set, systemd kills the run and marks the unit failed, which is a state you can actually alert on.

Run it once by hand before trusting the schedule:

systemctl start backup.service
journalctl -u backup.service -n 30 --no-pager

7. Retention, and why forget is not enough

restic forget removes snapshots from the index. restic prune removes the data those snapshots referenced. They are separate operations, and running only the first is a very popular way to have a retention policy that does nothing.

Watch what each one does. restic stats --mode raw-data reports the size of the repository itself, which works against object storage where du does not:

restic snapshots --compact | tail -1     # 16 snapshots
restic stats --mode raw-data             # 98 MiB

restic forget --keep-last 3
restic snapshots --compact | tail -1     # 3 snapshots
restic stats --mode raw-data             # 98 MiB

restic prune
restic stats --mode raw-data             # 26 MiB

Sixteen snapshots became three and not one byte was freed. The space came back only after prune. So either pass --prune to forget, as the script above does, or run prune on its own schedule.

One thing that confused me the first time: --keep-last 3 keeps three snapshots per group, and restic groups by host and paths by default. Back up two different path sets and you keep three of each, so the count you land on may be higher than the number you typed.

8. The restore, which is the whole point

Everything so far is preparation. This is the only part that proves anything, and it is the part nobody does.

Do it on a machine that has never seen this data, with nothing but the repository URL and the password. That is the actual disaster scenario: the original box is gone.

That machine needs restic installed, plus sqlite3 and Docker if you want to check the dumps the way I do below. A throwaway cloud box is ideal, because you want somewhere with genuinely none of your data on it, and you want to destroy it afterwards.

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export RESTIC_REPOSITORY="s3:https://nbg1.your-objectstorage.com/your-bucket/homelab"
export RESTIC_PASSWORD="..."

restic snapshots
restic restore latest --target /bare --include /srv/dumps

Then check what came back, rather than admiring the file listing:

sqlite3 /bare/srv/dumps/app.db 'PRAGMA integrity_check;'

ok

and put the Postgres dump into a fresh server to prove it loads:

docker run -d --name pgr -e POSTGRES_PASSWORD=x postgres:17

# Wait for it. `docker run -d` returns long before Postgres accepts connections,
# and running createdb immediately gives you a confusing socket error.
until docker exec pgr pg_isready -U postgres -q; do sleep 1; done

docker cp /bare/srv/dumps/labdb.dump pgr:/tmp/d.dump
docker exec pgr createdb -U postgres restored
docker exec pgr pg_restore -U postgres -d restored /tmp/d.dump
docker exec pgr psql -U postgres -d restored -tAc 'select count(*) from t;'

Your number will differ, obviously. That it is a number at all is the entire point of the post. Not that the backup ran, that the data came back and could be queried on a machine that had never seen it.

Pulling a single file out of an older snapshot is the other restore worth practicing, because it is the one you will actually use:

restic snapshots                     # copy an ID from the first column
restic restore <id> --target /tmp/oops --include /etc/fstab

And confirm the repository itself is readable, which is a different question from whether the last backup succeeded:

restic check

check snapshots, trees and blobs
no errors were found

Put restic check on a monthly timer. A repository that cannot be read is not a backup either, and you want to find that out on a Tuesday rather than during a rebuild.

Gotchas I hit

A single successful restore proves nothing. My first live database restore came back clean. Across three machines the same test then failed between nine and nineteen times out of twenty. Test more than once, and test the thing you actually care about.

forget frees no space. Covered above, and worth repeating because a retention policy that never prunes looks like it is working right up until the storage bill.

Two backups at once do not collide. I expected a repository lock error and got two clean snapshots instead, because backups take a non exclusive lock. prune is the one that takes an exclusive lock, so the collision you can actually hit is a manual backup landing while the timer is pruning.

--unsafe-allow-remove-all refuses to run bare. Trying to empty a repository gives you Fatal: --unsafe-allow-remove-all is not allowed unless a snapshot filter option is specified, so it needs something like --host or --tag alongside it.

Restic reports a warning and still exits zero when a file vanishes mid backup, which happens constantly with sockets and pid files. at least one source file could not be read is worth reading, not worth alerting on.

Under systemd, restic complains about a cache it cannot find. unable to open cache: unable to locate cache directory: neither $XDG_CACHE_HOME nor $HOME are defined, on every run. It is harmless, it just means no local cache and so slower metadata operations. Set Environment=HOME=/root in the service unit to quiet it and get the cache back.

backup.service has no [Install] section, on purpose. It is started by the timer, so systemctl enable backup.service is not a thing you want. Enable backup.timer instead. systemctl status backup.service still works for reading the last run.

That shrinkage on the first snapshot is compression, not dedup. 2.7 MiB of files stored as 874 KiB, with no previous snapshot to deduplicate against. Restic creates version 2 repositories now and compresses by default. Dedup is real too, it just needs a second snapshot before it has anything to work with.

Quick reference

set -a; . /etc/restic.env; set +a      # load credentials into the shell

restic init                            # once, at the start
restic backup /srv/dumps /etc --tag nightly
restic snapshots                       # what have I got
restic snapshots --compact             # the same, readable

restic restore latest --target /bare                        # everything
restic restore <id> --target /tmp/x --include /etc/hosts    # one file
restic check                                                # is the repo sound

restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune

rm -f /srv/dumps/app.db.tmp                                 # VACUUM INTO wants a free path
sqlite3 live.db "VACUUM INTO '/srv/dumps/app.db.tmp';"      # not cp, and not .backup
docker exec pg pg_dump -U postgres -d mydb -Fc -f /tmp/d.dump && docker cp pg:/tmp/d.dump /srv/dumps/

I went into this assuming the hard part was picking a tool. The hard part was finding out that the obvious way to back up a database works often enough to fool you. Go and restore something.

[ restored ✓ ]

Originally published at peculiarengineer.com.

Running Kimi Code on Claude Code

Minor Keith — Thu, 13 Aug 2026 20:40:23 +0000

I pay for Claude and I pay for Kimi Code, and I want both available from the same terminal without choosing one in the morning. Kimi publishes a guide for pointing Claude Code at their endpoint. It tells you to export a handful of environment variables in your shell profile, which works, and which also routes every Claude Code session on the machine to Kimi from then on. Your Anthropic subscription sits there unused while you wonder why Opus got worse.

The fix is boring: a second settings file and a shell function that loads it. Ten minutes. The part worth writing down is that the headline model ID in Kimi's guide is rejected by Kimi's own endpoint, and it's rejected with a 401 that says your credentials are bad. I lost time auditing a key that was never the problem. Then I found a second version of the same lie living in ~/.claude.json, and that one is better disguised.

TL;DR. Put the Kimi config in ~/.claude/kimi-settings.json rather than ~/.claude/settings.json, then add a claude-kimi() function that runs claude --settings "$HOME/.claude/kimi-settings.json" "$@". Base URL is https://api.kimi.com/coding/ with the key in ANTHROPIC_API_KEY. The model is k3, not k3[1m], whatever the docs say. Set CLAUDE_CODE_MAX_CONTEXT_TOKENS to 1048576 because a bare k3 gives Claude Code no window to infer. If interactive mode 401s while curl and claude -p both work, your key is in the rejected list in ~/.claude.json and no amount of key rotation will fix it.

Kimi has two API surfaces and only one of them is yours

This is the distinction that decides whether anything else in this post works, so get it straight first.

A Kimi Code subscription authenticates against https://api.kimi.com/coding/, using ANTHROPIC_API_KEY, which is sent as an x-api-key header. Model IDs are short: k3, k3-256k, kimi-for-coding.

The Moonshot platform is a different thing. It lives at https://api.moonshot.ai/anthropic, uses ANTHROPIC_AUTH_TOKEN as a bearer token, and wants model IDs with a kimi- prefix. That is the surface you pay for by the token. It's also what most of the search results describe.

They are different products. Your subscription key will not authenticate against the platform host, and the failure is a flat 401 that tells you nothing about why. If you follow the popular guide with a subscription key, this is where you stop.

1. Get a key

Go to kimi.com/code/console and create one.

Two things worth knowing before you click. You get at most five keys, and each is shown exactly once, at creation. There is no reveal button later, only delete and make a new one. Copy it somewhere before you close the dialog.

2. Write the settings file

Create it empty and lock it down before the key goes anywhere near it, so it's never briefly readable by anything else on the machine:

touch ~/.claude/kimi-settings.json
chmod 600 ~/.claude/kimi-settings.json

Then paste this in, replacing the placeholder:

{
  "env": {
    "ANTHROPIC_BASE_URL": "https://api.kimi.com/coding/",
    "ANTHROPIC_API_KEY": "PASTE_YOUR_KIMI_CODE_KEY_HERE",

    "ANTHROPIC_MODEL": "k3",
    "ANTHROPIC_DEFAULT_OPUS_MODEL": "k3",
    "ANTHROPIC_DEFAULT_SONNET_MODEL": "k3",
    "ANTHROPIC_DEFAULT_FABLE_MODEL": "k3",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL": "kimi-for-coding",
    "CLAUDE_CODE_SUBAGENT_MODEL": "k3",

    "ANTHROPIC_DEFAULT_OPUS_MODEL_NAME": "Kimi K3",
    "ANTHROPIC_DEFAULT_OPUS_MODEL_DESCRIPTION": "Kimi Code K3 (1M context)",
    "ANTHROPIC_DEFAULT_HAIKU_MODEL_NAME": "Kimi K2.7 Code",

    "CLAUDE_CODE_MAX_CONTEXT_TOKENS": "1048576",
    "CLAUDE_CODE_AUTO_COMPACT_WINDOW": "950000",
    "CLAUDE_CODE_EFFORT_LEVEL": "high"
  }
}

Check the permissions again after you save. Some editors write a temp file and rename it into place, which lands at your umask and quietly undoes the chmod you just ran:

ls -l ~/.claude/kimi-settings.json   # want -rw-------

There are six model slots, not one, and that's deliberate. Claude Code uses the haiku slot for background chores like naming your conversation, and pointing those at K3 spends your quota on titles. kimi-for-coding is ungated on every tier, so it's the safe floor for throwaway work.

The two window numbers are the part people get wrong. CLAUDE_CODE_AUTO_COMPACT_WINDOW is a trigger threshold, not a statement of how much context you have, so it needs headroom underneath the real window. Kimi's guide sets it to 1048576, which is out of range anyway: Claude Code accepts 100000 to 1000000. It also has to be a plain integer. Write 500k and it parses as 500 and clamps to the floor, and you'll be wondering why compaction fires roughly immediately.

3. The model ID in the official guide does not work

Kimi's own Claude Code guide tells you to set ANTHROPIC_MODEL to k3[1m]. Their endpoint rejects it. I sent each ID at api.kimi.com/coding/ with curl, and here's what came back:

Model ID	HTTP	What comes back
`k3`	200	Works. Main conversation model
`k3-256k`	200	Works. Smaller window variant
`kimi-for-coding`	200	Works. Ungated on all tiers
`k3[1m]`	401	`Your model id does not exist, recognized as other:k3[1m]. Please set model id as k3.`

Read that 401 again. It's an authentication_error, and the message body is about a model ID. Nothing in the status code suggests you should go looking at your model configuration, so you go and audit your key instead, and your key is fine. It cost me more time than anything else in this setup.

Use the bare k3. It is natively a 1M context model and the suffix was never what unlocked that.

There is a consequence though, and it's why CLAUDE_CODE_MAX_CONTEXT_TOKENS is in the config above rather than being decorative. With a [1m] style ID, Claude Code can read the window size out of the name. A bare k3 tells it nothing, so it falls back to a conservative guess and starts compacting your session far earlier than it needs to. Declaring the real number fixes that.

If you're on k3-256k instead, change ANTHROPIC_MODEL, ANTHROPIC_DEFAULT_OPUS_MODEL, ANTHROPIC_DEFAULT_SONNET_MODEL, ANTHROPIC_DEFAULT_FABLE_MODEL and CLAUDE_CODE_SUBAGENT_MODEL to k3-256k, leave the haiku slot alone, and set CLAUDE_CODE_MAX_CONTEXT_TOKENS to 262144 with the compact window at 240000. I'm listing all five by name on purpose. Forget CLAUDE_CODE_SUBAGENT_MODEL and your subagents keep running on the 1M model while you believe you moved everything down.

4. Add the launcher

cat >> ~/.zshrc <<'EOF'

# Claude Code against the Kimi Code subscription.
# Plain `claude` is unaffected and keeps the claude.ai subscription.
claude-kimi() {
  command claude --settings "$HOME/.claude/kimi-settings.json" "$@"
}
EOF

exec zsh

Use $HOME rather than ~ so the path resolves however the function gets called. The --settings flag layers that file on top of your normal user config, and because ~/.claude/settings.json has no env block on my machine, there's nothing for it to collide with. Check yours before you assume the same:

python3 -c "import json,os;print('env' in json.load(open(os.path.expanduser('~/.claude/settings.json'))))"

If that prints True, whatever is in there will fight with the Kimi config, and you should move it out first.

5. Test the endpoint before you test Claude Code

Confirm the key, host and model at the HTTP layer while there's only one thing that can be wrong. Then any failure in the next step is a Claude Code problem, and you already know it isn't credentials.

read -rs "KEY?Kimi Code key: "; echo

curl -sS https://api.kimi.com/coding/v1/messages \
  -H "x-api-key: $KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"k3","max_tokens":16,
       "messages":[{"role":"user","content":"Say OK."}]}'

You want JSON with a content array in it. read -rs keeps the key out of your shell history, and note the header is x-api-key rather than a bearer token, which is the practical tell that you're on the subscription surface and not the platform one. Run unset KEY when you're finished with it.

Two notes on this. If you get a 404 on everything, your base URL includes the full path and Claude Code is appending v1/messages to something that already ends in it. Set the base, keep the trailing slash, nothing more.

And k3 needs the Moderato tier or above. If it comes back with a 401 naming the model and linking an upgrade page, that's tier gating rather than a bad ID, and you want every slot on kimi-for-coding instead. Both failures arrive as authentication_error, so the status code can't tell them apart and the message body is the only thing that can.

A curl probe here is fine, by the way. Kimi's terms require tools to identify themselves honestly, and curl does. Forging a User-Agent to make one client look like another is the thing they take issue with.

6. Confirm both lanes

In one tab:

claude-kimi
# then, inside the session:
/status

/status should show https://api.kimi.com/coding/ and a Kimi model. Send it a real message so you've exercised the round trip rather than just the startup path.

Then open a separate tab and run plain claude, and check /status there too. This is the whole point of the exercise, so actually do it. Verifying that the isolation held is more important after a Claude Code upgrade than it is today: model variable precedence changed in v2.1.195, and the next release that touches settings layering is not going to send you a note.

Expect one prompt on the first interactive claude-kimi, asking whether to use the detected API key. Say yes.

The rejected key list, which is the worst one

Answer no to that prompt, deliberately or with a stray keypress on a dialog you weren't expecting, and Claude Code writes the key's last 20 characters into a rejected list in ~/.claude.json. From then on, interactive sessions refuse it and tell you this:

Please run /login · API Error: 401

Here is why this one is genuinely nasty. None of the obvious diagnostics consult that list, so every one of them clears the key. I had curl returning 200, claude -p "hello" returning 200, a config file that was correct, an endpoint that was up, and a key that was valid. Only interactive mode failed, and it failed with a message pointing at my credentials.

Do not run /login here. The error asks you to, and it's wrong, and running it inside the Kimi lane is the one action in this whole setup that can disturb the Anthropic credential you were trying to protect.

Check for it instead:

python3 -c "
import json,os
d=json.load(open(os.path.expanduser('~/.claude.json')))
print(json.dumps(d.get('customApiKeyResponses',{}),indent=2))"

If your key's tail is sitting under rejected, move it across:

python3 - <<'PY'
import json, os, shutil
p = os.path.expanduser('~/.claude.json')
shutil.copy(p, p + '.bak-apikey')
d = json.load(open(p))
key = json.load(open(os.path.expanduser(
    '~/.claude/kimi-settings.json')))['env']['ANTHROPIC_API_KEY']
tail = key[-20:]
r = d.setdefault('customApiKeyResponses', {'approved': [], 'rejected': []})
r['rejected'] = [x for x in r.get('rejected', []) if x != tail]
if tail not in r.setdefault('approved', []):
    r['approved'].append(tail)
json.dump(d, open(p, 'w'), indent=2)
print('approved:', r['approved'])
PY

Close every Claude Code session before you run that, since ~/.claude.json is live state and you don't want two writers. Then start fresh, because the list is read at launch. If the prompt appears again, accept it this time.

While we're on things read at launch: settings are too. Editing kimi-settings.json does nothing to a session that's already open, and a session started before your last edit will keep sending the old model ID and keep 401ing on every turn, with an error that once again blames the key. To find stale ones:

ps -eo pid,lstart,command | grep kimi-settings
stat -f '%Sm' ~/.claude/kimi-settings.json

Compare the two and kill anything older than the file.

Living with the request quota

This is the part that changes how you work, and it isn't the setup. Your plan is metered in requests per rolling five hour window, somewhere around 300 to 1,200 depending on tier, with up to 30 running at once. Tokens are not the constraint. Round trips are.

Claude Code burns through that faster than a chat client does, because one instruction can fan out into a lot of requests. Subagents are the big one, since each agent runs its own request loop and several can run concurrently, so a single "go investigate this" can cost dozens. Workflows and ultracode fan out hard enough to approach the 30 concurrent ceiling on their own, which is reason enough to keep them off this lane. Raising the effort level means more tool calls per turn, and every tool call is a request. Background chores are small but constant, which is exactly why the haiku slot points at kimi-for-coding rather than K3.

One quirk on effort. Kimi Code collapses Claude Code's five effort levels onto K3's three, so medium and xhigh land in the same place as their neighbors. Tune between low, high and max and don't expect the in between settings to do anything.

Gotchas worth knowing

Plain claude also goes to Kimi. The variables reached your global environment. Run env | grep ANTHROPIC, and check you didn't paste the config into ~/.claude/settings.json out of habit.

A warning that claude.ai connectors are disabled. Expected, and actually a good sign. An API key outranks your claude.ai login for that session, which is the isolation working. The other tab is unaffected.

WebFetch says it's temporarily unavailable. The endpoint doesn't implement it. Paste the content in, or use an MCP scraping server. WebSearch still works on K3.

/model shows unhelpful names. That's what the _NAME and _DESCRIPTION keys in the config are for. You can also set CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1 and let the picker populate itself from the endpoint's model list.

Timestamps in the console request log are UTC+8, and a request that failed on a bad model ID logs the model as -. A row with a dash in the model column is a rejected ID, not a credential fault, whatever the status column says.

400 invalid thinking: only type=enabled is allowed. The K2.7 Code models used to reject requests that arrived without thinking enabled. I retested on 12 August 2026 and it no longer reproduces, with kimi-for-coding returning 200 to a request carrying no thinking block at all. I'm leaving it here because my retest used a minimal raw HTTP body rather than Claude Code's real request shape, and I'd rather you recognize it than rediscover it. If you do hit it, stay on K3.

Quick reference

Variable	Value	What it does
`ANTHROPIC_BASE_URL`	`https://api.kimi.com/coding/`	Where requests go. Base only, keep the trailing slash
`ANTHROPIC_API_KEY`	your Kimi Code key	Sent as `x-api-key`. Outranks the claude.ai login for this session
`ANTHROPIC_MODEL`	`k3`	Main conversation. No `[1m]` suffix
`ANTHROPIC_DEFAULT_OPUS_MODEL`	`k3`	What the `opus` alias resolves to
`ANTHROPIC_DEFAULT_SONNET_MODEL`	`k3`	What the `sonnet` alias resolves to
`ANTHROPIC_DEFAULT_FABLE_MODEL`	`k3`	What the `fable` alias resolves to
`ANTHROPIC_DEFAULT_HAIKU_MODEL`	`kimi-for-coding`	Background chores. Ungated on every tier
`CLAUDE_CODE_SUBAGENT_MODEL`	`k3`	Subagents and workflow agents. Your main quota burn
`CLAUDE_CODE_MAX_CONTEXT_TOKENS`	`1048576`	Required. A bare `k3` gives Claude Code nothing to infer from
`CLAUDE_CODE_AUTO_COMPACT_WINDOW`	`950000`	Compaction trigger. `100000` to `1000000`, plain integer
`CLAUDE_CODE_EFFORT_LEVEL`	`high`	Kimi collapses five levels onto three

Backing it out

rm ~/.claude/kimi-settings.json
# then delete the claude-kimi() block from ~/.zshrc
exec zsh

Your claude.ai login was never touched, so claude keeps working the whole time. To pause Kimi without uninstalling anything, just stop typing claude-kimi.

One honest caveat on the cleanup. If you accepted that API key prompt, the last 20 characters of your key are still in ~/.claude.json under customApiKeyResponses, and if you ran the repair script there's a ~/.claude.json.bak-apikey next to it. Neither is dangerous, both are residue, and you'll want to clear them out if you're rotating the key or handing the machine on.

The one thing to remember

Every failure in this setup arrives dressed as an authentication error. A rejected model ID, a model above your tier, and a key you once declined all surface as 401, and two of them will send you to the console to rotate a credential that was never broken. Read the message body, not the status code. That's the habit worth keeping.

[ two lanes, one terminal ]

Originally published at peculiarengineer.com.

Kubernetes, Part Three: GitOps with Argo CD, let Git drive your cluster

Minor Keith — Wed, 05 Aug 2026 02:58:58 +0000

In Part One I deployed nginx on a single k3s node and killed a Pod to watch a replacement appear. In Part Two I pulled the config and the passwords out of the image. Both parts had the same weak spot, and it's the one nobody mentions when they teach you kubectl: I was still standing at a terminal typing kubectl apply at a cluster.

That works fine until it doesn't. A month later nobody can tell you what's actually running, or who changed it, or what the cluster looked like before someone "just quickly fixed" something at 11pm. The YAML on your laptop and the YAML in the cluster quietly drift apart, and the only way to find out is to go and look.

Argo CD fixes that by taking the terminal away from you. You put your manifests in Git, you tell Argo CD where that repo is, and from then on the repo is the truth. Change the cluster by hand and Argo CD changes it back. I ran the whole thing on the same 4GB box that already hosts my Forgejo instance, and I'm writing it down because six separate things bit me and I'd rather not rediscover any of them.

TL;DR Put your manifests in a Git repo first. Install Argo CD with kubectl apply --server-side (plain apply fails on one oversized CRD). Create an Application object that names three things: which repo, which folder, where it goes. Sync it once by hand and watch OutOfSync become Synced. Then set syncPolicy.automated with selfHeal: true and prune: true, and try to fight it: scale a Deployment, delete a Service, swap an image. It puts all of them back within fifteen seconds. Deploy by pushing to Git, roll back with git revert. And remember that Synced means "matches Git", not "works".

The one idea worth holding onto

Part One's idea was that you declare the state you want and Kubernetes makes reality match it. Part Three is the same sentence moved one level out:

You declare the state you want in Git. Argo CD makes the cluster match it.

Kubernetes watches its own objects. Argo CD watches your repo. That's the entire product, and every feature below is a variation on it.

What you get for free is the part I didn't expect. Because every change to the cluster is now a commit, Git becomes your audit log, your rollback mechanism, and your review gate without you building any of those things. "Who scaled this to five replicas and why" stops being an unanswerable question.

Before you start

You need a working cluster and a kubectl that reaches it without sudo. That's Part One, Step 0, the KUBECONFIG fix that catches everyone. You also need a GitHub account, and about forty minutes.

Mine is a single k3s node, v1.36.2, on a 4GB Hetzner box in Falkenstein that was already running Forgejo in Docker. Argo CD cost me about 500MB across seven Pods, taking the box from 2.3GB free to 1.8GB free. It fits on a small machine comfortably. Nothing got OOM killed.

One warning that matters if your box has a public IP like mine does. Argo CD ships with an admin account and a bootstrap password, and Step 6 is where I make sure the web UI is reachable from my devices and from nowhere else. Don't skip it and don't reorder it.

Step 1: the repo comes first

This ordering is the whole philosophy in miniature, so do it in this order even though it feels backwards: the repo exists before Argo CD does.

Create a new public repo. Mine is peculiarengineer-gitops, and it holds exactly two files under apps/hello/.

Save the first as apps/hello/deployment.yaml:

# A Deployment says "keep N copies of this container running, forever."
# Argo CD's job is to make sure this file and the cluster always agree.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  replicas: 2 # change this number, push, and watch Argo CD notice
  selector:
    matchLabels:
      app: hello # which Pods this Deployment owns
  template:
    metadata:
      labels:
        app: hello # must match the selector above, or nothing happens
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            # Small box, so be explicit about what this is allowed to eat.
            requests:
              cpu: 10m
              memory: 16Mi
            limits:
              memory: 64Mi

And the second as apps/hello/service.yaml:

# A Service gives the Pods above one stable address inside the cluster.
# ClusterIP means "reachable from inside the cluster only", so nothing is
# published to the internet by this file.
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: ClusterIP
  selector:
    app: hello # sends traffic to any Pod carrying this label
  ports:
    - port: 80
      targetPort: 80

Nothing new here. This is Part One's Deployment and Service with resource limits added, because a 4GB box deserves them. Commit and push both.

💡 What happened: You wrote down what you want running without running any of it. From here on, editing these files is how you change the cluster. That's the habit the rest of the post builds.

Step 2: install Argo CD

Argo CD installs into the cluster as ordinary Kubernetes objects in its own namespace. No operator, no Helm required, no packages on the host.

▶ Do:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

❌ And that fails. Most of it applies, then right at the end:

The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes

This one confused me for a minute because the error blames the CRD, and the CRD is fine. kubectl apply stashes a copy of the entire manifest it just applied into a last-applied-configuration annotation so it can compute diffs later. The ApplicationSet CRD is bigger than the 256KB limit Kubernetes puts on annotations, so the copy can't be stored.

Server side apply doesn't use that annotation at all. It hands the whole document to the API server and lets it track ownership properly.

▶ Do:

kubectl apply -n argocd --server-side -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

✅ You should see three CRDs land:

customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io serverside-applied
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io serverside-applied

Now wait for the Pods. Seven of them come up, and on my box that took about forty seconds:

kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s

💡 What happened: Argo CD is now a workload in your cluster like any other, watching for Application objects that don't exist yet. Worth noticing that it installed by declaring a pile of YAML, which is the same trick it's about to do on your behalf.

Step 3: the CLI and the admin password

Grab the CLI on the node. This is the Linux x86 build, so if you'd rather run it from a Mac or an ARM box, take the matching binary from the releases page.

▶ Do:

sudo curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo chmod +x /usr/local/bin/argocd
argocd version --client --short

✅ You should see a version line and nothing else:

argocd: v3.4.5+564b949

The installer generates a random admin password and leaves it in a Secret. Reading it is a callback to Part Two, base64 and all:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ; echo

Same reminder as Part Two: base64 is encoding, not encryption. You just decoded a password with a pipe. Step 12 replaces this one properly.

Step 4: the Application, and the moment it clicks

This is the object the whole product hangs off, and it's smaller than you'd think. An Application answers three questions and nothing else. Which repo. Which folder inside it. Where the result goes.

You can create it with argocd app create, but write the YAML instead. Seeing the object is what makes the idea stick. Save it as hello-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello
  namespace: argocd # Applications live in the argocd namespace
spec:
  project: default
  source:
    repoURL: https://github.com/peculiarengineer-mk/peculiarengineer-gitops.git
    targetRevision: main # which branch to track
    path: apps/hello # which folder in the repo to apply
  destination:
    server: https://kubernetes.default.svc # this same cluster
    namespace: default # where the manifests land
  # No syncPolicy yet. This first one syncs only when we tell it to.

Swap in your own repo URL. Leaving syncPolicy out is deliberate, and it's the best decision in this whole tutorial, because it lets you see the next bit.

▶ Do:

kubectl apply -f hello-app.yaml
sleep 5
kubectl get application hello -n argocd \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status

✅ You should see two words that are the entire pitch:

NAME    SYNC        HEALTH
hello   OutOfSync   Missing

💡 What happened, and this is the paragraph I'd underline. Nobody asked it to do that. In the five seconds since you created the object, Argo CD cloned your repo, read apps/hello, compared both files against the live cluster, and reported the gap: OutOfSync means Git and the cluster disagree, Missing means those objects don't exist yet. It will keep doing that comparison forever, whether or not you ever sync anything.

The sync is almost an afterthought. The diff is what you're actually buying.

Step 5: the first sync

The argocd CLI has two modes and they behave differently, which caused me the second gotcha. --core talks straight to the Kubernetes API with no Argo CD server in the middle. It's the quickest way to work from the node.

▶ Do:

argocd app sync hello --core

❌ And it fails with a message that is actively misleading:

{"level":"fatal","msg":"configmap \"argocd-cm\" not found"}

Go and look and argocd-cm is sitting right there in the argocd namespace. The problem is that core mode has no server session telling it where Argo CD lives, so it reads the namespace from your kubeconfig context. On k3s that context has no namespace set, so it looks in default and finds nothing.

Setting ARGOCD_NAMESPACE does not help. I tried, twice.

The obvious fix is kubectl config set-context --current --namespace=argocd, and that's the third gotcha, because it poisons every kubectl command you run on that box afterwards. Suddenly kubectl get pods shows you Argo CD's internals instead of your app, forever, and you will not remember why. Use a separate kubeconfig for the argocd CLI instead.

▶ Do:

cp ~/.kube/config ~/.kube/argocd-core.yaml
KUBECONFIG=~/.kube/argocd-core.yaml kubectl config set-context --current --namespace=argocd
KUBECONFIG=~/.kube/argocd-core.yaml argocd app sync hello --core

That copies the kubeconfig Part One had you set up, so you're not going back to the root owned /etc/rancher/k3s/k3s.yaml that Step 0 told you to stop touching.

✅ You should see:

Sync Status:        Synced to main (d1395f1)
Phase:              Succeeded
Message:            successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME   STATUS  HEALTH       MESSAGE
       Service     default    hello  Synced  Healthy      service/hello created
apps   Deployment  default    hello  Synced  Progressing  deployment.apps/hello created

Check what landed:

kubectl get deploy,pods -l app=hello -n default

deployment.apps/hello   2/2   2   2   22s
pod/hello-5d6b7fbfc4-f6kxf   1/1   Running   0   23s
pod/hello-5d6b7fbfc4-p4x64   1/1   Running   0   23s

That KUBECONFIG= prefix is needed on every argocd command from here on, and typing it each time gets old fast. Set an alias for the rest of the session:

alias argocd='KUBECONFIG=~/.kube/argocd-core.yaml argocd'

Every argocd command below assumes you did that. If one of them ever comes back with configmap "argocd-cm" not found, you're in a new shell and the alias is gone.

💡 What happened: You never ran kubectl apply on those manifests. Argo CD read GitHub and created both objects itself. That's a small thing on two files and a very large thing on two hundred.

Step 6: reach the UI without publishing it to the internet

The CLI does everything, but the web UI is genuinely good and the resource tree is the bit that makes Kubernetes legible to people who don't live in it.

Here's the trap. A k3s NodePort binds to 0.0.0.0, which on a box with a public IP means the open internet. Argo CD has a known admin username and a bootstrap password. So the firewall goes on first and the service gets exposed second. If you get that order backwards you spend a few minutes with an unauthenticated door open, and that's plenty.

My box wasn't on my tailnet at all. Only the Forgejo container was, through its sidecar. So the host joins properly:

curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --hostname=k3s-1 --accept-dns=false --accept-routes=false

Both of those flags are deliberate. --accept-dns=false stops Tailscale rewriting /etc/resolv.conf, and --accept-routes=false stops it touching the routing table. On a box already running k3s and Docker I want Tailscale to add an interface and change nothing else.

Now the firewall, and here's the fourth gotcha, which is the one that can genuinely take your services down. Ubuntu ships this in /etc/default/ufw:

DEFAULT_FORWARD_POLICY="DROP"

Enable ufw with that set and you break pod to pod networking in k3s and Docker's bridge. On my box that would have taken Forgejo down as collateral. Fix it before you enable anything:

sed -i 's/^DEFAULT_FORWARD_POLICY=.*/DEFAULT_FORWARD_POLICY="ACCEPT"/' /etc/default/ufw

Before you turn the firewall on, arm a dead man switch. Enabling a firewall over SSH on a box whose only access is SSH is how people lose servers, and a Hetzner rescue console at midnight is a bad time.

▶ Do:

setsid nohup bash -c "sleep 300; ufw --force disable" >/dev/null 2>&1 < /dev/null &
echo $! > /root/ufw-deadman.pid

That undoes the firewall in five minutes unless you cancel it. Now the rules. SSH, the tailnet, and the k3s internal networks:

ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow in on tailscale0
ufw allow 41641/udp        # tailscale wireguard
ufw allow in on cni0       # k3s pods
ufw allow in on flannel.1
ufw allow in on docker0
ufw allow from 10.42.0.0/16   # k3s pod cidr
ufw allow from 10.43.0.0/16   # k3s service cidr
ufw --force enable

Open a second SSH session and confirm you can still get in. Check your cluster and anything else on the box is still healthy. Then cancel the timer using the PID file:

kill $(cat /root/ufw-deadman.pid)

Use the PID file, not pkill -f "sleep 300". I tried the pkill version and it killed my own shell, because the command line running that pkill contains the string sleep 300 and therefore matches itself. Exit code 255, dropped connection, and a few seconds of wondering whether I'd just locked myself out.

Now expose the UI:

kubectl -n argocd patch svc argocd-server -p '{"spec":{"type":"NodePort","ports":[
  {"name":"http","port":80,"targetPort":8080,"protocol":"TCP"},
  {"name":"https","port":443,"targetPort":8080,"protocol":"TCP","nodePort":30443}]}}'

If you go looking for a listening socket you won't find one, and that threw me for a second:

ss -tlnp | grep 30443    # returns nothing, and that's correct

kube-proxy implements NodePorts with iptables rules, not a bound socket. The test that actually matters is reaching it from two places:

https://100.87.x.y:30443  ->  HTTP 200     # over the tailnet
https://203.0.113.88:30443  ->  HTTP 000     # public IP, blocked

💡 What happened: The UI is reachable from every device on my tailnet, including the iPad, and invisible from everywhere else. The cert is self signed so your browser will complain once. Log in as admin with the password from Step 3, click into the hello app, and you get the resource tree: Application, Deployment, ReplicaSet, Pods, each with a health dot. That view is why people keep the UI around.

Step 7: break the cluster on purpose

Now the interesting part. Do the exact thing GitOps is supposed to prevent.

▶ Do:

kubectl scale deployment hello -n default --replicas=5
sleep 15
kubectl get application hello -n argocd \
  -o custom-columns=SYNC:.status.sync.status,HEALTH:.status.health.status

✅ You should see:

SYNC        HEALTH
OutOfSync   Healthy

Five Pods running, and Argo CD spotted it within seconds and did nothing at all.

That's not a bug, it's the lesson. Detecting drift and correcting drift are separate features, and right now you only have the first one. Seeing Argo CD notice and deliberately sit on its hands is what makes the next step land.

Healthy sitting next to OutOfSync is worth its own thought too. Health and sync are different axes. Health means the five Pods are fine. Sync means this matches Git. An app can be in perfect health and completely wrong.

Ask what's wrong and it tells you exactly:

argocd app diff hello --core

===== apps/Deployment default/hello ======
120c120
<   replicas: 5
---
>   replicas: 2

< is the live cluster, > is Git. Four lines that answer "how does reality differ from what I said I wanted". You can run that against anything, at any time, and get a real answer.

Step 8: turn on self-heal and try to win

Three settings. selfHeal corrects drift, prune deletes objects whose files vanish from the repo, and automated means it acts without being asked.

▶ Do:

kubectl -n argocd patch application hello --type=merge -p '{
  "spec": {"syncPolicy": {"automated": {"prune": true, "selfHeal": true}}}
}'

Then watch, without running any sync command:

 5s  OutOfSync Healthy   replicas=5  pods=5
10s  Synced Healthy      replicas=2  pods=2

Five Pods became two on their own. I changed a policy, not the cluster, and Argo CD dragged reality back to what the repo says. For my money this is the moment the whole idea clicks, more than the first sync was.

Now try to win. I made three attempts and lost all three inside fifteen seconds:

What I did	What happened
`kubectl scale --replicas=4`	back to 2
`kubectl delete svc hello`	Service recreated
`kubectl set image nginx=nginx:1.25`	back to `nginx:1.27`

💡 What happened, plus the fifth gotcha. Look closely at that recreated Service and it has a different ClusterIP and an age of fifteen seconds. Argo CD did not restore the object you deleted. It created a new one that matches the file. Anything that cached the old address is now talking to nothing. Self-heal keeps your cluster matching your repo, and that is not the same as a backup. Don't let it talk you out of having real backups.

Step 9: deploy by pushing to Git

The whole point. Edit the repo, not the cluster.

▶ Do change replicas: 2 to replicas: 3 in apps/hello/deployment.yaml, then:

git commit -am "Scale hello to 3 replicas"
git push

Now touch nothing and watch:

  0s  revision=d1395f1  replicas=2
...
124s  revision=d1395f1  replicas=2
136s  revision=e6c57cb  replicas=3

136 seconds. That isn't Argo CD being slow, it's the default polling interval. With no webhook configured it asks GitHub "anything new?" every three minutes. In production you point a repo webhook at it and deployments become instant. While you're writing a tutorial and don't want to wait, force it:

argocd app get hello --hard-refresh --core

💡 What happened: Check the Pod ages afterwards and you get 28m, 28m, 2s. It scaled up. It did not redeploy. Argo CD works out the difference between the repo and the cluster and applies only that, so a one line change doesn't churn your running workload.

Step 10: delete a file, delete the object

prune is the setting that decides whether removing a file removes the thing it described. I tested it with something disposable.

Add apps/hello/temp-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: hello-temp
data:
  note: 'delete this file from git and prune should remove the object'

Commit, push, hard refresh, and the ConfigMap appears in the cluster. Now git rm the file, commit, push, hard refresh again:

kubectl get configmap hello-temp -n default

Error from server (NotFound): configmaps "hello-temp" not found

💡 What happened: Deleting a file deleted the object. Worth knowing that prune: false is the default, and with it off Argo CD will happily add and update things but never remove them. That sounds safer and mostly it is, but you end up with orphaned objects in your cluster that no file in any repo describes and nobody remembers creating. Turn prune on early, while your cluster is small enough that a mistake is obvious.

Step 11: ship something broken, then roll it back

This is the most important section in the post, so I deliberately shipped a release that could not possibly work. In deployment.yaml:

image: nginx:9.9.9-doesnotexist

Commit, push, refresh, wait thirty seconds:

=== app health ===
Synced   Progressing

=== pods ===
hello-5d6b7fbfc4-f6kxf 1/1 Running       32m
hello-5d6b7fbfc4-p4x64 1/1 Running       32m
hello-5d6b7fbfc4-vzq7x 1/1 Running       6m
hello-6d96c85587-lgxml 0/1 ErrImagePull  31s

=== is the site still up? ===
HTTP 200

Read that first line again. Synced, and completely broken.

Synced means the cluster matches Git. It does not mean the app works. I put something wrong in the repo, so Argo CD faithfully made the cluster wrong. GitOps guarantees fidelity to your repo, not correctness, and if you remember one sentence from this post make it that one. Your repo is now the thing that needs code review, because it's the thing that ships.

The site stayed up for a reason worth knowing: Kubernetes' rolling update will not kill a healthy Pod until its replacement reports Ready, so three good Pods kept serving while the fourth failed to pull. It also sits in Progressing for a full ten minutes before it admits to being Degraded, which is progressDeadlineSeconds defaulting to 600 rather than Argo CD dragging its feet. I stared at Progressing for a while assuming something was stuck.

Now the fix, and notice what the fix is:

git revert --no-edit <bad-sha>
git push

Synced   Healthy
image: nginx:1.27
HTTP 200

Pod ages afterwards: 32m, 32m, 6m. Not one Pod died. The bad release never took hold, the old Pods served throughout, and recovery was a normal commit.

💡 What happened: Your rollback procedure is the same command as every other change. No special runbook, no "quick fix" applied straight to prod that nobody writes down. The revert is right there in the log next to the mistake it undoes.

Small thing that cost me a minute: git revert -q is not a flag. It quietly prints the usage text and reverts nothing, and if you're not reading closely you'll push an empty change and wonder why nothing recovered.

Step 12: rotate that admin password

The bootstrap password from Step 3 is meant to be temporary. The documented command is argocd account update-password, and here's the sixth gotcha:

$ argocd account update-password --account admin ... --core
failed to get issue time: unable to extract token claims

Core mode has no server session, so there's no token to authenticate a password change. You either log in through the API server properly, or you write the hash yourself:

NEW="pick-something-better-than-this"
HASH=$(argocd account bcrypt --password "$NEW")
kubectl -n argocd patch secret argocd-secret \
  -p "{\"stringData\":{\"admin.password\":\"$HASH\",\"admin.passwordMtime\":\"$(date +%FT%T%Z)\"}}"
kubectl -n argocd rollout restart deployment argocd-server
kubectl -n argocd delete secret argocd-initial-admin-secret

Confirm both directions. The old password should be refused:

$ argocd login <host>:30443 --username admin --password "<old>" --insecure --grpc-web
Invalid username or password

$ argocd login <host>:30443 --username admin --password "<new>" --insecure --grpc-web
'admin:login' logged in successfully

Worth understanding that those two CLI modes really are different tools wearing the same name. --core skips the server and talks to the Kubernetes API, which is fast and needs no login but can't do anything that depends on a session. --server <host> --grpc-web goes through argocd-server with a real login, which is what you want for anything touching accounts, tokens, or RBAC.

Gotchas I hit

Plain kubectl apply fails on the install manifest. The ApplicationSet CRD exceeds the 256KB annotation limit that apply needs for its last-applied-configuration copy. Use --server-side. The error blames the CRD, which sends you looking in the wrong place.
argocd --core reads its namespace from the kubeconfig context. Not from ARGOCD_NAMESPACE, which does nothing. On k3s the context has no namespace, so you get configmap "argocd-cm" not found while the ConfigMap is plainly sitting in the argocd namespace.
Fixing that by repointing your main kubeconfig poisons everything else. kubectl config set-context --current --namespace=argocd means every later kubectl get pods on that box shows Argo CD internals. Keep a separate kubeconfig for the argocd CLI.
DEFAULT_FORWARD_POLICY="DROP" breaks k3s and Docker when you enable ufw. Set it to ACCEPT in /etc/default/ufw first, and allow the pod and service CIDRs. Skip this and you take down pod networking and every container on the box at the same time.
Self-heal is not a backup. Delete a Service and Argo CD creates a new one matching the file, with a new ClusterIP. It restores the description, not the object. Anything holding the old address is now pointing at nothing.
argocd account update-password does not work in core mode. No session, no token, no password change. Patch the bcrypt hash into argocd-secret instead.
Synced does not mean working. It means the cluster matches Git. Put a broken image tag in the repo and you get a proudly Synced broken app.
Don't cancel a background timer with pkill -f "sleep 300". The shell running that command contains the string, matches itself, and kills your session. Use a PID file.

Quick command reference

Goal	Command
Install Argo CD	`kubectl apply -n argocd --server-side -f <install.yaml>`
Read bootstrap password	`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" \
App status	{% raw %}`kubectl get application <app> -n argocd`
See Git versus cluster	`argocd app diff <app> --core`
Sync now	`argocd app sync <app> --core`
Skip the 3 minute poll	`argocd app get <app> --hard-refresh --core`
List apps through the server	`argocd app list --server <host>:30443 --insecure --grpc-web`
Turn on self-heal and prune	patch `spec.syncPolicy.automated` with `selfHeal` and `prune`
Roll back	`git revert --no-edit <sha> && git push`

Where this series goes next

You now have a cluster that argues with you, which is the correct behaviour. Change something by hand and it changes it back. Deploy by pushing a commit. Roll back with git revert and lose nothing.

There's one loose thread I left deliberately, and it's the obvious next part. That hello-app.yaml from Step 4 is still sitting on the node, applied by hand with kubectl. The thing driving all my GitOps is itself not under GitOps, which is a slightly embarrassing place to stop. The fix has a good name, app of apps, and it answers the question of who deploys the deployer.

After that: repo webhooks to kill the three minute poll, private repo authentication so this can point at my own Forgejo rather than GitHub, and Helm and Kustomize as sources instead of plain YAML. The Kubernetes series hub tracks the lot, and Labels, Volumes, health probes and Ingress are all still on the list.

Go and scale something by hand, then watch it change back while you're still looking at it. That's the bit that made it real for me. [ synced ]

Originally published at peculiarengineer.com.

Self-host Forgejo on your tailnet with Docker Compose and Tailscale

Minor Keith — Thu, 30 Jul 2026 02:53:00 +0000

You want your own Git host. Maybe you're getting off GitHub, maybe you just want somewhere private to keep the repos that shouldn't be on someone else's servers. So you go looking, and every guide hands you the same shape: a public DNS record, ports 80 and 443 open to the entire internet, a reverse proxy, a certificate, and a login page that anyone on earth can now knock on.

For a private code host, that's a strange trade. Nothing about "my repos, for me and three collaborators" requires a public address. If the only people who should reach it are people you already trust, put it on your tailnet and the whole category of internet facing problems stops existing. No open ports, and no login page getting scanned at three in the morning.

The Compose file for this is short. What makes it worth writing down is the sidecar pattern it uses, which is not obvious the first time, and a handful of settings that decide whether it works at all. I built this from nothing on a fresh Ubuntu 26.04 box to check it, and the settings that went wrong were not the ones I expected, so those get their own section at the end.

TL;DR. Run Tailscale as its own container and give Forgejo network_mode: service:ts-forgejo so Forgejo has no published ports and its own tailnet identity. Point TS_SERVE_CONFIG at a serve JSON file and you get real HTTPS on forgejo.your-tailnet.ts.net with no port 80 and no HTTP-01 challenge. Persist /var/lib/tailscale or every restart creates a brand new node. Tag the device so its node key never expires. Do not set START_SSH_SERVER, because the image already runs sshd on port 22.

Prerequisites

A box with Docker installed. Mine runs Ubuntu 26.04, but nothing here is version specific.
A Tailscale account and a tailnet you can already log into. If you're starting cold, install Tailscale on the host first so you have something to test from.
Admin access to the Tailscale console, because two of the steps happen there and not on the box.
MagicDNS turned on. It's on by default for new tailnets, and without it you get an IP instead of a name.

You do not need a domain, a DNS record, a certificate, or a single open port in your firewall. That's the point.

The one thing to get straight

Forgejo is not sitting behind a reverse proxy here. Forgejo is the Tailscale node.

That distinction is the whole post. In the usual setup you'd run Caddy or Nginx on the host, publish ports, and forward traffic to a container. Here, the Tailscale container owns a network namespace, and Forgejo is placed inside that same namespace with network_mode: service:ts-forgejo. The two containers share one network stack the way two processes on one machine do.

Once that clicks, the rest follows on its own:

Forgejo publishes no ports to the host, so docker ps shows nothing listening and the host firewall has nothing to do.
The machine appears in your tailnet as its own device, with its own name and its own ACL rules, separate from the host it happens to run on.
Port 22 inside that namespace belongs to Forgejo, and the host's real sshd is somewhere else entirely as far as the network stack is concerned. The port collision that makes the public version of this setup annoying never happens.
HTTPS arrives without opening port 80. The certificate still comes from Let's Encrypt, but through a DNS-01 challenge that Tailscale completes for you by publishing the _acme-challenge TXT record under ts.net for your node. Nothing has to be reachable from the public internet for validation to pass.

The result is a real, publicly trusted certificate on a machine with no public presence at all. Your browser sees a normal padlock. Nothing is exposed.

1. Turn on HTTPS for your tailnet

Do this before anything else. Skipping it does not stop the stack coming up, which is exactly why it wastes your time later.

In the Tailscale admin console, go to DNS, and under HTTPS Certificates click Enable HTTPS. Note the tailnet name it shows you, something like tail1234.ts.net. Every device in your tailnet gets a name under it, so your Forgejo node will end up at forgejo.tail1234.ts.net.

If you skip this, the sidecar still starts and still says it's running. It logs a line telling you HTTPS is not enabled and links the docs, which is fair enough, but tailscale serve status just answers No serve config with no reason attached. The command that gives you a straight answer is:

docker compose exec ts-forgejo tailscale cert forgejo.tail1234.ts.net

With HTTPS off you get your Tailscale account does not support getting TLS certs. With it on you get two files written and you can move on. I use that as the check before touching anything else, because every other symptom of this is ambiguous.

2. Make a tagged auth key

The container needs a key to join the tailnet unattended, and the way expiry works here catches people out.

Auth keys cap out at 90 days, but that expiry only stops new devices from joining. A node that already registered keeps working until its own node key expires, and that defaults to 180 days. So a plain reusable key gets you roughly six months before your Git host quietly drops off the tailnet on a day you weren't touching anything.

The fix is a tag. Key expiry is disabled by default for tagged devices, because they're owned by the tailnet rather than by a user.

First define the tag in your ACL file, under Access controls:

{
  "tagOwners": {
    "tag:container": ["autogroup:admin"]
  }
}

Then go to Settings, Keys, and generate an auth key with Reusable on and the tag tag:container applied. Copy it somewhere safe now, because the console shows it exactly once.

Put it in a .env file next to your Compose file:

# .env
TS_AUTHKEY=tskey-auth-xxxxxxxxxxxx

And keep that file out of Git:

echo ".env" >> .gitignore

3. The Compose file

Two services. The first is the network, the second is the application.

services:
  ts-forgejo:
    image: tailscale/tailscale:latest
    container_name: ts-forgejo
    hostname: forgejo                      # becomes forgejo.your-tailnet.ts.net
    environment:
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_EXTRA_ARGS=--advertise-tags=tag:container
      - TS_STATE_DIR=/var/lib/tailscale    # persisted, see the volume below
      - TS_SERVE_CONFIG=/config/serve.json
      - TS_USERSPACE=false                 # use the kernel networking path
    volumes:
      - ts-forgejo-state:/var/lib/tailscale
      - ./ts-config:/config
    devices:
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped

  forgejo:
    image: codeberg.org/forgejo/forgejo:15
    container_name: forgejo
    network_mode: service:ts-forgejo       # the whole trick, one line
    environment:
      - USER_UID=1000
      - USER_GID=1000
      - FORGEJO__server__ROOT_URL=https://forgejo.tail1234.ts.net/
      - FORGEJO__server__SSH_DOMAIN=forgejo.tail1234.ts.net
      - FORGEJO__server__SSH_PORT=22
      - FORGEJO__service__DISABLE_REGISTRATION=true
    volumes:
      - forgejo-data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - ts-forgejo
    restart: unless-stopped

volumes:
  ts-forgejo-state:
  forgejo-data:

A few of those lines are doing more work than they look like they are.

network_mode: service:ts-forgejo is the pattern. Notice what is absent: there is no ports: block anywhere in this file. Forgejo listens on 3000 and 22 inside the shared namespace, reachable over the tailnet and nowhere else.

TS_USERSPACE=false matters more than it looks. The Tailscale image defaults to userspace networking, and in that mode the TUN device and the net_admin capability sit there unused. Turning it off puts the container on the kernel networking path those lines exist for, which is what you want when something in the namespace needs to answer on a real port.

hostname: forgejo is what decides the name. Tailscale registers the container under it, so this is the value that ends up in your URL. Pick it before first boot, because renaming a node later means fixing ROOT_URL too.

TS_STATE_DIR with a real volume behind it is the difference between one node and a hundred. More on that in the gotchas below.

FORGEJO__server__ROOT_URL uses Forgejo's double underscore convention, where FORGEJO__section__KEY maps to a key in app.ini. Get this value right before you ever start the container. Forgejo bakes it into the clone URLs it shows you, the links in its emails, and its webhook targets. Wrong here means every clone command your users copy points at the wrong place, and it looks fine on screen right up until someone tries it.

There is also something deliberately missing. You will see START_SSH_SERVER=true in a lot of Forgejo examples, and it does not belong here. The rootful image runs OpenSSH on port 22 by itself, unconditionally. Setting that variable starts Forgejo's own Go SSH server as well, and now two SSH servers want the same port in the same namespace. Leave it out and the image does the right thing on its own. That variable is for the rootless image, which is a different setup with SSH on 2222.

DISABLE_REGISTRATION=true from the first boot means nobody gets to sign up while you're still setting things up. You'll create your admin account through the installer instead.

4. The serve config

Create ts-config/serve.json next to your Compose file:

{
  "TCP": {
    "443": {
      "HTTPS": true
    }
  },
  "Web": {
    "${TS_CERT_DOMAIN}:443": {
      "Handlers": {
        "/": {
          "Proxy": "http://127.0.0.1:3000"
        }
      }
    }
  },
  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": false
  }
}

${TS_CERT_DOMAIN} is substituted by the Tailscale container at startup with the node's real name, so you don't have to hardcode your tailnet name in a second place.

The proxy target is 127.0.0.1:3000, and that is only correct because of the shared namespace. Forgejo really is on localhost from Tailscale's point of view. If you ever pull these two apart into separate networks, this line breaks first.

AllowFunnel is set to false on purpose. It's the switch that would put this node on the public internet, and having it present and off is better than having it absent, because you can see what the answer currently is.

Bring it up:

docker compose up -d
docker compose logs -f ts-forgejo

Watch for the node registering and the certificate being issued. Then check it from another machine on your tailnet:

tailscale status | grep forgejo
curl -I https://forgejo.tail1234.ts.net/

A 200 and a valid certificate means the hard part is done.

5. First boot and locking the installer

Open https://forgejo.tail1234.ts.net/ in a browser on any device in your tailnet and you get Forgejo's setup page.

Two things matter here. Leave the database as SQLite unless you have a reason not to. For a personal or small team instance it's genuinely fine, it's one less container to run and one less thing to back up, and you can migrate later if the instance grows into something that needs Postgres. And check that the URL fields on the form match the ROOT_URL you set, because the installer will happily write different values into app.ini and leave you with two sources of truth.

Create your admin account on that same form. Do not skip it and do it later.

The reason for the urgency is smaller here than on a public box, but it still applies: until the installer is completed, whoever reaches that page can complete it and become the administrator. On a tailnet that's limited to devices you've already authorised, which is a much shorter list than "the internet". It's still a list. Finish the form.

Once you're in, confirm the install lock landed:

docker compose exec forgejo grep INSTALL_LOCK /data/gitea/conf/app.ini
# INSTALL_LOCK = true

6. Cloning over SSH

This is the part that would have been a whole section of fighting in the public version, and here it's almost nothing.

git clone git@forgejo.tail1234.ts.net:yourname/yourrepo.git

Port 22, no custom port, no -p 2222 to remember, no conflict with the host's own sshd. The image's SSH server owns port 22 inside its namespace, and the host's sshd owns port 22 on the host, and they never meet.

Add your public key in the Forgejo UI under Settings, SSH / GPG Keys, the same as you would anywhere else.

One thing to check: the clone URL Forgejo displays comes from SSH_DOMAIN and SSH_PORT. If those disagree with reality, the button copies a command that fails, and the error the user sees is a connection timeout that tells them nothing about why.

Who can actually reach it

Right now, every device in your tailnet can reach it. That's fine for a tailnet of one person and three laptops. It stops being fine the moment you add a contractor's machine or a server that runs someone else's code.

ACLs fix that. This is a fragment to merge into your existing policy, not a whole policy file. If you paste it over the top of everything, you delete the default rule that lets your tailnet talk to itself, and every other connection you have stops working at the same moment.

{
  "groups": {
    "group:devs": ["you@example.com", "collaborator@example.com"]
  },
  "tagOwners": {
    "tag:container": ["autogroup:admin"]
  },
  "acls": [
    // ... your existing rules stay here ...
    {
      "action": "accept",
      "src": ["group:devs"],
      "dst": ["tag:container:443,22"]
    }
  ]
}

If you already worked through the ACL section of Tailscale for private networking, this is the same machinery pointed at a container instead of a server.

When you want it public after all

Sometimes you need to hand someone a link without adding them to your tailnet. Tailscale Funnel does that, and it's less of a one liner than it looks.

Two things have to be true first. Your tailnet policy needs the funnel node attribute, which for a tagged container node you will be adding by hand:

"nodeAttrs": [
  {
    "target": ["tag:container"],
    "attr": ["funnel"]
  }
]

Then flip the switch in serve.json and restart, which is the durable way to do it since TS_SERVE_CONFIG reapplies that file every time the container starts:

  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": true
  }

Restart with docker compose restart and no service name after you edit that file. Restarting only the Tailscale container is what leaves Forgejo answering 502 behind a network stack that moved out from under it, which is the gotcha below and the one that wastes the most time.

Funnel only works on ports 443, 8443, and 10000, so 443 is the one you want anyway.

The node attribute is not optional, and skipping it is the worst kind of failure. I flipped AllowFunnel to true without it and everything told me it had worked. tailscale funnel status printed Funnel on with the URL under it, tailscale serve status agreed, and the logs said nothing at all. The name simply never appeared in public DNS, so from outside the tailnet it did not resolve, let alone serve. The node did not have the capability and no part of the tooling mentioned it.

If you turn Funnel on, verify it from something that is not on your tailnet. A phone with WiFi off is enough:

dig +short @1.1.1.1 forgejo.tail1234.ts.net

An empty answer means Funnel is not really on, whatever the CLI told you.

Once it's on, you have inherited every problem this post was avoiding. Registration lockdown and rate limiting become your concern again. It's a good escape hatch and a bad default. Set it back to false when you're done.

The honest limitation: if the people you collaborate with will not join your tailnet, this setup is not for you. Tailnet only means tailnet only. Everyone who touches these repos needs Tailscale on their machine and a place in your ACLs. For a solo developer or a small team that already uses Tailscale, that cost is zero. For an open source project taking drive by contributions, it's a wall.

Gotchas I hit

No state volume, so every restart mints a new node. This is the big one, and I watched it happen. If TS_STATE_DIR has no volume behind it, the container loses its identity, registers again as a fresh device, and Tailscale appends a suffix to keep the name unique. Mine came back as forgejo-1 on a new address while the old forgejo sat there marked offline.

What makes it nasty is how healthy the result looks. The new node got its own certificate within seconds and served Forgejo on 443 without complaint. But ROOT_URL and SSH_DOMAIN live in app.ini inside the data volume, so they still held the old name, and the API cheerfully handed out clone URLs pointing at a node nothing is listening on. The server is fine. Every clone command it gives your users times out. Persist /var/lib/tailscale.

What losing that state does to you depends on your auth key. With a reusable key you get the duplicate node above. With a single use key the container cannot register at all, fails with invalid key: API key ... not valid, and never comes up. Two completely different mornings from the same missing volume.

Auth failures loop instead of stopping. Every registration failure I hit, whether a tag the policy did not allow or a spent key, left the container restarting rather than exiting. With restart: unless-stopped that continues forever. docker compose ps shows restarting, not an error, so if you run up -d and walk away you come back to something that has been failing quietly for however long you were gone. The reason is only ever in docker compose logs.

An untagged node drops off at six months, not three. The auth key expiring at 90 days is the number everyone quotes, but that only blocks new registrations. The node itself runs until its node key hits the 180 day default, which is a much worse way to find out, because by then you've forgotten the setup entirely. Tag the device and key expiry is off.

The tag has to exist before you advertise it. tag:container is my example name, not a default. If it is not in tagOwners in your policy the node refuses to join with requested tags [tag:container] are invalid or not permitted, which is at least an honest error. Use whatever tag your tailnet already has if you have one.

Tailscale SSH swallows Git over SSH. If you add --ssh to TS_EXTRA_ARGS, tailscaled intercepts inbound tailnet connections to port 22 before they ever reach the SSH server in the container. There's no bind conflict and nothing looks broken in the logs. Your clones just stop working. Leave Tailscale SSH off on this node and use it on your other machines.

Restarting the sidecar alone breaks Forgejo in a way that looks like something else. This one cost me the most time, because I did it to myself repeatedly while testing and kept diagnosing the wrong thing. docker compose restart ts-forgejo gives Forgejo a network stack that has moved out from under it. Forgejo keeps running, docker compose ps says both are Up, and the tailnet name still resolves.

What you get is a broken instance that points nowhere useful. The web side answers 502, because tailscaled terminates TLS perfectly well and then cannot reach 127.0.0.1:3000 any more. Git over SSH gives you Connection refused, or just hangs with no banner. Neither symptom points at the container you actually restarted.

The fix is docker compose restart with no service name, so both come back together. I checked this twice: sidecar alone gives 502 and a refused clone, both together gives 200 and a clean clone. Any time you touch the Tailscale container, take Forgejo with it.

ROOT_URL left over from an earlier attempt. If you tried this once with a different hostname, changing the environment variable is not always enough, because the installer wrote the old value into app.ini and that file lives in the volume. Check /data/gitea/conf/app.ini and fix it there, or start from a clean volume.

Quick reference

Setting	What it controls	What breaks without it
`network_mode: service:ts-forgejo`	Puts Forgejo in the sidecar's network namespace	Forgejo is not on the tailnet at all
`TS_USERSPACE=false`	Kernel networking instead of userspace	The TUN device and `net_admin` do nothing
`hostname` on the sidecar	The node name, and so the URL	Random or wrong `ts.net` name
`TS_STATE_DIR` plus a volume	Node identity across restarts	New device on every restart, name drifts
`TS_EXTRA_ARGS=--advertise-tags`	Marks the node as tailnet owned	Node key expires at 180 days, node drops off
`TS_SERVE_CONFIG`	HTTPS and the proxy to port 3000	No TLS, nothing served on 443
`FORGEJO__server__ROOT_URL`	Clone URLs, emails, webhooks	Everything points somewhere wrong
`FORGEJO__server__SSH_DOMAIN`	The SSH clone URL shown in the UI	Copy button hands out a command that times out
no `START_SSH_SERVER`	Lets the image's own sshd own port 22	Two SSH servers want the same port

Useful commands:

docker compose logs -f ts-forgejo                      # node registration and cert issuance
docker compose exec ts-forgejo tailscale status        # is it on the tailnet
docker compose exec ts-forgejo tailscale serve status   # what is being served on 443

# backup, written somewhere writable and then copied out
docker compose exec -u 1000 -w /tmp forgejo \
  forgejo dump -c /data/gitea/conf/app.ini

# the archive is named forgejo-dump-<timestamp>.zip, so read the name back
DUMP=$(docker compose exec -T forgejo sh -c 'ls -1 /tmp/forgejo-dump-*.zip | tail -1')
docker compose cp "forgejo:${DUMP%$'\r'}" ./

That timestamp is the part that bites. forgejo dump prints the filename it wrote and then you are on your own, so a copy command with a fixed name in it fails every time and you find out when you need the backup.

[ no open ports · real certs · git@ over the tailnet ]

Originally published at peculiarengineer.com.

Your MCP server's search is bad: ranking, embeddings, and what each one fixes

Minor Keith — Tue, 28 Jul 2026 02:34:37 +0000

At the end of Part One I left the notes server with a search I described as dumb, and promised to come back for it. Here is where it stands today, on the same query:

search_notes("firewall")

build-your-first-mcp-server-python: Build your first MCP server in Python: give Claude your own notes
caddy-reverse-proxy-docker-compose-ubuntu-26-04: Reverse Proxy Your Containers with Caddy and Docker Compose on Ubuntu 26.04
create-sudo-user-ubuntu-26-04: Create a Sudo User on Ubuntu 26.04
enable-ssh-on-ubuntu-desktop: Enable SSH on an Ubuntu desktop
extend-azure-windows-disk-run-command: Extending C: on locked-down Azure Windows VMs without RDP

It got worse. The top hit is now Part One itself, the post complaining that this search is bad, which mentions "firewall" only because it was quoting this exact output. The blog is up to 51 posts, 17 of them mention firewalls somewhere, and UFW Firewall Basics is still not in the list.

This post fixes it in two stages, because there are two different problems here and only one of them is the one everybody reaches for. The first is a ranking bug and it costs about twenty lines. The second is a query that no amount of ranking will ever answer, and that one costs an embedding model.

TL;DR Score every hit and sort, instead of taking the first five in alphabetical order. That alone puts the right post on top. Then add local embeddings with ollama pull nomic-embed-text, chunked on ## headings and cached in an .npz keyed by path, mtime and model name. Prefix documents with search_document: and queries with search_query:, which doubled how often my index put the right post first. Build the index from a command, not on the first search, because 496 chunks took four minutes on CPU. And keep both searches, because they fail in different places: the exact match misses brute force over a hyphen, and the embeddings miss it because two words are not enough meaning to work with.

Before you start

The finished server from Part One. Everything below edits that server.py.
Ollama running locally. That post is about feeding a GPU for chat models, and none of that applies here: embedding models are small and a CPU is fine for this job.
Two dependencies: uv add ollama numpy

1. The bug was ranking, not matching

Part One already named this, and it is worth quoting because the diagnosis is the whole of Stage One:

The substring match did not fail. It found the UFW post fine. The problem is that there is no ranking at all.

Look at what the original loop does:

hits = []
for path in _posts():
    text = path.read_text(encoding="utf-8")
    if query.lower() in text.lower():
        hits.append(f"{path.stem}: {_title(text, path.stem)}")
    if len(hits) == 5:
        break

_posts() globs the .md files and the .mdx files separately and concatenates the two sorted lists, so the order is alphabetical within each extension and any .mdx file lands at the end no matter what it is called. The loop appends in that order and breaks at five. So search_notes is not returning the five best matches for "firewall," it is returning the first five files in that order that contain the string anywhere, code blocks and links included. u sorts after b, c, and e, so the post that is entirely about firewalls loses to four posts that mention them in passing.

That is worse than returning nothing, because five confident, plausible, wrong results are an answer the model will happily build on.

The fix is to stop treating a match as a boolean. Collect every hit, score it, sort, then truncate:

def _frontmatter(text: str) -> str:
    parts = text.split("---", 2)
    return parts[1] if len(parts) > 2 else ""


def _fm_line(fm: str, key: str) -> str:
    return next((l for l in fm.splitlines() if l.startswith(key)), "")


def _score(path: Path, text: str, q: str) -> int:
    low = text.lower()
    if q not in low:
        return 0

    fm = _frontmatter(low)
    score = 0
    if q in _fm_line(fm, "title: "\"):"
        score += 100
    if q in path.stem.lower():
        score += 60
    if q in _fm_line(fm, "tags:"):
        score += 40
    # Body mentions count for something, but cap them so a 4,000 word post
    # cannot win on length alone.
    score += min(low.count(q), 10) * 3
    return score

Four signals, weighted by how much intent each carries. A title match scores highest because a title is the strongest claim a post makes about its own subject. The slug is close behind, since I hand-write slugs on this blog and they say what the post is about. Tags sit below that, and body count is the tiebreaker, capped at ten so a long post cannot grind out a win on length alone.

Then pull the ranked lookup into its own function, because Stage Two is going to need it separately from the tool:

def _keyword_hits(query: str, k: int = 5) -> list[tuple[str, str]]:
    """Every post containing the phrase, best first, as (slug, title)."""
    q = query.lower()
    scored = []
    for path in _posts():
        text = path.read_text(encoding="utf-8")
        score = _score(path, text, q)
        if score:
            scored.append((score, path.stem, _title(text, path.stem)))

    scored.sort(key=lambda row: (-row[0], row[1]))
    return [(slug, title) for _, slug, title in scored[:k]]


@mcp.tool()
def search_notes(query: str) -> str:
    """Search my blog posts for a word or phrase.

    Matches the literal phrase anywhere in a post. Results are ranked: a match
    in the title, slug or tags outranks a passing mention in the body. Returns
    the 5 best hits with slug and title. Use get_note with a slug to read a
    full post.
    """
    hits = _keyword_hits(query)
    if not hits:
        return f"No posts mention {query!r}."
    return "\n".join(f"{slug}: {title}" for slug, title in hits)

Same query, same corpus:

ufw-firewall-basics-ubuntu: UFW Firewall Basics on Ubuntu
hardening-ubuntu-desktop: Hardening an Ubuntu Desktop
ssh-connection-refused-port-22-ubuntu: Troubleshooting "ssh: connect to host port 22: Connection refused"
build-your-first-mcp-server-python: Build your first MCP server in Python: give Claude your own notes
enable-ssh-on-ubuntu-desktop: Enable SSH on an Ubuntu desktop

The UFW post scores 230 against 61 for the runner up, because it hits the title, the slug, the tags and the body. Part One drops to fourth where it belongs.

The docstring changed too. Part One's said "There is no ranking" and told the model to search repeatedly with different wording. That was honest then and is wrong now, and left alone it would keep the model burning three calls to work around a limitation that no longer exists.

If you have fifty markdown files, you could stop reading here. Twenty lines, no new dependencies, no model, nothing to keep in sync. That is a real answer and I am not going to pretend otherwise to justify the rest of the post.

2. The query ranking cannot touch

Here is the one that sent me looking further. I have a post about stopping repeated SSH login attempts. Ask for it the way a person would ask:

search_notes("brute force")

No posts mention 'brute force'.

Not badly ranked, not buried at position nine. Zero, across 51 posts.

Two separate failures are stacked on top of each other there. UFW Firewall Basics says "throttle brute-force knocking," so the word is right there in the corpus, and it does not match because I typed a space where the post has a hyphen. Substring matching is not looking for a concept, it is looking for a byte sequence, and those are two different byte sequences.

The second failure is the one that matters. The post that actually answers this question is the Fail2ban post, and it does not contain the word in any spelling. It never says "brute" or "attack" at all. It opens like this:

A public SSH server starts collecting failed logins almost as soon as it gets an address. Key-only authentication makes those guesses useless, but it does not stop the same addresses filling the journal all day.

"Failed logins," "guesses," "the same addresses." Every one of those describes the thing better than "brute force" does, and not one of them is the phrase somebody would type into a search box. I wrote a whole post about defending against brute force attacks without ever using the term.

This is where ranking runs out. Ranking orders the matches you already have. It cannot manufacture one for a word you never wrote.

And the caller here is not me. It is a model turning somebody's half-remembered question into search terms, and it will not guess my vocabulary. Part One patched around this by telling the model in the docstring to try different wording, which works about as well as it sounds. What the server needs is a search that matches on meaning instead of spelling.

3. What an embedding actually is

A model reads a piece of text and hands back a fixed-length list of floats. That is all. nomic-embed-text gives you 768 of them. The useful property is that texts about similar things land near each other in that 768-dimensional space, so "brute force attack" and "the same addresses filling the journal" end up close together despite sharing no words.

"Near" is cosine similarity, which for unit-length vectors is just a dot product: multiply the two lists element by element, add up the results, get a number between -1 and 1. Higher is more related. That is the entire retrieval algorithm.

Here is what you do not need, despite what a search for this will tell you: a vector database. No Chroma, no pgvector, no Pinecone, no index structure of any kind. Fifty-one posts chunked by heading comes to 496 sections, so the entire index is a 496 by 768 array of float32, about 1.5 MB. Comparing a query against every single one of them is one matrix multiply. You need a real vector store somewhere in the tens of thousands of documents, when scanning everything stops being free. Below that it is a file you can delete when it gets weird, which is worth more than it sounds.

4. Embeddings locally with Ollama

ollama pull nomic-embed-text

The current SDK call is embed(), which takes a list and returns a list:

import ollama

r = ollama.embed(model="nomic-embed-text", input=["hello", "goodbye"])
print(len(r.embeddings), len(r.embeddings[0]))

Two vectors of 768 floats each. The older ollama.embeddings(prompt=...) is the deprecated single-string version. Use embed(): batching matters later.

Two things about this model are not optional, and both fail silently.

Prefix your inputs. nomic-embed-text is trained with task prefixes. Documents go in as search_document: {text} and queries as search_query: {text}. Ollama will not add them for you. Leave them off and nothing breaks: you get vectors, plausible similarity scores, and ranked results. There is no error to chase.

I nearly talked myself out of this one. Comparing raw cosine scores between two documents, prefixed and not, the numbers barely move, which looks like proof that the prefixes are folklore. They are not. Raw cosine is the wrong thing to measure, because a change that shifts every score equally changes no rankings. What matters is where the right post lands.

So I built the index twice over all 496 chunks and ran eight questions with known answers through both:

	Mean reciprocal rank	Correct post first	Mean rank
No prefixes	0.480	2 of 8	3.6
With prefixes	0.682	4 of 8	2.4

Prefixing doubles the number of questions answered correctly on the first result. "Something keeps trying to log into my server over and over" goes from seventh place to first. That is not folklore, and it costs one f-string.

Know where it truncates. Ollama serves this model with a 2,048 token context. Feed it a longer chunk and it drops the tail and tells you nothing, which is the second silent failure in a row.

ollama show will tell you, if you think to ask:

$ ollama show nomic-embed-text
  Model
    architecture        nomic-bert
    parameters          137M
    context length      2048
    embedding length    768
    quantization        F16

  Capabilities
    embedding

  Parameters
    num_ctx    8192

Read those last two numbers together, because between them they are the whole trap. The architecture stops at 2,048, and the model file already asks for 8,192. The setting you were about to reach for is set, and it is not doing anything.

The obvious move is options={"num_ctx": 8192}, since nomic's own model card describes an 8,192 window. It does not work, and it is worth showing how I know, because "the flag had no effect" is a hard thing to prove by staring at vectors.

Take one long document, copy it, and change only the last sentence. If the model reads the whole thing, the two copies must embed slightly differently. If it truncates, the two copies are byte-identical up to the cut and the vectors come back the same:

filler = "The firewall configuration is stored in the usual place. " * 400
a = filler + " ZEBRA QUASAR MARMALADE."
b = filler + " The cat sat quietly on the warm mat."

That is roughly 5,700 tokens of identical text with two very different endings. On Ollama 0.32.4:

num_ctx=None  -> cosine(a,b)=1.000000
num_ctx=8192  -> cosine(a,b)=1.000000

Identical to six decimal places, with and without the flag. Everything past the limit was thrown away in both cases. The clamp is in the Ollama log if you go looking, one WARN line and then the model loads at 2,048 regardless:

level=WARN source=server.go:114 msg="requested context size too large for model"
  num_ctx=8192 n_ctx_train=2048
llama_context: n_ctx = 2048

Treat num_ctx as a request, not a setting, and check it the way I did rather than trusting either the model card or me.

The better answer is to not need it. Chunking on ## headings does most of the work for you. Across this blog, 496 sections, the longest one in any prose post is 7,825 characters, which at the usual four-characters-per-token rule of thumb lands just under the limit. The only sections that clearly blow through it are the four complete-source dumps at the end of the LÖVE posts, and the largest of those are past 8,192 as well, so raising the window would not have rescued them either. A wall of Lua that size should not be competing for search hits anyway. Split those or skip them.

5. Wiring it into the server

Four pieces: chunk, embed, cache, search.

Chunking is not optional. One vector for a 4,000 word post averages every idea in it into mush. The server hardening post covers SSH, the firewall, unattended upgrades, fail2ban and livepatch; embedded whole, it is a vector for "Ubuntu things," equally mediocre at all five. Chunk it and each section gets to be about one thing.

Every post on this blog uses ## headings, so the chunk boundaries already exist. That also means a hit can report which section matched, which is strictly more useful to the model than a slug.

import numpy as np
import ollama

MODEL = "nomic-embed-text"
INDEX = Path(__file__).parent / "index.npz"
FLOOR = 0.5          # minimum cosine score worth returning, see below


def _chunks(path: Path) -> list[tuple[str, str]]:
    """Split a post into (heading, text) pairs on ## boundaries."""
    text = path.read_text(encoding="utf-8")
    title = _title(text, path.stem)
    body = text.split("---", 2)[-1]

    sections, heading, buf = [], "intro", []
    for line in body.splitlines():
        if line.startswith("## "):
            if buf:
                sections.append((heading, "\n".join(buf)))
            heading, buf = line[3:].strip(), []
        else:
            buf.append(line)
    if buf:
        sections.append((heading, "\n".join(buf)))

    # Carry the title into every chunk so a section knows what post it is from.
    return [
        (h, f"{title}\n\n## {h}\n\n{b}")
        for h, b in sections
        if b.strip()
    ]


def _embed(texts: list[str], prefix: str) -> np.ndarray:
    try:
        r = ollama.embed(model=MODEL, input=[f"{prefix}: {t}" for t in texts])
    except (ollama.ResponseError, ConnectionError) as e:
        # ConnectionError is the daemon not running. stderr, never stdout.
        print(f"ollama embed failed: {e}", file=sys.stderr)
        raise
    v = np.array(r.embeddings, dtype="float32")
    # Normalize once, here, so search is a plain dot product later.
    return v / np.linalg.norm(v, axis=1, keepdims=True)

Now the cache, and this is the part where my first design was wrong.

Embedding the whole blog takes real time. On my Mac, CPU only, 496 chunks indexed in 239 seconds. That is 482 ms per chunk, and it is four minutes of wall clock before the first search can return anything.

My instinct was to build the index lazily on the first search, so mcp.run() starts instantly and the handshake never blocks. That is the right instinct and the wrong conclusion: it moves a four minute wait from startup, where nobody is looking, onto a tool call the model is waiting on. Build the index ahead of time instead, from a command you run yourself, and have the server load a file that already exists. The lazy path stays as the fallback for a cold cache, but it should be the exception, not the plan.

So: build it once, keep it on disk, and only re-embed what changed.

def _load_cache() -> tuple[np.ndarray | None, list[tuple]]:
    if not INDEX.exists():
        return None, []
    data = np.load(INDEX, allow_pickle=True)
    if str(data["model"]) != MODEL:      # a model swap invalidates every vector
        return None, []
    return data["vecs"], [tuple(row) for row in data["meta"]]


def _build_index() -> tuple[np.ndarray, list[tuple]]:
    """Load the cached index, re-embedding only posts whose mtime moved."""
    cached_vecs, cached_meta = _load_cache()
    live = {path.stem: str(path.stat().st_mtime_ns) for path in _posts()}

    keep_rows, keep_meta = [], []
    for i, row in enumerate(cached_meta):
        slug, mtime = row[0], row[1]
        if live.get(slug) == mtime:      # unchanged file, reuse its vectors
            keep_rows.append(cached_vecs[i])
            keep_meta.append(row)

    done = {row[0] for row in keep_meta}
    todo_texts, todo_meta = [], []
    for path in _posts():
        if path.stem in done:
            continue
        title = _title(path.read_text(encoding="utf-8"), path.stem)
        for heading, text in _chunks(path):
            todo_texts.append(text)
            todo_meta.append((path.stem, live[path.stem], heading, title))

    if not todo_texts:
        vecs = np.array(keep_rows)
        if len(keep_meta) != len(cached_meta):   # a post was deleted, persist the prune
            np.savez(INDEX, vecs=vecs, meta=np.array(keep_meta, dtype=object), model=MODEL)
        return vecs, keep_meta

    print(f"embedding {len(todo_texts)} chunks", file=sys.stderr)
    new_vecs = _embed(todo_texts, "search_document")   # one batched call
    vecs = np.vstack([np.array(keep_rows), new_vecs]) if keep_rows else new_vecs
    meta = keep_meta + todo_meta
    np.savez(INDEX, vecs=vecs, meta=np.array(meta, dtype=object), model=MODEL)
    return vecs, meta


_INDEX_CACHE: tuple | None = None


def _index():
    """Loaded on first search. Warm cache is instant; a cold one costs minutes."""
    global _INDEX_CACHE
    if _INDEX_CACHE is None:
        _INDEX_CACHE = _build_index()
    return _INDEX_CACHE


if __name__ == "__main__":
    # This replaces Part One's block. Build the index from the shell before
    # starting the server:
    #     uv run --directory /abs/path server.py --reindex
    if "--reindex" in sys.argv:
        vecs, meta = _build_index()
        print(f"indexed {len(meta)} chunks", file=sys.stderr)
        raise SystemExit

    print(f"serving {len(_posts())} posts from {NOTES}", file=sys.stderr)
    mcp.run(transport="stdio")

Four things in there matter. The model name goes into the cache file, so swapping models throws the whole index out instead of comparing 768-dimensional rows against a 1024-dimensional query. st_mtime_ns is the freshness check, so editing one post re-embeds one post and leaves the other fifty alone, which turns a four minute rebuild into a two second one. Deleting a post is the quiet case: its slug never makes it into keep_meta, and the save on the early return writes the pruned index back to disk, because otherwise the dead vectors sit in index.npz forever and every run prunes them again. And the --reindex path means the expensive build happens when you ask for it, not in the middle of somebody's question.

Search itself is four lines:

def _similar(query: str, k: int = 5) -> list[tuple[float, str, str, str]]:
    vecs, meta = _index()
    q = _embed([query], "search_query")[0]
    scores = vecs @ q                      # both sides normalized, so this is cosine
    top = np.argsort(-scores)[:k]
    # meta rows are (slug, mtime, heading, title)
    return [(float(scores[i]), meta[i][0], meta[i][2], meta[i][3]) for i in top]

That vecs @ q is the whole search engine. One matrix multiply against every chunk on the blog.

6. What it actually retrieves

Here is where I found out my plan for this post was wrong.

The whole setup in Stage Two was that brute force should find the Fail2ban post, and that meaning-based search is the thing that gets you there. So I built the index, 496 chunks across 51 posts, and asked it:

query: "brute force"
  0.6237  ufw-firewall-basics-ubuntu        ## Rate limiting the front door
  0.6048  hardening-ubuntu-desktop          ## First, know what you are defending against
  0.5961  hardening-ubuntu-26-04-server     ## See what is actually listening
  0.5951  dev-handoff-claude-code-skill     ## How it works, section by section
  0.5950  hardening-ubuntu-26-04-desktop    ## Per-app permission prompts

The Fail2ban post is not in the top five. It is eighth. Fourth place is a post about handing work to AI sub-agents, which has nothing to do with any of this.

That is a better result than the substring match, which returned nothing at all, and the top hit is defensible: the UFW post's rate limiting section really is about throttling repeated connection attempts. But it is not the answer I promised, and the spread across those five is 0.03, which is another way of saying the index has no strong opinion.

Now ask the same thing as a sentence:

query: "how do I stop repeated failed SSH login attempts"
  0.7546  set-up-fail2ban-ubuntu-26-04      ## intro
  0.7425  set-up-ssh-keys-ubuntu            ## 5. Turn off password login
  0.7232  hardening-ubuntu-26-04-server     ## Order matters: do not lock yourself out
  0.7185  set-up-fail2ban-ubuntu-26-04      ## Before you install it
  0.7168  create-sudo-user-ubuntu-26-04     ## Now lock down root

First place, and second place is the post it tells you to read first. Phrase it more vaguely and it still holds: "something keeps trying to log into my server over and over" also puts the Fail2ban intro on top.

Two words is not enough to embed. That is the lesson, and I would not have believed it without the numbers. An embedding is a summary of meaning, and "brute force" on its own carries almost none: no target, no symptom, no context. The model has to guess whether you mean SSH, or a search algorithm, or a way of solving a puzzle. A sentence gives it something to summarize.

Which is fine, and it is worth understanding why. The caller is not a human typing two words into a box. It is a model turning somebody's question into a tool call, and models write sentences. The query shape this search is worst at is the one it will rarely be handed. But it does mean the docstring should tell it so, and mine now does.

7. Keep both

I expected this section to be about embeddings falling over on exact strings. It is not, because they do not. Ask the index for _ctypes and the top five hits are all sections of the right post, starting at 0.7004. That is the single cleanest result in this whole experiment, and it is the opposite of what I sat down to write.

So the case for keeping both is not that one is bad at literals. It is that they fail in different places, and you now have both failures on record. The substring match returned nothing for brute force over a hyphen. The embedding index returned nothing useful for the same two words because two words carry no meaning to summarize. Neither of those is a flaw you can fix from inside the other approach.

The other reason is cheaper and more practical. When I paste an error string into my own notes, I want to know that a post literally contains it, not that a post is thematically nearby. Exact match is a different kind of answer, not a worse one, and it costs one function call to keep.

So the answer is both, in one tool:

@mcp.tool()
def search_notes(query: str) -> str:
    """Search my blog posts.

    Runs two searches and merges them: an exact-phrase search ranked by where
    the phrase appears, and a meaning-based search that finds related posts
    even when they never use your words. Good for both error strings and
    plain-English questions. Returns up to 5 posts, best first, with the
    section that matched. Use get_note with a slug to read a full post.
    """
    exact = _keyword_hits(query)
    # Ask for more chunks than we need: several may come from one post and
    # collapse into a single line below.
    related = _similar(query, k=15)

    seen, out = set(), []
    for slug, title in exact:                    # literal matches win the top
        if slug not in seen:
            seen.add(slug)
            out.append(f"{slug}: {title}")
    for score, slug, heading, title in related:  # semantic fills the rest
        if slug not in seen and score > FLOOR:
            seen.add(slug)
            out.append(f"{slug}: {title}  (matched section: {heading})")

    return "\n".join(out[:5]) or f"Nothing found for {query!r}."

The merge rule is deliberately blunt: anything the keyword search found literally goes first, because if you typed an exact string you almost certainly meant it, and semantic hits fill whatever is left. Chunk hits collapse to one line per post, keeping the best-scoring section as the label, since the model wants a slug it can pass to get_note and not four near-duplicate rows from the same file.

FLOOR exists so a query with no real answer returns nothing instead of the five least-unrelated posts on the blog, which is the semantic version of the bug in Stage One. I set it to 0.5 by guessing, and the numbers say that was useless: on this corpus every hit that came back for every query I tried scored above 0.59, including the sub-agent post that has nothing to do with brute force attacks. A floor of 0.5 filters nothing at all.

Then I went looking for a number that would work, and did not find one. Real scores here sit between 0.59 and 0.76, and across five queries neither obvious rule separates the good answers from the bad ones:

Query	Top score	Gap to second	Right answer?
`brute force`	0.6237	0.0189	no
`how do I stop repeated failed SSH login attempts`	0.7546	0.0121	yes
`something keeps trying to log into my server over and over`	0.6540	0.0009	yes
`_ctypes`	0.7004	0.0443	yes
`my server forgets its IP address after reboot`	0.6985	0.0080	no

A floor of 0.70 would have thrown away a correct answer at 0.6540. And the gap between first and second place, which I assumed would be the tell, does not sort them either. The widest gap on that list is a success and the narrowest gap is also a success, a factor of fifty apart, with both of the failures sitting comfortably in between. Both of the tidy heuristics I expected to find are wrong on my own data.

So FLOOR stays at 0.5 as insurance, not as a demonstrated filter. It never fired on any query I tried, and I am keeping it for the query that is genuinely about nothing on this blog, which none of my test queries were. The honest position is that I do not have a reliable confidence signal for this yet, and anyone who tells you a fixed cosine threshold means "relevant" has not printed their own numbers.

One tool, not two. It is tempting to ship search_notes and search_notes_semantic and let the model choose. Do not. The model would have to know which search strategy suits a query it has not run yet, which is a harder problem than the one you are solving, and it will get it wrong in the direction of whichever docstring reads more confidently. Every tool you add is another thing it can pick wrong. Give it one door and do the routing yourself.

8. Ask it the way a person would

Testing the function at a terminal is not the point. The caller is a model, so restart Claude Code and ask it the way somebody would actually ask.

how do I stop people repeatedly trying to log into my server?

mcp-server-search-ranking-embeddings-ollama: Your MCP server's search is bad: ranking, embeddings, and what each one fixes
set-up-fail2ban-ubuntu-26-04: Set Up Fail2ban on Ubuntu 26.04 Server  (matched section: intro)
hardening-ubuntu-26-04-server: Hardening Ubuntu 26.04 Server (Resolute Raccoon)  (matched section: Order matters: do not lock yourself out)
create-sudo-user-ubuntu-26-04: Create a Sudo User on Ubuntu 26.04  (matched section: Now lock down root)
set-up-ssh-keys-ubuntu: Set up SSH keys for Ubuntu  (matched section: 5. Turn off password login)

The Fail2ban post is there in second place, matched on its intro, which is the section that never says "brute" or "attack." That is the result section 6 promised and the whole point of the exercise.

Now look at what is sitting on top of it.

That is this post. And look at what the top line is missing: every other row carries a matched section, because every other row came out of the embedding index. The top row does not, which means it came back from the exact-phrase half.

When I ran this, the sentence I had just typed existed in exactly one file on the blog. It was in this section, inside an HTML comment, in a note to myself that said to ask something like "how do I stop people repeatedly trying to log into my server?" I wrote the question down before I wrote the section. The literal search found my own unfinished to-do list and ranked it above the answer.

Part One opened with the search returning Part One. Two stages and a rewrite later, it opens with the search returning a comment I had not finished writing.

There is a real lesson under the joke, and it is about the merge rule in Stage One. Exact matches go first because if you typed a literal string you almost certainly meant it. That reasoning holds for _ctypes. It does not hold for a twelve word question, because a twelve word question matching a document verbatim does not mean the document answers it. It means the document quotes it. Length should have been part of that decision and it is not.

The contamination is not only literal, either. Ask something with no phrase in common with anything I have written:

someone keeps guessing passwords on my ssh server and filling the logs

hardening-ubuntu-26-04-server: Hardening Ubuntu 26.04 Server (Resolute Raccoon)  (matched section: Order matters: do not lock yourself out)
set-up-ssh-keys-ubuntu: Set up SSH keys for Ubuntu  (matched section: Gotchas I hit)
ssh-config-file-explained: The ~/.ssh/config file explained: aliases, jump hosts, and a key per host  (matched section: A key per host)
mcp-server-search-ranking-embeddings-ollama: Your MCP server's search is bad: ranking, embeddings, and what each one fixes  (matched section: 2. The query ranking cannot touch)
set-up-fail2ban-ubuntu-26-04: Set Up Fail2ban on Ubuntu 26.04 Server  (matched section: intro)

Fail2ban falls to fifth, and this post is fourth on the strength of section 2, which is three hundred words about failed SSH logins that solve nothing. Every measurement in section 6 was taken on an index of 51 posts. There are 52 now, and the new one is a long post about searching for the answer rather than a post containing it.

I do not have a clean fix for that and I am not going to invent one. A post about retrieval is a genuinely hard document to keep in a retrieval index, because it is made of other posts' vocabulary. What I have is the knowledge that my corpus now contains a decoy, which is worth more than a confidence score.

9. Gotchas I hit

Part One's worst bug was a print() that worked fine until a buffer filled weeks later, and the first two here have the same shape: no crash, no error, just an index that is quietly worse than you think.

I built the first index without the task prefixes. Nothing told me. Vectors came back, the scores looked reasonable, the results were ranked, and I nearly shipped it. It took building the whole index a second time and scoring both against eight questions to see that the first one was measurably worse. There is no error to chase here, only the version of the index you did not build.
I reached for num_ctx before I tested whether it did anything. Ollama serves this model at 2,048 tokens, drops the rest without a word, and clamps a larger num_ctx back down to the trained context. Two documents differing only after the cut embed to cosine 1.000000, which is how I found out. Measure your longest chunk instead of trusting the flag.
A four minute index build in the wrong place. 496 chunks took 239 seconds on CPU. Put that on the first search, the way I first designed it, and the model waits four minutes for a tool call. Build ahead of time and let the server load a file.
A model swap poisons a cache keyed only on mtime. No file changed, so nothing re-embeds, and now a 768-dimensional index meets a 1024-dimensional query. That one at least fails loudly, though in numpy's terms rather than yours:

  ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0,
  with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1024 is different from 768)

Nothing in there says "you changed embedding models." Put the model name in the cache file and check it on load, which is what _load_cache() above is for.

The two halves of the merged tool can disagree about the corpus. _INDEX_CACHE loads once per process, and Part One's rule that config is read at session start applies to it too. Edit a post mid-session and the keyword half sees the change on the next call, because it reads the files live every time, while the embedding half keeps serving the old vectors until the server restarts. One tool, two answers about what the blog says.
Ollama being down fails at search time, not startup. The index builds lazily, so claude mcp list reports ✔ Connected, the tools list normally, and the failure waits until someone searches. The try in _embed covers both paths that need it, the index build and the query, because _similar embeds the query on every single search whether the index is warm or not. The daemon being down surfaces as a plain ConnectionError, and the reason goes to stderr, which Part One established is the only channel that will not corrupt the transport.
The post you are reading poisoned its own index. Section 8 has the whole story. A twelve word question matched this file literally, because I had written that question into a to-do comment before I had written the section under it, and the merge rule puts literal matches first. The lesson is not about comments. It is that the corpus is not a fixed thing you index once and reason about afterwards. It contains whatever you are currently writing, including the parts that are not finished, and a post about search is made almost entirely of other posts' vocabulary.
The docstring changed three times in two posts. No ranking, then ranking, then two searches at once. Every time it went stale the model kept working around a limitation that no longer existed. It is still the only description of this tool it will ever see.

10. Quick reference

Task	Command / code
Add the deps	`uv add ollama numpy`
Pull the model	`ollama pull nomic-embed-text`
Embed a batch	`ollama.embed(model=MODEL, input=[...])`
Keep chunks under	2,048 tokens, where Ollama truncates this model
Check that limit yourself	`ollama show nomic-embed-text`
Prefix documents	`search_document: {text}`
Prefix queries	`search_query: {text}`
Build the index	`uv run --directory /abs/path server.py --reindex`
Normalize	`v / np.linalg.norm(v, axis=1, keepdims=True)`
Search	`vecs @ q` then `np.argsort(-scores)[:k]`
Cache key	path + `st_mtime_ns` + model name
Force a rebuild	`rm index.npz`
Inspect the index	`np.load("index.npz", allow_pickle=True)["meta"]`
Reload the server	restart Claude Code, config is read at session start
Pre-allow headless	`claude -p "..." --allowedTools "mcp__notes__search_notes"`

Part One's closing line was that the protocol is the easy part and what you put behind the door is your problem. Two stages later that still holds. The ranking fix is twenty lines and should have shipped in Part One. The embeddings are a 768-float vector per section and a matrix multiply.

Neither is the interesting part. The interesting part is that I wrote a whole post about stopping brute force attacks and never once typed the words. Any search I put in front of these notes has to survive that, because the person asking will not know what I called it.

[ two searches, one door ]

Originally published at peculiarengineer.com.

Reverse Proxy Your Containers with Caddy and Docker Compose on Ubuntu 26.04

Minor Keith — Fri, 24 Jul 2026 02:15:03 +0000

Once you have Docker running on 26.04 and a couple of containers up, you hit the wall everyone hits: your apps live on a pile of random ports. Plex on :32400, something else on :8080, a dashboard on :9000, and none of them speak HTTPS. You want plex.example.com and books.example.com to just work, with a real certificate, without hand-rolling Nginx configs or running certbot on a cron job.

Caddy is the boring, dependable answer to that, and it is short. A hostname in a Caddyfile is all it takes for Caddy to go get a Let's Encrypt certificate, serve it on 443, and renew it forever without you thinking about it again. This is the setup I use to front my own containers, and this post is where I keep the parts that are not obvious the first time.

TL;DR Run Caddy in the same Compose project as your apps, put every container on one shared Docker network, and in the Caddyfile reverse_proxy to the container name and its internal port, never localhost. Caddy gets HTTPS automatically as long as your domain's DNS points at the box and ports 80 and 443 are reachable. While testing, set acme_ca to the Let's Encrypt staging endpoint so a broken config does not burn your weekly certificate quota.

Prerequisites

Ubuntu 26.04 with Docker and the Compose plugin. If you are not there yet, start with Install Docker on Ubuntu 26.04.
A domain you control, with an A record (and AAAA if you have IPv6) pointing at the box's public IP.
Ports 80 and 443 reachable from the internet: open in UFW and forwarded at your router if you are behind NAT.

The mental model

Two ideas carry this whole setup, and if you hold them straight nothing else is confusing.

First, Caddy and your apps have to share a Docker network so Caddy can reach them by name. Containers on the same user-defined network get automatic DNS: a service called whoami is reachable at the hostname whoami from any other container on that network. Caddy proxies to those names. Your apps then do not need to publish any ports to the host at all, which is the point. Only Caddy is exposed, everything behind it is private.

Second, automatic HTTPS is triggered by using a real hostname, not a port. When the Caddyfile says books.example.com { ... }, Caddy registers an ACME account, proves it controls that name over port 80, installs the certificate, and serves 443. When it says :8080 { ... } it does not, because there is nothing to get a certificate for. The hostname is the switch.

1. The Compose file

Here is a complete, working stack: Caddy plus one example backend (whoami, a tiny container that prints request info, perfect for confirming the proxy works before you point it at anything real).

services:
  caddy:
    image: caddy:2.9
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"        # HTTP/3
    volumes:
      - /opt/dockerapp/caddy:/etc/caddy   # mount the dir, not the file
      - caddy_data:/data                  # certificates live here, keep it
      - caddy_config:/config
    networks:
      - web

  whoami:
    image: traefik/whoami
    container_name: whoami
    restart: unless-stopped
    networks:
      - web
    # note: no `ports:` at all. Only Caddy is exposed.

networks:
  web:

volumes:
  caddy_data:
  caddy_config:

Two lines earn their comments. caddy_data is a named volume holding your certificates and ACME account key; if you delete it you throw away real Let's Encrypt certificates and have to fetch them again, which matters because of the rate limit below. And the config mount is the directory /opt/dockerapp/caddy, not the single Caddyfile. Mounting a lone file bites you when an editor replaces it on save: the container keeps pointing at the old inode and your edits appear to do nothing. Mount the folder and that problem disappears.

Create the config directory before you bring the stack up:

sudo mkdir -p /opt/dockerapp/caddy

2. The Caddyfile

Drop this at /opt/dockerapp/caddy/Caddyfile:

{
      email you@example.com
}

whoami.example.com {
      reverse_proxy whoami:80
}

That is the entire config for one site. The global block at the top sets the email Let's Encrypt uses for expiry notices. The site block says "for this hostname, hand the request to the whoami container on port 80." Bring it up:

cd /opt/dockerapp/caddy
docker compose up -d

Point whoami.example.com at the box, wait for DNS, and load it over HTTPS. You should see the whoami output with a valid padlock and no certificate warning. Adding a second app is another three lines:

books.example.com {
      reverse_proxy calibre-web:8083
}

The port you write is the container's internal port, the one the app listens on inside its own container, not any host port you might have published elsewhere. Caddy is talking to it over the Docker network, so the host port mapping is irrelevant here.

3. The shared network is the whole trick

This is the part that sends people to the search bar, so it gets its own section. If Caddy and the backend are not on the same Docker network, reverse_proxy whoami:80 fails with a DNS error in the Caddy logs:

dial tcp: lookup whoami on 127.0.0.11:53: no such host

That message means exactly what it says: Caddy asked Docker's internal DNS for whoami and got nothing, because from Caddy's network that name does not exist. The fix is always the same: put both services on the same network. In the Compose file above they share web, so it works.

The trap shows up when your apps live in a different Compose file (a common way to organize a homelab, one project per app). Compose names each project's default network after the project, and those networks are isolated. To let Caddy reach a container in another project, declare a shared external network and attach both sides to it. Create it once:

docker network create web

Then in every Compose file, Caddy's and each app's, mark that network external:

networks:
  web:
    external: true

and add - web to each service's networks: list. Now they all share one bridge and Caddy can resolve every container by name across projects.

One real exception worth knowing: a container running with network_mode: host is not on any bridge network and has no Docker DNS name. My Plex container uses host networking because it needs it for discovery, so Caddy cannot reach it as plex. For those, proxy to the host's LAN address and the published port instead:

plex.example.com {
      reverse_proxy 192.168.1.10:32400
}

4. How automatic HTTPS actually happens

Caddy's certificate magic is not magic, and knowing the steps tells you exactly what to fix when it fails. On first request for a hostname, Caddy:

Registers an ACME account with Let's Encrypt (once, stored in caddy_data).
Requests a certificate for the hostname.
Proves control by answering an HTTP-01 challenge on port 80, or a TLS-ALPN challenge on 443.
Installs the certificate and serves HTTPS on 443, redirecting HTTP to HTTPS.
Renews automatically around 30 days before expiry.

So the requirements are concrete: the DNS record must resolve to this box, and port 80 must be reachable from the public internet for the challenge. If 80 is closed at the firewall or not forwarded through your router, the challenge fails and you get no certificate, no matter how correct the Caddyfile is. Check the logs with docker compose logs caddy and you will see the ACME error spelled out.

If the box is LAN-only and has no public DNS, public certificates are not an option, because Let's Encrypt cannot reach it to run the challenge. Two ways out: use Caddy's built-in local CA with tls internal (you then trust its root on your devices), or use the DNS-01 challenge, which proves control by writing a TXT record instead of answering on port 80. DNS-01 needs a Caddy image built with your provider's DNS module, so it is a custom build, not the stock caddy:2.9. For a normal internet-facing box, none of that applies and the default just works.

5. Reloading without downtime

After editing the Caddyfile, you do not need to restart the container and drop connections. Caddy reloads its config in place:

docker exec -w /etc/caddy caddy caddy reload

The -w /etc/caddy runs the command from the config directory so Caddy finds the Caddyfile without a --config flag. If the new config has a syntax error, the reload is rejected and the old config keeps serving, so a typo does not take your sites down. Validate a change before reloading if you want the check without applying it:

docker exec -w /etc/caddy caddy caddy validate

Gotchas I hit

The rate limit that burns a real certificate. Let's Encrypt allows 5 certificates per exact hostname per week. Fight a broken config with the production endpoint and you can exhaust that quota fast, then you are locked out of new certificates for that name for days. While you are still getting things working, point Caddy at the staging CA, which issues certificates your browser will not trust but that never run out:

{
      email you@example.com
      acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

Your browser will warn about the staging certificate, that is expected. Once the setup is solid, delete that acme_ca line, reload, and Caddy fetches a real one.

Editing the mounted single file did nothing. This is the inode trap from earlier. I mounted ./Caddyfile:/etc/caddy/Caddyfile directly, edited it with an editor that writes a new file and renames it over the old one, and the container went on serving the original because it still held the old inode. Mounting the directory instead of the file fixes it for good.

Port 80 closed after I set up the firewall. I locked the box down with UFW, opened 443, and forgot 80. Everything looked fine until the certificate came up for renewal weeks later and silently failed the HTTP-01 challenge. Open both:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

Port 80 is not just for a redirect, Caddy needs it for the ACME challenge on renewal, so leaving it closed is a time bomb set for 60 days out.

Quick reference

Job	Command / config
Proxy to a container	`reverse_proxy servicename:INTERNALPORT`
Proxy to a host-networked app	`reverse_proxy 192.168.1.10:32400`
Shared network across projects	`docker network create web` + `external: true`
Reload config	`docker exec -w /etc/caddy caddy caddy reload`
Validate config	`docker exec -w /etc/caddy caddy caddy validate`
Staging CA while testing	`acme_ca https://acme-staging-v02.api.letsencrypt.org/directory`
Open the ACME ports	`sudo ufw allow 80/tcp && sudo ufw allow 443/tcp`
Read the logs	`docker compose logs -f caddy`

Put every container on one shared network, proxy to names and not localhost, keep port 80 open for the challenge, and test against staging so a bad afternoon does not cost you a week of certificates. Do that and Caddy turns a pile of random ports into clean hostnames with real HTTPS that renews itself while you forget it exists. That last part, forgetting it exists, is the whole reason to use it.

[ 443 up · certs on autopilot ]

Originally published at peculiarengineer.com.

Install Docker on Ubuntu 26.04 (the right way, with the docker-group truth)

Minor Keith — Wed, 22 Jul 2026 21:24:32 +0000

The wrong way to install Docker on Ubuntu is the one that looks easiest: sudo apt install docker.io. That package exists, it installs, and it runs a container. It is also whatever version happened to be frozen into the archive when 26.04 was cut, it lags the real releases by months, and it ships without the Compose and Buildx plugins you will want by the end of the week. Use Docker's own apt repository instead, and this is the post I keep open so I do not re-derive the repo setup from memory each time.

This is short on purpose. The steps are the official ones, and the only place worth slowing down is step 5, where adding yourself to the docker group quietly hands out root. That tradeoff is the part most guides skip, and it is the one thing here actually worth reading twice.

TL;DR Remove any distro docker.io/containerd packages, add Docker's GPG key and the deb822 .sources repo, then sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin. Verify with sudo docker run hello-world. Add yourself to the docker group to drop the sudo (it is root-equivalent, more below).

Prerequisites

Ubuntu 26.04 (Resolute Raccoon), server or desktop, on amd64 or arm64.
A user with sudo. If you are still on root, create a sudo user first.
Outbound HTTPS to download.docker.com.

1. Remove the distro Docker packages first

Ubuntu ships its own docker.io, docker-compose, and containerd packages, and any of them will fight the official ones over the same files and the same containerd socket. Clear them out before you add Docker's repo. This is safe on a fresh box because there is nothing to lose yet; on a box that already ran the distro Docker, it removes the packages but leaves your images and volumes in /var/lib/docker alone.

sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc | cut -f1)

The dpkg --get-selections wrapper is just so the command does not error out on packages you never had installed. If none of them are present, nothing happens, which is exactly what you want.

2. Add Docker's apt repository

Two steps: trust Docker's signing key, then point apt at their repo. Docker's current docs use the newer deb822 format (a .sources file), which is more readable than the old one-line .list entry and is what 26.04's apt prefers.

First, the key:

sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

The key lives in /etc/apt/keyrings/, not the deprecated apt-key store, and the repo file below points at it with Signed-By. That pairing is what tells apt "only trust packages from this repo if they are signed by this specific key," which is the whole reason you are not just piping a script into your shell.

Now the repo itself:

sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

The Suites: line reads your release codename out of /etc/os-release, so on 26.04 it resolves to resolute and on 24.04 it would be noble. That final apt update pulls in Docker's package list, and you are ready to install.

3. Install the engine and plugins

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Five packages, and it is worth knowing what each one is rather than pasting them as a magic incantation:

docker-ce is the daemon, the thing that actually runs containers.
docker-ce-cli is the docker command you type.
containerd.io is the lower-level runtime the daemon drives.
docker-buildx-plugin is the modern builder, so docker build uses BuildKit.
docker-compose-plugin gives you docker compose as a subcommand.

That last one is the part people miss. Compose v2 is a plugin now, so the command is docker compose up with a space, not the old standalone docker-compose with a hyphen. If you have muscle memory for the hyphenated one, this is where it stops working, and installing this plugin is how you get the replacement.

The daemon starts and enables itself on install, so there is nothing to systemctl enable. Confirm it:

systemctl is-active docker    # -> active

4. Verify it actually runs

sudo docker run hello-world

This pulls a tiny image, runs it, and prints a paragraph confirming the daemon, the runtime, and the network path all work end to end. If you see "Hello from Docker!", the install is done. If it hangs on the pull, that is a network or DNS problem reaching Docker Hub, not a broken install.

5. Run Docker without sudo (and what that really costs)

Typing sudo before every docker command gets old fast. The fix is to add yourself to the docker group, which owns the daemon's socket:

sudo groupadd docker            # usually already exists; harmless if so
sudo usermod -aG docker $USER

Group membership is read at login, so it does not apply to your current shell. Log out and back in, or start a fresh session with:

newgrp docker

Then prove it without sudo:

docker run hello-world

Here is the honest caveat, because most guides drop you in the docker group and move on. Membership in the docker group is root-equivalent. The daemon runs as root, and anyone who can talk to its socket can mount the host filesystem into a container and walk straight out as root. There is no privilege boundary between "in the docker group" and "root," full stop. On your own laptop or a single-admin homelab box, that is a fine trade for convenience. On a shared server, do not hand out docker group membership as if it were a lesser permission, because it is not. If that trade bothers you, rootless mode runs the whole daemon as your user instead, at the cost of a few limitations around ports below 1024 and some networking.

Quick reference

Job	Command
Remove distro packages	`sudo apt remove $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc \
Add GPG key	{% raw %}`sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc`
Install engine + plugins	`sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin`
Verify	`sudo docker run hello-world`
Drop the sudo	`sudo usermod -aG docker $USER` then re-login
Compose command	`docker compose up` (space, not `docker-compose`)
Check the daemon	`systemctl is-active docker`

Use Docker's repo, not the distro package, so you get real versions and the Compose and Buildx plugins. Clear out the old packages first, trust the key, and let apt pull the real thing. Then decide with open eyes whether the docker group is a trade you want, because it hands out root. With that done, the box is ready for the actual reason you installed Docker: running something. A natural next step is putting Caddy in front of those containers for automatic HTTPS.

Originally posted on Peculiar Engineer.

DEV Community: Minor Keith

Prometheus node exporter on Ubuntu 26.04, and it scrapes over the private network

Prerequisites

1. Install the exporter from the archive

2. What the collectors package adds, and what it costs

3. Bind it to the private interface and fence it with ufw

4. A metric of your own with the textfile collector

5. Prometheus from the archive, scraping it

6. Three alert rules worth having

7. The upstream binary, when the package is too old

8. Reading a down target

Gotchas I hit

Quick reference

How To - Nginx reverse proxy on Ubuntu 26.04 without Docker

Prerequisites

1. Install nginx and open the firewall

2. The site file

3. Why the default site keeps winning

4. The headers your app actually needs

5. Websockets

6. HTTPS with certbot from the archive

7. Reading the errors: 502, 413, and a reload that has not landed yet

Gotchas I hit

Quick reference

Guide to DNS on Ubuntu 26.04

The one idea worth holding onto

1. Start where the applications start

2. resolv.conf is a symlink, and there are two of them

3. Who is 127.0.0.53

4. Watch a lookup happen

5. Why dig and ping disagree

6. Changing the upstream, and what survives a reboot

7. Split DNS: a different server for one domain

8. Search domains and single labels

9. DNS over TLS, and the setting that takes the box offline

10. Three ways it gets broken

11. Debugging it

Gotchas worth knowing

Quick reference

Systemd units and timers on Ubuntu 26.04, helpful guide

The one idea worth holding onto

What is already running on your box

1. The job

2. The service unit

3. daemon-reload, and why your edit did nothing

4. The timer unit

5. Reading and writing OnCalendar

6. Why it fired late

7. The box was off at 03:30

8. Timers that count from boot, and jitter

9. When the job fails

10. Your script is not running in your shell

There is no shell

No tilde, no globs

$ means something to systemd first

% means something to systemd too

Set the environment on purpose

11. When the job takes longer than the interval

12. Testing without waiting until 03:30

13. Timers for a normal user

14. Changing one of Ubuntu's own timers

Gotchas worth knowing

Quick reference

A disposable Hetzner lab with OpenTofu: real Ubuntu box, run a script, destroy it

The shape of it

Prerequisites

1. The OpenTofu is the small part

2. One state object per lab

3. The wrapper

4. Runbooks, not shell history

5. Teardown is the actual feature

Three footguns I found in my own tool

The public IP lookup fails open

Hetzner will price a server type it will not sell you

The teardown handler has two blind spots

Gotchas worth knowing

Quick reference

Backing up a homelab with restic

Why restic

Before you start

`$` means something to systemd first

`%` means something to systemd too