DEV Community

Cover image for 4 self-hosting failures that return success
Maxime Baelde
Maxime Baelde

Posted on

4 self-hosting failures that return success

#docker #selfhosted #homelab #devops

The failures that cost me the most in three years of self-hosting were never the ones that threw an error. An error is a gift: it tells you where to look. The expensive ones are the failures that report success while being broken. A page that returns 200 OK. A healthcheck that says the container is fine. A backup that exits cleanly. A command that prints nothing wrong. Everything green, everything lying.

Here are four of them, all from the same box (a 2016 desktop, i7-6700 / 32 GB, Docker behind Caddy, reachable only over Tailscale). Each fails by handing you a success signal. Each cost me an evening the first time. The fixes are boring once you know them, the point is knowing the failure exists.

Sanitized skeleton with all the config at the end.

1. A loading page that returns 200

This one I could find nothing written about, so it cost me the most.

To keep the box quiet, I run the heavy services on-demand: Sablier stops idle containers and starts them on the first request. Caddy (with the Sablier plugin) gates a virtual host behind a container group, serves a "please wait, starting up" page while the group boots, then proxies through:

myhost.my-tailnet.ts.net:8081 {
    route {
        sablier http://sablier:10000 {
            group office
            session_duration 30m
        }
        reverse_proxy nextcloud:80
    }
}
Enter fullscreen mode Exit fullscreen mode

I gate my whole Nextcloud vhost, WebDAV included, this way. And here is the silent failure: if the gated group is not healthy, Sablier serves that HTML loading page for every request, and it serves it with 200 OK. A browser shows a spinner, fine. But my Obsidian vault syncs over WebDAV, and a WebDAV client asking for a directory listing got a 200 with a chunk of HTML instead of the XML it expected. Sync died with a cryptic no root multistatus found. Nextcloud itself was up and perfectly healthy the whole time. Every uptime check I had was green, because the gate in front kept answering 200.

The structural lesson: the moment you put a service on-demand behind a reverse proxy, that gate's health is the service's health, and the gate fails by returning success with the wrong body. Which is exactly how failure 2 kept the gate stuck.

2. A healthcheck that lies

The group above stayed unhealthy because Docker had marked Collabora unhealthy, and Docker had marked it unhealthy because its healthcheck was quietly broken, while the container served traffic perfectly.

Collabora's documented healthcheck probes its discovery endpoint with curl:

# Silently fails on collabora/code 25.04 and later
test: ["CMD", "curl", "-f", "http://localhost:9980/hosting/discovery"]
Enter fullscreen mode Exit fullscreen mode

Since the 25.04 image, curl was removed to shrink it (CollaboraOnline/online#11856). The healthcheck does not complain that curl is gone, it just exits non-zero every interval, forever. So you have a container that answers every real request correctly, flagged unhealthy by a check that is testing nothing. Sablier trusts the flag, keeps the group "starting", and failure 1 does the rest.

This is not a Collabora quirk, the same missing-curl break hit Ollama (#9781), litellm, and others: minimal images drop curl, and every healthcheck copy-pasted from an old tutorial silently rots. The fix is to depend on nothing the image might not ship. /dev/tcp is a bash builtin that opens a TCP socket with no external binary:

healthcheck:
  test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/localhost/9980 2>/dev/null && echo ok' || exit 1"]
  interval: 5s
  timeout: 3s
  retries: 20
Enter fullscreen mode Exit fullscreen mode

Socket opens, service is listening, healthcheck honest. Use it for anything too minimal to include an HTTP client.

3. A backup that is empty

My offsite job dumps every database, then rclone-syncs to an encrypted remote. The dumps exit 0. The files exist. The sync reports success. And two of them were empty for a while before I noticed.

Postgres is the honest one, pg_dumpall in the running container. MariaDB lied to me twice: a MariaDB container reports "started" a few seconds before it actually accepts connections. A script that starts a stopped DB and dumps immediately gets a truncated or empty .sql, mariadb-dump exits 0, and you find out the day you try to restore. Poll before dumping:

dump_mariadb() {
    local container="$1" out="$2" pass="$3"
    for _ in $(seq 15); do
        docker exec "$container" mariadb-admin ping -h localhost --silent 2>/dev/null && break
        sleep 2
    done
    docker exec "$container" mariadb-dump --all-databases -u root -p"$pass" > "$out"
}
Enter fullscreen mode Exit fullscreen mode

SQLite (Vaultwarden) failed differently: the DB file in the volume is owned by root, so the host user cannot read it to run .backup. Instead of sudo, borrow a throwaway container that already ships sqlite3, mount the volume, run the backup there:

docker run --rm --entrypoint='/usr/bin/sqlite3' \
    -v "$(realpath ./data/vaultwarden)":/data \
    louislam/uptime-kuma:1 /data/db.sqlite3 ".backup '/data/db.sqlite3.bak'"
Enter fullscreen mode Exit fullscreen mode

Any image with sqlite3 works. And use .backup, not cp: it gives a consistent snapshot even while the app is writing, which a file copy silently does not. The rule that saved me: a backup you have never restored is a rumor. Test-restore it, or you are trusting an exit code.

4. A packet that vanishes

A quiet box spends most of its time asleep, so I lean on Wake-on-LAN to bring machines up remotely. My first attempt: send the magic packet to the target's Tailscale IP from wherever I was. It ran, printed nothing wrong, and nothing happened. No error, no wake.

The reason is a layer confusion that WoL never warns you about: a sleeping machine has no OS running, so there is no IP stack and no Tailscale to receive anything. WoL only works as a broadcast on the local L2 segment, caught by the network card in hardware. A packet aimed at a layer-3 address the machine no longer has just evaporates, silently.

The fix is a relay through a node that stays on. SSH into it over Tailscale from anywhere, and have it send the local broadcast:

you (remote) --Tailscale SSH--> always-on node (on LAN) --broadcast--> sleeping target
Enter fullscreen mode Exit fullscreen mode 
# run on the always-on node, on the target's LAN
wakeonlan -i 192.168.1.255 AA:BB:CC:DD:EE:FF
Enter fullscreen mode Exit fullscreen mode

Two things to enable once on the target. Arm the NIC at the OS level:

nmcli con modify <profile> 802-3-ethernet.wake-on-lan magic
# verify: ethtool <iface> shows "Wake-on: g"
Enter fullscreen mode Exit fullscreen mode

And enable "Power On By PCI-E" (or "Wake on LAN") in the BIOS. This is the one people miss: it is what allows waking from a full shutdown (S5), not just from sleep (S3). Without it the card is powered down and, again, the packet vanishes with no complaint.

The pattern across all four: the system returned a success signal (200, exit 0, a printed command, a green dashboard) while being broken underneath. That is the failure class worth training your instincts on. When something in a self-hosted stack misbehaves, my first question now is not "what threw an error", it is "what is reporting success that I never actually verified end to end".

Sanitized skeleton, compose + Caddyfile + backup and WoL scripts, MIT:

github.com/mbaelde/sober-selfhost

If you have hit failure 1 with a different sync client, I would like to hear it in the comments, I suspect it is more common than the zero search results suggest.

Top comments (0)