The failures that cost me the most in three years of self-hosting were never the ones that threw an error. An error is a gift: it tells you where to look. The expensive ones are the failures that report success while being broken. A page that returns 200 OK. A healthcheck that says the container is fine. A backup that exits cleanly. A command that prints nothing wrong. Everything green, everything lying.
Here are four of them, all from the same box (a 2016 desktop, i7-6700 / 32 GB, Docker behind Caddy, reachable only over Tailscale). Each fails by handing you a success signal. Each cost me an evening the first time. The fixes are boring once you know them, the point is knowing the failure exists.
Sanitized skeleton with all the config at the end.
1. A loading page that returns 200
This one I could find nothing written about, so it cost me the most.
To keep the box quiet, I run the heavy services on-demand: Sablier stops idle containers and starts them on the first request. Caddy (with the Sablier plugin) gates a virtual host behind a container group, serves a "please wait, starting up" page while the group boots, then proxies through:
myhost.my-tailnet.ts.net:8081 {
route {
sablier http://sablier:10000 {
group office
session_duration 30m
}
reverse_proxy nextcloud:80
}
}
I gate my whole Nextcloud vhost, WebDAV included, this way. And here is the silent failure: if the gated group is not healthy, Sablier serves that HTML loading page for every request, and it serves it with 200 OK. A browser shows a spinner, fine. But my Obsidian vault syncs over WebDAV, and a WebDAV client asking for a directory listing got a 200 with a chunk of HTML instead of the XML it expected. Sync died with a cryptic no root multistatus found. Nextcloud itself was up and perfectly healthy the whole time. Every uptime check I had was green, because the gate in front kept answering 200.
The structural lesson: the moment you put a service on-demand behind a reverse proxy, that gate's health is the service's health, and the gate fails by returning success with the wrong body. Which is exactly how failure 2 kept the gate stuck.
2. A healthcheck that lies
The group above stayed unhealthy because Docker had marked Collabora unhealthy, and Docker had marked it unhealthy because its healthcheck was quietly broken, while the container served traffic perfectly.
Collabora's documented healthcheck probes its discovery endpoint with curl:
# Silently fails on collabora/code 25.04 and later
test: ["CMD", "curl", "-f", "http://localhost:9980/hosting/discovery"]
Since the 25.04 image, curl was removed to shrink it (CollaboraOnline/online#11856). The healthcheck does not complain that curl is gone, it just exits non-zero every interval, forever. So you have a container that answers every real request correctly, flagged unhealthy by a check that is testing nothing. Sablier trusts the flag, keeps the group "starting", and failure 1 does the rest.
This is not a Collabora quirk, the same missing-curl break hit Ollama (#9781), litellm, and others: minimal images drop curl, and every healthcheck copy-pasted from an old tutorial silently rots. The fix is to depend on nothing the image might not ship. /dev/tcp is a bash builtin that opens a TCP socket with no external binary:
healthcheck:
test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/localhost/9980 2>/dev/null && echo ok' || exit 1"]
interval: 5s
timeout: 3s
retries: 20
Socket opens, service is listening, healthcheck honest. Use it for anything too minimal to include an HTTP client.
3. A backup that is empty
My offsite job dumps every database, then rclone-syncs to an encrypted remote. The dumps exit 0. The files exist. The sync reports success. And two of them were empty for a while before I noticed.
Postgres is the honest one, pg_dumpall in the running container. MariaDB lied to me twice: a MariaDB container reports "started" a few seconds before it actually accepts connections. A script that starts a stopped DB and dumps immediately gets a truncated or empty .sql, mariadb-dump exits 0, and you find out the day you try to restore. Poll before dumping:
dump_mariadb() {
local container="$1" out="$2" pass="$3"
for _ in $(seq 15); do
docker exec "$container" mariadb-admin ping -h localhost --silent 2>/dev/null && break
sleep 2
done
docker exec "$container" mariadb-dump --all-databases -u root -p"$pass" > "$out"
}
SQLite (Vaultwarden) failed differently: the DB file in the volume is owned by root, so the host user cannot read it to run .backup. Instead of sudo, borrow a throwaway container that already ships sqlite3, mount the volume, run the backup there:
docker run --rm --entrypoint='/usr/bin/sqlite3' \
-v "$(realpath ./data/vaultwarden)":/data \
louislam/uptime-kuma:1 /data/db.sqlite3 ".backup '/data/db.sqlite3.bak'"
Any image with sqlite3 works. And use .backup, not cp: it gives a consistent snapshot even while the app is writing, which a file copy silently does not. The rule that saved me: a backup you have never restored is a rumor. Test-restore it, or you are trusting an exit code.
4. A packet that vanishes
A quiet box spends most of its time asleep, so I lean on Wake-on-LAN to bring machines up remotely. My first attempt: send the magic packet to the target's Tailscale IP from wherever I was. It ran, printed nothing wrong, and nothing happened. No error, no wake.
The reason is a layer confusion that WoL never warns you about: a sleeping machine has no OS running, so there is no IP stack and no Tailscale to receive anything. WoL only works as a broadcast on the local L2 segment, caught by the network card in hardware. A packet aimed at a layer-3 address the machine no longer has just evaporates, silently.
The fix is a relay through a node that stays on. SSH into it over Tailscale from anywhere, and have it send the local broadcast:
you (remote) --Tailscale SSH--> always-on node (on LAN) --broadcast--> sleeping target
# run on the always-on node, on the target's LAN
wakeonlan -i 192.168.1.255 AA:BB:CC:DD:EE:FF
Two things to enable once on the target. Arm the NIC at the OS level:
nmcli con modify <profile> 802-3-ethernet.wake-on-lan magic
# verify: ethtool <iface> shows "Wake-on: g"
And enable "Power On By PCI-E" (or "Wake on LAN") in the BIOS. This is the one people miss: it is what allows waking from a full shutdown (S5), not just from sleep (S3). Without it the card is powered down and, again, the packet vanishes with no complaint.
The pattern across all four: the system returned a success signal (200, exit 0, a printed command, a green dashboard) while being broken underneath. That is the failure class worth training your instincts on. When something in a self-hosted stack misbehaves, my first question now is not "what threw an error", it is "what is reporting success that I never actually verified end to end".
Sanitized skeleton, compose + Caddyfile + backup and WoL scripts, MIT:
github.com/mbaelde/sober-selfhost
If you have hit failure 1 with a different sync client, I would like to hear it in the comments, I suspect it is more common than the zero search results suggest.
Top comments (3)
"Everything green, everything lying" deserves to be the title of a whole ops textbook. I run internal tools for a hospital on a NAS (I'm a physical therapist, not a developer — everything built with AI), and my infrastructure taught me the same lesson from the opposite direction: the NAS's built-in WAF rewrites real error statuses into misleading ones. A legitimate 4xx JSON response from my own API gets disguised into something else entirely. So I ended up doing the perverse-sounding thing your post implies: my APIs return 200 for everything, with the real success/failure flag in the body, and every healthcheck asserts on content, never on status codes.
Your empty-backup case is the scariest of the four, because it fails at the exact moment you're relying on it having worked — you discover it during a restore, which is the worst possible classroom. After a similar scare, my rule became: a backup job isn't "done" when it exits 0, it's done when something independently verifies the artifact (size sanity, table count, a test query against the dump). Same principle as your polling loop: never trust a component's self-report about its own readiness.
Great write-up. The failures that page you at 3am are annoying; the ones that smile and wave are expensive.
Better comment than post, honestly. Thank you.
The WAF twist is the inverse of my four cases: mine were components lying about their own health, yours is infrastructure lying for a component that was telling the truth. A correct 4xx rewritten into a misleading status is worse, because even an operator who carefully asserts on status codes gets fed a fiction by a layer they can't see into.
Which is exactly why "assert on content, not status" isn't a hack: status codes are owned by whatever sits between you and the client, but the body is the one channel your own code controls end to end. (Just wire the WAF's own logs into monitoring too, so a rewritten-away 500 doesn't vanish silently.)
And your backup rule is the keeper: "done" is not exit 0, it's an independent check that the artifact exists and is plausible. Restore-time is the worst place to learn the difference.
Stealing your last line, with attribution.
"The body is the one channel your own code controls end to end" — that's a cleaner articulation than I've ever managed for a rule I arrived at by bruises. Stealing that back, so we're even.
Your WAF-logs tip is well taken, with one wrinkle from my world: this WAF is a consumer-NAS black box, logs barely included. So my workaround became monitoring from both sides of the wall — an external probe hitting the public endpoints like a real user, and an internal one asking services directly, with disagreement between the two being itself an alert. When outside says broken and inside says fine, I don't know what's wrong yet, but I know exactly which layer is lying. Crude, but it's caught every rewrite-shenanigan so far.
Enjoyed this exchange a lot — two different stacks, same epistemology: trust nothing that grades its own homework.