DEV Community

John
John

Posted on

Should Ntfy Run on the Home Server It Is Meant to Warn You About? Where Alerts Fail and How to Avoid It

#ntfy #selfhosting #homelab #monitoring

Yes, if Ntfy is your only way of getting alerts, it should not run only on the server it watches. A power cut, an ISP outage, a kernel panic or a dead Docker daemon takes down the thing that failed and the thing that would tell you, in the same moment. Self-hosting Ntfy on your home server is still fine for app-level messages such as "backup finished" or "cron job failed", because the host is alive to send them. For "my server is down" alerts, something outside that box has to notice the silence and send the message.

TL;DR

  • Single-box homelabber (one mini PC running Jellyfin, Nextcloud and Ntfy): keep the local Ntfy for job notifications and add an external heartbeat check, because a dead host cannot report its own death.
  • Privacy-first tinkerer who refuses third-party relays: run a second Ntfy instance on a small VPS or at another site, because being independent of other services only helps if the second instance fails separately.
  • Backup-and-cron notifier who only wants "job done" messages: self-host on the same server and switch to "alert me when the message does not arrive" logic, because a missing success message is the real signal.
  • iPhone user: sending critical alerts to ntfy.sh with an access-protected topic costs you little, because a self-hosted server already hands iOS push wake-ups to the ntfy.sh upstream relay.
  • Two-site owner (home server plus a NAS at a relative's house): have each site host the Ntfy that watches the other, because two homes rarely lose power and internet at the same time.
  • Freelancer hosting client sites from home: do not rely on a home-hosted Ntfy for uptime alerts at all, because clients will notice the outage before you do.

The core tradeoff is keeping your notification pipeline fully under your control versus having an alert path that survives the exact failure you need to hear about.

Table of contents

What actually goes silent when your home server goes down?

If Ntfy runs on the box it watches, one failure can shut down three things at once: the Ntfy server, the tools that publish to it (Uptime Kuma, cron scripts, smartd, a UPS daemon) and the cache holding recent messages. How much you lose depends on how the server failed.

  • Power loss: every process stops at the same moment, so nothing gets a chance to send a last message. A UPS only helps if something like a NUT shutdown hook runs curl -d "on battery" ntfy.example.com/alerts before the battery runs out.
  • ISP or router outage: the server stays up and local publishing still works. Your phone on mobile data just can't reach Ntfy, so the messages wait until the connection is back.
  • Kernel panic or hard freeze: no userspace code runs, so no monitor on the host can notice the failure, let alone report it.
  • Docker daemon crash: the host still answers ping, but the Ntfy container and the Uptime Kuma container stop together. Running Ntfy as a systemd service from the official package instead of in Docker avoids this particular case.
  • Disk full: Ntfy's cache-file database, logs and your scripts' temp files all fail to write. You usually find out when things start behaving strangely, not from an alert.

None of this depends on the hardware. A home server, a NAS, a self-managed VPS or Yundera are each a single failure domain when they both run the monitored apps and send the alerts about them. Yundera is a managed Personal Cloud Server, built on CasaOS, that runs self-hosted apps as Docker containers on a server dedicated to the user. The rest of this article is about what to move off that one machine.

Which outages can a same-host Ntfy still report, and which can it never report?

Useful rule: a same-host Ntfy can report a failure only if the kernel, the network stack and Ntfy itself are still running. That covers more than you might expect, but not the failures that matter most.

Failure Can a same-host Ntfy report it? Why
One app container crashes Yes Uptime Kuma or a healthcheck script sees it and publishes locally
Backup or cron job exits non-zero Yes The script can call curl on its failure path before it exits
Disk usage passes 90% Yes A threshold check fires while there is still room to write
SMART warning from smartd Yes The disk is degraded, not dead yet
Disk at 100% Unreliable The cache database and the sending scripts may fail to write
Docker daemon crash Only if Ntfy runs outside Docker Containerised publishers and the containerised server stop together
ISP or router outage Late The message is published locally and reaches your phone only after the link comes back
Power loss or kernel panic Never Nothing is running to notice or publish

The pattern is clear. A same-host Ntfy is good at "something inside the server went wrong" and useless for "the server itself is gone". The first group is the everyday noise of a homelab, so local Ntfy still earns its place. The second group is the one you set up alerting for in the first place.

This does not change with the platform. A home server, a NAS, a self-managed VPS or Yundera all show the same split. When the bottom rows of the table happen, only an observer running somewhere else can tell you.

How Ntfy's message cache and missed-message recovery behave during an outage

Ntfy is built to cope with clients that disconnect. It stores messages on the server so a phone that was offline can catch up later. That design helps with some outages and does nothing for others, so it is worth knowing exactly how it works.

  • The cache is only as durable as its configuration: without cache-file set in server.yml, Ntfy keeps messages in memory only, so a container restart after a crash wipes everything that had not been delivered yet. Point it at a SQLite file on a persistent volume, such as /var/cache/ntfy/cache.db.
  • Messages expire after cache-duration: the default is 12 hours. If your ISP drops overnight and comes back 14 hours later, alerts from the start of the outage have already been deleted before your phone reconnects.
  • Clients catch up with since: the apps reconnect and ask for everything after the last message ID they saw. You can do the same by hand with curl "https://ntfy.example.com/alerts/json?poll=1&since=all" to check what the server actually holds.
  • Attachments have their own shorter clock: attachment-expiry-duration defaults to 3 hours, so a log file attached to an alert may already be gone while the text is still there.
  • Messages sent with X-Cache: no are never stored: that is handy for chatty status pings, but a phone that is offline when one arrives never gets it.
  • Failed publishes are not queued anywhere: when the server itself is unreachable, a plain curl from a remote script just fails. Unless you add --retry 5 or your own queue, the message is lost before the cache ever sees it.

The cache recovers messages that were delayed. It cannot recover messages that were never published.

Why silence is the signal: dead man's switch patterns for Ntfy alerts

A server that has lost power cannot send a message, so the alert has to come from something that notices the messages stopped. This is a dead man's switch: the server checks in on a schedule, and a watcher somewhere else alerts you when the check-ins stop. Ntfy has no built-in "alert me if no message arrives" feature, so the watcher is always a separate tool.

  • Hosted heartbeat checks: a crontab line like */5 * * * * curl -fsS -m 10 --retry 3 https://hc-ping.com/<uuid> pings Healthchecks.io every 5 minutes. When a ping is missed and the grace period runs out, Healthchecks sends an alert through its native Ntfy integration to whatever Ntfy server you point it at.
  • Uptime Kuma push monitors off-site: run Uptime Kuma somewhere other than home and add a monitor of type Push. Your server calls the push URL on a timer, and Kuma marks it down after one heartbeat interval with no call, then notifies you through its Ntfy provider.
  • Remote pull checks: the same off-site Kuma can probe a public URL or TCP port on your home server. Unlike a push heartbeat, this also catches the case where the server is fine but your ISP has dropped.
  • Grace periods sized to your maintenance: if a kernel update reboot takes 4 minutes, a 5-minute schedule with a 10-minute grace keeps routine reboots quiet and still alerts you in under a quarter of an hour.
  • Heartbeats from the end of the chain: put the ping at the end of the backup script, not in its own cron entry. That way it proves the job finished, not only that cron started.

The watcher's location decides everything. If it runs on the home server, it goes silent along with everything else.

Is ntfy.sh reliable and private enough for critical home server alerts?

The public instance at ntfy.sh runs the same open source server you would host yourself, on infrastructure that has nothing in common with your home. That alone makes it a strong candidate for the one alert that has to survive a power cut. The tradeoffs are about trust and limits, not features.

  • Topic names are the password on anonymous use: anyone who guesses homelab-alerts can read your messages or send you fake ones. A random topic such as hl-7f3k9q2xw8 makes guessing impractical. An account with access tokens, sent as Authorization: Bearer tk_..., closes the hole properly.
  • Message bodies are readable by the operator: Ntfy does not encrypt messages end to end, so whatever you send sits in plain text in ntfy.sh's cache until it expires. Keep hostnames, IP addresses and file paths out of alert text.
  • Anonymous use has rate limits: ntfy.sh caps daily messages and attachment sizes per visitor, and the paid ntfy Pro tiers raise those caps and add reserved topics. A heartbeat alert that fires only on failure stays far below any cap. A chatty Uptime Kuma pointed at it may not.
  • No uptime guarantee on the free service: the public instance is run by the project itself, and free use comes with no guarantee. For a backup alert path that is acceptable, because it only has to be up at the rare moment your home server is down.
  • The split design works best: send routine job notifications to your local Ntfy and send only "heartbeat missed" alerts from Healthchecks.io to ntfy.sh. That way ntfy.sh sees almost nothing about your systems, while the alert that matters travels outside your home.

For most homelabs, ntfy.sh is private enough for a single, carefully worded outage alert.

Is a second, off-site Ntfy instance worth the extra maintenance?

The Ntfy server is a single Go binary with modest resource needs, so compute is not the cost of a second instance. The cost is that you now run a public service, and a public service needs more care than a LAN-only container.

Task Same-host Ntfy only Adding an off-site instance
Access control Often left open on the LAN auth-default-access: deny-all, then ntfy user add, ntfy access and ntfy token add for every publisher
TLS and proxy Optional behind the home router Public HTTPS, behind-proxy: true and certificate renewal you have to keep an eye on
Upgrades One docker pull binwiederhier/ntfy Two instances to keep on the same release
Backups cache-file if you care auth-file as well, or every token has to be reissued after a rebuild
Watching the watcher Not applicable The VPS needs its own heartbeat, or it can fail silently for weeks
Phone setup One server in the app Subscriptions on two servers, with different topics and credentials
Monthly cost Electricity you already pay A small VPS fee, plus your time for patching the OS

The row that usually decides it is watching the watcher. An off-site instance that nobody checks gives you false confidence: the day your home server dies may be months after the VPS quietly ran out of disk.

A second instance makes sense in two cases. The first is when you refuse to let any third-party service carry your alerts. The second is when you already run a VPS for other things, so the patching and backups are already done. If the only reason is one "server down" message, a hosted heartbeat check sending to ntfy.sh gets you the same failure-domain separation with no new server to maintain.

Where should the Ntfy instance that watches your home server run?

Separate two roles before choosing a location. The watcher notices the silence, and the messenger delivers the alert. Both have to survive the failures they are meant to report, and each place you might put them covers a different set of failures.

  • The same home server: goes down with everything it watches. Keep it for job notifications and nothing that says "the server is down".
  • A second device on the same LAN: a Raspberry Pi running the arm64 Ntfy build outlives a kernel panic, a Docker crash or a full disk on the main box. It still dies in a power cut unless it has its own UPS, and during an ISP outage it cannot reach your phone either.
  • A NAS or Pi at a second site: a relative's house gives you separate power and a separate uplink. You depend on their router, their outages and your ability to fix things remotely.
  • A small self-managed VPS: separate power, network and hardware, at the cost of the public-service maintenance covered in the previous section.
  • ntfy.sh as the messenger: no maintenance and fully separate infrastructure, with the privacy and rate-limit tradeoffs already covered.
  • A hosted watcher with any messenger: Healthchecks.io spots the missed heartbeat on its own infrastructure, so even a same-LAN messenger only has to be reachable when it matters.

Before you pick a host, whether a self-managed VPS, a NAS at a friend's house or Yundera, ask three questions. Does it share a power circuit with the monitored machine? Does it share an internet uplink? Does it resolve its name through DNS running on that machine? One "yes" means that failure will silence your alerts again.

Does the phone side of Ntfy add its own failure points?

Getting the alert to the server is only half of the path. The other half is waking a phone that may be asleep, in a pocket, on battery saver or connected to a flaky mobile network, and each platform does that differently.

Client How it gets woken What else must be working
Android, Google Play build Firebase for ntfy.sh topics, its own always-on connection for self-hosted servers Battery optimisation exemption so Android does not kill the connection
Android, F-Droid build Always its own connection, no Firebase at all The same exemption, plus the ongoing foreground service notification
iOS app Your server sends a poll request through upstream-base-url: "https://ntfy.sh", Apple's push service wakes the app, and the app then fetches the message from your server ntfy.sh, Apple's push service and your server, all at once
Web app in a browser Web Push through the browser vendor's push service, once VAPID keys are configured The browser's push infrastructure and an active subscription
Email forwarding X-Email header, sent through the server's smtp-sender-addr A working SMTP relay and a mailbox you actually check

The iOS row matters most here. If your self-hosted Ntfy is the messenger and ntfy.sh has a problem at the same moment, iPhone alerts show up late or only when you open the app. If your own server is the thing that is down, the app has nothing to fetch even when the wake-up does arrive.

Two cheap fixes cover most of the phone-side risk. Send outage alerts with X-Priority: 5 and allow that max-priority notification channel to override Do Not Disturb in Android settings. Then add a second delivery channel, such as email, for the heartbeat alert only.

Wiring Uptime Kuma, Healthchecks and cron to Ntfy without a shared failure domain

The goal is simple to state: every alert path ends at a messenger that does not share power, network or a Docker daemon with whatever raised the alert. In practice that means deliberately choosing the destination for each tool.

  • Cron jobs with a fallback URL: wrap your publish call as curl -fsS -m 5 -d "backup failed" http://ntfy.lan/jobs || curl -fsS -m 10 -d "backup failed" https://ntfy.sh/hl-7f3k9q2xw8. When the local container is down, the second call still gets the message out.
  • Local Uptime Kuma for container health: point its Ntfy notification at your local server with a dedicated health topic and credentials. It is the right tool for "Nextcloud returns 502", and it is fine if it dies along with the host.
  • Healthchecks.io for host death: in its Ntfy integration, set an off-site server URL and a separate outage topic, with the highest priority for down events and a low priority for recovery events.
  • A remote Uptime Kuma pointed away from home: the most common mistake is an off-site Kuma whose notification still targets ntfy.yourdomain.com at home. It detects the outage perfectly and then tries to report it to a server that is down.
  • A boot announcement: a systemd oneshot unit with After=network-online.target that runs ntfy publish outage "host rebooted" tells you that a short power cut happened, even when it ended inside the heartbeat grace period.
  • One topic per severity: keeping jobs, health and outage separate lets you mute routine noise on your phone without also muting the alert that matters.

Before trusting any path, write down the full chain for each alert: the tool, the host, the messenger, the phone. Any host that appears twice in the same chain is a shared failure domain.

Top comments (0)