This is a short post about a systemd timer that looked healthy and did nothing for about six hours. I'll say up front: I have not found the root cause. What follows is the symptom, how it was caught, the command that fixed it, and the check I now run. I'm writing it up because the failure is easy to miss, and the detection is the useful part.
The context: a messy power day
On 24 September my homelab lost power more than once. The NAS dropped hard at 10:13 BST: its journal simply stops, with no shutdown sequence. Both Proxmox nodes came back at 12:02 with their previous boot marked as a crash. The same thing had happened on 2 September. The container this post is about kept its last journal entry at 10:13:37 and started its current boot at 12:04:09.
The power problem has since been fixed. Everything came back, including the Cloudflare tunnel, which re-registered its connections by itself about four minutes after the NAS rebooted.
The timer
One of the things that runs in that container is a small checker for the Cloudflare tunnel. It resolves two of my public hostnames against a public resolver, and then uses curl --resolve to send the requests to the Cloudflare edge. Testing from inside my LAN would hit my local DNS and prove nothing. If the tunnel is fine, it pushes "up" to an Uptime Kuma push monitor.
The timer is as simple as they come:
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min
Persistent=true
Two minutes after boot, then every five minutes after the service last ran. (Persistent=true is in there too. systemd documents it as applying to OnCalendar= timers, so it wasn't going to help here either way.)
The symptom
Before the power loss, the journal shows a run every five minutes, each logging "tunnel ok" with a 200 from both hostnames. The last one was at 10:10:21.
After the reboot, there is exactly one line for the timer:
Sep 24 12:06:29 … systemd[1]: Started cf-tunnel-check.timer - Cloudflare tunnel check every 5 min.
After that, nothing. No service runs at all until I intervened at 16:49. The timer was enabled. systemctl status said it was active. But in systemctl list-timers --all its NEXT column was -. It had no next elapse scheduled. Its LAST was still 10:10, from the previous boot.
Every other timer in that container had a next run time. This was the only one stuck.
How it was caught
This is why that check is a push monitor.
It used to be a poll monitor, and the poll monitor was wrong in a different way. Kuma had been polling the local web UI of a cloudflared wrapper container, not the tunnel. It sat DOWN for more than 2,700 heartbeats while the tunnel worked perfectly, and a monitor that is always red gets ignored. On 13 September I changed it to a push monitor with a 600-second interval. It goes DOWN only if both hostnames return 530 or don't answer through the edge.
A push monitor has a property I didn't really appreciate until this happened. It also goes DOWN when the checker stops running. With no pushes, Kuma marks it DOWN once the interval passes. So on 24 September the "Cloudflare" monitor was DOWN for about six hours while the tunnel was fine.
On its own, that alert was misleading: the tunnel really was up. But it was right that something was broken. The thing watching the tunnel had stopped watching. A poll monitor, or a checker that only alerts on failure, would have stayed quiet, and I'd have had no tunnel monitoring without knowing it.
I found it during a pre-holiday sweep of anything that was red. I checked the tunnel through the edge (fine), noticed the checker's last log line was from before the power cut, and then looked at the timer.
The re-arm
systemctl restart cf-tunnel-check.timer
The journal shows the timer stopping and starting at 16:49:16, the service running in the same second, and a normal run every five minutes after that (16:49:17, 16:54:35, and so on), each one "tunnel ok" with both hostnames returning 200.
list-timers now shows a real NEXT, and the monitor is green again.
What I don't know
I haven't worked out why the timer came up without a next elapse. I have a question but no answer. This runs in an LXC container, where "boot" for OnBootSec= may mean the host kernel's boot rather than the container's start. The timer started more than two minutes after the host came up. But systemd's documentation says an OnBootSec= point that has already passed when the timer starts should elapse immediately, and that isn't what happened.
I haven't tested any of that, and I'd rather say so than dress up a guess as a diagnosis. If it happens again, I'll capture systemctl show for the timer before restarting it. This time I restarted first, and that destroyed the evidence.
The check I run now
After any reboot of that container:
systemctl list-timers --all
Look for any enabled timer whose NEXT is -. An enabled timer with nothing scheduled is not doing its job, whatever systemctl status says. active only means the timer unit is loaded and started. It does not mean anything is scheduled.
The broader lesson is about how the monitoring is built, not about systemd. Anything that runs on a schedule and matters should report that it ran, not only that it failed. A job that stops completely sends no failure alert, but it does stop sending success pings, and a push monitor notices that.
🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.
Top comments (0)