We built a small posting scheduler: a YAML queue of things to publish, each with a time, and a runner that reads the queue and sends whatever is due. The runner keeps its own record of what it has already sent, so running it twice in a minute posts once. It has tests. It is about two hundred lines and none of them are clever.
Then we wrote the launchd job that runs it into the README, as a fenced block, with the launchctl command underneath. And moved on.
Six days later somebody noticed the posts were not happening.
The queue looked perfect
This is the part worth sitting with. Here is what the status command printed the morning we found out:
fb-2026-09-02-smoke-testing facebook 2026-09-02 09:12 queued
x-2026-09-03-cors x 2026-09-03 15:07 queued
fb-2026-09-04-staging facebook 2026-09-04 09:12 queued
x-2026-09-04-connection-errors x 2026-09-04 15:07 queued
Eleven entries, all valid, all with times in the past, all saying queued. Nothing is wrong with any of them. queued is exactly what the state should say for a post that has not gone out yet, and it is also exactly what it says when nothing has ever read the file.
Those two situations are indistinguishable from inside the queue. A runner that has never executed and a runner with nothing to do produce identical output, because the evidence of the runner working is not stored anywhere the queue can see.
The plist was never installed. launchctl list | grep session-replay returned nothing, and had returned nothing since the day it was written. A documented snippet in a README reads exactly like a thing that is set up.
The check we would have run could not have caught it
We had a health check in mind, and it was the wrong one: is the queue healthy? Valid YAML, no duplicate ids, no entries pointing at articles that do not exist, no two entries at the same timestamp. All of that passed. All of it would have passed every day for six days.
The queue was healthy. The queue was never the problem. The missing piece was one level up, in a place none of the queue's own diagnostics can reach, and no amount of validating the queue harder would have found it.
The generalisable version: if the only evidence your scheduled job is running is the absence of a complaint, you have no evidence. Silence is produced by working, by having nothing to do, and by not existing, and you cannot tell which one you have.
Two more things that were quietly wrong
Once it was installed, the schedule itself had a bug we had written into the README months before anybody could hit it. It ran hourly, on minute 7. A post timed for 09:12 would not go out until 10:07 - fifty-five minutes late, silently, forever.
Worse, the X entries were written for 15:07, the same minute the runner fired. Whether the afternoon post went out on time or an hour late came down to which of the two won a race measured in milliseconds. It would have looked flaky and been almost impossible to reproduce deliberately.
Both fixed by running every ten minutes on offsets nothing else uses: 2, 12, 22, 32, 42, 52.
And the log was going to /tmp, which macOS clears on reboot. So the one artefact that would have explained a failure was configured to delete itself.
What actually fixes this
Not a better queue check. A positive trace, written by the thing you care about, checked somewhere the thing itself cannot influence.
Ours now logs every run, including the boring ones, to a path that survives a reboot:
nothing due (20 in /Users/.../social/queue.yml)
nothing due (20 in /Users/.../social/queue.yml)
That is not noise. That is the runner saying I exist and I looked, which is the exact fact the queue could not tell us. Forty of those lines and one posted entry is a healthy system. An empty file is a dead one, no matter how good the queue looks.
One last trap while wiring this up, which cost twenty minutes: the first run after loading a launchd job produces nothing for a couple of minutes. launchd spawns through xpcproxy, macOS scans the binary on first execution, and Ruby holds a redirected stdout in a buffer until the process exits. An empty log thirty seconds in reads exactly like a failure and is not one. launchctl list <label> shows the PID while it is up, and LastExitStatus once it is not. Check that, not the log, in the first two minutes.
Top comments (1)
Logging the boring runs is what makes an empty file mean something, since without those lines a dead runner and a quiet week look identical. It also has a limit worth knowing. Ours ran every morning, logged that it ran, and collected zero rows for three days, because the machine's wifi is off overnight.
systemd, the heartbeat and the dashboards were all green, because all three answer "did it run", and it did. What closed it was making each run record how many rows it wrote and treating a completed run with zero as a failure, so the trace asserts the collection rather than the execution. Is anything watching your log from outside, or is reading it still manual?