DEV Community

prodbymarcu
prodbymarcu

Posted on

My watchdog killed the gateway twelve seconds after start

cover

A guard that verifies by waiting has a deadline baked into it, and if that deadline is shorter than the cold start of the thing it guards, the guard becomes a demolition crew. Mine did exactly that on my Mac. An hourly helper around notification cadence kept restarting the shared messaging gateway behind my two chat bridges while everything inside the gateway was healthy. Twelve seconds after a live process declared itself connected, a SIGUSR1 arrived from nowhere, both adapters dropped off mid-handshake, and the gateway exited 75.

The timestamps came out of the log before I formed any theory about cause. This one wasn't a crash at all.

What the log actually showed

I stopped theorizing and read the incident window straight out of ~/.hermes/logs/gateway.log. Connected at 11:55:56. Then SIGUSR1 at 11:56:08.958, under fourteen seconds into a live process. Both adapters disconnected right behind the signal, exit code 75, and a new startup control socket appeared at 11:56:15.251. Platforms came fully ready at 12:00:27.

So the replacement needed about four minutes twelve seconds of cold boot before it could answer anything, the heavy stretch of it wiring MCP servers into place. Nothing inside that process was broken when it died, the kill came from outside, and the log line for it doesn't name the sender. That's what pushed me to hunt for the owner of the signal directly.

Four timestamps across the incident window

Ever set a wait window from how long the thing felt while you were typing? That's the exact trap here. My helper had one relevant number, it waited 180 seconds after requesting a restart, then verified, then exited 1 whenever verification came back short. Startup measured closer to 250 seconds. The window never fit on a cold boot, which meant every single run of that helper ended in a failure exit, and launchd relaunches failed submitted jobs.

Why a finished job kept firing

The owner hunt was mechanical. launchctl listings named the initiator directly: a submitted hourly job with the keepalive property, running /usr/bin/python3 against my notification-cadence helper. Read together with the log, the contract made the loop obvious. After two idle observations the helper invokes the official default-profile gateway restart, waits 180 seconds for the replacement, verifies, exits 1 on any verification failure. The relaunch of a failed submitted job treats already-complete work as pending again, so its startup overwrote prior terminal results and asked for another restart.

The cadence value of 3600 seconds was already saved and activated months of runs earlier, which made this job pure liability once it started firing on schedule. No ongoing helper was necessary at all after that state landed. The job wasn't maintaining anything, it existed only to re-apply a config value that hadn't changed since I wrote it, and each re-application pointed at a process that was still starting up. A guard that re-does settled work can only find trouble.

The receipt pattern that stopped it

Three changes, all small, none of them touching the running stack while it worked.

First, the helper checks a receipt file before doing anything else. A prior done or retired receipt carrying notify_seconds=3600 returns success immediately, no config write, no restart subprocess, nothing for launchd to escalate against. The exit path never asks the question it kept answering wrong.

Second, I marked the helper itself retired with the reason written into the file, then tested both paths in an isolated temporary HOME. Done and retired receipts exit 0 even without the Hermes executable present and even without any gateway state to verify against, and they preserve the receipt across runs. Invoking the live retired helper on my actual machine also exits 0 without restarting anything.

Third, one launchctl remove for the obsolete submitted job. The shared gateway and both active chat agents stayed exactly where they were.

The test rig matters as much as the guard here. If your idempotency check needs the guarded system alive to pass, you haven't written a guard, you've written a dependency, and the loop survives it. Running the done-path in a temp HOME with no executable and no daemon proves the receipt short-circuits before the first side effect, and side effects are the only place a relaunch finds trouble.

Three rules your own guard needs

  1. Measure the cold start of what you guard before you pick the wait window. Mine chose 180 seconds against about 250 measured, so the verification lost by seventy seconds flat on every boot nobody had warmed.
  2. Encode completion in state the relaunch can read. A supervisor that restarts on failure cannot tell "the work failed" apart from "the check timed out", those are different answers and receipts separate them cheaply.
  3. When the function lives in saved config, delete the automation. I kept rescheduling a helper whose entire job was re-writing a value that hadn't changed since I set it.

What shipped

The gateway finished its startup at 12:00:27 and stayed connected with the restart initiator absent afterward, the loop died together with the job that caused it. Real ping tests I sent over iMessage came back with typed answers on my phone, both bridges admitted their turns normally and every test request ran on my local model, no cloud fallback slipped in during the recovery window.

How much margin do you leave between a verification window and a measured cold start? Mine was 180 seconds picked by feel against a boot that later measured closer to 250, seventy seconds short on every cold run. Numbers decide whether a guard protects or demolishes, and I found that out only because launchd kept handing the helper back. If you've written receipts for one-shot jobs that must stay honest about finished work, show me yours in the comments.

Top comments (0)