My coding agents kept failing in ways nothing reported. Not crashes — crashes announce themselves. The expensive failures were silent: a session looping on the same three tool calls, convinced it was busy. A process OOM-killed at 3am, taking its ability to tell me with it. Six hours of nothing, discovered at breakfast.
The platforms are closing the other half of this. Claude Code's Remote Control now pushes a phone notification when a session needs a decision, and you can answer from the couch. That covers every failure the agent can self-report. But an agent that stopped making progress has, by definition, nothing to say — and a dead one has no way to say it. Everything that can be self-reported already is. What's left needs an observer outside the process.
So I built nightsitter: a small MIT-licensed CLI that reads Claude Code's on-disk transcripts and classifies sessions by progress, not process health — 🟡 stalled (alive, mid-task, silent), 🔴 dead (gone mid-task) — and wraps anything else with a dead-man switch: nightsitter run --max-minutes 45 -- python nightly.py.
On day two I installed it as a daemon on the machine I was building it on. Which means it was watching the very Claude Code sessions that were writing it, and its false alarms landed on my own phone, in real time, about itself. Every bug below was found that way, not by testing. This post is the list, because the bugs turned out to be more interesting than the product.
Bug 1: the process name is a version number
The watcher needs to know which sessions have a live process. Transcripts don't record a PID, so I matched running processes by name: "claude" in name.lower(). Zero sessions matched. The daemon declared my entire fleet dead — including the session that was, at that moment, writing the process-matching code.
Claude Code's native installer puts the binary at ~/.local/share/claude/versions/2.1.228, and on macOS both the kernel process name and the exe basename report the bare version string: 2.1.228. Only argv[0] stays claude. You have to match on any of name / exe basename / argv[0].
Bug 2: mtime is a liar
With no PID, "is this session active" comes down to the transcript file. File mtime looks like a perfect heartbeat — the agent appends as it works.
Except Claude Code also appends bookkeeping to idle transcripts: session titles, mode records, file-history snapshots. Roughly hourly, forever. So an untouched session's mtime keeps refreshing, and my watcher read that as activity: idle session → looks ACTIVE → alert gate re-arms → ten minutes later it's "waiting" again → new phone ping. Every hour. All night. It also displayed "37m idle" for a session whose last real message was 38 hours old.
The fix: activity means the timestamp of the last conversational event (user/assistant), never mtime. And because file-history snapshots embed file contents, they can push every conversational event out of your tail-read window — so the window has to widen until it finds one, or you fall back to mtime and resurrect the bug you just fixed.
Bug 3: a notification that times out may have arrived
ntfy.sh publishes from my ISP take 20–30 seconds — right on my 30s timeout boundary. Half succeed, half "fail". Here's the ugly part: a request that times out client-side has often already been accepted server-side. So my retry logic — built to guarantee delivery — was re-sending alerts that had already reached my phone.
Final score one night: 8 real situations, 334 notifications.
Worse: each retry re-ran a blocking retry ladder (3 attempts × 30s timeout), freezing the poll loop ~100 seconds per alert. A broken notifier was blinding the watchdog — the exact failure the tool exists to prevent. The fixes, in order of hard-won: single-shot sends with exponential backoff instead of in-call ladders; retry only the channels that failed; a circuit breaker that skips a channel after 3 consecutive failures; and a give-up rule that abandons a broken channel only after another channel has actually delivered — with breaker skips not counting as attempts, because "gave up after 3 attempts" must never mean "3 attempts that were zero real sends".
Bug 4: alert history that lies about what the human knows
The deepest one. I recorded "alerted" when an alert was attempted. Everything downstream inherited that lie:
- An alert that failed to send still suppressed its own follow-up ("already said that") — for up to 4 hours, to a human who had heard nothing.
- A restart during a network outage restored the session as "already announced" and silently dropped the one alert nobody received. Reboots and network outages travel together, so this fired exactly when it mattered.
- A session that escalated (waiting → dead) mid-retry left stale history behind, and every subsequent poll looked like a fresh escalation — a guaranteed duplicate death ping 15 seconds after the real one.
The fix is one sentence: history records what was delivered, not what was attempted. An undelivered alert isn't history; it's an obligation. Once I moved the recording to delivery time, all three bugs became structurally impossible instead of individually patched.
What I actually learned
Alert discipline is the product. Detection took a day. Not-alerting took two weeks: quiet periods that double on repeats (a healthy hourly cron session should not ping you six times a day just for existing), escalation that always breaks through (waiting→dead is news; the reverse isn't), history that survives restarts. A monitor that cries wolf gets uninstalled on day two — mine nearly earned it, from me, while I was building it.
Dogfood the failure path, not the happy path. Every bug here shipped past a green test suite. They were caught because the tool ran, for real, on the machine that was building it, with its own alerts pointed at my own phone. If your product's job is handling failure, arrange for your own failures to hit it daily.
The heuristics are the moat, and they're all falsifiable. Everything above — liveness attribution without PIDs, conversational-timestamp activity, delivery-time history — is a heuristic I got wrong at least once and now hold with tests. I'd genuinely like to hear where they break for you.
pipx install nightsitter — free, MIT, two dependencies. Loop detection (agent repeating the same tool calls) is next, and I'd rather build it against real failure shapes than guesses — so: what does your silent agent failure look like? Comments welcome, issues even more so.
Top comments (0)