How I built a scheduler that can never double-fire: PID liveness, a recovery sidecar, and a takeover handshake

Ankita is my open-source desktop AI companion — a little companion, a lot more possible. It runs scheduled routines in the background: morning briefings, page watches, a quiet half-hour heartbeat. The one bug that scared me most wasn't a failed job. It was the same job succeeding twice — two scheduler owners, one on the desktop app and one from a CLI invocation, both firing the same routine, the companion sending the same Telegram message twice like it had a stutter.

Classic lock files solve this until they don't. A laptop sleeps. A process dies holding the lock. The next owner can't tell "lock held by a live process that paused for sleep" from "lock left behind by a corpse". Kill the lock aggressively and you get double owners; respect the lock blindly and your scheduler never recovers. Here's the ownership protocol I landed on — 72 lines in src/automation/scheduler-ownership.mjs — and the decision at its core that I'm still not sure about.

The lock file is a claim, not a lease

Acquiring ownership starts with the atomic part: fs.openSync(this.file, 'wx') — exclusive create. If it succeeds, the lock is yours and you write your claim: { pid, host, token, heartbeat }. That's the whole happy path. Everything interesting is the contention path, when the file already exists.

The first rule: a live PID is authoritative even when its heartbeat is stale. alive(pid) uses process.kill(pid, 0) — the zero-signal liveness check — and a process whose laptop just woke up from an hour of sleep will have a heartbeat older than LOCK_STALE_MS (2 minutes) but still be very much alive. If you steal locks on heartbeat staleness alone, you double-fire across every laptop sleep. So freshness never overrides liveness. The heartbeat exists to make dead owners detectable, not to expire live ones.

This also means the lock can never be force-stolen while its owner breathes. acquire() loops, and without --takeover it throws this instead of proceeding:

desktop scheduler owns routines — stop the desktop app or pass --takeover (cooperative shutdown required)

No --force. I refused to write one (more on why at the end).

Dead owners need an atomic recovery

When the owner is dead (!alive(pid) and heartbeat older than 2 minutes — or the lock file is malformed), a new owner has to delete the file and re-acquire. There's a cross-process gap here that took me a while to see: between "check that the owner is stale" and "unlink the lock", another process can do the same check and also unlink. Two processes both believing they recovered the lock = the exact double-owner bug we were trying to prevent. A token comparison alone doesn't close the gap either, because the check and the unlink aren't atomic.

So recovery is serialized with its own exclusive sidecar file — <lock>.recovery. A process creates it with wx (exclusive); whoever wins re-reads the lock, re-verifies staleness against the fresh read, and only then unlinks. There's even a fallback for a lock that exists but is garbage JSON: fall back to mtimeMs staleness. And the test that guards the worst interleaving is named exactly for it: 'separate processes recover a dead owner without deleting a newly acquired lock'. A second test covers the truncated-lock case and asserts heartbeat itself uses atomic file replacement (write temp + rename, via writeTextFile), because a heartbeat torn mid-write by a crash would look like a corrupt dead lock.

The cooperative takeover handshake

The trickiest case isn't dead owners — it's a live owner you actually want to replace. Say you ran ankita daemon from the CLI while the desktop app is also scheduling. Killing the desktop's scheduler process risks mid-job work; two owners double-fire. So --takeover runs a handshake:

1. The challenger writes a <lock>.takeover sidecar: { token: <owner's token>, requestedBy: <challenger> }.
2. The live owner's heartbeat() — called on its own schedule — reads the takeover file and checks the token. If the request's token matches its own (i.e., the request is addressed to it), it declines to refresh its heartbeat and returns false — a graceful "I hear you, I'm standing down".
3. The challenger polls every 50ms (up to TAKEOVER_WAIT_MS, 15 seconds) for the lock to go stale, then takes the normal dead-owner recovery path.
4. release() only deletes the lock if the caller's token matches, and also cleans up the takeover sidecar.

Nobody sends SIGTERM. The owner volunteers its exit on its own tick. The 15-second deadline exists because "please stand down" isn't a suicide pact — if the owner never sees the request, the challenger errors out instead of stealing.

What this cost, and what's still open

The whole class is injectable: now, takeoverWaitMs, host, and file are constructor options precisely so the tests can simulate stale heartbeats without sleeping 2 minutes. That's 3 of the 3 tests passing in ~1 second. The protocol treats the lock file as a single-writer claim with multi-reader observers, which is really all you can say about anything shared between unsynchronized processes.

But here's the decision I keep circling: there is no --force. A wedged owner — not dead, not sleeping, just stuck in a hung job loop — can hold the lock forever, and the only resolution is manual: kill the desktop app yourself. JOB_TIMEOUT_MS bounds each job at 3 minutes, so a truly hung job gets reaped, but a wedged owner is a human's problem. I chose "a scheduler you can't murder" as the price of "a scheduler that can never double-fire", because in my experience the failure mode users actually hit is sleep, not deadlock, and a --force flag would become the default answer to every "scheduler won't start" support thread — reintroducing the exact bug the protocol exists to kill.

Am I wrong? Would you add an escalated --force that SIGKILLs after the handshake times out, or is the absence of a kill switch the whole point? Drop a comment — I'm building this companion in the open, and the 16-year-old maintainer is still the entire review board.

Feedback welcome: akyourowngames/A.N.K.I.T.A (MIT, local-first, your models).