How I built a scheduler that can never double-fire: PID liveness, a recovery sidecar, and a takeover handshake

Ankita is my open-source desktop AI companion — a little companion, a lot more possible. It runs scheduled routines in the background: morning briefings, page watches, a quiet half-hour heartbeat. The one bug that scared me most wasn't a failed job. It was the same job succeeding twice — two scheduler owners, one on the desktop app and one from a CLI invocation, both firing the same routine, the companion sending the same Telegram message twice like it had a stutter.

Classic lock files solve this until they don't. A laptop sleeps. A process dies holding the lock. The next owner can't tell "lock held by a live process that paused for sleep" from "lock left behind by a corpse". Kill the lock aggressively and you get double owners; respect the lock blindly and your scheduler never recovers. Here's the ownership protocol I landed on — 72 lines in src/automation/scheduler-ownership.mjs — and the decision at its core that I'm still not sure about.

The lock file is a claim, not a lease

Acquiring ownership starts with the atomic part: fs.openSync(this.file, 'wx') — exclusive create. If it succeeds, the lock is yours and you write your claim: { pid, host, token, heartbeat }. That's the whole happy path. Everything interesting is the contention path, when the file already exists.

The first rule: a live PID is authoritative even when its heartbeat is stale. alive(pid) uses process.kill(pid, 0) — the zero-signal liveness check — and a process whose laptop just woke up from an hour of sleep will have a heartbeat older than LOCK_STALE_MS (2 minutes) but still be very much alive. If you steal locks on heartbeat staleness alone, you double-fire across every laptop sleep. So freshness never overrides liveness. The heartbeat exists to make dead owners detectable, not to expire live ones.

This also means the lock can never be force-stolen while its owner breathes. acquire() loops, and without --takeover it throws this instead of proceeding:

desktop scheduler owns routines — stop the desktop app or pass --takeover (cooperative shutdown required)

No --force. I refused to write one (more on why at the end).

Dead owners need an atomic recovery

When the owner is dead (!alive(pid) and heartbeat older than 2 minutes — or the lock file is malformed), a new owner has to delete the file and re-acquire. There's a cross-process gap here that took me a while to see: between "check that the owner is stale" and "unlink the lock", another process can do the same check and also unlink. Two processes both believing they recovered the lock = the exact double-owner bug we were trying to prevent. A token comparison alone doesn't close the gap either, because the check and the unlink aren't atomic.

So recovery is serialized with its own exclusive sidecar file — <lock>.recovery. A process creates it with wx (exclusive); whoever wins re-reads the lock, re-verifies staleness against the fresh read, and only then unlinks. There's even a fallback for a lock that exists but is garbage JSON: fall back to mtimeMs staleness. And the test that guards the worst interleaving is named exactly for it: 'separate processes recover a dead owner without deleting a newly acquired lock'. A second test covers the truncated-lock case and asserts heartbeat itself uses atomic file replacement (write temp + rename, via writeTextFile), because a heartbeat torn mid-write by a crash would look like a corrupt dead lock.

The cooperative takeover handshake

The trickiest case isn't dead owners — it's a live owner you actually want to replace. Say you ran ankita daemon from the CLI while the desktop app is also scheduling. Killing the desktop's scheduler process risks mid-job work; two owners double-fire. So --takeover runs a handshake:

1. The challenger writes a <lock>.takeover sidecar: { token: <owner's token>, requestedBy: <challenger> }.
2. The live owner's heartbeat() — called on its own schedule — reads the takeover file and checks the token. If the request's token matches its own (i.e., the request is addressed to it), it declines to refresh its heartbeat and returns false — a graceful "I hear you, I'm standing down".
3. The challenger polls every 50ms (up to TAKEOVER_WAIT_MS, 15 seconds) for the lock to go stale, then takes the normal dead-owner recovery path.
4. release() only deletes the lock if the caller's token matches, and also cleans up the takeover sidecar.

Nobody sends SIGTERM. The owner volunteers its exit on its own tick. The 15-second deadline exists because "please stand down" isn't a suicide pact — if the owner never sees the request, the challenger errors out instead of stealing.

What this cost, and what's still open

The whole class is injectable: now, takeoverWaitMs, host, and file are constructor options precisely so the tests can simulate stale heartbeats without sleeping 2 minutes. That's 3 of the 3 tests passing in ~1 second. The protocol treats the lock file as a single-writer claim with multi-reader observers, which is really all you can say about anything shared between unsynchronized processes.

But here's the decision I keep circling: there is no --force. A wedged owner — not dead, not sleeping, just stuck in a hung job loop — can hold the lock forever, and the only resolution is manual: kill the desktop app yourself. JOB_TIMEOUT_MS bounds each job at 3 minutes, so a truly hung job gets reaped, but a wedged owner is a human's problem. I chose "a scheduler you can't murder" as the price of "a scheduler that can never double-fire", because in my experience the failure mode users actually hit is sleep, not deadlock, and a --force flag would become the default answer to every "scheduler won't start" support thread — reintroducing the exact bug the protocol exists to kill.

Am I wrong? Would you add an escalated --force that SIGKILLs after the handshake times out, or is the absence of a kill switch the whole point? Drop a comment — I'm building this companion in the open, and the 16-year-old maintainer is still the entire review board.

Feedback welcome: akyourowngames/A.N.K.I.T.A (MIT, local-first, your models).

Comments

[Arhan Canli]

Treating a live PID as authoritative is the right default, and the sidecar closes the check-then-unlink gap. Three holes I'd test.

1. PID reuse. After a reboot or a long-dead owner, the old PID can belong to an unrelated process, so process.kill(pid, 0) says alive and the lock never recovers. The claim has host but no process start time, so a live-but-wrong PID looks like the real owner. Storing the start time (or checking the command line) fixes it.

2. process.kill(pid, 0) throws EPERM when the process exists but belongs to another user. If alive() treats any throw as dead, a root-owned CLI run against a user-owned desktop lock steals it. EPERM should count as alive and only ESRCH as dead.

3. A crash while holding .recovery leaves the sidecar behind. Does it have its own staleness rule? Otherwise one crash at the wrong moment blocks recovery for good.

On the wedged owner with no --force, I'd put a progress timestamp in the heartbeat (last job tick, not just "timer fired") and let takeover escalate to a token-checked kill after the 15 seconds, instead of a bare force flag. Is the 'separate processes' test run with real spawned processes?

[Krish Verma]

Three holes, three different verdicts — one you caught me on, one the code already handles, and one that's a real unbuilt gap.

1. PID reuse — you're right, it's a real hole. The claim written into the lock file is { pid, host, token, heartbeat } — no process start time, no cmdline. And alive() in src/automation/scheduler-ownership.mjs is literally process.kill(pid, 0), so after a reboot or a long-dead owner, a recycled PID belonging to some unrelated process reads as alive, and recovery never fires: the stale branch requires both !alive(owner.pid) and the heartbeat to be past LOCK_STALE_MS (2 minutes, from src/automation/job-policy.mjs). I store no start time today. The fix is what you said — capture the start time at claim and compare it, or gate on the command line. Adding it to the TODO list.

2. EPERM — already handled. The actual line is:

function alive(pid) { try { process.kill(pid, 0); return true; } catch (error) { return error.code === 'EPERM'; } }

EPERM counts as alive, only ESRCH (or anything else) counts as dead. So a root-owned CLI run against a user-owned desktop lock sees the owner as alive and backs off to OWNERSHIP_ERROR instead of stealing it. This one was deliberate.

3. Crashed sidecar — real gap, fair catch. The .recovery sidecar in acquire() is created with 'wx' and deleted in the finally block. A finally only runs for exits the process survives — a SIGKILL or power loss between sidecar creation and lock deletion leaves .recovery behind for good. The next attempt hits EEXIST on the sidecar, which is caught and skipped, and since the dead owner's lock file still exists, recovery never runs — every future attempt throws OWNERSHIP_ERROR. There is no staleness rule on the sidecar itself. You're right: one crash at the wrong moment blocks recovery until someone deletes the sidecar file by hand. The fix I'd take: write the holder's own PID and timestamp into the sidecar at creation, and let a later process claim the sidecar itself when its holder is dead. Not built yet.

On the wedged owner: takeover is cooperative by design, never a kill. acquire({ takeover: true }) — the --takeover CLI flag — writes a .takeover request file, polls every 50ms for up to TAKEOVER_WAIT_MS (15s), and the owner's own heartbeat() returns false when it sees a request addressed to its token, which is how the owner notices and steps down. A wedged owner (PID alive, never heartbeating) just burns the deadline and the requester throws. No token-checked kill exists; the constant's comment says it plainly: "Cooperative owner shutdown deadline; never forcibly steal a live owner." A progress timestamp in the heartbeat (last job tick, not just heartbeat: this.now()) would be the right first step toward telling wedged apart from busy — worth doing.

And yes — the "separate processes" test uses real spawned processes: spawn(process.execPath, …) × 4, each running the actual SchedulerOwnership module coordinated over IPC, with a monkeypatched unlinkSync staggering the deletes via Atomics.wait to force the check-then-unlink race, against a lock file left by a genuinely exited probe process. No mocks in that one.