A Pidfile Lock Is Only as Good as Its Stale Check

Every backup script eventually tells the same story. A cron job ran long, the
machine rebooted mid-run, and some morning I noticed the nightly job had quietly
stopped firing for three days. Nothing had crashed — the opposite. The lock file
the wrapper creates on startup was still sitting in /tmp, holding a PID from a
boot that no longer existed, and every subsequent run politely refused to start.

That's the trap with pidfile locking: the lock protects you from overlap, but a
dead lock protects you from everything, including the job itself. I wrote
pidlock (https://github.com/Raknaos/pidlock), a zero-dependency Python CLI that
wraps a command in a pidfile and handles this case, and the interesting part
isn't the wrapper — it's what "is this process actually alive?" really means when
you answer it once per run for a year.

The mechanism

Acquisition is a single atomic create. No lock daemon, no flock, no parsing:

def acquire(path):
    """Try to create the pidfile atomically. Returns True on success."""
    fd = None
    try:
        fd = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644)
        os.write(fd, ("%d\n" % os.getpid()).encode())
        return True
    except FileExistsError:
        return False
    except OSError as e:
        if e.errno == errno.EEXIST:
            return False
        raise
    finally:
        if fd is not None:
            os.close(fd)

O_CREAT | O_EXCL guarantees two racing processes never both succeed — the
kernel picks one, the other gets EEXIST. That half is easy. The hard half is
what you do after the EEXIST, because that's where every stale-lock bug lives.

When acquisition fails, pidlock reads the file, checks the PID, and only
removes it if it's confirmed dead — a corrupt or unreadable pidfile counts as
dead, a live one never does:

def pid_alive(pid):
    if pid <= 0:
        return False
    try:
        os.kill(pid, 0)
        return True
    except ProcessLookupError:
        return False
    except PermissionError:
        return True
    except OSError:
        return False

def acquire_with_stale_retry(path):
    if acquire(path):
        return True
    if is_stale(path):
        try:
            os.unlink(path)
        except OSError:
            pass
        if acquire(path):
            return True
    return False

Two details in there that I'd have gotten wrong the first time I needed them.

PermissionError returns True. On Linux, a signal probe on a process owned
by another user fails with EPERM, not ESRCH. pid 4231 is alive is the
correct answer even though the probe failed — deleting that pidfile would kill
another user's job's lock. The asymmetry is deliberate: never delete a lock
unless you're sure the owner is gone.

Release verifies ownership before unlinking. If your lock was evicted as
stale while your process ran (see below), and a new instance already wrote its
own PID into the same path, an unconditional os.unlink at exit deletes the
wrong lock. So release_if_ours re-reads the file and unlinks only if it still
contains its own os.getpid(). Seven lines, invisible when it works, load-bearing
when it doesn't.

What a live PID doesn't prove

Here's the part I had to state out loud once the reboot incident was explained:
os.kill(pid, 0) proves some process owns that number. It does not prove it's
the process that wrote the file.

PIDs are recycled. After a reboot, the counter restarts from the top — a fresh
rsyslogd can inherit the exact number your stale backup wrote yesterday. The
pid_alive check says "alive", the stale-lock logic says "not my problem", and
your backup stays locked out indefinitely by a process that has nothing to do
with it. The textbook fix is to store a signature alongside the PID — the
process starttime field from /proc/<pid>/stat, or a hash of cmdline — and
verify it matches before trusting liveness.

pidlock does not do this. The pidfile contains one number, full stop. That's a
decision, not an oversight, and it comes with the rest of what the tool refuses
to be:

• Advisory only. A job that never calls pidlock doesn't see the lock. The README says so in one line: only processes that use pidlock respect it.
• No flock. A kernel flock dies with its holder — no stale state is even possible across a reboot — but it doesn't play well with NFS, and flock(2) availability varies enough across platforms that the CLI stays portable. The README lists this under "What it does NOT do" rather than pretending.
• TOCTOU between the staleness check and the unlink is real and documented. Two processes can both observe a stale lock at once and both race to evict it; the O_EXCL re-acquire still lets only one win, but the losing side has already deleted a file it thought was stale. --wait N (retry acquisition for N seconds) shrinks the practical blast radius for cron-style collisions.

The PID-recycle false-positive needs a reboot plus an unlucky PID collision on
a busy-enough box to matter — rare in practice, and rarer still for short-lived
backup jobs whose stale window is hours, not weeks. On machines where the
consequence justifies the coupling, flock or a systemd PathExists= guard is
the honest recommendation, and I'd rather say that in the README than have you
discover it during an outage. Same discipline for the check against the
--check flag: it reports the state it can observe, and where the staleness
signal is ambiguous on a recycled PID it stays conservative rather than
destructive.

Trying it

Wrap any command; the exit code propagates, and signals clean up after
themselves (130 on SIGINT, 143 on SIGTERM):

pidlock --pidfile /tmp/backup.pid -- /home/me/backup.sh
pidlock --pidfile /tmp/backup.pid --wait 60 -- ./backup.sh   # wait instead of fail
pidlock --pidfile /tmp/backup.pid --check                     # 0 free, 1 held

No dependencies, Python 3.6+, 201 lines in the single pidlock.py, and
--self-test proves the semantics hermetically (live pid blocks, dead pid is
evicted and retried) without touching your real cron.

Repo: https://github.com/Raknaos/pidlock