DEV Community

mattleeee
mattleeee

Posted on Originally published at hkcode.dpdns.org

Overlap-Proof Cron: File Locks with O_EXCL and Stale-Lock Self-Healing

Windows Task Scheduler has no concept of mutual exclusion between runs. If you tell it to fire a job every 15 minutes, it will do exactly that, whether or not the previous instance is still alive. For a job that normally finishes in 10-30 seconds, this rarely matters. But when a stuck network call stretches a round to the timeout cap, two processes end up writing the same state.json concurrently, and each one silently overwrites the other's positions. The result is a corrupted state file and positions that vanish or double.

The fix is a lock file created with O_EXCL, plus a staleness rule so that a killed process can't wedge the scheduler forever. This post walks through the implementation, the failure modes, and a small test suite.

Why a lock file at all

The scheduler fires rounds on a fixed cadence. The job itself has no idea another instance is running. You have three broad options:

  1. Let the scheduler serialize. Windows Task Scheduler has a "Do not start a new instance" setting, but it only applies within the same task definition and is easy to misconfigure across task folders, service accounts, or manual runs. It also does nothing if someone triggers the script by hand.
  2. Use an in-process lock. Useless here: the two contenders are separate processes.
  3. Use an OS-level lock file. This works regardless of who started the process.

The lock file approach is the most portable and the easiest to reason about, provided you get two details right: atomic creation and stale-lock recovery.

Atomic creation with O_EXCL

The naive version of a lock is:

if not os.path.exists(lock_path):
    open(lock_path, "w").close()
    run_job()
Enter fullscreen mode Exit fullscreen mode

This is a check-then-act race. Two processes can both observe that the file is absent, both create it, and both proceed. Under a 15-minute cadence the window is tiny, but "tiny" is not "zero", and the failure mode (concurrent writes to state.json) is exactly the one you're trying to prevent.

O_EXCL closes the race at the kernel level. os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY) is a single atomic operation: it creates the file only if it does not already exist, and fails with FileExistsError otherwise. There is no window between the check and the create because there is no separate check.

import os

def try_acquire(lock_path: str, pid: int) -> bool:
    try:
        fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except FileExistsError:
        return False
    try:
        os.write(fd, str(pid).encode("ascii"))
    finally:
        os.close(fd)
    return True
Enter fullscreen mode Exit fullscreen mode

Writing the PID is not strictly required for correctness, but it is invaluable when you're staring at a stuck lock at 2 a.m. and want to know whether the owning process is still alive.

The stale-lock problem

O_EXCL gives you atomicity, but it also gives you a lock that never releases itself if the process dies. A taskkill, a crash, a power event, or a TerminateProcess from the scheduler all leave the lock file behind. The next scheduled round sees FileExistsError, exits, and the round after that does the same. The job is now permanently blocked, and nothing in the logs says why.

You need a rule for deciding when an existing lock is abandoned. The lock file's modification time is the natural arbiter. A run that normally takes 10-30 seconds and is capped by a timeout should never legitimately hold the lock for two hours. So:

  • If the lock is younger than the staleness threshold, treat it as live and skip this round.
  • If the lock is older than the threshold, treat it as abandoned, remove it, and try to acquire again.

The threshold has to be comfortably larger than the worst-case legitimate runtime. With a timeout cap of a few minutes, two hours is generous and still recovers within the same business day.

There is a subtle race in the takeover path: two processes can both decide the lock is stale, both unlink it, and both create a new one. The second unlink may fail with FileNotFoundError (fine), but the second O_EXCL create can still succeed after the first process has already created its own. To narrow this, do the takeover as a two-step: unlink, then attempt O_EXCL. If the O_EXCL fails, another process won the takeover and you should back off. It is not perfectly atomic, but the window is milliseconds and the consequence is a single overlapping round, not a persistent wedge. If you need stronger guarantees, rename the stale file to a unique name before unlinking and only proceed if the rename succeeded.

The full implementation

import os
import time
import logging

log = logging.getLogger(__name__)

LOCK_PATH = r"C:\trading\state.lock"
STALE_SECONDS = 2 * 60 * 60  # 2 hours


def _lock_age_seconds(lock_path: str) -> float:
    try:
        return time.time() - os.path.getmtime(lock_path)
    except FileNotFoundError:
        return float("inf")  # vanished; treat as stale so we retry


def acquire_lock(lock_path: str = LOCK_PATH,
                 stale_seconds: int = STALE_SECONDS) -> bool:
    """Return True if this process owns the lock, False if another run holds it."""
    pid = os.getpid()

    # Fast path: atomic exclusive create.
    try:
        fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except FileExistsError:
        pass
    else:
        try:
            os.write(fd, str(pid).encode("ascii"))
        finally:
            os.close(fd)
        return True

    # Lock exists. Is it stale?
    age = _lock_age_seconds(lock_path)
    if age < stale_seconds:
        log.warning("Lock held by another run (age %.1fs); skipping.", age)
        return False

    log.warning("Stale lock detected (age %.1fs); attempting takeover.", age)
    try:
        os.unlink(lock_path)
    except FileNotFoundError:
        pass  # someone else already removed it

    # Second attempt; may still lose the race to another process.
    try:
        fd = os.open(lock_path, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
    except FileExistsError:
        log.warning("Lost stale-lock takeover race; skipping.")
        return False
    try:
        os.write(fd, str(pid).encode("ascii"))
    finally:
        os.close(fd)
    return True


def release_lock(lock_path: str = LOCK_PATH) -> None:
    try:
        os.unlink(lock_path)
    except FileNotFoundError:
        pass
Enter fullscreen mode Exit fullscreen mode

Usage in the job entry point:

def main() -> int:
    if not acquire_lock():
        return 0  # nothing to do; next scheduled round will pick it up
    try:
        run_trading_round()
    finally:
        release_lock()
    return 0
Enter fullscreen mode Exit fullscreen mode

The try/finally matters. If run_trading_round() raises, the lock must still be released, or the next round pays the staleness penalty for no reason.

Lock failure must not block the business path

There is a design decision hiding in the return 0 above. When the lock is held, the process exits cleanly without doing work. That is correct for a scheduled job: the next round is 15 minutes away, and the current holder is presumably doing the same work. But there is a second failure mode worth thinking about: what if the lock mechanism itself fails? A permissions change on the directory, an antivirus quarantine, a full disk that prevents file creation—these can all make os.open raise something other than FileExistsError.

The rule should be: if you cannot determine lock state due to an unexpected error, do not block the business path. Log loudly, then proceed. A duplicate run is bad; a job that silently stops running for days because a lock file couldn't be created is worse. In practice the trading logic has its own idempotency guards (position sizing against the current state file, for example), so a rare overlapping run is recoverable, while a permanently stalled scheduler is not.

def acquire_lock_or_proceed(lock_path: str = LOCK_PATH) -> bool:
    """Returns True if this run should proceed. Never raises."""
    try:
        return acquire_lock(lock_path)
    except OSError:
        log.exception("Lock subsystem failed; proceeding without lock.")
        return True
Enter fullscreen mode Exit fullscreen mode

Use this wrapper at the top of main(). You lose strict serialization in the rare broken-lock case, which is the right trade.

Why not flock?

Python's fcntl.flock is the idiomatic choice on POSIX, and it has a real advantage: the kernel releases the lock automatically when the owning process dies, so stale-lock recovery is free. On Windows, fcntl does not exist, and the portable substitutes are uneven:

  • msvcrt.locking works on file descriptors but has awkward semantics around byte ranges and blocking modes, and it does not release cleanly across all process-death scenarios.
  • portalocker wraps the platform differences reasonably well, but adds a dependency and its Windows backend still relies on msvcrt or LockFileEx via ctypes.
  • Third-party file-locking libraries vary in how they behave when the process is killed by the scheduler versus terminated by the user.

O_EXCL sidesteps all of this. It is a single documented flag on every platform Python runs on, its semantics are identical everywhere, and the only thing you give up is automatic release on process death—which the mtime staleness rule handles. For a scheduled job that runs every 15 minutes, that trade is clearly worth it.

Testing the lock

Three tests cover the behavior that matters. They run against a temporary directory and don't touch the real lock path.

import os
import time
import pytest

from lock import acquire_lock, release_lock


@pytest.fixture
def lock_path(tmp_path):
    return str(tmp_path / "state.lock")


def test_second_acquire_fails(lock_path):
    assert acquire_lock(lock_path) is True
    assert acquire_lock(lock_path) is False
    release_lock(lock_path)


def test_release_then_acquire_succeeds(lock_path):
    assert acquire_lock(lock_path) is True
    release_lock(lock_path)
    assert acquire_lock(lock_path) is True
    release_lock(lock_path)


def test_stale_lock_self_heals(lock_path):
    # Simulate a crashed process: lock file exists, mtime is old.
    with open(lock_path, "w") as f:
        f.write("99999")
    old = time.time() - 3 * 60 * 60  # 3 hours
    os.utime(lock_path, (old, old))

    assert acquire_lock(lock_path, stale_seconds=2 * 60 * 60) is True
    release_lock(lock_path)
Enter fullscreen mode Exit fullscreen mode

The third test is the one that catches regressions in the takeover logic. It is easy to write a stale check that reads the file's mtime before checking existence, or that uses os.path.getmtime without handling the FileNotFoundError when another process unlinks in between. The test forces the stale path and verifies that acquisition succeeds.

A fourth test worth having, though not strictly required: assert that a fresh lock is not taken over. Set the mtime to now and confirm acquire_lock returns False. That guards against a threshold bug that turns every lock into a stale lock.

Operational notes

  • Put the lock file on the same volume as the state file. Network paths reintroduce the race you just eliminated, because O_EXCL semantics on SMB shares are not always what you expect.
  • Log the PID and age whenever you skip a round. When the job silently stops doing work, that log line is the first thing you'll look for.
  • Keep the staleness threshold in configuration, not in code, so you can tighten it if the timeout cap changes.
  • The lock file should never be read by anything except the lock logic. Treat it as opaque.

The pattern generalizes: any scheduled job that mutates shared state and can occasionally overrun its interval needs exactly this. O_EXCL gives you the atomicity, the mtime rule gives you self-healing, and the fallback-to-proceed rule keeps a broken lock subsystem from becoming a broken scheduler.

More notes like this ship every week on this site.

Top comments (0)