Why a Lease Is Not Enough to Stop Duplicate Scheduled Jobs

A cron job that matters enough to run on two machines has changed category. It is no longer a timer with a function attached. It is a distributed system.

On one machine, the scheduler starts a job and assumes it owns that execution. Add a second machine for availability, and both schedulers may observe that the same job is due. If duplicate work means sending the same cleanup query twice, that may be acceptable. If the job submits a settlement instruction or broadcasts a blockchain transaction, the second execution may be irreversible.

This is where “at most once” stops being free.

The basic lease

A common coordination mechanism is a lease stored in a shared database. The row usually contains:

• A job identifier
• The current owner
• An expiry timestamp
• A monotonically increasing fencing token

Acquisition must be atomic. Two workers cannot read an expired row and then both decide they own it. A transaction, row lock, or compare-and-swap update must select one winner.

Conceptually:

with transaction:
    lease = lock(job_id)

    if lease.expires_at <= database_time():
        lease.owner = this_worker
        lease.expires_at = database_time() + lease_duration
        lease.fence = lease.fence + 1
        acquired_fence = lease.fence
    else:
        acquisition_failed

Using the lease store’s clock avoids making acquisition depend on the wall clocks of separate workers. The owner can renew the lease while running, provided the update checks both its identity and fencing token.

This prevents normal overlap. It does not stop a stale process.

The paused-worker failure

Suppose scheduler A acquires a lease with fencing token 41. It reads the input and then pauses for longer than the lease duration. Causes include a long garbage-collection pause, CPU starvation, a frozen virtual machine, or a network partition.

The lease expires. Scheduler B acquires it with token 42 and performs the job.

Scheduler A then resumes. Its local process still has the state saying it acquired the lease. Checking its own wall clock does not repair the situation. Even if it checks the database before acting, another pause can occur between that check and the external side effect.

A lease answers, “Who may work during this time window?” It cannot, by itself, make an old worker forget what it was doing.

Longer lease durations reduce how often this race appears, but also delay recovery when a worker actually dies. Frequent renewal narrows the window but does not remove process pauses or network uncertainty. A timeout is a failure detector, not proof that the old worker has stopped.

Fencing the side effect

A fencing token turns each successful acquisition into a newer generation. Every operation sent to the protected resource includes that token.

The resource records the highest token it has accepted. Once it has seen token 42, it rejects an operation carrying token 41. Scheduler A can wake up, but it can no longer act as the current owner.

For a database-owned resource, the check can be part of the same transaction as the write:

if request.fence < resource.highest_fence:
    reject_as_stale
else:
    resource.highest_fence = request.fence
    apply_operation()

Equal tokens may need separate operation identifiers so retries from the current owner are deduplicated. Fencing distinguishes lease generations; it does not replace request-level idempotency.

The important limitation is that the resource performing the side effect must participate. Putting a token in application logs does nothing. If an external API cannot inspect a fencing token, the scheduler cannot force it to reject a stale caller.

Irreversible execution exposes the gap

Consider a scheduled job that constructs and submits a blockchain transaction. Scheduler A prepares the action and pauses. Scheduler B takes the newer lease and submits its transaction. Scheduler A resumes and submits a separately constructed valid transaction.

The old lease cannot unsend either transaction. Both may be accepted if they are distinct and valid under the relevant account rules.

Rebroadcasting identical signed bytes is a different case because the transaction identity can provide deduplication. That protection disappears when separate executions create distinct transactions, use different nonces, or represent the same business action through different requests.

A duplicated email is usually an annoyance that can be observed and handled. A duplicated asset transfer or swap changes durable state. Detection after the fact is not coordination.

What to use when the destination cannot fence

If the destination has no fencing mechanism, move deduplication to the narrowest durable boundary you control:

1. Assign a stable execution ID to each logical schedule slot, such as job-name/2026-09-02T10:00Z.
2. Store that ID under a uniqueness constraint before creating the external action.
3. Use an outbox so job-state changes and dispatch intent are committed together.
4. Pass the execution ID as an idempotency key when the destination supports one.
5. Separate retries of one logical execution from creation of a new execution.

An outbox alone still has an ambiguity window: a dispatcher can send a request and crash before recording the acknowledgement. Endpoint idempotency or a fenced single writer is needed to close that gap.

If no layer can reject duplicate requests, honest semantics are limited. Marking a job complete before the side effect can avoid duplicate attempts but may lose work after a crash. Recording completion afterward supports retries but permits duplicate execution. That is a product decision, not a database trick.

Disclosure: I build Tradevo, software where creators describe an automated crypto strategy in plain English, historically test and paper-trade its deterministic rules, submit it for human review, set a free or monthly creator price, and publish approved strategies, while other users can discover, subscribe to, allocate USDC to, and run published strategies from a wallet they control.

The practical rule

Use leases to select an active worker. Use fencing tokens to reject workers from expired lease generations. Use stable operation IDs to deduplicate retries. For irreversible side effects, verify that the final boundary enforces one of those mechanisms.

Without that final enforcement point, “we have a distributed lock” describes an intention, not at-most-once execution.