DEV Community

Zira
Zira

Posted on

Your AI Agent Scheduler Needs a Clock-Skew Budget, Not Just Cron

A scheduler can be perfectly healthy and still run the wrong job at the wrong time.

The failure is usually not the cron expression. It is the boundary between wall-clock time, monotonic elapsed time, leases, retries, and a process that may pause or restart.

A reliable agent scheduler needs an explicit clock contract. Without one, a clock correction can make a job run twice, never run, or run after its authorization window has expired.

The three clocks an agent should not conflate

Use wall-clock time for human meaning and durable records:

  • scheduled_at: when the user asked for the run
  • not_before: the earliest acceptable dispatch time
  • expires_at: the latest acceptable dispatch time

Use a monotonic clock for elapsed-time decisions inside one process:

  • lease renewal deadlines
  • backoff timers
  • watchdog intervals
  • drain deadlines

Use a database or provider sequence for ordering across processes:

  • scheduler ownership
  • fencing tokens
  • attempt numbers
  • reconciliation order

A monotonic timestamp cannot be compared across hosts, and a wall-clock timestamp cannot safely measure a five-minute lease if NTP steps the clock backward. Store both kinds of evidence instead of pretending one timestamp answers every question.

A small scheduling contract

Here is a deliberately boring record shape:

action: send_digest
run_id: 01J...
scheduled_at: 2026-08-19T08:00:00Z
not_before: 2026-08-19T08:00:00Z
expires_at: 2026-08-19T08:05:00Z
lease_owner: worker-7
lease_token: 1842
attempt: 1
state: READY
Enter fullscreen mode Exit fullscreen mode

The important part is not the field names. It is the decision rule:

  1. The scheduler claims the run with a durable lease and fencing token.
  2. It checks wall-clock eligibility against not_before and expires_at.
  3. The worker checks that its lease token is still current before starting.
  4. The effect layer checks the token again before a side effect.
  5. If the outcome is ambiguous, record UNKNOWN and reconcile by the provider's idempotency key instead of blindly retrying.

That last step matters after restarts. A clean restart is not proof that the previous effect did not happen. I covered the effect-side version of this problem in restart-safe agent deduplication.

Define the clock-skew budget

A clock-skew budget is the maximum uncertainty you will tolerate between the clock used to schedule a run and the clock used to authorize dispatch.

For example:

  • expected host skew: 2 seconds
  • NTP alarm threshold: 10 seconds
  • dispatch grace period: 30 seconds
  • job freshness window: 5 minutes

Do not silently turn the budget into a larger retry window. If the observed offset exceeds the budget, stop dispatching new work or move runs to CLOCK_UNCERTAIN. Existing in-flight work needs its own lease and effect policy.

A simple gate can look like this:

action = now_wall < run.not_before
expired = now_wall >= run.expires_at
clock_uncertain = abs(host_offset) > CLOCK_SKEW_BUDGET

if clock_uncertain:
    return CLOCK_UNCERTAIN
if action:
    return NOT_READY
if expired:
    return EXPIRED
return DISPATCHABLE
Enter fullscreen mode Exit fullscreen mode

The ordering is intentional. A scheduler should not dispatch merely because a job is due if the host's clock is outside the authority's accepted uncertainty.

What changes when the clock jumps?

Test at least these cases:

Fault Unsafe symptom Safer result
Clock jumps backward due work appears early or leases live too long use monotonic lease timers and hold new dispatch
Clock jumps forward future work runs immediately or expires reject dispatch outside the freshness window
NTP becomes unavailable stale schedule decisions continue silently enter CLOCK_UNCERTAIN with an alert
Worker pauses during a lease two workers perform one effect fencing token rejects the stale worker
Scheduler restarts at a boundary a run is lost or duplicated reload durable state and reconcile by run ID
DST or timezone conversion changes local-time jobs shift unexpectedly store UTC plus the original schedule zone

A useful failure-injection test does not just mock now(). Pause a worker after it claims a lease, advance the authority clock, start a replacement worker, and then let the old worker attempt the effect. The expected result is a rejected stale token, not a second provider call.

Keep local time out of the effect boundary

Human schedules may be expressed as “every weekday at 09:00 Europe/Berlin.” Convert that schedule to an unambiguous UTC occurrence before creating the run record. Persist the timezone and the resolved occurrence together.

Do not let a browser session, email worker, or MCP tool reinterpret the local schedule. By the time work reaches an effect boundary, it should carry a concrete run ID, expiry, authority version, and idempotency key.

This also makes audits possible. When a user asks why a run happened at 08:00 UTC, you can distinguish:

  • the requested local schedule
  • the timezone rule used for conversion
  • the resolved UTC occurrence
  • the clock offset observed at dispatch
  • the lease and fencing decision
  • the provider's final effect result

Deployment checklist

Before running an always-on agent scheduler, verify:

  • [ ] Wall-clock and monotonic time are used for different decisions.
  • [ ] Every run has not_before, expires_at, a lease, and a fencing token.
  • [ ] Clock offset is measured and has an explicit budget.
  • [ ] New dispatch pauses when the budget is exceeded.
  • [ ] Workers cannot renew or apply effects with stale fencing tokens.
  • [ ] Unknown outcomes reconcile through a stable idempotency key.
  • [ ] UTC occurrence and source timezone are both persisted.
  • [ ] Restart, NTP step, DST, pause, and duplicate-dispatch tests are automated.
  • [ ] Alerts distinguish process liveness from schedule correctness.

If you need a managed always-on runtime for an OpenClaw or browser-based agent, managed agent hosting on Ampere is one option to evaluate. Hosting can keep a process running, but it does not define the clock contract, lease semantics, effect idempotency, or recovery policy. Those still belong in the application.

The practical lesson is simple: cron tells you when to try. A clock contract tells you whether the attempt is still authorized, current, and safe.

If this kind of control-plane detail is useful, follow for practical agent reliability patterns rather than model demos.

Top comments (0)