A scheduler can be perfectly healthy and still run the wrong job at the wrong time.
The failure is usually not the cron expression. It is the boundary between wall-clock time, monotonic elapsed time, leases, retries, and a process that may pause or restart.
A reliable agent scheduler needs an explicit clock contract. Without one, a clock correction can make a job run twice, never run, or run after its authorization window has expired.
The three clocks an agent should not conflate
Use wall-clock time for human meaning and durable records:
- scheduled_at: when the user asked for the run
- not_before: the earliest acceptable dispatch time
- expires_at: the latest acceptable dispatch time
Use a monotonic clock for elapsed-time decisions inside one process:
- lease renewal deadlines
- backoff timers
- watchdog intervals
- drain deadlines
Use a database or provider sequence for ordering across processes:
- scheduler ownership
- fencing tokens
- attempt numbers
- reconciliation order
A monotonic timestamp cannot be compared across hosts, and a wall-clock timestamp cannot safely measure a five-minute lease if NTP steps the clock backward. Store both kinds of evidence instead of pretending one timestamp answers every question.
A small scheduling contract
Here is a deliberately boring record shape:
action: send_digest
run_id: 01J...
scheduled_at: 2026-08-19T08:00:00Z
not_before: 2026-08-19T08:00:00Z
expires_at: 2026-08-19T08:05:00Z
lease_owner: worker-7
lease_token: 1842
attempt: 1
state: READY
The important part is not the field names. It is the decision rule:
- The scheduler claims the run with a durable lease and fencing token.
- It checks wall-clock eligibility against not_before and expires_at.
- The worker checks that its lease token is still current before starting.
- The effect layer checks the token again before a side effect.
- If the outcome is ambiguous, record UNKNOWN and reconcile by the provider's idempotency key instead of blindly retrying.
That last step matters after restarts. A clean restart is not proof that the previous effect did not happen. I covered the effect-side version of this problem in restart-safe agent deduplication.
Define the clock-skew budget
A clock-skew budget is the maximum uncertainty you will tolerate between the clock used to schedule a run and the clock used to authorize dispatch.
For example:
- expected host skew: 2 seconds
- NTP alarm threshold: 10 seconds
- dispatch grace period: 30 seconds
- job freshness window: 5 minutes
Do not silently turn the budget into a larger retry window. If the observed offset exceeds the budget, stop dispatching new work or move runs to CLOCK_UNCERTAIN. Existing in-flight work needs its own lease and effect policy.
A simple gate can look like this:
action = now_wall < run.not_before
expired = now_wall >= run.expires_at
clock_uncertain = abs(host_offset) > CLOCK_SKEW_BUDGET
if clock_uncertain:
return CLOCK_UNCERTAIN
if action:
return NOT_READY
if expired:
return EXPIRED
return DISPATCHABLE
The ordering is intentional. A scheduler should not dispatch merely because a job is due if the host's clock is outside the authority's accepted uncertainty.
What changes when the clock jumps?
Test at least these cases:
| Fault | Unsafe symptom | Safer result |
|---|---|---|
| Clock jumps backward | due work appears early or leases live too long | use monotonic lease timers and hold new dispatch |
| Clock jumps forward | future work runs immediately or expires | reject dispatch outside the freshness window |
| NTP becomes unavailable | stale schedule decisions continue silently | enter CLOCK_UNCERTAIN with an alert |
| Worker pauses during a lease | two workers perform one effect | fencing token rejects the stale worker |
| Scheduler restarts at a boundary | a run is lost or duplicated | reload durable state and reconcile by run ID |
| DST or timezone conversion changes | local-time jobs shift unexpectedly | store UTC plus the original schedule zone |
A useful failure-injection test does not just mock now(). Pause a worker after it claims a lease, advance the authority clock, start a replacement worker, and then let the old worker attempt the effect. The expected result is a rejected stale token, not a second provider call.
Keep local time out of the effect boundary
Human schedules may be expressed as “every weekday at 09:00 Europe/Berlin.” Convert that schedule to an unambiguous UTC occurrence before creating the run record. Persist the timezone and the resolved occurrence together.
Do not let a browser session, email worker, or MCP tool reinterpret the local schedule. By the time work reaches an effect boundary, it should carry a concrete run ID, expiry, authority version, and idempotency key.
This also makes audits possible. When a user asks why a run happened at 08:00 UTC, you can distinguish:
- the requested local schedule
- the timezone rule used for conversion
- the resolved UTC occurrence
- the clock offset observed at dispatch
- the lease and fencing decision
- the provider's final effect result
Deployment checklist
Before running an always-on agent scheduler, verify:
- [ ] Wall-clock and monotonic time are used for different decisions.
- [ ] Every run has not_before, expires_at, a lease, and a fencing token.
- [ ] Clock offset is measured and has an explicit budget.
- [ ] New dispatch pauses when the budget is exceeded.
- [ ] Workers cannot renew or apply effects with stale fencing tokens.
- [ ] Unknown outcomes reconcile through a stable idempotency key.
- [ ] UTC occurrence and source timezone are both persisted.
- [ ] Restart, NTP step, DST, pause, and duplicate-dispatch tests are automated.
- [ ] Alerts distinguish process liveness from schedule correctness.
If you need a managed always-on runtime for an OpenClaw or browser-based agent, managed agent hosting on Ampere is one option to evaluate. Hosting can keep a process running, but it does not define the clock contract, lease semantics, effect idempotency, or recovery policy. Those still belong in the application.
The practical lesson is simple: cron tells you when to try. A clock contract tells you whether the attempt is still authorized, current, and safe.
If this kind of control-plane detail is useful, follow for practical agent reliability patterns rather than model demos.
Top comments (0)