DEV Community

Cover image for Delayed Jobs, Schedulers & Clock Skew Gotchas
Muhammad Ali Mughal
Muhammad Ali Mughal

Posted on

Delayed Jobs, Schedulers & Clock Skew Gotchas

#architecture #laravel #devops #node

"Send the reminder in 24 hours" sounds like the simplest job in the system. In practice, delayed and scheduled work is where time itself becomes a dependency: the producer's clock, the worker's clock, the database's clock, the broker's maximum delay, the server's timezone and daylight saving rules all get a vote. I have debugged trial-expiry emails that fired twice on DST night, invoices that ran on three servers at once, and delayed jobs that sat for 40 seconds past due because one worker's clock drifted. None of those were queue bugs. They were time bugs.

I wrote this with help from AI tools and checked every example and claim myself. At Loop Origin we build Laravel and Node SaaS systems where queues are part of the product surface, not an afterthought. This is Part 4 of Reliable Queues & Async Work in SaaS. Part 1 covered idempotent jobs and exactly-once effects. Part 2 covered backpressure, fairness and tenant isolation. Part 3 covered dead letter queues and safe replay. Here we make "run later" mean the same thing on every machine.

📚 Series Roadmap — Reliable Queues & Async Work in SaaS

Earlier Loop Origin series: Zero-Downtime Database Migrations in Laravel & Node.js · Production Observability on a Budget

How do you run delayed jobs and schedulers reliably across servers?

Direct answer: Store long delays as due-at rows in your database, not as multi-day broker delays. Run the scheduler on exactly one node using a distributed lock, keep every server on NTP, compute times in UTC, and convert to tenant timezones only at the edge. Make every scheduled job idempotent so a DST overlap or failover double-run is harmless.

Close-up of a stopwatch dial in a dim server room with soft bokeh lights, AI-generated

Three kinds of "later"

Teams often blur these together, and each fails differently:

Kind Example Where it should live Main failure mode
Short delay (seconds–minutes) Retry backoff, debounce a webhook Broker delay (delay(), BullMQ delay, SQS DelaySeconds) Worker clock skew, broker limits
Long delay (hours–days) Trial-ending email in 14 days Database row with due_at + sweeper Lost on deploy/flush, cancelled-but-still-fires
Recurring schedule Nightly invoices, hourly sync Scheduler (cron, Laravel scheduler, BullMQ job scheduler) Runs on N servers, DST skips/doubles

The broker is a great timer for seconds. It is a poor calendar for weeks.

Broker limits you should know

  • Amazon SQS: DelaySeconds maxes out at 900 seconds (15 minutes). Laravel's SQS driver cannot delay a job longer than that, so "dispatch in 3 days" on SQS needs another mechanism.
  • Redis-backed queues (Laravel, BullMQ): delayed jobs sit in a sorted set scored by an availability timestamp. They survive restarts only if Redis persistence is configured, and a FLUSHDB or failover without AOF silently drops them.
  • Database queues: available_at is a column, so long delays are durable, but polling cost grows with backlog.

Long delays in a broker also freeze the payload at dispatch time. If the user upgrades, cancels or changes their timezone, the queued job still carries the old state.

Pattern: due-at rows plus a sweeper

For anything beyond a few minutes, I store intent in a table and let a short-interval sweeper enqueue due work:

CREATE TABLE scheduled_actions (
  id            BIGSERIAL PRIMARY KEY,
  tenant_id     BIGINT NOT NULL,
  action        TEXT   NOT NULL,          -- 'trial_ending_email'
  subject_id    BIGINT NOT NULL,
  due_at        TIMESTAMPTZ NOT NULL,
  status        TEXT NOT NULL DEFAULT 'pending', -- pending|enqueued|done|cancelled
  idempotency_key TEXT NOT NULL UNIQUE,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX ON scheduled_actions (due_at) WHERE status = 'pending';
Enter fullscreen mode Exit fullscreen mode 
// Sweeper: runs every minute, claims due rows using the DATABASE clock
$rows = DB::transaction(function () {
    $due = DB::select("
        SELECT id FROM scheduled_actions
        WHERE status = 'pending' AND due_at <= now()
        ORDER BY due_at
        LIMIT 500
        FOR UPDATE SKIP LOCKED
    ");
    $ids = array_column($due, 'id');
    if ($ids) {
        DB::table('scheduled_actions')->whereIn('id', $ids)->update(['status' => 'enqueued']);
    }
    return $ids;
});

foreach ($rows as $id) {
    RunScheduledAction::dispatch($id); // handler re-checks state + uses idempotency_key
}
Enter fullscreen mode Exit fullscreen mode

Two details matter. due_at <= now() uses the database's clock, so app server skew does not change what is due. FOR UPDATE SKIP LOCKED lets several sweepers run without claiming the same rows. Cancelling is just UPDATE ... SET status = 'cancelled', and the handler re-reads current state before acting, so a downgraded user does not get an upgrade email.

Schedulers: run once, not once per server

The most common scheduler bug in Laravel is deploying the schedule:run cron to every web and worker node. Every node then sends the nightly invoices.

// routes/console.php (Laravel 11+)
use Illuminate\Support\Facades\Schedule;

Schedule::job(new GenerateMonthlyInvoices)
    ->monthlyOn(1, '02:00')
    ->timezone('UTC')
    ->onOneServer()          // needs a shared cache store with atomic locks (e.g. Redis)
    ->withoutOverlapping(30) // lock expires after 30 minutes
    ->onFailure(fn () => Log::critical('invoices.schedule_failed'));

Schedule::command('actions:sweep')->everyMinute()->onOneServer()->withoutOverlapping();
Enter fullscreen mode Exit fullscreen mode 
# Only on nodes allowed to schedule; onOneServer still guards duplicates
* * * * * cd /var/www/app && php artisan schedule:run >> /dev/null 2>&1
Enter fullscreen mode Exit fullscreen mode

onOneServer() requires all servers to share the same central cache (Redis, Memcached, DynamoDB or database). If one node points at a local file cache, the lock is local too and the guard silently does nothing.

In Node, BullMQ job schedulers give you the same "one definition, one run" behaviour because the schedule lives in Redis, not in each process:

import { Queue } from "bullmq";

const billing = new Queue("billing", { connection });

// Idempotent upsert: deploying 6 pods still yields one schedule
await billing.upsertJobScheduler(
  "monthly-invoices",
  { pattern: "0 2 1 * *", tz: "UTC" },
  { name: "generate-invoices", data: {}, opts: { attempts: 5 } }
);
Enter fullscreen mode Exit fullscreen mode

Avoid node-cron inside every replica for anything with side effects. It has no coordination, so N pods means N runs.

Clock skew: the bug that looks like a queue bug

Delayed job timestamps are computed by the producer and compared by the worker. If they disagree, jobs fire early or late, and lock TTLs expire at the wrong moment.

# Check sync state on every node (chrony)
chronyc tracking | grep -E "System time|Leap status"
timedatectl show -p NTPSynchronized

# Compare app server time with the database clock
php artisan tinker --execute="echo now()->toIso8601String().' vs '.DB::selectOne('select now() as t')->t;"
Enter fullscreen mode Exit fullscreen mode

Rules I apply:

  • Run chrony or systemd-timesyncd on every host and alert when offset exceeds ~100 ms.
  • Use one authority for "is it due?": the database (now()) or the broker, never a mix of app clocks.
  • Set lock TTLs with margin. A 30-second lock with 2 seconds of skew and a 29-second job is a race.
  • Containers inherit the host clock. Fix skew on the node, not in the image.

DST and tenant timezones

Run everything in UTC internally. Convert at the edge when the business rule is "9 AM in the customer's timezone".

Situation Naive local-time cron UTC + edge conversion
Spring forward (02:00–03:00 missing) 02:30 job skipped Computes next valid UTC instant
Fall back (01:00–02:00 repeats) 01:30 job runs twice Runs once at a single UTC instant
Tenant moves timezone Old schedule keeps firing Recompute due_at on change
Server TZ changed by ops Every schedule shifts No effect 
// Next "09:00 tenant local" as a UTC due_at
$dueAt = CarbonImmutable::now($tenant->timezone)
    ->addDay()->setTime(9, 0)
    ->utc();
Enter fullscreen mode Exit fullscreen mode

Even with UTC, Part 1 still applies: give each scheduled run an idempotency key such as invoice:{tenant}:{2026-12} so a failover double-run becomes a no-op.

When we audit client systems, whether through Laravel development engagements or platform reviews, duplicated schedulers and multi-day broker delays are among the first things we look for. If you are fighting one of these, get in touch.

Diagnostic checklist

  1. Are delays longer than a few minutes stored as due_at rows instead of broker delays?
  2. Do you know your broker's max delay (SQS: 15 minutes) and respect it?
  3. Does the sweeper use the database clock and SKIP LOCKED?
  4. Is schedule:run guarded by onOneServer() with a shared lock store?
  5. Are BullMQ schedules created with upsertJobScheduler, not per-pod cron?
  6. Is NTP sync monitored with an offset alert on every node?
  7. Are all internal times UTC, with tenant timezones applied only at the edge?
  8. Have you tested DST transitions for any local-time schedule?
  9. Does every scheduled handler re-check current state before acting?
  10. Does each run carry an idempotency key so double-runs are harmless?

FAQ

Can I just use dispatch()->delay(now()->addDays(14))?

On Redis or database queues it works mechanically, but the job is fragile: lost on a flush, hard to cancel, and frozen with stale data. On SQS it fails past 15 minutes. Use due-at rows.

Is onOneServer() enough to prevent double runs?

It prevents concurrent duplicates when every node shares the lock store. It does not protect against a run that finishes and is retried. Idempotency covers that.

How much clock skew is acceptable?

With NTP, offsets should be milliseconds. Alert above ~100 ms and design lock TTLs with seconds of margin.

Should the scheduler run on a dedicated node?

Often, yes. A small scheduler role simplifies reasoning, but keep onOneServer() so a failover node cannot double-fire.

What about Kubernetes CronJobs?

Fine for coarse schedules. Set concurrencyPolicy: Forbid and remember CronJobs can still occasionally run twice, so idempotency remains mandatory.

How do I see which scheduled run caused a side effect?

Correlation IDs and job traces, which is exactly what Part 5 covers.

Next in this series, Part 5 covers observability for async work: correlation IDs that survive queue hops and job traces that show where a delayed task actually spent its time.

Follow & Connect

Top comments (0)