Deadline-bound renewal reminders: cron timezone drift, missed runs, seconds of jitter

#go #cron #scheduling #sre

The trade-off in a renewal reminder pipeline is latency against cost, and you settle it by deciding how often something wakes up. Use cron only to wake an idempotent due-window scan, never to deliver the reminder itself: a five-minute tick is 288 scans a day, a ten-second tick is 8,640, and the expensive one still doesn't protect you from a wrong timezone, a missed run after a pause and resume, or the jitter every scheduler inserts between "scheduled" and "started". Delivery timing belongs in your data. The cron expression only decides when the process opens its eyes.

Everything below follows from that one boundary.

Where a renewal deadline stops being a scheduling problem

The system I have in mind is a logistics platform: carriers upload a certificate of insurance, the certificate expires on a fixed date, and a haulage contract auto-renews unless the shipper cancels 30 days out. The product promise is a reminder at 09:00 depot-local, 72 hours before the deadline, to the account owner. Two depots, Rotterdam on Europe/Amsterdam and Chicago on America/Chicago, are enough to break a naive implementation. A reminder that arrives four hours late is mildly embarrassing; a reminder that arrives after the certificate lapses means a truck is legally not allowed to move, which is a claims problem, not a notification problem. So the real service level has two clauses, and they behave differently under load: a soft one about lateness measured in seconds, and a hard one about the business deadline, where late equals never. Once you write those two clauses down, the design question stops being "which scheduler" and becomes "which component is allowed to forget".

An occurrence is a row before it is a delivery. Reminder id, IANA zone id, civil time, resolved UTC instant, deadline, state, delivery key. That row is written by the scanner and only moved to delivered by a worker that succeeded. Every recovery story in this article — pause, resume, backfill, a queue outage, a redeploy in the middle of a scan — is easy if that row exists and impossible if the only record of what was owed is the scheduler's run history.

The scheduler is a wake-up call. It is not the ledger.

How do I stop a cron reminder from missing its run after a pause and resume?

You stop relying on the scheduler's opinion about missed runs, because that opinion is different in every implementation and none of them are wrong.

A systemd timer with Persistent=true will fire once, immediately, for an OnCalendar schedule it slept through — that is catch-up, and if the machine was off for a week you get exactly one run, not seven. A Kubernetes CronJob does the opposite kind of thing: it counts the schedules it missed, and if it finds more than 100 of them it refuses to start and logs that it cannot determine whether the job needs to run, telling you to set or lower .spec.startingDeadlineSeconds or check for clock skew. Most managed schedulers replay nothing at all; resuming a paused job restores future ticks and says nothing about the past. Three defensible behaviours, three different ways to be surprised at 09:04 on a Monday.

So the catch-up rule has to live in the scan, expressed as a bounded window rather than as trust: read from a durable cursor up to now, clamp the cursor to a maximum catch-up horizon, and apply the expiry policy per occurrence. For renewal reminders the policy is usually blunt — anything whose business deadline has already passed is expired, not delivered — because a notice telling someone to renew a certificate that lapsed yesterday is worse than silence.

Then make the scan safe to repeat. Overlapping windows are normal (a deploy, a retry, a manual re-run after an incident), so the occurrence needs a stable identity, and the uniqueness has to be enforced by something durable. Queue-level deduplication is not that thing: SQS FIFO deduplicates on a five-minute interval, which covers a burst of retries and does nothing for a catch-up scan an hour later. A unique index on the occurrence key does cover it. If the database write and the enqueue can't share a transaction, the transactional outbox pattern gives you the same guarantee with one extra hop, and it is the boring choice for a reason.

Jitter is the last piece, and the honest framing is that a scheduler promises "not before", never "at". systemd defaults AccuracySec to one minute and offers RandomizedDelaySec to spread load on purpose; hosted schedulers queue their invocations behind whatever else fired on that boundary. If your reminder needs second-level precision, don't buy it by shrinking the tick — enqueue the occurrence with the exact target instant attached and let the worker hold or delay it. The tick interval, the queue latency and the worker service time all belong in the delivery error budget as separate terms, because they scale differently and only one of them gets cheaper when you spend money.

Europe and the United States don't change clocks on the same weekend

The most common wrong-timezone bug I have seen reviewed is not a typo. It is a UTC instant computed months in advance and stored as if it were a fact.

The European Union switches on the last Sunday of March and the last Sunday of October, at 01:00 UTC. The United States switches on the second Sunday of March and the first Sunday of November. In 2026 that means Chicago moves on 8 March and Amsterdam on 29 March, so for three weeks the offset between those two depots is six hours instead of the usual seven, and again for a week between 25 October and 1 November. A due instant frozen in January lands an hour off for every depot on the wrong side of that gap. The fix is to store the civil time plus the IANA zone id and resolve to UTC as late as you reasonably can, so a tzdata update — governments do move these dates — reaches the reminder before the reminder reaches the customer.

Standard crontab has no timezone field at all. Kubernetes added .spec.timeZone to CronJob for exactly this reason; without it the schedule is interpreted in the controller's zone, which is UTC on most clusters and something else on a laptop. And while you are re-reading the expression: if you restrict both day-of-month and day-of-week, Vixie-style cron runs the job when either field matches, not both. 0 9 1 * 1 is not "the first of the month, if it is a Monday". It is every Monday, plus every first.

Two more traps that only show up in a test suite. A civil time inside a spring-forward gap does not exist, and Go's time.Date normalizes it rather than returning an error, so 02:30 quietly becomes 03:30 and the standard library documents that the choice is not guaranteed. A civil time inside a fall-back overlap happens twice. Pick reminder hours in the middle of the working day and both problems disappear; if the product insists on 02:30, decide the policy yourself in code instead of inheriting it.

Buy, build, or rent the tick

The buy-versus-build question here is narrower than it looks, because nobody sells you the part that actually hurts. The occurrence ledger, the expiry policy, the zone resolution and the delivery SLO stay yours in every row of this table.

Option	What you stop operating	What you still own	Where the cost shows up
Managed scheduler plus managed queue	Timer hosts, leader election, queue storage	Ledger, timezone policy, idempotency, catch-up window	Per-invocation and per-message billing that scales with tick frequency
Cron on a VM plus a database table	Nothing, but it is small	All of it, including the second host you add after the first single point of failure	On-call load, not the invoice
Container orchestrator CronJob	Timer hosts	Missed-start semantics, deadline configuration, worker capacity	Cluster capacity and the operational knowledge to debug it
Durable workflow engine	Retry, state machine and timer durability	Workflow versioning, worker fleet, a separate deployment story	Engineering time to model a periodic scan as a workflow
Batch orchestrator	Scheduling and dependency graphs	Per-user latency, which the tool is not built for	Latency for reminders, which is the thing you were optimising

The decision rule I would defend in a review is boring. If the tolerance is measured in minutes — and for renewal reminders anchored to a business day it almost always is — take the coarse tick, put the accuracy in the ledger, and spend the savings on worker capacity for month-end peaks, when renewals cluster on the first. If the tolerance is genuinely sub-minute, keep the coarse tick anyway and buy precision at the queue with per-message delays, because a ten-second global scan is a thirty-fold increase in query volume to fix a problem that lives in three percent of your rows. A durable workflow engine earns its keep when the reminder is a multi-step process with escalation, cancellation and compensation; for a periodic due scan it is a lot of machinery. Batch orchestrators are not suitable here at all — they optimise dependency graphs, not per-user delivery latency.

What the scan looks like, and what to measure after it

Here is the shape in Go. The scheduler calls it, it claims work, and it can be run twice in a row without sending anything twice.

package reminders

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "time"
)

// Store is the durable side of the ledger. The scheduler owns none of it.
type Store interface {
    Cursor(ctx context.Context) (time.Time, error)
    // Due over-selects on a stored estimate; dueAt below is the authority.
    Due(ctx context.Context, from, to time.Time) ([]Reminder, error)
    Claim(ctx context.Context, key string, due time.Time) (bool, error)
    Advance(ctx context.Context, to time.Time) error
}

type Queue interface {
    Enqueue(ctx context.Context, key string, notBefore time.Time) error
}

type Reminder struct {
    ID       string
    Zone     string    // IANA zone id, e.g. "Europe/Amsterdam"
    LocalDay time.Time // calendar day, civil
    Hour     int       // civil hour in Zone
    Deadline time.Time // the renewal instant this reminder has to beat
}

// dueAt resolves civil time in a named zone at scan time, so a tzdata change
// reaches the reminder instead of a UTC instant frozen months ago.
func dueAt(r Reminder) (time.Time, error) {
    loc, err := time.LoadLocation(r.Zone)
    if err != nil {
        return time.Time{}, fmt.Errorf("zone %q: %w", r.Zone, err)
    }
    d := r.LocalDay
    // A DST gap deletes some civil times; time.Date normalizes rather than
    // erroring, so keep reminder hours out of the small hours.
    return time.Date(d.Year(), d.Month(), d.Day(), r.Hour, 0, 0, 0, loc).UTC(), nil
}

func occurrenceKey(id string, due time.Time) string {
    sum := sha256.Sum256([]byte(id + "@" + due.UTC().Format(time.RFC3339)))
    return hex.EncodeToString(sum[:16])
}

// Scan claims every occurrence due in (cursor, now], clamped to maxCatchup so a
// scanner resumed after a long pause never replays a quarter of history.
func Scan(ctx context.Context, s Store, q Queue, now time.Time, maxCatchup time.Duration) error {
    cursor, err := s.Cursor(ctx)
    if err != nil {
        return err
    }
    if floor := now.Add(-maxCatchup); cursor.Before(floor) {
        cursor = floor
    }
    rs, err := s.Due(ctx, cursor, now)
    if err != nil {
        return err
    }
    for _, r := range rs {
        due, err := dueAt(r)
        if err != nil {
            return err
        }
        if !due.After(cursor) || due.After(now) {
            continue // outside this window once resolved exactly
        }
        if !now.Before(r.Deadline) {
            continue // the renewal deadline passed: expire it, don't deliver it late
        }
        key := occurrenceKey(r.ID, due)
        fresh, err := s.Claim(ctx, key, due)
        if err != nil {
            return err
        }
        if !fresh {
            continue // a previous overlapping scan already owns this one
        }
        if err := q.Enqueue(ctx, key, due); err != nil {
            return err
        }
    }
    return s.Advance(ctx, now)
}

The test that matters is the one nobody writes, and it is three lines of setup.

func TestDueAtHonoursTheAmericanDstSwitch(t *testing.T) {
    // 2026: Chicago moves on 8 March, Amsterdam not until 29 March.
    r := Reminder{
        ID:       "coi-4711",
        Zone:     "America/Chicago",
        Hour:     9,
        LocalDay: time.Date(2026, time.March, 9, 0, 0, 0, 0, time.UTC),
    }
    got, err := dueAt(r)
    if err != nil {
        t.Fatal(err)
    }
    want := time.Date(2026, time.March, 9, 14, 0, 0, 0, time.UTC)
    if !got.Equal(want) {
        t.Fatalf("got %s, want %s", got, want)
    }
}

Run the same table for Europe/Amsterdam on 9 March and you should get 08:00 UTC, six hours apart rather than seven. If your suite only ever runs on today's date, this class of bug is invisible until the customer finds it.

Now the part that turns troubleshooting into operations. A green scheduler dashboard tells you a process started; it says nothing about what was owed. Measure the distribution of delivered-minus-due in seconds, the count of occurrences currently overdue, the count expired against a business deadline, and the count suppressed as duplicates — that last one going to zero is a warning sign, because it usually means the claim step stopped running rather than that retries stopped happening. Alert on overdue and expired. Chart the lateness percentiles per zone, because a regional queue backlog shows up there first, and page on the hard clause only.

I am not sure a single tick interval is right for every reminder population, and I would not guess it from a whiteboard: measure the due-window size at your peak day, the enqueue rate, and the worker service rate, then set the cadence against the soft clause and the worker count against the hard one. Your mileage will differ if your renewals cluster on quarter boundaries instead of month boundaries. What does not differ is the boundary — the schedule wakes the scanner, the ledger owes the reminder, and the worker is the only thing allowed to say it was delivered.

DEV Community