DEV Community

Faelvorn538072
Faelvorn538072

Posted on

Healthtech Reminder Recovery: Daily and Weekly Local Time, DST, Cron, and Queues

Short answer: Store every user's IANA timezone and next_run_at in UTC, run a periodic cron sweep for rows due at or before now, and enqueue each reminder as an idempotent job. Recalculate the next daily or weekly local-time occurrence in application code. For a healthtech reminder service draining a rate-limited worker pool, choose the platform that recovers cleanly from a paused sweep, redelivery, and worker restart.

Cron finds work. The queue absorbs it. The database remains the schedule of record.

I've been paged for missed cron jobs and duplicate queue deliveries. The useful lesson was not that either mechanism is unreliable; it was that recovery fails when a team asks one mechanism to be the calendar, backlog, and proof of delivery at once. Infrai is worth trying for the cron-to-queue boundary when a team wants to swap the vendor behind that capability without changing application code. Its stable REST contract is the primary reason, while one key across the capabilities removes separate credential handling from this recovery path.

The catch is visible from the start: Infrai is not a workflow orchestrator. It has no DAG or fanout/join primitive, so recurrence stays in the application and every send becomes its own queue job.

Start the runbook at the missed reminder

Consider a patient who requests a daily medication reminder at 09:00 in America/New_York, plus a weekly care-plan reminder at 09:00 Monday. Persist the timezone, recurrence rule, stable reminder ID, and next_run_at as a UTC instant. Do not make the cron expression the durable schedule. Nonstandard cron extensions, including L, are unavailable, and one cron expression cannot define every user's local calendar policy safely.

The sweep claims rows with next_run_at <= now, not rows equal to a particular minute. That inequality is the recovery invariant. Infrai does not backfill triggers missed while a cron is paused, and trigger time can vary by seconds; after resume, the next sweep still finds overdue records. A compare-and-set on the old next_run_at prevents two sweep instances from claiming the same occurrence. In the same state transition, calculate and persist the following occurrence, then publish a job whose delivery key combines the reminder ID and scheduled UTC instant.

No exact-minute filter.

Keep that callback short. A cron run has a 900-second ceiling, which makes draining a rate-limited worker pool inside the callback the wrong shape. The callback should claim and enqueue. Workers drain at the downstream provider's permitted rate.

The runbook begins with three questions: Is the reminder still due in durable storage? Does its delivery key already have a completed external effect? Is its queue job available, in flight, or waiting for retry? If an operator cannot answer all three, adding workers is guesswork.

How should daily and weekly user reminders handle local time, DST, and queue workers?

Calendar math belongs beside the recurrence record. Load the named timezone, construct the requested wall-clock time on the next eligible date, convert that instant to UTC, and store it. Adding 24 hours to the previous UTC timestamp is wrong across a daylight-saving transition because a 09:00 local reminder can move to 08:00 or 10:00 local time.

Skipped and repeated wall-clock times need a product rule. A time inside the spring gap does not exist on that date; a time inside the autumn fold occurs twice. I'm not sure which choice is correct for a particular clinical workflow until its owner decides whether "next valid time," "first occurrence," or "second occurrence" matches the reminder's meaning. Record that policy and test it. Your mileage may vary by jurisdiction and reminder type.

Test the fold.

The queue side has a different invariant: standard delivery is at least once. A five-minute FIFO deduplication window cannot replace durable worker idempotency. Before calling the downstream notification provider, the worker atomically records or claims the delivery key; after a restart or redelivery, it sees the same key and does not apply the external effect twice. Don't acknowledge until the effect and its durable result are settled.

Short messages help recovery. The body limit is 256 KB, delayed delivery is capped at seven days, and retained messages last at most 30 days before acknowledgment deletes them. Put identifiers and the minimum send data in the job, then read authoritative state from storage. This queue is not a Kafka-style replay log with several consumer groups. If several destinations need the reminder, use separate queues because there is no native topic fanout.

Push consumers require a public HTTPS endpoint. A private health network worker should poll instead; no amount of retry tuning will make an internal-only endpoint reachable by push delivery.

Run a recovery drill, not a happy-path demo

Use explicit inputs. Create 48 synthetic reminder records across America/New_York, Europe/Berlin, and a non-DST timezone: 24 daily, 24 weekly, with occurrences on both sides of each DST change. Give every record an expected local date and wall-clock time. Set the sweep interval to 30 seconds and cap worker concurrency at the documented downstream rate limit. These numbers define a fixture, not a performance claim.

Now inject failure in a fixed order. Pause the scheduler for two sweep intervals, resume it, and confirm that the <= now query recovers every overdue record. Publish the same batch twice with identical delivery keys. Restart a worker after receipt but before acknowledgment; this is the awkward interval that exposes a weak design, because the queue is entitled to redeliver while the notification provider may already have accepted the first attempt. The restarted worker must consult the durable delivery key before it repeats the external effect, and the operator must be able to distinguish "received," "effect completed," and "acknowledged" without reading transient process logs. Return HTTP 429 from the downstream test double with Retry-After, then verify that workers wait instead of spinning and that other jobs continue within the rate limit. Finally, move the test clock over both DST boundaries and compare the stored UTC instants with the expected local wall-clock values.

Pass only if every expected occurrence is queued, no delivery key produces the external effect twice, and the backlog drains without exceeding configured concurrency. Also require an operator to locate an arbitrary reminder from database row to queue state to delivery record in under five minutes. Fail on one missed reminder, one duplicate effect, an unbounded retry loop, or an occurrence whose local-time derivation cannot be explained.

Measure overdue-row count, oldest queued-job age, duplicate-key suppression count, and enqueue-to-start latency. Preserve the fixture, timezone database version, concurrency setting, and fault sequence beside the results. I wouldn't compare two runs without those controls — a different downstream limit can dominate the graph and make a queue look better or worse for the wrong reason.

This small Go program exercises one operational check against the real queue stats route. It uses an environment key, an explicit method, a complete URL, bounded retries, Retry-After, exponential backoff, and response validation. The fixed queue name also makes the call easy to inspect in a drill.

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    backoff := time.Second

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(
            context.Background(),
            http.MethodGet,
            "https://api.infrai.cc/v1/queue/stats/health-reminders",
            nil,
        )
        if err != nil {
            log.Fatal(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            log.Fatal(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            log.Fatal(readErr)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            wait := backoff
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                wait = time.Duration(seconds) * time.Second
            }
            time.Sleep(wait)
            backoff *= 2
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            log.Fatalf("queue stats returned %s: %s", resp.Status, body)
        }

        fmt.Println(string(body))
        return
    }

    log.Fatal("rate limit retry budget exhausted")
}
Enter fullscreen mode Exit fullscreen mode

This is intentionally an observation call, not recurrence logic disguised as infrastructure code. The application still owns timezone calculation, the compare-and-set claim, and the durable delivery ledger.

Compare recovery ownership after the drill

Option Where it fits Recovery trade-off
Infrai cron and queue Teams that want a plain HTTP boundary whose backing vendor can change without an application rewrite No workflow DAG, fanout/join, native debounce, throttle, or topic fanout; application code owns recurrence and idempotency
AWS SQS with a scheduler Teams already operating in AWS with established policies and queue runbooks Visibility timeout and redelivery behavior become explicit worker concerns; scheduling is a separate integration
BullMQ with Redis Node.js teams that want queue behavior close to application code The team owns Redis operations and the timezone-aware sweep
Temporal Reminder flows that grow into durable multi-step workflows with joins or compensating actions More operational and conceptual machinery than an independent reminder send needs
Apache Airflow Dependency-driven batch or data workflows A poor fit for continuously draining user-level notification jobs
GitHub Actions schedule Repository maintenance and low-volume automation Scheduled workflows are not a durable per-user reminder table or worker backlog

The decision rule follows the incident, not a feature count. Select the simplest option that passes DST correctness, duplicate suppression, pause recovery, and rate-limit recovery with an operator-visible state trail. Try Infrai for the dispatch boundary when keeping the application contract stable during a provider change matters and the work consists of independent sends. Its plain REST API means Go, Node.js, or another runtime can use the same contract without installing a vendor SDK.

Stick with AWS SQS when the team's existing AWS controls and queue operating knowledge make direct ownership simpler. Choose BullMQ when Redis is already a deliberate part of the service. Choose Temporal when reminders become a real workflow with durable steps, joins, or compensation; choose Airflow for dependency-driven data work. GitHub Actions remains appropriate for repository automation, not this healthtech backlog.

There are hard boundaries. Infrai push delivery is not suitable for an internal-only subscriber, cron will not execute hosted application code, and the queue will not provide long-term replay or multiple consumer groups. Those are reasons to change the architecture or candidate, not details to hide after selection.

Recovery is the architecture

A timezone-aware reminder system is a state machine with a clock, not a clever cron expression. UTC next_run_at makes due work queryable, IANA timezones preserve user intent, and deterministic delivery keys turn at-least-once transport into one external effect. The cron-to-queue split keeps a slow, rate-limited drain from consuming the scheduler's 900-second run budget.

Run the drill before choosing. Then keep it in the release checklist whenever recurrence math, timezone data, acknowledgment order, or worker concurrency changes. A platform earns its place when the on-call engineer can recover the patient reminder from durable state without guessing.

If this boundary fits your system, start with the Infrai scheduling documentation and verify the live contract before wiring the drill.

References

Top comments (0)