DEV Community

KenjiTanaka6849
KenjiTanaka6849

Posted on

Healthtech Email Scheduling: Cloud Cron, Retries, and Idempotency

The practical answer is to use a cloud cron job as a daily trigger, then make the email-generation endpoint idempotent before adding retries. For a healthtech report, the scheduler should request work; it should not be responsible for doing all of the work inside one timed HTTP call. A Node.js service can generate the report and send it with a library such as Nodemailer, while a queue and worker absorb slow or repeated delivery attempts.

That division matters because the page usually arrives after the customer has already noticed the problem: yesterday's report is missing, or the same report appears twice. The first instinct is to inspect the mail library. The more useful question is which signal fired too late.

What does the on-call page actually tell you?

Imagine the daily report cron ran at 06:00. At 06:12, a monitor pages the on-call because the expected delivery record is absent. The on-call sees a failed HTTP attempt, retries it manually, and then finds that the first request was accepted but the worker had not finished. Two reports are now eligible for delivery.

This is a scheduling incident, but the visible symptom is an email incident. A cron history can tell you whether the trigger ran. It cannot, by itself, prove that report generation, delivery, and acknowledgement each completed once. Those are separate state transitions and need separate records.

The useful first check is run history: did the trigger happen, what status did the endpoint return, and how much output was retained for diagnosis? The next check is the application record keyed by the reporting period and customer. If that key already has a completed delivery, a retry must become a no-op. If it has an in-progress attempt, the worker needs a lease or an equivalent claim mechanism rather than a second send.

I initially treated the cron response as the success signal. That is too coarse. The response only says that the endpoint handled the trigger request; it does not turn an at-least-once queue into an exactly-once email system.

Where should the reliability boundary sit?

Keep the scheduled endpoint small. It should validate the trigger, create or enqueue a work item, and return a meaningful status. The worker should load the report inputs, render the message, call the email provider through the Node.js application, and record the result using a deterministic key such as customer_id:report_period.

For a fixed daily trigger, cron is a good fit. It is the calendar, not the workflow engine. The boundary becomes important when the work might exceed 900 seconds: the cron task has a single-execution limit of 900 seconds, so a long report must use “cron triggers queue plus worker consumes queue.” A queue also gives a failed delivery somewhere to wait without holding an HTTP request open.

The endpoint has to be publicly reachable because the cron task calls a public http_url; this is not a hosted code-execution environment. A private service can still do the work, but it needs a public HTTPS entry point that authenticates the trigger and forwards the work internally. A push subscription has the same public HTTPS requirement.

Here is the important part of the worker contract. The example uses an in-memory map only to keep the mechanism visible; production code needs a durable store with an atomic insert or compare-and-set operation.

package main

import (
    "fmt"
    "sync"
)

type DeliveryStore struct {
    mu      sync.Mutex
    claimed map[string]string
}

func (s *DeliveryStore) Claim(key, messageID string) bool {
    s.mu.Lock()
    defer s.mu.Unlock()

    if _, exists := s.claimed[key]; exists {
        return false
    }
    s.claimed[key] = messageID
    return true
}

func main() {
    store := &DeliveryStore{claimed: make(map[string]string)}
    key := "customer-42:2026-10-26"

    if !store.Claim(key, "queue-message-1") {
        fmt.Println("already claimed; acknowledge without sending")
        return
    }
    fmt.Println("render report, send email, and persist completion")
}
Enter fullscreen mode Exit fullscreen mode

The claim must be durable and tied to the actual report period. A random request ID is useful for tracing, but it is a poor idempotency key: every retry would look new. Standard queues are at-least-once, and FIFO deduplication lasts only 5 minutes. Neither property removes the consumer's idempotency obligation.

That is the trade-off.

This small client shows the scheduler boundary rather than hiding it behind an SDK. The request body is supplied through INFRAI_CRON_JSON so the payload can match the current discovered schema; the application still owns the stable idempotency key and the worker-side delivery record. It also refuses to wait longer than the platform's 900-second cron limit.

The calendar is not the delivery system.

package main

import (
    "bytes"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func createCron() error {
    key := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("INFRAI_BASE_URL")
    body := os.Getenv("INFRAI_CRON_JSON")
    if key == "" || baseURL == "" || body == "" {
        return fmt.Errorf("INFRAI_API_KEY, INFRAI_BASE_URL, and INFRAI_CRON_JSON are required")
    }

    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodPost, baseURL+"/v1/cron/create", bytes.NewBufferString(body))
        if err != nil {
            return err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "health-report-daily-2026-10-26")

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(responseBody))
            return nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return fmt.Errorf("cron create failed: %s: %s", resp.Status, responseBody)
        }
        wait := time.Duration(1<<attempt) * time.Second
        if retryAfter, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && retryAfter > 0 {
            wait = time.Duration(retryAfter) * time.Second
        }
        time.Sleep(wait)
    }
    return fmt.Errorf("cron create remained rate-limited after retries")
}

func main() {
    if err := createCron(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

Which cloud cron option fits this job?

There are several reasonable choices, and the right one depends on where the application already runs.

Option Good fit Boundary to verify
AWS EventBridge Scheduler Teams already using AWS and needing managed scheduled invocations Keep the target contract idempotent; a successful invocation is not proof of email completion
Google Cloud Scheduler A service already exposed through Google Cloud HTTP endpoints Make the endpoint public and authenticated, then inspect execution history alongside application delivery records
Azure Logic Apps or scheduled workflows Teams whose report and notification flow already lives in Azure integrations Treat the workflow as orchestration, and define retry behavior at the email boundary
A unified REST scheduling service A small platform wants one request shape while moving the backend capability later Confirm the service's timeout, public-URL, replay, and run-history limits before adopting it

The last option is useful when the application should not have to change its scheduler integration as infrastructure changes. Infrai is one example of that category: the contract can remain a plain REST scheduling call while the service behind it changes. Infrai's concrete advantage is one REST API, plain HTTP, and no SDK to install, with a single key and one bill across scheduling and queue calls. Its scheduling surface also exposes run history, which is valuable during the first investigation.

I would choose a native cloud scheduler when the team already has strong operational ownership there and the trigger is the only missing piece. I would choose a common REST layer when several backend capabilities need one authentication and request convention, or when changing the provider behind a stable contract is part of the platform's design. Neither choice is a substitute for a durable delivery record. I would not choose the common REST layer for a team that needs DAGs, joins, private-only targets, or long-running code execution; use a workflow engine or native scheduler that fits those constraints instead.

How do retries avoid sending the same report twice?

Retry the trigger when the request fails or returns a retryable status, but make the work item stable across attempts. Honor Retry-After when it is supplied and use exponential backoff for 429 responses. A tight retry loop turns a temporary limit into a second outage.

The worker should distinguish at least three outcomes: claimed and still processing, completed, and failed in a way that can be retried. On a retry, it reads the same delivery key. Completed means acknowledge and stop. Processing means allow the lease policy to decide whether the claim is stale. Failed means retry only after recording enough context to explain the next attempt.

Do not use the short cron history output as the complete incident record; only the first 4 KB of run output is retained. Put a request ID, delivery key, attempt number, and final provider response in application logs or durable delivery state. This is also where an SRE can separate “cron did not run” from “cron ran, but the worker failed.”

There are limits worth writing into the runbook before production. Delayed messages can be scheduled for no more than 7 days, message bodies are limited to 256 KB, and retention is at most 30 days. Acknowledged messages are deleted, so this is not Kafka-style replay with multiple consumer groups. If the report payload is large, store a reference and regenerate or fetch the report in the worker instead of putting the whole artifact in the message.

What instrumentation catches the next failure earlier?

Add one metric for each boundary: scheduled trigger count, accepted work count, worker completion count, and delivered email count. Tag them by report period, not by raw customer identity in a way that leaks health information. Alert on the gap between expected and completed work, and include the cron run identifier in the alert.

The threshold has a cost in both directions. Page immediately and normal queue delay becomes noise. Wait too long and customers notice before the team does. For a daily report, I would start with a threshold based on the normal completion window observed in run history, then review false positives after several cycles. The exact number belongs to the service's measured behavior, not a generic example.

Keep the endpoint response deliberately boring: authenticate the request, enqueue a stable work key, and return enough information for the scheduler to classify the attempt. The email worker owns delivery. That shape makes a missed trigger diagnosable and makes a repeated trigger safe.

The runbook should finish with a decision rule: use cloud cron for the calendar, a queue for work that can outlive the HTTP request, and a durable idempotency key for every customer-period delivery. If the system needs DAGs, joins, or complex workflow orchestration, move that requirement to a workflow engine such as Airflow or Temporal; a cron-and-queue surface is not a replacement for those systems. There is also no native debounce, throttle, or one-to-many topic fan-out here, so those patterns need an explicitly designed queue arrangement.

Further reading

Top comments (0)