ThatcherCole8235

Posted on Nov 13

Node.js Schedule for Daily Email — Cloud Cron Examples with Retry State

#node #scheduling #email #distributedsystems

Short answer: use cloud cron to create one idempotent daily run, then let a durable queue and leased Node.js workers drain the email jobs with bounded retries.

A daily report email is easy to trigger and surprisingly hard to finish reliably. The schedule is only the first event. The real system has to drain a rate-limited worker pool, survive a timeout after the mail provider accepted a message, and avoid sending the same report twice.

My default design is a thin cloud cron trigger that enqueues one durable job, followed by Node.js workers that claim jobs with a lease, record an idempotency key, and retry only when the result is genuinely unknown or transient. Before copying that choice, measure queue age, attempt count, send latency, duplicate suppression, and the number of reports that miss their delivery window. A cron expression cannot tell you any of those things.

What should a daily report job retry?

Retry the work, not the whole schedule. A scheduler should create a run for a defined reporting date, perhaps 2026-10-25, and then stop being responsible for delivery. If the scheduler fires again after a deployment or a missed invocation, the run key still identifies the same logical report.

The useful distinction is between a known failure and an unknown result. A rejected request before the provider accepted it is usually retryable. A connection that dies after the request leaves the process is not safe to replay blindly. The provider may have accepted the email while the worker is preparing to retry.

That is the uncomfortable case.

For a daily report, I would store a durable record keyed by a stable value such as daily-report:2026-10-25:account-42. The record contains the intended recipient set, report date, current state, attempt count, lease expiry, and a provider-independent idempotency key. The worker checks that record before doing expensive report generation and checks it again before accepting a retry. The trade-off is deliberate: the extra state costs a database write, but it makes an ambiguous send inspectable and prevents a blind replay.

A queue visibility timeout helps with ownership, but it is not an idempotency system. The SQS visibility timeout documentation describes the period in which a received message is hidden from other consumers; if the consumer does not delete it before that period ends, the message can become visible again. That behavior is useful for recovering abandoned work. It also means the handler must tolerate redelivery.

The small state machine that keeps the email boring

I use explicit states instead of a boolean named sent. queued means a worker may claim the job. running means a lease is active. accepted means the outbound operation returned a durable provider receipt. delivered is a separate state when delivery confirmation exists. retryable carries the next attempt time. dead requires an operator or a separate replay policy.

The important transition is running -> accepted, and it must be guarded by the idempotency key. The worker can crash after the provider accepts the request and before the database transaction completes; a later worker must discover the existing result instead of producing a second message. That requires an outbound API or a sending boundary that can replay the same key safely. If the email boundary cannot provide that property, keep the ambiguity visible and reconcile by provider receipt, rather than pretending a timeout is a clean failure.

Here is a deliberately small TypeScript-shaped worker. The interfaces are the boundary, so the queue, database, and email implementation can be swapped without changing the retry decision.

type JobState = "queued" | "running" | "accepted" | "retryable" | "dead";

type ReportJob = {
  id: string;
  idempotencyKey: string;
  state: JobState;
  attempts: number;
  leaseUntil: number | null;
  nextAttemptAt: number;
};

type SendResult =
  | { kind: "accepted"; receipt: string }
  | { kind: "retryable"; reason: string }
  | { kind: "permanent"; reason: string };

interface Store {
  claim(id: string, leaseMs: number): Promise<ReportJob | null>;
  recordAccepted(id: string, key: string, receipt: string): Promise<void>;
  scheduleRetry(id: string, nextAttemptAt: number, reason: string): Promise<void>;
  markDead(id: string, reason: string): Promise<void>;
}

interface MailBoundary {
  send(input: { idempotencyKey: string; to: string[]; body: string }): Promise<SendResult>;
}

async function processReport(
  store: Store,
  mail: MailBoundary,
  jobId: string,
  to: string[],
  body: string,
  now = Date.now(),
): Promise<void> {
  const job = await store.claim(jobId, 120_000);
  if (!job || job.state === "accepted") return;

  const result = await mail.send({
    idempotencyKey: job.idempotencyKey,
    to,
    body,
  });

  if (result.kind === "accepted") {
    await store.recordAccepted(job.id, job.idempotencyKey, result.receipt);
    return;
  }

  if (result.kind === "permanent" || job.attempts >= 5) {
    await store.markDead(job.id, result.reason);
    return;
  }

  const backoffMs = Math.min(15 * 60_000, 2 ** job.attempts * 30_000);
  await store.scheduleRetry(job.id, now + backoffMs, result.reason);
}

The 120_000 millisecond lease is not a universal setting. It should exceed normal report generation and send time while remaining short enough to recover a crashed worker. The five-attempt ceiling is also a policy, not a law. I would derive it from the delivery window and the provider's transient-error behavior, then test the boundary where the next retry would arrive after the report is stale.

There is a hard limit here: leases recover abandoned work, but they cannot prove that a timed-out email was never accepted.

How do cloud cron and workers divide the job?

Cloud cron should answer one narrow question: when should a new logical run be created? It should not hold a worker open while a report is rendered, and it should not own the retry loop. The trigger can call a small Node.js endpoint or publish a message; that handler performs a database insert with a uniqueness constraint on the run key and returns.

This separation also makes local testing less awkward. A test can submit a run directly, advance a fake clock, and assert that a redelivery does not create a second accepted receipt. The scheduled path remains covered by one integration test that checks the trigger-to-queue handoff.

There is no single meaning of “cloud cron.” AWS EventBridge Scheduler is a managed schedule that can invoke configured targets. Google Cloud Scheduler is designed to send scheduled requests, including HTTP requests. GitHub Actions supports scheduled workflow triggers, which run workflows rather than acting as a general-purpose queue. Azure Functions timer triggers start function code on a schedule. These boundaries matter: an HTTP trigger, a workflow runner, a function runtime, and a queue consumer have different timeout, identity, and retry semantics.

Scheduling option	Useful boundary	Main limitation for this job
AWS EventBridge Scheduler	Invokes a configured target	It creates the trigger; queue ownership and send idempotency remain application concerns
Google Cloud Scheduler	Sends a scheduled HTTP request	An HTTP request is only a handoff, not durable per-recipient progress
GitHub Actions schedule	Starts a scheduled workflow	A workflow runner is a poor substitute for a rate-limited email queue
Azure Functions timer trigger	Starts function code on a schedule	Function execution and outbound delivery still need explicit retry state

The table is a boundary map, not a ranking. A lightweight HTTP trigger may be the right fit for one daily run, while a workflow runner may be unsuitable when thousands of account jobs must drain under a provider limit.

Treat the product's schedule feature as an adapter. Keep the run key, enqueue operation, and state transitions in application code. That leaves the decision portable without pretending all schedulers behave the same way.

The rate limit changes the drain algorithm

Suppose the report run creates 2,400 account jobs and the outbound boundary allows 20 sends per second. Launching 2,400 promises at once does not drain faster; it creates a burst, increases memory pressure, and makes a timeout harder to classify. A fixed worker count plus a token bucket gives the system a visible operating limit. That is slower than unrestricted concurrency by design, and the queue-age alarm tells you when the trade-off has become unacceptable.

The queue consumer should claim only the next available job, respect nextAttemptAt, and release or extend its lease deliberately. When the provider returns a throttling response, the worker should lengthen backoff and record the event. A permanent address failure should not consume the retry budget meant for a transient network failure.

I also separate report rendering from sending when rendering is expensive. The rendered artifact gets a content hash and an expiry policy. A retry can reuse that artifact, while the send record still uses the stable report idempotency key. This reduces repeated database reads and makes the cost of a slow report visible instead of hiding it inside every attempt.

What I would test before shipping

Start with the failure matrix, not the happy-path cron demo. Test a worker crash before acknowledgement, after the outbound request, and after the accepted receipt is stored. Test two workers claiming the same job, a lease expiring during a long render, a scheduler firing twice, a five-minute provider outage, and a report whose delivery window has closed.

The most valuable test is the awkward middle one. Create a job for one account, let the fake mail boundary record the idempotency key as accepted, then make the worker throw before recordAccepted completes. On the next delivery, return the existing receipt for that same key. The expected result is one accepted send and one stored receipt, not two calls with two newly generated keys. Repeat the sequence with two workers whose leases overlap, because a lease check and a database update can be separated by a process pause. Then advance the clock beyond the report window and verify that the retry policy records a terminal reason instead of quietly sending stale data. This test does not prove a provider's behavior, but it proves that the application's contract remains explicit when the provider's answer is missing.

That is the boundary worth testing.

For each case, assert invariants: at most one accepted send for one idempotency key, every claimed job eventually becomes terminal or retryable, a retry does not reset the original run date, and the queue can make progress after one poisoned job. Those assertions are more valuable than a screenshot showing a cron invocation.

Instrumentation should expose queue age, lease expiry count, retry reason, attempt histogram, accepted-to-delivered delay, and duplicate-suppression count. Keep recipient data out of ordinary logs. A report identifier and hashed account identifier are enough for most operational traces.

The cost decision is tied to these measurements. If rendering dominates, cache the artifact. If idle workers dominate, reduce concurrency and let queue age set the alarm. If retries dominate, classify failures before adding capacity. More workers cannot repair an ambiguous send result.

The durable pattern is small: cron creates a unique run, a queue holds work, workers claim with leases, the send boundary accepts an idempotency key, and state transitions are observable. The exact scheduler can change. The retry contract cannot be hand-waved away.

DEV Community