DEV Community

ThatcherCole8235
ThatcherCole8235

Posted on

Weekly Customer Reminder Digests — Recovering Delayed Jobs with Postgres and Node.js

Use Postgres as the schedule of record for a weekly customer-support digest, let cron wake a Node.js worker, and treat any delayed-job queue as an optional accelerator rather than the only copy of future work. The deciding constraint is operational recovery: after a bad deploy, a paused sender, or an exhausted email quota, an operator must be able to see which customer reminders are due, reclaim abandoned work, and resend without duplicating a digest.

This isn't the lowest-latency design. It is the one whose state is easiest to inspect and repair for a small team serving customers in Europe and the US. A queue-first design can be a better fit at higher volume; a pure cron loop can be cheaper to run at very low volume. Neither choice removes the need for durable delivery state.

The useful distinction is authority versus transport.

How should a Node.js reminder system choose delayed jobs, cron, or Postgres?

Start with the failure you need to recover from. For a weekly digest, being a few seconds late is usually less damaging than sending twice or silently skipping a week. That makes a Postgres row a practical authority: it can hold the customer's time-zone rule, the next eligible instant, the delivery state, the attempt count, and the stable idempotency key in one inspectable record. Cron then has one modest job: wake a worker often enough to find due rows.

Delayed jobs solve a different problem. They can keep workers from polling empty tables and can distribute a burst across consumers, but a message scheduled six days ahead is awkward as the sole record when a customer changes time zone, becomes inactive, or opts out. The application still needs a durable place to answer, “What should happen next?” Put that answer in the database. If a queue is present, publish only a job identifier and verify the current database state when the worker receives it.

The simple approach is tempting: run one cron expression for every customer and send immediately inside the callback. It fails the recovery test. Per-user schedules become hard to enumerate, the callback has no durable lease, and a process restart leaves no obvious distinction between “never attempted” and “sent but not recorded.” Keeping one global wake-up schedule and materializing each customer's next run makes those states explicit.

There is a cost trade-off — no universal “cheapest” architecture exists without request volume, database load, queue retention, and operator time. At small scale, reusing an existing Postgres database and one bounded poller avoids another moving part. At sustained scale, repeatedly scanning for sparse work may cost more than emitting delayed messages. Your mileage may vary because the crossover depends on the workload, not the product category.

Retry recovery begins with an expiring lease

The schedule table needs a small state machine. scheduled means the digest may be claimed after nextRunAt; running means one worker owns a lease; sent means the provider accepted the delivery operation and the application committed that outcome; retryable means the next attempt is allowed after backoff; and cancelled means policy says not to send. Keep the customer activity check close to the claim or send transaction so an old queued message cannot revive an obsolete reminder.

The critical move is claiming a bounded batch atomically. Multiple workers may wake at once, so each due row must be leased to one worker while the others skip it. The lease needs an expiry. Without one, a process that dies after claiming work can strand the row forever; with one, a recovery pass can return expired work to the eligible set. Keep the batch small enough that the lease comfortably exceeds the normal processing time, then renew deliberately if processing may run longer. Sending an email and committing a database transaction cannot be one atomic operation, so idempotency has to bridge that boundary. Derive a stable key from the logical delivery, such as customerId + digestWeek + channel, store it before sending, and pass it to a provider that honors idempotency when one is available. If provider-level idempotency isn't available, keep a delivery ledger and make duplicates observable; don't pretend a local sent boolean creates exactly-once behavior across a network call. This is also where an outbox earns its keep: in the same database transaction that makes a digest eligible, insert an outbox record describing the intended delivery. A dispatcher claims outbox rows, sends them, and records the result. If publishing to a queue is part of the design, the dispatcher publishes from the outbox instead of asking request-handling code to update Postgres and a queue independently. The queue can redeliver; the stable delivery key and current row state decide whether redelivery has work to do.

Keep recovery boring.

No guesswork.

An operator should be able to answer four questions with ordinary queries: which rows are overdue, which leases expired, which customers have repeated attempts, and which logical digest keys already have a successful delivery. Dashboards are useful, but the database queries are the recovery contract. Test them before the first incident, including the path that moves an expired running row back to an eligible state.

Code the claim and lease in TypeScript

The following sketch keeps vendor details outside the scheduling core. It assumes the database adapter implements a transaction and parameterized queries; the SQL uses row locking so competing workers can claim separate due rows. The example intentionally returns identifiers and scheduling data, not a fully rendered email, because rendering and model calls shouldn't lengthen the claim transaction.

type DigestJob = {
  id: string;
  customerId: string;
  nextRunAt: Date;
  timeZone: string;
  leaseUntil: Date;
  deliveryKey: string;
};

type Transaction = {
  query<T>(sql: string, values: unknown[]): Promise<{ rows: T[] }>;
};

type Database = {
  transaction<T>(fn: (tx: Transaction) => Promise<T>): Promise<T>;
};

async function claimWeeklyDigests(
  db: Database,
  now: Date,
  leaseUntil: Date,
  batchSize: number,
): Promise<DigestJob[]> {
  return db.transaction(async (tx) => {
    const result = await tx.query<DigestJob>(
      `
        WITH due AS (
          SELECT id
          FROM reminder_schedule
          WHERE status IN ('scheduled', 'retryable')
            AND next_run_at <= $1
            AND active = true
          ORDER BY next_run_at, id
          FOR UPDATE SKIP LOCKED
          LIMIT $2
        )
        UPDATE reminder_schedule AS schedule
        SET status = 'running',
            lease_until = $3,
            attempt_count = attempt_count + 1
        FROM due
        WHERE schedule.id = due.id
        RETURNING
          schedule.id,
          schedule.customer_id AS "customerId",
          schedule.next_run_at AS "nextRunAt",
          schedule.time_zone AS "timeZone",
          schedule.lease_until AS "leaseUntil",
          schedule.delivery_key AS "deliveryKey"
      `,
      [now, batchSize, leaseUntil],
    );

    return result.rows;
  });
}
Enter fullscreen mode Exit fullscreen mode

The sender should perform its expensive work after this transaction commits. Before delivery, it reloads the row and confirms that the customer remains active, the lease still belongs to this attempt, and the delivery key has no successful ledger entry. After a successful delivery, a short transaction records the ledger result and computes the next occurrence.

Keep regional time-zone data in the schedule

Time zones deserve special treatment. Store the customer's named time zone and the local scheduling rule, while using a UTC instant for nextRunAt. After a terminal outcome, calculate the following occurrence from the local calendar rule; don't advance by adding a fixed seven times 24 hours. Europe and the US change clocks on different calendars, and a local-morning digest should stay a local-morning digest. The exact library is a deployment choice, but its time-zone data must be current and the calculation needs tests around both forward and backward clock changes.

One subtle failure remains: a worker may send successfully and die before recording success. The next worker sees an expired lease and tries again. A provider idempotency key can collapse those attempts; without that guarantee, the delivery ledger narrows the ambiguity but cannot prove the external side effect did not happen. Surface that state for review instead of retrying forever. Honest ambiguity is operationally safer than an invented exactly-once claim.

What did the recovery test actually measure?

Measure overdue age, claim-to-send latency, expired lease count, attempts per logical digest, duplicate-suppression count, database rows scanned per claimed job, and the age of the oldest outbox row. Split the delivery metrics by region and time zone. A global average can look healthy while a clock-change boundary or regional provider limit creates a narrow backlog.

Signal What it reveals Change to consider
Oldest overdue age Whether recovery is keeping pace Reduce claim size variance or add workers
Expired leases Work abandoned after claim Revisit lease duration and worker shutdown
Rows scanned per claim Polling cost for sparse schedules Add a delayed-job execution layer
Duplicate suppressions Redelivery pressure at the send boundary Audit delivery keys and retry policy
Oldest outbox age Separation between intent and dispatch Scale or isolate dispatchers

Run recovery drills, too. Pause the worker for one scheduling interval, resume it, and confirm that overdue rows drain in bounded batches rather than creating an uncontrolled send spike. Terminate a worker after claim and verify that its lease expires and another worker can reclaim the job. Change a customer's time zone while a delayed message is outstanding and verify that the consumer consults the current schedule instead of trusting stale payload data. Finally, replay the same delivery identifier and confirm the ledger or provider idempotency behavior matches the documented contract.

Cost follows the execution boundary

The catch is that this Postgres-led design is not suitable when reminders require sub-second precision, when the due set is so large that polling and row contention threaten transactional traffic, or when delivery must continue independently of the primary database. In those cases, use a dedicated scheduling or queue service as the execution plane and retain a separate application record for cancellation, audit, and deduplication. FIFO queues can help where ordering within a defined group matters, while cron-trigger products are better treated as wake-up mechanisms than per-customer state stores.

Stick with a plain cron scan when volume is low, lateness tolerance is broad, and the existing database has headroom. Add delayed jobs when measurements show empty polling, burst distribution, or worker isolation is the actual bottleneck. Move scheduling out of the primary database when contention or availability boundaries demand it. Choose from observed recovery behavior, not from the elegance of the happy path.

References

Top comments (0)