DEV Community

FletcherVance3712
FletcherVance3712

Posted on

Node.js Background Job Queue: Cron Enqueue and Worker Checkpoints Beyond a 15-Minute Timeout

The operational constraint changes the answer: a cron trigger that can run for only 15 minutes should never own the renewal-reminder work itself. Short answer: let the cron handler enqueue a small, idempotent command, let a worker split the customer set into bounded tasks, and persist each checkpoint before asking for the next task. This applies to a Node.js background job queue just as well as to a Go or Java worker; the language is secondary to the recovery boundary.

In customer support, the concrete job is easy to describe. A business asks that its renewal reminder wait until a contract deadline, then the system should send the reminder once, record what happened, and survive a process restart without either losing the reminder or sending it twice. That last requirement is where the design becomes an architecture decision rather than a cron expression.

A renewal reminder is a state machine, not a timer

The system has four invariants. A reminder has a stable business identity, such as account_id + renewal_id + reminder_kind; the deadline is stored as data rather than inferred from a worker's wall clock; the send attempt and its audit entry have a recoverable relationship; and a retry can safely repeat the command. A queue can deliver a message more than once. It cannot turn a non-idempotent email provider into an exactly-once system.

The scheduler should therefore do only four things: authenticate the trigger, find or create a schedule record, enqueue a command containing an immutable schedule ID, and return. It should not scan every account, wait for an external email provider, or keep progress in process memory. An HMAC construction is appropriate for authenticating a signed request; RFC 2104 defines the construction and its keyed-hashing requirements. Authentication proves who called the handler, while the schedule record prevents that caller from creating duplicate work.

The worker claims a bounded task, reads authoritative customer data, and commits the business effect with a unique idempotency key. The transaction also records an audit event: schedule ID, reminder ID, attempt identity, decision, and timestamp. After the checkpoint commits, the worker publishes the successor task. If publication and the database commit cannot be made atomic, an outbox row is the durable bridge: a relay can publish it repeatedly, and the consumer can recognize the same outbox ID.

This is an exactly-once mindset about effects, not a claim that delivery is exactly once. That distinction is important in a payment or ledger backend, where “the message disappeared” is not an audit explanation.

Option Boundary it gives you Cost or limitation
Cron handler performs all work One process and little setup A timeout leaves ambiguous partial progress and makes retries dangerous
Cron enqueues one bounded worker task Scheduling and execution have separate failure boundaries The application must own checkpoints, idempotency, and task continuation
Durable workflow engine Timers, branching, joins, and compensation are first-class More runtime state and operational ownership than a simple queue needs
Database-backed task table Queries, leases, and audit history can share one store The team owns claiming, retry timing, and cleanup semantics

The second option is the default for this scenario. It is small enough to reason about and explicit enough to test. It is not automatically the right answer for a workflow with dozens of branches or human approvals.

Where does the 15-minute timeout leave evidence?

Give the scheduled occurrence a deterministic ID. For example, a support account's reminder might be renewal:acct_482:2026-09-30:seven_day, while the execution record has a separate schedule occurrence ID. A repeated trigger performs an insert-or-read operation on that occurrence. If the record already exists, the handler returns the existing state instead of enqueuing a second logical run.

The payload should be boring: an occurrence ID, a task ID, and perhaps a cursor or partition. Do not put a mutable customer snapshot in the message. The worker should re-read the source of truth and decide whether the deadline is still valid. This matters when a customer moves the contract date after the first enqueue but before processing.

The public handler also needs a deliberate request boundary. Check the signature before creating a schedule record, include a timestamp in the signed material, and reject stale requests. The scheduler's delivery time is not the business deadline. A trigger may arrive late, and a paused schedule may not replay every missed occurrence, so the application must decide whether a missed reminder is still valid rather than quietly treating arrival time as truth. The scheduling documentation from Cloudflare is a useful reference for the distinction between cron timing and application work.

Here is the critical path in Go. The storage calls are interfaces on purpose: the production implementation should put the uniqueness check, domain write, audit event, checkpoint, and outbox row behind one database transaction.

package main

import (
    "context"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "fmt"
    "time"
)

type RenewalTask struct {
    OccurrenceID string
    Cursor       string
}

type Store interface {
    Begin(ctx context.Context) (Tx, error)
}

type Tx interface {
    RecordReminder(ctx context.Context, idempotencyKey string, task RenewalTask) (bool, error)
    WriteAudit(ctx context.Context, idempotencyKey string, outcome string) error
    SaveCheckpoint(ctx context.Context, occurrenceID string, nextCursor string) error
    WriteOutbox(ctx context.Context, task RenewalTask) error
    Commit(ctx context.Context) error
    Rollback(ctx context.Context) error
}

func validSignature(secret, body, supplied string) bool {
    m := hmac.New(sha256.New, []byte(secret))
    _, _ = m.Write([]byte(body))
    want := hex.EncodeToString(m.Sum(nil))
    return hmac.Equal([]byte(want), []byte(supplied))
}

func processTask(ctx context.Context, store Store, task RenewalTask) error {
    tx, err := store.Begin(ctx)
    if err != nil {
        return err
    }
    defer tx.Rollback(ctx)

    key := fmt.Sprintf("%s:%s", task.OccurrenceID, task.Cursor)
    newEffect, err := tx.RecordReminder(ctx, key, task)
    if err != nil {
        return err
    }
    if newEffect {
        if err := tx.WriteAudit(ctx, key, "reminder recorded"); err != nil {
            return err
        }
    } else if err := tx.WriteAudit(ctx, key, "duplicate ignored"); err != nil {
        return err
    }

    next, done := nextCursor(task.Cursor)
    if err := tx.SaveCheckpoint(ctx, task.OccurrenceID, next); err != nil {
        return err
    }
    if !done {
        if err := tx.WriteOutbox(ctx, RenewalTask{OccurrenceID: task.OccurrenceID, Cursor: next}); err != nil {
            return err
        }
    }
    return tx.Commit(ctx)
}

func nextCursor(cursor string) (string, bool) {
    if cursor == "last" {
        return cursor, true
    }
    return "last", false
}

var _ = errors.New
var _ = time.Second
Enter fullscreen mode Exit fullscreen mode

The example omits the email call because its semantics must be specified separately. If the provider supports an idempotency key, use the same stable key and retain the provider's result. If it does not, “recorded reminder” should mean a durable intent handed to a delivery component, not a claim that an external inbox received the message. Compliance requirements may also require retention of the request, response, and operator action; confirm those requirements before choosing an acknowledgement policy that removes queue messages permanently.

The queue is only a transport.

How can a Node.js background job queue split long-running work after a worker timeout?

Use a keyset cursor, not an offset. If reminders are ordered by an immutable renewal_id, a task can process the next bounded range and save the last committed ID. A retry starts after that ID and therefore does not depend on rows shifting between pages. A fixed batch size is a starting point, not a guarantee: large customer records, slow provider responses, or database locks can make the same batch exceed the worker's time budget.

Leave headroom. A worker that waits until the final second to checkpoint has converted a predictable limit into a recovery lottery. Measure tail latency, reserve time for the transaction and acknowledgement, and stop before the deadline. I’m not sure there is a universal percentage that works across teams; the useful measurement is the job's worst normal completion time under representative load, not the median on an empty database.

Retries need classification. A malformed payload should be quarantined for inspection; a transient network failure should be retried with bounded backoff; a duplicate idempotency key should be treated as a known result. Do not retry every error for the same duration. A retry budget, an attempt number, and a next-eligible timestamp belong in the task state so operations can explain why a reminder is pending.

The long section of this design is the part teams tend to skip. Suppose an account changes its deadline from September 30 to October 15 after the scheduler has created a task. The worker loads the current renewal row, compares its version with the task's expected version, and records “superseded” if the old reminder is no longer applicable. It must not send first and discover the change later. If the send intent was already committed, the audit trail needs to show that fact and a compensating policy must govern the next action; silently deleting the old task destroys the evidence needed to reconcile support tickets, ledger entries, and customer communication.

Small boundaries help.

The implementation boundary matters more than the queue brand

Choose a managed queue when the team wants delivery, visibility, and retry primitives without operating a broker, and choose a database-backed task table when transactional inspection and local audit queries are more valuable than independent queue scaling. A self-hosted broker can be the right choice for high throughput or existing platform expertise, but its operational burden is part of the decision: persistence, consumer leases, dead-letter handling, alerting, and upgrades do not disappear because the producer is a cron handler.

For a Node.js service, keep the enqueue route and the worker process separate, even if one repository contains both. The route validates the signed trigger and writes the occurrence; the worker owns task claiming and checkpointing. Keep the business idempotency key in the database, where a unique constraint can enforce it. A JavaScript promise, an in-memory lock, or a process-local “already sent” flag cannot survive a restart or coordinate two worker instances.

One failure deserves a slower walkthrough. A task for acct_482 is enqueued at 09:00, the worker reads the old deadline at 09:01, and the support agent changes that deadline at 09:02. At 09:03 the worker loses its process after sending the provider request but before committing its checkpoint. A second worker receives the delivery, reads the current version, and sees both an old task and an uncertain external request. The idempotency key must make the second provider request harmless where the provider supports it; the database must make the reminder record and audit row unique; and the reconciliation view must expose “sent, checkpoint pending” rather than pretending that either no send or two sends occurred. If the provider offers no idempotency facility, the application needs an explicit delivery-intent state and a manual or provider-side reconciliation rule. There is no honest database transaction that reaches into an unrelated inbox.

The rejected option is a monolithic cron request. It is valid for a bounded action that is naturally idempotent and completes comfortably within the platform's limit, such as creating one known task. It is unsuitable for scanning a growing customer table, calling many external providers, or performing a workflow whose duration depends on tenant data. For those cases, the queue-and-checkpoint boundary makes the partial states visible and replayable.

The catch is that a basic queue does not provide the domain's exactly-once guarantee for you. If you need branching, joins, long timers, or compensation as first-class workflow concepts, use a workflow system or build those states explicitly in the database. Stick with a simpler task queue when the job is a sequence of bounded, auditable effects; change the boundary when the workflow itself becomes the product.

References

Top comments (0)