thomasmoore5082

Posted on Oct 23

Node.js Message Queue Setup: Create, Publish, and Run Background Tasks

#node #messagequeues #scheduling

The least complex Node.js setup to create a message queue and publish a background task is a durable queue with a worker that owns the deadline, an idempotency key, and a visibility timeout longer than normal processing time. For a B2B SaaS renewal reminder, a timer in the web process is a poor fit: deploys, crashes, and multiple instances turn a timer into an unreliable promise.

TL;DR: put a reminder job on a durable queue with its due time, have a worker claim it, and make the send operation idempotent. Start with an SLO such as 99.9% of reminders delivered within 60 seconds of their business deadline. Then size worker concurrency and retention from the deadline volume and retry policy, rather than from the queue's marketing description.

The page fires before the reminder does

The on-call page usually arrives as a queue-age alert: renewal reminders are sitting longer than the allowed delay, and the dashboard shows a growing count of due jobs. The visible symptom is late mail. The earlier signal was quieter: the worker's successful completions per minute had fallen below the rate at which reminders became eligible.

That distinction matters. A queue can be empty while the system is still wrong if the producer has stopped publishing, and it can be full while delivery remains inside the SLO if jobs are scheduled for tomorrow. Alerting on raw depth alone confuses planned waiting with operational backlog.

For a renewal workflow, I would trace the job through four timestamps: created_at, eligible_at, claimed_at, and completed_at. The alert should evaluate completed_at - eligible_at for eligible work, with a separate alert for a missing producer signal. The instrumentation change is small but consequential: publish a metric for eligible jobs, record queue delay and processing duration separately, and include the tenant and job id in structured logs without putting customer content there.

The threshold has a cost in both directions. Set it too low and a brief traffic burst wakes the on-call team for a queue that will drain before the deadline. Set it too high and the first page arrives after the renewal message has missed its business purpose. False positives consume the same finite on-call budget needed for database and payment incidents.

What should a Node.js background task actually own?

The application should publish a durable description of intent, not hold an open request until a calendar date. A renewal job might contain a tenant identifier, a renewal identifier, an eligible_at timestamp, a schema version, and an idempotency key. The worker resolves the current account state before sending; it does not trust a stale copy of the entire customer record embedded in the message.

The queue's job is delivery and temporary exclusion. The worker's job is business correctness. Those boundaries prevent a retry from accidentally sending a second reminder after a timeout, and they let the application change its worker pool without changing the API request path. A queue does not solve an ambiguous business deadline, an unbounded downstream dependency, or a missing idempotency record; those limits stay in the application design.

The main limitation is the extra operational boundary: someone must own retention, redelivery, dead letters, and capacity alarms. The trade-off is reasonable only when losing the job or holding a request open is more expensive than operating that boundary. For a low-volume internal reminder, a transactional database row and a periodic poller may be the better choice.

That is the whole trick.

Here is the smallest useful shape for a worker-side decision. The queue adapter is deliberately generic; a production adapter must implement the acknowledgement and visibility semantics of its chosen transport.

package reminder

import (
    "context"
    "time"
)

type Job struct {
    ID           string
    RenewalID    string
    EligibleAt   time.Time
    Idempotency  string
}

type Queue interface {
    Receive(context.Context) (Job, string, error)
    Ack(context.Context, string) error
    Retry(context.Context, string, time.Duration) error
}

type Sender interface {
    SendIfNew(context.Context, string, string) error
}

func Handle(ctx context.Context, q Queue, s Sender, now func() time.Time) error {
    job, receipt, err := q.Receive(ctx)
    if err != nil {
        return err
    }
    if wait := time.Until(job.EligibleAt); wait > 0 {
        return q.Retry(ctx, receipt, wait)
    }
    if err := s.SendIfNew(ctx, job.Idempotency, job.RenewalID); err != nil {
        return q.Retry(ctx, receipt, 30*time.Second)
    }
    return q.Ack(ctx, receipt)
}

The SendIfNew operation needs a durable uniqueness constraint, usually keyed by the business event and reminder type. An acknowledgement is not a transaction with the email provider or notification service. The worker can send successfully and crash before acknowledging, so at-least-once delivery requires deduplication. Exactly-once processing is not a property I would put in an SLO unless the complete side-effect path supports it.

EligibleAt also deserves a precise definition. If the business deadline is 09:00 in an account's local timezone, resolve that timezone when the reminder is created, store the resulting instant, and test daylight-saving transitions explicitly. A queue delay is measured from eligibility, not from the moment a customer first configured a renewal date.

Which queue boundary fits the failure mode?

The important comparison is operational, not fashionable. A managed queue generally reduces broker patching and quorum work, while a self-hosted broker can offer richer routing and local control at the cost of storage, upgrades, and a larger on-call surface. A Redis-backed library inside Node.js can be pleasant for application teams, but its durability and failure behavior depend on how Redis is operated.

Option	Useful boundary	Cost or risk to plan for	Fit for renewal reminders
Amazon SQS	Managed durable queue with visibility timeout and redelivery	Less control over broker internals and routing semantics	Good when the team values lower broker operations and can model work with simple queues
RabbitMQ	Self-managed broker with acknowledgements and routing patterns	Cluster storage, upgrades, partition behavior, and paging become team responsibilities	Useful when routing topology is a real requirement, not just a preference
BullMQ	Node.js job library backed by Redis, with delayed jobs and worker APIs	Redis persistence, failover, and queue semantics remain part of the platform design	Practical when the team already operates Redis and accepts that boundary
Build a database scheduler	Familiar storage and transactional writes	Polling, locking, index growth, and retry recovery become application code	Reasonable at modest volume when a queue would add more machinery than it removes

This is a buy-vs-build decision with a capacity consequence. Suppose 120,000 reminders become eligible in a busy hour and the worker's measured end-to-end service time is 250 ms. A single worker slot can process about four jobs per second before overhead; 120,000 jobs in 3,600 seconds needs an average of 33.3 jobs per second. The design therefore needs more than nine continuously healthy slots, plus headroom for retries, downstream throttling, and deploys. I would provision and test closer to the peak concurrency implied by the SLO, not the average arrival rate. If the downstream provider permits only a much smaller rate, adding queue consumers makes the failure worse by increasing rejected work, so the worker pool needs a limiter as well as more capacity. That is a real boundary of queue-based scaling: storage absorbs bursts, but it cannot manufacture notification throughput.

The exact number is not a promise about any queue product. It is a capacity-planning example showing why queue depth is an output of worker capacity and deadline distribution. Measure the service time at the notification boundary, then include retry traffic in the arrival rate.

More workers are not always better.

Why does visibility timeout change the retry story?

Visibility timeout is the interval during which a received message is hidden from other consumers. It is not a processing lease that makes the work safe forever. If the worker runs longer than the interval without extending it, another worker can receive the same job while the first worker is still sending. If the timeout is enormous, a crashed worker can delay redelivery and push the reminder past its SLO.

Choose an initial timeout above the high percentile of normal processing time, then extend it while a long-running operation is genuinely making progress. Cap total attempts and route exhausted jobs to a dead-letter destination with enough metadata to diagnose them. The retry delay should respect downstream rate limits; immediate retries can turn one provider failure into a synchronized storm.

A useful test sequence is deliberately boring: claim a job, stop the worker, wait for the timeout, confirm redelivery, make the first attempt appear successful but skip acknowledgement, and confirm that the second delivery does not create a second reminder. Repeat it with a worker deployment and with a slow notification dependency. These are the cases that expose a design that looked correct in a unit test.

The queue API should also distinguish a transient failure from a permanent one. A malformed schema, deleted renewal, or rejected recipient may deserve a terminal state and an operator-visible record. A timeout from a notification dependency usually deserves retry. Treating every error alike produces either silent data loss or a retry queue that never drains.

The operational contract is the implementation

Before choosing a library, write down the contract. For this workflow I would specify a 60-second delivery SLO after eligible_at, a maximum retry age, a duplicate-send policy, an outage behavior, and the ownership of the dead-letter queue. The contract should say what happens when the business record changes between scheduling and execution; “send the latest valid reminder once” is more useful than “process the job exactly once.”

The dashboard needs at least these views: eligible backlog age, processing duration, retry count, dead-letter count, producer publication rate, and downstream response class. Logs should correlate the request, job, renewal, and idempotency identifiers. Traces are valuable across the publish, receive, and side-effect boundaries, but trace retention should not become the only audit record.

Deployments deserve a queue-specific runbook. Drain or pause consumers when a schema change is incompatible, preserve old workers until old messages are safe to consume, and use a versioned payload so a delayed reminder does not become an accidental migration test. Capacity alarms should distinguish a worker pool that is saturated from a downstream service that is rejecting requests.

My decision rule is conservative: use the simplest durable boundary that meets the latency SLO, supports idempotent side effects, and leaves the team able to explain redelivery at 02:00. A richer broker is justified by a concrete routing or throughput requirement. A database scheduler is justified by low volume and a willingness to own polling and locking. A queue is a poor choice for a single small process that can safely retry from its database, while a database poller is a poor choice once deadline volume makes lock contention and polling lag material. The queue name matters less than the failure contract around it.

DEV Community