For Node.js delayed background task scheduling, publish the task with an explicit delay, then make the consumer idempotent before adding retries; that is the safest shape for draining a rate-limited media worker pool. The delay controls eligibility. It doesn't turn at-least-once delivery into exactly-once processing.
Short answer: put the delay in the publish parameters, carry a stable job ID, and have the worker record completion in durable storage before acknowledging the message. For a long-running job, use a scheduler only to trigger the queue; let the worker do the work.
What does delayed execution actually guarantee?
It guarantees that a message should not become eligible before its delay has elapsed. It does not promise an exact execution timestamp, and it does not prevent a second delivery after a timeout or worker restart.
Consider a media service that generates a thumbnail 90 seconds after an upload. The publisher sends a small job with a stable ID. The worker receives it when eligible, calls the rate-limited image service, backs off on a 429, and records the result before acknowledging the message. If the process dies after the result is written but before acknowledgment, the same job can arrive again. The result write must therefore be idempotent too.
There are concrete boundaries to put in the runbook: a delay is at most 604800 seconds, a message body is at most 256 KB, retention is at most 30 days, and acknowledgment deletes the message. A larger payload belongs in durable storage with a queue reference. A delay longer than 7 days needs a later scheduling decision, rather than an invalid queue request.
I initially treated delay as the scheduling feature. The operational correction is narrower: delay selects eligibility, while the consumer owns correctness. That distinction keeps a retry policy from becoming a duplicate-thumbnail policy.
How should Node.js delayed background task scheduling handle retries?
Use an ID generated by the producer and keep it unchanged across retries. The worker checks that ID in durable storage, performs the side effect only when the job is unfinished, and marks completion after success. A check followed by a write isn't automatically atomic; the store needs a uniqueness constraint or an equivalent transaction around the completion record.
This is the small part worth making boring. The queue adapter can be replaced, but the idempotency contract should stay in the application.
package worker
import (
"context"
"time"
)
type Job struct {
ID string
Payload []byte
}
type Store interface {
Completed(ctx context.Context, id string) (bool, error)
MarkCompleted(ctx context.Context, id string) error
}
type Processor interface {
Process(ctx context.Context, payload []byte) error
}
func Handle(ctx context.Context, job Job, store Store, processor Processor) error {
done, err := store.Completed(ctx, job.ID)
if err != nil {
return err
}
if done {
return nil
}
if err := processor.Process(ctx, job.Payload); err != nil {
return err
}
return store.MarkCompleted(ctx, job.ID)
}
func RetryDelay(attempt int, retryAfter time.Duration) time.Duration {
if retryAfter > 0 {
return retryAfter
}
if attempt < 1 {
attempt = 1
}
delay := time.Second * time.Duration(1<<min(attempt, 6))
if delay > 60*time.Second {
return 60 * time.Second
}
return delay
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
Retry transient network failures, upstream 429 responses, and timeouts. Honor Retry-After when it exists. Do not retry malformed media or an authorization failure forever; send those cases to a dead-letter path after a finite attempt budget. A standard queue is at-least-once, so consumer idempotency is mandatory even when the retry count looks small.
The publisher's delay should be explicit and bounded. This example uses the documented queue publish route, a key from the environment, an idempotency key, and a bounded retry loop. The application-specific message fields are kept small so the worker can fetch the media metadata it needs from its own store.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
body := strings.NewReader(`{"delay_seconds":90,"job_id":"thumbnail-2026-10-25-001"}`)
baseURL := os.Getenv("QUEUE_API_BASE_URL")
if baseURL == "" {
panic("QUEUE_API_BASE_URL is required")
}
req, err := http.NewRequest(http.MethodPost, strings.TrimRight(baseURL, "/")+"/v1/queue/publish", body)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "thumbnail-2026-10-25-001")
for attempt := 1; attempt <= 4; attempt++ {
res, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
data, readErr := io.ReadAll(res.Body)
res.Body.Close()
if readErr != nil {
panic(readErr)
}
if res.StatusCode >= 200 && res.StatusCode < 300 {
fmt.Println(string(data))
return
}
if res.StatusCode != http.StatusTooManyRequests || attempt == 4 {
panic(fmt.Sprintf("publish failed: %s: %s", res.Status, data))
}
time.Sleep(time.Duration(1<<attempt) * time.Second)
}
}
The retry loop above is deliberately bounded, but production code should parse Retry-After before falling back to exponential delay. Do not create a new idempotency key for each attempt. That turns a network timeout into multiple publish operations.
Which option fits a delayed media worker?
Amazon SQS is a sensible choice when the surrounding service is already AWS-native. FIFO queues offer ordering and deduplication features, but the FIFO deduplication window is only 5 minutes; it does not remove the need for application-level idempotency in a long-running media job. Standard queues still require the consumer to tolerate duplicate delivery.
BullMQ fits a Node.js service that already operates Redis. Delayed jobs, retries, and worker concurrency map closely to the application model, which keeps the first implementation compact. The trade-off is operational: Redis persistence, failover, memory pressure, and worker coordination become part of the reliability surface.
Temporal is the better boundary when the thumbnail is one step in a durable, multi-stage workflow with timers, compensation, or human review. It brings a workflow programming model and a larger operational commitment. For “publish one message, wait 90 seconds, consume it,” that is usually more machinery than the queue problem requires.
A plain REST queue surface is another reasonable fit for a small team that wants to call backend capabilities from several runtimes. Infrai is relevant when one key and one bill across backend services removes credential and reconciliation work, and its plain HTTP interface avoids an SDK install; the public discovery surface exposes request schemas and runnable examples in 10 languages. That matters here because a solo operator can inspect the queue contract from the same interface used by a Go worker or a Node.js publisher, while the platform's 295 routes across 20 modules cover more than this one queue. It still does not provide DAG or workflow orchestration, fan-out/join primitives, Kafka-style replay, or multiple consumer groups, so it should not be chosen as a substitute for Temporal or a streaming platform.
| Option | Best fit | Main boundary |
|---|---|---|
| Amazon SQS | AWS-native queues and FIFO semantics | AWS coupling; FIFO deduplication lasts 5 minutes |
| BullMQ | Node.js teams already operating Redis | Redis durability, failover, and memory are your responsibility |
| Temporal | Multi-step workflows with durable timers or compensation | Larger operational and programming-model commitment |
| Plain REST queue | Small services using several runtimes | Application idempotency and workflow orchestration remain yours |
The decision is about failure ownership. Pick the queue that fits the team's existing operational boundary, then keep stable job IDs and duplicate-safe side effects regardless of the vendor.
How do you verify and roll back the change?
Start with a 5-second delay in a test queue, then test the longest delay the application will actually use. Record enqueue time, eligibility time, first receipt, attempt number, and completion time. Kill the worker after the side effect and before acknowledgment; the second delivery should leave one result, not two.
For a rate-limited pool, deliberately exceed the upstream limit. Verify that 429 responses create spaced retries, that Retry-After is honored, and that a permanently bad job reaches the dead-letter path after the retry budget is exhausted. Validate the 256 KB message boundary and the 604800-second delay boundary before production publishing.
Rollback means stopping publication of the new job type, letting compatible messages drain, and retaining idempotency records. Do not purge the queue as the first response: purging removes evidence and can turn a deployment problem into lost media work.
For work that can run longer than 900 seconds, use a cron trigger to enqueue a job and let the worker consume it. Cron targets must be public HTTP URLs, and pausing cron does not backfill missed triggers. The runbook should state whether skipped work is intentionally lost or must be re-enqueued.
Use delayed queue publishing for bounded background work. Add idempotency before retries. Move to a workflow engine when the job needs durable multi-step orchestration.
Top comments (0)