DEV Community

GarrisonSterling2693
GarrisonSterling2693

Posted on

Background Job Queue: Create Node.js Worker Ack, Retry, Idempotency in 6 Stages Explained

#backgroundjobs #queues #reliability

A cleanup alert fires at 02:14. The API is healthy, but a background job queue has left 18,000 expired developer-tool artifacts in storage because its worker acknowledged messages before doing the work.

Short answer: create a durable queue and a separately scaled worker, consume jobs with an explicit ack or nack policy, retry only transient failures with bounded backoff, dead-letter poison messages, and make the side effect idempotent. That keeps a periodic cleanup out of the web request path while making latency and cost measurable.

Start with the page, then trace the missing signal

The page is the last symptom, so I work backwards from it. For a scheduled cleanup, the producer should attach a job ID, tenant ID, intended run time, and deadline. The worker should put those fields in structured logs and a trace span. I want enqueue age, oldest-unprocessed age, handler duration, acknowledgement latency, attempts per job, and the ratio of successful side effects to consumed messages. Queue depth alone is a weak proxy: a fast consumer can drain a growing stream while repeatedly failing the same work.

The useful SLO might be “99% of due jobs begin within 10 minutes.” Alert on that age against the SLO, then add a separate dead-letter-rate alert. During a deploy, those signals distinguish planned capacity from a handler that is receiving HTTP 429 (Too Many Requests), where the downstream service may provide a Retry-After value.

One short page.

How should a Node.js queue worker create, consume, ack, nack, and retry jobs?

The implementation language is less important than the contract. A scheduler creates a durable record; a worker consumes it; the handler validates input, performs an idempotent operation, persists the result, and acknowledges only after the side effect is committed. A nack is a routing decision, not a generic error button: requeue a transient failure when the broker can redeliver, and route a permanent failure to a dead-letter queue.

This Go sketch keeps the broker behind a tiny interface. It also makes the dangerous ordering visible: acknowledging before cleanup lowers apparent latency but loses work when the process dies.

type Job struct {
    ID       string
    TenantID string
    Attempt  int
}

type Delivery struct {
    Job  Job
    Ack  func() error
    Nack func(requeue bool) error
}

type Store interface {
    Completed(id string) (bool, error)
    MarkCompleted(id string) error
}

type Cleanup interface {
    Expired(tenantID string) error
}

func handle(d Delivery, store Store, cleanup Cleanup) error {
    if d.Job.Attempt > 5 {
        return d.Nack(false) // bounded retries; broker applies dead-letter policy
    }

    seen, err := store.Completed(d.Job.ID)
    if err != nil {
        return d.Nack(true)
    }
    if seen {
        return d.Ack()
    }

    if err := cleanup.Expired(d.Job.TenantID); err != nil {
        if isTransient(err) {
            return d.Nack(true)
        }
        return d.Nack(false)
    }
    if err := store.MarkCompleted(d.Job.ID); err != nil {
        return d.Nack(true)
    }
    return d.Ack()
}
Enter fullscreen mode Exit fullscreen mode

The completed marker must be durable and written in a transaction with any result that needs to be observed later. The check makes a redelivered message a safe no-op, which is the practical answer to at-least-once delivery. Your mileage may vary on the limit of five attempts; choose it from downstream recovery time and the cleanup SLO, then record the reason in the runbook.

Keep delay outside the hot receive loop. Exponential backoff with jitter stops every worker from retrying at once, while a maximum attempt count prevents one invalid tenant record from consuming capacity forever.

What belongs in a dead-letter queue?

A dead-letter queue is a quarantine lane, not a trash can. Preserve the original job ID, failure class, last error, attempt count, and first-seen timestamp. Operators also need a replay policy that says which fixes make replay safe.

Separate permanent validation failures from temporary dependency pressure. A malformed payload belongs with the producer that created it. A 429 should wait according to the provider’s guidance. Immediate nack-and-requeue loops can starve unrelated cleanup work and make queue latency look healthy while useful jobs wait.

Consider a scheduled run that emits 600 tenant jobs at 02:00. The first five workers may all receive the same dependency throttle response, requeue immediately, and then pick those same jobs again before any other tenant is served. The queue appears active, acknowledgements continue to arrive for unrelated messages, and a dashboard based only on depth stays green. An attempt counter, a delayed retry lane, and an oldest-job-age alert expose the pattern; the dead-letter record then preserves enough context to decide whether the tenant data or the dependency needs attention.

The catch is operational ownership. Every dead-letter item needs retention, access control, and someone who can investigate it. A small team may choose a managed broker; a regulated team may need self-hosted storage and an auditable replay tool. Neither choice removes the runbook, and a queue is not suitable when nobody owns upgrades, retention, and recovery.

Capacity planning: latency, cost, and the false-positive tax

Treat this as a capacity calculation. Estimate arrival rate in jobs per minute, service time in seconds per job, and a burst multiplier for scheduled runs. If one worker handles two jobs per second and a schedule emits 600 jobs at once, the ideal drain time is about five minutes before retries or downstream throttling. Set concurrency from the dependency’s limit, then add workers until oldest-job age has headroom.

More workers reduce latency but increase broker connections, downstream load, and spend. Fewer workers cost less until oldest-unprocessed age breaches the SLO; then the cost appears as an incident and manual cleanup. A threshold that is too low pages on every harmless burst. One that is too high hides a stuck consumer. I would start with a burn-rate alert and a small warning window, then tune it against two weeks of actual schedule variance.

For modest throughput, strict ordering, and a team that already operates a relational database, a database-backed job table can be the better fit. Choose a broker when independent worker scaling, backpressure, or durable fan-out matter. The decision is about the failure you are prepared to operate, not a queue brand.

Decision Queue broker Database-backed jobs
Burst latency Independent consumers can drain faster Polling and database load bound throughput
Operating cost Broker capacity plus worker capacity Database capacity plus polling overhead
Failure handling Ack/nack and dead-letter semantics vary Leases, retries, and cleanup are explicit
Portability Protocol and delivery semantics differ SQL is portable; tuning still varies

References

Further reading

Top comments (0)