DEV Community

Trkfpn392751
Trkfpn392751

Posted on

Node.js Background Queue for Auditable Scheduled Cleanup Retries

#sre #scheduling #node

Short answer: use the scheduler as a trigger, then put a uniquely identified cleanup command in durable storage. Let a worker own leases, idempotent side effects, bounded retries, and a reviewable dead-letter state. That is the simplest setup I trust when missing a cleanup run has consequences.

I arrived at that boundary after being paged for missed jobs. The timer had fired, the process had stayed healthy, and the queue looked quiet. The actual cleanup had not completed. I followed one item from the scheduled trigger through the producer, lease claim, dependency call, and acknowledgement, expecting to find a dramatic outage. Instead, each component had reported a locally reasonable result: the trigger had emitted work, the dependency had accepted a request, and the handler had returned without an exception. No component had recorded the business fact we needed, which was that the targeted records had reached the intended generation. The incident was a reminder that “the schedule ran” and “the data reached its intended state” are different facts. It also changed the review question I ask now: where, exactly, is completion proven, and what durable state lets an operator distinguish a delayed attempt from a completed effect?

No silent success.

What should a simple scheduled cleanup queue guarantee?

Start with a contract small enough to test. A scheduled scan emits a stable job ID, such as tenant-42/2026-08-07/policy-3, and records the work before reporting success to the scheduler. A worker claims the row or message with a lease, performs an idempotent operation, and acknowledges only after that operation succeeds.

The state machine is deliberately plain: ready -> leased -> succeeded, with leased -> ready for a retryable failure and leased -> dead for a permanent or exhausted failure. Lease expiry returns abandoned work to ready. Every transition is conditional on the current state and lease owner, so a slow worker cannot acknowledge work that another worker has reclaimed. In one review, this detail exposed why a superficially tidy retry loop was unsafe: the loop caught a dependency timeout, waited, and then returned the final error to a wrapper that treated any returned value as a completed handler invocation. The queue acknowledged the message even though the deletion had never been verified. The repair was not a larger retry library. We made completion an explicit queue operation and made every other outcome carry a state transition, a reason class, and a next action. That made the postmortem legible and gave the on-call a replayable record instead of a quiet green metric.

Keep the clocks separate. The calendar clock decides when to enqueue a scan. The delivery clock decides when a failed item may be tried again. If the next daily scan is also your retry mechanism, a transient outage can turn into a 24-hour delay, while overlapping scans can produce duplicate effects.

One sentence matters most: delivery may repeat, but the effect must converge.

For a Node.js producer, this contract does not require a queue-specific SDK. Enqueue through the system's normal HTTP or database boundary, pass the stable ID and cleanup generation, and keep payloads small. The worker language is an implementation choice; the acknowledgement semantics are not.

How do retries and dead letters change the cleanup architecture?

Retry policy belongs beside the job, not inside an opaque helper. Store the attempt number, not_before timestamp, and a reason class. Classify dependency timeouts, rate limits, and temporary unavailability as retryable only when repeating the operation is safe. Validation failures, authorization failures, and an expired policy should go straight to a terminal review state.

Bound both attempts and age. A job that can retry forever is an outage amplifier, and a job that retries after its data contract has changed may perform the wrong cleanup. Exponential backoff with jitter reduces synchronized load; the exact ceiling depends on the downstream service and the retention promise. I'm not sure there is one universal number that works for a small database and a multi-tenant object store.

Dead letters are an operator workflow, not a trash can. Record the stable job ID, attempt, reason class, timestamps, and a redacted summary. Define who may replay an item, whether replay keeps the same ID, and how a changed policy is represented. A replay should be an explicit state transition with an audit record.

The useful alerts are boring: age of the oldest ready job, lease-expiry rate, retry volume by reason, and dead-letter growth. Worker error count alone can be zero while a backlog quietly ages. I want the dashboard to answer “what is stuck, why, and what happens if I replay it?”

Choosing an execution model without hiding the trade-off

The least complex option depends on the consequence of a missed run. I use this comparison in design reviews:

Model Restart recovery Retry and dead letters Good fit Main trade-off
In-process timer Only if a later scan reconstructs work Application-owned Disposable cache eviction Timing and recovery live in one process
Repository-hosted schedule Run history survives the process Must be implemented by the workflow or app Low-frequency maintenance with tolerant timing Runs can be delayed; default-branch rules apply
Database job table Rows survive worker restarts Explicit timestamps and terminal states Teams already operating a transactional database Polling, leases, indexes, and replay UI are yours
Durable queue plus workers Delivery state survives restarts Broker policy plus job metadata Variable load and independent worker scaling More components and operational surface

A repository-hosted schedule is a trigger, not a precision clock. Its documentation notes that scheduled workflows may be delayed during high load, especially around the start of an hour, and that they run from the latest commit on the default branch. That is acceptable when the workflow only enqueues durable work. It is risky when the workflow performs the entire cleanup and a deadline is strict.

A database table is often the most understandable middle ground. Claim rows with a lease, index the next eligible time, and retain terminal records long enough to investigate. The catch is ownership: if nobody will maintain lease recovery, replay authorization, and backlog visibility, the “simple table” is an undocumented queue.

A worker path that makes acknowledgement boring

The handler below keeps business completion separate from delivery mechanics. Complete is reachable only after the idempotent cleanup effect returns success. The queue could be a table or a broker; the interface leaves that decision outside the handler.

package cleanup

import (
    "context"
    "errors"
    "fmt"
    "time"
)

type Job struct {
    ID          string
    Generation  int64
    Attempt     int
    MaxAttempts int
}

type Queue interface {
    Complete(context.Context, string) error
    Retry(context.Context, string, time.Time, string) error
    DeadLetter(context.Context, string, string) error
}

type Cleaner interface {
    DeleteGeneration(context.Context, int64) error
}

type RetryableError struct{ Err error }

func (e RetryableError) Error() string { return e.Err.Error() }
func (e RetryableError) Unwrap() error { return e.Err }

func Handle(ctx context.Context, q Queue, c Cleaner, job Job, now time.Time) error {
    if job.ID == "" || job.Generation <= 0 || job.MaxAttempts <= 0 {
        return fmt.Errorf("invalid cleanup job")
    }

    err := c.DeleteGeneration(ctx, job.Generation)
    if err == nil {
        return q.Complete(ctx, job.ID)
    }

    var retryable RetryableError
    nextAttempt := job.Attempt + 1
    if errors.As(err, &retryable) && nextAttempt < job.MaxAttempts {
        return q.Retry(ctx, job.ID, now.Add(retryDelay(nextAttempt)), "retryable_dependency")
    }
    return q.DeadLetter(ctx, job.ID, "cleanup_not_completed")
}

func retryDelay(attempt int) time.Duration {
    delay := time.Second << min(attempt, 8)
    if delay > 5*time.Minute {
        return 5 * time.Minute
    }
    return delay
}
Enter fullscreen mode Exit fullscreen mode

Test the effect twice against the same fixture. Then test lease loss between deletion and Complete; redelivery should observe the completed generation and leave the same end state. Use a fake clock to assert retry timing and terminal transitions instead of sleeping in tests. A killed worker must be recoverable by lease expiry, and an exhausted job must appear in the operator view with a safe reason.

If cleanup fetches a user-supplied URL, queue correctness is not enough. Apply the OWASP SSRF guidance at the network boundary: prefer an allowlist, validate the resolved destination, and control redirects so validation cannot be bypassed. Do not put credentials or raw response bodies in a dead-letter record. A poison job should be inspectable without becoming a second data leak.

When is this design the wrong fit?

This setup is not suitable when cleanup is genuinely disposable and a fresh scan can cheaply reconstruct every missed item. Stick with an in-process timer for an evictable cache when duplicate or missed eviction has no correctness impact. Use a repository-hosted schedule when timing can drift and durable run history is enough.

The opposite boundary matters too. Do not use this small queue as a substitute for a workflow engine when work spans human approvals, waits days between steps, or needs compensating actions across several systems. A queue delivers commands; it does not automatically provide a comprehensible long-running process model.

Before rollout, prove three things in a staging environment: duplicate delivery converges, a killed worker's lease is recoverable, and exhausted work is replayable by an authorized operator. Start with one worker, watch oldest-ready age and downstream rate limits, then raise concurrency. The safest schedule is the one that can be rerun without creating a new, unrelated wave of destructive work.

References

Further reading

Top comments (0)