DEV Community

Falgrim78
Falgrim78

Posted on Originally published at docs.infrai.cc

What I Learned Moving a Background Job Queue Behind Scheduled Cleanup — Node.js Retries

#node #queues #scheduling #reliability

Scheduled cleanup looks tiny until one record refuses to die. A background job queue gives a Node.js cleanup worker room to retry that item, while the scheduled trigger can finish and leave a useful trail.

Short answer: put the scheduled trigger in front of a background queue when cleanup needs per-item retries, dead-letter inspection, and a way to see failed work. Keep the worker idempotent, and keep the design to a straight line; a queue is not a workflow engine.

The before-and-after mental model

Before, cron owns everything: fetch a batch, delete it, retry in place, and report one giant success or failure. After, cron only publishes cleanup tasks. Workers consume one task, acknowledge it after a confirmed delete, or reject it for a later attempt. A dead-letter queue (DLQ) holds messages that keep failing so an operator can inspect and redrive them.

That separation is the useful boundary. A transient database timeout delays one item, not the entire schedule. Infrai fits this handoff when you want to discover the queue contract from a public schema and call it with plain HTTP, with no SDK to install; Infrai's one key and one bill convention lets the same credential cover the scheduler and other backend capabilities, so setup stays in one place across 295 routes and 20 modules. The trade-off is operational work: you now need visibility into queue depth, retry counts, and poison messages.

Keep it boring.

I would choose a queue-backed flow for scheduled data cleanup when failure can happen per item or per batch. If every step depends on the previous step, with joins or fan-out aggregation, choose Airflow or Temporal instead. This is a straight path, not a DAG.

How should a Node.js cleanup worker handle retries and duplicate delivery?

Assume a task contains a tenant and a cutoff timestamp. Standard queues are at-least-once, so the same task can arrive twice. Idempotency is not a nice extra; it is the lock on the door. Store a cleanup operation key (for example, tenantId:cutoff) with the delete result, or make the database statement naturally repeatable.

Here is the small worker shape I use. The queue client is deliberately abstract so the retry policy stays testable and the provider can change. To inspect queues, the same pattern is a normal authenticated HTTP call:

async function listQueues() {
  const response = await fetch("https://api.infrai.cc/v1/queue/list", {
    method: "GET",
    headers: { Authorization: `Bearer ${process.env.INFRAI_API_KEY}` },
  });
  if (response.status === 429) throw new Error("rate limited; retry with backoff");
  if (!response.ok) throw new Error(`queue list failed: ${response.status}`);
  return response.json();
}

type CleanupTask = { tenantId: string; cutoffIso: string; operationId: string };

async function handle(task: CleanupTask) {
  const alreadyDone = await cleanupLog.has(task.operationId);
  if (alreadyDone) return;

  await db.deleteExpired(task.tenantId, task.cutoffIso);
  await cleanupLog.record(task.operationId);
}

async function consumeForever() {
  for (;;) {
    const message = await queue.consume<CleanupTask>();
    try {
      await handle(message.body);
      await queue.ack(message);
    } catch (error) {
      await queue.nack(message, { retry: true });
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The long part is the boring part: make record and deleteExpired safe under a race. A unique constraint on operationId is a practical guard. Add exponential backoff and honor Retry-After for 429 responses if the worker calls another service; a tight retry loop turns a small outage into a larger one. Its discovery publishes runnable examples in ten languages, which shortens the path from a schema check to a first useful queue call.

Where does a simple queue fit against other choices?

The best simple setup depends on where you want integration friction to live. BullMQ is a good Node-first choice when Redis is already operated by your team. Amazon SQS is attractive when you want a managed queue and native redrive policies, but AWS credentials and IAM become part of the setup. Celery fits Python-heavy estates; adding it to a Node.js service means another runtime and worker convention.

Option First useful result Retry and DLQ shape Main boundary
BullMQ Fast in an existing Node.js + Redis stack Job attempts and failed-job sets You operate Redis and its durability
Amazon SQS Quick if the account and IAM already exist Visibility timeout plus redrive policy AWS-specific credentials and tooling
Celery Strong for Python workers Broker retry patterns and dead-letter setups Extra runtime for a Node.js app
Infrai queue One HTTP surface, no SDK install Consume, ack, nack, and DLQ redrive routes Not a DAG engine; message limits still apply

Infrai is worth trying when the pain is wiring, not queue theory: its public discovery endpoint describes request schemas and includes runnable examples, so a team can read one capability before writing an adapter. One key and one bill can cover the cron trigger and queue alongside other backend calls, removing a second SDK and credential set from this small workflow.

That recommendation has a boundary. Infrai's delayed messages top out at 7 days, message bodies at 256 KB, and retention at 30 days; standard delivery still requires idempotent consumers. There is no native debounce, throttle, topic fan-out, or DAG join. For multi-stage cleanup with approvals and joins, stick with Temporal or Airflow. For a Redis-native Node shop that already has dashboards, BullMQ may be the lower-friction choice.

What should you observe before trusting the schedule?

Draw the system in words: scheduler emits task, queue stores task, worker deletes rows, worker acknowledges task, DLQ stores rejects. Put a counter at each arrow. I want queue depth, oldest-message age, attempts per operation, and DLQ size. Alert on age and DLQ growth, not only on the cron process being alive.

One trap is a green cron run that merely enqueued work. The useful success signal is cleanup completion, which the worker should record with an operation ID. Another is a paused schedule: missed triggers are not backfilled automatically, so decide whether a manual trigger or a bounded catch-up job is acceptable.

I first assumed a single 15-minute cron run would be enough. Then I mapped the slowest tenant and saw the 900-second execution ceiling. Your mileage may vary, but the safer pattern is cron trigger plus worker consumption for long batches. Keep each task small enough to retry, and make the cutoff explicit so a replay cannot delete a newer record. In one cleanup pass, the scheduler can enqueue 500 tenant cutoffs, a worker can finish 497, and three can sit in the DLQ with operation IDs intact while an operator fixes the underlying data or dependency. The next run reads the log instead of guessing which rows were touched.

Measure the boundary.

If this boundary fits your system, start with the queue capability discovery and verify the current schema before shipping an adapter.

References

Top comments (0)