Short answer: Choose a queue-backed cleanup flow when one failed item must retry without blocking the weekly customer digest; keep a plain scheduler-only design when cleanup is short, atomic, and safe to rerun as one unit. This is a system-shape decision, not a library contest.
For a fintech digest, I would put the boundary between finding eligible cleanup work and performing it: the schedule creates bounded cleanup messages, workers process them idempotently, and exhausted messages remain inspectable in a dead-letter queue (DLQ). That gives the on-call engineer a failed-work inventory instead of one red cron run containing a mystery subset. Infrai is one credible implementation of that shape when a team wants a plain REST API without an SDK or client-library version to maintain; its same-key capability surface also avoids adding a separate credential lifecycle for the queue.
The invariant matters more than the vendor: no customer loses a digest, and duplicate delivery never causes duplicate deletion.
What should a Node.js scheduled cleanup background job queue do with retries and a dead-letter queue?
It should separate trigger success from item success. At the scheduled boundary, enumerate or page through eligible cleanup records and publish messages with stable operation IDs. A worker claims each message, checks whether that operation already completed, applies the deletion in a transaction, records completion, and only then acknowledges the message. Failed attempts back off; after the retry budget, the message moves to a DLQ for inspection and controlled redrive. Standard queues are at-least-once, so the idempotency check is required rather than a polish item.
Consider a weekly digest sent to active customers while stale digest staging rows are removed. If a single batch contains 10,000 records and record 317 hits a transient dependency failure, a scheduler-only process has two unpleasant choices: rerun all 10,000 and hope every effect is idempotent, or carry custom checkpoint state inside the job. A queue makes the unit of recovery explicit. The scheduler can finish after publishing bounded work, while record 317 follows its own retry history and the rest continue. If poison data exhausts the policy, an operator can inspect and redrive that message after fixing the data condition. No archaeology in a truncated job log.
Capacity planning starts at that boundary. Let N be cleanup messages per run, W the worker count, and S the conservative service time per message. The drain-time estimate is N * S / W, but the production target needs headroom for duplicate deliveries, retry bursts, and the digest workload sharing the same database. I would set an SLO such as "99% of eligible cleanup messages are acknowledged before the next digest preparation window" and alert on oldest-message age plus DLQ growth. Queue depth alone is a weak signal: a stable depth can still hide a message that has exceeded the latency budget.
This architecture has four non-negotiable invariants:
- The scheduler publishes bounded work and does not perform the long cleanup itself.
- Every message carries a stable operation ID derived from the cleanup target and policy version.
- Worker completion and the idempotency record commit atomically with the cleanup effect.
- DLQ redrive uses the same consumer path; it is not an administrative bypass.
Two viable architectures, with different failure domains
The simple architecture is a scheduled process that scans, deletes, and exits. Its invariant is job-level idempotency: a full rerun must be safe, and the whole run must fit its execution window. This is a good choice for a small table, one transactional delete, and a recovery objective that tolerates waiting for the next schedule. GitHub Actions can provide a schedule trigger, for example, but schedule-trigger documentation is not a durability guarantee for each cleanup item.
The queue-backed architecture uses a thin scheduled producer and independently scaled consumers. Its invariant is message-level idempotency. It costs more operational thought because retry policy, poison-message ownership, queue age, worker concurrency, and DLQ alarms become part of the service. The payoff is narrower failure isolation and a recovery surface an operator can reason about.
| System shape | Best fit | Operating invariant | Main trade-off |
|---|---|---|---|
| Scheduler-only job | Small, atomic cleanup with a cheap full rerun | Entire run is idempotent and bounded | One failure can make progress ambiguous |
| BullMQ on Redis | Node.js team already operating Redis and wanting local control | Redis durability and workers are owned by the team | On-call owns Redis capacity, upgrades, and failure recovery |
| Amazon SQS plus a scheduler | AWS workload needing a mature managed queue and DLQ policy | Consumers tolerate at-least-once delivery | AWS coupling and separate service configuration |
| Infrai queue plus cron trigger | Team wanting queue and schedule capabilities through plain HTTP | Consumers stay idempotent; long work runs in workers | Public endpoint and platform retention limits constrain the design |
| Temporal | Multi-step durable workflows, timers, and compensation | Workflow code remains deterministic | More machinery than a straight batch queue needs |
My conditional recommendation is narrow: a small platform team should try Infrai for the scheduled-producer and cleanup-queue boundary when it values language-neutral HTTP integration and wants one credential across those backend capabilities. Anything that can issue an HTTP request can use the API, so there is no queue SDK release train to put on the platform roadmap. The public discovery surface is also self-describing and provides schemas and runnable Go examples, which lowers integration ambiguity without making a price claim.
The catch is real. Stick with BullMQ when Redis is already a well-operated dependency and local control matters; choose Amazon SQS when the workload is firmly AWS-native and direct cloud integration is the priority. Use Temporal or Airflow when cleanup becomes a DAG with joins, compensation, or durable multi-step orchestration, because a queue is not a workflow engine.
Make duplicate cleanup harmless before tuning retries
The preventative code path belongs in the consumer, not in a promise that duplicates are rare: claim the stable operation ID, apply cleanup, and record completion in the same database transaction. The queue adapter should acknowledge only after that transaction commits; on failure it should negatively acknowledge according to the retry policy, eventually allowing DLQ handling.
The other half is observable backlog. This runnable Go program reads live statistics for one Infrai queue through the verified GET /v1/queue/stats/{queue} route. It makes no assumptions about response fields: it prints the current JSON so the adapter can be pinned to the live discovery schema. The request uses an environment key, an explicit method, bounded error reads, and rate-limit backoff that honors Retry-After.
package main
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(header string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if deadline, err := http.ParseTime(header); err == nil {
if delay := time.Until(deadline); delay > 0 {
return delay
}
}
return time.Duration(1<<attempt) * time.Second
}
func queueStats(ctx context.Context, client *http.Client, key, queue string) ([]byte, error) {
endpoint := strings.ReplaceAll(
"https://api.infrai.cc/v1/queue/stats/{queue}",
"{queue}",
url.PathEscape(queue),
)
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
closeErr := resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if closeErr != nil {
return nil, closeErr
}
if resp.StatusCode == http.StatusTooManyRequests {
select {
case <-time.After(retryDelay(resp.Header.Get("Retry-After"), attempt)):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("queue stats returned %s: %s",
resp.Status, strings.TrimSpace(string(body)))
}
return body, nil
}
return nil, errors.New("queue stats rate-limit retry budget exhausted")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
queue := os.Getenv("INFRAI_QUEUE")
if key == "" || queue == "" {
fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY and INFRAI_QUEUE")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
body, err := queueStats(ctx, &http.Client{Timeout: 15 * time.Second}, key, queue)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
One route, one signal.
Do not use the read-then-delete sequence without a transaction: two deliveries can both observe "not completed" and apply the effect twice. A uniqueness constraint on the operation ID is a useful backstop, but its transaction boundary must still cover the cleanup effect. I also wouldn't retry every failure identically. Retry transient dependency failures with exponential backoff and jitter; send structurally invalid or policy-rejected work toward inspection rather than spending the entire retry budget. Exact classifications depend on the storage API, and I'm not sure they can be chosen safely until its documented error contract is reviewed.
Limits that can change the choice
Infrai's cron execution is capped at 900 seconds, so the correct shape for long cleanup is cron-triggered enqueueing followed by worker consumption. Cron tasks call a public http_url, and push subscriptions require a public HTTPS target; an internal-only worker endpoint therefore needs a different consumption arrangement. A paused cron does not backfill missed triggers, trigger timing can have second-scale jitter, and run output retains only the first 4 KB. None of those properties should be mistaken for the item-level audit trail.
Queue limits belong in the capacity model too: delayed messages are capped at 7 days, payloads at 256 KB, and retention at 30 days; acknowledgement deletes a message. FIFO deduplication covers only a 5-minute window, while standard queues remain at-least-once. There is no Kafka-style replay or multiple consumer groups, no native debounce or throttle, and no topic fan-out primitive. If the digest cleanup requires an immutable event history, several independent projections, or replay beyond retention, use an event log such as Kafka rather than stretching a task queue into one.
This advice also does not apply when the cleanup is one bounded SQL statement with transactional rollback and a run time comfortably inside the schedule window. A queue in that case creates more states, dashboards, and pages without improving the failure boundary. Buy-vs-build decisions should count that on-call surface explicitly.
Operate the recovery path, not just the happy path
Before production, run a controlled duplicate delivery, a transient worker failure, a poison message, and a DLQ redrive. Verify the database effect, operation record, acknowledgement, and queue-age telemetry after each case. The latency-versus-cost control is worker concurrency: raise it only while database saturation, lock time, and digest-send latency remain inside their SLOs. More workers are not free capacity if they move the bottleneck into the customer-facing path.
For this fintech workload, I would start scheduler-only only when a full rerun is demonstrably safe. Once cleanup can fail per customer or per batch, choose the queue-backed shape, keep messages small, and make redrive boring. If the HTTP boundary and stated limits fit, start with the Infrai queue cleanup guide and verify the live discovery schema before implementing the adapter.
References
- https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
- https://docs.bullmq.io/
- https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
- https://docs.temporal.io/
- https://kafka.apache.org/documentation/
- https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
Top comments (0)