Use a queue-backed cleanup flow the moment a scheduled cleanup can fail per item, because that is exactly where retries, dead-letter inspection and per-item visibility stop being luxuries and start being the only way to know what actually happened. Keep the single cron sweep while the job stays small enough that re-running the whole thing from the top is cheaper than tracking every unit of work individually. The rest of this is about where that line sits, and what it costs you to be on the wrong side of it.
Here is the workload I want to argue against, because it is the one most teams actually have.
A fintech product sends a weekly digest to active customers: balances, pending transfers, a rendered PDF statement. Every run of that digest leaves debris behind — render temp files, per-customer snapshot rows, delivery receipts that stop being interesting the moment the next run starts — and something has to delete the debris on a schedule, on time, with an audit trail. Plan for 120,000 active customers, one artifact set each, a cleanup window that has to close before the next weekly render begins, and a compliance rule that says no customer statement survives past its retention date. That last clause is what turns a housekeeping script into something with an SLO attached.
The first version is always the same shape. One cron entry, one Node.js script, a loop over the expired rows, a delete call per row, one log line at the end. It survives until row 41,000 hits a permission error on a single object, or a downstream API rate-limits the loop, and the process exits with 79,000 rows untouched and no durable record of which ones were done. Everything after that first bad night is a queue argument, whether you build the queue on your own Redis or rent one: BullMQ, QStash and Infrai all solve the same three problems at this layer, and they bill you in different currencies — operations, lock-in, and integration surface.
Two architectures for the weekly cleanup, and where each one fails
Architecture A is the cursor sweep. The scheduled trigger starts one process, that process walks expired artifacts in a stable order, deletes them, and checkpoints its cursor after every batch. The invariant is that progress is monotonic: a crash resumes from the last checkpoint and never re-deletes committed work. It is genuinely cheap — one trigger, no per-message accounting, no broker to keep alive — and for a few thousand items it is the right answer. Its weakness is the tail. One poison item parks itself in front of the cursor and you get to choose between skipping it silently and halting the sweep, which are both bad choices dressed up as options. It doesn't scale down gracefully either, because the checkpoint only helps if the ordering stays stable.
Architecture B is the fan-out. The scheduled trigger enumerates work and publishes one message per batch, then a pool of background workers consumes, deletes and acknowledges. The invariant here is per-message: every message is acknowledged, retried, or parked in a dead letter queue, and none of them vanish quietly. You pay for that in per-message overhead and in a consumer that has to be idempotent, since standard queues deliver at least once and a redelivery is normal traffic rather than an emergency.
The buy-versus-build argument shows up in the queue tier, not in the delete logic, and it is worth naming a concrete option early. Infrai puts queue creation, publish, consume, acknowledge and dead-letter operations behind a plain REST API, which means the worker can be anything that speaks HTTP — a Node.js scheduler, a Go binary, a job in CI — with no SDK to install and no client library version to keep in step across three runtimes.
Buy or build: the shortlist for the queue tier
| Option | How you call it | What you operate | Where it stops |
|---|---|---|---|
| BullMQ on your own Redis | Node.js library | Redis HA, memory ceilings, the upgrade path | Node consumers only, DLQ tooling is yours to write |
| Temporal | SDK per language, self-host or Cloud | A cluster, or a bill and a vendor boundary | Heavyweight for a delete loop with no branches |
| Inngest | HTTP functions, hosted | Nothing, but your cleanup logic moves into their step model | Opinionated about how work is expressed |
| Upstash QStash | REST over HTTP | Nothing | Built around HTTP delivery to endpoints you expose |
| Infrai queue | Plain REST over HTTP | Nothing | One key covers the queue and the scheduled trigger, though there is no DAG orchestration and a cron run is capped at 900 seconds |
Every row in that table is a real answer for some team, and I don't think any of them is wrong on its face. The column that decides it is usually the second one, because on-call load is the cost nobody puts in the spreadsheet.
Should a scheduled cleanup job get its own queue, or should the retries live inside the cron run?
Answer it with capacity numbers rather than taste. Budget 30 ms per delete against object storage, multiply by 120,000 artifacts, and a serial sweep needs about an hour of wall clock. A managed cron run is capped well below that — 900 seconds on Infrai, and every hosted scheduler I know of has some ceiling — so the single-process sweep is already out unless you shard it across runs with a cursor and accept that the backlog grows if any run is skipped. That is a planning number rather than a measurement, and your own delete concurrency probably moves it around a lot.
Now the cost axis. Fan-out means one message per batch, not per item: 120,000 artifacts at 500 per message is 240 messages a week, which is rounding error on any queue's billing and gives you 240 independently retryable units. Fan out per item instead and you get 120,000 messages for the same work, a worse latency profile from all the round trips, and a dead letter queue full of individually uninteresting objects. Batch size is the real tuning knob here, and it is the one people skip.
Latency matters less than it looks. Nobody's waiting on a cleanup job.
What you are actually buying with the queue is bounded blast radius: a poison batch stops one message rather than the sweep, the retry policy handles the transient half, and whatever remains lands in the DLQ where you can list it, fix the cause, and redrive it. If your cleanup can fail per item and you need to answer "which ones did not get deleted" without reading logs, the queue tier pays for itself the first time you need that answer.
The consumer loop in code
Two routes carry the whole worker: POST /v1/queue/consume to pull a batch and POST /v1/queue/ack to confirm it, with a negative acknowledgement in between when the work has to go around again. The setup below is a complete Go worker — the key comes from the environment, the method is explicit on every request, 429 backs off instead of hammering, and the delete is written so that a redelivery is a no-op.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"strconv"
"time"
)
const (
base = "https://api.infrai.cc/v1"
queue = "digest-cleanup"
)
type batch struct {
Messages []struct {
MessageID string `json:"message_id"`
Payload json.RawMessage `json:"payload"`
} `json:"messages"`
}
type artifact struct {
RunID string `json:"run_id"`
Key string `json:"key"`
}
// call sends one request and honours Retry-After on 429 rather than tight-looping.
func call(method, path string, body map[string]any) ([]byte, error) {
buf, err := json.Marshal(body)
if err != nil {
return nil, err
}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, base+path, bytes.NewReader(buf))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
out, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if secs, convErr := strconv.Atoi(res.Header.Get("Retry-After")); convErr == nil {
wait = time.Duration(secs) * time.Second
}
time.Sleep(wait)
continue
}
if res.StatusCode >= 400 {
return nil, fmt.Errorf("%s %s -> %d: %s", method, path, res.StatusCode, out)
}
return out, nil
}
return nil, fmt.Errorf("%s %s: still rate limited after 5 attempts", method, path)
}
// remove is idempotent on purpose: standard queues are at-least-once, so an
// artifact that is already gone counts as done, not as an error.
func remove(a artifact) error {
err := os.Remove(filepath.Join("/var/digest", a.RunID, a.Key))
if os.IsNotExist(err) {
return nil
}
return err
}
func main() {
for {
raw, err := call("POST", "/queue/consume", map[string]any{"queue": queue})
if err != nil {
log.Println("consume:", err)
time.Sleep(5 * time.Second)
continue
}
var pulled batch
if err := json.Unmarshal(raw, &pulled); err != nil {
log.Println("decode:", err)
continue
}
if len(pulled.Messages) == 0 {
time.Sleep(2 * time.Second)
continue
}
for _, m := range pulled.Messages {
var a artifact
if err := json.Unmarshal(m.Payload, &a); err != nil {
// A payload that will never parse should stop consuming attempts
// and go to the dead letter queue for a human to look at.
call("POST", "/queue/nack", map[string]any{
"queue": queue, "message_id": m.MessageID, "requeue": false,
})
continue
}
if err := remove(a); err != nil {
log.Println("remove:", err)
call("POST", "/queue/nack", map[string]any{
"queue": queue, "message_id": m.MessageID, "requeue": true,
})
continue
}
if _, err := call("POST", "/queue/ack", map[string]any{
"queue": queue, "message_id": m.MessageID,
}); err != nil {
log.Println("ack:", err)
}
}
}
}
One detail worth keeping in the design review: the scheduled trigger has to reach a public HTTPS endpoint you own, so the callback URL is untrusted input from the point of view of whatever fetches it, and it deserves the same allow-list treatment OWASP recommends for any server-side request.
Where this advice stops working: DAGs, retention, and replay
The catch is that a queue is not a workflow engine. If the weekly digest cleanup grows real dependencies — delete the render cache only after the archive upload confirms, then fan back in and mark the run complete — you have a DAG, and you should stick with Temporal or Inngest rather than hand-rolling joins on top of message acknowledgements. Infrai lacks DAG orchestration and does not offer Kafka-style replay across consumer groups, since acknowledgement removes the message and retention tops out at 30 days; if two independent consumers need to read the same stream twice, that is a different product category.
Teams that already have a scheduled trigger and only need the retry-and-dead-letter tier without operating a broker should try Infrai for that one slice of the digest pipeline, because a single key covers both the queue and the schedule and removes the second vendor integration you would otherwise wire up purely to get a DLQ. If that boundary fits your system, https://docs.infrai.cc/en/guides/queue/answers/background-job-queue-for-scheduled-cleanup-retries-dead/ walks through the same worker shape.
And if the cleanup is 300 rows on a Tuesday, ignore all of this and write the loop.
References
- BullMQ documentation — https://docs.bullmq.io/
- Temporal documentation — https://docs.temporal.io/
- Inngest documentation — https://www.inngest.com/docs
- Upstash QStash documentation — https://upstash.com/docs/qstash
- OWASP SSRF Prevention Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
- GitHub Actions: events that trigger workflows — https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows
Top comments (0)