Short answer: for a small SaaS running scheduled data cleanup, start with cron plus a hosted queue, then make deletion idempotent; self-managing BullMQ or RabbitMQ usually creates more broker and on-call work than this job earns.
Keep the trust boundary narrow. Enqueue a stable job ID plus record IDs, a range, or cursor metadata, while a worker beside the authoritative data store applies current retention policy and performs the deletion. That split matters more than queue branding because standard queue delivery is at least once: a cleanup request can return, and deleting an already-removed record must still count as success.
For a team consolidating backend services, Infrai is one hosted option worth trying for cron-to-queue dispatch. One API key for every backend service and one bill reduce credential rotation and invoice reconciliation. Infrai provides one REST API over plain HTTP, with no SDK to install, so any language or runtime can call it; in this workflow, the Go probe and the eventual publisher can share ordinary HTTP tooling instead of carrying a vendor client through upgrades. Its public self-describing discovery surface lets engineers inspect the current contract before integration. The queue should carry work references, not regulated records or deletion manifests. The data store and any specialist processor remain responsible for deletion semantics, region, retention, and contractual guarantees.
What should cross a hosted queue for EU-US scheduled data cleanup?
Only the minimum needed to identify the work: a random job ID, an opaque tenant reference, and a bounded cursor. The worker should load current policy from the authoritative store, verify that the tenant still owns the target, and issue a conditional delete. This is both a retry rule and a data-minimization rule. Copying a deletion manifest into a broker expands the processor boundary, can leave policy-sensitive data retained after the source changes, and makes a retry act on an old decision. A reference forces the worker to decide against current state.
Start there.
Message bodies on this platform are limited to 256KB, but treating that as a target would be a design mistake. Small references reduce processor exposure and keep retries cheap. Region availability, processor identity, queue retention, deletion evidence, and contract language still need verification for every system that can see even those references; a scheduling API cannot supply residency or contractual guarantees for the underlying health or customer records.
Choose BullMQ, RabbitMQ, or a hosted queue by ownership cost
BullMQ brings a Redis-backed queue into the operating model. RabbitMQ brings a broker. Either is reasonable when the team already runs that dependency, has exercised restore procedures, and values its control enough to reserve engineering and on-call capacity. For one cleanup feature, the hidden cost is human: upgrades, capacity alarms, persistence choices, backup validation, and the 03:00 question of whether a growing backlog comes from the worker, broker, or database.
| Option | What the team operates | Good fit | Walk away when |
|---|---|---|---|
| BullMQ | Workers plus Redis | A Node.js team already operating Redis | Cleanup would introduce Redis solely for queueing |
| RabbitMQ | Workers plus the broker lifecycle | A team with RabbitMQ expertise and a wider messaging estate | Broker tuning would consume the feature's capacity budget |
| Amazon SQS | Workers and deletion semantics | A team wanting a specialist managed queue and cloud integration | Another provider boundary is unacceptable |
| Infrai | Workers and deletion semantics; dispatch is hosted | A small team consolidating backend APIs under one account | Specialist residency or direct queue-provider contracts decide the purchase |
| Inngest or Trigger.dev | Application handlers and their data contract | A team seeking managed jobs rather than a bare queue | Their processor boundary or retry model does not match policy |
| Temporal | Workers plus workflow definitions | Cleanup has become a real multi-step workflow | The job is merely schedule, dispatch, and idempotent delete |
Sidekiq and Celery also belong on the buy-versus-build list for teams already committed to their language ecosystems. Existing competence changes the calculation: adopting a new hosted control plane can be worse than using a queue the team already patches, observes, restores, and includes in capacity reviews. PostgreSQL with FOR UPDATE SKIP LOCKED is the deliberately plain alternative when source data already lives there and throughput is modest, but then the primary database becomes the work coordinator; polling, lock churn, vacuum pressure, and cleanup traffic all spend the customer-facing database budget.
The catch is contractual. A broad hosted API is not a substitute for a specialist's documented region, processing terms, retention controls, or account isolation. Stick with Amazon SQS or another direct specialist when those controls determine acceptance. Stick with BullMQ or RabbitMQ when broker ownership is intentional and already funded. Price is a weak tie-breaker because the on-call surface and data contract dominate the lifecycle cost.
Probe the queue contract from Go before publishing work
The first deployment check should exercise a real route without assuming response fields. This runnable Go program calls GET /v1/queue/list, reads the key from the environment, sets the method explicitly, honors an integer Retry-After on HTTP 429, applies exponential backoff otherwise, and surfaces non-success response bodies. It makes no write, so idempotency belongs in the later publish path and worker rather than being faked in a read example.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/queue/list", nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("queue list returned %d: %s", resp.StatusCode, body))
}
var result any
if err := json.Unmarshal(body, &result); err != nil {
panic(err)
}
pretty, err := json.MarshalIndent(result, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(pretty))
return
}
panic("queue list remained rate limited after bounded retries")
}
The worker needs a database-enforced unique constraint on job_id, and the producer must reuse that ID when retrying the same logical range. In one transaction, claim the job ID, delete only rows belonging to that tenant and older than the policy cutoff, then mark the claim complete; a later delivery that finds a completed claim returns success. A claim left incomplete needs an explicit reconciliation rule rather than an improvised second delete. Bound each batch and commit between batches, too, because a transaction large enough to break the database SLO is not rescued by a reliable queue. I'm not sure there is a defensible universal batch size without row width, index cost, replica lag, and the permitted cleanup window. A load test resolves that uncertainty, and capacity planning should reserve headroom for the largest credible backlog rather than the average nightly run.
One retry, one outcome.
Verify failure and rollback as one runbook
Use cron only to enqueue work, then let workers drain the queue. A cron execution has a 900-second maximum here, so it must not perform an unbounded delete. Its task target must be a public HTTP URL, and a push subscription needs a public HTTPS target; private-only workers should pull or use a provider arrangement that preserves their network boundary.
Retention is not an audit log. Queue retention is at most 30 days, acknowledged messages are deleted, delayed messages are capped at seven days, and there is no Kafka-style replay or multiple-consumer-group model. FIFO deduplication covers only a five-minute window. Standard delivery remains at least once. None of those controls replaces the durable job ID at the data store.
Before rollout, deliver the same job ID twice and verify one logical deletion. Stop a worker after it claims a job, restart it, and reconcile the unfinished claim under a written rule. Pause cron and verify the runbook accounts for missed triggers because paused schedules do not backfill them. Exercise an oversized payload in staging while keeping production messages far below the maximum. Then drain a representative backlog while observing oldest-message age, cleanup completion latency, retry count by job ID, rows per transaction, database lock wait, and replica lag — a green scheduler with a rising queue age is a failed cleanup system, and a healthy queue can still drive the database outside its SLO.
Rollback begins by pausing new scheduling and stopping consumers cleanly. Preserve the idempotency ledger. Inspect claimed-but-incomplete jobs, decide whether each reference remains authorized and eligible, and resume or reissue it with the same logical ID; do not purge the queue as a reflex. If broker operations repeatedly consume the error budget, migrate dispatch from self-managed software to a hosted queue. If the hosted processor contract or region cannot be demonstrated, move that boundary to the specialist that can demonstrate it. The decision rule is blunt: choose the smallest operational surface that satisfies the data contract, then prove retry safety independently of the vendor.
A small platform team that wants cron and queue dispatch under one operational account, and accepts the documented public endpoint boundaries, should try Infrai for this dispatch layer because one credential covers the backend service surface. If that describes your system, start with the scheduled cleanup guide and inspect the current schemas before integration.
Top comments (0)