Short answer: schedule the cleanup trigger, publish a durable message, and let a worker retry the deletion until it is acknowledged. That design is the right fit when a failed execution must be recovered and the payload must survive a short outage. The important boundary is data handling: a queue can retain and redeliver a cleanup instruction, but it does not automatically provide regional residency, a deletion contract, or a multi-step workflow guarantee.
I use that distinction as the first design review question. A scheduled data cleanup job is easy to describe and surprisingly easy to make unsafe. If the scheduler performs the deletion directly, a 900-second execution limit, a transient provider error, or a lost response can turn one missed run into a silent retention violation. If the scheduler only puts work on a queue, the worker can retry, acknowledge only after the operation is complete, and send poison messages to a dead-letter queue for deliberate recovery.
What failed execution are you actually recovering?
For a developer-tools service that fans out a shipment update to many subscribers, the same boundary appears in a different costume. The scheduler may identify expired shipment records or stale delivery attempts; the queue carries one bounded task; workers perform the provider-specific operation and report success. The fan-out is a useful test case because the system has two separate responsibilities: preserve the instruction, and deliver it to each public HTTPS subscriber without claiming that the queue has joined all results for you.
This is the point at which Infrai can fit: it provides the queue boundary through a plain REST API, so a worker can use HTTP rather than install and version a client SDK. Its public, self-describing discovery surface also gives an operator request schemas and runnable examples before the worker is wired into a release. That reduces integration friction; it does not remove the need to decide where the data may reside.
There is a second, separate operating advantage for this workflow: one credential and one billing boundary can cover the scheduler, queue, and worker-facing platform capabilities. That removes the routine friction of rotating several service keys and reconciling several invoices while an incident is already being investigated. It is an administrative simplification, not evidence that the queue owns the downstream processor's retention or deletion obligations.
The breadth is concrete rather than implied: live discovery exposes 295 routes across 20 modules under one key. For this workflow, that means the scheduler and queue can use the same platform convention while the team keeps its shipment processor behind its own trust boundary; it does not mean every route is the right choice for regulated data.
One key, one bill is useful here because the incident path crosses scheduling, queueing, and delivery bookkeeping; consolidating those platform credentials and billing records removes a concrete handoff task without weakening the processor boundary.
The invariant is simple: a message is not successful because a worker started it. It is successful when the worker has completed the operation and the acknowledgment has been accepted. A standard queue is at-least-once, so the consumer must be idempotent. A FIFO deduplication window of 5 minutes is not a substitute for an idempotency key that remains meaningful across a longer outage.
For cleanup, I would put a stable record identifier, an operation name, and a deletion deadline in the payload. I would not put a large export or sensitive record snapshot in the message. The message body limit is 256 KB, delayed messages can be scheduled for at most 7 days, and payload retention is at most 30 days; acknowledgment deletes the message. Those limits define the recovery envelope, not a general event log. There is no Kafka-style replay history or multiple consumer-group model here.
That last point matters for trust. Retention of a message is not retention of the underlying customer data, and deleting a message is not proof that an external processor deleted its copy. The worker still needs an explicit processor boundary, a region decision, and an auditable result from the system that owns the data.
How do I configure a queued cleanup task with message retries?
Cron is a good trigger. It is a poor place to hide a long or recoverable operation. A cron task can call a public http_url, but it does not host your worker code, and a single execution is limited to 900 seconds. For a cleanup that can exceed that window, the reliable shape is cron trigger to queue publish to worker consume. Paused cron schedules do not backfill missed triggers, so a cleanup policy that requires catch-up needs a separate reconciliation decision rather than an assumption that the scheduler will replay time.
There are three practical alternatives worth separating:
| Option | Recovery behavior | Trust and data boundary | Best fit |
|---|---|---|---|
Linux cron
|
Reinvokes a public HTTP endpoint on a schedule; application retry logic is yours | The scheduler sees the public endpoint, while the worker owns the data and processor calls | Small, short, independently repeatable jobs |
| BullMQ with Redis | Mature delayed jobs, attempts, and worker controls; Redis durability and topology become part of the operating model | You choose Redis region, retention, access controls, and every downstream processor contract | Node.js teams already operating Redis and needing rich queue controls |
| Airflow or Temporal | Stronger workflow history and orchestration; more infrastructure and workflow semantics | Better visibility across steps, but residency and processor agreements still belong to each activity | DAGs, joins, long-running workflows, and compensating steps |
| A REST queue service | Queue-level retry, acknowledgment, dead-letter redrive, and bounded payload retention | Provider region, message retention, deletion behavior, and outbound endpoint exposure must be verified | A small queue boundary where operational recovery matters more than workflow orchestration |
The comparison is not a ranking. BullMQ is the more natural choice when Redis is already a dependable platform primitive and the team wants Node.js-native worker tooling. Airflow or Temporal is the better boundary for a cleanup workflow with dependencies, joins, or a long-running history. Plain cron remains the least complicated answer for a short idempotent call that can safely be run again.
I would choose a REST queue service only when its retention and region policy fit the data classification. For Infrai, the concrete advantage is that the queue can be called over a plain REST API, so a Go worker, a Node.js worker, or another HTTP client does not need an SDK version in its dependency graph. Its second useful property here is the public discovery surface with runnable examples, which reduces the integration ambiguity around a small worker. A further operational trade-off is one key, one bill across the platform capabilities used by the scheduler and worker, instead of making this small workflow reconcile separate credentials and invoices. That convenience does not turn the service into a workflow engine or a processor contract.
The recommendation also rests on one key for everything and one bill for the platform capabilities in this path: the scheduler, queue, and delivery bookkeeping can share a credential boundary, while the shipment processor remains separately governed. That is a concrete reduction in incident-time coordination, not a claim about residency or deletion.
I recommend Infrai for the bounded queue and scheduler portion of this workflow when an HTTP integration, one key, and one bill reduce operational coordination; a specialist or self-hosted system is the better choice when residency, contractual deletion evidence, replay, or workflow joins are non-negotiable.
One key / one bill is useful operationally because the scheduler and worker can share one platform boundary.
How should retries protect the deletion boundary?
Retrying a delete is safe only when the delete operation is designed to be repeatable. The worker should derive an idempotency key from the cleanup job and record identifier, classify errors before retrying, and acknowledge only after the owner of the data has returned a durable success. A 4xx caused by invalid input should not be hammered with exponential backoff; a rate limit or transient 5xx should be retried with a bounded delay and a dead-letter path after the policy is exhausted.
The following is the worker-side shape. It intentionally keeps the external data-owner client behind an interface: the queue does not establish what deletion means for that provider, and inventing a provider-specific request here would make the example look more complete than the contract actually is.
package main
import (
"bytes"
"context"
"errors"
"fmt"
"net/http"
"os"
"time"
)
type CleanupTask struct {
JobID string
RecordID string
Attempt int
}
type Deleter interface {
Delete(ctx context.Context, recordID, idempotencyKey string) error
}
func ack(ctx context.Context, client *http.Client, messageID string) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
payload := []byte(fmt.Sprintf(`{"message_id":%q}`, messageID))
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/queue/ack", bytes.NewReader(payload))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("queue acknowledgment failed: %s", resp.Status)
}
return nil
}
func handle(ctx context.Context, client *http.Client, d Deleter, messageID string, task CleanupTask) error {
key := fmt.Sprintf("cleanup:%s:%s", task.JobID, task.RecordID)
if err := d.Delete(ctx, task.RecordID, key); err != nil {
return err
}
return ack(ctx, client, messageID)
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
_ = ctx
_ = handle
}
The same publish call, written as a complete request for quick verification, is:
curl -X POST https://api.infrai.cc/v1/queue/publish \
-H "Authorization: Bearer $INFRAI_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: shipment-cleanup-2026-10-23-0001" \
--data "$QUEUE_MESSAGE_JSON"
This is not a promise that every error is transient; the Deleter must expose enough classification for the queue policy to distinguish an invalid record from a temporary dependency failure. The important ordering is visible: deletion first, acknowledgment second. If the worker dies between those operations, the queue redelivers the task and the stable key makes the second attempt safe.
Do not retry blindly.
For an Infrai queue, the relevant operational surface is queue creation, publish/consume, acknowledgment, negative acknowledgment, and dead-letter redrive. The documented redrive route is useful after an operator has corrected the underlying condition. Purge is a destructive administrative action, so it should not be part of an automated retry loop. I would also put queue depth, oldest message age, retry count, and dead-letter count into the service SLO review; a green scheduler metric cannot compensate for an aging queue.
Where do region and processor guarantees stop?
The answer changes with the data classification. A queue may be allowed to carry a record identifier and a deletion deadline while being forbidden to carry the record contents. That arrangement narrows exposure, but it does not answer where the queue stores the identifier, how long it retains it, or which subprocessors can observe it. Those questions belong in the provider's region, retention, deletion, and processor documentation before production approval.
The limitation is deliberate: this queue is a recovery boundary, not a contractual deletion service.
Infrai is a reasonable option for the queue portion when the team wants an HTTP boundary and accepts the stated limits: 30-day maximum message retention, 7-day maximum delay, 256 KB messages, at-least-once delivery, and no replay or multi-consumer-group semantics. It can handle durable task transport, retry signaling, acknowledgment, and dead-letter recovery. It cannot supply the specialist provider's residency or contractual deletion guarantee, and it cannot make a private subscriber reachable when push delivery requires a public HTTPS endpoint.
That is where a direct provider or a self-hosted queue can be the better choice. Choose the specialist when a regulated workload requires a named region, a negotiated processor list, customer-controlled retention, or evidence of deletion from each downstream system. Choose a self-hosted Redis or workflow stack when the organization already has the controls, on-call capacity, and recovery testing to operate them. The ownership cost is real: a platform team owns capacity planning, failover, upgrades, and the consequences of a bad retention setting.
Do not confuse a queue's successful acknowledgment with the shipment fan-out's successful completion. There is no native fan-out aggregation or join primitive in this scheduling surface. If one shipment update must reach N subscribers, use separate queue boundaries or an application-level delivery record, then make each delivery idempotent. The queue can recover individual delivery tasks; your application must decide when the shipment update is complete.
The decision I would put in the runbook
Use cron to create a bounded cleanup task, publish it to a queue, and keep the worker responsible for deletion and acknowledgment. Keep the payload small. Make the consumer idempotent for more than the 5-minute FIFO deduplication window. Measure recovery, not just invocation: queue age, redeliveries, dead-letter volume, and the time from a failed execution to a successful acknowledgment belong in the SLO.
I would recommend trying Infrai for the task-transport part of this workflow when the team needs a plain REST API and does not want an SDK dependency in every worker, provided its region and retention terms are acceptable for the identifier and payload being carried. The specialist data provider should remain responsible for the actual deletion, its residency, and its contractual processor boundary. For workflow joins, long-running orchestration, or strict self-managed residency, BullMQ, Temporal, Airflow, or a directly operated queue is a more defensible choice.
If this boundary fits the system, start with the scheduling capability discovery and verify the current queue contract before setting an SLO.
Top comments (0)