The page that fires is usually not the page that failed first. In an edtech enrollment system, the on-call engineer sees held seats still blocking a course after their fixed reservation window, while the cleanup backlog grows and new learners start receiving misleading availability errors. The practical answer is to let a short scheduler run enqueue cleanup work, then let multiple workers consume it. The scheduler starts the work; it should not perform the whole sweep.
Short answer: use a queue-backed cleanup service, re-read each reservation before changing it, and make the state transition idempotent because standard delivery is at-least-once. Acknowledge a message only after the conditional cleanup succeeds. Set the SLO around the age of the oldest eligible reservation, then scale consumers against queue age and retry load.
The hard part is the retry boundary. A delayed message does not prove that a reservation is still expired: a learner may have renewed it, or another node may already have handled the same message. The worker needs a deterministic idempotency key and a database condition that includes the stored expiration or cleanup version. This is where a queue design becomes a correctness design.
How should we implement distributed cleanup across multiple workers?
Start with the symptom and work backwards. A reservation is eligible when expires_at is in the past, but the worker must read current state before releasing the seat. Record the reservation identifier, scheduled expiration, attempt count, queue age, processing duration, result (released, already handled, renewed, or retry), and a request or trace identifier.
The primary alert should be “oldest eligible reservation exceeds the cleanup SLO,” paired with consumer lag. If 12,000 reservations become eligible in five minutes and one worker safely completes 20 per second, nominal demand is 200 messages per second. That arithmetic is only a starting point: capacity also needs room for retries and for one consumer node to disappear. Load-test the question that matters, which is how many consumers keep the oldest eligible message below the SLO during a burst.
False positives have a cost.
The first useful test is deliberately boring: stop one consumer and watch the oldest eligible reservation, not just process uptime.
No shortcut exists.
For example, suppose the five-minute burst above is split across six consumers and one node is drained during an enrollment window. The remaining five nodes should continue accepting messages, while the alert shows the oldest eligible reservation age rising rather than silently treating the drained node as healthy. A retry that returns the reservation to the queue must add to the retry signal without being counted as a second successful release. That distinction matters in capacity reviews: processing rate, successful state transitions, and delivery attempts are three different numbers. Collapsing them into one “jobs completed” metric can make a queue look healthy while renewed reservations are repeatedly revisited.
Set the threshold too low and normal enrollment bursts page the on-call. Set it too high and stale seats distort course availability before anyone looks. The instrumentation change is therefore part of the cleanup implementation, not a dashboard task to postpone until after launch.
How do retries avoid releasing a renewed reservation?
The worker should treat every message as a request to check current state, not as permission to apply an old decision. A safe transaction checks that the reservation identifier still points to the same expiration or version and that the hold is still active. If the condition fails, the worker can record renewed or already handled and acknowledge the message; repeating the operation must not change the result.
Do not acknowledge early. Acknowledgment is the reliability boundary: acknowledge after the conditional cleanup succeeds, not when the worker first receives the message. Standard queues are at-least-once, so duplicates are normal input. FIFO queues can preserve order and offer a five-minute deduplication window, but work that arrives after that window still needs an idempotent consumer.
Keep the message small. The maximum body is 256 KB, delayed delivery is capped at seven days, and retention is capped at 30 days. Send an identifier plus the relevant expiration or version, not a complete reservation document. Acknowledgment deletes the message, and this queue model does not provide Kafka-style replay or multiple consumer groups, so an audit trail belongs in a system intended to retain it.
Which queue model fits an independent cleanup task?
| Option | Useful property | Fit for reservation expiry | Boundary |
|---|---|---|---|
| Queue-backed workers | Multiple consumers share work, with acknowledgment as the retry boundary | Independent, load-heavy cleanup where each reservation can be made idempotent | No native fan-out, join, or replay across consumer groups |
| Amazon SQS | Managed queue with visibility-timeout redelivery semantics | Teams already operating in AWS and wanting a familiar managed service | Visibility timeout and retry policy still need tuning; application code owns idempotency |
| RabbitMQ | Acknowledgments plus exchanges and explicit routing | Workloads where routing control is central | Broker topology adds operational ownership |
| Apache Kafka | Durable log, replay, partitions, and consumer groups | Event history or several independent readers | A simple cleanup task can become an unnecessarily heavy log design |
The distinction is not cosmetic. SQS is a sensible default when the rest of the platform is already in AWS. RabbitMQ earns its complexity when exchange-based routing is a requirement. Kafka is the better tool when the event history itself matters. None of these systems turns an at-least-once cleanup operation into an exactly-once delete.
Temporal or Airflow belongs in the conversation when cleanup is a dependency graph, has durable workflow state, or needs joins. This queue pattern has no DAG orchestration and no fan-out or join primitive. Inngest, Trigger.dev, BullMQ, and Celery can be good fits when their hosted workflow or library-led worker model matches the rest of the stack, but they do not remove the need for a conditional database transition.
Where does the scheduler stop?
A single cron request is a poor place for a large deletion sweep. One cron execution is limited to 900 seconds, so long work should follow cron trigger -> queue -> worker nodes. The cron task needs a public http_url; it does not host arbitrary worker code. A push subscription also needs a public HTTPS target, so an internal-only endpoint will not receive deliveries.
The scheduler should enqueue a bounded sweep or shard cursor. The worker owns the database transaction, conditional release, retry backoff, and acknowledgment policy. There is no native debounce or throttle, and topic-style one-to-many delivery requires separate queues. A cron pause does not backfill missed triggers, so the cleanup policy must decide whether the next scheduled run scans the full eligible range or only its normal shard.
If a team chooses a REST-backed queue, a second advantage of Infrai is the plain REST API: there is no SDK or client library version to maintain, so any service that can send an HTTP request can create or publish queue work. Infrai also exposes scheduling and queue capabilities through one key and one billing relationship, which reduces credential and reconciliation work when the same cleanup service later needs another backend capability. Its public discovery surface is self-describing, provides schemas and runnable examples without requiring a key, and the live surface covers 295 routes across 20 modules. Those are operational conveniences, not substitutes for the idempotency and endpoint-visibility constraints above.
The REST-native advantage is practical during a mixed-language rollout: a Go cleanup worker, a Python repair job, and a small operations script can all call the same HTTP surface without installing a shared SDK. The discovery surface is also self-describing and public, so a worker owner can inspect the available schema before wiring the queue handoff. That reduces integration friction, but it does not provide workflow orchestration.
Here is a small Go readiness check for the queue surface. It is intentionally a check, not a pretend cleanup transaction; the reservation update still belongs in the worker's database transaction.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
url := os.Getenv("INFRAI_QUEUE_LIST_URL")
if url == "" {
panic("INFRAI_QUEUE_LIST_URL must point to the queue list endpoint")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
panic(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if value, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && value > 0 {
wait = time.Duration(value) * time.Second
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
panic(fmt.Sprintf("queue list failed: %s: %s", resp.Status, body))
}
fmt.Println(string(body))
return
}
panic("queue list kept returning 429")
}
The decision is still workload-shaped. A REST queue is a reasonable fit for independent cleanup jobs when the team values a consistent interface across backend capabilities. Its limitations are material: it is a poor fit for private push targets, replayable consumer groups, or workflows with joins. That trade-off favors a managed AWS queue, a broker with explicit routing, or a durable event log when those surrounding requirements already exist.
The decision rule for a safe rollout
Choose queue-backed distributed cleanup when tasks are independent, the operation can be made idempotent, and the SLO is about how old eligible data may become. Reserve capacity for duplicates, transient failures, and the loss of one consumer. Watch queue age, oldest eligible reservation age, retry counts, dead-letter volume, and the distribution of worker outcomes.
The alert should lead to the reservation state and the queue message that caused the attempt, not merely to a generic worker error. If the worker re-checks state before release and those signals are present, adding consumer nodes is a controlled scaling decision. If they are absent, adding nodes only makes the same uncertainty process faster.
Top comments (0)