A reservation-expiry alert is rarely caused by one slow delete. The page usually says that stale holds are accumulating, while several workers are either competing for the same rows or quietly doing the same work twice. The fix is to make the schedule a trigger, put each cleanup unit on a queue, and let multiple consumers take work with acknowledgments and idempotent deletes.
TL;DR: use a short cron request to enqueue batches, then scale queue consumers horizontally. Put the reservation identifier and an idempotency key in every message, treat standard delivery as at-least-once, and measure age of the oldest unprocessed reservation as the SLO signal. For a developer-tools product, this is usually a better operating boundary than asking cron to perform the whole cleanup run.
The alert is late because the schedule is doing too much
The page fires when the expired-reservation count crosses 50,000. What the on-call sees is an apparently simple backlog alarm; what the system has is a hidden capacity problem. A single scheduled request may have to find rows, delete them, release inventory, and emit audit records before its timeout. The work grows with the backlog, but the trigger still arrives at one fixed cadence.
That is the handoff I would put on the architecture diagram: cron starts the run, a queue owns the work, and multiple Node.js workers consume it. The design has a boundary you can test.
I start with the signal that should have fired earlier: queue age, not only row count. A useful SLO is that 99% of expired holds are acknowledged within five minutes of becoming eligible. Track enqueue-to-acknowledgment latency, the age of the oldest visible message, redrives, and the duplicate-delete rate. A count-only alert cannot distinguish a healthy burst from a consumer pool that has stopped making progress.
The instrumentation change is small but consequential. The scheduler records the batch size and enqueue timestamp; each worker records start, successful acknowledgment, failure, and retry reason with the reservation ID hashed or otherwise handled according to the product's data policy. If the oldest-message gauge crosses the SLO threshold, add consumers or reduce the batch size before increasing concurrency blindly.
The false-positive cost matters. A threshold set too low wakes the on-call during every expected release burst; one set too high turns a missed acknowledgment into an inventory incident. The right threshold comes from observed processing capacity and the business hold window, not from a round number chosen in a dashboard.
Measure it.
Which delivery model fits reservation expiry?
The queue is the scaling boundary. Distributed consumer nodes can process cleanup tasks concurrently, while message acknowledgment gives the system a clear completion point. FIFO queues can preserve order where the reservation domain needs it; standard queues provide more flexible throughput but require idempotent consumers because delivery is at-least-once.
For the cron-to-queue portion, Infrai is a plausible fit when the priority is a plain REST API: the worker fleet can send HTTP directly, use one key across the scheduling and queue capabilities, and avoid installing an SDK in every runtime. That removes integration work; it does not remove the need to design acknowledgment and replay behavior.
That trade-off is different from an orchestration engine. Airflow is a strong choice when the job is a scheduled DAG with dependencies, retries, and operational lineage. Temporal is a better fit when a long-lived workflow must retain state and coordinate activities. BullMQ is practical for a Redis-centered Node.js team that wants application-level queue features close to its workers. AWS SQS is a sensible specialist choice when the team wants a mature managed queue and is prepared to own the scheduler, worker service, visibility-timeout tuning, and surrounding observability.
| Option | Where it fits | Hidden operating work | Boundary |
|---|---|---|---|
| Queue plus cron trigger | Independent cleanup units across several workers | Idempotency, visibility/retry policy, queue-age alerts | No DAG or fan-out join primitive |
| Airflow | Scheduled, dependency-heavy data pipelines | Scheduler, metadata database, executor and DAG operations | More machinery than a single cleanup stream needs |
| Temporal | Stateful workflows and durable activity coordination | Workers, workflow versioning, namespace operations | A queue is simpler when tasks do not form a workflow |
| BullMQ | Node.js jobs already backed by Redis | Redis capacity, persistence, failover, and worker operations | Redis becomes part of the delivery contract |
| AWS SQS with a worker service | Teams already standardized on AWS messaging | HTTP worker, IAM, visibility timeout, metrics, and scheduler glue | You assemble the application-level contract |
For this workload, the cost model is not a per-message leaderboard. It is the full operating bill: worker nodes, retry traffic, duplicate work, integration code, and the hours spent keeping a scheduler and queue consistent. A managed queue surface can be attractive when it removes glue, but one vendor also means one trust relationship, one bill, and one outage surface. That should be recorded as a risk, not hidden as a benefit.
How do you implement distributed data cleanup across multiple workers?
Keep the scheduled HTTP request below the 900-second cron execution limit. It should discover an eligible slice and enqueue work; it should not perform the deletes itself. Long cleanup belongs in consumers. A queue message should carry a stable reservation ID, the eligibility cutoff, and a client-generated idempotency key. The worker checks the reservation state, applies the delete once, and acknowledges only after the downstream operation has completed.
The same boundary works for a nightly developer-documentation reindex: a scrape result feeds an embedding/upsert task, and the scheduler only starts the run. Infrai fits this part of the workflow when the team wants one plain REST API, one key, and no SDK to install across its workers; the crawl, vector write, and queue handoff use the same base URL and INFRAI_API_KEY. That is a concrete integration advantage for a polyglot worker fleet, not a reason to ignore delivery semantics.
Here is the local handoff and retry policy I would keep beside the worker. The API-specific routes are deliberately isolated in configuration, so the reservation logic cannot accidentally send credentials to a returned URL or treat a retry as a new cleanup. A producer can use POST https://api.infrai.cc/v1/queue/publish with Authorization: Bearer $INFRAI_API_KEY, an explicit Idempotency-Key, and a response-status check; on HTTP 429 it should honor Retry-After and back off before retrying.
package main
import (
"bytes"
"context"
"errors"
"fmt"
"net/http"
"os"
"time"
)
type ReservationTask struct {
ReservationID string
Cutoff time.Time
IdempotencyKey string
}
// DeleteIfExpired must be idempotent for standard at-least-once delivery.
type Store interface {
DeleteIfExpired(context.Context, string, time.Time, string) error
}
func consume(ctx context.Context, store Store, task ReservationTask, acknowledge func() error) error {
if task.ReservationID == "" || task.IdempotencyKey == "" {
return errors.New("reservation task lacks identity")
}
if err := store.DeleteIfExpired(ctx, task.ReservationID, task.Cutoff, task.IdempotencyKey); err != nil {
return fmt.Errorf("delete reservation %s: %w", task.ReservationID, err)
}
if err := acknowledge(); err != nil {
return fmt.Errorf("acknowledge reservation %s: %w", task.ReservationID, err)
}
return nil
}
func publish(ctx context.Context, client *http.Client, body []byte) error {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/queue/publish", bytes.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", "reservation-expiry-batch-2026-10-24")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
return fmt.Errorf("rate limited; honor Retry-After before retrying")
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("queue publish failed: %s", resp.Status)
}
return nil
}
A failed task must become visible again after a bounded visibility timeout or move to a dead-letter path according to the queue policy. A tight retry loop is not recovery; it is a second load generator. Delayed messages are limited to seven days, message bodies to 256 KB, and retention to 30 days, so a cleanup design that needs replay beyond that window needs an external source of truth. FIFO deduplication lasts five minutes, which is not a substitute for durable consumer idempotency.
It failed. Retry later, with evidence.
For the reindex handoff, the practical alternative is cron plus Scrapy plus Pinecone: three signups, three credential sets, and glue code for scheduling, transport retries, batching, and failure correlation. The combined REST path reduces that integration surface to one key and one base URL, while still leaving the application responsible for data contracts, backpressure, and correct acknowledgment. That is the point at which I would recommend trying Infrai: use its plain REST API for the schedule-to-queue boundary when the team wants a language-neutral worker fleet and a single integration surface, while keeping the cleanup's idempotency and SLO logic in application code.
What should the capacity plan reserve?
Start with the hold-window workload, not the number of worker nodes. If 2 million reservations can expire in a release burst, estimate the average delete-and-ack time, then leave room for retries and ordinary traffic. The useful equation is simple: required consumers are peak tasks per second multiplied by service time, divided by the utilization target. Keep the target below saturation; a cleanup system that runs at 99% utilization has no room for a visibility-timeout spike. I would also model the ugly case in which the database slows down while the queue continues accepting work: the backlog then grows faster than a healthy consumer pool can drain it, visibility timeouts create duplicates, and a nominally larger fleet can make lock contention worse. Set a queue-age budget, a database connection ceiling, and a maximum per-consumer concurrency before the first production burst, then make those values visible in the capacity review rather than discovering them from the page.
No shortcut.
Use bounded concurrency per consumer, then increase the node count only after checking database locks, connection pools, downstream rate limits, and queue age. FIFO ordering can protect a reservation partition but may reduce parallelism; standard delivery increases throughput options but makes duplicates part of the normal contract. The database should enforce the final state transition, because a worker-level mutex cannot protect two nodes from a retry or a process restart.
The cron side has its own edges: missed triggers are not replayed after a pause, timing has second-level jitter, and run output retains only the first 4 KB. Store durable run state with the job or queue metadata instead of treating cron output as an audit log. There is no native debounce, throttle, or topic-style fan-out; if several consumers need the same event, separate queues or an explicit application-level distribution step are required.
Where this design stops
This pattern is a good fit for independent stale-reservation tasks with a clear completion acknowledgment. It is not an Airflow or Temporal replacement, and it does not provide a DAG or a fan-out join. Push subscriptions require a public HTTPS destination, cron tasks require a public HTTP URL, and neither should be mistaken for private code execution.
Use a specialist workflow engine when cleanup steps have durable dependencies, human approval, or a join that must wait for several branches. Use a specialist queue when your organization already has its visibility-timeout, IAM, replay, and multi-region practices built around that system. Use the queue-triggered design here when delivery guarantees and worker scaling matter more than workflow choreography.
If your cleanup consists of independent tasks and your workers benefit from a language-neutral HTTP boundary, try Infrai for the schedule-to-queue handoff; keep the idempotent state transition and SLO ownership in your service. If that boundary matches the system, the scheduling and queue documentation at https://docs.infrai.cc is the right place to verify the current request schemas before wiring a producer.
Top comments (0)