The hard limit is not midnight; it is the delete quota downstream. Short answer: schedule one nightly enqueue operation, then let a rate-limited worker queue drain stale uploads at a pace the dependency can sustain, with every delete safe to repeat. For a healthtech SaaS in the EU, that separates cleanup latency from request volume and keeps an ordinary quota response such as HTTP 429 from turning the nightly job into an uncontrolled retry wave.
Don't put the scan, every deletion, and any follow-up webhook into one cron request. A cron execution can run for at most 900 seconds, standard queue delivery is at least once, and a paused schedule does not replay missed triggers. Those are design inputs, not footnotes.
What can make nightly scheduled data cleanup duplicate stale uploads in an EU SaaS worker queue?
Use four boundaries: the cron trigger finds or requests a bounded page of stale upload IDs; the producer publishes those IDs as cleanup work; consumers claim work with limited concurrency and a shared rate budget; the delete operation records an idempotent terminal state before acknowledgement. If a cleanup also needs an outbound webhook, publish that action to a separate queue. There is no native topic fan-out, so one message cannot stand in for several independently acknowledged downstream actions.
This is a queueing problem before it is a scheduling problem. A nightly timestamp answers when work becomes eligible. It does not answer how many delete calls the storage provider accepts, how quickly a growing backlog must clear, or what happens when the same upload ID arrives twice. Native debounce and throttle are not available here, so the worker owns pacing, or consumer concurrency becomes the coarser control.
The platform choice follows from how much machinery the workflow actually needs:
| Option | Prefer it when | The catch |
|---|---|---|
| Managed cron plus queue through Infrai | The team wants scheduling and queueing behind one consistent REST contract, with one key and bill across 295 routes in 20 modules | It has no DAG orchestration, join primitive, native throttle, or Kafka-style replay and multiple consumer groups |
| Celery | The team already operates its worker stack and wants worker-side rate control close to application code | The platform team retains the operational ownership that comes with the self-managed path |
PostgreSQL with FOR UPDATE SKIP LOCKED
|
Cleanup rows already live in PostgreSQL and the team deliberately accepts a database-backed work queue | Queue traffic now competes inside the database capacity plan |
| Temporal or Airflow | Cleanup is really a workflow with DAG dependencies, fan-out and join behavior, or richer orchestration | This is more machinery than a nightly enqueue-and-drain loop requires |
My decision rule is blunt: choose managed cron plus queue for the narrow cleanup loop when reducing integrations and on-call surface matters; stick with Temporal or Airflow when the missing workflow primitives are requirements, Celery when its operating model is already paid for, or PostgreSQL when keeping the queue in the existing database is an explicit capacity decision. I'm not sure which option wins for a given team until its backlog SLO and ownership boundary are written down. Vendor count is not an architecture.
Turn the cleanup SLO into a rate budget
Start with a completion SLO. Suppose the nightly scan can identify 36,000 stale uploads and the downstream quota allows 20 deletes per second. The service-time floor is 1,800 seconds, or 30 minutes, before retries and variance. Two hundred workers do not improve that floor; without a shared limiter they merely reach 429 together. A practical plan adds headroom, caps concurrency separately from request rate, and alerts on the age of the oldest unacknowledged item rather than treating “cron fired” as success.
Short math helps:
package main
import (
"fmt"
"time"
)
func main() {
items := 36000
ratePerSecond := 20
floor := time.Duration(items/ratePerSecond) * time.Second
fmt.Printf("service-time floor: %s\n", floor)
}
Run it with go run main.go; it prints a 30-minute floor. Your mileage may vary because delete latency, retry frequency, and the actual stale-upload count are workload measurements, not constants supplied by the scheduler. Record them. Then size the queue retention against the worst credible recovery interval, remembering that retention is at most 30 days and an acknowledged message is deleted. Delayed messages can be delayed by at most seven days, so they are not a substitute for an indefinite retry archive.
For the latency-versus-cost decision, increase rate only until the completion SLO has useful margin. Past that point, extra consumer capacity buys little when a third-party quota is the bottleneck. If the SLO is “done before the morning shift,” a 45-minute drain may be entirely adequate; if an upload must disappear within minutes of becoming stale, nightly scheduling is the wrong trigger even though the worker design still applies.
Keep messages small. The body limit is 256 KB, so send an upload identifier and the minimum deletion context, not the object itself. Standard delivery can repeat, and FIFO deduplication covers only a five-minute window; neither removes the need for application idempotency.
Connect the scheduled producer to the worker control loop
The producer should use the queue's verified discovery schema rather than a payload copied from an old article. The program below makes the real Infrai batch-publish call. It reads that schema-compliant JSON from INFRAI_PUBLISH_BATCH_JSON, sends the key from INFRAI_API_KEY, reuses one idempotency key across retries, and backs off on 429 while honoring an integer Retry-After value. The route is intentionally the only Infrai route in the example; creating the cron and queue is control-plane setup, not part of every nightly tick.
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const publishPath = "/v1/queue/publish_batch"
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func publishBatch(ctx context.Context, payload []byte, idempotencyKey string) error {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return errors.New("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
return errors.New("INFRAI_BASE_URL is required")
}
for attempt := 0; attempt < 5; attempt++ {
request, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+publishPath, bytes.NewReader(payload))
if err != nil {
return err
}
request.Header.Set("Authorization", "Bearer "+key)
request.Header.Set("Content-Type", "application/json")
request.Header.Set("Idempotency-Key", idempotencyKey)
response, err := http.DefaultClient.Do(request)
if err != nil {
return err
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return readErr
}
if response.StatusCode >= 200 && response.StatusCode < 300 {
fmt.Println(string(body))
return nil
}
if response.StatusCode != http.StatusTooManyRequests {
return fmt.Errorf("publish batch returned %d: %s", response.StatusCode, body)
}
timer := time.NewTimer(retryDelay(response, attempt))
select {
case <-ctx.Done():
timer.Stop()
return ctx.Err()
case <-timer.C:
}
}
return errors.New("publish batch remained rate limited after five attempts")
}
func main() {
payload := []byte(os.Getenv("INFRAI_PUBLISH_BATCH_JSON"))
if len(payload) == 0 {
panic("INFRAI_PUBLISH_BATCH_JSON is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := publishBatch(ctx, payload, "cleanup-2026-08-14-page-0001"); err != nil {
panic(err)
}
}
Use a stable page identity for the idempotency key instead of the sample date. On the consumer side, persist each upload's terminal state transactionally, acknowledge only after the delete has reached its idempotent outcome, and retry a downstream rate-limit response with the same backoff rule. Don't tight-loop. A Node.js worker needs the same invariants even though its concurrency primitive will look different.
There is a subtle ordering issue for the healthtech webhook path. Deleting an upload and reporting the deletion are two independently retryable effects, so publishing them to separate queues lets each consumer acknowledge its own result; pretending they are one atomic operation creates an ambiguity after a process exits between the two calls. No built-in fan-out or join resolves that ambiguity. If the business process requires a durable multi-step state machine, this is the point where the recommendation changes to a workflow engine.
Prove deletion safety before rollout and rollback
Verification should prove the outcome, not merely the trigger. In a staging run, publish duplicate IDs on purpose, confirm that only one terminal deletion state is recorded, lower the worker rate, and watch the oldest-item age rise predictably. Then restore the planned rate and confirm the backlog drains within the stated SLO. Also test a quota response: workers should back off, honor Retry-After, and resume without a duplicate side effect.
Track at least the enqueue count, acknowledged count, retry count, oldest message age, and idempotent no-op count. The exact alert threshold depends on the completion SLO, but the relationship is fixed: oldest age approaching the allowed cleanup window is the page; a cron run record is diagnostic context. Cron trigger timing can have seconds of jitter, and recorded run output retains only its first 4 KB, so neither is a precise ledger.
Rollback is small. Pause future scheduling, leave already queued work available, and reduce consumer concurrency or request rate while investigating. Remember that resuming cron does not backfill the paused nights; after the cause is understood, run one bounded reconciliation scan and enqueue only identifiers whose terminal state is absent. Do not purge the queue as a reflex because that discards the evidence and work needed for recovery.
One more boundary matters: cron tasks call a public http_url, push subscriptions require public HTTPS, and cron does not host application code. This pattern is therefore not suitable when every worker endpoint must remain private with no public ingress. Use a deployment model that can reach the private worker directly, or keep consumption pull-based from inside the private environment.
Ship only after the duplicate test and the drain-time test pass.
Top comments (0)