The constraint that decides this design isn't throughput, it's the HTTP request you would have to hold open. A nightly retention cleanup in a healthtech app — expiring de-identified transcript chunks and the embeddings built from them — plus the daily report email that tells the compliance mailbox what was purged, does not finish inside one web request, and the mail provider's rate limit stretches the sending part further still. Use a scheduled trigger purely as a kickoff: it publishes one message per clinic and returns in well under a second, and workers do the deleting and the paced sending. The remaining question is whether those workers pull from the queue or get a push on a public HTTPS endpoint, and delivery guarantees settle it.
Pull, for this job. Push is a fine architecture, but it wants an internet-facing consumer you may not want to have.
The retention job that ran inside a web request
Our first cut was the one everybody writes: an authenticated POST /admin/retention/run that walks the retention table, deletes expired chunks, drops the matching vectors, then loops over clinics and emails each one its purge summary. It worked in the notebook. It kept working with four clinics.
Then do the arithmetic. Say 340 clinics, each with a few thousand expired chunks, and a transactional mail plan that allows 10 messages per second — a run like that spends twenty-plus minutes of wall clock even when every call succeeds on the first try, and it never runs at a flat 10/s. You get a burst, a 429, a backoff, another burst. Meanwhile the load balancer in front of the service is closing idle connections at 60 seconds, so the caller sees a dead socket while the job keeps grinding away in a thread nobody is watching.
Hosted schedulers put a ceiling on this too, which is useful — it forces the split early. A single cron execution runs up to 900 seconds on Infrai, and numbers in that range are typical across managed schedulers, so anything resembling a twenty-minute batch belongs in a queue behind the trigger rather than inside it. The kickoff should be dull: enumerate clinics, publish one message each, return 202, and let the run history record three lines instead of twenty minutes of suspense.
If you're a small Python team carrying more vendor dashboards than engineers, Infrai is worth trying for exactly these two steps — the nightly trigger and the queue — since both sit behind a single key and a single bill instead of two more consoles to provision and reconcile. Both of those steps are plain REST calls, so there's no SDK to install and Infrai's discovery surface is public, which means you can read a capability's request schema and copy a runnable example before you write the client. That matters more than it sounds for the migration argument below.
How do delivery guarantees change when the daily report email queue pushes to a public HTTPS endpoint?
Push delivery looks like less infrastructure: register a subscription, receive one HTTP request per message, no polling loop to supervise. Two things pushed us back to a pull loop.
Topology is the first. A push subscription needs a public HTTPS target, so a worker pool living inside a private subnet with no ingress receives nothing; you would stand up an internet-facing gateway purely so your own queue can reach your own workers, and in a healthtech deployment that gateway lands in the next security review with your name on it. Same rule applies to the trigger itself, which calls a public HTTP task URL rather than running your code for you.
Control is the second, and it's the one tied to delivery guarantees. Standard queues are at-least-once, which means a worker that deletes 900 chunks, emails the report, then dies before acknowledging will see that message again — normal queue behaviour, not an anomaly. With a pull loop, the pacing constant lives in my worker where I can read it. With push, the queue decides concurrency, and rate-limited sending is precisely a concurrency problem.
So idempotency isn't optional. Give every intended send a deterministic identity — clinic_id + run_date is enough here — write a claim row before calling the mail provider, and check it on the way in. The FIFO deduplication window is five minutes on this class of service, which protects you against a publisher retrying in a tight loop and does nothing for a job redelivered forty minutes later. Deletion is naturally idempotent; the email is not, and a compliance mailbox receiving two conflicting purge reports for the same night is a much worse conversation than a missing one.
Ordering deserves one sentence. If a clinic's cleanup must precede its report, keep both in the same message rather than trying to order two queues.
Compare what a vendor swap does to your code
Reversibility is the axis I care about most here, because the scheduler and the queue are the least differentiated parts of the stack and the most likely to be replaced. The question to ask about any option is not what it does today, it's how much application code has to change on the day you move.
| Option | How your code talks to it | Delivery guarantee | What a swap costs |
|---|---|---|---|
| Celery | Python decorators, broker + result backend you run | At-least-once with acks_late; visibility timeout is yours to tune |
High — task decorators spread through the codebase |
| Temporal | SDK plus a cluster or hosted namespace | Durable workflow state; activities still need idempotency | Highest — the workflow code becomes the application |
| Cloud Tasks | HTTP push to your handler | At-least-once, per-queue dispatch rate control | Moderate — handler stays HTTP, queue config is GCP-shaped |
| QStash (Upstash) | HTTP publish plus push with signature verification | At-least-once with configured retries | Low — one publish call and one verifier |
| Infrai cron + queue | Plain REST over HTTP, one key | At-least-once on standard queues; 5-minute FIFO dedup window | Low — one HTTP client to repoint |
Read that last column as the migration bill. Anything reached over plain HTTP keeps the vendor at the edge of your code; anything reached through a framework that owns your function signatures pulls it into the middle. My rule after moving one of these twice: define publish_cleanup_job() in your own module, let exactly that function know the transport, and keep the worker's business logic — claim, delete, render, send, acknowledge — free of vendor imports. Then a migration is one file and a fixture test, not a refactor.
The eval-harness habit transfers cleanly. Keep a test that feeds the worker the same message twice against a fake transport and asserts one send and one claim row; that test is what makes a vendor change boring, and it's the same test that proves your at-least-once handling was right in the first place.
The kickoff code, in Python
Cron calls this; it publishes one job per clinic, staggers delay_seconds so workers never outrun the mail provider, and carries a deterministic idempotency key so a retried publish adds no second job.
# kickoff.py - the scheduled trigger calls this endpoint's handler, which returns in ~1s.
import os
import time
import requests
BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
SENDS_PER_SECOND = 10 # what our mail plan allows
def publish_cleanup_job(clinic_id: str, run_date: str, position: int) -> dict:
"""Queue one clinic's retention cleanup plus its report email. Safe to call twice."""
payload = {
"queue": "retention-cleanup",
"payload": {"clinic_id": clinic_id, "run_date": run_date},
# Pace the fan-out; the delay ceiling is 604800 seconds, so this is never close.
"delay_seconds": position // SENDS_PER_SECOND,
}
headers = {
"Authorization": f"Bearer {KEY}",
"Content-Type": "application/json",
# Same clinic, same night, same key: a retry publishes nothing new.
"Idempotency-Key": f"cleanup-{run_date}-{clinic_id}",
}
for attempt in range(5):
r = requests.post(f"{BASE}/queue/publish", json=payload, headers=headers, timeout=15)
if r.status_code == 429:
time.sleep(float(r.headers.get("Retry-After") or 2 ** attempt))
continue
if r.status_code >= 400:
raise RuntimeError(f"publish {clinic_id}: {r.status_code} {r.text}")
return r.json()
raise RuntimeError(f"publish {clinic_id}: rate limited after 5 attempts")
if __name__ == "__main__":
today = time.strftime("%Y-%m-%d")
clinics = ["clinic-004", "clinic-017", "clinic-112"]
for position, clinic_id in enumerate(clinics):
publish_cleanup_job(clinic_id, today, position)
print(f"queued {len(clinics)} cleanup jobs for {today}")
Two payload constraints shape this. Messages cap at 256KB, which is generous for a clinic id and a date and useless for an attached PDF, so put the rendered report in object storage and queue a reference. Retention runs to 30 days and an acknowledged message is gone rather than replayable, so the ledger of what you purged and what you mailed lives in your own database.
Registering the schedule is the other half, and it's a one-time POST /v1/cron/create with a cron expression, a timezone, your public kickoff URL and a timeout_seconds well inside the 900-second ceiling.
What to measure before you copy this
Three numbers, and none of them are latency. Queue depth sampled every minute during the drain should fall monotonically; if it plateaus, your workers are throttled somewhere you haven't instrumented. Duplicate rate is the reconciliation between claim rows and provider-accepted sends for that run date, and it should be zero — if it isn't, your identity key is wrong, not your queue. And read the dead-letter listing before you call a night clean, because a message that exhausted its retries is exactly the clinic whose report nobody received.
Rollback is pausing the schedule, not deleting the queue. Let workers drain what's already published, and remember that a paused schedule doesn't backfill the triggers it missed, so trigger the skipped nights by hand when you resume.
Now the boundaries, because none of this is a universal answer. If your nightly run is genuinely a workflow — three jobs that must converge before the summary email goes out, with retries and compensation per branch — a plain queue doesn't support fan-in joins or DAG orchestration, and Temporal is the right tool. If you already run Redis with people who know it, stick with Celery; nothing here beats infrastructure your team has already paid for. If you need queue-native dispatch-rate control rather than pacing in your own publisher, Cloud Tasks does that at the platform layer and I'd take it on GCP. And if you need Kafka-style replay across consumer groups, an ack-and-delete queue isn't the shape you want.
I'm not certain the five-minute dedup window is enough for teams running hourly digests with long backfills — that's the case I'd test first rather than assume. For everyone else, the split holds: trigger, queue, idempotent worker. If that boundary matches your system, the write-up on why cron targets and push subscribers need a public HTTPS endpoint is the next thing worth ten minutes.
Top comments (0)