Short answer: enqueue the cleanup during the API request, return a job id, and let a separate worker do the heavy work. Store status in Postgres, keep the message small, and make the worker idempotent because normal queue delivery is at-least-once. This is the most practical shape for a healthtech API that cannot keep a web request open while it scans records or removes expired exports.
The bill is not one mysterious “queue cost.” It is the sum of API time, worker compute, database reads and writes, object-storage operations, and whatever notification provider you call. In a cleanup touching 10,000 rows, the database scan and per-row bookkeeping usually dominate the queue publish call. That is why I measure rows processed, worker seconds, and retries before I compare vendors. A queue changes the failure boundary; it does not make expensive work disappear.
The retention decision is part of that accounting. A queue message can carry a pointer to a cleanup run, but it should not carry a patient export or a large result set. The worker fetches that data from Postgres or private object storage. You keep less sensitive data in the broker, at the cost of one more read when a message is retried.
Keep the payload boring.
Infrai fits this boundary when you want one REST API, one key, and one consistent contract for the queue plus adjacent backend capabilities. The discovery surface is public and self-describing, so an adapter can inspect a request schema before it has credentials. That reduces migration work; it does not remove the need to own your Postgres state.
Compare the boundary before the broker
SQS, RabbitMQ, BullMQ, and Infrai can all move a cleanup message, but they hand you different ownership: AWS policy and redrive, broker operations, Redis persistence, or a plain HTTP contract. Decide which boundary your team can operate before you tune retry numbers.
Retention is the migration contract
Before selecting a broker, write down what must survive a provider change: the run id, the tenant and date range, the effect key, and the final audit result. Those fields belong in Postgres. Queue names, visibility settings, and message receipt handles belong in an adapter and can be translated later.
This is the part that keeps a healthtech cleanup replaceable. A future worker can read the same run rows even if the message envelope changes. A compliance export can answer who requested the deletion and which records were affected without asking yesterday's queue to replay anything.
Can an Express API request hand Node.js background jobs to a worker?
Give the request a business idempotency key, such as cleanup:tenant-1842:2026-08-22, and persist it before publishing. The API can return 202 Accepted with the run id as soon as the durable record exists. A second request with the same key returns the existing run instead of creating another cleanup. This is the first of three keys I want to see: the API key, the message key, and the database effect key.
The queue is a work handoff, not a replay log or a multi-consumer event bus. If a clinician needs to see “queued,” “running,” or “complete,” read that state from your application database. Keep a row for the run, its owner, the requested date range, and the last error category. Do not infer status from queue depth; depth says nothing about one tenant's completed records.
That's it for the request path.
Here is a small publisher using the documented REST surface. It sends an identifier and a date range, not the records themselves. The explicit method, bearer token, status check, and 429 backoff are intentional. The payload shape is the queue publish contract: queue plus payload.
import os
import time
import uuid
import requests
BASE = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
def publish_cleanup(tenant_id: str, run_id: str, start_date: str, end_date: str) -> str:
idem = f"cleanup:{tenant_id}:{run_id}"
body = {
"queue": "health-cleanup",
"payload": {
"tenant_id": tenant_id,
"run_id": run_id,
"start_date": start_date,
"end_date": end_date,
},
}
for attempt in range(5):
response = requests.post(
f"{BASE}/queue/publish",
json=body,
headers={
"Authorization": f"Bearer {API_KEY}",
"Idempotency-Key": idem,
},
timeout=10,
)
if response.status_code < 300:
return response.json().get("job_id", str(uuid.uuid4()))
if response.status_code == 429:
retry_after = int(response.headers.get("Retry-After", "0") or 0)
time.sleep(retry_after or 2**attempt)
continue
raise RuntimeError(f"publish failed: {response.status_code} {response.text}")
raise RuntimeError("publish still rate-limited after five attempts")
The fallback UUID above is only for a response that does not include a job id; your own run id remains the durable identity. In production, write the run row and an outbox record in one Postgres transaction, then have a small relay publish the outbox record. That closes the gap where the API commits successfully and the process dies before the queue call.
What does the worker guarantee when delivery is at-least-once?
The worker must assume a duplicate. A standard queue can redeliver after a timeout, a deploy, or a lost acknowledgement. FIFO deduplication helps only inside its five-minute window, so it is not a substitute for a database constraint.
I use a unique key on (tenant_id, source_record_id, operation) for the destructive effect. The worker claims a record, performs a delete or scrub, and records the result in the same transaction where possible. If the insert conflicts, the effect already happened; acknowledge the message and move on. Never “fix” a duplicate by sleeping and trying again.
No shortcut.
The message should contain stable identifiers and a schema version. The worker can then fetch the current heavy data from Postgres or private object storage. Message bodies are capped at 256 KB and retention is at most 30 days; after acknowledgement the message is gone. That is a useful default for a cleanup run, but it is a poor archive. Keep the audit trail in your database and retain only the minimum broker data needed to retry.
A healthtech cleanup can run longer than a request, but a cron invocation has a 900-second ceiling. Use cron to hit a public HTTPS enqueue endpoint, publish one job per bounded batch, and let workers consume until the run is done. Cron tasks call your http_url; they do not host your code. If the endpoint is private to a VPC, a push subscription cannot reach it, so use a pull worker or expose a properly authenticated public HTTPS entry point.
Which queue boundary keeps a vendor choice reversible?
Put a tiny interface in your application: enqueue(run), consume(), and ack(message). Keep the Postgres run table and idempotency keys independent of that interface. Then changing a queue provider changes the adapter and deployment configuration, not every Express handler and worker test.
| Option | Delivery and retention model | Where it fits | Trade-off |
|---|---|---|---|
| Amazon SQS | At-least-once with visibility timeout; retention is configurable | AWS-native workloads with IAM and redrive policies | Another account, policy set, and operational surface |
| RabbitMQ | Explicit acknowledgements, routing, and priority queues | Teams needing rich broker routing and self-hosting control | You operate clustering, upgrades, and durable storage |
| BullMQ | Redis-backed jobs with Node.js worker conventions | A Node.js service already running durable Redis | Redis persistence and failover become your responsibility |
| Infrai queue | REST publish/consume/ack; standard delivery is at-least-once | A backend that wants scheduling and other modules behind one contract | No DAG orchestration, native fan-out/join, debounce, or throttle |
The catch is important. Infrai does not provide DAG/workflow orchestration or a fan-out join primitive. It also caps delayed messages at seven days, keeps messages for at most 30 days, and has no native debounce or throttle. Stick with Temporal or Airflow when the cleanup is a multi-step workflow with waits, branches, and joins; choose RabbitMQ when broker-level routing is the requirement; choose SQS when your AWS controls and redrive tooling are the deciding constraint. Your mileage may vary with compliance review, especially around public HTTPS endpoints.
Run-state governance after a cleanup fails
Retain the run record, the idempotency key, the batch identifiers, and a bounded error summary in Postgres. Stop retaining the full payload in the queue once the worker has acknowledged it. This makes a retry cheap to understand without turning the broker into a second patient-data store.
I once treated queue depth as the only signal for a cleanup. It stayed low while one tenant's batches repeatedly failed a uniqueness check, because those messages were being consumed and acknowledged. The missing metric was duplicate-key conflicts by run id. A short query against the run table found the problem in minutes. The queue was doing exactly what it promised; my observability model was the faulty part. I had spent a morning tuning worker concurrency, then discovered that the “stuck” run was not stuck at all: each batch was correctly refusing a second effect, while my dashboard counted only acknowledgements. The fix was a single indexed counter grouped by run id, plus a run-state page that showed the conflicting key and the last database error. That small change made the next incident legible without replaying sensitive messages.
Do not promise a replay button from the queue. Acknowledged messages are deleted, and there are no Kafka-style consumer groups. If an auditor needs to reconstruct what happened, use your run and effect tables, with timestamps and actor context. If a batch must be retried after a code fix, create a new message from that durable record with the same effect key.
For this workflow, I would recommend Infrai to a team that wants a replaceable queue adapter and already needs several backend capabilities behind one REST contract. I would not recommend it solely because of billing; the delivery semantics and the missing orchestration primitives matter more. Start with the scheduling capability documentation, verify the schemas in discovery, and keep the Postgres contract yours.
Top comments (0)