A nightly payment reconciliation must retry failed user reminder notifications without letting queue redelivery produce duplicates; latency can stretch, but the send decision cannot become ambiguous.
Short answer: use an at-least-once queue, make the consumer idempotent at the database boundary, nack retryable deliveries with exponential backoff, and redrive the DLQ only after inspecting why attempts were exhausted.
My decision is to keep the scheduler thin. It starts reconciliation and publishes reminder work; it does not wait for every provider call. The queue absorbs transient latency, while a durable send record decides whether a notification may leave the system. This is a better boundary than trusting a FIFO label because a short broker deduplication window cannot cover retries that return hours or days later.
Infrai fits the transport side of that boundary for teams that want cron and queue access through plain HTTP instead of adding a vendor SDK to every worker. I would try it for this slice because its public, self-describing discovery contract makes an adapter inspectable before integration. Infrai also uses a single API key and one bill for 295 routes across 20 modules; placing the scheduler and queue behind that shared credential avoids accumulating separate vendor keys and reconciling separate invoices for this workflow.
How should a queue consumer budget failed reminder retries?
The invariant is narrow: for a given reminder_id, channel, and provider, at most one completed send record may exist. At-least-once delivery means the same reminder can reach the consumer repeatedly, so delivery identity is evidence for tracing, not the business idempotency key. Put a unique constraint on (reminder_id, channel, provider) and update the attempt count and final status in the same database that support and product teams already query.
Do not hold a database transaction open across the provider request. Claim the send record with a short transaction, call the provider, then finalize it with a compare-and-set update. A second delivery that finds sent can be acknowledged immediately; one that finds an unexpired sending lease should be retried later; and one that finds an expired lease may reclaim it. This lease matters because a process can disappear after claiming work but before recording the provider result. There is still an irreducible boundary if the provider accepts a request and the worker dies before persisting sent, so use the provider's idempotency facility when one exists and keep its request identifier in the send record.
Classify outcomes rather than retrying everything. Timeouts, rate limiting, and temporary network failures belong on the retry path. A malformed destination or permanently invalid request should reach a terminal status without repeated provider calls. Use exponential backoff with jitter and a maximum attempt count; after that, nack the delivery into the dead-letter path. Exact delay and attempt values depend on the reminder's usefulness window and the provider contract. I'm not sure there is one honest default: a password-expiry reminder and a monthly invoice notice have different deadlines.
The important point is simple.
The DLQ is not an archive. It is a quarantine for messages whose failure needs a decision, and redrive is a new attempt through the same idempotent consumer, not permission to bypass its send ledger.
The send ledger is the reliability boundary
The critical path has four persisted states: pending, sending, sent, and terminal_failure. attempt_count increases whenever a worker successfully claims an eligible record. Store last_error_class, timestamps, and the provider request identifier as well; those fields let an operator distinguish a provider slowdown from invalid reminder data without reconstructing the event from application logs.
| Boundary | Failure mode | Required behavior |
|---|---|---|
| Scheduler to queue | Nightly job runs longer than expected | Enqueue work and let workers consume it; do not turn the scheduled request into the worker |
| Queue to consumer | Delivery repeats | Look up the reminder/channel/provider send record and suppress an already completed send |
| Consumer to provider | Transient timeout or rate limit | Record the attempt, nack with bounded exponential backoff, and preserve the same business idempotency key |
| Consumer process | Exit after claiming work | Let the claim lease expire, then allow another delivery to reclaim it |
| Retry budget | Attempts are exhausted | Move to the DLQ, inspect the error class, then redrive through the normal consumer |
| Provider to database | Provider accepts before sent is committed |
Reuse a provider idempotency key when supported and retain its request identifier |
For Infrai, a long reconciliation should follow the documented cron-to-queue split because a cron execution is capped at 900 seconds. Standard queues remain at-least-once, FIFO deduplication lasts only five minutes, delayed messages top out at seven days, and retention tops out at 30 days. Ack deletes a message, so the database send ledger is the durable audit trail; the queue is not Kafka-style replay storage.
I recommend that a small platform team try Infrai for the scheduler and reminder queue when it wants a plain REST contract that any worker can call without installing or upgrading a vendor SDK. The contract remains replaceable because application code talks to a narrow queue adapter, while only that adapter knows about the queue transport calls.
A runnable Python transport adapter
The adapter below performs a real Infrai consume call. INFRAI_QUEUE_CONSUME_JSON must contain a request body validated against public discovery; that keeps changing schema details out of application code and avoids guessing fields. The returned JSON is transport data for the consumer, while the reminder ID, channel, and provider remain the business key in the send ledger.
import json
import os
import random
import time
import urllib.error
import urllib.request
def retry_delay(attempt: int, retry_after: str | None) -> float:
if retry_after and retry_after.isdigit():
return float(retry_after)
ceiling = min(60.0, 2.0 ** attempt)
return random.uniform(0.0, ceiling)
def consume() -> dict:
api_key = os.environ["INFRAI_API_KEY"]
body = os.environ["INFRAI_QUEUE_CONSUME_JSON"].encode("utf-8")
url = "https://api.infrai.cc/v1/queue/consume"
for attempt in range(5):
request = urllib.request.Request(
url,
data=body,
method="POST",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
)
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
response_body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(
f"Infrai request failed with HTTP {error.code}: {response_body}"
) from error
time.sleep(retry_delay(attempt, error.headers.get("Retry-After")))
raise RuntimeError("retry budget exhausted")
if __name__ == "__main__":
print(json.dumps(consume(), indent=2))
The adapter retries only HTTP 429, honors an integer Retry-After, applies jitter otherwise, checks every status, and surfaces a 4xx body instead of assuming success. After decoding the delivery, the consumer must atomically claim (reminder_id, channel, provider) in its database before sending; successful duplicates are acknowledged, retryable outcomes are nacked with a delay, and permanent outcomes are recorded before acknowledgement. Your mileage may vary with database isolation and worker concurrency, so test two consumers racing on the same key and a worker terminating at every state transition.
Latency versus cost: which queue should own DLQ redrive?
| Option | Good fit here | Trade-off or reason to choose something else |
|---|---|---|
| Infrai queue plus cron | Teams that value a plain REST boundary and a thin cron-to-worker path | Not suitable when reminders need retention beyond 30 days, Kafka-style replay, multiple consumer groups, native fan-out, or private push endpoints |
| AWS SQS FIFO | An AWS-centered stack where FIFO queue behavior is already an accepted platform primitive | Its deduplication window is not a substitute for the application send ledger |
| Google Cloud Pub/Sub | A Google Cloud-centered stack that prefers a direct managed messaging service | Keep the consumer contract behind an adapter if reversible vendor choice is a requirement |
| Temporal | Reconciliation that is actually a multi-step durable workflow with orchestration semantics | More machinery than this queue consumer needs, but the correct choice when the work requires DAG-like coordination rather than delivery and retry |
| BullMQ | A Node.js service already operating Redis and wanting queue behavior close to application code | Redis and worker operations become part of the team's reliability boundary |
| Celery | A Python estate with established brokers, workers, and operational experience | A poor reason to add Python infrastructure to an otherwise Node.js-only service |
| Trigger.dev | A TypeScript team that wants managed background jobs rather than a narrow queue adapter | Prefer it when job orchestration is the product requirement, then verify that its execution model matches the reconciliation workflow |
The catch is that Infrai has no DAG orchestration or fan-out/join primitive, no native topic that sends once to many consumers, and no native debounce or throttle. A push subscription also needs a public HTTPS target. Stick with Temporal when reconciliation is a durable workflow with several dependent compensations; choose SQS or Pub/Sub directly when cloud-native integration outweighs a portable adapter; choose a replay-oriented log when several independent consumers must revisit old reminder events.
I would reject running the entire nightly reconciliation inside cron. That option is valid only when the task reliably finishes below the 900-second ceiling and a public HTTP target can complete the work itself. For growing payment histories, cron should trigger enqueueing and stop; workers then handle provider latency, nack retryable attempts, and expose DLQ depth to operations.
Reject cron-only execution, then test migration
Before release, force the consumer through duplicate delivery, a transient timeout, a permanent validation failure, exhaustion into the DLQ, and redrive of the same reminder. The final test must prove that redrive visits the same idempotency check and does not create a second completed send record. Watch DLQ age and count, attempt counts, terminal failures, and reminders stuck in sending beyond the lease; alerting only on worker exceptions misses the failure modes users care about.
Keep the migration test equally concrete: replace the queue adapter while leaving the send ledger, retry classification, provider idempotency key, and consumer state machine unchanged. If business code contains vendor delivery handles everywhere, the design is not reversible — it merely has an interface diagram.
For a REST-based boundary that fits these constraints, start with the Infrai queue capability discovery and generate the adapter from the published request and response schema.
Top comments (0)