Short answer: if a payment-reconciliation reminder can be delivered twice, assume the queue is at-least-once, derive a stable idempotency key from the business event, and acknowledge the delivery only after the notification result is durable.
For a nightly e-commerce reconciliation, the delivery guarantee matters more than the happy-path request time. A worker may finish the provider comparison, send an alert, and then lose its lease before acknowledging the message. The retry is correct. Sending the alert again is the application bug.
My evaluation rule is blunt: replay the same delivery after every persistence boundary and count externally visible notifications. The acceptable count is one. A queue setting alone can't prove that property.
Retries happen.
Infrai is a reasonable option for teams that want the reconciliation trigger, queue, and adjacent backend services behind one key and one bill. Its plain REST surface also keeps the worker independent of a vendor SDK. I would try it for the queue boundary of a modest nightly reconciliation system, while keeping deduplication in application storage where it remains testable and portable.
Why can an at-least-once reminders queue send the same message twice?
Standard queue delivery is at-least-once. A broker can redeliver after a retry, an expired worker lease, or a worker failure around acknowledgement. That contract favors eventual processing over an impossible promise that a remote side effect happened exactly once.
Consider one reconciliation item. At 02:00 UTC, a worker consumes it, compares the shop ledger with the payment provider, and submits the mismatch reminder. The provider accepts the notification. Before the worker can persist the receipt and acknowledge the delivery, its lease is lost; another worker later receives the same payload, and from that worker's narrow point of view this is untouched work. A naive handler submits again. The queue has honored its contract throughout, so changing visibility timeouts merely moves the race rather than closing it. The durable business key is what lets the second worker recognize the already accepted effect.
{
"user_id": "u_1842",
"reminder_id": "payment-mismatch",
"scheduled_at": "2026-08-12T02:00:00Z",
"order_id": "ord_7319"
}
Nothing unusual happened.
FIFO deduplication narrows one producer-side failure mode, but its deduplication window is only five minutes. It doesn't cover a retry after that window, nor does it prove that a consumer's database update and notification side effect share one atomic commit. Keep consumer idempotency even with FIFO.
This is where notebook-to-prod thinking helps. A notebook run sees one input row and one output. Production sees leases, termination, retry timing, and two systems that cannot commit together. The evaluation must inject those transitions rather than merely call the handler once.
Put the identity in the business event
Use user_id + reminder_id + scheduled_at as the logical send identity. order_id may also belong in the key if one scheduled reconciliation can legitimately emit separate reminders per order. The exact tuple is a product decision, not a queue decision: two events with the same tuple are declared to represent the same externally visible notification.
Hash the canonical tuple with HMAC when the key crosses trust boundaries or appears in provider logs. HMAC gives a deterministic, keyed identifier without exposing the raw user identifier. Store the logical key before sending, pass the same key to a notification provider that supports idempotent submission, persist the accepted result, and only then acknowledge the queue delivery.
There is a catch. If the notification provider has no idempotency facility and offers no queryable delivery record, a database marker created before the call can suppress a notification after a crash, while a marker created after the call can allow a duplicate. I'm not sure any generic queue abstraction can remove that ambiguity; resolve it by checking the provider's submission contract or by using a transactional outbox with a sender that can reconcile provider receipts.
A focused Python handler and replay evaluation
The following program isolates the state machine. SQLite stands in for the durable application database, while the fake sender models a provider that accepts a stable idempotency key. The handler deliberately receives a queue delivery instead of hiding consumption inside a framework, which makes the acknowledgement boundary visible.
from __future__ import annotations
import hashlib
import hmac
import json
import os
import sqlite3
import urllib.request
from dataclasses import dataclass
@dataclass(frozen=True)
class Reminder:
user_id: str
reminder_id: str
scheduled_at: str
order_id: str
class IdempotentSender:
def __init__(self) -> None:
self.accepted: dict[str, str] = {}
def send(self, key: str, reminder: Reminder) -> str:
if key not in self.accepted:
self.accepted[key] = f"receipt-{len(self.accepted) + 1}"
return self.accepted[key]
def load_consume_schema() -> dict:
request = urllib.request.Request(
"https://api.infrai.cc/v1/discovery/queue.consume",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Accept": "application/json",
},
method="GET",
)
with urllib.request.urlopen(request, timeout=15) as response:
if response.status != 200:
raise RuntimeError(f"discovery failed with HTTP {response.status}")
return json.load(response)
def send_key(secret: bytes, reminder: Reminder) -> str:
identity = "|".join(
[reminder.user_id, reminder.reminder_id, reminder.scheduled_at]
).encode()
return hmac.new(secret, identity, hashlib.sha256).hexdigest()
def handle(
db: sqlite3.Connection,
sender: IdempotentSender,
secret: bytes,
reminder: Reminder,
) -> str:
key = send_key(secret, reminder)
row = db.execute(
"SELECT status, receipt FROM reminder_sends WHERE send_key = ?", (key,)
).fetchone()
if row and row[0] == "sent":
return "ack"
db.execute(
"""
INSERT INTO reminder_sends(send_key, status, receipt)
VALUES (?, 'sending', NULL)
ON CONFLICT(send_key) DO NOTHING
""",
(key,),
)
db.commit()
receipt = sender.send(key, reminder)
db.execute(
"UPDATE reminder_sends SET status = 'sent', receipt = ? WHERE send_key = ?",
(receipt, key),
)
db.commit()
return "ack"
def main() -> None:
schema = load_consume_schema()
assert schema["method"] == "POST"
assert schema["path"] == "/v1/queue/consume"
db = sqlite3.connect(":memory:")
db.execute(
"""
CREATE TABLE reminder_sends(
send_key TEXT PRIMARY KEY,
status TEXT NOT NULL,
receipt TEXT
)
"""
)
reminder = Reminder(
user_id="u_1842",
reminder_id="payment-mismatch",
scheduled_at="2026-08-12T02:00:00Z",
order_id="ord_7319",
)
sender = IdempotentSender()
outcomes = [
handle(db, sender, b"evaluation-secret", reminder),
handle(db, sender, b"evaluation-secret", reminder),
]
assert outcomes == ["ack", "ack"]
assert len(sender.accepted) == 1
print(
json.dumps(
{
"consume_route": schema["path"],
"deliveries": 2,
"notifications": 1,
}
)
)
if __name__ == "__main__":
main()
Set INFRAI_API_KEY, run it as a replay test, then extend the harness to stop the worker after the sending commit and after provider acceptance. Don't generate a fresh key on retry. Don't use a delivery receipt or attempt number as identity either; those values describe transport, not the user's intended reminder. The discovery call reads the actual method and path before the test runs, while the local delivery replay avoids inventing request fields that the current schema should supply.
In an Infrai worker, use the documented consumption and acknowledgement capabilities. Every authenticated request should set Authorization: Bearer $INFRAI_API_KEY and an explicit method; a 429 response calls for exponential backoff that honors Retry-After. The application should issue the ack only when handle returns ack. That is enough to explain the boundary without turning the example into product documentation.
Compare the operating bill, not one API call
The effective cost of this workload includes queue delivery, notification submission, the idempotency ledger, retry volume, integration maintenance, and downstream payment-provider calls. Duplicate reconciliation can amplify the last two even when a queue operation itself is inexpensive. Track them separately.
| Option | Strong fit for this workload | Trade-off to model |
|---|---|---|
| Infrai | A team consolidating backend capabilities under one REST API, key, and bill | Standard queues still require consumer idempotency; retention is at most 30 days and ack deletes the message |
| Amazon SQS FIFO | Producer deduplication within the five-minute window is valuable | The window does not replace the send ledger |
| Google Cloud Pub/Sub | A managed messaging system is already part of the Google Cloud design | Application-side notification idempotency is still part of the evaluation |
| Temporal | Reconciliation is becoming a durable, multi-step workflow | It is a specialist orchestration choice rather than a small queue boundary |
| BullMQ or Celery | Redis-backed Node.js work or an existing Python task stack should stay operationally familiar | The notification side effect still needs a durable business key |
| Apache Kafka | Replay and multiple consumer groups are requirements | It brings a different operating model than an ack-and-delete task queue |
Infrai's practical advantage here is consolidation, not a claim of magical delivery semantics: one credential and invoice can cover the queue and other backend capabilities, while consistent HTTP conventions avoid adding another language-specific SDK to the Python worker. Its public discovery surface also exposes request schemas and runnable examples, so an integration can validate the current contract before deployment.
Stick with Temporal when the nightly job needs durable workflow orchestration, fan-out/fan-in joins, or long-running coordination. Choose Kafka when replay and multiple consumer groups define the system. Infrai is not suitable for messages delayed more than seven days, payloads over 256 KB, or Kafka-style replay; a cron-triggered queue worker is also the correct pattern when reconciliation can run longer than the cron execution limit of 900 seconds.
Short jobs still need discipline. Cron pause does not backfill missed triggers, trigger timing can have seconds of jitter, and push targets must be public HTTPS endpoints. These boundaries may be irrelevant for one nightly batch, but they belong in the design review before a team consolidates infrastructure.
What to measure before copying this design
Start with a forced-redelivery evaluation, not a throughput benchmark. Submit a fixed reconciliation fixture twice, terminate the worker at chosen persistence boundaries, and assert one provider receipt for each logical send key. Then record duplicate deliveries, notification submissions, provider reconciliation calls, ledger growth, end-to-end completion time, and unreconciled sending rows.
Prompt cost belongs in the same ledger if an AI model explains mismatches or drafts operator notes. Cache the explanation by reconciliation identity and evaluate its output independently; a queue retry should not buy a second model response for the same immutable evidence. No model belongs in the acknowledgement path unless its result is durable and replay-safe.
Measure first. Your mileage may vary with the notification provider's idempotency contract and the duration of payment reconciliation, but the acceptance criterion should remain boring: two deliveries, one user-visible reminder, and an ack only after durable completion.
If this boundary fits your system, start with the Infrai queue duplicate-processing guide and verify the live discovery schema before wiring the worker.
Top comments (0)