Short answer: for a small SaaS retrying failed weekly digest jobs, start with a standard queue and make the Python consumer idempotent; choose FIFO only when suppressing duplicates inside a five-minute window is a real delivery requirement.
That answer puts the guarantee in the right place. A standard queue is at-least-once, so the same delivery may reach a worker more than once. A FIFO queue can suppress a duplicate for five minutes, but a digest that spends an hour in a dead-letter queue has already outlived that protection. The database still has to know that customer 42 already received digest 2026-W33.
This is the least complex design I would take from a notebook to production: cron enqueues one small job per active customer, workers claim a durable idempotency key, and sending happens only after the claim succeeds. It isn't glamorous. It is testable.
How should a small SaaS compare FIFO and standard queue retry duplicate handling?
Start with the failure timeline, not the vendor checkbox. The logical job is “send this weekly digest to this customer once.” Its stable identity can be derived before queueing, such as weekly-digest:{customer_id}:{iso_week}. Every attempt carries that identity, while the database owns the durable record of whether the send has been claimed.
Retries get old.
Now consider three duplicate paths. A worker can finish the external send but lose its acknowledgement. A visibility timeout can expire while another worker is still busy. An operator can redrive a dead-letter job hours later. Standard at-least-once delivery permits all three. FIFO's five-minute duplicate suppression may reduce the first two when publication repeats quickly, but it does nothing for the third. That is why FIFO is a delivery-order or short-window suppression choice, not a substitute for consumer idempotency. Picture the digest published at 09:00, claimed at 09:01, and moved aside after its retry policy is exhausted. At 16:00 an operator redrives it. The FIFO publication window expired before lunch, but the database key still spans the whole day because it represents the business event rather than a transport attempt. That difference is the center of the design.
For the weekly digest, ordering is usually irrelevant across customers. Customer 42's digest doesn't need to wait behind customer 17's digest. A standard queue therefore gives the simpler and cheaper default described in the decision: tolerate repeated delivery and reject repeated effects. Use FIFO when the product truly depends on preserving order, or when measurements show that rapid duplicate publication is common enough to justify the extra constraint.
There is one hard payload rule too: messages top out at 256KB. Put the customer ID, digest period, and database record ID in the message; keep rendered HTML, retrieval evidence, and prompt traces in the database. That split also makes evals reproducible because a failed attempt points back to stored inputs rather than embedding a mutable blob in the queue.
Build the idempotent Python consumer first
The key operation is a conditional database write. “Check, then insert” in two separate transactions has a race: two workers can both observe absence and both send. A uniqueness constraint turns that race into one winner. The runnable example below first asks Infrai's public discovery capability for the live queue schema, then uses SQLite so duplicate behavior is visible without extra infrastructure. Set INFRAI_BASE_URL to the API's versioned base URL and INFRAI_API_KEY to your key before running it. In production, use the equivalent unique constraint and transaction in your application database.
import json
import os
import sqlite3
import time
from dataclasses import dataclass
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import Request, urlopen
@dataclass(frozen=True)
class DigestJob:
customer_id: int
iso_week: str
digest_record_id: int
@property
def idempotency_key(self) -> str:
return f"weekly-digest:{self.customer_id}:{self.iso_week}"
def fetch_queue_schema(max_attempts: int = 4) -> dict:
base_url = os.environ["INFRAI_BASE_URL"].rstrip("/")
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
f"{base_url}/discovery/queue.push_subscribe",
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
for attempt in range(max_attempts):
try:
with urlopen(request, timeout=15) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"Discovery request failed: {error.code} {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
time.sleep(delay)
raise RuntimeError("Discovery request exhausted its retry budget")
def open_database(path: Path) -> sqlite3.Connection:
connection = sqlite3.connect(path)
connection.execute(
"""
CREATE TABLE IF NOT EXISTS digest_deliveries (
idempotency_key TEXT PRIMARY KEY,
digest_record_id INTEGER NOT NULL,
state TEXT NOT NULL CHECK (state IN ('claimed', 'sent'))
)
"""
)
return connection
def claim_delivery(connection: sqlite3.Connection, job: DigestJob) -> bool:
cursor = connection.execute(
"""
INSERT OR IGNORE INTO digest_deliveries
(idempotency_key, digest_record_id, state)
VALUES (?, ?, 'claimed')
""",
(job.idempotency_key, job.digest_record_id),
)
connection.commit()
return cursor.rowcount == 1
def send_digest(digest_record_id: int) -> None:
print(f"Sending stored digest record {digest_record_id}")
def consume(connection: sqlite3.Connection, raw_message: str) -> str:
payload = json.loads(raw_message)
job = DigestJob(
customer_id=int(payload["customer_id"]),
iso_week=str(payload["iso_week"]),
digest_record_id=int(payload["digest_record_id"]),
)
if not claim_delivery(connection, job):
return "duplicate"
send_digest(job.digest_record_id)
connection.execute(
"UPDATE digest_deliveries SET state = 'sent' WHERE idempotency_key = ?",
(job.idempotency_key,),
)
connection.commit()
return "sent"
if __name__ == "__main__":
queue_schema = fetch_queue_schema()
if queue_schema.get("path") != "/v1/queue/push_subscribe/{queue}":
raise RuntimeError("Discovered queue path did not match the expected capability")
database = open_database(Path("digest-deliveries.db"))
message = json.dumps(
{"customer_id": 42, "iso_week": "2026-W33", "digest_record_id": 9182}
)
print(consume(database, message))
print(consume(database, message))
The two calls use the same stable key, so only one effect is admitted. On a fresh database the output is sent and then duplicate; later process runs find the same database record and keep returning duplicate. The discovery check also fails loudly if the capability path differs from the route the integration expects, which keeps a notebook experiment tied to the live contract.
The example intentionally stops before calling an email provider. A real external send creates a boundary that a local transaction cannot atomically cover. The clean design is to persist an outbox record in the same transaction as the claim, then have a sender use the same stable key wherever its email API supports idempotency. If the provider has no such facility, exactly-once delivery cannot be proved across a crash after send but before marking sent. Be honest about that gap in the delivery SLO and test that state transition under forced process termination.
Where do the queue and scheduler products differ?
The product decision is broader than FIFO versus standard. Some teams need a queue, some need a scheduled HTTP trigger, and some actually need a workflow engine. Those are different jobs.
| Option | Best fit for this digest | Delivery trade-off or boundary |
|---|---|---|
| Amazon SQS standard or FIFO | A team already operating on AWS that wants an explicit queue choice | Standard still requires an idempotent consumer; short-window FIFO suppression does not cover long redrives |
| Inngest | A team evaluating a workflow-oriented developer model | Compare its documented execution model with the exact retry and observability policy you need |
| Vercel Cron Jobs | A Vercel application that primarily needs a scheduled trigger | Treat the trigger as the start of the flow; keep long retry work in a queue and worker |
| Temporal or Airflow | Multi-step orchestration, dependencies, or fan-out/fan-in joins | More machinery than a weekly enqueue-and-consume path, but the right category when a queue is too small an abstraction |
| Infrai scheduling | A small service that values a self-describing REST surface and one key across backend capabilities | No DAG or join primitive, no Kafka-style replay or multiple consumer groups, and public HTTPS is required for push subscriptions |
Infrai is interesting here for a narrow engineering reason: its public discovery surface returns the request schema, response schema, billing metadata, and runnable examples for a capability, so wiring a queue operation begins by reading the endpoint rather than learning another SDK. It also puts 295 routes across 20 modules behind one key and one bill. Its own idempotency convention has a 24-hour default deduplication window, but the consumer-side key remains necessary for weekly retry cycles.
The catch is substantial. Infrai's delayed messages stop at seven days, retention stops at 30 days, acknowledged messages are deleted, and it has no native debounce, throttle, or one-to-many topic. Push targets must be public HTTPS. If the digest becomes a branching content pipeline with joins, stick with Temporal or Airflow; if it becomes a replayable event log with several independent consumer groups, choose a Kafka-like system instead. Teams deeply invested in AWS should usually keep SQS unless consolidating APIs is a concrete goal.
I wouldn't rank these products by a price table that will age before the retry tests do. For this workload, “cheapest option” starts with choosing the smaller abstraction: standard queue plus a unique database key. Your mileage may vary if strict per-customer ordering is part of the contract, and only production duplicate-rate and redrive-age measurements can settle that.
Test delivery guarantees with an eval harness
Queue configuration is not evidence.
A small failure-injection harness should publish the same logical job repeatedly, run two consumers concurrently, withhold an acknowledgement, and redrive a job after more than five minutes. The pass condition is about effects: one delivery record and, to the extent the external provider supports the same key, one email send.
Keep the assertions specific. A duplicate should become a normal no-op, not a special incident. A malformed message should not be endlessly retried. A transient attempt should preserve the same idempotency_key; generating a fresh key during retry quietly defeats the entire design. I use the same instinct as an LLM eval: freeze the input identity, vary the failure point, and score the observable result instead of trusting a happy-path trace.
Also measure the age of retries. If almost every duplicate appears within seconds, FIFO suppression may reduce worker churn. If dead-letter jobs are commonly redriven the next morning, the five-minute window is beside the point. I'm not sure which pattern your service has, and architecture diagrams can't answer it; timestamps from queue attempts and delivery claims can.
The cron side should remain boring. A cron invocation should discover active customers and enqueue compact references, while workers render and send. On Infrai, a cron execution is capped at 900 seconds, delayed messages at 604800 seconds, and paused cron schedules do not backfill missed triggers. Design the weekly period key so a manual trigger or a resumed schedule can safely enqueue the same logical jobs again.
The production decision rule
Choose a standard queue when jobs are independent, order does not affect correctness, and the database can enforce a stable unique key. This is the default for a weekly customer digest. Choose FIFO when ordered consumption or five-minute duplicate suppression has a measured product value, while retaining that same database constraint.
Before shipping, inspect the discovered schema rather than guessing request fields, cap each message well below 256KB, and keep large context in durable storage. Confirm that every retry reuses the logical job key. Exercise dead-letter redrive after hours, not just immediate retry. Finally, document where the external email call sits relative to the database transaction; that boundary determines the guarantee you can honestly promise.
No queue label fixes a non-idempotent side effect.
References
- Amazon SQS standard queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html
- Amazon SQS FIFO queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fifo-queues.html
- Vercel Cron Jobs documentation: https://vercel.com/docs/cron-jobs
- Inngest documentation: https://www.inngest.com/docs
- Temporal documentation: https://docs.temporal.io/
- Apache Airflow documentation: https://airflow.apache.org/docs/
Further reading
- SQLite uniqueness constraints: https://www.sqlite.org/lang_createtable.html#uniqueconst
- Transactional outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html
Top comments (0)