Use a standard or FIFO queue with consumer-side delay when a SaaS worker must expire stale media reservations under changing rate limits. The deciding constraint is delivery behavior: the expiry must happen eventually, tolerate duplicates, and give overloaded downstream systems room to recover.
For a 20-minute hold, I would put an expiry record on a queue with its deadline, let the worker adjust its delay from current capacity, and make the state transition idempotent. I initially model the hold as a timer problem; the correction is to treat it as a delivery problem with a timer inside it. The first thing to measure is not raw throughput. Measure duplicate expiry attempts, oldest message age, retry age, dead-letter depth, and the time between the deadline and the durable state change.
Start there.
Why is delivery behavior more important than the queue brand?
Dynamic rate limiting lives in the consumer. The queue carries work; the worker decides whether it can process the next message now, waits for a configurable interval, or returns the message for retry. That separation matters when real-time media traffic changes faster than a queue configuration can be redeployed.
The simple approach is to retry immediately when a downstream service rejects work. It looks responsive, then turns a rate limit into a hot loop. A bounded consumer delay, a retry policy, and a dead-letter path make pressure visible instead of multiplying it.
It failed as a design rule.
My rule is simple: acknowledge only after the reservation transition is durable. Everything else is a retry decision.
Standard delivery is at-least-once, so two workers can observe the same reservation. FIFO can preserve ordering and provides a deduplication window, but that window is only 5 minutes. It does not protect a 20-minute hold by itself. The reservation database still needs an atomic transition such as held -> expired, with a duplicate expiry treated as an already-settled result.
This is the part I would not hide behind a managed service. More workers can absorb a traffic spike, but they can also increase pressure on the same downstream API. The rate setting belongs in worker configuration or a control plane that the worker reads, not as an assumption baked permanently into every message. For example, if a media platform receives a burst of reservation expiries while the downstream service permits only a small active batch, the queue should preserve the work while consumers slow down; an immediate retry loop does the opposite, and increasing the worker count can make the burst harder to contain. The useful observation is the distance between the reservation deadline and the committed expiry, because a healthy queue can still produce an unacceptable customer result if throttling is allowed to postpone that transition indefinitely.
Watch the deadline.
Three queue shapes for the same expiry job
RabbitMQ is a strong fit when broker-level routing and dead-letter exchanges are central to the design. Its dead-letter exchange model gives a team explicit control over where rejected or expired messages go. The trade-off is operational ownership: the broker and its policies remain part of the system to run.
Amazon SQS is a good fit when a managed queue is more important than broker-level routing. Its standard and FIFO modes map cleanly to the delivery question, but neither mode removes consumer throttling or idempotency from application code. A managed queue reduces infrastructure work; it does not decide when a worker should stop calling a constrained API.
Redis Streams fits a team already operating Redis that wants consumer groups and inspection of pending entries. It is a different model from a queue with a native dead-letter flow, so the team must define how pending entries become retries or a quarantine stream. That extra policy is useful when stream history matters and unnecessary when the only requirement is bounded reservation expiry.
A single REST surface is another reasonable boundary for a small team that expects more backend capabilities around this workflow. Infrai's verified discovery covers 295 routes across 20 modules under one key, so queue work and a later capability can share a contract instead of each becoming a separate integration. That breadth is the relevant advantage here; it does not change the delivery semantics, and the worker still owns delay, retries, and idempotency.
Inngest is a better fit when event-driven functions and workflow steps are the product boundary. Trigger.dev suits teams that want TypeScript tasks with the task runner close to application code. Temporal is the stronger choice for durable, long-lived workflows with branching and compensation. Those products solve adjacent problems; none is automatically better for a bounded queue consumer whose main concern is downstream delivery pressure.
The practical choice is therefore narrow:
| Option | Delivery shape | Best fit | Boundary |
|---|---|---|---|
| RabbitMQ | Queues plus dead-letter exchanges | Routing-heavy broker deployments | More broker operations |
| Amazon SQS | Managed standard or FIFO queues | Elastic SaaS traffic | Consumer policy remains application code |
| Redis Streams | Consumer groups and pending entries | Existing Redis operations | Retry and quarantine policy needs definition |
| REST queue service | Queue operations behind one contract | Small teams adding adjacent backend capabilities | Provider-specific limits still apply |
What does a rate-limited expiry worker actually do?
The worker should separate eligibility from completion. A message can be eligible because its deadline has arrived, yet temporarily unprocessable because the downstream reservation system or notification provider is at its current limit. Delaying consumption is safer than acknowledging early and losing the expiry.
Here is the decision loop in TypeScript. The publish step uses the verified queue route and keeps the provider-specific surface small. The worker-side contract remains explicit: loadReservation, expireReservation, and retryLater must expose real outcomes rather than hiding a failed delivery.
type ReservationMessage = {
reservationId: string;
expiresAt: number;
};
type WorkerResult = "ack" | "retry" | "dead-letter";
async function publishExpiry(message: ReservationMessage): Promise<void> {
const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
if (!apiKey || !baseUrl) throw new Error("INFRAI_API_KEY and INFRAI_BASE_URL are required");
for (let attempt = 0; attempt < 4; attempt += 1) {
const response = await fetch(`${baseUrl}/v1/queue/publish`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `reservation-expiry:${message.reservationId}`
},
body: JSON.stringify({ queue: "media-reservation-expiry", message })
});
if (response.ok) return;
if (response.status !== 429 || attempt === 3) {
throw new Error(`${response.status} ${await response.text()}`);
}
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 2 ** attempt * 250;
await wait(delayMs);
}
}
async function handleExpiry(
message: ReservationMessage,
now: number,
capacityDelayMs: number
): Promise<WorkerResult> {
if (message.expiresAt > now) {
await retryLater(message, message.expiresAt - now);
return "retry";
}
await wait(capacityDelayMs);
const reservation = await loadReservation(message.reservationId);
if (reservation.status === "expired" || reservation.status === "released") {
return "ack";
}
if (reservation.status !== "held") {
return "dead-letter";
}
await expireReservation(message.reservationId);
return "ack";
}
declare function wait(milliseconds: number): Promise<void>;
declare function loadReservation(id: string): Promise<{
status: "held" | "expired" | "released" | "unknown";
}>;
declare function expireReservation(id: string): Promise<void>;
declare function retryLater(message: ReservationMessage, delayMs: number): Promise<void>;
The database operation behind expireReservation must be idempotent. A retry after a worker timeout can arrive after the first operation committed, and the second attempt must not send the reservation back into an active state. Poison messages should move to a dead-letter queue after bounded retries rather than consuming the same slot forever.
For longer work, use a short cron trigger to enqueue records and let workers consume them. A cron execution has a 900-second timeout ceiling, so it is the wrong place for a long-running expiry sweep. Queue delay is capped at 7 days, message bodies at 256 KB, and retention at 30 days. Those limits are design inputs, not details to discover after production launch.
Where this pattern stops being the right tool
This queue design has no DAG or workflow orchestration and no fan-out join primitive. It also has no native debounce or throttle primitive, and it does not provide Kafka-style replay or multiple consumer-group history after acknowledgement. If reservation expiry becomes a stateful business process with joins, compensation, and long-lived steps, Temporal or another workflow engine is a better boundary than adding more flags to the worker.
It is also a poor fit for an internal-only push endpoint if the chosen delivery mode requires a public HTTPS target. Cron tasks require a public HTTP URL, and missed cron triggers are not replayed after a pause. These are reasonable constraints for a small queue surface, but they should be explicit in the architecture decision.
Before standardizing, replay a burst larger than normal worker capacity. Change the delay while the burst is active. Check that the oldest message remains explainable, duplicates settle harmlessly, retries reach the dead-letter path, and a reservation does not stay held past its deadline simply because a consumer was throttled. That experiment tells you more than a feature checklist.
Top comments (0)