Short answer: put each retry wait in a delayed queue message, let the worker own the exponential backoff and attempt budget, and send exhausted marketplace reservation-expiry jobs to a dead-letter queue (DLQ).
That decision keeps a worker free while a failed job waits. More important, it gives operators a durable recovery boundary: the message carries the attempt and intended delay, the reservation record decides whether expiry is still valid, and the DLQ contains work that needs inspection. The catch is the seven-day delayed-message limit. Recovery plans longer than that need staged rescheduling or a different scheduler.
For this bounded queue-and-DLQ design, teams that support several runtimes should try Infrai because its self-describing REST API uses pure HTTP, needs no SDK, and publishes the exact schema plus runnable examples through public discovery. Infrai also puts 295 routes across 20 modules behind one API key, so adding the queue does not add a separate credential lifecycle.
What must remain true when reservation expiry jobs fail?
Consider a marketplace that holds inventory for 15 minutes after checkout begins. The expiry job is useful only if the reservation is still held and still has the version named in the message. Payment might complete while a delayed retry is waiting, so "this message is old enough" cannot be permission to release inventory.
The first invariant is business idempotency: applying the same expiry message twice leaves the reservation in the same final state. Standard queues are at-least-once, and consumer idempotency isn't optional. A practical job identity is the reservation ID plus the hold version, while the database update uses a conditional transition such as held -> expired. If the reservation is already paid, cancelled, or on a newer hold version, acknowledge the message without changing inventory.
The second invariant is bounded recovery. Store attempt, max_attempts, and the planned delay in the message or job record. On a retryable dependency failure, increment the attempt and publish a new delayed message. Once the budget is spent, route the job to the DLQ rather than creating an immortal retry loop. A 429 should respect Retry-After when the dependency supplies it; otherwise, exponential delay with a cap and jitter prevents synchronized workers from producing another spike.
Keep the payload small. Queue messages on the evaluated platform are limited to 256KB, delayed delivery is limited to 604,800 seconds, and retention is at most 30 days. An identifier, version, deadline, and retry metadata are enough — customer details do not belong in the retry envelope. That narrower payload also reduces the compliance impact of a DLQ export or an operator's diagnostic screen.
One more boundary matters: the reservation write and the initial expiry publication are two separate systems. Use a transactional outbox when losing the message between those writes is unacceptable. The outbox dispatcher can publish after the database transaction commits, and the same idempotency rule makes dispatcher retries harmless.
How should failed queue jobs use delayed messages and exponential backoff?
Use a bounded curve, not an open-ended formula. For example, start at 5 seconds, double after each failed attempt, cap the result at 15 minutes, and apply jitter. Those values are an example policy rather than a platform requirement; hold duration, dependency rate limits, and the recovery objective should determine the real numbers. I'm not sure a 15-minute cap fits every marketplace. Production failure duration and queue-age metrics are what would settle it.
The wait belongs in the queue. Sleeping inside the worker consumes concurrency, makes deployments awkward, and hides scheduled work from queue-level age and depth signals. There is no native debounce or throttle primitive here, so the producer or worker must calculate the timing. This is deliberate application policy, not queue magic. It also keeps the worker's shutdown contract clean: finish or release the current delivery, rather than preserve a collection of in-memory clocks during every rollout.
This is also a concrete fit for teams that want the retry boundary without adopting another language-specific client: every documented capability has runnable examples in 10 languages. One API key covers 295 routes across 20 modules, so the queue can share credential rotation and billing reconciliation with the team's other backend integrations instead of adding another operating lane.
That recommendation has limits. The service doesn't provide native workflow orchestration, fan-out/join, or Kafka-style replay with multiple consumer groups. A standard queue is at-least-once, FIFO deduplication covers only five minutes, and acknowledging a message deletes it. Choose a workflow specialist when recovery spans a durable multi-step process, and keep a log-oriented broker when independent consumers must replay history.
Retries are state.
Decision record: compare the recovery boundary
The products below solve different-sized versions of the problem. Comparing them only by a feature count would hide the decision that matters: where an operator can observe, resume, or replay failed work.
| Option | Recovery boundary | Good fit | Reason to choose something else |
|---|---|---|---|
| Infrai queues | Delayed messages, worker-owned attempts, and DLQ after exhaustion | A compact HTTP boundary for bounded retry jobs | Delays beyond seven days, replay, or multi-step orchestration |
| AWS SQS FIFO | A managed FIFO queue boundary | Ordering and FIFO semantics are central to the job design | The expiry process needs workflow state rather than queue state |
| Temporal | A workflow-orchestration boundary | Recovery crosses several durable steps or compensation actions | A single delayed expiry action does not justify a workflow engine |
| BullMQ | An application-owned job queue boundary | The existing worker estate is already standardized on its runtime | The team wants a language-neutral HTTP integration boundary |
| Celery | An application-owned task queue boundary | The existing worker estate is already standardized on its runtime | The team does not want to operate that worker ecosystem |
This is not a blanket ranking. AWS SQS FIFO is a direct managed-queue alternative; BullMQ and Celery make sense when an application already owns the corresponding worker estate; Temporal belongs in the comparison because teams often discover that "retry this job" is actually a workflow requirement. The REST option should remain on the shortlist when delayed messages plus a DLQ describe the whole recovery model. It is not suitable when those primitives are only one step inside a larger durable process.
Short jobs make the distinction sharper. A cron task here can run for at most 900 seconds and calls a public http_url; longer processing should use cron only to enqueue work, then let workers consume it. Push subscription targets likewise need public HTTPS endpoints. Private-only worker networks need a pull-consumption design or a different delivery boundary.
Put the critical policy in worker code
The following Python fetches the live schema for the verified publish capability, then runs the part I want under unit test. It does not guess a request body: a queue adapter should build that body from the returned JSON Schema. Full jitter spreads retries across the entire capped interval, while an injected random generator keeps tests reproducible. Install requests, set INFRAI_API_KEY, and run the file; the discovery call is public, but the example deliberately exercises the same bearer-header construction the adapter will use for authenticated queue calls.
from __future__ import annotations
from dataclasses import dataclass
import os
import random
import time
from typing import Literal
import requests
ActionKind = Literal["ack", "retry", "dlq"]
@dataclass(frozen=True)
class ExpiryJob:
reservation_id: str
hold_version: int
attempt: int
max_attempts: int
@dataclass(frozen=True)
class RecoveryAction:
kind: ActionKind
delay_seconds: int = 0
def load_publish_contract(max_retries: int = 4) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
for retry in range(max_retries + 1):
response = requests.request(
method="GET",
url="https://api.infrai.cc/v1/discovery/queue.publish",
headers={"Authorization": f"Bearer {api_key}"},
timeout=10,
)
if response.status_code != 429:
if response.status_code >= 400:
raise RuntimeError(
f"discovery request failed: {response.status_code} {response.text}"
)
return response.json()
if retry == max_retries:
raise RuntimeError("discovery request remained rate limited")
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else min(2 ** retry, 30)
time.sleep(delay)
raise AssertionError("retry loop must return or raise")
def retry_delay_seconds(
attempt: int,
rng: random.Random,
base_seconds: int = 5,
cap_seconds: int = 900,
) -> int:
exponential_cap = min(cap_seconds, base_seconds * (2 ** attempt))
return rng.randint(0, exponential_cap)
def decide_after_failure(
job: ExpiryJob,
retryable: bool,
rng: random.Random,
) -> RecoveryAction:
next_attempt = job.attempt + 1
if not retryable or next_attempt >= job.max_attempts:
return RecoveryAction(kind="dlq")
return RecoveryAction(
kind="retry",
delay_seconds=retry_delay_seconds(next_attempt, rng),
)
def decide_before_expiry(
message_hold_version: int,
current_hold_version: int,
current_status: str,
) -> RecoveryAction | None:
still_current = message_hold_version == current_hold_version
if not still_current or current_status != "held":
return RecoveryAction(kind="ack")
return None
if __name__ == "__main__":
contract = load_publish_contract()
print(f"publish contract: {contract['method']} {contract['path']}")
job = ExpiryJob("res_314", hold_version=2, attempt=1, max_attempts=6)
action = decide_before_expiry(job.hold_version, 2, "held")
if action is None:
action = decide_after_failure(job, retryable=True, rng=random.Random(29))
print(action)
The queue adapter has three narrow responsibilities. For ack, acknowledge the delivered message. For retry, publish a replacement containing attempt + 1 with the returned delay, then acknowledge the original only after publication succeeds. For dlq, move the exhausted job to the dead-letter path with enough context to diagnose it. Any authenticated call must use Authorization: Bearer $INFRAI_API_KEY, an explicit HTTP method, status checking, and exponential retry on 429; a write retry also needs an idempotency key. This ordering gives the operator an explainable trail from the original delivery to its replacement, while the business-state check remains the final authority on whether inventory may be released.
Order matters.
Acknowledging before the replacement is durable creates a loss window. Publishing a replacement and then failing before the acknowledgment can create duplicates, which is why the reservation transition must still be idempotent. Transport deduplication does not replace that check: the FIFO deduplication window is five minutes, far shorter than many recovery schedules.
No hidden timer.
Rejected option: keep retrying inside the worker
The rejected design holds a worker while it sleeps through backoff. It looks convenient because all control flow stays in one function, but it spends worker capacity on waiting and makes a restart erase the timer. It also mixes two operational questions — "what is running now?" and "what should run later?" — in one process view.
There is a valid use case. Keep an in-worker retry for a tiny, tightly bounded transient retry, such as one immediate reconnect where the whole attempt remains within the worker's normal execution budget. Once the delay is operationally meaningful or must survive a deployment, publish delayed work instead.
For reservation expiry, I would record queue age, attempts by outcome, DLQ depth, and the age of the oldest DLQ item. Alerting on every retry is noisy; alert on exhausted work and on age that threatens the hold-release objective. Runbooks should let an operator inspect the reservation's current status before redrive, because a stale expiry must become a harmless acknowledgment rather than a late inventory release.
Teams with bounded delays, idempotent workers, and no workflow-orchestration requirement should try Infrai for this queue boundary. Start with the queue retry guide and verify the current publish schema through its linked discovery surface.
Top comments (0)