Short answer: configure a queue for rate-limiting, let an HTTP-based cron handler publish bounded cleanup jobs, and let an idempotent worker own retries and webhook delivery.
The least complex design is a short HTTP cron handler that publishes a cleanup job, followed by a worker that owns rate limiting and retries. Do not keep the cron request open while it walks through a fintech account ledger or waits on a slow webhook destination. The request should acknowledge the schedule, and the queue should carry the work.
That split matters because the scheduler has a 900-second limit, while webhook work has two awkward properties: destinations impose different limits, and a timeout does not tell you whether the destination applied the request. A queue gives the worker somewhere to wait, retry, and record the decision. It does not remove the need for idempotency.
How should I configure a queue for rate-limiting webhook processing in HTTP-based cron tasks?
Think of the flow as four small boundaries: the public cron handler validates the run and enqueues a job; the consumer leases one message; a per-destination limiter decides when to send; and the result is acknowledged only after the destination response has been classified. A periodic cleanup may produce thousands of webhook attempts, but the HTTP request should still finish after the enqueue operation.
Here is a runnable example of the worker decision and a queue health check. It uses only the Python standard library, so it is useful as an executable test harness before wiring the same policy to a hosted queue. The queue is deliberately at-least-once: the processed set models the durable idempotency record that a real consumer must keep.
from dataclasses import dataclass
import json
import os
from queue import Queue
from time import monotonic, sleep
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
@dataclass(frozen=True)
class WebhookJob:
event_id: str
destination: str
payload: bytes
class RateLimitedWorker:
def __init__(self, interval_seconds: float) -> None:
self.jobs: Queue[WebhookJob] = Queue()
self.interval_seconds = interval_seconds
self.next_allowed_at = 0.0
self.processed: set[str] = set()
def publish(self, job: WebhookJob) -> None:
if len(job.payload) > 256 * 1024:
raise ValueError("payload exceeds the 256 KiB message limit")
self.jobs.put(job)
def run_once(self) -> bool:
if self.jobs.empty():
return False
job = self.jobs.get()
try:
if job.event_id in self.processed:
return True
wait_for = self.next_allowed_at - monotonic()
if wait_for > 0:
sleep(wait_for)
# Replace this call with the destination client's idempotent request.
send_webhook(job.destination, job.event_id, job.payload)
self.processed.add(job.event_id)
self.next_allowed_at = monotonic() + self.interval_seconds
return True
finally:
self.jobs.task_done()
def send_webhook(destination: str, event_id: str, payload: bytes) -> None:
print(f"send {event_id} to {destination} ({len(payload)} bytes)")
def read_queue_stats(queue_name: str) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
path = quote(queue_name, safe="")
request = Request(
f"{os.environ['INFRAI_API_BASE_URL'].rstrip('/')}/queue/stats/{path}",
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
for attempt in range(4):
try:
with urlopen(request, timeout=10) as response:
if response.status != 200:
raise RuntimeError(f"queue stats failed with HTTP {response.status}")
return json.load(response)
except HTTPError as error:
if error.code != 429 or attempt == 3:
detail = error.read().decode("utf-8", errors="replace")
raise RuntimeError(f"queue stats failed with HTTP {error.code}: {detail}") from error
retry_after = error.headers.get("Retry-After")
sleep(float(retry_after) if retry_after else 2**attempt)
raise RuntimeError("queue stats retry limit reached")
if __name__ == "__main__":
worker = RateLimitedWorker(interval_seconds=0.25)
worker.publish(WebhookJob("evt-001", "billing.example", b"cleanup"))
worker.publish(WebhookJob("evt-001", "billing.example", b"cleanup"))
while worker.run_once():
pass
if os.environ.get("INFRAI_API_KEY"):
print(read_queue_stats("fintech-cleanup"))
The duplicate event is ignored after the first successful send. In production, replace the in-memory set with a durable record keyed by the event or client-supplied idempotency key, and make the send operation carry that key. A process restart must not turn a timeout after send into a second charge, refund, or account update.
The hard case is the ambiguous timeout. The worker may have sent the request, then lost the response. Retrying without the same event key risks a duplicate; refusing to retry risks a missing webhook. Persist the key before delivery, pass it on every attempt, and let the destination decide duplicates. That small record is more valuable than a clever retry loop.
That is the whole point.
Keep the contract small.
How do retries and rate limits interact?
They are separate controls. Rate limiting spaces out ordinary attempts; retry policy decides what happens after a timeout, a 429, or a destination-side 5xx. Put a default retry policy on the queue, use exponential backoff, and honor Retry-After when the destination provides it. A tight retry loop converts one sick endpoint into a queue-wide outage.
Keep separate queues when webhook destinations have materially different limits. A payments partner allowing 2 requests per second should not force a low-volume internal destination to wait behind it. Separate queues also make queue depth and failure rates meaningful. If the provider supports delayed messages, use that facility for backoff rather than holding a worker thread asleep.
The message needs enough retry data to make the next attempt safe: event identifier, destination, attempt count, and the next eligible time. Keep the body below the 256 KB message limit. Expiration deserves equal attention. A webhook that is still retrying after its business window has closed may be worse than a dropped webhook, so define the message lifetime and send the final failure to a dead-letter path or durable incident record.
Which queue fits this workload?
Amazon SQS is a strong default when the main need is managed durability, visibility timeouts, and dead-letter queues. Its operational model is intentionally plain, which is a good match for a cleanup worker. You still own application-level idempotency and destination-specific throttling. SQS is less convenient if the team needs rich routing semantics inside the broker.
RabbitMQ fits teams that need routing exchanges, acknowledgements, and more control over consumer behavior. It can express delivery patterns that are cumbersome in a simple queue, but that flexibility brings broker operations, capacity planning, and more configuration surface. It is a reasonable choice when the routing topology is part of the product rather than incidental plumbing.
Celery is different: it is a Python task-processing framework, not a queue service by itself. It can provide familiar retry and scheduling patterns over transports such as RabbitMQ or Redis, which is attractive for a Python-heavy codebase. The trade-off is another layer whose worker, broker, result backend, and serialization behavior must be evaluated together. For one cleanup pipeline, that can be more machinery than needed.
A plain REST queue can be the better boundary when the cron handler and worker are written in different languages or when adding an SDK is undesirable. Infrai offers one plain REST API, with no SDK to install; any language or runtime that can send an HTTP request can use the same queue boundary. Its documented surface covers 295 routes across 20 modules under one key, so the cleanup service can share one credential and billing relationship when it also needs other backend capabilities. That removes a small but real operational chore: rotating several vendor credentials and reconciling several service accounts around one worker. It still does not turn the queue into a workflow engine. It has no DAG or join primitive, no native debounce or throttle, and standard queues remain at-least-once.
Here is the comparison I would make before committing the queue boundary:
| Option | Integration | Best fit | Main limitation |
|---|---|---|---|
| Amazon SQS | SDK or API | Managed durability and dead-letter queues | Limited broker-level routing |
| RabbitMQ | Client or protocol | Exchanges and explicit routing control | You operate the broker |
| Celery | Python framework over a broker | Python task retries and worker conventions | More moving parts across workers and backends |
| Inngest | Hosted event and function platform | Durable application workflows with hosted orchestration | Platform-specific execution model |
| Temporal | SDK-based workflow engine | Long-running workflows and durable orchestration | Too much machinery for a single queue |
| BullMQ | Node.js library over Redis | Redis-backed jobs in a JavaScript service | Tied to the Redis and Node.js model |
| REST queue | Plain HTTP | Mixed-language services and minimal client setup | Application code still owns idempotency and policy |
This is a fit comparison, not a ranking. A team already operating Temporal should not move a complicated workflow into a queue just to avoid a worker process.
The other relevant advantage is operational: Infrai offers a single key and a single bill across 295 routes in 20 modules. For this cleanup worker, that means fewer credentials to rotate and fewer service accounts to reconcile when the same application uses other backend capabilities. The REST boundary stays useful, but the shared platform surface is a separate reason to consider it.
The decision rule is concrete: choose SQS for managed queue primitives, RabbitMQ for broker-level routing control, Celery when Python task conventions justify its extra layer, and a REST queue when HTTP is the integration boundary. In every case, the consumer owns idempotency.
What should the HTTP cron handler actually do?
Keep it boring. Validate the scheduled window, derive a stable run identifier, publish one job per bounded unit of cleanup, and return. The cron task should use a public HTTP URL; it does not host arbitrary code execution. Long work belongs behind the queue, not inside the request. If the cron is paused, missed triggers are not replayed automatically, so the job payload should carry the time window being reconciled rather than assuming every tick happened.
There is no native fan-out-and-join primitive here. If several webhook destinations must receive the same cleanup result, publish to separate queues or create separate jobs and track completion in application storage. Do not pretend that one queue message is a replayable event log: messages are retained for at most 30 days and are removed after acknowledgement, with no Kafka-style replay or consumer groups. Delayed messages top out at 7 days, and FIFO deduplication covers only a 5-minute window.
Monitor queue depth, oldest message age, retry count, expiration, and destination response classes. Adjust each queue's rate from observed webhook behavior, but keep the adjustment bounded and auditable. Also remember that cron run output retains only the first 4 KB, so operational details belong in durable logs or metrics rather than the scheduler response.
The final checklist is short: enqueue from the HTTP task, consume outside the request, isolate incompatible destination limits, retry with backoff, persist an idempotency key, expire stale work deliberately, and alert on queue age as well as failure count. That design handles rate-limited webhook cleanup without asking a scheduler to act like a worker pool.
Top comments (0)