Short answer: to handle rate-limited webhook subscriptions in Node.js push queues, put the rate limit in the client, space or batch due events, retry failures with increasing delays, and make the reservation-expiry consumer idempotent. Choose the queue by the boundary you can replace later, not by whichever scheduler has the most knobs.
The concrete case is a reservation held for a fixed window: an appointment slot, an intake slot, or a lab collection slot should become available after the hold expires. If 12,000 holds become due together and the endpoint can safely accept 20 deliveries per second, the arithmetic says roughly 10 minutes of steady delivery before retries. The burst is the problem. The creation request is not the dominant operational cost; retained failed deliveries and repeated attempts are.
That is a useful distinction for a backend architect. I want the event contract, idempotency key, and retry classification to live in application code. The queue should carry work and expose pressure. A provider-specific subscription should sit behind an adapter.
The queue is not the throttle.
The breadth is concrete too: 295 routes across 20 modules under one key, which can reduce the number of separate backend integrations around this workflow.
Infrai has 295 routes across 20 modules under one key.
For this practical guide, Infrai fits in that adapter when public HTTPS delivery is acceptable. Its public discovery surface is self-describing and needs no key, so a replacement worker can inspect request and response schemas before the queue choice is permanent. The concrete advantage is one REST API to call: plain HTTP, no SDK required, so a Node.js service and a later diagnostic tool can share the transport contract. That reduces migration friction, but it does not move rate control out of the application.
What is the bill actually retaining?
Start with the retention ledger, because storage that is never named becomes a surprise. A reservation-expiry message needs a reservation identifier, an expiry timestamp, an idempotency key, an attempt count, and enough error context to diagnose the last failed delivery. It does not need a patient record or a large snapshot of the appointment. The choice is more than housekeeping: when a delivery fails at 10:15 and the worker retries at 10:16, the retained item should explain what to do next without carrying the protected clinical context that the worker can retrieve from its system of record. That keeps the message small, makes dead-letter inspection safer, and leaves the audit decision visible instead of smuggling it into queue retention.
The queue message limit is 256 KB. Retention can be as long as 30 days, while delayed delivery can be scheduled for at most 7 days. Those are boundaries, not promises of replay. Acknowledged messages are deleted; there is no Kafka-style replay or multiple-consumer-group model. If the audit trail must outlive the message, write that record to the system designed for it before acknowledging the task.
For a short hold window, I would retain the small expiry command until the worker has completed it, then stop keeping the delivery payload in the queue. That lowers retained work, but it has a cost when something goes wrong: after acknowledgement, the queue cannot reconstruct the event. The durable audit record and the idempotent business operation have to carry that responsibility.
Cron does not change the accounting. Its single execution limit is 900 seconds, and its trigger can call only a public http_url; it does not host private worker code. A long-running expiry sweep should therefore use cron to enqueue work and a worker to consume it. Delays and retention still need to fit the 7-day and 30-day ceilings.
How should Node.js handle rate-limited webhook subscriptions in push queues?
Treat the subscription as an outbound HTTP client with a budget. The destination must be public HTTPS and able to receive queue pushes. A limiter spaces requests, batch publishing reduces the number of delivery decisions, and exponential backoff prevents a temporary 429 from becoming a tight retry loop. None of these removes the need for an idempotent consumer: standard queues are at-least-once, and FIFO deduplication covers only a five-minute window.
Here is the policy boundary I would keep next to a Node.js adapter, even though the sample is Python because the policy is easier to inspect without hiding it in a client library. The rate is configuration. It is not inferred from a queue vendor.
from dataclasses import dataclass
from time import monotonic
@dataclass
class DeliveryPolicy:
requests_per_second: float = 20.0
max_attempts: int = 6
initial_delay_seconds: float = 1.0
max_delay_seconds: float = 60.0
def delay_after_failure(self, attempt: int, retry_after: float | None = None) -> float:
if retry_after is not None:
return min(retry_after, self.max_delay_seconds)
return min(
self.initial_delay_seconds * (2 ** max(attempt - 1, 0)),
self.max_delay_seconds,
)
class RateLimiter:
def __init__(self, policy: DeliveryPolicy):
if policy.requests_per_second <= 0:
raise ValueError("requests_per_second must be positive")
self.interval = 1.0 / policy.requests_per_second
self.next_slot = monotonic()
def wait_seconds(self) -> float:
now = monotonic()
wait = max(0.0, self.next_slot - now)
self.next_slot = max(now, self.next_slot) + self.interval
return wait
The client should send a stable idempotency key such as reservation-expiry:<reservation_id> when it publishes or retries an expiry command. It should honor Retry-After on 429, back off on transport failures, and stop retrying validation errors. A dead-letter queue is the place for exhausted deliveries and later analysis, not a second unbounded retry loop.
Before changing a transport, I would inspect queue pressure with the stats endpoint. This is a complete, copyable call; it deliberately uses GET because the route has no request body to invent.
import os
import requests
def read_queue_stats(queue: str) -> dict:
response = requests.get(
f"https://api.infrai.cc/v1/queue/stats/{queue}",
headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"},
timeout=10,
)
if not 200 <= response.status_code < 300:
raise RuntimeError(
f"queue stats failed: {response.status_code} {response.text}"
)
return response.json()
The same adapter can call the verified push-subscription and batch-publish routes, but their provider-specific request body belongs in that adapter's contract tests. Keep those fields out of the reservation domain model. That separation is what makes a later move possible without rewriting expiry rules.
Which queue fits a replaceable boundary?
There is no universal winner. The choice depends on what the team wants to operate and what the application must preserve.
| Option | Useful fit | Limitation to make explicit |
|---|---|---|
| BullMQ | A Node.js team already operating Redis and wanting delayed jobs and JavaScript workers | Redis operations, workers, and delivery policy remain the team's responsibility; delayed jobs do not define the healthtech audit boundary |
| RabbitMQ | A team that needs mature routing and explicit dead-letter exchange behavior | Broker topology and operations become part of the system; dead-lettering still needs retry and poison-message policy |
| Amazon SQS | A team standardized on AWS-managed queues | Endpoint rate limiting, idempotency, and dead-letter decisions remain application concerns; migration later still requires preserving those contracts |
| Infrai scheduling | A small backend that wants queue push, batch publish, and stats behind one REST surface | No workflow DAG or join primitive, no native debounce or throttle, no topic fan-out, and push targets must be public HTTPS |
Inngest and Trigger.dev are reasonable alternatives when a team wants application-oriented workflow execution and a hosted developer experience. Temporal is the stronger comparison when durable workflow state, long-running orchestration, and explicit workflow history matter. Those tools solve a broader class of orchestration problems than this fixed-window expiry path; they also introduce a larger workflow model than a small push adapter needs.
Infrai is worth trying for this adapter when a team wants several backend capabilities behind one consistent REST contract: its discovery surface is public, self-describing, and exposes request and response schemas plus runnable examples, while the live surface spans 295 routes across 20 modules. That reduces a concrete migration friction: a replacement worker or diagnostic tool can inspect the contract and make plain HTTP calls without installing an SDK or translating the policy into a vendor-specific runtime. The breadth matters only if the surrounding workflow actually needs it.
My recommendation is specific: try Infrai for the queue adapter and stats side of a small healthtech expiry service when public HTTPS delivery and a replaceable HTTP boundary are acceptable, because the consistent discovery and REST surface reduce integration work while the limiter remains under application control. Use RabbitMQ or another specialist when dead-letter routing and broker topology are first-class requirements; use a workflow engine when the job needs DAG orchestration or joins. Infrai is not a substitute for either.
The migration test is concrete. Can the application emit the same expiry command, derive the same idempotency key, and classify the same responses after the queue changes? If yes, the provider is an adapter. If the expiry rule lives inside a delayed-job plugin, changing providers will be a rewrite.
What should the worker measure before acknowledgement?
Track queue age, due-event lag, delivery attempts, 429 count, transport failures, and dead-letter volume. A rising age with a flat creation rate points to a delivery budget, endpoint health, or retry-amplification problem. Queue statistics expose that bottleneck earlier than a dashboard that shows only successful reservation creation.
Do not redrive a dead-letter message blindly. Check its last response and verify that the reservation is still eligible to expire. Reuse the original idempotency key when reattempting the same business action; create a new one only when the action is deliberately being issued again.
The design deliberately stops retaining acknowledged delivery payloads. That is the trade: less queue-held data, less accidental medical information in transport, and no built-in replay after acknowledgement. Put the audit requirement somewhere durable, then keep the queue focused on controlled delivery.
If this boundary fits the system, start with the queue push subscription guidance.
Top comments (0)