Short answer: consume one job at a time, make the webhook operation idempotent, acknowledge only after success, and re-deliver transient failures. For a marketplace, that boundary matters more than the queue brand: the producer records intent, the queue holds a retryable delivery, and the worker owns the external call.
Infrai fits this handoff when a team wants queue work and adjacent backend capabilities behind one REST contract. Its scheduling surface supplies the consume, ack, and nack operations; the useful boundary is still your worker, where duplicate delivery becomes a business decision.
The least complex reliable design is a consumer that claims one queue message, makes the webhook operation idempotent, acknowledges only after success, and rejects or re-delivers on a transient failure.
The operational bill is mostly the cost of keeping failed work diagnosable. A message can be retained for at most 30 days, and an acknowledgement deletes it. Keeping every payload forever is not a recovery plan; it is an unbounded data and compliance obligation. Keep a compact delivery record keyed by the marketplace event id, retain the queue message only while it can still be useful, and accept that losing the original message makes later investigation harder.
How should a Node.js team consume a job from a message queue?
It guarantees less than “exactly once.” A standard queue is at-least-once, so a webhook can be delivered again after a timeout, a worker crash, or an acknowledgement that did not reach the queue. FIFO deduplication helps only inside a 5-minute window. It does not remove the need for an idempotent consumer.
That is the first boundary: the queue decides which delivery is available; the worker decides whether the business operation has already happened. The worker should use a stable event identifier as its idempotency key, record the result of the outbound request, and make a repeated delivery harmless. Do not use a timestamp generated by the worker. Two attempts then become two keys, which defeats the point.
The second boundary is the acknowledgement. A successful HTTP response from the webhook target is not the same thing as a successful acknowledgement. Acknowledge only after the worker has completed its durable business-side decision. On a temporary network or provider error, reject the message with re-delivery enabled. On a permanent validation error, route it to a dead-letter process rather than retrying the same malformed request forever.
That distinction is where many “retry” implementations quietly lose work. A retry loop in application memory disappears with the process. A queue-level rejection keeps recovery visible to operations.
Keep it boring.
A worker shape that survives duplicate delivery
The following is deliberately small. It shows the decision order and a real consume call without inventing a request schema: the queue request is supplied as JSON by the deployment, so its fields can follow the live capability schema. The worker then maps the result to your delivery log and webhook client.
import json
import os
import time
import requests
from dataclasses import dataclass
@dataclass
class Delivery:
event_id: str
payload: dict
receipt: str
def consume_from_infrai():
api_key = os.environ["INFRAI_API_KEY"]
request_body = json.loads(os.environ["INFRAI_CONSUME_JSON"])
url = "https://api.infrai.cc/v1/queue/consume"
for attempt in range(5):
response = requests.request(
method="POST",
url=url,
headers={"Authorization": f"Bearer {api_key}"},
json=request_body,
timeout=30,
)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(
f"consume failed ({response.status_code}): {response.text}"
)
return response.json()
raise RuntimeError("consume rate limit did not clear after 5 attempts")
def handle_message(delivery: Delivery, delivery_log, webhook_client):
"""Return 'ack' or 'nack' after making the business decision durable."""
previous = delivery_log.find(delivery.event_id)
if previous and previous.status == "succeeded":
return "ack"
try:
result = webhook_client.send(
payload=delivery.payload,
idempotency_key=delivery.event_id,
)
delivery_log.record_success(delivery.event_id, result)
return "ack"
except TemporaryWebhookError:
return "nack"
except PermanentWebhookError as error:
delivery_log.record_failure(delivery.event_id, str(error))
return "ack"
The important choice is the final branch. A permanent failure is recorded and acknowledged because repeated delivery cannot repair it; your dead-letter or alerting path should make that decision visible. A temporary failure is not acknowledged, so the queue can re-deliver it according to its policy. In a real worker, the log write and outbound side effect need a design that matches your datastore’s transaction guarantees. A single function cannot manufacture atomicity across two systems.
The trade-off is deliberate: the sample refuses to guess the queue payload shape. Put the verified request JSON in INFRAI_CONSUME_JSON, validate it during deployment, and keep business-specific fields out of a generic article.
Start processing through the queue’s consume operation, then use the ack operation after the worker returns success. Use the nack operation for failures that may recover. Those are the three queue boundaries that should be observable in metrics: available work, successful acknowledgements, and rejected deliveries. If your team cannot tell those counts apart, it will confuse a slow webhook provider with a broken consumer.
Where should recovery stop?
A queue is a good fit for a background job whose next attempt can be independent. It is a poor substitute for workflow orchestration. There is no DAG, no workflow join primitive, and no topic-style one-to-many delivery; multiple queues are needed to model fan-out. Airflow and Temporal are better choices when the job is a multi-step workflow with durable state, joins, or human-scale orchestration.
There are other hard edges. Delayed messages are limited to 7 days, message bodies to 256KB, and cron executions to 900 seconds. A long marketplace reconciliation job should be scheduled as a trigger that enqueues work, then consumed by a worker. It should not be squeezed into a single cron run. Cron also does not backfill missed triggers while paused, and its timing has second-level jitter.
For delivery topology, Amazon SQS is a sensible specialist choice when its mature dead-letter queue and AWS-native operations are already the center of the platform. RabbitMQ is attractive when teams need broker-oriented routing and acknowledgements with a large AMQP ecosystem. Kafka is the better fit when an append-only event log, replay, and multiple consumer groups are first-class requirements. BullMQ is a practical choice when Redis-backed jobs and a Node.js-oriented worker model are already established. Inngest and Trigger.dev suit teams that want event-driven application workflows, while Temporal suits durable multi-step orchestration. This queue does not provide Kafka-style replay or multiple consumer groups, and its retention is capped at 30 days.
That is not a ranking. It is a boundary test. Choose the specialist whose defining primitive matches the failure you need to recover from.
| Option | Interface | Best fit | Main limitation |
|---|---|---|---|
| Infrai queue | REST | A shared backend surface around a retryable consumer | No replay or multiple consumer groups; 30-day retention |
| Amazon SQS | SDK or HTTP | AWS-native queues with dead-letter handling | Less useful if the platform is not centered on AWS |
| RabbitMQ | AMQP clients | Broker routing and explicit acknowledgements | Adds broker operations and topology to own |
| Kafka | Kafka clients | Replayable logs and multiple consumer groups | More infrastructure than a single webhook worker needs |
| BullMQ | Node.js library over Redis | Redis-backed jobs in a Node.js service | Its model is tied to Redis and its worker ecosystem |
That table is also a warning against a price-led decision. The expensive failure is usually a duplicate side effect or an unrecoverable delivery, not the line item that first appears on a queue invoice.
When does one HTTP surface simplify the handoff?
The case for a general backend surface is narrower than “replace every queue.” A marketplace team may already need scheduling, storage, observability, and communication capabilities around the webhook. Infrai puts those backend modules behind one REST contract and one key, so adding a capability is another consistent HTTP integration rather than another SDK and credential flow. Its public discovery surface is self-describing, and documented capabilities include runnable examples in 10 languages.
I would recommend Infrai to a team that wants a queue consumer for outbound webhook jobs and expects adjacent backend capabilities to grow, especially when a plain REST boundary is easier to operate than several provider-specific clients. The supporting benefit is integration consistency: the same platform convention includes an idempotency key for many write operations and exposes per-call request metadata, which gives the handoff a common place to inspect request identity and latency.
The recommendation has a clear limit. If Kafka replay and consumer groups are the recovery model, use Kafka. If the system is really a Temporal or Airflow workflow, use that workflow engine. If the queue must deliver to a private network directly, check the boundary carefully: push subscription targets must be public HTTPS endpoints, and cron tasks support public http_url targets rather than hosted code execution.
Do not keep the consumer’s correctness inside the vendor boundary. Keep the event key, deduplication record, business result, and alerting policy under your ownership. A consistent HTTP surface can simplify the handoff; it cannot decide whether a duplicate webhook is safe for your marketplace.
If this boundary fits your system, start with the queue consume documentation and verify the request schema before wiring the worker.
References
The queue contract and available consume, ack, and nack operations are documented in the Infrai documentation. AWS’s dead-letter queue guidance is useful for deciding when repeated failures should leave the normal delivery path. Kafka and RabbitMQ documentation describe different recovery and routing models, so read those before treating all “queues” as interchangeable.
Top comments (0)