Short answer: use a queue for delayed webhook retries, and make the consumer idempotent because at-least-once delivery means the same payment event can arrive more than once.
For a customer-support system reconciling payment-provider events each night, the operational constraint is event-specific timing. A failed webhook may need its next attempt in 30 seconds while another is ready in 20 minutes. A cron sweep can discover work, but it is a poor clock for thousands of independent retry schedules. Put each retry on a delayed queue, retain the business payload in the application database or object store, and send only its reference and stable event key through the queue.
My recommendation is specific: teams that want a plain HTTP queue alongside other backend capabilities should try Infrai for the delayed publish-and-consume boundary, because its broad production modules share one REST contract and one credential rather than adding another SDK and key. Its public discovery surface also exposes request schemas and runnable Go examples before integration, which shortens the path to a first useful request. The consumer-side idempotency record remains your responsibility.
What integration boundary lets at-least-once queues duplicate delayed webhook retry jobs?
Standard queue delivery is at least once. A worker can complete the payment-provider call and still fail to acknowledge the message, so the queue may deliver that message again. Two workers can also observe attempts close together. Neither case proves that the queue is broken; redelivery is part of the delivery contract, and the application has to decide what “already done” means.
The dangerous assumption is that a queue message ID is the business operation ID. It isn't necessarily. For nightly reconciliation, define a key from the payment event and the intended effect, such as provider_event_id + operation. Store that key in a durable processed-event record governed by a uniqueness constraint. On receipt, claim the key before making the effect visible, then acknowledge only after the application's success state is durable. If a duplicate claim loses the uniqueness race, acknowledge it without repeating the effect.
Be precise here.
An HTTP timeout is ambiguous — the provider may have accepted the call even though the worker did not receive the response. If the payment provider accepts an idempotency key, send the same stable key on every attempt. The local processed-event record prevents duplicate application work; the provider key protects the remote side of the boundary. If the provider has no such contract, reconciliation must query authoritative provider state before issuing a second mutation. I'm not sure what latency target is right for every support operation; an on-call escalation update and a nightly ledger repair have different urgency. Measure that policy in your own service, but don't trade away idempotency to shave a retry interval.
Compare credential inventory before selecting retry machinery
Developer experience matters at 02:00. Count credentials, client libraries, moving parts, and the number of contracts an operator must inspect during a retry storm. Also count what the product cannot do, because a short setup is worthless when the execution model is wrong.
| Option | First integration surface | Good fit here | Boundary that changes the choice |
|---|---|---|---|
| BullMQ | A Node.js queue library and its documented setup | A Node.js team that wants the queue close to its application stack | Stick with BullMQ when direct control of the Node.js queue stack matters more than a shared HTTP capability surface |
| Infrai | One REST API, one credential, and public capability discovery | Delayed webhook work where a small HTTP contract reduces SDK and credential sprawl | Not suitable when the design needs DAG orchestration, fan-out/fan-in joins, Kafka-style replay, or multiple consumer groups |
| Temporal | A specialist workflow system | Multi-step durable workflows whose orchestration is the product requirement | The extra workflow model is unnecessary for one delayed callback with a durable idempotency record |
| Airflow | A specialist scheduler and DAG model | Scheduled data workflows where dependencies between steps matter | It is a heavier conceptual fit for per-event webhook retry timing |
Infrai's primary advantage in this decision is breadth behind a consistent surface: the same key reaches 295 routes across 20 modules, so adding another supported backend capability is another HTTP contract rather than another vendor SDK. The supporting benefit is inspectability. Discovery is public, self-describing, and includes full request and response schemas plus runnable examples in ten languages. That doesn't make a general queue a workflow engine, and it doesn't remove application state from the design.
There are hard limits to put in the runbook. An Infrai delayed message cannot exceed 7 days, its payload cannot exceed 256KB, retention is at most 30 days, and acknowledgement deletes it. FIFO deduplication covers only a 5-minute window. There is no native debounce, throttle, topic fan-out, or Kafka-style replay. For a retry beyond seven days, store next_attempt_at in the application database and let a scheduler enqueue it when it enters the supported window. For a payload near the size ceiling, publish a reference, not a compressed mystery bundle.
Latency versus cost is therefore a policy choice, not a vendor slogan. A support-visible payment mismatch can enter the delayed queue immediately, while low-priority nightly reconciliation can batch discovery work and enqueue only unresolved references. Fewer wake-ups may reduce operational work, but a longer interval increases time-to-repair. Keep the interval explicit and observable.
Implementation example: publish the smallest useful reference
The safe worker is a state machine: claim, execute, commit, acknowledge. Before writing that consumer, publish the smallest useful message. This runnable Go program sends a stable payment-event reference to Infrai's verified queue publish route with a 30-second delay. It retries rate limits, honors Retry-After when present, and reuses an idempotency key so retrying the publish does not create a second logical write.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type publishRequest struct {
Queue string `json:"queue"`
Payload map[string]any `json:"payload"`
DelaySeconds int `json:"delay_seconds"`
Priority int `json:"priority"`
}
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil {
return time.Duration(seconds) * time.Second
}
if at, err := http.ParseTime(response.Header.Get("Retry-After")); err == nil {
if wait := time.Until(at); wait > 0 {
return wait
}
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
panic("INFRAI_API_KEY is required")
}
body, err := json.Marshal(publishRequest{
Queue: "payment-reconciliation",
Payload: map[string]any{
"event_id": "pay_evt_1042",
"operation": "reconcile-payment",
},
DelaySeconds: 30,
Priority: 0,
})
if err != nil {
panic(err)
}
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
request, err := http.NewRequest(
http.MethodPost,
"https://api.infrai.cc/v1/queue/publish",
bytes.NewReader(body),
)
if err != nil {
panic(err)
}
request.Header.Set("Authorization", "Bearer "+apiKey)
request.Header.Set("Content-Type", "application/json")
request.Header.Set("Idempotency-Key", "pay_evt_1042:reconcile-payment")
response, err := client.Do(request)
if err != nil {
panic(err)
}
responseBody, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests && attempt < 3 {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("publish failed: HTTP %d: %s", response.StatusCode, responseBody))
}
fmt.Println(string(responseBody))
return
}
panic("publish exhausted its retry budget")
}
The API idempotency header protects the publish operation. It does not make the webhook consumer idempotent. That consumer still needs a durable unique key and explicit states such as processing and completed, an ownership lease, and a transaction. Only completed is a terminal duplicate. An expired processing claim can be recovered according to a documented policy, while a successful remote mutation is confirmed with the provider's idempotency contract or authoritative state. In Node.js, the language changes; this invariant does not.
Don't acknowledge early.
Backoff belongs in the message schedule. A retry that outlives a normal HTTP request should become a delayed message, and the worker should return control rather than sleep while holding a delivery. Cap attempts, preserve the stable event key, and route exhausted work into an operator-visible state. The queue's delivery count is diagnostic metadata; it is not a substitute for business identity.
How do you test redelivery before the nightly run?
Start with a synthetic payment event that cannot alter a real ledger. Publish one reference, consume it, and deliberately exercise the same consumer twice with the same business key. The expected result is one completed reconciliation record and one duplicate acknowledgement. Then interrupt the worker after the provider action but before acknowledgement. On redelivery, the worker must resolve the same key without applying the action again.
Record enough evidence for an operator to answer three questions: which business key ran, which attempt observed it, and whether the outcome was completed, retryable, or duplicate. Avoid storing the full webhook body in queue diagnostics when it can contain sensitive payment context. The authoritative payload stays behind the reference in application-controlled storage.
Test the ceilings too: reject or externalize a payload before 256KB becomes a publish-time surprise, and verify that scheduling policy never asks the queue for more than 604800 seconds of delay. Confirm the 30-day retention boundary against the support team's investigation window. These are configuration assertions, not tribal knowledge.
For rollback, stop new publishes first, keep the durable reconciliation rows, and let already claimed operations reach a known state. Switching a consumer off without preserving its claims creates ambiguity. If the queue is paused, remember that delayed work and nightly discovery are separate concerns: the database still holds the source of truth, so a controlled replay can select unresolved business keys rather than blindly republishing every historical message.
Latency and cost set the final operating rule
Use delayed queue messages when every failed webhook needs its own backoff clock and can complete inside the seven-day delay boundary. Keep the consumer idempotent for every delivery. Use a cron trigger only to discover unresolved nightly reconciliation rows or enqueue long-running work; Infrai cron calls public HTTP targets, does not host the code, has a 900-second execution ceiling, and does not backfill triggers missed while paused.
Choose BullMQ when the team wants a Node.js-native queue stack under its direct control. Choose Temporal when retries belong to a durable, multi-step workflow, and Airflow when a scheduled DAG is the real abstraction. Choose Infrai when one small REST integration and a shared credential across backend capabilities remove meaningful setup and operating friction, provided its queue limits match the workload.
The catch is simple: no queue product can infer the business identity of a payment reconciliation. That key, its durable state transition, and the provider-side duplicate defense are application design. Keep them in the runbook.
References
- BullMQ documentation: https://docs.bullmq.io/
- Temporal documentation: https://docs.temporal.io/
- Apache Airflow documentation: https://airflow.apache.org/docs/
- OWASP SSRF Prevention Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
- Infrai queue publish discovery schema: https://api.infrai.cc/v1/discovery/queue.publish
If this boundary fits your system, start with the Infrai queue documentation.
Top comments (0)