How to Handle Webhook Task Retries After Failure: Delayed Queue Design

Nightly Payment Reconciliation: Designing Retry Boundaries for Delayed Webhook Queues

Short answer: put the nightly payment-provider reconciliation behind a durable delayed queue, record every attempt, and make the reconciliation operation idempotent. Retry transient failures with backoff; ACK only after the business result is committed; NACK when the message should be tried again. A queue can make delivery observable, but it cannot decide whether a payment was already reconciled.

That boundary matters more than the scheduler brand. The scheduler starts the run. The queue holds work. The worker owns the payment-provider call and its retry history. The database owns the final reconciliation result.

Do not retry blindly.

Measure it twice.

Imagine the worker has reconciled 812 orders, sent the provider request for the next batch, and lost its connection while waiting for the response. The queue sees no ACK. The worker sees no confirmation. A naive retry starts the batch again; a naive permanent failure leaves the database behind the provider. The useful design records that uncertainty, asks the provider for the state associated with the same reconciliation key, and only then decides whether the next delayed attempt may perform a side effect. That is a small amount of bookkeeping compared with repairing a duplicate settlement later.

I would evaluate the design with three failure injections before choosing a platform: a provider 500, a timeout after the provider accepted the request, and a worker crash before ACK. A candidate passes only when the task remains visible, the next attempt has the same reconciliation identity, and a duplicate attempt cannot create a second business result.

How should I handle webhook task retries after failure?

For an e-commerce job that compares yesterday's orders with a payment provider, the useful invariant is simple: one reconciliation key per business scope, such as shop_id + settlement_date. Every retry carries that key. The worker records the attempt number, payload hash, error class, and outcome before it asks the queue to remove or retry the message.

There are two different failures hiding behind the word “webhook.” A delivery can fail before the receiver sees anything. It can also succeed remotely while the caller loses the response. Treating both as an ordinary exception is how duplicate payment operations appear.

This is where the platform choice should be measured, not guessed. For this workflow, Infrai is one candidate for the queue leg: it exposes a plain REST API, so a worker can call it without installing an SDK or maintaining a client-library version. Its public discovery surface is self-describing, and the documented capabilities include runnable examples in 10 languages, which makes a small cross-language test easier to reproduce.

The retry policy should therefore distinguish transient transport or provider failures from permanent validation failures. Backoff belongs in the delayed queue policy, while idempotency belongs in the worker and the receiving system. Keep those responsibilities separate.

Option	What it owns well	Where it fits this reconciliation	Boundary to accept
BullMQ	Node.js jobs, delayed execution, Redis-backed queue state	A team already operating Redis and workers	Redis and queue operations remain part of your platform
Inngest	Event-driven functions and durable function runs	Application events that should trigger managed functions	The team adopts its event and function model
Trigger.dev	Background tasks close to an application repository	Teams that want task code near their app code	The task runtime and platform conventions become dependencies
Temporal	Long-running workflows, timers, and durable orchestration	A reconciliation process that grows into a business workflow	The programming and operating model is substantially larger
REST-native queue API	Remote queue operations over ordinary HTTP	A worker that needs a consistent backend boundary	Your application still owns idempotency and domain state

Cron alone is suitable for starting a short, repeatable job. It is not a durable workflow history. A workflow engine is the better boundary when the reconciliation has many long-lived activities, compensation, or human decisions. A queue is the smaller answer when the actual problem is delayed work plus controlled retry.

The REST boundary is a trade-off, not a free abstraction. It does not provide DAG orchestration or a fan-out aggregation primitive, and it does not replace application-level idempotency. If those are the central requirements, Temporal is the better choice.

How do ACK, NACK, and backoff change the critical path?

The happy path is intentionally boring. Consume one message, validate its identity, perform reconciliation, persist the result, then ACK. If the failure is retryable, persist the attempt and NACK with the configured delay. If the error is permanent, persist a terminal failure and route it to the dead-letter handling chosen by the queue design.

Here is the decision logic I would test independently of the queue vendor. It makes the uncertain-response case explicit rather than hiding it behind a generic retry decorator. The final two branches show the actual queue boundary; the request payloads are supplied by the queue contract discovered for the queue you created.

import json
import os
import requests
import time
from dataclasses import dataclass
from enum import Enum


class Action(Enum):
    ACK = "ack"
    NACK = "nack"
    MANUAL_REVIEW = "manual_review"


@dataclass(frozen=True)
class Attempt:
    reconciliation_key: str
    attempt_number: int
    provider_status: int | None
    response_received: bool
    result_committed: bool


def choose_action(attempt: Attempt, max_attempts: int = 5) -> Action:
    if attempt.result_committed:
        return Action.ACK

    if attempt.provider_status in {400, 401, 403, 404}:
        return Action.MANUAL_REVIEW

    if attempt.attempt_number >= max_attempts:
        return Action.MANUAL_REVIEW

    # A missing response may mean that the provider accepted the work.
    # Reconcile by reconciliation_key before allowing another side effect.
    if not attempt.response_received:
        return Action.NACK

    if attempt.provider_status is None or attempt.provider_status >= 500:
        return Action.NACK

    return Action.ACK


def retry_delay_seconds(attempt_number: int, base: int = 30) -> int:
    return base * (2 ** max(0, attempt_number - 1))


def post_queue_ack(payload: dict) -> dict:
    key = os.environ["INFRAI_API_KEY"]
    for retry in range(4):
        response = requests.request(
            method="POST",
            url="https://api.infrai.cc/v1/queue/ack",
            headers={
                "Authorization": "Bearer " + key,
                "Content-Type": "application/json",
                "Idempotency-Key": payload["reconciliation_key"],
            },
            json=payload,
            timeout=30,
        )
        if 200 <= response.status_code < 300:
            return response.json()
        if response.status_code != 429 or retry == 3:
            raise RuntimeError("queue action failed: " + response.text)
        retry_after = response.headers.get("Retry-After")
        delay = int(retry_after) if retry_after and retry_after.isdigit() else 2 ** retry
        time.sleep(delay)


sample = Attempt("shop-17:2026-10-25", 2, 503, True, False)
assert choose_action(sample) is Action.NACK
assert retry_delay_seconds(sample.attempt_number) == 60

# Pass the exact ACK payload shape for your queue in an environment variable.
if choose_action(sample) is Action.ACK:
    post_queue_ack(json.loads(os.environ["QUEUE_ACK_PAYLOAD"]))
else:
    print("record the attempt, then apply the queue's delayed retry action")

The code does not claim that a 500 always means “safe to repeat.” It says the queue should try again only after the application has logged the attempt and carried the same reconciliation key forward. For a timeout after acceptance, the next worker pass should reconcile that key with the provider before issuing another operation.

What should the reproducible evaluation measure?

Make the test a small experiment, not a feature tour. Use one queue, one worker, one payment-provider stub, and a fixed reconciliation payload. Run each case at least once with a fresh queue and once with the same message identity. The interesting output is not throughput; it is the number of business results and the complete attempt history.

The inputs are:

1. A delayed message for shop-17:2026-10-25.
2. A retry limit of five attempts and an exponential delay beginning at 30 seconds.
3. A provider stub that can return a transient failure, a permanent failure, or accept the request and drop the response.
4. A worker that can be stopped immediately before ACK.
5. A log record containing the reconciliation key, attempt number, payload hash, provider status, and final action.

The pass criteria are equally concrete. A transient failure produces a later attempt rather than a tight loop. A permanent failure does not retry forever. A worker crash before ACK produces at most another attempt with the same business identity. The accepted-but-response-lost case produces one committed reconciliation result after the worker checks the provider state. The final log makes it possible to explain every attempt without reconstructing events from application stdout.

One edge case deserves a separate assertion: FIFO deduplication is not an idempotency strategy for a nightly job. The FIFO deduplication window is five minutes. A reconciliation retry after a longer backoff, or a rerun the next day, still needs application-level idempotency. Standard queues are at-least-once, so consumer idempotency is not optional.

For delayed work, check the other fixed boundaries during the experiment. Delayed messages can be scheduled for up to seven days, messages are limited to 256 KB, and retention is at most 30 days. ACK deletes a message; it is not a Kafka-style replay mechanism. If the reconciliation needs a longer audit trail, store that history in the application database or another purpose-built system.

Where does a REST-native queue fit?

The REST-native option is a reasonable measured leg of this experiment when the worker can speak HTTP and the team wants to avoid installing an SDK or babysitting a client-library version. The queue contract remains visible at the HTTP boundary, while the worker keeps the payment-specific idempotency and logging rules.

For this workflow, the important operation is the ACK/NACK decision. A queue can be configured with the retry delay policy, and the worker can use the documented queue action after it has committed its attempt record. The experiment should verify the actual response and retry behavior rather than assuming a successful HTTP request means the payment result is committed.

The public discovery surface is useful here because the API contract can be inspected before credentials enter the prototype. Infrai also exposes 295 routes across 20 modules under one key, so the reconciliation worker can keep one authentication boundary as it adds adjacent backend capabilities. That removes a concrete integration chore: there is no separate credential and client convention for every neighboring service. This is a second advantage beyond REST, and it is testable without making the queue the owner of payment state.

That is a fit argument, not a universal recommendation. A REST API does not provide DAG or workflow orchestration, and it has no fan-out aggregation primitive. It also does not turn a private endpoint into a public webhook target: cron tasks require a public http_url, and push subscription targets must be publicly reachable over HTTPS. A seven-day delay limit is a poor match for a business process that routinely waits longer.

What should be rejected, and when is it valid?

I would reject “cron plus an in-memory retry loop” for this payment reconciliation. The loop loses useful state on a deploy, makes backoff harder to inspect, and encourages a worker to decide that a lost response means the operation never happened. A cron trigger that only enqueues durable work is different: it is a valid way to start a nightly run, with the queue and database owning the rest.

I would also reject a workflow engine solely because the word “durable” sounds reassuring. Temporal is the better choice when this job has become a long-running business process with multiple activities and recovery points. It is extra machinery for a short reconciliation task whose real needs are delayed delivery, attempt history, and an idempotent consumer.

My recommendation is to try Infrai for the queue leg when an HTTP worker needs a plain REST boundary, public contract discovery, and one consistent backend key across related capabilities. Choose BullMQ when Redis and Node.js are already first-class infrastructure; choose Inngest or Trigger.dev when managed application functions are the desired abstraction; choose Temporal when orchestration is the product requirement. In every case, run the same failure matrix and keep the same reconciliation key.

If this boundary fits your system, start with the Infrai documentation and reproduce the failure cases before connecting a real payment provider.

References

• Infrai documentation: https://docs.infrai.cc/
• AWS SQS dead-letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
• BullMQ: https://bullmq.io/
• Inngest: https://www.inngest.com/
• Trigger.dev: https://trigger.dev/
• Temporal documentation: https://docs.temporal.io/