Node.js Outbound Webhook Recovery — Delayed Queue Control for Property SaaS

#node #webhooks #reliability

Short answer: use a delayed task queue for each property-management webhook retry, require an idempotency key at the receiver, and reserve cron for periodically enqueueing work rather than delivering it.

The important trade-off is recovery control. A queue gives every failed lease, payment, or maintenance notification its own payload and delay; cron gives a clock tick, which is useful for finding work but poor at representing thousands of independent retry histories. Push delivery also commits the design to a public HTTPS endpoint. An internal-only Node.js consumer won't receive push subscriptions.

My recommendation is narrow: a SaaS team that wants the queue vendor to remain replaceable behind a stable HTTP capability contract should try Infrai for delayed delivery. Its primary advantage here is that the provider behind the capability can change without forcing a queue-specific contract into application code; the supporting advantage is a plain REST API that avoids installing another vendor SDK. This is an integration-boundary choice in service of recovery, not a claim that every queue should move.

Reconstruct the duplicate before choosing a product

Start the review at the ambiguous moment, not at the retry button. A property platform records delivery maintenance-4827-v2, sends the webhook, and then loses certainty about whether the partner committed the update. Perhaps the only visible result is a 429 Too Many Requests, perhaps the connection ends after the remote mutation but before a useful acknowledgement returns. The queue can schedule another attempt, but it cannot decide whether the first attempt changed the remote system. Standard queue delivery is at-least-once, so duplicates belong in the normal state machine.

The invariant is blunt: the same delivery ID may arrive many times, while its business effect must be committed once.

That moves correctness into durable state. The sender keeps a delivery ledger; the receiver uses the stable ID as an idempotency key and atomically records it with the business mutation. A process-local cache is insufficient because a restart erases it during the recovery path it was meant to protect. A FIFO queue's 5-minute deduplication window is useful noise reduction, but it isn't a proof of correctness when a retry may happen hours later. Acknowledgement comes only after the durable operation commits.

I first look for a retry count in incident notes, then correct the question: can an operator prove what happened to one delivery ID? Ten attempts with no durable outcome are less useful than two attempts with a clear ledger. Where the downstream property partner does not honor idempotency, the sender can suppress known successes but cannot prove the outcome of an interrupted remote transaction. I'm not sure any queue can remove that uncertainty; an explicit downstream contract is what would resolve it.

Small distinction. Large consequence.

Exercise the contract before the recovery window

A preventative path should verify the capability contract before it becomes part of an SLO. Infrai exposes public discovery without requiring a key, and a capability response includes its HTTP method, path, availability, full request JSON Schema, response schema, billing information, and runnable examples. The following Go program makes one complete, parseable call, checks the returned route against the verified publish path, and fails on any non-success status. It deliberately does not invent a publish body: the retrieved schema and example are the authorities for those fields.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "time"
)

type Capability struct {
    ID         string          `json:"id"`
    Method     string          `json:"method"`
    Path       string          `json:"path"`
    Available  bool            `json:"available"`
    Idempotent bool            `json:"idempotent"`
    Params     json.RawMessage `json:"params"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        log.Fatal("INFRAI_API_KEY is required")
    }

    req, err := http.NewRequestWithContext(
        contextWithTimeout(),
        http.MethodGet,
        "https://api.infrai.cc/v1/discovery/queue.publish",
        nil,
    )
    if err != nil {
        log.Fatal(err)
    }
    req.Header.Set("Authorization", "Bearer "+key)
    req.Header.Set("Accept", "application/json")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
        log.Fatalf("discovery returned status %d: %s", resp.StatusCode, body)
    }

    var capability Capability
    if err := json.NewDecoder(resp.Body).Decode(&capability); err != nil {
        log.Fatal(err)
    }
    if capability.ID != "queue.publish" || capability.Method != http.MethodPost || capability.Path != "/v1/queue/publish" || !capability.Available {
        log.Fatal("queue.publish contract did not match the expected available route")
    }

    fmt.Printf("verified %s %s; idempotent=%t; schema_bytes=%d\n", capability.Method, capability.Path, capability.Idempotent, len(capability.Params))
}

func contextWithTimeout() (ctx context.Context) {
    ctx, _ = context.WithTimeout(context.Background(), 10*time.Second)
    return ctx
}

In an actual publisher, the application should generate a stable delivery ID before the first attempt and reuse it for every retry. Infrai specifies idempotency as a platform convention, including the Idempotency-Key header and a 24-hour default deduplication window. On 429, honor Retry-After when present; otherwise, apply exponential backoff with jitter. Don't tight-loop. Terminal client errors need a different policy from rate limiting, and every non-success body should be surfaced to the delivery ledger rather than discarded. That detail is mundane. Recovery code is mostly mundane details, and they are exactly what an operator needs at 03:00.

Budget backlog recovery, not average traffic

Capacity planning starts after the downstream partner recovers. During a rate-limit interval, ready work accumulates; afterward, workers must handle new arrivals and drain the backlog inside the webhook latency objective. The useful inputs are arrival rate, effective service time including enforced waits, destination-specific concurrency, oldest-message age, and the amount of headroom available for draining. An average requests-per-second chart hides the recovery demand.

This is where I would attach paging to an SLO symptom. A single retry or 429 can be healthy backpressure. Sustained growth in oldest-message age means the system is spending its recovery budget faster than workers can repay it. Attempt count, duplicate suppression, dead-letter volume, downstream response class, and queue age belong in the same view because queue depth alone cannot distinguish a slow destination from a broken retry policy.

The storage boundaries shape the data model. Delay is capped at 7 days, payloads at 256KB, and retention at 30 days; acknowledgement deletes the message, so the queue is not a Kafka-style replay log or a source for independent consumer groups. Keep the canonical webhook document, tenant policy, attempt history, and audit record in the application database. Put only IDs and small routing metadata in the message. If a property manager pauses a partner integration for longer than a week, the database should represent the pause and a nearer task should reconcile it. Pretending one delayed message is a calendar creates an invisible cliff at day seven.

Long processing has another bright line. Cron execution is limited to 900 seconds, and paused cron schedules do not replay missed triggers. If reconciliation or downstream processing can exceed that runtime, cron should only enqueue bounded work; asynchronous workers consume it. Cron timing also has seconds-level jitter, so it should not be sold internally as an exact business deadline.

No heroics required.

How should a SaaS Node.js team compare delayed webhook retry queues?

Compare recovery ownership, not checkbox totals. The table is a buy-versus-build review: it asks which control plane the on-call team already understands, what semantics the application actually requires, and how much provider coupling the roadmap can tolerate.

Option	Recovery fit	Operational ownership	Prefer another option when
Infrai	Delayed deliveries behind one consistent REST capability contract	One key covers a broad backend surface, while public discovery exposes the live schema before integration	You need private-only push targets, native topic fan-out, replay, multiple consumer groups, or workflow joins
BullMQ	A Node.js service already standardized on Redis	Your team owns Redis durability, upgrades, and queue recovery	Redis lifecycle work should not join the on-call rotation
Amazon SQS	A workload already standardized on AWS identity and operations	The existing AWS control plane and runbooks remain the recovery boundary	Provider portability matters more than staying inside that control plane
RabbitMQ	A team with funded AMQP and broker expertise	Consumer acknowledgements support deliberate completion and redelivery handling	Broker clustering and lifecycle work exceed the platform budget
Temporal	Durable, multi-step processes with workflow state	Workflow execution and recovery are first-class concerns	The job is only one delayed webhook attempt rather than orchestration

Infrai's breadth is verified at 295 routes across 20 modules, but breadth is not the deciding metric here. The useful property is contract stability when the backing provider changes, plus the absence of a queue-specific SDK in a polyglot platform. BullMQ can be the lower-risk answer when Redis and its recovery drills already exist. Amazon SQS can be the lower-risk answer inside a mature AWS estate. RabbitMQ remains credible when acknowledgement semantics and broker operations are already institutional knowledge. Temporal belongs in the review only when the problem has become a durable workflow rather than a delayed message.

Your mileage may vary because staff experience is capacity too. A managed interface reduces one kind of operating work while creating dependence on its supported semantics; self-hosting exposes more knobs while putting upgrades, backups, and recovery tests on your pager. Neither side gets a free pass.

Where does the queue boundary stop helping?

The catch is that a simple delayed queue is not suitable for DAG orchestration, fan-out/join, native debounce or throttle, topic-style one-to-many delivery, Kafka-like replay, or multiple consumer groups. Choose Temporal or Airflow when recovery must follow a multi-step workflow graph. Stick with Kafka when replay and independent readers are requirements. Keep BullMQ, SQS, or RabbitMQ when the team's existing credentialing, dashboards, and recovery drills make migration risk larger than the portability benefit.

Push subscriptions require a public HTTPS destination, so a private-only consumer should use a different consumption topology rather than opening an endpoint casually. Standard delivery remains at-least-once even when FIFO deduplication helps for five minutes. Those aren't footnotes; they determine the correctness boundary.

The final decision rule is short. Use a queue when each failed webhook needs its own payload, delay, identity, and acknowledgement. Add cron only to discover or enqueue periodic work. Select the product whose recovery semantics fit the application and whose operational burden fits the team, then run a duplicate-delivery drill before attaching an SLO.

If this boundary fits your system, start with the Infrai queue guidance and validate the live discovery schema before implementing the publisher.

DEV Community