DEV Community

oskarholm4968
oskarholm4968

Posted on

Business-Deadline Webhooks: A Simple Node.js SaaS Queue for Delayed Retries

Short answer: model a renewal reminder as a durable delivery record with a due time, send it through a delayed task queue, and make the public HTTPS handler idempotent. The queue decides when to attempt delivery; the application database decides whether that attempt may change business state.

That separation matters more than the choice of queue. A SaaS team can schedule a webhook after a business deadline with a timer, a cron process, or a queue, but none of those mechanisms makes a repeated HTTP request safe. Retry and idempotency are the primary decision axis here, not the number of buttons in a dashboard.

I use four invariants for this kind of developer-tool workflow: a committed reminder is discoverable, every attempt has a stable delivery ID, the receiver can repeat an attempt without repeating the business effect, and a terminal outcome can be reconciled against the original renewal event. The exactly-once mindset belongs at the domain boundary. Transport should be assumed at-least-once.

What data does a SaaS Node.js team record before a delayed webhook reaches a public HTTPS endpoint?

Write the renewal event and its outbox row in one database transaction. The row should contain a tenant ID, a delivery ID, the intended deadline, the current attempt, the next due time, and a reference to the authoritative payload. A publisher can then place a small task on the queue. If publishing fails after the database commit, an outbox scanner finds the row; if publishing succeeds and the worker dies before acknowledgement, the same delivery ID is published again.

The public HTTPS endpoint should receive a delivery ID and an authenticated request, claim that ID under a uniqueness constraint, and record the attempt before acknowledging transport. The claim and the business mutation need one transaction when they must be atomic. A response such as 202 Accepted means the receiver accepted responsibility; it does not prove that the reminder was processed to a terminal state.

Keep the queue message narrow. It is an instruction to locate work, not an audit archive. The database record should own the webhook body, retention policy, redaction rules, and compliance evidence. That choice also makes reconciliation possible: compare committed outbox rows, published delivery IDs, attempt rows, and terminal outcomes rather than trying to reconstruct history from worker logs.

The timing rule is equally plain. A task due at a business deadline belongs in a durable schedule until it is eligible; after that, the queue handles individual attempts and backoff. A periodic scan is useful for recovery, but a cron invocation is not a substitute for a per-delivery state machine. Short timers are easy. Long-lived obligations are where the ledger earns its keep.

Consider one concrete sequence. At 16:00, the subscription service commits renewal_due and an outbox row. At 16:05, a publisher emits delivery r-1842 for a reminder due at 09:00 the next business day. The worker sends the request, the endpoint commits its claim, and the network drops before the worker sees the response. A second attempt is now correct, even though it may look wasteful in a dashboard. The endpoint must find r-1842 already claimed, append a duplicate observation, and avoid sending a second reminder or applying a second ledger mutation. If the first transaction had failed instead, the claim would not exist and the next attempt would be allowed to perform the work. That distinction is why a boolean “sent” flag is insufficient: it cannot tell reconciliation whether the business effect committed, the transport acknowledgement vanished, or the request never reached the endpoint. Keep the status transitions explicit, and the recovery action becomes a query rather than a guess.

Duplicates happen.

Data ownership around the delivery ID

The following Go example shows the receiver boundary with an in-memory store so the ordering is visible. Production code should replace the map with a database transaction and a unique index on delivery_id; the example does not pretend that process memory survives a restart.

package main

import (
    "crypto/subtle"
    "log"
    "net/http"
    "os"
    "sync"
)

var (
    mu   sync.Mutex
    seen = map[string]struct{}{}
)

func claim(deliveryID string) bool {
    mu.Lock()
    defer mu.Unlock()
    if _, exists := seen[deliveryID]; exists {
        return false
    }
    seen[deliveryID] = struct{}{}
    return true
}

func webhook(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
        return
    }

    want := "Bearer " + os.Getenv("WEBHOOK_SECRET")
    if subtle.ConstantTimeCompare([]byte(r.Header.Get("Authorization")), []byte(want)) != 1 {
        http.Error(w, "unauthorized", http.StatusUnauthorized)
        return
    }

    deliveryID := r.Header.Get("X-Delivery-ID")
    if deliveryID == "" {
        http.Error(w, "missing delivery ID", http.StatusBadRequest)
        return
    }
    if !claim(deliveryID) {
        log.Printf("audit duplicate delivery_id=%q", deliveryID)
        w.WriteHeader(http.StatusNoContent)
        return
    }

    log.Printf("audit accepted delivery_id=%q", deliveryID)
    w.WriteHeader(http.StatusAccepted)
}

func main() {
    http.HandleFunc("/webhooks/renewals", webhook)
    log.Fatal(http.ListenAndServe(":8080", nil))
}
Enter fullscreen mode Exit fullscreen mode

The important line is not the handler registration. It is the stable identity crossing every boundary. If the sender retries after a timeout, the receiver sees the same ID. If a worker redelivers after a lost acknowledgement, reconciliation sees another attempt on the same obligation. A random ID generated inside the handler would destroy that property.

There is a small but consequential distinction between a duplicate and a failure. A duplicate whose first transaction committed should be recorded as an audit observation and answered consistently. A failed transaction must leave the claim available for a later attempt. Mixing those states turns a transient database problem into a permanently suppressed renewal reminder.

Queue, cron, and timer have different ownership

The comparison should measure ownership of state and recovery, not whether a mechanism can wait for a number of seconds.

Option Good fit Boundary to accept
Delayed task queue Each reminder has its own due time, payload reference, and retry policy Delivery is at-least-once; the receiver must be idempotent
Cron plus queue Periodic discovery of overdue outbox rows or recovery work The scan is a trigger, not the worker or the delivery ledger
Database scheduler The application needs transactional visibility of due rows The team owns polling, locking, backoff, and worker wake-up behavior
In-process timer A short-lived, non-critical action inside one process Restarts, deploys, and horizontal scaling make it unsuitable for obligations

For a renewal reminder, I would reject the in-process timer first. It has an attractive API and a weak recovery story. A database-backed scheduler can be correct, especially for a small self-hosted system, but its operating model is now part of the application: leases, stuck claims, clock handling, and polling load all need tests and alerts.

The queue-first design is also not universally suitable. If the requirement is a periodic report, a cron trigger may be clearer. If the workflow has durable joins, long-running compensation, or many dependent branches, a workflow engine is a better abstraction. If the team already operates a broker and needs its acknowledgement controls, adding a second scheduling layer may be needless complexity. Stick with the simpler mechanism when the state machine is genuinely small.

Retry policy fields that keep a deadline ledger honest

Retry policy should be data, not an undocumented loop. Store the attempt number, last response class, next due time, and terminal reason. Classify responses before applying backoff: a timeout or a 5xx generally deserves another attempt, while a permanent authentication failure needs an operator-visible terminal state. A 429 Too Many Requests response specifically signals that the client should respect the server's rate limit; when a Retry-After value is present, the scheduler should use it subject to a maximum delay and a tenant policy.

The schedule needs a cap. Without one, a reminder can remain “temporarily failed” forever while the customer-facing record says nothing. Exponential backoff with jitter avoids synchronized retries, but it does not decide when the business obligation expires. That deadline belongs to the domain record, along with the escalation or reconciliation action after expiry.

A useful audit trail includes the request timestamp, attempt number, response status, response classification, latency, redacted error detail, and the identity of the worker that made the attempt. Do not store secrets in a log merely because a webhook request had them. Compliance rules may require shorter payload retention than operational metadata, and a queue's retention setting cannot replace that policy.

I'm not sure any generic queue can infer the right terminal rule for a renewal reminder. The product owner has to define whether a missed deadline cancels the reminder, opens a support task, or requires a reconciliation job. That uncertainty is a domain decision, not an infrastructure feature gap. It isn't a reason to hide the state transition inside a worker. Don't let a retry loop decide a customer-facing policy by accident.

Rollout checks for reconciliation

Test the failure boundaries explicitly: commit the outbox row and kill the publisher; deliver the same task twice; acknowledge a message and crash before recording the terminal result; return 429; rotate the receiver credential; and deploy two workers that race for one delivery ID. A good test suite checks the resulting ledger, not just whether the handler returned a status code.

Metrics should join on the delivery ID and distinguish scheduled, published, attempted, accepted, completed, expired, and dead-lettered states. Alert on rows that have stayed in one state too long, on duplicate claims, and on a growing difference between committed outbox rows and published tasks. Those signals tell an operator where the obligation disappeared.

The final decision rule is therefore narrow: choose the simplest durable scheduler that can preserve a delivery identity, expose retries, and support a reconciliation query. For a delayed webhook to a public HTTPS endpoint, a queue plus an outbox and an idempotent receiver is usually the clearest shape. The recommendation is about boundaries and evidence, not a brand, a price, or a promise of exactly-once delivery.

References

Top comments (0)