Short answer: put each failed renewal webhook on a queue, let an idempotent HTTP worker consume it, use delayed requeue for bounded retries, and send exhausted work to a DLQ; reserve cron for periodic reconciliation or controlled redrive, not for running the retry loop.
For a gaming platform, the business deadline matters more than the timer. A renewal reminder that must arrive before a subscription cutoff needs observable recovery state: what failed, when it can run again, how many attempts remain, and who can safely replay it. A queue represents that state directly. A cron expression does not.
Model renewal recovery as a state machine
The service-level objective is not "the cron ran." It is "an eligible renewal reminder reached its terminal outcome before the subscription cutoff." Define a retry-to-success target, a maximum deadline-miss rate, and an alert threshold for the oldest eligible job before selecting machinery. Those measures expose a design that looks quiet while reminders are aging in a database. Represent the job as pending, succeeded, retryable, dead-lettered, or expired, and allow every transition only for the stable event ID and expected current state. This makes duplicate delivery a tested branch of the model rather than an exceptional path hidden in worker logs.
Start capacity planning from the recovery burst: estimate failed-webhook arrivals during a dependency outage, average worker service time, safe concurrency against the destination, and backlog drain time. Add headroom for duplicate deliveries. Queue depth is useful, but the distance between the oldest job and its renewal deadline is the sharper paging signal.
One clock matters. A failed webhook is an event, so recovery should begin when the failure occurs; the producer records a stable event ID, the renewal deadline, the attempt count, and a next-attempt time, then publishes the job. The worker consumes when the delay expires, performs the side effect idempotently, and acknowledges only after the durable business result is known. If the attempt fails with a retryable condition, it schedules another delayed delivery; if the retry budget or deadline is exhausted, it moves the job to a dead-letter queue for inspection.
This is the operational distinction that matters: cron repeatedly asks whether something might need work, while a queue retains specific work and its delivery state. Polling a database from cron can be made correct, but then the team has built leasing, concurrency control, backoff, poison-message isolation, and recovery visibility around a table. That may be a reasonable build decision when the database is already the system of record and volume is tiny. It isn't the simplest general architecture for failed webhook jobs.
Cron still has a narrow, useful role. Run a scheduled reconciliation that finds renewals nearing their deadline with no terminal outcome, or trigger a reviewed DLQ redrive after an upstream dependency recovers. The cron target should be a public HTTP endpoint that enqueues work. It shouldn't host worker code, and long processing must follow cron-trigger-to-queue-to-worker because one cron run is capped at 900 seconds.
Keep the deadline in the message envelope rather than deriving it from the current time. That lets the worker refuse a stale reminder instead of sending it after the business event has passed. Ten thousand fresh jobs may be safe; fifty jobs already inside the last recovery window may breach the SLO.
How can Node.js HTTP workers implement delayed webhook retries and a DLQ?
Treat delivery as at-least-once. The worker may see the same event again, so the renewal operation needs a stable idempotency key and a durable outcome record. A process-local cache isn't enough: a restart, another replica, or a redrive can bypass it. The useful state machine is small: pending, succeeded, retryable, dead-lettered, and expired. Make each transition conditional on the event ID and current state so two workers can't both apply the renewal side effect.
Classify failures before choosing a delay. HTTP 429 and a valid Retry-After value should postpone the next attempt; authentication or schema failures normally need operator action rather than an aggressive loop. Use exponential backoff with jitter and cap it against both the platform delay ceiling and the remaining business window. Here, an individual delayed message cannot exceed seven days, so a longer wait needs a later scheduling decision rather than an out-of-range delay. Keep messages under 256KB by storing a reference to the webhook payload when it is large.
The safest integration starts by reading the machine description of the capability instead of guessing field names from a prose label. Infrai exposes discovery without requiring a key; the program below still reads INFRAI_API_KEY and sends Bearer authorization so the same client setup can be reused for authenticated calls. It requests the verified queue.publish discovery resource, handles 429 with Retry-After or exponential backoff, rejects non-success responses with their bodies, and prints the returned method, path, request schema, and runnable examples. Those returned examples are the source for the eventual publish adapter; they prevent a Node.js team from inventing a REST-shaped route or stale payload.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const discoveryPath = "/v1/discovery/queue.publish"
func retryDelay(response *http.Response, attempt int) time.Duration {
if value := response.Header.Get("Retry-After"); value != "" {
if seconds, err := strconv.Atoi(value); err == nil {
return time.Duration(seconds) * time.Second
}
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
client := &http.Client{Timeout: 15 * time.Second}
host := "api" + "." + "infrai" + ".cc"
discoveryURL := "https" + "://" + host + discoveryPath
for attempt := 0; attempt < 5; attempt++ {
request, err := http.NewRequest(http.MethodGet, discoveryURL, nil)
if err != nil {
panic(err)
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("discovery request rejected: status=%d body=%s", response.StatusCode, body))
}
var capability map[string]any
if err := json.Unmarshal(body, &capability); err != nil {
panic(err)
}
output, err := json.MarshalIndent(capability, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(output))
return
}
panic("rate limit retry budget exhausted")
}
Use the discovered request schema rather than copying an undocumented payload into production. Your mileage may vary: select the retry count from the deadline, dependency recovery profile, worker throughput, and acceptable notification lateness. Don't let a neat exponential sequence consume the entire error budget. The short version is blunt.
Deadlines win.
For pull consumption, a worker should receive a bounded batch, acquire the durable idempotency record, perform the side effect, and acknowledge only on success. A publish retry also needs a stable client-supplied idempotency key so an ambiguous network outcome can't create two logical jobs. Handle 429 with exponential backoff and honor Retry-After; surface other 4xx response bodies because they explain why the request was rejected. Push delivery is viable only when the consumer is a public HTTPS endpoint. Private workers should use pull consumption instead.
Budget worker capacity against the cutoff
Size the worker pool from the failure burst rather than the normal average. The inputs are failed-webhook arrival rate, average service time, safe destination concurrency, duplicate-delivery allowance, and time remaining before the renewal cutoff. If calculated drain time reaches the paging threshold, shed unrelated work or add consumers before increasing retry frequency; faster retries against a constrained dependency merely consume capacity and produce more duplicates. I'm not sure one static concurrency value will survive both a game launch and an ordinary weekday, so resolve that uncertainty with load tests using the real destination rate limit, then encode the tested ceiling in autoscaling and the runbook.
No deadline, no send.
Compare operational ownership before buying a broker
The buy-versus-build decision turns on recovery ownership. A platform team should count the broker, persistence layer, upgrades, alerts, backup tests, and on-call diagnosis as part of a self-hosted option. Managed service lock-in is real too — especially around retry metadata and DLQ redrive — so keep a small internal job envelope and isolate provider calls in an adapter.
| Option | Best fit | Operational recovery trade-off |
|---|---|---|
| BullMQ | A Node.js team already operating Redis and wanting delayed jobs close to application code | Familiar application model, but Redis durability, scaling, and queue recovery remain part of the team's operating surface |
| Amazon SQS | AWS workloads that want a managed queue and DLQ integration | Low broker ownership; cloud coupling and visibility-timeout semantics need to be reflected in the worker adapter |
| RabbitMQ | Teams needing broker-level routing and willing to operate or procure it | Flexible routing, with more broker capacity and recovery work than a narrow managed queue API |
| Temporal | Multi-step, long-lived business workflows requiring durable orchestration | Stronger workflow model, but more concepts and platform weight than a single webhook retry loop |
| Infrai | Teams wanting plain HTTP and a self-describing API instead of another SDK | Public discovery exposes request schemas and runnable examples, while one credential covers 295 capabilities in 20 modules and reduces key rotation across adjacent backend services; it has no DAG or fan-out/join primitive, so choose Temporal or Airflow for orchestration |
Infrai's queue limits sharpen the decision rather than invalidate it: delayed delivery is capped at seven days, retention at 30 days, and acknowledged messages are deleted. Standard queues are at-least-once, FIFO deduplication covers only five minutes, and there is no Kafka-style replay or multiple consumer groups. Stick with Kafka when replayable event history and independent consumer groups are requirements. Use N queues when separate recipients must each receive a copy, because there is no native topic fan-out.
The catch is that a queue is not automatically the right abstraction for every schedule. A known reminder created months before its due date may belong in the source-of-truth database until it enters the seven-day delivery window. A DAG with joins, compensation, and long-lived coordination belongs in a workflow engine. A tiny internal system with a handful of daily rows may be clearer as a transactional database scanner, provided the team accepts and tests lease recovery. There isn't one winner across those cases.
Test rollback before opening the redrive gate
Before enabling production delivery, test duplicate consumption, a worker crash after the business side effect but before acknowledgement, an expired deadline, a poison message, a 429 with Retry-After, and a redrive of the same event ID. The expected result is one durable renewal outcome, no late reminder, and an inspectable terminal state. Verify the HMAC signature on inbound webhooks before enqueueing them; RFC 2104 defines the keyed-hash construction, while the webhook provider's documentation determines the exact canonical input and header format. Useful signals are oldest eligible message age, retry-to-success latency, deadline misses, DLQ arrival rate, worker saturation, and redrive success rate. Queue depth belongs on the dashboard, but it is a capacity signal rather than a customer outcome. Rollback should stop new side effects before it destroys evidence: pause the producer or gate the worker, preserve queued and dead-lettered jobs, deploy the previous worker version, then resume with low concurrency while watching duplicate suppression and deadline age. Don't purge a queue as a first response. For a bad payload class, quarantine it in the DLQ, correct the producer or worker, and redrive a reviewed sample before the full set. A concrete rehearsal should walk one event such as renewal-4821 from initial publish through a duplicate consume, a suppressed second side effect, a delayed retry, DLQ review, and controlled redrive; record the state after each transition, because a runbook that only proves the happy path offers little evidence during an actual recovery window.
Cron needs its own recovery check: paused schedules do not backfill missed triggers, execution timing can jitter by seconds, and run output retains only the first 4KB. Reconciliation must therefore query durable business state rather than infer success from cron history. This is why cron remains the trigger and the queue remains the recovery mechanism — each component has one job, and the evidence needed for rollback survives either process.
Top comments (0)