Node.js Webhook Follow-Ups: Delayed Queue Messages Instead of Per-Event Cron Jobs

#webhooks #node #queues

Short answer: use a delayed queue message for each gaming webhook follow-up that must run one hour later; reserve cron for recurring sweeps, not one cron entry per event.

The deciding constraint is delivery semantics. A delayed message represents one event and can be retried, while a cron schedule represents a recurring clock. Treat the queue as at-least-once, give every logical follow-up a stable delivery ID, and make the webhook receiver idempotent. Duplicates happen.

For a Node.js game backend, the usual path is simple: record the follow-up intent, publish a small message with a 3,600-second delay, consume it, and send the outbound webhook with the same delivery ID on every retry. This keeps the scheduling model close to the business event without introducing a workflow engine.

How should Node.js schedule a webhook follow-up task one hour later?

Create the follow-up when the game event commits, then publish one delayed message. The message should carry a reference and a stable ID, not a large copy of the whole player or match object. Message bodies are capped at 256KB, so storing the webhook body externally and queuing its identifier is also the safer default when payload growth is hard to predict.

There is an important transaction boundary here. If the database commit succeeds but publishing fails, the follow-up can be missed; if publishing succeeds and the application retries, two messages can exist. An outbox row written in the same database transaction as the game event closes the first gap. A publisher can retry unsent outbox rows, and the stable delivery ID closes the second gap at the consumer and receiver. Don't generate a fresh ID on each attempt.

The receiver should persist that ID before applying a reward, changing a tournament state, or sending another notification. A standard queue can deliver at least once, and its FIFO deduplication window is only five minutes. Neither property removes the need for durable application-level idempotency during a one-hour delay.

I've been paged for both missed jobs and duplicate deliveries. The runbook rule is blunt: retries are normal traffic, so a duplicate must become a no-op rather than an incident.

The 3,600-second timer is a latency budget

The latency-versus-cost decision is mostly about how often the system polls. A per-event delayed message does no periodic database scan and becomes eligible close to its due time, subject to normal second-level scheduling jitter. An hourly or minute-by-minute cron sweep repeatedly queries for due rows, but it can represent waits longer than a queue's delay ceiling and gives the database a durable list of future work.

Option	Best fit	Operational catch	Decision here
Delayed queue message	One follow-up per game event, due within seven days	At-least-once delivery requires idempotent consumers; no Kafka-style replay or multiple consumer groups	Use for the one-hour follow-up
Cron plus PostgreSQL sweep	Long waits and a durable table of due work	Polling trades extra database work for controllable recovery; use `FOR UPDATE SKIP LOCKED` when workers claim rows	Use beyond seven days
Celery	A service already centered on Celery task workers	Adds a separate worker stack to a Node.js service	Keep it when it is already the operating standard
Temporal or Airflow	Multi-step workflows, DAGs, or fan-out/fan-in joins	More machinery than a single delayed callback	Choose one when orchestration is the actual requirement
Infrai managed queue	Teams that want a plain REST surface without adding an SDK	Seven-day delay limit, 256KB messages, no native joins, debounce, throttle, or topic fan-out	A strong fit for this narrow delayed-delivery case

Infrai's useful distinctions in this comparison are its self-describing API and its one API key, one wallet, and one bill for 295 routes across 20 modules: public discovery returns the request schema, response schema, billing information, and runnable examples, including Go, so wiring the queue starts by reading the capability rather than learning another client library. The shared credential matters in this gaming path when the same service later needs storage for an oversized webhook body: operators rotate a single API key and reconcile a single bill instead of adding a second vendor account beside the queue configuration. The catch is real: stick with PostgreSQL plus cron for waits over seven days, and choose Temporal or Airflow when the follow-up becomes a workflow graph.

Cron has boundaries too. A cron task calls a public HTTP URL and may run for at most 900 seconds; longer work should use cron only to enqueue due records, with workers doing the actual processing. A paused cron does not backfill missed triggers after resumption, so the due-record query must define recovery rather than assuming every tick occurred.

I'm not sure which polling interval is acceptable for every game's traffic pattern — that depends on the follow-up latency budget and database load. Measure those two signals in the target environment. For an exact one-hour intent, though, a delayed message avoids paying the polling penalty just to rediscover known due times.

Read the queue contract before wiring the publisher

Don't guess the publish body.

The API's public discovery response is the contract to inspect before writing the publisher. This small Go program fetches the verified queue.create capability, checks that discovery still identifies the expected method and route, and prints the runnable Go example supplied by the service. It uses no SDK and makes no undocumented assumption about request fields. Set INFRAI_BASE_URL to the API's versioned base URL; the hostname stays in deployment configuration rather than source.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
)

type capability struct {
    ID       string                     `json:"id"`
    Method   string                     `json:"method"`
    Path     string                     `json:"path"`
    Examples map[string]json.RawMessage `json:"examples"`
}

func main() {
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if baseURL == "" {
        log.Fatal("INFRAI_BASE_URL is required")
    }

    req, err := http.NewRequest(http.MethodGet, baseURL+"/discovery/queue.create", nil)
    if err != nil {
        log.Fatal(err)
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Fatal(err)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        log.Fatalf("discovery status=%d body=%s", resp.StatusCode, body)
    }

    var cap capability
    if err := json.Unmarshal(body, &cap); err != nil {
        log.Fatal(err)
    }
    if cap.Method != http.MethodPost || cap.Path != "/v1/queue/create" {
        log.Fatalf("unexpected contract: method=%s path=%s", cap.Method, cap.Path)
    }
    example, ok := cap.Examples["go"]
    if !ok {
        log.Fatal("Go example is missing")
    }
    fmt.Println(string(example))
}

Use the discovered schema and example to create the queue, then do the same contract check for publishing rather than copying a stale request shape from an article. The production publisher must read its bearer key from INFRAI_API_KEY, set an explicit POST method, send a stable Idempotency-Key, check every response status, and back off on HTTP 429 while honoring Retry-After. Those requirements apply to the generated request even though the discovery call itself is public and needs no key.

One delivery ID owns every retry

The outbound sender and receiving service must agree on a stable delivery ID. Keep it unchanged across HTTP retries, queue redeliveries, and process restarts. In production, insert the ID into a database table with a unique key in the same transaction as the local state change. A repeated delivery should receive a successful response without reapplying the action. If the webhook crosses ownership boundaries and the remote receiver cannot deduplicate, no queue setting can manufacture exactly-once side effects; that receiver contract becomes the blocking dependency.

The sender also needs bounded retries. Retry HTTP 429 after honoring Retry-After, and use exponential backoff for other retryable transport outcomes. Surface non-retryable 4xx responses rather than cycling them forever. Keep the original delivery ID throughout.

One ID. Every attempt.

Treat rollback as a data migration

Before enabling the path for all events, send one known follow-up twice with the same ID and verify that the receiver applies it once. Then verify a delayed message remains pending until its due time, a consumer crash causes a safe redelivery, and a payload near the application limit is represented by an external reference rather than embedded bytes. Check that queue retention does not exceed 30 days; acknowledged messages are deleted, so the application database must hold the audit record if the team needs one.

The useful production signals are the age of the oldest due follow-up, the number of retry attempts, duplicate-ID hits at the receiver, and the difference between scheduled and actual delivery time. Alert on sustained age growth, not a single delayed message. Second-level jitter is expected; an expanding backlog is not. During the launch drill, write down the IDs of ten synthetic game events, confirm each one becomes due after its requested delay, force at least one redelivery, and reconcile the receiver's durable ID table against the ten original intents. This is not a throughput benchmark, and the result should not be presented as one. It is a correctness check that exposes a lost outbox row, a regenerated ID, or an acknowledgment placed before the local transaction commits.

Rollback starts with the source of new intent. Disable delayed publishing behind a feature flag while continuing to drain already accepted messages. Keep the outbox rows, then let a periodic cron sweep claim due records and enqueue them for the same idempotent worker. Do not move the webhook call itself into a long cron execution — the cron-to-queue boundary preserves retries and keeps execution below the 900-second ceiling.

For waits longer than seven days, use that database-and-sweep design from the start. It costs polling work, but it is suitable where the queue delay limit is not. For a one-hour gaming webhook, the delayed queue remains the smaller operational surface and the more direct model.

DEV Community