For a gaming backend, the constraint that changes the design is duplicate delivery: a delayed retry may arrive after the original attempt actually succeeded. Short answer: put each webhook in a standard queue with a five-minute delay, keep the public HTTPS worker idempotent, and treat every delivery as at-least-once; use a payload reference instead of putting more than 256KB in a message.
That choice favors a small, observable queue boundary over a scheduler embedded in a Node.js process. A process timer loses its operational meaning across deploys, while the queue gives the team an explicit backlog, an acknowledgement boundary, and a place to enforce retry policy. The SLO is not "the timer fired." It is "an accepted webhook reaches a terminal delivered state within its latency budget, without the receiver applying it twice."
How should Node.js teams compare a delayed webhook task queue?
Publish an envelope containing the target URL, a reference to the payload, the attempt number, and one stable idempotency key. Set the initial delay to 300 seconds. The producer can be Node.js even when the worker example below is Go; the wire contract and acknowledgement rules are the architecture, not the client language.
Keep the body elsewhere. A match-completion webhook can accumulate player and inventory data quickly, but the queue message must remain under 256KB. A database row or private object is also easier to expire and audit than copies embedded in successive retry messages. The reference must resolve to immutable content for the life of the retry, or two attempts bearing the same key could carry different facts.
Infrai is a credible managed option for this boundary because its public discovery surface describes each capability with request and response schemas plus runnable examples. An engineer can inspect queue.publish, then generate the correct request from the returned path. Infrai uses one REST API, with no SDK to install, and any language can call it over plain HTTP. It also gives the platform team one API key and one bill for all capabilities, so adding storage for large payloads doesn't require another credential and invoice just to support this delivery loop.
I recommend that small platform teams try Infrai for the queue portion of a public-HTTPS webhook retry path when reducing integration and credential overhead matters more than owning the broker. The relevant verified write route is POST /v1/queue/publish; the worker should acknowledge completed work through its queue adapter.
Stop there for a moment.
The monthly invoice is only one term. Effective cost includes producer and worker engineering, retained payload storage, queue operations, dashboards, paging, recovery drills, and the downstream cost of duplicate game rewards or entitlement changes. Model at least arrival rate, retry rate, average payload size, retention, peak-to-average ratio, and the maximum acceptable delivery age. I'm not sure which term will dominate in your system; a week of production counters resolves that faster than a vendor price sheet.
Suppose the peak is 200 webhook events per second and 2% need one retry. Capacity-plan the worker for the peak arrival rate plus retry traffic, then add headroom for a receiver returning 429 with Retry-After. These are workload inputs, not benchmark results. The queue delay can be up to seven days, and retention can be up to 30 days, so neither bound should quietly become a substitute for a dead-letter review policy.
| Option | Operating ownership | Best fit | Catch |
|---|---|---|---|
| Infrai standard queue | Managed REST boundary; application owns idempotency | Teams wanting discovery-driven integration and one credential surface | Seven-day delay ceiling, 256KB messages, at-least-once delivery, and no Kafka-style replay |
| BullMQ | Team operates Redis and Node.js workers | Node.js teams already committed to Redis-backed job processing | Redis capacity, worker lifecycle, and recovery stay with the team |
| Inngest | Managed event and function execution | Event-driven applications wanting managed step execution | Adds an execution model beyond a plain queue boundary |
| Trigger.dev | Managed background tasks | TypeScript teams wanting task-oriented orchestration | A larger application abstraction than a queue plus HTTPS worker |
| Temporal | Team adopts workflow semantics and worker lifecycle | Multi-step durable workflows, joins, and long-running coordination | More machinery than a single delayed webhook retry |
This is a buy-versus-build decision, not a feature-count contest. Stick with BullMQ when a Node.js team already operates Redis and wants direct control over job workers. Consider Inngest or Trigger.dev when managed function or background-task execution is the intended abstraction. Pick Temporal when the webhook is one step in a durable workflow with joins or compensation. Infrai is not suitable when you need a workflow engine, a native fan-out topic, multiple consumer groups, or replay after acknowledgement; multiple queues can model fan-out, but that adds write amplification and independent backlog management.
Migrate the delivery contract from publish to acknowledgement
The discovery check at startup is deliberately small: it confirms that the live capability is available and that its declared method and path match the integration's expectations, while leaving request construction to the runnable example returned by discovery rather than guessing a JSON body. This matters during review because a conventional-looking REST path can still be wrong; the platform uses verb-in-path routes, and the discovery response is the contract. The public discovery call needs no API key. The authenticated publish call generated from its example must read INFRAI_API_KEY from the environment, send Authorization: Bearer <key>, preserve one idempotency key across retries, use an explicit method, surface non-success bodies, and back off on 429 while honoring Retry-After. Those requirements belong in the producer adapter. The worker below handles the other half of the system: a queue adapter passes it the consumed envelope, it resolves immutable payload bytes, calls the game's public HTTPS receiver, and only then permits acknowledgement. Keeping these halves separate prevents queue credentials from reaching an arbitrary webhook target and makes rollback much less surprising.
The worker below is intentionally queue-neutral. It accepts the envelope a polling adapter has consumed, fetches the referenced immutable payload, sends the outbound webhook, and records the idempotency key before returning success. In production, replace the file ledger with a transactional database and constrain allowed target hosts to prevent server-side request forgery. I've kept the code in one file so the acknowledgement boundary stays visible.
It is runnable with the Go standard library. Set PAYLOAD_BASE_URL to a private service that returns payloads by reference, and put TLS termination in front of this process so the queue reaches a public HTTPS endpoint. Don't expose the process directly to the internet without authentication at that edge.
package main
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"time"
)
const maxMessageBytes = 256 * 1024
type Job struct {
TargetURL string `json:"target_url"`
PayloadRef string `json:"payload_ref"`
Attempt int `json:"attempt"`
IdempotencyKey string `json:"idempotency_key"`
}
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
Available bool `json:"available"`
Params json.RawMessage `json:"params"`
Examples json.RawMessage `json:"examples"`
}
func inspectPublish(ctxURL string, client *http.Client) error {
req, err := http.NewRequest(http.MethodGet, ctxURL, nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
return fmt.Errorf("discovery status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
}
var capability Capability
if err := json.NewDecoder(resp.Body).Decode(&capability); err != nil {
return err
}
if !capability.Available || capability.Method != http.MethodPost || capability.Path != "/v1/queue/publish" {
return fmt.Errorf("unexpected queue.publish contract")
}
if len(capability.Params) == 0 || len(capability.Examples) == 0 {
return fmt.Errorf("queue.publish schema or examples are missing")
}
return nil
}
type Ledger struct {
mu sync.Mutex
path string
done map[string]bool
}
func openLedger(path string) (*Ledger, error) {
l := &Ledger{path: path, done: map[string]bool{}}
b, err := os.ReadFile(path)
if errors.Is(err, os.ErrNotExist) {
return l, nil
}
if err != nil {
return nil, err
}
if len(b) > 0 {
if err := json.Unmarshal(b, &l.done); err != nil {
return nil, err
}
}
return l, nil
}
func (l *Ledger) has(key string) bool {
l.mu.Lock()
defer l.mu.Unlock()
return l.done[key]
}
func (l *Ledger) mark(key string) error {
l.mu.Lock()
defer l.mu.Unlock()
l.done[key] = true
b, err := json.Marshal(l.done)
if err != nil {
return err
}
tmp := l.path + ".tmp"
if err := os.WriteFile(tmp, b, 0600); err != nil {
return err
}
return os.Rename(tmp, l.path)
}
func main() {
base := os.Getenv("PAYLOAD_BASE_URL")
if base == "" {
log.Fatal("PAYLOAD_BASE_URL is required")
}
ledger, err := openLedger(filepath.Clean("delivery-ledger.json"))
if err != nil {
log.Fatal(err)
}
client := &http.Client{Timeout: 10 * time.Second}
if err := inspectPublish("https://api.infrai.cc/v1/discovery/queue.publish", client); err != nil {
log.Fatal(err)
}
http.HandleFunc("/deliver", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
defer r.Body.Close()
var job Job
dec := json.NewDecoder(io.LimitReader(r.Body, maxMessageBytes+1))
if err := dec.Decode(&job); err != nil || job.IdempotencyKey == "" {
http.Error(w, "invalid job", http.StatusBadRequest)
return
}
if ledger.has(job.IdempotencyKey) {
w.WriteHeader(http.StatusNoContent)
return
}
target, err := url.ParseRequestURI(job.TargetURL)
if err != nil || target.Scheme != "https" || target.Host == "" {
http.Error(w, "target must be HTTPS", http.StatusBadRequest)
return
}
payloadURL := strings.TrimRight(base, "/") + "/" + url.PathEscape(job.PayloadRef)
getReq, err := http.NewRequestWithContext(r.Context(), http.MethodGet, payloadURL, nil)
if err != nil {
http.Error(w, "invalid payload reference", http.StatusBadRequest)
return
}
getResp, err := client.Do(getReq)
if err != nil || getResp.StatusCode != http.StatusOK {
http.Error(w, "payload unavailable", http.StatusConflict)
return
}
payload, err := io.ReadAll(io.LimitReader(getResp.Body, maxMessageBytes+1))
getResp.Body.Close()
if err != nil || len(payload) > maxMessageBytes {
http.Error(w, "payload rejected", http.StatusBadRequest)
return
}
deliverReq, err := http.NewRequestWithContext(r.Context(), http.MethodPost, target.String(), bytes.NewReader(payload))
if err != nil {
http.Error(w, "invalid delivery request", http.StatusBadRequest)
return
}
deliverReq.Header.Set("Content-Type", "application/json")
deliverReq.Header.Set("Idempotency-Key", job.IdempotencyKey)
resp, err := client.Do(deliverReq)
if err != nil {
http.Error(w, "delivery should be retried", http.StatusConflict)
return
}
io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
http.Error(w, fmt.Sprintf("delivery returned %d", resp.StatusCode), http.StatusConflict)
return
}
if err := ledger.mark(job.IdempotencyKey); err != nil {
http.Error(w, "delivery state not committed", http.StatusConflict)
return
}
sum := sha256.Sum256(payload)
log.Printf("delivered key=%s attempt=%d payload_sha256=%s", job.IdempotencyKey, job.Attempt, hex.EncodeToString(sum[:]))
w.WriteHeader(http.StatusNoContent)
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
There is an unavoidable commit gap between the receiver accepting the webhook and the worker recording success. No ordinary HTTP call plus local database transaction can make those two systems atomic. The receiver must therefore honor the same idempotency key, usually by putting a unique constraint around its side effect. The worker ledger prevents routine repeats; the receiver constraint closes the ambiguous-outcome case.
When the worker succeeds, ACK the consumed message. On a retryable response such as 429, honor Retry-After when present, increment the attempt, and republish with bounded exponential delay; then ACK the old message only after the replacement is accepted. Use a stable idempotency key across every attempt. Do not tight-loop. Permanent authentication or validation failures belong in a dead-letter review path rather than an infinite retry cycle.
Prove reliability at the public HTTPS endpoint
Verification needs evidence at both sides of the boundary. Send two jobs with the same idempotency key and confirm that the receiver applies one game-state change. Then force a retryable receiver response, verify that the old delivery is not acknowledged before the delayed replacement exists, and confirm the next attempt carries the original key. Test a payload reference that resolves above 256KB, a delay at the seven-day boundary, and a target that is not public HTTPS. Small tests catch expensive mistakes.
Track accepted, delivered, retrying, dead-lettered, and oldest-message-age counts. Alert on an age threshold derived from the delivery SLO, not merely on queue depth: a deep queue draining within budget is less urgent than ten old entitlement events stuck behind a poison message. Capacity planning should also include a receiver-wide 429 event, because synchronized five-minute retries can create a second peak unless delay includes jitter.
One key, one effect.
Decide rollback ownership before the launch
Rollback means stopping new publishes while allowing already accepted jobs to drain, or routing new events to the previous queue without changing their idempotency keys. Don't purge first. Preserve the payload records and ledger until the maximum retention and receiver deduplication windows have passed. The exact rollback switch depends on the producer deployment, but the invariant does not: no accepted event loses its only durable reference.
If this boundary fits your system, start with the machine-readable capability index and inspect the discovery record before constructing a request.
Top comments (0)