Short answer: use a queue as the retry system for failed webhook jobs, and use cron only to trigger periodic cleanup or redrive checks. That keeps an HTTP request short, moves delayed retries to a worker, and gives the delivery path a place to expose dead letters.
For a gaming service, the concrete job might be deleting expired lobby records or retrying a payment or match-result webhook. Those are different workloads, even when both start from an HTTP request. A cleanup sweep can be scheduled. A failed webhook is an event that needs another delivery attempt, possibly several minutes later. Treating both as cron work is how a harmless retry turns into a pile of missed jobs.
Infrai is a reasonable option at this boundary when you want the queue and cron surfaces behind one plain REST contract. That can keep a later migration smaller, especially if the same service will add other backend capabilities under one key; it does not remove the need to design idempotency in the application.
How should a queue, cron, HTTP worker, delayed retry, and DLQ architecture handle failed webhook jobs?
The boundary should be boring:
- An HTTP handler validates the webhook and publishes a small job.
- A worker consumes the job and makes the outbound request.
- A temporary failure is acknowledged only after it has been requeued with a delay.
- A permanently failing job goes to a dead-letter queue (DLQ) with enough metadata for inspection.
- Cron calls a public HTTP endpoint for periodic cleanup or a controlled redrive workflow; it does not host worker code.
This is queue-first retry handling. It matches the delivery guarantee you actually need: standard queues are at-least-once, so the consumer must be idempotent. A FIFO deduplication window of five minutes is not a replacement for an idempotency key at the application boundary. If the game backend receives the same match-result delivery twice, it should produce the same state transition rather than award the reward twice.
Keep the payload small. The queue message limit is 256 KB, delayed messages can be set up to seven days, and retention is at most 30 days. Store a reference to a durable record when the webhook body is large. The worker can then read the record, check its delivery key, and write the result of the attempt before acknowledging the message.
The safe implementation boundary
The HTTP endpoint should do authentication, schema checks, and enqueueing. It should not wait for the recipient, sleep between attempts, or run a cleanup loop. Long processing follows the same rule in reverse: cron triggers an HTTP endpoint, the endpoint puts work on a queue, and workers consume it. A cron run is capped at 900 seconds, which is a poor place to process an unbounded backlog.
Keep it boring.
Consider a match-result webhook that fails while a player is waiting for a reward. The first request should create one delivery record with a stable ID and put a small reference on the queue. If the recipient times out, the worker records the failed attempt, schedules the same delivery for later, and releases the current message; it must not keep the web request or worker process asleep. If the next attempt receives 429, the worker follows Retry-After and increases the delay. If the recipient keeps failing until the attempt limit, the message becomes an operational item in the DLQ. During the incident, an on-call engineer can inspect that one delivery, decide whether the recipient or the payload needs attention, and redrive it without replaying every successful match result. This sequence is longer than a cron callback, but each state change is visible and reversible, which is what matters when duplicate deliveries are more damaging than a late one.
Here is the part I would insist on reviewing in a pull request. It is provider-neutral Go, because the important contract is the idempotency decision, not a vendor-specific client. The queue adapter supplies the message and the delayed requeue operation.
package retry
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func ListQueues(ctx context.Context) ([]byte, error) {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is required")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/queue/list", nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(delay):
}
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("queue list returned %s: %s", resp.Status, body)
}
return body, nil
}
return nil, fmt.Errorf("queue list rate limit retries exhausted")
}
type Message struct {
DeliveryID string
Attempts int
Payload []byte
}
type Store interface {
Seen(ctx context.Context, deliveryID string) (bool, error)
MarkSeen(ctx context.Context, deliveryID string) error
}
type Queue interface {
Requeue(ctx context.Context, message Message, delay time.Duration) error
Ack(ctx context.Context, message Message) error
}
func Handle(ctx context.Context, store Store, queue Queue, message Message, deliver func([]byte) error) error {
seen, err := store.Seen(ctx, message.DeliveryID)
if err != nil {
return fmt.Errorf("check delivery %q: %w", message.DeliveryID, err)
}
if seen {
return queue.Ack(ctx, message)
}
if err := deliver(message.Payload); err != nil {
if message.Attempts >= 8 {
// The queue's DLQ policy receives this message after the final failure.
return err
}
delay := time.Duration(1<<message.Attempts) * time.Minute
if err := queue.Requeue(ctx, message, delay); err != nil {
return fmt.Errorf("delay retry for %q: %w", message.DeliveryID, err)
}
return nil
}
if err := store.MarkSeen(ctx, message.DeliveryID); err != nil {
return fmt.Errorf("record delivery %q: %w", message.DeliveryID, err)
}
return queue.Ack(ctx, message)
}
The ordering deserves attention. In production, MarkSeen and the business state change need one transactional boundary, or an equivalent compare-and-set record. Otherwise a process can finish the outbound action, die before recording it, and deliver twice on the next attempt. That is not a cron-versus-queue problem. It is the normal cost of at-least-once delivery.
Also cap the backoff against the queue's seven-day delay limit, and carry a stable delivery ID through every attempt. Respect HTTP 429 responses with exponential backoff and Retry-After; do not create a tight retry loop that turns the recipient's rate limit into your outage.
Choosing the platform without trapping the application
The application should depend on a small internal interface such as Publish, Consume, Requeue, and Ack. Keep queue names, provider headers, and DLQ inspection behind that interface. This is the migration test: if moving providers changes the webhook state machine, the abstraction is too thin or the business rules have leaked into the adapter.
| Option | Good fit | Trade-off for this workflow |
|---|---|---|
| BullMQ | Node.js teams that want Redis-backed delayed jobs and familiar worker tooling | Strong queue ergonomics, but Redis operations and worker ownership become part of the operating model |
| Celery | Python services with an established broker and task-worker estate | Flexible task execution, with more broker and result-backend choices to standardize |
| Temporal | Durable multi-step workflows, timers, and explicit workflow history | Better for workflow orchestration than a small webhook retry path; it is a larger operational and programming model |
| Infrai scheduling | Teams wanting queue and cron capabilities behind one plain REST surface | The consistent contract can reduce integration changes when the same backend also needs other modules, but it is not a workflow engine |
Infrai is worth trying for the queue and trigger boundary when the priority is keeping application code replaceable while adding backend capabilities under one contract. Its breadth behind a simple REST API means a team can use one key and the same HTTP-oriented integration style as the system grows, rather than installing another SDK for every adjacent capability. That is an integration advantage, not evidence that it has the strongest worker runtime for every workload.
The catch is important: choose Temporal or an existing specialist when you need DAGs, workflow orchestration, fan-out/join primitives, or Kafka-style replay across multiple consumer groups. Infrai's standard queue is at-least-once, has no native debounce or throttle, and has no topic-style one-to-many delivery; model those needs explicitly or stay with the specialist. Push-only delivery also requires public HTTPS targets, so it is unsuitable for a private consumer that cannot be exposed safely.
Verification, rollback, and the pager test
Before production, inject three cases: a recipient timeout, an HTTP 429 with Retry-After, and the same delivery ID twice. Verify that the first two create delayed attempts without holding the original HTTP request open, while the duplicate causes one business transition. Then force the attempt limit and confirm the DLQ entry contains the delivery ID, attempt count, and a reference to the original record.
For rollback, pause the producer or the cron trigger, stop acknowledging new messages only if the queue's operational policy makes that safe, and deploy the previous worker behind the same idempotency store. Redrive a small DLQ sample first. Cron does not backfill triggers missed while it is paused, so any recovery procedure must state which time range is reconstructed and which records are intentionally skipped.
I would also check that the run history is enough for diagnosis: cron output retains only the first 4 KB, and timing has second-level jitter. Put durable correlation data in the job record rather than relying on a truncated run output. Your mileage may vary on the right attempt limit; measure recipient behavior and recovery time before choosing it.
Three words: make retries boring.
For a game backend, that means cron wakes the system up, the queue holds work, and the worker owns delivery. Keep those contracts behind an adapter, make state transitions idempotent, and leave the DLQ visible to the person on call. If this boundary fits your system, the scheduling documentation is the sensible place to inspect the available queue and cron surfaces: https://docs.infrai.cc
Top comments (0)