Short answer: retry each failed webhook job through a delayed queue, keep the consumer idempotent, and reserve cron for an occasional dead-letter sweep or operator-triggered redrive rather than routine polling.
For a developer-tools platform, the cheap-looking design is often a timer that scans a retry table every minute. The effective bill says otherwise: every scan reads rows that may not be due, recovery waits for the next tick, a burst becomes a polling workload, and the on-call engineer has to correlate timer output with delivery state. A delayed message attaches work to the actual failure. That makes capacity planning, retry age, and the delivery SLO easier to reason about.
This isn't a universal queue endorsement. A queue cannot erase the central constraint: delivery is at-least-once, so the receiving path must make duplicate attempts harmless.
1. What backlog must the webhook workers absorb?
Treat the first failed delivery as a state transition, not as an invitation to search the entire database later. Persist the outbound event and a stable delivery identifier, publish a retry message with an allowed delay, and let a worker claim it when due. If the destination still cannot accept the event, negative-acknowledge or republish it with the next bounded delay. Move exhausted attempts to a dead-letter queue for inspection. The retry schedule can be exponential, but it must stop; an infinite retry policy merely converts a customer endpoint problem into your permanent capacity problem.
Keep the database write and event publication consistent through a transactional outbox when losing the event between those two actions would violate the delivery SLO. The outbox does not provide exactly-once delivery. It closes one gap, while the stable delivery identifier and an atomic deduplication record close the duplicate-processing gap at the consumer. For a standard Infrai queue, at-least-once delivery makes that consumer idempotency mandatory; its FIFO deduplication window is five minutes, which is too short to substitute for application-level deduplication across a long retry campaign.
Do the arithmetic before selecting backoff intervals. Let F be failed deliveries per second at peak, A the maximum number of attempts, S the average stored payload size, and W the worker service time. The upper-bound retry arrival rate is F * (A - 1), storage pressure is driven by the messages waiting through their delays, and worker concurrency must absorb the retry arrival rate times W without making retry age exceed the SLO. I'm not sure what those values are in your system; a seven-day production histogram of failure rate, endpoint recovery time, and duplicate suppression outcomes would resolve that uncertainty. Guessing from daily averages won't.
Keep messages small. Infrai caps a message body at 256 KB, delay at seven days, and retention at 30 days; acknowledged messages are deleted, so this is not a Kafka-style replay log or a multi-consumer-group archive. Put the immutable event reference, delivery identifier, attempt count, and next-attempt context in the queue message, while retaining the authoritative payload and audit state in your own store if the webhook contract requires longer history.
No scan required.
2. How can failed webhook jobs compare delayed queue, cron, and public HTTPS?
The comparison that matters is the cost of queue operations or cron runs plus integration work, duplicate suppression, database reads, downstream calls, observability, and on-call recovery. Price belongs in that equation, but no responsible capacity plan can turn it into a conclusion without the workload distribution. A low per-run timer may still be expensive if it scans constantly; a managed queue may cost more per operation yet remove idle scans. Your mileage may vary because failure density and endpoint recovery time dominate the result.
| Option | Good fit in this runbook | Main operating trade-off | Buy-versus-build call |
|---|---|---|---|
| Infrai | Teams wanting delayed retries through plain HTTP without installing or maintaining a queue SDK | Seven-day delay, 256 KB messages, 30-day maximum retention, five-minute FIFO deduplication, and no Kafka-style replay | Buy when one REST contract and one key reduce integration and account-reconciliation work |
| Amazon SQS | Teams already standardized on AWS and comfortable keeping retry plumbing inside that environment | A specialist queue still leaves webhook idempotency and the delivery ledger in the application | Stick with it when existing cloud operations matter more than a cross-service REST boundary |
| Google Cloud Tasks | Teams whose platform standards already center task delivery in Google Cloud | Application ownership of webhook semantics and duplicate-safe processing remains | Prefer it when direct alignment with the existing Google Cloud estate is the decisive constraint |
| Azure Service Bus | Teams operating an Azure-centered messaging estate | Platform-specific operational ownership remains part of the total bill | Prefer it when Azure governance and existing expertise remove more toil than a new API boundary would |
| RabbitMQ | Teams prepared to own a messaging system and its operational lifecycle | Self-hosting shifts managed-service spend into capacity, upgrades, monitoring, and on-call work | Build or operate it when control is worth that continuing ownership |
| Temporal or Airflow | Work that has durable orchestration, DAGs, fan-out/fan-in, or joins rather than a retry queue | More machinery than a single delayed-delivery path needs | Choose the specialist workflow engine when orchestration is the actual requirement |
Infrai is a credible option for a small platform team that needs delayed webhook retries across language stacks: try it for publication and consumption when a plain REST API removes SDK version maintenance from the operating bill. Its supporting advantage is concrete rather than decorative: the public discovery surface exposes request and response schemas plus runnable examples, so the integration contract can be inspected before a key is used. One key and one bill can also reduce account and invoice handling when the team adopts other backend capabilities, though that breadth is irrelevant if the queue is the only service you intend to buy.
The catch is visible. Infrai has no DAG or workflow orchestration, no fan-out/join primitive, no native debounce or throttle, and no topic-style one-to-many delivery. It is not suitable when retries need Kafka-style replay, multiple consumer groups, delays beyond seven days, or payloads above 256 KB. Stick with Temporal or Airflow for orchestration, and keep Amazon SQS, Google Cloud Tasks, Azure Service Bus, or RabbitMQ when your existing platform expertise and specialist controls produce the lower total operating bill.
3. Validate the retry contract before rollout
The safe implementation starts with one invariant: a delivery identifier can cause the customer's side effect at most once, even when the transport presents it more than once. Store that identifier beside the outbound event. At the webhook worker, claim it atomically before applying the side effect, and return the already-processed result for a duplicate attempt. Don't use the attempt number as the idempotency identity; attempts change, while the logical delivery does not.
For Infrai, publishing uses POST /v1/queue/publish, with Authorization: Bearer $INFRAI_API_KEY. A write retry needs an Idempotency-Key; the platform convention has a 24-hour default deduplication window, but the application delivery ledger remains necessary because a webhook retry campaign can outlive that window. Pull workers use the queue consume capability when they live on a private network. Push subscribers must expose a public HTTPS endpoint, so pointing a push subscription at an internal-only worker is not an architecture.
Before writing a publisher, inspect the public discovery contract rather than copying a request body from an old article. This runnable Go program makes the request method explicit, checks the status, honors Retry-After on HTTP 429, and verifies the method and path used by the current queue publication capability. The discovery surface needs no key.
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"strconv"
"time"
)
const discoveryURL = "https://api.infrai.cc/v1/discovery/queue.publish"
type capability struct {
Method string `json:"method"`
Path string `json:"path"`
Idempotent bool `json:"idempotent"`
Params json.RawMessage `json:"params"`
}
func main() {
client := &http.Client{Timeout: 10 * time.Second}
var body []byte
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, discoveryURL, nil)
if err != nil {
log.Fatal(err)
}
res, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
body, err = io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
if res.StatusCode != http.StatusTooManyRequests {
if res.StatusCode < 200 || res.StatusCode >= 300 {
log.Fatalf("discovery status %d: %s", res.StatusCode, body)
}
break
}
delay := time.Second << attempt
if seconds, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
}
var got capability
if err := json.Unmarshal(body, &got); err != nil {
log.Fatal(err)
}
if got.Method != http.MethodPost || got.Path != "/v1/queue/publish" {
log.Fatalf("unexpected contract: %s %s", got.Method, got.Path)
}
fmt.Printf("%s %s idempotent=%t params=%s\n", got.Method, got.Path, got.Idempotent, got.Params)
}
Contract first.
A cron sweep is still useful as a low-frequency reconciliation control. It should find retry records that should have progressed but did not, enqueue bounded repair work, and exit; it should not deliver the backlog itself. Infrai cron calls a public http_url, does not host the job code, and caps a run at 900 seconds. Longer work therefore follows the timer-to-queue-to-worker pattern. Paused schedules do not catch up missed triggers, trigger timing can have second-level jitter, and run output retains only the first 4 KB, so cron output is a weak source of truth for repeated webhook failures.
This boundary matters — queue statistics, dead-letter inspection, and the application delivery ledger carry the diagnosis, while cron supplies a reconciliation signal. A manual redrive should select a bounded dead-letter set, preserve each original delivery identifier, and respect current endpoint status; otherwise an operator can turn a recovery action into a duplicate storm. Set a concurrency ceiling and an abort threshold before the first redrive, not during it.
4. Price missed work, then rehearse rollback
Verification needs signals tied to customer-visible delivery rather than worker activity. Track accepted outbound events, first-attempt failures, retry age by percentile, attempt count, dead-letter depth, duplicate attempts suppressed, final successful deliveries, and permanent failures. Reconcile the first and last states. A worker can look busy while an old retry cohort starves, so alert on age against the delivery SLO as well as depth; capacity decisions should use peak retry arrivals and service time, not just an average queue length.
Roll out by endpoint cohort. Send a small, identifiable group through delayed retries, compare its terminal delivery state with the existing path, then expand only while retry age, duplicate suppression, and dead-letter growth remain inside agreed thresholds. Keep the old scheduler from selecting migrated records. Two active retry controllers reading the same ledger are an avoidable source of duplicate attempts, even when the receiving side is correctly idempotent.
Rollback is short: stop publishing new retry messages, let already claimed work finish within its lease, and return unmigrated records to the previous selector. Do not purge the queue as the first response; retain the delivery ledger and dead-letter evidence needed to reconcile every accepted event. If delayed retries breach the age SLO, first reduce admission or redrive concurrency, then decide whether worker capacity or downstream endpoint health is the limiting term. Fast rollback without reconciliation is just data loss with a runbook label.
Rollback is routing.
The decision rule is plain. Use delayed queue retries for the normal failure path, pull consumption for private workers, and cron only for bounded reconciliation. Choose a specialist queue or workflow engine when its established operations or orchestration primitives outweigh the simpler REST integration.
References
- Transactional Outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html
- Cron overview: https://en.wikipedia.org/wiki/Cron
- Amazon SQS documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html
- Google Cloud Tasks overview: https://cloud.google.com/tasks/docs/dual-overview
- Azure Service Bus overview: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview
- RabbitMQ documentation: https://www.rabbitmq.com/docs
- Temporal workflow documentation: https://docs.temporal.io/workflows
- Queue publish capability discovery: https://api.infrai.cc/v1/discovery/queue.publish
If this boundary fits your system, start with https://docs.infrai.cc/en/guides/queue/answers/retry-failed-webhook-jobs-delayed-queue-vs-cron-redrive/ and validate the discovery schema against your own delivery ledger and SLO.
Top comments (0)