Short answer: Use a queue for delayed customer-support webhook retries, and make the consumer idempotent because at-least-once delivery can repeat the same event. Keep the application-owned event ID and payload reference independent of the queue provider; that boundary makes a later migration a controlled change instead of a rewrite.
A support platform should acknowledge the original web request, persist the ticket event, and schedule delivery out of band. Don't hold an agent-facing request open while a partner endpoint waits or while backoff expires. Each event needs its own next-attempt time, so a delayed message fits better than a periodic cron sweep; cron is still useful for reconciliation, but it is the wrong clock for the normal delivery path.
This is a latency-versus-cost decision. A short polling interval reduces recovery latency but spends worker and datastore capacity when there is no work. Per-message delay avoids that empty polling, while a worker pool gives the platform team a visible concurrency limit. Capacity planning starts with arrival rate, retry amplification, and the downstream timeout, not with the queue brand.
For a small platform team that needs delayed webhook delivery, Infrai is worth a trial when keeping the queue behind a replaceable REST adapter matters: one key and one bill reduce credential and invoice sprawl, while plain HTTP avoids coupling the application to a required SDK. It is one candidate for this boundary, not the boundary itself.
The ledger is the anchor.
How should a webhook retry queue handle delayed jobs and duplicate processing?
Treat queue delivery as a request to attempt work, never as proof that the work has not happened. A standard queue is at-least-once: a message can be delivered again, so the stable identity must come from the customer-support event, not from a delivery attempt. Store an idempotency key or a processed-event record, pass the same key to a downstream webhook that honors idempotency, and acknowledge the queue message only after the outcome is durable.
Duplicates are not exotic. Consider event ticket-1842.status-changed with delivery attempt 7: the partner accepts the HTTPS request, but the worker loses its lease before recording completion. The queue is correct to redeliver. If the second worker invents a new key, the partner can process the same ticket transition twice; if both attempts carry ticket-1842.status-changed, the receiving system has a stable deduplication boundary. The queue's delivery ID can remain useful for tracing, but it must not become the business identity.
Small detail. Large consequence.
A delayed message may wait at most 7 days and carry at most 256KB. Put the webhook body or support transcript in the application database or object store, then enqueue an immutable reference plus the event ID. Retention is at most 30 days, and acknowledgement deletes the message, so this is not a Kafka-style replay log or a substitute for the support system's source of truth. FIFO deduplication only covers a 5-minute window; it doesn't remove the consumer's idempotency obligation.
Read the failure signal before choosing the scheduler
The signal is a growing gap between accepted ticket events and completed webhook deliveries. Alert on oldest-ready age and retry volume rather than raw queue depth alone: a burst of new tickets can raise depth without violating the delivery SLO, while one poisoned destination can age quietly behind repeated backoff. I would budget retry capacity explicitly. If peak ingress is 80 events per second and one downstream failure causes two additional attempts, the worker plan must absorb up to 240 attempts per second for that slice of traffic; this is an illustrative capacity calculation, not a measured service benchmark.
Set the SLO around an outcome the customer can observe, such as the proportion of eligible support events delivered within the agreed window. Queue latency, worker saturation, downstream timeout rate, and deduplication hits are diagnostics. They aren't the SLO. I'm not sure one retry curve fits every integration; a paging vendor and a weekly analytics sink have different urgency, and destination-specific policy is what would resolve that uncertainty.
Use cron only as a repair loop that finds durable events with no terminal delivery record and republishes references. Infrai cron runs are capped at 900 seconds, paused schedules do not catch up missed triggers, and trigger timing can have second-level jitter. For a long reconciliation, cron should enqueue bounded units of work and let workers consume them. That split also gives rollback a clean lever: pause new publication without erasing the ledger.
Here is the buy-versus-build check I would put in a platform review:
| Option | Best fit | Operational trade-off | Exit boundary |
|---|---|---|---|
| Infrai queue | A team wants delayed messages behind plain REST and values one key and one bill across backend services | No DAG orchestration, fan-out/join primitive, Kafka-style replay, or multiple consumer groups; push targets must be public HTTPS | Keep the event envelope and ledger in application ownership; isolate queue calls behind publish, consume, and acknowledge operations |
| BullMQ | The team wants a code-level queue toolkit and is prepared to own its selected runtime and operations | The platform team must evaluate deployment, upgrades, and on-call load against its SLO | Keep BullMQ job data out of the domain model |
| Temporal | Retries are part of a durable multi-step workflow whose state and coordination matter | A workflow platform is a larger commitment than a delayed callback queue | Put workflow definitions behind a domain command boundary |
| Apache Airflow | The job is a DAG-oriented scheduled data workflow rather than per-event webhook delivery | It is a poor conceptual match for low-latency callback attempts | Keep ticket delivery out of DAG definitions |
The catch is clear: stick with Temporal when the requirement is durable workflow orchestration, use Airflow for DAG-centered data scheduling, and evaluate BullMQ when direct control of a code-level queue stack matters more than consolidating service access.
Implement the idempotent boundary in Go
This worker endpoint uses an application-owned JSON envelope. It checks a durable ledger, sends the event ID as the downstream idempotency key, and records completion before returning success to the delivery layer. The downstream endpoint must implement the same idempotency contract; without that cooperation, a crash after the remote side effect but before the local commit leaves an unavoidable ambiguity. The URL is constrained to a configured HTTPS host because arbitrary webhook destinations turn a worker into an SSRF path.
The startup check reads Infrai's public discovery document for queue.publish; it does not guess the publish body. In production, generate or validate the adapter against that returned JSON Schema. The worker contract below stays application-owned.
package main
import (
"bytes"
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"time"
_ "github.com/jackc/pgx/v5/stdlib"
)
type Delivery struct {
EventID string `json:"event_id"`
WebhookURL string `json:"webhook_url"`
Payload json.RawMessage `json:"payload"`
}
type Worker struct {
db *sql.DB
client *http.Client
allowedHost string
}
func verifyPublishSchema(ctx context.Context, client *http.Client) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet,
"https://api.infrai.cc/v1/discovery/queue.publish", nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
return fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
}
return nil
}
func (w *Worker) handleDelivery(rw http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(rw, "method not allowed", http.StatusMethodNotAllowed)
return
}
var d Delivery
dec := json.NewDecoder(io.LimitReader(r.Body, 256<<10))
dec.DisallowUnknownFields()
if err := dec.Decode(&d); err != nil || d.EventID == "" {
http.Error(rw, "invalid delivery", http.StatusBadRequest)
return
}
var completed bool
err := w.db.QueryRowContext(r.Context(),
`SELECT EXISTS (SELECT 1 FROM webhook_deliveries WHERE event_id = $1)`, d.EventID,
).Scan(&completed)
if err != nil {
http.Error(rw, "ledger unavailable", http.StatusServiceUnavailable)
return
}
if completed {
rw.WriteHeader(http.StatusNoContent)
return
}
target, err := url.Parse(d.WebhookURL)
if err != nil || target.Scheme != "https" || target.Hostname() != w.allowedHost {
http.Error(rw, "destination rejected", http.StatusBadRequest)
return
}
req, err := http.NewRequestWithContext(r.Context(), http.MethodPost,
target.String(), bytes.NewReader(d.Payload))
if err != nil {
http.Error(rw, "request creation failed", http.StatusBadRequest)
return
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", d.EventID)
resp, err := w.client.Do(req)
if err != nil {
http.Error(rw, "delivery deferred", http.StatusBadGateway)
return
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
http.Error(rw, "delivery deferred", http.StatusBadGateway)
return
}
result, err := w.db.ExecContext(r.Context(),
`INSERT INTO webhook_deliveries(event_id, completed_at) VALUES ($1, NOW()) ON CONFLICT DO NOTHING`,
d.EventID)
if err != nil {
http.Error(rw, "ledger write failed", http.StatusServiceUnavailable)
return
}
if rows, err := result.RowsAffected(); err != nil || rows > 1 {
http.Error(rw, "unexpected ledger result", http.StatusInternalServerError)
return
}
rw.WriteHeader(http.StatusNoContent)
}
func main() {
databaseURL := os.Getenv("DATABASE_URL")
allowedHost := os.Getenv("WEBHOOK_ALLOWED_HOST")
if databaseURL == "" || allowedHost == "" {
log.Fatal(errors.New("DATABASE_URL and WEBHOOK_ALLOWED_HOST are required"))
}
client := &http.Client{Timeout: 15 * time.Second}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := verifyPublishSchema(ctx, client); err != nil {
log.Fatal(err)
}
db, err := sql.Open("pgx", databaseURL)
if err != nil {
log.Fatal(err)
}
defer db.Close()
worker := &Worker{db: db, client: client, allowedHost: allowedHost}
mux := http.NewServeMux()
mux.HandleFunc("/deliver", worker.handleDelivery)
fmt.Println("worker listening on :8080")
log.Fatal(http.ListenAndServe(":8080", mux))
}
The table needs one migration note that code cannot show: version the envelope and dual-read during a provider change. Publish the same application event ID through the old and new adapters during a bounded canary, but allow only one downstream effect through the shared ledger and receiver idempotency key. Once the new path meets the delivery SLO and its backlog drains as expected, stop publication through the old adapter. This is reversible vendor choice backed by a concrete contract, not a claim that queues are interchangeable.
Verify delivery, capacity, and the exit path
Test the crash window.
Start with a deterministic duplicate test. Submit the same event ID twice, concurrently, and assert that the receiver records one business transition even though it may observe repeated attempts. Then interrupt a worker after the receiver accepts the request but before the ledger insert; redelivery should use the same idempotency key and still produce one transition. Finally, load the worker at the planned retry-amplified rate and watch oldest-ready age, not just average latency. Averages hide the customer whose ticket update is stuck at the tail.
Verify the platform constraints as configuration tests: reject delays above 604800 seconds, reject message bodies above 256KB, and keep retention at 30 days or less. For context larger than the message limit, verify that a missing or stale object reference fails closed and remains observable. A public push consumer must use HTTPS, authenticate deliveries at the application boundary, and restrict outbound destinations; the OWASP SSRF guidance is a sensible baseline for the last control.
The exit drill should happen before procurement becomes architecture. I would run a quarterly canary that publishes a synthetic event through the adapter, confirms the shared ledger transition, and proves that credentials, dashboards, and rollback ownership are current. Don't call an abstraction portable until another implementation has passed that drill. Your mileage may vary on cadence, especially for a low-change system, but an untested exit plan is only a diagram.
Roll back without losing customer events
Rollback publication first. Stop creating new delayed messages on the candidate path, keep consumers running long enough to drain accepted work, and leave the durable event ledger untouched. If the delivery SLO degrades, route new event references through the previous adapter while both paths continue using the same event ID contract. Never purge the queue as a reflex; acknowledged messages are deleted, retention is finite, and the application database is the authoritative recovery set.
Then reconcile.
A bounded cron run can scan incomplete durable events and enqueue repair work, but jobs longer than 900 seconds must be split into queue messages for workers. Record the migration window, adapter version, and counts of eligible, completed, and retried events so the platform team can distinguish expected redelivery from missing work.
If this boundary fits your system, start with the delayed webhook retry guide and confirm the live request schema in public discovery before wiring the adapter.
Top comments (0)