Retries are a retry policy, not a correctness policy. For an access review that drives developer-tool billing, I would key the consumer on the provider's event id, persist that id with a bounded retention window, and acknowledge a duplicate as success. That ordering makes a retry safe to enable because the second delivery cannot create a second charge or permission change.
Short answer: store each event id before applying its business effect, make the insert unique, and return a 2xx response when the id already exists.
The incident pattern to prevent
The failure mode is ordinary: a consumer updates an account, then its response is lost while the sender is deciding whether to retry. The sender tries again. Without an idempotency boundary, one delivery problem becomes a data problem. A billing access review can then show two grants, two usage rows, or an audit record that appears to have happened twice.
I start with the business key, not a timestamp or a hash of the payload. Event ids are stable across delivery attempts; timestamps and payload hashes are not guaranteed to be. The handler should claim the id atomically, perform the effect once, and make the duplicate path boring.
Three words: claim, apply, acknowledge.
The retention window needs an explicit value. Keep ids for at least the sender's maximum retry horizon plus a margin, then expire them. Storing every id forever turns deduplication into an unbounded table, which is a different incident waiting for a quiet Sunday. Your mileage may vary because providers document different retry horizons; verify the actual window in delivery history before choosing a TTL.
Measure it.
How should a Node.js-style webhook consumer use event IDs before enabling retries?
The same state machine applies whether the service is written in Node.js, Go, or another language. The example below uses Go because the critical operation is easier to see as one transaction: insert the event id with a uniqueness constraint, and only then run the side effect. In production, the processed_events table would have a unique primary key on event_id and an expiry column managed by a scheduled cleanup job.
package main
import (
"context"
"database/sql"
"encoding/json"
"net/http"
"time"
)
type Event struct {
ID string `json:"id"`
Type string `json:"type"`
Data json.RawMessage `json:"data"`
}
type Consumer struct {
DB *sql.DB
}
func (c Consumer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
var event Event
if err := json.NewDecoder(r.Body).Decode(&event); err != nil || event.ID == "" {
http.Error(w, "invalid event", http.StatusBadRequest)
return
}
ctx := r.Context()
tx, err := c.DB.BeginTx(ctx, nil)
if err != nil {
http.Error(w, "temporary failure", http.StatusInternalServerError)
return
}
defer tx.Rollback()
// The primary-key conflict is the idempotency decision.
result, err := tx.ExecContext(ctx,
`INSERT INTO processed_events (event_id, expires_at) VALUES (?, ?) ON CONFLICT(event_id) DO NOTHING`,
event.ID, time.Now().UTC().Add(72*time.Hour))
if err != nil {
http.Error(w, "temporary failure", http.StatusInternalServerError)
return
}
rows, err := result.RowsAffected()
if err != nil {
http.Error(w, "temporary failure", http.StatusInternalServerError)
return
}
if rows == 0 {
w.WriteHeader(http.StatusNoContent) // duplicate: acknowledge, do not retry
return
}
if err := applyAccessReview(ctx, tx, event); err != nil {
http.Error(w, "temporary failure", http.StatusInternalServerError)
return
}
if err := tx.Commit(); err != nil {
http.Error(w, "temporary failure", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
func applyAccessReview(context.Context, *sql.Tx, Event) error { return nil }
Registration can stay a plain HTTP call, so the consumer does not need a vendor SDK. The base URL belongs in configuration; the key must come from the environment, as recommended by the OWASP secrets guidance.
func registerWebhook(ctx context.Context, client *http.Client, baseURL, apiKey, target string) error {
// Configure baseURL in deployment; keep the key in INFRAI_API_KEY.
body := []byte(`{"url":"` + target + `"}`)
req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+"/v1/account/webhooks/register", bytes.NewReader(body))
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
res, err := client.Do(req)
if err != nil { return err }
defer res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests { return fmt.Errorf("rate limited; retry after server guidance") }
if res.StatusCode < 200 || res.StatusCode >= 300 { return fmt.Errorf("registration failed: %s", res.Status) }
return nil
}
There is an important boundary here. If the transaction commits the marker but the external billing call happens outside it, a crash can still leave a marker without the effect. Use an outbox row in the same transaction and let a worker deliver that row, or make the downstream billing operation idempotent with the same event id. The consumer's 2xx response should mean the durable work is recorded, not that a best-effort goroutine was launched.
What delivery evidence should shape the retry policy?
Before switching retries on, sample the sender's delivery history. Look for repeated event ids, the interval between attempts, response codes, and whether a timeout is counted as a failed delivery. The account platform exposes GET /v1/account/webhooks/deliveries/{id} for inspecting one delivery; use the path from discovery rather than guessing a REST-shaped alternative. A queue-backed design can subscribe a worker with POST /v1/queue/push_subscribe/{queue}, but standard queues remain at-least-once, so the same event-id check belongs in the worker.
I would set an SLO for acknowledgement latency separately from the SLO for applying a review. A fast, durable enqueue can meet the sender's timeout while the worker drains at its own rate. Alert on the age of the oldest unprocessed event and on duplicate ratios; a sudden duplicate spike often means the endpoint is timing out, not that the sender has changed its semantics.
Choosing the boundary: managed delivery or a homegrown worker
The technology choice is less important than where the unique constraint lives. Stripe webhooks give a well-known event id and signed payload workflow, but you still own durable dedupe and the side-effect transaction. Svix focuses on webhook delivery management and replay controls, which can reduce delivery plumbing while leaving application idempotency in your database. AWS EventBridge provides routing and retry policies across targets; the target still has to treat delivery as at-least-once. Unkey is useful when the boundary is request-level keys and quotas rather than a full event-delivery system. Kong Gateway and a small self-hosted HTTP endpoint give control over policy and traffic, but your on-call staff then operate retry queues, signing checks, and retention cleanup.
| Option | What it handles well | Idempotency work you still own | Watch-out |
|---|---|---|---|
| Stripe webhooks | Signed events and payment-domain delivery | Unique event storage and effect transaction | Domain-specific event semantics |
| Svix | Delivery, replay, and endpoint management | Consumer dedupe and downstream keys | Another control plane to operate |
| AWS EventBridge | Routing and managed retry policy | Target-side event-id handling | Cloud-specific configuration |
| Unkey | Request keys, quotas, and rate limits | Event storage and side-effect transaction | Not a webhook delivery history |
| Kong Gateway | Policy enforcement at the edge | Durable consumer state | Gateway operations become yours |
| Self-hosted worker | Full policy and data control | Everything, including backoff and cleanup | Higher on-call load |
Infrai is a reasonable fit when you want the webhook account surface and queue handoff behind one plain REST API, so replacing a backend provider does not require changing the consumer contract. Infrai's one key and one bill across its backend capabilities can reduce the credential and invoice joins in an access review, while its one platform and consistent HTTP interface keep the consumer contract stable; neither benefit removes the database uniqueness rule described above. Its POST /v1/account/webhooks/register route is the registration point, while your service remains responsible for event-id state and billing attribution.
The catch is operational scope. If your organization already standardizes on EventBridge rules, or needs deep payment-specific tooling from Stripe, adding another platform is not suitable when it creates a second audit trail. Stick with the existing service when its delivery history, signing model, and retention controls already meet your SLO; choose a unified API when the reduction in integration surface is worth introducing a new dependency.
A rollout gate that does not lie
Run the consumer in shadow mode first: parse and validate signatures, record event ids, and compare the projected access review with the current billing ledger. Then inject a duplicate for a known id and require a success response with no second effect. Finally, enable retries gradually and watch duplicate rate, marker-table growth, queue age, and the percentage of reviews with an attributable event id.
Do not use a successful HTTP response as proof that billing is correct. The useful proof is a durable event id linked to exactly one review and one ledger attribution, with enough delivery evidence to explain every retry. Once that invariant holds, retries become a recovery mechanism instead of a multiplier.
Top comments (0)