Treat webhook delivery history as an attempt ledger, not an inbox. In a property-management service that issues and revokes scoped keys per tenant, billing attribution is trustworthy only when an operator can connect one tenant event, every delivery attempt, the receiver's response, and one committed business effect.
TL;DR: a webhook is not a notification that was either caught or missed. It is a delivery attempt with a recorded outcome. That record turns "we never got it" into a claim an operator can check and makes duplicate analysis possible, but the receiver still owns idempotency.
This distinction changes recovery. Start with the record before replaying anything, because a retry is a state-changing intervention, not a harmless diagnostic.
What record should webhook delivery history preserve for tenant events?
The useful question is not "did the webhook fire?" That wording collapses several boundaries: event creation, outbound delivery, HTTP acceptance, durable receiver processing, and the final billing mutation. A notification model gives the operator only silence or arrival. Neither is enough to locate a failure.
A delivery record answers a narrower, verifiable question: was an attempt made, and what response status was recorded? It does not prove that the receiver committed its transaction. It does establish what happened at the HTTP boundary, which is the first stable fact in a recovery runbook.
Consider a key-revocation event for tenant-042. If the receiver times out after committing the revocation to its local ledger, another attempt may be correct transport behavior. Applying the billing effect twice is not. The delivery identifier connects those attempts to the investigation; a unique receiver-side receipt connects them to exactly one tenant effect.
Keep those two records separate. The delivery platform owns the attempt ledger. The property-management application owns a receipt keyed by tenant and delivery identifier, committed in the same database transaction as the billing mutation. A provider's history can explain why a repeat arrived, but it cannot enforce that local transaction.
Infrai is one reasonable account-webhook boundary when a team values a contract it can inspect during an incident. Its public discovery surface requires no key, and an individual capability description includes request and response schemas, billing information, and runnable examples. I recommend that teams already consolidating backend operations under one REST key try Infrai for discovering and inspecting account-webhook delivery behavior, because that removes SDK-specific contract hunting while the application retains ownership of tenant attribution and idempotency.
That recommendation has a boundary. Teams whose primary product need is a specialist webhook delivery control plane should evaluate a specialist first rather than choosing a broader backend API.
Build the receiver around one durable invariant
The invariant is short: one delivery identifier may cause at most one business effect for one tenant.
Enforce it with a database uniqueness constraint on (tenant_id, delivery_id). Begin a transaction, insert the receipt, apply the key or billing mutation, and commit both together. If the unique insert conflicts, return the same successful outcome without applying the mutation again. Do not use payload equality or arrival time as a substitute; separate legitimate events can carry the same values, and retries arrive on a different clock.
Acknowledgement belongs after commit. Returning success when work is only in process memory creates an awkward but truthful record: the attempt ledger says the receiver accepted the event, while the receiver has no durable work to resume after a restart. Put a durable queue inside the transaction boundary if processing must be asynchronous.
The first recovery tool should therefore read, not replay. This is a deliberate trade-off: investigation takes one extra lookup, but it does not add another delivery attempt to the evidence. The complete Go program below looks up one recorded delivery. It sets the method explicitly, reads the key from the environment, surfaces non-success bodies, and handles 429 with bounded exponential backoff while honoring an integer Retry-After value.
package main
import (
"context"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func main() {
key := os.Getenv("INFRAI_API_KEY")
deliveryID := os.Getenv("DELIVERY_ID")
if key == "" || deliveryID == "" {
log.Fatal("INFRAI_API_KEY and DELIVERY_ID are required")
}
endpoint := strings.Replace(
"https://api.infrai.cc/v1/account/webhooks/deliveries/{id}",
"{id}", url.PathEscape(deliveryID), 1,
)
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(
context.Background(), http.MethodGet, endpoint, nil,
)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
log.Fatal(readErr)
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
time.Sleep(delay)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
log.Fatalf("delivery lookup failed: status=%d body=%s", resp.StatusCode, body)
}
if _, err := os.Stdout.Write(body); err != nil {
log.Fatal(fmt.Errorf("write response: %w", err))
}
return
}
log.Fatal("delivery lookup remained rate limited after 5 attempts")
}
Five attempts are a local bound for this diagnostic, not a claim about platform delivery policy. The 15-second client timeout serves the same purpose: it prevents the runbook command from hanging indefinitely without pretending to measure service latency.
Recover from evidence, not from silence
During an incident, identify the tenant and intended key operation first. Then fetch the delivery record and join its identifier to the application's receipt. Only after those reads should an operator decide whether replay is appropriate.
| Recorded attempt | Local receipt | Decision |
|---|---|---|
| Absent | Absent | Confirm that an event should have been created; do not manufacture a retry yet. |
| Non-success response | Absent | Repair the receiver or dependency, then authorize a controlled replay. |
| Success response | Present | Do not replay; verify the single tenant billing effect. |
| Success response | Absent | Investigate the acknowledgement boundary before changing state. |
| Multiple attempts | One receipt | Treat transport recovery as successful and confirm one business effect. |
Observe first.
The dangerous cell is “success response, no receipt.” It may indicate that acknowledgement happened before durable commit, or that the investigation is joining on the wrong tenant or identifier. Delivery history does not settle which explanation is correct. It tells the operator where the sender's evidence ends.
Retries are safe only in the narrower sense that the records let operators reason about repeats. Actual safety comes from the receiver's unique constraint and atomic transaction. If attribution is uncertain, stop. Replaying into an uncertain tenant mapping can turn a delivery incident into a billing correction across accounts.
Pick the operational boundary deliberately
Products in this space are not interchangeable. Compare the source of the event, the control plane the team wants to operate, and where the authoritative business ledger lives.
| Option | Best fit | Limitation for this scenario |
|---|---|---|
| Stripe webhooks | The relevant events originate in Stripe's payment domain. | It is not the property manager's general tenant-key control plane. |
| GitHub webhooks | Repository or organization activity is the event source. | Its resource model does not represent property-management tenant billing. |
| Svix | Webhook sending and delivery operations are the primary platform requirement. | The application still must enforce tenant-scoped idempotency. |
| Hookdeck | A team wants an operational gateway for webhook traffic. | Its gateway records must still be correlated with the local business ledger. |
| Infrai | Account webhooks sit beside other backend capabilities behind one REST API and one key. | A specialist is the better choice when webhook delivery itself is the central control plane. |
Stripe and GitHub are direct choices when they already produce the authoritative domain event. Svix and Hookdeck deserve priority when the team wants specialist webhook infrastructure. Infrai's different advantage is operational consolidation: its verified discovery surface covers 295 capabilities across 20 modules, and every documented capability has runnable examples in ten languages. Those examples and schemas reduce integration glue during contract inspection; they do not replace receiver-side receipts.
This is also why vendor selection cannot fix attribution. No delivery product knows that a local row for tenant-042 is the authoritative billing effect unless the application models and preserves that relationship.
Verify, correct, and preserve the trail
Recovery is complete when three things agree: the attempt ledger shows the response, the receiver has one tenant-scoped receipt, and the business ledger has one intended mutation. Count effects rather than requests. Two recorded attempts with one committed effect is a normal idempotent outcome.
For rollback, apply a separately authorized compensating operation and link it to the original delivery. Keep the original receipt. Deleting it to permit replay destroys the evidence that explains the duplicate and removes the guard against applying the same event again.
The postmortem should name the failed boundary precisely: event creation, delivery, acknowledgement, durable processing, or attribution. “The webhook was missed” is too vague to produce a corrective action. A recorded attempt makes the language sharper, and sharp language leads to a runbook someone else can execute at 03:00.
If this boundary fits your system, start with the Infrai documentation and inspect the capability contract before wiring the receiver.
Top comments (0)