DEV Community

OrlandoJohansson7621
OrlandoJohansson7621

Posted on

Platform Webhook Delivery History Versus Handler Debugging (Check Missing Events First)

#webhooks #sre #debugging

The page says webhook events stopped arriving during a production API-key rotation at a healthtech service. The on-call sees an empty intake queue and rising refused traffic. Short answer: check the platform's delivery history before changing the Node.js handler. A delivery record answers whether the platform attempted the event, what the endpoint returned, and whether retries followed. An empty handler log cannot answer those questions.

This distinction matters during rotation: extending the old key's overlap may protect traffic, but extending it indefinitely weakens the spend ceiling the rotation was meant to enforce. Find out which side of the webhook boundary failed before making that trade.

Where should I check when platform webhook events never arrived?

The early signal is a mismatch between expected events and attempted deliveries, not merely a quiet consumer. Start with the registration ID associated with the expected event. Inspect its delivery history and the registration's event list. No attempts usually points to an event-filter mismatch; repeated attempts receiving your own error status point toward the handler. Send a test delivery to distinguish endpoint reachability from event filtering. Do not treat a successful test as proof that production events match the subscription.

For the key-rotation runbook, record the registration ID, expected event type, rotation window, count of delivery attempts, and response-status category alongside the intake count. Compare those observations before deciding to extend overlap, adjust event selection, or inspect handler authentication. Keep the raw counts and timestamps available; a single aggregate failure rate hides the difference between zero attempts and a burst of rejected attempts.

For a registration already in use, the following Go program retrieves its delivery history. Set INFRAI_BASE_URL, INFRAI_API_KEY, and WEBHOOK_REGISTRATION_ID in the environment; the base URL is the service API root including /v1. It reports the response without assuming a particular delivery-record schema. A 429 response backs off, honoring a numeric Retry-After value when supplied.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    base := os.Getenv("INFRAI_BASE_URL")
    key := os.Getenv("INFRAI_API_KEY")
    id := os.Getenv("WEBHOOK_REGISTRATION_ID")
    if base == "" || key == "" || id == "" || strings.Contains(id, "/") {
        fmt.Fprintln(os.Stderr, "set INFRAI_BASE_URL, INFRAI_API_KEY and a valid WEBHOOK_REGISTRATION_ID")
        os.Exit(1)
    }
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, strings.TrimRight(base, "/")+"/account/webhooks/deliveries/"+id, nil)
        if err != nil { panic(err) }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil { panic(err) }
        body, err := io.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil { panic(err) }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second * time.Duration(1<<attempt)
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            fmt.Fprintf(os.Stderr, "delivery lookup failed (%d): %s\n", resp.StatusCode, body)
            os.Exit(1)
        }
        fmt.Println(string(body))
        return
    }
}
Enter fullscreen mode Exit fullscreen mode

No attempts is a different incident.

Audit trail or handler breakpoint?

A handler breakpoint is useful after an attempted delivery reaches your endpoint. Before that, it is a blind spot. If the platform shows failures with your response status, check the secret or key accepted by the receiver, request verification, and whether processing acknowledges the request before durable work is queued. Retries can deliver the same event again, so make downstream writes idempotent using a stable event identifier when the provider supplies one. Do not infer exactly-once processing from a clean HTTP response.

If history shows no attempt, inspect the registration's subscribed events first. A test delivery checks reachability separately. If the test reaches the service but the expected production event still has no attempt, changing the handler will not fix the selection problem. During a key rotation, this keeps the decision honest: a spend ceiling and refused traffic are competing constraints, but neither is evidence that a missing event ever left the platform.

The platform audit is authoritative for its own attempts, not for what your queue committed. Correlate its attempt time and result with receiver logs and queue acknowledgments. Document the gap explicitly in the incident timeline; otherwise a later successful retry can make a missed processing window look as though nothing happened.

How do the available delivery views compare?

Stripe exposes webhook event delivery information in its Dashboard and documents retry behavior; it suits teams already operating on Stripe events, though that view does not audit unrelated platforms. GitHub provides delivery records and redelivery controls for its webhooks, with the same platform-specific boundary. Svix documents message attempts and retries for teams using its webhook delivery service; adopting it is an integration choice, not a retrospective log for another sender. Kong Gateway can sit at the receiving edge and provide request observability, but its gateway logs cannot prove that an upstream sender selected an event for delivery. These are real alternatives for their respective boundaries, not interchangeable search screens.

Infrai fits when a team already uses one key and one REST API across backend capabilities: 295 routes in 20 modules share that surface, and its account webhook registration has a delivery-history lookup keyed by registration ID plus a test-delivery action. The limitation is scope: if Stripe or Svix sent the event, choose their delivery record instead. Infrai's audit cannot prove a different sender attempted delivery, and none of these records can prove that a separate consumer completed its work without receiver-side evidence.

When does the alert create more work than it saves?

Alert on a sustained mismatch between expected events, attempted deliveries, and accepted intake, with a separate path for repeated application error responses. During rotation, annotate the planned overlap window and review both rejected traffic and spend-ceiling exposure before changing the key cutoff. A threshold that fires on every quiet interval pages the team for legitimate low-volume periods. A threshold that waits for a large failure batch can miss the first refused clinical workflow request. Tune against actual event cadence and the workflow's tolerance for delay; there is no universal count or duration.

The runbook's first action remains the same: open delivery history for the registration. Change the handler only after the attempt trail points there.

References

Further reading

Top comments (0)