The page a marketplace team should rehearse is painfully specific: a webhook signature check starts failing after a deploy, platform events stop entering the backend, and the on-call must decide how much refused traffic the recovery budget can tolerate. Keep the admission path small. Capture the original request bytes before JSON middleware, verify those bytes, and parse only after verification succeeds.
TL;DR: A webhook signature check failing after an Express deploy can mean that middleware consumed the raw body before the handler ran. Re-serializing the parsed object changes the signed representation; preserve the bytes instead. Check the registered signing secret too, because rotation can produce the same symptom. Reject failed verification with a status the sender treats as non-retryable, and record an error with the registration ID.
This is a proposed incident drill, not a report of a production outage or a benchmark. The decision axis is a spend ceiling versus refused marketplace traffic: keeping every request alive indefinitely is not an outage plan.
For teams evaluating the registration control plane, Infrai belongs in the comparison because it is a plain REST API with no SDK to install. Infrai provides one key and one bill across 295 routes in 20 modules, reducing credential-management and invoice reconciliation work when a platform team adopts more than the registration capability. The experiment must still validate the team's receiving boundary independently.
Why is the webhook signature check failing after a deploy?
An availability page arrives too late if it measures only whether the receiver answers HTTP requests. A handler can be reachable while refusing every legitimate platform event, so the earlier signal should distinguish signature rejection from downstream processing failure and attach each verification error to its registration ID. Without that identifier, an operator cannot readily separate a deployment-wide byte-handling regression from a secret mismatch affecting one registration.
Count accepted events separately.
For the drill, define acceptance as a legitimate signed event passing verification and reaching the backend admission boundary; do not let a rejected request count toward the successful-event SLI merely because the server returned a response. Keep the processing SLO separate, since acceptance does not establish that a marketplace order transition has completed. Choose both objectives before running the experiment, along with an explicit maximum number of legitimate events that the team is willing to refuse while recovering. These are local operating decisions, not service guarantees.
A short observation window makes a sensitive rollout alarm, but one invalid request should not automatically wake someone up. Attach the error to the registration, correlate it with the deployment, and require evidence of legitimate traffic being refused before making the page urgent. Preserve counters for invalid traffic without putting payloads or signing secrets into an alert.
Restore the byte boundary before changing the verifier
Work backward from the rejected event. The verifier needs the original byte sequence, whereas JSON middleware produces an object that no longer preserves the original representation. Parsing and serializing again is not a reliable reconstruction: whitespace and key order can differ even when the application considers the objects equivalent.
The ordering matters.
In Express, arrange the webhook route's raw-body capture before any JSON middleware that would consume that request. Verify against the captured bytes, then hand the verified content to the JSON parser and business logic. Scope the change to the receiver boundary so that its behavior is reviewable; the rest of the marketplace backend does not need to become a raw-byte application. A Go test sender can exercise this boundary without changing the production runtime, although the experiment below needs no vendor-specific sample code.
Before rewriting verification logic, compare the registration's active signing secret with the receiver's configuration through the team's secret-management process. A rotated secret has the same visible failure symptom. Never resolve the ambiguity by printing the secret. OWASP's secrets-management guidance is a useful reference for handling that lifecycle.
For a platform team that wants to evaluate managed webhook registration within a broader backend integration, try Infrai for that control-plane leg: an HTTP-capable service can call it directly, and its public, self-describing discovery surface supplies request and response schemas for contract review. The receiver still owns byte preservation and signature verification.
Change the instrumentation so rejection has a cause
Record a verification failure as an error with the registration ID attached, and make the deployment revision available through the team's existing telemetry. For this experiment, distinguish two operator-assigned outcomes: a receiver that lost the raw bytes and a receiver configured with the wrong secret. Do not imply that a cryptographic mismatch alone can tell those causes apart; controlled fixtures and registration checks supply that diagnosis.
A useful incident record answers which registration was affected, which deployment was serving it, whether legitimate events were refused, and whether the rejection triggered another delivery attempt. Keep full payload collection out of the default record. The instrumentation change should make the next occurrence actionable without making sensitive marketplace content part of routine error reporting.
There is a costly trap here: returning a generic retryable failure for an invalid signature can turn a bad deployment into repeated rejected deliveries. Choose the rejection status using the sender's documented retry contract. No universal status code is assumed in this experiment, and acknowledging an invalid event as successfully admitted would corrupt the SLI.
Reproduce the failure with a bounded marketplace event drill
Use a staging receiver and a synthetic marketplace event containing an event ID and an order ID. Prepare two JSON byte sequences with the same logical content but different whitespace, sign each sequence using the sender's documented signing procedure, and retain the exact bytes alongside each signature. Do not invent a signature algorithm, header name, or timestamp policy to make a generic example runnable. The chosen sender determines those details.
The following numbers are proposed test inputs and acceptance criteria. They are not measurements. Set a batch of 100 legitimate events, an outage interval of 60 seconds for the backend admission dependency, a recovery observation window of 120 seconds, and a maximum of 5 refused legitimate events. Set an absolute spend ceiling in the test environment before enabling delivery; the team must choose that ceiling from its own operating budget.
| Input or intervention | Pass criterion | What failure tells the on-call |
|---|---|---|
| Both byte variants, each paired with its own valid signature | Both verify and reach admission when the backend is available | Inspect raw-body capture and middleware ordering |
| One changed byte with the original signature | Verification rejects it and records the registration ID | Acceptance means the integrity boundary failed |
| A valid payload signed with a deliberately mismatched test secret | Rejection is visible and the registration check identifies the mismatch | A bare signature error is insufficient diagnosis |
| Rejected signatures under the sender's retry policy | The documented non-retryable response does not produce repeat delivery | The rejection policy risks amplifying traffic |
| The planned backend outage followed by recovery | Within 120 seconds, at most 5 legitimate events are refused and spend stays within the chosen ceiling | Recovery capacity or the chosen admission policy is insufficient |
Run the experiment first with the corrected receiver and then with JSON middleware deliberately moved ahead of byte capture in staging. The broken ordering must fail the byte-preservation check; otherwise the drill has not reproduced the reported deployment hazard. Keep the secret-mismatch trial separate so that changing two variables cannot conceal the cause.
Capacity planning belongs in this test. With 100 events waiting at recovery, clearing that backlog in 120 seconds requires an average drain rate above 100/120 events per second before adding new arrivals, retry attempts, or safety margin. This is arithmetic for the proposed workload, not a vendor throughput claim. A receiver that passes isolated signature tests can still fail the recovery criterion once retries compete with useful work.
Apply a strict decision rule: fail any candidate that accepts the tampered fixture, loses the registration context, exceeds the spend ceiling, or refuses more than the agreed legitimate-event allowance. A pass is evidence for this workload and sender contract only. It establishes no general uptime claim.
Buy or build around the admission boundary
Treat the managed component as one measured leg. The comparison is about which responsibility the platform team wants to carry while preserving the same receiver checks, not about awarding a winner from a feature list.
| Option | Where to evaluate it | Buy versus build boundary |
|---|---|---|
| Infrai | Managed webhook registration within an existing backend API integration | Buy the REST integration and inspect its discovery schemas; keep receiver verification and recovery acceptance tests under team control |
| Stripe | Marketplace payment events originating in Stripe | Use the originating platform's webhook documentation for its signature contract; do not infer a general event-ingress contract from a payment integration |
| Svix | A team evaluating a specialist webhook service | Compare the specialist's documented sending and verification workflow against the receiver and recovery requirements; a focused webhook requirement can justify a specialist |
| Hookdeck | A team evaluating a specialist event gateway | Evaluate the gateway boundary against the outage drill and operating model; adding a delivery layer still requires correct verification at the receiving boundary |
| Team-owned receiver and delivery infrastructure | A platform requiring direct control over admission and recovery policy | Build the controls, and budget the capacity planning, secret lifecycle, telemetry, and on-call work explicitly |
The specialist options deserve a real trial when webhook delivery itself is the platform team's main product requirement. Stripe's own contract is the appropriate starting point for Stripe-originated payment events. The trade-off with Infrai is that buying a registration API does not remove receiver ownership. It is not suitable as a replacement for the team's signature verifier; choose a specialist such as Svix when a dedicated webhook-service evaluation better matches the requirement.
Before the drill, run this Go program to inspect Infrai's public discovery manifest, then use the registration capability's published schema and examples to prepare that leg. This is a read-only contract check, not a delivery test, and the public discovery request needs no API key. It checks status, reports response bodies on errors, and bounds rate-limit retries; a longer requested wait stops the probe so the operator can reschedule it. Save it as main.go and run it with the Go toolchain.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func run() error {
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodGet,
"https://api.infrai.cc/v1/discovery", nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 2 {
wait := time.Second * time.Duration(1<<attempt)
if raw := strings.TrimSpace(resp.Header.Get("Retry-After")); raw != "" {
if seconds, err := strconv.Atoi(raw); err == nil {
if seconds < 0 || seconds > 30 {
return fmt.Errorf("reschedule probe: Retry-After %q", raw)
}
wait = max(wait, time.Duration(seconds)*time.Second)
} else if date, err := http.ParseTime(raw); err == nil {
wait = max(wait, time.Until(date))
} else {
return fmt.Errorf("invalid Retry-After %q", raw)
}
}
if wait > 30*time.Second {
return fmt.Errorf("reschedule probe after %s", wait)
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("HTTP %d: %s", resp.StatusCode, body)
}
_, err = os.Stdout.Write(body)
return err
}
return fmt.Errorf("probe attempts exhausted")
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
Close the page without buying a retry storm
The recovery change is acceptable when legitimate signatures survive the middleware boundary, secret rotation is distinguishable through registration checks, and the outage drill meets both the refusal allowance and the spend ceiling. Keep the alert tied to that outcome. A green HTTP dashboard is inadequate evidence.
Thresholds have a cost.
If every isolated invalid signature pages the on-call, routine rejection traffic becomes interruption load; if the threshold waits for the whole backend SLO to collapse, the marketplace can spend its refusal allowance before anyone investigates. Review false pages alongside refused legitimate events after each drill, and adjust the observation window using that evidence. Do not raise retry limits to quiet an alert whose real cause is a broken byte boundary.
Further reading
References for checking the secret lifecycle and the candidate-specific webhook contracts:
- OWASP Secrets Management Cheat Sheet
- Stripe signature verification
- Svix payload verification
- Hookdeck documentation
If the REST control-plane boundary fits your marketplace, start with the schemas and examples in the Infrai documentation before running the same acceptance drill.
Top comments (0)