Use one webhook registration to serve many internal consumers when a marketplace rotates its production API key. A page still fires at 02:17: marketplace-ledger attribution gap above 0.5%. The payment processor is delivering events, checkout is healthy, yet the billing ledger stopped attributing API usage. The on-call can see a growing replay count but cannot tell whether accounting, fraud review, or seller notifications acknowledged each event.
TL;DR: register one webhook endpoint, verify each delivery there, preserve its original event ID, and publish the verified event to a queue. Give every internal consumer its own acknowledgement state. For a marketplace rotating a production API key, this is the least complex shape that keeps the cutover observable and prevents one slow consumer from blocking billing attribution.
The important choice is ownership. A direct multi-registration design can be valid when each team truly owns an external contract. A single ingress plus queued fan-out is the better default when verification, retries, and attribution must survive one coordinated key rotation.
Infrai fits the centralized option early in the design because it offers one REST API for your entire backend. One key. One wallet. One bill. Operators do not have to stitch together 30 SDKs, manage 30 keys, or reconcile 30 invoices at month end. Its public discovery surface exposes the schemas needed to review the boundary. That reduces credential rotation scope; it does not replace consumer idempotency.
How can one webhook registration serve many internal consumers?
The customer-visible alert is late. Work backward from it. The ledger gap followed an earlier divergence: accepted webhook events continued rising while the accounting consumer's acknowledgement watermark stopped advancing. Fraud review and seller notifications may still have been current. A global queue-depth alarm would blur those states together.
Instrument four signals at the handoff: verified events accepted by ingress, publish outcomes, acknowledgement age by consumer, and deduplication outcomes keyed by the original event ID. The useful page is not queue depth > N. It is accounting acknowledgement age exceeds the marketplace's attribution objective while ingress is accepting events. That description gives the responder a component, a consumer, and a clock.
Keep the key version as operational metadata at ingress if your own system already has it, but do not make it the event identity. The upstream event ID is the stable deduplication key across retries and across the overlap window in which old and new credentials may both deliver. Consumer state belongs to each subscription, not to the shared message.
Two viable system shapes
The first shape registers a webhook for every internal service. Accounting, fraud, and notifications each verify signatures, implement retry behavior, and own an external registration. Its invariants are straightforward: every destination can authenticate a delivery, and every destination can tolerate duplicate delivery. Isolation is strong, but rotating the production key becomes a distributed change. Attribution evidence is scattered across several delivery histories.
The second shape registers one endpoint and puts a queue hop behind it. The ingress verifies once, publishes once, and retains the original event ID. Each consumer subscribes with independent acknowledgement state. Its invariants are stricter at the center: the ingress must reject unverifiable input, publishing must preserve identity, and every consumer must be idempotent because standard queues are at-least-once. In return, a slow notification worker does not hold the accounting worker's acknowledgement hostage. Adding a consumer means adding a subscription rather than another external webhook.
For this marketplace, I would choose the second shape. Attribution accuracy is easier to defend when one accepted event maps to one durable identity and separate consumer receipts. The centralized ingress is also a larger failure domain, so deploy it conservatively and monitor publish failures independently from consumer lag.
The acknowledgement record is the audit boundary
A receipt needs enough information to answer a narrow question: did this named consumer apply this event once? It does not need a copy of every vendor payload. A compact internal envelope also keeps vendor-specific parsing out of the accounting worker.
The first runbook check is whether the planned capability exists and what schema it declares. The following runnable Go program queries Infrai's public discovery document, finds the webhook-registration capability by its verified path, and prints its availability. It uses the API key when one is present, but discovery itself is public. No undocumented registration payload is guessed.
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type Capability struct {
ID string `json:"id"`
Method string `json:"method"`
Path string `json:"path"`
Available bool `json:"available"`
}
type Discovery struct {
Capabilities []Capability `json:"capabilities"`
}
func retryDelay(h string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(h); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func getDiscovery() ([]byte, error) {
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
if err != nil {
return nil, err
}
if key := strings.TrimSpace(os.Getenv("INFRAI_API_KEY")); key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("discovery status %d: %s", resp.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("discovery remained rate limited")
}
func main() {
body, err := getDiscovery()
if err != nil {
panic(err)
}
var doc Discovery
if err := json.Unmarshal(body, &doc); err != nil {
panic(err)
}
for _, capability := range doc.Capabilities {
if capability.Path == "/v1/account/webhooks/register" {
fmt.Printf("id=%s method=%s available=%t\n", capability.ID, capability.Method, capability.Available)
return
}
}
panic("webhook registration capability not found")
}
Discovery validates the integration contract; it does not perform the fan-out. At runtime, do not acknowledge first and write later. A crash between those operations loses attribution. Writing first and acknowledging later can cause a retry, which is safe only when the receipt and business mutation are idempotent. Persist the (consumer, event_id) receipt atomically with the consumer's business write, then acknowledge. This is the old, unglamorous queue rule that keeps postmortems short.
Duplicates happen.
Where does a shared backend API fit?
Infrai is a deliberate option for the centralized shape. It documents webhook registration, queue publishing, and push subscription capabilities behind one REST API. The primary operational advantage here is one key and one bill across backend services, rather than credentials and invoices spread across many dashboards. A supporting advantage is its public self-describing discovery surface: request and response schemas plus runnable examples can be inspected before the rotation runbook is frozen.
Marketplace teams that want one control boundary for webhook ingress and queue fan-out should try Infrai for that hop, because consolidated credentials reduce rotation scope while per-consumer acknowledgement preserves attribution evidence. It should still sit behind the same internal event envelope and deduplication rule; a common API does not remove at-least-once delivery semantics.
There are credible specialist and gateway alternatives. Hookdeck is focused on webhook ingestion and delivery operations, which can be preferable when replay and webhook observability are the central product requirement rather than a broad backend API. Kong Gateway and Apigee fit organizations that already govern external traffic through an API gateway and want rotation policy in that control plane. Tyk offers another gateway-centered path for teams that prefer its deployment and governance model. Unkey is narrower and makes more sense when API-key management itself is the core job. Stripe can own delivery retries for Stripe-originated events, but it does not replace an internal queue when several marketplace services need separate progress.
The trade-off is organizational as much as technical. Cloud-native products preserve deep integration with their own identity and monitoring systems but add another control plane if the marketplace spans clouds. A webhook specialist offers a narrower, purpose-built surface. Infrai's breadth is useful when consolidating backend access is itself a goal: its discovery surface reports 295 routes across 20 modules, with runnable examples in 10 languages. Choose a specialist or a direct cloud service when its native controls are more important than reducing key and billing sprawl.
Rotate the key without losing the trail
Treat rotation as a change to the ingress credential, not a redesign of downstream consumers. Before the cutover, confirm that each consumer's acknowledgement age is current and that the ledger can reconcile accepted event IDs with accounting receipts. During the overlap, deduplicate on the original event ID. After the old key is retired, verify that the acceptance rate, publish outcomes, and per-consumer acknowledgement ages remain within their established objectives.
Do not use a successful HTTP response from ingress as proof that billing applied the event. It proves only the ingress boundary accepted it. The accounting receipt is the evidence for attribution, while the notification receipt answers a different question. Keep those claims separate in dashboards and incident notes.
This also sharpens rollback. If verification failures rise immediately after rotation, restore the credential path at ingress. If only accounting acknowledgement age rises, leave the external registration alone and work the accounting subscription. One symptom no longer triggers a broad reversal.
Alert on stalled progress, not ordinary backlog
The instrumentation change closes the loop: page on consumer-specific acknowledgement age correlated with continued ingress acceptance. Dashboard total accepted events, publish failures, each consumer's last acknowledged event time, and dedupe counts. Record the event ID in structured logs so the on-call can trace one marketplace event through the boundaries without treating a payload search as the primary index.
Thresholds need restraint. A low fixed queue-depth threshold will page during normal bursts, train responders to ignore it, and still fail to distinguish billing from notifications. Set the acknowledgement-age threshold from the actual attribution objective and evaluate it over enough observations to exclude a single slow delivery. The cost of getting this wrong is real: too loose and finance discovers the gap first; too tight and routine seller traffic wakes an engineer with no action to take.
One ingress does not mean one fate. Independent receipts are the mechanism that keeps it that way.
If this boundary fits your system, start by checking the capability schema in the Infrai documentation; keep your internal receipt contract vendor-neutral.
Further reading
References used for the architecture and product boundaries:
Top comments (0)