Set the spend ceiling at each consumer, not at the shared webhook. Register one endpoint, verify each event there, preserve its original event ID when publishing it to a queue, and give every internal consumer its own acknowledgement state. That decision lets a slow analytics job fall behind without holding up billing or entitlement processing, while each workload can refuse new work at its own limit before the invoice arrives.
TL;DR: the safe default is one controlled intake and independently metered subscriptions. A shared acknowledgement couples unrelated workloads; separate webhook registrations duplicate verification and retry behavior. Neither tells the on-call engineer which consumer crossed its allowance.
For a B2B SaaS product, the uncomfortable trade-off is explicit: a hard ceiling can refuse traffic, while a soft ceiling can allow unbounded spend. Pick the refusal behavior per consumer, record it, and page on the boundary that actually fired. A dashboard showing total queue depth is not enough.
How can one webhook registration serve many internal consumers?
Imagine one customer event feeding three internal workloads: entitlements, invoice-projection, and usage-analytics. Entitlements is latency-sensitive. Invoice projection can tolerate a short lag. Analytics may be paused after reaching its daily processing allowance. If all three share one receipt, analytics can prevent acknowledgement and trigger delivery retries for work the other two already completed; if the receipt is acknowledged early, analytics can silently miss work.
That is the incident shape to avoid. The page should say that usage-analytics refused event evt_01892 because its admitted-work counter reached the configured ceiling, not merely that "the webhook queue is growing." The former has an owner and a decision. The latter is a graph asking someone at 3 a.m. to infer the system.
One registration keeps verification and retry handling at an ingress boundary you control. After verification, the ingress publishes an immutable message carrying the original event ID. Each subscription then owns its delivery attempts and acknowledgement. Adding a fourth workload means adding a subscription, not another public webhook.
This does not eliminate refusal. It makes refusal local and observable.
Draw the three boundaries before writing code
The design has three boundaries, and each answers a different operational question.
| Boundary | Responsibility | Useful page condition |
|---|---|---|
| Webhook intake | Verify once and accept or reject the external event | Valid events cannot enter the queue |
| Queue publication | Preserve the original event ID and make fan-out possible | Verified events cannot be published |
| Consumer subscription | Admit work against its ceiling, deduplicate, then acknowledge | One named consumer refuses or repeatedly fails work |
The acknowledgement belongs after a consumer's durable effect, not after receipt. Standard queues should be treated as at-least-once delivery, so every consumer needs an idempotency record keyed by the original event ID. Keep that record even when the consumer declines work because of its ceiling; otherwise a retry can repeatedly spend admission capacity or emit repeated refusal noise.
There is a policy decision hiding here. A consumer that reaches its ceiling can reject immediately, defer within the queue's permitted retention window, or route the event to a separately controlled review path. The right choice depends on business semantics, not queue mechanics: refusing an analytics refresh is different from refusing an entitlement change. Document which work is allowed to lag and which must page before deploying the subscription.
A minimal consumer with a hard admission ceiling
The following Go program is deliberately local: it models the consumer-side contract without inventing a vendor payload or signature scheme. It accepts an event ID, reserves one unit under a ceiling, performs an idempotent effect, and records the acknowledgement. In production, the queue adapter should pass the original webhook event ID into Handle; the durable store must replace the in-memory maps.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strings"
"sync"
"time"
)
var ErrCeilingReached = errors.New("consumer spend ceiling reached")
type Ledger struct {
mu sync.Mutex
limit int
admitted int
done map[string]bool
}
func NewLedger(limit int) *Ledger {
return &Ledger{limit: limit, done: make(map[string]bool)}
}
// Admit reserves capacity and deduplicates using the original webhook event ID.
func (l *Ledger) Admit(eventID string) (alreadyDone bool, err error) {
l.mu.Lock()
defer l.mu.Unlock()
if l.done[eventID] {
return true, nil
}
if l.admitted >= l.limit {
return false, ErrCeilingReached
}
l.admitted++
return false, nil
}
func (l *Ledger) Complete(eventID string) {
l.mu.Lock()
defer l.mu.Unlock()
l.done[eventID] = true
}
type Consumer struct {
name string
ledger *Ledger
}
type Capability struct {
Method string `json:"method"`
Path string `json:"path"`
}
type Discovery struct {
Capabilities []Capability `json:"capabilities"`
}
// Discover verifies route paths against the live manifest before integration.
func Discover(ctx context.Context) (Discovery, error) {
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
if baseURL == "" {
return Discovery{}, errors.New("INFRAI_BASE_URL is required")
}
var lastErr error
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/v1/discovery", nil)
if err != nil {
return Discovery{}, err
}
if key := os.Getenv("INFRAI_API_KEY"); key != "" {
req.Header.Set("Authorization", "Bearer "+key)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
lastErr = err
continue
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Discovery{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if retryAfter := resp.Header.Get("Retry-After"); retryAfter != "" {
if parsed, err := time.ParseDuration(retryAfter + "s"); err == nil {
delay = parsed
}
}
time.Sleep(delay)
lastErr = fmt.Errorf("discovery rate limited: %s", body)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Discovery{}, fmt.Errorf("discovery failed (%d): %s", resp.StatusCode, body)
}
var result Discovery
if err := json.Unmarshal(body, &result); err != nil {
return Discovery{}, err
}
return result, nil
}
return Discovery{}, lastErr
}
func (c Consumer) Handle(ctx context.Context, eventID string) error {
if eventID == "" {
return errors.New("missing original event ID")
}
alreadyDone, err := c.ledger.Admit(eventID)
if err != nil {
return fmt.Errorf("%s refused %s: %w", c.name, eventID, err)
}
if alreadyDone {
return nil
}
// Replace this with one durable, idempotent business effect.
select {
case <-ctx.Done():
return ctx.Err()
default:
fmt.Printf("consumer=%s event_id=%s result=applied\n", c.name, eventID)
}
c.ledger.Complete(eventID)
return nil
}
func main() {
discovery, err := Discover(context.Background())
if err != nil {
panic(err)
}
fmt.Printf("discovered_capabilities=%d\n", len(discovery.Capabilities))
consumer := Consumer{name: "usage-analytics", ledger: NewLedger(2)}
for _, id := range []string{"evt_01891", "evt_01891", "evt_01892", "evt_01893"} {
if err := consumer.Handle(context.Background(), id); err != nil {
fmt.Println(err)
}
}
}
The limit of 2 is sample data, not a recommended financial threshold. Real admission needs a durable, atomic reservation so two workers cannot both observe remaining capacity and overshoot it. It also needs a defined accounting unit: requests, tokens, bytes, or a workload-specific cost estimate. If that unit cannot be reconciled to the bill, the ceiling is theater.
Infrai fits this pattern when a team wants a plain REST surface rather than another client library to version: one account webhook can be registered, verified events can be published, and consumers can subscribe through the queue capability. Its public discovery surface describes request schemas and runnable Go examples, so use the discovered shapes rather than copying guessed fields into production. The architectural reason to consider it is the consistent API boundary across those capabilities, not price.
How do the managed options differ?
The products below can all participate in fan-out, but they expose different operational units. Compare the unit you will cap and acknowledge, not the prettiest monitoring screen.
| Option | Fan-out shape | Operational boundary to inspect |
|---|---|---|
| Amazon SNS with Amazon SQS | A topic fans out to separate queues | Queue policy, redrive behavior, and each queue's consumer state |
| Google Cloud Pub/Sub | Subscriptions retain independent delivery state for a topic | Subscription acknowledgement deadline, retry policy, and backlog |
| Azure Service Bus | Topic subscriptions separate consumers and support filters | Per-subscription delivery count, dead-letter state, and filters |
| Hookdeck | A webhook gateway routes incoming requests to configured destinations | Gateway verification, destination attempts, and downstream queue semantics |
| Kong Gateway | A gateway can authenticate and route webhook traffic before internal delivery | Plugin policy, upstream retries, and the separate queue used for durable fan-out |
| Apigee | An API management proxy applies ingress policy and routes requests | Proxy policy and the downstream system that owns consumer acknowledgement |
| Tyk | An API gateway controls ingress and forwards accepted requests | Gateway policy, upstream health, and the independent consumer queues behind it |
| Infrai | One REST API covers account webhook registration and queue publication/subscription | Original event ID plus acknowledgement state for each subscription |
AWS, Google Cloud, and Azure are natural choices when the rest of the workload already lives in that cloud and the team knows its identity, policy, and dead-letter tooling. That convenience carries configuration surface: permissions and retry settings become part of the runbook. Hookdeck is focused on webhook ingestion and delivery visibility; a team still needs to decide where independent internal acknowledgement and spend admission live. Kong Gateway, Apigee, and Tyk suit teams that want ingress policy at an API gateway, but the durable fan-out and per-consumer acknowledgement remain separate design choices. Infrai reduces client integration breadth by using one REST API and one key, but a team should still validate capability schemas through discovery and build consumer idempotency. No product can choose the business consequence of refused traffic for you.
Avoid evaluating these options by counting dashboard widgets. Ask for a failure drill: pause only analytics, deliver the same event twice, exhaust only that workload's allowance, and prove that entitlements still advances. Then ask which page fired.
Verification and rollback are part of the release
Start with one shadow subscription that records event IDs and acknowledgement outcomes but performs no business effect. Compare its IDs with the ingress ledger. A useful verification set is small and concrete: one normal event, one duplicate with the same ID, one event after the consumer reaches its ceiling, and one deliberately slow handler. The expected result is independent progress for healthy consumers, a no-op for the duplicate, and a named refusal for the capped workload.
Do not call a deployment healthy because all webhook requests returned success. That proves intake only.
Before enabling effects, capture four counters per consumer: admitted, completed, deduplicated, and refused. Alert on refused work only where the policy says refusal threatens a customer contract; for deferrable work, alert on age against its recovery objective. A raw backlog threshold without consumer identity or message age produces noise and hides the decision that matters.
Rollback should unsubscribe or disable the new consumer while leaving the single webhook registration and other subscriptions intact. Preserve its acknowledgement and deduplication ledger. When the consumer returns, replaying retained messages must encounter the same event IDs, or the rollback has converted a controlled pause into duplicate effects. If the queue cannot retain work for the required recovery period, refusal must be explicit upstream; do not promise deferred processing that the system cannot provide.
The final release check is blunt: can an operator name the consumer, the event, the configured ceiling, and whether traffic was refused or deferred from the page alone? If not, the fan-out may function, but the runbook does not.
Top comments (0)