API webhook routing
asynchronous event processing
automated webhook distribution
broadcast webhook to multiple endpoints
decouple microservices webhooks
distribute webhook events
enterprise webhook routing
event broadcasting
event delivery fanout
event-driven architecture webhooks
event-driven microservices
event fanout architecture
event fan-out microservices
event gateway microservices
event ingestion pipeline
HTTP event broadcasting
InstaWebhook
InstaWebhook fanout
microservice communication webhooks
microservice event routing
microservices event gateway
microservices webhook gateway
multi-destination webhooks
multi service webhook dispatcher
publish subscribe webhooks
pub sub webhook routing
real-time event broadcasting
reliable webhook delivery
route one webhook to multiple destinations
route webhooks to multiple URLs
route webhooks to slack
route webhook to multiple microservices
scalable webhook architecture
sendgrid webhook integration
serverless webhook fanout
slack webhook notifications
stripe webhook fanout
stripe webhook integration
stripe webhook microservices routing
webhook architecture best practices
webhook broker architecture
webhook consumer decoupling
webhook dispatch pattern
webhook distribution engine
webhook event pipeline
webhook fan-out architecture
webhook fan out pattern
webhook fanout pattern
webhook forwarding service
webhook gateway pattern
webhook listener pattern
webhook load balancing
webhook multiplexing
webhook payload routing
webhook processing pipeline
webhook proxy microservices
webhook queue management
webhook retry mechanism
webhook router
webhooks multiple endpoints
The Webhook Fan Out Pattern Routing One Event To Multiple Microservices
The Webhook Fan-Out Pattern: Routing One Event to Multiple Microservices
When a customer completes a checkout on your platform, a single event occurs: a payment succeeds. But inside a modern distributed system, that one event triggers a cascade of downstream work across several microservices:
Order Service — updates the core database (orders.status = 'paid')
Notification Service — sends a transactional receipt
Internal Ops — posts an alert to a Slack channel
CRM Integration — updates customer lifetime value
Data/Analytics — streams the transaction into a warehouse for reporting
If your backend receives that payment webhook and tries to handle all five operations synchronously, inside one HTTP request handler, your system will eventually break — usually at the worst possible moment (a traffic spike, a slow third party, a partial outage). That's the problem the webhook fan-out pattern solves.
This article covers why inline webhook handling fails at scale, how fan-out architecture fixes it, the real timeout/retry rules used by major webhook providers, the distributed-systems edge cases (idempotency, ordering, rate limits) you have to handle either way, and the actual build-vs-buy landscape as of 2026.
The Naive Approach: Why Synchronous Webhook Handling Fails
The Monolithic Handler Anti-Pattern
In a naive implementation, your API exposes a single endpoint (e.g. POST /api/webhooks/stripe). The handler receives a payload and synchronously works through every downstream dependency in sequence:
Code example
Copy code
// ANTI-PATTERN: Monolithic Synchronous Webhook Handler
app.post('/api/webhooks/stripe', async (req, res) => {
const event = req.body;
if (event.type === 'payment_intent.succeeded') {
try {
await updateDatabase(event.data); // Step 1: DB write
await sendEmailReceipt(event.data); // Step 2: external API call
await postSlackNotification(event.data); // Step 3: external API call
await updateCRM(event.data); // Step 4: external API call
return res.status(200).send({ received: true });
} catch (error) {
// Which step failed? What should be retried, and what shouldn't?
return res.status(500).send({ error: error.message });
}
}
});
Why this breaks in production
- You're racing a real, provider-enforced clock. Webhook senders don't wait indefinitely for a response, and the limits are tighter than most people assume:
Provider Response timeout Retry behavior
GitHub 10 seconds No automatic retry — failed deliveries must be manually redelivered, and only deliveries from the last 3 days are available for redelivery
Shopify 5 seconds Since a September 2024 policy change: up to 8 retries over a 4-hour window with exponential backoff (older docs citing "19 retries over 48 hours" are outdated)
Stripe Not a single published number — Stripe's own docs simply say to return a 2xx quickly, before any complex logic runs Retries with exponential backoff for up to about 3 days in live mode; some subscription-related events extend this to 72 hours
Sentry (integration platform webhooks) 1 second After 1,000 timeouts in 24 hours, the webhook is automatically unsubscribed
If your handler chains four sequential API calls that each take a few hundred milliseconds to a few seconds, you can blow through any of these windows — and a slow fourth step causes the entire delivery to be marked failed, even though steps 1–3 already succeeded.
Cascading failures cause duplicate side effects. If your CRM call throws an unhandled exception, the function returns a 500. The provider retries the whole webhook later. On retry, the database write and the email send — which already succeeded the first time — run again, causing duplicate charges, duplicate emails, or duplicate records.
Your uptime becomes coupled to every third party you call. If Slack, your CRM, or your email provider has a slow day, your core webhook ingestion — the thing that has to stay reliable — inherits that slowness.
Bursts overwhelm your worker pool. A batch of GitHub push events or a bulk product-sync from an e-commerce platform can spike inbound volume by orders of magnitude in seconds. If webhook processing shares infrastructure with your customer-facing API, that spike can degrade or take down endpoints that have nothing to do with the webhook.
What Is the Webhook Fan-Out Pattern?
The fan-out pattern (also called event broadcasting) ingests an event exactly once, acknowledges it immediately, and then distributes it asynchronously into independent delivery pipelines — one per destination.
Code example
Copy code
┌─────────────────────┐ ──> [Database Service]
│ Delivery Queue 1 │
└─────────────────────┘
▲
┌────────────────┐ ┌──────────┐ │
│ Webhook Source │──>│ Ingestion│─────┼──────────────> [Email Service]
│ (e.g., Stripe) │ │ Gateway │ │
└────────────────┘ └──────────┘ ▼
┌─────────────────────┐ ──> [Slack Notification]
│ Delivery Queue 3 │
└─────────────────────┘
Instead of looping through destinations inside a request handler, the event is written to durable storage or a message broker, which fans it out into N independent delivery tasks for N destinations. Each destination then operates in isolation:
A failure in Service B doesn't delay delivery to Service A.
A slow response from Service C doesn't block acknowledgment of the original event.
Each destination gets its own retry schedule, rate limit, and dead-letter queue (DLQ).
Architectural comparison
Feature Monolithic Synchronous Handling Webhook Fan-Out Pattern
Ingestion speed Slow — sum of every downstream call Fast — event is durably stored and acknowledged in milliseconds
Fault isolation Poor — one destination failure can fail the whole batch High — destinations fail independently
Retry granularity All-or-nothing, often triggers duplicate side effects Per-destination retry schedules
Extensibility Requires touching core handler code Register a new target; no core code changes
Observability Scattered across application logs Centralized, per-endpoint delivery history
Anatomy of a Fan-Out System
A production-grade fan-out architecture has four layers:
Code example
Copy code
[Ingestion] ──> [Storage & Routing] ──> [Dispatch Workers] ──> Destinations
- Ingestion layer. A stable, highly available public endpoint that:
Validates the payload's HMAC signature (Stripe-Signature, X-Hub-Signature-256, etc.) to confirm authenticity before trusting the payload.
Writes the raw payload to durable storage immediately.
Returns a 2xx response — ideally in well under 100ms, and always inside whatever window the sending provider enforces.
Fan-out / routing engine. Once stored, the router checks which destinations are subscribed to this event type and creates one delivery job per match. Filtering and transformation rules can be applied here — e.g., route payment_intent.failed only to an on-call alerting tool, but route all payment events to a data warehouse.
Asynchronous worker pool. Independent workers pull jobs and execute the HTTP calls to each destination in parallel, with per-destination concurrency and rate limits — you don't want a burst of events to blow past Slack's rate limits just because your database can absorb the same burst fine.
Retry engine and dead-letter queue. On a transient failure (HTTP 429, 502, 503, timeout), the specific delivery job is retried with backoff — without affecting other subscribers. If retries are exhausted, the job lands in a per-endpoint DLQ for inspection and manual replay.
Overcoming the Real Distributed-Systems Challenges
Fan-out solves coupling and latency, but distributing one event to many services introduces the usual distributed-systems problems. These apply whether you build the fan-out layer yourself or use a managed gateway.
- Idempotency Nearly every major webhook provider — Stripe, GitHub, Shopify — delivers on an at-least-once basis. Retries, timeout ambiguity, and manual redeliveries all mean your downstream consumers will eventually see the same event more than once. Stripe's own documentation is explicit about this: it recommends checking whether an event has already been processed and returning a success response for duplicates rather than reprocessing them.
The standard fix is a unique event ID plus a fast atomic lock:
Code example
Copy code
// Idempotency guard using an atomic "set if not exists"
async function handleWebhook(req, res) {
const eventId = req.headers['x-event-id'] || req.body.id;
const lockKey = idempotency:${eventId};
// NX = only set if the key doesn't already exist
const isNewEvent = await redis.set(lockKey, 'processing', 'EX', 86400, 'NX');
if (!isNewEvent) {
console.log(Duplicate event received: ${eventId}. Skipping.);
return res.status(200).json({ status: 'ignored', reason: 'duplicate_event' });
}
try {
await processBusinessLogic(req.body);
await redis.set(lockKey, 'completed', 'EX', 86400);
return res.status(200).json({ status: 'success' });
} catch (error) {
// Release the lock on failure so a legitimate retry can run
await redis.del(lockKey);
return res.status(500).json({ error: 'Processing failed' });
}
}
- Out-of-order delivery Because fan-out workers process concurrently, events can arrive at a destination out of sequence — an order.updated event might land before order.created finishes retrying. The fix is to carry a monotonic sequence number or timestamp in the payload and discard stale updates:
Code example
Copy code
async function updateCustomerRecord(customerId, payloadTimestamp, newData) {
const customer = await db.findCustomer(customerId);
if (customer && new Date(customer.last_updated_at) > new Date(payloadTimestamp)) {
console.warn(Stale event for customer ${customerId}. Discarding.);
return;
}
await db.updateCustomer(customerId, { ...newData, last_updated_at: payloadTimestamp });
}
Rate limits and the "noisy neighbor" problem
A burst of thousands of events in a few seconds (a bulk price sync, a mass CSV import) will crash a downstream service if it's pushed through unthrottled. Fan-out systems should enforce per-destination rate limits — a high-capacity database endpoint and a Slack webhook (which enforces its own strict rate limits) need very different throttles.Backoff needs jitter, not just exponentiation
Plain exponential backoff reduces retry frequency, but if many clients fail at the same moment, they stay synchronized and retry in clusters, which just moves the thundering herd to a later time instead of eliminating it. AWS's widely cited 2015 engineering write-up on this ("Exponential Backoff and Jitter," still referenced in AWS's Well-Architected Framework today) showed that adding randomized jitter on top of exponential backoff is what actually spreads retries out and prevents synchronized retry storms. Most current SDKs and retry libraries implement this combination by default now, so in practice you rarely need to hand-roll it — but it's worth knowing why plain backoff alone is not enough.Signature verification on the way out, too
If you're operating a fan-out relay in front of your own microservices, sign your outgoing requests (a timestamped HMAC is the common pattern, mirroring how Stripe and GitHub sign their own webhooks) so your internal services can verify a delivery actually came from your gateway and not from an internal network compromise.
Build vs. Buy: DIY Queues vs. a Managed Webhook Gateway
Teams generally choose between two approaches:
DIY on cloud messaging infrastructure. The classic pattern on AWS is SNS (or EventBridge) publishing to multiple SQS queues, each consumed independently — you publish once, and every subscribed queue gets its own durable copy, so a slow or failing consumer never blocks the others. This is the same underlying pattern described above, built from primitives you already have if you're on AWS.
Code example
Copy code
Provider ──> API Gateway ──> SNS Topic ──> SQS Queue A ──> Lambda / Worker A
└─> SQS Queue B ──> Lambda / Worker B
└─> SQS Queue C ──> Lambda / Worker C
This gives you full control and no vendor lock-in, at the cost of building and maintaining the retry logic, DLQ wiring, per-destination rate limiting, HMAC verification, and a debugging UI yourself.
Managed webhook gateways. A newer category of dedicated infrastructure exists specifically to remove that operational burden. As of 2026, the more established options include:
Hookdeck Event Gateway — purpose-built for receiving inbound webhooks, with fan-out to multiple destinations, durable queueing, filtering/transformation rules, and delivery-level observability.
Svix — split into Svix Ingest (inbound, with JS-based fan-out routing) and Svix Dispatch (outbound delivery for products that send webhooks to their customers), plus an open-source self-hosted server.
Convoy — an open-source, self-hostable option that unifies sending and receiving.
Smaller/regional options such as Hook0 (EU-hosted, lower-volume focus).
Before committing to any of these, check their current docs and pricing directly — this space has moved fast, feature sets and plan tiers change, and open-source maintenance status varies project to project.
Dimension DIY (SNS/SQS/EventBridge) Managed Gateway
Setup time Days to weeks (IaC, IAM, workers) Minutes to hours
Maintenance You own scaling, patching, backpressure Handled by the vendor
Per-endpoint debugging Requires stitching together CloudWatch/log data Usually a built-in delivery timeline UI
Manual replay You write the tooling Typically built in
HMAC signing/verification You implement it Often built in
Data residency/compliance Full control, inside your own VPC Varies by vendor — check for SOC 2/HIPAA/PCI-DSS as needed
Neither approach eliminates the idempotency, ordering, and rate-limiting work described above — that logic still has to live somewhere, either in your own workers or configured into the gateway you choose.
Production Checklist
Acknowledge fast. Return a 2xx well inside the sending provider's timeout window — GitHub gives you 10 seconds, Shopify gives you 5. Don't perform external network calls before responding.
Enforce idempotency. Every downstream consumer checks a unique event ID before taking action, because at-least-once delivery means duplicates will happen.
Verify signatures. Validate HMAC signatures on both the inbound provider webhook and any internal fan-out deliveries.
Isolate destinations. Give each destination its own queue/retry schedule so a slow or broken service never blocks the others.
Configure DLQs. Route exhausted-retry payloads somewhere inspectable, with alerting.
Use exponential backoff with jitter, not backoff alone, to avoid synchronized retry storms.
Have a redelivery/reconciliation plan. Some providers (GitHub) don't auto-retry at all, and even the ones that do have finite windows (hours to a few days) — build a way to reconcile via the provider's API for anything that falls outside that window.
Summary
Coupling downstream microservices directly to an inbound webhook handler creates brittle systems: slow API responses, duplicate side effects on retry, and availability that's hostage to every third party you call. The fan-out pattern fixes this by ingesting an event once, acknowledging it immediately, and distributing it into independent, isolated delivery pipelines — whether you build that on top of SNS/SQS, or adopt a managed gateway to skip the undifferentiated infrastructure work. Either way, idempotency, ordering guards, and per-destination rate limiting are the pieces that make the pattern actually reliable in production, not optional extras.
Sources & further reading
GitHub: Best practices for using webhooks — 10-second response window
GitHub: Handling failed webhook deliveries — no automatic retry
Shopify Engineering: Webhook best practices
Stripe: Receive Stripe events in your webhook endpoint
Stripe: Process undelivered webhook events — idempotent processing guidance
Sentry: Integration platform webhook timeout change
AWS Architecture Blog: Exponential Backoff and Jitter
AWS Well-Architected Framework: Control and limit retry calls
Hookdeck: Webhook platform guides
Svix: Webhook resources
Top comments (0)