First of all what is a "webhook" and what is a "webhook delivery" system?
So a webhook is a way for one application to notify another application when an event occurs.
Instead of repeatedly asking, "Has anything changed?", an application can send an HTTP request as soon as something happens. For example, when a customer makes a payment, a payment provider can immediately notify your application that the payment was successful.
In simple terms, webhooks are event-driven HTTP callbacks that enable systems to communicate in real time.
A webhook delivery system is the infrastructure responsible for reliably delivering those event notifications to subscriber endpoints.
Its job goes beyond sending a single HTTP request. A production-grade webhook delivery system must handle retries, failures, rate limiting, monitoring, security, and large volumes of events while ensuring that important notifications reach their destinations.
Examples of companies that rely heavily on webhook delivery systems include Stripe, GitHub, and Shopify, where millions of webhook events are delivered every day.
Now that we understand what webhooks are, let's design a webhook delivery system.
V1: The Naive Approach
Naive approach to send webhook events
The first solution most developers think of is straightforward:
Receive an event.
Look up the subsc
- riber's webhook URL.
- Send an HTTP request to that URL.
- Mark the delivery as successful or failed. This approach works perfectly for small projects and low traffic. If a few webhook events are generated each minute, a single API server can handle everything.
However, problems start appearing as traffic grows.
What's Wrong With This Design?
A webhook endpoint is outside of our control. It might be slow, unavailable, or return errors.
If a receiver takes 30 seconds to respond, our application is forced to wait. During traffic spikes, thousands of slow webhook requests can consume server resources and delay other deliveries.
Even worse, if the server crashes while processing events, some deliveries may be lost entirely.
V2: Introducing a Queue
common improvement is to separate event ingestion from delivery.
Instead of sending webhooks immediately, the API stores events in a queue. Worker processes consume events from the queue and perform the actual HTTP deliveries.
This makes the API fast and resilient because receiving events is no longer blocked by slow webhook endpoints.
New Problems Appear
While queues solve many issues, new challenges emerge:
- A single failing endpoint can create a growing backlog.
- Continuous retries can waste resources.
- High-volume customers can dominate the queue.
- Rate limits must be respected.
- Temporary outages can trigger retry storms. As the number of customers grows, ensuring reliable and fair delivery becomes increasingly difficult.
Building Toward a Production-Ready System
To address these challenges, modern webhook platforms introduce concepts such as retries with backoff, dead-letter queues, rate limiting, circuit breakers, delivery tracking, and tenant isolation.
These additions improve reliability, but they also increase system complexity.
The architecture used in HookRelay was designed with these challenges in mind. Rather than treating webhook delivery as a simple HTTP request, it treats delivery as a distributed workflow that must remain reliable under failures, traffic spikes, and multi-tenant workloads.
V3: HookRelay Design
The previous architectures are a good starting point, but they begin to break down as traffic increases and more customers start relying on the platform. A production webhook delivery system must do much more than simply send HTTP requests—it needs to ingest events quickly, survive failures, isolate tenants, retry intelligently, and scale without becoming a bottleneck.
These requirements shaped the design of HookRelay.
Instead of treating webhook delivery as a single operation, HookRelay breaks it into multiple independent stages. Each stage is responsible for one part of the delivery pipeline and communicates asynchronously with the next. This separation keeps the API responsive, improves reliability, and allows different parts of the system to scale independently.
At a high level, every event follows this journey:
Producer → API Service → Redis Buffer → Ingest Worker → PostgreSQL → Fanout Worker → Per-Tenant Delivery Queues → Delivery Workers → Consumer Endpoints
Each stage exists for a specific reason. Some stages improve throughput by batching operations, while others ensure fairness between tenants or increase reliability through retries and circuit breakers. Together, they form a webhook delivery pipeline that can continue operating even when endpoints are slow, temporarily unavailable, or receiving a high volume of traffic.
Let's walk through the architecture one component at a time and understand why each piece exists.
1. API Service
Every webhook event enters the system through the API Service. Its responsibility is not to deliver webhooks directly, but to validate incoming requests and accept events as quickly as possible. The faster the API responds, the more events the system can ingest under heavy load.
Before an event enters the delivery pipeline, the API performs several important checks.
Tenant Authentication
Since HookRelay is a multi-tenant platform, every request must belong to a valid tenant. Producers authenticate using an API key, allowing the system to identify which customer owns the event and which delivery rules should be applied.
To avoid querying PostgreSQL on every request, tenant information is cached in Redis, significantly reducing database load during high traffic.
Rate Limiting
No single tenant should be able to overwhelm the system. The API enforces per-tenant rate limits before accepting an event.
HookRelay implements rate limiting using atomic Lua scripts in Redis, ensuring accurate counters even when multiple API instances are running concurrently. Requests that exceed the configured limit receive an HTTP 429 Too Many Requests response.
Idempotency
Network failures often cause producers to retry requests. Without protection, the same event could be processed multiple times, resulting in duplicate webhook deliveries.
To prevent this, every event can include an idempotency key. If the API receives the same key again, it recognizes the request as a duplicate instead of creating another event.
Redis Buffer
After passing all validation, the event is not written directly to PostgreSQL. Instead, it is pushed into a Redis buffer, and the API immediately returns 202 Accepted.
This keeps request latency low because the client doesn't wait for database writes or webhook processing. The actual persistence and delivery pipeline are handled asynchronously by background workers.
Why Not Write Directly to PostgreSQL?
A common question is: Why introduce Redis at all? Why not insert every event directly into PostgreSQL?
The answer is throughput.
If every incoming request performs an individual database insert, PostgreSQL quickly becomes the bottleneck under high traffic. Each request must wait for disk I/O before receiving a response, increasing latency and reducing overall throughput.
By using Redis as a temporary ingestion buffer, the API only performs a fast in-memory operation before responding. Background workers later drain the buffer and insert hundreds of events into PostgreSQL in batches, dramatically reducing database overhead and allowing the API to remain responsive even during traffic spikes.
This separation between event ingestion and event persistence is one of the key design decisions that enables HookRelay to scale efficiently.
2. Ingest Worker
At this point, the event has been accepted by the API and stored safely in the Redis buffer. However, Redis is only a temporary staging area, it isn't the system's source of truth.
The job of the Ingest Worker is to continuously drain events from the Redis buffer, persist them in PostgreSQL, and prepare them for the next stage of the delivery pipeline. By moving these operations into a background worker, the API remains lightweight and can continue accepting new events without waiting for database operations.
Batch Pop from Redis
Instead of processing one event at a time, the Ingest Worker pulls multiple events from the Redis buffer in a single operation.
This reduces the number of Redis operations and allows the worker to process large bursts of incoming traffic much more efficiently.
Batch Insert into PostgreSQL
Once a batch of events is retrieved, they are inserted into PostgreSQL using a single bulk insert instead of individual database writes.
PostgreSQL performs significantly better when inserting many rows in one query rather than executing hundreds of separate insert statements. This reduces database overhead, minimizes transaction costs, and improves overall throughput.
Bulk Enqueue Fanout Jobs
After the events have been persisted successfully, the Ingest Worker creates fanout jobs in bulk.
Each fanout job represents the next stage of the pipeline, where the system determines which webhook endpoints should receive the event. Enqueuing these jobs in bulk further reduces Redis overhead and keeps the pipeline efficient.
Why Batching?
Batching is one of the biggest performance optimizations in HookRelay.
Without batching, every incoming event would result in:
- One Redis read
- One PostgreSQL insert
- One queue operation At thousands of events per second, these individual operations quickly become expensive.
By processing events in batches, the system performs fewer database transactions, fewer queue operations, and fewer network round trips. The result is lower latency, higher throughput, and much better resource utilization.
This batching strategy was one of the key optimizations that significantly improved HookRelay's ingestion performance while allowing the system to scale under heavy workloads.
3. Fanout Worker
Once an event has been safely stored in PostgreSQL, the next question is:
Who should receive this event?
This is the responsibility of the Fanout Worker.
Rather than sending HTTP requests directly, the Fanout Worker determines which endpoints have subscribed to the event, creates delivery records for each endpoint, and distributes those deliveries across tenant-specific queues.
Resolve Subscribed Endpoints
Every tenant can register one or more webhook endpoints, and each endpoint can subscribe to different event types.
When a new event arrives, the Fanout Worker queries the registered endpoints for that tenant and selects only those that are interested in the event being processed.
For example, if a tenant has three endpoints but only one subscribes to order.created, only that endpoint will receive the event.
Create Delivery Records
After identifying the matching endpoints, the Fanout Worker creates a delivery record for each one.
If an event needs to be delivered to five endpoints, five independent delivery records are created.
Tracking deliveries separately allows HookRelay to monitor the status of every webhook independently. One endpoint may succeed immediately, another may require retries, while another may eventually end up in the dead letter queue. Each delivery follows its own lifecycle.
Per-Tenant Delivery Queues
Instead of placing every delivery into a single shared queue, HookRelay creates a dedicated delivery queue for each tenant.
This isolates workloads between customers. If one tenant suddenly generates millions of events or has slow webhook endpoints, their deliveries remain confined to their own queue and do not delay deliveries for other tenants.
Why Is Fanout a Separate Stage?
It might seem tempting to let the Ingest Worker perform endpoint resolution and enqueue delivery jobs immediately. However, separating fanout into its own stage keeps each worker focused on a single responsibility.
The Ingest Worker is optimized for high-throughput persistence moving events from Redis into PostgreSQL as quickly as possible. The Fanout Worker, on the other hand, performs business logic by resolving subscriptions, creating delivery records, and scheduling deliveries.
This separation allows each stage to scale independently. If endpoint resolution becomes expensive as the number of tenants grows, additional Fanout Workers can be added without affecting event ingestion. It also keeps the architecture modular, making each stage simpler to reason about, monitor, and optimize.
At the end of this stage, every delivery has been created and placed into the appropriate tenant queue, ready for the final step: sending the actual webhook requests.
4. Delivery Workers
Once the delivery jobs have been distributed into tenant-specific queues, the Delivery Workers take over.
Their responsibility is straightforward: consume delivery jobs, send HTTP requests to the appropriate webhook endpoints, and record the result of each attempt. Since every tenant has its own queue, multiple Delivery Workers can process deliveries concurrently without one tenant affecting another.
HTTP POST
For every delivery job, the worker constructs an HTTP POST request containing the original event payload and sends it to the registered webhook endpoint.
A successful 2xx response marks the delivery as successful. Any non-2xx response, network error, or timeout is treated as a failed delivery and handed over to the retry mechanism.
HMAC Signing
Webhook payloads should never be trusted blindly. To allow consumers to verify that a request genuinely originated from HookRelay, every outgoing request is signed using HMAC-SHA256.
The worker includes a signature header generated from the payload and the endpoint's secret. On the receiving side, consumers can generate the same signature and compare the two values to verify the authenticity and integrity of the request.
Timeouts
A webhook endpoint is an external service and is completely outside of our control. Some endpoints may respond immediately, while others may become slow or stop responding altogether.
To prevent workers from waiting indefinitely, every HTTP request is executed with a timeout. If the endpoint fails to respond within the configured time, the attempt is marked as failed and the delivery becomes eligible for retry. This ensures that slow endpoints cannot block workers from processing other deliveries.
Concurrency
Webhook deliveries are independent of one another, making them ideal for concurrent processing.
Instead of processing deliveries sequentially, multiple Delivery Workers can execute many HTTP requests in parallel. Combined with tenant-specific queues, this allows HookRelay to achieve high throughput while maintaining fairness between customers. If one tenant experiences slow or failing endpoints, only that tenant's workers are affected, while deliveries for other tenants continue uninterrupted.
At this stage, the webhook has either been delivered successfully or has failed. In the next section, we'll look at how HookRelay handles failures through retries, exponential backoff, circuit breakers, and dead letter queues.
5. Reliability Features
In the real world, webhook deliveries rarely succeed 100% of the time. Endpoints can be temporarily unavailable, networks can fail, or servers can become overloaded. A reliable webhook platform must expect these failures and recover from them automatically.
HookRelay includes several reliability mechanisms to ensure that temporary failures don't result in permanently lost events.
Retries
A failed delivery doesn't necessarily mean the webhook should be discarded. Many failures are temporary - a server may be restarting, a network connection may briefly drop, or an application may be experiencing high load.
Instead of giving up immediately, HookRelay automatically retries failed deliveries. Every failed attempt is recorded, allowing each delivery to progress independently until it either succeeds or reaches the maximum retry limit.
Exponential Backoff
Retrying immediately after a failure usually makes the situation worse. If a consumer's server is already overloaded, continuously sending new requests only increases the pressure.
Instead, HookRelay spaces retries using an exponential backoff strategy. The delay between attempts gradually increases from seconds to minutes, and eventually hours giving downstream services time to recover before another delivery attempt is made.
Circuit Breaker
Repeatedly sending requests to an endpoint that is clearly unavailable wastes CPU, network bandwidth, and worker capacity.
To avoid this, HookRelay uses a circuit breaker for every endpoint. After several consecutive failures, the circuit enters the OPEN state, temporarily preventing new requests from being sent to that endpoint. After a cooldown period, a single probe request is allowed. If it succeeds, normal delivery resumes; otherwise, the circuit remains open until the endpoint recovers. This prevents failing endpoints from consuming resources while still allowing automatic recovery.
Dead Letter Queue
Not every delivery can be recovered. Some endpoints may remain permanently unavailable or continue returning errors despite multiple retries.
After the maximum number of delivery attempts has been exhausted, HookRelay marks the delivery as dead instead of retrying forever. Dead deliveries can later be inspected, replayed, or manually retried by an administrator without affecting the rest of the delivery pipeline.
Together, these reliability mechanisms allow HookRelay to continue delivering webhooks even when downstream services are unreliable, while preventing temporary failures from impacting the overall health of the system.
6. Why Per-Tenant Queues?
So far, we've seen how events are ingested, expanded into deliveries, and processed by Delivery Workers. One important design decision remains:
Why does HookRelay create a separate delivery queue for every tenant instead of using a single shared queue?
The answer lies in fairness, isolation, and scalability.
The Noisy Neighbor Problem
Imagine two customers using the platform.
- Tenant A generates 50 webhook events per minute.
- Tenant B suddenly generates 100,000 events in a few minutes.
If both tenants share the same delivery queue, Tenant B's deliveries will flood the queue, causing Tenant A's webhook deliveries to wait behind thousands of unrelated jobs. Even though Tenant A has very little traffic, their webhooks will experience increased latency simply because another tenant is much busier.
This is commonly known as the Noisy Neighbor Problem.
Tenant Isolation
HookRelay avoids this problem by creating a dedicated delivery queue for each tenant.
Every tenant's deliveries are processed independently. If one tenant experiences traffic spikes, slow webhook endpoints, or repeated failures, those issues remain isolated to that tenant's queue. Other tenants continue receiving their webhook deliveries without interruption.
This isolation makes the platform far more predictable and provides a fairer experience for every customer.
Horizontal Scaling
Per-tenant queues also make the system easier to scale.
As traffic grows, additional Delivery Workers can be added to process more queues concurrently. Since workers consume jobs independently, increasing processing capacity doesn't require changing the overall architecture it simply means running more worker instances.
This design allows HookRelay to scale horizontally while maintaining tenant isolation and consistent delivery performance.
By isolating tenants into independent queues, HookRelay ensures that one customer's workload never becomes another customer's problem. This simple architectural decision significantly improves fairness, resilience, and scalability in a multi-tenant webhook delivery platform.
7. Performance Optimizations
Building a reliable webhook delivery system isn't enough, it also needs to handle high traffic efficiently. Throughout the development of HookRelay, several optimizations were introduced to reduce latency, minimize database load, and improve overall throughput.
Redis Authentication Cache
Every incoming request must be authenticated to identify the tenant and apply the correct delivery rules.
Performing a PostgreSQL lookup for every request would quickly become expensive under heavy traffic. To avoid this, tenant information is cached in Redis with a short TTL. Most authentication requests are therefore served directly from memory, significantly reducing database load and improving request latency.
Lua Scripts
Some operations, such as rate limiting and circuit breaker state transitions, must remain atomic even when multiple API or worker instances are running simultaneously.
Instead of implementing these operations with multiple Redis commands, HookRelay uses Lua scripts that execute atomically on the Redis server. This eliminates race conditions, reduces network round trips, and ensures consistent behavior under concurrent workloads.
Batch Inserts
Database writes are one of the most expensive operations in the pipeline.
Rather than inserting events one at a time, the Ingest Worker collects multiple events from Redis and performs a single bulk insert into PostgreSQL. This significantly reduces transaction overhead, improves database efficiency, and allows the system to process much higher event volumes with the same resources.
Asynchronous Ingestion
Perhaps the biggest optimization is separating event ingestion from event processing.
Instead of waiting for database writes, endpoint resolution, and webhook delivery to complete, the API validates the request, pushes the event into the Redis buffer, and immediately returns 202 Accepted. The remaining work is performed asynchronously by background workers.
This keeps the API responsive even during traffic spikes while allowing each stage of the pipeline to scale independently. These optimizations work together to reduce latency on the critical request path while maximizing throughput in the background. The result is a webhook delivery platform capable of accepting events quickly and processing them efficiently under heavy workloads.
8. Performance Results
Designing a scalable architecture is only valuable if it performs well under load. To evaluate HookRelay, I ran a series of load tests using k6, simulating concurrent producers continuously sending events to the POST /events endpoint.
The goal of these tests wasn't to measure webhook delivery speed, but to measure how efficiently the system could ingest events while keeping API latency low.
Load Test Configuration
Tool: k6
- Duration: 5 minutes
- Concurrent Users: 200 Virtual Users
- Target Endpoint: POST /events
Environment: Docker Desktop running PostgreSQL, Redis, API, and Worker on a single machine.
Results
MetricValue
- Throughput 1,259 req/s
- Total Requests 377,843
- p50 Latency 73 ms
- p95 Latency 166 ms
- Error rate (>500ms) 0.12%
- HTTP Failure Rate 0.00% These results demonstrate that the ingestion pipeline remains responsive even under sustained concurrent traffic. The API consistently returned low response times while successfully accepting every request during the test.
Optimization Journey
The final numbers weren't achieved in a single iteration. The system evolved through several performance optimizations, including Redis-based authentication caching, Lua-powered rate limiting, asynchronous event ingestion, and batch inserts into PostgreSQL.
Together, these improvements increased throughput from 342 req/s to 1,259 req/s, while reducing median latency from 247 ms to 73 ms and decreasing the error rate from 22.71% to 0.12%.
While there is still room for further optimization, these benchmarks show that the architectural decisions discussed throughout this article particularly asynchronous ingestion, batching, and workload isolation translate into measurable performance improvements under load.
Top comments (0)