Enterprise Webhook Architecture at Scale: High-Throughput Ingress, Multi-Region CRDTs, and Adaptive Backpressure

Webhook traffic is bursty and outside your control. Senders fire when something happens, such as a flash sale, a settlement batch or a fleet of devices reconnecting, and they all do it at once. This article covers three layers that together keep an ingestion tier alive under that load: an in-memory ingress buffer, active-active multi-region replication, and adaptive load shedding. For each layer it also covers the trade-offs, which matter as much as the mechanisms.

A note on numbers: “500,000 requests per second” is used here as a design target, not a benchmark. The real ceiling depends on payload size, TLS termination, core count and NIC bandwidth, so measure on your own hardware.

Part 1: RAM-Disk Ingress Buffers — Using Linux tmpfs and Shared Memory to Absorb Webhook Bursts

Why senders force this design

Major webhook senders expect a fast acknowledgment and do the waiting for you. Stripe’s documentation tells receivers to return a 2xx immediately and defer complex logic. In live mode it retries failed deliveries for up to three days with exponential backoff. GitHub expects a 2xx within 10 seconds and otherwise terminates the connection and records a failure. It also does not automatically redeliver failed deliveries, so a missed delivery stays missed until someone redelivers it.

The consequence is that the hot path should do as little as possible before acknowledging. That means validating the signature, appending the payload to a buffer and returning. Writing each payload straight into a relational database, or into a broker configured for synchronous disk persistence, puts storage latency inside the sender’s timeout window. During a spike this produces timeouts, and the timeouts produce retries that add to the spike.

How tmpfs behaves

tmpfs keeps its contents in the kernel’s page cache and, if swap is enabled, can push unneeded pages out to swap. According to the kernel documentation, a few details matter for an ingress buffer:

• The default size limit is half of physical RAM, and nr_inodes caps the inode count.
• Swap is enabled by default. The noswap mount option disables it, which you want here, because a buffer that silently spills to swap gives up the latency profile you built it for.
• /dev/shm is normally a tmpfs mount, so POSIX shared memory (shm_open) and files under a dedicated tmpfs mount draw on the same memory pool.

# 64 GiB buffer, no swap, owner-only access
mount -t tmpfs -o size=64G,nr_inodes=2M,mode=0700,noswap tmpfs /mnt/ingress-buffer

Memory in tmpfs is real RAM that other processes can no longer use, so alert on df usage and size the mount against your worst-case burst.

Sizing arithmetic (illustrative). Assume an average payload of 4 KiB at 500,000 RPS. That is roughly 1.9 GiB/s of inbound payload data, so a 64 GiB buffer absorbs about 33 seconds of sustained peak if nothing drains. Two consequences follow:

1. The buffer buys you time but is not unlimited. The drain path must be sized to catch up after the burst.
2. Do not write one file per webhook. At 500,000 RPS, a 2M-inode limit would be exhausted in about four seconds. Append payloads to large segment files instead, for example 64–256 MiB each, with a small length-prefixed record header.

Ring buffers and io_uring

The pipeline looks like this:

Client TLS / HTTP ──► Edge proxy (Rust or Go; epoll or io_uring)
                            │  validate signature, enforce size limits
                            ▼
                 Append-only segment files on tmpfs
                            │
                            ▼
               Drain workers (user space, batched)
                            │
                            ▼
           Local NVMe log ► Kafka / object storage

A webhook endpoint needs TCP and TLS termination, so the data path goes through the normal kernel network stack. XDP and eBPF are useful earlier in the path, for dropping obviously bad traffic at the NIC before it costs you CPU. They don’t replace the HTTP server.

Within the proxy, lock-free single-producer/single-consumer rings (one per worker thread) are a common way to hand records to a writer without contention. io_uring can then batch the file writes.

io_uring has an IORING_SETUP_SQPOLL mode in which the kernel creates a thread that polls the submission queue. The man page says this lets an application issue I/O without context switching into the kernel, with two caveats worth knowing:

• If the polling thread is idle for longer than sq_thread_idle milliseconds, it goes to sleep and sets a flag. The application must then call io_uring_enter(2) to wake it. SQPOLL reduces system calls under sustained load, but it does not eliminate them.
• Unprivileged SQPOLL requires a newer kernel (5.13, per the man page). The polling thread also occupies a CPU, so budget cores for it. The io_uring_sqpoll(7) page notes that it only affects submissions, so completion handling still costs CPU.

Some hardened or containerized environments restrict io_uring, so confirm it is permitted on your hosts before building around it.

An illustrative sketch with the Rust io-uring crate follows. Check the API against the crate version you pin:

use io_uring::{opcode, types, IoUring};
use std::{fs::OpenOptions, os::unix::io::AsRawFd};

fn main() -> std::io::Result<()> {
    // SQPOLL: a kernel thread polls the submission queue and sleeps after 1000 ms idle.
    let mut ring = IoUring::builder().setup_sqpoll(1000).build(256)?;

    let file = OpenOptions::new()
        .create(true)
        .write(true)
        .open("/mnt/ingress-buffer/segment-000001.log")?;

    let payload = b"{\"event\":\"example\"}\n";
    let write = opcode::Write::new(
        types::Fd(file.as_raw_fd()),
        payload.as_ptr(),
        payload.len() as _,
    )
    .offset(0)
    .build()
    .user_data(1);

    // Safety: `payload` and `file` outlive the completion below.
    unsafe { ring.submission().push(&write).expect("submission queue full") };
    ring.submit_and_wait(1)?;

    let cqe = ring.completion().next().expect("completion missing");
    assert!(cqe.result() >= 0, "write failed: {}", cqe.result());
    Ok(())
}

The durability trade-off

A buffer in RAM does not survive a power failure or kernel panic, and fsync() on tmpfs buys nothing because there is no backing store. Acknowledging with 202 Accepted after a tmpfs write therefore creates a window in which the sender believes the event is safe but your platform could lose it. Because senders only retry on failure, a lost event that was acknowledged is not retried.

RFC 9110 defines 202 as acceptance for processing, not completion. That fits the pattern, but it doesn’t make the data durable. There are three common ways to narrow the window:

1. Replicate before acknowledging. Copy the record into a second host’s memory in another failure domain and acknowledge after both writes succeed. This costs a network round trip inside a region, typically on the order of a millisecond between availability zones, according to ThousandEyes measurements.
2. Group-commit to local NVMe. Acknowledge after a batched, sequential append to a local log. This is slower than RAM but far cheaper than per-request database writes.
3. Accept the window deliberately for low-value event classes, and reconcile against the sender’s API or redelivery tools after an incident.

Pick one per event class and document it.

Draining to persistent storage

Drain workers run in user space. They read sealed segments from tmpfs, batch them and write them onward. A few practical points:

• Object storage. S3 multipart parts must be 5 MiB to 5 GiB, except the last part, with at most 10,000 parts per upload. Use chunks of 8–16 MiB or larger. Chunks below 5 MiB are rejected for non-final parts.
• Direct I/O. O_DIRECT is relevant when writing to a local disk log, not when uploading over HTTP. tmpfs itself gained O_DIRECT support in Linux 6.6; on older kernels, opening a tmpfs file with it fails. The drain side can simply use ordinary reads or mmap.
• Idempotency. Draining can crash midway and replay a segment. Give every record a stable ID, such as the sender’s delivery ID (GitHub supplies X-GitHub-Delivery) or a hash, so downstream consumers can deduplicate.

Part 2: Active-Active Cross-Region Webhook Ingress — Convergence with CRDTs and Vector Clocks

Why active-active

Consider a provider serving customers from AWS us-east-1 (N. Virginia), eu-central-1 (Frankfurt) and ap-southeast-1 (Singapore). Each region should accept webhooks locally, so that a regional outage or a slow inter-region link doesn’t block ingestion.

To steer clients to a nearby region, there are two options:

• Anycast. The same IP is announced from multiple locations and network routing (BGP) delivers each client to a nearby one. Services such as AWS Global Accelerator work this way.
• DNS-based routing (latency or geolocation policies). This is a separate mechanism from anycast. Don’t conflate them: anycast operates at the network layer, not through DNS.

A webhook sender such as Stripe will use whichever endpoint you give it. If that is a single hostname, steering happens on your side.

Why not synchronous consensus across regions

Quorum protocols such as Paxos or Raft need at least one round trip to a majority of replicas per decision. Inside a region that is around a millisecond. Across continents it is limited by distance and routing, and typically ranges from tens to hundreds of milliseconds. AWS publishes measured inter-region numbers through Network Manager Infrastructure Performance. Putting that on the acknowledgment path eats into the sender’s timeout budget and ties availability to the slowest link. The alternative is to accept writes locally and replicate asynchronously, and to design the data model so that replicas converge.

Terminology: conflicts, not split-brain

“Split-brain” usually describes two nodes both believing they are the sole leader. In an active-active design, every region is supposed to accept writes, so the real problem is concurrent, possibly conflicting updates to the same entity, plus out-of-order delivery. Two tools address this: vector clocks, which detect concurrency, and CRDTs, which merge it deterministically.

Vector clocks

Each entity carries a vector clock, a map from region ID to a counter. A region increments its own entry when it records an event and takes the element-wise maximum when it receives remote state.

Region A: {A:1, B:0, C:0}  ← event 1
Region B: {A:0, B:1, C:0}  ← event 2 (neither saw the other: concurrent)
After exchange: merged clock {A:1, B:1, C:0}

For vector clocks V₁ and V₂ over N regions:

• V₁ ≤ V₂ if V₁[i] ≤ V₂[i] for every region i.
• V₁ causally precedes V₂ if V₁ ≤ V₂ and V₁[j] < V₂[j] for at least one j.
• If neither V₁ ≤ V₂ nor V₂ ≤ V₁, the events are concurrent, and the merge rule decides the outcome.

Vector clocks grow with the number of participants. That is fine for three or five regions but not for per-client identifiers. Keep them per region, not per sender.

Many systems pair this with hybrid logical clocks, which combine physical time with a logical counter, so that timestamps stay close to wall-clock time yet never go backwards. They are useful as a tiebreaker or for ordering display, but wall clocks alone are not a safe basis for correctness.

CRDTs

Shapiro, Preguiça, Baquero and Zawirski’s 2011 study formalized two families of conflict-free replicated data types. Both guarantee that replicas converge without foreground coordination:

• State-based (CvRDT): replicas exchange full or delta state and merge it with an associative, commutative, idempotent function.
• Operation-based (CmRDT): replicas exchange operations that commute, which requires reliable, exactly-once delivery of those operations.

Useful building blocks for a webhook platform include:

• PN-Counters for approximate global metrics such as delivery attempts or events ingested. They converge, but they cannot enforce a strict global limit, because that would require coordination. Treat a PN-Counter-based rate limit as approximate.
• Observed-Remove Sets (OR-Sets) for subscription or endpoint registries, where concurrent adds and removes must converge predictably. An add concurrent with a remove of the same element wins.
• LWW registers for single-value fields where “latest write wins” is acceptable.

Here is a last-writer-wins register that uses the vector clock for causality and falls back to a timestamp only for truly concurrent writes. The original idea of merging clocks but then deciding on the timestamp alone can silently discard a causally newer update when clocks are skewed, so the causal check comes first:

use std::collections::HashMap;

#[derive(Clone)]
pub struct LwwRegister<T> {
    pub value: T,
    pub clock: HashMap<String, u64>, // vector clock: region -> counter
    pub hlc_micros: u64,             // hybrid logical timestamp
    pub replica_id: String,          // deterministic tiebreaker
}

enum Causality { Before, After, Equal, Concurrent }

fn compare(a: &HashMap<String, u64>, b: &HashMap<String, u64>) -> Causality {
    let (mut a_less, mut b_less) = (false, false);
    for k in a.keys().chain(b.keys()) {
        let x = a.get(k).copied().unwrap_or(0);
        let y = b.get(k).copied().unwrap_or(0);
        if x < y { a_less = true } else if x > y { b_less = true }
    }
    match (a_less, b_less) {
        (false, false) => Causality::Equal,
        (true, false) => Causality::Before,
        (false, true) => Causality::After,
        (true, true) => Causality::Concurrent,
    }
}

impl<T: Clone> LwwRegister<T> {
    pub fn merge(&mut self, remote: &Self) {
        let remote_wins = match compare(&self.clock, &remote.clock) {
            Causality::Before => true,                      // remote is causally newer
            Causality::After | Causality::Equal => false,   // local is newer or identical
            Causality::Concurrent => {
                // Timestamp first, replica ID as a deterministic tiebreaker
                (remote.hlc_micros, &remote.replica_id) > (self.hlc_micros, &self.replica_id)
            }
        };

        let mut merged = self.clock.clone();
        for (node, &r) in &remote.clock {
            let e = merged.entry(node.clone()).or_insert(0);
            *e = (*e).max(r);
        }

        if remote_wins {
            self.value = remote.value.clone();
            self.hlc_micros = remote.hlc_micros;
            self.replica_id = remote.replica_id.clone();
        }
        self.clock = merged;
    }
}

Two cautions apply. First, with a concurrent update, LWW keeps one value and discards the other. That is acceptable for something like “current endpoint URL” but not for data where both writes matter. For those, use a multi-value register or a domain-specific merge. Second, because senders may deliver the same event more than once and don’t guarantee ordering, receivers should be idempotent regardless. Stripe’s documentation says as much, and GitHub exposes a per-delivery ID header for deduplication.

Part 3: Adaptive Backpressure — Shedding Load Gracefully Before Downstream Saturation Hits 100%

The failure mode

Even with a RAM buffer and multiple regions, downstream consumers can saturate: a database exhausts connections, a partner endpoint slows down, or workers fall behind. Fixed rate limits struggle here because the right limit depends on how much the downstream can handle at that moment. The goal is to start slowing intake before the queue is full, and to slow the least important traffic first.

Little’s Law

For a stable system in steady state, Little’s Law states:

L = λ × W

where L is the average number of items in the system, λ is the average arrival rate, and W is the average time an item spends in the system.

The practical use is as a sizing and sanity check. If downstream slows and W rises while λ stays constant, L must rise. More precisely, once the arrival rate exceeds the service rate, the steady-state assumption no longer holds and the queue grows until it hits a limit, which for an unbounded in-memory queue means running out of memory. Queueing delay also climbs sharply as utilization approaches 100%, so waiting until the queue is nearly full is too late. Shedding reduces λ to something the downstream can sustain. A bounded queue turns an eventual crash into an explicit, controlled rejection.

Saturation signal and dynamic token bucket

Define queue saturation as:

S = current queue depth / maximum queue capacity

A token bucket with a fixed refill rate cannot react to S. Instead, scale the refill rate down as S rises between a low and high watermark. For example, with S_low = 0.75 and S_high = 0.90:

R = R_max × (1 − clamp((S − S_low) / (S_high − S_low), 0, 1))

Below S_low, requests get the full rate. Above S_high, low-priority traffic is refused entirely. Many systems use AIMD (additive increase, multiplicative decrease) on the admitted rate instead, borrowing from TCP congestion control. Either way, feed the controller a smoothed signal such as queue depth plus drain rate, so it doesn’t oscillate.

Incoming webhook ──► classify priority ──► token bucket (rate scaled by S)
                                              │
                          ┌───────────────────┴──────────────────┐
                      tokens available                       no tokens
                          │                                      │
                  append to buffer ──► 202                 429 + Retry-After

HTTP semantics: 429 and Retry-After

RFC 6585 defines 429 Too Many Requests. A Retry-After header may accompany it, either as a number of seconds or as an HTTP date. Add jitter so that all clients don’t retry at the same instant. AWS’s “Exponential Backoff And Jitter” article is a good reference on why. A formula consistent with the thresholds below:

delay = min(MaxBackoff, BaseDelay × 2^(k × p)) × jitter

where p = clamp((S − S_low) / (1 − S_low), 0, 1), k is a steepness constant and jitter is a random factor, for example between 0.5 and 1.0.

package backpressure

import (
    "math"
    "math/rand"
    "strconv"
)

const (
    lowWatermark  = 0.75
    baseDelaySecs = 5.0
    maxDelaySecs  = 120.0
    steepness     = 4.0
)

// RetryAfter returns a Retry-After value in seconds, or ok=false when no
// throttling applies.
func RetryAfter(saturation float64) (value string, ok bool) {
    if saturation <= lowWatermark {
        return "", false
    }
    p := math.Min((saturation-lowWatermark)/(1-lowWatermark), 1.0)
    delay := math.Min(baseDelaySecs*math.Pow(2, steepness*p), maxDelaySecs)
    delay *= 0.5 + 0.5*rand.Float64() // jitter between 50% and 100% of delay
    return strconv.Itoa(int(math.Ceil(delay))), true
}

With these constants, the pre-jitter delay is 5 s at S = 0.75, about 26 s at S = 0.90 and 80 s at S = 1.0.

Not every sender will cooperate

Shedding with 429 only helps if the sender retries. The differences between senders are large:

• Stripe retries failed deliveries in live mode for up to three days with exponential backoff, so a 429 or 503 costs a delay, not data.
• GitHub does not automatically redeliver failed deliveries, so a shed GitHub webhook is lost until someone redelivers it manually or via the API.

So classify by what the sender will do, not only by business importance. Events whose senders won’t retry should be the last to be shed, and should preferably go to the durable buffer or a spill queue. Where you must drop them, log enough to reconcile later. A 503 Service Unavailable with Retry-After is also a reasonable shed response for tiers that are unavailable rather than rate-limited.

Shedding states

State	Queue saturation (S)	Ingress action	Response
Normal	S ≤ 0.75	Accept all traffic into the buffer	202 Accepted
Degraded	0.75 < S ≤ 0.90	Shed low-priority events (for example, analytics pings) with a growing delay	429 with Retry-After of roughly 5–26 s
Tripped	S > 0.90	Reject all non-critical events; keep accepting critical ones until the hard limit	429 with Retry-After of roughly 26–80 s; 503 once the hard limit is hit

These are starting points, not universal constants. Tune watermarks against measured drain rates and your senders’ retry behavior.

Export the state machine and its inputs (queue depth, drain rate, admitted vs. rejected counts per priority, current Retry-After) as Prometheus or OpenTelemetry metrics. Alert on transitions, not only on levels, because a system that flaps between Normal and Degraded is a tuning problem you want to see.

Putting it together

The three layers address different failure modes:

• RAM-backed ingress keeps acknowledgment latency low during bursts, in exchange for a deliberate durability decision (replicate, group-commit or accept the window) and careful sizing of tmpfs, inodes and swap.
• Active-active regions with vector clocks and CRDTs keep ingestion available across regions without cross-continent consensus on the hot path. The cost is eventual consistency, and conflict semantics you must choose per data type.
• Adaptive backpressure keeps downstream failures from becoming memory exhaustion, provided the shedding policy accounts for which senders retry.

No design makes webhook ingestion lossless under all failures. What it can do is make the trade-offs explicit and measurable, so that overload degrades the least valuable traffic first and the data you’ve promised to keep survives.

---

Originally published at https://instawebhook.com/blog/enterprise-webhook-architecture-at-scale-high-throughput-ingress-multi-region-cr