High-volume webhook ingestion sits where network performance, security enforcement and multi-tenant data architecture meet. An event gateway that handles very high request rates has to keep noisy neighbors from starving other tenants and resist low-and-slow denial-of-service attacks. It also has to plan for the post-quantum cryptography transition that regulators are now putting on a calendar.
This article covers four pillars of a modern ingress design: post-quantum signature verification, early packet filtering with XDP and AF_XDP, tenant-aware sharding and queueing, and stateful defenses against slow-client attacks. Each section notes where the common “textbook” claims overreach, so you know what to benchmark in your own environment.
1. Post-Quantum Webhook Signatures: ML-DSA and Hybrid Crypto-Agility
Why the clock is ticking
Classical public-key signatures (RSA, ECDSA over P-256 or secp256k1, Ed25519) rely on problems that Shor’s algorithm would solve on a sufficiently large quantum computer. No such machine exists today, but standards bodies are already setting retirement dates. NIST published the final version of NIST IR 8547 on October 27, 2026. It deprecates quantum-vulnerable public-key algorithms after 2030 and disallows them under NIST standards and guidelines after 2035. This now applies to all parameter strengths, not just the 112-bit ones the 2024 draft singled out.
HMAC-SHA256, the most common webhook signature scheme, is symmetric and is not broken by Shor’s algorithm. Grover’s algorithm only reduces its effective security margin, and a 256-bit MAC stays comfortable. Its drawback is operational rather than quantum: the sender and receiver share a secret, so every receiver that can verify can also forge. Asymmetric signatures avoid that, which is why public-key schemes remain attractive for webhooks.
NIST standardized ML-DSA (Module-Lattice-Based Digital Signature Algorithm) in FIPS 204 in August 2024. It is the standardized form of CRYSTALS-Dilithium. NIST also published FIPS 205 (SLH-DSA, a stateless hash-based signature) as a second signature option, and FN-DSA is still under development.
Sizes and what they cost on the wire
ML-DSA signatures and public keys are far larger than Ed25519’s. The sizes below come from FIPS 204. The base64 column is simple arithmetic and matters because HTTP headers carry text, not raw bytes.
| Algorithm | NIST security category | Public key | Signature | Signature as base64 |
|---|---|---|---|---|
| Ed25519 (classical) | n/a (about 128-bit classical) | 32 bytes | 64 bytes | 88 chars |
| ML-DSA-44 | Category 2 | 1,312 bytes | 2,420 bytes | 3,228 chars |
| ML-DSA-65 | Category 3 | 1,952 bytes | 3,309 bytes | 4,412 chars |
| ML-DSA-87 | Category 5 | 2,592 bytes | 4,627 bytes | 6,172 chars |
ML-DSA-65 is commonly recommended as the general-purpose default. ML-DSA-87 is the parameter set the NSA’s CNSA 2.0 requires for national security systems. Verification speed depends heavily on the implementation and hardware, so benchmark your own library rather than trusting generic per-signature timings.
A dual-header verification pipeline
No widely adopted webhook specification defines post-quantum signature headers today, so the header names below are illustrative. The practical way to migrate without breaking existing senders is to let them send both a classical and a post-quantum signature over the same canonical message (typically timestamp, delivery ID and raw body, which also gives you replay protection).
Incoming HTTP Request
│
┌────────────────┴────────────────┐
▼ ▼
[X-Signature-Classical] [X-Signature-PQC]
(Ed25519 / HMAC-SHA256) (ML-DSA-65)
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Verify classical│ │ Verify PQC │
└────────┬────────┘ └────────┬────────┘
│ (pass) │ (pass)
└────────────────┬────────────────┘
▼
Apply policy, forward upstream
Dual headers are not the only way to do hybrids. The IETF LAMPS working group’s Composite ML-DSA draft defines single combined signature algorithms that pair ML-DSA with RSA, ECDSA, Ed25519 or Ed448, aimed at X.509/PKIX use. When I checked its status, it had reached the RFC Editor queue. NIST’s transition guidance also accommodates dual and composite signatures as long as at least one component is NIST-approved.
Engineering considerations
-
Header limits. Many proxies cap header sizes, and ML-DSA signatures push against those caps.
- NGINX’s default
large_client_header_buffersis4 8k, and each header field must fit in one buffer. A base64 ML-DSA-87 signature (about 6.2 KB) fits, but a hex-encoded one (about 9.3 KB) does not. Raise the buffer size or use base64. - Envoy’s default
max_request_headers_kbis 60 KiB, with a configurable maximum of 96 KiB. Oversized requests get a431. - Every hop must agree. Upstream services with smaller limits will still reject large headers even if your edge accepts them.
- NGINX’s default
-
Public key caching. Keys of up to about 2.6 KB should be cached in process memory (for example in a concurrent map keyed by
kid) with a sensible TTL and an invalidation path for rotation. This avoids a remote key-set lookup on the verification hot path. -
Policy modes. Run the gateway in one of three modes per sender:
- Migration: accept if either signature verifies, and emit metrics on PQC adoption. Treat this strictly as a rollout aid. While it is on, an attacker who can break the classical scheme can still get through.
- Strict hybrid: both signatures must verify. This defends against a break in either algorithm.
- PQC-only: use this after a sender has retired the classical scheme.
- Prevent downgrade. Once a sender has published a PQC key, require the PQC header for that sender. Otherwise an attacker can strip it and present only a classical signature.
2. Early Packet Filtering: XDP and AF_XDP
What XDP does and does not buy you
XDP (eXpress Data Path) runs eBPF programs in the network driver’s receive path, before the kernel allocates an sk_buff. A program can drop a packet (XDP_DROP), hand it to the normal stack (XDP_PASS) or redirect it (XDP_REDIRECT). Cloudflare’s L4Drop system uses XDP for volumetric DDoS mitigation and has reported dropping more than 10 million packets per second on a single core. Typical iptables-style filtering is far slower per core.
Two clarifications keep expectations realistic:
- Filtering is where XDP shines. Legitimate HTTPS webhook traffic at high request rates is usually bound by TLS, parsing and application logic rather than by per-packet kernel overhead. XDP’s biggest win is discarding junk before it consumes those resources.
-
AF_XDP delivers raw Ethernet frames to user space, not HTTP requests. An AF_XDP socket bypasses the kernel stack, so there is no TCP or TLS processing. If you redirect port-443 traffic to an AF_XDP socket, your application must supply its own TCP and TLS implementation. Most gateways instead use XDP only for filtering and rate limiting, then
XDP_PASSsurviving traffic to the normal kernel stack and a conventional proxy.
AF_XDP also depends on the driver. In XDP_DRV mode with zero-copy support, frames land directly in user-space memory (the UMEM). Otherwise the kernel falls back to a mode with an extra copy. A UMEM has a fill ring and a completion ring, and a socket has RX and TX rings. The application must keep the fill ring stocked with free frames, or no packets will arrive.
┌────────────────────────────────────────────────────────┐
│ USER SPACE │
│ ┌──────────────────────────────────────────────┐ │
│ │ Gateway with its own TCP/TLS (AF_XDP socket)│ │
│ └──────────────────────▲───────────────────────┘ │
│ │ RX / fill rings over UMEM │
├─────────────────────────┼──────────────────────────────┤
│ KERNEL SPACE │
│ ┌──────────────────────┴───────────────────────┐ │
│ │ XDP program in NIC driver (drop/pass/redirect)│─────┼─► XDP_PASS → normal
│ └──────────────────────▲───────────────────────┘ │ kernel TCP stack
├─────────────────────────┼──────────────────────────────┤
│ HARDWARE │
│ ┌──────────────────────┴───────────────────────┐ │
│ │ Network Card │ │
│ └──────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────┘
XDP filter (C)
This program validates Ethernet, IPv4 and TCP headers with bounds checks, drops obviously malformed TCP flag combinations, and redirects traffic for the gateway port to an AF_XDP socket bound to the receiving queue. Everything else goes to the normal stack, so DNS, QUIC, ICMP and other host traffic keep working.
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/in.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
#define GATEWAY_PORT 443
struct {
__uint(type, BPF_MAP_TYPE_XSKMAP);
__uint(max_entries, 64);
__type(key, __u32);
__type(value, __u32);
} xsks_map SEC(".maps");
SEC("xdp")
int xdp_webhook_ingress_filter(struct xdp_md *ctx)
{
void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
return XDP_DROP;
if (eth->h_proto != bpf_htons(ETH_P_IP))
return XDP_PASS; /* non-IPv4 handled by the kernel */
struct iphdr *iph = (void *)(eth + 1);
if ((void *)(iph + 1) > data_end)
return XDP_DROP;
__u32 ip_hlen = iph->ihl * 4; /* honor IP options */
if (ip_hlen < sizeof(*iph))
return XDP_DROP;
if (iph->protocol != IPPROTO_TCP)
return XDP_PASS; /* leave UDP/ICMP etc. alone */
struct tcphdr *tcph = (void *)iph + ip_hlen;
if ((void *)(tcph + 1) > data_end)
return XDP_DROP;
if (tcph->dest != bpf_htons(GATEWAY_PORT))
return XDP_PASS;
/* Illegal flag combination (SYN+FIN) */
if (tcph->syn && tcph->fin)
return XDP_DROP;
/* Redirect to the AF_XDP socket for this RX queue; if none is
* registered, fall back to the normal stack (flags = XDP_PASS). */
return bpf_redirect_map(&xsks_map, ctx->rx_queue_index, XDP_PASS);
}
char _license[] SEC("license") = "GPL";
If you keep the kernel TCP stack, replace the final redirect with return XDP_PASS; and use the program purely as a filter and rate limiter.
AF_XDP consumer (Rust sketch)
The sketch below assumes bindgen-generated bindings to libxdp’s xsk.h. Two details matter. First, what arrives is a raw frame, so you must walk the Ethernet, IP and TCP headers to reach any payload. Second, frames must be returned to the fill ring once you are done with them, and in zero-copy mode you cannot recycle a frame while a worker is still reading it.
use std::slice;
const BATCH: u32 = 64;
pub struct XskSocket {
umem_area: *mut std::ffi::c_void,
rx_ring: xsk_ring_cons,
fill_ring: xsk_ring_prod,
}
/// Returns the TCP payload of an Ethernet/IPv4/TCP frame, if well formed.
fn tcp_payload(frame: &[u8]) -> Option<&[u8]> {
if frame.len() < 14 + 20 { return None; }
if u16::from_be_bytes([frame[12], frame[13]]) != 0x0800 { return None; } // IPv4
let ihl = ((frame[14] & 0x0f) as usize) * 4;
if ihl < 20 || frame[14 + 9] != 6 { return None; } // TCP
let tcp = 14 + ihl;
if frame.len() < tcp + 20 { return None; }
let doff = ((frame[tcp + 12] >> 4) as usize) * 4;
if doff < 20 || frame.len() < tcp + doff { return None; }
Some(&frame[tcp + doff..])
}
impl XskSocket {
pub fn process_rx(&mut self) {
let mut rx_idx: u32 = 0;
let rcvd = unsafe { xsk_ring_cons__peek(&mut self.rx_ring, BATCH, &mut rx_idx) };
if rcvd == 0 { return; }
let mut fill_idx: u32 = 0;
let reserved =
unsafe { xsk_ring_prod__reserve(&mut self.fill_ring, rcvd, &mut fill_idx) };
for i in 0..rcvd {
let desc = unsafe { &*xsk_ring_cons__rx_desc(&self.rx_ring, rx_idx + i) };
let frame = unsafe {
slice::from_raw_parts(
xsk_umem__get_data(self.umem_area, desc.addr) as *const u8,
desc.len as usize,
)
};
if let Some(_payload) = tcp_payload(frame) {
// Feed the segment to your user-space TCP/TLS stack here.
}
// Hand the frame back to the kernel (only once nothing still reads it).
if i < reserved {
unsafe {
*xsk_ring_prod__fill_addr(&mut self.fill_ring, fill_idx + i) = desc.addr;
}
}
}
unsafe {
xsk_ring_cons__release(&mut self.rx_ring, rcvd);
xsk_ring_prod__submit(&mut self.fill_ring, reserved);
}
}
}
Production code must also handle partial reservations, retry and wake-up logic, and the TX and completion rings. Libraries such as libxdp or xsk-rs handle much of that bookkeeping.
3. Tenant-Aware Sharding and Isolation
The noisy-neighbor problem
In a shared ingestion pipeline, one tenant’s backfill can fill a common queue and delay everyone else’s events. This is head-of-line (HoL) blocking. The fix is to isolate tenants so that a flood from one only fills that tenant’s own buffer.
Incoming Webhook Stream
│
▼
┌──────────────────────────┐
│ Tenant-aware Router │
└─────────────┬────────────┘
┌──────────────────────────┼──────────────────────────┐
▼ ▼ ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Tenant A (dedicated)│ │ Tenant B (shared) │ │ Shared pool queues │
│ saturated │ │ normal │ │ │
└──────────┬──────────┘ └──────────┬──────────┘ └──────────┬──────────┘
▼ ▼ ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Worker group │ │ Fair-share workers │ │ Fair-share workers │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
Consistent hashing with virtual nodes
Consistent hashing maps tenants to queues so that adding or removing a queue only remaps a small fraction of tenants. Virtual nodes smooth out the distribution. Note that hashing alone does not solve the noisy-neighbor problem. A single hot tenant still hashes to a single queue, which is why the dedicated-queue mechanism below is needed.
use std::collections::BTreeMap;
use std::hash::{Hash, Hasher};
use std::collections::hash_map::DefaultHasher;
pub struct ConsistentHashRing {
nodes: BTreeMap<u64, String>,
}
impl ConsistentHashRing {
pub fn new() -> Self {
Self { nodes: BTreeMap::new() }
}
pub fn add_queue(&mut self, queue_id: &str, vnodes: usize) {
for i in 0..vnodes {
let key = Self::hash_key(&format!("{queue_id}-vnode-{i}"));
self.nodes.insert(key, queue_id.to_string());
}
}
pub fn get_queue(&self, tenant_id: &str) -> Option<&String> {
if self.nodes.is_empty() {
return None;
}
let hash = Self::hash_key(tenant_id);
self.nodes
.range(hash..)
.next()
.map(|(_, q)| q)
.or_else(|| self.nodes.values().next()) // wrap around the ring
}
// NOTE: DefaultHasher's algorithm is not guaranteed stable across Rust
// releases. For a ring shared by several services or persisted routing
// decisions, use a fixed, specified hash (e.g. xxHash or keyed SipHash).
fn hash_key(val: &str) -> u64 {
let mut hasher = DefaultHasher::new();
val.hash(&mut hasher);
hasher.finish()
}
}
Dedicated queues, deficit round robin and backpressure
These policies work with brokers such as Apache Kafka, RabbitMQ or NATS JetStream. The thresholds below are starting points to tune against your own traffic, not standards.
- Dedicated queues for heavy tenants. When a tenant’s share of ingress stays above a threshold (for example 15% of capacity over a moving window), move it to its own queue or partition set, with its own consumer group, so its backlog cannot delay others.
- Deficit round robin (DRR) scheduling. DRR, introduced by Shreedhar and Varghese in 1995, gives each queue a byte “quantum” per round and carries unused credit forward. Weighted variants give larger tenants bigger quanta. Workers move on to the next tenant when a tenant exhausts its credit, so no tenant monopolizes a worker.
-
Per-tenant backpressure. When a tenant’s queue depth crosses a high-water mark (for example 85%), respond to that tenant’s requests with
429 Too Many Requestsand aRetry-Afterheader, both defined in HTTP standards (RFC 6585 and RFC 9110). Other tenants are unaffected. Better still, keep the ingress path thin: validate, persist to a durable log, return2xxquickly, and do the real work asynchronously, so backpressure is a last resort rather than the normal path.
4. Stateful Defenses Against Slow-Client Attacks
Webhook endpoints accept POST requests from the open internet, which makes them targets for application-layer resource exhaustion.
Attack mechanics
- Slowloris-style attacks. The client opens many connections and sends headers (or body bytes) extremely slowly, never finishing the request. Each connection holds a socket, a file descriptor and some memory until a timeout fires, so a modest number of connections can exhaust a server’s capacity.
-
Unbounded chunked bodies. With
Transfer-Encoding: chunkedthere is noContent-Lengthup front, so a client can keep streaming chunks. A proxy that buffers the body without a cumulative cap can run out of memory or disk.
HTTP/2 brings related abuse patterns, such as the Rapid Reset attack disclosed in 2023 (CVE-2023-44487). Those need protocol-specific limits on concurrent streams and stream resets on top of what follows.
Decision model
Raw TCP Stream Connection
│
▼
┌─────────────────────────────────────┐
│ Connection tracking (per socket) │
└──────────────────┬──────────────────┘
│
┌────────────────────────┴────────────────────────┐
▼ ▼
[Header deadline] [Total body size]
Headers complete within N seconds? Cumulative bytes ≤ limit?
│ │
no ──► 408 / close no ──► 413 / close
Use the knobs your proxy already has
Before writing custom code, set the built-in limits:
-
NGINX:
client_header_timeout(default 60s) bounds the time to receive the full request header.client_body_timeout(default 60s) applies between two successive reads of the body, not to the body as a whole, so a trickle can slip under it.client_max_body_size(default 1m) is enforced for chunked bodies too and returns413. Lowering the header and body timeouts to the single-digit to tens-of-seconds range is typical hardening. -
Envoy: the HTTP connection manager exposes
request_headers_timeout,request_timeoutand idle timeouts, along withmax_request_headers_kbfor header size. -
Minimum data rates: NGINX has no direct minimum-rate directive. Apache’s
mod_reqtimeoutsupports minimum transfer rates, and a custom proxy can enforce the same idea.
Choose values from your real traffic. Legitimate webhook senders usually deliver small bodies in well under a second, so a header deadline of a few seconds and a tight body timeout are reasonable for many deployments. Senders on slow links, or with large payloads, may need more room.
Example: minimum-rate and size enforcement (Rust, Tokio)
This sketch reads a request body while enforcing a per-read timeout, a minimum average rate over a rolling window, and a cumulative size cap. A real implementation would also parse HTTP framing (Content-Length and chunked encoding) to know when the request is complete, and apply the rate check only while a request is still in flight.
use std::io;
use tokio::io::AsyncReadExt;
use tokio::net::TcpStream;
use tokio::time::{timeout, Duration, Instant};
const MAX_PAYLOAD_BYTES: usize = 10 * 1024 * 1024; // 10 MB cumulative cap
const MIN_RATE_BYTES_PER_SEC: u64 = 1024; // 1 KB/s average
const RATE_WINDOW: Duration = Duration::from_secs(5);
const PER_READ_TIMEOUT: Duration = Duration::from_secs(5);
pub async fn read_body_guarded(stream: &mut TcpStream) -> io::Result<Vec<u8>> {
let mut body = Vec::new();
let mut buf = [0u8; 8192];
let mut window_start = Instant::now();
let mut window_bytes: u64 = 0;
loop {
let n = match timeout(PER_READ_TIMEOUT, stream.read(&mut buf)).await {
Ok(Ok(0)) => return Ok(body), // peer closed
Ok(Ok(n)) => n,
Ok(Err(e)) => return Err(e),
Err(_) => {
return Err(io::Error::new(io::ErrorKind::TimedOut, "read timeout"));
}
};
if body.len() + n > MAX_PAYLOAD_BYTES {
return Err(io::Error::new(io::ErrorKind::InvalidData, "payload too large"));
}
body.extend_from_slice(&buf[..n]);
window_bytes += n as u64;
// Check the average rate once per full window, not per read.
let elapsed = window_start.elapsed();
if elapsed >= RATE_WINDOW {
if window_bytes < MIN_RATE_BYTES_PER_SEC * elapsed.as_secs() {
return Err(io::Error::new(io::ErrorKind::TimedOut, "client too slow"));
}
window_start = Instant::now();
window_bytes = 0;
}
}
}
Judging each read against a fixed size, as simpler implementations do, would kill legitimate small payloads. Measuring the average over a window avoids that.
Summary
A resilient high-throughput webhook ingress needs defenses at every layer, and each layer has a realistic scope:
- Protocol layer: Plan for ML-DSA (FIPS 204) alongside your existing signatures, using dual headers or composite signatures. Enforce strict verification and downgrade protection once senders migrate. NIST’s final IR 8547 sets 2030 and 2035 as the deprecation and disallowance dates for quantum-vulnerable public-key algorithms.
- Network layer: Use XDP to drop junk and rate-limit before it reaches the kernel stack, where published results show on the order of 10 million dropped packets per second per core. Reserve AF_XDP for cases where you are prepared to bring your own TCP and TLS stack.
- Routing layer: Combine consistent hashing, dedicated queues for heavy tenants, deficit round robin scheduling and per-tenant backpressure so one sender cannot block the rest.
- Transport layer: Set strict header and body timeouts, size caps and minimum-rate checks, using your proxy’s native controls first and custom code only where gaps remain.
References
- NIST FIPS 204, Module-Lattice-Based Digital Signature Standard: https://csrc.nist.gov/pubs/fips/204/final
- Encryption Consulting, What Is ML-DSA (FIPS 204)?: https://www.encryptionconsulting.com/education-center/ml-dsa-fips-204/
- Help Net Security, NIST IR 8547 roadmap coverage: https://www.helpnetsecurity.com/2026/10/28/nist-post-quantum-cryptography-migration-roadmap/
- PostQuantum.com, NIST IR 8547 final: https://postquantum.com/security-pqc/nist-ir-8547-final/
- IETF LAMPS, Composite ML-DSA for use in X.509 PKI: https://datatracker.ietf.org/doc/draft-ietf-lamps-pq-composite-sigs/
- Linux kernel documentation, AF_XDP: https://docs.kernel.org/networking/af_xdp.html
- Cloudflare, L4Drop: XDP DDoS Mitigations: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/
- NGINX,
ngx_http_core_moduledocumentation: https://nginx.org/en/docs/http/ngx_http_core_module.html - Envoy, HTTP connection manager configuration: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto
- RFC 6585 (HTTP 429): https://www.rfc-editor.org/rfc/rfc6585
- RFC 9110 (HTTP Semantics,
Retry-After): https://www.rfc-editor.org/rfc/rfc9110
Originally published at https://instawebhook.com/blog/high-throughput-post-quantum-webhook-ingress-engineering
Top comments (0)