DEV Community: InstaWebhook

Ingress Security & Cryptographic Verification: Enterprise Webhook Hardening Guide

InstaWebhook — Fri, 06 Nov 2026 02:20:06 +0000

Webhooks are how platforms such as Stripe, GitHub and Shopify tell your systems that something happened: a payment cleared, a repository changed, an order shipped. Each one is an unsolicited HTTP request that your infrastructure has to authenticate before acting on it. Every webhook therefore has two security questions: is this payload really from the sender (ingress verification), and, if you are the sender, is this delivery really reaching the intended customer (outbound path integrity)?

This guide covers both. It looks at how to prepare webhook signature verification for post-quantum cryptography with an ML-DSA (FIPS 204) dual-signature verifier, and how senders can reduce the risk of BGP hijacking and DNS spoofing with RPKI, DNSSEC and DANE. It also says where each control stops helping, because several popular claims about these technologies are overstated.

Information is current as of November 2026.

1. Where Webhook Signatures Stand Today

The major providers’ documented schemes are still symmetric HMAC-SHA256, and the details differ:

Stripe sends a Stripe-Signature header of the form t=<timestamp>,v1=<hex>. The signed string is <timestamp>.<raw body>, and the official libraries reject timestamps outside a default tolerance of five minutes. During secret rotation the header can carry several v1 values.
GitHub sends X-Hub-Signature-256: sha256=<hex>, computed over the raw body only. There is no signed timestamp, so receivers should deduplicate on the delivery ID to limit replay.
Shopify sends a base64-encoded HMAC-SHA256 in X-Shopify-Hmac-SHA256, with no timestamp in the signed data.

HMAC-SHA256 is not broken, and it is not threatened by quantum computers in any practical sense. Quantum algorithms give only a quadratic speedup against symmetric primitives. The post-quantum standards bodies’ deprecation schedules target public-key algorithms (RSA, ECDSA, EdDSA, Diffie-Hellman), not SHA-2 or AES-256.

HMAC does have structural limits at enterprise scale:

Secret distribution and rotation. Every sender–receiver pair needs its own shared secret, and every receiver has to store it. A leak at either end lets an attacker mint valid signatures.
No non-repudiation. Because both parties hold the key, a signature cannot prove to a third party which side created it. That matters if you archive webhooks as audit evidence.
Receiver-side forgery. A compromised receiver can fabricate “valid” events as easily as verify them.

Asymmetric signatures fix these problems: the sender holds a private key, and receivers hold only a public key. Today that means Ed25519 or ECDSA, which are exactly the algorithms a future quantum computer would break. This is the real argument for post-quantum signatures in webhooks, and it is a narrower claim than “HMAC is obsolete”.

2. Post-Quantum Signatures for Webhooks: The Honest Threat Model

In August 2024, NIST published its first three finalized post-quantum standards:

FIPS 203: ML-KEM (derived from CRYSTALS-Kyber), a key-encapsulation mechanism
FIPS 204: ML-DSA (derived from CRYSTALS-Dilithium), a lattice-based signature scheme
FIPS 205: SLH-DSA (derived from SPHINCS+), a stateless hash-based signature scheme

NIST’s transition plan, IR 8547, was finalized on October 27, 2026. It deprecates quantum-vulnerable public-key algorithms after 2030 and disallows them after 2035. The final version removes the draft’s distinction between key sizes, so RSA, ECDSA and EdDSA are deprecated at every approved strength. NIST has said the 2035 disallowance is not intended to cover hybrid modes that pair an approved post-quantum algorithm with a classical one. US federal directives set earlier targets for high-value assets: post-quantum key establishment by the end of 2030 and post-quantum signatures by the end of 2031.

Two points keep this in proportion:

Signature forgery needs a quantum computer at verification time. A webhook signed today with ECDSA cannot be forged retroactively by a machine that appears in 2035 and used to authenticate a request that was accepted in 2026. The “harvest now, decrypt later” threat applies mainly to confidentiality (TLS key exchange), not to short-lived authentication tokens. Signatures that must stay verifiable for years, such as archived audit trails, are the exception.
Compliance deadlines are the practical driver. If you sell to regulated or government customers, you will be asked about your cryptographic inventory and migration plan well before a quantum computer exists.

The sensible goal is therefore cryptographic agility: build verification so that the algorithm is a configuration choice, then migrate on your own schedule.

3. Designing a Dual-Signature Verification Proxy

A dual-signature verifier sits at your ingress, whether as a gateway plugin or as middleware in front of your services. It authenticates each request against whichever signature schemes the sender supplies, applies a policy, and forwards only verified requests.

            Incoming webhook (raw body + headers)
                            │
                            ▼
              ┌───────────────────────────┐
              │  Dual-signature verifier  │
              │  1. check timestamp       │
              │  2. build signed message  │
              └───────┬───────────┬───────┘
                      ▼           ▼
             HMAC-SHA256      ML-DSA-65
             (shared secret)  (sender public key)
                      │           │
                      └─────┬─────┘
                            ▼
                    Policy evaluation
          (accept either / require both / require PQ)
                            │
                            ▼
              Forward to internal service

An illustrative request during a migration window looks like this. The header names are examples, because no industry standard for dual webhook signatures currently exists:

POST /webhooks/events HTTP/1.1
Host: api.example.com
Content-Type: application/json
Webhook-Timestamp: 1767225600
Webhook-Signature-HMAC: 9f8a...4b2e
Webhook-Signature-MLDSA65: MEUCIQ...(base64, about 4.4 kB)

Three design rules matter more than the algorithms:

Verify the raw bytes. Parsing and re-serializing JSON before verification breaks signatures and opens parser-differential attacks.
Sign a timestamp together with the body (as Stripe does) and enforce a tolerance window. Also deduplicate on an event or delivery ID, because a timestamp window only narrows replay and does not eliminate it.
Use a domain-separation context for ML-DSA. FIPS 204 lets the signer bind a context string of up to 255 bytes into the signature, which prevents a signature made for one purpose from being replayed for another.

Choosing a policy

The policy you pick determines what the dual signature actually buys you:

Policy	Passes when	Use it for
`AcceptEither`	HMAC or ML-DSA is valid	Early migration and telemetry. This adds no forgery resistance beyond HMAC alone: anyone who can strip the ML-DSA header falls back to HMAC.
`RequireBoth`	HMAC and ML-DSA are both valid	Hybrid assurance. An attacker must compromise both the shared secret and the signing key.
`RequirePQ`	ML-DSA is valid	End state, once all senders have migrated.

4. Implementation in Go

Go 1.27, released in August 2026, ships crypto/mldsa in the standard library. It implements FIPS 204 with MLDSA44, MLDSA65 and MLDSA87 parameter sets, and Verify accepts an Options value carrying the context string. The following verifier uses only the standard library:

package webhookverify

import (
    "crypto/hmac"
    "crypto/mldsa"
    "crypto/sha256"
    "encoding/base64"
    "encoding/hex"
    "errors"
    "strconv"
    "time"
)

type Policy int

const (
    AcceptEither Policy = iota // migration: HMAC or ML-DSA
    RequireBoth                // hybrid: HMAC and ML-DSA
    RequirePQ                  // end state: ML-DSA only
)

// Domain-separation context bound into every ML-DSA signature.
const pqContext = "webhook-v1"

var (
    ErrBadTimestamp     = errors.New("invalid timestamp")
    ErrStale            = errors.New("timestamp outside tolerance window")
    ErrInvalidSignature = errors.New("signature verification failed")
)

type Verifier struct {
    HMACSecret []byte           // may be nil once HMAC is retired
    PQKey      *mldsa.PublicKey // parsed once with mldsa.NewPublicKey(mldsa.MLDSA65(), raw)
    Tolerance  time.Duration    // e.g. 5 * time.Minute
    Policy     Policy
}

// Verify checks the raw request body against the timestamp and signature headers.
// The signed message is "<timestamp>.<raw body>".
func (v *Verifier) Verify(body []byte, tsHeader, hmacHex, pqBase64 string, now time.Time) error {
    ts, err := strconv.ParseInt(tsHeader, 10, 64)
    if err != nil {
        return ErrBadTimestamp
    }
    age := now.Sub(time.Unix(ts, 0))
    if age > v.Tolerance || age < -v.Tolerance {
        return ErrStale
    }

    msg := append([]byte(tsHeader+"."), body...)
    hmacOK := v.checkHMAC(msg, hmacHex)
    pqOK := v.checkPQ(msg, pqBase64)

    switch v.Policy {
    case AcceptEither:
        if hmacOK || pqOK {
            return nil
        }
    case RequireBoth:
        if hmacOK && pqOK {
            return nil
        }
    case RequirePQ:
        if pqOK {
            return nil
        }
    }
    // Log which check failed internally; do not tell the caller.
    return ErrInvalidSignature
}

func (v *Verifier) checkHMAC(msg []byte, sigHex string) bool {
    if len(v.HMACSecret) == 0 {
        return false
    }
    got, err := hex.DecodeString(sigHex)
    if err != nil {
        return false
    }
    mac := hmac.New(sha256.New, v.HMACSecret)
    mac.Write(msg)
    return hmac.Equal(mac.Sum(nil), got) // constant-time comparison
}

func (v *Verifier) checkPQ(msg []byte, sigB64 string) bool {
    if v.PQKey == nil {
        return false
    }
    sig, err := base64.StdEncoding.DecodeString(sigB64)
    if err != nil {
        return false
    }
    return mldsa.Verify(v.PQKey, msg, sig, &mldsa.Options{Context: pqContext}) == nil
}

Also cap the request body size before reading it, and cache the parsed public key rather than parsing it on every request.

5. The Operational Cost: Sizes, Headers and Key Distribution

ML-DSA verification is not computationally heavy, but the objects are large. The FIPS 204 sizes for ML-DSA-65 are below. The ECDSA figures are the raw encodings used by JOSE’s ES256.

	HMAC-SHA256	ECDSA P-256	ML-DSA-65
Verification key	Shared secret	64 bytes	1,952 bytes
Signature / tag	32 bytes	64 bytes	3,309 bytes
Signature as base64	44 characters	88 characters	4,412 characters

Benchmark verification on your own hardware rather than relying on published microsecond figures, because results vary widely by CPU, library and implementation. Signature size is the more predictable cost.

A 4.4 kB header fits comfortably in most defaults, but not in all of them. Per-line and total limits vary: nginx’s large_client_header_buffers defaults to four buffers of 8 KB (and a single header line must fit within one buffer), Apache’s LimitRequestFieldSize defaults to 8,190 bytes, and Envoy’s max_request_headers_kb defaults to 60 KiB. Older or constrained intermediaries, such as embedded proxies and legacy application servers, may have much lower limits. Test the full path from sender to verifier before you rely on a header-carried ML-DSA signature. If it does not fit, a signature in the request body envelope or a trailer is an alternative.

Publishing public keys

Receivers need a way to fetch and rotate sender public keys. In May 2026 the IETF published RFC 9964, which defines how ML-DSA keys and signatures are expressed in JOSE and COSE. It introduces a new key type, AKP (Algorithm Key Pair), where the algorithm is named in alg and the public key is carried base64url-encoded in pub. A published JWKS for an ML-DSA-65 key therefore looks like this, not like the OKP type used for Ed25519:

{
  "keys": [
    {
      "kty": "AKP",
      "alg": "ML-DSA-65",
      "kid": "webhooks-2026-11",
      "pub": "<base64url, 1,952 raw bytes, about 2.6 kB encoded>"
    }
  ]
}

Receivers should cache keys by kid, refresh them on a schedule or on an unknown kid (with rate limiting), and never fetch the JWKS inside the request-verification path.

6. Outbound Path Integrity: BGP Hijacking and DNS Spoofing

If you send webhooks, your concern is different: you are making an outbound HTTPS request to a customer-controlled URL, and you want to be sure it reaches that customer.

Why this threat is real

The best-known illustration is the April 24, 2018 incident. Attackers announced more-specific BGP prefixes covering Amazon Route 53 name servers for about two hours, redirected DNS queries to a rogue DNS server, and pointed myetherwallet.com at a phishing site, draining roughly $150,000 in Ethereum. AWS stated that neither it nor Route 53 had been hacked or compromised. The phishing site used a self-signed certificate that browsers flagged, which is why the attack was only partly successful.

A harder variant targets certificate issuance. If an attacker hijacks routes to a victim’s server, a Certificate Authority performing a single-vantage-point domain check may issue the attacker a valid certificate for the victim’s name, and TLS then validates against the attacker’s server. This was demonstrated by Princeton researchers and prompted the industry’s response.

What has already improved

The CA/Browser Forum’s Ballot SC-067 requires Multi-Perspective Issuance Corroboration (MPIC): CAs must check domain control from several network locations and compare the results. It took effect for publicly trusted TLS certificates on March 15, 2025, with enforcement strengthening through September 2025 and further requirements scheduled for 2026. Let’s Encrypt had already been using multi-perspective validation in production since 2020. This sharply reduces the “hijack the route, win the certificate” attack against public CAs, but it does not eliminate the underlying routing problem.

Defense layers and their limits

Layer	What it protects	What it does not do
RPKI route origin validation (ROV)	Detects announcements where the origin AS is not authorized by a ROA for that prefix	Does nothing for prefixes without ROAs. It does not stop forged-origin hijacks that keep the legitimate AS at the end of the path, and it does not validate the full AS path.
DNSSEC	Authenticates DNS answers, so spoofed records are rejected	Does nothing to routing, and only works for zones that are signed.
DANE (TLSA records)	Pins which certificate or key a host may present, taken from DNSSEC-signed data	Not supported by web browsers. It is practical only for software you control, such as a webhook dispatcher.
Mutual TLS or signed payloads	Lets the receiver prove identity independently of the path	Requires customer cooperation.

Adoption is improving: IPinfo reported in October 2026 that roughly 70% of routed prefixes were RPKI-valid, up from about 54% a year earlier (figures vary by methodology and by region). Many networks still do not publish ROAs, and ROV only helps where networks along the path actually reject invalid routes.

The practical consequence is that DANE with DNSSEC gives a sender the strongest guarantee, because it ties the expected key to the domain name rather than to the network path. If a hijacker diverts your connection, the rogue server cannot present a key matching the DNSSEC-signed TLSA record, and the connection fails. RPKI checks are useful defense in depth, mainly for detecting anomalies and limiting the exposure window, but they should not be your only control.

Step 1: Check route origin validity with RPKI

Routinator, an open-source RPKI validator, exposes an HTTP API (default 127.0.0.1:8323) with a validity endpoint for a given origin AS and prefix. It also speaks the RPKI-to-Router protocol (RTR, default port 3323) to routers, which is a separate interface from the HTTP API.

The check needs a prefix and an origin AS that you observe, for example from your own BGP feed or a route collector such as RIPE RIS or RouteViews. It does not take a bare IP address and an “expected” AS.

import requests

ROUTINATOR = "http://127.0.0.1:8323"

def rpki_state(origin_asn: int, prefix: str) -> str:
    """Return 'valid', 'invalid' or another state (e.g. not-found) from a local Routinator."""
    resp = requests.get(
        f"{ROUTINATOR}/api/v1/validity/AS{origin_asn}/{prefix}",
        timeout=2,
    )
    resp.raise_for_status()
    return resp.json()["validated_route"]["validity"]["state"].lower()

# Policy sketch:
#   invalid   -> do not connect; alert
#   valid     -> proceed
#   not found -> proceed, but rely on DANE/mTLS and log it
#   error     -> decide per your risk tolerance (fail open vs closed)

Step 2: Pin certificates with DNSSEC and DANE

DANE is defined in RFC 6698, with operational guidance in RFC 7671. A TLSA record published at _443._tcp.<host> states which certificate or public key a server must present. The common “DANE-EE” form 3 1 1 means: usage 3 (end-entity certificate, no CA trust required), selector 1 (SubjectPublicKeyInfo), matching type 1 (SHA-256).

_443._tcp.api.example.com. IN TLSA 3 1 1 <SHA-256 of the server's SPKI, hex>

The sender below resolves the TLSA record, requires the resolver to report DNSSEC validation, completes the TLS handshake, and compares the presented key to the pinned hash on the same connection that then carries the payload. Checking on one connection and sending on another would leave a gap in which the destination could change. The connection still uses normal certificate validation, which is stricter than DANE-EE requires but harmless as defense in depth.

import hashlib
import http.client
import ssl

import dns.flags
import dns.resolver
from cryptography import x509
from cryptography.hazmat.primitives import serialization


class PathValidationError(Exception):
    pass


def tlsa_records(host: str, port: int = 443, resolver_ip: str = "127.0.0.1"):
    """Fetch TLSA records via a *trusted local validating resolver*."""
    r = dns.resolver.Resolver(configure=False)
    r.nameservers = [resolver_ip]
    r.use_edns(0, dns.flags.DO, 1232)  # request DNSSEC data
    answer = r.resolve(f"_{port}._tcp.{host}", "TLSA")
    # The AD bit is only meaningful if the path to the resolver is trusted
    # (loopback or a private link) and the resolver itself validates DNSSEC.
    if not (answer.response.flags & dns.flags.AD):
        raise PathValidationError("TLSA answer was not DNSSEC-validated (AD flag missing)")
    return [(a.usage, a.selector, a.mtype, a.cert) for a in answer]


def post_with_dane(host: str, path: str, body: bytes, port: int = 443):
    records = tlsa_records(host, port)

    conn = http.client.HTTPSConnection(
        host, port, context=ssl.create_default_context(), timeout=5
    )
    conn.connect()
    try:
        der = conn.sock.getpeercert(binary_form=True)
        spki = x509.load_der_x509_certificate(der).public_key().public_bytes(
            serialization.Encoding.DER,
            serialization.PublicFormat.SubjectPublicKeyInfo,
        )
        digest = hashlib.sha256(spki).digest()

        # Support only DANE-EE / SPKI / SHA-256 (3 1 1) in this sketch.
        if not any(u == 3 and s == 1 and m == 1 and c == digest for u, s, m, c in records):
            raise PathValidationError("Server key does not match DNSSEC-signed TLSA record")

        conn.request("POST", path, body=body, headers={"Content-Type": "application/json"})
        resp = conn.getresponse()
        return resp.status, resp.read()
    finally:
        conn.close()

Operational caveats:

The resolver is part of your trust base. Run a validating resolver locally or on a protected link. The AD flag from an untrusted resolver proves nothing.
Many customers will not have DNSSEC or TLSA records. Treat DANE as an opt-in tier (for example, for customers who publish TLSA records), with a defined fallback policy for everyone else.
Rotate carefully. Publish the new TLSA record before switching keys, and keep both during the overlap, or deliveries will fail during rotation.

7. Comparing Approaches

Concern	Typical today	Hardened design
Payload authenticity	HMAC-SHA256 with a shared secret	HMAC plus ML-DSA, moving to ML-DSA only
Non-repudiation	None (symmetric key)	Public-key signature
Replay protection	Timestamp window (Stripe) or none (GitHub, Shopify)	Signed timestamp, tolerance window and event-ID deduplication
Algorithm change	Code change per integration	Configuration-driven policy
Route hijack detection	None at the application layer	RPKI origin check (where ROAs exist)
DNS integrity	Unauthenticated lookups	DNSSEC-validated resolution
Endpoint identity	Public CA trust (hardened by MPIC)	DANE pin, plus optional mutual TLS

8. Action Plan

Inventory your webhook flows. List every provider you receive from and every customer endpoint you send to, with the signature scheme and TLS configuration of each. A cryptographic inventory is the first step in NIST-aligned migration plans.
Fix the basics first. Verify HMACs over raw bytes with constant-time comparison, enforce a timestamp window where the provider signs one, deduplicate by event ID, and support multiple active secrets for rotation.
Make the algorithm configurable. Introduce a verifier with an explicit policy like the one above, and run in AcceptEither while collecting telemetry on which senders support ML-DSA.
Test your header limits. Send a 4.4 kB signature header through every proxy, CDN and load balancer on the path before relying on it.
Publish PQ keys when you send. Offer ML-DSA public keys in an RFC 9964 JWKS alongside your classical keys, and run dual signatures during the transition.
Tier your outbound protection. Apply RPKI origin checks for observability, DNSSEC and DANE where customers publish TLSA records, and offer mutual TLS or signed payloads for high-value integrations.
Plan the end state. Move from AcceptEither to RequireBoth, then to RequirePQ once all senders have migrated, and retire shared secrets.

Conclusion

HMAC-signed webhooks remain sound against quantum computers; the case for ML-DSA rests on asymmetric key management, non-repudiation and meeting 2030–2035 public-key deprecation timelines. Network-path attacks are real and documented, but RPKI alone only covers part of the problem. The strongest sender-side protection comes from binding the expected certificate key to the domain name with DNSSEC and DANE, with RPKI and multi-perspective certificate issuance as supporting layers. Build the verifier so the algorithm is a policy choice, and then migrate on your schedule rather than someone else’s.

References

NIST FIPS 203, 204 and 205 (August 2024): https://csrc.nist.gov/pubs/fips/204/final
NIST IR 8547, final report (October 27, 2026), summarized at https://postquantum.com/security-pqc/nist-ir-8547-final/ and https://labs.cloudsecurityalliance.org/research/csa-research-note-nist-ir8547-pqc-transition-20261030-csa-st/
RFC 9964, ML-DSA for JOSE and COSE: https://www.rfc-editor.org/rfc/rfc9964.html
Go crypto/mldsa package: https://pkg.go.dev/crypto/mldsa
Webhook signature schemes for Stripe and GitHub: https://blog.openreplay.com/verify-webhook-signature/
RFC 6698 (DANE) and RFC 7671 (DANE operational guidance): https://www.rfc-editor.org/rfc/rfc6698
2018 Route 53 / MyEtherWallet incident: https://www.helpnetsecurity.com/2018/04/25/myetherwallet-dns-hijacking/ and https://www.theregister.com/2018/04/24/myetherwallet_dns_hijack/
Multi-Perspective Issuance Corroboration (Ballot SC-067): https://www.securityweek.com/new-issuance-requirements-improve-https-certificate-validation/
Routinator HTTP API usage: https://oneuptime.com/blog/post/2026-03-20-monitor-rpki-validation-ipv6/view

Originally published at https://instawebhook.com/blog/ingress-security-cryptographic-verification-enterprise-webhook-hardening-guide

Architecting a Bring-Your-Own (BYO) Database Schema for Webhook Ingress: Balancing Customer Data Control and Metadata

InstaWebhook — Thu, 05 Nov 2026 02:17:50 +0000

Webhooks connect payment gateways, identity providers and SaaS platforms to the systems that act on their events. As enterprise adoption grows, two requirements pull against each other: data sovereignty and operational orchestration.

Customers increasingly want sensitive payloads (PII, financial records, health data) to stay inside infrastructure they control. Webhook relays and automation platforms still need to coordinate delivery, track retries and report health. This article describes a Bring-Your-Own (BYO) database pattern that handles both. Raw webhook bodies are persisted only in a customer-controlled PostgreSQL database. An orchestration layer sees operational metadata only.

Along the way, it covers where the pattern’s guarantees stop. A pattern like this is easy to oversell, so the limits are stated plainly.

1. The Architectural Dilemma: Sovereignty vs. Orchestration

The conventional approach is a hosted webhook router. It receives provider requests, stores the bodies in a shared multi-tenant datastore, and runs processing and retries centrally. This is operationally simple, but it has costs:

Compliance scope: If a vendor stores or processes regulated data on your behalf, it becomes part of your compliance footprint. HIPAA generally requires a business associate agreement with such a vendor, and GDPR requires a data processing agreement. PCI DSS applies to any system that stores, processes or transmits cardholder data. Keeping payloads out of a third party’s storage can shrink that footprint, but it does not remove your own obligations.
Trust: Many security teams are reluctant to send raw event bodies to a multi-tenant system, or to share database credentials with one.

The BYO Infrastructure Paradigm

The pattern splits payload persistence from job orchestration:

Data plane (customer infrastructure): A PostgreSQL database holds the raw event store, headers and business processing tables. Workers run here too.
Orchestration plane (relay or control plane): It manages routing rules, delivery status, retry policy, alerting and dashboards. It works from metadata such as job IDs, status codes, timings and error classes, not payload contents.

One constraint shapes the whole design. Providers sign the exact raw bytes of the request body. Stripe, for example, signs the raw payload and sends the signature in a Stripe-Signature header with a timestamp, and its libraries reject timestamps outside a default five-minute tolerance. Verification must therefore happen wherever the raw body is available. There are two ways to honor “the orchestrator never stores the payload”:

Customer-side ingress: The provider’s webhook points at an endpoint you host. It verifies the signature, writes the event to your database and reports metadata to the control plane.
Pass-through relay: A relay receives the request, verifies it, and forwards it to your ingress without persisting the body. The relay still sees the bytes in memory in transit, so this is a weaker guarantee than “never possesses the payload.” Describe it accurately to customers.

                  ┌───────────────────────────────┐
                  │   ORCHESTRATION PLANE         │
                  │  - Routing & retry policy     │
                  │  - Delivery status & metrics  │
                  │  - Alerting & dashboards      │
                  └──────────────▲────────────────┘
                                 │ metadata only
                                 │ (job_id, status, timings, error class)
┌──────────┐  signed webhook  ┌──┴────────────────────────────────┐
│ Provider │ ───────────────► │   CUSTOMER INFRASTRUCTURE         │
│ (Stripe, │                  │  - Ingress: verify signature      │
│  etc.)   │                  │  - PostgreSQL: events + jobs      │
└──────────┘                  │  - Workers: lease & process jobs  │
                              └───────────────────────────────────┘

2. Designing the Minimal-Privilege PostgreSQL Schema

The schema below stores events and jobs in a dedicated schema. It uses gen_random_uuid(), which is built into core PostgreSQL since version 13. The uuid-ossp extension is therefore unnecessary for random UUIDs. pgcrypto is only needed if you use its functions.

Two design choices differ from a naive implementation:

Raw body is stored as bytea. Signature verification needs the original bytes, and JSONB does not preserve whitespace, key order or duplicate keys. Keep the original body for audit and replay. Add a JSONB column if you want to query fields.
Retries are scheduled in the row. A next_attempt_at column supports backoff, and a bounded max_attempts stops infinite retry loops.

CREATE SCHEMA IF NOT EXISTS webhook_ingress;

-- 1. Events: raw payloads live only in the customer's database
CREATE TABLE webhook_ingress.events (
    id                 UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    provider           VARCHAR(64)  NOT NULL,
    external_event_id  VARCHAR(255) NOT NULL,
    event_type         VARCHAR(128) NOT NULL,

    raw_body           BYTEA        NOT NULL,   -- exact bytes that were signed
    payload            JSONB,                   -- parsed copy for querying
    headers            JSONB        NOT NULL,

    signature_verified BOOLEAN      NOT NULL DEFAULT FALSE,
    received_at        TIMESTAMPTZ  NOT NULL DEFAULT now(),

    CONSTRAINT uk_provider_external_id UNIQUE (provider, external_event_id)
);

CREATE INDEX idx_events_received_at ON webhook_ingress.events (received_at DESC);

-- 2. Jobs: lease-based work queue
CREATE TABLE webhook_ingress.jobs (
    job_id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id         UUID NOT NULL REFERENCES webhook_ingress.events(id) ON DELETE CASCADE,

    status           VARCHAR(16) NOT NULL DEFAULT 'pending'
                     CHECK (status IN ('pending','leased','completed','failed','dead_letter')),

    lease_holder_id  VARCHAR(128),
    lease_expires_at TIMESTAMPTZ,
    next_attempt_at  TIMESTAMPTZ NOT NULL DEFAULT now(),

    attempt_count    INT  NOT NULL DEFAULT 0,
    max_attempts     INT  NOT NULL DEFAULT 5,
    last_error       TEXT,

    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Partial index keeps the polling set small as completed jobs accumulate
CREATE INDEX idx_jobs_claimable
    ON webhook_ingress.jobs (next_attempt_at)
    WHERE status IN ('pending', 'leased');

CREATE INDEX idx_jobs_event_id ON webhook_ingress.jobs (event_id);

Enforcing Least Privilege with Roles

Use separate roles for ingress and workers. PostgreSQL supports column-level UPDATE grants, so workers can be limited to the lease-related columns.

CREATE ROLE webhook_ingress_writer LOGIN;
CREATE ROLE webhook_worker LOGIN;

REVOKE ALL ON SCHEMA webhook_ingress FROM PUBLIC;
GRANT USAGE ON SCHEMA webhook_ingress TO webhook_ingress_writer, webhook_worker;

-- Ingress: insert events and enqueue jobs; no reads, updates or deletes
GRANT INSERT ON webhook_ingress.events, webhook_ingress.jobs TO webhook_ingress_writer;

-- Workers: read events, and update only lease and result columns on jobs
GRANT SELECT ON webhook_ingress.events TO webhook_worker;
GRANT SELECT ON webhook_ingress.jobs   TO webhook_worker;
GRANT UPDATE (status, lease_holder_id, lease_expires_at, next_attempt_at,
              attempt_count, last_error, updated_at)
    ON webhook_ingress.jobs TO webhook_worker;

TRUNCATE, DELETE, DROP and DDL are separate privileges that none of these grants include. A role that is not the table owner cannot alter the schema. Note that the ingress role needs INSERT only. If you use INSERT ... ON CONFLICT DO NOTHING, PostgreSQL also requires SELECT on the columns in the conflict target, so grant column-level SELECT (provider, external_event_id) as well.

3. Job Leasing with `FOR UPDATE SKIP LOCKED`

Workers claim jobs directly from the customer database, with no separate broker such as Redis.

PostgreSQL added SKIP LOCKED in version 9.5. With it, rows that cannot be locked immediately are skipped instead of waited on. The PostgreSQL documentation warns that this yields an inconsistent view of the data, which makes it unsuitable for general-purpose queries. It is intended for queue-like tables, where each consumer should receive a different subset of available rows.

-- Claim up to 5 jobs: pending and due, or leased with an expired lease
WITH candidates AS (
    SELECT job_id
    FROM webhook_ingress.jobs
    WHERE attempt_count < max_attempts
      AND (
            (status = 'pending' AND next_attempt_at <= now())
         OR (status = 'leased'  AND lease_expires_at < now())
      )
    ORDER BY next_attempt_at
    LIMIT 5
    FOR UPDATE SKIP LOCKED
)
UPDATE webhook_ingress.jobs j
SET status           = 'leased',
    lease_holder_id  = $1,                        -- worker identity
    lease_expires_at = now() + interval '5 minutes',
    attempt_count    = j.attempt_count + 1,
    updated_at       = now()
FROM candidates c
WHERE j.job_id = c.job_id
RETURNING j.job_id, j.event_id, j.attempt_count;

The worker then reads the event body by event_id. Keeping the claim and the payload read separate means a claim never copies large payloads through the queue path.

Run the claim as a single statement, as above, or inside one transaction. If the row lock is taken in one transaction and the update happens in another, the lock is released at the first commit and another worker can claim the same row.

Completing, Failing and Dead-Lettering

Completion must verify the lease. Otherwise a slow worker whose lease expired could overwrite the result of the worker that took over.

-- Success
UPDATE webhook_ingress.jobs
SET status = 'completed', lease_holder_id = NULL, lease_expires_at = NULL, updated_at = now()
WHERE job_id = $1 AND lease_holder_id = $2 AND status = 'leased';

-- Failure: retry with backoff, or dead-letter when attempts are exhausted
UPDATE webhook_ingress.jobs
SET status = CASE WHEN attempt_count >= max_attempts THEN 'dead_letter' ELSE 'pending' END,
    next_attempt_at = now() + (interval '30 seconds' * power(2, attempt_count)),
    lease_holder_id = NULL,
    lease_expires_at = NULL,
    last_error = $3,
    updated_at = now()
WHERE job_id = $1 AND lease_holder_id = $2 AND status = 'leased';

Because the claim query skips jobs with attempt_count >= max_attempts, a job whose worker crashed on its final attempt would stay leased forever. Run a periodic reaper that moves such rows to dead_letter:

UPDATE webhook_ingress.jobs
SET status = 'dead_letter', updated_at = now()
WHERE status = 'leased' AND lease_expires_at < now() AND attempt_count >= max_attempts;

What `SKIP LOCKED` Does and Doesn’t Give You

Less lock waiting: Workers move on to unlocked rows instead of blocking behind a competitor.
Crash recovery: If a worker dies, its lease expires and another worker can reclaim the job.
At-least-once delivery, not exactly-once: A worker that is slow but alive can still be processing when its lease expires. Handlers must be idempotent. Set the lease longer than your worst-case processing time, or extend it with a heartbeat.
Throughput limits: A Postgres-backed queue is a good fit for moderate volumes and for teams that already run Postgres. Very high volumes may call for table partitioning, aggressive vacuuming of the jobs table, or a dedicated broker. Benchmark with your own workload.

4. Metadata Synchronization and State Reporting

The orchestration layer still needs to know whether deliveries are succeeding. After each job finishes, the customer-side worker or an outbox publisher sends a small status record to the control plane over HTTPS:

{
  "instance_id": "cust-cluster-us-east-01",
  "job_id": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d",
  "status": "completed",
  "attempt_count": 1,
  "duration_ms": 142,
  "error_class": null,
  "processed_at": "2026-11-05T07:43:00Z"
}

Keep the record minimal and make the contract explicit:

Send: job ID, status, attempt count, duration, a coarse error class and timestamps.
Never send: payload bodies, headers, customer identifiers inside error messages, or stack traces that might echo payload fields.
Treat some metadata as sensitive. Fields like provider and event_type can reveal business activity. Under GDPR, an identifier that links back to a person can itself count as personal data. If that matters to the customer, use opaque job IDs and let them opt out of event-type reporting.

To keep this reporting reliable, write the status change and a pending-sync row in the same database transaction (a transactional outbox). A publisher then ships outbox rows to the control plane and retries on failure, so a network blip never loses a status update.

5. Provider Behavior Your Schema Must Absorb

Your ingress design has to cope with how providers actually deliver. Using Stripe as an example:

Retries: In live mode, Stripe retries failed deliveries with exponential backoff for up to three days. In test and sandbox mode, it makes three attempts over a few hours.
Duplicates and ordering: The same event can arrive more than once, and events are not guaranteed to arrive in order. This is why UNIQUE (provider, external_event_id) matters. Insert with ON CONFLICT DO NOTHING, return 2xx for duplicates, and avoid assuming the latest-arriving event reflects the latest state.
Acknowledge quickly: Stripe’s exact timeout is not publicly documented, but handlers should acknowledge fast. Persist the event and enqueue the job, then return 2xx and do the heavy work asynchronously. This is the main reason for the events/jobs split.

INSERT INTO webhook_ingress.events
    (provider, external_event_id, event_type, raw_body, payload, headers, signature_verified)
VALUES ($1, $2, $3, $4, $5, $6, TRUE)
ON CONFLICT (provider, external_event_id) DO NOTHING
RETURNING id;
-- If a row is returned, insert the matching job in the same transaction.

6. Security Hardening

Network isolation and transport security: Place the database in private subnets. Require TLS on all connections, set ssl_min_protocol_version to at least TLS 1.2 (TLS 1.3 where your clients support it), and have clients connect with sslmode=verify-full. For service-to-service access, use client certificates (mTLS). Certificate pinning is rarely practical for database connections, so prefer a private CA with verification.
Encryption at rest: Use volume or storage encryption from your cloud provider or filesystem layer as the baseline. For field-level protection, prefer application-side or envelope encryption with keys from a KMS.
Be careful with pgcrypto column encryption: pgp_sym_encrypt() returns bytea, so it cannot be inserted into a JSONB column. It needs a bytea column. Also, pgcrypto runs inside the database server, so plaintext and keys reach the server, and a key written as a literal in SQL can end up in statement logs. Disable statement logging for those queries, or encrypt in the application before the data reaches Postgres.
Verify before you store as trusted: Verify the provider’s signature on the raw body, with its timestamp tolerance, before setting signature_verified = TRUE. Reject anything outside the tolerance window.
Retention and deletion: Add a scheduled job to purge old events and completed jobs. Because jobs.event_id cascades on delete, deleting expired events removes their jobs as well. Retention limits are also a common compliance requirement.
Audit access: Log connections and role usage (for example with log_connections, and the pgaudit extension where available) so customers can see exactly what the orchestration layer touched.

Summary: Centralized vs. BYO Architecture

Dimension	Centralized webhook storage	BYO database pattern
Payload storage	Vendor-managed, multi-tenant	Customer-controlled PostgreSQL
Compliance scope	Vendor holds regulated data; agreements such as a BAA or DPA are typically required	Smaller vendor footprint; customer keeps full responsibility for the data
Job coordination	Vendor-operated queues or brokers	Database-native leasing with `FOR UPDATE SKIP LOCKED`
Metadata visibility	Full event visibility for the vendor	Operational status only, by design
Operational burden	Low for the customer	Higher: the customer runs the database, workers and backups
Debugging	Vendor can inspect payloads	Needs customer-side tooling, since the vendor cannot see payloads

The BYO pattern shifts risk and responsibility rather than eliminating them. Customers get stronger control over payloads and a smaller vendor footprint. In exchange, they run more infrastructure and must build idempotent handlers. Combining least-privilege roles, lease-verified job updates and a deliberately small metadata contract gives you reliable delivery orchestration without a third party storing the raw bodies.

References

PostgreSQL documentation: SELECT locking clause (FOR UPDATE, NOWAIT, SKIP LOCKED), https://www.postgresql.org/docs/current/sql-select.html
PostgreSQL documentation: UUID type and gen_random_uuid() (core since PostgreSQL 13), https://www.postgresql.org/docs/current/datatype-uuid.html
PostgreSQL documentation: pgcrypto, https://www.postgresql.org/docs/current/pgcrypto.html
Stripe documentation: webhooks, signatures, retries, https://docs.stripe.com/webhooks
Hookdeck, Guide to Stripe Webhooks: Features and Best Practices, https://hookdeck.com/webhooks/platforms/guide-to-stripe-webhooks-features-and-best-practices

Originally published at https://instawebhook.com/blog/architecting-a-bring-your-own-byo-database-schema-for-webhook-ingress-balancing

High-Throughput & Post-Quantum Webhook Ingress Engineering

InstaWebhook — Wed, 04 Nov 2026 02:18:53 +0000

High-volume webhook ingestion sits where network performance, security enforcement and multi-tenant data architecture meet. An event gateway that handles very high request rates has to keep noisy neighbors from starving other tenants and resist low-and-slow denial-of-service attacks. It also has to plan for the post-quantum cryptography transition that regulators are now putting on a calendar.

This article covers four pillars of a modern ingress design: post-quantum signature verification, early packet filtering with XDP and AF_XDP, tenant-aware sharding and queueing, and stateful defenses against slow-client attacks. Each section notes where the common “textbook” claims overreach, so you know what to benchmark in your own environment.

1. Post-Quantum Webhook Signatures: ML-DSA and Hybrid Crypto-Agility

Why the clock is ticking

Classical public-key signatures (RSA, ECDSA over P-256 or secp256k1, Ed25519) rely on problems that Shor’s algorithm would solve on a sufficiently large quantum computer. No such machine exists today, but standards bodies are already setting retirement dates. NIST published the final version of NIST IR 8547 on October 27, 2026. It deprecates quantum-vulnerable public-key algorithms after 2030 and disallows them under NIST standards and guidelines after 2035. This now applies to all parameter strengths, not just the 112-bit ones the 2024 draft singled out.

HMAC-SHA256, the most common webhook signature scheme, is symmetric and is not broken by Shor’s algorithm. Grover’s algorithm only reduces its effective security margin, and a 256-bit MAC stays comfortable. Its drawback is operational rather than quantum: the sender and receiver share a secret, so every receiver that can verify can also forge. Asymmetric signatures avoid that, which is why public-key schemes remain attractive for webhooks.

NIST standardized ML-DSA (Module-Lattice-Based Digital Signature Algorithm) in FIPS 204 in August 2024. It is the standardized form of CRYSTALS-Dilithium. NIST also published FIPS 205 (SLH-DSA, a stateless hash-based signature) as a second signature option, and FN-DSA is still under development.

Sizes and what they cost on the wire

ML-DSA signatures and public keys are far larger than Ed25519’s. The sizes below come from FIPS 204. The base64 column is simple arithmetic and matters because HTTP headers carry text, not raw bytes.

Algorithm	NIST security category	Public key	Signature	Signature as base64
Ed25519 (classical)	n/a (about 128-bit classical)	32 bytes	64 bytes	88 chars
ML-DSA-44	Category 2	1,312 bytes	2,420 bytes	3,228 chars
ML-DSA-65	Category 3	1,952 bytes	3,309 bytes	4,412 chars
ML-DSA-87	Category 5	2,592 bytes	4,627 bytes	6,172 chars

ML-DSA-65 is commonly recommended as the general-purpose default. ML-DSA-87 is the parameter set the NSA’s CNSA 2.0 requires for national security systems. Verification speed depends heavily on the implementation and hardware, so benchmark your own library rather than trusting generic per-signature timings.

A dual-header verification pipeline

No widely adopted webhook specification defines post-quantum signature headers today, so the header names below are illustrative. The practical way to migrate without breaking existing senders is to let them send both a classical and a post-quantum signature over the same canonical message (typically timestamp, delivery ID and raw body, which also gives you replay protection).

                   Incoming HTTP Request
                             │
            ┌────────────────┴────────────────┐
            ▼                                 ▼
   [X-Signature-Classical]           [X-Signature-PQC]
   (Ed25519 / HMAC-SHA256)           (ML-DSA-65)
            │                                 │
            ▼                                 ▼
   ┌─────────────────┐               ┌─────────────────┐
   │ Verify classical│               │   Verify PQC    │
   └────────┬────────┘               └────────┬────────┘
            │ (pass)                          │ (pass)
            └────────────────┬────────────────┘
                             ▼
                  Apply policy, forward upstream

Dual headers are not the only way to do hybrids. The IETF LAMPS working group’s Composite ML-DSA draft defines single combined signature algorithms that pair ML-DSA with RSA, ECDSA, Ed25519 or Ed448, aimed at X.509/PKIX use. When I checked its status, it had reached the RFC Editor queue. NIST’s transition guidance also accommodates dual and composite signatures as long as at least one component is NIST-approved.

Engineering considerations

Header limits. Many proxies cap header sizes, and ML-DSA signatures push against those caps.
- NGINX’s default large_client_header_buffers is 4 8k, and each header field must fit in one buffer. A base64 ML-DSA-87 signature (about 6.2 KB) fits, but a hex-encoded one (about 9.3 KB) does not. Raise the buffer size or use base64.
- Envoy’s default max_request_headers_kb is 60 KiB, with a configurable maximum of 96 KiB. Oversized requests get a 431.
- Every hop must agree. Upstream services with smaller limits will still reject large headers even if your edge accepts them.
Public key caching. Keys of up to about 2.6 KB should be cached in process memory (for example in a concurrent map keyed by kid) with a sensible TTL and an invalidation path for rotation. This avoids a remote key-set lookup on the verification hot path.
Policy modes. Run the gateway in one of three modes per sender:
- Migration: accept if either signature verifies, and emit metrics on PQC adoption. Treat this strictly as a rollout aid. While it is on, an attacker who can break the classical scheme can still get through.
- Strict hybrid: both signatures must verify. This defends against a break in either algorithm.
- PQC-only: use this after a sender has retired the classical scheme.
Prevent downgrade. Once a sender has published a PQC key, require the PQC header for that sender. Otherwise an attacker can strip it and present only a classical signature.

2. Early Packet Filtering: XDP and AF_XDP

What XDP does and does not buy you

XDP (eXpress Data Path) runs eBPF programs in the network driver’s receive path, before the kernel allocates an sk_buff. A program can drop a packet (XDP_DROP), hand it to the normal stack (XDP_PASS) or redirect it (XDP_REDIRECT). Cloudflare’s L4Drop system uses XDP for volumetric DDoS mitigation and has reported dropping more than 10 million packets per second on a single core. Typical iptables-style filtering is far slower per core.

Two clarifications keep expectations realistic:

Filtering is where XDP shines. Legitimate HTTPS webhook traffic at high request rates is usually bound by TLS, parsing and application logic rather than by per-packet kernel overhead. XDP’s biggest win is discarding junk before it consumes those resources.
AF_XDP delivers raw Ethernet frames to user space, not HTTP requests. An AF_XDP socket bypasses the kernel stack, so there is no TCP or TLS processing. If you redirect port-443 traffic to an AF_XDP socket, your application must supply its own TCP and TLS implementation. Most gateways instead use XDP only for filtering and rate limiting, then XDP_PASS surviving traffic to the normal kernel stack and a conventional proxy.

AF_XDP also depends on the driver. In XDP_DRV mode with zero-copy support, frames land directly in user-space memory (the UMEM). Otherwise the kernel falls back to a mode with an extra copy. A UMEM has a fill ring and a completion ring, and a socket has RX and TX rings. The application must keep the fill ring stocked with free frames, or no packets will arrive.

┌────────────────────────────────────────────────────────┐
│                       USER SPACE                       │
│  ┌──────────────────────────────────────────────┐      │
│  │  Gateway with its own TCP/TLS (AF_XDP socket)│      │
│  └──────────────────────▲───────────────────────┘      │
│                         │ RX / fill rings over UMEM    │
├─────────────────────────┼──────────────────────────────┤
│                      KERNEL SPACE                      │
│  ┌──────────────────────┴───────────────────────┐      │
│  │ XDP program in NIC driver (drop/pass/redirect)│─────┼─► XDP_PASS → normal
│  └──────────────────────▲───────────────────────┘      │   kernel TCP stack
├─────────────────────────┼──────────────────────────────┤
│                        HARDWARE                        │
│  ┌──────────────────────┴───────────────────────┐      │
│  │                 Network Card                 │      │
│  └──────────────────────────────────────────────┘      │
└────────────────────────────────────────────────────────┘

XDP filter (C)

This program validates Ethernet, IPv4 and TCP headers with bounds checks, drops obviously malformed TCP flag combinations, and redirects traffic for the gateway port to an AF_XDP socket bound to the receiving queue. Everything else goes to the normal stack, so DNS, QUIC, ICMP and other host traffic keep working.

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/in.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

#define GATEWAY_PORT 443

struct {
    __uint(type, BPF_MAP_TYPE_XSKMAP);
    __uint(max_entries, 64);
    __type(key, __u32);
    __type(value, __u32);
} xsks_map SEC(".maps");

SEC("xdp")
int xdp_webhook_ingress_filter(struct xdp_md *ctx)
{
    void *data     = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_DROP;

    if (eth->h_proto != bpf_htons(ETH_P_IP))
        return XDP_PASS;                  /* non-IPv4 handled by the kernel */

    struct iphdr *iph = (void *)(eth + 1);
    if ((void *)(iph + 1) > data_end)
        return XDP_DROP;

    __u32 ip_hlen = iph->ihl * 4;         /* honor IP options */
    if (ip_hlen < sizeof(*iph))
        return XDP_DROP;

    if (iph->protocol != IPPROTO_TCP)
        return XDP_PASS;                  /* leave UDP/ICMP etc. alone */

    struct tcphdr *tcph = (void *)iph + ip_hlen;
    if ((void *)(tcph + 1) > data_end)
        return XDP_DROP;

    if (tcph->dest != bpf_htons(GATEWAY_PORT))
        return XDP_PASS;

    /* Illegal flag combination (SYN+FIN) */
    if (tcph->syn && tcph->fin)
        return XDP_DROP;

    /* Redirect to the AF_XDP socket for this RX queue; if none is
     * registered, fall back to the normal stack (flags = XDP_PASS). */
    return bpf_redirect_map(&xsks_map, ctx->rx_queue_index, XDP_PASS);
}

char _license[] SEC("license") = "GPL";

If you keep the kernel TCP stack, replace the final redirect with return XDP_PASS; and use the program purely as a filter and rate limiter.

AF_XDP consumer (Rust sketch)

The sketch below assumes bindgen-generated bindings to libxdp’s xsk.h. Two details matter. First, what arrives is a raw frame, so you must walk the Ethernet, IP and TCP headers to reach any payload. Second, frames must be returned to the fill ring once you are done with them, and in zero-copy mode you cannot recycle a frame while a worker is still reading it.

use std::slice;

const BATCH: u32 = 64;

pub struct XskSocket {
    umem_area: *mut std::ffi::c_void,
    rx_ring: xsk_ring_cons,
    fill_ring: xsk_ring_prod,
}

/// Returns the TCP payload of an Ethernet/IPv4/TCP frame, if well formed.
fn tcp_payload(frame: &[u8]) -> Option<&[u8]> {
    if frame.len() < 14 + 20 { return None; }
    if u16::from_be_bytes([frame[12], frame[13]]) != 0x0800 { return None; } // IPv4
    let ihl = ((frame[14] & 0x0f) as usize) * 4;
    if ihl < 20 || frame[14 + 9] != 6 { return None; }                        // TCP
    let tcp = 14 + ihl;
    if frame.len() < tcp + 20 { return None; }
    let doff = ((frame[tcp + 12] >> 4) as usize) * 4;
    if doff < 20 || frame.len() < tcp + doff { return None; }
    Some(&frame[tcp + doff..])
}

impl XskSocket {
    pub fn process_rx(&mut self) {
        let mut rx_idx: u32 = 0;
        let rcvd = unsafe { xsk_ring_cons__peek(&mut self.rx_ring, BATCH, &mut rx_idx) };
        if rcvd == 0 { return; }

        let mut fill_idx: u32 = 0;
        let reserved =
            unsafe { xsk_ring_prod__reserve(&mut self.fill_ring, rcvd, &mut fill_idx) };

        for i in 0..rcvd {
            let desc = unsafe { &*xsk_ring_cons__rx_desc(&self.rx_ring, rx_idx + i) };
            let frame = unsafe {
                slice::from_raw_parts(
                    xsk_umem__get_data(self.umem_area, desc.addr) as *const u8,
                    desc.len as usize,
                )
            };

            if let Some(_payload) = tcp_payload(frame) {
                // Feed the segment to your user-space TCP/TLS stack here.
            }

            // Hand the frame back to the kernel (only once nothing still reads it).
            if i < reserved {
                unsafe {
                    *xsk_ring_prod__fill_addr(&mut self.fill_ring, fill_idx + i) = desc.addr;
                }
            }
        }

        unsafe {
            xsk_ring_cons__release(&mut self.rx_ring, rcvd);
            xsk_ring_prod__submit(&mut self.fill_ring, reserved);
        }
    }
}

Production code must also handle partial reservations, retry and wake-up logic, and the TX and completion rings. Libraries such as libxdp or xsk-rs handle much of that bookkeeping.

3. Tenant-Aware Sharding and Isolation

The noisy-neighbor problem

In a shared ingestion pipeline, one tenant’s backfill can fill a common queue and delay everyone else’s events. This is head-of-line (HoL) blocking. The fix is to isolate tenants so that a flood from one only fills that tenant’s own buffer.

                           Incoming Webhook Stream
                                      │
                                      ▼
                        ┌──────────────────────────┐
                        │  Tenant-aware Router     │
                        └─────────────┬────────────┘
           ┌──────────────────────────┼──────────────────────────┐
           ▼                          ▼                          ▼
┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│ Tenant A (dedicated)│    │ Tenant B (shared)   │    │ Shared pool queues  │
│ saturated           │    │ normal              │    │                     │
└──────────┬──────────┘    └──────────┬──────────┘    └──────────┬──────────┘
           ▼                          ▼                          ▼
┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│ Worker group        │    │ Fair-share workers  │    │ Fair-share workers  │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘

Consistent hashing with virtual nodes

Consistent hashing maps tenants to queues so that adding or removing a queue only remaps a small fraction of tenants. Virtual nodes smooth out the distribution. Note that hashing alone does not solve the noisy-neighbor problem. A single hot tenant still hashes to a single queue, which is why the dedicated-queue mechanism below is needed.

use std::collections::BTreeMap;
use std::hash::{Hash, Hasher};
use std::collections::hash_map::DefaultHasher;

pub struct ConsistentHashRing {
    nodes: BTreeMap<u64, String>,
}

impl ConsistentHashRing {
    pub fn new() -> Self {
        Self { nodes: BTreeMap::new() }
    }

    pub fn add_queue(&mut self, queue_id: &str, vnodes: usize) {
        for i in 0..vnodes {
            let key = Self::hash_key(&format!("{queue_id}-vnode-{i}"));
            self.nodes.insert(key, queue_id.to_string());
        }
    }

    pub fn get_queue(&self, tenant_id: &str) -> Option<&String> {
        if self.nodes.is_empty() {
            return None;
        }
        let hash = Self::hash_key(tenant_id);
        self.nodes
            .range(hash..)
            .next()
            .map(|(_, q)| q)
            .or_else(|| self.nodes.values().next()) // wrap around the ring
    }

    // NOTE: DefaultHasher's algorithm is not guaranteed stable across Rust
    // releases. For a ring shared by several services or persisted routing
    // decisions, use a fixed, specified hash (e.g. xxHash or keyed SipHash).
    fn hash_key(val: &str) -> u64 {
        let mut hasher = DefaultHasher::new();
        val.hash(&mut hasher);
        hasher.finish()
    }
}

Dedicated queues, deficit round robin and backpressure

These policies work with brokers such as Apache Kafka, RabbitMQ or NATS JetStream. The thresholds below are starting points to tune against your own traffic, not standards.

Dedicated queues for heavy tenants. When a tenant’s share of ingress stays above a threshold (for example 15% of capacity over a moving window), move it to its own queue or partition set, with its own consumer group, so its backlog cannot delay others.
Deficit round robin (DRR) scheduling. DRR, introduced by Shreedhar and Varghese in 1995, gives each queue a byte “quantum” per round and carries unused credit forward. Weighted variants give larger tenants bigger quanta. Workers move on to the next tenant when a tenant exhausts its credit, so no tenant monopolizes a worker.
Per-tenant backpressure. When a tenant’s queue depth crosses a high-water mark (for example 85%), respond to that tenant’s requests with 429 Too Many Requests and a Retry-After header, both defined in HTTP standards (RFC 6585 and RFC 9110). Other tenants are unaffected. Better still, keep the ingress path thin: validate, persist to a durable log, return 2xx quickly, and do the real work asynchronously, so backpressure is a last resort rather than the normal path.

4. Stateful Defenses Against Slow-Client Attacks

Webhook endpoints accept POST requests from the open internet, which makes them targets for application-layer resource exhaustion.

Attack mechanics

Slowloris-style attacks. The client opens many connections and sends headers (or body bytes) extremely slowly, never finishing the request. Each connection holds a socket, a file descriptor and some memory until a timeout fires, so a modest number of connections can exhaust a server’s capacity.
Unbounded chunked bodies. With Transfer-Encoding: chunked there is no Content-Length up front, so a client can keep streaming chunks. A proxy that buffers the body without a cumulative cap can run out of memory or disk.

HTTP/2 brings related abuse patterns, such as the Rapid Reset attack disclosed in 2023 (CVE-2023-44487). Those need protocol-specific limits on concurrent streams and stream resets on top of what follows.

Decision model

                      Raw TCP Stream Connection
                                  │
                                  ▼
               ┌─────────────────────────────────────┐
               │   Connection tracking (per socket)  │
               └──────────────────┬──────────────────┘
                                  │
         ┌────────────────────────┴────────────────────────┐
         ▼                                                 ▼
 [Header deadline]                                 [Total body size]
 Headers complete within N seconds?                Cumulative bytes ≤ limit?
         │                                                 │
   no ──► 408 / close                               no ──► 413 / close

Use the knobs your proxy already has

Before writing custom code, set the built-in limits:

NGINX: client_header_timeout (default 60s) bounds the time to receive the full request header. client_body_timeout (default 60s) applies between two successive reads of the body, not to the body as a whole, so a trickle can slip under it. client_max_body_size (default 1m) is enforced for chunked bodies too and returns 413. Lowering the header and body timeouts to the single-digit to tens-of-seconds range is typical hardening.
Envoy: the HTTP connection manager exposes request_headers_timeout, request_timeout and idle timeouts, along with max_request_headers_kb for header size.
Minimum data rates: NGINX has no direct minimum-rate directive. Apache’s mod_reqtimeout supports minimum transfer rates, and a custom proxy can enforce the same idea.

Choose values from your real traffic. Legitimate webhook senders usually deliver small bodies in well under a second, so a header deadline of a few seconds and a tight body timeout are reasonable for many deployments. Senders on slow links, or with large payloads, may need more room.

Example: minimum-rate and size enforcement (Rust, Tokio)

This sketch reads a request body while enforcing a per-read timeout, a minimum average rate over a rolling window, and a cumulative size cap. A real implementation would also parse HTTP framing (Content-Length and chunked encoding) to know when the request is complete, and apply the rate check only while a request is still in flight.

use std::io;
use tokio::io::AsyncReadExt;
use tokio::net::TcpStream;
use tokio::time::{timeout, Duration, Instant};

const MAX_PAYLOAD_BYTES: usize = 10 * 1024 * 1024; // 10 MB cumulative cap
const MIN_RATE_BYTES_PER_SEC: u64 = 1024;          // 1 KB/s average
const RATE_WINDOW: Duration = Duration::from_secs(5);
const PER_READ_TIMEOUT: Duration = Duration::from_secs(5);

pub async fn read_body_guarded(stream: &mut TcpStream) -> io::Result<Vec<u8>> {
    let mut body = Vec::new();
    let mut buf = [0u8; 8192];
    let mut window_start = Instant::now();
    let mut window_bytes: u64 = 0;

    loop {
        let n = match timeout(PER_READ_TIMEOUT, stream.read(&mut buf)).await {
            Ok(Ok(0)) => return Ok(body),               // peer closed
            Ok(Ok(n)) => n,
            Ok(Err(e)) => return Err(e),
            Err(_) => {
                return Err(io::Error::new(io::ErrorKind::TimedOut, "read timeout"));
            }
        };

        if body.len() + n > MAX_PAYLOAD_BYTES {
            return Err(io::Error::new(io::ErrorKind::InvalidData, "payload too large"));
        }
        body.extend_from_slice(&buf[..n]);
        window_bytes += n as u64;

        // Check the average rate once per full window, not per read.
        let elapsed = window_start.elapsed();
        if elapsed >= RATE_WINDOW {
            if window_bytes < MIN_RATE_BYTES_PER_SEC * elapsed.as_secs() {
                return Err(io::Error::new(io::ErrorKind::TimedOut, "client too slow"));
            }
            window_start = Instant::now();
            window_bytes = 0;
        }
    }
}

Judging each read against a fixed size, as simpler implementations do, would kill legitimate small payloads. Measuring the average over a window avoids that.

Summary

A resilient high-throughput webhook ingress needs defenses at every layer, and each layer has a realistic scope:

Protocol layer: Plan for ML-DSA (FIPS 204) alongside your existing signatures, using dual headers or composite signatures. Enforce strict verification and downgrade protection once senders migrate. NIST’s final IR 8547 sets 2030 and 2035 as the deprecation and disallowance dates for quantum-vulnerable public-key algorithms.
Network layer: Use XDP to drop junk and rate-limit before it reaches the kernel stack, where published results show on the order of 10 million dropped packets per second per core. Reserve AF_XDP for cases where you are prepared to bring your own TCP and TLS stack.
Routing layer: Combine consistent hashing, dedicated queues for heavy tenants, deficit round robin scheduling and per-tenant backpressure so one sender cannot block the rest.
Transport layer: Set strict header and body timeouts, size caps and minimum-rate checks, using your proxy’s native controls first and custom code only where gaps remain.

References

NIST FIPS 204, Module-Lattice-Based Digital Signature Standard: https://csrc.nist.gov/pubs/fips/204/final
Encryption Consulting, What Is ML-DSA (FIPS 204)?: https://www.encryptionconsulting.com/education-center/ml-dsa-fips-204/
Help Net Security, NIST IR 8547 roadmap coverage: https://www.helpnetsecurity.com/2026/10/28/nist-post-quantum-cryptography-migration-roadmap/
PostQuantum.com, NIST IR 8547 final: https://postquantum.com/security-pqc/nist-ir-8547-final/
IETF LAMPS, Composite ML-DSA for use in X.509 PKI: https://datatracker.ietf.org/doc/draft-ietf-lamps-pq-composite-sigs/
Linux kernel documentation, AF_XDP: https://docs.kernel.org/networking/af_xdp.html
Cloudflare, L4Drop: XDP DDoS Mitigations: https://blog.cloudflare.com/l4drop-xdp-ebpf-based-ddos-mitigations/
NGINX, ngx_http_core_module documentation: https://nginx.org/en/docs/http/ngx_http_core_module.html
Envoy, HTTP connection manager configuration: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto
RFC 6585 (HTTP 429): https://www.rfc-editor.org/rfc/rfc6585
RFC 9110 (HTTP Semantics, Retry-After): https://www.rfc-editor.org/rfc/rfc9110

Originally published at https://instawebhook.com/blog/high-throughput-post-quantum-webhook-ingress-engineering

Building an In-Memory Circuit Breaker for Webhook Ingress: Shedding Load Without Losing Events

InstaWebhook — Tue, 03 Nov 2026 02:10:44 +0000

Webhooks connect cloud systems. Payment providers like Stripe, code hosts like GitHub and commerce platforms like Shopify all use them to tell your services that something changed. When a sender has a bad day, you pay for it. A bulk export, a backfill or a burst of retries can hit your ingress endpoint, and the endpoint is the first thing that fails.

When a burst arrives, worker threads or goroutines block on slow downstream calls and connection pools saturate. Memory fills with queued request bodies and garbage-collection pressure pushes latency higher. Health checks start failing, the load balancer moves traffic to the remaining nodes, and those nodes tip over too.

This guide describes an in-memory admission controller for webhook ingress. It combines an adaptive concurrency limit, a CoDel-style delay signal and priority-aware shedding. It also covers what “not losing events” means when you reject a request, because that depends on how each sender retries.

1. How an ingress meltdown unfolds

Webhook senders enforce short response deadlines, and each provider sets its own:

Sender	Documented behavior
GitHub	Expects a 2XX within 10 seconds, otherwise it terminates the connection and counts a failed delivery. It does not redeliver automatically; failed deliveries must be redelivered manually or via the API.
Shopify	One-second connection timeout and five-second timeout for the whole request. Failed deliveries are retried 8 times over 4 hours.
Slack (Events API)	Expects a valid response within three seconds.
Stripe (live mode)	Retries for up to 3 days with exponential backoff.

Because the deadlines are short, a slow endpoint produces timeouts, and timeouts produce more traffic. A sender that retries adds those retries to the stream of fresh events. A sender that doesn’t retry, like GitHub, leaves you with lost events and manual redelivery.

The failure then plays out in a predictable order:

Connection saturation: new connections queue in the socket backlog.
Worker starvation: handlers blocked on database calls hold their threads or goroutines.
Memory growth: request bodies pile up while waiting for a worker.
Latency amplification: garbage-collection work and contention raise latency, which causes more sender timeouts.

[Upstream Sender] ──(flood / retries)──> [Load Balancer]
                                                │
                                                ▼
                                  [Ingress Worker Node]
                                    ├── Connection queue (full)
                                    ├── Workers (blocked)
                                    └── Memory / GC (pressure) ──> [meltdown]

The first defense is architectural. GitHub’s documentation recommends putting a queue in front of slow work so the server can respond immediately and process the payload in the background. Shopify gives the same advice for bursts. Admission control is the second line of defense. It protects the thin “validate, enqueue, acknowledge” path itself.

2. Why the admission decision should be local

Resilience4j’s circuit breaker is the standard tool for outbound calls. It is an in-process library and doesn’t need Redis or any other shared store. The same reasoning applies at ingress:

No extra dependency on the hot path. Consulting a shared store for every inbound request adds a network round trip and a new failure mode. If that store is down, your ingress should still be able to shed load.
The overload is local. Thread pools, memory and database connections are per-node resources. A node-local signal reflects what that node can actually handle.
Senders only see HTTP. They react to status codes and timing, so the cleanest signal you can give them is a fast, explicit rejection.

The trade-off is that each node decides independently. With uneven load balancing, one node may shed while another has spare capacity. Per-node metrics (section 7) show you when that happens.

3. Two mechanisms: adaptive concurrency and queue delay

Adaptive concurrency limits

Little’s Law says that concurrency equals throughput times latency. A fixed limit such as “100 concurrent requests” goes stale as soon as latency changes. If the database slows down, 100 is too high. If it speeds up, 100 is too low.

Netflix’s open-source concurrency-limits library, described in their 2018 post “Performance Under Load”, adjusts the limit from observed latency instead. Its gradient algorithm uses:

newLimit = gradient × currentLimit + queueSize

In the library’s Gradient2 variant, the gradient is the long-term RTT divided by the current RTT, clamped to the range 0.5 to 1.0. The result is then smoothed (default factor 0.2):

gradient = max(0.5, min(1.0, longTermRTT / currentRTT))
newLimit = gradient × currentLimit + queueSize
newLimit = currentLimit × (1 − smoothing) + newLimit × smoothing

When latency rises above its long-term average, the gradient drops below 1 and the limit shrinks. When latency is stable, the queueSize term lets the limit creep upward. It can be a fixed value or a function of the limit, such as a square root.

Queue delay (CoDel-style)

Counting 5xx errors reacts late, because by the time errors appear the damage is done. A better early signal is how long requests wait for a slot.

CoDel (Controlled Delay) was designed by Kathleen Nichols and Van Jacobson as an active queue management algorithm for network buffers (RFC 8289). It tracks a packet’s sojourn time, the time spent in the queue. It acts only when the minimum delay over a sliding window stays above a target, which allows short bursts but prevents a persistent standing queue. RFC 8289’s defaults are a 5 ms target and a 100 ms interval.

Facebook adapted a variant of CoDel for request queues, as described in “Fail at Scale”. Its open-source Wangle library shows the idea. Within each interval it tracks the minimum queueing delay and declares overload if that minimum exceeded the target. While overloaded, it sheds requests that have waited longer than twice the target. This is an adaptation rather than textbook CoDel. Treat 5 ms and 100 ms as starting points and tune them against your handler’s normal service time.

4. Architecture

Requests pass through classification and then a limiter that both enforces the adaptive limit and measures queue delay:

Incoming webhook request
       │
       ▼
 ┌──────────────────────────────────────────┐
 │ 1. Priority classifier                   │
 │    (route or verified partner identity)  │
 └──────────────────────────────────────────┘
       │
       ▼
 ┌──────────────────────────────────────────┐
 │ 2. Admission: slot free?                 │
 │    yes → admit; no → queue briefly, or   │
 │    shed if overloaded (by priority)      │
 └──────────────────────────────────────────┘
       │ (slot acquired)
       ▼
 ┌──────────────────────────────────────────┐
 │ 3. Thin handler: verify signature,       │
 │    enqueue durably, return 2xx           │
 └──────────────────────────────────────────┘
       │
       ▼
 Release slot → feed RTT back to the limit

Classify by something you control, such as the endpoint path or the partner identified after signature verification. Don’t trust a priority header that arrives from the public internet, because anyone can set it.

5. A working Go sketch

The following sketch uses only the standard library. It combines a Gradient2-style limit with a Wangle-style overload flag. The parameter values are illustrative and need tuning against your own traffic.

package ingress

import (
    "math"
    "net/http"
    "sync"
    "time"
)

type Priority int

const (
    PriorityLow Priority = iota
    PriorityDefault
    PriorityHigh
)

type Config struct {
    InitialLimit float64       // e.g. 20
    MinLimit     float64       // e.g. 4
    MaxLimit     float64       // e.g. 200
    Smoothing    float64       // e.g. 0.2
    TargetDelay  time.Duration // e.g. 5ms
    Interval     time.Duration // e.g. 100ms
    MaxWait      time.Duration // longest a request may wait for a slot, e.g. 2 * TargetDelay
    MaxQueue     int           // hard cap on waiting requests
}

type waiter struct {
    ready   chan struct{}
    granted bool
}

type Limiter struct {
    cfg Config

    mu          sync.Mutex
    limit       float64
    inflight    int
    waiters     []*waiter
    shortRTT    float64 // seconds, fast EWMA
    longRTT     float64 // seconds, slow EWMA
    intervalEnd time.Time
    minDelay    time.Duration
    overloaded  bool
}

func NewLimiter(cfg Config) *Limiter {
    return &Limiter{cfg: cfg, limit: cfg.InitialLimit}
}

// Acquire returns a release func and true if the request is admitted.
func (l *Limiter) Acquire(p Priority) (func(time.Duration), bool) {
    l.mu.Lock()
    if l.inflight < int(l.limit) && len(l.waiters) == 0 {
        l.inflight++
        l.observeDelay(0)
        l.mu.Unlock()
        return l.release, true
    }
    if l.shouldShed(p) {
        l.mu.Unlock()
        return nil, false
    }
    w := &waiter{ready: make(chan struct{})}
    l.waiters = append(l.waiters, w)
    l.mu.Unlock()

    start := time.Now()
    timer := time.NewTimer(l.cfg.MaxWait)
    defer timer.Stop()

    timedOut := false
    select {
    case <-w.ready:
    case <-timer.C:
        timedOut = true
    }

    l.mu.Lock()
    defer l.mu.Unlock()
    l.observeDelay(time.Since(start))
    if timedOut && !w.granted {
        l.removeWaiter(w)
        return nil, false
    }
    return l.release, true
}

func (l *Limiter) shouldShed(p Priority) bool {
    switch p {
    case PriorityHigh:
        return len(l.waiters) >= l.cfg.MaxQueue // only when the queue is full
    case PriorityDefault:
        return l.overloaded || len(l.waiters) >= l.cfg.MaxQueue
    default: // PriorityLow: no queueing at all once slots are exhausted
        return true
    }
}

// observeDelay tracks the minimum queueing delay per interval (CoDel-style).
func (l *Limiter) observeDelay(d time.Duration) {
    now := time.Now()
    if now.After(l.intervalEnd) {
        l.overloaded = l.minDelay > l.cfg.TargetDelay
        l.minDelay = d
        l.intervalEnd = now.Add(l.cfg.Interval)
        return
    }
    if d < l.minDelay {
        l.minDelay = d
    }
}

func (l *Limiter) release(rtt time.Duration) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.inflight--
    l.updateLimit(rtt.Seconds())
    for len(l.waiters) > 0 && l.inflight < int(l.limit) {
        w := l.waiters[0]
        l.waiters = l.waiters[1:]
        w.granted = true
        l.inflight++
        close(w.ready)
    }
}

// updateLimit is a simplified Gradient2-style update.
func (l *Limiter) updateLimit(rtt float64) {
    if rtt <= 0 {
        return
    }
    if l.longRTT == 0 {
        l.longRTT, l.shortRTT = rtt, rtt
    }
    l.shortRTT += 0.2 * (rtt - l.shortRTT)
    l.longRTT += 0.01 * (rtt - l.longRTT)

    gradient := math.Max(0.5, math.Min(1.0, l.longRTT/l.shortRTT))
    next := gradient*l.limit + math.Sqrt(l.limit)
    next = l.limit*(1-l.cfg.Smoothing) + next*l.cfg.Smoothing
    l.limit = math.Max(l.cfg.MinLimit, math.Min(l.cfg.MaxLimit, next))
}

func (l *Limiter) removeWaiter(w *waiter) {
    for i, x := range l.waiters {
        if x == w {
            l.waiters = append(l.waiters[:i], l.waiters[i+1:]...)
            return
        }
    }
}

// Guard wraps a webhook handler with admission control.
func Guard(l *Limiter, classify func(*http.Request) Priority, next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        release, ok := l.Acquire(classify(r))
        if !ok {
            w.Header().Set("Retry-After", "10")
            http.Error(w, "service temporarily overloaded", http.StatusServiceUnavailable)
            return
        }
        start := time.Now()
        defer func() { release(time.Since(start)) }()
        next.ServeHTTP(w, r)
    })
}

Some limits of this sketch:

Waiters are served in FIFO order. Facebook’s design switches to LIFO once a queue forms, so the newest requests, which are most likely to still meet their sender’s deadline, go first.
RTT is measured for every request. In production you would typically exclude requests that failed fast.
A single mutex is fine at moderate rates. Profile before optimizing it.

6. Rejecting requests without losing events

Whether a rejected webhook is lost depends on the sender, so you need to know how each one behaves:

Senders with retries: Stripe retries with exponential backoff for up to 3 days in live mode. Shopify retries 8 times over 4 hours.
Senders without automatic retries: GitHub does not redeliver automatically. If you shed a GitHub delivery, you need to redeliver it manually or through the API once you recover.
Subscription risk: Shopify deletes a webhook subscription created through the Admin API after 8 consecutive failures, and Stripe emails you and eventually disables an endpoint that fails for multiple days. A long shedding episode can cost you the subscription itself.

For these reasons, shedding should be short, targeted and backed by a reconciliation job. Shopify itself recommends building one if you can’t afford to miss webhooks. The job periodically fetches data you may have missed through the provider’s API.

Status codes

503 Service Unavailable with Retry-After: the standard way to signal temporary overload. RFC 9110 defines Retry-After as either a number of seconds or an HTTP date.
429 Too Many Requests: RFC 6585 defines it for rate violations by a particular sender. It also accepts Retry-After.
Don’t just drop connections. An abrupt reset gives the sender no information, and it will treat the delivery as a failed attempt. A clean 503 tells it explicitly that the failure is temporary.

Do senders actually honor Retry-After? Some do. Webhook infrastructure vendors such as Hookdeck and Kit document that they respect Retry-After on 429 and 503, and AWS EventBridge API destinations are documented to as well. Svix advises that good senders will respect it when you return a 429. Stripe’s public documentation describes exponential backoff but doesn’t mention Retry-After, so check its behavior in your delivery logs rather than assuming. Because support varies, treat the header as a hint and rely on the sender’s own backoff as the fallback.

Add jitter to the values you return, for example 5 to 15 seconds instead of a constant 10. Otherwise senders that honor the header will retry in lockstep.

7. Priority-based shedding

Not all webhooks are equally urgent. A payment confirmation matters more than a bulk analytics sync. Classify by endpoint or authenticated partner, then apply different behavior per tier. The sketch above implements this policy:

Tier	Examples	Behavior in the sketch
`PriorityHigh`	Payment confirmations, security alerts	Queue for a slot; shed only when the wait queue is full.
`PriorityDefault`	Standard operational updates	Queue, but shed once the CoDel-style overload flag is set.
`PriorityLow`	Bulk syncs, analytics, non-essential notifications	Shed immediately when no slot is free.

This is a policy choice rather than a standard. Choose tiers based on which events you can safely replay later.

8. Observability

Since each node decides for itself, you need per-node metrics. Export them to Prometheus or OpenTelemetry:

webhook_ingress_concurrency_limit: gauge for the current adaptive limit.
webhook_ingress_inflight_requests: gauge for requests being processed.
webhook_ingress_shed_total{priority}: counter of rejected requests.
webhook_ingress_queue_delay_seconds: histogram of time spent waiting for a slot. Use millisecond-scale buckets, since the targets are in the single-digit-millisecond range.
webhook_ingress_overloaded: gauge (0 or 1) for the overload flag.

Alert on a sustained rise in shed counts or on queue delay staying above target. Both mean you should look at downstream capacity, not just the limiter.

9. Checklist

Acknowledge fast: Verify the signature, enqueue durably and return 2xx. Do the real work asynchronously.
Keep admission state local: Avoid a shared-store call on every inbound request.
Use a latency-driven limit: An adaptive, gradient-style limit beats a fixed number.
Watch queue delay, not just errors: Use a CoDel-style minimum-delay signal, and tune the target to your handler.
Reject explicitly: Return 503 (or 429 for per-sender limits) with a jittered Retry-After.
Know your senders: Document each provider’s timeout, retry window and subscription-deletion rules.
Reconcile: Run a job that backfills events missed during shedding, and redeliver GitHub deliveries manually.
Classify safely: Derive priority from routes or verified identity, never from unauthenticated headers.

References

Stripe webhooks documentation: https://docs.stripe.com/webhooks
GitHub, “Best practices for using webhooks”: https://docs.github.com/webhooks/using-webhooks/best-practices-for-using-webhooks
Shopify, “Deliver webhooks through HTTPS”: https://shopify.dev/docs/apps/build/webhooks/subscribe/https
Slack Events API response deadline: https://www.vincentschmalbach.com/setting-webhook-response-deadlines-to-prevent-failures/
Netflix Technology Blog, “Performance Under Load”: https://netflixtechblog.medium.com/performance-under-load-3e6fa9a60581
Netflix concurrency-limits library: https://github.com/Netflix/concurrency-limits
RFC 8289, “Controlled Delay Active Queue Management”: https://datatracker.ietf.org/doc/rfc8289/
CoDel pseudocode (ACM Queue): https://queue.acm.org/appendices/codel.html
Facebook Wangle CoDel implementation: https://github.com/facebook/wangle/blob/master/wangle/concurrent/Codel.cpp
Svix, “Webhook retry strategies”: https://www.svix.com/resources/webhook-university/reliability/webhook-retry-strategies/
Hookdeck, “Webhook Delivery Guarantees”: https://hookdeck.com/webhooks/platforms/webhook-delivery-guarantees-comparison

Originally published at https://instawebhook.com/blog/building-an-in-memory-circuit-breaker-for-webhook-ingress-shedding-load-without

Memory-Mapped Webhook Buffers: Using `mmap` and Ring Buffers to Cut Kernel-User Crossings at 500k RPS

InstaWebhook — Mon, 02 Nov 2026 02:21:37 +0000

Webhooks connect payment gateways, CI/CD pipelines and SaaS platforms. At scale, the service that receives them has to accept a request, verify it, acknowledge it quickly and hand it off for processing, hundreds of thousands of times per second.

This article looks at what actually costs CPU time at 500,000 requests per second. It shows where a shared-memory ring buffer built on mmap helps, how to write one correctly, and how modern io_uring features fit around it. It also covers where the approach does not help, because those limits matter as much as the gains.

1. Where the time goes at 500k RPS

500,000 RPS leaves a very small time budget per request. A single core handling all of that traffic would have 2 µs per request. Spread across 32 cores, each request gets about 64 µs of core time, and that has to cover TCP, TLS (if you terminate it), HTTP parsing, signature verification and the handoff to workers. A 1 KB payload at this rate is about 512 MB/s, or roughly 4 Gbit/s of body data before protocol overhead, so a 25 GbE NIC is not the limit. CPU time per request is.

The traditional receive path

A typical blocking or epoll-based server handles an incoming webhook like this:

The NIC writes the packet into a kernel-owned receive ring by DMA.
An interrupt (or a polled NAPI cycle) triggers softirq processing, and the kernel runs the packet through IP and TCP.
The payload lands in the socket’s receive queue (sk_buff structures in kernel memory).
The application calls recv/read, which switches into the kernel.
The kernel copies the data into the user-space buffer (copy_to_user) and returns to user mode.

+-------------------------------------------------------------+
| USER SPACE                                                  |
|   Application buffer                                        |
|        ^                                                    |
|        | copy_to_user()  (copy + syscall entry/exit)        |
+--------|----------------------------------------------------+
| KERNEL SPACE                                                |
|   Socket receive queue (sk_buff)                            |
|        ^                                                    |
|        | TCP/IP stack processing                            |
+--------|----------------------------------------------------+
| HARDWARE                                                    |
|   NIC receive ring (DMA)                                    |
+-------------------------------------------------------------+

A system call is not a context switch

The terminology matters here. A system call is a privilege-mode transition within the same thread. A context switch is the scheduler replacing one thread or process with another, which usually means saving and restoring state and possibly changing address spaces.

Both cost something, but they are different amounts. In one widely cited 2010 measurement on older Intel hardware, the cheapest system call (gettid) cost roughly 52 to 105 ns depending on the CPU. The same benchmark measured process context switches at about 1.3 to 1.9 µs, and about 3 µs when futex calls were included. A University of Rochester study put the direct cost of a context switch at about 3.4 µs on 2000s-era hardware. It also found that the indirect cost of refilling caches ranged from several microseconds to hundreds of microseconds, depending on working-set size. More recent measurements report a direct kernel-level context switch of around 0.62 µs. These numbers vary with CPU, kernel, mitigations and measurement method, so treat them as orders of magnitude rather than constants. Always measure your own hardware.

The practical conclusion is that per-request system calls, copies, and scheduler wake-ups add up. At 500k RPS they can take a meaningful share of your CPU budget. The size of that share depends on your workload, so profile with perf before assuming a number.

2. What `mmap` and ring buffers actually buy you

mmap() maps a file or shared-memory object into a process’s address space. If two processes map the same object with MAP_SHARED, they can exchange data through plain loads and stores. The kernel is only involved when the mapping is created and when pages fault in, not on every message.

That makes it a good fit for handing parsed webhooks from an ingestion process to worker processes. It does not remove the kernel from the network receive path, because TCP still runs in the kernel unless you bypass the stack entirely (see section 4).

+-----------------------------------------------------------------------+
| SHARED MEMORY REGION (mmap of a shm object)                           |
|                                                                       |
|   +------------+------------+------------+------------+               |
|   | Slot 0     | Slot 1     | Slot 2     | Slot 3     | ...           |
|   | (Payload)  | (Payload)  | (Payload)  | (Empty)    |               |
|   +------------+------------+------------+------------+               |
|         ^                                     ^                       |
|    [Head: consumer reads]             [Tail: producer writes]         |
+-----------------------------------------------------------------------+

A ring buffer is a good structure for this handoff because:

No allocation on the hot path. Slots are preallocated, so enqueue and dequeue are a few arithmetic and atomic operations in constant time.
Cache-friendly layout. Slots are contiguous and sequentially accessed, which suits hardware prefetchers. Aligning the head and tail counters to separate cache lines also avoids false sharing.
No locks for the single-producer, single-consumer (SPSC) case. One side only writes the tail, the other only writes the head, so acquire/release atomics are enough. Multiple producers or consumers on one ring need additional machinery (section 3).

If ingestion and workers are threads in one process, an ordinary in-process ring buffer gives you the same benefits without mmap. Use shared memory when you need process isolation (a worker crash must not take down the listener), independent deploys, or the buffer to survive an ingestion restart.

3. A correct SPSC shared-memory ring

Here is a compact C++ implementation for one producer process and one consumer process. It differs from naive versions in a few ways that matter:

It uses a power-of-two capacity, so the index is a bitmask rather than a modulo.
It uses monotonically increasing 64-bit counters, so full and empty are unambiguous and every slot is usable.
It stores each payload’s length in its slot.
It separates “create” from “attach”, so two processes never race to initialize the header.

#include <atomic>
#include <cstdint>
#include <cstring>
#include <stdexcept>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

constexpr size_t kCacheLine = 64;

struct RingHeader {
    alignas(kCacheLine) std::atomic<uint64_t> head;  // written by consumer
    alignas(kCacheLine) std::atomic<uint64_t> tail;  // written by producer
    alignas(kCacheLine) uint32_t capacity;           // power of two
    uint32_t slot_size;                              // includes 4-byte length prefix
};

static_assert(std::atomic<uint64_t>::is_always_lock_free,
              "atomics in shared memory must be lock-free");

class ShmRing {
public:
    // Producer side creates; consumer side attaches.
    static ShmRing create(const char* name, uint32_t capacity, uint32_t slot_size) {
        if (capacity == 0 || (capacity & (capacity - 1)) != 0)
            throw std::invalid_argument("capacity must be a power of two");
        size_t bytes = sizeof(RingHeader) + size_t(capacity) * slot_size;
        int fd = shm_open(name, O_CREAT | O_EXCL | O_RDWR, 0600);
        if (fd < 0) throw std::runtime_error("shm_open failed");
        if (ftruncate(fd, bytes) != 0) { close(fd); throw std::runtime_error("ftruncate failed"); }
        ShmRing r(fd, bytes);
        r.h_->capacity = capacity;
        r.h_->slot_size = slot_size;
        r.h_->head.store(0, std::memory_order_relaxed);
        r.h_->tail.store(0, std::memory_order_release);
        return r;
    }

    static ShmRing attach(const char* name) {
        int fd = shm_open(name, O_RDWR, 0600);
        if (fd < 0) throw std::runtime_error("shm_open failed");
        // Map the header first to learn the full size, then remap.
        void* hdr = mmap(nullptr, sizeof(RingHeader), PROT_READ, MAP_SHARED, fd, 0);
        if (hdr == MAP_FAILED) { close(fd); throw std::runtime_error("mmap failed"); }
        auto* ph = static_cast<RingHeader*>(hdr);
        size_t bytes = sizeof(RingHeader) + size_t(ph->capacity) * ph->slot_size;
        munmap(hdr, sizeof(RingHeader));
        return ShmRing(fd, bytes);
    }

    bool push(const void* data, uint32_t len) {            // producer only
        if (len > h_->slot_size - sizeof(uint32_t)) return false;
        uint64_t tail = h_->tail.load(std::memory_order_relaxed);
        uint64_t head = h_->head.load(std::memory_order_acquire);
        if (tail - head == h_->capacity) return false;     // full: apply backpressure
        char* slot = slots_ + (tail & (h_->capacity - 1)) * h_->slot_size;
        std::memcpy(slot, &len, sizeof(len));
        std::memcpy(slot + sizeof(len), data, len);
        h_->tail.store(tail + 1, std::memory_order_release);
        return true;
    }

    bool pop(void* dst, uint32_t dst_cap, uint32_t& out_len) {  // consumer only
        uint64_t head = h_->head.load(std::memory_order_relaxed);
        uint64_t tail = h_->tail.load(std::memory_order_acquire);
        if (head == tail) return false;                    // empty
        const char* slot = slots_ + (head & (h_->capacity - 1)) * h_->slot_size;
        uint32_t len;
        std::memcpy(&len, slot, sizeof(len));
        if (len > dst_cap) return false;
        std::memcpy(dst, slot + sizeof(len), len);
        out_len = len;
        h_->head.store(head + 1, std::memory_order_release);
        return true;
    }

private:
    ShmRing(int fd, size_t bytes) : bytes_(bytes) {
        void* m = mmap(nullptr, bytes, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
        close(fd);  // the mapping stays valid after the descriptor is closed
        if (m == MAP_FAILED) throw std::runtime_error("mmap failed");
        h_ = static_cast<RingHeader*>(m);
        slots_ = static_cast<char*>(m) + sizeof(RingHeader);
    }
    RingHeader* h_;
    char* slots_;
    size_t bytes_;
};

A few things to note about this design:

It is SPSC. The producer only reads head and writes tail, and the consumer does the reverse. The release store on one side pairs with the acquire load on the other, so a consumer that sees the new tail also sees the slot contents. A fetch_add on the tail does not make this multi-producer safe, because a producer could claim a slot before finishing its write. To get multiple producers, use one ring per producer (simple and fast), or use a per-slot sequence-number scheme such as Dmitry Vyukov’s bounded MPMC queue.
Use one ring per worker, or per ingestion thread, as the scaling strategy. The ingestion layer can shard by connection or hash, and each worker drains its own ring. This keeps every ring SPSC and avoids cross-core contention.
Slot size is a trade-off. Fixed slots make indexing trivial but waste space for small payloads. If your webhooks vary widely in size, consider variable-length records with a length header and wrap-around handling, or a second tier for oversized payloads.
The ring only moves bytes. Authentication, parsing and idempotency live in the code on either side of it.

4. Pairing the ring with `io_uring` for network ingress

io_uring is the Linux async I/O interface merged in kernel 5.1. It uses a submission queue and a completion queue shared between user space and the kernel, so many operations can be queued and reaped with few or no system calls.

Several features matter for a webhook receiver, and each needs a minimum kernel version:

Batching. One io_uring_enter() call can submit many queued operations and wait for completions. In ordinary mode you still enter the kernel, but once per batch rather than once per operation.
SQPOLL. With IORING_SETUP_SQPOLL, a kernel thread polls the submission queue, so the application can submit and reap I/O without issuing system calls in the steady state. The cost is that the polling thread consumes a CPU core. Unprivileged use of SQPOLL has been allowed since kernel 5.12.
Multishot receive and provided buffer rings. Multishot recv (kernel 6.0 and later) lets a single submission produce a completion for every arrival, with the kernel picking buffers from a pre-registered pool. This removes the need to re-arm a receive after every packet. Receive bundles, which fill several buffers per operation, arrived in kernel 6.10.
Zero-copy receive (zcrx). Kernel 6.15 added io_uring zero-copy receive, which lets the NIC deliver payload data directly into application memory. This is the feature that actually removes copy_to_user from the receive path. It has strict requirements: the NIC must support hardware header/data split, you must configure flow steering and RSS so that only the intended flows hit a zero-copy receive queue, and the ring must be created with IORING_SETUP_SINGLE_ISSUER, IORING_SETUP_DEFER_TASKRUN and IORING_SETUP_CQE32. The NIC configuration is done outside io_uring. In a demonstration described by the pull request coverage, a 200G link was saturated from a single CPU core.

Without zcrx, a normal io_uring socket receive still copies data from kernel buffers into your buffers. What you save is system-call and wake-up overhead, not the copy itself. An accurate description of the pipeline is:

+-------------------------------------------------------------------+
| NETWORK: kernel TCP/IP stack, io_uring multishot recv             |
|          (zcrx on supported NICs and kernels >= 6.15)             |
+-------------------------------------------------------------------+
                                  │
                                  ▼
+-------------------------------------------------------------------+
| INGESTION THREADS: HTTP parsing, auth, size limits                |
+-------------------------------------------------------------------+
                                  │
                                  ▼  (memcpy into shared slot, no syscall)
+-------------------------------------------------------------------+
| SHARED MEMORY: one SPSC ring per worker                           |
+-------------------------------------------------------------------+
        │                         │                         │
        ▼                         ▼                         ▼
+---------------+         +---------------+         +---------------+
| Worker 1      |         | Worker 2      |         | Worker 3      |
+---------------+         +---------------+         +---------------+

Workers can verify HMAC-SHA256 signatures, apply business logic and dispatch downstream. The ingestion layer should acknowledge the sender only after the payload is safely enqueued (and, if you need durability, persisted), so a slow downstream never stretches your response time.

Full kernel bypass is a different tool. DPDK and AF_XDP move packet handling into user space. DPDK uses poll-mode drivers that poll NIC queues from user space, and its AF_XDP driver redirects packets from a NIC queue into a user-space memory buffer. Both bypass the kernel network stack, which means you must supply your own TCP, and TLS if needed. That is a large step up in complexity, and for a typical HTTP webhook receiver it is usually unnecessary. io_uring keeps the kernel stack and its tooling (ss, tcpdump, firewalling, congestion control).

5. Production hardening

Backpressure and durability

When consumers fall behind, the ring fills, and push returns false. Decide in advance what happens:

Shed load. For low-value telemetry, answer with 429 Too Many Requests (or 503 with a Retry-After) and let the sender retry. Dropping the oldest entries is a deliberate data-loss policy, so only use it where loss is acceptable.
Persist before acknowledging. For payments or other critical events, a shared-memory ring is not durable. A shm_open object lives in memory (on Linux typically on tmpfs), so a machine reboot loses it. Write to an append-only log or a durable queue before returning 2xx, and make processing idempotent, because senders retry and at-least-once delivery means duplicates.

Be careful with file-backed `mmap`

A MAP_SHARED mapping of a regular file lets the kernel write dirty pages back to storage on its own schedule, and it can evict pages under memory pressure. The 2022 CIDR paper “Are You Sure You Want to Use MMAP in Your Database Management System?” lists the resulting hazards. Page faults can happen at any memory access and stall the thread. Evicting pages requires TLB shootdowns, which are inter-processor interrupts that can hurt multi-core scaling. I/O errors surface as SIGBUS signals rather than return codes. The paper is about database buffer pools, but the same mechanisms apply to a file-backed ingest buffer. For a hot ring, prefer an anonymous or shm_open-backed region, pre-fault it at startup (for example with MAP_POPULATE or by touching every page), and use mlock or huge pages if you need to keep it resident. If you need a durable log, write it with explicit I/O so you control when data is flushed and how errors are handled.

NUMA and CPU affinity

On multi-socket machines, memory access cost depends on which socket owns the memory. Keep the NIC’s interrupt or polling cores, the ingestion threads and the shared-memory region on the same NUMA node where possible, and pin threads to cores with pthread_setaffinity_np or taskset. This reduces scheduler migrations and cross-socket cache traffic. Verify the effect with perf rather than assuming it.

Check the rest of the path

Shared memory only fixes the handoff. At this rate you should also look at TLS (many deployments terminate it at a load balancer so the webhook service sees plain HTTP), connection reuse, HTTP parsing cost and signature verification, which is a real CPU cost at 500k verifications per second.

6. What measurements actually show

You will see articles quote tidy tables claiming a 5x throughput gain and a 10x p99 improvement for an mmap/io_uring design. Treat any such table with suspicion unless the benchmark setup is published, because results depend on hardware, kernel version, payload size, connection count and what the application does per request.

Published, independent measurements are more modest:

A University of Edinburgh MSc project comparing io_uring with epoll in HTTP server benchmarks found io_uring ahead by up to 33% in throughput.
A single-threaded TCP echo benchmark with 1024-byte payloads measured about 176k requests/s for io_uring against 169k for epoll with one connection, and nearly identical results (about 421k) at 256 connections. At 512 connections epoll was ahead.
A Netty benchmark on Linux 6.6 found io_uring with half as many worker threads delivered about 3% higher throughput than epoll, with over twice the per-thread efficiency and far fewer CPU migrations.
The maintainers of uWebSockets have reported that io_uring can lose to epoll for larger messages when it uses more memory and falls out of CPU cache, and that localhost benchmarks can mislead.

The gains are real, but they are usually tens of percent, not an order of magnitude, and they depend heavily on the workload. For reference, public HTTP framework benchmarks (such as HttpArena) show well-tuned epoll and io_uring servers exceeding a million simple requests per second on large multi-core machines. So 500k RPS is reachable without exotic techniques when the per-request work is small. The real constraint is how much work each webhook needs.

To evaluate this design for your own workload:

Start with a baseline: a well-tuned epoll server with in-process queues, under realistic payloads and connection counts.
Profile with perf and flame graphs to see how much time is actually spent in system calls, copies, scheduling and TLS.
Change one thing at a time (batching or multishot receive, then SQPOLL, then shared-memory handoff) and measure throughput, CPU per request and p99/p999 latency.
Test on separate client and server machines over a real network, because loopback results can mislead.

Summary

Reaching 500k RPS on a webhook ingestion service means spending a few microseconds of CPU per request, so syscalls, copies, wake-ups and lock contention all matter. A shared-memory SPSC ring buffer gives you a clean, allocation-free, syscall-free handoff between ingestion and worker processes. io_uring reduces the cost of the network side through batching, multishot receive, optional SQPOLL, and zero-copy receive on kernel 6.15+ with supporting NICs. Neither technique removes the kernel TCP stack, and independent benchmarks suggest incremental rather than dramatic gains over a well-tuned epoll server. The reliable path is to measure your baseline, adopt these techniques where profiling shows a bottleneck, and design for backpressure and durability from the start.

References

Linux 6.15 io_uring zero-copy receive: https://www.phoronix.com/news/Linux-6.15-IO_uring
Kernel documentation for io_uring zero-copy Rx: https://kernel.org/doc/html/v6.15/networking/iou-zcrx.html
Evolution of io_uring Zero-Copy Receive (netdev 0x1a): https://netdevconf.info/0x1A/docs/netdev-0x1a-paper41-talk-paper.pdf
io_uring_setup(2) and SQPOLL: https://docs.rs/crate/axboe-liburing/latest/source/liburing-rs/docs/io_uring_setup.md
io_uring_prep_recv(3) (multishot, buffer rings, bundles): https://man7.org/linux/man-pages/man3/io_uring_prep_recv.3.html
io_uring SQPOLL overview: https://man.archlinux.org/man/extra/liburing/io_uring_sqpoll.7.en
Quantifying the Cost of Context Switch (Li, Ding, Shen): https://www.cs.rochester.edu/~kshen/papers/expcs2007.pdf
Context-switch measurements: https://web.eecs.umich.edu/%7Echshibo/files/microkiller_report.pdf
System call and context switch measurements (2010): https://git.captnemo.in/nemo/hn-classics/src/branch/master/_stories/2010/13930305.md
Are You Sure You Want to Use MMAP in Your Database Management System? (CIDR 2022): https://www.pdl.cmu.edu/ftp/Database/p13-crotty.pdf
DPDK AF_XDP poll mode driver: https://doc.dpdk.org/guides-22.07/nics/af_xdp.html
io_uring vs epoll HTTP server study (University of Edinburgh): https://project-archive.inf.ed.ac.uk/msc/20236779/msc_proj.pdf
io_uring and epoll echo server benchmark: https://github.com/joakimthun/io-uring-echo-server
Netty io_uring vs epoll benchmark: https://www.besthub.dev/articles/does-netty-s-io-uring-make-the-2-cpu-thread-rule-obsolete-3358ba3ca620
uWebSockets io_uring discussion: https://git.tdem.in/uNetworking/uWebSockets/issues/1603

Originally published at https://instawebhook.com/blog/memory-mapped-webhook-buffers-using-mmap-and-ring-buffers-to-cut-kernel-user-cro

Building a Resilient Webhook Ingress Proxy in Go: Handling Slow Consumers Without Thread Starvation

InstaWebhook — Sun, 01 Nov 2026 02:17:23 +0000

Payment gateways, CI/CD systems, e-commerce platforms and chat tools all push events to your servers as HTTP POST requests. Accepting those requests is cheap. The hard part is what happens when something behind your endpoint slows down: a database under lock contention, a downstream microservice in a garbage-collection spiral, or a third-party API having a bad afternoon.

A naive receiver that does the downstream work inside the request path, or spawns a goroutine for every event, degrades in a predictable way. Memory grows, sockets pile up, latency climbs, and eventually the process stops accepting new webhooks.

A note on terminology: Go multiplexes goroutines over a small set of OS threads, so the failure here is not classic thread-pool starvation. It is exhaustion of memory, sockets and file descriptors, plus unbounded queueing delay. This article uses “thread starvation” loosely, as most people searching for this problem do. The fix is the same: decouple ingestion from delivery and put hard limits on everything in between.

By the end you will have a small, complete Go service that verifies signatures, acknowledges quickly, buffers into a bounded queue, delivers through a fixed worker pool with a tuned HTTP transport, sheds load explicitly, and shuts down without losing queued work. The code was compiled with Go 1.22 and smoke-tested locally.

1. The Anatomy of the Webhook Ingress Problem

Every webhook provider gives you a deadline and a retry policy, and they differ a lot:

Provider	Response deadline	Automatic retries
GitHub	2xx within 10 seconds	None; failed deliveries must be redelivered manually or via the API
Shopify	2xx within 5 seconds	Up to 8 retries over about 4 hours; persistent failures can remove the subscription
Slack Events API	2xx within 3 seconds	Up to 3 retries with exponential backoff
Stripe	No specific timeout published; return 2xx before any complex logic	Live mode: up to 3 days with exponential backoff. Sandbox: 3 retries over a few hours

Three practical conclusions follow:

There is no universal “200–500 ms” rule. The tightest documented deadline among these providers is Slack’s 3 seconds. Acknowledge as fast as you can, but design for the strictest sender you integrate with.
Slow responses cause duplicates. If your handler misses a deadline, the provider counts the delivery as failed even if you eventually process it. Providers that retry will send the event again.
Dropping an event is not equally recoverable everywhere. If you shed load with a 503, Stripe, Shopify and Slack will retry. GitHub will not, so you must redeliver missed events yourself once you recover.

The architecture that satisfies all of this looks like this:

[ Third-Party Senders ]
         │  HTTP POST (webhooks at scale)
         ▼
┌─────────────────────────────────────────┐
│         Go Webhook Ingress Proxy        │
│  - Bounded body read                    │
│  - HMAC signature verification          │
│  - Fast validation, then 202 Accepted   │
└─────────────────────────────────────────┘
         │ non-blocking enqueue (full → 503 + Retry-After)
         ▼
[ Bounded in-memory queue ]  ──(optional)──►  [ Durable broker ]
         │
         ▼
[ Fixed worker pool ] ──► tuned HTTP transport ──► [ Downstream service ]

The Naive Anti-Pattern: Unbounded Goroutines

A common trap is spawning a goroutine per request to handle downstream delivery:

// ANTI-PATTERN: DO NOT USE IN PRODUCTION
func WebhookHandler(w http.ResponseWriter, r *http.Request) {
    payload := parseBody(r)

    go func() {
        if err := sendToSlowDownstream(payload); err != nil { // can block for a long time
            // handle error
        }
    }()

    w.WriteHeader(http.StatusOK)
}

Goroutines are cheap, not free. A goroutine starts with a 2 KB stack, and since Go 1.19 the runtime adapts the initial stack size to the program’s observed average stack usage. That is why the old “2–8 KB” rule of thumb is only approximate.

The real cost is everything each goroutine holds. Little’s Law says average concurrency equals arrival rate multiplied by time in the system. At 15,000 events per second:

With a 50 ms downstream, you have about 750 deliveries in flight.
When the downstream slows to 5 seconds, you have about 75,000.

The stacks alone are roughly 150 MB at 2 KB each. On top of that, each goroutine pins its payload and typically an open connection. The failure modes:

Memory growth. Stacks, payload copies and per-connection buffers accumulate with no upper bound.
GC pressure. The garbage collector has to scan every live goroutine’s stack, so tens of thousands of parked goroutines make each cycle more expensive.
Socket and file-descriptor exhaustion. Each in-flight call to a slow downstream holds a TCP connection, and every open connection consumes a file descriptor, which the OS caps per process (ulimit -n on Linux). Once you hit the cap, new inbound connections fail too.
Silent data loss. The handler returned 200 before the event was durable. If the process dies, those events are gone and the provider believes they were delivered.

Go’s HTTP server already runs each connection in its own goroutine, so the handler is exactly where you must not block on downstream work.

2. Architecture Blueprint: Decoupling Ingress and Egress

A resilient proxy splits an event’s lifecycle into three phases:

Ingress and validation. Read a size-limited body, verify the HMAC signature, check that the payload is well-formed, and return 202 Accepted.
Buffer. Enqueue into a bounded structure. A buffered Go channel is the simplest option, but it lives in memory only. A channel cannot be “backed by persistent storage”; durability requires writing to something external (see section 11).
Dispatch. A fixed pool of workers pulls from the buffer and delivers through a shared, tuned HTTP client, with timeouts, bounded retries and (optionally) a circuit breaker.

Every stage has a hard limit: body size, queue length, worker count, connection count, request timeout. When the downstream is slow, pressure propagates back to the one place designed to absorb it, the queue, and then to an explicit “no” at the front door.

3. Sizing the Pool: Little’s Law Again

Before writing code, decide the numbers:

Delivery capacity ≈ workers ÷ average delivery latency. With 100 workers and a 100 ms downstream, you can deliver about 1,000 events per second. If latency rises to 1 second, capacity falls to about 100 per second.
Queue runway ≈ queue size ÷ (arrival rate − delivery rate). If events arrive at 3,000 per second against a capacity of 1,000, a 5,000-slot queue fills in about 2.5 seconds.

Two things follow. A queue is a shock absorber for bursts, not a fix for sustained overload. And the queue size should be chosen from how much delay and memory you can tolerate, not set as large as possible. A very large queue only converts overload into long delays and eventually an out-of-memory crash.

4. Deep Dive 1: A Bounded Worker Pool

You will often see a two-level pattern in tutorials: a dispatcher, plus a chan chan Job that workers register their private channels into. It is unnecessary in Go. Goroutines blocked on receiving from a single channel are queued by the runtime, and each job goes to exactly one waiting worker. A single shared channel is simpler and easier to shut down correctly:

func (p *WorkerPool) Start(workers int) {
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go func() {
            defer p.wg.Done()
            for job := range p.jobs { // exits when the channel is closed and drained
                p.deliver(job)
            }
        }()
    }
}

The worker count is the concurrency limit toward the downstream. Whatever the downstream does, you never have more than workers deliveries in flight.

5. Deep Dive 2: Non-Blocking Enqueue and Backpressure

When arrivals outpace delivery, the handler must never block waiting for space. A select with a default case makes the enqueue non-blocking:

func (p *WorkerPool) Enqueue(job Job) bool {
    select {
    case p.jobs <- job:
        return true
    default:
        return false // queue full: shed load
    }
}

When Enqueue returns false, the handler answers 503 Service Unavailable (or 429 Too Many Requests) with a Retry-After header. Providers are not obliged to honor Retry-After, but providers that retry will use their own backoff schedule and try again. Choose your overflow policy deliberately:

Shed load. Return 503 and rely on sender retries. This works for Stripe, Shopify and Slack, but for GitHub you need your own redelivery process.
Spill to a durable queue. Write overflow to Redis Streams, RabbitMQ or Kafka instead of rejecting.
Drop selectively. For low-value events (heartbeats, noisy notifications), dropping may be acceptable. Make that an explicit per-event-type decision.

6. Deep Dive 3: Connection Pooling in Go’s HTTP Client

Go’s http.DefaultTransport is reasonable for general use, but it is tuned for modest concurrency: MaxIdleConns is 100, IdleConnTimeout is 90 seconds, and DefaultMaxIdleConnsPerHost is only 2. A webhook proxy that sends many concurrent requests to a single downstream host will keep at most 2 idle connections for it. Every other connection is closed after use and re-dialed, with a new TCP handshake and, for HTTPS, a new TLS handshake. Under load, that churn also produces many sockets in TIME_WAIT.

Two clarifications on common advice:

Creating a new http.Client is not itself the problem. A client with a nil Transport shares DefaultTransport. The problem is creating a new http.Transport per request, which defeats connection reuse entirely. Create one transport and share it.
To reuse a connection, the response body must be read to EOF and closed. Skipping the drain makes Go close the connection instead of returning it to the pool.

A transport tuned for a proxy delivering to one primary downstream:

transport := &http.Transport{
    Proxy: http.ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   5 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    ForceAttemptHTTP2:     true,
    MaxConnsPerHost:       100, // hard cap on connections to a single downstream
    MaxIdleConns:          100,
    MaxIdleConnsPerHost:   100, // the default is 2
    IdleConnTimeout:       90 * time.Second,
    TLSHandshakeTimeout:   5 * time.Second,
    ResponseHeaderTimeout: 8 * time.Second,
    ExpectContinueTimeout: 1 * time.Second,
}
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

Why these values matter for slow consumers:

MaxConnsPerHost and MaxIdleConnsPerHost should track the worker count. There is no point keeping 500 idle connections to a host when only 100 workers can use them.
Timeouts bound how long a worker can be held hostage. Without Client.Timeout and ResponseHeaderTimeout, a single hung downstream can occupy every worker indefinitely.
ForceAttemptHTTP2 is needed here. Go disables automatic HTTP/2 on a custom transport that sets DialContext or a TLS config unless you ask for it. With HTTP/2, requests are multiplexed over few connections, so the per-host idle settings matter less for HTTPS targets.

7. A Complete, Runnable Example

The program below combines everything: size-limited reads, HMAC verification, a non-blocking bounded queue, a worker pool with bounded retries and jitter, a tuned transport, and a graceful shutdown that drains the queue. It requires Go 1.22 or newer (method-based ServeMux patterns, math/rand/v2).

package main

import (
    "bytes"
    "context"
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "log"
    "math/rand/v2"
    "net"
    "net/http"
    "os"
    "os/signal"
    "strings"
    "sync"
    "syscall"
    "time"
)

const (
    maxWorkers   = 100
    queueSize    = 5000
    maxBodyBytes = 1 << 20 // 1 MiB
    maxAttempts  = 4
)

// Job is one accepted webhook waiting for delivery.
type Job struct {
    EventID string
    Body    []byte // the original bytes, forwarded untouched
}

// WorkerPool delivers jobs with a fixed number of goroutines.
type WorkerPool struct {
    ctx    context.Context // cancelled to abort in-flight work on a hard shutdown
    jobs   chan Job
    wg     sync.WaitGroup
    client *http.Client
    target string
}

func NewWorkerPool(ctx context.Context, client *http.Client, target string, queueSize int) *WorkerPool {
    return &WorkerPool{
        ctx:    ctx,
        jobs:   make(chan Job, queueSize),
        client: client,
        target: target,
    }
}

func (p *WorkerPool) Start(workers int) {
    for i := 0; i < workers; i++ {
        p.wg.Add(1)
        go func() {
            defer p.wg.Done()
            for job := range p.jobs { // exits when the channel is closed and drained
                p.deliver(job)
            }
        }()
    }
}

// Enqueue never blocks: it returns false when the queue is full.
func (p *WorkerPool) Enqueue(job Job) bool {
    select {
    case p.jobs <- job:
        return true
    default:
        return false
    }
}

// Drain stops accepting work and waits for queued jobs to finish or ctx to expire.
// Call it only after the HTTP server has stopped, so nothing can send on a closed channel.
func (p *WorkerPool) Drain(ctx context.Context) error {
    close(p.jobs)
    done := make(chan struct{})
    go func() { p.wg.Wait(); close(done) }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

func (p *WorkerPool) deliver(job Job) {
    backoff := 500 * time.Millisecond
    for attempt := 1; attempt <= maxAttempts; attempt++ {
        if p.ctx.Err() != nil {
            log.Printf("event %s abandoned: shutting down", job.EventID)
            return
        }
        retry, err := p.post(job)
        if err == nil {
            return
        }
        if !retry || attempt == maxAttempts {
            // In production: write to a dead-letter queue instead of only logging.
            log.Printf("event %s failed after %d attempt(s): %v", job.EventID, attempt, err)
            return
        }
        sleep := backoff + rand.N(backoff/2) // jitter avoids synchronized retries
        select {
        case <-time.After(sleep):
        case <-p.ctx.Done():
            return
        }
        backoff *= 2
    }
}

func (p *WorkerPool) post(job Job) (retry bool, err error) {
    req, err := http.NewRequestWithContext(p.ctx, http.MethodPost, p.target, bytes.NewReader(job.Body))
    if err != nil {
        return false, err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-Event-ID", job.EventID)

    resp, err := p.client.Do(req)
    if err != nil {
        return true, err // timeouts and connection errors are retryable
    }
    defer resp.Body.Close()
    // Read the body to EOF (bounded) so the connection can be reused.
    _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 64<<10))

    switch {
    case resp.StatusCode >= 200 && resp.StatusCode < 300:
        return false, nil
    case resp.StatusCode == http.StatusTooManyRequests,
        resp.StatusCode == http.StatusRequestTimeout,
        resp.StatusCode >= 500:
        return true, fmt.Errorf("downstream returned %d", resp.StatusCode)
    default:
        return false, fmt.Errorf("downstream rejected event with %d", resp.StatusCode)
    }
}

func validSignature(secret, body []byte, header string) bool {
    sig, ok := strings.CutPrefix(header, "sha256=")
    if !ok {
        return false
    }
    got, err := hex.DecodeString(sig)
    if err != nil {
        return false
    }
    mac := hmac.New(sha256.New, secret)
    mac.Write(body)
    return hmac.Equal(got, mac.Sum(nil)) // constant-time comparison
}

func webhookHandler(pool *WorkerPool, secret []byte) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
        body, err := io.ReadAll(r.Body)
        if err != nil {
            var tooBig *http.MaxBytesError
            if errors.As(err, &tooBig) {
                http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
                return
            }
            http.Error(w, "bad request", http.StatusBadRequest)
            return
        }

        // Authenticate before doing anything else with the payload.
        if !validSignature(secret, body, r.Header.Get("X-Hub-Signature-256")) {
            http.Error(w, "invalid signature", http.StatusUnauthorized)
            return
        }

        var evt struct {
            EventID string `json:"event_id"`
        }
        if err := json.Unmarshal(body, &evt); err != nil || evt.EventID == "" {
            http.Error(w, "invalid payload", http.StatusBadRequest)
            return
        }

        if !pool.Enqueue(Job{EventID: evt.EventID, Body: body}) {
            w.Header().Set("Retry-After", "5")
            http.Error(w, "overloaded, retry later", http.StatusServiceUnavailable)
            return
        }

        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusAccepted)
        _, _ = w.Write([]byte(`{"status":"accepted"}`))
    }
}

func main() {
    secret := os.Getenv("WEBHOOK_SECRET")
    target := os.Getenv("DOWNSTREAM_URL")
    if secret == "" || target == "" {
        log.Fatal("WEBHOOK_SECRET and DOWNSTREAM_URL must be set")
    }

    transport := &http.Transport{
        Proxy: http.ProxyFromEnvironment,
        DialContext: (&net.Dialer{
            Timeout:   5 * time.Second,
            KeepAlive: 30 * time.Second,
        }).DialContext,
        ForceAttemptHTTP2:     true,
        MaxConnsPerHost:       maxWorkers, // hard cap on connections to one downstream
        MaxIdleConns:          maxWorkers,
        MaxIdleConnsPerHost:   maxWorkers, // the default is only 2
        IdleConnTimeout:       90 * time.Second,
        TLSHandshakeTimeout:   5 * time.Second,
        ResponseHeaderTimeout: 8 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }
    client := &http.Client{Transport: transport, Timeout: 10 * time.Second}

    workCtx, cancelWork := context.WithCancel(context.Background())
    defer cancelWork()
    pool := NewWorkerPool(workCtx, client, target, queueSize)
    pool.Start(maxWorkers)

    mux := http.NewServeMux()
    mux.HandleFunc("POST /webhook", webhookHandler(pool, []byte(secret)))
    mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })

    srv := &http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second,
        ReadTimeout:       10 * time.Second,
        WriteTimeout:      10 * time.Second,
        IdleTimeout:       60 * time.Second,
    }

    sigCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    serverErr := make(chan error, 1)
    go func() { serverErr <- srv.ListenAndServe() }()
    log.Println("webhook ingress proxy listening on :8080")

    select {
    case err := <-serverErr:
        log.Fatalf("HTTP server error: %v", err)
    case <-sigCtx.Done():
        log.Println("shutdown signal received")
    }
    stop()

    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    // 1. Stop accepting requests and wait for in-flight handlers to return.
    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("HTTP shutdown: %v", err)
    }
    // 2. Let workers finish what is already queued, up to the deadline.
    if err := pool.Drain(shutdownCtx); err != nil {
        log.Printf("drain timed out with %d event(s) still queued", len(pool.jobs))
    }
    cancelWork() // abort anything still in flight
    log.Println("stopped")
}

Design notes on the code

http.MaxBytesReader instead of io.LimitReader. LimitReader silently truncates at the limit, so an oversized payload turns into confusing invalid JSON. MaxBytesReader returns a typed error, so the handler can answer 413.
Signature verification uses the raw bytes. Re-serializing JSON before checking can change the bytes and break the signature. The check uses hmac.Equal for constant-time comparison, and runs before JSON parsing and before the queue, so unauthenticated traffic never consumes queue capacity. The proxy also forwards the original bytes untouched.
Shutdown ordering matters. After Shutdown is called, ListenAndServe returns ErrServerClosed immediately, and the Go documentation warns that the program must not exit before Shutdown itself returns. A common bug is to call Shutdown in a goroutine and let main return as soon as ListenAndServe does. Here main waits for Shutdown, then closes the queue (safe, because no handler can still be enqueueing), then waits for workers.
Server timeouts are set. ReadHeaderTimeout, ReadTimeout, WriteTimeout and IdleTimeout protect the ingress side from slow senders, the mirror image of the slow-consumer problem.
Method routing. Since Go 1.22, ServeMux accepts patterns like "POST /webhook" and automatically replies 405 Method Not Allowed with an Allow header for other methods.

Try it against a slow consumer

Start any stub downstream that sleeps before replying, point DOWNSTREAM_URL at it, and send a signed request:

BODY='{"event_id":"evt_1","data":{}}'
SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$WEBHOOK_SECRET" | sed 's/^.* //')
curl -i -X POST localhost:8080/webhook \
  -H "X-Hub-Signature-256: sha256=$SIG" \
  -d "$BODY"

You should see 202 Accepted within milliseconds regardless of how slow the stub is. A request with a bad signature returns 401, and a GET returns 405. To see load shedding, make the stub sleep for several seconds and fire more than maxWorkers + queueSize events quickly. The surplus gets 503 with Retry-After, while the process’s goroutine count stays flat.

8. Observability: Know When You Are Absorbing Pain

Building the proxy is half the job. These signals tell you whether a slow consumer is hurting you:

Ingestion rate and response codes. Track 202, 401, 413 and 503 separately. A rising 503 rate means the queue is overflowing.
Queue depth (len(jobs)) and oldest-event age. Depth near capacity means your workers cannot keep up. Age is often more useful than depth because it is the delay your users actually experience.
In-flight deliveries against the worker count. If every worker is permanently busy, you are at capacity.
Downstream latency and error rate (p95/p99 of outgoing calls), plus retry counts and dead-lettered events.
Goroutine count via runtime.NumGoroutine(). In this design it should be roughly constant: workers plus per-connection server goroutines. A steady climb indicates a leak. Go 1.27 (released August 2026) also makes the goroutineleak profile in runtime/pprof generally available, after it was an experiment in Go 1.26.

9. Idempotency: Assume Duplicates

Retries mean duplicates, so your downstream must tolerate seeing an event more than once. Stripe’s documentation says events can be delivered more than once and are not guaranteed to arrive in order. Slack documents retry headers (X-Slack-Retry-Num, X-Slack-Retry-Reason). GitHub reuses the same X-GitHub-Delivery GUID when you request a redelivery.

Practical rules:

Deduplicate on the provider’s event ID, with a TTL longer than the provider’s retry window.
Authenticate first, then deduplicate. GitHub’s delivery header is not itself a signed event ID, so don’t trust identifiers until the signature has been verified.
Don’t assume ordering. Design handlers so that a late or repeated event cannot overwrite newer state.
The proxy above forwards an X-Event-ID header so the downstream can implement idempotency cheaply.

10. Security Notes

Use the right header. GitHub’s SHA-256 signature is in X-Hub-Signature-256. Don’t rely on the older SHA-1 X-Hub-Signature.
Reject stale requests where the provider supports it. Some providers include a timestamp in the signed material, which lets you reject replays. Check your provider’s documentation.
Consider IP allow-lists as an extra layer. GitHub publishes its delivery IP ranges through its /meta endpoint and notes that they change occasionally, so refresh the list periodically. Treat it as defense in depth, not a replacement for signatures.
Terminate TLS properly and require HTTPS. Both GitHub and Stripe expect HTTPS endpoints.

11. Advanced Enhancements

Durability. An in-memory queue loses everything on a crash or restart, and a 202 only promises that the event reached memory. For high-stakes events such as payments, write to a durable broker (Redis Streams, RabbitMQ, Kafka) before acknowledging, and let the workers consume from there. A common compromise is the in-memory queue as a fast path with a durable spillover when it fills.

Per-destination isolation. One shared pool means one slow destination can occupy every worker and starve healthy ones (head-of-line blocking). If you fan out to several downstreams, give each its own queue and concurrency limit.

Circuit breaking. When a downstream is failing, retrying only adds load. The sony/gobreaker library (use the generic v2 module) wraps calls in a breaker that fails fast once failures cross a threshold:

import "github.com/sony/gobreaker/v2"

cb := gobreaker.NewCircuitBreaker[int](gobreaker.Settings{
    Name:    "downstream",
    Timeout: 30 * time.Second, // how long the breaker stays open before probing
    ReadyToTrip: func(c gobreaker.Counts) bool {
        return c.Requests >= 20 &&
            float64(c.TotalFailures)/float64(c.Requests) >= 0.5
    },
})

status, err := cb.Execute(func() (int, error) {
    // perform the POST and return the status code
})
if errors.Is(err, gobreaker.ErrOpenState) {
    // don't burn a retry attempt; requeue with a delay or pause consumption
}

Decide what workers do while the breaker is open. Failing fast without pausing consumption simply moves events from the queue into your retry or dead-letter path faster.

Rate limiting. Cap per-sender or per-tenant ingress and per-destination egress so one noisy source cannot consume the shared queue.

12. Conclusion

A webhook proxy is resilient when every resource is bounded and every limit has an explicit failure behavior:

Bounded body size protects memory at the edge.
A fixed worker pool caps concurrency toward the downstream, regardless of how slow it gets.
A bounded, non-blocking queue absorbs bursts and turns sustained overload into a fast, explicit 503.
A shared, tuned http.Transport with timeouts, connection caps and body draining prevents connection churn and hung workers.
Ordered graceful shutdown drains accepted work instead of dropping it.
Idempotent consumers and durable queues cover duplicates and crashes.

None of this makes failures impossible. It makes them bounded, visible and recoverable, which is what you want when someone else’s system is slow and your ingress tier still has to stay up.

Sources and Further Reading

Originally published at https://instawebhook.com/blog/building-a-resilient-webhook-ingress-proxy-in-go-handling-slow-consumers-without

High-Performance Ingress & Network Architecture: Building Resilience at Scale

InstaWebhook — Sat, 31 Oct 2026 02:15:30 +0000

Webhook ingress gateways have to absorb bursty traffic from third-party senders and stay responsive when some of those connections are slow or hostile. This article covers three techniques for hardening such a gateway: using HTTP/3 and QUIC where they fit, defending against slow-connection attacks (Slowloris and Slow-Read), and using memory-mapped (mmap) ring buffers to absorb bursts. It also states where each technique stops helping, because each one is often oversold.

The article is aimed at teams who run their own ingress layer, whether they receive webhooks from providers such as Stripe or GitHub, or ingest events from their own devices and agents.

1. Know the Contract Before Tuning the Network

Network tuning only matters relative to what the sender expects from you. Two documented examples:

Stripe expects a webhook endpoint to use HTTPS, accept POST requests with a JSON event payload, and quickly return a 2xx status. In live mode, Stripe attempts delivery of an event for up to three days with exponential backoff. Events created in a sandbox are retried three times over a few hours.
GitHub expects a 2xx response within 10 seconds. If your server takes longer, GitHub terminates the connection and counts the delivery as failed. GitHub’s guidance is to enqueue the payload and process it in the background. GitHub does not automatically redeliver failed deliveries, so recovering from an outage means redelivering missed webhooks yourself once the server is back.

Deadlines and retry behavior vary by provider, so check each sender’s documentation rather than assuming a general standard. Two consequences follow for gateway design:

The gateway’s main job is to acknowledge fast and persist durably. Everything else (parsing, business logic, fan-out) belongs behind a queue.
Your own endpoint must tolerate duplicates. Retries and redeliveries mean the same event can arrive more than once, so deduplicate on an event or delivery ID.

2. HTTP/3 and QUIC: What They Fix and What They Don’t

Where TCP-based HTTP hurts

Handshake cost. A new HTTPS connection over TCP needs a TCP handshake plus a TLS handshake before the first request byte, which is roughly two round trips with TLS 1.3 and more with TLS 1.2. QUIC integrates TLS 1.3 into its own handshake, so a new connection completes in one round trip. Resumed connections can send application data in the first flight (0-RTT).
Transport-level head-of-line (HoL) blocking. TCP delivers a byte stream in order. When a segment is lost, data that arrived after it waits in the receive buffer until the retransmission arrives. HTTP/2 multiplexes many streams over one TCP connection, so one lost packet can stall every stream on that connection. HTTP/1.1 sidesteps this by using many connections, at the cost of more handshakes and sockets.

TCP head-of-line blocking (HTTP/2 over one TCP connection):
Packet 1 [OK] -> Packet 2 [LOST] -> Packet 3 [HELD] -> Packet 4 [HELD]
                 Delivery to all streams stalls until Packet 2 is retransmitted

What QUIC changes

HTTP/3 (RFC 9114) runs over QUIC (RFC 9000), a UDP-based transport with TLS 1.3 built in. QUIC streams are independent at the transport layer, so loss on one stream does not block delivery on another.

QUIC independent streams:
Stream A: frame [OK]   -> delivered immediately
Stream B: frame [LOST] -> retransmitted, only Stream B waits
Stream C: frame [OK]   -> delivered immediately

Two caveats matter for webhook gateways:

0-RTT data is replayable. An attacker who captures a 0-RTT flight can replay it, so 0-RTT should only carry idempotent requests. The quic-go library makes this explicit: its client allows 0-RTT only for GET requests and notes that 0-RTT provides no replay protection. A webhook is a POST that triggers side effects, so for webhook endpoints leave 0-RTT off or make processing strictly idempotent.
Plain HTTP/3 already gives you stream independence. You do not need datagrams to get the head-of-line benefit. Ordinary HTTP/3 request streams are reliable and independent.

Where HTTP/3 Datagrams fit

QUIC DATAGRAM frames (RFC 9221) carry unreliable data. They are not retransmitted if lost, but unlike raw UDP they remain subject to the connection’s congestion control. HTTP/3 Datagrams (RFC 9297) map them onto HTTP requests. They are associated with a request stream, and in practice they are used with extended CONNECT, as in WebTransport and CONNECT-UDP (RFC 9298).

That has direct consequences for webhook design:

Standard webhook providers do not send datagrams. They send HTTPS POST requests, and the documentation of the providers cited above specifies exactly that. Datagrams are relevant only when you control both ends, for example telemetry agents, IoT devices, or location trackers that you operate.
Datagrams suit data where the newest value supersedes older ones. For a status heartbeat or a position update, dropping a stale value is better than delaying fresh ones behind a retransmission. They are a poor fit for anything that must not be lost, such as payment events.
Datagrams are size-limited. A datagram must fit in a single QUIC packet, so payloads are limited to roughly the path MTU minus protocol overhead. Larger payloads need ordinary streams.

Server sketch in Go (`quic-go`)

The quic-go HTTP/3 package supports HTTP Datagrams (RFC 9297) on both client and server, and the HTTP/3 API changed substantially across releases. For example, Settingser was removed in v0.43.0, and Settings.EnableDatagram was renamed EnableDatagrams. At the time of writing, pkg.go.dev lists v0.61.0 as the latest release. Treat the following as a sketch and verify the types against the version you pin.

package main

import (
    "log"
    "net/http"

    "github.com/quic-go/quic-go"
    "github.com/quic-go/quic-go/http3"
)

// Bounded hand-off to the persistence layer. When it is full, datagrams
// are dropped, which is acceptable for supersedable telemetry.
var ingest = make(chan []byte, 4096)

func datagramHandler(w http.ResponseWriter, r *http.Request) {
    // HTTP Datagrams ride on an HTTP request stream, typically extended CONNECT.
    if r.Method != http.MethodConnect {
        http.Error(w, "CONNECT required", http.StatusMethodNotAllowed)
        return
    }
    w.WriteHeader(http.StatusOK)

    // The ResponseWriter implements http3.HTTPStreamer, which gives access
    // to the underlying request stream and its datagram methods.
    str := w.(http3.HTTPStreamer).HTTPStream()
    defer str.Close()

    for {
        data, err := str.ReceiveDatagram(r.Context())
        if err != nil {
            return // connection closed or request canceled
        }
        select {
        case ingest <- data:
        default: // queue full: drop rather than spawn unbounded goroutines
        }
    }
}

func main() {
    server := http3.Server{
        Addr:            ":4433",
        EnableDatagrams: true,
        QUICConfig:      &quic.Config{EnableDatagrams: true},
        Handler:         http.HandlerFunc(datagramHandler),
    }
    log.Fatal(server.ListenAndServeTLS("cert.pem", "key.pem"))
}

Note the bounded channel: spawning a goroutine per datagram, as many examples do, lets an attacker or a traffic spike create unbounded memory growth.

QUIC also has its own amplification and address-validation concerns, so the usual UDP-facing protections (rate limiting, retry tokens, and a fronting proxy or CDN that terminates QUIC) apply to any public HTTP/3 listener.

3. Preventing Slowloris and Slow-Read Attacks

Webhook endpoints are public by necessity, which makes them targets for application-layer “low-and-slow” attacks that exhaust connection capacity rather than bandwidth.

The attack patterns

Slowloris (popularized by Robert Hansen in 2009). The attacker opens many connections and sends partial HTTP request headers, adding a header line every few seconds to keep each connection alive. If the server dedicates a thread, process, or fixed connection slot to each connection, the pool fills and legitimate requests are refused.
Slow request body. The attacker sends valid headers, then trickles the body (including chunked bodies) at a few bytes per interval.
Slow-Read. The attacker sends a valid request but advertises a very small TCP receive window, so the server can only transmit its response in tiny segments. Response buffers and connection state stay allocated for a long time.

Slowloris lifecycle:
Client --[ POST /webhook HTTP/1.1 ]--------> Server allocates connection state
Client --[ ...wait... one more header ]-----> connection stays open
Client --[ ...wait... one more header ]-----> connection stays open
Result: connection slots exhausted, legitimate senders refused

Event-driven servers such as NGINX handle idle connections cheaply, so thread-per-connection servers are the most exposed. Even so, every server has a finite number of connections and file descriptors, so timeouts and limits still matter.

Key principle: most timeouts measure gaps, not totals

A common mistake is assuming client_header_timeout-style settings bound the total time to receive a request. In NGINX, client_header_timeout, client_body_timeout, and send_timeout apply to the interval between two successive read or write operations, not to the whole transfer. An attacker who sends one byte every few seconds can therefore stay under a gap-based timeout indefinitely. Effective defense combines several layers:

gap timeouts (so completely idle connections are dropped quickly),
caps on header and body size,
per-IP or per-client connection limits,
an absolute deadline for receiving the request head, where the server supports one,
and a front proxy or CDN that absorbs the slow clients.

NGINX has no directive for a minimum transfer rate. If you need rate-based eviction, you must implement it in your own server code or rely on a CDN or WAF feature.

NGINX example

http {
    # Timeouts between successive reads/writes, not whole-request limits
    client_header_timeout 5s;
    client_body_timeout   10s;
    send_timeout          10s;

    # Drop timed-out connections immediately instead of lingering
    reset_timedout_connection on;

    # Bound request size; webhooks are usually small
    client_max_body_size 2m;

    # Per-IP concurrent connection cap
    limit_conn_zone $binary_remote_addr zone=per_ip:10m;

    server {
        listen 443 ssl;
        http2 on;   # the "http2" parameter of listen is deprecated since 1.25.1
        server_name ingress.example.com;

        limit_conn per_ip 100;

        location /api/v1/webhooks {
            # NGINX buffers the request body by default, so a slow client
            # does not hold an upstream connection while it trickles data.
            proxy_pass http://internal_webhook_cluster;
        }
    }
}

Two practical notes. First, the http2 directive was introduced in NGINX 1.25.1, which is why listen 443 ssl http2; is now deprecated. Second, per-IP limits can punish legitimate senders: providers often deliver from a small set of shared IP ranges, so give published sender ranges (such as those GitHub exposes through its /meta endpoint) higher limits or exempt them, and apply the strictest limits to everyone else.

Go `net/http` example

srv := &http.Server{
    Addr:              ":8443",
    ReadHeaderTimeout: 4 * time.Second,  // absolute deadline for request headers
    ReadTimeout:       15 * time.Second, // absolute deadline for the whole request
    WriteTimeout:      15 * time.Second,
    IdleTimeout:       60 * time.Second,
    MaxHeaderBytes:    16 << 10,
}

// In the handler, also cap the body size:
// r.Body = http.MaxBytesReader(w, r.Body, 2<<20)

Go’s server timeouts are absolute, per-request deadlines, which is stricter than NGINX’s gap-based timeouts. The trade-off is that a fixed ReadTimeout can cut off a large but legitimate upload on a slow link. If that matters, use http.ResponseController to extend the read deadline as data arrives.

Rust (Actix Web) example

use actix_web::{web, App, HttpResponse, HttpServer};
use std::time::Duration;

async fn handle_webhook(_body: web::Bytes) -> HttpResponse {
    // Persist or enqueue the payload, then acknowledge quickly.
    HttpResponse::Ok().finish()
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(web::resource("/webhook").to(handle_webhook)))
        // Time allowed for the client to send the full request head.
        // Slow clients receive 408 Request Timeout. Default: 5 seconds.
        .client_request_timeout(Duration::from_secs(4))
        // Time allowed for connection shutdown. Default: 1 second.
        .client_disconnect_timeout(Duration::from_secs(1))
        .bind(("0.0.0.0", 8080))?
        .run()
        .await
}

client_request_timeout covers only reading the request head, and client_disconnect_timeout governs connection shutdown; neither is a minimum-throughput control. Enforce body size limits and body-read deadlines separately.

Related protocol-level attacks

HTTP/2 introduced its own resource-exhaustion vector: the “Rapid Reset” attack (CVE-2023-44487), disclosed in October 2023, abuses stream creation and immediate cancellation to overload servers. Keep your HTTP/2 implementation and proxies patched and configure concurrent-stream limits. Slow-connection defenses and HTTP/2 hardening are separate concerns.

4. Surviving Traffic Bursts: Durable Buffering with `mmap`

The problem

Bursts of webhooks happen when an upstream system emits many events at once, such as a batch job, a retry storm after your own outage, or a bulk operation. If the ingress process holds every in-flight payload in heap memory (Go slices, Rust Vec<u8>), memory grows with the burst. In a container, exceeding the memory limit gets the process killed by the kernel’s OOM killer and in-flight data is lost.

The most reliable defense is architectural, not a buffer trick:

Cap request body sizes and concurrent in-flight requests.
Apply backpressure: return 429 or 503 with Retry-After when capacity is exhausted. Providers that retry, such as Stripe, will redeliver later.
Acknowledge only after the payload is durably stored, so an acknowledged event is never lost.

What an `mmap` ring buffer gives you

mmap maps a file into the process’s address space. Writes go to kernel page-cache pages that the kernel writes back to disk in the background. A fixed-size file used as a circular buffer gives you a bounded, disk-backed staging area with these properties:

Bounded, preallocated capacity. The buffer cannot grow past the file size. When it is full you reject new work explicitly instead of crashing.
Survives process crashes. Pages written through a MAP_SHARED mapping belong to the kernel’s page cache, so data written before the process dies is not lost when the process is killed. It is not durable against power loss or a kernel crash unless you flush with msync or fsync. Each flush costs latency, so choose a flush policy deliberately.
Less heap pressure. Payload bytes live in file-backed pages rather than the Go or Rust heap, which relieves garbage-collector and allocator pressure.

What it does not give you

It is not “zero OOM.” On Linux, page cache and file-backed memory are charged to the container’s memory cgroup. Kubernetes’ working-set metric, which is what the OOM decision tracks, includes active file-backed memory. The kernel can reclaim clean pages and write back dirty ones, but under heavy write load into a tight limit, dirty pages can cause memory-cgroup OOM kills. Test your configuration under load, tune dirty-page writeback, and size limits with headroom.
It is not zero-copy. Copying the payload from the network buffer into the mapping is still a memory copy. The savings are in allocation and GC behavior, not in copy count.
It needs a defined recovery protocol. After a crash, the consumer must be able to find valid records. Production designs add record checksums or sequence numbers and flush ordering rules.

A correct single-producer, single-consumer ring in Go

Head and tail are monotonically increasing byte counters, with the position in the buffer computed modulo capacity. Records are length-prefixed and may wrap around the end of the region, which the copy helpers handle. This version supports exactly one producer goroutine and one consumer goroutine; with multiple producers, serialize writes with a mutex or give each producer its own ring.

package ringbuf

import (
    "encoding/binary"
    "errors"
    "fmt"
    "os"
    "sync/atomic"
    "syscall"
    "unsafe"
)

const (
    headerSize = 64 // head at offset 0, tail at offset 8, rest reserved
    lenPrefix  = 8
)

var (
    ErrFull  = errors.New("ring buffer full")
    ErrEmpty = errors.New("ring buffer empty")
)

type Ring struct {
    file   *os.File
    mem    []byte // whole mapping, including header
    region []byte // data area
    head   *uint64
    tail   *uint64
}

// Open maps a file-backed ring of the given total size. Reopening an
// existing file of the same size preserves its contents and positions.
func Open(path string, size int) (*Ring, error) {
    f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o600)
    if err != nil {
        return nil, err
    }
    if err := f.Truncate(int64(size)); err != nil {
        f.Close()
        return nil, err
    }
    mem, err := syscall.Mmap(int(f.Fd()), 0, size,
        syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
    if err != nil {
        f.Close()
        return nil, fmt.Errorf("mmap failed: %w", err)
    }
    return &Ring{
        file:   f,
        mem:    mem,
        region: mem[headerSize:],
        head:   (*uint64)(unsafe.Pointer(&mem[0])),
        tail:   (*uint64)(unsafe.Pointer(&mem[8])),
    }, nil
}

// Write appends one record. It returns ErrFull instead of overwriting
// unread data, so the caller can apply backpressure (for example HTTP 503).
func (r *Ring) Write(p []byte) error {
    need := uint64(lenPrefix + len(p))
    capacity := uint64(len(r.region))
    tail := atomic.LoadUint64(r.tail)
    head := atomic.LoadUint64(r.head)
    if tail-head+need > capacity {
        return ErrFull
    }
    var hdr [lenPrefix]byte
    binary.LittleEndian.PutUint64(hdr[:], uint64(len(p)))
    r.copyIn(tail, hdr[:])
    r.copyIn(tail+lenPrefix, p)
    atomic.StoreUint64(r.tail, tail+need) // publish only after data is written
    return nil
}

// Read removes and returns the oldest record.
func (r *Ring) Read() ([]byte, error) {
    head := atomic.LoadUint64(r.head)
    tail := atomic.LoadUint64(r.tail)
    if head == tail {
        return nil, ErrEmpty
    }
    var hdr [lenPrefix]byte
    r.copyOut(head, hdr[:])
    n := binary.LittleEndian.Uint64(hdr[:])
    out := make([]byte, n)
    r.copyOut(head+lenPrefix, out)
    atomic.StoreUint64(r.head, head+lenPrefix+n)
    return out, nil
}

func (r *Ring) copyIn(pos uint64, src []byte) {
    off := pos % uint64(len(r.region))
    n := copy(r.region[off:], src)
    copy(r.region, src[n:]) // wrap around
}

func (r *Ring) copyOut(pos uint64, dst []byte) {
    off := pos % uint64(len(r.region))
    n := copy(dst, r.region[off:])
    copy(dst[n:], r.region) // wrap around
}

// Sync flushes dirty pages to disk. Call it on your chosen durability schedule.
func (r *Ring) Sync() error {
    _, _, errno := syscall.Syscall(syscall.SYS_MSYNC,
        uintptr(unsafe.Pointer(&r.mem[0])), uintptr(len(r.mem)), uintptr(syscall.MS_SYNC))
    if errno != 0 {
        return errno
    }
    return nil
}

func (r *Ring) Close() error {
    _ = syscall.Munmap(r.mem)
    return r.file.Close()
}

This sketch uses Linux-specific system calls and omits checksums, corruption recovery, and multi-producer support. Treat it as a starting point rather than production code; mature alternatives such as a write-ahead log library or an embedded queue are often a better choice than writing your own.

Choosing a buffering strategy

I deliberately omit latency figures for the options below, because they depend heavily on hardware, payload size, and configuration. Benchmark on your own infrastructure.

Strategy	Memory behavior	Durability	Main risk
In-process heap queue	Grows with the burst	Lost on process crash	OOM kill under bursts
Remote queue or broker (Redis, Kafka, RabbitMQ)	Offloaded to a separate system	Depends on broker configuration	Network hop and an extra dependency on the ingest path
`mmap`-backed ring on local disk	Bounded file size; page cache counts toward cgroup limits	Survives process crash; power loss needs `msync`/`fsync`	Dirty-page pressure and a custom recovery protocol

5. Putting It Together

A defensible layered design looks like this:

Public senders
      │
      ▼
┌──────────────────────────────────────────────────────┐
│ Edge layer: CDN / reverse proxy (NGINX, Envoy, etc.) │
│  • Header/body gap timeouts, size caps, conn limits  │
│  • Terminates TLS (and HTTP/2, optionally HTTP/3)    │
│  • Buffers slow clients away from your application   │
└───────────────────────────┬──────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────┐
│ Ingress worker (Go / Rust)                           │
│  • Verify signature, enforce body limit              │
│  • Append to durable buffer; ack only after success  │
│  • Return 503 + Retry-After when the buffer is full  │
└───────────────────────────┬──────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────┐
│ Durable buffer (mmap ring, WAL, or message broker)   │
└───────────────────────────┬──────────────────────────┘
                            ▼
              Async consumers: dedupe, process, route

The priorities, in order:

Acknowledge fast, process later. Providers publish deadlines (10 seconds for GitHub, for example), and handlers that do real work inline miss them and trigger retries.
Bound everything. Connection counts, header and body sizes, queue depth, and buffer capacity all need explicit limits, with backpressure when they are reached.
Use HTTP/3 where you control the clients and it measurably helps. For third-party webhook providers, expect HTTPS POST over HTTP/1.1 or HTTP/2 and design for that. Reserve datagrams for loss-tolerant, supersedable telemetry from your own clients.
Make duplicates harmless. Retries and redelivery are part of every provider’s delivery model, so idempotent processing is mandatory.
Load-test the failure modes. Simulate slow clients, bursts at several times your normal peak, and a process kill mid-write, and watch the container’s memory working set, not just the process RSS.

Deterministic Reordering of Concurrent Webhooks: Handling Out-of-Order State Transitions with CRDTs

InstaWebhook — Fri, 30 Oct 2026 02:13:23 +0000

Introduction: The Multi-Region Webhook Delivery Dilemma

Webhooks are the default way for SaaS providers to push state changes to your systems. Once you accept them through several regional ingress points, you inherit a hard problem: the order in which events arrive is not the order in which they happened.

Providers say this openly. Stripe states that it does not guarantee events arrive in the order they were generated, and that the same event can be delivered more than once. In live mode it retries failed deliveries with exponential backoff for up to three days. Its own example is a subscription creation that can emit customer.subscription.created, invoice.created, invoice.paid and charge.created, with no promise about their arrival order. Shopify and GitHub give similar warnings. GitHub says deliveries can arrive in a different order than the events occurred. It also does not automatically redeliver failed deliveries, so you must redeliver them manually. Analyses of dozens of webhook platforms find the same pattern: most providers disclaim ordering without supplying the metadata a consumer needs to rebuild it.

Deploying ingress in several regions adds more reordering. Different network paths, queue workers that finish jobs in a different order than they started them, and retries all interleave events for the same resource.

Retries also mean you cannot get exactly-once delivery. If a sender never retries, it loses data when a delivery fails. If it does retry, it will occasionally duplicate events that had already arrived. The practical target is therefore at-least-once delivery plus an idempotent, order-insensitive consumer.

This article shows one way to build such a consumer. It combines Conflict-Free Replicated Data Types (CRDTs) with logical timestamps and a deterministic tie-breaker, so that every replica converges to the same state no matter how events are interleaved. It also covers when this design is overkill.

1. The Anatomy of Out-of-Order Webhook Failures

HTTP itself gives you no delivery guarantee. Reliability comes from the provider’s retry logic, and that logic is the source of both duplicates and reordering.

Common Failure Modes in Multi-Region Ingress

The ghost update. An updated event arrives before the created event it depends on. A handler that does a strict update fails. An upsert that fills in missing fields with defaults or nulls creates a half-populated record.
State reversal. An older event is retried or delayed and arrives after a newer one has been committed. If the handler applies events in arrival order, it overwrites newer state with older state.
Concurrent mutation divergence. Two regions ingest different events for the same resource at nearly the same time and apply them in different orders. Without a merge rule that gives the same result regardless of order, the replicas diverge permanently.

Provider emits:  E1 (created, t=1)   E2 (updated, t=2)

                 E1 ── slow route / retry ──────────► Region B ── applies 2nd
                 E2 ── fast route ───────► Region A ── applies 1st

Region A sees: E2, E1     Region B sees: E1, E2
Arrival-order handling → A and B end in different states

A common first fix is Last-Write-Wins (LWW) on the payload’s wall-clock timestamp. That fix is only as good as the clocks behind it. If the timestamps come from machines with skewed clocks, or you compare them against your own server’s Date.now(), “latest” is not reliable. Plain LWW also silently discards one side of a true concurrent write. LWW is not inherently broken, though. As shown below, an LWW register with logical timestamps and a deterministic tie-breaker is itself a valid CRDT.

2. Foundations: CRDTs and Strong Eventual Consistency

CRDTs were formalized by Shapiro, Preguiça, Baquero and Zawirski in 2011. They defined Strong Eventual Consistency (SEC) and gave sufficient conditions under which replicas of a data type are guaranteed to converge without rollback or coordination. SEC requires three things:

Eventual delivery. An update applied at one correct replica is eventually applied at all correct replicas.
Termination. Every update execution finishes.
Strong convergence. Replicas that have applied the same set of updates are in equivalent states, regardless of order.

State-based CRDTs (CvRDTs)

A state-based CRDT models its state as a join-semilattice. This is a partially ordered set in which every pair of states has a least upper bound, and the merge function computes that bound (⊔). The merge must satisfy three laws:

Commutative: A ⊔ B = B ⊔ A, so arrival order does not matter.
Associative: (A ⊔ B) ⊔ C = A ⊔ (B ⊔ C), so batching does not matter.
Idempotent: A ⊔ A = A, so duplicate deliveries are harmless.

Updates must also be monotonic: a state may only move up the lattice. This is why a plain counter cannot support decrement in a naive state-based design, since merging by max would ignore it.

Operation-based CRDTs (CmRDTs)

Operation-based CRDTs send operations instead of state. They are compact on the wire, but they push guarantees onto the transport: each operation must be delivered exactly once, and in causal order where the data type requires it.

That is the wrong fit for webhooks, which are at-least-once and unordered, and whose operation history you do not control. Webhook handlers should therefore treat each payload as a state fragment and merge it with a state-based join. State-based merges tolerate duplicate and out-of-order delivery by construction.

Logical Clocks

To order events without trusting physical clocks, use logical ones:

Lamport timestamps (Lamport, 1978) guarantee that if event a causally precedes b, then L(a) < L(b). The converse does not hold, so they cannot tell you whether two events were concurrent.
Vector clocks (Fidge and Mattern, 1988) do capture causality, so concurrent events can be detected. They cost one counter per participant.
Provider-supplied versions are the best source of all when they exist: a monotonic sequence number per resource, an updated_at or version field on the object, or an explicit parent event reference.

If the provider supplies none of these, anything you stamp at the edge reflects arrival order at your edge, not the provider’s order. Edge stamping can still give you a consistent internal order for deduplication and replay. It cannot recover the provider’s true event order.

Deterministic Tie-Breaking

When two events carry the same logical timestamp, compare a fixed sequence of keys: (logicalTimestamp, actorId, eventId). Every replica applies the same comparison, so every replica picks the same winner. This works only if that tuple is unique per event. That is why the example below includes eventId as the final key.

3. Architecture of a Deterministic Reordering Layer

+--------------------------------------------------------------+
|                   Multi-Region Edge Ingress                  |
|   verify signature · return 2xx fast · persist raw event     |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|             Dedup + Metadata Extraction                      |
|   eventId · resourceId · logical timestamp / version · actor |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|                  CRDT State Resolution                       |
|   event → state fragment → join (⊔) with stored state        |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|        Visibility Gate (dependencies) + Reconciliation       |
|   hold side effects until `created` seen; TTL → API fetch    |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|                    Persisted State Storage                   |
+--------------------------------------------------------------+

Ingestion. Verify the signature, acknowledge with a 2xx quickly, and process asynchronously. Providers enforce delivery timeouts. For example, Primer documents a 10-second timeout. A slow synchronous handler therefore causes retries, which cause duplicates.
Deduplication. Store processed event IDs with a unique constraint. Stripe events carry a stable id for this purpose. Deduplication is not the same as domain idempotency, though. A provider can emit two distinct events that describe the same business transition. The idempotent join below handles that case as well.
Join. Convert the event into a state fragment and merge it into the stored state. Because the merge is commutative, associative and idempotent, arrival order is irrelevant to the final state.
Visibility gate. The join converges even when updated arrives before created. What you still need to hold back are side effects and foreign-key-dependent writes. Keep the merged state internal until the creation event has been seen. If it never arrives within a time limit, fall back to fetching the resource from the provider’s API.

4. Reference Implementation (TypeScript)

This sketch models a resource as a map of per-field LWW registers, plus a grow-only created flag and a delete tombstone. It is not a drop-in production system.

interface WebhookEvent {
  eventId: string;
  resourceId: string;
  eventType: 'CREATED' | 'UPDATED' | 'DELETED';
  logicalTimestamp: number;   // provider version/sequence, or a Lamport value
  actorId: string;            // issuing node/region, used for tie-breaking
  data: Record<string, unknown>;
}

interface Stamp { ts: number; actor: string; eventId: string }
interface Field { value: unknown; stamp: Stamp }

interface ResourceState {
  fields: Map<string, Field>; // per-field last-writer-wins registers
  created: boolean;           // grow-only flag: once true, stays true
  deleted?: Stamp;            // tombstone (delete-wins policy)
}

// Total order on stamps: (ts, actor, eventId). Unique per event.
const cmp = (a: Stamp, b: Stamp): number =>
  a.ts !== b.ts ? a.ts - b.ts
  : a.actor !== b.actor ? (a.actor < b.actor ? -1 : 1)
  : a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0;

const EMPTY: ResourceState = { fields: new Map(), created: false };

// The join (⊔): commutative, associative, idempotent.
function merge(a: ResourceState, b: ResourceState): ResourceState {
  const fields = new Map(a.fields);
  for (const [key, fb] of b.fields) {
    const fa = fields.get(key);
    if (!fa || cmp(fb.stamp, fa.stamp) > 0) fields.set(key, fb);
  }
  let deleted = a.deleted;
  if (b.deleted && (!deleted || cmp(b.deleted, deleted) > 0)) deleted = b.deleted;
  return { fields, created: a.created || b.created, deleted };
}

function toFragment(e: WebhookEvent): ResourceState {
  const stamp: Stamp = { ts: e.logicalTimestamp, actor: e.actorId, eventId: e.eventId };
  const fields = new Map<string, Field>();
  for (const [k, v] of Object.entries(e.data)) fields.set(k, { value: v, stamp });
  return {
    fields,
    created: e.eventType === 'CREATED',
    deleted: e.eventType === 'DELETED' ? stamp : undefined,
  };
}

class WebhookReorderingEngine {
  private states = new Map<string, ResourceState>();
  private waitingSince = new Map<string, number>(); // local TTL bookkeeping only

  ingest(e: WebhookEvent, now: number = Date.now()): void {
    const merged = merge(this.states.get(e.resourceId) ?? EMPTY, toFragment(e));
    this.states.set(e.resourceId, merged);

    if (merged.created) this.waitingSince.delete(e.resourceId);
    else if (!this.waitingSince.has(e.resourceId)) this.waitingSince.set(e.resourceId, now);
  }

  // Visible only after `created` is known and while not tombstoned.
  read(resourceId: string): Record<string, unknown> | undefined {
    const s = this.states.get(resourceId);
    if (!s || !s.created || s.deleted) return undefined;
    return Object.fromEntries([...s.fields].map(([k, f]) => [k, f.value]));
  }

  // Resources stuck without a `created` event: reconcile via the provider's API.
  expiredWaiting(now: number, ttlMs: number): string[] {
    return [...this.waitingSince]
      .filter(([, since]) => now - since > ttlMs)
      .map(([id]) => id);
  }
}

A few design notes:

Date.now() appears only in the local TTL bookkeeping. It never influences event ordering.
The version number is not stored in the state. A counter incremented on every applied event would differ across replicas that drop or reorder events differently, which breaks convergence.
Merging per field means the result can combine fields from different events into a state the sender never actually had. If an invariant spans several fields, keep the whole entity in one register keyed by the provider’s version.
Delete-wins is a policy choice. A later “recreate” would need a new incarnation ID in the key.
State lives in memory here. In production it belongs in a durable store, and the merge must run atomically per resource, for example inside a transaction or compare-and-set.

5. Production Resilience Patterns

A. Unbounded buffers and missing events

If a created event is lost, for instance because the provider’s retries were exhausted, resources waiting for it accumulate. Bound the wait with a TTL, move the stuck events to a dead-letter queue, and reconcile by fetching the resource’s current state from the provider’s API. Choose the TTL against the provider’s behavior. Stripe retries over days, so a short TTL means you will usually reconcile by fetching rather than wait. Stripe’s List Events API only returns events from the last 30 days, so a lost event can only be backfilled within that window. Treat that as a hard limit when you plan recovery.

B. Clock drift and sequence sources

Do not order by physical Date.now() across regions. Prefer, in this order:

A version or sequence that the provider supplies.
A Lamport timestamp that your consumers maintain.
An edge-assigned counter per resource. If you use one, note that Redis Cluster does not provide strong consistency. Its replication is asynchronous, so it can lose acknowledged writes during failovers. The WAIT command reduces that risk but does not eliminate it. A counter that must never go backwards needs a store with consensus, such as Raft-based systems like etcd.

Remember that an edge-assigned counter records arrival order at your edge, not the provider’s order.

C. Tombstone garbage collection

You cannot drop a tombstone arbitrarily. If it is pruned while a delayed created or updated event is still in flight, that event can resurrect the deleted resource. The CRDT literature handles this with causal stability: a tombstone is safe to remove only once every replica is known to have seen the delete. For webhooks, a practical rule is to keep tombstones longer than the provider’s maximum retry window plus your own replay window. Stripe retries for up to three days. Adyen queues events for up to 30 days. Hookdeck documents both. A fixed 72-hour window would therefore be too short for some providers, so make it configurable per provider.

D. When to skip the CRDT

If the provider has an API that returns an authoritative current object, the simplest approach is often to treat the webhook as a notification. You fetch the object by ID and overwrite your local copy from that response. Many engineers recommend this for Stripe subscription, customer and invoice objects, at the cost of one extra API call per event. The CRDT approach is worth its complexity when:

the payload is the only source of truth,
the API is rate limited or slow,
you need replicas in several regions to converge without calling the provider on each event.

Summary: Approaches Compared

Strategy	Out-of-order delivery	Concurrent multi-region writes	Duplicates
Apply in arrival order	Overwrites newer data with older data	Replicas diverge	Unsafe without dedup
LWW on physical timestamps	Correct only if clocks are reliable	Subject to clock skew; one concurrent write is silently lost	Safe if writes are idempotent
LWW/CRDT with logical stamps + tie-breaker	Converges regardless of arrival order	Deterministic winner at every replica	Idempotent join, so safe
Fetch current state on notification	Not affected by order	Not affected by order	Safe, but costs an API call per event

Conclusion

Webhook ordering is not something to rely on. Major providers document that it is not guaranteed, and retries guarantee that duplicates will happen. A robust consumer acknowledges quickly, deduplicates on a stable event ID, and merges each event into state with a function that is commutative, associative and idempotent. A fixed tie-breaker keeps every replica’s choice identical, and logical clocks or provider versions replace unreliable wall-clock time. Where the provider offers an authoritative API, combine that approach with reconciliation by fetch. Together these give you convergence in every region, even when the network interleaves events differently at each one.

References

Shapiro, Preguiça, Baquero, Zawirski, “A Comprehensive Study of Convergent and Commutative Replicated Data Types,” INRIA RR-7506, 2011: https://inria.hal.science/inria-00555588/document
Shapiro et al., “Conflict-free Replicated Data Types,” SSS 2011: https://www.cs.tufts.edu/~nr/cs257/archive/marc-shapiro/CRDTs_SSS-2011.pdf
Lamport, “Time, Clocks, and the Ordering of Events in a Distributed System,” CACM, 1978
Fidge, “Timestamps in Message-Passing Systems That Preserve the Partial Ordering,” 1988; Mattern, “Virtual Time and Global States of Distributed Systems,” 1988
Stripe webhook behavior (retries, ordering, duplicates): https://dev.to/shannonllc/stripe-webhooks-retries-ordering-duplicates-timeouts-and-ips-46f0
Hookdeck, “Common Mistakes with Outbound Webhooks”: https://hookdeck.com/outpost/guides/common-outbound-webhook-mistakes.md
Hookdeck, “At-Least-Once vs. Exactly-Once Webhook Delivery Guarantees”: https://hookdeck.com/webhooks/guides/webhook-delivery-guarantees.md
Provider ordering summary (Stripe, Shopify, GitHub): https://www.vincentschmalbach.com/why-webhook-events-arrive-out-of-order/
Primer webhook documentation: https://primer.io/docs/api-reference/get-started/configure-webhooks
Redis Cluster consistency guarantees: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/

Originally published at https://instawebhook.com/blog/deterministic-reordering-of-concurrent-webhooks-handling-out-of-order-state-tran

High-Velocity Webhook Engineering: Low-Level Ingress, Dynamic Flow Control, and Multi-Tenant Isolation

InstaWebhook — Thu, 29 Oct 2026 02:07:54 +0000

Webhook traffic is bursty by nature. A payment provider, a Git host, or a SaaS platform can fire a large batch of events at your endpoint at once. When a router fans those events out to many downstream customers, three problems show up: the cost of network I/O at the edge, one slow customer stalling everyone else, and noisy senders consuming shared capacity.

This guide covers three building blocks of a resilient multi-tenant webhook router:

Efficient ingress with Linux io_uring: multishot receive and provided buffer rings, plus the kernel’s separate zero-copy receive feature.
Dynamic flow control: bounded per-tenant queues, credit-style backpressure, and load shedding to stop “slow consumer” cascades.
Per-tenant rate limiting: an atomic token bucket in Redis (Lua), with notes on Redis Cluster.

One caveat applies throughout. Many webhook workloads are limited by TLS termination, HTTP parsing, signature verification, and downstream I/O rather than by recv() calls. Profile before you adopt anything in section 1.

1. Ingress with `io_uring`: Multishot Receive and Provided Buffers

Where the standard epoll model spends its time

Most high-performance Linux network services use readiness-based event loops. Tokio, for example, runs on epoll through the mio library. Epoll tells you a socket is ready, and your code then issues the actual system call. Reading one chunk of a webhook request looks like this:

The NIC DMA-copies the packet into kernel memory, and the kernel’s network stack processes it and queues the payload on the socket.
epoll_wait() returns, telling userspace the socket is readable.
Userspace calls read() or recv(), which crosses the user/kernel boundary.
The kernel copies the payload from the socket buffer into the application’s buffer.
The application parses and processes the bytes.

Each step is cheap. At very high request rates, though, per-call overhead and the kernel-to-user copy become measurable. Gains from io_uring vary widely with workload, kernel version, mitigations, and batching, so benchmark your own stack instead of relying on generic figures.

What `io_uring` changes

io_uring is a Linux asynchronous I/O interface built around two ring buffers shared between the kernel and the application: a submission queue (SQ) and a completion queue (CQ). Instead of waiting for readiness and then making a call, the application submits operations and later reaps completions. Many operations can be submitted and completed with a single io_uring_enter() call.

Two features matter most for webhook ingestion:

Multishot receive. The kernel’s io_uring_prep_recv documentation describes a multishot variant in which one receive request repeatedly posts completions as data arrives, instead of needing a new submission per receive. Multishot receive is available since kernel 6.0.
Provided buffer rings. The application registers a pool of buffers with the kernel, and each completion consumes one buffer from the pool. The completion reports which buffer was used. The application hands buffers back to the ring when it has finished with them. This allows memory to be allocated per-pool rather than per-connection.

A correction on “zero-copy”

Multishot receive with provided buffers is not zero-copy. The kernel still receives packets into its own memory, runs the TCP stack, and copies the payload into your provided buffer. What you save is submission overhead and per-connection buffer pinning.

True zero-copy receive is a separate feature, io_uring zero-copy Rx (zcrx), added in Linux 6.15. The kernel documentation says it removes the kernel-to-user copy on the receive path, so payload data lands directly in userspace memory. Unlike full kernel-bypass stacks such as DPDK, packet headers are still processed by the kernel TCP stack. It also has strict hardware requirements:

The NIC must support header/data split, so headers go to kernel memory and payloads go to userspace memory.
Flow steering is needed to direct only the intended flows to the zero-copy queues.
RSS is needed to steer all other traffic away from those queues.

The kernel API does not configure the NIC for you; you have to set it up yourself. For a typical webhook gateway behind a load balancer on cloud VMs, zcrx may simply be unavailable. Multishot receive with provided buffers is the more practical starting point.

How the data flows

 Userspace                                   Kernel
 ┌──────────────────────┐   shared rings    ┌──────────────────────────┐
 │ Submission queue (SQ)│ ───────────────►  │ Executes multishot recv  │
 │ Completion queue (CQ)│ ◄───────────────  │ on the socket            │
 │ Provided buffer ring │ ◄── payload ───── │ Copies payload from the  │
 └──────────────────────┘    copied into    │ socket buffer into a     │
                             a free buffer  │ provided buffer          │
                                            └────────────▲─────────────┘
                                                         │
                                        NIC → kernel network stack (TCP)

A Rust sketch

Rust has several io_uring options. The io-uring crate from the tokio-rs project is a low-level binding, and several runtimes (for example tokio-uring, monoio, and glommio) build on or implement io_uring. Tokio’s main runtime itself uses epoll.

The sketch below submits a multishot receive and handles completions. It is simplified: it omits buffer-ring registration and error handling, and the exact API differs between crate versions, so check the docs for the version you use. Buffer rings are registered through the Submitter, for example with register_buf_ring_with_flags, and require careful unsafe memory management.

use io_uring::{cqueue, opcode, types, IoUring};
use std::{io, os::fd::RawFd};

const BUF_GROUP: u16 = 1; // buffer group id of a registered provided buffer ring

fn submit_multishot_recv(ring: &mut IoUring, conn_fd: RawFd) -> io::Result<()> {
    // One submission; the kernel posts a completion each time data arrives.
    let sqe = opcode::RecvMulti::new(types::Fd(conn_fd), BUF_GROUP)
        .build()
        .user_data(conn_fd as u64);

    unsafe {
        ring.submission()
            .push(&sqe)
            .map_err(|_| io::Error::other("submission queue full"))?;
    }
    ring.submit()?;
    Ok(())
}

fn reap_completions(ring: &mut IoUring) {
    for cqe in ring.completion() {
        let bytes = cqe.result(); // number of bytes received, or a negative errno
        let buf_id = cqueue::buffer_select(cqe.flags()); // which provided buffer was filled
        let multishot_still_armed = cqueue::more(cqe.flags());

        // 1. If bytes > 0 and buf_id is Some, read the payload from that buffer,
        //    then return the buffer to the ring.
        // 2. If the multishot request is no longer armed (no MORE flag),
        //    re-submit it for this connection.
        let _ = (bytes, buf_id, multishot_still_armed);
    }
}

Security and operational caveats

io_uring has a large kernel attack surface. In June 2023, Google’s security team reported that 60% of the Linux kernel exploits submitted to its bug bounty program in 2022 involved io_uring. As a result, Google disabled it for Android apps, on ChromeOS, and on its servers. Docker also removed it from its default seccomp profile, which means io_uring calls are blocked by default in many container environments. Check your runtime’s seccomp policy before assuming it will work in production. Administrators can also restrict it system-wide using the kernel.io_uring_disabled sysctl. For a service that handles untrusted internet input, such as a webhook gateway, weigh the throughput gains against that exposure.

2. Preventing the Slow Consumer Cascade

Head-of-line blocking in a multi-tenant router

A webhook router accepts events and forwards them to many customer endpoints. If one customer’s endpoint slows or goes offline and the router does not isolate that customer, the failure spreads:

Shared workers block while trying to deliver to the slow tenant.
Healthy tenants’ events queue behind them.
Threads, connections, and memory run out, and the failure becomes platform-wide.

Incoming events
   │
   ├─► Tenant A queue (full) ──► Endpoint A (slow or failing)  → circuit opens, A's events shed or parked
   │
   └─► Tenant B queue (healthy) ─► Endpoint B (fast)           → unaffected

The core rule is that a delivery problem for one tenant must never consume a resource that another tenant needs.

Bounded queues as credits

Credit-based flow control is an established idea in networking. It appears in technologies such as Fibre Channel, InfiniBand, and PCIe, where a sender may transmit only while it holds credits granted by the receiver. The same idea works at the application layer.

Give each tenant a fixed budget of in-flight events:

Accepting an event consumes one credit from that tenant’s budget.
A credit is returned only when the event leaves the queue, either after successful delivery or after it is moved to durable storage.
When a tenant has no credits left, the ingress path does not block. It sheds load or spills to durable storage.

A bounded Tokio channel is a natural fit for this. Its capacity is the credit budget, and try_send (or try_reserve) fails immediately instead of waiting.

What “shedding” means depends on who is sending

How you reject an event matters because the upstream provider’s retry behavior varies. Stripe, for instance, documents that in live mode it retries delivery for up to three days with exponential backoff, and recommends returning a 2xx quickly before doing heavy work. If your router returns 429 or 503 to such a sender, the event will come back later. Other senders may retry only briefly or not at all. The safer pattern is:

Verify the signature and persist the event durably first, then return 2xx quickly.
Deliver downstream asynchronously from a per-tenant queue.
When a tenant’s in-memory queue is full, spill to a disk-backed or broker-backed queue (RocksDB, Kafka, SQS, or similar) rather than dropping.
Reserve 429 or 503 with a Retry-After header for cases where durable storage itself is saturated, and only for providers you know will retry.

Per-tenant isolation in Rust

The sketch below gives each tenant a bounded channel and its own consumer task. It uses a read-write lock so lookups for different tenants do not serialize on a single mutex.

use std::{collections::HashMap, sync::Arc, time::Duration};
use tokio::sync::{mpsc, RwLock};

#[derive(Debug)]
pub struct WebhookPayload {
    pub event_id: String,
    pub body: Vec<u8>,
}

#[derive(Debug)]
pub enum IngestError {
    UnknownTenant,
    TenantSaturated, // out of credits: spill to durable storage or shed
    TenantClosed,
}

pub struct RouterManager {
    tenants: RwLock<HashMap<String, mpsc::Sender<WebhookPayload>>>,
}

impl RouterManager {
    pub fn new() -> Arc<Self> {
        Arc::new(Self { tenants: RwLock::new(HashMap::new()) })
    }

    pub async fn register_tenant(&self, tenant_id: String, capacity: usize) {
        let (tx, mut rx) = mpsc::channel::<WebhookPayload>(capacity);
        let id = tenant_id.clone();

        // One isolated consumer per tenant: a slow endpoint only stalls its own queue.
        tokio::spawn(async move {
            while let Some(event) = rx.recv().await {
                // Bound every delivery attempt so a hung endpoint cannot block forever.
                let _ = tokio::time::timeout(
                    Duration::from_secs(10),
                    deliver(&id, &event),
                )
                .await;
                // On timeout or error: schedule a retry, and trip a per-tenant circuit breaker.
            }
        });

        self.tenants.write().await.insert(tenant_id, tx);
    }

    pub async fn ingest(&self, tenant_id: &str, event: WebhookPayload) -> Result<(), IngestError> {
        let tenants = self.tenants.read().await;
        let tx = tenants.get(tenant_id).ok_or(IngestError::UnknownTenant)?;

        match tx.try_send(event) {
            Ok(()) => Ok(()),
            Err(mpsc::error::TrySendError::Full(_)) => Err(IngestError::TenantSaturated),
            Err(mpsc::error::TrySendError::Closed(_)) => Err(IngestError::TenantClosed),
        }
    }
}

async fn deliver(_tenant_id: &str, _event: &WebhookPayload) {
    // HTTP POST to the customer's endpoint with signing, timeouts, and retry scheduling.
}

Two design notes. First, a bounded channel gives you fairness in memory, but you still need per-tenant concurrency limits on outbound connections, otherwise one tenant with thousands of queued events can monopolize the shared HTTP client pool. Second, a circuit breaker per endpoint, which stops attempts after repeated failures and probes occasionally for recovery, prevents wasted work against a dead endpoint.

3. Per-Tenant Rate Limiting with a Redis Token Bucket

Why rate limit

Failures and bugs can create floods without any malicious intent. A customer’s endpoint that returns errors triggers retries, and a buggy integration can loop and resend events. Per-tenant limits keep one sender’s problem from consuming shared capacity.

Atomic token bucket in Lua

A token bucket stores the number of tokens and the time of the last refill. A read-modify-write sequence done through separate GET and SET commands is racy when several gateway nodes run it concurrently. A Redis Lua script runs atomically as a single command, so no other command executes in the middle of it.

The script below improves on the common version in three ways:

It reads time from Redis with TIME instead of trusting each gateway’s clock, which avoids skew between nodes. Reading time in scripts works with current Redis replication behavior, where scripts replicate by their effects.
It uses HSET, because HMSET has been deprecated since Redis 4.0.
It returns floats as strings, because Redis converts Lua numbers in replies to integers and truncates them.

-- KEYS[1]: bucket key, e.g. "{tenant_xyz}:rate_limit"
-- ARGV[1]: bucket capacity (maximum burst)
-- ARGV[2]: refill rate in tokens per second
-- ARGV[3]: tokens requested for this call (usually 1)

local key         = KEYS[1]
local capacity    = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested   = tonumber(ARGV[3])

local t   = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local bucket = redis.call("HMGET", key, "tokens", "ts")
local tokens = tonumber(bucket[1])
local ts     = tonumber(bucket[2])

if tokens == nil or ts == nil then
  tokens = capacity
else
  local elapsed = math.max(0, now - ts)
  tokens = math.min(capacity, tokens + elapsed * refill_rate)
end

local allowed = 0
local retry_after = 0
if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
else
  retry_after = (requested - tokens) / refill_rate
end

redis.call("HSET", key, "tokens", tokens, "ts", now)
-- Expire idle buckets once they would have fully refilled anyway.
redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate * 1000))

return { allowed, tostring(tokens), tostring(retry_after) }

On a denied request, use retry_after to populate a Retry-After header.

Redis 7.0 also introduced Redis Functions, which are invoked with FCALL. Unlike EVAL scripts, functions are persisted with the dataset and replicated to replicas, so you do not need to resend script source from every client. Consider them if you run Redis 7 or later.

Redis Cluster considerations

Declare every key in KEYS. In Redis Cluster, all keys a script touches must be passed in KEYS and must hash to the same slot. Accessing undeclared keys inside a script breaks cluster routing.
Use hash tags when a script touches multiple keys. The script above uses one key per tenant, so it works as is. If you extend it to also read, say, a per-tenant configuration key, name both with the same tag (for example {tenant_xyz}:bucket and {tenant_xyz}:config) so they land in the same slot.
Expect hot slots. Each tenant’s bucket lives on one node, so a single very busy tenant is bounded by what that node can serve. Spread tenants across slots and monitor slot-level load.
Limit round trips. Redis executes commands on a main thread per node, so every limiter call costs capacity there. For very high-volume tenants, let each gateway keep a local bucket and lease tokens from Redis in batches (for example, take 100 tokens at a time and refill when they run low). This reduces Redis calls and latency at the cost of precision: the global limit becomes approximate by up to the total size of outstanding leases. Choose the batch size according to how strict the limit must be. The reduction in Redis traffic depends on your batch size and traffic pattern, so measure it.

Summary

Layer	Common baseline	Alternative discussed here	What you gain, and what to watch
Network ingress	epoll plus a `recv()` call per read	`io_uring` multishot receive with provided buffer rings; zcrx on supported NICs and kernels (6.15+)	Fewer syscalls and simpler buffer management. Multishot is not zero-copy, and the security exposure of `io_uring` needs review.
Flow control	Shared worker pool, unbounded queues, blocking writes	Bounded per-tenant queues with credit-style limits, timeouts, circuit breakers, and durable spill	One tenant’s slowness stays contained; durable persistence avoids data loss when shedding.
Rate limiting	Non-atomic `GET`/`INCR`/`SET` sequences	Atomic Lua token bucket using Redis `TIME`, with hash-tagged keys and optional local token leasing	No race conditions and cluster-safe routing, traded against slot hot spots and approximate global limits when leasing.

Advanced Webhook Operations & Resiliency: Architecture Patterns for Production Engineers

InstaWebhook — Wed, 28 Oct 2026 02:05:19 +0000

Webhooks are the main way SaaS platforms, payment processors and internal microservices tell other systems that something happened. At a few thousand events a day, a loop that POSTs to a URL and retries on failure is enough. At hundreds of millions of events a day, the same loop turns small outages into large ones. Retries pile up, healthy customers get starved by unhealthy ones, and the infrastructure sitting in front of your consumers starts rejecting requests for reasons your application never sees.

This guide covers four areas of production webhook operations:

Adaptive backoff combined with per-endpoint circuit breakers.
Preventing retry storms and cascading failures.
Zero-downtime migration of delivery infrastructure and payload schemas.
Header bloat and the request-size limits of reverse proxies and load balancers.

The numbers in the examples (timeouts, thresholds, cooldowns) are illustrative starting points. Tune them against your own traffic.

1. Adaptive Backoff with Circuit Breakers

Plain exponential backoff (delay = base * 2^attempt) treats every failure the same way. Failures differ in what they say about the consumer, and the dispatcher should react to each differently:

HTTP 429 (Too Many Requests): the consumer, or something in front of it, is shedding load on purpose. Retrying on your normal schedule ignores that signal and can get your delivery IPs throttled or blocked. If the response carries a Retry-After header, honor it. RFC 9110 defines the value as either a number of seconds or an HTTP date.
HTTP 5xx: the consumer is crashing, deploying, or out of database connections. Fast retries add load to a system that is trying to recover.
Timeouts and connection resets: the consumer is slow, overloaded or unreachable. Every slow attempt also holds one of your own workers and sockets, so keep client timeouts short.

                +----------------------------------------+
                |           Incoming Event Queue         |
                +----------------------------------------+
                                    |
                                    v
                +----------------------------------------+
                |     Circuit Breaker State Inspection   |
                +----------------------------------------+
                         /                      \
                [CLOSED / HALF-OPEN probe]    [OPEN]
                       /                          \
                      v                            v
     +----------------------------+     +--------------------------+
     |  Attempt webhook delivery  |     | Skip the network call,   |
     +----------------------------+     | schedule for later retry |
        /          |            \       +--------------------------+
 [2xx Success] [429 Rate Limit] [5xx / Timeout]
      |             |                 |
      v             v                 v
 Reset failure  Honor Retry-After  Increment failure
 counter, close or scaled backoff  counter, trip if
 circuit                           threshold is met

Retry Policy by Response Type

Response / failure	Likely cause	Suggested action
429 Too Many Requests	Consumer or its gateway is rate limiting	Honor `Retry-After` (seconds or HTTP date) and pause delivery to that endpoint. If the header is missing, use a steeper backoff multiplier than for 5xx.
502 / 503 / 504	Gateway or app server down, or mid-deployment	Exponential backoff with full jitter.
500	Bug or unhandled exception in the consumer’s handler	Slower backoff and a lower attempt cap. Retrying rarely fixes a code bug, so don’t let these events clog shared queues.
408, connection or read timeout	Thread pool exhaustion, slow queries, network problems	Short client timeouts (a few seconds) so workers are freed quickly. Route retries to a slower queue.
400, 401, 403, 404	Bad request, expired credentials, misconfigured or removed endpoint	Usually not worth retrying quickly. Dead-letter after a small number of attempts, or retry slowly, because 401/403/404 can appear transiently during consumer deployments.
410 Gone	The consumer has explicitly removed the endpoint	Stop delivering and disable the endpoint.

Providers differ on this. Stripe, for example, treats any non-2xx response as a failed delivery and retries in live mode with exponential backoff for up to three days (Stripe docs). Other platforms stop retrying on most 4xx responses but still retry 408 and 429. Whichever policy you pick, document it for your consumers.

Why Full Jitter

If many events fail at the same moment and all use the same backoff formula, they all retry at the same moments too. Jitter spreads those retries out. In full jitter, described in the AWS Architecture Blog’s Exponential Backoff and Jitter, the delay is a random value between 0 and the capped exponential ceiling:

sleep = random_between(0, min(cap, base * 2^attempt))

Full jitter can occasionally return a very short delay. If that matters, use a variant that keeps a minimum floor (often called “equal jitter”), or add a minimum delay yourself.

Distributed Circuit Breakers for Outbound Webhooks

When one consumer endpoint fails for a long time, retrying millions of events against it wastes worker time and sockets, and it slows delivery to healthy consumers that share the worker pool. A circuit breaker keyed per endpoint (or per consumer domain) stops that. Keep its state in a fast shared store such as Redis, so every dispatcher instance sees the same state.

Closed (normal): events are dispatched normally, and failures are counted.
Open (tripped): when failures cross a threshold (for example, 10 failures within a window, or an error rate above a set percentage), the breaker opens. Events for that endpoint skip the network and go straight to a delayed retry queue.
Half-open (probing): after a cooldown, one probe request is let through.
- If the probe succeeds, the breaker closes and normal delivery resumes.
- If it fails, the breaker re-opens, ideally with a longer cooldown.

Count only failures that indicate an unhealthy consumer (timeouts, connection errors, 5xx, and arguably 429). A 400 or 401 usually indicates a configuration problem and shouldn’t trip a breaker on its own.

Python Sketch: Adaptive Delay, Jitter and a Redis-Backed Breaker

This is a simplified sketch. The Redis operations are not atomic as written. In production, move the read-modify-write logic into a Lua script or use a library that provides atomic breaker state.

import math
import random
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from typing import Mapping, Optional


def parse_retry_after(value: str) -> Optional[float]:
    """Retry-After is either delay-seconds or an HTTP-date (RFC 9110)."""
    try:
        return max(0.0, float(value))
    except ValueError:
        pass
    try:
        when = parsedate_to_datetime(value)
        if when.tzinfo is None:
            when = when.replace(tzinfo=timezone.utc)
        return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())
    except (TypeError, ValueError):
        return None


class AdaptiveWebhookDispatcher:
    def __init__(self, redis_client):
        self.redis = redis_client          # redis-py client, decode_responses=True
        self.base_delay = 2.0              # seconds
        self.max_delay = 3600.0            # cap retry delay at 1 hour
        self.failure_threshold = 10        # failures per window before tripping
        self.failure_window = 300          # fixed 5-minute counting window
        self.cooldown_period = 120         # first OPEN period, seconds
        self.max_cooldown = 3600           # cap on escalated cooldowns
        self.probe_ttl = 30                # how long a half-open probe slot is held

    # ---- retry delay -------------------------------------------------
    def retry_delay(self, attempt: int, status: Optional[int],
                    headers: Mapping[str, str]) -> float:
        h = {k.lower(): v for k, v in headers.items()}

        if status == 429 and "retry-after" in h:
            requested = parse_retry_after(h["retry-after"])
            if requested is not None:
                requested = min(requested, self.max_delay)
                # Never retry earlier than asked; add a little jitter on top.
                return requested + random.uniform(0, requested * 0.1)

        # Steeper growth for 429 without Retry-After, standard growth otherwise.
        multiplier = 3 if status == 429 else 2
        ceiling = min(self.max_delay, self.base_delay * math.pow(multiplier, attempt))
        return random.uniform(0, ceiling)  # full jitter

    # ---- circuit breaker --------------------------------------------
    def _k(self, endpoint: str, name: str) -> str:
        return f"cb:{endpoint}:{name}"

    def get_state(self, endpoint: str) -> str:
        if self.redis.exists(self._k(endpoint, "open")):
            return "OPEN"
        if self.redis.exists(self._k(endpoint, "tripped")):
            return "HALF_OPEN"          # cooldown expired, not yet proven healthy
        return "CLOSED"

    def allow_request(self, endpoint: str) -> bool:
        state = self.get_state(endpoint)
        if state == "CLOSED":
            return True
        if state == "OPEN":
            return False
        # HALF_OPEN: allow exactly one probe at a time
        return bool(self.redis.set(self._k(endpoint, "probe"), 1,
                                   nx=True, ex=self.probe_ttl))

    def record_failure(self, endpoint: str) -> None:
        if self.get_state(endpoint) == "HALF_OPEN":
            self._trip(endpoint)        # probe failed: re-open with longer cooldown
            return
        key = self._k(endpoint, "failures")
        count = self.redis.incr(key)
        if count == 1:
            self.redis.expire(key, self.failure_window)  # fixed window, set once
        if count >= self.failure_threshold:
            self._trip(endpoint)

    def _trip(self, endpoint: str) -> None:
        trips = self.redis.incr(self._k(endpoint, "trips"))
        cooldown = min(self.cooldown_period * 2 ** (trips - 1), self.max_cooldown)
        pipe = self.redis.pipeline()
        pipe.setex(self._k(endpoint, "open"), int(cooldown), "1")
        pipe.set(self._k(endpoint, "tripped"), 1)
        pipe.delete(self._k(endpoint, "failures"), self._k(endpoint, "probe"))
        pipe.execute()

    def record_success(self, endpoint: str) -> None:
        self.redis.delete(*(self._k(endpoint, n) for n in
                            ("failures", "open", "tripped", "trips", "probe")))

2. Webhook Storms: Detecting and Preventing Cascading Failures

A webhook storm (or retry storm) happens when an outage creates a large backlog of failed deliveries, and the retries of that backlog, added to normal live traffic, overload the consumer as it recovers and knock it over again.

          [ Consumer outage or bad deployment ]
                          |
                          v
             Many deliveries fail at once
                          |
                          v
          Failed events enter retry pipelines
                          |
             Retries scheduled at identical
             offsets (no jitter): T+30s, T+60s ...
                          |
                          v
        Consumer recovers, receives a retry spike
        on top of live traffic, and fails again
                          |
                          v
              Repeated or cascading outage

Common Causes of Amplification

Synchronized retries. Without jitter, every event that failed in the same outage retries on the same schedule, so the consumer sees repeated spikes instead of a smooth ramp.
Head-of-line blocking. If fresh events share a queue with thousands of failing retries, they wait behind them. Healthy consumers are delayed because one consumer is down.
Naive re-enqueueing. Pushing a failed message straight back to the front of the main queue creates a tight failure loop. Failed work should go through a delayed path instead.
No per-consumer limits. Without concurrency and rate caps per endpoint, a backlog drains as fast as your workers allow, which may be far faster than the consumer can absorb.

Architecture to Contain Retry Storms

Separate real-time delivery from retries, and put limits in front of every consumer:

                 +-------------------------+
                 |     Primary event       |
                 |        ingress          |
                 +-------------------------+
                             |
                             v
                 +-------------------------+
                 | Tier 1: ingestion queue |
                 | (Kafka, Pulsar, etc.)   |
                 +-------------------------+
                             |
                             v
                 +-------------------------+
                 |  Real-time worker pool  |
                 |  (per-endpoint limits)  |
                 +-------------------------+
                      /               \
               [2xx Success]       [Failure]
                    |                  |
                    v                  v
           +----------------+  +---------------------------+
           |   Completed    |  | Tier 2: delayed retry     |
           +----------------+  | queues (short/medium/long)|
                               +---------------------------+
                                          |
                                          | attempts exhausted or
                                          | non-retriable error
                                          v
                               +---------------------------+
                               | Dead Letter Queue (DLQ)   |
                               | inspection and replay     |
                               +---------------------------+

Design Principles

Two tiers of queues.
- Tier 1 handles first-attempt deliveries with short timeouts, so slow consumers can’t hold workers for long.
- Tier 2 holds retries, split into delay buckets (for example 1 minute, 15 minutes, 1 hour). Retry traffic never competes with first attempts for the same workers.
- Implementation note: Kafka has no built-in per-message delay, so delay buckets are usually implemented as separate topics or a scheduler that re-publishes when a message is due. Pulsar supports delayed message delivery natively.
Per-endpoint concurrency and rate limits.
- Cap in-flight requests per endpoint (a semaphore or lease in Redis) so a deep backlog cannot open hundreds of simultaneous connections to one consumer.
- Add a per-endpoint token bucket for request rate. When a consumer returns 429, reduce its rate and recover gradually.
Circuit breakers (Section 1) so persistently failing endpoints stop consuming worker capacity.
Bounded retries and a dead letter queue.
- Stop retrying after a fixed budget, for example a maximum number of attempts or a maximum age (24 hours in your policy, 3 days like Stripe’s live-mode window, or whatever you document).
- Move exhausted or non-retriable events to a DLQ.
- Offer replay, via API or dashboard, so consumers can re-ingest events after they fix the problem. Replays should also be rate limited.
Visibility. Track retry queue depth, age of the oldest retry, per-endpoint failure rate and breaker state. A growing retry backlog is an early warning of a storm.

3. Zero-Downtime Migration of Webhook Delivery

Moving webhook delivery to a new region, a new gateway or a new delivery engine, or introducing a new payload version, must not drop events, send duplicates or break the contract consumers rely on.

Dual-Run with Shadow Delivery

Run the new pipeline alongside the old one, but let only one of them deliver to real customer endpoints.

                 +----------------------------------+
                 |       Upstream event publisher   |
                 +----------------------------------+
                                  |
                                  v
                 +----------------------------------+
                 |          Routing layer           |
                 +----------------------------------+
                      /                        \
                     v                          v
   +-------------------------------+  +-------------------------------+
   | Legacy engine                 |  | New engine                    |
   | ACTIVE: delivers to customers |  | SHADOW: delivers to test sink |
   +-------------------------------+  +-------------------------------+
                     |                          |
                     v                          v
        Customer webhook endpoint       Validation sink and logs

Phase 1: Shadow Mirroring and Validation

Publish each event to both engines. The new engine builds payloads, signs them and applies its routing logic, but sends the HTTP request to an internal validation sink instead of the customer URL. Compare outputs with the legacy engine: payload bytes, signatures, headers, latency and error behavior. Because the shadow engine never reaches customers, it can’t prove that customer endpoints are reachable from the new network. Check reachability separately, for example by testing egress IPs and DNS from the new region.

Phase 2: Payload Versioning

When the payload schema changes, for example a timestamp format change or a restructured object, avoid forcing every consumer to upgrade at once:

Pin a version per endpoint. Each webhook endpoint is configured with a payload version, and the dispatcher renders that version. Stripe takes this approach: events are rendered according to the API version configured for the endpoint.
Prefer additive changes. Adding fields is usually backward compatible. Renaming fields, changing types or changing units is not.
Include a version field in the payload and in your documentation, and publish a deprecation timeline.
Support old versions for a defined period and track which endpoints still use them before turning a version off.

Avoid sending both the old and new payload shapes in a single body. It makes payloads larger, ambiguous for consumers and harder to deprecate.

{
  "id": "evt_987654321",
  "version": "2026-10-01",
  "type": "payment.succeeded",
  "data": {
    "account_id": "acc_123",
    "amount_in_cents": 4999,
    "currency": "USD"
  }
}

Phase 3: Idempotency and Standard Headers

Failover, replays and migrations all raise the chance that a consumer receives the same event twice. Give consumers a stable ID to deduplicate on, and sign requests in a way libraries already understand. The Standard Webhooks specification defines a widely adopted set of headers:

webhook-id: unique per message and stable across all delivery attempts, so consumers can use it as a deduplication key. Any unique string works (it doesn’t have to be a UUID).
webhook-timestamp: Unix timestamp in seconds, set when that attempt is sent (retries carry a new timestamp). Consumers use it to reject stale or replayed requests.
webhook-signature: one or more space-separated signatures, each in the form v1,<base64>, where the value is a Base64-encoded HMAC-SHA256 over {webhook-id}.{webhook-timestamp}.{raw body}.

Because the header holds a list of signatures, you can rotate signing secrets by sending signatures from both the old and new secret during the transition. Stripe uses a similar scheme with its own format: the Stripe-Signature header carries t=<timestamp> and v1=<hex HMAC-SHA256 of "timestamp.payload">.

Consumers must verify the signature against the raw, unparsed request body. A framework that parses and re-serializes the JSON first will produce a different byte sequence and the signature will fail.

Phase 4: Canary Cutover

Move traffic to the new engine gradually, by tenant. Assign each tenant to exactly one active engine using a stable hash of the tenant ID, so that no tenant is delivered to by both engines at once. Raise the share in steps (for example 1%, 10%, 50%, 100%), watching delivery success, latency, signature-verification failures reported by consumers, and retry volume. Keep the routing switch simple enough to roll back immediately.

4. Header Bloat and Proxy Limits

Webhook requests often pass through several hops: a CDN or WAF, a load balancer, an API gateway, and an ingress proxy, before reaching the application. Each hop enforces its own limits on request line and header size. Requests that exceed them are rejected before they reach your consumer’s application code, so the consumer’s application logs show nothing. The evidence is in the proxy or load balancer access logs.

Common sources of large headers:

Large cookies and JWT bearer tokens (a JWT with many claims can be several kilobytes on its own).
X-Forwarded-For and Via chains that grow with each proxy hop.
Multiple custom signature or key-identifier headers.
tracestate and baggage headers from distributed tracing. Note that traceparent itself is small and fixed-length (55 characters per the W3C Trace Context spec), so the tracing headers that grow are tracestate and baggage.

Incoming request
  |
  +-- Headers:
  |    |-- Authorization: Bearer <large JWT>
  |    |-- traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
  |    |-- tracestate / baggage: <vendor and application entries>
  |    |-- Cookie: <many cookies>
  |    |-- X-Forwarded-For: 203.0.113.7, 10.0.0.1, 172.16.0.1 ...
  |    +-- Custom signature / metadata headers
  v
+---------------------------------------------------------+
| Reverse proxy or load balancer                          |
| Header size limit exceeded                              |
+---------------------------------------------------------+
  |
  v
Request rejected at the proxy (400 / 431 / other), never reaching the app

Default Limits of Common Proxies

Defaults vary by version and configuration. Verify against the version you run.

Proxy	Default limits	Typical failure	Tuning
NGINX	`client_header_buffer_size` 1k; `large_client_header_buffers` 4 × 8k. The request line and each individual header line must fit in one 8k buffer, and all headers together must fit in the allocated buffers (32k at defaults).	`414` if the request line is too long; `400 Bad Request` (“Request Header Or Cookie Too Large”) for oversized headers	`large_client_header_buffers 4 16k;`
Envoy	`max_request_headers_kb`: 60 KiB; `max_headers_count`: 100	`431 Request Header Fields Too Large`	Raise `max_request_headers_kb` in the `HttpConnectionManager` (older versions cap it at 96; recent versions allow much higher values)
Apache HTTPD	`LimitRequestLine` 8190 bytes; `LimitRequestFieldSize` 8190 bytes per header; `LimitRequestFields` 100 headers	Request rejected	`LimitRequestFieldSize 16384`, `LimitRequestFields 200`
AWS Application Load Balancer	Request line 16 K; single header 16 K; entire request header 64 K	Request rejected	Hard limits that cannot be changed. Reduce header size.

Two notes on status codes. 431 is the status defined by RFC 6585 for oversized headers. 413 refers to the request body being too large, so a 413 from a webhook endpoint points to payload size, not header size. A 502 can also appear when an upstream closes the connection because of its own limits, so check the logs of every hop.

Also remember that raising the limit on your edge proxy only helps if every proxy and application server behind it accepts the larger size too. An Envoy that admits 128 KB headers in front of a backend that rejects anything above 8 KB just moves the failure one hop inward.

Mitigations

1. Keep Headers Small

Keep authentication to a single compact credential. Where you control token contents, avoid stuffing large claim sets into a bearer token.
Limit tracestate and baggage to what you actually use, and drop them at the edge when sending to third-party consumers.
Don’t add per-attempt or per-hop debug headers to outbound webhooks unless they are needed.
If you sign with several secrets during a rotation, use the standard single webhook-signature header with space-separated signatures, rather than one header per version.

# Anti-pattern: one header per signature version plus extras
X-Webhook-Signature-V1: <signature>
X-Webhook-Signature-V2: <signature>
X-Webhook-Key-Identifier: key_prod_9921381

# Better: one header carrying all active signatures
webhook-signature: v1,<base64-signature-A> v1,<base64-signature-B>

2. Don’t Move Metadata into a Signed Body

It is tempting to push tracing or delivery metadata into the JSON body to save header space. Be careful with this for outbound webhooks. The body is what gets signed, so anything in it that changes per attempt (such as a delivery-attempt counter) changes the signature and makes it harder for consumers to deduplicate by content. Keep the event payload immutable across retries, and carry per-attempt data only where it is needed, such as the timestamp in the signed headers.

3. Tune NGINX Ingress

For a Kubernetes cluster running the NGINX-based ingress controller, the relevant ConfigMap keys are client-header-buffer-size (default 1k) and large-client-header-buffers (default 4 8k):

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  client-header-buffer-size: "4k"
  large-client-header-buffers: "4 16k"   # up to 64k total, 16k per header line

Two cautions:

The http2-max-field-size and http2-max-header-size keys are deprecated in favor of large-client-header-buffers, so don’t rely on them.
The community Ingress NGINX project was retired by the Kubernetes project in March 2026, meaning it no longer receives maintenance or security fixes. The Kubernetes Steering and Security Response Committees urged users to plan a migration to Gateway API or another supported controller. If you still run it, treat the migration as a priority. The F5/NGINX-maintained “NGINX Ingress Controller” is a different project.

4. Raise Envoy’s Header Limit Where Needed

In Envoy, set max_request_headers_kb on the HttpConnectionManager. The configuration below is a minimal, complete listener, including the router filter that an HTTP connection manager requires:

static_resources:
  listeners:
  - name: ingress_webhook_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 443 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          max_request_headers_kb: 96     # default is 60
          route_config:
            name: local_route
            virtual_hosts:
            - name: webhook_services
              domains: ["webhooks.example.com"]
              routes:
              - match: { prefix: "/v1/ingest" }
                route: { cluster: webhook_backend_cluster }
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

A production listener would also need TLS configuration (transport_socket) and a defined webhook_backend_cluster, which are omitted here for brevity. Older Envoy releases cap max_request_headers_kb at 96. Newer ones accept larger values, but raising it far above what you actually need weakens the protection the limit provides.

Production Checklist

Error classification: Do 429, 5xx, timeouts, and 4xx responses each follow their own retry policy? Does the dispatcher honor Retry-After in both its formats?
Jitter: Do all backoff schedules use jitter?
Circuit breakers: Are persistently failing endpoints isolated automatically, with half-open probes and escalating cooldowns?
Queue separation: Are first-attempt deliveries decoupled from retry traffic?
Per-endpoint limits: Are concurrency and request rate capped per consumer?
Bounded retries and DLQ: Is there a retry budget, a dead letter queue and a replay path?
Idempotency: Is webhook-id (or equivalent) stable across retries, and are consumers told to deduplicate on it?
Signatures: Is the signed content documented (id, timestamp, raw body), and are timestamps checked on the consumer side?
Versioning: Is the payload version pinned per endpoint, with a published deprecation policy?
Migration safety: Does each tenant have exactly one active delivery engine during cutover, with an instant rollback?
Header budget: Are header sizes measured, and are every hop’s limits (CDN, WAF, load balancer, ingress, app server) known and consistent?
Observability: Do you monitor retry backlog age, per-endpoint failure rates, breaker state and proxy-level rejections?

References

RFC 9110, HTTP Semantics (Retry-After): https://www.rfc-editor.org/rfc/rfc9110
RFC 6585, Additional HTTP Status Codes (429, 431): https://www.rfc-editor.org/rfc/rfc6585
AWS Architecture Blog, “Exponential Backoff And Jitter”: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Standard Webhooks specification: https://www.standardwebhooks.com/
Stripe documentation, receiving webhook events and retry behavior: https://stripe.com/docs/webhooks
W3C Trace Context: https://www.w3.org/TR/trace-context/
AWS, Quotas for your Application Load Balancers: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html
NGINX, large_client_header_buffers: https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers
Envoy, HttpConnectionManager (max_request_headers_kb): https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto
Apache HTTP Server, core directives (LimitRequestFieldSize, LimitRequestFields, LimitRequestLine): https://httpd.apache.org/docs/2.4/mod/core.html
Kubernetes blog, “Ingress NGINX Retirement: What You Need to Know”: https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/
Kubernetes blog, “Ingress NGINX: Statement from the Kubernetes Steering and Security Response Committees”: https://www.kubernetes.io/blog/2026/01/29/ingress-nginx-statement/

Originally published at https://instawebhook.com/blog/advanced-webhook-operations-resiliency-architecture-patterns-for-production-engi

Enterprise Webhook Architecture at Scale: High-Throughput Ingress, Multi-Region CRDTs, and Adaptive Backpressure

InstaWebhook — Tue, 27 Oct 2026 02:05:03 +0000

Webhook traffic is bursty and outside your control. Senders fire when something happens, such as a flash sale, a settlement batch or a fleet of devices reconnecting, and they all do it at once. This article covers three layers that together keep an ingestion tier alive under that load: an in-memory ingress buffer, active-active multi-region replication, and adaptive load shedding. For each layer it also covers the trade-offs, which matter as much as the mechanisms.

A note on numbers: “500,000 requests per second” is used here as a design target, not a benchmark. The real ceiling depends on payload size, TLS termination, core count and NIC bandwidth, so measure on your own hardware.

Part 1: RAM-Disk Ingress Buffers — Using Linux `tmpfs` and Shared Memory to Absorb Webhook Bursts

Why senders force this design

Major webhook senders expect a fast acknowledgment and do the waiting for you. Stripe’s documentation tells receivers to return a 2xx immediately and defer complex logic. In live mode it retries failed deliveries for up to three days with exponential backoff. GitHub expects a 2xx within 10 seconds and otherwise terminates the connection and records a failure. It also does not automatically redeliver failed deliveries, so a missed delivery stays missed until someone redelivers it.

The consequence is that the hot path should do as little as possible before acknowledging. That means validating the signature, appending the payload to a buffer and returning. Writing each payload straight into a relational database, or into a broker configured for synchronous disk persistence, puts storage latency inside the sender’s timeout window. During a spike this produces timeouts, and the timeouts produce retries that add to the spike.

How `tmpfs` behaves

tmpfs keeps its contents in the kernel’s page cache and, if swap is enabled, can push unneeded pages out to swap. According to the kernel documentation, a few details matter for an ingress buffer:

The default size limit is half of physical RAM, and nr_inodes caps the inode count.
Swap is enabled by default. The noswap mount option disables it, which you want here, because a buffer that silently spills to swap gives up the latency profile you built it for.
/dev/shm is normally a tmpfs mount, so POSIX shared memory (shm_open) and files under a dedicated tmpfs mount draw on the same memory pool.

# 64 GiB buffer, no swap, owner-only access
mount -t tmpfs -o size=64G,nr_inodes=2M,mode=0700,noswap tmpfs /mnt/ingress-buffer

Memory in tmpfs is real RAM that other processes can no longer use, so alert on df usage and size the mount against your worst-case burst.

Sizing arithmetic (illustrative). Assume an average payload of 4 KiB at 500,000 RPS. That is roughly 1.9 GiB/s of inbound payload data, so a 64 GiB buffer absorbs about 33 seconds of sustained peak if nothing drains. Two consequences follow:

The buffer buys you time but is not unlimited. The drain path must be sized to catch up after the burst.
Do not write one file per webhook. At 500,000 RPS, a 2M-inode limit would be exhausted in about four seconds. Append payloads to large segment files instead, for example 64–256 MiB each, with a small length-prefixed record header.

Ring buffers and `io_uring`

The pipeline looks like this:

Client TLS / HTTP ──► Edge proxy (Rust or Go; epoll or io_uring)
                            │  validate signature, enforce size limits
                            ▼
                 Append-only segment files on tmpfs
                            │
                            ▼
               Drain workers (user space, batched)
                            │
                            ▼
           Local NVMe log ► Kafka / object storage

A webhook endpoint needs TCP and TLS termination, so the data path goes through the normal kernel network stack. XDP and eBPF are useful earlier in the path, for dropping obviously bad traffic at the NIC before it costs you CPU. They don’t replace the HTTP server.

Within the proxy, lock-free single-producer/single-consumer rings (one per worker thread) are a common way to hand records to a writer without contention. io_uring can then batch the file writes.

io_uring has an IORING_SETUP_SQPOLL mode in which the kernel creates a thread that polls the submission queue. The man page says this lets an application issue I/O without context switching into the kernel, with two caveats worth knowing:

If the polling thread is idle for longer than sq_thread_idle milliseconds, it goes to sleep and sets a flag. The application must then call io_uring_enter(2) to wake it. SQPOLL reduces system calls under sustained load, but it does not eliminate them.
Unprivileged SQPOLL requires a newer kernel (5.13, per the man page). The polling thread also occupies a CPU, so budget cores for it. The io_uring_sqpoll(7) page notes that it only affects submissions, so completion handling still costs CPU.

Some hardened or containerized environments restrict io_uring, so confirm it is permitted on your hosts before building around it.

An illustrative sketch with the Rust io-uring crate follows. Check the API against the crate version you pin:

use io_uring::{opcode, types, IoUring};
use std::{fs::OpenOptions, os::unix::io::AsRawFd};

fn main() -> std::io::Result<()> {
    // SQPOLL: a kernel thread polls the submission queue and sleeps after 1000 ms idle.
    let mut ring = IoUring::builder().setup_sqpoll(1000).build(256)?;

    let file = OpenOptions::new()
        .create(true)
        .write(true)
        .open("/mnt/ingress-buffer/segment-000001.log")?;

    let payload = b"{\"event\":\"example\"}\n";
    let write = opcode::Write::new(
        types::Fd(file.as_raw_fd()),
        payload.as_ptr(),
        payload.len() as _,
    )
    .offset(0)
    .build()
    .user_data(1);

    // Safety: `payload` and `file` outlive the completion below.
    unsafe { ring.submission().push(&write).expect("submission queue full") };
    ring.submit_and_wait(1)?;

    let cqe = ring.completion().next().expect("completion missing");
    assert!(cqe.result() >= 0, "write failed: {}", cqe.result());
    Ok(())
}

The durability trade-off

A buffer in RAM does not survive a power failure or kernel panic, and fsync() on tmpfs buys nothing because there is no backing store. Acknowledging with 202 Accepted after a tmpfs write therefore creates a window in which the sender believes the event is safe but your platform could lose it. Because senders only retry on failure, a lost event that was acknowledged is not retried.

RFC 9110 defines 202 as acceptance for processing, not completion. That fits the pattern, but it doesn’t make the data durable. There are three common ways to narrow the window:

Replicate before acknowledging. Copy the record into a second host’s memory in another failure domain and acknowledge after both writes succeed. This costs a network round trip inside a region, typically on the order of a millisecond between availability zones, according to ThousandEyes measurements.
Group-commit to local NVMe. Acknowledge after a batched, sequential append to a local log. This is slower than RAM but far cheaper than per-request database writes.
Accept the window deliberately for low-value event classes, and reconcile against the sender’s API or redelivery tools after an incident.

Pick one per event class and document it.

Draining to persistent storage

Drain workers run in user space. They read sealed segments from tmpfs, batch them and write them onward. A few practical points:

Object storage. S3 multipart parts must be 5 MiB to 5 GiB, except the last part, with at most 10,000 parts per upload. Use chunks of 8–16 MiB or larger. Chunks below 5 MiB are rejected for non-final parts.
Direct I/O. O_DIRECT is relevant when writing to a local disk log, not when uploading over HTTP. tmpfs itself gained O_DIRECT support in Linux 6.6; on older kernels, opening a tmpfs file with it fails. The drain side can simply use ordinary reads or mmap.
Idempotency. Draining can crash midway and replay a segment. Give every record a stable ID, such as the sender’s delivery ID (GitHub supplies X-GitHub-Delivery) or a hash, so downstream consumers can deduplicate.

Part 2: Active-Active Cross-Region Webhook Ingress — Convergence with CRDTs and Vector Clocks

Why active-active

Consider a provider serving customers from AWS us-east-1 (N. Virginia), eu-central-1 (Frankfurt) and ap-southeast-1 (Singapore). Each region should accept webhooks locally, so that a regional outage or a slow inter-region link doesn’t block ingestion.

To steer clients to a nearby region, there are two options:

Anycast. The same IP is announced from multiple locations and network routing (BGP) delivers each client to a nearby one. Services such as AWS Global Accelerator work this way.
DNS-based routing (latency or geolocation policies). This is a separate mechanism from anycast. Don’t conflate them: anycast operates at the network layer, not through DNS.

A webhook sender such as Stripe will use whichever endpoint you give it. If that is a single hostname, steering happens on your side.

Why not synchronous consensus across regions

Quorum protocols such as Paxos or Raft need at least one round trip to a majority of replicas per decision. Inside a region that is around a millisecond. Across continents it is limited by distance and routing, and typically ranges from tens to hundreds of milliseconds. AWS publishes measured inter-region numbers through Network Manager Infrastructure Performance. Putting that on the acknowledgment path eats into the sender’s timeout budget and ties availability to the slowest link. The alternative is to accept writes locally and replicate asynchronously, and to design the data model so that replicas converge.

Terminology: conflicts, not split-brain

“Split-brain” usually describes two nodes both believing they are the sole leader. In an active-active design, every region is supposed to accept writes, so the real problem is concurrent, possibly conflicting updates to the same entity, plus out-of-order delivery. Two tools address this: vector clocks, which detect concurrency, and CRDTs, which merge it deterministically.

Vector clocks

Each entity carries a vector clock, a map from region ID to a counter. A region increments its own entry when it records an event and takes the element-wise maximum when it receives remote state.

Region A: {A:1, B:0, C:0}  ← event 1
Region B: {A:0, B:1, C:0}  ← event 2 (neither saw the other: concurrent)
After exchange: merged clock {A:1, B:1, C:0}

For vector clocks V₁ and V₂ over N regions:

V₁ ≤ V₂ if V₁[i] ≤ V₂[i] for every region i.
V₁ causally precedes V₂ if V₁ ≤ V₂ and V₁[j] < V₂[j] for at least one j.
If neither V₁ ≤ V₂ nor V₂ ≤ V₁, the events are concurrent, and the merge rule decides the outcome.

Vector clocks grow with the number of participants. That is fine for three or five regions but not for per-client identifiers. Keep them per region, not per sender.

Many systems pair this with hybrid logical clocks, which combine physical time with a logical counter, so that timestamps stay close to wall-clock time yet never go backwards. They are useful as a tiebreaker or for ordering display, but wall clocks alone are not a safe basis for correctness.

CRDTs

Shapiro, Preguiça, Baquero and Zawirski’s 2011 study formalized two families of conflict-free replicated data types. Both guarantee that replicas converge without foreground coordination:

State-based (CvRDT): replicas exchange full or delta state and merge it with an associative, commutative, idempotent function.
Operation-based (CmRDT): replicas exchange operations that commute, which requires reliable, exactly-once delivery of those operations.

Useful building blocks for a webhook platform include:

PN-Counters for approximate global metrics such as delivery attempts or events ingested. They converge, but they cannot enforce a strict global limit, because that would require coordination. Treat a PN-Counter-based rate limit as approximate.
Observed-Remove Sets (OR-Sets) for subscription or endpoint registries, where concurrent adds and removes must converge predictably. An add concurrent with a remove of the same element wins.
LWW registers for single-value fields where “latest write wins” is acceptable.

Here is a last-writer-wins register that uses the vector clock for causality and falls back to a timestamp only for truly concurrent writes. The original idea of merging clocks but then deciding on the timestamp alone can silently discard a causally newer update when clocks are skewed, so the causal check comes first:

use std::collections::HashMap;

#[derive(Clone)]
pub struct LwwRegister<T> {
    pub value: T,
    pub clock: HashMap<String, u64>, // vector clock: region -> counter
    pub hlc_micros: u64,             // hybrid logical timestamp
    pub replica_id: String,          // deterministic tiebreaker
}

enum Causality { Before, After, Equal, Concurrent }

fn compare(a: &HashMap<String, u64>, b: &HashMap<String, u64>) -> Causality {
    let (mut a_less, mut b_less) = (false, false);
    for k in a.keys().chain(b.keys()) {
        let x = a.get(k).copied().unwrap_or(0);
        let y = b.get(k).copied().unwrap_or(0);
        if x < y { a_less = true } else if x > y { b_less = true }
    }
    match (a_less, b_less) {
        (false, false) => Causality::Equal,
        (true, false) => Causality::Before,
        (false, true) => Causality::After,
        (true, true) => Causality::Concurrent,
    }
}

impl<T: Clone> LwwRegister<T> {
    pub fn merge(&mut self, remote: &Self) {
        let remote_wins = match compare(&self.clock, &remote.clock) {
            Causality::Before => true,                      // remote is causally newer
            Causality::After | Causality::Equal => false,   // local is newer or identical
            Causality::Concurrent => {
                // Timestamp first, replica ID as a deterministic tiebreaker
                (remote.hlc_micros, &remote.replica_id) > (self.hlc_micros, &self.replica_id)
            }
        };

        let mut merged = self.clock.clone();
        for (node, &r) in &remote.clock {
            let e = merged.entry(node.clone()).or_insert(0);
            *e = (*e).max(r);
        }

        if remote_wins {
            self.value = remote.value.clone();
            self.hlc_micros = remote.hlc_micros;
            self.replica_id = remote.replica_id.clone();
        }
        self.clock = merged;
    }
}

Two cautions apply. First, with a concurrent update, LWW keeps one value and discards the other. That is acceptable for something like “current endpoint URL” but not for data where both writes matter. For those, use a multi-value register or a domain-specific merge. Second, because senders may deliver the same event more than once and don’t guarantee ordering, receivers should be idempotent regardless. Stripe’s documentation says as much, and GitHub exposes a per-delivery ID header for deduplication.

Part 3: Adaptive Backpressure — Shedding Load Gracefully Before Downstream Saturation Hits 100%

The failure mode

Even with a RAM buffer and multiple regions, downstream consumers can saturate: a database exhausts connections, a partner endpoint slows down, or workers fall behind. Fixed rate limits struggle here because the right limit depends on how much the downstream can handle at that moment. The goal is to start slowing intake before the queue is full, and to slow the least important traffic first.

Little’s Law

For a stable system in steady state, Little’s Law states:

L = λ × W

where L is the average number of items in the system, λ is the average arrival rate, and W is the average time an item spends in the system.

The practical use is as a sizing and sanity check. If downstream slows and W rises while λ stays constant, L must rise. More precisely, once the arrival rate exceeds the service rate, the steady-state assumption no longer holds and the queue grows until it hits a limit, which for an unbounded in-memory queue means running out of memory. Queueing delay also climbs sharply as utilization approaches 100%, so waiting until the queue is nearly full is too late. Shedding reduces λ to something the downstream can sustain. A bounded queue turns an eventual crash into an explicit, controlled rejection.

Saturation signal and dynamic token bucket

Define queue saturation as:

S = current queue depth / maximum queue capacity

A token bucket with a fixed refill rate cannot react to S. Instead, scale the refill rate down as S rises between a low and high watermark. For example, with S_low = 0.75 and S_high = 0.90:

R = R_max × (1 − clamp((S − S_low) / (S_high − S_low), 0, 1))

Below S_low, requests get the full rate. Above S_high, low-priority traffic is refused entirely. Many systems use AIMD (additive increase, multiplicative decrease) on the admitted rate instead, borrowing from TCP congestion control. Either way, feed the controller a smoothed signal such as queue depth plus drain rate, so it doesn’t oscillate.

Incoming webhook ──► classify priority ──► token bucket (rate scaled by S)
                                              │
                          ┌───────────────────┴──────────────────┐
                      tokens available                       no tokens
                          │                                      │
                  append to buffer ──► 202                 429 + Retry-After

HTTP semantics: `429` and `Retry-After`

RFC 6585 defines 429 Too Many Requests. A Retry-After header may accompany it, either as a number of seconds or as an HTTP date. Add jitter so that all clients don’t retry at the same instant. AWS’s “Exponential Backoff And Jitter” article is a good reference on why. A formula consistent with the thresholds below:

delay = min(MaxBackoff, BaseDelay × 2^(k × p)) × jitter

where p = clamp((S − S_low) / (1 − S_low), 0, 1), k is a steepness constant and jitter is a random factor, for example between 0.5 and 1.0.

package backpressure

import (
    "math"
    "math/rand"
    "strconv"
)

const (
    lowWatermark  = 0.75
    baseDelaySecs = 5.0
    maxDelaySecs  = 120.0
    steepness     = 4.0
)

// RetryAfter returns a Retry-After value in seconds, or ok=false when no
// throttling applies.
func RetryAfter(saturation float64) (value string, ok bool) {
    if saturation <= lowWatermark {
        return "", false
    }
    p := math.Min((saturation-lowWatermark)/(1-lowWatermark), 1.0)
    delay := math.Min(baseDelaySecs*math.Pow(2, steepness*p), maxDelaySecs)
    delay *= 0.5 + 0.5*rand.Float64() // jitter between 50% and 100% of delay
    return strconv.Itoa(int(math.Ceil(delay))), true
}

With these constants, the pre-jitter delay is 5 s at S = 0.75, about 26 s at S = 0.90 and 80 s at S = 1.0.

Not every sender will cooperate

Shedding with 429 only helps if the sender retries. The differences between senders are large:

Stripe retries failed deliveries in live mode for up to three days with exponential backoff, so a 429 or 503 costs a delay, not data.
GitHub does not automatically redeliver failed deliveries, so a shed GitHub webhook is lost until someone redelivers it manually or via the API.

So classify by what the sender will do, not only by business importance. Events whose senders won’t retry should be the last to be shed, and should preferably go to the durable buffer or a spill queue. Where you must drop them, log enough to reconcile later. A 503 Service Unavailable with Retry-After is also a reasonable shed response for tiers that are unavailable rather than rate-limited.

Shedding states

State	Queue saturation (S)	Ingress action	Response
Normal	S ≤ 0.75	Accept all traffic into the buffer	`202 Accepted`
Degraded	0.75 < S ≤ 0.90	Shed low-priority events (for example, analytics pings) with a growing delay	`429` with `Retry-After` of roughly 5–26 s
Tripped	S > 0.90	Reject all non-critical events; keep accepting critical ones until the hard limit	`429` with `Retry-After` of roughly 26–80 s; `503` once the hard limit is hit

These are starting points, not universal constants. Tune watermarks against measured drain rates and your senders’ retry behavior.

Export the state machine and its inputs (queue depth, drain rate, admitted vs. rejected counts per priority, current Retry-After) as Prometheus or OpenTelemetry metrics. Alert on transitions, not only on levels, because a system that flaps between Normal and Degraded is a tuning problem you want to see.

Putting it together

The three layers address different failure modes:

RAM-backed ingress keeps acknowledgment latency low during bursts, in exchange for a deliberate durability decision (replicate, group-commit or accept the window) and careful sizing of tmpfs, inodes and swap.
Active-active regions with vector clocks and CRDTs keep ingestion available across regions without cross-continent consensus on the hot path. The cost is eventual consistency, and conflict semantics you must choose per data type.
Adaptive backpressure keeps downstream failures from becoming memory exhaustion, provided the shedding policy accounts for which senders retry.

No design makes webhook ingestion lossless under all failures. What it can do is make the trade-offs explicit and measurable, so that overload degrades the least valuable traffic first and the data you’ve promised to keep survives.

Originally published at https://instawebhook.com/blog/enterprise-webhook-architecture-at-scale-high-throughput-ingress-multi-region-cr

Zero-Trust Webhook Ingress: Enforcing Cryptographic Sender Identity with DNSSEC, DANE, and Verified mTLS

InstaWebhook — Mon, 26 Oct 2026 02:03:58 +0000

Webhooks are how many B2B SaaS integrations learn that something happened: a payment settled, a resource was provisioned, an order shipped. Each webhook endpoint is also a public HTTPS listener that accepts state-changing requests from systems you don’t control.

Most webhook security still lives at the application layer: shared HMAC secrets, static tokens, or IP allowlists. These are workable, and for most providers they are all that’s offered. They also have well-known limits. This guide looks at how to push sender authentication down to the transport layer with mutual TLS (mTLS) enforced at the edge, how DNSSEC and DANE-style TLSA records can serve as a tamper-evident place to publish sender key pins, and how to degrade gracefully for senders that can’t support any of this.

One caveat up front. mTLS for webhooks is real and supported by some providers. Using DNSSEC-signed TLSA records to pin client certificates is not a standardized practice. It is a design pattern built from standardized parts, and this article is explicit about where the standards stop and the engineering begins.

1. Why Shared Secrets and IP Allowlists Fall Short

How HMAC webhook signing works

In the common model, the provider and consumer share a secret. For each event, the provider computes an HMAC (typically HMAC-SHA256) over the raw request body and sends it in a header:

POST /v1/webhooks/orders HTTP/1.1
Host: api.example.com
X-Hub-Signature-256: sha256=a3f5c9e2b1...
Content-Type: application/json

{"event": "order.fulfilled", "order_id": "99214"}

GitHub, for example, signs webhook deliveries with X-Hub-Signature-256, the SHA-256 successor to the older SHA-1 X-Hub-Signature header. Stripe uses a Stripe-Signature header that carries a timestamp and one or more v1 HMAC-SHA256 signatures computed over the string {timestamp}.{body}.

Done correctly, this gives you payload integrity and proof that the sender held the secret. The limitations are structural:

Symmetric secrets must be distributed and rotated. Both sides hold the same key, so anyone who can read it can forge events, and the consumer can forge events “from” the provider. Rotation without dropping deliveries needs an overlap window in which two secrets are accepted. Stripe’s header format supports multiple signatures for this reason.
Replay protection depends on the scheme. A signature over only the body can be replayed indefinitely. Stripe signs a timestamp together with the body, and its official libraries reject events outside a default tolerance of five minutes. GitHub’s signature covers only the payload, so receivers need other controls, such as deduplicating on the delivery ID. Timestamp checks in turn require reasonably synchronized clocks, and clock drift is a common cause of spurious signature failures.
The signature identifies a secret, not a network or cryptographic identity. It says “someone with this key sent this,” nothing more. It is checked after your edge, TLS terminator, and HTTP parser have already accepted and processed the request.

Why IP allowlists are a weak identity signal

Many providers publish source IP ranges. GitHub exposes its hook delivery ranges through its meta API, and Stripe publishes lists in its documentation. Allowlisting them is reasonable defense in depth, but a poor primary control:

Lists change. Providers add and remove egress ranges, and consumers have to track the changes or drop legitimate traffic.
IP says nothing about which tenant sent the request. An allowlisted range identifies the provider’s platform at best. It cannot tell you which customer or workload on that platform originated the request.
Shared egress widens the trust boundary. Anything that can send traffic from the allowlisted range inherits the trust you granted.

Spoofing the source address of a full TCP/TLS exchange is hard in practice, so the real weakness is coarse granularity rather than easy forgery. Use allowlists to narrow exposure, not to establish identity.

2. mTLS: Authenticating the Sender During the Handshake

With mutual TLS, the sender presents an X.509 client certificate during the TLS handshake, and the receiver verifies it before any HTTP request is processed. Compared with HMAC headers:

The sender proves possession of a private key that never leaves it, rather than a secret both parties know.
Authentication happens before the request body is accepted, so unauthenticated senders are rejected at the handshake or edge.
Identity is a certificate (subject, SAN, fingerprint), which maps cleanly to a partner record.

This is already offered by some webhook senders. Truv documents mTLS for webhook delivery, including delivery using a client-supplied certificate. Google’s Dialogflow supports mTLS when calling a customer’s webhook server: Dialogflow presents a client certificate, and the webhook server verifies it. Support is still the exception, so plan on a mix of sender capabilities (see Section 5).

A significant change: public CAs and client authentication

Many teams used to run mTLS with certificates from public CAs, relying on certificates that carried both the serverAuth and clientAuth Extended Key Usages. That is ending. Driven by Chrome Root Program policy, public CAs are removing the clientAuth EKU from newly issued certificates through 2026. Let’s Encrypt, for instance, stopped including it in certificates issued from mid-May 2026 (with an extended deadline for some users of its TLS-client profile), and other large CAs followed similar schedules. Failures tend to appear quietly when a certificate renews and no longer carries clientAuth.

The practical consequence for webhook ingress is that you should not design around public-CA client certificates. Use a private CA (yours or the partner’s), or pin individual partner certificates or public keys directly.

3. Where DNSSEC and DANE Fit (and Where They Don’t)

What the standards define

DANE (DNS-Based Authentication of Named Entities) is defined in RFC 6698, with operational guidance in RFC 7671. It lets a domain publish TLSA records in DNS that bind a TLS server’s certificate or public key to a service name, and it depends on DNSSEC to make those records tamper-evident. DNSSEC signs DNS data using DNSKEY and RRSIG records and links zones through DS records, forming a chain of trust from the DNS root zone down through the TLD (such as .com) to the domain.

A TLSA record is published at a name built from the port, protocol, and server hostname, for example _443._tcp.webhooks.example.com, and has four fields:

_443._tcp.webhooks.example.com. IN TLSA 3 1 1 (
    a1b2c3d4e5f67890123456789abcdef0123456789abcdef0123456789abcdef0 )

Certificate usage 3 (DANE-EE): the end-entity certificate or key itself is the trust anchor. No public CA is needed, so self-signed or privately issued certificates work.
Selector 1: match the SubjectPublicKeyInfo (SPKI) rather than the whole certificate, so the certificate can be re-issued without changing the record as long as the key pair stays the same.
Matching type 1: compare a SHA-256 digest.
Association data: the digest itself.

RFC 7671 recommends publishing records for both the current key and the next key ahead of a rollover, so key changes don’t cause validation failures while DNS caches expire.

DANE is widely deployed for server authentication in SMTP, defined in RFC 7672. For HTTPS, major browsers do not support it. This is why most web traffic still relies on the public CA ecosystem.

The gap for webhooks

Standard DANE authenticates a server by looking up TLSA records at a name derived from the server’s hostname. In webhook delivery, the sender is the TLS client. A client sends no SNI naming itself, so there is no standardized way for a receiver to derive a DNS name from the handshake and check the client’s certificate against it. Any scheme that uses TLSA records for client identity is therefore a custom convention, not an interoperable standard.

A workable version of the pattern looks like this:

At onboarding, each partner gives you a DNS name they control (for example webhook-client.partner.example) and agrees to publish its client-key pin there as a TLSA-format record, in a DNSSEC-signed zone.
A scheduled sync job, not the request path, queries that name using a DNSSEC-validating resolver, rejects results that do not validate, and writes the resulting fingerprints into your edge key-value store.
The edge worker checks incoming client certificates against that store.

This gives you a partner-controlled place to rotate keys without a manual exchange for every rotation, plus tamper-evidence from DNSSEC. Know its costs before committing:

Partner adoption is limited. DNSSEC signing is far from universal. Verisign reported a signing rate of about 4.3% for second-level domains in .com and 5.3% in .net in 2023, and a Q3 2025 report on DNSSEC validation put the global average validation rate at roughly 35%. Many partners will not be able to publish signed records.
You are trusting a validator. Unless you run DNSSEC validation yourself, your trust rests on the resolver that performs it and on the path to it.
Operational risk shifts to DNS. A broken DS record or expired signatures can invalidate a partner’s pin set. Decide in advance whether the sync job fails closed (keep last-known-good pins for a bounded period) or open.
It is non-standard. Document the convention, version it, and expect each partner to need explanation.

For many organizations, a simpler out-of-band registry of partner certificate fingerprints, populated at onboarding and updated through an authenticated portal or API, provides most of the benefit with less moving machinery. DNSSEC-published pins are best treated as an optional enhancement for partners who can support them.

4. Enforcing mTLS at the Edge

Terminating mTLS at an edge platform lets you reject unauthenticated senders before traffic reaches your origin, and lets you update trusted CAs and pinned certificates without redeploying application servers.

On Cloudflare, for example, mTLS can be enabled per hostname and enforced with API Shield. Client certificate details are exposed to Workers in request.cf.tlsClientAuth, with fields such as certPresented, certVerified, certRevoked, certFingerprintSHA256, certSubjectDN, and certSerial. The edge can also forward the certificate to the origin, either through Cloudflare-specific headers (Cf-Client-Cert-Sha256, Cf-Client-Cert-Der-Base64) or through headers following RFC 9440, which standardizes Client-Cert and Client-Cert-Chain headers for conveying client certificates from a TLS-terminating proxy to a backend. Other CDNs and gateways offer comparable features, but their field names and capabilities differ, so check your provider’s documentation.

Worker pattern

The worker below is a sketch of the pattern, written for Cloudflare Workers. Field value formats should be checked against the current documentation before use.

// Edge gatekeeper: accept webhooks only from verified, pinned partner certificates.
// env.PARTNERS is a KV namespace mapping "fp:<sha256-fingerprint>" -> partner record.

const TRUST_HEADERS = ["x-verified-sender-id", "x-sender-cert-fingerprint", "x-ingress-auth-method"];

export default {
  async fetch(request, env) {
    if (request.method !== "POST") {
      return new Response("Method Not Allowed", { status: 405, headers: { Allow: "POST" } });
    }

    const auth = request.cf?.tlsClientAuth;

    // 1. A certificate must have been presented and validated by the edge
    //    against the CA(s) you configured. Verify the exact value formats
    //    for these fields in your platform's documentation.
    if (!auth || auth.certPresented !== "1") {
      return new Response("Unauthorized", { status: 401 });
    }
    if (auth.certVerified !== "SUCCESS" || auth.certRevoked === "1") {
      return new Response("Forbidden", { status: 403 });
    }

    // 2. Pin: the certificate must match a known partner.
    const fingerprint = String(auth.certFingerprintSHA256 || "").toLowerCase();
    const partner = await env.PARTNERS.get(`fp:${fingerprint}`, "json");
    if (!partner) {
      return new Response("Forbidden", { status: 403 });
    }

    // 3. Strip any client-supplied copies of our trust headers, then set our own.
    const headers = new Headers(request.headers);
    for (const name of TRUST_HEADERS) headers.delete(name);
    headers.set("X-Verified-Sender-ID", partner.id);
    headers.set("X-Sender-Cert-Fingerprint", fingerprint);
    headers.set("X-Ingress-Auth-Method", "mtls-pinned");

    return fetch(new Request(request, { headers }));
  },
};

Points worth noting in this design:

The edge already checks validity. Chain, expiry, and revocation status are evaluated by the platform when it verifies the client certificate, so the worker reads the result instead of re-parsing dates. Revocation behavior depends on the platform and how you configured it, so confirm what your provider checks and how quickly revocations take effect.
Pinning adds a second control. Chain validation says “a CA you trust issued this.” The fingerprint lookup says “this is a specific partner we onboarded.” If you use a private CA scoped to a single partner, the fingerprint check can be relaxed to a subject or SAN check.
Strip inbound trust headers. Clients can send their own Client-Cert-style or custom headers. Cloudflare’s documentation explicitly warns about this. Remove any such headers before setting your own, as the worker does.
Protect the origin path. The origin must accept traffic only from the edge (for example, via authenticated origin pulls, a private tunnel, or network restrictions). Otherwise anyone who reaches the origin directly can forge the assertion headers. For higher assurance, have the edge sign the identity headers or pass a short-lived token the origin verifies.
Application-layer checks still matter. mTLS authenticates the sender’s connection. It does not make a payload idempotent or protect against a compromised partner. Keep schema validation, idempotency keys, and replay detection.

What edge verification does and doesn’t buy you

Moving authentication to the edge means unauthenticated traffic is dropped before it consumes origin resources, and trusted-certificate changes become a configuration update at the edge instead of an origin redeploy. Claims that it removes all cryptographic load from the origin or “eliminates” attack surface overreach. The origin still terminates its own TLS (or trusts an edge tunnel), parses requests, and handles business logic.

5. Tiered Fallback for Senders That Can’t Do mTLS

Most webhook senders will not support client certificates, let alone DNSSEC-published pins. A realistic ingress layer treats trust as tiers and routes accordingly:

Tier	Mechanism	What it establishes	Handling
1: Pinned mTLS	mTLS with a pinned partner certificate or key (optionally DNSSEC-published)	Sender proves possession of a private key tied to an onboarded partner	Normal processing; strongest identity signal
2: Signed with replay protection	HMAC signature over timestamp and body, per-partner secrets, optional provider IP ranges	Sender knew a shared secret; replay limited to the tolerance window	Normal processing with signature, timestamp, and dedupe checks
3: Legacy	Static token or HMAC without timestamp	Weakest; vulnerable to leaked secrets and replay	Isolated queue, tighter rate limits, tracked for migration

Implementation notes for the fallback paths:

Route by what the sender can do, not by what it claims. Decide the tier from the verified handshake result and the partner registry, not from request headers the sender controls.
Enforce replay windows where the scheme allows. If the signature covers a timestamp, reject stale events. Stripe’s default tolerance is five minutes, and tighter windows are possible if clocks are well synchronized. Where the signature has no timestamp, deduplicate on the sender’s delivery or event ID.
Use IP allowlists as an additional filter only. Combine provider-published ranges with signature checks, and refresh the lists automatically since they change.
Emit telemetry per tier. Log which tier accepted each event and which partners remain on Tier 3, so migration progress is measurable and you can raise it with vendors.

Conclusion

Static shared secrets and IP allowlists remain the baseline in the webhook ecosystem, and they work when combined with timestamped signatures, secret rotation, and replay protection. Where a partner can support it, mTLS at the edge, with certificates pinned per partner and a private CA instead of a public one, moves sender authentication into the handshake and rejects unauthenticated traffic before it reaches your application.

DNSSEC and TLSA records can add a tamper-evident, partner-controlled way to publish and rotate those pins. This is a custom pattern, not a standard, and DNSSEC adoption among partners is limited, so treat it as an optional layer on top of a straightforward partner registry. Around all of it, keep a tiered fallback so that senders who cannot meet the highest bar are still accepted under compensating controls instead of being cut off.

References

RFC 6698, The DNS-Based Authentication of Named Entities (DANE) Transport Layer Security (TLS) Protocol: TLSA: https://www.rfc-editor.org/rfc/rfc6698
RFC 7671, The DANE Protocol: Updates and Operational Guidance: https://www.rfc-editor.org/rfc/rfc7671
RFC 7672, SMTP Security via Opportunistic DANE TLS: https://www.rfc-editor.org/rfc/rfc7672
RFC 9440, Client-Cert HTTP Header Field: https://www.rfc-editor.org/rfc/rfc9440
Cloudflare Docs, mTLS with Workers: https://developers.cloudflare.com/learning-paths/mtls/mtls-workers
Cloudflare Docs, Enable mTLS: https://developers.cloudflare.com/ssl/client-certificates/enable-mtls/
Truv Docs, Security and mTLS: https://docs.truv.com/api-reference/security
Google Cloud, Dialogflow CX mutual TLS authentication: https://docs.cloud.google.com/dialogflow/cx/docs/concept/mtls
Stripe Docs, Webhooks: https://docs.stripe.com/webhooks

Originally published at https://instawebhook.com/blog/zero-trust-webhook-ingress-enforcing-cryptographic-sender-identity-with-dnssec-d

DEV Community: InstaWebhook

Ingress Security & Cryptographic Verification: Enterprise Webhook Hardening Guide

1. Where Webhook Signatures Stand Today

2. Post-Quantum Signatures for Webhooks: The Honest Threat Model

3. Designing a Dual-Signature Verification Proxy

Choosing a policy

4. Implementation in Go

5. The Operational Cost: Sizes, Headers and Key Distribution

Publishing public keys

6. Outbound Path Integrity: BGP Hijacking and DNS Spoofing

Why this threat is real

What has already improved

Defense layers and their limits

Step 1: Check route origin validity with RPKI

Step 2: Pin certificates with DNSSEC and DANE

7. Comparing Approaches

8. Action Plan

Conclusion

References

Architecting a Bring-Your-Own (BYO) Database Schema for Webhook Ingress: Balancing Customer Data Control and Metadata

1. The Architectural Dilemma: Sovereignty vs. Orchestration

The BYO Infrastructure Paradigm

2. Designing the Minimal-Privilege PostgreSQL Schema

Enforcing Least Privilege with Roles

3. Job Leasing with FOR UPDATE SKIP LOCKED

Completing, Failing and Dead-Lettering

What SKIP LOCKED Does and Doesn’t Give You

4. Metadata Synchronization and State Reporting

5. Provider Behavior Your Schema Must Absorb

6. Security Hardening

Summary: Centralized vs. BYO Architecture

References

High-Throughput & Post-Quantum Webhook Ingress Engineering

1. Post-Quantum Webhook Signatures: ML-DSA and Hybrid Crypto-Agility

Why the clock is ticking

Sizes and what they cost on the wire

A dual-header verification pipeline

Engineering considerations

2. Early Packet Filtering: XDP and AF_XDP

What XDP does and does not buy you

XDP filter (C)

AF_XDP consumer (Rust sketch)

3. Tenant-Aware Sharding and Isolation

The noisy-neighbor problem

Consistent hashing with virtual nodes

Dedicated queues, deficit round robin and backpressure

4. Stateful Defenses Against Slow-Client Attacks

Attack mechanics

Decision model

Use the knobs your proxy already has

Example: minimum-rate and size enforcement (Rust, Tokio)

Summary

References

Building an In-Memory Circuit Breaker for Webhook Ingress: Shedding Load Without Losing Events

1. How an ingress meltdown unfolds

2. Why the admission decision should be local

3. Two mechanisms: adaptive concurrency and queue delay

Adaptive concurrency limits

Queue delay (CoDel-style)

4. Architecture

5. A working Go sketch

6. Rejecting requests without losing events

Status codes

7. Priority-based shedding

8. Observability

9. Checklist

References

Memory-Mapped Webhook Buffers: Using `mmap` and Ring Buffers to Cut Kernel-User Crossings at 500k RPS

1. Where the time goes at 500k RPS

The traditional receive path

A system call is not a context switch

2. What mmap and ring buffers actually buy you

3. A correct SPSC shared-memory ring

4. Pairing the ring with io_uring for network ingress

5. Production hardening

Backpressure and durability

Be careful with file-backed mmap

NUMA and CPU affinity

Check the rest of the path

6. What measurements actually show

3. Job Leasing with `FOR UPDATE SKIP LOCKED`

What `SKIP LOCKED` Does and Doesn’t Give You

2. What `mmap` and ring buffers actually buy you

4. Pairing the ring with `io_uring` for network ingress

Be careful with file-backed `mmap`

Server sketch in Go (`quic-go`)

Go `net/http` example

4. Surviving Traffic Bursts: Durable Buffering with `mmap`

What an `mmap` ring buffer gives you

1. Ingress with `io_uring`: Multishot Receive and Provided Buffers

What `io_uring` changes