High-Performance Ingress & Network Architecture: Building Resilience at Scale

Webhook ingress gateways have to absorb bursty traffic from third-party senders and stay responsive when some of those connections are slow or hostile. This article covers three techniques for hardening such a gateway: using HTTP/3 and QUIC where they fit, defending against slow-connection attacks (Slowloris and Slow-Read), and using memory-mapped (mmap) ring buffers to absorb bursts. It also states where each technique stops helping, because each one is often oversold.

The article is aimed at teams who run their own ingress layer, whether they receive webhooks from providers such as Stripe or GitHub, or ingest events from their own devices and agents.

---

1. Know the Contract Before Tuning the Network

Network tuning only matters relative to what the sender expects from you. Two documented examples:

• Stripe expects a webhook endpoint to use HTTPS, accept POST requests with a JSON event payload, and quickly return a 2xx status. In live mode, Stripe attempts delivery of an event for up to three days with exponential backoff. Events created in a sandbox are retried three times over a few hours.
• GitHub expects a 2xx response within 10 seconds. If your server takes longer, GitHub terminates the connection and counts the delivery as failed. GitHub’s guidance is to enqueue the payload and process it in the background. GitHub does not automatically redeliver failed deliveries, so recovering from an outage means redelivering missed webhooks yourself once the server is back.

Deadlines and retry behavior vary by provider, so check each sender’s documentation rather than assuming a general standard. Two consequences follow for gateway design:

1. The gateway’s main job is to acknowledge fast and persist durably. Everything else (parsing, business logic, fan-out) belongs behind a queue.
2. Your own endpoint must tolerate duplicates. Retries and redeliveries mean the same event can arrive more than once, so deduplicate on an event or delivery ID.

---

2. HTTP/3 and QUIC: What They Fix and What They Don’t

Where TCP-based HTTP hurts

1. Handshake cost. A new HTTPS connection over TCP needs a TCP handshake plus a TLS handshake before the first request byte, which is roughly two round trips with TLS 1.3 and more with TLS 1.2. QUIC integrates TLS 1.3 into its own handshake, so a new connection completes in one round trip. Resumed connections can send application data in the first flight (0-RTT).
2. Transport-level head-of-line (HoL) blocking. TCP delivers a byte stream in order. When a segment is lost, data that arrived after it waits in the receive buffer until the retransmission arrives. HTTP/2 multiplexes many streams over one TCP connection, so one lost packet can stall every stream on that connection. HTTP/1.1 sidesteps this by using many connections, at the cost of more handshakes and sockets.

TCP head-of-line blocking (HTTP/2 over one TCP connection):
Packet 1 [OK] -> Packet 2 [LOST] -> Packet 3 [HELD] -> Packet 4 [HELD]
                 Delivery to all streams stalls until Packet 2 is retransmitted

What QUIC changes

HTTP/3 (RFC 9114) runs over QUIC (RFC 9000), a UDP-based transport with TLS 1.3 built in. QUIC streams are independent at the transport layer, so loss on one stream does not block delivery on another.

QUIC independent streams:
Stream A: frame [OK]   -> delivered immediately
Stream B: frame [LOST] -> retransmitted, only Stream B waits
Stream C: frame [OK]   -> delivered immediately

Two caveats matter for webhook gateways:

• 0-RTT data is replayable. An attacker who captures a 0-RTT flight can replay it, so 0-RTT should only carry idempotent requests. The quic-go library makes this explicit: its client allows 0-RTT only for GET requests and notes that 0-RTT provides no replay protection. A webhook is a POST that triggers side effects, so for webhook endpoints leave 0-RTT off or make processing strictly idempotent.
• Plain HTTP/3 already gives you stream independence. You do not need datagrams to get the head-of-line benefit. Ordinary HTTP/3 request streams are reliable and independent.

Where HTTP/3 Datagrams fit

QUIC DATAGRAM frames (RFC 9221) carry unreliable data. They are not retransmitted if lost, but unlike raw UDP they remain subject to the connection’s congestion control. HTTP/3 Datagrams (RFC 9297) map them onto HTTP requests. They are associated with a request stream, and in practice they are used with extended CONNECT, as in WebTransport and CONNECT-UDP (RFC 9298).

That has direct consequences for webhook design:

• Standard webhook providers do not send datagrams. They send HTTPS POST requests, and the documentation of the providers cited above specifies exactly that. Datagrams are relevant only when you control both ends, for example telemetry agents, IoT devices, or location trackers that you operate.
• Datagrams suit data where the newest value supersedes older ones. For a status heartbeat or a position update, dropping a stale value is better than delaying fresh ones behind a retransmission. They are a poor fit for anything that must not be lost, such as payment events.
• Datagrams are size-limited. A datagram must fit in a single QUIC packet, so payloads are limited to roughly the path MTU minus protocol overhead. Larger payloads need ordinary streams.

Server sketch in Go (quic-go)

The quic-go HTTP/3 package supports HTTP Datagrams (RFC 9297) on both client and server, and the HTTP/3 API changed substantially across releases. For example, Settingser was removed in v0.43.0, and Settings.EnableDatagram was renamed EnableDatagrams. At the time of writing, pkg.go.dev lists v0.61.0 as the latest release. Treat the following as a sketch and verify the types against the version you pin.

package main

import (
    "log"
    "net/http"

    "github.com/quic-go/quic-go"
    "github.com/quic-go/quic-go/http3"
)

// Bounded hand-off to the persistence layer. When it is full, datagrams
// are dropped, which is acceptable for supersedable telemetry.
var ingest = make(chan []byte, 4096)

func datagramHandler(w http.ResponseWriter, r *http.Request) {
    // HTTP Datagrams ride on an HTTP request stream, typically extended CONNECT.
    if r.Method != http.MethodConnect {
        http.Error(w, "CONNECT required", http.StatusMethodNotAllowed)
        return
    }
    w.WriteHeader(http.StatusOK)

    // The ResponseWriter implements http3.HTTPStreamer, which gives access
    // to the underlying request stream and its datagram methods.
    str := w.(http3.HTTPStreamer).HTTPStream()
    defer str.Close()

    for {
        data, err := str.ReceiveDatagram(r.Context())
        if err != nil {
            return // connection closed or request canceled
        }
        select {
        case ingest <- data:
        default: // queue full: drop rather than spawn unbounded goroutines
        }
    }
}

func main() {
    server := http3.Server{
        Addr:            ":4433",
        EnableDatagrams: true,
        QUICConfig:      &quic.Config{EnableDatagrams: true},
        Handler:         http.HandlerFunc(datagramHandler),
    }
    log.Fatal(server.ListenAndServeTLS("cert.pem", "key.pem"))
}

Note the bounded channel: spawning a goroutine per datagram, as many examples do, lets an attacker or a traffic spike create unbounded memory growth.

QUIC also has its own amplification and address-validation concerns, so the usual UDP-facing protections (rate limiting, retry tokens, and a fronting proxy or CDN that terminates QUIC) apply to any public HTTP/3 listener.

---

3. Preventing Slowloris and Slow-Read Attacks

Webhook endpoints are public by necessity, which makes them targets for application-layer “low-and-slow” attacks that exhaust connection capacity rather than bandwidth.

The attack patterns

1. Slowloris (popularized by Robert Hansen in 2009). The attacker opens many connections and sends partial HTTP request headers, adding a header line every few seconds to keep each connection alive. If the server dedicates a thread, process, or fixed connection slot to each connection, the pool fills and legitimate requests are refused.
2. Slow request body. The attacker sends valid headers, then trickles the body (including chunked bodies) at a few bytes per interval.
3. Slow-Read. The attacker sends a valid request but advertises a very small TCP receive window, so the server can only transmit its response in tiny segments. Response buffers and connection state stay allocated for a long time.

Slowloris lifecycle:
Client --[ POST /webhook HTTP/1.1 ]--------> Server allocates connection state
Client --[ ...wait... one more header ]-----> connection stays open
Client --[ ...wait... one more header ]-----> connection stays open
Result: connection slots exhausted, legitimate senders refused

Event-driven servers such as NGINX handle idle connections cheaply, so thread-per-connection servers are the most exposed. Even so, every server has a finite number of connections and file descriptors, so timeouts and limits still matter.

Key principle: most timeouts measure gaps, not totals

A common mistake is assuming client_header_timeout-style settings bound the total time to receive a request. In NGINX, client_header_timeout, client_body_timeout, and send_timeout apply to the interval between two successive read or write operations, not to the whole transfer. An attacker who sends one byte every few seconds can therefore stay under a gap-based timeout indefinitely. Effective defense combines several layers:

• gap timeouts (so completely idle connections are dropped quickly),
• caps on header and body size,
• per-IP or per-client connection limits,
• an absolute deadline for receiving the request head, where the server supports one,
• and a front proxy or CDN that absorbs the slow clients.

NGINX has no directive for a minimum transfer rate. If you need rate-based eviction, you must implement it in your own server code or rely on a CDN or WAF feature.

NGINX example

http {
    # Timeouts between successive reads/writes, not whole-request limits
    client_header_timeout 5s;
    client_body_timeout   10s;
    send_timeout          10s;

    # Drop timed-out connections immediately instead of lingering
    reset_timedout_connection on;

    # Bound request size; webhooks are usually small
    client_max_body_size 2m;

    # Per-IP concurrent connection cap
    limit_conn_zone $binary_remote_addr zone=per_ip:10m;

    server {
        listen 443 ssl;
        http2 on;   # the "http2" parameter of listen is deprecated since 1.25.1
        server_name ingress.example.com;

        limit_conn per_ip 100;

        location /api/v1/webhooks {
            # NGINX buffers the request body by default, so a slow client
            # does not hold an upstream connection while it trickles data.
            proxy_pass http://internal_webhook_cluster;
        }
    }
}

Two practical notes. First, the http2 directive was introduced in NGINX 1.25.1, which is why listen 443 ssl http2; is now deprecated. Second, per-IP limits can punish legitimate senders: providers often deliver from a small set of shared IP ranges, so give published sender ranges (such as those GitHub exposes through its /meta endpoint) higher limits or exempt them, and apply the strictest limits to everyone else.

Go net/http example

srv := &http.Server{
    Addr:              ":8443",
    ReadHeaderTimeout: 4 * time.Second,  // absolute deadline for request headers
    ReadTimeout:       15 * time.Second, // absolute deadline for the whole request
    WriteTimeout:      15 * time.Second,
    IdleTimeout:       60 * time.Second,
    MaxHeaderBytes:    16 << 10,
}

// In the handler, also cap the body size:
// r.Body = http.MaxBytesReader(w, r.Body, 2<<20)

Go’s server timeouts are absolute, per-request deadlines, which is stricter than NGINX’s gap-based timeouts. The trade-off is that a fixed ReadTimeout can cut off a large but legitimate upload on a slow link. If that matters, use http.ResponseController to extend the read deadline as data arrives.

Rust (Actix Web) example

use actix_web::{web, App, HttpResponse, HttpServer};
use std::time::Duration;

async fn handle_webhook(_body: web::Bytes) -> HttpResponse {
    // Persist or enqueue the payload, then acknowledge quickly.
    HttpResponse::Ok().finish()
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(web::resource("/webhook").to(handle_webhook)))
        // Time allowed for the client to send the full request head.
        // Slow clients receive 408 Request Timeout. Default: 5 seconds.
        .client_request_timeout(Duration::from_secs(4))
        // Time allowed for connection shutdown. Default: 1 second.
        .client_disconnect_timeout(Duration::from_secs(1))
        .bind(("0.0.0.0", 8080))?
        .run()
        .await
}

client_request_timeout covers only reading the request head, and client_disconnect_timeout governs connection shutdown; neither is a minimum-throughput control. Enforce body size limits and body-read deadlines separately.

Related protocol-level attacks

HTTP/2 introduced its own resource-exhaustion vector: the “Rapid Reset” attack (CVE-2023-44487), disclosed in October 2023, abuses stream creation and immediate cancellation to overload servers. Keep your HTTP/2 implementation and proxies patched and configure concurrent-stream limits. Slow-connection defenses and HTTP/2 hardening are separate concerns.

---

4. Surviving Traffic Bursts: Durable Buffering with mmap

The problem

Bursts of webhooks happen when an upstream system emits many events at once, such as a batch job, a retry storm after your own outage, or a bulk operation. If the ingress process holds every in-flight payload in heap memory (Go slices, Rust Vec<u8>), memory grows with the burst. In a container, exceeding the memory limit gets the process killed by the kernel’s OOM killer and in-flight data is lost.

The most reliable defense is architectural, not a buffer trick:

• Cap request body sizes and concurrent in-flight requests.
• Apply backpressure: return 429 or 503 with Retry-After when capacity is exhausted. Providers that retry, such as Stripe, will redeliver later.
• Acknowledge only after the payload is durably stored, so an acknowledged event is never lost.

What an mmap ring buffer gives you

mmap maps a file into the process’s address space. Writes go to kernel page-cache pages that the kernel writes back to disk in the background. A fixed-size file used as a circular buffer gives you a bounded, disk-backed staging area with these properties:

• Bounded, preallocated capacity. The buffer cannot grow past the file size. When it is full you reject new work explicitly instead of crashing.
• Survives process crashes. Pages written through a MAP_SHARED mapping belong to the kernel’s page cache, so data written before the process dies is not lost when the process is killed. It is not durable against power loss or a kernel crash unless you flush with msync or fsync. Each flush costs latency, so choose a flush policy deliberately.
• Less heap pressure. Payload bytes live in file-backed pages rather than the Go or Rust heap, which relieves garbage-collector and allocator pressure.

What it does not give you

• It is not “zero OOM.” On Linux, page cache and file-backed memory are charged to the container’s memory cgroup. Kubernetes’ working-set metric, which is what the OOM decision tracks, includes active file-backed memory. The kernel can reclaim clean pages and write back dirty ones, but under heavy write load into a tight limit, dirty pages can cause memory-cgroup OOM kills. Test your configuration under load, tune dirty-page writeback, and size limits with headroom.
• It is not zero-copy. Copying the payload from the network buffer into the mapping is still a memory copy. The savings are in allocation and GC behavior, not in copy count.
• It needs a defined recovery protocol. After a crash, the consumer must be able to find valid records. Production designs add record checksums or sequence numbers and flush ordering rules.

A correct single-producer, single-consumer ring in Go

Head and tail are monotonically increasing byte counters, with the position in the buffer computed modulo capacity. Records are length-prefixed and may wrap around the end of the region, which the copy helpers handle. This version supports exactly one producer goroutine and one consumer goroutine; with multiple producers, serialize writes with a mutex or give each producer its own ring.

package ringbuf

import (
    "encoding/binary"
    "errors"
    "fmt"
    "os"
    "sync/atomic"
    "syscall"
    "unsafe"
)

const (
    headerSize = 64 // head at offset 0, tail at offset 8, rest reserved
    lenPrefix  = 8
)

var (
    ErrFull  = errors.New("ring buffer full")
    ErrEmpty = errors.New("ring buffer empty")
)

type Ring struct {
    file   *os.File
    mem    []byte // whole mapping, including header
    region []byte // data area
    head   *uint64
    tail   *uint64
}

// Open maps a file-backed ring of the given total size. Reopening an
// existing file of the same size preserves its contents and positions.
func Open(path string, size int) (*Ring, error) {
    f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o600)
    if err != nil {
        return nil, err
    }
    if err := f.Truncate(int64(size)); err != nil {
        f.Close()
        return nil, err
    }
    mem, err := syscall.Mmap(int(f.Fd()), 0, size,
        syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)
    if err != nil {
        f.Close()
        return nil, fmt.Errorf("mmap failed: %w", err)
    }
    return &Ring{
        file:   f,
        mem:    mem,
        region: mem[headerSize:],
        head:   (*uint64)(unsafe.Pointer(&mem[0])),
        tail:   (*uint64)(unsafe.Pointer(&mem[8])),
    }, nil
}

// Write appends one record. It returns ErrFull instead of overwriting
// unread data, so the caller can apply backpressure (for example HTTP 503).
func (r *Ring) Write(p []byte) error {
    need := uint64(lenPrefix + len(p))
    capacity := uint64(len(r.region))
    tail := atomic.LoadUint64(r.tail)
    head := atomic.LoadUint64(r.head)
    if tail-head+need > capacity {
        return ErrFull
    }
    var hdr [lenPrefix]byte
    binary.LittleEndian.PutUint64(hdr[:], uint64(len(p)))
    r.copyIn(tail, hdr[:])
    r.copyIn(tail+lenPrefix, p)
    atomic.StoreUint64(r.tail, tail+need) // publish only after data is written
    return nil
}

// Read removes and returns the oldest record.
func (r *Ring) Read() ([]byte, error) {
    head := atomic.LoadUint64(r.head)
    tail := atomic.LoadUint64(r.tail)
    if head == tail {
        return nil, ErrEmpty
    }
    var hdr [lenPrefix]byte
    r.copyOut(head, hdr[:])
    n := binary.LittleEndian.Uint64(hdr[:])
    out := make([]byte, n)
    r.copyOut(head+lenPrefix, out)
    atomic.StoreUint64(r.head, head+lenPrefix+n)
    return out, nil
}

func (r *Ring) copyIn(pos uint64, src []byte) {
    off := pos % uint64(len(r.region))
    n := copy(r.region[off:], src)
    copy(r.region, src[n:]) // wrap around
}

func (r *Ring) copyOut(pos uint64, dst []byte) {
    off := pos % uint64(len(r.region))
    n := copy(dst, r.region[off:])
    copy(dst[n:], r.region) // wrap around
}

// Sync flushes dirty pages to disk. Call it on your chosen durability schedule.
func (r *Ring) Sync() error {
    _, _, errno := syscall.Syscall(syscall.SYS_MSYNC,
        uintptr(unsafe.Pointer(&r.mem[0])), uintptr(len(r.mem)), uintptr(syscall.MS_SYNC))
    if errno != 0 {
        return errno
    }
    return nil
}

func (r *Ring) Close() error {
    _ = syscall.Munmap(r.mem)
    return r.file.Close()
}

This sketch uses Linux-specific system calls and omits checksums, corruption recovery, and multi-producer support. Treat it as a starting point rather than production code; mature alternatives such as a write-ahead log library or an embedded queue are often a better choice than writing your own.

Choosing a buffering strategy

I deliberately omit latency figures for the options below, because they depend heavily on hardware, payload size, and configuration. Benchmark on your own infrastructure.

Strategy	Memory behavior	Durability	Main risk
In-process heap queue	Grows with the burst	Lost on process crash	OOM kill under bursts
Remote queue or broker (Redis, Kafka, RabbitMQ)	Offloaded to a separate system	Depends on broker configuration	Network hop and an extra dependency on the ingest path
mmap-backed ring on local disk	Bounded file size; page cache counts toward cgroup limits	Survives process crash; power loss needs msync/fsync	Dirty-page pressure and a custom recovery protocol

---

5. Putting It Together

A defensible layered design looks like this:

Public senders
      │
      ▼
┌──────────────────────────────────────────────────────┐
│ Edge layer: CDN / reverse proxy (NGINX, Envoy, etc.) │
│  • Header/body gap timeouts, size caps, conn limits  │
│  • Terminates TLS (and HTTP/2, optionally HTTP/3)    │
│  • Buffers slow clients away from your application   │
└───────────────────────────┬──────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────┐
│ Ingress worker (Go / Rust)                           │
│  • Verify signature, enforce body limit              │
│  • Append to durable buffer; ack only after success  │
│  • Return 503 + Retry-After when the buffer is full  │
└───────────────────────────┬──────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────┐
│ Durable buffer (mmap ring, WAL, or message broker)   │
└───────────────────────────┬──────────────────────────┘
                            ▼
              Async consumers: dedupe, process, route

The priorities, in order:

1. Acknowledge fast, process later. Providers publish deadlines (10 seconds for GitHub, for example), and handlers that do real work inline miss them and trigger retries.
2. Bound everything. Connection counts, header and body sizes, queue depth, and buffer capacity all need explicit limits, with backpressure when they are reached.
3. Use HTTP/3 where you control the clients and it measurably helps. For third-party webhook providers, expect HTTPS POST over HTTP/1.1 or HTTP/2 and design for that. Reserve datagrams for loss-tolerant, supersedable telemetry from your own clients.
4. Make duplicates harmless. Retries and redelivery are part of every provider’s delivery model, so idempotent processing is mandatory.
5. Load-test the failure modes. Simulate slow clients, bursts at several times your normal peak, and a process kill mid-write, and watch the container’s memory working set, not just the process RSS.

---

Further Reading

• RFC 9000 (QUIC), RFC 9114 (HTTP/3), RFC 9221 (QUIC DATAGRAM), RFC 9297 (HTTP Datagrams), RFC 9298 (CONNECT-UDP), at rfc-editor.org
• Stripe: Receive Stripe events in your webhook endpoint
• GitHub: Best practices for using webhooks
• quic-go: HTTP/3 package documentation
• NGINX: ngx_http_v2_module

---

Originally published at https://instawebhook.com/blog/high-performance-ingress-network-architecture-building-resilience-at-scale