DEV Community: InstaWebhook

Routing Webhooks Through Cloudflare Workers KV and Durable Objects: Building a Globally Distributed Ingress Buffer

InstaWebhook — Sat, 17 Oct 2026 02:07:00 +0000

Inbound webhooks are deceptively hard to handle at scale. Providers such as Stripe, GitHub, and Shopify don’t know about your database connection pool, your autoscaling lag, or a brief microservice outage. They send events when things happen, and a busy period for one of your customers can turn into a burst of requests against your origin.

If the origin sits directly behind a load balancer, one of those bursts can exhaust database connections and memory and cause timeouts. Some senders then retry, which adds more load while you are already struggling.

One way to protect the origin is an ingress buffer at the edge. With Cloudflare Workers, Workers KV, and Durable Objects, you can verify signatures, deduplicate, rate-limit, and durably queue incoming webhooks before a single request reaches your infrastructure. Cloudflare runs Workers in more than 330 cities and says that puts code within roughly 50 ms of about 95% of the world’s Internet-connected population.

One caveat up front: only the entrypoint Worker is “everywhere”. Each tenant’s Durable Object is a single globally unique instance that coordinates all requests sent to it. The buffer is therefore distributed across tenants, not replicated for any one tenant.

Why Webhook Ingestion Fails

Each of these failure modes shapes the design below.

Retry behavior differs by provider. Stripe retries failed live-mode deliveries with exponential backoff for up to three days and disables endpoints that keep failing. Shopify retries 8 times over the following 4 hours. GitHub does not automatically redeliver failed deliveries at all. A short outage can therefore produce a burst of retries from some senders and permanently lost events from others. You can’t assume one retry contract.
Senders give up quickly. Shopify allows a one-second connection timeout and a five-second timeout for the whole request. If your origin does real work before responding, slow responses are treated as failures and trigger the retries you were trying to avoid.
Duplicates are normal. Providers deliver at least once. Shopify says its app “might receive the same webhook more than once”, for example after a network timeout or a retry, and recommends deduplicating on the X-Shopify-Webhook-Id header. Stripe events carry a stable event ID that stays the same across retries.
One tenant can starve the rest. A misconfigured integration that fires thousands of events can consume shared processing capacity unless you isolate and limit it per tenant.

Acknowledging quickly after a durable write handles the first two, deduplication handles the third, and a per-tenant limiter handles the fourth.

Architecture

[Webhook sender]
       │
       ▼
[Worker: validate tenant, verify HMAC]
       │            │
       │            └──► [Workers KV: cached tenant config, secrets, limits]
       ▼
[Durable Object (one per tenant, SQLite-backed)]
   ├─ dedupe by event ID
   ├─ per-tenant rate limit
   ├─ durable event queue
   └─ alarm → batched delivery to origin (or to Cloudflare Queues)

Worker (entrypoint): Parses the path, loads tenant config, verifies the signature, and forwards verified events to the tenant’s Durable Object. Anycast routing and Cloudflare’s scheduling typically place it close to the sender.
Workers KV: A read-optimized, eventually consistent cache for data that changes rarely, such as tenant config and rate limits. Cloudflare cites sub-5 ms hot read latency. Writes can take 60 seconds or more to be visible in other locations.
Durable Objects: Provide per-tenant coordination with a private, transactional SQLite database. Because each object processes one tenant’s events, dedupe checks and rate-limit counters don’t need distributed locks.
Alarms: Each object can schedule a single alarm, which lets it drain its buffer in batches without waiting for another incoming request.

Step 1: The Entrypoint Worker

The Worker reads the raw bytes, verifies the HMAC over exactly those bytes, and only then touches the Durable Object. Several details differ from naive implementations:

Fail closed. If there is no secret for the tenant, the request is rejected. A shared fallback secret would let anyone who knows it forge events for every tenant.
Constant-time comparison. Workers provide crypto.subtle.timingSafeEqual. Cloudflare’s documentation notes it is not constant-time with respect to length, which is harmless here because an HMAC-SHA256 digest is always 32 bytes.
Meaningful dedupe keys. If the sender provides no event ID, the code falls back to a hash of the body. A random UUID would defeat deduplication, because every retry would look new.
Minimal forwarding. Only what the origin needs is passed on, not every header. Forwarding all headers would persist signatures and other sensitive values.

// src/index.ts
import type { TenantWebhookBuffer } from "./tenant-webhook-buffer";

export interface Env {
  TENANT_DO: DurableObjectNamespace<TenantWebhookBuffer>;
  WEBHOOK_KV: KVNamespace;
  ORIGIN_BASE_URL: string; // e.g. https://api.example.com
  ORIGIN_TOKEN: string;    // set with `wrangler secret put ORIGIN_TOKEN`
}

interface TenantConfig {
  active: boolean;
  rateLimit: number;   // max accepted events per 10-second window
  secrets: string[];   // current + previous secret during rotation
}

const TENANT_ID = /^[A-Za-z0-9_-]{1,64}$/;
const encoder = new TextEncoder();

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    if (request.method !== "POST") {
      return new Response("Method Not Allowed", { status: 405, headers: { Allow: "POST" } });
    }

    // Expected path: /webhook/<tenantId>
    const [, prefix, tenantId] = new URL(request.url).pathname.split("/");
    if (prefix !== "webhook" || !tenantId || !TENANT_ID.test(tenantId)) {
      return new Response("Not Found", { status: 404 });
    }

    const config = await getTenantConfig(env, ctx, tenantId);
    if (!config?.active) {
      return new Response("Unknown tenant", { status: 404 });
    }

    // Verify against the exact bytes the sender signed.
    const bodyBytes = new Uint8Array(await request.arrayBuffer());
    const signature = request.headers.get("X-Signature") ?? "";

    let valid = false;
    for (const secret of config.secrets) {
      if (await verifyHmacHex(bodyBytes, signature, secret)) valid = true;
    }
    if (!valid) {
      return new Response("Invalid signature", { status: 401 });
    }

    const eventId = request.headers.get("X-Event-ID") ?? (await sha256Hex(bodyBytes));

    try {
      const stub = env.TENANT_DO.getByName(tenantId);
      const result = await stub.ingestWebhook(
        {
          eventId,
          receivedAt: Date.now(),
          contentType: request.headers.get("Content-Type") ?? "application/octet-stream",
          rawBody: new TextDecoder().decode(bodyBytes), // assumes UTF-8 payloads (JSON)
        },
        config.rateLimit,
      );
      return Response.json(result.body, {
        status: result.status,
        headers: result.retryAfter ? { "Retry-After": String(result.retryAfter) } : undefined,
      });
    } catch {
      // The event was NOT durably stored, so ask the sender to retry.
      return new Response("Temporarily unavailable", { status: 503, headers: { "Retry-After": "30" } });
    }
  },
};

async function verifyHmacHex(body: Uint8Array, headerSig: string, secret: string): Promise<boolean> {
  const provided = hexToBytes(headerSig);
  if (!provided) return false;

  const key = await crypto.subtle.importKey(
    "raw",
    encoder.encode(secret),
    { name: "HMAC", hash: "SHA-256" },
    false,
    ["sign"],
  );
  const expected = new Uint8Array(await crypto.subtle.sign("HMAC", key, body));
  return provided.length === expected.length && crypto.subtle.timingSafeEqual(provided, expected);
}

function hexToBytes(hex: string): Uint8Array | null {
  if (hex.length % 2 !== 0 || !/^[0-9a-fA-F]*$/.test(hex)) return null;
  const out = new Uint8Array(hex.length / 2);
  for (let i = 0; i < out.length; i++) out[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16);
  return out;
}

async function sha256Hex(data: Uint8Array): Promise<string> {
  const digest = await crypto.subtle.digest("SHA-256", data);
  return Array.from(new Uint8Array(digest), (b) => b.toString(16).padStart(2, "0")).join("");
}

This example assumes a generic sender that sends a hex HMAC-SHA256 of the body in X-Signature. Real providers differ, so adapt the verification step:

GitHub sends sha256=<hex> in X-Hub-Signature-256.
Shopify sends a base64-encoded digest in X-Shopify-Hmac-SHA256.
Stripe signs a timestamp plus the payload, which also gives you replay protection.

For providers with a stable delivery ID, use it as the dedupe key: X-Shopify-Webhook-Id for Shopify, X-GitHub-Delivery for GitHub, and the event id in the body for Stripe.

The Worker calls the Durable Object through Workers RPC. Public methods on a class that extends DurableObject are callable through a stub. Arguments and return values must be serializable, so this example returns a plain object and builds the Response in the Worker.

Step 2: Per-Tenant State in a Durable Object

Each tenant’s Durable Object holds four things in its SQLite database:

processed event IDs, for deduplication
a short rate-limit window
the pending event queue
a small counter for delivery backoff

SQLite-backed Durable Objects are Cloudflare’s recommended storage backend for new classes, and the SQL API is synchronous. Writes made without an intervening await are committed together, and an output gate holds the response until the writes are confirmed. The 202 therefore means the event has been durably stored, not just accepted into memory.

Keeping deduplication and rate limiting in separate tables matters. Pruning old rate-limit rows from the same table that records processed IDs would erase your dedupe history after seconds.

// src/tenant-webhook-buffer.ts
import { DurableObject } from "cloudflare:workers";
import type { Env } from "./index";

export interface WebhookEvent {
  eventId: string;
  receivedAt: number;
  contentType: string;
  rawBody: string;
}

export interface IngestResult {
  status: 200 | 202 | 429;
  body: Record<string, unknown>;
  retryAfter?: number;
}

const RATE_WINDOW_MS = 10_000;
const DEDUPE_RETENTION_MS = 4 * 24 * 60 * 60 * 1000; // longer than Stripe's 3-day retry window
const BATCH_SIZE = 50;
const FLUSH_DELAY_MS = 2_000;
const MAX_BACKOFF_MS = 5 * 60 * 1000;

export class TenantWebhookBuffer extends DurableObject<Env> {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env);
    ctx.blockConcurrencyWhile(async () => {
      const sql = ctx.storage.sql;
      sql.exec(`CREATE TABLE IF NOT EXISTS processed_events (
        event_id TEXT PRIMARY KEY, received_at INTEGER NOT NULL)`);
      sql.exec(`CREATE TABLE IF NOT EXISTS rate_window (received_at INTEGER NOT NULL)`);
      sql.exec(`CREATE INDEX IF NOT EXISTS idx_rate_window ON rate_window (received_at)`);
      sql.exec(`CREATE TABLE IF NOT EXISTS event_queue (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        event_id TEXT NOT NULL, payload TEXT NOT NULL, received_at INTEGER NOT NULL)`);
      sql.exec(`CREATE TABLE IF NOT EXISTS flush_state (
        id INTEGER PRIMARY KEY CHECK (id = 1), failures INTEGER NOT NULL DEFAULT 0)`);
      sql.exec(`INSERT OR IGNORE INTO flush_state (id, failures) VALUES (1, 0)`);
    });
  }

  async ingestWebhook(event: WebhookEvent, limit = 100): Promise<IngestResult> {
    const sql = this.ctx.storage.sql;
    const now = Date.now();

    // 1. Deduplicate. Duplicates are acknowledged with 200 and cost almost nothing.
    const seen = sql.exec("SELECT 1 FROM processed_events WHERE event_id = ?", event.eventId).toArray();
    if (seen.length > 0) {
      return { status: 200, body: { status: "deduplicated", eventId: event.eventId } };
    }

    // 2. Sliding-window rate limit over accepted events.
    sql.exec("DELETE FROM rate_window WHERE received_at < ?", now - RATE_WINDOW_MS);
    const { count } = sql.exec<{ count: number }>("SELECT COUNT(*) AS count FROM rate_window").one();
    if (count >= limit) {
      return { status: 429, body: { status: "rate_limited" }, retryAfter: 10 };
    }

    // 3. Record and enqueue atomically.
    this.ctx.storage.transactionSync(() => {
      sql.exec("INSERT INTO processed_events (event_id, received_at) VALUES (?, ?)", event.eventId, now);
      sql.exec("INSERT INTO rate_window (received_at) VALUES (?)", now);
      sql.exec(
        "INSERT INTO event_queue (event_id, payload, received_at) VALUES (?, ?, ?)",
        event.eventId,
        JSON.stringify(event),
        now,
      );
    });

    // Schedule a flush only if one isn't pending. Calling setAlarm() again would
    // overwrite the pending alarm and keep pushing it back under steady traffic.
    if ((await this.ctx.storage.getAlarm()) === null) {
      await this.ctx.storage.setAlarm(now + FLUSH_DELAY_MS);
    }

    return { status: 202, body: { status: "buffered", eventId: event.eventId } };
  }

  async alarm(): Promise<void> {
    const sql = this.ctx.storage.sql;

    // Housekeeping: expire old dedupe records.
    sql.exec("DELETE FROM processed_events WHERE received_at < ?", Date.now() - DEDUPE_RETENTION_MS);

    const rows = sql
      .exec<{ id: number; payload: string }>("SELECT id, payload FROM event_queue ORDER BY id LIMIT ?", BATCH_SIZE)
      .toArray();
    if (rows.length === 0) return;

    let delivered = false;
    try {
      const response = await fetch(`${this.env.ORIGIN_BASE_URL}/internal/webhooks/batch`, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${this.env.ORIGIN_TOKEN}`,
        },
        body: JSON.stringify({ events: rows.map((r) => JSON.parse(r.payload)) }),
        signal: AbortSignal.timeout(10_000),
      });
      delivered = response.ok;
    } catch {
      delivered = false;
    }

    if (delivered) {
      sql.exec("DELETE FROM event_queue WHERE id <= ?", rows[rows.length - 1].id);
      sql.exec("UPDATE flush_state SET failures = 0 WHERE id = 1");
      const { n } = sql.exec<{ n: number }>("SELECT COUNT(*) AS n FROM event_queue").one();
      if (n > 0) await this.ctx.storage.setAlarm(Date.now() + FLUSH_DELAY_MS); // keep draining
    } else {
      const { failures } = sql.exec<{ failures: number }>("SELECT failures FROM flush_state WHERE id = 1").one();
      sql.exec("UPDATE flush_state SET failures = ? WHERE id = 1", failures + 1);
      const delay = Math.min(FLUSH_DELAY_MS * 2 ** failures, MAX_BACKOFF_MS);
      await this.ctx.storage.setAlarm(Date.now() + delay);
    }
  }
}

A few design points are worth explaining:

Alarm retries are limited. Cloudflare retries a throwing alarm() handler with exponential backoff starting at 2 seconds, up to 6 retries. After that the alarm is not retried. That is not enough to ride out a long origin outage, so the handler above catches errors and manages its own backoff, capped at five minutes.
Delivery to the origin is at-least-once. If the Durable Object is interrupted after the origin accepted a batch but before the rows are deleted, the batch is sent again. Each event carries its eventId, so the origin should deduplicate on it.
A permanently failing batch blocks the head of the queue. In production, cap per-event attempts and move poison events to a dead-letter table.
The 429 is a safety valve, not a guarantee. Providers that retry, such as Stripe and Shopify, will treat a non-2xx as a failure and try again later. Don’t rely on every sender honoring Retry-After, and GitHub won’t redeliver automatically.

Step 3: Workers KV for Tenant Config

KV suits data that is read often and changed rarely, such as tenant status, rate limits, and signing secrets. It is eventually consistent. A write is visible immediately in the location that made it, but can take 60 seconds or more to propagate elsewhere, and a missing key can be cached for up to that long too. KV also enforces a minimum TTL of 60 seconds and a maximum of one write per second to the same key.

That has practical consequences:

Use short TTLs. A 300-second expirationTtl is a reasonable default for rules that change occasionally.
Accept old and new secrets during rotation. The secrets array in the config above does this, because a rotated secret can lag at some edge locations.
Don’t write on every cache miss. Concurrent misses for one key would hit the write limit, so the write happens in ctx.waitUntil and failures are ignored.
Don’t cache unknown tenants. Otherwise anyone can fill your namespace with junk keys by inventing tenant IDs.

async function getTenantConfig(env: Env, ctx: ExecutionContext, tenantId: string): Promise<TenantConfig | null> {
  const key = `config:${tenantId}`;

  const cached = await env.WEBHOOK_KV.get<TenantConfig>(key, { type: "json", cacheTtl: 60 });
  if (cached) return cached;

  // Cache miss: load from the system of record. Only known tenants are cached.
  const response = await fetch(`${env.ORIGIN_BASE_URL}/internal/tenants/${tenantId}/webhook-config`, {
    headers: { Authorization: `Bearer ${env.ORIGIN_TOKEN}` },
    signal: AbortSignal.timeout(3_000),
  });
  if (!response.ok) return null;

  const fresh = (await response.json()) as TenantConfig;
  ctx.waitUntil(
    env.WEBHOOK_KV.put(key, JSON.stringify(fresh), { expirationTtl: 300 }).catch(() => {}),
  );
  return fresh;
}

Treat KV as a cache of data you can reload from your system of record. Never hardcode secrets in source.

Wrangler Configuration

Cloudflare now documents a declarative exports field for Durable Object class lifecycle in new Workers. Older projects use the migrations array with new_sqlite_classes, and a Worker can use only one of the two. Cloudflare recommends the SQLite backend for all new namespaces.

// wrangler.jsonc
{
  "name": "webhook-ingress",
  "main": "src/index.ts",
  "compatibility_date": "2026-10-01",
  "vars": { "ORIGIN_BASE_URL": "https://api.example.com" },
  "kv_namespaces": [{ "binding": "WEBHOOK_KV", "id": "<KV_NAMESPACE_ID>" }],
  "durable_objects": {
    "bindings": [{ "name": "TENANT_DO", "class_name": "TenantWebhookBuffer" }]
  },
  "exports": {
    "TenantWebhookBuffer": { "type": "durable-object", "storage": "sqlite" }
  }
}

If you are on the legacy flow, replace exports with "migrations": [{ "tag": "v1", "new_sqlite_classes": ["TenantWebhookBuffer"] }].

Failure Modes and Mitigations

Component	Consistency model	Primary failure mode	Mitigation
Worker	Stateless	Uncaught exception or timeout	Wrap the Durable Object call in `try/catch` and return a deliberate 503 so retrying senders try again
Workers KV	Eventually consistent	Stale config or secrets for 60 seconds or more	Short TTLs, accept old and new secrets during rotation, don’t cache unknown tenants
Durable Object	Strongly consistent per object	Per-object limits: 10 GB storage on Workers Paid (1 GB on Free), soft limit of about 1,000 requests per second	Alert on backlog depth, split very large tenants across several objects, and shard dedupe by event ID
Alarm	At-least-once	Handler keeps failing after the built-in retries	Custom backoff with a cap; origin deduplicates by `eventId`
Origin	Yours	Outage or slow responses	The buffer absorbs it; drain in controlled batches

Backpressure and Origin Outages

If the origin goes down, each tenant’s Durable Object keeps accepting and storing events up to its storage limit, and the sender sees fast 202 responses. When the origin recovers, the alarm drains the backlog in batches of 50 with a pause between them, which avoids a thundering herd against your database.

Buffered events are only safe while there is space. When a SQLite-backed Durable Object hits its storage cap, writes fail. In this design that surfaces as a 503, and providers that retry will redeliver later. Providers that don’t retry, GitHub among them, will have lost the event. Monitor queue depth and age so you can act well before the cap.

Alternative: Cloudflare Queues

If you prefer to hand batches to a managed queue, Cloudflare Queues offers at-least-once delivery, consumer batches of up to 100 messages, and dead-letter queues. A single message is limited to 128 KB, so large webhook bodies need to go to object storage with only a reference queued. A Durable Object buffer, as built above, gives you per-tenant ordering, deduplication, and rate limiting in one place, and Queues can still sit behind it as the delivery mechanism.

Production Checklist

Fail closed on authentication. No fallback secrets, constant-time comparison, and provider-specific verification. Check timestamps where the provider signs them.
Deduplicate on the provider’s ID and retain it long enough. Keep dedupe records for longer than the longest provider retry window. Stripe’s is three days, hence four above.
Make the origin idempotent. Delivery from the buffer is at-least-once, so the origin must tolerate repeats.
Set limits deliberately. Choose per-tenant rate limits from real traffic, and plan for tenants that approach a single object’s throughput ceiling.
Add a dead-letter path for events the origin permanently rejects, so they don’t block a tenant’s queue.
Alert on backlog depth and oldest-event age per tenant.
Test the failure paths: origin down, KV stale, duplicate floods, and a rotated secret.

Zero-Downtime Webhook Schema Evolution: Managing Breaking Payloads Without Dropping Events

InstaWebhook — Fri, 16 Oct 2026 02:04:12 +0000

Webhooks carry a lot of B2B integration traffic: payment notifications, inventory syncs, CRM updates. If you receive them from third parties, you have probably seen the failure mode. A producer silently renames a JSON key, turns an integer ID into a prefixed string, or drops a field you depended on. Your parser throws, your endpoint starts returning 4xx or 5xx responses, and events pile up in a dead letter queue.

Webhooks have what we can call the distributed publisher problem. In an internal event stream you can often coordinate producer and consumer releases through shared CI, contract tests, or a schema registry you both control. With a third-party webhook you do not control the producer’s release cadence. Your ingestion pipeline is the part that has to absorb the change.

This guide covers how to make that absorption safe: durable capture before parsing, tolerant readers, schema validation at the edge, and versioned mapping layers backed by contract tests. One caveat first. “Zero downtime” here means you never lose an event because of a schema change. It does not mean the change costs nothing, and it does not remove the need to talk to your partners.

1. The Schema Drift Dilemma in Multi-Producer Integrations

Schema drift is the gap between the data a producer sends and the structure your consumer expects. Internally you can close that gap with contract tests and coordinated rollouts. Across dozens of external producers you mostly cannot, so you have to design for it.

Common Breaking Changes

Type mutation. A resource id changes from a number (123456) to a prefixed string ("usr_9f83j2"), or the reverse. A strictly typed model throws on the first payload that doesn’t match. Providers do consider this kind of change safe. Stripe’s API upgrade documentation, for example, lists adding or removing fixed prefixes on object IDs among its backward-compatible changes. Treat IDs as opaque strings unless the provider promises otherwise.
Key renaming and restructuring. A team refactors its domain model and renames customer_email to email_address, or promotes billing.address.zip to a top-level zip_code. Stripe’s own upgrade notes list this kind of change, such as renaming a user_id property on Connect events to account, as breaking enough to require a new API version.
Field removal. A producer deletes a field it assumed nobody used, and your fraud check or reconciliation job depended on it.

Not every change is breaking. Stripe treats adding new API resources, new properties on existing responses, and new event types as backward-compatible, and tells consumers to ignore unfamiliar fields and event types. A consumer that rejects unknown fields or event types will break on changes the producer considers routine.

Large Integers Are a Hidden Type Problem

Numeric IDs also have a precision problem. JavaScript’s Number type is an IEEE 754 double, so integers above 2^53 − 1 can’t be represented exactly. A consumer that parses a 64-bit ID with a naive JavaScript JSON.parse can silently corrupt it. For this reason many APIs send IDs as strings, and you should too if you produce webhooks.

The Cost of Rigid Ingestion

Fail-fast validation protects your database, but in an event-driven system it has real costs:

Dead letter queue overflow. Valid business events pile up behind a minor schema change and need manual replay.
Stalled workflows. Payment fulfillment, inventory locking, and similar flows wait on events that were rejected rather than processed.
Limited provider retries. Providers retry for a bounded time. Stripe, for instance, retries live-mode deliveries for up to three days with exponential backoff, then disables an endpoint that keeps failing. If your fix takes longer than the provider’s retry window, you can lose events permanently.

That last point shapes the rest of the design. The goal is to accept and persist first, interpret later.

2. Capture First: Acknowledge Fast, Parse Later

Before any schema handling, decide what your HTTP handler does. The pattern most providers recommend is:

Read the raw request body and verify the signature against it.
Write the raw payload and headers to durable storage or a queue.
Return a 2xx immediately.
Parse, validate, and map asynchronously.

Stripe’s documentation tells receivers to defer complex logic and return a successful response quickly. If you parse synchronously and fail, you depend on the provider’s retry policy. If you persist first and fail later, you can fix the parser and replay from your own store at any time.

Verify signatures against the raw bytes. Frameworks that auto-parse JSON and re-serialize it can change whitespace or key order and break the signature check. The open Standard Webhooks specification, used by providers such as Svix, Resend, and Clerk, defines webhook-id, webhook-timestamp, and webhook-signature headers. The signature is an HMAC over <id>.<timestamp>.<body>, and receivers are advised to reject messages whose timestamp falls outside a tolerance of about five minutes. The webhook-id stays the same across retries of a delivery, so it works as a deduplication key. Because providers retry, your handlers must tolerate duplicates regardless of which spec the provider follows.

3. Writing Resilient Parsers: The Tolerant Reader

Martin Fowler’s 2011 bliki entry on the Tolerant Reader builds on Postel’s Law, the robustness principle from early TCP specifications: be conservative in what you do, be liberal in what you accept from others. His recommendation is to be as tolerant as possible when reading data from a service. In practice that means extracting only the fields you need, ignoring the rest, and not making assumptions about structure you don’t depend on. He also suggests that the consumer can send its reader and tests to the provider to run in their build, so they can catch breakages early.

A caveat is worth keeping in mind. Postel’s Law has critics, including a 2023 argument by Martin Thomson and David Schinazi that unconditional liberal acceptance can undermine robustness and security over time. Tolerance without feedback hides problems. The approach in this article is therefore tolerant to keep processing, loud to keep producers honest. Accept what you can, but log and count every deviation.

Tolerant Parsing in Go

Go’s encoding/json ignores unknown object keys by default. Decoder.DisallowUnknownFields() is an opt-in method that makes the decoder strict. It takes no arguments, so tolerance is the default and you just don’t call it.

Type mismatches are the problem. Decoding a JSON string into an integer field produces an UnmarshalTypeError. The simplest tolerant approach for IDs is to store them as strings and accept either JSON form:

package main

import (
    "encoding/json"
    "fmt"
)

// FlexibleID accepts a JSON string or number and stores it as a string,
// so integer IDs, prefixed IDs and UUIDs all survive without loss.
type FlexibleID string

func (f *FlexibleID) UnmarshalJSON(data []byte) error {
    var s string
    if err := json.Unmarshal(data, &s); err == nil {
        *f = FlexibleID(s)
        return nil
    }
    var n json.Number
    if err := json.Unmarshal(data, &n); err == nil {
        *f = FlexibleID(n.String())
        return nil
    }
    return fmt.Errorf("flexible id: expected string or number, got %s", data)
}

type WebhookPayload struct {
    EventID string     `json:"event_id"`
    UserID  FlexibleID `json:"user_id"`
}

func main() {
    payloads := []string{
        `{"event_id": "evt_123", "user_id": "usr_9f83j2"}`, // new prefixed string ID
        `{"event_id": "evt_124", "user_id": 123456}`,       // original integer ID
        `{"event_id": "evt_125", "user_id": 5, "plan": "pro"}`, // unknown field is ignored
    }
    for _, p := range payloads {
        var wp WebhookPayload
        if err := json.Unmarshal([]byte(p), &wp); err != nil {
            fmt.Println("parse error:", err)
            continue
        }
        fmt.Printf("event=%s user=%s\n", wp.EventID, wp.UserID)
    }
}

The earlier common approach of coercing a non-numeric string to 0 is dangerous. Many IDs would collapse into the same value and you would corrupt data silently. Keeping the original string avoids that. Using json.Number also avoids float64 precision loss for large integers.

Key Strategies

Ignore unknown fields and unknown event types. Route unrecognized event types to a “seen but unhandled” counter rather than an error path.
Use union types for fields prone to change: IDs, timestamps (epoch seconds versus ISO 8601), and booleans sent as "true" versus true.
Separate required from nice-to-have. Fail only when a field your logic cannot work without is missing. For everything else, use a default and emit a metric.
Keep a thin boundary. Parse the payload into a small internal model at the edge, as Fowler suggests with a Data Transfer Object, so the rest of your system never touches raw partner JSON.

4. Schema Validation at the Ingress Gateway

Tolerant parsing keeps you running, but it does not tell you that a producer changed. For that you need validation and telemetry at the edge.

[Webhook producer]
       │  HTTP POST
       ▼
[Ingress: verify signature → persist raw → return 2xx]
       │
       ▼
[Validator] ──(fetch schema by partner + event type + version)──► [Schema store]
       │
       ├─► conforms   ──► [Mapper] ──► [Event bus / queue]
       └─► deviates   ──► metric + alert, still mapped if tolerable, else quarantine

Choosing a Schema Format and Registry

JSON Schema is the most widely used way to describe JSON documents. The current version is 2020-12, which the project’s site lists as the latest specification. You can store versioned schemas per partner and event type in a registry or in a simple database table.

Check what your tooling supports before standardizing on a draft. AWS Glue Schema Registry’s JSON support covers Draft-04, Draft-06, and Draft-07, with validation done by the Everit library, so it does not accept 2020-12 schemas. Glue also defines eight compatibility modes: NONE, DISABLED, BACKWARD, BACKWARD_ALL, FORWARD, FORWARD_ALL, FULL, and FULL_ALL. Other registries such as Confluent Schema Registry and Apicurio Registry also support JSON Schema, with their own draft coverage, so verify the draft against the version you run.

Tiered Validation Modes

Strict edge validation defeats the point if every producer change causes rejections. Use tiers:

Shadow mode. Validate and record deviations, but never block. This is where you discover drift.
Enforcement for critical fields. Once telemetry shows a schema is stable, reject or quarantine payloads that break the few invariants your business logic requires.
Compatibility checks on your own schema versions. Registries enforce compatibility modes such as BACKWARD or FULL between versions of a schema you register. This governs your view of a partner’s payload. You cannot enforce it on the partner. If a partner publishes a changelog or an OpenAPI/JSON Schema description, diff against it in CI.

Record the deviation rate per partner and event type. A sudden rise after a partner’s release date is often the first sign of drift, usually before any customer complains.

5. Dual-Schema Mapping Layers

Sometimes a breaking change is unavoidable, such as a restructured order object, and you will see old and new shapes at the same time. A mapping layer based on the Adapter pattern translates each variant into one canonical internal event, so only the adapter knows about partner formats.

Version detection. Identify the variant from whatever the producer gives you. Stripe, for example, ties the payload shape to the API version configured on each webhook endpoint, and its documentation recommends updating webhook code to handle both the old and new version of each object before upgrading. If a provider offers no version marker, detect by shape (presence of keys) as a fallback.
Transformation. Use declarative mappings where they fit, such as jq or JSONata, or plain functions for logic that needs types and tests.
Canonical event. Emit one internal structure that downstream services consume. Version the canonical model too, so internal changes follow the same discipline.

Keep the raw payload alongside the canonical event so you can re-map historical events when a mapping bug is found.

6. Contract Tests for Producers You Don’t Control

Consumer-driven contract testing tools such as Pact were built around services where both sides can run the tests. External webhook producers won’t run your suite, so treat recorded payloads as the contract instead:

Keep a fixture library of real payloads per partner, event type, and version, with secrets and personal data removed.
Run every fixture through the parser and mapper in CI.
When a partner announces or ships a change, add the new payload shape as a fixture before it reaches production.
Where a partner is willing, share your fixtures or schemas with them, as Fowler suggests for the Tolerant Reader.

Here is a pytest example. Parsing monetary values from floats is a classic trap: int(19.99 * 100) evaluates to 1998 in Python because 19.99 * 100 is slightly under 1999. The example converts through Decimal instead.

from dataclasses import dataclass
from decimal import Decimal

import pytest


@dataclass(frozen=True)
class NormalizedOrder:
    order_id: str
    total_cents: int
    customer_id: str


def parse_incoming_webhook(payload: dict, schema_version: str) -> NormalizedOrder:
    if schema_version == "v1_legacy":
        # Legacy: integer id, amount in decimal dollars, "client_id"
        cents = int(Decimal(str(payload["amount"])) * 100)
        return NormalizedOrder(
            order_id=str(payload["id"]),
            total_cents=cents,
            customer_id=str(payload["client_id"]),
        )
    if schema_version == "v2_current":
        # Current: string order id, integer cents, "customer_id"
        return NormalizedOrder(
            order_id=str(payload["order_id"]),
            total_cents=int(payload["total_amount_cents"]),
            customer_id=str(payload["customer_id"]),
        )
    raise ValueError(f"Unsupported schema version: {schema_version}")


@pytest.mark.parametrize(
    "payload,version,expected",
    [
        (
            {"id": 9988, "amount": 49.99, "client_id": 42},
            "v1_legacy",
            NormalizedOrder("9988", 4999, "42"),
        ),
        (
            {"id": 9989, "amount": 19.99, "client_id": 42},
            "v1_legacy",
            NormalizedOrder("9989", 1999, "42"),
        ),
        (
            {"order_id": "ord_abc123", "total_amount_cents": 5000, "customer_id": "usr_789"},
            "v2_current",
            NormalizedOrder("ord_abc123", 5000, "usr_789"),
        ),
        (
            # Unknown extra field must not break parsing
            {"order_id": "ord_abc124", "total_amount_cents": 100, "customer_id": "usr_1", "new_flag": True},
            "v2_current",
            NormalizedOrder("ord_abc124", 100, "usr_1"),
        ),
    ],
)
def test_webhook_schema_compatibility(payload, version, expected):
    assert parse_incoming_webhook(payload, version) == expected


def test_unknown_version_is_rejected_explicitly():
    with pytest.raises(ValueError):
        parse_incoming_webhook({}, "v9_future")

Operational Checklist

Persist before you interpret. Verify the signature on the raw body, store the payload, and return 2xx quickly.
Be tolerant in parsing. Ignore unknown fields and event types, accept multiple representations of IDs and timestamps, and avoid lossy coercions.
Be loud in monitoring. Validate against versioned schemas in shadow mode, track deviation rates per partner, and alert on spikes.
Isolate with a canonical model. Map every partner variant through versioned adapters and keep raw payloads for replay.
Test against recorded reality. Maintain a fixture matrix of past, current, and announced payloads in CI.
Make handlers idempotent. Deduplicate on the provider’s event or message ID, because retries and replays will deliver the same event more than once.
Talk to your partners. Subscribe to their changelogs, pin API versions where they offer it, and upgrade deliberately. Stripe, for example, ships monthly releases with no breaking changes and issues major releases with breaking changes twice a year.

Treat webhook schemas as contracts that evolve and you can keep processing events through producer changes. The unavoidable work is noticing a change quickly and mapping it correctly.

Edge-Native Webhook Ingress: Running Webhook Gateways on Cloudflare Workers and WebAssembly (Wasm)

InstaWebhook — Thu, 15 Oct 2026 03:51:58 +0000

Modern distributed systems lean on webhooks to drive event-driven workflows, sync data between SaaS platforms, trigger deployments, and process billing events. The traditional design routes webhook traffic from providers such as Stripe, GitHub, or Shopify straight into a centralized Kubernetes cluster or cloud region. That design has real security and capacity drawbacks.

A public webhook endpoint accepts requests from anyone, so it is only as safe as its signature verification. If every request, valid or not, has to travel to your origin to be rejected, bursts and floods consume ingress, API gateway, and worker capacity. That can end in connection starvation, resource exhaustion, and cascading failures.

Moving signature verification, payload checks, and tenant routing to the edge with Cloudflare Workers and, where useful, WebAssembly (Wasm) changes this. Cloudflare reports a network of more than 330 cities, and your gateway code runs inside it. Bad requests are dropped before they reach your origin, and valid events are acknowledged quickly and handed off asynchronously.

This article walks through the architecture, the code, and the limits you need to design around. Where a claim depends on platform limits or provider behavior, it follows the current documentation. Links are at the end.

1. The Architectural Shift: Moving Ingress to the Edge

In a legacy webhook architecture, an HTTP POST hits your load balancer and passes through an API gateway. It then consumes application memory to parse a body, runs signature verification in your application runtime, and finally writes state to a database.

[Legacy Flow]
Webhook Provider ──> Cloud LB ──> API Gateway ──> App Cluster (CPU/memory spent on invalid requests)

[Edge-Native Flow]
Webhook Provider ──> Cloudflare Worker (limits, verify, drop) ──> Queue ──> Origin consumers

The legacy model has several weaknesses:

Wasted compute on invalid requests. Misconfigured clients, scanners, and attackers can send garbage or replayed payloads. Your origin still spends CPU reading the body and running signature checks before it can reject them.
Shared failure domains. A webhook flood competes with your user-facing APIs for the same pods, connection pools, and database connections.
Tight coupling to provider timeouts. Providers expect a fast response. GitHub treats a delivery as failed if your server does not return a 2xx within 10 seconds, and Stripe’s guidance is to return a 2xx quickly, before running slow logic. If your origin is slow or down, retries pile on top of the load you already have.

An edge gateway acts as a stateless front door. It checks size, content type, rate, and signature, then writes accepted events to a durable queue and returns a 2xx.

One caveat: geographic latency is a weaker argument than it first appears. Webhook senders are usually cloud services calling from a handful of regions, not end users spread around the world. The main benefits are shielding your origin and acknowledging deliveries fast, not shaving round-trip time for every sender.

2. Choosing the Right Verification Primitive: WebCrypto First, Wasm When Needed

Know what your providers actually use

Signature schemes differ by provider, so check the documentation of each sender you support:

Provider	Scheme	Notes
Stripe	HMAC-SHA256 over `timestamp.payload`, in the `Stripe-Signature` header (`t=…,v1=…`)	Default timestamp tolerance is 5 minutes. The header can carry multiple `v1` signatures during secret rotation.
GitHub	HMAC-SHA256 over the raw body, in `X-Hub-Signature-256` (`sha256=<hex>`)	Payloads are capped at 25 MB. GitHub does not automatically redeliver failed deliveries.
Discord	Ed25519, via `X-Signature-Ed25519` and `X-Signature-Timestamp`	Asymmetric. You verify with Discord’s public key.
Telnyx (Webhook API v2)	Ed25519, via `telnyx-signature-ed25519` and `telnyx-timestamp`	Signs the timestamp and raw body together.
Standard Webhooks	HMAC (`v1`) or Ed25519 (`v1a`), with `webhook-id`, `webhook-timestamp`, and `webhook-signature` headers	An open specification with an asymmetric variant.

Two things follow. Most big providers use HMAC-SHA256, not Ed25519. And in every scheme above, the timestamp is part of the signed data or checked alongside it. That matters for the replay protection discussed below.

Workers’ WebCrypto already covers the common cases

The Workers runtime implements the Web Crypto API, and Cloudflare notes that WebCrypto operations are significantly faster than doing the same work in pure JavaScript. HMAC is supported, and so is Ed25519 (as the standard Ed25519 algorithm, plus a legacy NODE-ED25519 variant). For many gateways, crypto.subtle.verify is therefore the right first choice. It needs no extra build step and no extra bundle size.

Here is a Stripe-style HMAC verifier using only WebCrypto:

const enc = new TextEncoder();

function hexToBytes(hex: string): Uint8Array | null {
  if (hex.length % 2 !== 0 || !/^[0-9a-fA-F]*$/.test(hex)) return null;
  const out = new Uint8Array(hex.length / 2);
  for (let i = 0; i < out.length; i++) {
    out[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16);
  }
  return out;
}

function concat(a: Uint8Array, b: Uint8Array): Uint8Array {
  const out = new Uint8Array(a.length + b.length);
  out.set(a, 0);
  out.set(b, a.length);
  return out;
}

export async function verifyStripeStyle(
  rawBody: Uint8Array,
  header: string,
  secret: string,
  toleranceSec = 300,
): Promise<boolean> {
  const pairs = header.split(',').map((p) => p.split('=') as [string, string]);
  const t = pairs.find(([k]) => k === 't')?.[1];
  const sigs = pairs.filter(([k]) => k === 'v1').map(([, v]) => v);
  if (!t || sigs.length === 0) return false;

  const ts = Number(t);
  if (!Number.isFinite(ts) || Math.abs(Date.now() / 1000 - ts) > toleranceSec) {
    return false;
  }

  const key = await crypto.subtle.importKey(
    'raw',
    enc.encode(secret),
    { name: 'HMAC', hash: 'SHA-256' },
    false,
    ['verify'],
  );
  const signedPayload = concat(enc.encode(`${t}.`), rawBody);

  for (const sig of sigs) {
    const sigBytes = hexToBytes(sig);
    // crypto.subtle.verify performs the comparison for you
    if (sigBytes && (await crypto.subtle.verify('HMAC', key, sigBytes, signedPayload))) {
      return true;
    }
  }
  return false;
}

Note the Number.isFinite guard. Without it, a non-numeric timestamp becomes NaN, and Math.abs(NaN) > 300 is false, so the stale-timestamp check would silently pass.

Where Wasm fits

Wasm earns its place when you need something WebCrypto does not offer. Examples are a provider-specific or non-standard scheme, a Rust library you already trust and want to reuse, or heavier parsing and canonicalization logic. Compiling Rust to Wasm gives you near-native execution speed and a sandboxed linear memory, and you can reuse audited code.

Some claims about Wasm need correcting. The sandbox isolates a module from the host, but it does not prevent leaks or guarantee predictable performance. That comes from how you write the module. Rust’s ownership model helps, but Wasm does not make memory management magically safe. Treat any speed claim as something to benchmark in your own Worker.

CPU limits are also not a blocker. Workers on the Free plan allow 10 ms of CPU time per request, while Paid plans default to 30 seconds and can be raised to 5 minutes. Time spent waiting on I/O does not count as CPU time.

An Ed25519 verification module in Rust

This module uses the ed25519-dalek 2.x API. It uses verify_strict, which also rejects some malleable or weak-key edge cases that plain verify accepts. For webhooks, strict is the safer default.

use ed25519_dalek::{Signature, VerifyingKey};
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn verify_webhook_signature(
    public_key_bytes: &[u8],
    message: &[u8],
    signature_bytes: &[u8],
) -> Result<bool, JsValue> {
    let key_array: &[u8; 32] = public_key_bytes
        .try_into()
        .map_err(|_| JsValue::from_str("Invalid public key length"))?;

    let verifying_key = VerifyingKey::from_bytes(key_array)
        .map_err(|_| JsValue::from_str("Malformed public key"))?;

    let signature = Signature::from_slice(signature_bytes)
        .map_err(|_| JsValue::from_str("Invalid signature format"))?;

    Ok(verifying_key.verify_strict(message, &signature).is_ok())
}

Build it with the web target, for example wasm-pack build --target web.

Loading the module in a Worker

Cloudflare’s runtime does not let you compile Wasm from bytes at request time. Instead, Wrangler bundles any file ending in .wasm and your code imports it as a precompiled WebAssembly.Module. Cloudflare’s guidance is to instantiate it once at the top level, not on every request. With wasm-bindgen’s web target, that means calling initSync with the imported module:

import { initSync, verify_webhook_signature } from './pkg/webhook_crypto.js';
import wasmModule from './pkg/webhook_crypto_bg.wasm';

// Instantiate once per isolate, at module scope.
initSync({ module: wasmModule });

Recent wasm-bindgen versions take an object ({ module }). Older versions took the module directly, so check the signature in your generated .js file.

The full gateway handler

The handler below puts the layers in order and verifies a generic Ed25519 scheme. The signed message is the timestamp plus the raw body, as in Telnyx’s scheme (which joins them with |) and Stripe’s (.). Adapt the signed-message format to your provider.

Binding the timestamp into the signature is what makes replay protection real. If you only check a timestamp header that is not covered by the signature, an attacker can replay an old valid body with a fresh header.

import { initSync, verify_webhook_signature } from './pkg/webhook_crypto.js';
import wasmModule from './pkg/webhook_crypto_bg.wasm';

initSync({ module: wasmModule });

export interface Env {
  TENANT_KV: KVNamespace;
  WEBHOOK_QUEUE: Queue;
  WEBHOOK_LIMITER: RateLimit; // Workers Rate Limiting binding
}

interface TenantConfig {
  publicKeysHex: string[]; // multiple entries during key rotation
}

// Cloudflare Queues messages are limited to 128 KB, so keep the accepted size well below that.
const MAX_BODY_BYTES = 64 * 1024;
const MAX_SKEW_SECONDS = 300;

const enc = new TextEncoder();
const dec = new TextDecoder();

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== 'POST') {
      return new Response('Method Not Allowed', { status: 405 });
    }

    // 1. Route: /webhook/:tenantId/events
    const match = new URL(request.url).pathname.match(/^\/webhook\/([\w-]{1,64})\/events$/);
    if (!match) return new Response('Not Found', { status: 404 });
    const tenantId = match[1];

    // 2. Cheap per-location throttle (permissive and eventually consistent; see section 3)
    const ip = request.headers.get('cf-connecting-ip') ?? 'unknown';
    const { success } = await env.WEBHOOK_LIMITER.limit({ key: `${tenantId}:${ip}` });
    if (!success) return new Response('Too Many Requests', { status: 429 });

    // 3. Size and content-type guards before touching the body
    const declared = Number(request.headers.get('content-length') ?? '0');
    if (declared > MAX_BODY_BYTES) {
      return new Response('Payload Too Large', { status: 413 });
    }
    if (!request.headers.get('content-type')?.startsWith('application/json')) {
      return new Response('Unsupported Media Type', { status: 415 });
    }

    // 4. Tenant config (cached at the edge)
    const config = await getTenantConfig(env, tenantId);
    if (!config) return new Response('Unauthorized', { status: 401 });

    // 5. Required headers and timestamp window
    const signatureHex = request.headers.get('X-Webhook-Signature');
    const timestamp = request.headers.get('X-Webhook-Timestamp');
    if (!signatureHex || !timestamp) {
      return new Response('Missing security headers', { status: 400 });
    }
    const ts = Number(timestamp);
    if (!Number.isFinite(ts) || Math.abs(Date.now() / 1000 - ts) > MAX_SKEW_SECONDS) {
      return new Response('Stale or invalid timestamp', { status: 401 });
    }

    // 6. Raw bytes. Never parse JSON before verification.
    const rawBody = new Uint8Array(await request.arrayBuffer());
    if (rawBody.byteLength > MAX_BODY_BYTES) {
      return new Response('Payload Too Large', { status: 413 });
    }

    // 7. Verify against every active key (supports rotation)
    const signature = hexToBytes(signatureHex);
    if (!signature) return new Response('Invalid Signature', { status: 401 });

    const message = concat(enc.encode(`${timestamp}.`), rawBody);
    const valid = config.publicKeysHex.some((keyHex) => {
      const key = hexToBytes(keyHex);
      return key !== null && verify_webhook_signature(key, message, signature);
    });
    if (!valid) return new Response('Invalid Signature', { status: 401 });

    // 8. Durably enqueue BEFORE acknowledging
    await env.WEBHOOK_QUEUE.send({
      tenantId,
      eventId: request.headers.get('X-Webhook-Id') ?? crypto.randomUUID(),
      receivedAt: Date.now(),
      body: dec.decode(rawBody), // JSON is UTF-8 text
    });

    return new Response('Accepted', { status: 202 });
  },
};

function hexToBytes(hex: string): Uint8Array | null {
  if (hex.length % 2 !== 0 || !/^[0-9a-fA-F]*$/.test(hex)) return null;
  const out = new Uint8Array(hex.length / 2);
  for (let i = 0; i < out.length; i++) {
    out[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16);
  }
  return out;
}

function concat(a: Uint8Array, b: Uint8Array): Uint8Array {
  const out = new Uint8Array(a.length + b.length);
  out.set(a, 0);
  out.set(b, a.length);
  return out;
}

Because the Worker awaits send() before returning, a 202 means the event is in the queue. Do not use waitUntil for this step, or you could acknowledge events you later fail to enqueue.

3. CDN Boundary Protection: Layered Early Rejection

Application resilience depends on early rejection. The cheapest request to handle is one that never invokes your code, and the next cheapest is one rejected before the body is read.

A sensible order of defenses looks like this:

[Edge Security Pipeline]
1. Cloudflare network-level protections and WAF / rate limiting rules (before the Worker runs)
2. Optional provider IP allowlisting (defense in depth)
3. Worker: method, path, content-type, size guards
4. Worker: per-key rate limiting
5. Worker: tenant lookup, timestamp window, signature verification
6. Valid   -> enqueue, return 2xx
   Invalid -> 4xx at the edge, zero origin compute

Protective measures in practice

Platform and WAF rules first. Cloudflare processes traffic across its network and mitigates DDoS attacks there. A rejected request still invokes your Worker, so for the cheapest possible drop, put WAF custom rules or rate limiting rules in front of it rather than relying only on Worker code.
IP allowlisting as defense in depth. Some providers publish the IP ranges they send from. GitHub documents an allowlist approach and Stripe publishes its addresses. Signatures remain the real control, because IP ranges change and allowlists drift.
Size enforcement. Check Content-Length early, but also check the actual byte length after reading, because the header can be absent or wrong. Request body limits depend on your Cloudflare plan, and Cloudflare returns a 413 for requests that exceed them. Pick your own, much lower limit based on what your providers send. GitHub, for example, caps payloads at 25 MB, but most events are tiny.
Content-type validation. Accept only the types each provider documents.
Rate limiting with realistic expectations. The Workers Rate Limiting binding takes a limit and a period of either 10 or 60 seconds. Counters are tracked per key per Cloudflare location, and Cloudflare describes the API as permissive and eventually consistent, not an accurate accounting system. It is a good throttle against abuse, not a billing-grade quota. If you need exact, globally consistent counts, use a Durable Object, which provides strongly consistent state at the cost of routing those requests to a single instance.
Do not use KV for counters. KV is eventually consistent, and concurrent writes to the same key can overwrite each other. Writes to a single key are also rate limited.

4. Real-Time Tenant Routing with Edge KV

In a multi-tenant SaaS product, one gateway endpoint receives events for many customers, each with its own signing keys and routing rules. Querying a central database on every request adds latency and creates a new origin dependency.

Workers KV suits this read-heavy, rarely-changing data. It is a globally distributed, eventually consistent key-value store.

Know KV’s consistency model

KV writes are visible immediately in the location where they were made. In other locations they can take up to 60 seconds, or the cacheTtl you set on reads, to appear. Two other limits apply:

The minimum expirationTtl on writes is 60 seconds.
The minimum cacheTtl on reads is 60 seconds, which is also the default.

For webhooks this has a practical consequence. Key and secret changes are not instantaneous. After a rotation, some locations will keep serving the old config for a while, so your gateway must accept both old and new keys during an overlap window.

Durable Objects are not a replicated edge cache. Each object has a single active instance, which gives strong consistency but means requests may travel to wherever that instance lives. Use them for coordination such as exact counters or per-tenant serialization, not as a global config cache.

Tenant routing flow

Identify the tenant from the path (for example /webhook/:tenant_id/events) or a custom header. Prefer the path, because it is covered by your routing and WAF rules.
Look up the config in KV, including the active public keys or secrets and the destination.
Verify the signature with the tenant’s key, then enqueue with the tenant ID attached.

Implementation example

const NEGATIVE_MARKER = '__none__';

export async function getTenantConfig(
  env: Env,
  tenantId: string,
): Promise<TenantConfig | null> {
  const key = `tenant_config:${tenantId}`;

  // cacheTtl keeps hot keys in the local cache (minimum 60 seconds)
  let raw = await env.TENANT_KV.get(key, { cacheTtl: 60 });

  if (raw === null) {
    // Fall back to your control plane
    raw = await fetchTenantFromOrigin(env, tenantId);

    if (raw === null) {
      // Cache the miss briefly so unknown tenant IDs can't hammer your origin
      await env.TENANT_KV.put(key, NEGATIVE_MARKER, { expirationTtl: 60 });
      return null;
    }
    await env.TENANT_KV.put(key, raw, { expirationTtl: 300 });
  }

  if (raw === NEGATIVE_MARKER) return null;
  return JSON.parse(raw) as TenantConfig;
}

The negative cache is a real production concern. Without it, anyone can send requests for random tenant IDs and force a control-plane call every time, which recreates the origin-exhaustion problem at the edge.

Protect secrets stored at the edge

Anything you place in KV is data your Worker can read, so treat tenant signing secrets accordingly. Prefer storing public keys for asymmetric schemes, since a leaked public key cannot forge signatures. For HMAC secrets, consider encrypting values and keeping the data-encryption key in a Worker secret, or using a dedicated secrets service.

Measure your gateway’s latency in your own deployment with Workers observability instead of assuming a fixed number. Cache hits are fast, but cold reads, cache misses, and control-plane fallbacks all take longer.

5. Production Blueprint and Best Practices

Never parse JSON before verification. Read the raw request body as bytes (ArrayBuffer) or text. HMAC and Ed25519 sign bytes, not objects, and re-serializing parsed JSON changes whitespace and key order, which breaks the signature. Parse only after the check passes.
Bind the timestamp into verification. Check that the timestamp is a finite number inside your tolerance window, and make sure it is part of what is signed. Stripe’s default tolerance is 5 minutes.
Acknowledge fast, process asynchronously. Return a 2xx once the event is durably enqueued. Never run slow database writes or third-party API calls inline.
Use Cloudflare Queues as the buffer, within its limits. Delivery is at-least-once, so duplicates are possible. Messages are limited to 128 KB (including a small amount of metadata), consumer batches go up to 100 messages, and default retention is 4 days (configurable up to 14). For larger payloads, write the body to R2 and enqueue a reference. Configure a dead-letter queue so poison messages don’t retry forever.
Make consumers idempotent. Duplicates can come from the provider and from the queue itself. Stripe retries failed deliveries for up to three days in live mode, and its guidance is to expect repeat events. GitHub does not automatically redeliver failed deliveries, but it sends an X-GitHub-Delivery GUID you can use for deduplication, and you can redeliver manually or via the API. Track event IDs in a uniqueness table or similar store.
Plan for provider differences in retries. GitHub will not retry for you, so if your gateway or queue is unavailable you may need to reconcile with its delivery API afterwards. Stripe retries for days, so a long outage creates a backlog to absorb when you recover.
Handle key rotation gracefully. Support multiple active keys per tenant. Stripe may include more than one v1 signature while a secret is being rolled, so accept any valid one. Remember KV propagation delays when you roll your own keys.
Test with provider tooling. Use the provider’s test events or CLI against a staging Worker before going live. Test mode and live mode typically use different signing secrets.

Conclusion

An edge-native webhook gateway moves the cheapest and most abuse-prone work, rejecting bad requests, to Cloudflare’s network. Your origin then sees only verified, queued events. For HMAC and Ed25519, Workers’ built-in WebCrypto is often enough. Wasm is the right tool when you need custom schemes or reusable Rust code, and it loads as a precompiled module rather than being compiled at request time. KV gives you fast tenant lookups, as long as you plan for its eventual consistency and encrypt or avoid sensitive secrets. Cloudflare Queues decouples ingestion from processing, as long as your consumers are idempotent.

Next steps for taking this further:

Add per-tenant metrics and alerts for rejection reasons (bad signature, stale timestamp, rate limited).
Add a dead-letter queue consumer and a replay tool for failed events.
Add R2 offloading for payloads that exceed the Queues message limit.
Run load tests that include invalid-signature floods to confirm origin load stays flat.

References

Cloudflare Workers limits: https://developers.cloudflare.com/workers/platform/limits
Cloudflare Workers: Wasm in JavaScript: https://developers.cloudflare.com/workers/runtime-apis/webassembly/javascript
Cloudflare Workers Web Crypto: https://developers.cloudflare.com/workers/runtime-apis/web-crypto/
Cloudflare Workers KV, writing key-value pairs: https://developers.cloudflare.com/kv/api/write-key-value-pairs
Cloudflare Workers Rate Limiting binding: https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/
Cloudflare Queues documentation: https://developers.cloudflare.com/queues/
Stripe webhooks: https://stripe.com/docs/webhooks
GitHub webhooks best practices: https://docs.github.com/webhooks/using-webhooks/best-practices-for-using-webhooks
GitHub handling failed webhook deliveries: https://docs.github.com/webhooks/using-webhooks/handling-failed-webhook-deliveries
GitHub webhook events and payloads: https://docs.github.com/en/webhooks/webhook-events-and-payloads

Originally published at https://instawebhook.com/blog/edge-native-webhook-ingress-running-webhook-gateways-on-cloudflare-workers-and-w

HTTP/3 and QUIC for Webhook Delivery: What It Fixes, What It Doesn’t, and How to Adopt It

InstaWebhook — Wed, 14 Oct 2026 02:36:11 +0000

Webhooks are the standard way services push events to each other: payment notifications, repository events, order updates. A webhook sender has an unusual job. It makes outbound HTTPS requests to thousands of independent endpoints it doesn’t control. Many are slow, some are on lossy links, and most receive only occasional bursts of traffic.

HTTP/3, built on the QUIC transport, changes several things that matter for this workload. It is not a universal speed-up, though. This article covers where HTTP/3 helps webhook delivery, where it doesn’t, and what it takes to adopt it safely.

1. Where Webhook Delivery Hits TCP’s Limits

Handshake cost for sporadic endpoints

A new HTTPS connection over TCP with TLS 1.3 needs one round trip for the TCP handshake and one for the TLS handshake. The first request byte therefore goes out roughly two round trips after the dial begins. A QUIC connection combines the transport and TLS 1.3 handshakes (RFC 9001) and needs one round trip before the client can send application data.

For a sender that keeps long-lived, busy connections to a few endpoints, this saving is amortized away. For a sender that delivers one event every few minutes to each of thousands of subscribers, many deliveries pay the full handshake cost, so the saving is real.

Ephemeral port and `TIME_WAIT` pressure

Each outbound TCP connection is identified by its 4-tuple (source IP, source port, destination IP, destination port). A sender with a fixed set of egress IPs and a busy destination can run short of source ports. On Linux, the default ephemeral range (net.ipv4.ip_local_port_range) is 32768–60999, which gives about 28,000 ports per source IP and destination tuple. Connections that close actively also linger in TIME_WAIT, which is 60 seconds on Linux, so the usable number is lower in practice.

QUIC runs over UDP, and a single UDP socket can carry many QUIC connections, which are distinguished by connection IDs rather than by the 4-tuple alone. A QUIC client therefore doesn’t need a separate local port per connection. There is also no TIME_WAIT state in the TCP sense. This relieves port pressure, though it does not remove other limits such as file descriptors, CPU, or NAT table capacity.

Head-of-line blocking

Head-of-line (HoL) blocking has several distinct forms:

HTTP/1.1. One request occupies a connection at a time. Pipelining exists in the spec but is rarely used in practice. A slow response ties up the connection, so senders open many parallel connections.
HTTP/2 over TCP. Many streams share one TCP connection, which removes request-level blocking. TCP, however, delivers a single ordered byte stream. When one packet is lost, the kernel withholds all later data from the application until the retransmission arrives, so every stream on that connection stalls, including streams whose packets all arrived.
HTTP/3 over QUIC. Streams are independent at the transport layer. Loss on one stream delays only that stream, and other streams can keep delivering data.

QUIC does not eliminate HoL blocking entirely. Data within a single stream is still delivered in order, and an overloaded receiving application can still queue work.

2. How QUIC Works

QUIC is specified in RFC 9000 (transport), RFC 9001 (TLS integration) and RFC 9002 (loss detection and congestion control). HTTP/3 is specified in RFC 9114 and its header compression, QPACK, in RFC 9204. All were published in 2021–2022. The protocol grew out of work Google began around 2012, with Jim Roskind as its original designer, and was then standardized by the IETF with significant changes.

+-------------------------------------------------------+
| HTTP/3 (Semantics: Headers, Methods, QPACK)           |
+-------------------------------------------------------+
| QUIC (Streams, Loss Recovery, Congestion Control,     |
|       TLS 1.3 handshake and encryption)               |
+-------------------------------------------------------+
| UDP (Datagrams)                                       |
+-------------------------------------------------------+
| IP                                                    |
+-------------------------------------------------------+

QUIC implementations live in user space, which makes them easier to update than kernel TCP stacks. They also have to build features such as pacing, batching, and socket buffer management themselves.

0-RTT: powerful, but dangerous for webhooks

QUIC supports 0-RTT resumption: a client that has previously connected to a server can send application data in its first flight. That sounds ideal for webhooks, but 0-RTT data is not protected against replay. An attacker who captures the first flight can resend it, and the server may process it twice.

Because of this, HTTP/3 clients and libraries restrict 0-RTT to idempotent requests. The Go library quic-go, for example, exposes 0-RTT only for GET and HEAD requests and documents that 0-RTT offers no replay protection. A webhook is a POST that triggers side effects, so a dispatcher should not send it in 0-RTT unless the receiver has made its handler idempotent and explicitly accepts early data. Treat the 1-RTT handshake as the realistic baseline.

Connection migration: narrower than it sounds

QUIC connections are identified by connection IDs rather than by the IP and port 4-tuple. A client can therefore continue a connection after its address changes, for example after a NAT rebinding or a switch between network interfaces. Two limits apply:

Migration is initiated by the client. A server cannot migrate an established connection to a new address on its own.
A dispatcher pod that is evicted and rescheduled with a new IP does not carry its connections with it. The process is gone, and so is its connection state. New connections have to be established, as with TCP.

For webhook dispatchers, migration mainly helps with NAT rebinding and path changes. It does not help with pod churn, so don’t plan around it for that.

3. Designing a Dispatcher for HTTP/3

From connection pools to multiplexed sessions

HTTP/2 and HTTP/3 both let one connection carry many concurrent requests. A dispatcher can therefore keep one or a few long-lived connections per subscriber host rather than a deep pool. The peer limits the number of concurrent streams through the SETTINGS/transport parameters it advertises, and dispatchers should respect that limit.

Prioritization

Prioritization in HTTP/3 uses the Extensible Prioritization scheme from RFC 9218, which signals urgency and incrementality hints through the Priority header field. Servers treat these signals as hints and are not required to honor them. Don’t rely on them to guarantee that a critical event overtakes a routine one. Handle that ordering in your own queues before events reach the transport.

Loss recovery and congestion control

QUIC’s loss recovery works per packet and retransmits the affected stream data while other streams continue. RFC 9002 describes a NewReno-style baseline, and implementations commonly offer Cubic and, in some cases, BBR variants. The available algorithms depend on the library, so check the one you use.

Discovery and fallback

A dispatcher needs to know whether an endpoint supports HTTP/3. There are two standard mechanisms:

Alt-Svc response header (RFC 7838): an HTTP/1.1 or HTTP/2 response advertises an HTTP/3 alternative on a UDP port, usually 443.
DNS HTTPS records (RFC 9460), which can advertise h3 support before the first connection.

Whichever you use, UDP can be blocked or rate-limited by a firewall or network path, so you need a fallback to TCP. A practical design:

Race HTTP/3 and TCP when no preference is cached, or try HTTP/3 with a short timeout.
Cache the outcome per endpoint with a TTL, and retry HTTP/3 after the TTL expires.
Treat the fallback as a normal path, not an error.

4. Reference Architecture

[Event Source / Queue] ---> [Dispatcher Workers]
                                      |
                              [Protocol Selector + Cache]
                                      |
       +------------------------------+------------------------------+
       | HTTP/3 (QUIC/UDP)            | HTTP/3 (QUIC/UDP)            | HTTP/2 or 1.1 (TCP)
       v                              v                              v
[Subscriber A]                [Subscriber B]                [Subscriber C]

Durable queue. Events are buffered in a system such as Kafka, SQS, or RabbitMQ.
Workers. They serialize the payload and sign it. Providers differ here: GitHub sends an HMAC-SHA256 signature in X-Hub-Signature-256, Stripe uses a Stripe-Signature header, and Shopify uses X-Shopify-Hmac-Sha256. Subscribers need to know which scheme you use.
Protocol selector. It holds the per-endpoint cache described above.
Retries. Use exponential backoff with jitter, and give every event a stable ID so receivers can deduplicate.

Go sketch

This example uses quic-go’s HTTP/3 transport and sends a real request body. The transport is created once and reused, because creating one per request throws away connection reuse. (In recent quic-go releases http3.RoundTripper was renamed to http3.Transport; check the version you depend on.)

package main

import (
    "bytes"
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "time"

    "github.com/quic-go/quic-go/http3"
)

var h3Transport = &http3.Transport{}

var h3Client = &http.Client{
    Transport: h3Transport,
    Timeout:   10 * time.Second,
}

// Standard TCP client (HTTP/2 or HTTP/1.1) used as a fallback.
var tcpClient = &http.Client{Timeout: 10 * time.Second}

func newRequest(ctx context.Context, url string, payload []byte, sig string) (*http.Request, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(payload))
    if err != nil {
        return nil, err
    }
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("X-Webhook-Signature", sig)
    req.Header.Set("User-Agent", "ExampleWebhookDispatcher/1.0")
    return req, nil
}

func send(client *http.Client, req *http.Request) error {
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    _, _ = io.Copy(io.Discard, resp.Body)

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("subscriber returned status %d", resp.StatusCode)
    }
    return nil
}

// dispatch tries HTTP/3 first (if the endpoint is known to support it)
// and falls back to TCP on transport-level failure.
func dispatch(ctx context.Context, url string, payload []byte, sig string, supportsH3 bool) error {
    if supportsH3 {
        h3Ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
        defer cancel()
        req, err := newRequest(h3Ctx, url, payload, sig)
        if err != nil {
            return err
        }
        if err := send(h3Client, req); err == nil {
            return nil
        } else if !isTransportError(err) {
            return err // e.g. a non-2xx response: do not resend over TCP blindly
        }
        // transport failure: record it in the protocol cache, then fall through
    }

    req, err := newRequest(ctx, url, payload, sig)
    if err != nil {
        return err
    }
    return send(tcpClient, req)
}

func isTransportError(err error) bool {
    var ne interface{ Timeout() bool }
    return errors.As(err, &ne) || errors.Is(err, context.DeadlineExceeded)
}

Note the retry caveat: a timeout can mean the receiver processed the request but the response was lost. Falling back to another transport can therefore deliver the event twice. Include an event ID in the payload and require receivers to be idempotent.

5. Operational Challenges

UDP blocking and throttling

Some enterprise firewalls and cloud security groups block or rate-limit UDP port 443. Make sure your egress rules allow it, and make the TCP fallback automatic and cached.

CPU cost and tuning

QUIC does packet processing and encryption in user space, and implementations are generally more CPU-intensive per byte than kernel TCP with offloads, though the gap varies by library and hardware. Practical steps:

Load-test your actual library and payload sizes rather than assuming parity with TCP.
Raise the UDP socket buffer limits (net.core.rmem_max and net.core.wmem_max on Linux). The quic-go documentation recommends this and logs a warning when the buffers are small.
Watch for kernel and NIC features, such as UDP segmentation offload, that your library can exploit.

Amplification and abuse

QUIC’s address validation and its anti-amplification limit (a server may send at most three times the data it has received from an unvalidated address) mainly protect servers. A dispatcher acting as a client is less exposed, but the usual outbound controls still apply: egress rate limits, SSRF protections on subscriber-supplied URLs, and sensible timeouts.

Observability

QUIC encrypts most transport metadata that TCP exposes, so passive tools such as tcpdump with TCP flag analysis show much less. Instrument the application instead:

handshake duration and failure reasons
fraction of deliveries by protocol (HTTP/3 vs. fallback)
per-endpoint latency and error rates
stream resets and connection closures

Many QUIC libraries can emit qlog traces for deep debugging.

6. What the Evidence Says

Public measurements show the general pattern rather than webhook-specific numbers, so treat any exact figure with caution, including the ones below.

Adoption on the receiving side is substantial but uneven. W3Techs reported HTTP/3 in use on about 39.7% of websites in July 2026. Much of that is concentrated behind CDNs and reverse proxies; for sites using Cloudflare, W3Techs reported 88.8% in March 2026.
Actual request share is lower. One analysis of Q1 2026 Cloudflare Radar data put HTTP/3 at about 21% of requests, compared with about 51% for HTTP/2 and about 28% for HTTP/1.x, with bot and server-to-server traffic a large contributor to the gap.
Benefits are largest on high-latency and lossy paths. Field measurements consistently find larger gains over long distances and on mobile networks than on clean, low-latency datacenter links.

For a dispatcher, this means two things. First, only some of your subscribers can receive HTTP/3 today, so fallback is a day-one requirement. Second, the gain for any individual endpoint depends on its network path, so measure your own traffic: compare latency percentiles, failure rates, and CPU use between the two protocols for the same endpoints before committing.

7. Should You Adopt It?

HTTP/3 is a good fit for a webhook dispatcher when:

many endpoints are contacted infrequently, so handshake cost matters,
you deliver across long or lossy internet paths,
you are hitting ephemeral port limits or connection churn on TCP, and
your subscribers sit behind CDNs or proxies that already speak HTTP/3.

It offers little when your traffic goes to a handful of well-connected endpoints over warm, long-lived HTTP/2 connections.

A sensible rollout:

Add HTTP/3 as an opt-in transport behind a feature flag, with cached fallback to TCP.
Start with endpoints that advertise Alt-Svc or DNS h3 support.
Don’t use 0-RTT for POST deliveries.
Compare latency, success rate, and CPU against the TCP baseline per endpoint.
Expand gradually as the data supports it.

References and Further Reading

RFC 9000: QUIC: A UDP-Based Multiplexed and Secure Transport, IETF.
RFC 9001: Using TLS to Secure QUIC, IETF.
RFC 9002: QUIC Loss Detection and Congestion Control, IETF.
RFC 9114: HTTP/3, IETF.
RFC 9204: QPACK: Field Compression for HTTP/3, IETF.
RFC 9218: Extensible Prioritization Scheme for HTTP, IETF.
RFC 7838: HTTP Alternative Services, IETF.
RFC 9460: Service Binding and Parameter Specification via the DNS (SVCB and HTTPS Resource Records), IETF.
W3Techs, Usage statistics of HTTP/3 (w3techs.com).
Cloudflare Radar, HTTP protocol version trends (radar.cloudflare.com).
quic-go documentation (quic-go.net), including its notes on 0-RTT and UDP buffer sizes.
Ilya Grigorik, High Performance Browser Networking (O’Reilly, 2013). Pre-dates HTTP/3, but remains a good foundation for TCP and TLS performance.

Originally published at https://instawebhook.com/blog/http-3-and-quic-for-webhook-delivery-what-it-fixes-what-it-doesn-t-and-how-to-ad

The Multi-Tenant Neighbor Problem: Why Traditional API Rate Limiters Fail Under Webhook-Heavy Bursts

InstaWebhook — Tue, 13 Oct 2026 02:20:58 +0000

Modern software architectures rely heavily on webhooks. Whether your application processes payments via Stripe, synchronizes inventory with Shopify, or reacts to repository events from GitHub, webhooks are how many real-time, event-driven systems learn that something happened. As applications scale, engineering teams run into an architectural problem known as the multi-tenant neighbor problem, often called the noisy neighbor problem.

Consider a hypothetical scenario. Your SaaS platform serves thousands of businesses, each a tenant with its own data and API credentials. One of your largest customers launches a flash sale on Shopify, and your ingestion endpoint suddenly receives tens of thousands of webhook requests per second. For illustration, this article uses 50,000 requests per second (RPS). That figure is an assumption, not a published Shopify number. It is a plausible order of magnitude for a platform that aggregates events from many merchants, and a single merchant would rarely produce it alone.

If your system uses rate-limiting strategies designed for ordinary user traffic, it may crash, bottleneck, or starve unrelated customers.

This guide explains why traditional rate limiters struggle with webhook spikes, how the neighbor problem plays out in a real ingestion pipeline, and which patterns help you build a resilient one. Provider behavior described here (timeouts, retries) comes from the providers’ own documentation, linked at the end.

1. Understanding the Multi-Tenant Neighbor Problem in Webhooks

In a shared multi-tenant architecture, CPU, memory, database connection pools, and network capacity are pooled across customers. This improves hardware efficiency and lowers cost, but it exposes every tenant to the behavior of the loudest one.

How Webhook Traffic Differs from User Traffic

Ordinary user traffic is spread across many endpoints (login, dashboard, search) and paced by human interaction. Webhook traffic behaves differently:

Event-driven bursts. Providers push data when events occur. Bulk operations, such as a flash sale, a mass product update, or a batch import, can generate thousands of events in a very short window. For scale, Shopify reported that during Black Friday–Cyber Monday 2025 its API requests peaked at 31.8 million per minute and merchant sales peaked at $5.1 million per minute. Those are platform-wide figures rather than webhook counts, but they show how sharply commerce traffic can spike.
Tight response deadlines. Providers expect a fast acknowledgment. Shopify fails a delivery if your app doesn’t respond within five seconds, and GitHub gives your server 10 seconds before it terminates the connection and records a failure. Stripe’s documentation asks endpoints to return a 2xx status code quickly, before doing complex work.
Provider-side retries. Retry behavior varies by provider, and it can amplify load on a struggling receiver:
- Stripe, in live mode, attempts delivery for up to three days with exponential backoff. In sandbox mode it retries three times over a few hours.
- Shopify retries failed deliveries up to eight times over four hours. If failures persist, it removes the webhook subscription.
- GitHub does not automatically redeliver failed deliveries. You have to redeliver them manually or with a script.
Per-request work. Webhook handlers typically verify a cryptographic signature, resolve which tenant the event belongs to, and persist or enqueue the payload.

When one tenant generates a traffic surge, it competes with every other tenant for the same shared resources. The others see higher latency, timeouts, or dropped events even though their own traffic hasn’t changed.

2. Why Traditional API Rate Limiters Fail

Common algorithms such as token bucket, leaky bucket, and fixed or sliding window counters, often backed by Redis, work well for protecting CRUD APIs from scrapers and runaway client scripts. They break down for webhook ingestion for several reasons.

A. IP-Based Limiting Doesn’t Identify Tenants

Webhooks from a given provider come from that provider’s infrastructure, not from your tenant’s. Every tenant’s events arrive from the same pool of source addresses. Limiting by IP therefore either throttles the provider as a whole, which punishes every tenant using that integration, or fails to single out the one tenant causing the burst. IP allowlists are still useful as an additional security layer where providers publish ranges (GitHub, for example, exposes its addresses through its GET /meta endpoint), but IP is the wrong key for fairness between tenants.

B. Global or Per-Route Limits Cause Cross-Tenant Starvation

Suppose you cap /api/webhooks/shopify at 10,000 RPS. If Tenant A generates 12,000 RPS, the limiter has no concept of who is responsible. Tenant B’s legitimate events are throttled or rejected alongside A’s, which causes missed order updates and out-of-sync data.

C. A Centralized Counter Becomes a Hot Spot

Distributed rate limiters commonly rely on Redis atomic operations (INCR, EXPIRE, or Lua scripts). Redis executes commands on a single main thread, so the problem is not lock contention. It is that one hot key lives on one shard and every limiter check is a network round trip to it. At tens of thousands of checks per second, the limiter’s own latency, network bandwidth, and availability become part of your request path. If the limiter is unreachable, you have to decide whether to fail open (and risk overload) or fail closed (and reject legitimate traffic).

D. Rejecting a Webhook Doesn’t Make It Go Away

Traditional limiters are binary: allow the request or reject it with 429 Too Many Requests. Webhook senders aren’t ordinary API clients. Shopify counts any non-2xx response as a failed delivery and retries it, and it may eventually remove the subscription. Stripe retries for days. A rejected event is therefore either redelivered later, adding to future load, or lost if the provider gives up or never retries (as with GitHub). Neither outcome is what you want from a load-protection mechanism.

Providers’ documentation does not state that they honor a Retry-After header on webhook deliveries, so don’t design your protection strategy around it. The more reliable approach is to acknowledge quickly and absorb the burst in a durable buffer.

3. Anatomy of a Webhook Ingestion Failure

Here is how an unprotected, synchronous pipeline typically fails under a sudden burst from one tenant:

Edge saturation. The load balancer or gateway (NGINX, Envoy, a cloud API gateway) handles a flood of new connections. TLS handshakes and connection handling consume CPU.
Per-request overhead. Each request needs signature verification and a lookup of the tenant’s signing secret. The HMAC computation itself is cheap relative to TLS and JSON parsing, but if the secret comes from a database on every request, that lookup becomes the bottleneck.
Connection pool exhaustion. Each handler tries to read tenant metadata or write the event to a relational database. Connection pools (for example HikariCP, or PgBouncer in front of PostgreSQL) fill up, threads block waiting for connections, and timeouts cascade.
Memory pressure. Requests pile up in application memory while waiting for workers. Eventually the process or container exceeds its memory limit and the kernel’s OOM killer, or the orchestrator, restarts it. A restarted instance that immediately faces the same backlog can fall into a crash loop.
Provider retries amplify the damage. Once responses exceed the provider’s timeout (five seconds for Shopify, ten for GitHub), the provider records failures. Where it retries, as Stripe and Shopify do, it adds more load while you are already down.

4. Modern Architectural Patterns to Solve the Neighbor Problem

The core shift is from synchronous processing and flat rate limits to asynchronous ingestion, per-tenant isolation, and tenant-keyed quotas.

Pattern 1: Tenant-Aware Rate Limiting with Dynamic Quotas

Key limits by tenant rather than by route or IP. Depending on the provider, the tenant can be identified from the URL path (for example /webhooks/shopify/{tenant_id}), a subdomain, or a header. Shopify, for instance, includes the originating shop’s domain in a request header.

One caution: the tenant identifier is unauthenticated until the signature has been verified. Anyone who can reach your endpoint can forge a header or URL. Use the identifier to look up the signing secret and verify the signature before charging the request to that tenant’s quota or trusting the payload. Otherwise an attacker can exhaust a victim’s quota with forged requests.

Hierarchical limiting. Use two tiers:
1. A global safety cap that protects the infrastructure as a whole.
2. A per-tenant quota, set by plan tier or historical volume, so Tenant A cannot consume capacity reserved for Tenant B.
Shed after acknowledging, not before. Because rejected webhooks come back as retries (or are lost), apply per-tenant limits primarily to how fast you process events from the buffer, not to whether you accept them.

Pattern 2: Decouple Ingestion from Processing

Don’t process webhook payloads synchronously. Both GitHub and Shopify advise responding quickly and handling the work in the background, and GitHub specifically suggests a queue for this. The ingestion endpoint should do the minimum:

Read the raw request body.
Verify the signature against the raw bytes.
Write the event to a durable buffer.
Return a 2xx response.

Durable buffers. Apache Kafka, Redpanda, Amazon Kinesis, or a managed queue such as Amazon SQS all work. Choose based on throughput, ordering, and operational needs.
Partitioning by tenant. In Kafka-style logs, keying messages by tenant ID sends all of a tenant’s events to the same partition, which preserves per-tenant ordering. Ordering is guaranteed only within a partition, and several tenants can share one. A very large tenant can also create a hot partition, so consider sub-keying (for example by entity ID) for heavy tenants if you don’t need strict tenant-wide ordering.
Idempotency. Delivery is at-least-once, so consumers must tolerate duplicates. Use the provider’s delivery identifier, such as X-Shopify-Webhook-Id for Shopify or X-GitHub-Delivery for GitHub, as an idempotency key. GitHub keeps the same X-GitHub-Delivery value on manual redeliveries.

[Webhook Provider]
       │  burst of requests
       ▼
[Edge / API Gateway]
       │  verify signature, enqueue, return 2xx
       ▼
[Durable Log / Queue]  (partitioned by tenant)
       │
       ▼
[Per-Tenant Rate-Limited Worker Pools]
       │
       ▼
[Tenant Data Stores]

Pattern 3: Prioritization and Graceful Degradation

When queues or worker pools approach saturation, degrade deliberately instead of failing everywhere.

Priority lanes. Route business-critical events (for example, payment confirmations) to a high-priority queue and lower-value events (analytics, profile updates) to a lower-priority one that can lag during congestion.
Backlog-aware processing. Slow or pause consumption for the offending tenant while others continue, and alert on queue depth and consumer lag rather than only on request rates.
Treat non-2xx as a last resort. If you must reject traffic, such as when the buffer itself is unavailable, remember that Shopify caps retries at eight and may remove the subscription, and that GitHub won’t retry at all. Build a reconciliation path: GitHub lets you list and redeliver failed deliveries through its API, and Shopify recommends re-importing data from the outage window after downtime.

Pattern 4: Resource Isolation (Bulkheading)

Bulkheads, borrowed from ship design and resilience engineering, partition resources so that failure or overload in one compartment doesn’t sink everything.

Dedicated worker pools or queues for high-volume enterprise tenants, with smaller shared pools for everyone else.
Per-tenant concurrency caps so one tenant cannot occupy all database connections.
Data partitioning. Shard or partition tenant data so heavy write load from one tenant is confined to its own partitions or databases where practical.

5. A Token Bucket Rate Limiter in Redis (Lua)

To avoid race conditions and extra round trips when enforcing per-tenant limits, run the token bucket logic as an atomic Lua script. Redis executes a script as a single unit, so concurrent callers can’t interleave between the read and the write.

This version reads the clock from Redis itself (TIME) so that clock skew between application servers cannot corrupt the bucket. It requires Redis 5.0 or later, where scripts that call non-deterministic commands are replicated by their effects. It also returns a suggested wait time you can use to pace your own consumers.

-- Tenant token bucket
-- KEYS[1]: bucket key, e.g. ratelimit:tenant:12345
-- ARGV[1]: bucket capacity (maximum burst)
-- ARGV[2]: refill rate (tokens per second)
-- ARGV[3]: tokens requested (usually 1)

local t = redis.call('TIME')
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested = tonumber(ARGV[3])

local bucket = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(bucket[1])
local ts = tonumber(bucket[2])

if tokens == nil or ts == nil then
  tokens = capacity
  ts = now
end

-- Refill based on elapsed time
tokens = math.min(capacity, tokens + math.max(0, now - ts) * refill_rate)

local allowed = 0
local retry_after = 0

if tokens >= requested then
  tokens = tokens - requested
  allowed = 1
else
  retry_after = (requested - tokens) / refill_rate
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
-- Let idle buckets expire (twice the time needed to fully refill)
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / refill_rate) * 2)

return {allowed, tostring(retry_after)}

Two practical notes:

Where to apply it. Use this limiter to meter processing per tenant, such as consumers pulling from the queue, rather than as a gate that rejects webhook deliveries at the edge (see Pattern 3).
Limiter failure mode. Decide up front what happens if Redis is slow or unavailable. For ingestion, failing open (accept and buffer) is usually safer than failing closed, because the queue provides the real protection. Keep local, in-process fallback limits as a backstop.

6. Summary and Best Practices Checklist

Protecting your ingestion pipeline from the multi-tenant neighbor problem means moving from reactive scaling to proactive isolation:

Acknowledge fast, process later. Verify the signature, write to a durable buffer, and return 2xx well inside the provider’s deadline (five seconds for Shopify, ten for GitHub).
Key quotas by tenant, not by IP or route. Authenticate the request before charging a tenant’s quota.
Verify signatures on the raw body, and cache signing secrets so verification doesn’t hit your database on every request.
Meter consumption, not acceptance. Apply per-tenant rate limits and concurrency caps to workers reading from the queue.
Design for duplicates. Use delivery IDs for idempotency, since providers retry and consumers may reprocess.
Isolate with bulkheads. Give heavy tenants dedicated pools, and cap per-tenant database concurrency.
Plan for reconciliation. Know how to recover missed events: GitHub supports redelivery through its API, and Shopify recommends re-importing data after an outage.
Monitor the right signals. Watch queue depth, consumer lag, and per-tenant processing time, along with request rate.

A system built this way absorbs bursts in the buffer, throttles the noisy tenant at the processing stage, and keeps its neighbors online.

Sources

Shopify, Troubleshoot webhooks: five-second response limit, retry behavior, and subscription removal.
Stripe, Receive Stripe events in your webhook endpoint: retry behavior in live and sandbox modes.
GitHub Docs, Best practices for using webhooks: 10-second response window, queue recommendation, delivery header.
GitHub Docs, Handling failed webhook deliveries: no automatic redelivery.
Shopify, BFCM 2025 data: peak sales and API request volumes.

Originally published at https://instawebhook.com/blog/the-multi-tenant-neighbor-problem-why-traditional-api-rate-limiters-fail-under-w

Webhook Payload Encryption at Rest: Application-Layer Cryptography for Sensitive Webhook Pipelines

InstaWebhook — Mon, 12 Oct 2026 01:58:16 +0000

Webhooks synchronize data between microservices, trigger automated workflows, and deliver events from third-party SaaS providers. More and more of them carry regulated data: payment card numbers, health records, and personal data covered by GDPR and CCPA. That makes every system a webhook touches after your ingress point part of your security perimeter.

A typical pipeline terminates HTTPS at a gateway, pushes the JSON body into a queue such as Redis or Amazon SQS, and archives it in a relational database like PostgreSQL. Every hop after TLS termination usually holds plaintext. An attacker who reads your queue, restores a database backup, or finds an unredacted log line gets the full payloads.

This guide describes a practical blueprint for application-layer (envelope) encryption of webhook payloads, so that plaintext exists only at the ingress service and at the workers that actually need it. It covers envelope encryption, a per-tenant key hierarchy, and how to keep events searchable in PostgreSQL. It also covers retries and idempotency, and what “cryptographic erasure” can and cannot promise.

A note on terminology: the sender’s TLS session ends at your edge, so this is not end-to-end encryption between you and the webhook provider. It is end-to-end inside your own infrastructure, from the ingress service to the worker that processes the event.

1. The Threat Model of Traditional Webhook Pipelines

A standard webhook carrying sensitive data goes through these stages:

Ingress: A provider (Stripe, Salesforce, a healthcare partner) sends an HTTP POST with a JSON body to your public endpoint.
Termination: Your load balancer or API gateway terminates TLS, and the body exists as plaintext bytes in memory. Your receiver also needs those raw bytes to verify the provider’s signature (see Section 6).
Queueing: The receiver pushes the body to a broker (Redis Streams, RabbitMQ, SQS) for asynchronous processing. In many setups, the queue holds raw JSON.
Worker processing: A worker consumes the message, runs business logic, and writes the event to an audit table or event store, often in a PostgreSQL JSONB column. Without payload-level encryption, those rows hold the sensitive values indefinitely, and they propagate into replicas and backups.
Logging: Debug statements and exception handlers print request bodies into Datadog, Elasticsearch, CloudWatch, or similar systems.

Platform-level encryption helps but does not close these gaps. SQS supports server-side encryption, and managed databases encrypt volumes. That protects against stolen disks and storage-layer access. It does not protect against any principal that is authorized to read the queue or query the database, such as an over-privileged service role, a compromised application credential, or a developer with production read access. Application-layer encryption narrows that exposure to principals that are also allowed to use the key.

Compliance Implications

Encryption is not the only control these frameworks require, but each one points toward protecting stored data in this way.

PCI DSS (v4.0.1): Requirement 3.5.1 says the primary account number (PAN) must be rendered unreadable anywhere it is stored. Acceptable methods are one-way hashes, truncation, index tokens, or strong cryptography with key-management processes. The requirement’s applicability notes cover primary storage and also backups, audit logs, and exception or troubleshooting logs. In PCI DSS v3.2.1 this was Requirement 3.4. Requirement 3.5.1.2 also limits disk-level or partition-level encryption as a way to render PAN unreadable. It is acceptable only on removable media, otherwise another mechanism meeting 3.5.1 is needed. This is a strong argument for field-level or payload-level encryption of non-removable storage. Separately, Requirement 3.3.1 prohibits storing sensitive authentication data after authorization, even if encrypted. The best design is to keep PANs out of webhook payloads entirely by using tokens.
HIPAA Security Rule: 45 CFR § 164.312(a)(2)(iv) covers encryption and decryption of ePHI. Today it is an “addressable” implementation specification: you must implement it or document an equivalent alternative. HHS published a proposed rule on January 6, 2025 that would make encryption of ePHI at rest and in transit mandatory with limited exceptions. As of mid-2026 reporting, it had not been finalized, and sources differ on when it might be, with some pointing to a 2027 target. Encryption also matters for breach response. Under HHS guidance, ePHI encrypted consistent with NIST guidance, with the keys not compromised, is generally not considered “unsecured PHI” for breach notification purposes.
GDPR Article 32: It lists pseudonymisation and encryption of personal data as examples of appropriate technical measures, to be chosen according to risk, cost, and context. It does not mandate encryption in every case. Article 17 provides the right to erasure.
CCPA/CPRA: California’s consumer right is a right to delete personal information, not a literal “right to be forgotten.” The statute’s breach-related private right of action (Civil Code § 1798.150) applies to nonencrypted and nonredacted personal information, which gives encryption direct legal value.

2. Core Pattern: Envelope Encryption

To protect payloads across queues, caches, and general-purpose databases, you encrypt before the data touches any of them. The standard technique for this is envelope encryption, which uses two tiers of keys:

Data Encryption Key (DEK): A random symmetric key generated for each webhook payload. It encrypts the payload with an authenticated mode such as AES-GCM or ChaCha20-Poly1305.
Key Encryption Key (KEK): A longer-lived key that never encrypts payloads directly. It only wraps (encrypts) DEKs. In production the root KEK lives in a KMS or HSM, such as AWS KMS, Google Cloud KMS, Azure Key Vault, or HashiCorp Vault’s transit engine.

Incoming webhook body (plaintext bytes)
        │
        ▼
[ Generate random DEK ] ──encrypts──> [ Ciphertext + nonce + auth tag ]
        │                                          │
        ▼                                          │
[ Wrap DEK with KEK ]                              │
        │                                          │
        ▼                                          ▼
   [ Wrapped DEK ]  ───────── stored together ─────────>  queue / database

Why an AEAD mode?

Use an Authenticated Encryption with Associated Data (AEAD) cipher. AES-256-GCM (specified in NIST SP 800-38D) and ChaCha20-Poly1305 (RFC 8439) provide confidentiality and integrity. If anyone modifies the ciphertext, nonce, or associated data, decryption fails with an authentication error instead of returning altered plaintext. Alert on those failures.

Two details are often missed:

Bind context with associated data. Pass values such as the tenant ID, event ID, and envelope version as AAD. They are not encrypted but are authenticated, so an attacker cannot move a valid ciphertext into another tenant’s record or swap it between events.
Respect nonce limits. GCM is catastrophically weak if a nonce repeats under the same key. NIST SP 800-38D requires that the probability of reusing an IV under one key stay at or below 2^-32. With random 96-bit IVs this is commonly interpreted as a cap of about 2^32 (roughly 4.3 billion) invocations per key. A fresh DEK per message makes the payload encryption safe by construction. But if you wrap every DEK under one long-lived tenant key with random nonces, you are accumulating invocations under that key. Version your keys and rotate them well before the limit, or use a KMS call for wrapping.

3. Multi-Tenant Key Hierarchy

In a multi-tenant system, encrypting everyone’s data under one global key creates a large blast radius. It also makes per-tenant deletion impossible without re-encrypting everything. You want each tenant’s data to be separable cryptographically.

One KMS key per tenant is the simplest model and works well for tens or hundreds of tenants. It gets expensive and operationally heavy at thousands. AWS KMS cryptographic operations also share a per-account, per-region request quota that varies by region (the AWS Security Blog cites a 10,000 requests-per-second default in several major regions). The quota is adjustable, and AWS recommends reducing KMS calls with data key caching, for example in the AWS Encryption SDK.

A scalable alternative is a three-level hierarchy:

Root key (in KMS/HSM): Never leaves the KMS boundary. It is used only to wrap and unwrap tenant keys.
Tenant key (random, 256-bit): Generated per tenant, for example with KMS GenerateDataKey. You store only its KMS-wrapped form in a tenant_keys table, and workers unwrap it once and cache it in memory with a TTL.
Message DEK (random, 256-bit): Generated per webhook and wrapped with the tenant key.

HKDF (RFC 5869) fits in this hierarchy for domain separation: deriving independent subkeys (payload wrapping, blind indexes, and so on) from one tenant key so the same key material is never reused across purposes.

A common design mistake is to derive tenant keys with HKDF directly from a root key held in KMS. KMS does not expose raw key material or an HKDF operation, so your code would have to hold the root key in application memory, defeating the point of a KMS boundary. If you want deterministic derivation, derive from a root secret you manage yourself, and understand that it then becomes your most sensitive asset. Also note that a purely derived key cannot be deleted. It can always be recomputed from the root and the tenant ID. That matters for erasure (Section 7), and it is why the hierarchy above stores a random tenant key you can destroy.

import boto3
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.hkdf import HKDF

kms = boto3.client("kms")
ROOT_KEY_ID = "alias/webhook-root"  # your KMS key


def create_tenant_key(tenant_id: str) -> tuple[bytes, bytes]:
    """Create a random tenant key. Persist only the wrapped copy."""
    resp = kms.generate_data_key(
        KeyId=ROOT_KEY_ID,
        KeySpec="AES_256",
        EncryptionContext={"tenant_id": tenant_id},
    )
    return resp["Plaintext"], resp["CiphertextBlob"]  # store CiphertextBlob


def load_tenant_key(tenant_id: str, wrapped_key: bytes) -> bytes:
    """Unwrap once, then cache in memory with a short TTL."""
    resp = kms.decrypt(
        CiphertextBlob=wrapped_key,
        EncryptionContext={"tenant_id": tenant_id},
    )
    return resp["Plaintext"]


def derive_subkey(tenant_key: bytes, purpose: str) -> bytes:
    """Domain-separated 256-bit subkey (e.g. 'dek-wrap', 'blind-index')."""
    return HKDF(
        algorithm=hashes.SHA256(),
        length=32,
        salt=None,
        info=f"webhook-pipeline/v1/{purpose}".encode("utf-8"),
    ).derive(tenant_key)

The KMS encryption context is itself authenticated additional data. The same tenant ID must be presented at decryption, and it shows up in CloudTrail logs. Scope IAM and key policies so the ingress service can only generate and wrap keys, while only workers (and, if necessary, an admin service) may decrypt.

Caching unwrapped tenant keys in an in-memory LRU cache with a TTL means workers call KMS once per tenant per interval instead of once per message. The trade-off is that plaintext keys then sit in process memory. Keep TTLs short and avoid logging or dumping them.

4. Securing the Pipeline: Redis, SQS, and PostgreSQL

Step 1: Encrypt immediately at ingestion

Verify the provider’s signature first, since it needs the original bytes, then encrypt the raw body bytes before they go anywhere else. Do not re-serialize the JSON, because that changes the bytes and breaks later signature checks or forensic comparisons.

import base64
import os

from cryptography.hazmat.primitives.ciphers.aead import AESGCM


def b64(data: bytes) -> str:
    return base64.b64encode(data).decode("ascii")


def encrypt_webhook(raw_body: bytes, tenant_id: str, event_id: str,
                    tenant_key: bytes, key_version: int) -> dict:
    wrap_key = derive_subkey(tenant_key, "dek-wrap")
    aad = f"v1|{tenant_id}|{event_id}|{key_version}".encode("utf-8")

    # Per-message DEK encrypts the payload
    dek = AESGCM.generate_key(bit_length=256)
    nonce = os.urandom(12)
    ciphertext = AESGCM(dek).encrypt(nonce, raw_body, aad)  # tag appended

    # Tenant-derived key wraps the DEK
    wrap_nonce = os.urandom(12)
    wrapped_dek = AESGCM(wrap_key).encrypt(wrap_nonce, dek, aad)

    return {
        "v": 1,
        "kid": key_version,
        "nonce": b64(nonce),
        "ct": b64(ciphertext),
        "wrap_nonce": b64(wrap_nonce),
        "wrapped_dek": b64(wrapped_dek),
    }


def decrypt_webhook(env: dict, tenant_id: str, event_id: str,
                    tenant_key: bytes) -> bytes:
    wrap_key = derive_subkey(tenant_key, "dek-wrap")
    aad = f"v1|{tenant_id}|{event_id}|{env['kid']}".encode("utf-8")
    dek = AESGCM(wrap_key).decrypt(
        base64.b64decode(env["wrap_nonce"]),
        base64.b64decode(env["wrapped_dek"]), aad)
    return AESGCM(dek).decrypt(
        base64.b64decode(env["nonce"]),
        base64.b64decode(env["ct"]), aad)  # raises InvalidTag if tampered

The kid field records which tenant-key version was used, which makes rotation possible: new messages use the new version while old ones stay decryptable.

Step 2: Queue only envelopes

Serialize the envelope and publish it to Redis Streams, RabbitMQ, or SQS. Someone who dumps the Redis keyspace, reads persistence files (RDB/AOF), or intercepts SQS messages sees only opaque base64 strings. Platform encryption such as SQS server-side encryption is still worth enabling as defense in depth, since it covers different threats.

Remember that encoding costs space. Hex doubles the size of the ciphertext, and base64 adds about 33%. If message size matters, store raw bytes (for example, BYTEA in PostgreSQL) instead of text encodings.

Step 3: Store and query in PostgreSQL

The common objection is: “If we encrypt the payload, how do we search or audit events?” The answer is to leave routing metadata unencrypted and encrypt only the sensitive body. Make sure that metadata is itself non-sensitive. An event_type like invoice.paid is fine, while a patient identifier is not.

CREATE TABLE webhook_events (
    id                UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id         TEXT        NOT NULL,
    event_type        TEXT        NOT NULL,   -- non-sensitive, indexed
    external_event_id TEXT        NOT NULL,   -- provider's event id, for dedupe
    received_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    key_version       INTEGER     NOT NULL,
    envelope          JSONB       NOT NULL,   -- nonce, ciphertext, wrapped DEK
    UNIQUE (tenant_id, external_event_id)
);

CREATE INDEX idx_webhook_events_lookup
    ON webhook_events (tenant_id, event_type, received_at DESC);

(gen_random_uuid() is built into PostgreSQL 13 and later.)

Queries on tenant_id, event_type, and received_at run on plain indexed columns. When an authorized user opens one event in an admin dashboard, the application fetches that single row, decrypts it in memory, and returns the plaintext. There is no table scan and no bulk decryption.

For equality search on a sensitive field, such as finding events for a given email address, use a blind index: store an HMAC-SHA256 of the normalized value, keyed with a separate HKDF-derived subkey (derive_subkey(tenant_key, "blind-index")), in its own indexed column. The database can match on the HMAC without seeing the value. The trade-off is that blind indexes reveal when two rows share the same value, and they support only exact matches, so use them sparingly and only for fields that truly need lookup.

PostgreSQL’s pgcrypto extension can encrypt in SQL, but then plaintext and keys pass through the database server, its logs, and statement history. For this threat model, encrypt in the application so the database never sees them.

5. Performance and Overhead

Cryptography adds cost, but for typical webhook sizes it is rarely the bottleneck.

CPU: With AES-NI hardware acceleration, AES-GCM generally runs at gigabytes per second per core on modern x86 servers, so encrypting a payload of a few kilobytes takes microseconds. JSON handling, base64 encoding, network I/O, and any remote key call will outweigh it. Don’t trust a generic figure, including this one. Measure on your own hardware with openssl speed -evp aes-256-gcm and benchmark your actual encrypt path.
Size: With the scheme above, fixed binary overhead is about 88 bytes: a 12-byte payload nonce, a 16-byte payload tag, a 12-byte wrap nonce, and a 60-byte wrapped DEK (32-byte key plus 16-byte tag, plus its 12-byte nonce). Text encoding is the bigger factor. Base64 inflates everything by roughly a third, and hex would double it, so a 15 KB payload becomes about 20 KB as base64 and 30 KB as hex. Field names and JSON syntax add a little more.
Network and KMS: The expensive operation is a remote KMS call. Unwrap tenant keys once and cache them, and monitor your account’s KMS request quota. With caching, per-message overhead stays local and throughput stays close to a plaintext pipeline, though you should confirm this under load.

6. Retries, Idempotency, and Failure States

Webhook delivery is at-least-once. Providers retry on failures and timeouts, and the same event can arrive more than once. Stripe, for example, retries failed deliveries for up to 72 hours. Your encryption layer has to work with that.

Verify before you encrypt. Providers sign the raw request body. Stripe sends an HMAC-SHA256 signature in the Stripe-Signature header, computed over the timestamp and the exact raw body, and verification fails if you parse or re-serialize the JSON first. Verify the signature, then encrypt the same raw bytes.
Deduplicate on the event ID, not the signature. The signature header is a signature, not an idempotency key. Use the provider’s stable event identifier, such as the Stripe event id (evt_...), stored in the unencrypted external_event_id column with a unique constraint, as in the schema above. An INSERT ... ON CONFLICT DO NOTHING lets you discard replays before spending any decryption or queue work on them. (Stripe’s Idempotency-Key header is a different mechanism, used on your outbound API requests.)
Acknowledge fast, process asynchronously. Return a 2xx once the event is verified, encrypted, and durably enqueued. Do the slow work in workers.
Dead-letter queues stay encrypted. When a worker exhausts its retries, the message moves to a DLQ still in envelope form. Operators can inspect metadata, count failures, and purge or redrive messages without ever seeing PII or PHI. Decrypting for debugging should require an explicit, audited action.
Log defensively. Never log request bodies or decrypted payloads. Log the event ID, tenant ID, key version, and outcome, and scrub exception messages that may embed request data.

7. Cryptographic Erasure and Its Limits

Deleting one tenant’s records from backups, archives, and queues is slow and error-prone. Cryptographic erasure offers a shortcut: if every record for a tenant is encrypted under keys that only exist in one place, destroying those keys renders the data unreadable even where ciphertext copies persist. NIST SP 800-88 recognizes cryptographic erase as a media sanitization technique.

With the hierarchy from Section 3, the procedure is:

Delete the tenant’s wrapped tenant-key records (all versions) from your key store.
Purge cached plaintext copies from worker memory, for example by restarting workers or invalidating the cache.
Make sure backups of the key store do not retain the keys longer than your retention policy allows. Erasure is only as strong as the oldest surviving copy of the key.
If you use per-tenant KMS keys, schedule key deletion. AWS KMS requires a waiting period of 7 to 30 days (30 by default) before deletion completes. During that period the key is in a “pending deletion” state and cannot be used for cryptographic operations. The key can be recovered by cancelling deletion until the period ends, so plan for that delay in your deletion SLA.

Be careful with the claims you make about this:

It is not “instantaneous” in practice, because of caches, key-store backups, and KMS waiting periods.
“Mathematically impossible” overstates it. The accurate statement is that decryption becomes computationally infeasible, assuming the keys are truly gone and the cipher remains secure.
Whether crypto-shredding satisfies a particular erasure request is a legal question. Regulators and auditors may expect that other copies, such as plaintext in logs, analytics stores, or unencrypted metadata, are removed too. Treat cryptographic erasure as one tool within your erasure process and confirm the approach with your data protection officer or counsel.

Production Checklist

Use envelope encryption. Generate a fresh random 256-bit DEK per webhook, and never encrypt payloads directly with a long-lived key.
Use AEAD with bound context. Use AES-256-GCM or ChaCha20-Poly1305, include tenant ID, event ID, and version as associated data, and alert on authentication failures.
Keep the root key in KMS/HSM. Use random per-tenant keys wrapped by the root key, and use HKDF for domain-separated subkeys rather than for reconstructing deletable keys.
Cache and version keys. Cache unwrapped tenant keys briefly, track a key version in every envelope, and rotate before approaching GCM invocation limits.
Verify, then encrypt raw bytes. Check provider signatures on the raw body, and encrypt those exact bytes.
Keep non-sensitive metadata queryable. Leave routing fields such as tenant, event type, timestamp, and provider event ID unencrypted and indexed, and use blind indexes only where a sensitive field needs equality search.
Deduplicate on provider event IDs with a unique constraint, and keep DLQs encrypted.
Plan erasure honestly. Design key deletion, cache invalidation, and key-store backup retention together, and document what your erasure guarantees do and don’t cover.
Scrub logs. No payloads in logs, traces, or error reports.

References

PCI DSS v4.0.1, Requirements 3.3.1, 3.5.1, 3.5.1.2: https://www.pcisecuritystandards.org/document_library/
HIPAA Security Rule NPRM status (mid-2026): https://luxsci.com/hipaa-security-rule-final-rule-missed-may-deadline/ and https://www.myworkdrive.com/blog/hipaa-security-rule-update
GDPR Article 32: https://gdpr-info.eu/art-32-gdpr/
RFC 5869, HKDF: https://www.rfc-editor.org/rfc/rfc5869
RFC 8439, ChaCha20-Poly1305: https://www.rfc-editor.org/rfc/rfc8439
NIST SP 800-38D, GCM: https://csrc.nist.gov/publications/detail/sp/800-38d/final
NIST SP 800-88, Media Sanitization: https://csrc.nist.gov/publications/detail/sp/800-88/rev-1/final
AWS KMS request quotas: https://docs.aws.amazon.com/kms/latest/developerguide/requests-per-second.html
AWS KMS key deletion: https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys.html

Originally published at https://instawebhook.com/blog/webhook-payload-encryption-at-rest-application-layer-cryptography-for-sensitive

Web-Scale Webhook Ingress & Traffic Management: Architecting Resilient, High-Throughput Gateways

InstaWebhook — Sun, 11 Oct 2026 02:09:26 +0000

Webhooks connect SaaS platforms, payment processors, CI/CD pipelines, and cloud-native microservices. Unlike internal RPCs or authenticated API clients, a webhook ingress tier faces an untrusted external ecosystem. Senders include well-behaved platforms with documented retry policies, misconfigured integrations that replay events in tight loops, and attackers who find your public endpoint and flood it.

The cost of failure is concrete. Providers cap how long they wait for your response, and they give up after a finite retry window:

Provider	Response timeout	Retry behavior	Other limits
GitHub	10 seconds	Not covered here	Payloads capped at 25 MB; larger events are not delivered
Shopify	5 seconds (1-second connection timeout)	Up to 8 retries over 4 hours	Subscriptions are removed if failures persist
Stripe	Not published in the sources reviewed	Live mode: exponential backoff for up to 3 days	Test mode retries 3 times over a few hours

If your gateway is down for longer than a provider’s retry window, events are lost permanently. Surviving that takes more than a reverse proxy. It takes layered defenses: distributed rate limiting, a geo-distributed edge, regional failover with a realistic idempotency model, and streaming request handling that never buffers a whole body in RAM.

1. Rate Limiting for Webhook Ingress

The Webhook Traffic Problem

Conventional API rate limiting keys off API keys, OAuth tokens, or user accounts, and clients are expected to honor 429 Too Many Requests and Retry-After. Webhooks invert this. The publisher is an external system pushing data into yours, and you have no control over its retry logic.

Well-run platforms treat non-2xx responses as failures and retry them with backoff. Rejecting traffic with a 429 therefore usually delays delivery rather than dropping it, but only while you are inside the provider’s retry window. Misconfigured senders, retry storms, and buggy emitters may ignore backpressure entirely. Retry behavior also varies widely between providers (see the table above), so a rate-limit policy tuned for one provider can silently drop events from another.

Without fine-grained limits, one runaway publisher can exhaust your connection pool and worker threads. That starves every other tenant on the same tier.

Token Bucket vs. Leaky Bucket

Two algorithms dominate traffic shaping:

Token bucket allows bursts up to a configured capacity while enforcing an average refill rate. It suits legitimate webhook spikes, such as a merchant platform emitting a batch of order events during a sale.
Leaky bucket (as a queue or meter) drains at a constant rate regardless of arrival burstiness. It smooths traffic ahead of fragile downstream systems. NGINX’s limit_req module is a well-known implementation of this model.

Layer Local and Global Limits

A single gateway instance cannot enforce a per-publisher limit across a horizontally scaled fleet, so you need shared state. Envoy documents this as two stages. A local token-bucket filter absorbs large bursts in each proxy, and a global gRPC rate limit service enforces fine-grained limits. Envoy’s reference rate limit service is written in Go and uses a Redis backend. The local stage reduces load on the global service, so a burst does not turn your rate limiter into your bottleneck.

Two design points matter for webhooks:

Do the cheap checks first. HMAC verification requires reading the whole body, so limiting by “verified signature” means attackers can make you spend CPU and bandwidth before you can reject them. Key the first-stage limits on things you know before reading the body: the source IP or network, and a per-tenant endpoint identifier in the URL path. Apply per-publisher limits after authentication.
Fail open or closed deliberately. If Redis is unreachable, decide in advance whether to admit traffic (risking overload) or shed it (risking lost events inside the retry window). A local limiter is a sensible fallback.

A Distributed Token Bucket in Redis and Lua

Redis executes a Lua script atomically, so the read-modify-write cycle cannot interleave with another client’s. The version below fixes problems that commonly appear in examples of this pattern:

It reads time from Redis (TIME) rather than trusting a caller-supplied timestamp, which avoids clock skew between gateway nodes.
It uses HSET with multiple fields, because HMSET has been deprecated since Redis 4.0.
It uses sub-second precision so fractional refill rates work.
It returns the token count as a string, because Redis truncates Lua floats to integers in replies.

-- KEYS[1]: rate limit key, e.g. "rl:tenant:acme"
-- ARGV[1]: capacity (max tokens)
-- ARGV[2]: refill rate (tokens per second)
-- ARGV[3]: cost of this request (usually 1)

local key         = KEYS[1]
local capacity    = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local requested   = tonumber(ARGV[3])

-- Use the Redis server clock so all gateway nodes agree on "now"
local t   = redis.call("TIME")
local now = tonumber(t[1]) + tonumber(t[2]) / 1000000

local data   = redis.call("HMGET", key, "tokens", "ts")
local tokens = tonumber(data[1])
local ts     = tonumber(data[2])

if tokens == nil or ts == nil then
  tokens = capacity
else
  local elapsed = math.max(0, now - ts)
  tokens = math.min(capacity, tokens + elapsed * refill_rate)
end

local allowed     = 0
local retry_after = 0

if tokens >= requested then
  tokens  = tokens - requested
  allowed = 1
else
  retry_after = math.ceil((requested - tokens) / refill_rate)
end

redis.call("HSET", key, "tokens", tokens, "ts", now)
-- Expire idle buckets after twice the time needed to refill from empty
redis.call("PEXPIRE", key, math.ceil(capacity / refill_rate * 1000) * 2)

return { allowed, tostring(tokens), retry_after }

Notes for production use:

In Redis Cluster, each bucket lives under a single key, so the script stays single-slot. Always pass the key via KEYS, never build it inside the script.
Return 429 with a Retry-After header built from retry_after. Compliant senders will use it, and others will at least be easy to identify in logs.
Cross-region active-active Redis deployments typically replicate asynchronously and reconcile conflicts. Do not assume a token bucket stays exact across regions. Give each region its own share of the budget, or accept approximate global limits.

Where this logic runs depends on your stack. Envoy supports both the rate limit service above and Wasm extensions. OpenResty (NGINX plus Lua) and custom Go gateways can call Redis directly.

2. Geo-Distributed Ingress: Anycast, Edge Routing, and Regional Failover

Why a Single Region Is Not Enough

Pointing every publisher at one cloud region adds latency for distant senders and makes that region a single point of failure. Because providers only retry for a bounded time, a regional outage that outlasts the window means permanent event loss, not just delay.

What Anycast Actually Does

Anycast means multiple locations announce the same IP prefix via BGP, and Internet routing delivers each client’s packets to a topologically nearby announcer. This is how large CDNs and DDoS-mitigation networks absorb attack traffic across many sites. Managed products such as AWS Global Accelerator also expose static anycast IPs.

Be precise about terminology. “Anycast DNS” refers to authoritative DNS servers sharing anycast addresses. That is a different thing from fronting your HTTPS endpoint with an anycast IP. For webhook ingress you usually want the latter, optionally combined with DNS-based steering and health checks to move traffic between regions.

[Webhook publishers worldwide]
          │
          ▼  (anycast IP)
┌──────────────────────────────┐
│ Edge network (CDN / WAF)     │
│  - TLS termination           │
│  - DDoS mitigation           │
│  - Coarse rate limits        │
│  - Body size caps            │
└───────┬──────────────┬───────┘
        │              │
        ▼              ▼
┌──────────────┐ ┌──────────────┐
│ Region A     │ │ Region B     │
│ ingress tier │ │ ingress tier │
└──────┬───────┘ └──────┬───────┘
       ▼                ▼
  durable queue    durable queue
  (per region)     (per region)

What to Do at the Edge, and What Not To

Edge nodes are a good place for cheap, stateless work: TLS termination, IP reputation and allowlisting, coarse rate limits, and rejecting oversized or malformed requests. Some providers publish their source IP ranges. GitHub does, through its meta API, so allowlisting can be an extra layer, though not a substitute for signature verification.

Be careful about moving HMAC verification into edge functions. Edge platforms impose body-size limits that can make verification of large payloads impossible or incorrect:

With CloudFront, Lambda@Edge only exposes the request body if you enable the include-body option, and it truncates the body at 1 MB for origin request events. A signature computed over a truncated body will not match.
Cloudflare enforces a request body limit by plan: 100 MB on Free and Pro, 200 MB on Business, and 500 MB by default on Enterprise. Larger requests get a 413 at the edge.

The practical rule is to verify signatures at your own ingress tier, where you control buffering and can stream the entire body. Use the edge for filtering that does not depend on the full payload.

Idempotency Across Regions: Be Honest About the Consistency Model

Cross-region deduplication is where many designs go wrong. Webhook delivery is at-least-once, so duplicates are normal. They come from provider retries after a timeout, and from your own failover when two regions both receive the same event.

Consider a multi-region key-value store used for “have I seen this event ID?” checks. In DynamoDB global tables, the default multi-Region eventual consistency (MREC) mode replicates asynchronously and resolves conflicting writes with last-writer-wins. A conditional “insert if not exists” is evaluated against the local replica. If the same event lands in two regions before replication completes, both writes can succeed, and you process the event twice.

AWS also offers multi-Region strong consistency (MRSC), introduced in June 2025. A successful write is immediately readable from any replica, but MRSC tables are limited to certain Region sets and the consistency mode cannot be changed after creation. Strong cross-region coordination also costs latency.

A robust design combines several measures rather than relying on any one:

Partition by publisher or tenant. Route each publisher to a home region, for example with consistent hashing at the edge. Fail over to another region only when the home region is unhealthy. This keeps almost all deduplication local and strongly consistent.
Use a strongly consistent unique constraint in the home region. For a relational store (PostgreSQL syntax shown):

   INSERT INTO webhook_events (event_id, received_at, status)
   VALUES ($1, now(), 'RECEIVED')
   ON CONFLICT (event_id) DO NOTHING
   RETURNING event_id;

If no row is returned, the event is a duplicate. Acknowledge it with a 2xx and skip further processing.

Make downstream handlers idempotent anyway. During failover, a small window of cross-region duplicates is unavoidable with asynchronous replication. Side effects such as charging a card, sending an email, or provisioning a resource should carry their own idempotency keys.
Use provider-supplied identifiers. Don’t invent a header name. GitHub sends a unique X-GitHub-Delivery header per delivery. Providers following the Standard Webhooks convention send webhook-id, which stays constant across retries of the same message, and Stripe puts an event id in the body. There is no universal X-Webhook-ID.

CRDTs can model a replicated “seen set,” but they converge eventually. They cannot give you the first-writer-wins exclusivity that prevents two regions from both acting on one event, so don’t present them as a substitute for the measures above.

3. Streaming Large Payloads Without Running Out of Memory

How Big Are Webhooks, Really?

Most webhook bodies are small JSON documents. Large ones exist, but provider limits cap them. GitHub, for example, does not deliver payloads over 25 MB. Your own internal publishers, partner integrations, and bulk-export callbacks may be larger, and attackers choose their own sizes. The defensive posture is the same either way: set an explicit maximum size, and never assume a body fits in memory.

For truly large data, prefer a “thin event” or claim-check pattern. The webhook carries a reference such as an ID or a pre-signed URL, and the consumer fetches the data separately.

The common anti-pattern is reading the whole body into memory before verifying it. Buffering 50 concurrent 200 MB bodies means roughly 10 GB of RAM. On a container limited to 2 GB, that ends in an out-of-memory kill that takes legitimate traffic down with it.

Signature Verification Constraints

Two constraints shape the design:

HMAC verification must run over the exact raw bytes received. Parsing the JSON first and re-serializing it will break the signature.
A hash can be computed incrementally while bytes stream past, so verification never needs the full body in memory.

The pattern is: stream the body through the hasher into a size-limited spool (a temporary file or object storage), verify the signature, durably hand off, return a 2xx quickly, and parse asynchronously. Returning quickly matters because of the provider timeouts above. Shopify, for instance, advises delaying processing until after you’ve sent a response.

Go

Several details are easy to get wrong in Go:

io.LimitReader silently truncates at the limit and returns EOF. A truncated body would then be hashed and accepted as if complete. Use http.MaxBytesReader, which returns an error (*http.MaxBytesError, Go 1.19+) and signals the server to close the connection.
hmac.New requires a key. hmac.Equal is already a constant-time comparison.
Limiting body size does not stop slow-sender attacks. You also need server timeouts such as ReadHeaderTimeout, and per-request read deadlines for large uploads.

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "errors"
    "io"
    "log"
    "net/http"
    "os"
    "path/filepath"
    "strings"
    "time"
)

const (
    maxWebhookSize = 50 << 20 // 50 MiB
    spoolDir       = "/var/spool/webhooks"
)

func webhookHandler(secret []byte) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if r.Method != http.MethodPost {
            http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
            return
        }

        // GitHub-style header: "sha256=<hex>"
        hexSig := strings.TrimPrefix(r.Header.Get("X-Hub-Signature-256"), "sha256=")
        provided, err := hex.DecodeString(hexSig)
        if err != nil || len(provided) == 0 {
            http.Error(w, "missing or malformed signature", http.StatusUnauthorized)
            return
        }

        // Allow slow-but-legitimate large uploads, but not forever
        rc := http.NewResponseController(w)
        _ = rc.SetReadDeadline(time.Now().Add(60 * time.Second))

        // Hard cap: returns an error (not silent truncation) when exceeded
        r.Body = http.MaxBytesReader(w, r.Body, maxWebhookSize)

        tmp, err := os.CreateTemp(spoolDir, "incoming-*.json")
        if err != nil {
            http.Error(w, "internal error", http.StatusInternalServerError)
            return
        }
        accepted := false
        defer func() {
            tmp.Close()
            if !accepted {
                os.Remove(tmp.Name())
            }
        }()

        // Bytes flow to the hasher and to disk in one pass, using a small
        // fixed-size copy buffer instead of the whole payload in memory
        mac := hmac.New(sha256.New, secret)
        if _, err := io.Copy(io.MultiWriter(mac, tmp), r.Body); err != nil {
            var tooBig *http.MaxBytesError
            if errors.As(err, &tooBig) {
                http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
            } else {
                http.Error(w, "read failed", http.StatusBadRequest)
            }
            return
        }

        // hmac.Equal compares in constant time
        if !hmac.Equal(mac.Sum(nil), provided) {
            http.Error(w, "invalid signature", http.StatusUnauthorized)
            return
        }

        // Hand the verified file to a worker (rename into a queue directory,
        // publish a pointer to a durable queue, etc.), then acknowledge fast
        final := filepath.Join(spoolDir, "verified-"+filepath.Base(tmp.Name()))
        if err := os.Rename(tmp.Name(), final); err != nil {
            http.Error(w, "internal error", http.StatusInternalServerError)
            return
        }
        accepted = true
        w.WriteHeader(http.StatusAccepted)
    }
}

func main() {
    secret := []byte(os.Getenv("WEBHOOK_SECRET"))
    if len(secret) == 0 {
        log.Fatal("WEBHOOK_SECRET is required")
    }
    mux := http.NewServeMux()
    mux.Handle("/webhook", webhookHandler(secret))

    srv := &http.Server{
        Addr:              ":8080",
        Handler:           mux,
        ReadHeaderTimeout: 5 * time.Second, // mitigates slow-header (Slowloris-style) clients
        IdleTimeout:       60 * time.Second,
    }
    log.Fatal(srv.ListenAndServe())
}

Verification here has no timestamp check. A leaked signed request can be replayed forever. Providers that sign a timestamp alongside the body, such as Stripe and the Standard Webhooks convention, expect receivers to reject messages outside a tolerance window. The Standard Webhooks specification recommends 300 seconds, and Stripe’s libraries default to the same. Combine that with the event-ID deduplication from section 2.

A worker can then parse the spooled file token by token, without ever materializing the whole document. For a top-level JSON array of events:

func processEvents(path string, handle func(json.RawMessage) error) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()

    dec := json.NewDecoder(f)
    if _, err := dec.Token(); err != nil { // consume the opening '['
        return err
    }
    for dec.More() {
        var ev json.RawMessage
        if err := dec.Decode(&ev); err != nil {
            return err
        }
        if err := handle(ev); err != nil {
            return err
        }
    }
    return nil
}

(Add encoding/json to the imports.) Decoding the whole body into a map[string]interface{} would defeat the purpose, because that holds the entire document in memory.

Node.js / TypeScript

Express’s express.json() reads and parses the whole body into memory, with a default limit of 100 KB. Raising the limit to 100mb for webhook routes means a flood of large requests can exhaust the V8 heap. Instead, consume the request as a stream and apply backpressure.

Common mistakes in naive stream handlers:

Calling fileStream.write() in a loop ignores backpressure. Use stream.pipeline.
crypto.timingSafeEqual throws if the two buffers differ in length, so check lengths first.
Without a byte counter, nothing stops an unbounded upload.

import http from 'node:http';
import crypto from 'node:crypto';
import path from 'node:path';
import { createWriteStream } from 'node:fs';
import { rename, unlink } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { Transform } from 'node:stream';

const SECRET = process.env.WEBHOOK_SECRET;
if (!SECRET) throw new Error('WEBHOOK_SECRET is required');

const MAX_BYTES = 50 * 1024 * 1024;
const SPOOL_DIR = '/var/spool/webhooks';

class PayloadTooLargeError extends Error {}

const server = http.createServer(async (req, res) => {
  if (req.method !== 'POST' || req.url !== '/webhook') {
    res.writeHead(404).end();
    return;
  }

  const header = req.headers['x-hub-signature-256'];
  if (typeof header !== 'string' || !header.startsWith('sha256=')) {
    res.writeHead(401).end();
    return;
  }
  const provided = Buffer.from(header.slice('sha256='.length), 'hex');

  // Cheap early rejection; the byte counter below is the real enforcement
  if (Number(req.headers['content-length'] ?? 0) > MAX_BYTES) {
    res.writeHead(413, { Connection: 'close' }).end();
    return;
  }

  const hmac = crypto.createHmac('sha256', SECRET);
  let received = 0;
  const meter = new Transform({
    transform(chunk: Buffer, _enc, cb) {
      received += chunk.length;
      if (received > MAX_BYTES) return cb(new PayloadTooLargeError());
      hmac.update(chunk);
      cb(null, chunk);
    },
  });

  const id = crypto.randomUUID();
  const tmpPath = path.join(SPOOL_DIR, `${id}.part`);

  try {
    // pipeline handles backpressure and cleans up streams on error
    await pipeline(req, meter, createWriteStream(tmpPath));

    const expected = hmac.digest();
    if (provided.length !== expected.length ||
        !crypto.timingSafeEqual(provided, expected)) {
      await unlink(tmpPath).catch(() => {});
      res.writeHead(401).end();
      return;
    }

    await rename(tmpPath, path.join(SPOOL_DIR, `${id}.json`)); // visible to workers
    res.writeHead(202).end();
  } catch (err) {
    await unlink(tmpPath).catch(() => {});
    if (err instanceof PayloadTooLargeError) {
      res.writeHead(413, { Connection: 'close' }).end();
    } else {
      console.error('webhook ingestion error', err);
      res.writeHead(500).end();
    }
  }
});

server.requestTimeout = 60_000;  // total time allowed to receive the request
server.headersTimeout = 10_000;  // time allowed to receive headers
server.listen(8080);

A worker process can then parse the verified file incrementally with a streaming parser such as stream-json:

const { createReadStream } = require('node:fs');
const { pipeline } = require('node:stream/promises');
const { parser } = require('stream-json');
const { streamArray } = require('stream-json/streamers/StreamArray');

async function processEvents(filePath, handleEvent) {
  await pipeline(
    createReadStream(filePath),
    parser(),
    streamArray(), // emits { key, value } for each element of a top-level array
    async function (source) {
      for await (const { value } of source) {
        await handleEvent(value);
      }
    }
  );
}

Summary & Architectural Checklist

Layered rate limiting. Use a local token bucket in each proxy for burst absorption and a Redis-backed global limiter for fine-grained per-tenant limits. Do cheap, body-independent checks first, and decide fail-open versus fail-closed behavior for your limiter in advance.
Geo-distributed ingress. Front your endpoint with an anycast edge for TLS termination and DDoS absorption. Verify signatures at an ingress tier where you control full-body streaming, not in edge functions that truncate or cap bodies.
Honest idempotency. Partition publishers to home regions and enforce uniqueness with a strongly consistent constraint there. Treat cross-region replication as eventually consistent unless you deliberately pay for strong consistency, and make downstream side effects idempotent regardless.
Bounded, streaming ingestion. Enforce explicit size limits that error instead of truncating. Hash and spool in one pass, verify in constant time, check timestamps for replay protection, and set server timeouts.
Acknowledge fast, process later. Provider timeouts are as short as 5 to 10 seconds. Durably enqueue, return a 2xx, and do the heavy parsing and business logic asynchronously.
Know your providers’ retry windows. Your outage tolerance is the shortest retry window among your publishers (about 4 hours for Shopify, up to 3 days for Stripe). Plan reconciliation jobs for anything longer.

Sources Referenced

GitHub webhook payload cap and delivery headers: GitHub Docs
GitHub 10-second timeout: GitHub Developer changelog
Stripe retry behavior: Stripe webhook documentation
Shopify timeouts and retries: Shopify webhook troubleshooting
Envoy local and global rate limiting: Envoy global rate limiting overview and local rate limit filter
Lambda@Edge body limits: AWS CloudFront documentation
Cloudflare request body limits: Cloudflare limits documentation
DynamoDB global table consistency modes: AWS Prescriptive Guidance
Standard Webhooks signing and replay tolerance: Svix glossary

Originally published at https://instawebhook.com/blog/web-scale-webhook-ingress-traffic-management-architecting-resilient-high-through

High-Performance Webhook Architecture: Scaling Ingress, Partitioning Queues, and Neutralizing Poison Pills

InstaWebhook — Sat, 10 Oct 2026 02:04:26 +0000

Webhooks drive asynchronous, event-driven integrations between microservices, SaaS vendors, and enterprise customers. At millions of events per day, the usual failure modes are a slow downstream consumer, a burst from one tenant, or a malformed payload that keeps crashing workers. Any of these can turn a local problem into a platform-wide one.

This guide covers three pillars of a resilient webhook pipeline:

Partitioning and fair scheduling to contain noisy tenants and head-of-line blocking.
Low-allocation ingress, meaning ring buffers and zero-copy techniques, plus what they can and cannot deliver.
Poison pill detection and quarantine, using dead-letter queues, bounded retries, and safe replay.

The patterns apply to a webhook platform that receives events and fans them out to tenant endpoints. Where a technique applies only to inbound traffic or only to outbound delivery, the text says so.

1. Partition Strategy: Containing Noisy Tenants and Head-of-Line Blocking

The head-of-line blocking problem

In a multi-tenant pipeline, events are written to a queue or table and then picked up by workers. When all tenants share one global queue, a single slow tenant becomes a problem. If workers stall on its slow endpoint, they hold connections, threads, and in-flight slots, and healthy tenants wait behind them. This is head-of-line (HoL) blocking, a close relative of the “noisy neighbor” problem.

Senders impose their own deadlines, so the cost of being slow is real. GitHub, for example, expects a 2xx response within 10 seconds, treats a slower response as a failed delivery, and recommends queueing payloads and processing them asynchronously. Stripe tells receivers to return a successful status quickly and defer complex logic.

Partitioning helps, but it is not a complete fix

Partitioning by tenant (or by a hash of the tenant ID) narrows the blast radius, but two caveats apply:

Hash partitioning still co-locates tenants. If tenants 17 and 902 hash to the same partition, one slow tenant still delays the other. In a classic Kafka consumer group, each partition is assigned to exactly one consumer in the group, so a stalled consumer stalls everything on its partitions.
Partitioning a database table is about data layout, not worker isolation. It helps with pruning, index size, and cheap retention (detaching or dropping old partitions). It does not stop a stalled worker pool from starving other tenants.

Incoming webhooks
       │
       ▼
[Ingress: authenticate, size-limit, persist, ack]
       │
       ├─► Partition 0 (tenant hash bucket 0) ──► Worker pool 0
       ├─► Partition 1 (tenant hash bucket 1) ──► Worker pool 1
       ├─► Partition N ...                    ──► Worker pool N
       └─► Dedicated queue for very large or unreliable tenants

Practical techniques

Tenant-aware keys in relational storage. PostgreSQL requires that a primary key or unique constraint on a partitioned table include all partition key columns, and the partition key cannot contain expressions. If you hash-partition on tenant_id, a key such as (tenant_id, webhook_id) works. If you also range-partition on created_at for retention, created_at must appear in the key too. Note that time-based partitioning separates data by time, not by tenant, so use hash or list partitioning on tenant_id (optionally sub-partitioned by time) when tenant segregation is the goal.
Route by tenant at the edge. Extract the tenant identifier from the URL path, a header, or an authenticated token. The URL or header is usually better than the payload, because you can read it without parsing the body. Use it as the Kafka message key, the SQS message group ID, or the Redis Stream key.
Use fair scheduling, not only partitioning. There are now managed building blocks for this:
- Amazon SQS fair queues (announced July 2025) work on standard queues. You set a MessageGroupId as the tenant identifier. When one tenant builds a backlog, SQS prioritizes delivery of other tenants’ messages. The noisy tenant’s messages are not dropped or throttled and are delivered as capacity allows. Messages without a group ID are each treated as their own tenant.
- Kafka share groups (KIP-932, “queues for Kafka”) let consumers scale beyond the partition count and provide per-message processing controls. They became production-ready with Apache Kafka 4.2.
Add per-tenant limits as well. A token bucket per tenant caps request rate at ingress. Pair it with a concurrency limit for outbound delivery. Rate limiting stops a tenant from flooding you, while fair queueing shares worker capacity among tenants that are all legitimately busy. You usually want both.

2. Low-Allocation Ingress: Ring Buffers and Zero-Copy Techniques

Where the cost actually is

At 50,000 requests per second with 10 KB payloads, the pipeline moves about 500 MB/s of payload data. That is a modest rate for modern hardware, and in practice most webhook platforms are limited by durable writes, TLS, JSON parsing, and slow downstream HTTP calls rather than by raw memory copies. Measure before optimizing.

Allocation pressure still matters in managed runtimes, because frequent short-lived allocations increase garbage collection work and can hurt tail latency (p99/p999). Reducing allocations is a legitimate goal, as long as you are clear about which techniques achieve it.

Ring buffers: the Disruptor pattern

The LMAX Disruptor is an inter-thread messaging library for the JVM built around a pre-allocated ring buffer. Its main ideas:

Pre-allocation. Ring buffer entries are created once at startup. Publishers claim an existing entry and overwrite its fields instead of allocating new objects, which reduces garbage in steady state.
Single-writer principle. One thread writing to a given piece of state avoids write contention.
Sequence counters instead of locks. Producers and consumers coordinate through sequence numbers rather than mutexes.
Cache-line padding. The sequence counters are padded so that independent variables written by different threads do not share a cache line (typically 64 bytes on common x86 CPUs). This avoids false sharing.

[Network I/O thread] ──► [Ring buffer: pre-allocated slots]
                                  │
                      (sequence barriers / dependencies)
                                  │
                                  ▼
                       [Parse / validate stage]
                          │               │
                          ▼               ▼
                   [Route stage]    [Journal stage]

Three limits to keep in mind:

A ring buffer is in-process and in-memory. It is not durable, so it does not make a webhook safe to acknowledge. For inbound webhooks, only return a 2xx after the event has been written to durable storage or a durable queue.
It is bounded. When the buffer is full, you must apply backpressure. At the HTTP edge that means returning a retryable status such as 429 or 503 rather than dropping events silently. Stripe retries non-2xx responses for up to three days in live mode, and many senders retry in a similar way, so a retryable failure beats data loss.
Its latency numbers do not describe your whole request. In-process handoff can be very fast, but end-to-end webhook latency also includes the network, TLS, and the durable write.

What “zero-copy” really means on Linux

The claim that mmap lets network data flow from the NIC into shared memory without copies is not accurate. mmap maps files or devices into a process’s address space, and it is useful for file-backed logs and queues, but it does not by itself remove copies on the socket receive path. The real options are narrower:

Transmit side. MSG_ZEROCOPY lets the kernel send from user pages without copying them first. It mainly benefits large sends, such as outbound webhook delivery with big payloads, and less so small ones.
Receive side, in-kernel TCP. Linux 6.15 added io_uring zero-copy receive, which delivers packet payload directly into user memory. It needs specific NIC features (header/data split, flow steering, and RSS configuration), and the kernel still processes packet headers and runs the TCP stack as normal.
Receive side, kernel bypass. Frameworks such as DPDK take the network stack out of the kernel entirely, which means you must supply your own TCP/TLS/HTTP handling. This is rarely justified for an HTTP webhook receiver.
TLS. Webhook traffic is HTTPS (Stripe requires TLS 1.2 or higher), so the payload has to be decrypted before it can be parsed. In practice most teams terminate TLS at a load balancer or proxy and focus on efficient buffer reuse downstream.

For persistence, an append-only log is the proven pattern. Kafka, for example, leans on the OS page cache and sequential I/O. If you build your own memory-mapped journal, treat durability settings (when data is flushed to disk) as an explicit design decision.

Realistic benefits

Fewer allocations in the hot path, which means less GC pressure in JVM-based ingress.
Lower lock contention between pipeline stages inside one process.
Better CPU cache behavior from contiguous, pre-allocated structures.

These are targeted gains for a specific hot path, not a replacement for sound queueing, partitioning, and acknowledgment design.

3. Handling Poison Pills: Detection, Quarantine, and Safe Replay

What is a poison pill?

A poison pill is a message that repeatedly fails processing, for example a payload that crashes a parser, triggers an unhandled exception, or hangs a worker. The exact loop depends on the broker:

With Amazon SQS, a consumer that fails to delete a message sees it reappear after the visibility timeout, and it is then retried.
With Kafka, a consumer that fails before committing its offset re-reads the same record after a restart or rebalance.

Without a limit, the same message keeps consuming worker capacity. In a partitioned or ordered queue, it can also block everything behind it.

Phase 1: Reject early, but choose responses carefully

For inbound webhooks from third parties, what you return to the sender affects what happens next:

Verify signatures on the raw request body. Both Stripe and GitHub document signature verification. A request that fails verification should get a 4xx response and should not be queued.
Apply cheap, strict limits at the edge. Enforce maximum body size, content type, and basic structural checks.
Be careful about rejecting well-formed requests from legitimate senders. Stripe retries any non-2xx response, and if an endpoint fails to return 2xx for multiple days, Stripe emails the account owner and may disable the endpoint. If you reject a payload with a 400 because of your own schema or processing problem, the sender will keep retrying a message that can never succeed. For authenticated senders, it is usually better to acknowledge after durable persistence and do deep schema validation asynchronously, sending failures to quarantine.
Return 5xx or 429 only for transient problems on your side, such as the queue being unavailable, so the sender’s retry logic can help you.

Phase 2: Bounded retries and dead-letter queues

Bounded retries, then isolation, is the core defense:

Count attempts per message. Brokers can do this for you. In SQS, when a message’s receive count exceeds the queue’s maxReceiveCount, SQS moves it to the configured dead-letter queue. A dead-letter queue for a standard queue must itself be a standard queue (and FIFO for FIFO). SQS also supports redriving messages from a DLQ back to the source queue.
Pick the threshold deliberately. Too low and transient failures (a brief database blip) land in the DLQ. Too high and a true poison pill burns capacity. Use exponential backoff between attempts, and consider distinguishing crashes and hangs from ordinary retryable errors.
On Kafka, a consumer group has no built-in per-message retry or dead-letter mechanism, so teams typically write failed records to a separate “dead-letter” topic from application code. Kafka 4.2 adds built-in dead-letter-queue handling to Kafka Streams (KIP-1034), and share groups bring per-message acknowledgment semantics.

[Active queue]
     │
     ├──► [Worker] ──► success ──► ack
     │
     └──► [Worker] ──► failure ──► retry with backoff
                                       │
                                       ▼ (attempts exceeded)
                               [Quarantine / DLQ] ──► [Classifier] ──► [Review & replay]

Phase 3: Classify, notify, and replay safely

Enrich with metadata. Record the error signature, stack trace, attempt count, timestamps, tenant ID, payload size, and a payload hash. Be careful with the payload itself: webhook bodies often contain sensitive customer or payment data. Encrypt quarantined payloads, restrict access, and set a retention limit.
Classify automatically. Simple rules on error type give you categories such as JSON_SYNTAX_ERROR, SCHEMA_MISMATCH, PAYLOAD_TOO_LARGE, or HANDLER_EXCEPTION. Group by signature so one bug that produces thousands of failures shows up as one alert.
Use circuit breakers on outbound delivery. If a tenant’s endpoint keeps failing, open a circuit breaker, pause delivery to it, keep buffering events (within retention limits), and probe periodically. Notify the tenant through email or a dashboard, not only by webhook, since the webhook endpoint is the thing that is broken. Stripe behaves similarly: it retries for up to three days with exponential backoff and warns account owners about misconfigured endpoints.
Make replay idempotent. Replays and sender retries both produce duplicates, so deduplicate on a stable event ID. GitHub sends an X-GitHub-Delivery header that stays the same when you request a redelivery, and Stripe events carry their own event ID. Store processed IDs and make handlers safe to run twice.
Let senders help. Both Stripe and GitHub support redelivering or resending events, so a quarantined event that you cannot recover may be recoverable from the sender.

Summary Table: Architectural Comparison

Dimension	Naive pipeline	Partitioned and fair-scheduled	Hardened ingress and quarantine
Ingestion bottleneck	Single table or global queue	Tenant-keyed partitions or fair queues	Durable log behind a low-allocation ingress stage
Failure isolation	Low: one slow tenant can delay all	Moderate: tenants sharing a partition can still interfere	Higher: per-tenant limits, circuit breakers, and DLQs
Memory and GC behavior	Frequent per-request allocation	Unchanged	Reduced by pre-allocated buffers in hot paths
Poison pill handling	Endless retries or worker crashes	Basic DLQ after max attempts	Bounded retries, quarantine, classification, and idempotent replay

Conclusion

Reliable webhook infrastructure comes from layering several measures, not from a single trick:

Acknowledge fast and durably. Persist first, process asynchronously, and return retryable errors only for genuinely transient problems.
Isolate tenants at several levels. Partition by tenant, add per-tenant rate and concurrency limits, and use fair queueing (such as SQS fair queues or Kafka share groups) so a noisy tenant slows itself rather than everyone.
Optimize the hot path only where measurements justify it. Ring buffers and zero-copy I/O can reduce allocation and contention, but they do not provide durability and their gains are narrower than often claimed.
Bound retries and quarantine failures. Dead-letter queues, classification, circuit breakers, and idempotent replay keep one bad message or tenant from taking down the fleet.

References

Amazon SQS fair queues announcement and documentation: https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-sqs-introduces-fair/ and https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-fair-queues.html
Amazon SQS dead-letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
Stripe webhooks documentation: https://docs.stripe.com/webhooks
GitHub webhook best practices: https://docs.github.com/webhooks/using-webhooks/best-practices-for-using-webhooks
PostgreSQL table partitioning: https://www.postgresql.org/docs/current/ddl-partitioning.html
LMAX Disruptor: https://github.com/lmax-exchange/disruptor
Linux io_uring zero-copy receive: https://docs.kernel.org/6.15/networking/iou-zcrx.html
Queues for Kafka (KIP-932) and Kafka 4.2: https://www.confluent.io/blog/kafka-queue-semantics-share-consumer-ga/ and https://www.instaclustr.com/support/documentation/announcements/apache-kafka-and-kafka-connect/instaclustr-for-apache-kafka-and-kafka-connect-4-2-1-are-now-generally-available/

Originally published at https://instawebhook.com/blog/high-performance-webhook-architecture-scaling-ingress-partitioning-queues-and-ne

eBPF-Based Webhook Observability: Catching Dropped Packets Before the Application Layer

InstaWebhook — Fri, 09 Oct 2026 04:54:34 +0000

Executive Summary: The Invisible Webhook Drop Problem

Webhooks are the connective tissue of event-driven systems. Payment processors, code hosts and commerce platforms push state changes to subscriber backends over plain HTTP POST requests. Each provider has its own delivery rules, and those rules decide what a lost connection costs you:

Provider	Delivery behavior (per provider docs)
Stripe	In live mode, retries for up to three days with exponential backoff. In a sandbox, retries three times over a few hours.
Shopify	Requires a 2xx response within five seconds. Retries failed deliveries up to eight times over about four hours, then removes the subscription if failures persist.
GitHub	Records a failure if your server is down or takes longer than 10 seconds to respond. It does not automatically redeliver failed deliveries.

Retries can mask a kernel-level drop, but they also add delay and duplicate deliveries. With a sender that never retries, the event is simply lost.

Application Performance Monitoring (APM) is the usual way to watch webhook endpoints, and it has a structural blind spot. SDK-based agents (Datadog, New Relic, OpenTelemetry SDKs) run inside your process. Even the newer eBPF-based auto-instrumentation, such as OpenTelemetry eBPF Instrumentation (OBI, donated by Grafana Labs as Beyla in 2025), observes requests that have already reached a process. A connection the kernel never hands to your application produces no span, no log line and no status code.

[ Incoming Webhook Burst ]
           │
           ▼
┌───────────────────────────────────────────┐
│          Linux Kernel Network Stack       │
│ ┌───────────────────────────────────────┐ │
│ │ NIC Rx ring → (XDP) → sk_buff alloc   │ │
│ └──────────────────┬────────────────────┘ │
│ ┌──────────────────▼────────────────────┐ │
│ │ TCP: SYN queue → Accept queue         │ │ ──► [ DROPPED HERE ]
│ └──────────────────┬────────────────────┘ │     (APM never sees it)
└────────────────────┼──────────────────────┘
                     │ accept()
                     ▼
┌───────────────────────────────────────────┐
│       User-Space Application Runtime      │
│ ┌───────────────────────────────────────┐ │
│ │ Webhook worker / HTTP framework       │ │ ──► Seen by APM (HTTP 2xx/5xx)
│ └───────────────────────────────────────┘ │
└───────────────────────────────────────────┘

When the kernel discards a SYN or the final ACK of the handshake because a listening socket’s queues are full, the sender sees a connect timeout, a stalled request, or a reset. Your APM dashboard shows nothing unusual.

This article shows how to find those drops with eBPF, how to measure the latency they hide, and where XDP does and does not help. It also covers the kernel and application tuning that prevents most of them.

1. Anatomy of a Kernel-Level Webhook Drop

1.1 The ingress path

NIC and DMA. Packets land in a receive ring buffer on the network interface card (NIC).
XDP (optional). If an XDP program is attached in native mode, it runs in the driver’s receive path, before the kernel allocates an sk_buff.
NAPI and SoftIRQ. The driver polls the ring and the kernel builds a struct sk_buff for each packet.
IP and TCP processing. The stack validates the packet and looks up the listening socket.
Listener queues. New connections pass through the SYN queue and then the accept queue.
accept(). Only now does your application see the connection, whether it is a Node.js event loop, a Go server or a Python worker.

1.2 The two queues where webhooks die

As Cloudflare describes in SYN packet handling in the wild, every listening TCP socket has two queues:

The SYN queue holds half-open connections (state SYN_RECV) that have been sent a SYN+ACK and are waiting for the client’s final ACK.
The accept queue holds fully established connections waiting for the application to call accept().

The maximum length of both queues derives from the backlog argument the application passes to listen(2). The kernel silently caps that value at net.core.somaxconn. The default for somaxconn was 128 before Linux 5.4 and has been 4096 since 5.4. The default for tcp_max_syn_backlog scales with machine memory, and the 5.4 kernel raised its maximum default from 2048 to 4096.

Accept queue overflow is the common webhook killer. If your runtime is busy, whether from a blocked event loop, an exhausted worker pool or a GC pause, it calls accept() late and the accept queue fills. Per Cloudflare’s analysis, once the accept queue is full, the kernel drops both inbound SYN packets and inbound final-handshake ACK packets. The counters that move are TcpExtListenOverflows and TcpExtListenDrops.

What the sender experiences depends on net.ipv4.tcp_abort_on_overflow:

0 (default). The kernel silently drops the final ACK. From the client’s side the handshake looks complete. Its request data goes unacknowledged until the server’s SYN+ACK retransmission timer and the client’s own retransmissions eventually line up with free space in the queue, or the connection times out. Dropped SYNs appear to the client as connect timeouts that are retried with exponential backoff.
1. The kernel sends a RST after the final ACK, and the sender sees a connection reset. Cloudflare’s advice is that it is better to leave this setting alone, because resets make overload worse for clients that would otherwise retry.

SYN queue overflow is rarer than it used to be. Linux enables SYN cookies by default (net.ipv4.tcp_syncookies=1). When a listener’s SYN queue is full, the kernel answers with a stateless SYN cookie instead of storing the request, so legitimate connections generally survive a SYN flood. Plain SYN queue drops still show up in TCPReqQFullDrop-style counters when cookies are disabled or unavailable.

Either way, the application receives zero bytes for the affected connection, so an APM agent has nothing to record.

2. Tracing Ingress Drops with eBPF

Counters like nstat -az TcpExtListenOverflows TcpExtListenDrops tell you that drops happened, but not which source, port or connection was hit. For a quick check on a live host, ss -ltn is the fastest tool. For listening sockets, Recv-Q is the current accept queue depth and Send-Q is its configured maximum. A Recv-Q pinned at Send-Q means the application is not accepting fast enough.

eBPF adds per-event context without recompiling the kernel or attaching intrusive debuggers.

2.1 The `skb:kfree_skb` tracepoint and drop reasons

Since Linux 5.17, kfree_skb carries a drop reason that explains why a socket buffer was discarded. These reasons are available in RHEL 8.8 and 9.2 and later, and the tracepoint is the main interface for reading them. Coverage has grown over time but is not complete. Kernel patches in 2022 added reasons for accept queue overflow and request queue full, originally named LISTENOVERFLOWS and TCP_REQQFULLDROP. Enum names have been reorganized in later releases, so check the symbols your kernel actually exposes in /sys/kernel/tracing/events/skb/kfree_skb/format.

On older kernels, some listen-queue drops are not reported through kfree_skb at all, because the packet is freed as if it had been consumed normally. If a trace shows nothing while ListenDrops keeps climbing, this is the likely reason. Rely on the counters, and use the kprobe approach in Script 2.

A note on the snippets below. They are illustrative sketches. They assume a kernel with BTF (CONFIG_DEBUG_INFO_BTF), bpftrace 0.21 or later (older releases use args->field instead of args.field), and IPv4 traffic. Kprobe targets are not a stable ABI, so confirm the function names exist on your kernel (bpftrace -l 'kprobe:tcp_v4_*') and test in staging before running them in production.

Script 1: Who is being dropped, and why

This script counts dropped inbound TCP packets for webhook ports, by source address and drop reason.

#!/usr/bin/env bpftrace

tracepoint:skb:kfree_skb
/args.protocol == 0x0800/
{
    $skb = (struct sk_buff *)args.skbaddr;
    $iph = (struct iphdr *)($skb->head + $skb->network_header);

    if ($iph->protocol == 6) {                      // TCP
        $tcp   = (struct tcphdr *)($skb->head + $skb->transport_header);
        $dport = bswap($tcp->dest);

        if ($dport == 8080 || $dport == 443) {
            @drops[ntop($iph->saddr), $dport, args.reason] = count();
        }
    }
}

interval:s:10
{
    time("%H:%M:%S  drops by (source, dport, reason):\n");
    print(@drops);
    clear(@drops);
}

The reason field prints as a number. Map it to a name using the format file mentioned above. Packets freed before the TCP header is parsed may carry an unset transport_header, so treat results as indicative rather than exhaustive.

Script 2: Accept queue pressure per listening port

The kernel considers the accept queue full when sk_ack_backlog exceeds sk_max_ack_backlog. The check runs on the listener socket in both tcp_v4_conn_request (incoming SYN) and tcp_v4_syn_recv_sock (final ACK). Both functions receive the listener as their first argument, so one probe covers both paths.

#!/usr/bin/env bpftrace

kprobe:tcp_v4_conn_request,
kprobe:tcp_v4_syn_recv_sock
{
    $sk = (struct sock *)arg0;

    @max_depth[$sk->sk_num] = max($sk->sk_ack_backlog);

    if ($sk->sk_ack_backlog > $sk->sk_max_ack_backlog) {
        @queue_full[probe, $sk->sk_num] = count();
    }
}

interval:s:10
{
    time("%H:%M:%S\n");
    print(@max_depth);
    print(@queue_full);
    clear(@max_depth);
    clear(@queue_full);
}

If you would rather not maintain probes, BCC and libbpf-tools ship maintained utilities for this area. tcpdrop, tcpretrans and tcpsynbl cover drops, retransmissions and SYN backlog depth.

3. Correlating Kernel Waits with Latency Spikes

When the kernel drops a SYN, the sender waits for its retransmission timer. Per RFC 6298, the initial retransmission timeout is 1 second, and each failed attempt doubles it:

RTO_n = RTO_0 × 2^n        (RTO_0 = 1 s for the initial SYN)

So a client’s SYN retries leave at roughly 1 s, 3 s, 7 s, 15 s and so on after the first attempt. Linux’s tcp_syn_retries setting caps how many times a client tries before giving up. Once a connection has RTT measurements, the minimum RTO in Linux is 200 ms, so established connections recover faster than new ones. A first-SYN drop is therefore especially expensive.

A webhook whose first SYN is dropped and whose retry succeeds arrives about a second late, even though your handler runs in milliseconds:

 Sender (Webhook Provider)                      Receiver (Your Pod)
───────────┬───────────                        ───────────────┬───────────────
           │ 1. SYN                                           │
           │─────────────────────────────────────────────────►│ ──► Dropped (accept queue full)
           │                                                  │
           │ [ sender waits: initial RTO ≈ 1 s ]              │
           │                                                  │
           │ 2. SYN (retransmission #1)                       │
           │─────────────────────────────────────────────────►│ ──► Accepted
           │ 3. SYN-ACK / 4. ACK                              │
           │◄────────────────────────────────────────────────►│
           │ 5. HTTP POST /webhook                            │
           │─────────────────────────────────────────────────►│ ──► Handled in ~15 ms
           ▼                                                  ▼
   [ Sender-side latency ≈ 1,015 ms ]               [ APM duration ≈ 15 ms ]

This matters because the delay is short enough to hide inside a retry policy but long enough to cross a hard deadline. Shopify’s five-second response limit and GitHub’s ten-second limit are both reachable after two or three SYN retransmissions.

Measuring the hidden wait

You can measure this on the receiver. The idea is to remember when each client’s first SYN was dropped, then measure the gap when the same client port tries again. That gap is the retransmission wait your APM never sees.

#!/usr/bin/env bpftrace

tracepoint:skb:kfree_skb
/args.protocol == 0x0800/
{
    $skb = (struct sk_buff *)args.skbaddr;
    $iph = (struct iphdr *)($skb->head + $skb->network_header);
    if ($iph->protocol == 6) {
        $tcp = (struct tcphdr *)($skb->head + $skb->transport_header);
        if (bswap($tcp->dest) == 8080 && $tcp->syn == 1 && $tcp->ack == 0) {
            if (@first_drop[$iph->saddr, $tcp->source] == 0) {
                @first_drop[$iph->saddr, $tcp->source] = nsecs;
            }
            @syn_drops = count();
        }
    }
}

kprobe:tcp_v4_conn_request
{
    $skb = (struct sk_buff *)arg1;
    $iph = (struct iphdr *)($skb->head + $skb->network_header);
    $tcp = (struct tcphdr *)($skb->head + $skb->transport_header);
    if (bswap($tcp->dest) == 8080) {
        $t0 = @first_drop[$iph->saddr, $tcp->source];
        if ($t0 != 0) {
            @syn_retry_gap_ms = hist((nsecs - $t0) / 1000000);
            delete(@first_drop[$iph->saddr, $tcp->source]);
        }
    }
}

A histogram with clusters near 1,000 ms, 2,000 ms and 4,000 ms is the signature of SYN retransmission backoff. Bpftrace maps have a bounded size, so for long-running production use, port this logic to a libbpf program with an LRU hash map. Then export the histogram with a tool such as Cloudflare’s ebpf_exporter.

Metrics worth alerting on

You do not need a custom exporter to start. The Prometheus node_exporter netstat collector exposes the relevant kernel counters by default:

node_netstat_TcpExt_ListenOverflows and node_netstat_TcpExt_ListenDrops track accept queue overflows and listener drops.
node_netstat_TcpExt_TCPSynRetrans tracks SYN retransmissions.
node_netstat_TcpExt_SyncookiesSent shows when a SYN queue overflowed and cookies kicked in.

Alert on any sustained increase in ListenDrops. Compare the rate of change to your APM request rate, and a divergence tells you traffic is arriving that your application never sees.

4. Mitigation Patterns: Where eBPF/XDP Helps and Where It Doesn’t

Tracing finds the problem. Fixing it is mostly about how fast your application accepts connections, and secondarily about what the kernel does under load.

4.1 First, fix the queue

Raise the effective backlog. The kernel uses the smaller of the application’s backlog and net.core.somaxconn, so raise both. Frameworks differ: Node.js’s server.listen() defaults to a backlog of 511, and Go’s standard library derives its default from somaxconn.
Know the container caveat. net.core.somaxconn is namespaced per container. In Kubernetes it is classed as an unsafe sysctl, so setting it from a pod’s securityContext.sysctls requires the kubelet to allow it with --allowed-unsafe-sysctls.
Accept fast, process later. Shopify’s own guidance is to delay processing until after you have sent a response. A small receiver that verifies the signature, enqueues the payload to a durable queue (Kafka, SQS) and returns 2xx keeps accept() and response times short, even when downstream work is slow.

4.2 What XDP is good for

XDP is a driver-level eBPF hook. In native mode it runs before sk_buff allocation, so dropping a packet costs very little. In Cloudflare’s benchmark, XDP_DROP discarded roughly 10 million packets per second on a single core. Newer drivers and hardware report substantially higher figures. Dropping at the socket or in the application costs far more per packet, and that gap is why XDP is used for DDoS mitigation at Cloudflare and for L4 load balancing in Meta’s Katran.

That makes XDP a strong fit for removing traffic you never wanted before it can compete with real webhooks for CPU and queue slots. It is a poor fit for rate-limiting legitimate webhook senders.

Do not rate-limit webhook providers per source IP at the packet level. Providers send from a small set of addresses. Stripe, for example, publishes a list of about a dozen webhook IPs, and GitHub publishes hook ranges through its /meta API. A per-IP, per-packet limiter would throttle your most important traffic, including its retransmissions and the data packets of a single request.

A safer use is a blocklist, or the inverse, an allowlist. Drop traffic to your webhook port that does not come from provider-published ranges:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

#define ETH_P_IP 0x0800

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 65536);
    __type(key, __u32);      // IPv4 source address
    __type(value, __u8);
} blocklist SEC(".maps");

SEC("xdp")
int xdp_blocklist(struct xdp_md *ctx)
{
    void *data     = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;
    if (eth->h_proto != bpf_htons(ETH_P_IP))
        return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;
    if (ip->protocol != 6)           // TCP only
        return XDP_PASS;

    __u32 src = ip->saddr;
    if (bpf_map_lookup_elem(&blocklist, &src))
        return XDP_DROP;

    return XDP_PASS;
}

char LICENSE[] SEC("license") = "GPL";

For provider ranges (CIDR blocks rather than single addresses), use a BPF_MAP_TYPE_LPM_TRIE instead of a hash map. A user-space controller can refresh the map from the providers’ published lists.

Three limits to keep in mind:

XDP cannot read TLS-protected HTTP. It sees packet headers, not webhook payloads, endpoints or signatures. Anything that needs the HTTP layer still belongs in your ingress or application.
Native XDP needs driver support. Generic (SKB-mode) XDP works anywhere but runs after sk_buff allocation, losing most of the performance advantage.
XDP_REDIRECT does not forward to another node. BPF_MAP_TYPE_DEVMAP redirects to another network device on the same host, and BPF_MAP_TYPE_CPUMAP redirects to another CPU. Steering overflow traffic to a buffering tier on a different machine needs an encapsulating load balancer (Katran uses IPIP encapsulation) or a normal L4/L7 proxy in front of your workers. It is not something a stock XDP program does on its own.

5. Production Checklist for SREs

Layer	Recommended action	Why it matters
Kernel tuning	Confirm `net.core.somaxconn` (4096 default on 5.4+, 128 before) and `net.ipv4.tcp_max_syn_backlog`. Keep `tcp_syncookies=1` and leave `tcp_abort_on_overflow=0` unless you have measured a reason to change it.	Prevents low queue limits on older kernels from dropping bursts.
Application	Set an explicit listen backlog, and return 2xx quickly. Move processing off the request path.	Keeps `accept()` fast; this is the only fix for the root cause of accept queue overflow.
Kubernetes	Allow and set `net.core.somaxconn` per pod where needed (unsafe sysctl).	The setting is per network namespace, so node-level tuning does not reach pods.
Counters	Alert on `node_netstat_TcpExt_ListenDrops`, `ListenOverflows` and `TCPSynRetrans`.	Cheapest possible signal that the kernel is dropping what the app never sees.
eBPF tracing	Use `skb:kfree_skb` (kernel 5.17+ for drop reasons) and listener-queue kprobes; try BCC/libbpf-tools `tcpdrop`, `tcpretrans`, `tcpsynbl`.	Adds source, port and reason context to the raw counters.
Edge filtering	Use XDP blocklists or provider-range allowlists for webhook ports; do not packet-rate-limit providers.	Removes unwanted traffic cheaply without harming real senders.
Resilience	Reconcile against provider APIs (GitHub’s delivery API, Stripe’s event list), and make handlers idempotent.	Covers deliveries lost without retry, and duplicates created by retry.

Conclusion

Application-layer observability starts after accept(). The drops that matter most during a webhook burst happen before it, in listener queues the application cannot see. eBPF closes that gap: kernel counters tell you something is wrong, kfree_skb drop reasons and listener-queue probes tell you where and why, and a retransmission-gap histogram turns invisible kernel waits into a number you can alert on.

The fix usually lives in the application and its queue sizing, with XDP as an efficient front-line filter for traffic you do not want. Add idempotent handlers and provider-side reconciliation, because providers differ sharply in whether they retry at all.

Sources and Further Reading

Cloudflare, SYN packet handling in the wild
Cloudflare, How to drop 10 million packets per second
Red Hat, How to retrieve packet drop reasons in the Linux kernel
Linux man-pages, listen(2)
IETF, RFC 6298: Computing TCP’s Retransmission Timer
Stripe docs: webhooks and IP addresses
GitHub docs, Handling failed webhook deliveries
Shopify docs, Troubleshoot webhooks
Grafana Labs, OpenTelemetry and Grafana Labs: what’s new in 2026

Originally published at https://instawebhook.com/blog/ebpf-based-webhook-observability-catching-dropped-packets-before-the-application

Debugging Webhooks in Local Dev Environments: Intercepting, Replaying, and Mocking Downstream Failures

InstaWebhook — Thu, 08 Oct 2026 02:12:04 +0000

Event-driven architectures lean heavily on webhooks to connect third-party platforms such as Stripe, Shopify, GitHub, and Twilio to internal backend services. Debugging them locally is still a common source of friction. Providers send HTTP callbacks over the public internet, so they cannot reach a server running on localhost:3000 or 127.0.0.1.

Closing that gap raises follow-up questions:

How do you inspect the exact delivery payload and headers without touching production data?
How do you replay a captured event locally without re-triggering the upstream action?
How do you force your handler through failure paths such as slow downstream calls, 429 Too Many Requests responses, 5xx errors, and duplicate deliveries?

This guide covers how to receive webhooks locally, inspect deliveries, replay them, and simulate failures deterministically. Provider limits quoted below come from each provider's own documentation as of October 2026. They change, so check the current docs before you depend on a specific number.

1. The Local Webhook Debugging Architecture

In production, the provider sends an HTTPS POST to a public ingress route. Locally, you place a tunnel or proxy between the provider and your machine.

[ Upstream Provider ] (e.g., Stripe, Shopify, GitHub)
       │
       │ HTTP POST (event payload + signature headers)
       ▼
[ Public Endpoint / Tunnel / Webhook Gateway ] (e.g., ngrok, Hookdeck, Cloudflare Tunnel)
       │
       ├─► [ Inspection UI ] (headers, body, status, timing)
       │
       │ Forwarded request
       ▼
[ Local Dev Server ] (localhost:3000/api/webhooks)
       │
       └─► [ Mocked or faulty dependencies ] (simulated downstream failures)

Three Common Approaches

Public tunnels. Tools such as ngrok and Cloudflare Tunnel give your local port a public HTTPS URL. ngrok also ships a local request inspector with a replay button (see Section 3). Shopify's CLI uses a Cloudflare tunnel by default when you run shopify app dev, and the tunnel URL changes on every run, so a webhook subscription pointing at an old URL will stop working.
Provider CLIs. Tools such as the Stripe CLI listen to your account's events and forward them to a local endpoint, so no public URL is needed. The Shopify CLI can send a sample payload for a chosen topic to any address.
Capture-and-replay gateways. Services such as Hookdeck sit between the provider and your machine. The Hookdeck CLI (hookdeck listen <port> <source-name>) forwards events to your local server while the gateway keeps them, so you can retry any event on demand. If your local handler returns a non-2xx response, the event stays available for retry.

2. Inspecting Deliveries and Raw Payloads

When a webhook "does nothing", the cause is usually in the details: a signature mismatch, a re-serialized body, or a response that arrived too late. Check these first.

What to inspect	What to look for	Typical failure
Response status	`2xx` returned promptly	Handler throws (`500`), or the provider gives up waiting.
Raw request body	The exact bytes the provider sent	Middleware parsed and re-serialized the JSON, changing the bytes.
Signature headers	`Stripe-Signature`, `X-Shopify-Hmac-SHA256`, `X-Hub-Signature-256` (GitHub)	Wrong secret, wrong body, or clock skew.
Response time	Time until your handler returns	Slow work done before responding.
Delivery IDs	`X-Shopify-Webhook-Id`, `X-GitHub-Delivery`, Stripe's `event.id`	Same ID arriving more than once (retry or redelivery).

Know the Provider Limits

Timeout and retry behavior differs by provider, so test against the one you actually integrate with:

Provider	Documented behavior
Shopify	Expects a response within about 5 seconds, with a 1-second connection timeout. On no response or an error, it retries 8 times over the next 4 hours.
Stripe	Tells you to return a `2xx` before any complex logic that could cause a timeout. In live mode it retries for up to 3 days with exponential backoff. In a sandbox it retries 3 times over a few hours. Stripe does not publish a fixed response-time limit.
GitHub	Expects a `2xx` within 10 seconds. GitHub does not automatically redeliver failed deliveries, but you can redeliver them manually or through the REST API.
Twilio	Connection timeouts and retries are configurable per webhook URL through connection overrides. For call-related requests there is a hard upper timeout of 15 seconds.

Preserve the Raw Body for Signature Verification

A very common local bug is signature verification failing because a body parser such as express.json() ran first. Stripe signs the exact bytes it sent, so a re-serialized body no longer matches. The same applies to Shopify's HMAC, which is computed over the raw request body.

With Stripe's official Node library, use express.raw() on the webhook route only, and register that route before any global JSON parser:

import express from 'express';
import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);
const endpointSecret = process.env.STRIPE_WEBHOOK_SECRET; // whsec_...

const app = express();

// Raw body parser applies to this route only
app.post(
  '/api/webhooks',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signature = req.headers['stripe-signature'];
    let event;

    try {
      // req.body is a Buffer here, which is what verification needs
      event = stripe.webhooks.constructEvent(req.body, signature, endpointSecret);
    } catch (err) {
      console.error(`Signature verification failed: ${err.message}`);
      return res.status(400).send(`Webhook Error: ${err.message}`);
    }

    // Acknowledge first, then do the real work in the background
    res.status(200).json({ received: true });
    processWebhookAsync(event).catch(console.error);
  }
);

// Register JSON parsing for the rest of the app after the webhook route
app.use(express.json());

Two details to remember when testing locally:

Use the right signing secret. When you run stripe listen, the CLI prints a webhook signing secret for that forwarding session. It differs from the secret of an endpoint registered in the Dashboard.
Mind the timestamp tolerance. Stripe's libraries reject signatures whose timestamp is more than five minutes from the current time by default. Large clock drift on your machine makes every verification fail, and so does replaying an old captured Stripe request (see Section 3).

3. Replaying Webhooks Locally Without Upstream Triggers

Generating a real event, such as completing a test checkout, is slow and hard to repeat. Replaying a captured event is faster and deterministic.

Method 1: Replay from the ngrok Inspector

Start a tunnel:

   ngrok http 3000

Open the agent's web inspection interface at http://127.0.0.1:4040 (or http://localhost:4040).
Select a captured webhook request.
Click Replay in the top-right corner of the request to resend it to your local server. Use the dropdown on the Replay button to Replay with modifications, which lets you edit the method, path, headers, trailers, and body first.

ngrok also offers a cloud-based Traffic Inspector in its dashboard, which adds longer retention and search. Its replay feature works only for fully captured, non-truncated requests, and it adds an Ngrok-Replay-Original-Request-ID header to the replayed request.

Signature caveat: a replay resends the original request, including its original signature headers. Shopify's HMAC has no timestamp, so a replay still verifies. A Stripe replay older than the tolerance window will fail timestamp validation. When replaying older Stripe events, either use the Stripe CLI (below) or temporarily widen the tolerance in your development configuration only.

Method 2: Use the Provider's CLI

Stripe CLI. Forward events to your local endpoint and trigger test events on demand:

# Terminal 1: forward events and print the signing secret for this session
stripe listen --forward-to localhost:3000/api/webhooks

# Terminal 2: create a test event
stripe trigger payment_intent.succeeded

You can also run stripe listen --load-from-webhooks-api --forward-to localhost:4242/webhook to forward to your local server using the configuration of the endpoints you've registered. To re-send an event that already exists, use stripe events resend <event_id> --webhook-endpoint=<endpoint_id>. Stripe's documentation says manual retries from the CLI work for events up to 30 days old, and retries from the Dashboard work for up to 15 days.

Shopify CLI. shopify app webhook trigger sends a sample payload for an Admin API topic to an address you choose. Shopify says to use it for experimentation, initial configuration checks, and unit testing, and to verify the real end-to-end flow by performing the related action in Shopify. For local HTTP testing, use an address like http://localhost:{port}/{path}. If you target an ngrok URL, leave the local port out of it:

shopify app webhook trigger \
  --topic=orders/create \
  --address=http://localhost:3000/webhooks \
  --delivery-method=http

Samples produced this way are not real store events, and in the Shopify Remix app template the admin object is undefined for CLI-triggered webhooks.

GitHub. In a repository, organization, or app webhook's settings you can inspect recent deliveries and redeliver them. Because GitHub doesn't redeliver automatically, redelivery is also your recovery tool after your local server was down.

Hookdeck CLI. Run hookdeck listen 3000 <source-name> to forward events to your local server. In the CLI's interactive view, press r to retry the selected event, or use Retry in the Hookdeck dashboard.

Method 3: Replay a Saved Payload with cURL

For offline work or fully scripted tests, save a captured body and its headers to disk and replay them yourself:

curl -X POST http://localhost:3000/api/webhooks \
  -H "Content-Type: application/json" \
  -H "X-Shopify-Topic: orders/create" \
  -H "X-Shopify-Hmac-SHA256: <valid_hmac_for_this_exact_body>" \
  --data-binary @payloads/order_created_sample.json

Use --data-binary rather than --data so the file's bytes are sent unchanged. Newline handling in --data can alter the body and break HMAC verification. If the HMAC isn't valid for the saved body, your handler should reject the request, so either regenerate the signature with your app's secret or enable a dev-only verification bypass.

4. Mocking Downstream Failures: Timeouts, Rate Limits, and Duplicates

Making the happy path return 200 OK is only part of the job. Your handler also has to behave when something it depends on is slow, rate-limited, or down, and when the provider sends the same event more than once.

[ Inbound Webhook ] ──► [ Local Ingress Handler ]
                                  │
                 ┌────────────────┴────────────────┐
                 ▼                                 ▼
        [ Idempotency Guard ]            [ Processing Pipeline ]
        - duplicate event check          - queue / workers / database
        - skip already-processed         - calls to downstream APIs
                                                   │
                                                   ▼
                                       [ Fault Injection Layer ]
                                       - HTTP 429 with Retry-After
                                       - added latency / timeouts
                                       - 5xx errors, dropped connections

Scenario 1: Slow Handlers and Provider Timeouts

If your handler does heavy work before responding (image processing, email, multi-table writes), the provider may treat the delivery as failed and retry it. Shopify allows about 5 seconds, GitHub 10. The fix is to acknowledge quickly and process in the background:

app.post('/api/webhooks/orders', express.raw({ type: 'application/json' }), async (req, res) => {
  // ...verify the signature against req.body first...
  const event = JSON.parse(req.body.toString('utf8'));

  // Enqueue and return immediately
  await taskQueue.add('processWebhook', { eventId: event.id, payload: event });
  res.status(202).json({ status: 'queued', id: event.id });
});

To check that your system is correct when the worker is slow, inject an artificial delay on the processing side and confirm the HTTP response still returns quickly:

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function handleWebhookJob(job) {
  if (process.env.SIMULATE_SLOW_DOWNSTREAM === 'true') {
    await sleep(6000); // longer than Shopify's ~5 s limit, to prove the ack doesn't wait for it
  }
  await updateDatabase(job.data);
}

To test the opposite case, where the provider really does time out, temporarily make the handler wait longer than the limit before it responds. Then watch how the provider reports the failure and retries. With Stripe's sandbox this happens a few times over a few hours. With Shopify it follows its 8-retry schedule, and with GitHub it does not retry at all.

For network-level faults (added latency, dropped connections, hard timeouts) between your app and a dependency such as Redis or a database, Shopify's open-source Toxiproxy is a TCP proxy that can inject "latency" and "timeout" toxics, among others, and it has client libraries for several languages.

Scenario 2: Downstream `429 Too Many Requests`

When a webhook makes your server call another API, bursts of webhooks can trigger rate limiting there. Use an HTTP interceptor such as Nock (or MSW) to return a 429 with a Retry-After header and confirm your code backs off:

import nock from 'nock';

nock('https://api.internal-service.com')
  .post('/v1/sync')
  .reply(
    429,
    { error: 'rate_limit_exceeded', message: 'Too many requests. Retry after 30 seconds.' },
    { 'Retry-After': '30' }
  );

Interceptors like this only affect requests made from the same Node process, so they suit integration tests. They won't affect a separate worker process. For those, run a small stub server that returns the failure and point the worker's base URL at it.

Then assert the behavior you want: the job is rescheduled using the Retry-After value rather than retried in a tight loop, and the webhook itself was already acknowledged.

Scenario 3: Duplicate Deliveries and Idempotency

Providers can deliver the same event more than once, so handlers must tolerate duplicates. Stripe notes that endpoints might occasionally receive the same event more than once, and Shopify says the same after a network timeout or retry. Shopify recommends the X-Shopify-Webhook-Id header for detecting duplicate deliveries. It also sends an X-Shopify-Event-Id header that stays the same across deliveries originating from the same merchant action, so you can use that one to correlate them. For Stripe, use event.id. For GitHub, use the X-GitHub-Delivery header.

Here is a Redis-based guard you can exercise locally. The first caller claims the event, duplicates are skipped, and a failure releases the claim so a retry can try again:

import Redis from 'ioredis';
const redis = new Redis();

const CLAIM_TTL_SECONDS = 24 * 60 * 60;

async function processIdempotentWebhook(eventId, payload) {
  const key = `webhook:event:${eventId}`;

  // SET ... NX returns 'OK' only if the key did not already exist
  const claimed = await redis.set(key, 'processing', 'EX', CLAIM_TTL_SECONDS, 'NX');

  if (claimed !== 'OK') {
    console.log(`[Idempotency] Duplicate event skipped: ${eventId}`);
    return { status: 'ignored', reason: 'duplicate_event' };
  }

  try {
    await executeOrderFulfillment(payload);
    await redis.set(key, 'completed', 'EX', CLAIM_TTL_SECONDS);
    return { status: 'success' };
  } catch (error) {
    // Release the claim so a later retry can reprocess the event
    await redis.del(key);
    throw error;
  }
}

Choose the retention period to cover the provider's retry window. Stripe retries for up to 3 days in live mode, so a 24-hour claim does not cover that window. In production, a durable database table with a unique constraint on the event ID is a safer record than a cache key alone.

Test it:

Replay a webhook locally.
Confirm the database was updated once and the log shows success.
Replay the exact same request again, with the same event ID.
Confirm the log shows Duplicate event skipped and no second record exists.
Make executeOrderFulfillment throw once, replay, and confirm the claim was released and the next replay succeeds.

Choose Your Error Responses Deliberately

What your endpoint returns controls what the provider does next. A non-2xx response, including a 400, generally causes Stripe and Shopify to retry the delivery. Return 2xx once you have safely accepted the event, even if downstream processing will happen later or may fail. Return a non-2xx code when you want the provider to try again. Reserve 4xx responses for requests you genuinely want rejected, such as failed signature verification. Handle invalid business payloads inside your own queue and dead-letter process instead of relying on the provider's retries.

5. Local Webhook Debugging Checklist

Signature verification

The raw body reaches the verification code, and the webhook route is registered before any global JSON parser.
The signing secret comes from an environment variable and matches the source of the events (for example, the secret printed by stripe listen).
Your system clock is accurate. Stripe's default tolerance is five minutes.

Transport and delivery

The tunnel or CLI is running, and the registered URL matches the current one (Shopify's CLI tunnel URL changes on each run).
The handler responds well inside the shortest provider limit you support (about 5 seconds for Shopify, 10 seconds for GitHub).
Heavy work runs in a background queue, not in the request handler.

Resilience and idempotency

Duplicate deliveries are detected using the provider's delivery or event ID.
Transient downstream failures (429, 502, 503, timeouts) are retried with backoff inside your own job system.
You have tested a failed delivery and its retry or redelivery on each provider you rely on.

Conclusion

Local webhook debugging works best when you can reproduce a problem on demand. Capture real payloads and headers, verify signatures against the raw body, replay events through your tunnel, provider CLI, or a capture gateway, and inject slow, rate-limited, failing, and duplicate scenarios deliberately. Because each provider has its own timeout and retry rules, test against the specific limits of the platforms you integrate with, and confirm them in the provider's current documentation before relying on them.

Originally published at https://instawebhook.com/blog/debugging-webhooks-in-local-dev-environments-intercepting-replaying-and-mocking

Modern Webhook Engineering: Beyond Tunnels, Race Conditions, and Flaky Pipelines

InstaWebhook — Wed, 07 Oct 2026 05:49:37 +0000

Webhooks are the standard way for payment providers, SaaS platforms, and job systems to tell your application that something happened. Reliable ingestion matters, but the testing and local-development workflows around webhooks are often fragile.

Public tunnels such as ngrok are excellent for a quick demo. They are a poor foundation for automated pipelines, because they add an external dependency, a public ingress path, and usage quotas to every run. Replaying production-shaped payloads against a local container causes a different class of problems: state pollution, stale signatures, and races with hot reloads.

This guide covers three things:

Deterministic webhook testing in CI without tunnels.
Correct HMAC signing and verification in tests.
Isolated replay and queue-based processing in local development.

Part 1: Beyond Localhost Tunnels: Testing Webhooks in CI/CD

Where tunnels hurt in automation

A tunnel exposes a local port through a public URL, which is exactly what you want when a vendor's sandbox must reach your laptop. In CI the same property causes problems:

Quotas and rate limits. ngrok's free plan currently allows up to 20,000 HTTP requests per month, 1 GB of outbound data transfer, and a single assigned dev domain. A busy pipeline can exhaust these, and a quota failure looks like a flaky test. Limits change, so check the current ngrok limits.
Shared, non-ephemeral URLs. With one fixed dev domain, parallel jobs would compete for the same public endpoint. Per-job URLs generally require a paid plan.
External availability. Every run now depends on the tunnel provider's uptime and on routing between the provider and your runner.
Public ingress to a build environment. Anything reachable from the internet can be probed. A webhook endpoint that verifies signatures is still an unauthenticated public surface. CI runners are often less hardened than production hosts, and you rarely need that exposure.
Time-sensitive signatures. Many providers sign a timestamp together with the body and reject old requests. Stripe's libraries, for example, reject signatures whose timestamp is more than five minutes old by default. Slow or stalled pipelines can trip such windows, and replaying captured payloads later will always trip them.

None of this makes tunnels wrong. It means they belong in manual, exploratory work, not in the automated test path.

Shift the webhook source inside the test network

In CI you do not need a vendor to call you. You need something that produces a correctly formed, correctly signed request. Run that producer inside the same network as the application under test:

+----------------------------------------------------------------------+
| CI job / local Docker network                                        |
|                                                                      |
|  +-------------------+   signed POST    +-----------------------+    |
|  | Test runner or    | ---------------> | Webhook receiver      |    |
|  | webhook sender    |                  | (application)         |    |
|  +-------------------+                  +-----------+-----------+    |
|            |                                        |                |
|            | asserts on state                       | writes         |
|            v                                        v                |
|  +-------------------+                  +-----------------------+    |
|  | Assertions        | <--------------- | Test DB / Redis       |    |
|  +-------------------+     reads        +-----------------------+    |
+----------------------------------------------------------------------+

Three tools fit this role:

WireMock mocks the vendor's HTTP API and can send callbacks. From version 3.1.0, webhook support is part of WireMock core and is enabled by default. Before 3.1.0 it required a separate extension.
Prism is a mock server driven by an OpenAPI description. It is useful for mocking the vendor's request/response API, not for signing callbacks.
Vendor CLIs such as the Stripe CLI can forward real sandbox events to a local endpoint (stripe listen --forward-to ...) and fire test events (stripe trigger ...). This still depends on network access to the vendor, so it suits local development more than hermetic CI.

Signing and verifying correctly

Most of the "my test signature is rejected" bugs come from one mistake: signatures are computed over the raw request bytes, not over a re-serialized JSON object. If your framework parses the body and you re-stringify it, whitespace and key order can differ from what was signed. Capture the raw body and verify against that.

An Express receiver that does this:

// src/app.ts
import express from 'express';
import crypto from 'crypto';

export const app = express();

const WEBHOOK_SECRET = process.env.WEBHOOK_SECRET ?? '';

// Keep the raw bytes for signature verification.
app.use(
  express.json({
    verify: (req, _res, buf) => {
      (req as any).rawBody = buf;
    },
  })
);

function verifySignature(rawBody: Buffer, header: string | undefined): boolean {
  if (!header) return false;
  const expected = crypto
    .createHmac('sha256', WEBHOOK_SECRET)
    .update(rawBody)
    .digest('hex');

  const a = Buffer.from(expected);
  const b = Buffer.from(header);
  // timingSafeEqual throws if lengths differ, so compare lengths first.
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

app.post('/api/v1/webhooks/payment', (req, res) => {
  const ok = verifySignature((req as any).rawBody, req.header('X-Signature-256'));
  if (!ok) return res.status(401).json({ error: 'invalid signature' });

  // Enqueue or process the event here (see Part 2).
  return res.status(200).json({ received: true });
});

This example uses a simple scheme (HMAC-SHA256 of the body, hex-encoded, in a custom header). Real providers differ. Stripe, for instance, signs timestamp.body and sends it in a Stripe-Signature header containing the timestamp and one or more v1 signatures. Always follow your provider's documented scheme and prefer its official verification helper, such as stripe.webhooks.constructEvent, when one exists.

In-process tests: the fastest feedback loop

For most suites, skip the network and containers for the sender entirely. Call the app in-process with supertest and sign the exact string you send:

// test/integration/webhook.spec.ts
import crypto from 'crypto';
import request from 'supertest';
import { app } from '../../src/app';

const SECRET = process.env.WEBHOOK_SECRET ?? 'ci_secret_key_12345';

function sign(rawBody: string, secret: string): string {
  return crypto.createHmac('sha256', secret).update(rawBody).digest('hex');
}

describe('payment webhook', () => {
  it('accepts a correctly signed event', async () => {
    const body = JSON.stringify({
      event: 'payment.succeeded',
      id: `evt_test_${crypto.randomUUID()}`,
      amount: 2500,
      currency: 'usd',
    });

    const res = await request(app)
      .post('/api/v1/webhooks/payment')
      .set('Content-Type', 'application/json')
      .set('X-Signature-256', sign(body, SECRET))
      .send(body); // send the same string that was signed

    expect(res.status).toBe(200);
    expect(res.body).toEqual({ received: true });
  });

  it('rejects a tampered body', async () => {
    const body = JSON.stringify({ event: 'payment.succeeded', id: 'evt_x', amount: 1 });
    const res = await request(app)
      .post('/api/v1/webhooks/payment')
      .set('Content-Type', 'application/json')
      .set('X-Signature-256', sign(body, SECRET))
      .send(body.replace('"amount":1', '"amount":999999'));

    expect(res.status).toBe(401);
  });
});

Two details make this deterministic. The ID is a random UUID rather than a timestamp, so parallel tests never collide. The signed string and the sent string are the same value, so no serialization difference can break verification.

Also add tests for the failure modes that matter in production: duplicate delivery of the same event ID (the handler must be idempotent), out-of-order events, and a timestamp outside your tolerance window if your scheme uses one.

Containerized mock: WireMock as the vendor

Use a containerized mock when you need to test a flow in which your application calls the vendor API and the vendor later calls back. Define it in docker-compose.ci.yml:

services:
  app-receiver:
    build: .
    environment:
      NODE_ENV: test
      WEBHOOK_SECRET: ci_secret_key_12345
      DATABASE_URL: postgres://user:pass@postgres:5432/testdb
      VENDOR_API_URL: http://mock-webhook-provider:8080
    depends_on:
      postgres:
        condition: service_healthy
      mock-webhook-provider:
        condition: service_started

  mock-webhook-provider:
    image: wiremock/wiremock:3.13.2
    volumes:
      - ./wiremock/mappings:/home/wiremock/mappings

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
      POSTGRES_DB: testdb
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d testdb"]
      interval: 2s
      timeout: 3s
      retries: 15

A few notes on this file:

The top-level version: key is obsolete in current Docker Compose and can be omitted.
No ports are published, so nothing is exposed on the host. Containers on the same Compose network reach each other by service name.
Pin the WireMock image tag. At the time of writing, 3.13.2 is the 3.x release shown on the WireMock download page. A WireMock 4.x beta line also exists, but it is labeled beta and expected to contain breaking changes.

WireMock fires the callback through a serve event listener named webhook. Put this in ./wiremock/mappings/payment_success.json:

{
  "request": {
    "method": "POST",
    "url": "/mock-vendor/checkout"
  },
  "response": {
    "status": 200,
    "headers": { "Content-Type": "application/json" },
    "jsonBody": {
      "status": "initiated",
      "checkout_id": "chk_99887766"
    }
  },
  "serveEventListeners": [
    {
      "name": "webhook",
      "parameters": {
        "method": "POST",
        "url": "http://app-receiver:3000/api/v1/webhooks/payment",
        "headers": {
          "Content-Type": "application/json",
          "X-Signature-256": "14caa87134ad8255a2b2337f38d95738842e4b35021cff21d664203cbccdfe29"
        },
        "body": "{\"event\":\"payment.succeeded\",\"id\":\"evt_123\",\"amount\":4999,\"customer_id\":\"cust_abc\"}"
      }
    }
  ]
}

Several points matter here:

WireMock has no built-in HMAC helper. Its templating ships many helpers (JSON/XML, dates, random values, base64, and others), but none that computes an HMAC. Webhook parameters can be templated, but a signature has to come from somewhere else.
The signature above is precomputed. It is the HMAC-SHA256 of the exact body string using the secret ci_secret_key_12345. You can reproduce it with printf '%s' '<body>' | openssl dgst -sha256 -hmac 'ci_secret_key_12345'.
This only works for schemes that sign the body alone. If your provider signs a timestamp too, a static signature cannot work. In that case generate the request in your test code (as in the in-process example) or register a custom WireMock template helper through its extension mechanism.
Older tutorials differ. Many online examples use postServeActions and the separate webhooks extension from WireMock 2.x. The current documentation uses serveEventListeners.
Template data comes from originalRequest. If you template the callback from the triggering request, reference originalRequest (for example {{jsonPath originalRequest.body '$.transactionId'}}), not request.
Callbacks are asynchronous. WireMock fires them after the triggering response, so tests should poll or wait for the expected state rather than asserting immediately.

Part 2: Developing Webhook Receivers with Hot Reload

Local development brings its own problems. You run the receiver under a watcher such as nodemon (Node.js), air (Go), or next dev, and you want realistic payloads. Replaying captured production payloads causes three recurring problems:

State pollution. Payloads carry real IDs and keys. Replaying them inserts foreign records into your local database, collides with unique constraints, or leaves orphaned rows after a restart.
Restarts mid-request. If a file change restarts the process while a delivery is in flight, that request fails or half-completes.
Signature mismatches. A captured payload has a signature (and possibly a timestamp) computed for the original secret and time. It will not verify against your local secret, and a time-bound scheme rejects it as stale.

Replaying safely: a local sanitizer

A local relay can rewrite a captured payload so it is safe to replay: namespace the IDs, refresh timestamps, and sign with the local secret. The version below fails safe. It runs only when explicitly enabled, instead of whenever NODE_ENV is not production, so a misconfigured environment cannot silently accept unsigned traffic.

captured payload                relay / sanitizer                  local receiver
+---------------------+     +---------------------------+     +----------------------+
| id: evt_prod_99     | --> | 1. namespace the ID       | --> | verifies with local  |
| created: 1600000000 |     | 2. refresh timestamp      |     | secret, writes to    |
| (old signature)     |     | 3. re-sign with dev secret|     | local DB only        |
+---------------------+     +---------------------------+     +----------------------+

The simplest design is a standalone replay script rather than middleware inside the receiver. That keeps test-only code out of the production code path and lets the receiver verify signatures exactly as it does in production:

// scripts/replay-webhook.ts
import crypto from 'crypto';
import fs from 'fs';

const [, , file, url = 'http://localhost:3000/api/v1/webhooks/payment'] = process.argv;
const secret = process.env.WEBHOOK_SECRET;
if (!file || !secret) {
  console.error('usage: WEBHOOK_SECRET=... ts-node replay-webhook.ts <payload.json> [url]');
  process.exit(1);
}

const runId = process.env.DEV_RUN_ID ?? 'local_dev';
const payload = JSON.parse(fs.readFileSync(file, 'utf8'));

// 1. Namespace the ID so replays never collide with real or previous data.
payload.original_id = payload.id;
payload.id = `${runId}_${payload.id}_${Date.now()}`;

// 2. Refresh timestamps so freshness checks pass.
if (payload.created) payload.created = Math.floor(Date.now() / 1000);

// 3. Sign the exact bytes we send, using the local dev secret.
const body = JSON.stringify(payload);
const signature = crypto.createHmac('sha256', secret).update(body).digest('hex');

const res = await fetch(url, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', 'X-Signature-256': signature },
  body,
});
console.log(res.status, await res.text());

Two cautions apply to any replay tooling:

Scrub personal data. Production payloads usually contain customer data. Redact or synthesize it before it lands on a laptop or in a repository.
Never copy production secrets locally. Use a dedicated dev secret. If you must receive real vendor events locally, use the vendor's test mode and its CLI forwarder.

Surviving hot reloads: acknowledge fast, process in a worker

When a watcher restarts your process, any work still running in the HTTP handler is lost. The standard fix is the same one used in production: verify the signature, persist or enqueue the event, return a 2xx quickly, and do the real work in a separate worker. Most providers retry on timeouts or non-2xx responses, and Stripe documents that you should return a 2xx quickly and that it may deliver the same event more than once. So your processing must be idempotent regardless.

Using BullMQ (a Redis-backed queue for Node.js), use the event ID as the job ID. BullMQ ignores a second job added with a job ID that already exists, which gives you cheap deduplication:

// src/queue/webhookQueue.ts
import { Queue } from 'bullmq';
import IORedis from 'ioredis';

// BullMQ workers require maxRetriesPerRequest to be null on their connection.
export const connection = new IORedis(process.env.REDIS_URL ?? 'redis://localhost:6379', {
  maxRetriesPerRequest: null,
});

export const webhookQueue = new Queue('incoming-webhooks', { connection });

export async function enqueueWebhook(event: { id: string; type: string; data: unknown }) {
  // Re-adding the same jobId is ignored, so duplicate deliveries collapse into one job.
  await webhookQueue.add(event.type, event, {
    jobId: event.id,
    attempts: 5,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: 1000,
    removeOnFail: 5000,
  });
}

// src/queue/webhookWorker.ts
import { Worker } from 'bullmq';
import { connection } from './webhookQueue';

const worker = new Worker(
  'incoming-webhooks',
  async (job) => {
    await handleWebhookBusinessLogic(job.name, job.data); // your logic, written to be idempotent
  },
  { connection, concurrency: 1 } // serialize work locally to avoid races on a shared dev DB
);

async function shutdown() {
  // close() waits for jobs that are currently running to finish.
  await worker.close();
  await connection.quit();
  process.exit(0);
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

A few operational notes:

Setting concurrency: 1 is a local-development convenience. In production you will usually run higher concurrency and rely on idempotent handlers and database constraints instead.
Job IDs deduplicate only while the job still exists in Redis. Keep a durable record of processed event IDs (for example, a unique constraint in your database) if you need long-term idempotency.
Docker sends SIGTERM when a container stops, followed by SIGKILL after a grace period. Make sure your process receives the signal (running it directly, or using init: true in Compose) and set stop_grace_period long enough for in-flight jobs to finish.

Approach Comparison

Approach	Determinism	Exposure	Network dependency	State isolation	Setup effort
Public tunnel (ngrok, etc.)	Low	Public ingress to a local port	High (provider and internet)	Poor unless you add it	Very low
WireMock in the CI network	High	Low (internal network only)	None at test time	Good (fresh containers per run)	Moderate
In-process integration tests	High	None	None	Excellent (per-test data)	Low
Replay script plus queue	High	Low	None for local replays	Good (namespaced IDs)	Moderate
Vendor CLI forwarding (e.g. `stripe listen`)	Medium	Low (outbound connection)	Yes (vendor sandbox)	Depends on your setup	Low

Summary and Best Practices

Keep tunnels out of automated pipelines. Use them for manual exploration and vendor sandbox demos. For CI, produce the webhook inside the test network.
Sign and verify the raw bytes. Capture the raw request body for verification, compare signatures in constant time, and follow your provider's exact scheme, including timestamp tolerance.
Start with in-process tests. They are the fastest and most deterministic option. Add a WireMock layer when you need to test vendor API calls and callbacks end to end.
Know WireMock's limits. Use serveEventListeners on WireMock 3.1 and later, template from originalRequest, and remember there is no built-in HMAC helper.
Sanitize replays, and make it opt-in. Namespace IDs, refresh timestamps, re-sign with a dev secret, and scrub personal data. Never copy production secrets to a development machine.
Acknowledge fast, process asynchronously, and be idempotent. Deduplicate by event ID, return a 2xx quickly, and let a worker with graceful shutdown do the work so hot reloads do not lose events.

References

Originally published at https://instawebhook.com/blog/modern-webhook-engineering-beyond-tunnels-race-conditions-and-flaky-pipelines

High-Throughput Webhook Systems: Delivery Timelines and Split-Brain Prevention

InstaWebhook — Tue, 06 Oct 2026 03:12:46 +0000

Webhooks are the standard way to integrate systems asynchronously. Scaling a delivery platform to thousands of requests per second exposes two infrastructure problems:

Debugging the "black box": Most webhook logs record only Received and Delivered. When a transient failure or partition occurs, engineers can't see the intermediate states (encrypted, leased, attempted, retried, dead-lettered).
Duplicate processing after lease expiry: In large worker clusters, a stalled thread or latency spike can let a lease expire while the original worker is still alive. A second worker then claims the job, and the receiver may see the same event twice.

This guide covers a state-machine design for full-lifecycle visibility. It also covers lease-and-fencing patterns that keep your own state consistent, and the part fencing cannot solve: duplicates seen by the receiver.

1. Full-Lifecycle Timelines for Asynchronous Webhook Jobs

The limits of `Received` vs. `Delivered` logging

A basic setup writes to a central log sink (Elasticsearch, CloudWatch, Loki) at two points:

When the payload reaches the ingest API (Received).
When the receiver answers with an HTTP 2xx status (Delivered).

This works at low volume but breaks down at scale:

Unaccounted queue latency: If an event sits for 30 seconds before a worker picks it up, two-point logging can't show whether the cause was queue lag, database lock contention, or worker starvation.
Opaque retry behavior: After a 503, a retry is scheduled. Without state transitions, operators can't tell a job sleeping in backoff from one whose worker crashed.
Audit gaps: If payloads contain sensitive data, you may need to show that encryption happened before storage. Without structured state events, that means correlating logs across several services.

Retry windows at the sending side are long, so visibility matters. Stripe, for example, attempts to deliver a given event to your webhook endpoint for up to 3 days with an exponential back off in live mode, while test mode retries three times over a few hours. A job can stay in flight for days, and operators need to see where it stands. stripe

A state-machine-driven delivery pipeline

Model each delivery as a finite state machine (FSM). Every transition writes an immutable, append-only audit record with the state, worker identity, timestamp, and relevant metadata.

 RECEIVED ──► ENCRYPTED ──► LEASED ──► ATTEMPTED ──► DELIVERED
                              ▲            │
                              │            ├── retryable failure ──► RETRIED ──┐
                              └────────────┼───────────────────────────────────┘
                                           │   (re-queued, then leased again)
                                           └── retries exhausted / non-retryable ──► DLQ

State descriptions

State	Scope	Trigger
`RECEIVED`	Ingestion tier	Payload validated and durably persisted.
`ENCRYPTED`	Security / KMS layer	Payload encrypted (for example with AES-256-GCM) before it is placed in queue storage.
`LEASED`	Queue / worker pool	A worker claims the job via a lease with a fixed time-to-live (TTL).
`ATTEMPTED`	Outbound transport	Worker sends the HTTP POST; status code and latency are recorded.
`RETRIED`	Scheduler	Non-2xx response or timeout; next attempt time computed with exponential backoff and jitter.
`DELIVERED`	Terminal	Receiver returned a `2xx` response.
`DLQ`	Dead-letter storage	Retry budget exhausted, or a non-retryable error such as most `4xx` responses.

Treat 408 and 429 as retryable rather than as permanent client errors.

Jitter matters. Exponential backoff alone reduces call volume but does not break up synchronized retry clusters, which is why production systems add randomness to the delay.

State transition schema

Store delivery metadata separately from event payloads, so the hot path touches small rows and payload storage stays isolated. The schema below includes the lease columns used later in this guide.

CREATE TABLE webhook_deliveries (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    event_id         UUID NOT NULL REFERENCES webhook_events(id),
    subscription_id  UUID NOT NULL,
    current_state    VARCHAR(32) NOT NULL,
    attempt_count    INT NOT NULL DEFAULT 0,
    max_attempts     INT NOT NULL DEFAULT 5,
    next_attempt_at  TIMESTAMPTZ,
    worker_id        VARCHAR(64),
    lease_expires_at TIMESTAMPTZ,
    fencing_token    BIGINT NOT NULL DEFAULT 0,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

-- Append-only audit trail: one row per state transition
CREATE TABLE webhook_delivery_attempts (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    delivery_id      UUID NOT NULL REFERENCES webhook_deliveries(id),
    state            VARCHAR(32) NOT NULL,
    worker_id        VARCHAR(64),
    fencing_token    BIGINT,
    http_status_code INT,
    latency_ms       INT,
    error_message    TEXT,
    created_at       TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

CREATE INDEX idx_deliveries_claimable
    ON webhook_deliveries (next_attempt_at)
    WHERE current_state IN ('ENCRYPTED', 'RETRIED', 'LEASED');

CREATE INDEX idx_attempts_delivery_time
    ON webhook_delivery_attempts (delivery_id, created_at);

On PostgreSQL, gen_random_uuid() is built in from version 13 (earlier versions need the pgcrypto extension). clock_timestamp() returns the actual current time, unlike now(), which is frozen at transaction start. Use it for lease arithmetic. If you run multiple database nodes, prefer the database's clock over worker clocks for all lease comparisons.

To let many workers claim jobs without blocking each other, combine the claim query with FOR UPDATE SKIP LOCKED. Concurrent workers then skip rows another worker has already locked instead of waiting on them.

End-to-end timeline view

With one record per transition, an SRE can open a single delivery and read its history instead of grepping log streams:

[00:00.000] RECEIVED  - Payload ingested via gateway node gw-us-east-1a
[00:00.004] ENCRYPTED - Encrypted via KMS (key id: ...8f2a)
[00:00.012] LEASED    - Worker wrk-402 claimed lease (30 s), token=1
[00:00.015] ATTEMPTED - POST https://api.client.example/webhook
[00:00.120] FAILED    - Endpoint returned HTTP 503 (latency 105 ms)
[00:00.122] RETRIED   - Rescheduled for +15 s (exponential backoff + jitter)
[00:15.125] LEASED    - Worker wrk-109 claimed lease, token=2
[00:15.128] ATTEMPTED - POST https://api.client.example/webhook
[00:15.210] DELIVERED - HTTP 200 OK (latency 82 ms)

Useful derived metrics from this table include time-in-state percentiles (queue wait is LEASED minus ENCRYPTED), retry rate per subscription, DLQ inflow, and lease-expiry counts. As one webhook-reliability guide notes, a single failed delivery is expected in an at-least-once system, so alert on sustained increases in failure rate, retry rate, queue depth, or dead-letter volume rather than on isolated events.

Build vs. buy

Visual timelines, FSM tracking, and query pipelines are real engineering work. Managed options include Hookdeck, Svix, the open-source and self-hostable Convoy, and relay services such as InstaWebhook. Hookdeck's Event Gateway, for example, provides ingestion, signature verification, deduplication, retries, and replay. Whichever you evaluate, check that it exposes per-attempt history, replay, and DLQ inspection. Also check how it handles duplicate delivery, since that determines how much work stays on your side. hookdeck

2. Preventing Split-Brain Processing: Heartbeats, Leases, and Fencing

When many workers pull from shared queues, you must decide what happens when two workers both believe they own the same job.

Root cause: lease expiry during a pause

Worker A                      Lease store (database)                 Worker B
   |--- 1. Acquire lease (TTL 10 s, token=1) -->|                        |
   |                                            |                        |
   | [GC pause / VM stall / network partition]  |                        |
   |                                            |<-- 2. Lease expired ---|
   |                                            |<-- 3. Acquire (token=2)|
   |                                            |                        |--> 5. POST to receiver
   |--- 4. Resume; still believes it owns it -->|                        |
   |        POST to receiver                    |                        |

Worker A claims a delivery with a 10-second lease.
Worker A stalls, for example in a long garbage-collection pause or a partition.
The lease expires without renewal. The system assumes Worker A is dead and makes the job claimable.
Worker B claims it and sends the HTTP request.
Worker A wakes up, unaware its lease is gone, and sends the same request.

The receiver now sees two requests for one event. This is the failure mode described in Martin Kleppmann's analysis of distributed locking. He calls leases a flawed basis for correctness unless every write is checked with a fencing token, a number that increases with each lock acquisition, which the storage service uses to reject stale writers. dnitza

Fencing tokens and atomic lease renewal

1. Acquire the lease and increment the token atomically

UPDATE webhook_deliveries
SET current_state    = 'LEASED',
    worker_id        = 'worker-node-102',
    lease_expires_at = clock_timestamp() + INTERVAL '10 seconds',
    fencing_token    = fencing_token + 1,
    updated_at       = clock_timestamp()
WHERE id = 'd82c4f74-32e7-4b11-9a99-923f6e9112a2'
  AND (
        current_state IN ('ENCRYPTED', 'RETRIED')
        OR (current_state = 'LEASED' AND lease_expires_at < clock_timestamp())
      )
RETURNING fencing_token;

If the query returns no row, another worker holds a live lease and this worker must walk away. Because the increment and the claim happen in one statement, tokens are strictly increasing per delivery.

2. Heartbeat loop

While a job runs, a background thread extends the lease. The WHERE clause checks both the worker ID and the token, so a stale worker cannot renew a lease it has lost.

import threading

class WorkerHeartbeat:
    def __init__(self, db, delivery_id, worker_id, fencing_token,
                 interval_sec=3, lease_sec=10):
        self.db = db
        self.delivery_id = delivery_id
        self.worker_id = worker_id
        self.fencing_token = fencing_token
        self.interval = interval_sec
        self.lease_sec = lease_sec
        self._stop = threading.Event()
        self.lease_lost = threading.Event()   # checked by the main work loop
        self._thread = threading.Thread(target=self._loop, daemon=True)

    def start(self):
        self._thread.start()

    def stop(self):
        self._stop.set()
        self._thread.join()

    def _loop(self):
        # Event.wait returns True when stop is set, ending the loop promptly
        while not self._stop.wait(self.interval):
            rows = self.db.execute(
                """
                UPDATE webhook_deliveries
                SET lease_expires_at = clock_timestamp() + make_interval(secs => %s)
                WHERE id = %s
                  AND worker_id = %s
                  AND fencing_token = %s
                  AND current_state = 'LEASED'
                """,
                (self.lease_sec, self.delivery_id, self.worker_id, self.fencing_token),
            ).rowcount

            if rows == 0:
                # Lease lost or reassigned. Signal the worker; do not just raise here.
                self.lease_lost.set()
                break

Two details matter. First, check the affected row count, not just whether the query succeeded, because an UPDATE that matches nothing still succeeds. Second, an exception raised in a background thread does not stop the main worker. The thread must set a flag or cancel the in-flight request, and the main loop must check it before every side effect. Run heartbeats at roughly a third of the lease TTL (3 seconds for a 10-second lease) so one missed beat doesn't expire the lease.

3. Guard every state write with the token

UPDATE webhook_deliveries
SET current_state = 'DELIVERED',
    updated_at    = clock_timestamp()
WHERE id = %s
  AND fencing_token = %s       -- fails if another worker has since taken over
  AND worker_id = %s;

If this updates zero rows, the worker has been fenced out. It must discard its result and write nothing further. The same check should apply to inserts into the audit table.

What fencing does not protect: the receiver

Fencing tokens work when the resource being written can check them, as your database can. The receiver's HTTP endpoint cannot. If Worker A has already sent its request, or sends it in the instant before it discovers it lost the lease, no token check can recall it. Fencing also does not give mutual exclusion. Kleppmann has agreed that fencing tokens would not protect against two processes both believing they hold the lock, with the lower token reaching the shared resource before the higher one. surfingcomplexity

So the honest guarantee of a lease-and-fencing design is this:

Your internal state stays consistent. Only the current lease holder can record outcomes.
The window for duplicate sends is minimized. Check lease_lost and re-validate the lease immediately before sending.
Duplicate delivery to receivers remains possible. Treat delivery as at-least-once.

The correct complement is idempotency. Send a stable event or delivery ID with every request, for example in a header, identical across retries and across workers. Document that receivers must deduplicate on it. Webhook providers behave this way in practice, which is why guidance for webhook consumers consistently stresses idempotent handlers.

Comparison of concurrency-control strategies

	Optimistic locking	Redis single-node lock	Redlock (multi-node Redis)	Database lease + fencing token
Main risk	High retry rate under heavy contention	Lock lost on failover; no fencing	Relies on timing and clock assumptions	Extra database write load
Fencing support	Version column can act as one	None built in	None built in; critics note no obvious way to add monotonic tokens	Yes, if the token is checked on every write
Safe for correctness-critical work	Yes, for the guarded row	No	Disputed (see below)	Yes, for state in the same database
Complexity	Low	Low	Medium	Medium to high
Best for	Low-contention updates	Efficiency-only locks (avoiding duplicate work)	Efficiency-style locks	Job queues where correctness matters

Redlock is contested. Kleppmann argues it is unsuitable where correctness depends on the lock. Its author, Salvatore Sanfilippo (antirez), responded with a rebuttal, and the two disagree on the timing assumptions. His position is that you can generate an incremental ID for each Redlock acquired if you have a linearizable store. Kleppmann's recommendation for correctness-critical locking is a consensus-backed system such as ZooKeeper, or a database with transactional guarantees plus fencing tokens. For efficiency-only locks, where an occasional duplicate is harmless, a single Redis node is usually enough. antirez

Many teams skip external lock services altogether. They use the database as the serialization point (SELECT … FOR UPDATE SKIP LOCKED or the conditional UPDATE above) or a managed queue with a visibility timeout. Amazon SQS works this way: a received message is hidden for a configurable period and becomes visible again if it isn't deleted in time. It is also at-least-once, so the same duplicate-handling caveat applies. SQS dead-letter queues use a maxReceiveCount threshold to move repeatedly failing messages aside while keeping the original message ID.

Architecture Summary Checklist

Granular state engine: Every delivery moves through explicit states (RECEIVED, ENCRYPTED, LEASED, ATTEMPTED, RETRIED, DELIVERED, DLQ), with each transition recorded.
Immutable audit log: Each attempt stores a timestamp, worker ID, fencing token, HTTP status, and latency.
Backoff with jitter: Retries use capped exponential backoff plus randomness, with a defined retry budget and a clear retryable/non-retryable error policy.
Atomic leases and fencing tokens: Claims are single conditional statements, and every state write is guarded by the token.
Heartbeats with real abort handling: Renewal checks affected rows, and the main worker loop stops before any further side effect once the lease is lost.
Idempotency end to end: A stable delivery ID is sent on every attempt, and receivers are told to deduplicate.
Recoverable DLQ: Dead-lettered events can be inspected and replayed in bulk, with alerts on sustained DLQ growth rather than single failures.

Originally published at https://instawebhook.com/blog/high-throughput-webhook-systems-delivery-timelines-and-split-brain-prevention

DEV Community: InstaWebhook

Routing Webhooks Through Cloudflare Workers KV and Durable Objects: Building a Globally Distributed Ingress Buffer

Why Webhook Ingestion Fails

Architecture

Step 1: The Entrypoint Worker

Step 2: Per-Tenant State in a Durable Object

Step 3: Workers KV for Tenant Config

Wrangler Configuration

Failure Modes and Mitigations

Backpressure and Origin Outages

Alternative: Cloudflare Queues

Production Checklist

Further Reading

Zero-Downtime Webhook Schema Evolution: Managing Breaking Payloads Without Dropping Events

1. The Schema Drift Dilemma in Multi-Producer Integrations

Common Breaking Changes

Large Integers Are a Hidden Type Problem

The Cost of Rigid Ingestion

2. Capture First: Acknowledge Fast, Parse Later

3. Writing Resilient Parsers: The Tolerant Reader

Tolerant Parsing in Go

Key Strategies

4. Schema Validation at the Ingress Gateway

Choosing a Schema Format and Registry

Tiered Validation Modes

5. Dual-Schema Mapping Layers

6. Contract Tests for Producers You Don’t Control

Operational Checklist

Further Reading

Edge-Native Webhook Ingress: Running Webhook Gateways on Cloudflare Workers and WebAssembly (Wasm)

1. The Architectural Shift: Moving Ingress to the Edge

2. Choosing the Right Verification Primitive: WebCrypto First, Wasm When Needed

Know what your providers actually use

Workers’ WebCrypto already covers the common cases

Where Wasm fits

An Ed25519 verification module in Rust

Loading the module in a Worker

The full gateway handler

3. CDN Boundary Protection: Layered Early Rejection

Protective measures in practice

4. Real-Time Tenant Routing with Edge KV

Know KV’s consistency model

Tenant routing flow

Implementation example

Protect secrets stored at the edge

5. Production Blueprint and Best Practices

Conclusion

References

HTTP/3 and QUIC for Webhook Delivery: What It Fixes, What It Doesn’t, and How to Adopt It

1. Where Webhook Delivery Hits TCP’s Limits

Handshake cost for sporadic endpoints

Ephemeral port and TIME_WAIT pressure

Head-of-line blocking

2. How QUIC Works

0-RTT: powerful, but dangerous for webhooks

Connection migration: narrower than it sounds

3. Designing a Dispatcher for HTTP/3

From connection pools to multiplexed sessions

Prioritization

Loss recovery and congestion control

Discovery and fallback

4. Reference Architecture

Go sketch

5. Operational Challenges

UDP blocking and throttling

CPU cost and tuning

Amplification and abuse

Observability

6. What the Evidence Says

7. Should You Adopt It?

References and Further Reading

The Multi-Tenant Neighbor Problem: Why Traditional API Rate Limiters Fail Under Webhook-Heavy Bursts

1. Understanding the Multi-Tenant Neighbor Problem in Webhooks

How Webhook Traffic Differs from User Traffic

2. Why Traditional API Rate Limiters Fail

A. IP-Based Limiting Doesn’t Identify Tenants

B. Global or Per-Route Limits Cause Cross-Tenant Starvation

C. A Centralized Counter Becomes a Hot Spot

D. Rejecting a Webhook Doesn’t Make It Go Away

3. Anatomy of a Webhook Ingestion Failure

Ephemeral port and `TIME_WAIT` pressure

2.1 The `skb:kfree_skb` tracepoint and drop reasons