A viral video does not trend politely. One morning a clip about a Dutch canal rescue jumped from 400 to 90,000 views in under an hour, and our ingest pipeline emitted roughly 12,000 video.trending events in a tight burst. Every one of those events had to fan out to subscriber webhooks: partner dashboards, a Slack relay, two analytics sinks, and a customer's own CRM. The naive delivery loop we started with — dequeue an event, curl the subscriber URL, move on — did exactly what you would expect. It hammered a partner endpoint at 300 requests per second, earned us an HTTP 429 with a Retry-After: 120 header, and then kept hammering because nothing in our code read that header. Within minutes the partner's rate limiter had banned our egress IP for an hour, and we were silently dropping events for every other subscriber queued behind the poisoned one.
That incident is why I rebuilt webhook delivery at ViralVidVault around per-subscriber rate limiting, honest backoff, and a delivery ledger that survives a crash. This article walks through the design and the code — real PHP 8.4 and Go you can adapt, plus a Cloudflare Workers edge shim — and it is deliberately opinionated about the parts that bit us. The goal is not "send webhooks." The goal is: never exceed a subscriber's advertised rate, never lose an event to a transient failure, and never let one slow consumer starve the rest.
The Problem Is Per-Subscriber, Not Global
The first instinct is to throttle globally: "send at most N webhooks per second." That is the wrong axis. A global limiter that caps you at 50 rps still lets you send all 50 to a single subscriber whose endpoint tops out at 10 rps, while a fast subscriber that could happily take 200 rps sits idle. Rate limiting for webhooks is fundamentally a per-destination concern, because the limit that matters is the one enforced on the other side.
So the model is: each subscriber has a bucket. Events are keyed by subscriber. A worker may only send to a subscriber if that subscriber's bucket has a token. This isolates a slow or angry consumer to its own lane — the canal-rescue partner can be throttled to a crawl while everyone else drains at full speed.
Our event shape is small and boring on purpose. Video events are cheap to produce and expensive to deliver, so the payload stays minimal and the subscriber pulls detail if it wants it:
{
"id": "evt_01JMZ8K3",
"type": "video.trending",
"video_id": "vv_9f31c2",
"region": "NL",
"velocity": 224.0,
"occurred_at": "2026-07-11T08:14:02Z"
}
A Token Bucket Backed by SQLite WAL
We run SQLite in WAL mode for the whole platform — it handles our write volume comfortably and removes an entire class of operational pain that a networked database would add. The delivery queue and the rate-limiter state both live there. WAL matters here specifically because the delivery worker writes constantly (marking attempts, updating buckets) while a separate reader tallies metrics, and WAL lets those not block each other.
A token bucket has two columns per subscriber: the current token count and the timestamp of the last refill. Instead of a background thread topping up buckets, we refill lazily at read time — compute how many tokens should have accrued since last_refill and clamp to the burst capacity. This is cheaper and it is crash-safe: there is no in-memory state to lose.
CREATE TABLE subscriber_bucket (
subscriber_id TEXT PRIMARY KEY,
capacity REAL NOT NULL, -- burst size, e.g. 20
refill_rate REAL NOT NULL, -- tokens per second, e.g. 10
tokens REAL NOT NULL,
last_refill REAL NOT NULL -- unix seconds, fractional
);
CREATE TABLE delivery (
id INTEGER PRIMARY KEY,
subscriber_id TEXT NOT NULL,
event_id TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending', -- pending|sent|failed|dead
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt REAL NOT NULL DEFAULT 0,
created_at REAL NOT NULL,
UNIQUE(subscriber_id, event_id) -- idempotent enqueue
);
CREATE INDEX idx_delivery_due ON delivery(status, next_attempt);
The UNIQUE(subscriber_id, event_id) constraint gives us idempotent enqueue for free: if the ingest pipeline re-emits an event after a retry, INSERT OR IGNORE collapses the duplicate instead of double-delivering it.
Here is the token acquisition in PHP 8.4. The whole thing runs inside a transaction so the refill-and-decrement is atomic against other workers:
<?php
declare(strict_types=1);
final class TokenBucket
{
public function __construct(private readonly \PDO $db) {}
/** Try to spend one token for a subscriber. Returns true if allowed. */
public function tryAcquire(string $subscriberId, float $now): bool
{
$this->db->beginTransaction();
try {
$row = $this->db->prepare(
'SELECT capacity, refill_rate, tokens, last_refill
FROM subscriber_bucket WHERE subscriber_id = ?'
);
$row->execute([$subscriberId]);
$b = $row->fetch(\PDO::FETCH_ASSOC);
if ($b === false) {
$this->db->rollBack();
return false; // unknown subscriber, drop
}
$elapsed = max(0.0, $now - (float) $b['last_refill']);
$tokens = min(
(float) $b['capacity'],
(float) $b['tokens'] + $elapsed * (float) $b['refill_rate']
);
if ($tokens < 1.0) {
// Persist the refill so we don't lose accrued fractional tokens.
$this->persist($subscriberId, $tokens, $now);
$this->db->commit();
return false;
}
$this->persist($subscriberId, $tokens - 1.0, $now);
$this->db->commit();
return true;
} catch (\Throwable $e) {
$this->db->rollBack();
throw $e;
}
}
private function persist(string $id, float $tokens, float $now): void
{
$up = $this->db->prepare(
'UPDATE subscriber_bucket
SET tokens = ?, last_refill = ?
WHERE subscriber_id = ?'
);
$up->execute([$tokens, $now, $id]);
}
}
The subtle bug to avoid: if tokens < 1.0 you must still write back the refilled value and the new last_refill. Skip that and every failed acquire resets the clock, so a busy subscriber never accrues tokens and stalls forever. We shipped that bug once. It looked exactly like the endpoint being down.
The Delivery Worker and Honest Backoff
A worker pulls due deliveries, checks the bucket, and either sends or defers. The order matters: check the bucket before you dequeue for sending, otherwise you pull an event, discover you cannot send it, and have to put it back — extra writes and a window where the event looks in-flight but isn't.
The non-negotiable rule from the outage: when a subscriber returns 429 (or 503 with Retry-After), you obey the header. You do not apply your own guess. The subscriber is telling you exactly when to come back, and ignoring it is how you get IP-banned.
<?php
declare(strict_types=1);
final class DeliveryWorker
{
private const MAX_ATTEMPTS = 8;
public function __construct(
private readonly \PDO $db,
private readonly TokenBucket $bucket,
) {}
public function tick(float $now): void
{
$due = $this->db->prepare(
"SELECT id, subscriber_id, event_id, payload, attempts
FROM delivery
WHERE status = 'pending' AND next_attempt <= ?
ORDER BY next_attempt LIMIT 100"
);
$due->execute([$now]);
foreach ($due->fetchAll(\PDO::FETCH_ASSOC) as $d) {
if (!$this->bucket->tryAcquire($d['subscriber_id'], $now)) {
// Bucket empty: defer a short, jittered slice. Cheap re-check.
$this->defer((int) $d['id'], $now + 0.25 + (mt_rand(0, 200) / 1000));
continue;
}
$this->send($d, $now);
}
}
private function send(array $d, float $now): void
{
$endpoint = $this->endpointFor($d['subscriber_id']);
[$code, $retryAfter] = $this->post($endpoint, $d['payload'], $d['subscriber_id']);
if ($code >= 200 && $code < 300) {
$this->finish((int) $d['id'], 'sent');
return;
}
$attempts = (int) $d['attempts'] + 1;
if ($attempts >= self::MAX_ATTEMPTS) {
$this->finish((int) $d['id'], 'dead');
return;
}
// Honor server-advertised backoff; else exponential with full jitter.
$delay = $retryAfter !== null
? (float) $retryAfter
: min(600.0, (2 ** $attempts) * (mt_rand(50, 100) / 100));
$this->reschedule((int) $d['id'], $attempts, $now + $delay);
}
private function post(string $url, string $body, string $subId): array
{
$sig = hash_hmac('sha256', $body, $this->secretFor($subId));
$ch = curl_init($url);
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $body,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 10,
CURLOPT_CONNECTTIMEOUT => 4,
CURLOPT_HTTPHEADER => [
'Content-Type: application/json',
'X-VV-Signature: sha256=' . $sig,
'X-VV-Event-Id: ' . $this->currentEventId,
],
CURLOPT_HEADERFUNCTION => function ($ch, $h) use (&$retryAfter) {
if (stripos($h, 'Retry-After:') === 0) {
$retryAfter = (int) trim(substr($h, 12));
}
return strlen($h);
},
]);
$retryAfter = null;
curl_exec($ch);
$code = (int) curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);
return [$code === 0 ? 599 : $code, $retryAfter];
}
}
A few decisions baked in there:
-
Full jitter, not fixed backoff.
2 ** attemptsalone means every worker that failed at the same second retries at the same second — a thundering herd that recreates the overload. Multiplying by a random0.5–1.0factor spreads the retries out. This is the AWS "exponential backoff and jitter" result, and it is worth taking literally. -
A dead-letter state, not infinite retries. After 8 attempts (roughly 10 minutes of escalating backoff) the delivery goes to
dead. A trending event is worthless a day later; retrying it forever just wastes egress and hides the fact that a subscriber is actually broken. We alert ondeadcount, not on individual failures. -
HMAC signatures on every request. Subscribers verify
X-VV-Signatureso a leaked endpoint URL cannot be spammed by a third party. Cheap to add, painful to retrofit.
Pushing the Fast Path to the Edge
Most of our subscribers sit behind Cloudflare, and so do we. For high-volume, low-value events — video.view_milestone, for instance — we do not want a PHP worker holding a connection open for 10 seconds per delivery. We offload those to a Cloudflare Worker that does the actual fan-out and rate limiting at the edge, close to the subscriber, using a Durable Object as the per-subscriber bucket. The origin just enqueues; the edge delivers.
This also keeps GDPR sane for our European subscribers: the edge worker strips any subscriber-identifying header from logs before it forwards, and view-milestone payloads carry no personal data at all — just a video id and a count.
// Cloudflare Worker: per-subscriber token bucket in a Durable Object.
export class SubscriberBucket {
constructor(state) {
this.state = state;
}
async fetch(request) {
const { capacity, refillRate } = await request.json();
const now = Date.now() / 1000;
let tokens = (await this.state.storage.get('tokens')) ?? capacity;
let last = (await this.state.storage.get('last')) ?? now;
tokens = Math.min(capacity, tokens + (now - last) * refillRate);
if (tokens < 1) {
const wait = (1 - tokens) / refillRate;
await this.state.storage.put({ tokens, last: now });
return new Response(JSON.stringify({ allowed: false, retryIn: wait }), {
status: 429,
});
}
await this.state.storage.put({ tokens: tokens - 1, last: now });
return new Response(JSON.stringify({ allowed: true }), { status: 200 });
}
}
export default {
async fetch(request, env) {
const { subscriberId, endpoint, payload } = await request.json();
const id = env.BUCKET.idFromName(subscriberId);
const gate = await env.BUCKET.get(id).fetch('https://bucket/', {
method: 'POST',
body: JSON.stringify({ capacity: 20, refillRate: 10 }),
});
if (gate.status === 429) {
// Bounce back to origin queue; do not drop.
return new Response('deferred', { status: 202 });
}
return fetch(endpoint, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(payload),
});
}
};
A Durable Object gives you a single-threaded, strongly-consistent instance per subscriber id, which is exactly the coordination primitive a token bucket wants. No lock contention, no read-modify-write race, and it lives at the edge nearest the request. The origin still owns the durable queue and the retry ledger — the edge is a fast delivery lane, not the source of truth.
A Load-Test Harness Worth Keeping
The only way I trust a rate limiter is to point a hostile client at it and watch the numbers. This tiny Go program floods the delivery endpoint and reports the achieved rate, so you can confirm the limiter actually caps at the configured refill_rate under burst.
package main
import (
"fmt"
"net/http"
"sync"
"sync/atomic"
"time"
)
func main() {
const workers, duration = 64, 10 * time.Second
var ok, throttled int64
deadline := time.Now().Add(duration)
var wg sync.WaitGroup
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
client := &http.Client{Timeout: 5 * time.Second}
for time.Now().Before(deadline) {
resp, err := client.Post(
"http://localhost:8787/deliver",
"application/json",
nil,
)
if err != nil {
continue
}
if resp.StatusCode == http.StatusTooManyRequests {
atomic.AddInt64(&throttled, 1)
} else {
atomic.AddInt64(&ok, 1)
}
resp.Body.Close()
}
}()
}
wg.Wait()
rate := float64(ok) / duration.Seconds()
fmt.Printf("accepted=%d throttled=%d achieved=%.1f rps\n", ok, throttled, rate)
}
With refill_rate = 10 and capacity = 20, a healthy run reports an achieved rate that hovers just above 10 rps — the burst capacity drains in the first two seconds, then delivery settles to the sustained refill rate. If you see the achieved rate track the offered load instead of the configured cap, your limiter is not limiting, and better to learn that from Go than from a partner's abuse team.
What Actually Mattered in Production
After running this for several months across our European regions, the parts that earned their keep were not the clever ones:
-
Obeying
Retry-Aftereliminated every repeat ban. It is one header. Read it. - Per-subscriber isolation meant one broken consumer never affected the others. Slow lanes stay in their lane.
-
The
deadstate turned silent data loss into an alert. We now know within minutes when a subscriber's endpoint is genuinely down, instead of discovering it from a support ticket a week later. - Idempotent enqueue via a unique constraint removed a whole category of double-delivery bugs that used to appear only under retry storms.
Rate-limited webhook delivery is not glamorous, but for an event-driven video platform it is load-bearing. The events that matter most — a clip going viral right now — are exactly the ones that arrive in the biggest bursts, and a delivery layer that folds under burst is a delivery layer that fails precisely when you need it. Build the bucket, honor the backoff, keep a durable ledger, and load-test it like an adversary. The boring version that survives the spike beats the clever version that doesn't.
Top comments (0)