Webhooks Are Harder Than You Think — 4 Lessons from Building a Reliable Webhook System

I used to think webhooks were simple. You get a POST request, you process it, you return 200. Done.

Then I built a system that processed 50,000 webhooks a day. And everything broke.

Here are the four things I wish someone had told me before I started.

---

Lesson 1: "At-Least-Once" Is a Lie You Need to Handle

Every webhook provider says "at-least-once delivery." What they don't tell you is that "at-least-once" can mean "three times in the same second."

I learned this the hard way when a payment processor retried a payment.succeeded event — three times, all within 800ms. My system processed all three, charged the customer's card for a second subscription, and triggered a very angry support email at 3 AM.

The fix isn't complicated, but you have to do it before anything else:

import hashlib
import redis

redis_client = redis.Redis()

def process_webhook(event_id: str, payload: dict):
    idempotency_key = hashlib.sha256(event_id.encode()).hexdigest()

    # Try to claim this event. If we can't, it's already being processed.
    if not redis_client.set(idempotency_key, "processing", nx=True, ex=300):
        return {"status": "duplicate", "event_id": event_id}

    try:
        # Your actual processing logic here
        handle_payment(payload)
        redis_client.set(idempotency_key, "completed", ex=86400)
        return {"status": "ok"}
    except Exception as e:
        redis_client.delete(idempotency_key)  # Allow retry
        raise

Key points:

• Use SET NX (set if not exists) — it's atomic, so no race conditions
• Set a TTL (300s here) so a crashed worker doesn't lock the event forever
• Delete the key on failure so the event can be retried

We went from 3-5 duplicate processing incidents per week to zero. It's the single highest-ROI change I've ever made to a webhook pipeline.

---

Lesson 2: Return 200 Fast, Process Later

Webhook providers have short timeouts. Stripe gives you 20 seconds. GitHub gives you 10. If your processing takes longer than that — and it will, once you add database writes, cache updates, and downstream calls — the provider marks it as failed and retries.

Now you're in a death spiral: each retry creates more work, which takes longer, which causes more retries.

The pattern that saved us:

from fastapi import FastAPI, Request, BackgroundTasks
import asyncio
from datetime import datetime

app = FastAPI()
event_queue = asyncio.Queue()

@app.post("/webhooks/stripe")
async def stripe_webhook(request: Request, bg: BackgroundTasks):
    payload = await request.json()
    event_id = request.headers.get("stripe-signature", "")

    # Verify signature first (always do this!)
    if not verify_stripe_signature(payload, request.headers):
        return {"status": "invalid_signature"}

    # Acknowledge immediately — don't wait for processing
    bg.add_task(process_in_background, event_id, payload)

    return {"status": "received", "timestamp": datetime.utcnow().isoformat()}

async def process_in_background(event_id: str, payload: dict):
    # Now you can take as long as you need
    await event_queue.put({"id": event_id, "payload": payload, "received_at": datetime.utcnow()})

The provider sees a fast 200. Your system processes at its own pace. Everybody wins.

One caveat: this means you need a retry mechanism on your side. If the background worker crashes after acknowledging the webhook but before finishing, the event is lost. We solved this by writing raw events to a database table first, then processing from the queue.

---

Lesson 3: Webhook Signatures Are Not Optional

I've seen production systems that skip signature verification because "it's behind a firewall" or "the endpoint URL is secret." Both are wrong.

An unverified webhook endpoint is an open POST endpoint on the internet. Anyone who discovers the URL can inject fake events. And URL discovery is easier than you think — DNS logs, misconfigured CORS headers, leaked in client-side code.

Here's what proper verification looks like for Stripe:

import stripe
import os

stripe.api_key = os.getenv("STRIPE_SECRET_KEY")

def verify_stripe_signature(payload: bytes, headers: dict) -> bool:
    try:
        signature = headers.get("stripe-signature", "")
        stripe.Webhook.construct_event(
            payload=payload,
            sig_header=signature,
            secret=os.getenv("STRIPE_WEBHOOK_SECRET")
        )
        return True
    except stripe.error.SignatureVerificationError:
        return False
    except ValueError:
        return False

For custom webhooks where the provider doesn't give you an SDK, HMAC works fine:

import hmac
import hashlib

def verify_hmac(payload: bytes, signature_header: str, secret: str) -> bool:
    expected = hmac.new(
        secret.encode(),
        payload,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature_header)

Always use hmac.compare_digest() — it's constant-time, which prevents timing attacks.

---

Lesson 4: Monitor Everything (Or Regret It Later)

The scariest moment of my webhook journey wasn't a crash. It was discovering that a critical webhook had been silently failing for six hours because we didn't have monitoring.

A downstream API went down. Our worker retried, failed, retried again, and eventually gave up. We had no alert. We found out when a customer emailed us.

Here's the minimum monitoring I now set up before deploying any webhook endpoint:

from prometheus_client import Counter, Histogram, generate_latest
from fastapi import Response

webhook_received = Counter(
    'webhook_received_total',
    'Total webhooks received',
    ['provider', 'event_type']
)
webhook_processed = Counter(
    'webhook_processed_total', 
    'Successfully processed webhooks',
    ['provider', 'event_type']
)
webhook_failures = Counter(
    'webhook_failures_total',
    'Failed webhook processing attempts',
    ['provider', 'event_type', 'error_type']
)
webhook_latency = Histogram(
    'webhook_processing_seconds',
    'Webhook processing duration'
)

@app.post("/webhooks/{provider}")
async def webhook_endpoint(provider: str, request: Request):
    payload = await request.json()
    event_type = payload.get("type", "unknown")

    webhook_received.labels(provider=provider, event_type=event_type).inc()

    try:
        with webhook_latency.time():
            result = await process_event(provider, event_type, payload)
        webhook_processed.labels(provider=provider, event_type=event_type).inc()
        return result
    except Exception as e:
        webhook_failures.labels(
            provider=provider,
            event_type=event_type,
            error_type=type(e).__name__
        ).inc()
        raise

@app.get("/metrics")
async def metrics():
    return Response(content=generate_latest(), media_type="text/plain")

Set up an alert on rate(webhook_failures_total[5m]) > 0. You'll sleep better.

---

What I'd Do Differently Next Time

If I were starting a new webhook system tomorrow, I'd build these four things before writing any business logic:

1. Idempotency layer (Redis SET NX with TTL)
2. Async acknowledge pattern (return 200, process in background)
3. Signature verification (HMAC or provider SDK)
4. Prometheus metrics + alerts

Everything else — the actual payment processing, the notification sending, the data syncing — comes after.

Webhooks aren't hard because the concept is complex. They're hard because the failure modes are subtle and the consequences are expensive. A duplicate event costs you real money. A dropped event costs you customer trust.

Build the guardrails first. The business logic can wait.

---

Building reliable API infrastructure. Follow for more lessons from production.