The 3 Things That Make Webhooks Actually Work (With Python Code)

Webhooks look simple from the outside. Your service makes an HTTP POST to a customer's URL when something happens, and... that's it, right?

Then you run one in production and learn the hard way that webhooks are a distributed systems problem wearing a friendly costume. The customer's endpoint goes down. A request times out. The same event gets delivered twice. An attacker starts sending fake events to your endpoint.

I've built webhook delivery for two different products, and both times the naive version (just POST and forget) broke within a week. This post walks through the three things that make webhooks actually work — retries, signatures, and idempotency — with code you can copy.

The three hard problems

1. Delivery is not guaranteed. The receiver's server is down, their firewall drops the request, they deploy mid-flight. A single POST has maybe a 99% success rate, which sounds fine until you realize that means roughly one in a hundred events is silently lost.

2. Duplication and ordering. If you retry a failed delivery, you might deliver the same event twice. And if two workers deliver events concurrently, the receiver can see them out of order.

3. Security. Anyone who can reach the receiver's endpoint can POST a fake event. The receiver needs to verify the event actually came from you.

Let me handle each one with real code.

1. Retries with exponential backoff

A retry loop is table stakes, but the details matter. You want exponential backoff with jitter, a cap on attempts, and a cap on total time.

import time
import random
import requests

MAX_ATTEMPTS = 8
BASE_DELAY = 5  # seconds

def deliver_with_retries(url, payload, signature_header):
    attempt = 0
    while attempt < MAX_ATTEMPTS:
        try:
            resp = requests.post(
                url,
                json=payload,
                headers={"X-Signature": signature_header},
                timeout=10,
            )
            if resp.status_code < 500:
                # 2xx/3xx/4xx: don't retry client errors
                return resp.status_code
            # 5xx: server error, retry below
        except requests.exceptions.RequestException:
            # network error or timeout, retry below
            pass

        attempt += 1
        if attempt == MAX_ATTEMPTS:
            return None  # give up, log for manual review

        # exponential backoff with jitter
        delay = BASE_DELAY * (2 ** (attempt - 1))
        jitter = random.uniform(0, delay * 0.3)
        time.sleep(delay + jitter)

    return None

Two decisions in here matter more than the loop itself:

• Only retry 5xx and network errors, never 4xx. If the receiver says 400 or 401, that's their bug, not a transient failure. Retrying a 400 just spams their logs and doesn't fix anything.
• Add jitter. If a receiver goes down and every sender retries on the exact same schedule, they all hammer the endpoint at once the moment it comes back. Jitter spreads the retries out.

A real system would use a queue and a background worker instead of time.sleep, but the logic is identical — I kept it inline so it's easy to read.

2. Signatures so the receiver trusts you

The receiver needs to answer one question: did this request really come from the sender? The standard answer is an HMAC signature over the raw request body, using a shared secret.

import hashlib
import hmac
import json

def sign_payload(secret: bytes, payload: dict) -> str:
    body = json.dumps(payload, separators=(",", ":"), sort_keys=True)
    return hmac.new(secret, body.encode(), hashlib.sha256).hexdigest()

And on the receiver's side:

def verify_signature(secret: bytes, raw_body: bytes, header: str) -> bool:
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)

Three details that trip people up:

• Sign the raw bytes, not the re-serialized JSON. JSON has no canonical form — whitespace and key order can differ. If the receiver parses the body and re-serializes it before verifying, the signature won't match. Sign the exact bytes that went over the wire.
• Use hmac.compare_digest, not ==. == short-circuits and leaks timing information; compare_digest runs in constant time.
• Rotate secrets. If a customer's secret leaks, they need to be able to revoke and rotate it without breaking everyone else.

3. Idempotency so duplicate delivery is harmless

Retries mean duplicates are inevitable. The fix is an idempotency key: every event carries a unique ID, and the receiver ignores events it has already processed.

import sqlite3

def handle_event(event_id: str, handler):
    # atomic claim: INSERT succeeds only if the ID is new
    conn = sqlite3.connect("events.db")
    try:
        conn.execute(
            "INSERT INTO processed_events (event_id) VALUES (?)", (event_id,)
        )
        conn.commit()
    except sqlite3.IntegrityError:
        return  # already processed, skip
    handler()

The INSERT with a unique constraint is the whole trick — it gives you an atomic "only process this once" guarantee even if two workers race on the same event.

What actually goes wrong in production

These are bugs I've personally shipped (and then fixed):

• Out-of-order delivery. With concurrent workers and retries, event #2 can arrive before event #1. If the receiver assumes order, add a sequence field to the payload so they can detect and buffer out-of-order events.
• Retry storms. A receiver with a slow endpoint backs up thousands of events, and your retries make it worse. Cap concurrent deliveries per receiver, and drop events into a dead-letter queue after N failed attempts instead of retrying forever.
• No timeout on the receiving side. Set a connect timeout and a read timeout. A receiver that hangs the connection for 60 seconds is worse than one that fails fast.

Wrap-up

If you're adding webhooks to your API, ship all three from day one: retries with backoff and jitter, HMAC signatures over the raw body, and idempotency keys. Retrofitting any of them after customers have built against your API is twice the work.

And if you're consuming webhooks, verify the signature and deduplicate by event ID before you touch your database. Your sender will eventually deliver the same event twice — plan for it.

Comments

[오피셜메일]

재시도·서명·멱등성을 한 흐름으로 묶은 구성이 명확했습니다. 4xx를 모두 재시도하지 않는 원칙에는 408과 429처럼 일시적인 상태를 예외로 두고 Retry-After를 따르는 분기를 추가하면 운영 환경에서 더 안전할 것 같습니다.