A file gets hashed. The hash goes off to get anchored. Sometime later, a webhook arrives saying the anchor is done. That gap between "submitted" and "confirmed" is where most of the hard problems in an async proof pipeline actually live. Not the hashing. Not the anchoring. The webhook.
I hit this building the anchoring flow behind ProofLedger, and it's a pattern that applies to any system where a client kicks off work and gets notified later: payment confirmations, video transcoding, background exports. If you're building or consuming webhooks for anything time-sensitive, these four problems show up in the same order every time.
Verify the signature before you trust the payload
A webhook endpoint is a URL on the open internet. Anyone can POST to it. If your handler reads status: "anchored" from the body and acts on it without checking where it came from, you've built an endpoint that lets a stranger fake completion events.
The standard fix is HMAC-SHA256. The sender computes a signature over the raw request body using a shared secret, puts it in a header, and the receiver recomputes it and compares.
import hashlib
import hmac
from flask import Flask, request, abort
app = Flask(__name__)
WEBHOOK_SECRET = b"shared-secret-from-sender"
def verify_signature(payload_body: bytes, signature_header: str) -> bool:
expected = hmac.new(WEBHOOK_SECRET, payload_body, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header)
@app.route("/webhooks/proof-status", methods=["POST"])
def handle_webhook():
signature = request.headers.get("X-Signature-256", "")
if not verify_signature(request.get_data(), signature):
abort(401, "invalid signature")
event = request.get_json()
process_event(event)
return "", 204
Two details matter here and both are easy to get wrong. First, hmac.compare_digest instead of ==. A regular string comparison short-circuits on the first mismatched byte, which leaks timing information an attacker can use to guess the signature byte by byte. Second, sign the raw bytes, not the parsed JSON. If you verify against request.get_json() re-serialized, key ordering or whitespace differences between the sender's serializer and yours will produce a different signature than the one that was actually sent, and you'll get false rejections that are miserable to debug.
Idempotency: the same event will arrive more than once
Webhook senders retry on timeout, because from their side a timeout looks identical to a dropped delivery. That means your handler needs to survive receiving the exact same event two, three, sometimes ten times.
The fix is an idempotency key, some sender-provided ID that's unique per logical event, checked against a store before you do anything with side effects.
import sqlite3
import time
conn = sqlite3.connect("webhook_events.db")
conn.execute("""
CREATE TABLE IF NOT EXISTS processed_events (
event_id TEXT PRIMARY KEY,
received_at REAL NOT NULL
)
""")
def already_processed(event_id: str) -> bool:
row = conn.execute(
"SELECT 1 FROM processed_events WHERE event_id = ?", (event_id,)
).fetchone()
return row is not None
def mark_processed(event_id: str) -> None:
conn.execute(
"INSERT OR IGNORE INTO processed_events (event_id, received_at) VALUES (?, ?)",
(event_id, time.time()),
)
conn.commit()
def process_event(event: dict) -> None:
event_id = event["id"]
if already_processed(event_id):
return
mark_processed(event_id)
apply_status_update(event)
The INSERT OR IGNORE plus a PRIMARY KEY constraint does the real work. Two requests racing for the same event_id will have one succeed and one silently no-op at the database level, which is a stronger guarantee than an in-process check-then-set that a second thread can slip through. A Redis SETNX with a TTL does the same job if you're not on SQLite. Either way, the key insight is that idempotency has to be enforced at the storage layer, not in application logic that can race.
Retry and backoff on the sending side
If you're the one firing webhooks (say, notifying a customer's endpoint when an anchor confirms), the receiving server will sometimes be down, slow, or rate-limiting you. Retrying immediately in a tight loop makes that worse. Exponential backoff with jitter spreads retries out and gives the receiver room to recover.
import random
import time
import requests
def send_webhook(url: str, payload: dict, max_attempts: int = 5) -> bool:
for attempt in range(max_attempts):
try:
response = requests.post(url, json=payload, timeout=5)
if response.status_code < 300:
return True
if response.status_code == 429:
retry_after = float(response.headers.get("Retry-After", 0))
time.sleep(retry_after or backoff_delay(attempt))
continue
if response.status_code < 500:
return False # client error, retrying won't help
except requests.RequestException:
pass
time.sleep(backoff_delay(attempt))
return False
def backoff_delay(attempt: int) -> float:
base = min(2 ** attempt, 60)
return base + random.uniform(0, base * 0.1)
Notice the branch on status code. A 429 gets special handling because the receiver told you exactly when to come back, a 4xx (other than 429) stops retrying because the request itself is malformed and resending it changes nothing, and anything else falls through to exponential backoff. Treating all failures the same, whether it's a 400 or a 503, wastes retry budget on requests that were never going to succeed.
Replay protection closes the last gap
Signature verification proves the payload came from the real sender. It doesn't prove the payload is fresh. If an attacker captures a valid signed request (from a compromised log, a proxy, a misconfigured CDN cache) they can replay it later and your handler will accept it, because the signature still checks out.
The fix is a timestamp in the payload, checked against a tolerance window, combined with the idempotency check you already have.
def is_replay(event: dict, tolerance_seconds: int = 300) -> bool:
event_time = event.get("timestamp", 0)
now = time.time()
if abs(now - event_time) > tolerance_seconds:
return True
return already_processed(event["id"])
A five-minute tolerance window is generous enough to absorb clock drift and network delay between well-behaved systems, tight enough that a captured request is useless a few minutes after the fact. Sign the timestamp as part of the HMAC payload too, not just the event body, otherwise an attacker can strip an old timestamp and paste in a fresh one without invalidating the signature.
Putting it together
None of these four pieces is complicated on its own. What makes webhook handling hard is that skipping any one of them creates a gap the other three don't cover. Signature verification without idempotency means a legitimate but duplicated event can double-charge, double-anchor, or double-anything. Idempotency without replay protection means an old captured request stays valid forever. Backoff without respecting Retry-After just hammers a struggling server on your own schedule instead of theirs.
If you're building this for anchoring workflows specifically, the shape doesn't change: a hash goes out, a confirmation comes back on its own schedule, and your handler needs to be safe whether that confirmation shows up once, twice, or three minutes late with a stale timestamp attached.
What's your tolerance window for replay protection, and have you had to tune it after a real incident?
Top comments (0)