Webhook reliability: delivering and receiving events without losing them

Cross-post. Original: stellarbytecapital.com/blog/webhook-reliability

Webhooks are the duct tape of system integration: one service POSTs an event to another's URL when something happens. The happy path is a five-minute tutorial. The unhappy paths — the receiver was down, the request timed out, the same event arrived three times, two events landed out of order, someone forged a payload — are where real money and data get lost. Reliable webhooks are a distributed-systems problem in a simple costume.

The fundamental truth: delivery is at-least-once

Over an unreliable network you can guarantee "at least once" or "at most once," not "exactly once." Everyone sane chooses at-least-once: keep retrying until the receiver confirms, and accept duplicates. That one decision drives everything — the sender must retry, so the receiver must handle repeats safely.

Exactly-once delivery is a myth. Exactly-once processing is achievable — by an at-least-once sender talking to an idempotent receiver.

Building a reliable sender

• Deliver asynchronously via a queue. Don't fire the webhook inline with the business transaction. Commit the event to an outbox/queue; a separate worker delivers it. A slow or down receiver never blocks your core operation.
• Retry with exponential backoff and jitter, spaced over minutes to hours, capped.
• Give every event a stable unique ID that stays constant across retries, so the receiver can dedupe.
• Sign the payload with an HMAC over the body using a shared secret. Never make a security decision on an unsigned webhook.
• Dead-letter after max retries and expose a way to inspect and replay, plus a delivery log the customer can see.

Building a reliable receiver

• Verify the signature first. Check the HMAC against the raw body with a constant-time comparison before trusting anything. An endpoint that acts on unverified webhooks lets anyone forge "payment succeeded."
• Be idempotent on the event ID. Record processed IDs and skip repeats. This turns "the webhook fired twice" from a double refund into a no-op.
• Acknowledge fast, process later. Return 2xx as soon as you've durably stored the event; do the work in a background job. Heavy inline processing that times out makes the sender retry — multiplying load and duplicates.
• Don't assume order. Events arrive out of sequence. Use timestamps/version numbers and ignore stale events, or reconcile to current state rather than replaying a strict sequence.

When webhooks aren't enough: reconcile

Even a good webhook system drops events occasionally. For anything critical, webhooks should be an optimization for latency, not your only source of truth. Periodically pull authoritative state from the source and reconcile — exactly as a payment or trading system reconciles against the provider. The webhook makes you fast; the reconciliation makes you correct.

What to avoid

• Processing inline and timing out — the classic cause of duplicate storms.
• No signature verification — an unauthenticated endpoint is a public API for forging your events.
• No idempotency on the receiver — at-least-once guarantees you'll double-process eventually.
• Assuming ordered, exactly-once delivery — neither is real.
• Treating webhooks as the sole source of truth — one dropped event becomes a silent, permanent inconsistency.

It collapses to one pairing: an at-least-once sender with retries, signing, and dead-lettering, talking to an idempotent, signature-verifying receiver that acks fast and reconciles for safety.

---

We're Xingyao Byte — building payment platforms, quant trading systems, secure AI-execution layers, and reliable backends. Remote, async-first → stellarbytecapital.com