Webhook Ingestion Pipelines: Idempotency, Ordering, Dead-Letter Queues & Replay

webhook ingestion is the deceptively simple-looking HTTP endpoint that quietly decides whether your billing state, your CRM, and your analytics warehouse agree with the outside world — or drift silently out of sync every time a provider retries a delivery. A webhook is just a POST that some external system (Stripe, GitHub, Shopify, Twilio, a partner API) fires at your URL when something happens. What makes it hard is everything the naive "read the JSON, update a row, return 200" handler ignores: providers deliver at-least-once, so the same event arrives two, three, or ten times; the network reorders deliveries, so a subscription.updated can land before the subscription.created it depends on; anyone who learns your URL can forge a payload unless you verify a signature; and your handler will crash halfway through, leaving the question of what happens to the event it was processing.

This guide is the senior-engineering walkthrough for building a receiver that is correct under all four of those pressures. It treats webhook ingestion as four orthogonal problems — authenticity (signature verification over the raw body plus a timestamp tolerance that closes the replay-attack hole), idempotency (the provider's event id as a dedup key, backed by a UNIQUE constraint so reprocessing is a no-op), event ordering (version-guarded state so an out-of-order at-least-once delivery never clobbers newer data), and failure handling (bounded retries feeding a dead-letter queue you can replay once the bug is fixed). Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works — so you can defend every design choice the way an interviewer wants to hear it.

When you want hands-on reps immediately after reading, drill the streaming practice library →, rehearse dedup and sequencing on the event-processing practice library →, and sharpen the idempotency SQL on the SQL practice library →.

---

On this page

• Why webhook ingestion is a correctness problem
• Signature verification and the fast-ACK boundary
• Idempotency and deduplication
• Event ordering across retries
• Dead-letter queues and replay
• Cheat sheet — webhook ingestion recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why webhook ingestion is a correctness problem

Four failure modes, one endpoint — the guarantees the naive handler silently violates

The one-sentence invariant: a webhook receiver is a distributed-systems boundary where an untrusted sender delivers each event at-least-once, in no guaranteed order, over a channel anyone can forge, to a handler that can crash mid-processing — so a correct receiver must authenticate every payload, deduplicate every event, order state changes per entity, and route unprocessable events to a dead-letter queue it can replay, all without ever double-charging a customer or dropping a delete. The "read JSON, update row, return 200" handler assumes exactly-once, in-order, trusted, never-fails delivery — four assumptions the real world violates on day one, usually invisibly, until a reconciliation job or an angry customer surfaces the drift months later.

The four axes interviewers actually probe.

• Authenticity. Can anyone who discovers your URL POST a fake payment.succeeded? Every serious provider signs the request with an HMAC over the raw body plus a timestamp; your handler recomputes the signature with the shared secret and rejects mismatches in constant time. Skip this and your webhook endpoint is an unauthenticated write API. Interviewers open here because a candidate who forgets signature verification has never shipped a webhook to production.
• Idempotency. At-least-once delivery means duplicates are not an edge case — they are guaranteed. A provider that doesn't get a timely 2xx retries the same event, and network timeouts mean you sometimes did process an event whose ACK never arrived. The dedup key is the provider's event id; a UNIQUE constraint turns reprocessing into a no-op. Getting this wrong double-applies side effects.
• Ordering. Deliveries reorder. A customer.updated (new email) can arrive after a later customer.updated (newer email), and last-arrival-wins would resurrect stale data. Correct receivers order per entity using a version or sequence number carried in the payload, applying a change only if it is newer than the stored state.
• Failure handling. Your handler will throw — a downstream is down, a payload has an unexpected shape, a bug ships. The choice is: retry (with backoff, bounded), then dead-letter the event (with enough context to debug and replay) — or lose it. A receiver with no dead-letter queue silently drops every event that fails processing.

The 2026 reality — retries and reordering are the contract, not the exception.

• Every major provider retries. Stripe retries with exponential backoff for up to ~3 days; GitHub redelivers on demand and on failure; Shopify retries 19 times over 48 hours; AWS SNS/EventBridge retry aggressively. If your endpoint returns non-2xx, times out, or is briefly down, you will see the same event again. Duplicates are designed in.
• Delivery is unordered by default. Providers fan out webhooks across workers and regions; two events emitted a millisecond apart can arrive seconds apart in swapped order. Only a few providers offer ordered delivery, and even those don't guarantee it across retries.
• The signature is the only trust anchor. The payload's own "verified": true field means nothing — an attacker sets it too. Trust comes from the HMAC the attacker can't forge without the secret, plus a timestamp window so a captured-and-replayed request goes stale.
• The endpoint must ACK fast. Providers treat a slow response as a failure and retry, so heavy processing on the request path causes the duplicates it then has to dedup. The correct shape is accept → verify → enqueue → return 200 in milliseconds, then process asynchronously.

What interviewers listen for.

• Do you name all four axes — authenticity, idempotency, ordering, failure handling — without prompting? — senior signal.
• Do you say "at-least-once means duplicates are guaranteed, so processing must be idempotent" in the first minute? — required answer.
• Do you separate verify + ACK fast from process asynchronously, rather than doing the work inline? — senior signal.
• Do you describe a dead-letter queue with replay, not "log the error and move on"? — senior signal.
• Do you treat a webhook as an untrusted, at-least-once, unordered event rather than "an API call from Stripe"? — required framing.

Worked example — the four-axis webhook checklist

Detailed explanation. The single most useful artifact for a webhook-ingestion interview is a checklist that, for any incoming event type, tells you which of the four controls must fire and what breaks if it doesn't. Every senior webhook discussion converges on this list within the first ten minutes; having it memorised is what separates a fluent answer from a stumbling one. Walk through building it for a hypothetical payment.succeeded webhook from a payments provider into a billing service.

• The event. payment.succeeded with { id, type, created, data: { payment_id, customer_id, amount_cents, version } }.
• The side effect. Mark the invoice paid, grant entitlements, send a receipt — all things you must do exactly once.
• The pressures. The provider retries on any non-2xx; deliveries reorder; the URL is public; the entitlement grant can fail if the auth service is down.

Question. For the payment.succeeded event, state which control handles each of the four axes and the concrete failure if it is missing.

Input.

Axis	Control	Failure if missing
Authenticity	HMAC signature + timestamp guard	anyone can forge a paid invoice
Idempotency	UNIQUE event id + ON CONFLICT	customer double-charged / double-granted
Ordering	version guard per payment	a stale refund overwrites a newer capture
Failure handling	retry → dead-letter queue	entitlement silently never granted

Code.

Incoming POST /webhooks/payments
=================================

1. AUTHENTICITY   verify HMAC over raw body; check timestamp within ±5m
                  -> reject 401 if invalid

2. FAST ACK       enqueue {event_id, type, payload} to a durable queue
                  -> return 200 immediately (do NOT process inline)

--- async worker ---------------------------------------------------

3. IDEMPOTENCY    INSERT event_id INTO processed_events ON CONFLICT DO NOTHING
                  -> if row already existed, STOP (already handled)

4. ORDERING       apply only if payload.version > stored payment.version

5. FAILURE        on exception: retry with backoff; after N attempts,
                  move to dead_letter_queue with error + attempt count

Step-by-step explanation.

1. Authenticity is the gate: the handler recomputes HMAC-SHA256(secret, raw_body) and compares it, in constant time, to the signature header. A payload whose signature doesn't match — or whose timestamp is outside the tolerance window — is rejected with 401 before any work happens. This is non-negotiable for any event with a side effect.
2. The fast-ACK step decouples accepting the event from processing it. The endpoint verifies and enqueues, then returns 200 in milliseconds. This keeps the provider from retrying due to slow processing, which is the single biggest source of self-inflicted duplicates.
3. Idempotency runs first inside the worker: attempt to record event_id in a dedup table with a UNIQUE constraint. If the insert conflicts, this exact event was already processed — the worker stops. This is what makes at-least-once delivery safe.
4. Ordering guards the state write: apply the change only if the payload's version (or sequence) exceeds the version already stored for that payment. A reordered, older event is dropped rather than clobbering newer state.
5. Failure handling wraps the whole worker: an exception triggers a bounded retry with backoff; after the attempt budget is exhausted, the event lands in a dead-letter queue with its payload, error, and attempt count so an engineer can debug and replay it.

Output.

Control	Where it runs	Cost
Signature + timestamp	request path, before ACK	one HMAC per request
Fast ACK / enqueue	request path	one durable enqueue
Dedup (UNIQUE)	worker, first step	one indexed insert
Version guard	worker, on write	one compare per state change
Retry → DLQ	worker, on failure	bounded retries + DLQ row

Rule of thumb. Never design a webhook endpoint as "parse and update." Design it as five stations — verify, ACK, dedup, order, dead-letter — and satisfy each one explicitly. Draw the five stations on the whiteboard first; the implementation falls out of them.

Worked example — what interviewers actually probe

Detailed explanation. The senior webhook interview has a predictable shape: the interviewer opens with an innocent-sounding "how would you build an endpoint that receives events from Stripe?", then progressively narrows to test whether you know the four axes. Candidates who name the guarantees score highest; candidates who describe "a Flask route that updates the database" score lowest. Walk through the grading rubric.

• Ambiguous opener. "Build an endpoint that receives payment events." — invites you to name the guarantees.
• Follow-up 1. "The same event arrives twice — what happens?" — probes idempotency.
• Follow-up 2. "How do you know the request is really from the provider?" — probes authenticity.
• Follow-up 3. "Two updates for the same order arrive out of order." — probes ordering.
• Follow-up 4. "Processing throws because a downstream is down." — probes failure handling and replay.

Question. Draft a five-point senior webhook answer that pre-empts all four follow-ups without waiting to be asked.

Input.

Interview signal	Weak answer	Senior answer
Delivery model	"Stripe calls my endpoint"	"at-least-once; duplicates and reordering are guaranteed"
Duplicate handling	"check if it exists first"	"UNIQUE event_id + ON CONFLICT DO NOTHING; idempotent"
Authenticity	"it's HTTPS so it's fine"	"HMAC over raw body + timestamp tolerance, constant-time compare"
Ordering	"process in the order received"	"version guard per entity; drop stale events"
Failure	"log and return 500"	"bounded retries → dead-letter queue → replay"

Code.

Senior webhook answer template (5 points)
=========================================

1 — name the delivery model up front
  "Webhooks are at-least-once and unordered, so my handler must be
   idempotent and must order state changes per entity."

2 — authenticity
  "I verify an HMAC-SHA256 signature over the RAW request body plus a
   timestamp, comparing in constant time, and reject requests whose
   timestamp is outside a ~5 minute window to stop replay attacks."

3 — accept vs process
  "The endpoint verifies, enqueues, and returns 200 in milliseconds.
   Processing happens on an async worker so slow work never triggers
   provider retries."

4 — idempotency + ordering
  "The worker records the event id in a dedup table with a UNIQUE
   constraint; a conflicting insert means already-processed, so it
   stops. State writes apply only if the payload version is newer than
   what's stored, so reordered deliveries can't clobber newer data."

5 — failure + replay
  "On a processing exception I retry with exponential backoff and
   jitter; after the attempt budget, the event goes to a dead-letter
   queue with its payload, headers, error, and attempt count. Once the
   bug is fixed I replay the DLQ — safe because processing is idempotent."

Step-by-step explanation.

1. Point 1 is the crucial framing. Naming the delivery model — "at-least-once and unordered" — signals you understand the problem is distributed systems, not CRUD. Weak candidates describe the framework ("a FastAPI route") before naming the guarantees.
2. Point 2 addresses authenticity concretely: HMAC over the raw body (not the re-serialized JSON), a constant-time compare, and a timestamp window. Saying "it's HTTPS" is the classic junior tell — TLS authenticates the channel, not the sender.
3. Point 3 is the accept-versus-process split. Verifying and enqueuing on the request path, then processing asynchronously, is what keeps you from causing the very duplicates you then dedup. This is the single highest-signal architectural point.
4. Point 4 covers idempotency and ordering together because they are the two invariants at-least-once delivery forces. The dedup table with a UNIQUE constraint plus the version guard is the whole correctness story for the happy path.
5. Point 5 is the reliability axis: bounded retries, a dead-letter queue with enough context to debug, and idempotent replay. "Log and return 500" loses the event; the senior answer never loses an event.

Output.

Grading criterion	Weak score	Senior score
Names at-least-once / unordered	rare	mandatory
Verifies signature over raw body	occasional	required
Splits ACK from processing	rare	senior signal
Names dedup + version guard	rare	mandatory
Names DLQ + idempotent replay	rare	senior signal

Rule of thumb. The senior webhook answer is a five-point monologue that covers delivery model, authenticity, accept-vs-process, idempotency + ordering, and failure + replay — without waiting for the follow-ups. Rehearse it once; deploy it every time.

Worked example — the accept-then-process pipeline shape

Detailed explanation. Given a new webhook integration, the senior engineer draws the same pipeline every time: a thin, fast ingress that authenticates and durably records the raw event, and a separate worker that processes it with dedup, ordering, and dead-lettering. Codifying the shape makes the design reproducible — any interviewer can hand you a provider and you can draw the two-stage pipeline in under a minute. Walk through it for three scenarios: a low-volume GitHub push webhook, a high-volume Stripe payment stream, and a partner API with no signatures.

• Stage A — ingress (request path). Verify signature + timestamp; write the raw event to a durable buffer (a queue, a table, or Kafka); return 200. Milliseconds. No business logic.
• Stage B — worker (async). Dedup by event id; order by version; do the business work; on failure, retry then dead-letter.
• Why two stages. The provider's SLA is "respond fast or I retry." Business work is slow and can fail. Splitting them lets ingress be fast and reliable while the worker is slow and fallible without causing duplicates.

Question. Walk the two-stage pipeline for the three scenarios and record what each stage does.

Input.

Scenario	Signed?	Volume	Ingress buffer
GitHub push	yes (HMAC)	low	Postgres raw_events table
Stripe payments	yes (HMAC)	high	Kafka topic / SQS queue
Partner API (no sig)	no	low	table + IP allowlist + mutual TLS

Code.

# Stage A — ingress: verify, persist raw, ACK fast (framework-agnostic)
def ingest(request) -> "Response":
    raw = request.get_data()                      # RAW bytes, not parsed JSON
    sig = request.headers.get("X-Signature", "")
    ts  = request.headers.get("X-Timestamp", "")

    if not verify_signature(raw, ts, sig):        # HMAC + timestamp window
        return Response(status=401)

    event = json.loads(raw)
    # Durably record the raw event; processing happens elsewhere.
    enqueue({
        "event_id": event["id"],
        "type":     event["type"],
        "payload":  event,
        "received_at": now_iso(),
    })
    return Response(status=200)                    # ACK in milliseconds


# Stage B — worker: dedup, order, process, dead-letter (pseudocode)
def process(job) -> None:
    if already_processed(job["event_id"]):        # UNIQUE dedup
        return
    if not is_newer(job):                         # version guard
        mark_processed(job["event_id"]); return
    try:
        apply_side_effects(job)                    # the actual business work
        mark_processed(job["event_id"])
    except Exception as exc:
        retry_or_dead_letter(job, exc)             # backoff, then DLQ

Step-by-step explanation.

1. Scenario 1 — GitHub push, low volume, signed. Ingress verifies the X-Hub-Signature-256 HMAC and writes the raw event to a raw_events Postgres table; a worker polls the table. Postgres-as-a-queue is perfectly adequate at low volume and keeps the stack simple.
2. Scenario 2 — Stripe payments, high volume, signed. Ingress verifies the Stripe-Signature header and pushes to Kafka or SQS; a fleet of workers consumes. The durable queue absorbs bursts and decouples ingress throughput from worker throughput.
3. Scenario 3 — partner API with no signatures. Authenticity can't come from an HMAC, so it comes from network controls: an IP allowlist plus mutual TLS. The two-stage shape is otherwise identical — verify (by network identity), persist, ACK, then process.
4. In all three, Stage A never runs business logic. It authenticates, records the raw event durably, and ACKs. If the process crashes after ACK, the durable buffer still holds the event, so nothing is lost — the worker picks it up.
5. Stage B is where dedup, ordering, and dead-lettering live. Because the raw event is already durable, the worker can be retried freely: it re-reads the buffered event, the dedup step makes reprocessing safe, and failures dead-letter rather than vanish.

Output.

Scenario	Stage A auth	Stage A buffer	Stage B consumer
GitHub push	HMAC-SHA256	raw_events table	table-poller worker
Stripe payments	HMAC-SHA256	Kafka / SQS	worker fleet
Partner (no sig)	IP allowlist + mTLS	raw_events table	table-poller worker

Rule of thumb. Always split webhook ingestion into a fast authenticated ingress that durably records the raw event and returns 200, and a separate worker that deduplicates, orders, processes, and dead-letters. The durable buffer between them is what makes "the process crashed" a non-event.

Senior interview question on webhook ingestion design

A senior interviewer often opens with: "You're building the receiver for Stripe payment webhooks into a billing service. The provider retries on any non-2xx and can deliver out of order. Walk me through the end-to-end ingestion pipeline — how you authenticate, how you avoid double-charging on a duplicate, how you avoid a stale event overwriting newer state, and what happens when processing throws because the entitlements service is down."

Solution Using an accept-verify-enqueue-process pipeline with dedup, version guard, and DLQ

# ingress.py — Stage A: authenticate, persist raw, ACK fast
import hmac, hashlib, json, time
from flask import Flask, request, Response

app = Flask(__name__)
WEBHOOK_SECRET = b"whsec_...."            # shared secret from the provider
TOLERANCE_SEC  = 300                       # 5-minute replay window

def verify(raw: bytes, ts: str, sig: str) -> bool:
    # Reject stale requests (replay-attack defense)
    if abs(time.time() - int(ts)) > TOLERANCE_SEC:
        return False
    signed = f"{ts}.".encode() + raw
    expected = hmac.new(WEBHOOK_SECRET, signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, sig)     # constant-time

@app.post("/webhooks/payments")
def ingress():
    raw = request.get_data()                       # RAW body — verify before parse
    ts  = request.headers.get("X-Timestamp", "0")
    sig = request.headers.get("X-Signature", "")
    if not verify(raw, ts, sig):
        return Response(status=401)

    event = json.loads(raw)
    enqueue_durably(                               # SQS / Kafka / outbox table
        event_id=event["id"],
        event_type=event["type"],
        payload=raw.decode(),                      # store the exact bytes
    )
    return Response(status=200)                     # ACK in milliseconds

# worker.py — Stage B: dedup, order, process, dead-letter
import json, psycopg2

MAX_ATTEMPTS = 5

def handle(job: dict, conn) -> None:
    event = json.loads(job["payload"])
    event_id = event["id"]

    with conn:
        with conn.cursor() as cur:
            # 1. Idempotency: claim the event id, or bail if already claimed
            cur.execute("""
                INSERT INTO processed_events(event_id)
                VALUES (%s) ON CONFLICT (event_id) DO NOTHING
            """, (event_id,))
            if cur.rowcount == 0:
                return                              # duplicate — no-op

            # 2. Ordering: apply only if this version is newer
            d = event["data"]
            cur.execute("""
                UPDATE payments
                SET    status = %s, amount_cents = %s, version = %s
                WHERE  payment_id = %s AND version < %s
            """, (d["status"], d["amount_cents"], d["version"],
                  d["payment_id"], d["version"]))

            # 3. Side effect (same transaction as the dedup claim)
            grant_entitlements(cur, d["customer_id"], d["payment_id"])
    # commit = everything above is atomic

def consume(job, conn):
    try:
        handle(job, conn)
    except Exception as exc:
        if job["attempts"] + 1 >= MAX_ATTEMPTS:
            move_to_dlq(job, exc)                   # dead-letter with context
        else:
            requeue_with_backoff(job)               # 1s, 2s, 4s, 8s, ...

-- Schema behind the worker
CREATE TABLE processed_events (
    event_id     TEXT        PRIMARY KEY,           -- the idempotency key
    processed_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

CREATE TABLE payments (
    payment_id   TEXT        PRIMARY KEY,
    customer_id  TEXT        NOT NULL,
    status       TEXT        NOT NULL,
    amount_cents BIGINT      NOT NULL,
    version      BIGINT      NOT NULL DEFAULT 0      -- drives the ordering guard
);

Step-by-step trace.

Step	Naive handler	This pipeline
Authenticity	none / "trusts HTTPS"	HMAC over raw body + timestamp window
Slow processing	inline → provider retries	ACK first, process async
Duplicate event	double-charges	UNIQUE event_id → second attempt is a no-op
Out-of-order event	stale overwrites new	version guard WHERE version < incoming
Processing throws	500, event lost on give-up	bounded retries → dead-letter queue
Recover after bug fix	manual, error-prone	replay DLQ (idempotent, so safe)

After the design, a duplicate payment.succeeded conflicts on processed_events.event_id and does nothing; a reordered older event fails the version < predicate and updates zero rows; an entitlements outage retries with backoff and, if still failing, dead-letters the event with full context; and once the outage clears, the DLQ replays without any risk of double-granting because every step is idempotent.

Output:

Metric	Naive handler	This pipeline
Forged payloads accepted	possible	rejected (401)
Double-charge on retry	yes	never (dedup)
Stale overwrite	yes	never (version guard)
Events lost on failure	yes	never (DLQ)
Recovery after outage	manual replay	idempotent DLQ replay

Why this works — concept by concept:

• Verify before parse, over raw bytes — the HMAC is computed over the exact bytes the provider signed. Parsing and re-serializing JSON changes whitespace and key order, breaking the signature, so verification runs on request.get_data() before json.loads.
• Accept then process — ingress does the minimum (verify, enqueue, ACK) so the provider never times out and retries; all slow, fallible work moves to the worker. This removes the self-inflicted duplicate storm that inline processing causes.
• UNIQUE event_id claimed in the same transaction as the side effect — the dedup insert and the state write commit together. Either the event is claimed and applied, or neither happens; there is no window where the event is marked processed but the work didn't land.
• Version guard — WHERE version < incoming makes the state write a monotonic, last-writer-by-version operation. Reordered deliveries update zero rows instead of resurrecting stale data.
• Cost — one HMAC and one enqueue per request on the ingress path (O(1)); one indexed insert plus one guarded update per event on the worker (O(1)); a bounded number of retries and at most one DLQ row per permanently-failing event. The eliminated cost is the reconciliation job, the duplicate-charge refunds, and the 3 AM "where did that event go" incident.

Streaming
Topic — streaming
Streaming ingestion and event-pipeline problems

Practice →

Design Topic — design Design problems on event-ingestion architectures

Practice →

---

2. Signature verification and the fast-ACK boundary

HMAC over the raw body, a timestamp window against replay, and a 2xx returned before any work

The mental model in one line: signature verification is the pattern where the receiver recomputes an HMAC-SHA256 over the exact raw request bytes (plus a signed timestamp) using the shared webhook secret and compares it, in constant time, to the signature header — rejecting any request whose signature doesn't match or whose timestamp is outside a tolerance window — and it must happen on the request path, before parsing, alongside a fast 2xx ACK that hands the real work to an async worker. TLS authenticates the channel; the HMAC authenticates the sender; the timestamp window authenticates the freshness. All three are required.

The four things signature verification must get right.

• Verify over the raw bytes. The provider signs the literal request body. If your framework parses JSON and you re-serialize it to verify, whitespace and key ordering differ and every signature fails. Capture the raw body before any JSON parsing and HMAC that.
• Constant-time compare. Comparing the computed and provided signatures with == leaks timing information an attacker can use to forge a signature byte by byte. Use hmac.compare_digest (Python), crypto.timingSafeEqual (Node), or the equivalent — a compare whose duration doesn't depend on where the first mismatch is.
• Timestamp tolerance. The signature alone doesn't stop a replay attack: an attacker who captures one valid signed request can resend it forever. Providers include a signed timestamp; the receiver rejects requests older than a tolerance (Stripe uses 5 minutes) so captured requests go stale.
• Reject before work. Verification is a gate. An invalid signature returns 401 (or 400) immediately — no parsing, no enqueue, no processing. The gate is the cheapest possible operation and runs first.

The fast-ACK boundary — accept is not process.

• The SLA. Providers expect a 2xx within a few seconds (Stripe ~ a handful of seconds; many providers stricter). Anything slower is treated as a failure and retried.
• The trap. Doing the business work inline — DB writes, calling downstreams, sending email — makes the response slow, which triggers retries, which floods you with duplicates. Slow processing causes the duplicate problem.
• The fix. Verify, durably enqueue the raw event, return 200. The enqueue is the only write on the request path and it is fast. All heavy work is a worker's problem.
• Durability of the buffer. "Enqueue" must be durable: SQS, Kafka, or an outbox/raw_events table committed before the 200. If you ACK and then lose the event because it was only in memory, you've told the provider "got it" and dropped it.

Common interview probes on signature verification.

• "Why verify the raw body, not the parsed JSON?" — re-serialization changes bytes; the HMAC breaks.
• "Why a timestamp window?" — to stop replay of a captured valid request.
• "Why constant-time compare?" — to prevent timing side-channel signature forgery.
• "Why return 200 before processing?" — so slow work doesn't trigger provider retries and duplicates.

Worked example — HMAC-SHA256 verification

Detailed explanation. The canonical verification routine: read the raw body and the signature header, recompute HMAC-SHA256(secret, signed_payload), and compare in constant time. Providers differ only in what string they sign (some sign timestamp.body, some sign just the body, some send multiple candidate signatures during secret rotation). Build a verifier for a Stripe-style scheme where the signed payload is "{timestamp}.{raw_body}".

• Header. X-Signature: t=1718000000,v1=5f3c... — a timestamp and one or more signatures.
• Signed payload. The string "{t}.{raw_body}".
• Compare. Constant-time against each v1 candidate (to support key rotation).

Question. Implement a verifier that parses the header, recomputes the HMAC, and returns True only on a constant-time match within the tolerance window.

Input.

Element	Value
Header format	t=<unix>,v1=<hex>[,v1=<hex>]
Signed payload	"{t}.{raw_body}"
Algorithm	HMAC-SHA256, hex digest
Tolerance	300 seconds

Code.

import hmac, hashlib, time

def verify_webhook(raw_body: bytes, header: str, secret: bytes,
                   tolerance: int = 300) -> bool:
    # 1. Parse "t=...,v1=...,v1=..." into a timestamp and candidate sigs
    parts = dict(kv.split("=", 1) for kv in header.split(","))
    ts_str = parts.get("t")
    if ts_str is None:
        return False
    candidates = [v for k, v in
                  (kv.split("=", 1) for kv in header.split(",")) if k == "v1"]

    # 2. Freshness check — reject stale (replay) requests
    try:
        ts = int(ts_str)
    except ValueError:
        return False
    if abs(time.time() - ts) > tolerance:
        return False

    # 3. Recompute the expected signature over "{t}.{raw_body}"
    signed_payload = f"{ts}.".encode() + raw_body
    expected = hmac.new(secret, signed_payload, hashlib.sha256).hexdigest()

    # 4. Constant-time compare against every candidate (supports rotation)
    return any(hmac.compare_digest(expected, c) for c in candidates)

Step-by-step explanation.

1. Step 1 parses the signature header into its timestamp t and one or more v1 signatures. Multiple v1 values appear during secret rotation, when the provider signs with both the old and new secret so you can roll over without downtime.
2. Step 2 is the freshness gate. abs(now - ts) > tolerance rejects a request whose signed timestamp is more than five minutes from now — in either direction, to also reject clock-skewed or future-dated forgeries. This is what makes a captured-and-replayed request fail.
3. Step 3 reconstructs the exact string the provider signed — "{t}.{raw_body}" — using the raw body bytes, and computes the HMAC-SHA256 hex digest with the shared secret. Any mutation of the body (pretty-printing, key reordering) changes this digest.
4. Step 4 compares the computed digest against each candidate signature using hmac.compare_digest, whose runtime is independent of the position of the first differing byte. any(...) accepts if any candidate matches, which is what enables zero-downtime secret rotation.
5. If parsing fails, the timestamp is stale, or no candidate matches, the function returns False and the caller responds 401 — before parsing JSON or doing any work.

Output.

Case	Result
Valid signature, fresh timestamp	True (accept)
Valid signature, timestamp 10m old	False (stale — replay defense)
Body mutated after signing	False (digest mismatch)
Old secret during rotation window	True (matched second v1)
Tampered signature	False (constant-time mismatch)

Rule of thumb. Verify over the raw bytes, include the timestamp in the signed payload, compare with a constant-time function, and support multiple candidate signatures so secret rotation never causes an outage. Never == on a signature.

Worked example — the replay-window guard

Detailed explanation. The signature proves the payload came from someone holding the secret, but it does not prove when. Without a freshness check, an attacker who captures one valid request (or a well-meaning proxy that retries an old one) can replay it indefinitely, re-triggering the side effect. The fix is a signed timestamp plus a tolerance window, and — for defense in depth on sensitive events — recording recently-seen (event_id, timestamp) pairs so an in-window replay is also caught. Walk through both layers.

• Layer 1 — tolerance window. Reject requests whose signed timestamp is outside ±5 minutes of now.
• Layer 2 — seen-set (optional). For high-value events, remember event ids seen in the last window; reject a second arrival within the window as a replay (distinct from the idempotency dedup, which accepts duplicates as no-ops).
• The nuance. Legitimate provider retries also resend the same event; the timestamp window plus idempotency dedup handles those gracefully. The seen-set is only for rejecting malicious in-window replays where required.

Question. Add a timestamp tolerance and an optional short-TTL seen-set to the verifier, and explain how it coexists with legitimate retries.

Input.

Component	Value
Tolerance window	±300 s
Seen-set store	Redis, TTL = 600 s
Seen-set key	seen:{event_id}
Legit retry handling	idempotency dedup (not rejection)

Code.

import time

def check_freshness(ts: int, tolerance: int = 300) -> bool:
    """Layer 1 — reject requests outside the tolerance window."""
    return abs(time.time() - ts) <= tolerance

def check_replay(redis, event_id: str, ttl: int = 600) -> bool:
    """Layer 2 — reject a second in-window arrival for sensitive events.
    Returns True if this is the FIRST time we've seen the id in the window."""
    # SET key value NX EX ttl  → set only if absent, auto-expire
    first_seen = redis.set(f"seen:{event_id}", "1", nx=True, ex=ttl)
    return bool(first_seen)

def accept_request(raw_body, header, secret, redis, sensitive=False) -> bool:
    if not verify_webhook(raw_body, header, secret):   # HMAC + freshness
        return False
    if sensitive:
        event_id = json.loads(raw_body)["id"]
        if not check_replay(redis, event_id):
            # Seen inside the window: treat as replay for sensitive ops.
            # For ordinary events we'd let it through and rely on idempotency.
            return False
    return True

Step-by-step explanation.

1. Layer 1 (check_freshness) is the always-on guard already inside verify_webhook: any request whose signed timestamp is more than the tolerance from now is rejected. This alone defeats an attacker replaying a request captured hours ago.
2. Layer 2 (check_replay) uses Redis SET ... NX EX — set the key only if it doesn't exist, with a TTL matching the tolerance window. The first arrival sets the key and returns True (accept); a second arrival within the TTL finds the key present and returns False.
3. The subtlety is that legitimate provider retries also resend the same event_id. For ordinary events you do not want to reject those at the edge — you want to accept them and let the idempotency dedup in the worker make them no-ops. The seen-set rejection is reserved for sensitive=True operations where an in-window replay must be refused outright.
4. The tolerance window and the seen-set TTL should match: if the window is 5 minutes, a seen-set TTL of ~10 minutes covers the window plus clock skew without growing unbounded.
5. Together the two layers mean: stale replays fail the timestamp check; in-window malicious replays of sensitive events fail the seen-set; and legitimate retries flow through to the idempotency layer, which absorbs them.

Output.

Scenario	Layer 1 (window)	Layer 2 (seen-set)	Outcome
Captured request replayed after 1h	fail	—	rejected
In-window replay of sensitive event	pass	fail	rejected
Legitimate provider retry (ordinary)	pass	(skipped)	accepted → deduped in worker
First delivery	pass	pass	accepted

Rule of thumb. Always enforce the timestamp tolerance window; add a short-TTL seen-set only for sensitive operations, and never let the seen-set reject legitimate retries — those are the idempotency layer's job. Keep the seen-set TTL slightly larger than the tolerance window.

Worked example — the fast-ACK ingestion endpoint

Detailed explanation. The endpoint's job is to authenticate and durably record the event, then return 200 as fast as possible. Any business logic on the request path is a latency and duplicate liability. Build the endpoint so it commits the raw event to a durable buffer before the 200, then let a worker do everything else. Walk through the two durable-buffer options: a database raw_events table (transactional outbox style) and a managed queue.

• Option A — outbox table. Insert the raw event into raw_events and commit; return 200. A worker polls the table. Simplest; no extra infrastructure.
• Option B — managed queue. Send to SQS/Kafka; return 200. Scales to high volume; the queue is the durable buffer.
• The invariant. The durable write must succeed before the 200. If you ACK without durably recording, a crash loses an event you claimed to accept.

Question. Implement the endpoint both ways and show why the durable write must precede the ACK.

Input.

Element	Option A (table)	Option B (queue)
Durable buffer	raw_events Postgres table	SQS / Kafka
Commit point	before 200	send acked before 200
Worker	table poller	queue consumer
Best for	low/medium volume	high volume / bursts

Code.

# Option A — outbox table as the durable buffer
@app.post("/webhooks/payments")
def ingress_table():
    raw = request.get_data()
    if not verify_webhook(raw, request.headers.get("X-Signature", ""), SECRET):
        return Response(status=401)
    event = json.loads(raw)

    with db() as conn, conn.cursor() as cur:
        # Durably record BEFORE acking. ON CONFLICT so a duplicate
        # delivery at the edge doesn't error the insert.
        cur.execute("""
            INSERT INTO raw_events(event_id, event_type, payload, received_at)
            VALUES (%s, %s, %s, now())
            ON CONFLICT (event_id) DO NOTHING
        """, (event["id"], event["type"], raw.decode()))
        conn.commit()                              # committed → safe to ACK
    return Response(status=200)


# Option B — managed queue as the durable buffer
@app.post("/webhooks/payments")
def ingress_queue():
    raw = request.get_data()
    if not verify_webhook(raw, request.headers.get("X-Signature", ""), SECRET):
        return Response(status=401)
    event = json.loads(raw)

    sqs.send_message(                              # returns only after durable
        QueueUrl=QUEUE_URL,
        MessageBody=raw.decode(),
        MessageAttributes={"event_id": {"DataType": "String",
                                        "StringValue": event["id"]}},
        MessageDeduplicationId=event["id"],        # FIFO edge-dedup (optional)
    )
    return Response(status=200)                     # ACK after send succeeds

Step-by-step explanation.

1. Both handlers verify the signature first and return 401 on failure — no durable write, no ACK for an unauthenticated request.
2. Option A inserts the raw event into raw_events and commit()s before returning 200. The ON CONFLICT (event_id) DO NOTHING makes an edge-level duplicate delivery a harmless no-op rather than a primary-key violation that would 500 and trigger a retry.
3. Option B calls sqs.send_message, which returns only after SQS has durably stored the message. The 200 is returned after that call succeeds, so the provider is only told "accepted" once the event is safely buffered.
4. The ordering — durable write, then ACK — is the load-bearing invariant. If you returned 200 first and the process died before the write, the provider considers the event delivered and never resends it: a silent loss. Committing first means a crash after the commit is fine (the worker still has it) and a crash before the commit means no 200, so the provider retries.
5. Neither handler does business logic. The worker (table poller or queue consumer) owns dedup, ordering, processing, and dead-lettering. The endpoint stays in the single-digit-millisecond range regardless of how slow the downstream work is.

Output.

Failure point	Option A / B behaviour	Event lost?
Crash before durable write	no 200 → provider retries	no
Crash after write, before 200	no 200 → provider retries → edge dedup	no
Crash after 200	event is buffered; worker processes	no
Downstream slow	irrelevant — not on request path	no

Rule of thumb. The endpoint's contract is "durably record, then ACK." Commit the raw event to a queue or table before returning 200, keep all business logic off the request path, and make the edge write idempotent so a duplicate delivery at ingress can't error.

Senior interview question on signature verification and fast ACK

A senior interviewer might ask: "Design the request-path half of a webhook receiver for a payments provider. Cover exactly how you verify authenticity, how you defend against replay attacks, why you return 200 before processing, and what could still go wrong between 'return 200' and 'the event is safely stored.'"

Solution Using raw-body HMAC + timestamp guard + durable enqueue before ACK

import hmac, hashlib, json, time
from flask import Flask, request, Response

app = Flask(__name__)
SECRET = b"whsec_..."
TOLERANCE = 300

def verify(raw: bytes, header: str) -> bool:
    try:
        fields = dict(kv.split("=", 1) for kv in header.split(","))
        ts = int(fields["t"])
    except (ValueError, KeyError):
        return False
    if abs(time.time() - ts) > TOLERANCE:          # replay-window guard
        return False
    expected = hmac.new(SECRET, f"{ts}.".encode() + raw,
                        hashlib.sha256).hexdigest()
    candidates = [v for k, v in
                  (kv.split("=", 1) for kv in header.split(",")) if k == "v1"]
    return any(hmac.compare_digest(expected, c) for c in candidates)

@app.post("/webhooks/payments")
def ingress():
    raw = request.get_data()                        # RAW bytes
    if not verify(raw, request.headers.get("X-Signature", "")):
        return Response("invalid signature", status=401)

    event = json.loads(raw)
    with db() as conn, conn.cursor() as cur:
        cur.execute("""
            INSERT INTO raw_events(event_id, event_type, payload, received_at)
            VALUES (%s, %s, %s, now())
            ON CONFLICT (event_id) DO NOTHING       -- edge idempotency
        """, (event["id"], event["type"], raw.decode()))
        conn.commit()                               # durable BEFORE ack
    return Response(status=200)

-- The durable ingress buffer (outbox pattern for webhooks)
CREATE TABLE raw_events (
    event_id     TEXT        PRIMARY KEY,           -- edge dedup + worker key
    event_type   TEXT        NOT NULL,
    payload      TEXT        NOT NULL,              -- exact raw bytes
    received_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    processed_at TIMESTAMPTZ                        -- NULL until a worker finishes
);

CREATE INDEX idx_raw_events_unprocessed
    ON raw_events (received_at) WHERE processed_at IS NULL;

Step-by-step trace.

Concern	Mechanism	Result
Sender authenticity	HMAC over raw body	forged payloads rejected 401
Freshness	timestamp within ±5 min	captured replays go stale
Timing side channel	hmac.compare_digest	no byte-by-byte forgery
Secret rotation	multiple v1 candidates	zero-downtime rollover
Slow processing	ACK before processing	no retry-storm duplicates
Crash after ACK	committed to raw_events first	worker still has the event

After deployment, the endpoint verifies each request in microseconds, commits the raw event to raw_events, and returns 200 in a few milliseconds. An invalid signature never reaches the database. A crash anywhere before the commit means no 200, so the provider retries; a crash after the commit is harmless because the event is durably buffered for the worker.

Output:

Metric	Value
Request-path latency (p99)	single-digit ms
Forged requests accepted	0 (401)
Replayed stale requests accepted	0 (window)
Events lost between ACK and storage	0 (commit-before-ACK)
Secret rotation downtime	0 (multi-candidate verify)

Why this works — concept by concept:

• Raw-body HMAC — computing the digest over request.get_data() before any JSON parsing guarantees the bytes match what the provider signed. This is the authenticity anchor TLS cannot provide.
• Timestamp tolerance window — folding the signed timestamp into both the HMAC input and a freshness check turns a valid-forever signature into a valid-for-five-minutes one, which is what actually stops replay attacks.
• compare_digest — a constant-time comparison denies the attacker the timing side channel needed to brute-force a signature one byte at a time.
• Commit before ACK — the durable write to raw_events precedes the 200, so "the provider thinks it's delivered" is only ever true when the event is genuinely stored. The ON CONFLICT makes an edge duplicate a no-op.
• Cost — one HMAC computation and one indexed insert per request; no business logic on the hot path. The eliminated cost is the retry storm from slow processing and the silent event loss from acking before persisting. O(1) per request, and the request path stays fast no matter how slow the downstream is.

Validation
Topic — data-validation
Data-validation and signature-verification problems

Practice →

JSON Topic — json JSON payload parsing and extraction problems

Practice →

---

3. Idempotency and deduplication

At-least-once delivery guarantees duplicates — a UNIQUE event id makes reprocessing a no-op

The mental model in one line: idempotency for webhooks is the property that processing the same event any number of times has the same effect as processing it once, achieved by recording the provider's globally-unique event id in a durable dedup store with a UNIQUE constraint and claiming that id in the same transaction as the side effect — so a second (or tenth) delivery of the same event conflicts on the id and becomes a no-op instead of a double-charge. Because delivery is at-least-once, deduplication is not an optimization you add later; it is the correctness invariant without which retries corrupt state.

Why duplicates are guaranteed, not rare.

• Timeouts create phantom successes. You process an event and start writing the 200; the provider's connection times out before the ACK arrives. From the provider's view you failed, so it retries — but you already did the work. Only idempotency saves you.
• Retries on any non-2xx. A transient blip — a 502 from a proxy, a brief deploy, a slow response — makes the provider resend. This is by design and happens constantly at scale.
• Provider at-least-once semantics. Most webhook systems are built on at-least-once messaging internally; even without your endpoint failing, the same event can be emitted twice.
• Manual redelivery. Dashboards (GitHub, Stripe) let operators redeliver events for debugging. Your handler must treat a redelivery exactly like a first delivery.

The dedup key — use the provider's event id.

• Prefer the provider's id. Stripe evt_..., GitHub X-GitHub-Delivery UUID, Shopify X-Shopify-Webhook-Id. These are globally unique and stable across retries of the same event — exactly what a dedup key needs.
• Not a content hash (usually). Hashing the payload dedups byte-identical bodies, but two genuinely distinct events could hash-collide semantics, and a legitimately re-emitted event with a new id would be wrongly dropped. Use the id the provider promises is per-event-unique.
• Composite when needed. If one delivery can carry multiple sub-events, dedup on (event_id, sub_id). If you consume the same event in multiple independent services, dedup on (event_id, consumer_name) so each consumer processes independently.

The dedup store — durable, indexed, TTL'd.

• Durable table with UNIQUE PK. processed_events(event_id PK, processed_at). The PK is the dedup mechanism; INSERT ... ON CONFLICT DO NOTHING returns rowcount 0 on a duplicate.
• Claim in the same transaction as the side effect. Insert the id and do the work in one transaction. If they were separate, a crash between them either double-processes or marks-processed-without-doing.
• Redis front line for volume. At very high volume, a Redis SET NX EX gives a fast first-line dedup, backed by the durable table as the source of truth (Redis can evict; the table cannot).
• TTL / retention. Providers retry for a bounded window (hours to days). Keep dedup rows longer than the max retry horizon plus a margin, then partition-drop or TTL old ids so the table doesn't grow forever.

Common interview probes on idempotency.

• "What's the dedup key?" — the provider's event id, unique across retries.
• "Where do you dedup?" — a durable UNIQUE store, claimed in the same transaction as the side effect.
• "How long do you keep dedup records?" — longer than the provider's retry window plus a margin.
• "Isn't a SELECT-then-INSERT enough?" — no; it races. Use INSERT ... ON CONFLICT (atomic) or a UNIQUE-violation catch.

Worked example — the dedup table with a UNIQUE event id

Detailed explanation. The canonical dedup store is a table whose primary key is the event id. Processing claims the id with INSERT ... ON CONFLICT DO NOTHING; if the insert affected zero rows, the event was already processed and the worker returns early. The claim and the business write share one transaction so they commit together. Build it for an order.updated webhook.

• Table. processed_events(event_id PK, event_type, processed_at).
• Claim. INSERT ... ON CONFLICT (event_id) DO NOTHING; check rowcount.
• Atomicity. Claim + side effect in one transaction.

Question. Write the dedup table and a handler that processes each event exactly once even under duplicate delivery.

Input.

Element	Value
Dedup key	event_id (provider UUID)
Store	Postgres processed_events
Claim	ON CONFLICT DO NOTHING
Transaction	claim + side effect together

Code.

CREATE TABLE processed_events (
    event_id     TEXT        PRIMARY KEY,           -- the idempotency key
    event_type   TEXT        NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

import json, psycopg2

def handle_event(conn, raw_payload: str) -> str:
    event = json.loads(raw_payload)
    event_id = event["id"]

    with conn:                                      # BEGIN ... COMMIT / ROLLBACK
        with conn.cursor() as cur:
            # 1. Atomically claim the event id
            cur.execute("""
                INSERT INTO processed_events(event_id, event_type)
                VALUES (%s, %s)
                ON CONFLICT (event_id) DO NOTHING
            """, (event_id, event["type"]))

            if cur.rowcount == 0:
                # Someone already claimed this id → duplicate → no-op
                return "duplicate"

            # 2. Do the real work in the SAME transaction
            apply_order_update(cur, event["data"])

    return "processed"                              # both committed together

Step-by-step explanation.

1. The processed_events table's primary key is the dedup mechanism — Postgres enforces uniqueness, so two concurrent workers cannot both claim the same id. No separate SELECT is needed.
2. INSERT ... ON CONFLICT (event_id) DO NOTHING is atomic: it either inserts the row (rowcount 1, first time) or does nothing (rowcount 0, duplicate). This avoids the classic SELECT-then-INSERT race where two workers both see "not present" and both process.
3. When rowcount == 0, this exact event id is already recorded, so the handler returns "duplicate" and does no work. A retried or redelivered event lands here.
4. When rowcount == 1, the handler does the business work (apply_order_update) inside the same with conn: transaction. The claim and the side effect commit atomically — there is no window where the id is marked processed but the work didn't happen, or vice versa.
5. If apply_order_update raises, the whole transaction rolls back — including the claim. The event is not marked processed, so a later retry can process it cleanly. Idempotency and correct failure handling come from the shared transaction.

Output.

Delivery	processed_events	Side effect	Return
1st delivery of evt_A	row inserted	applied	processed
2nd delivery of evt_A	conflict, no-op	skipped	duplicate
Redelivery from dashboard	conflict, no-op	skipped	duplicate
1st delivery of evt_B	row inserted	applied	processed

Rule of thumb. Make the primary key the dedup key, claim with INSERT ... ON CONFLICT DO NOTHING, check the rowcount, and put the claim and the side effect in one transaction. Never SELECT-then-INSERT — it races under concurrency.

Worked example — idempotent upsert of the side effect

Detailed explanation. Sometimes the dedup table alone isn't enough: the side effect itself writes to a business table, and you want that write to be idempotent too, so that even a bug that bypasses the dedup check can't create duplicate rows. The tool is a natural key plus INSERT ... ON CONFLICT DO UPDATE (upsert). Walk through making an invoices write idempotent on top of the dedup layer.

• The risk. Two workers, or a redelivery after a partial failure, could both try to insert the same invoice.
• The fix. A UNIQUE natural key on the business table (payment_id) plus an upsert, so a second write updates rather than duplicates.
• Belt and braces. Dedup table prevents reprocessing; the upsert makes the write itself idempotent. Defense in depth.

Question. Write an idempotent upsert for the invoice side effect keyed on the business id.

Input.

Element	Value
Business table	invoices
Natural key	payment_id UNIQUE
Write	ON CONFLICT (payment_id) DO UPDATE
Guard	version to avoid stale overwrite

Code.

CREATE TABLE invoices (
    payment_id   TEXT        PRIMARY KEY,
    customer_id  TEXT        NOT NULL,
    status       TEXT        NOT NULL,
    amount_cents BIGINT      NOT NULL,
    version      BIGINT      NOT NULL,
    updated_at   TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

def apply_invoice(cur, data: dict) -> None:
    """Idempotent upsert: same event applied twice = one row, correct state."""
    cur.execute("""
        INSERT INTO invoices(payment_id, customer_id, status, amount_cents, version)
        VALUES (%(payment_id)s, %(customer_id)s, %(status)s,
                %(amount_cents)s, %(version)s)
        ON CONFLICT (payment_id) DO UPDATE
          SET status       = EXCLUDED.status,
              amount_cents = EXCLUDED.amount_cents,
              version      = EXCLUDED.version,
              updated_at   = clock_timestamp()
          WHERE invoices.version < EXCLUDED.version   -- only if newer
    """, data)

Step-by-step explanation.

1. The invoices table uses the business id payment_id as its primary key. This natural key is what makes the write idempotent: the same payment can only ever occupy one row.
2. INSERT ... ON CONFLICT (payment_id) DO UPDATE upserts — first delivery inserts, any later delivery updates the existing row rather than creating a duplicate. Even if the dedup table were somehow bypassed, no duplicate invoice row can exist.
3. The WHERE invoices.version < EXCLUDED.version clause on the DO UPDATE folds in the ordering guard (section 4): the update only takes effect if the incoming event carries a newer version. A stale, reordered event upserts nothing.
4. EXCLUDED refers to the row proposed for insertion, so EXCLUDED.status is the incoming value. This is the standard Postgres upsert idiom for "apply the new values, but only conditionally."
5. Combined with the processed_events dedup, this is defense in depth: the dedup table stops the worker from redoing the work, and the upsert guarantees the write is idempotent and monotonic even if it runs twice. Two independent safeguards protect the business table.

Output.

Sequence	invoices row	Note
Insert v5 (pending)	1 row, version 5	created
Re-apply v5 (duplicate)	1 row, version 5	conflict → update, no change
Apply v6 (paid)	1 row, version 6	updated (newer)
Late v4 arrives	1 row, version 6	version < fails → ignored

Rule of thumb. Layer idempotency: a dedup table on the event id to skip reprocessing, plus an upsert on the business natural key with a version guard so the write itself is idempotent and monotonic. Two safeguards, because one bug shouldn't be able to double-write.

Worked example — Redis SETNX dedup window for high volume

Detailed explanation. At very high volume, hitting the durable dedup table on every single delivery — including the flood of duplicates — can be a bottleneck. A Redis SET key value NX EX ttl gives a fast, cheap front-line dedup: the first delivery sets the key, duplicates within the TTL are rejected in memory before touching Postgres. The durable table remains the source of truth because Redis can evict under memory pressure. Walk through the two-tier design.

• Tier 1 — Redis. SET seen:{event_id} 1 NX EX <ttl>. First delivery returns OK; duplicates return nil.
• Tier 2 — Postgres. The processed_events table, claimed transactionally with the side effect, is the durable truth.
• Why both. Redis is fast but can lose keys (eviction, restart). The table can't lose rows but is slower. Redis filters the duplicate flood; the table guarantees correctness.

Question. Add a Redis front-line dedup in front of the durable table and explain the failure semantics if Redis loses the key.

Input.

Tier	Store	Role	Failure mode
1	Redis SET NX EX	fast filter	eviction → falls through to Tier 2
2	Postgres UNIQUE	durable truth	none (authoritative)
TTL	~ retry window + margin	bound memory	—

Code.

import redis, json

r = redis.Redis()
DEDUP_TTL = 3 * 24 * 3600      # keep longer than the provider's retry window

def handle_high_volume(conn, raw_payload: str) -> str:
    event = json.loads(raw_payload)
    event_id = event["id"]

    # Tier 1 — fast in-memory filter for the duplicate flood
    if not r.set(f"seen:{event_id}", "1", nx=True, ex=DEDUP_TTL):
        return "duplicate (redis)"                  # already seen recently

    # Tier 2 — durable, authoritative dedup + side effect in one txn
    try:
        with conn:
            with conn.cursor() as cur:
                cur.execute("""
                    INSERT INTO processed_events(event_id, event_type)
                    VALUES (%s, %s) ON CONFLICT (event_id) DO NOTHING
                """, (event_id, event["type"]))
                if cur.rowcount == 0:
                    return "duplicate (db)"          # Redis missed it; DB caught it
                apply_side_effect(cur, event["data"])
        return "processed"
    except Exception:
        # Roll back the Redis claim so a retry isn't wrongly filtered
        r.delete(f"seen:{event_id}")
        raise

Step-by-step explanation.

1. Tier 1 runs SET seen:{event_id} 1 NX EX. The NX means "set only if absent"; the first delivery gets a truthy result and proceeds, while duplicates within the TTL get nil and return early — filtering the duplicate flood without touching Postgres.
2. Tier 2 is the authoritative dedup. Even if Redis evicted the key (memory pressure) or restarted (lost all keys), the processed_events UNIQUE constraint still catches the duplicate: the insert conflicts, rowcount == 0, and the handler returns "duplicate (db)". Redis is an optimization, not the source of truth.
3. The side effect runs in the same transaction as the durable claim, exactly as in the single-tier design. Redis only decides whether to attempt the transaction; correctness still comes from the database.
4. The failure path is critical: if the side effect throws, the code r.deletes the Redis key before re-raising. Otherwise the Redis claim would survive, and the subsequent retry would be wrongly filtered as a duplicate in Tier 1 even though the work never completed. Rolling back the Redis claim keeps the two tiers consistent.
5. The TTL is set longer than the provider's maximum retry window so Redis catches essentially all duplicates; the durable table catches the rare ones that slip through after a Redis eviction. Together they give fast dedup with a correctness floor.

Output.

Scenario	Tier 1 (Redis)	Tier 2 (DB)	Result
First delivery	set OK	insert OK	processed
Duplicate within TTL	nil	(skipped)	duplicate (redis)
Duplicate after Redis eviction	set OK	conflict	duplicate (db)
Side effect throws	claim deleted	rolled back	retried cleanly

Rule of thumb. Use Redis SET NX EX as a fast front-line filter and a durable UNIQUE table as the authoritative dedup. Always roll back the Redis claim if processing fails, and never treat Redis as the source of truth — it can evict; the table cannot.

Senior interview question on idempotency

A senior interviewer might ask: "Your webhook handler processes 5,000 events/sec from a provider that delivers at-least-once and lets operators manually redeliver. Design the deduplication so that no event is ever double-processed, the duplicate flood doesn't overload Postgres, and a crash mid-processing never leaves an event marked done-but-not-actually-done."

Solution Using a Redis front-line filter + durable UNIQUE dedup claimed with the side effect

import redis, json, psycopg2

r = redis.Redis()
DEDUP_TTL = 4 * 24 * 3600           # > provider's ~3-day retry window

def process(conn, raw: str) -> str:
    event = json.loads(raw)
    eid = event["id"]

    # Front line: filter the duplicate flood in memory
    if not r.set(f"seen:{eid}", "1", nx=True, ex=DEDUP_TTL):
        return "dup-redis"

    try:
        with conn:
            with conn.cursor() as cur:
                # Durable claim (authoritative even if Redis evicts)
                cur.execute("""
                    INSERT INTO processed_events(event_id, event_type)
                    VALUES (%s, %s) ON CONFLICT (event_id) DO NOTHING
                """, (eid, event["type"]))
                if cur.rowcount == 0:
                    return "dup-db"

                # Idempotent, version-guarded side effect (same txn)
                d = event["data"]
                cur.execute("""
                    INSERT INTO invoices(payment_id, customer_id, status,
                                         amount_cents, version)
                    VALUES (%s, %s, %s, %s, %s)
                    ON CONFLICT (payment_id) DO UPDATE
                      SET status = EXCLUDED.status,
                          amount_cents = EXCLUDED.amount_cents,
                          version = EXCLUDED.version
                      WHERE invoices.version < EXCLUDED.version
                """, (d["payment_id"], d["customer_id"], d["status"],
                      d["amount_cents"], d["version"]))
        return "processed"
    except Exception:
        r.delete(f"seen:{eid}")     # keep tiers consistent on failure
        raise

CREATE TABLE processed_events (
    event_id     TEXT        PRIMARY KEY,
    event_type   TEXT        NOT NULL,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);
-- Retention: drop dedup rows older than the retry window + margin
DELETE FROM processed_events WHERE processed_at < now() - INTERVAL '7 days';

Step-by-step trace.

Step	Value	Reasoning
Front-line filter	Redis SET NX EX	absorbs the duplicate flood off Postgres
Authoritative dedup	processed_events UNIQUE	survives Redis eviction/restart
Claim + side effect	one transaction	no done-but-not-done window
Side effect	version-guarded upsert	idempotent and monotonic write
Failure	delete Redis key + raise	retry not wrongly filtered
Retention	7-day TTL	> provider retry horizon

After deployment, the vast majority of duplicate deliveries are rejected in Redis without a database round trip; the rare duplicate that arrives after a Redis eviction is caught by the processed_events UNIQUE constraint; the claim and the invoice upsert commit atomically so a crash never leaves an event half-processed; and the nightly retention keeps the dedup table bounded.

Output:

Metric	Value
Duplicates filtered in Redis	~99%+ at steady state
Duplicates caught by DB (post-eviction)	remainder
Double-processed events	0
Done-but-not-done events	0 (shared transaction)
Dedup table size	bounded by 7-day TTL

Why this works — concept by concept:

• Provider event id as the key — the id is unique per event and stable across retries and manual redeliveries, so it is the correct dedup key. Content hashing would misfire on re-emitted events.
• Two-tier dedup — Redis SET NX EX filters the duplicate flood cheaply; the durable UNIQUE table is the correctness floor that Redis eviction can't undermine.
• Claim in the side-effect transaction — recording the event id and doing the work in one transaction removes the window where a crash could mark an event processed without doing it (or vice versa).
• Version-guarded upsert — makes the business write itself idempotent and monotonic, so even a bug bypassing dedup can't create a duplicate or a stale overwrite.
• Cost — one Redis op plus, for non-duplicates, one indexed insert and one guarded upsert per event; retention is an O(rows-in-window) periodic delete. The eliminated cost is the double-charge refunds and the reconciliation tooling. O(1) per event, and the duplicate flood never reaches the database.

SQL
Topic — sql
SQL deduplication and idempotent-upsert problems

Practice →

Events Topic — event-processing Event-processing deduplication problems

Practice →

---

4. Event ordering across retries

Arrival time is not causal order — a version guard keeps state monotonic per entity

The mental model in one line: event ordering for webhooks is the guarantee that the state an entity ends up in reflects the newest event about it, not the last one to arrive — achieved by carrying a monotonic version or sequence number in each event and applying a change only when its version exceeds the version already stored for that entity, so that out-of-order, delayed, or replayed deliveries can never overwrite newer data. Under at-least-once delivery over an unordered channel, wall-clock arrival order is meaningless; the version guard restores per-entity causal order.

Why webhooks arrive out of order.

• Fan-out across workers. Providers deliver from a pool of senders across regions; two events emitted a millisecond apart can traverse different paths and arrive seconds apart, swapped.
• Retries reshuffle. A failed-then-retried older event can land after a newer event that succeeded on the first try. Retries are the biggest source of reordering.
• No global order guarantee. Only a handful of providers offer ordered delivery, and even those scope it narrowly and don't preserve it across retries. Assume unordered.
• Consumer parallelism. Even if the provider sent in order, your own worker fleet processes in parallel, so two events for the same entity can be handled concurrently in either order.

The version guard — the core mechanism.

• Carry a monotonic version. The event payload must include a per-entity monotonically-increasing value: a version, a sequence number, or a reliable server-side updated_at from the source. The provider assigns it; you trust it.
• Apply only if newer. The state write is UPDATE ... WHERE version < :incoming (or the equivalent guard in an upsert). A stale event updates zero rows.
• Prefer version over timestamp. A version/sequence is exact; timestamps suffer clock skew and ties. If only a timestamp is available, treat equal timestamps carefully and prefer a tiebreaker.
• Per entity, not global. Ordering only needs to hold within an entity (one order, one customer). Global ordering across all entities is neither needed nor achievable cheaply.

Per-entity ordering via partition key.

• Partition by aggregate id. If you use Kafka or a partitioned queue, key messages by the entity id so all events for one entity land on one partition and are processed by one consumer in offset order — turning "unordered globally" into "ordered per entity."
• The guard is still required. Partitioning gives in-order arrival to a single consumer, but retries and reprocessing still mean the version guard is the durable correctness mechanism. Partitioning is an optimization; the guard is the guarantee.

Handling late and stale events.

• Stale update → drop. An older-version event for an entity you've already advanced is simply ignored (the guard updates zero rows). This is correct, not a loss.
• Missing earlier event. If you receive v3 before v1/v2 and your model needs the intermediate states, either request a resync/snapshot from the provider, or design the state to be fully described by the latest event (so intermediate states aren't required).
• Deletes and tombstones. A delete must also be version-ordered; a late "update" must not resurrect a deleted entity. Model deletes as a versioned state (deleted_at, status='deleted') so the guard applies to them too.

Common interview probes on ordering.

• "Webhooks arrive out of order — how do you handle it?" — version guard per entity; apply only if newer.
• "Timestamp or sequence for ordering?" — sequence/version (exact); timestamps skew and tie.
• "How do you keep a late update from resurrecting a deleted row?" — model delete as versioned state; the guard drops the stale update.
• "How does partitioning help?" — key by entity id so one consumer sees one entity's events in order; the guard still backs it.

Worked example — the version-guarded upsert

Detailed explanation. The core ordering primitive is an upsert whose DO UPDATE is gated on the incoming version being newer than the stored version. Any event — first, duplicate, in-order, or reordered — runs the same statement; the guard decides whether it takes effect. Build it for a subscription entity whose events carry a version.

• State. subscriptions(subscription_id PK, status, plan, version).
• Event. { id, type, data: { subscription_id, status, plan, version } }.
• Guard. ON CONFLICT ... DO UPDATE ... WHERE subscriptions.version < EXCLUDED.version.

Question. Write the version-guarded upsert and trace three events arriving out of order.

Input.

Event	version	arrives
A (active)	3	1st
B (past_due)	1	2nd
C (active)	2	3rd

Code.

CREATE TABLE subscriptions (
    subscription_id TEXT   PRIMARY KEY,
    status          TEXT   NOT NULL,
    plan            TEXT   NOT NULL,
    version         BIGINT NOT NULL
);

def apply_subscription(cur, d: dict) -> int:
    cur.execute("""
        INSERT INTO subscriptions(subscription_id, status, plan, version)
        VALUES (%(subscription_id)s, %(status)s, %(plan)s, %(version)s)
        ON CONFLICT (subscription_id) DO UPDATE
          SET status  = EXCLUDED.status,
              plan    = EXCLUDED.plan,
              version = EXCLUDED.version
          WHERE subscriptions.version < EXCLUDED.version
    """, d)
    return cur.rowcount        # 1 = applied, 0 = stale (ignored)

Step-by-step explanation.

1. Every event runs the identical statement — there is no branching on "is this in order?" The guard WHERE subscriptions.version < EXCLUDED.version does all the ordering work declaratively.
2. Event A (version 3) arrives first. The row doesn't exist, so the INSERT path runs and creates subscription = active, version 3. rowcount == 1 (applied).
3. Event B (version 1) arrives second. The row exists at version 3; the ON CONFLICT DO UPDATE fires but its WHERE 3 < 1 is false, so it updates zero rows. rowcount == 0 (stale, correctly ignored). Without the guard, B would have overwritten the newer state with past_due.
4. Event C (version 2) arrives third. Again WHERE 3 < 2 is false → zero rows. The stored state stays at version 3. Both out-of-order older events were dropped.
5. The end state is active, version 3 — the newest event — regardless of the scrambled arrival order. The guard made the outcome depend on causal version, not on wall-clock arrival.

Output.

Step	Arrives	Guard	Stored state
1	A v3	insert	active, v3
2	B v1	3 < 1 false	active, v3 (B dropped)
3	C v2	3 < 2 false	active, v3 (C dropped)
final	—	—	active, v3

Rule of thumb. Run the same version-guarded upsert for every event; let WHERE stored.version < incoming.version decide. The final state converges to the highest version no matter the arrival order, and duplicates are naturally absorbed (equal version also fails <).

Worked example — per-entity ordering via partition key

Detailed explanation. When throughput demands a worker fleet, you still want all events for one entity handled in a consistent order. Keying the durable buffer by the entity id routes every event for that entity to one partition and thus one consumer, which sees them in offset order. Walk through partitioning a Kafka-backed pipeline by subscription_id.

• Producer. Key each message by subscription_id so the partitioner is deterministic.
• Consequence. One entity's events all land on one partition, consumed by one worker in order.
• Still guarded. Retries and reprocessing mean the version guard remains the durable correctness mechanism; partitioning reduces contention and reordering, it doesn't replace the guard.

Question. Configure the producer to partition by entity and explain what partitioning does and does not guarantee.

Input.

Element	Value
Partition key	subscription_id
Effect	one entity → one partition → one consumer
Guarantees	in-order arrival to that consumer
Does not guarantee	correctness under retries (guard does)

Code.

# Producer — key by entity so one entity's events share a partition
def publish(producer, event: dict) -> None:
    key = event["data"]["subscription_id"].encode()   # partition key
    producer.produce(
        topic="subscription-events",
        key=key,                                       # same key → same partition
        value=json.dumps(event).encode(),
    )

# Consumer — process in offset order, still apply the version guard
def consume_loop(consumer, conn):
    for msg in consumer:
        event = json.loads(msg.value())
        with conn:
            with conn.cursor() as cur:
                # dedup + version-guarded upsert as before
                cur.execute("""
                    INSERT INTO processed_events(event_id, event_type)
                    VALUES (%s, %s) ON CONFLICT DO NOTHING
                """, (event["id"], event["type"]))
                if cur.rowcount:
                    apply_subscription(cur, event["data"])   # guarded
        consumer.commit(msg)

Step-by-step explanation.

1. The producer keys each message by subscription_id. Kafka's default partitioner hashes the key, so every event for a given subscription deterministically lands on the same partition.
2. A partition is consumed by exactly one consumer in a group, in strictly increasing offset order. So one consumer sees all of subscription_42's events in the order they were produced — turning a globally-unordered stream into per-entity ordering.
3. This drastically reduces reordering and eliminates concurrent processing of the same entity by two workers, which removes a class of race conditions on the state row.
4. It does not make the version guard optional. A message can still be redelivered (consumer restart before commit), and produce order isn't guaranteed to equal causal order if the provider itself reordered. So the consumer still deduplicates and applies the version-guarded upsert.
5. The division of labor: partitioning is the throughput-and-contention optimization (one entity, one consumer, in order); the version guard is the durable correctness guarantee (final state is the newest version regardless). Use both; rely on the guard.

Output.

Property	Provided by
One entity handled by one consumer	partition key
In-order arrival to that consumer	partition offsets
No two workers racing one entity	partition key
Correct final state under retries	version guard
Duplicate suppression	dedup table

Rule of thumb. Partition the durable buffer by entity id to get per-entity in-order processing and eliminate same-entity races, but keep the version-guarded upsert as the correctness backstop. Partitioning optimizes; the guard guarantees.

Worked example — stale events and delete resurrection

Detailed explanation. The nastiest ordering bug is a late "update" that resurrects a deleted entity: you process delete (version 5), then a delayed update (version 4) arrives and re-inserts the row. The fix is to model deletion as versioned state rather than a physical DELETE, so the version guard applies to it too. Walk through a soft-delete design that is immune to resurrection.

• The bug. Physical DELETE erases the version, so a late lower-version update's guard has nothing to compare against and re-creates the row.
• The fix. Model delete as status='deleted', version=N. The row (and its version) persists, so a late update with a lower version fails the guard.
• Cleanup. Physically purge deleted rows only after a window longer than the provider's retry horizon, out of band.

Question. Design a soft-delete that the version guard protects, and trace a delete followed by a late lower-version update.

Input.

Event	version	effect intended
delete	5	mark deleted
update (late)	4	must NOT resurrect

Code.

-- Soft-delete: the row and its version survive a delete
ALTER TABLE subscriptions ADD COLUMN deleted_at TIMESTAMPTZ;

def apply_event(cur, event: dict) -> int:
    d = event["data"]
    if event["type"] == "subscription.deleted":
        # Delete is a versioned UPDATE, not a physical DELETE
        cur.execute("""
            UPDATE subscriptions
            SET status = 'deleted', deleted_at = now(), version = %(version)s
            WHERE subscription_id = %(subscription_id)s
              AND version < %(version)s
        """, d)
    else:
        cur.execute("""
            INSERT INTO subscriptions(subscription_id, status, plan, version)
            VALUES (%(subscription_id)s, %(status)s, %(plan)s, %(version)s)
            ON CONFLICT (subscription_id) DO UPDATE
              SET status = EXCLUDED.status, plan = EXCLUDED.plan,
                  version = EXCLUDED.version
              WHERE subscriptions.version < EXCLUDED.version
                AND subscriptions.deleted_at IS NULL        -- never un-delete
        """, d)
    return cur.rowcount

Step-by-step explanation.

1. The delete is modeled as an UPDATE ... SET status='deleted', version=5 WHERE version < 5. The row and its version survive; the entity is logically gone but the version marker remains for future guards to compare against.
2. When the late update (version 4) arrives, its upsert runs WHERE subscriptions.version < 4. The stored version is 5, so 5 < 4 is false → zero rows. The stale update is dropped and the entity stays deleted.
3. The extra AND subscriptions.deleted_at IS NULL guard is belt-and-braces: even a higher-version non-delete event won't un-delete a soft-deleted entity unless your domain explicitly allows resurrection. Choose the policy deliberately.
4. Had the delete been a physical DELETE, the row and its version 5 would be gone; the late version-4 update would find no conflicting row and insert it, resurrecting the entity at a stale state. Soft-delete is what makes the guard work across deletes.
5. Physical cleanup of soft-deleted rows happens out of band, on a schedule longer than the provider's retry window, so no late event can arrive after the row is purged.

Output.

Step	Event	Guard	Stored
1	delete v5	version←5	deleted, v5
2	update v4 (late)	5 < 4 false	deleted, v5 (dropped)
final	—	—	deleted, v5

Rule of thumb. Never physically DELETE in a webhook pipeline where late events are possible — model deletion as versioned soft-delete so the ordering guard protects it, and purge deleted rows out of band after the retry window closes. A physical delete is how a late update resurrects a dead entity.

Senior interview question on event ordering

A senior interviewer might ask: "Your subscription service receives webhooks that arrive out of order and are redelivered on failure. Design the state model so that the stored subscription always reflects the newest event, an out-of-order older event never overwrites it, and a delayed update can never bring back a subscription that was already cancelled."

Solution Using version-guarded state with per-entity partitioning and versioned soft-delete

import json

def handle(cur, event: dict) -> str:
    eid = event["id"]
    # 1. Dedup (idempotency)
    cur.execute("""
        INSERT INTO processed_events(event_id, event_type)
        VALUES (%s, %s) ON CONFLICT (event_id) DO NOTHING
    """, (eid, event["type"]))
    if cur.rowcount == 0:
        return "duplicate"

    d = event["data"]
    # 2. Version-guarded state, delete-safe
    if event["type"] == "subscription.deleted":
        cur.execute("""
            UPDATE subscriptions
            SET status='deleted', deleted_at=now(), version=%(version)s
            WHERE subscription_id=%(subscription_id)s AND version < %(version)s
        """, d)
    else:
        cur.execute("""
            INSERT INTO subscriptions(subscription_id, status, plan, version)
            VALUES (%(subscription_id)s, %(status)s, %(plan)s, %(version)s)
            ON CONFLICT (subscription_id) DO UPDATE
              SET status=EXCLUDED.status, plan=EXCLUDED.plan,
                  version=EXCLUDED.version
              WHERE subscriptions.version < EXCLUDED.version
                AND subscriptions.deleted_at IS NULL
        """, d)
    return "applied" if cur.rowcount else "stale-ignored"

# Producer keys by entity so one subscription's events share a partition
def publish(producer, event: dict) -> None:
    producer.produce(
        topic="subscription-events",
        key=event["data"]["subscription_id"].encode(),   # per-entity order
        value=json.dumps(event).encode(),
    )

CREATE TABLE subscriptions (
    subscription_id TEXT        PRIMARY KEY,
    status          TEXT        NOT NULL,
    plan            TEXT        NOT NULL,
    version         BIGINT      NOT NULL,
    deleted_at      TIMESTAMPTZ                    -- versioned soft-delete
);

Step-by-step trace.

Concern	Mechanism	Result
Duplicate delivery	dedup table	reprocessing is a no-op
Out-of-order update	version < EXCLUDED.version	older event updates 0 rows
Newest event wins	version guard	state converges to max version
Delete then late update	versioned soft-delete + guard	entity stays deleted
Same-entity races	partition by subscription_id	one consumer, in order

After deployment, whatever order the events arrive in, the stored subscription converges to the highest-version event; an older reordered event updates zero rows; a cancelled subscription cannot be resurrected by a delayed update because the delete kept its version and the guard rejects the older event; and partitioning by subscription_id means one consumer handles one subscription's events without racing another worker.

Output:

Metric	Naive (last-arrival-wins)	Version-guarded
Stale overwrite	frequent	never
Delete resurrection	possible	impossible
Final state correctness	order-dependent	order-independent
Same-entity race	possible	eliminated (partitioned)
Duplicate effect	double-applied	no-op

Why this works — concept by concept:

• Monotonic version per entity — a per-entity increasing version makes "newer" an exact, skew-free comparison, so the state write can be conditioned on it deterministically.
• Guarded upsert — WHERE stored.version < incoming.version makes every write monotonic; the final state is the maximum version regardless of arrival order, and equal-version duplicates are absorbed for free.
• Versioned soft-delete — modeling delete as a versioned state keeps the version marker alive, so a late lower-version update fails the guard instead of resurrecting the entity.
• Partition by entity id — routing one entity's events to one consumer removes same-entity concurrency and most reordering; the guard covers the residual reordering from retries.
• Cost — one guarded upsert (or update) per event plus one dedup insert, all O(1); partitioning adds no per-event cost. The eliminated cost is the data-corruption incidents from stale overwrites and delete resurrection, which are the hardest webhook bugs to detect after the fact.

Events
Topic — event-processing
Event-ordering and sequencing problems

Practice →

Streaming Topic — streaming Streaming ordering and partitioning problems

Practice →

---

5. Dead-letter queues and replay

Bounded retries, then a dead-letter queue you can replay — because processing is idempotent

The mental model in one line: a dead-letter queue is the durable holding area for events that failed processing after a bounded number of retries, capturing the full payload, headers, error, and attempt count so an engineer can diagnose the failure and, once fixed, replay the event back through the normal handler — and this whole scheme is only safe because the processing path is idempotent, so replaying an event that partially succeeded can't double-apply its side effect. Retries handle transient failures; the DLQ isolates poison messages so one bad event can't block the stream; replay is the recovery valve.

Retry first — but bounded, with backoff and jitter.

• Transient vs permanent. Most failures are transient: a downstream is briefly down, a lock timed out, a rate limit hit. A retry after a short wait usually succeeds. Retrying is the first line of defense.
• Exponential backoff. Wait base * 2^attempt between retries (1s, 2s, 4s, 8s...) so a struggling downstream isn't hammered. Add jitter (randomize the wait) so a fleet of workers doesn't retry in a synchronized thundering herd.
• Bounded attempts. Retrying forever turns a permanently-bad event into an infinite loop that blocks the queue. Cap attempts (e.g. 5); after that, dead-letter.
• Distinguish retryable from fatal. A 503 from a downstream is retryable; a schema-violation or a business-rule rejection is fatal — dead-letter it immediately rather than wasting retries.

The dead-letter queue — what to capture.

• The full raw payload. Store the exact bytes so you can replay verbatim (and re-verify if needed).
• Headers and metadata. Signature, timestamp, event_id, event_type, source — everything needed to reprocess and to debug.
• Failure context. The error message/stack, the failing attempt count, and the first-and-last failure timestamps.
• Status. new → investigating → replayed / discarded, so the DLQ is a workflow, not a graveyard.

Poison-message isolation.

• Don't let one bad event block the stream. In an ordered partition, a permanently-failing event at the head blocks everything behind it. Moving it to the DLQ after max attempts unblocks the partition.
• The trade-off with ordering. Skipping a poison event in an ordered stream means later events for that entity proceed without it. For entity-critical ordering, dead-lettering the whole entity's subsequent events (or pausing that key) may be required — a deliberate design choice.

Replay — safe because idempotent.

• Replay path = normal path. Re-inject the DLQ event through the same handler. The dedup table and version guard make a replay of an already-partially-applied event a no-op or a correct newer-only apply.
• Fix, then replay. Replay after the root cause is fixed (downstream back up, bug deployed, schema handled). Replaying into the same failure just re-dead-letters.
• Bulk vs selective. Replay one event, a filtered set (all investigating for a given error), or the whole DLQ. Idempotency makes bulk replay safe.
• Idempotency is the precondition. Without the dedup + version guard from sections 3–4, replay is dangerous. With them, replay is routine.

Common interview probes on DLQ and replay.

• "What goes in the DLQ record?" — raw payload, headers, error, attempt count, status.
• "Why backoff and jitter?" — avoid hammering a struggling downstream and synchronized retry storms.
• "How is replay safe?" — because processing is idempotent (dedup + version guard).
• "What about a poison message in an ordered stream?" — dead-letter after max attempts to unblock; decide the ordering trade-off deliberately.

Worked example — retry with exponential backoff and jitter

Detailed explanation. The retry layer wraps the handler: on a retryable exception it schedules another attempt after an exponentially growing, jittered delay; on a fatal exception or after the attempt budget, it dead-letters. Build the retry decision and delay calculation.

• Delay. min(cap, base * 2^attempt) plus random jitter.
• Retryable classification. Timeouts, 5xx, lock/rate-limit errors → retry. Validation/business errors → fatal.
• Budget. Max 5 attempts, then DLQ.

Question. Implement the backoff delay and the retry-or-dead-letter decision.

Input.

Parameter	Value
Base	1 s
Cap	300 s
Max attempts	5
Jitter	full jitter (0..computed)

Code.

import random

BASE, CAP, MAX_ATTEMPTS = 1.0, 300.0, 5

class Retryable(Exception): pass          # transient — retry
class Fatal(Exception): pass              # permanent — dead-letter now

def backoff_delay(attempt: int) -> float:
    """Exponential backoff with full jitter."""
    exp = min(CAP, BASE * (2 ** attempt))
    return random.uniform(0, exp)         # full jitter avoids thundering herd

def consume(job: dict) -> None:
    try:
        process(job)                      # dedup + version-guarded side effect
    except Fatal as exc:
        move_to_dlq(job, exc, reason="fatal")           # no retries wasted
    except Retryable as exc:
        if job["attempts"] + 1 >= MAX_ATTEMPTS:
            move_to_dlq(job, exc, reason="max-attempts")
        else:
            job["attempts"] += 1
            requeue_after(job, backoff_delay(job["attempts"]))

Step-by-step explanation.

1. backoff_delay computes base * 2^attempt, capped, then multiplies by full jitter (random.uniform(0, exp)). The exponential growth spaces out retries against a struggling downstream; the jitter desynchronizes a whole worker fleet so they don't all retry at the same instant (the thundering-herd problem).
2. The handler classifies exceptions. Fatal (schema violation, business rejection) means retrying is pointless — it dead-letters immediately with reason="fatal", saving the retry budget for failures that might actually recover.
3. Retryable (timeout, 5xx, lock contention) means try again — unless the attempt budget is exhausted. On the last allowed attempt it dead-letters with reason="max-attempts".
4. Otherwise it increments the attempt count and requeues the job with the computed backoff delay. The attempt count travels with the job (in the message or DLQ row), so the budget is enforced across worker restarts.
5. Because process is idempotent, a retry that runs after a partial success is safe: the dedup and version guard make the repeated side effect a no-op. Retrying can never double-apply.

Output.

Attempt	Backoff range	Outcome if still failing
1	0–2 s	retry
2	0–4 s	retry
3	0–8 s	retry
4	0–16 s	retry
5	—	move to DLQ (max-attempts)

Rule of thumb. Retry only retryable errors, with exponential backoff plus full jitter and a hard attempt cap; dead-letter fatal errors immediately and retryable errors after the cap. Carry the attempt count with the job so the budget survives restarts.

Worked example — the dead-letter queue schema and move-to-DLQ

Detailed explanation. The DLQ is a durable table (or a dedicated queue) that captures everything needed to debug and replay a failed event. Design the schema and the move-to-DLQ operation so no context is lost. Build it as a Postgres table with a status workflow.

• Schema. id, event_id, event_type, raw payload, headers, error, attempts, status, timestamps.
• Move. Insert the failed job with status='new' and the failure context.
• Workflow. new → investigating → replayed/discarded.

Question. Write the DLQ schema and the move-to-DLQ function.

Input.

Column	Purpose
event_id	dedup key for replay
payload	exact raw bytes for verbatim replay
error	failure diagnosis
attempts	how many tries before giving up
status	workflow state

Code.

CREATE TABLE dead_letter_queue (
    id           BIGSERIAL   PRIMARY KEY,
    event_id     TEXT        NOT NULL,
    event_type   TEXT        NOT NULL,
    payload      TEXT        NOT NULL,              -- exact raw bytes
    headers      JSONB,                             -- signature, timestamp, etc.
    error        TEXT        NOT NULL,
    attempts     INT         NOT NULL,
    status       TEXT        NOT NULL DEFAULT 'new', -- new|investigating|replayed|discarded
    first_failed_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    last_failed_at  TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);

CREATE INDEX idx_dlq_status ON dead_letter_queue (status);
CREATE INDEX idx_dlq_event  ON dead_letter_queue (event_id);

import json

def move_to_dlq(conn, job: dict, exc: Exception, reason: str) -> None:
    with conn:
        with conn.cursor() as cur:
            cur.execute("""
                INSERT INTO dead_letter_queue(
                    event_id, event_type, payload, headers, error, attempts, status)
                VALUES (%s, %s, %s, %s, %s, %s, 'new')
            """, (
                job["event_id"],
                job["event_type"],
                job["payload"],                      # verbatim for replay
                json.dumps(job.get("headers", {})),
                f"{reason}: {type(exc).__name__}: {exc}",
                job["attempts"],
            ))
    log.warning("dead-lettered %s (%s)", job["event_id"], reason)

Step-by-step explanation.

1. The DLQ schema stores the exact raw payload so replay is byte-for-byte identical to the original delivery — including anything needed to re-verify the signature. Storing a parsed/re-serialized version would risk changing bytes.
2. headers JSONB preserves the signature, timestamp, and delivery metadata, so a replay can go through the same verification and so an engineer has full context to diagnose the failure.
3. error, attempts, and the two timestamps capture why and how hard the system tried before giving up — the difference between "failed once fatally" and "retried five times over two minutes."
4. status turns the DLQ into a workflow: new events await triage, investigating marks ones an engineer is working, replayed/discarded are terminal. The idx_dlq_status index makes "give me all new failures" fast.
5. move_to_dlq runs in its own transaction so recording the failure is durable even if everything else about the job is broken. The DLQ insert must not itself be conditional on the failing work.

Output.

Field	Example
event_id	evt_9f3a
event_type	payment.succeeded
error	max-attempts: TimeoutError: entitlements down
attempts	5
status	new

Rule of thumb. Capture the raw payload, headers, error, and attempt count in the DLQ, index it by status and event_id, and treat it as a workflow with explicit states — a DLQ nobody can query or replay is just a slower way to lose events.

Worked example — the replay job

Detailed explanation. Replay reads events from the DLQ and re-injects them through the normal handler. Because the handler is idempotent (dedup + version guard), replaying an event that partially succeeded is safe. Build a replay job that can target one event, a filtered set, or all new failures, and that updates status as it goes.

• Select. Pull DLQ rows by status/error/event_type.
• Re-inject. Run the same process used in production.
• Update. On success, mark replayed; on repeat failure, keep in DLQ (bump attempts / leave for investigation).

Question. Write a replay job that reprocesses DLQ events safely and idempotently.

Input.

Element	Value
Source	dead_letter_queue WHERE status='new'
Handler	production process (idempotent)
Success	status='replayed'
Repeat failure	leave in DLQ, log

Code.

def replay_dlq(conn, where_sql: str = "status = 'new'", limit: int = 500) -> dict:
    stats = {"replayed": 0, "failed": 0}
    with conn.cursor() as cur:
        cur.execute(f"""
            SELECT id, event_id, event_type, payload, headers
            FROM   dead_letter_queue
            WHERE  {where_sql}
            ORDER  BY id
            LIMIT  %s
        """, (limit,))
        rows = cur.fetchall()

    for dlq_id, event_id, event_type, payload, headers in rows:
        job = {"event_id": event_id, "event_type": event_type,
               "payload": payload, "headers": headers, "attempts": 0}
        try:
            process(conn, job)                       # SAME idempotent handler
            with conn, conn.cursor() as c:
                c.execute("""
                    UPDATE dead_letter_queue
                    SET status='replayed', last_failed_at=clock_timestamp()
                    WHERE id=%s
                """, (dlq_id,))
            stats["replayed"] += 1
        except Exception as exc:
            with conn, conn.cursor() as c:
                c.execute("""
                    UPDATE dead_letter_queue
                    SET error=%s, last_failed_at=clock_timestamp()
                    WHERE id=%s
                """, (f"replay-failed: {exc}", dlq_id))
            stats["failed"] += 1
    return stats

Step-by-step explanation.

1. The job selects DLQ rows by a filter — status='new' for a routine sweep, or a narrower predicate (a specific event_type, a specific error string) to replay only the events affected by a just-fixed bug. ORDER BY id LIMIT keeps batches bounded.
2. Each row is reconstructed into the same job shape the live worker uses, from the stored raw payload and headers, then fed to the identical production process function. Replay is not a special code path — it is the normal path, which is what keeps behavior consistent.
3. Idempotency is what makes this safe: if the event had partially applied before it originally failed, the dedup table now records it (or the version guard rejects the stale re-apply), so replay is a correct no-op rather than a double-apply. This is the payoff of sections 3–4.
4. On success, the DLQ row is marked replayed (terminal). On repeat failure, the row stays in the DLQ with an updated error — signalling the root cause isn't actually fixed, so replaying again would just fail again.
5. The returned stats (replayed, failed) give the operator a clear picture: replay after a fix should show all replayed; a batch of failed means the fix was incomplete and needs another look before re-running.

Output.

DLQ event	Replay result	New status
evt_A (downstream now up)	success	replayed
evt_B (duplicate, already applied)	no-op via dedup	replayed
evt_C (fix incomplete)	fails again	new (error updated)

Rule of thumb. Replay through the exact production handler, filter to just the events a fix addresses, and rely on idempotency to make re-injection safe. Mark successes replayed; leave repeat failures in the DLQ as the signal that the root cause isn't fixed.

Senior interview question on dead-letter queues and replay

A senior interviewer might ask: "A downstream service your webhook worker calls goes down for 30 minutes, so thousands of events fail processing. Walk me through the retry strategy, when and how events land in a dead-letter queue, how you keep one poison message from blocking the stream, and exactly how you replay the backlog once the downstream recovers — without double-applying any side effect."

Solution Using bounded jittered retries + a durable DLQ + idempotent replay

import random, json

BASE, CAP, MAX_ATTEMPTS = 1.0, 300.0, 5

class Retryable(Exception): pass
class Fatal(Exception): pass

def backoff(attempt: int) -> float:
    return random.uniform(0, min(CAP, BASE * (2 ** attempt)))   # full jitter

def consume(conn, job: dict) -> None:
    try:
        process(conn, job)                          # idempotent: dedup + version guard
    except Fatal as exc:
        move_to_dlq(conn, job, exc, "fatal")
    except Retryable as exc:
        if job["attempts"] + 1 >= MAX_ATTEMPTS:
            move_to_dlq(conn, job, exc, "max-attempts")   # poison isolation
        else:
            job["attempts"] += 1
            requeue_after(job, backoff(job["attempts"]))

def replay_dlq(conn, where_sql="status='new'", limit=500) -> dict:
    stats = {"replayed": 0, "failed": 0}
    with conn.cursor() as cur:
        cur.execute(f"SELECT id, event_id, event_type, payload, headers "
                    f"FROM dead_letter_queue WHERE {where_sql} ORDER BY id LIMIT %s",
                    (limit,))
        rows = cur.fetchall()
    for dlq_id, eid, etype, payload, headers in rows:
        job = {"event_id": eid, "event_type": etype,
               "payload": payload, "headers": headers, "attempts": 0}
        try:
            process(conn, job)                      # SAME handler → idempotent
            _set_status(conn, dlq_id, "replayed")
            stats["replayed"] += 1
        except Exception as exc:
            _bump_error(conn, dlq_id, f"replay-failed: {exc}")
            stats["failed"] += 1
    return stats

CREATE TABLE dead_letter_queue (
    id           BIGSERIAL   PRIMARY KEY,
    event_id     TEXT        NOT NULL,
    event_type   TEXT        NOT NULL,
    payload      TEXT        NOT NULL,
    headers      JSONB,
    error        TEXT        NOT NULL,
    attempts     INT         NOT NULL,
    status       TEXT        NOT NULL DEFAULT 'new',
    first_failed_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
    last_failed_at  TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX idx_dlq_status ON dead_letter_queue (status);

Step-by-step trace.

Phase	Behaviour during the 30-min outage	Behaviour after recovery
Retry	each event retries 5× with jittered backoff	n/a
Backoff	1→2→4→8→16 s (jittered) spreads load	downstream not hammered
Dead-letter	after 5 attempts → DLQ status='new'	thousands buffered, not lost
Poison isolation	failing events leave the stream	live traffic keeps flowing
Replay	—	replay_dlq("status='new'") re-injects
Idempotency	partial successes recorded	replay = no-op or newer-only apply

During the outage, each event exhausts its 5 jittered retries and lands in the DLQ with full context; the live stream keeps moving because poison events are isolated rather than blocking. When the downstream recovers, replay_dlq re-injects the buffered events through the same idempotent handler: any event that had partially applied is a no-op via the dedup table, and stale ones are rejected by the version guard, so nothing is double-applied. Events that still fail stay in the DLQ, flagging an incomplete fix.

Output:

Metric	Value
Events lost during outage	0 (all dead-lettered)
Live stream blocked	no (poison isolation)
Downstream hammered on retry	no (backoff + jitter)
Double-applied on replay	0 (idempotent handler)
Recovery action	one replay_dlq sweep

Why this works — concept by concept:

• Bounded jittered retries — exponential backoff spaces retries against a struggling downstream; full jitter desynchronizes the worker fleet; the attempt cap prevents an infinite loop on a permanently-bad event.
• Dead-letter with full context — capturing the raw payload, headers, error, and attempts makes every failed event diagnosable and replayable verbatim, so the DLQ is a recovery tool, not a graveyard.
• Poison isolation — moving a max-attempts event to the DLQ unblocks the stream, so one bad event can't stall thousands of good ones behind it.
• Idempotent replay — because the handler dedups by event id and guards by version, re-injecting a DLQ event that partially applied is a no-op or a correct newer-only apply; replay is routine, not risky.
• Cost — a bounded number of retries per failing event and one DLQ row per permanent failure; replay is one pass through the same O(1)-per-event handler. The eliminated cost is the lost events, the manual data-repair after an outage, and the blocked stream. Retries + DLQ + idempotent replay turn a 30-minute outage into a one-command recovery.

Streaming
Topic — streaming
Streaming dead-letter-queue and replay problems

Practice →

Design
Topic — design
Design problems on retry and dead-letter systems

Practice →

---

Cheat sheet — webhook ingestion recipes

• The five-station pipeline. Every webhook endpoint is: (1) verify signature + timestamp, (2) durably record the raw event and ACK 200 fast, (3) deduplicate by event id, (4) order state by version per entity, (5) retry then dead-letter on failure. Draw the five stations first; the implementation follows from them. Never design "parse and update."
• Signature-verify + timestamp-guard template. Compute HMAC-SHA256(secret, "{ts}." + raw_body) over the raw bytes (before JSON parse), compare with a constant-time function (hmac.compare_digest), and reject requests whose signed timestamp is outside a ±5-minute window. Support multiple candidate signatures so secret rotation is zero-downtime. TLS authenticates the channel; the HMAC authenticates the sender; the timestamp authenticates freshness.
• Fast-ACK endpoint shape. On the request path: verify → durably enqueue (raw_events table with ON CONFLICT DO NOTHING, or SQS/Kafka) → return 200 in single-digit ms. The durable write must commit before the ACK, so a crash after the ACK never loses an event. No business logic on the request path — slow processing causes the retries it then has to dedup.
• Dedup table DDL + claim. processed_events(event_id TEXT PRIMARY KEY, processed_at). Claim with INSERT ... ON CONFLICT (event_id) DO NOTHING; if rowcount == 0, it's a duplicate — stop. Put the claim and the side effect in one transaction so there is no "marked done but not done" window. Never SELECT-then-INSERT (it races). Add a Redis SET NX EX front line at high volume; roll back the Redis claim if processing fails.
• Version-guarded upsert. INSERT ... ON CONFLICT (id) DO UPDATE SET ... WHERE stored.version < EXCLUDED.version. Every event runs the same statement; the guard drops stale, reordered, and duplicate deliveries and keeps state monotonic. Prefer a provider version/sequence over a timestamp (skew, ties). Partition the buffer by entity id for per-entity in-order processing; the guard remains the durable correctness backstop.
• Versioned soft-delete. Never physically DELETE where late events are possible — model delete as status='deleted', version=N so the version guard protects it and a late lower-version update can't resurrect the entity. Purge soft-deleted rows out of band after the provider's retry window closes.
• DLQ schema + move. dead_letter_queue(id, event_id, event_type, payload, headers JSONB, error, attempts, status, first_failed_at, last_failed_at). Capture the raw payload verbatim (byte-for-byte replay), the headers (re-verify + debug), the error and attempt count, and a status workflow (new→investigating→replayed/discarded). Index by status and event_id.
• Retry policy. Retry only retryable errors (timeouts, 5xx, lock/rate-limit) with exponential backoff min(cap, base·2^attempt) plus full jitter, capped at ~5 attempts; dead-letter fatal errors (schema/business violations) immediately. Carry the attempt count with the job so the budget survives restarts.
• Replay query. SELECT ... FROM dead_letter_queue WHERE status='new' ORDER BY id LIMIT N, re-inject through the same production handler, mark success replayed, leave repeat failures in the DLQ. Safe only because processing is idempotent (dedup + version guard). Fix the root cause first; replaying into the same failure just re-dead-letters.
• Pattern decision matrix. Authenticity → HMAC over raw body + timestamp window (or mTLS/IP allowlist if unsigned). Idempotency → UNIQUE event id claimed in the side-effect transaction. Ordering → version guard per entity, partition by entity id. Failure → bounded jittered retries → DLQ → idempotent replay. Every axis has one canonical mechanism; name all four in the interview.
• Retention. Keep dedup rows and DLQ rows longer than the provider's maximum retry horizon plus a margin (e.g. Stripe ~3 days → keep ~7). Partition-drop or TTL old rows so neither table grows without bound. Keep the timestamp seen-set TTL slightly larger than the tolerance window.
• What to store from the provider. The event_id (dedup key), the event_type (routing), the raw bytes (verify + replay), the version/sequence (ordering), and the signature/timestamp headers (re-verify). Trust none of the payload's own "verified"/"authenticated" fields — trust only the HMAC.

Frequently asked questions

What is webhook ingestion in one sentence?

Webhook ingestion is the practice of receiving event notifications that an external system POSTs to your HTTP endpoint and processing them correctly under the real-world guarantees those deliveries carry — namely at-least-once delivery (so the same event can arrive many times), unordered delivery (so events can reorder), an untrusted channel (so payloads must be authenticated), and fallible processing (so failures must be retried and, if needed, dead-lettered). A correct receiver therefore verifies a signature, deduplicates by the provider's event id, orders state changes per entity by a version, and routes unprocessable events to a dead-letter queue it can replay — none of which the naive "parse the JSON and update a row" handler does. Every senior data- and platform-engineering interview probes webhook ingestion because it is the load-bearing integration pattern between SaaS providers, event buses, and your own systems.

Why do webhooks need idempotency?

Because webhook delivery is at-least-once, duplicates are guaranteed, not rare. A provider that doesn't receive a timely 2xx retries the same event; a network timeout can drop your ACK after you already did the work, so the provider retries something you completed; and dashboards let operators manually redeliver events for debugging. If your handler isn't idempotent, every one of those duplicates re-applies the side effect — a double charge, a double-granted entitlement, a double-counted metric. Idempotency is achieved by recording the provider's globally-unique event id in a durable store with a UNIQUE constraint and claiming that id in the same transaction as the side effect: INSERT ... ON CONFLICT (event_id) DO NOTHING, and if the insert affected zero rows, the event was already processed and you stop. This turns "processed N times" into "the same effect as processed once," which is the entire point.

How do I verify a webhook signature?

Compute an HMAC over the raw request body (the exact bytes, captured before any JSON parsing) using the shared secret the provider gave you, and compare it to the signature header in constant time. Concretely for a Stripe-style scheme: the provider sends X-Signature: t=<unix>,v1=<hex>, you reconstruct the signed payload as "{t}.{raw_body}", compute HMAC-SHA256(secret, signed_payload), and check it against each v1 candidate with hmac.compare_digest (constant-time avoids a timing side channel). Two more rules matter: reject any request whose signed timestamp t is outside a tolerance window (Stripe uses 5 minutes) to defeat replay attacks, and support multiple v1 candidates so secret rotation is zero-downtime. Do not verify over parsed-then-re-serialized JSON — re-serialization changes whitespace and key order and every signature will fail. And remember TLS only authenticates the channel; the HMAC is what authenticates the sender.

How do webhooks handle ordering?

They don't, by default — deliveries arrive in no guaranteed order because providers fan out across workers and regions and because retries reshuffle older events behind newer ones. You restore correctness with a version guard per entity: each event carries a monotonic version or sequence number for its entity, and your state write applies the change only if the incoming version exceeds the stored version (UPDATE ... WHERE version < :incoming, or the equivalent WHERE on an upsert's DO UPDATE). A reordered older event then updates zero rows instead of clobbering newer state, and the entity converges to the highest-version event regardless of arrival order. Prefer a provider-assigned version or sequence over a wall-clock timestamp (timestamps skew across senders and tie). If you use a partitioned buffer like Kafka, key messages by the entity id so one consumer handles one entity's events in order — but keep the version guard, because retries and reprocessing still need a durable correctness backstop.

What is a dead-letter queue for webhooks?

A dead-letter queue (DLQ) is a durable holding area for webhook events that failed processing after a bounded number of retries. Instead of losing a failed event (the "log and return 500" anti-pattern) or retrying it forever (which blocks the stream behind a poison message), you retry a capped number of times with exponential backoff and jitter, and on exhaustion move the event to the DLQ with everything needed to debug and reprocess it: the raw payload (for byte-for-byte replay), the headers (to re-verify), the error message, the attempt count, and a status you can drive through a triage workflow. The DLQ does two jobs: poison isolation — one permanently-failing event leaves the live stream so it can't block thousands of good events behind it — and recoverability — once the root cause is fixed, you replay the DLQ back through the normal handler. A webhook pipeline without a DLQ silently drops every event that fails processing.

How do I replay failed webhooks safely?

Replay by re-injecting the dead-letter-queue events through the exact same production handler, after you have fixed the root cause of their failure. The safety comes entirely from the pipeline already being idempotent: because the handler claims each event id in a UNIQUE dedup table and guards state writes by version, replaying an event that had partially applied before it failed is a no-op (the dedup catches it) or a correct newer-only apply (the version guard handles it) — it can never double-apply a side effect. Practically, select DLQ rows by a filter narrow enough to match just the events the fix addresses (a specific event_type or error), reprocess them in bounded batches, mark successes replayed, and leave any repeat failures in the DLQ as a signal that the fix is incomplete. This is why idempotency and ordering (sections 3 and 4) are prerequisites: without them, replay is dangerous; with them, replaying a backlog after an outage is a single routine command.

Practice on PipeCode

• Drill the streaming practice library → for the ingestion, ordering, partitioning, and dead-letter-queue problems senior interviewers love.
• Rehearse on the event-processing practice library → for deduplication, sequencing, and out-of-order event problems.
• Sharpen the idempotency SQL on the SQL practice library → for ON CONFLICT upserts, UNIQUE-constraint dedup, and version-guarded writes.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis webhook design against real graded inputs.

Lock in webhook ingestion muscle memory

Docs explain the endpoints. PipeCode drills explain the decision — when a duplicate double-charges, when a reordered event clobbers newer state, when a missing signature check turns your webhook into an open write API, when a poison message blocks the stream. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs senior data engineers actually face.

Practice streaming problems →
Practice event-processing problems →