A 200 OK Won’t Save Your Webhooks: Test Duplicates, Disorder, and Retries

Your webhook receiver returns 200 OK. The provider accepts the acknowledgment, but duplicates or previously scheduled retries can still arrive. Then the worker crashes before updating the order.

Your receiver acknowledged delivery before the worker completed the business operation.

Webhook testing needs to cover both boundaries: what the sender believes happened, and what your application durably accomplished. API virtualization lets QA and SDET teams control delivery schedules and receiver failures, while developers test the processing path against those schedules.

Store before acknowledging

A typical receiver verifies an event, stores it durably, acknowledges delivery, and processes it asynchronously:

Provider → Verify request → Durable inbox/queue → HTTP acknowledgment
                                  └───────────→ Worker → Business state

Store the event durably before acknowledging it, using the guarantees your architecture provides. Returning success before storing the event creates a loss window. Holding the HTTP connection until all downstream work finishes can create timeout and retry pressure.

There are two useful virtualization placements. To test a sender, replace its destination with a controllable endpoint that returns errors, delays, or success. To test a receiver, use a controlled event driver to deliver requests to the real receiver, optionally through a local tunnel. Inspecting captured requests confirms what crossed the HTTP boundary; it does not confirm worker completion.

Provider behavior is specific. Stripe, for example, documents duplicate events, non-guaranteed event ordering, signature verification, and retries. Use the provider's current documentation rather than assuming all webhook products have identical guarantees. Stripe webhook documentation.

Define delivery invariants

Choose an invariant such as “one payment event produces one fulfillment record.” Then deliberately deliver events in ways that challenge it.

Delivery schedule	Assertion on the real receiver
Same event ID twice, sequentially	One business effect; Receiver acknowledges duplicate according to policy
Same event ID twice, concurrently	One business effect under the race
Later state before earlier state	No invalid regression of business state
Worker fails after acknowledgment	Event remains recoverable and eventually processes or reaches a visible failure state
Invalid signature	Receiver rejects request before enqueueing
Valid signature but expired timestamp	Receiver rejects request under the provider-specific replay policy

The concurrent duplicate case matters. A “check whether processed, then insert” implementation can pass sequential tests and still race. Coordinate deduplication and business effects with durable uniqueness and transactions where appropriate, or another mechanism that gives equivalent guarantees. When an external side effect cannot share the transaction, use a stable idempotency key and explicit recovery logic.

Do not deduplicate solely by payload equality. Two legitimate events can have identical business fields. Conversely, providers may produce distinct event IDs for related changes to the same resource. Decide whether your domain also needs operation-level deduplication.

Assert the durable result and outbound effects, not only the acknowledgment code. An HTTP success can conceal a dropped message; an HTTP failure can follow a completed operation and cause a retry.

Drive duplicates and disorder

The following Python driver posts synthetic events to your real test receiver. It has no provider signature and is suitable for a receiver's isolated test mode. Keep signature verification enabled in separate tests that use the provider's signing algorithm or official testing tools.

import json
import os
import urllib.error
import urllib.request

target = os.environ["WEBHOOK_TEST_URL"]

def deliver(event):
    request = urllib.request.Request(
        target,
        data=json.dumps(event).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=5) as response:
            return response.status
    except urllib.error.HTTPError as error:
        return error.code

paid = {
    "id": "evt_test_paid_17",
    "type": "order.paid",
    "data": {"orderId": "order-17", "version": 2},
}
created = {
    "id": "evt_test_created_17",
    "type": "order.created",
    "data": {"orderId": "order-17", "version": 1},
}

# Explicit disorder and duplicate delivery; no automatic retries.
for event in (paid, created, paid):
    print(event["id"], deliver(event))

These event names and fields are illustrative, not Stripe payloads. Adapt them to your integration. After the driver finishes, poll the application's observable business state with a bounded deadline and assert the expected final result. Do not replace that assertion with an arbitrary sleep.

For a signed request, compute the signature from the exact transmitted bytes using the provider's prescribed format. Changing whitespace or reserializing JSON can invalidate a signature. Test raw-body handling, wrong secret, tampering, and replay expiration separately.

To test your sender, configure a Beeceptor destination rule with a chosen HTTP status or response delay, send an event, and inspect the recorded attempts. Change the rule to success and verify recovery. A capture endpoint does not manufacture duplicates or reorder events unless you explicitly build that behavior into the test driver. For setup, see Beeceptor request inspection.

Verify bounded recovery

Test more than “eventually succeeds.” Record attempt timestamps, event IDs, acknowledgment codes, queue state, worker failures, and final business effects.

Define a maximum retry budget and a terminal handling path. A poison event should not loop forever or block unrelated events. Verify the dead-letter or failure workflow, operator visibility, and safe replay using the same logical identity.

For local testing, a stable tunnel can keep the receiver URL unchanged while the service restarts. That improves connectivity, but availability and durability are separate: a reachable tunnel cannot preserve an event your application discarded. Beeceptor local tunneling.

Solutions architects should review where durability begins and which effects share a transaction. SDETs should own the delivery matrix and its assertions. Developers should make recovery inspectable enough that a failed test explains where the event stopped.

Test duplicate delivery, reversed events, and a worker crash after acknowledgment. Verify the durable outcome and the recovery path in each case. A resilient webhook integration turns repeated delivery into one intended business outcome and makes every recovery visible.