Webhook failures are rarely mysterious in production. They are usually incomplete: a timeout hid the response, a retry arrived after a deploy, or an event was accepted but its side effect did not finish. The hard part is reproducing the exact sequence without asking a teammate to click through a staging flow again.
A tiny replay lab gives a developer one durable event, one controlled receiver, and a way to repeat the delivery with different timing. It is a useful piece of automation because it shortens the path from “the webhook failed” to “here is the smallest test that explains why.”
Why a replay lab beats another retry
An automatic retry is good for availability, but it is not a debugging tool. A retry tells you that the system tried again. It does not preserve the original headers, payload, response, or delay between attempts.
The replay lab should answer four questions:
- What event did the sender produce?
- Which delivery attempt was this?
- What did the receiver return?
- What changed between attempts?
That last question is often the important one. A receiver may behave correctly for a fast 200 OK, then fail when the same event arrives while a database lock is held. The first version of a replay harness often look like a pile of shell commands, and that is fine. Keep the model small until the failure becomes visible.
The four pieces of a useful fixture
Store a fixture as a directory rather than a single opaque blob. A simple layout is enough:
fixtures/order-created-42/
event.json
headers.json
expected.json
scenario.json
event.json is the original payload. headers.json contains the event ID, signature metadata, and content type, with secrets removed or replaced by test values. expected.json records the outcome you care about, such as an accepted status and one created resource. scenario.json describes timing, for example a 300 ms delay before the receiver responds.
The event ID is not the same as a request ID. The event ID lets the receiver deduplicate the logical message; the request ID identifies one delivery attempt. Keeping both values in the fixture make concurrency bugs much easier to discuss. A fixture with one clear invariant are easier to review than a large bag of examples.
For email-related workflows, isolate the mailbox identity from the business assertion. A tem email value can be a test fixture, but it should not quietly become proof that a real person owns an account. If the flow needs to inspect a notification, use a controlled test inbox and retain only the message metadata needed for the assertion. Teams that run browser-based email checks may also benefit from triage snapshots for email tests.
A small replay script
The first tool can be a short Python command. It reads the fixture, sends the event, and writes a receipt for every attempt:
import json
import time
import uuid
from pathlib import Path
from urllib.request import Request, urlopen
fixture = Path("fixtures/order-created-42")
event = (fixture / "event.json").read_bytes()
headers = json.loads((fixture / "headers.json").read_text())
scenario = json.loads((fixture / "scenario.json").read_text())
time.sleep(scenario.get("delay_ms", 0) / 1000)
attempt_id = str(uuid.uuid4())
request = Request(
"http://localhost:8080/hooks/orders",
data=event,
headers={**headers, "X-Replay-Attempt": attempt_id},
method="POST",
)
with urlopen(request, timeout=5) as response:
receipt = {"attempt_id": attempt_id, "status": response.status}
Path("receipts").mkdir(exist_ok=True)
(Path("receipts") / f"{attempt_id}.json").write_text(json.dumps(receipt))
This example is intentionally plain. A production version should capture response headers, a bounded response body, elapsed time, and a hash of the request. It should also close resources reliably and avoid putting authorization values in a receipt. The point is the shape: immutable input plus explicit output.
Make failure evidence portable
A replay fixture becomes much more valuable when another developer can run it without reconstructing your local machine. Include the receiver contract, setup command, and a short README beside the JSON files. Record the expected invariant in words: “one event ID creates one invoice, even after three deliveries.”
Do not commit customer payloads merely because they are convenient examples. Replace names, addresses, tokens, and message contents with deterministic test values. A dummy e mail address is still data that can leak into logs or screenshots, so treat it like any other fixture input.
For CI, upload receipts only when a replay fails, and set a retention limit. The receipt should let someone compare attempt number, status, latency, and server correlation ID. It also keep the build log small enough to scan. If a test depends on a clock or an external queue, make that dependency a scenario option so the failure can be repeated locally. When the replay is boring, the next fix become easier to trust.
Questions to settle before CI
Should every webhook be replayable?
No. Start with events that have a clear idempotency rule and a safe synthetic payload. Payment, identity, and deletion events deserve a review before they are exposed to a general replay command.
How many attempts should a fixture run?
Use the fewest attempts that demonstrate the invariant. Three deliveries are usually enough to expose duplicate processing, while a delay scenario can reveal a race without creating a huge test matrix.
Should replay tests call a real queue?
Only when queue behavior is what you are testing. For handler logic, a local receiver and a recorded delivery receipt are faster. Keep one end-to-end test for the queue boundary, then use replay fixtures for the many failure combinations.
A practical checklist
- Store payload, headers, timing, and expected outcome separately.
- Keep event IDs stable and attempt IDs unique.
- Remove secrets and real customer data from fixtures.
- Assert an idempotency invariant, not just a
200response. - Write bounded receipts with latency and correlation IDs.
- Make the same command work locally and in CI.
- Retain failed evidence, then expire old receipts.
The goal is not a new testing platform. It is a small, boring failure lab that preserves enough context to make a flaky webhook repeatable. Once the fixture can explain one failure, you can add scenarios carefully: delayed responses, duplicate deliveries, malformed headers, and receiver restarts. That is when developer tools start feeling like leverage instead of more infrastructure.
Top comments (1)
Nice write-up. My take on your three questions: