Your Webhook Returned 200 OK. Did the Event Actually Get Processed?

A webhook provider sends an event. Your endpoint returns 200 OK. The provider stops retrying.

Everything worked, right?

Not necessarily.

A successful HTTP response proves that one network hop completed. It does not prove that the event was stored durably, processed exactly once, routed correctly, or delivered to every downstream system.

This is a focused follow-up to Building a Replay-Safe Webhook Event Gateway. That article covered the wider gateway architecture. Here, we will zoom in on the most important reliability boundary: what your service is actually promising when it returns 200.

200 OK acknowledges a boundary

Imagine this delivery path:

Provider -> Webhook endpoint -> Durable storage -> Processor -> Destination

A response from the webhook endpoint can only acknowledge work completed before that response was sent.

If your handler returns 200 immediately after parsing JSON, it acknowledges parsing.

If it returns 200 after writing to memory, it acknowledges a volatile write.

If it returns 200 after committing the request to durable storage, it acknowledges durable ingestion.

Those are very different guarantees.

The provider does not know what happens behind your endpoint. Once it receives a success response, it will normally consider the delivery complete. If your process crashes one millisecond later and the payload was never stored, the event may be gone permanently.

A useful rule is:

Do not acknowledge an event until you can recover it without the provider sending it again.

The reliable ingestion pattern

A production webhook handler should keep the synchronous path short, but not empty.

A practical ingestion flow is:

1. Read the raw body and required headers.
2. Verify the provider signature when one is available.
3. Apply basic size and method limits.
4. Persist the original request and its metadata durably.
5. Commit the write.
6. Return a success response.
7. Process and deliver the event asynchronously.

The critical ordering is persist first, acknowledge second.

Processing the full business workflow before returning 200 is usually a poor tradeoff. A slow database, third-party API, or internal service can push the response past the provider's timeout. The provider then retries, even though your first attempt may still be running.

Durable ingestion lets you respond quickly while preserving the ability to finish later.

Queueing is not the same as durability

"Put it on a queue" is good advice, but the exact guarantee matters.

Ask these questions about your queue or storage layer:

• Is the message durable before the publish call returns?
• Can an acknowledged message disappear during a process or region failure?
• What happens when a consumer crashes after receiving the message?
• Is delivery at least once, at most once, or effectively once?
• How long can an unprocessed event be retained?
• Can operators inspect and replay it?

A local in-memory queue is useful for smoothing work inside one process, but it should not define the acknowledgment boundary for business-critical events.

Idempotency belongs at two levels

Reliable webhook systems assume duplicates will happen.

A provider may retry because it did not receive your response. A network proxy may time out after your server committed the request. An operator may replay an old event. A failed destination may need another delivery attempt.

There are two related, but different, idempotency problems.

1. Ingestion idempotency

This prevents the same provider event from creating multiple logical requests.

Use a stable provider event ID when available, such as a Stripe event ID or GitHub delivery ID. Scope it by provider, account, or source so unrelated integrations cannot collide.

A typical uniqueness key looks like:

(source_id, provider_event_id)

Store the original request even when you mark a later copy as a duplicate. Duplicate attempts are valuable operational evidence.

2. Delivery idempotency

One accepted request may intentionally produce several outbound events. Each destination delivery needs its own stable identity.

A useful key is:

(request_id, connection_id, destination_id)

This lets you retry one failed destination without recreating every successful delivery.

Idempotency does not mean "never execute twice." In distributed systems, that promise is often unrealistic. It means repeated attempts converge on the same intended result.

Retry the right object

There is an important difference between retrying a request and retrying a delivery event.

Request replay runs the original request through routing again. Use it when filters, transformations, or connections changed and you intentionally want fresh routing decisions.

Event retry repeats one specific destination delivery. Use it when routing was correct but the destination timed out or returned an error.

Treating both operations as one generic retry button makes incident recovery unpredictable.

Backoff, jitter, and terminal failure

A retry loop should not hammer an unhealthy destination.

A common schedule uses exponential backoff with jitter:

delay = min(max_delay, base_delay * 2^attempt) + random_jitter

Jitter prevents thousands of events from retrying at the same instant after an outage.

Retries also need a terminal state. After the attempt limit is reached, the event should become visibly failed rather than silently disappearing. Operators need to see:

• the last destination status and response;
• the number of attempts;
• the next retry time;
• the request and event IDs;
• whether a manual replay is safe.

A dead-letter queue without an inspection and replay workflow is only a quieter failure mode.

Preserve the evidence

Debugging is much easier when the platform keeps both the inbound request and each outbound attempt.

For the request, preserve:

• method and path;
• headers;
• raw or losslessly stored body;
• receive time;
• signature verification result;
• source and provider identifiers;
• duplicate detection result.

For each delivery event, preserve:

• connection and destination;
• transformed payload;
• attempt number;
• start and finish timestamps;
• destination status, headers, and response body;
• retry reason and scheduling data.

Redact secrets before showing them in a UI or sharing a trace, but do not throw away the information required to understand what happened.

Measure the whole lifecycle

A single "webhook received" counter is not enough.

Useful metrics include:

• accepted and rejected requests;
• duplicate requests;
• ingestion latency;
• queued event age;
• deliveries by status;
• attempt count distribution;
• destination latency;
• retry success rate;
• terminal failures;
• replay volume.

The most useful operational question is not "Did the endpoint return 200?" It is:

Can I trace this provider event from ingestion through every routing and delivery decision?

If the answer is yes, incident response becomes evidence-based instead of guesswork.

Test the failure paths

Happy-path tests prove very little about webhook reliability.

Test at least these cases:

1. The process crashes after receiving the body but before committing storage.
2. The storage write succeeds but the HTTP response is lost.
3. The same provider event arrives twice.
4. A consumer crashes after reading an event.
5. The destination returns 429 with a retry hint.
6. The destination times out after performing its side effect.
7. A transformation fails.
8. An operator replays an event from last week.
9. Hundreds of failed events are retried after an outage.

For quick payload inspection, you can create a temporary endpoint with the FastHook Request Bin. It provides a disposable webhook URL and shows the incoming method, headers, and body. That is useful for verifying what the provider actually sends before connecting it to a production workflow.

A practical acknowledgment contract

A clear contract for a managed webhook gateway can be:

A successful response means the original request has been accepted and durably recorded. Downstream processing may continue asynchronously and remains observable and replayable.

That statement is narrow enough to be honest and strong enough to be useful.

It does not promise that every destination has already succeeded. It promises that the system now owns the event and can recover it.

That is the real meaning a reliable webhook endpoint should attach to 200 OK.

FastHook is being built around this model: durable webhook ingestion, explicit requests and events, routing through connections, destination delivery tracking, retries, replay, and inspection. You can explore it at fasthook.io.

Originally published on FastHook Engineering.