DEV Community: Andrew Lencmanis

Your Webhook Returned 200 OK. Did the Event Actually Get Processed?

Andrew Lencmanis — Wed, 15 Jul 2026 19:48:19 +0000

A webhook provider sends an event. Your endpoint returns 200 OK. The provider stops retrying.

Everything worked, right?

Not necessarily.

A successful HTTP response proves that one network hop completed. It does not prove that the event was stored durably, processed exactly once, routed correctly, or delivered to every downstream system.

This is a focused follow-up to Building a Replay-Safe Webhook Event Gateway. That article covered the wider gateway architecture. Here, we will zoom in on the most important reliability boundary: what your service is actually promising when it returns 200.

200 OK acknowledges a boundary

Imagine this delivery path:

Provider -> Webhook endpoint -> Durable storage -> Processor -> Destination

A response from the webhook endpoint can only acknowledge work completed before that response was sent.

If your handler returns 200 immediately after parsing JSON, it acknowledges parsing.

If it returns 200 after writing to memory, it acknowledges a volatile write.

If it returns 200 after committing the request to durable storage, it acknowledges durable ingestion.

Those are very different guarantees.

The provider does not know what happens behind your endpoint. Once it receives a success response, it will normally consider the delivery complete. If your process crashes one millisecond later and the payload was never stored, the event may be gone permanently.

A useful rule is:

Do not acknowledge an event until you can recover it without the provider sending it again.

The reliable ingestion pattern

A production webhook handler should keep the synchronous path short, but not empty.

A practical ingestion flow is:

Read the raw body and required headers.
Verify the provider signature when one is available.
Apply basic size and method limits.
Persist the original request and its metadata durably.
Commit the write.
Return a success response.
Process and deliver the event asynchronously.

The critical ordering is persist first, acknowledge second.

Processing the full business workflow before returning 200 is usually a poor tradeoff. A slow database, third-party API, or internal service can push the response past the provider's timeout. The provider then retries, even though your first attempt may still be running.

Durable ingestion lets you respond quickly while preserving the ability to finish later.

Queueing is not the same as durability

"Put it on a queue" is good advice, but the exact guarantee matters.

Ask these questions about your queue or storage layer:

Is the message durable before the publish call returns?
Can an acknowledged message disappear during a process or region failure?
What happens when a consumer crashes after receiving the message?
Is delivery at least once, at most once, or effectively once?
How long can an unprocessed event be retained?
Can operators inspect and replay it?

A local in-memory queue is useful for smoothing work inside one process, but it should not define the acknowledgment boundary for business-critical events.

Idempotency belongs at two levels

Reliable webhook systems assume duplicates will happen.

A provider may retry because it did not receive your response. A network proxy may time out after your server committed the request. An operator may replay an old event. A failed destination may need another delivery attempt.

There are two related, but different, idempotency problems.

1. Ingestion idempotency

This prevents the same provider event from creating multiple logical requests.

Use a stable provider event ID when available, such as a Stripe event ID or GitHub delivery ID. Scope it by provider, account, or source so unrelated integrations cannot collide.

A typical uniqueness key looks like:

(source_id, provider_event_id)

Store the original request even when you mark a later copy as a duplicate. Duplicate attempts are valuable operational evidence.

2. Delivery idempotency

One accepted request may intentionally produce several outbound events. Each destination delivery needs its own stable identity.

A useful key is:

(request_id, connection_id, destination_id)

This lets you retry one failed destination without recreating every successful delivery.

Idempotency does not mean "never execute twice." In distributed systems, that promise is often unrealistic. It means repeated attempts converge on the same intended result.

Retry the right object

There is an important difference between retrying a request and retrying a delivery event.

Request replay runs the original request through routing again. Use it when filters, transformations, or connections changed and you intentionally want fresh routing decisions.

Event retry repeats one specific destination delivery. Use it when routing was correct but the destination timed out or returned an error.

Treating both operations as one generic retry button makes incident recovery unpredictable.

Backoff, jitter, and terminal failure

A retry loop should not hammer an unhealthy destination.

A common schedule uses exponential backoff with jitter:

delay = min(max_delay, base_delay * 2^attempt) + random_jitter

Jitter prevents thousands of events from retrying at the same instant after an outage.

Retries also need a terminal state. After the attempt limit is reached, the event should become visibly failed rather than silently disappearing. Operators need to see:

the last destination status and response;
the number of attempts;
the next retry time;
the request and event IDs;
whether a manual replay is safe.

A dead-letter queue without an inspection and replay workflow is only a quieter failure mode.

Preserve the evidence

Debugging is much easier when the platform keeps both the inbound request and each outbound attempt.

For the request, preserve:

method and path;
headers;
raw or losslessly stored body;
receive time;
signature verification result;
source and provider identifiers;
duplicate detection result.

For each delivery event, preserve:

connection and destination;
transformed payload;
attempt number;
start and finish timestamps;
destination status, headers, and response body;
retry reason and scheduling data.

Redact secrets before showing them in a UI or sharing a trace, but do not throw away the information required to understand what happened.

Measure the whole lifecycle

A single "webhook received" counter is not enough.

Useful metrics include:

accepted and rejected requests;
duplicate requests;
ingestion latency;
queued event age;
deliveries by status;
attempt count distribution;
destination latency;
retry success rate;
terminal failures;
replay volume.

The most useful operational question is not "Did the endpoint return 200?" It is:

Can I trace this provider event from ingestion through every routing and delivery decision?

If the answer is yes, incident response becomes evidence-based instead of guesswork.

Test the failure paths

Happy-path tests prove very little about webhook reliability.

Test at least these cases:

The process crashes after receiving the body but before committing storage.
The storage write succeeds but the HTTP response is lost.
The same provider event arrives twice.
A consumer crashes after reading an event.
The destination returns 429 with a retry hint.
The destination times out after performing its side effect.
A transformation fails.
An operator replays an event from last week.
Hundreds of failed events are retried after an outage.

For quick payload inspection, you can create a temporary endpoint with the FastHook Request Bin. It provides a disposable webhook URL and shows the incoming method, headers, and body. That is useful for verifying what the provider actually sends before connecting it to a production workflow.

A practical acknowledgment contract

A clear contract for a managed webhook gateway can be:

A successful response means the original request has been accepted and durably recorded. Downstream processing may continue asynchronously and remains observable and replayable.

That statement is narrow enough to be honest and strong enough to be useful.

It does not promise that every destination has already succeeded. It promises that the system now owns the event and can recover it.

That is the real meaning a reliable webhook endpoint should attach to 200 OK.

FastHook is being built around this model: durable webhook ingestion, explicit requests and events, routing through connections, destination delivery tracking, retries, replay, and inspection. You can explore it at fasthook.io.

Originally published on FastHook Engineering.

How to Send Webhooks to Google Sheets with FastHook

Andrew Lencmanis — Wed, 08 Jul 2026 15:07:32 +0000

FastHook supports Google Sheets as a first-class destination type:

GOOGLE_SHEET

Do not treat it as an HTTP destination, and do not add spelling aliases such as
GOOGLE_SHEETS, GOOGLESHEET, or typo-based normalization. The database
constraint, API contract, dashboard UI, and workers should use the exact
GOOGLE_SHEET value.

Backend Contract

Supported destination types are:

HTTP | CLI | MOCK_API | GOOGLE_SHEET

For GOOGLE_SHEET, the destination config is sheet-specific and does not
require config.url or config.path.

Required config:

spreadsheet_id
auth_type
auth

Supported auth types:

GOOGLE_OAUTH_REFRESH_TOKEN
GOOGLE_SERVICE_ACCOUNT

OAuth config:

auth.client_id
auth.client_secret
auth.refresh_token

Service account config:

auth.client_email
auth.private_key

When using a service account, the target spreadsheet must be shared with
auth.client_email.

Sheet Targeting

Use:

sheet_name, default Sheet1
range, default <sheet_name>!A:Z

The dashboard must not ask users to type range. It should derive the value
from sheet_name and save <sheet_name>!A:Z. API callers may omit range;
the backend normalizes it from sheet_name.

Simple Unicode sheet names do not need quotes, so Лист1 should become
Лист1!A:Z. When the sheet name contains spaces or A1 notation punctuation,
quote the sheet name:

'Orders Archive'!A:Z

Row Layout

Google Sheet delivery writes delivered events through the Google Sheets
values.append API.

Row layout rules:

If columns is set, write those dotted payload/metadata paths in that exact order.
If columns is empty and include_metadata is true, write FastHook metadata columns plus a payload snapshot.
If columns is empty and include_metadata is false, auto-flatten the payload and spread payload values across columns.

Supported metadata paths include:

delivered_at
event_id
request_id
connection_id
event_data_id
team_id
source_id
destination_id

Payload paths can be written with or without the payload. prefix:

payload.customer.email
customer.email

Dashboard UI Rules

The dashboard should expose one destination editor shared by the Destinations
page and the Connections/Flow editor.

For Google Sheet destinations:

Show Spreadsheet ID, Sheet name, Payload fields, Value input, Include metadata, and Google auth.
Do not show Range; build it automatically from Sheet name as <sheet_name>!A:Z.
Support both OAuth2 login and Service account.
Hide the insert mode selector. The UI should save insert_data_option as INSERT_ROWS.
Keep Max delivery rate enabled for Google Sheet destinations. Google Sheets API write quota is 60 requests/minute per user/project, so fix the UI at 1/second to avoid burst writes.
Payload fields disables Include metadata, because explicit fields define the row layout.
Include metadata disables Value input, because metadata rows should be written as RAW.
If Include metadata is disabled and Payload fields is empty, payload data is auto-spread across columns.
Keep Max delivery rate as the shared destination rate-limit control: enabled toggle, numeric value, and second | minute | hour.

Operational Notes

Google Sheets API must be enabled in the Google Cloud project used by the OAuth client or service account.
Google Sheets API write quota is 300 requests/minute per project and 60 requests/minute per user/project. Treat one Google Sheet destination as a single user/account quota and default rate limiting to 1/second.
OAuth2 login requires the dashboard OAuth token exchange route and the public Google client id to be configured.
After changing a destination config in production, clear the destination config cache if the send worker reads cached destination records.
A failed Google Sheets delivery should be debugged from the event/attempt status first, then by checking auth, spreadsheet sharing, sheet name, and payload field mapping.

Building a Replay-Safe Webhook Event Gateway

Andrew Lencmanis — Sun, 10 May 2026 19:35:23 +0000

Webhooks look simple at first: receive an HTTP request, forward it somewhere, return 200 OK.

That simplicity disappears the moment the system becomes production-critical. Providers retry. Consumers fail. Payloads arrive twice. Teams need to debug what happened yesterday. A single endpoint becomes dozens of integrations, each with its own routing, retry, transformation, and observability needs.

At that point, webhooks stop being “just HTTP callbacks” and become an event gateway problem.

This is the layer FastHook is built around: receive webhook requests, route them through connections, deliver events to destinations, and make every step inspectable and retryable.

The Core Model
A useful webhook gateway needs clear internal concepts.

In FastHook, a source is where inbound webhooks arrive. It represents the public endpoint that providers send requests to.

A destination is where routed events are delivered. For example, an internal API endpoint, a worker, or another HTTP service.

A connection links one source to one destination. This is where delivery behavior lives: filtering, transformation, retry, delay, and deduplication rules.

A request is the inbound webhook received by FastHook. It contains the original payload and request metadata.

An event is the outbound delivery created from an accepted request and a matching connection. Events can be queued, processed, delivered, failed, or ignored.

That separation matters. If a provider sends one webhook request and it matches several routing paths in the future, you want to inspect the original request separately from each delivery attempt. Even with one source-to-destination connection, keeping request and event separate makes retries, debugging, and metrics much cleaner.

Why Replay Safety Matters
Retries are not optional in webhook systems. They are the normal path.

A destination can return 500. A network request can timeout. A deploy can briefly break an endpoint. If the only recovery path is “ask the provider to resend it,” your system is fragile.

A replay-safe gateway stores enough information to retry later without losing context.

FastHook exposes this at two levels:

curl -X POST \
  "https://api.fasthook.io/v1/requests/req_.../retry" \
  -H "Authorization: Bearer $API_KEY" \
  -H "x-team-id: tm_..."

Request retry replays an inbound request through routing again.

curl -X POST \
  "https://api.fasthook.io/v1/events/evt_.../retry" \
  -H "Authorization: Bearer $API_KEY" \
  -H "x-team-id: tm_..."

Event retry retries a specific delivery event.

Those two operations solve different problems. If routing or connection rules changed, retrying the request can create fresh routed events. If only one destination delivery failed, retrying the event is more precise.

Bulk Retry Is Where Operations Become Real
Manual retry is useful. Bulk retry is where the system becomes operationally useful.

Imagine a destination was down for 30 minutes. You do not want to click retry 800 times. You want to filter failed events by time range, source, status, or search term, then create a controlled bulk operation.

For example:

curl -X POST \
  "https://api.fasthook.io/v1/events/bulk_operations" \
  -H "Authorization: Bearer $API_KEY" \
  -H "x-team-id: tm_..." \
  -H "Content-Type: application/json" \
  -d '{
    "from": "2026-04-17T00:00:00.000Z",
    "to": "2026-04-17T23:59:59.999Z",
    "status": "failed",
    "source_id": "src_...",
    "q": "stripe"
  }'

The important part is that replay is filtered and tracked as an operation, not treated as an invisible background action.

Observability Starts With Count APIs
Before retrying, teams usually ask: how many objects match this filter?

FastHook supports count endpoints for requests and events:

curl \
  "https://api.fasthook.io/v1/events/count?from=2026-04-17T00%3A00%3A00.000Z&to=2026-04-17T23%3A59%3A59.999Z&status=failed&source_id=src_..." \
  -H "Authorization: Bearer $API_KEY" \
  -H "x-team-id: tm_..."

A typical response includes both the exact total and the applied time range:

{
  "total": 1234,
  "count": 1234,
  "total_exact": true,
  "applied_range": {
    "from": "2026-04-17T00:00:00.000Z",
    "to": "2026-04-17T23:59:59.999Z"
  }
}

That sounds small, but it changes the workflow. Operators can estimate blast radius before starting a replay.

Routing Rules Belong On Connections
Webhook providers usually send too much data to too few endpoints. A gateway needs routing rules between ingress and delivery.

FastHook connection rules can cover common operational needs:

filter rules decide whether a request should become an event for a connection.
transform rules modify payloads before delivery.
retry rules control retry behavior.
delay rules postpone delivery.
deduplicate rules help ignore duplicate deliveries inside a configured window.
This keeps source endpoints stable while letting each destination have its own behavior.

Transformations Need Their Own Audit Trail
Transformations are powerful, but they can also hide bugs. If a payload changes before delivery, teams need to know what ran and what came out.

A webhook gateway should treat transformation execution as inspectable data, not just inline code.

FastHook exposes transformation resources and execution history through the API, so teams can list transformations, run tests, and inspect executions tied to delivery behavior.

What To Avoid
The biggest mistake in webhook infrastructure is treating delivery as a single HTTP forward.

That loses the real story.

A production webhook gateway should not only answer “did we receive it?” It should answer:

Was the inbound request accepted or rejected?
Which connection matched it?
Which event was created?
Was the event delivered, failed, ignored, or retried?
What was the destination response?
Can we replay one event safely?
Can we bulk retry a filtered set?
Can we count the impact before doing it?
Once teams have those answers, webhook debugging stops being guesswork.

Closing
Webhooks are often the first integration surface a product exposes, but they are rarely treated as infrastructure until something breaks.

A replay-safe, observable event gateway gives teams a better foundation: stable source URLs, explicit routing, destination delivery tracking, retries, bulk operations, transformations, deduplication, and metrics.

That is the direction we are building with FastHook: not just receiving webhooks, but making webhook delivery understandable, recoverable, and safe to operate.

If you are building webhook infrastructure and want this as a managed product, FastHook is available at https://www.fasthook.io.