The webhook endpoint returns success, the support ticket appears, and everything looks normal. Then the same ticket is created again. This can happen without a faulty model because webhook delivery and business-event processing have different reliability guarantees.
Consider a hypothetical ecommerce order event that creates a customer-service ticket and sends a confirmation message. If the platform retries that delivery, the handler may repeat both actions. This article follows that failure through event identity, PostgreSQL constraints, durable work and tests.
Why Successful Webhook Delivery Does Not Guarantee Single Processing
Webhook providers generally promise at-least-once delivery, not exactly-once delivery. If your endpoint times out, returns an error, or loses the connection before the provider records your acknowledgment, the provider cannot know whether you processed the event, so it sends it again.
The dangerous case is a handler that does the work before responding. It creates the ticket, then a slow database call delays the response past the timeout. The provider sees a failure, the ticket already exists, and the retry creates another.
I separate two ideas here. Receiving an event means the payload arrived and was stored. Completing the business action means the ticket exists exactly once and the customer was notified once. Acknowledgment should depend on the first, never on the second.
Choosing the Correct Event Identifier for Deduplication
Shopify’s webhook delivery documentation describes X-Shopify-Webhook-Id as the identifier for an individual webhook delivery, which makes it the natural key for rejecting a repeated delivery. Treat it as a delivery key; other platforms name and scope identifiers differently, so verify each provider.
X-Shopify-Event-Id serves a different purpose. Per the same guidance, it correlates deliveries that arise from one merchant action. If an app has several subscriptions for the same change, each delivery can carry its own webhook ID while sharing an event ID.
Neither header is your business operation identity. I derive a separate key, such as ticket:order_id:topic, for the action itself. The delivery ID stops repeated deliveries, while the operation key stops two different events from opening two tickets for one need.
Preventing Duplicate Actions with Persistent Idempotency
Idempotency means repeating an operation leaves the same result as performing it once. In PostgreSQL, I enforce it with a unique constraint on the delivery ID, so the database, not application memory, decides whether an event is new, even across multiple servers.
The handler inserts the event row and returns quickly. A separate worker claims pending rows and performs the ticket work, a pattern I look for when reviewing ecommerce AI chatbot solutions. Storing the event and its job in one transaction keeps both together.
Workers can crash, so claims need recovery. I record a claim time and let another worker reclaim rows stuck in processing beyond a timeout. That recovery guarantees eventual progress, but it also means the same job can run twice.
sql
CREATE TABLE webhook_events (
webhook_id text PRIMARY KEY,
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'pending',
claimed_at timestamptz
);
js
// Illustrative only. HMAC verification omitted. Review and test before use.
await db.query(
`INSERT INTO webhook_events (webhook_id, payload)
VALUES ($1, $2) ON CONFLICT (webhook_id) DO NOTHING`,
[req.get('X-Shopify-Webhook-Id'), req.body]
);
res.sendStatus(200); // new or duplicate: the event is safely stored
sql
-- Atomic claim; SKIP LOCKED keeps concurrent workers off the same row
UPDATE webhook_events SET status = 'processing', claimed_at = now()
WHERE webhook_id = (
SELECT webhook_id FROM webhook_events
WHERE status = 'pending'
OR (status = 'processing' AND claimed_at < now() - interval '5 minutes')
ORDER BY claimed_at NULLS FIRST
LIMIT 1 FOR UPDATE SKIP LOCKED)
RETURNING *;
*The Race Condition in Check-Then-Insert Logic
*
A SELECT followed by an INSERT is unsafe. Two concurrent workers can both run the SELECT, find nothing, and then both INSERT before either commits. A unique constraint makes the second insert fail atomically, which closes the gap.
Handling Failures Between Event Storage and Ticket Creation
A transaction makes database state transitions atomic, but it cannot roll back a call to a remote ticketing API. If a worker creates the ticket and crashes before marking the row done, recovery reruns the job and creates a second ticket.
The durable work queue, often called an outbox, records intended side effects in the same transaction as the state change. Where the ticketing API supports idempotency keys, send the operation key with each request so repeated calls can return the original ticket.
Without downstream idempotency, I would reconcile instead. Before creating a ticket, search the ticketing system for the operation key stored in a custom field or tag. This reduces duplicates but is not perfectly atomic, so concurrent workers still need a claim.
Testing Replayed and Concurrent Webhook Deliveries
Replay tests turn these assumptions into checks. Run each scenario against a test database and a fake ticketing service that counts calls. The assertion is always the same: one ticket and one notification per business operation.
- Duplicate deliveries: post the same payload with an identical webhook delivery ID several times and expect one stored row and one ticket.
- Concurrent processing: start parallel workers on one pending event and expect exactly one successful claim.
- Crash after ticket creation: kill the worker before recording completion, then confirm recovery does not create a second ticket.
- Unprotected downstream retries: make the fake API time out after succeeding and confirm reconciliation finds the existing ticket.
Run the concurrency case many times, because races appear intermittently. A single passing run proves little, while a repeated failure usually points to a missing constraint, an unclaimed job, or an unguarded side effect in the worker.
Idempotency Must Cover the Business Action
The engineering test is simple. Retrying the same delivery, at any moment and from any worker, must not duplicate the intended business operation. Delivery IDs, unique constraints, durable queues and downstream keys each cover one failure mode, and none replaces the others.
Also observe real outcomes, not only logs. Count tickets and customer messages per order in the ticketing system and the email provider, because a successful insert in your own database never proves what happened in an external API.
Top comments (1)
The work-before-response ordering is the root cause worth underlining: once the side effect has happened and the acknowledgment is what times out, the retry is not a malfunction, it is the sender behaving exactly as promised. Recording the event identity durably before any side effect, as you describe, is the same pattern EDI flows settled on with control numbers — dedupe has to be a stored fact, not a hope that delivery will be polite.