I work with the Auto-Respond Team. I used AI assistance to edit this article, then checked the code and technical claims before publishing. The examples use fake adapters and fake identifiers. No customer data is involved.
Webhook providers retry. Your load balancer retries. A worker can crash after it writes to the database but before it acknowledges the message. If ingestion treats each delivery as a new lead, one real inquiry can produce two conversations, two timers, and a very confused customer.
The tempting fix is a seen flag. That works until the flag is written in a different transaction from the jobs it is meant to protect.
A safer design gives every provider event a stable identity, records that identity atomically with its downstream work, and lets each worker retry a bounded number of times. This article walks through that design in TypeScript.
Start with the delivery contract
An inbound delivery is not the same thing as a lead.
One lead may produce a creation event, an update, and a reply. Each event may arrive more than once. Deduplicating only on leadId would throw away legitimate updates. Deduplicating on a request timestamp would do the opposite: every retry would look new.
I model the boundary like this:
type ProviderEvent = {
provider: "yelp" | "thumbtack" | "other";
accountId: string;
eventId?: string;
leadId: string;
kind: "lead.created" | "lead.updated" | "lead.replied";
occurredAt: string;
payload: unknown;
};
Use the provider's immutable event ID when it exists. Scope it by provider and connected account, because IDs from different tenants may overlap.
When a provider does not supply an event ID, build a fallback from documented stable fields. Do not include a delivery timestamp, request ID, or retry counter. Those values change on redelivery.
import { createHash } from "node:crypto";
export function idempotencyKey(event: ProviderEvent): string {
const identity = event.eventId
? ["provider-event", event.eventId]
: ["lead-event", event.leadId, event.kind];
const scoped = [
event.provider,
event.accountId,
...identity,
];
return createHash("sha256")
.update(JSON.stringify(scoped))
.digest("hex");
}
This function is deterministic, but that alone does not make the system safe. The database still needs to enforce uniqueness.
A fallback key is only as good as the provider contract behind it. If lead.updated can happen several times with the same leadId, add a stable version or change identifier. If the provider exposes neither, ask for a contract change or accept a short dedupe window with an explicit collision risk. Do not quietly pretend the ambiguity is solved.
Let the database win the race
Two application instances can receive the same retry at the same time. A "select, then insert" sequence has a race between those statements.
Put the invariant in the database:
CREATE TABLE webhook_receipts (
id UUID PRIMARY KEY,
provider TEXT NOT NULL,
account_id TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
event_kind TEXT NOT NULL,
external_lead_id TEXT NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
payload_digest TEXT NOT NULL,
UNIQUE (provider, account_id, idempotency_key)
);
CREATE TABLE delivery_jobs (
id UUID PRIMARY KEY,
receipt_id UUID NOT NULL REFERENCES webhook_receipts(id),
job_key TEXT NOT NULL UNIQUE,
job_type TEXT NOT NULL,
status TEXT NOT NULL,
attempt_count INTEGER NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMPTZ NOT NULL,
last_error_code TEXT
);
The unique constraint decides which request created the receipt. Every concurrent retry gets the existing row.
The receipt and its jobs must be created in one transaction. Otherwise a process can write "seen" and crash before it schedules the work. The next delivery then looks like a duplicate and the lead is never processed.
type IngestResult =
| { state: "created"; receiptId: string }
| { state: "duplicate"; receiptId: string };
async function ingest(rawBody: Buffer, signature: string): Promise<IngestResult> {
verifyProviderSignature(rawBody, signature);
const event = decodeProviderEvent(rawBody);
const key = idempotencyKey(event);
const digest = createHash("sha256").update(rawBody).digest("hex");
return db.transaction(async (tx) => {
const receipt = await tx.receipts.insertOrGet({
provider: event.provider,
accountId: event.accountId,
idempotencyKey: key,
eventKind: event.kind,
externalLeadId: event.leadId,
payloadDigest: digest,
});
if (!receipt.created) {
return { state: "duplicate", receiptId: receipt.id };
}
await tx.jobs.insert({
receiptId: receipt.id,
jobKey: `${receipt.id}:initial-processing:v1`,
jobType: "initial-processing",
status: "queued",
nextAttemptAt: new Date(),
});
return { state: "created", receiptId: receipt.id };
});
}
insertOrGet needs to be an atomic upsert backed by the unique constraint. It is not shorthand for a read followed by a write.
Validate the provider signature against the raw request bytes before trusting fields from the payload. If persistence fails, return a retryable server error. Return success only after the transaction commits. A duplicate whose original transaction committed can receive the same success response as the first delivery.
This pattern shows up in marketplace lead systems, including the ingestion side of a Yelp auto responder. The important part is not the channel name. It is treating retries as a normal delivery state instead of an exceptional one.
Make jobs idempotent too
The webhook receipt protects ingestion. It does not protect a worker that times out after calling another service.
Give each logical side effect its own key. A job retry may run several times, but it should not create several logical operations.
For example:
type JobIdentity = {
receiptId: string;
operation: "initial-response" | "crm-upsert" | "audit-export";
version: number;
};
function jobKey(value: JobIdentity): string {
return [
value.receiptId,
value.operation,
`v${value.version}`,
].join(":");
}
Pass that key to downstream APIs that support idempotency. For systems that do not, store a local operation record before the call and reconcile uncertain outcomes instead of immediately repeating the side effect.
An HTTP timeout means "the client did not receive a response." It does not prove the server did nothing. That distinction matters for outbound messages and CRM writes.
If a worker crashes after a successful call but before marking the job complete, the recovery path should check the stored operation state or query the downstream system by a stable external reference. Blind retries are easy to code and hard to clean up.
Bound retries and preserve the job
Retries need an end. I use a short schedule for transient failures, with jitter added by the worker:
const RETRY_DELAYS_MS = [
1_000,
5_000,
30_000,
120_000,
] as const;
function nextAttempt(attemptCount: number, now = Date.now()): Date | null {
const base = RETRY_DELAYS_MS[attemptCount];
if (base === undefined) return null;
const jitter = Math.floor(Math.random() * Math.min(1_000, base / 4));
return new Date(now + base + jitter);
}
When the schedule is exhausted, move the job to a terminal review state. Keep the receipt and the job. Deleting either erases the evidence needed to diagnose the miss or replay it safely.
Not every failure deserves a retry. Invalid signatures, unsupported event kinds, and schema errors should fail without entering the queue. Rate limits and temporary upstream errors usually can retry. Authentication failures should pause the integration and surface a credential problem instead of hammering the endpoint.
Do not let retries delay the webhook acknowledgement. The request handler persists the receipt and queue entry, then returns. Workers own the slower work.
Store enough audit evidence, not the whole lead
An audit trail should answer:
- Which provider event did we receive?
- Which idempotency key won?
- Which jobs were created?
- How many attempts ran, and what state are they in now?
- Was a duplicate acknowledged without creating more work?
You usually do not need the full message body to answer those questions. Store the external identifiers, timestamps, event kind, payload digest, operation versions, and sanitized error codes. Keep raw payload retention short and access controlled if the business requires it at all.
A digest is useful for spotting a provider that reuses one event ID with different payloads. That should be an alert, not a second insert. Record both digests in a security-conscious diagnostic path and investigate the contract violation.
Test the failure boundaries
The happy path is one test. The useful tests force each boundary:
describe("webhook ingestion", () => {
it("creates one receipt and one job for repeated delivery", async () => {
await Promise.all([
postFixture("evt-123"),
postFixture("evt-123"),
postFixture("evt-123"),
]);
expect(await countReceipts("evt-123")).toBe(1);
expect(await countJobs("evt-123")).toBe(1);
});
it("keeps distinct updates for the same lead", async () => {
await postFixture("evt-created", { kind: "lead.created", leadId: "lead-7" });
await postFixture("evt-updated", { kind: "lead.updated", leadId: "lead-7" });
expect(await countReceiptsForLead("lead-7")).toBe(2);
});
it("does not lose work when the transaction rolls back", async () => {
failNextJobInsert();
await expect(postFixture("evt-rollback")).rejects.toThrow();
await postFixture("evt-rollback");
expect(await countReceipts("evt-rollback")).toBe(1);
expect(await countJobs("evt-rollback")).toBe(1);
});
});
Also test a worker timeout after the downstream side effect succeeds. That is where local idempotency often looks correct while the external system still gets duplicates.
The final property is simple to state: one provider event creates one durable receipt, and every required operation remains recoverable until it reaches a known terminal state. The implementation takes more than a seen flag, but it gives retries somewhere safe to land.
Top comments (0)