Webhook retries are one of those topics almost nobody thinks about until the first partner calls and asks why no orders have reached them since yesterday. A webhook is quick to build: an HTTP POST to someone else’s URL, done. What happens when the other side is in the middle of a deployment, returns 503 for a minute or simply does not answer is not covered in many implementations. Then the event is gone.
The topic matters to me because webhooks are where a cleanly built event-driven system leaves its own garden. Inside AWS, SQS, EventBridge and Lambda take care of retries. With a foreign URL, you are on your own. So in this eighth part of the series, I show you how I would build webhook retries today: with a queue, exponential backoff, signed messages and a dead-letter queue for the case where nothing works anymore.
In short
The best way to send webhooks is as jobs through an SQS queue: a Lambda function delivers them and retries failures with a growing delay via ChangeMessageVisibility. Every message carries a fixed ID and an HMAC signature so the receiver can spot duplicates and reject forgeries. The biggest pitfall is assuming that “send once” is enough: without retries, a DLQ and an alarm, you lose events without noticing.
What you will learn in this article
- How to deliver webhooks reliably through SQS with exponential backoff
- Which HTTP status codes trigger a new attempt and which do not
- How to sign and verify webhooks with HMAC in the Standard Webhooks format
- How to deploy and test everything with the Serverless Framework and TypeScript
Why webhooks get lost so easily
The obvious approach looks like this: an order gets paid, the code stores it and right after that calls every partner’s webhook URL. That always works in testing. In production, you run into receivers that are restarting, load balancers with their own timeouts and networks that simply drop a packet. A single attempt is therefore a bet that everything is fine on the receiver’s side at the exact moment of the event.
There is a second problem. If the call sits in the request path, your own customer waits for the response of a foreign system. If the partner answers slowly, your API becomes slow. Without any answer at all, your Lambda function runs into its timeout. That is exactly the coupling you wanted to avoid with an event-driven architecture.
A webhook is a message to a system you do not control. For messages like that, you need the same tools as for any other unreliable target: decoupling through a queue, retries with a delay and a place for messages that failed for good. I explained the basics of delays with a random component in detail in the article Exponential backoff with jitter. Here, the focus is on what that looks like for webhooks.
Webhook retries with SQS: the architecture at a glance
The example is once again a fictional ticket shop called “Kartenkiste”. When an order is paid or refunded, partners should learn about it, for example a point-of-sale system at the box office or an admission app. Each partner registers a URL and gets its own signing secret.
From event to partner: dispatch, delivery queue, deliverer and DLQ with alarm
The flow has two stages. The shop publishes an event such as OrderPaid on the EventBridge bus. A small dispatch function reads from a DynamoDB table who subscribed to this event and puts one delivery job per subscriber into an SQS queue. A second function, the deliverer, picks up the jobs from the queue, signs them and sends them to the partner via HTTP. If delivery fails, the job stays in the queue and comes back after a growing delay. After eight attempts, SQS moves it to a dead-letter queue (DLQ), a storage area for messages that cannot be processed. A CloudWatch alarm fires as soon as something lands there.
Why a queue instead of retries in code
You could simply write a loop with await sleep() in the deliverer. That has three drawbacks. First, Lambda bills you for every second the function waits. Second, the function stops after 15 minutes at the latest, because Lambda does not run longer. Third, the job is lost when the function crashes or gets redeployed. For webhook retries, a queue solves all three: the message sits safely in SQS while nobody pays for compute time, and the delay can be hours.
In addition, the queue keeps partners apart from each other. If one receiver hangs, only its own jobs are affected. The same principle is behind the fan-out with SNS and SQS: every consumer gets its own lane.
Aren’t EventBridge API destinations enough?
A fair question, because EventBridge can call HTTP endpoints directly through API destinations. According to the documentation, EventBridge retries errors such as 429 and 5xx for up to 24 hours by default, reads the Retry-After header and supports a DLQ. For internal targets or a handful of fixed partners, that is a good choice because you do not need any code of your own.
With real webhooks, however, I run into limits. A request may take five seconds at most, authentication is limited to basic auth, OAuth or an API key, and every new subscriber needs its own rule with a connection. An HMAC signature over the body per subscriber is not part of it. As soon as partners register themselves and are supposed to verify their messages, your own delivery through SQS is more flexible.
The rules: when is a webhook delivered?
Before you implement webhook retries, it pays to agree on clear rules with the receivers. For that, I follow the open specification Standard Webhooks, which several vendors developed together. It defines three headers: webhook-id is a unique ID that stays the same for every attempt. webhook-timestamp contains the time of sending in seconds. webhook-signature carries an HMAC-SHA256 signature over ID, timestamp and body, prefixed with v1,.
For the receiver’s responses, I use these rules:
| Receiver response | What the deliverer does |
|---|---|
| 2xx (200 to 299) | Delivered, the message is deleted |
| 410 Gone | The receiver no longer wants webhooks: disable the subscription, do not retry |
429 or 503 with Retry-After
|
Try again, at the earliest after the given time |
| 3xx (redirect) | Do not follow, treat as a failure and retry |
| All other 4xx and 5xx | Try again with a growing delay |
| Timeout or network error | Try again with a growing delay |
The other 4xx codes are debatable. A 400 or 404 often points to a permanent problem, and retries only postpone the error. On the other hand, many systems briefly return a 404 or 401 during a deployment. Here I stick to the specification and retry everything except 410, because lost events are usually more expensive for partners than a few unnecessary attempts.
Step by step: building webhook retries with Lambda
Now for the code. The project consists of a few small modules that you can test one by one.
Prerequisites: Node.js 22, an AWS account with configured credentials, the Serverless Framework v4 (npm i -g serverless) and basic TypeScript knowledge. I tested the code with TypeScript 7.0.2, Node.js 22.23 and the AWS SDK for JavaScript 3.1159.0.
Step 1: Create the project
Create an empty directory and install the dependencies. The tests run with the built-in Node.js test runner, so you need neither Jest nor Vitest.
mkdir kartenkiste-webhooks && cd kartenkiste-webhooks
npm init -y
npm i @aws-sdk/client-sqs @aws-sdk/client-ssm @aws-sdk/client-dynamodb
npm i -D typescript @types/node@22 @types/aws-lambda
mkdir src test
In package.json, set "type": "module" and two scripts. The --experimental-strip-types option lets Node.js run the TypeScript files directly without compiling them first.
{
"name": "webhook-retries",
"version": "1.0.0",
"private": true,
"type": "module",
"scripts": {
"typecheck": "tsc --noEmit",
"test": "node --test --experimental-strip-types test/*.test.ts"
},
"dependencies": {
"@aws-sdk/client-dynamodb": "^3.1159.0",
"@aws-sdk/client-sqs": "^3.1159.0",
"@aws-sdk/client-ssm": "^3.1159.0"
},
"devDependencies": {
"@types/aws-lambda": "^8.10.164",
"@types/node": "^22.19.0",
"typescript": "^7.0.2"
}
}
{
"compilerOptions": {
"target": "ES2023",
"module": "NodeNext",
"moduleResolution": "NodeNext",
"strict": true,
"noEmit": true,
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"erasableSyntaxOnly": true,
"skipLibCheck": true,
"types": ["node"]
},
"include": ["src", "test"]
}
Step 2: Sign webhooks
The signature is the first thing I build, because sender and receiver must understand it the same way. HMAC stands for “hash-based message authentication code”: a shared secret and the content produce a check value that only someone with the same secret can create. The string that gets signed is <id>.<timestamp>.<body>. Because the timestamp is included, an attacker cannot simply resend an intercepted message later.
// src/signature.ts
import { createHmac, timingSafeEqual } from 'node:crypto';
// Secrets in the Standard Webhooks format: "whsec_" + base64
function secretKey(secret: string): Buffer {
const raw = secret.startsWith('whsec_') ? secret.slice(6) : secret;
return Buffer.from(raw, 'base64');
}
// Signs "<id>.<timestamp>.<body>" with HMAC-SHA256 and returns "v1,<base64>"
export function signWebhook(secret: string, id: string, timestamp: number, body: string): string {
const mac = createHmac('sha256', secretKey(secret))
.update(`${id}.${timestamp}.${body}`)
.digest('base64');
return `v1,${mac}`;
}
export interface VerifyInput {
secret: string;
id: string | undefined;
timestamp: string | undefined;
signatureHeader: string | undefined;
body: string; // the raw body, never re-serialized JSON
nowSeconds?: number;
toleranceSeconds?: number;
}
// Receiver side: check the timestamp, then compare every signature in the header
export function verifyWebhook(input: VerifyInput): boolean {
const { secret, id, timestamp, signatureHeader, body } = input;
if (!id || !timestamp || !signatureHeader) return false;
const ts = Number(timestamp);
if (!Number.isInteger(ts)) return false;
const now = input.nowSeconds ?? Math.floor(Date.now() / 1000);
if (Math.abs(now - ts) > (input.toleranceSeconds ?? 300)) return false;
const expected = Buffer.from(signWebhook(secret, id, ts, body));
// Multiple signatures are separated by spaces (key rotation)
return signatureHeader.split(' ').some((candidate) => {
const given = Buffer.from(candidate);
return given.length === expected.length && timingSafeEqual(given, expected);
});
}
Two details matter. timingSafeEqual compares in constant time so that nobody can guess the signature character by character from response times. And the header may contain several signatures separated by spaces. That way you can rotate a secret without receivers rejecting messages during the transition.
Step 3: Calculate delays
The heart of webhook retries is the delay. It grows by a factor of three with every attempt: one minute, three, nine, 27 and so on, capped at six hours. To keep all failed messages from coming back at the same time, half of the delay is random. With eight attempts, this results in a window of roughly six to twelve hours. That covers a partner outage overnight without keeping messages around for days.
// src/backoff.ts
const BASE_SECONDS = 60;
const FACTOR = 3;
const MAX_DELAY_SECONDS = 6 * 60 * 60; // 6 hours, SQS allows at most 12
const MAX_RETRY_AFTER_SECONDS = 12 * 60 * 60;
// Exponentially growing delay (1, 3, 9, 27 … minutes) with "equal jitter":
// half of it is fixed, the other half is random
export function retryDelaySeconds(attempt: number, random: () => number = Math.random): number {
const exp = Math.min(MAX_DELAY_SECONDS, BASE_SECONDS * FACTOR ** Math.max(0, attempt - 1));
const half = exp / 2;
return Math.max(1, Math.round(half + random() * half));
}
// Retry-After may contain seconds or an HTTP date
export function parseRetryAfter(value: string | null, nowMs: number = Date.now()): number | undefined {
if (!value) return undefined;
const trimmed = value.trim();
let seconds: number;
if (/^\d+$/.test(trimmed)) {
seconds = Number(trimmed);
} else {
const date = Date.parse(trimmed);
if (Number.isNaN(date)) return undefined;
seconds = Math.ceil((date - nowMs) / 1000);
}
return Math.min(MAX_RETRY_AFTER_SECONDS, Math.max(1, seconds));
}
If the receiver sends a Retry-After header, it takes precedence. It may contain a number of seconds or an HTTP date, so the function handles both forms.
Step 4: Send a webhook
The sendWebhook function builds the body exactly once, signs this string and sends it. After that, it sorts the result into one of three categories: delivered, unsubscribed or try again.
// src/deliver.ts
import { parseRetryAfter } from './backoff.ts';
import { signWebhook } from './signature.ts';
export interface DeliveryJob {
webhookId: string; // stays the same for every attempt
subscriberId: string;
eventType: string;
url: string;
payload: unknown;
occurredAt: string;
}
export type DeliveryResult =
| { kind: 'delivered'; status: number }
| { kind: 'gone' }
| { kind: 'retry'; reason: string; retryAfterSeconds?: number };
export type FetchFn = typeof fetch;
export async function sendWebhook(
job: DeliveryJob,
secret: string,
fetchFn: FetchFn = fetch,
nowSeconds: number = Math.floor(Date.now() / 1000),
timeoutMs = 10_000,
): Promise<DeliveryResult> {
// Serialize the body exactly once: this string is signed and sent
const body = JSON.stringify({ type: job.eventType, timestamp: job.occurredAt, data: job.payload });
let response: Response;
try {
response = await fetchFn(job.url, {
method: 'POST',
redirect: 'manual', // do not follow redirects
signal: AbortSignal.timeout(timeoutMs),
headers: {
'content-type': 'application/json',
'user-agent': 'Kartenkiste-Webhooks/1.0',
'webhook-id': job.webhookId,
'webhook-timestamp': String(nowSeconds),
'webhook-signature': signWebhook(secret, job.webhookId, nowSeconds, body),
},
body,
});
} catch (error) {
// Timeout, DNS error, refused connection: try again later
const name = error instanceof Error ? error.name : 'Error';
return { kind: 'retry', reason: `network:${name}` };
}
// Discard the body so the connection is released
await response.body?.cancel().catch(() => undefined);
if (response.status >= 200 && response.status < 300) {
return { kind: 'delivered', status: response.status };
}
if (response.status === 410) {
return { kind: 'gone' };
}
return {
kind: 'retry',
reason: `http:${response.status}`,
retryAfterSeconds: parseRetryAfter(response.headers.get('retry-after')),
};
}
The ten-second limit comes from AbortSignal.timeout. The specification even recommends 15 to 30 seconds. I deliberately stay below that, because a Lambda function processes up to ten messages per invocation and a slow partner would otherwise hold up the whole batch. With redirect: 'manual', the deliverer does not follow redirects. A partner who has moved should update their URL instead of your messages silently ending up somewhere else.
Step 5: The deliverer with batch failures
Now comes the core of webhook retries. The function receives a batch from SQS and reports back through batchItemFailures which messages failed. Only those stay in the queue, Lambda deletes all others. To make this work, you have to enable the ReportBatchItemFailures option on the event source mapping.
The actual trick is ChangeMessageVisibility. Normally, a failed message becomes visible again after the queue’s visibility timeout expires, so always after the same time. If you set a new visibility timeout for exactly this message before reporting back, you decide yourself when it returns. SQS supplies the number of previous attempts in the ApproximateReceiveCount attribute.
🎬 Webhook retries with growing wait times – this animated diagram only runs on my site: watch it on sascha-ms.de
// src/handler.ts
import type { SQSBatchResponse, SQSEvent, SQSRecord } from 'aws-lambda';
import { ChangeMessageVisibilityCommand, SQSClient } from '@aws-sdk/client-sqs';
import { GetParameterCommand, SSMClient } from '@aws-sdk/client-ssm';
import { DynamoDBClient, UpdateItemCommand } from '@aws-sdk/client-dynamodb';
import { retryDelaySeconds } from './backoff.ts';
import { sendWebhook, type DeliveryJob, type DeliveryResult } from './deliver.ts';
export interface DeliveryDeps {
maxAttempts: number;
getSecret: (subscriberId: string) => Promise<string>;
send: (job: DeliveryJob, secret: string) => Promise<DeliveryResult>;
delayMessage: (receiptHandle: string, seconds: number) => Promise<void>;
disableSubscription: (job: DeliveryJob) => Promise<void>;
random?: () => number;
log?: (entry: Record<string, unknown>) => void;
}
export function createDeliveryHandler(deps: DeliveryDeps) {
const log = deps.log ?? ((entry) => console.log(JSON.stringify(entry)));
async function processRecord(record: SQSRecord): Promise<boolean> {
const job = JSON.parse(record.body) as DeliveryJob;
const attempt = Number(record.attributes.ApproximateReceiveCount);
const base = { webhookId: job.webhookId, subscriberId: job.subscriberId, attempt };
const secret = await deps.getSecret(job.subscriberId);
const result = await deps.send(job, secret);
if (result.kind === 'delivered') {
log({ ...base, outcome: 'delivered', status: result.status });
return true;
}
if (result.kind === 'gone') {
// 410 Gone: the receiver does not want any more webhooks
await deps.disableSubscription(job);
log({ ...base, outcome: 'disabled' });
return true;
}
if (attempt >= deps.maxAttempts) {
// Last attempt: SQS will move the message to the DLQ next
log({ ...base, outcome: 'exhausted', reason: result.reason });
return false;
}
const delay = result.retryAfterSeconds ?? retryDelaySeconds(attempt, deps.random);
await deps.delayMessage(record.receiptHandle, delay);
log({ ...base, outcome: 'retry', reason: result.reason, delaySeconds: delay });
return false;
}
return async (event: SQSEvent): Promise<SQSBatchResponse> => {
const results = await Promise.allSettled(event.Records.map(processRecord));
const batchItemFailures = results.flatMap((result, i) => {
const ok = result.status === 'fulfilled' && result.value;
if (result.status === 'rejected') {
log({ messageId: event.Records[i].messageId, outcome: 'error', error: String(result.reason) });
}
return ok ? [] : [{ itemIdentifier: event.Records[i].messageId }];
});
return { batchItemFailures };
};
}
// Wiring with real AWS clients
const sqs = new SQSClient({});
const ssm = new SSMClient({});
const dynamo = new DynamoDBClient({});
const secretCache = new Map<string, { value: string; expires: number }>();
async function getSecret(subscriberId: string): Promise<string> {
const cached = secretCache.get(subscriberId);
if (cached && cached.expires > Date.now()) return cached.value;
const out = await readParameter(`/kartenkiste/webhooks/${subscriberId}/signing-secret`);
secretCache.set(subscriberId, { value: out, expires: Date.now() + 5 * 60_000 });
return out;
}
async function readParameter(name: string): Promise<string> {
const out = await ssm.send(new GetParameterCommand({ Name: name, WithDecryption: true }));
const value = out.Parameter?.Value;
if (!value) throw new Error(`Parameter ${name} is empty`);
return value;
}
export const handler = createDeliveryHandler({
maxAttempts: Number(process.env.MAX_ATTEMPTS ?? '8'),
getSecret,
send: (job, secret) => sendWebhook(job, secret),
delayMessage: async (receiptHandle, seconds) => {
await sqs.send(new ChangeMessageVisibilityCommand({
QueueUrl: process.env.QUEUE_URL,
ReceiptHandle: receiptHandle,
VisibilityTimeout: seconds,
}));
},
disableSubscription: async (job) => {
await dynamo.send(new UpdateItemCommand({
TableName: process.env.TABLE_NAME,
Key: { eventType: { S: job.eventType }, subscriberId: { S: job.subscriberId } },
UpdateExpression: 'SET enabled = :off, disabledReason = :reason',
ExpressionAttributeValues: { ':off': { BOOL: false }, ':reason': { S: 'HTTP 410' } },
}));
},
});
The dependencies are passed in through createDeliveryHandler, which makes the function testable without AWS. The secrets are stored as SecureString in SSM Parameter Store and cached in memory for five minutes so that not every invocation queries Parameter Store. On the last attempt, the deliverer no longer sets a new delay. SQS then moves the message to the DLQ on the next receive. For other ways to catch errors in asynchronous invocations, see the article on Lambda Destinations.
Step 6: Queue one job per subscriber
The dispatch function is attached to the EventBridge bus. It loads all active subscriptions for the event type and writes one job per subscription into the queue, in batches of at most ten messages, because SendMessageBatch does not allow more.
// src/dispatch.ts
import type { EventBridgeEvent } from 'aws-lambda';
import { DynamoDBClient, QueryCommand, type AttributeValue } from '@aws-sdk/client-dynamodb';
import { SQSClient, SendMessageBatchCommand } from '@aws-sdk/client-sqs';
import type { DeliveryJob } from './deliver.ts';
export interface Subscription {
subscriberId: string;
url: string;
}
// One job per subscriber. The webhook-id is derived from the event and the subscriber:
// If the dispatcher runs twice, the receiver gets the same ID and can deduplicate.
export function buildJobs(event: EventBridgeEvent<string, unknown>, subscriptions: Subscription[]): DeliveryJob[] {
return subscriptions.map((s) => ({
webhookId: `msg_${event.id}_${s.subscriberId}`,
subscriberId: s.subscriberId,
eventType: event['detail-type'],
url: s.url,
payload: event.detail,
occurredAt: event.time,
}));
}
export function chunk<T>(items: T[], size: number): T[][] {
const out: T[][] = [];
for (let i = 0; i < items.length; i += size) out.push(items.slice(i, i + size));
return out;
}
const dynamo = new DynamoDBClient({});
const sqs = new SQSClient({});
async function loadSubscriptions(eventType: string): Promise<Subscription[]> {
const result: Subscription[] = [];
let startKey: Record<string, AttributeValue> | undefined;
do {
const out = await dynamo.send(new QueryCommand({
TableName: process.env.TABLE_NAME,
KeyConditionExpression: 'eventType = :t',
FilterExpression: 'enabled = :on',
ExpressionAttributeValues: { ':t': { S: eventType }, ':on': { BOOL: true } },
ExclusiveStartKey: startKey,
}));
for (const item of out.Items ?? []) {
result.push({ subscriberId: item.subscriberId.S!, url: item.url.S! });
}
startKey = out.LastEvaluatedKey;
} while (startKey);
return result;
}
export const handler = async (event: EventBridgeEvent<string, unknown>): Promise<void> => {
const jobs = buildJobs(event, await loadSubscriptions(event['detail-type']));
for (const batch of chunk(jobs, 10)) {
const out = await sqs.send(new SendMessageBatchCommand({
QueueUrl: process.env.QUEUE_URL,
Entries: batch.map((job, i) => ({ Id: String(i), MessageBody: JSON.stringify(job) })),
}));
if (out.Failed?.length) {
// Throw: EventBridge invokes the function again, the IDs stay the same
throw new Error(`${out.Failed.length} jobs could not be queued`);
}
}
console.log(JSON.stringify({ eventId: event.id, jobs: jobs.length }));
};
Pay attention to the webhookId. It combines the event ID and the subscriber. EventBridge delivers events at least once, so the dispatch function can run twice. Because the ID stays the same in that case, the receiver recognizes the duplicate. Why duplicate delivery is the normal case is explained in the article on idempotency in AWS Lambda.
Step 7: Infrastructure with the Serverless Framework
This time I use the Serverless Framework for the infrastructure. The file describes both functions, the subscriptions table, the queue with its DLQ and an alarm.
# serverless.yml
service: kartenkiste-webhooks
frameworkVersion: '4'
provider:
name: aws
runtime: nodejs22.x
architecture: arm64
region: eu-central-1
memorySize: 256
logRetentionInDays: 14
environment:
TABLE_NAME: !Ref SubscriptionsTable
QUEUE_URL: !Ref DeliveryQueue
MAX_ATTEMPTS: '8'
iam:
role:
statements:
- Effect: Allow
Action: [dynamodb:Query, dynamodb:UpdateItem]
Resource: !GetAtt SubscriptionsTable.Arn
- Effect: Allow
Action: [sqs:SendMessage, sqs:ChangeMessageVisibility]
Resource: !GetAtt DeliveryQueue.Arn
- Effect: Allow
Action: ssm:GetParameter
Resource: !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/kartenkiste/webhooks/*
functions:
dispatch:
handler: src/dispatch.handler
timeout: 30
events:
- eventBridge:
pattern:
source: [kartenkiste.shop]
detail-type: [OrderPaid, OrderRefunded]
deliver:
handler: src/handler.handler
timeout: 30
events:
- sqs:
arn: !GetAtt DeliveryQueue.Arn
batchSize: 10
functionResponseType: ReportBatchItemFailures
maximumConcurrency: 5
resources:
Resources:
SubscriptionsTable:
Type: AWS::DynamoDB::Table
Properties:
BillingMode: PAY_PER_REQUEST
AttributeDefinitions:
- { AttributeName: eventType, AttributeType: S }
- { AttributeName: subscriberId, AttributeType: S }
KeySchema:
- { AttributeName: eventType, KeyType: HASH }
- { AttributeName: subscriberId, KeyType: RANGE }
DeliveryDLQ:
Type: AWS::SQS::Queue
Properties:
MessageRetentionPeriod: 1209600 # 14 days
DeliveryQueue:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 180 # 6 x Lambda timeout
MessageRetentionPeriod: 345600 # 4 days
RedrivePolicy:
deadLetterTargetArn: !GetAtt DeliveryDLQ.Arn
maxReceiveCount: 8 # = MAX_ATTEMPTS
DeliveryDLQAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmDescription: Webhooks permanently undeliverable
Namespace: AWS/SQS
MetricName: ApproximateNumberOfMessagesVisible
Dimensions:
- { Name: QueueName, Value: !GetAtt DeliveryDLQ.QueueName }
Statistic: Maximum
Period: 300
EvaluationPeriods: 1
Threshold: 0
ComparisonOperator: GreaterThanThreshold
TreatMissingData: notBreaching
Three values belong together and should always be changed together. At 180 seconds, the queue’s visibility timeout is six times the function timeout, as AWS recommends for SQS triggers. maxReceiveCount and MAX_ATTEMPTS are both set to eight. And the retention period of four days is much longer than the retry window of at most twelve hours. maximumConcurrency also limits how many deliverers run at the same time, so that a large backlog does not flood your partners with requests.
You have to add the sqs:ChangeMessageVisibility permission yourself. For the SQS trigger, the framework only grants the permissions to read and delete. For the secrets, ssm:GetParameter is enough as long as you use the default aws/ssm key. With a customer managed KMS key, you also need kms:Decrypt on that key.
Step 8: Test and deploy
The tests check the signature, the delays, the classification of responses and the behavior of the handler with fake dependencies. Here are the tests for the handler:
// test/handler.test.ts
import { test } from 'node:test';
import assert from 'node:assert/strict';
import type { SQSEvent, SQSRecord } from 'aws-lambda';
import { createDeliveryHandler, type DeliveryDeps } from '../src/handler.ts';
import type { DeliveryResult } from '../src/deliver.ts';
function record(id: string, receiveCount: number): SQSRecord {
return {
messageId: id, receiptHandle: `rh-${id}`,
body: JSON.stringify({ webhookId: `msg_${id}`, subscriberId: 'kasse-nord', eventType: 'OrderPaid', url: 'https://partner.example', payload: {}, occurredAt: '2026-10-29T12:00:00Z' }),
attributes: { ApproximateReceiveCount: String(receiveCount) } as SQSRecord['attributes'],
messageAttributes: {}, md5OfBody: '', eventSource: 'aws:sqs', eventSourceARN: 'arn:aws:sqs:eu-central-1:111122223333:q', awsRegion: 'eu-central-1',
};
}
function setup(results: Record<string, DeliveryResult>) {
const delays: Array<[string, number]> = [];
const disabled: string[] = [];
const deps: DeliveryDeps = {
maxAttempts: 8,
getSecret: async () => 'whsec_dGVzdA==',
send: async (job) => results[job.webhookId],
delayMessage: async (rh, s) => { delays.push([rh, s]); },
disableSubscription: async (job) => { disabled.push(job.webhookId); },
random: () => 1,
log: () => undefined,
};
return { handler: createDeliveryHandler(deps), delays, disabled };
}
test('only failed messages end up in batchItemFailures', async () => {
const { handler, delays } = setup({
msg_a: { kind: 'delivered', status: 200 },
msg_b: { kind: 'retry', reason: 'http:500' },
});
const event: SQSEvent = { Records: [record('a', 1), record('b', 3)] };
assert.deepEqual(await handler(event), { batchItemFailures: [{ itemIdentifier: 'b' }] });
assert.deepEqual(delays, [['rh-b', 540]]);
});
test('Retry-After takes precedence over the backoff', async () => {
const { handler, delays } = setup({ msg_a: { kind: 'retry', reason: 'http:429', retryAfterSeconds: 600 } });
await handler({ Records: [record('a', 1)] });
assert.deepEqual(delays, [['rh-a', 600]]);
});
test('410 disables the subscription and is not retried', async () => {
const { handler, disabled } = setup({ msg_a: { kind: 'gone' } });
assert.deepEqual(await handler({ Records: [record('a', 1)] }), { batchItemFailures: [] });
assert.deepEqual(disabled, ['msg_a']);
});
test('no new delay on the last attempt, the DLQ takes over', async () => {
const { handler, delays } = setup({ msg_a: { kind: 'retry', reason: 'network:TypeError' } });
assert.deepEqual(await handler({ Records: [record('a', 8)] }), { batchItemFailures: [{ itemIdentifier: 'a' }] });
assert.deepEqual(delays, []);
});
test('unexpected errors only mark the affected message', async () => {
const { handler } = setup({ msg_a: { kind: 'delivered', status: 200 } });
const broken = { ...record('x', 1), body: 'not json' };
assert.deepEqual(await handler({ Records: [record('a', 1), broken] }), { batchItemFailures: [{ itemIdentifier: 'x' }] });
});
npm run typecheck
npm test
You should see output with # pass 19 and # fail 0. After that, deploy and create a test subscriber. A function URL running the receiver from the next section works well as a target.
serverless deploy
# Create the subscriber's secret (32 random bytes)
SECRET="whsec_$(openssl rand -base64 32)"
aws ssm put-parameter --type SecureString \
--name /kartenkiste/webhooks/kasse-nord/signing-secret --value "$SECRET"
# Add the subscription (take the table name from the output of serverless info)
aws dynamodb put-item --table-name <TABLE> --item '{
"eventType": {"S": "OrderPaid"}, "subscriberId": {"S": "kasse-nord"},
"url": {"S": "https://<your-function-url>/"}, "enabled": {"BOOL": true}}'
# Send a test event
aws events put-events --entries '[{"Source":"kartenkiste.shop",
"DetailType":"OrderPaid","Detail":"{\"orderId\":\"K-1001\"}"}]'
# Follow the deliverer's logs
serverless logs -f deliver --tail
In the deliverer’s log, you will then see a line like {"webhookId":"msg_…_kasse-nord","attempt":1,"outcome":"delivered","status":202}. To try out the retries, let the receiver function return a 503 for a while. Entries with "outcome":"retry" and a delaySeconds value that grows from attempt to attempt will appear.
The receiver side: verify the signature and deduplicate
A webhook is only as secure as the check on the receiving end. If you offer webhooks to partners yourself, give them a short example. This is what a receiver looks like as a Lambda function behind a function URL:
// src/receiver.ts
import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from 'aws-lambda';
import { verifyWebhook } from './signature.ts';
// Example for the other side: a partner receives the webhooks via a function URL
const seen = new Set<string>(); // demo only; use DynamoDB with a condition in production
export const handler = async (event: APIGatewayProxyEventV2): Promise<APIGatewayProxyResultV2> => {
const body = event.isBase64Encoded
? Buffer.from(event.body ?? '', 'base64').toString('utf8')
: event.body ?? '';
const id = event.headers['webhook-id'];
const valid = verifyWebhook({
secret: process.env.WEBHOOK_SECRET ?? '',
id,
timestamp: event.headers['webhook-timestamp'],
signatureHeader: event.headers['webhook-signature'],
body,
});
if (!valid) return { statusCode: 401 };
// Duplicate delivery? Just acknowledge it, do not process it again
if (seen.has(id!)) return { statusCode: 200 };
seen.add(id!);
const message = JSON.parse(body) as { type: string; data: unknown };
console.log(JSON.stringify({ id, type: message.type }));
// Respond quickly, do the actual work asynchronously
return { statusCode: 202 };
};
The order is crucial. First, the signature is verified over the raw body, before anything gets parsed. If you parse the JSON first and serialize it again, you often get a slightly different string, and the signature no longer matches. Next comes the duplicate check via webhook-id. The Set in the example only lives as long as the Lambda instance. In real systems, you write the ID to DynamoDB with a condition. I show the complete version, with queue and worker, in Receiving webhooks with Lambda. Finally, the receiver responds quickly with 202 and does the actual work asynchronously so that the sender does not run into a timeout.
Common mistakes with webhook retries
The delay depends on the queue timeout. Without ChangeMessageVisibility, every failed message comes back after three minutes, eight times in a row. After a little more than 20 minutes, it is in the DLQ, even though the partner may only have had an hour of maintenance.
The handler throws an exception. If an unhandled error escapes the handler, the whole batch counts as failed, including the messages that were already delivered. That is why the handler processes each message through Promise.allSettled and only reports the affected IDs.
The webhook-id changes with every attempt. Then the receiver cannot detect duplicates. The ID belongs to the event and the subscriber, not to the individual HTTP call.
Throttling counts too. ApproximateReceiveCount increases with every receive, including when Lambda could not process a message due to missing capacity. If you limit concurrency very tightly, messages can end up in the DLQ sooner than expected. The AWS recommendation of a sixfold visibility timeout gives you some headroom here.
Nobody looks at the DLQ. A dead-letter queue without an alarm is a trash can. Connect the alarm to an SNS topic or your chat channel, and plan to move the messages back to the main queue after an outage using the “redrive” feature in the SQS console.
Costs and security
The costs of webhook retries are moderate with this pattern. Every SQS action counts as a request, including ChangeMessageVisibility, and according to AWS one million requests per month are free. A webhook with three failed attempts generates roughly a dozen requests for sending, receiving, visibility and deleting. With Lambda, you mainly pay for the time the deliverer waits for the partner. If you want to work out what that means for your volume, the Lambda cost calculator helps.
Security: watch out for SSRF
On the security side, there is one point that is often overlooked: if partners enter their URL themselves, someone can also register an internal address, such as http://169.254.169.254/ or an IP from your VPC. This kind of attack is called server-side request forgery (SSRF). Therefore, only allow HTTPS URLs, check at registration time that the hostname does not resolve to private or link-local addresses, and run the deliverer without a VPC attachment so it cannot reach internal systems in the first place.
Signing secrets belong neither in the subscriptions table nor in environment variables. In Parameter Store they are encrypted, and the IAM role may only read parameters under /kartenkiste/webhooks/. To rotate a key, create the new secret, sign with both for a while and remove the old one afterwards.
My take
I consider webhooks one of the most underestimated integration patterns. They look trivial, and that is exactly why they often get written into a controller on the side. Yet a webhook is a promise to a partner: “You will hear about it when something changes.” If you keep that promise with a single HTTP call, you only keep it on good days.
The solution with SQS and ChangeMessageVisibility is, for me, the best trade-off between effort and reliability. You need no workflow service, no database for the delivery status and no cron job collecting stuck jobs. The queue is the state. Where I deliberately disagree: some vendors retry for days, and the specification gives an example spanning about three days. For order and status events, I find that too long. After twelve hours, the information is often outdated, and a human should look at the problem. If your partners need a longer window, increase the number of attempts and the queue’s retention period.
What this pattern does not guarantee
What this pattern does not provide is a guarantee about ordering. A refunded order can reach the partner before the payment confirmation if the first message is stuck in a delay. Receivers should therefore work with the timestamp in the body or fetch the current state through an API when needed.
FAQ
How often should you retry a webhook?
There is no fixed standard for webhook retries. Five to ten attempts with exponentially growing delays over a few hours to a few days are common. More important than the exact number is that delays include a random component and that failed messages land in a DLQ with an alarm.
Which HTTP status codes should trigger a retry?
Timeouts, network errors, 429 and all 5xx codes should always be retried. According to the Standard Webhooks specification, any response outside 2xx counts as a failure. 410 Gone is the exception: the receiver no longer wants messages, and the subscription gets disabled.
How do you prevent a webhook from being processed twice?
The sender sends the same webhook-id with every attempt. The receiver stores processed IDs, for example with a conditional write in DynamoDB, and acknowledges duplicates with 200 without processing them again.
How do you verify a webhook signature?
The receiver computes an HMAC-SHA256 over ID, timestamp and raw body with the shared secret and compares it with the header in constant time. In addition, it rejects messages whose timestamp differs from the current time by more than a few minutes.
Your next step
Take a look at how your application sends webhooks or other calls to external systems today. If the call sits directly in the request path, that is the first candidate for a queue with webhook retries. You do not have to rebuild everything at once: an SQS queue with a DLQ and an alarm already prevents events from silently disappearing. You can add backoff and signatures afterwards.
I am curious how you handle this: how long do you retry webhooks, and what do you do with partners that are permanently unreachable? Feel free to share it in the comments.
More in this series: Previous part: DynamoDB TTL: build seat holds that expire · Next part: Receiving webhooks with Lambda: signatures, duplicates, SQS · All parts of the series “Serverless patterns in practice”
Sources
- Standard Webhooks: specification and website
- Standard Webhooks: specification on GitHub
- AWS: Handling errors for an SQS event source in Lambda
- Amazon SQS visibility timeout (AWS documentation)
- AWS: ChangeMessageVisibility (API Reference)
- AWS re:Post: set the visibility timeout to at least six times the Lambda timeout
- API destinations as targets in Amazon EventBridge (AWS documentation)
- AWS re:Post: EventBridge API destination timeouts and retry policy
- AWS: Restricting decryption of SecureString parameter values
- Amazon SQS pricing (AWS)
This post was created with AI assistance.
👉 Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator – it runs right in your browser.
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Top comments (1)
The opening scenario is the real argument: the first sign you have lost events is a partner calling the next day to ask why no orders arrived, which means the failure was silent on your side the whole time. Putting a fixed ID on every message is what makes the retries safe to be generous with — the receiver can spot the duplicate instead of processing the same event twice, so backoff stops being a trade-off against double-delivery. The DLQ plus an alarm is the part I would not skip: a queue that quietly accumulates undeliverable webhooks is just a slower version of losing them.