DEV Community: Javier Mendoza

Monitoring Third-Party Webhook Delays with AWS Durable Functions

Javier Mendoza — Wed, 31 Dec 2025 20:49:47 +0000

In this blog, I'll explain how did we use Durable functions to monitor if delays occurs with the reception of third party webhooks.

When integrating with external third-party services, webhook notifications don't always arrive immediately or at all.

We needed visibility into:

Whether webhooks actually arrive
How long they take to reach our system

The idea was simple:

API GW -> Lambda function which makes a request to third party, start a Durable Function by sending an event to Eventbridge, and returns the sync response to the user because the client needed a synchronous response.

2.
Durable Function starts, generate a callbackId, stores the Callback in DynamoDB and waits until a callback is completed. This durable function will allow us to know if the webhook notification ever arrived and if it does how long did it take to land in our system.

3.
API GW-> SQS -> Webhook handler Lambda. This webhook handler consumes the Webhook notification from the third party and, gets the callbackId associated and resumes the Durable Function execution.
Having SQS prevent us from having from the race condition scenario where a webhook Notification arrives before the Durable Function has not started.

We monitor everything with AWS X-Ray and you can create Cloudwatch Dashboards to monitor how long does it take to arrive on average your notification.

Thank you so much for the reading.

A starters guide from SQS and some learning along the way

Javier Mendoza — Fri, 14 Nov 2025 22:59:40 +0000

In this guide, we will go through the basics of what SQS is and why it’s important for you to start using it.

What is it?

Is a service from AWS which is queue where messages wait their turn to be processed.

As a fun fact, SQS was the first Serverless Service launched by AWS in 2006

Why is it important?

SQS allows you to improve the resilence and retryability of your system.

Let's imagine that you've the following scenario:

Your API exposes a POST /create-order endpoint, and it’s connected directly to a Lambda function called 'CreateOrder'.

If your system receives a lot of traffic, your CreateOrder Lambda may start throttling, which means some orders could be lost.

In order to avoid this, we can put an SQS in the middle.

Having the SQS in the middle ensures that requests are not lost.

The queu will hold each request until it gets consumed or it has reached the maximum age of the message in the queue, by default is 4 days and the maximum is 14 days.).

When a consumer begins processing a message, SQS hides it temporarily so that no other consumer can process it at the same time.
This is controlled by the visibility timeout, which defines how long a message stays hidden while being processed.

The message is attempted to be consumed up to the amount of times defined by the maxReceiveCount.
This count represents how many attempts the message can go through before being considered “failed.”

DLQ

We can setup a Dead-Letter-Queue (DLQ) Attached to our SQS, so that once a message reaches the maxReceiveCount, it is moved to the DLQ for debugging.

When a message reaches your DLQ, you can redrive the message to the original queue through the AWS Console or through the CLI

Through the AWS Console:

You should go to your Queu and select ->

Start DLQ Redrive

by default it redrives to the source queu but you can select other queu to the messages to be delivered to.

Typical SQS Consumer

We've seen that you can consume SQS by using lambdas, but that's far from the only one. You can also use:

EventBridge pipes to connect to a state machine
SNS
EC2/ECS
Types of SQS

There are two types of SQS

Standard SQS where messages may be delivered out of order

Priority SQS messages are processed in order

and there are also another type of messages which deserve their own space 'fair queues' which will cover in the next post.

More to come!

How to reuse variables in your Lambda across invocations and know when it's about to terminate

Javier Mendoza — Sun, 08 Dec 2024 19:31:19 +0000

Reader: Why do I need to know this? Can't we just jump straight into the code?

Writer: Understanding AWS Lambda lifecycles is like learning how to navigate a river. The river flows continuously, with twists, turns, and varying currents. If you don't understand the flow, you might find yourself fighting against it or getting stuck. But if you know how it moves, when it speeds up, slows down, or where it pools. You can align with it and let it carry you where you need to go.

Reader: Alright, let's go with your analogy.

Writer: Here it is! the Lambda execution environment lifecycle

Reader: Woooow, I always wanted to read the official docs paraphrased by a junior dev.

Writer: Ignoring the jab Let me explain the lifecycle in simple terms.

The Init Phase and Variable Reuse

The init phase occurs only during a cold start. When a new Lambda execution environment is created. During this phase the lambda performs 3 tasks:

Starts up extensions
Sets up the runtime
Runs any initialization code placed outside the handler function

This initialization is important because the variables and resources you set up here can be reused across multiple function invocations.

Example Caching
In the code below, we store a token at the module level. The first time the Lambda environment is created, we fetch the token and save it. On subsequent invocations (warm starts), we simply reuse the token, saving time and resources.

let token: Token;

const getToken = async (): Token => {
  if (token) {
    console.log('Reusing Token');
    return token;
  }
  console.log('Generating Token');
  token = await getTokenFromThirdPartyAPI();
  return token;
};

export const handler = async () => {
  const myToken = await getToken();
  // Use myToken as needed
};

More use cases

In Jeremy Daly’s blog, he shows how declaring variables outside the handler enables you to reuse database connections across multiple invocations.
AWS Powertools for TypeScript: The AWS Powertools Parameters utility uses the same approach.

Declaration of variable at the runtime level

...
const DEFAULT_PROVIDERS: Record<string, BaseProviderInterface> = {};
...

Application of the singleton Pattern to reuse the variable if it has not already initialized

const getSecret = async (...) => {
  if (!Object.hasOwn(DEFAULT_PROVIDERS, 'secrets')) {
    DEFAULT_PROVIDERS.secrets = new SecretsProvider();
  }

  return (DEFAULT_PROVIDERS.secrets as SecretsProvider).get(name, options);
};

Usage inside the handler

import { getSecret } from '@aws-lambda-powertools/parameters/secrets';

export const handler = async (): Promise<void> => {
  // Retrieve a secret, cached for 1 hour.
  const secret = await getSecret('my-secret', {
    maxAge: 3600,
  });
  console.log(secret);
};

The Invoke Phase

The invoke phase is what happens every time your Lambda function is called(if it did not failed in the Init phase). This runs inside the handler function

The shutdown phase and how to know when your lambda is terminating

When the Lambda environment is about to be terminated, there’s a shutdown phase. If you have registered external or internal extensions, AWS will send a SIGTERM signal, giving you about 300ms (or 500ms for internal extensions) to gracefully shut down.

You can use any extension, in my case for the following example I've decided to use the Cloudwatch Lambda Insights Extensions, this extension allows you to get more data of your metrics and logs of a function.

In your CDK app


/// In your CDK App 
....
    const dummyFunction = new NodejsFunction(this, 'dummyFunction', {
      entry: path.join(__dirname, './src/sendAppleSessionRequest.ts'),
      insightsVersion: LambdaInsightsVersion.VERSION_1_0_333_0,  // Setup of Lambda Insights extension
    });

...

In your Lambda


process.on('SIGTERM', async () => {
  console.log('Terminating Lambda');
  // Close connections
  // Close any open resources
  // Close DB connections
  // Terminate any ongoing processes
});

export const handler = async (
  event: any
) => {
  console.log('Starting lambda');
};

And if you combine it with the power of reuse variables from the runtime you can get something pretty cool.

process.on('SIGTERM', async () => {
  console.log('Terminating Lambda');
  console.log('Closing connection');
});

let isConnectionInitialized: boolean;

const initializeConnection = async () => {
  if (!isConnectionInitialized) {
    console.log('Initializing connection');
    isConnectionInitialized = true;
  } else {
    console.log('Connection already initialized');
  }
};

export const handler = async (event: any) => {
  console.log('Starting Lambda');
  await initializeConnection();
};

Summary

AWS Lambda functions run through three main phases: init, invoke, and shutdown.
By setting up and reusing variables during init, you save time and resources during each invocation.
With a graceful shutdown, you ensure everything wraps up cleanly before the next run.

Retry of Messages Using Amazon EventBridge Pipes, Amazon SQS, And AWS Step Functions

Javier Mendoza — Mon, 16 Jan 2023 19:21:29 +0000

Amazon Eventbridge Pipes was launched on the last re: invent 2022. Amazon EventBridge Pipes allows you to glue different AWS services and optionally filter or enrich the structure of the events passed from consumers to producers.

You can use an SQS queue as a source of the Amazon EventBridge Pipe and an AWS Step Function as a target.

I attached a DLQ to the SQS source, but reading the documentation it was not clear to me when the message was going to be sent to the DLQ, so I decide to give it a try myself.

When the Amazon EventBridge Pipe is created with a State Machine as a target, it is required to set the type of Invocation as FIRE_AND_FORGET or REQUEST_RESPONSE.

Standard workflows

For Standard workflows, the only type of Invocation allowed is the FIRE_AND_FORGET which means that EventBridge doesn’t wait for a response before proceeding. I set up a Standard Workflow with a Lambda which throw an error when it was invoked.

The Standard Workflow was failing, but the Amazon EventBridge Pipe was not aware of the executions and counted each one of the invocations as successful. I polled the messages of the DLQ over various minutes through the AWS Console, but the DLQ did not have any messages.

The workflow was failing

But no Executions errors appeared on the Pipe

The DLQ was empty

Express workflows

For the test with an Express workflow, I set up the workflow as invocation type REQUEST_RESPONSE. The Amazon EventBridge Pipe this time was aware of the FailedExecutions and it did not take too much time until the messages started appearing in the DLQ. This means that the failed messages were able to be re-driven to the original source queue.

The EventBridgePipe showed the Failed Executions for Express workflows

The messages started appearing in the DLQ

Summary

A message will never arrive in the DLQ attached to an SQS if we use SQS as a source of the Amazon EventBridge Pipes and a Standard Workflow as a target.

Hopefully in the future, the EventBridge team will make this possible. But nowadays this does not help in the evolution to an asynchronous world.

DEV Community: Javier Mendoza

Monitoring Third-Party Webhook Delays with AWS Durable Functions

A starters guide from SQS and some learning along the way

What is it?

Why is it important?

DLQ

Typical SQS Consumer

How to reuse variables in your Lambda across invocations and know when it's about to terminate

The Init Phase and Variable Reuse

The Invoke Phase

The shutdown phase and how to know when your lambda is terminating

Summary

Recommended reading

Retry of Messages Using Amazon EventBridge Pipes, Amazon SQS, And AWS Step Functions

Standard workflows

Express workflows

Summary