DEV Community: Ian

Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes

Ian — Wed, 22 Oct 2025 14:09:31 +0000

Ian

Oct 21 '25

Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes

#serverless #aws #eventdriven #appsync

Comments

6 min read

Realtime Event-Driven Applications with AppSync Events and EventBridge Pipes

Ian — Tue, 21 Oct 2025 19:33:49 +0000

I want to share with you a powerful AWS serverless pattern that uses two of my favourite AWS services: AppSync Events and EventBridge Pipes. Using these services we can combine the power of event-driven architecture with realtime user notifications.

This article walks through how to build a demo application called Airspace Alerter, designed to showcase how these two services work together to deliver realtime user updates in a fully event-driven workflow.

Airspace Alerter is a map based web application, but this isn't a blog about geospatial data handling (although I'm passionate about that topic too!). This blog focuses on event management. If you’re interested in geospatial data in DynamoDB, check out my related post: Effective Handling of Geospatial Data in DynamoDB

Introduction

The full source code for this article can be found here:

👉 GitHub Repository – Airspace Alerter - CDK AppSync Events

In the demo, users can:

Submit and store their planned flight routes.
Simulate random airspace hazards (thunderstorms, drone sightings, restrictions).
See realtime alerts whenever a hazard impacts one of their routes.

These alerts are streamed back to the user through an open WebSocket connection using AppSync Events.

The Core Services

At the heart of this architecture are these two core AWS services:

1. EventBridge Pipes for Event Filtering and Enrichment

EventBridge Pipes act as a simple, fully-managed bridge between sources and targets, allowing filtering, transformation, and enrichment of events as they flow through. They are an alternative and more observable way to process events in your application, rather than using an EventBridge Bus with rules.

You can have multiple different sources for the pipes, but one of the most powerful is a DynamoDB stream. Streams is one the greatest features of DynamoDB, allowing for full Change Data Capture (CDC). This can be configured for all new, modified or deleted items to be passed to the pipes. The pipes allow a 'fan out' pattern where you can have multiple pipes, each with a specific filter. You only ever pay for a pipe when the filter matches and the pipe is invoked.

2. AppSync Events for Realtime Event Delivery

AppSync Events are AWS capability that provide realtime WebSocket event delivery. This is not AppSync as you might know it. They come under the same service name, but AppSync GraphQL API (a great GraphQL API service) is completely separate from AppSync Events API (a great JSON based event driven websocket).

AWS has two WebSocket services. API Gateway WebSocket API and AppSync Events. Think about AppSync events as API Gateway WebSocket APIs v2. WebSockets might not always be right for your application (see this great article by Allen Helton), but if they are, I would encourage you to use AppSync Events.

As a side note, I'm not quite sure why AWS chose to put AppSync Events under the AppSync service. It doesn't naturally fit. I understand the need to consolidate services, but I tend to think of AppSync as purely AWS's GraphQL offering.

Architecture Overview

The architecture looks like this:

User submits a flight route via the web application.
The route is stored in DynamoDB.
The user can trigger random hazard events (e.g., a thunderstorm, drone sighting) through API Gateway.
A DynamoDB Stream captures the new hazard record and sends it through an EventBridge Pipe.
The Pipe filters and then enriches the event: Fetching planned routes and checking if the hazard intersects with planned routes. If it does, it create an alert event.
Both alert events and hazard events are passed to the target of the EventBridge pipe. The target is an EventBridge API Destination which is configured to POST to the AppSync Events endpoint.
The connected user receives the hazards & alert in real time and sees it displayed on the map.

EventBridge Pipes handle filtering and enrichment. In our case, whenever a new hazard is inserted into the table, the event is picked up by the Pipe. Before the event is sent downstream, the Pipe performs an enrichment step, fetching flight routes from DynamoDB and determining whether any of them intersect with the hazard area.

If a hazard impacts a route, the event is marked as an alert. Both hazards and alerts are passed on to the target the pipe.

AppSync Events handle delivery to the web application. Clients simply subscribe once and wait for updates. AppSync Events handles some of the hidden complexity such as authentication, but more complex workflows requiring retries, specific authorisation will require additional logic that you build. Having said that, for a simple setup, it very easy to get working and the AWS Console provides some great tools to test and debug. Kudos to the AWS team here.

Building It with CDK

The demo is built entirely with the AWS CDK, which makes defining and connecting these event-driven components simple.

The key constructs include:

Pipe – EventBridge Pipe that defines the source (DynamoDB stream), enrichment (Lambda), and target (An API Destinations that publishes to AppSync Events). This is defined with an Alpha higher level construct @aws-cdk/aws-pipes-alpha. Alpha constructs are subject to breaking changes, so please be aware of this as you build. If you need stability, you can instead use the lower level cfnPipe construct.
EventApi – defines the AppSync Events WebSocket API and connection management. This is a really great, simple construct for deploying an AppSync Event API.
DynamoDB - Using a simple single-table design pattern for both the flight routes and the hazards. We can filter the stream in the EventBridge pipe if we are only interested in only one type of item.
Lambda – to handle route submission, hazard simulation, and enrichment logic.

Here’s an example of the enrichment pipe in CDK:

new Pipe(this, 'HazardEventsPipe', {
  source: new DynamoDBSource(props.airspaceAlerterTable, {
    startingPosition: DynamoDBStartingPosition.LATEST,
    batchSize: 1,
  }),
  target: new ApiDestinationTarget(airspaceAlertsAppsyncDestination),
  enrichment: new LambdaEnrichment(props.hazardsEnrichmentFunction),
  logLevel: LogLevel.TRACE,
  logIncludeExecutionData: [IncludeExecutionData.ALL],
  logDestinations: [cwlLogDestination],
  filter: sourceFilter,
});

This simple construct connects the DynamoDB stream, invokes the enrichment function, and publishes to the AppSync Events endpoint using a configured 'API Destination'. This is how the pipe looks once it is deployed:

For testing, you can log all the pipe actions and the execution data which allows for good observability of the pipe invocations.

The AppSync Events API is deployed with just a few lines:

const apiKeyProvider = { authorizationType: AppSyncAuthorizationType.API_KEY };
const eventsApi = new EventApi(this, 'AirspaceAlerterEventsApi', {
  apiName: 'AirspaceAlertsEvents',
  authorizationConfig: { authProviders: [apiKeyProvider] },
});

// add a channel namespace called `alerts`
eventsApi.addChannelNamespace('alerts');

This gets you up and running with an API key for authentication. This is great for a demo, but in the real world, it also supports IAM, Cognito, OpenID Connect and custom authentication with Lambda.

Publishing Events

In this demo application, we are using the HTTP endpoint to publish events, and this has been integrated into the application using API Destinations, a feature of EventBridge.

In Airspace Alerter, the Pipe target is the API Destinations, with an array of events output from the enrichment phase posted to the endpoint.

However, you could also use a Lambda function as the destination from the pipe, for example, using IAM permissions to invoke the AppSync API rather than managing secrets.

In the current way to create an API Destination with CDK, you must provide a secret from AWS Secrets Manager which contains the API key. API Destinations then creates an additional secret it uses to store a copy of this key and the other header information. If like me, this annoys you ($0.40 a month for each secret), the Lambda destination pattern avoids this for some minor overhead in additional code maintenance.

Getting Started

To get started with the example project, clone the repo and deploy following the steps in the README.

Costs and Considerations

Each component scales independently and costs only when used:

DynamoDB Streams are low-cost and event-based.
EventBridge Pipes are charged per event processed.
AppSync Events have lightweight connection-based pricing.
Lambda only runs when invoked for enrichment or simulation.

The only caveat is the current AppSync Event destination pattern — as noted earlier, it relies on a Secret to store the API key connection.

Next Steps

Future enhancements might include:

Alternative, more secure authentication mechanisms (e.g. IAM, OIDC)
Multi-user subscription models with fine-grained filters
Use of IoT Core as a managed MQTT message broker instead of AppSync Events.

About Me

I’m Ian, an AWS Serverless Specialist, AWS Community Builder, and AWS Certified Cloud Architect based in the UK.

I work as an independent consultant with a passion for aviation, helping teams build cloud-native and event-driven solutions.

Connect with me on LinkedIn or find out more at Crockwell Solutions.

Effective Handling of Geospatial Data in DynamoDB

Ian — Mon, 08 Sep 2025 10:49:14 +0000

Handling Geospatial data in DynamoDB doesn't feel like a natural fit, and can appear complex. With the right approach and some upfront work, you can leverage the superpowers of DynamoDB for your geospatial application.

There are some really good existing patterns for handling geospatial data, in particular the DynamoDB Geo Package. However, there are some limitations I wanted to avoid for my application. In particular, I wanted more control over the structure of my data, and I wanted to be able to return a geographically distributed set of results within a large result set without paginating. (woah, that's a mouthful... I'll explain what I mean further down)

This article walks through how I achieved this using DynamoDB, with an example dataset of weather data from ~5,000 global airports. The approach for structuring and querying the geospatial data includes making use of multiple geohash indexes and some de-clustering.

The end goal:

Real-time queries working at lightning speed at any precision level
Uniform response time, regardless of the bounding box size
Queries not impacted by the total dataset size: No scan operations, only optimised queries.
A random, well-distributed set of results, rather than clustered data from a single corner of the bounding box

Introduction

The code for this article can be found here:

👉 GitHub Repository – DynamoDB Geospatial Example

The project demonstrates how to:

Load real-world weather data into DynamoDB
Structure indexes for multi-level geohash precision
Support both large (worldwide) and small (town-level) bounding box queries
Perform "points of interest along a route" type queries
Return randomised results evenly distributed across space

The Challenge with Geospatial in DynamoDB

At its core, DynamoDB does not provide geospatial queries natively. You cannot simply say: "Give me all points within this bounding box."

We need to break the problem down:

Geohashing: Encode latitude and longitude into a compact, ordered string.
Precision levels: Wider bounding boxes use low-precision geohashes, smaller bounding boxes use higher precision.
Indexes: Leverage DynamoDB Primary Partition and GSIs to query data at the right level of precision.
Partition prefixes: Avoid hot partitions and enable randomised selection.

Table Design

Here’s the design I used that allows for multi-precision level queries:

Partition Key	Sort Key	GSI1 Partition Key	GSI1 Sort Key
PK	SK	GSI1PK	GSI1SK
`<ShardID>#<GeoHash Precision 1>`	`<GeoHash Precision 8>`	`<GeoHash Precision 4>`	`<GeoHash Precision 8>`

And importantly:

The ShardID is a random number (1–10 in the default case) that distributes data across partitions.
We fan out to parallel queries, returning a random spread of results.

The precision levels configurable based on your use case, but under test, I found this structure to work well from very low to very high precision queries.

The ShardID is used on the main partition key, while the GSI1 allows for mid-level precision queries without the shard. This provides two benefits:

We don't create a hot partition when using a very low precision geohash (e.g. world-level queries) on the main partition key.
We are able to make higher precision queries (e.g. city-level) without needing to include the shard, which would require multiple queries to cover all shard values.

Query Types

Depending on the zoom level of the bounding box, we dynamically adjust which keys/indexes are used:

PK only → World-scale queries
PK + SK → Country or region queries
GSI1PK only → City-level queries
GSI1PK + GSI1SK → Town-level queries

This tiered approach ensures we always query at the right precision without over-fetching.

Randomised Result Distribution

Without a partition prefix, DynamoDB queries naturally return clustered results (e.g. all from the "top-left" corner). By introducing a random ShardID prefix, we ensure queries fan out across partitions and return a spread of points across the bounding box.

Sharding alone doesn't completely solve the clustering issue, especially for large bounding boxes with low precision geohashes. Therefore we further randomise results by gridding the bounding box into smaller sections and selecting results evenly from these grids.

⚠️ Before Querying world-scale without gridding → clustered data:

✅ After Querying world-scale with partition prefixes and gridding → evenly spread data:

Performance Tuning with Lambda

Response times were tested response times using the truly excellent Lambda Power Tuning.

At 1024MB, with large (county level) bounding box queries averaged ~124ms. Scaling memory up or down adjusted response time linearly without any unpredictable spikes.

Example: Bounding Box Queries

Here’s a comparison of bounding box sizes, the index used, and the performance on my ~5,000 record dataset:

Level	Bounding Box	Diagonal Distance (Km)	Geohash Precision	Index Used	DDB Queries Made	Avg Response Time (1024MB)
World	(80, -170), (-80, 170)	17,845 km	1	PK	320	880 ms
Continent	(70, -25), (40, 30)	4,568 km	1	PK	40	150 ms
Country	(43, -10), (37, 2)	1,220 km	2	PK + SK	40	124 ms
Region	(50, 8), (47, 13)	497 km	3	PK + SK	150	324 ms
City	(51.6, -0.45), (51.3, 0.2)	56 km	4	GSI1	9	23 ms
Town	(37.041, -7.995), (37.006, -7.901)	9 km	4	GSI1PK + GSI1SK	3	11 ms

Even world-scale queries return results in under a second, with evenly distributed points. There is also much more we could do here to optimise the Lambda function performance.

This shows the approach delivers uniform response times regardless of bounding box size.

Getting Started

To get started with the example project:

Clone the repo
Deploy the CDK stack (see README)
Load the sample dataset (5,000 airports with METAR weather data)
Load up the frontend and try different bounding box queries

You’ll see that whether querying the whole world or a small town, response times remain fast and results are evenly distributed.

Configuration

The hashes and indexes used as configured within the environment configuration: config/environment-config.ts and queries configuration are configured within the config/geospatial-config.ts which includes the indexes and precision that should be used based on the size of the bounding box.

Architecture

The demo project consists of two main components:

Frontend: React-based web application served via CloudFront. Deployed with AWS CDK.
Backend: Serverless API built with AWS Lambda and API Gateway, using DynamoDB for data storage and event driven architecture for real-time updates. Deployed with AWS CDK.

AWS services consist of:

Amazon DynamoDB for geospatial data store
AWS Lambda for serverless compute
AWS API Gateway for REST API management
Amazon S3 for static hosting of the frontend
Amazon CloudFront for content delivery

API

The API also includes a route endpoint that uses similar logic to the bounding box query and will return points of interest along a route. The payload response will including the total distance of the route:

Next Steps

This approach provides a solid foundation for handling geospatial data in DynamoDB:

Scalable to millions of records
Uniform response times
Randomised and well-distributed query results

Future work could include:

Additional GSIs for intermediate precision levels
Support for more complex geospatial queries (e.g. radius, nearest neighbor)
Improved efficiency of the de-clustering. This is currently a very basic approach, and could be improved with more advanced algorithms.

About Me

I’m Ian, an AWS Serverless Specialist, AWS Community Builder and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.

Let's connect on LinkedIn, or find out more about my work at Crockwell Solutions.

Finally got some time to play with the new JSONata and Variables support for Step Functions, and I have to say, it is massive improvement. Check out my latest blog post, where I walk through a simple example of how easy it is to handle pagination now

Ian — Mon, 16 Dec 2024 19:07:55 +0000

Handling Paginated Results Seamlessly with AWS Step Functions

Ian ・ Dec 16

#aws #serverless #stepfunctions #jsonata

Handling Paginated Results Seamlessly with AWS Step Functions

Ian — Mon, 16 Dec 2024 18:09:08 +0000

...using Step Function Variables and JSONata

Paginated results exist everywhere. Whether you are fetching data from an API, or iterating through database queries. It has been traditionally very difficult to deal with paginated results natively within your AWS Step Function workflows. The primary reason is that there was no out-of-the-box way to keep the state of the results and then append to it. That all changed when AWS announced support for Step Function variables and JSONata. This made many people (myself included) very happy.

Introduction

The code for this article can be found here:

https://github.com/Crockwell-Solutions/cdk-stepfunctions-iterator

This CDK / TypeScript project gets you up and running with all the resources you need to iterate through some example records in DynamoDb using Step Functions. Also included is a seeding function to populate your DynamoDb table with 100 sample records to iterate through.

It is worth noting that at the time of writing this blog, the CDK does not support JSONata based Step Functions using the native fromChainable method. JSONata is supported using CDK when using ASL and the fromFile method.

Step Function Variables and JSONata

No more "Wait, where did my data go" ResultPath or hours spent manipulating data in the Data Flow Simulator to conclude it is just not possible to add data to the top level of the JSON object with OutputPath. The support for JSONata has completely eliminated the need. It is just Arguments (input), Output and optionally Variables (see below). JSONata is a query and transformation language for JSON data. More information is available at the official guide and a cool simulator.

When a string in the value of an Step Function field, a JSON object field, or a JSON array element is surrounded by {% %} characters, that string will be evaluated as JSONata. It is that simple. JSONata supports basic operations such as selecting attributes and complex operations such as sorting and grouping.

Step Function variables has complemented the support of JSONata by allowing assignment of variables for the Step Function rather than just for a task. This allows you to fetch and manipulate Step Function Variables through tasks.

We will use both of these new bits of functionality within our Step Function Iterator.

The Architecture

We are starting with a simple Step Function that will query DynamoDb. If there is a nextToken provided in the response, it will continue to query and append to the Step Function Variable $results. Before Step Functions Variables, this simple workflow was not possible like this.

Although we are focusing on DynamoDb as an example, this pattern equally applies to other paginated systems such as APIs that have a limit and page option.

The key elements of the Step Function are the instantiation of the Step Function Variable in the Query DynamoDb Table task:

"Assign": {
  "results": "{% $states.result.Items %}"
}

$states.result is the output from the task, used within an JSONata expression indicated but the {% %} encapsulation. And then we can now append to this array when we fetch the next page of results like this:

"Assign": {
  "results": "{% $append($results, $states.result.Items) %}"
}

Hooray to JSONata support 🎉

Getting Started

To get started with the example project, clone the repo and deploy following the steps in the README. When you invoke the SeedDynamoDbFunction, it will create 100 records in the DynamoDb table that we can iterate through.

If we wanted to return a query a set of results where the PK is ItemGroup, this could lead to paginated results (depending on the total size of the response).

You can see from the output that the Step Function Variable results now contains an array with the 100 DynamoDb results. The results were fetched across 4 queries, where the limit of 30 was applied to each query. Winner!

Next Steps

This simple pattern can be used to iterate results from alternative data sources. Prior to Step Function Variables and JSONata support, there was either the need for very complex task management, or more likely, you simply invoked a Lambda function to handle the logic. Now with native support, this is an effective way to get a complete result set in your Step Function workflow.

About Me

I’m Ian, an AWS Serverless Specialist and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.

Let's connect on linkedIn, or find out more about my work at Crockwell Solutions

Query your EventBridge Scheduled Events in DynamoDB

Ian — Sun, 24 Nov 2024 11:22:35 +0000

Querying Your EventBridge Scheduled Events

EventBridge Scheduled Events are a fantastic addition to the AWS event driven ecosystem. However, they do have an issue... observability. If you’re looking for a reliable way to track these events without the issues of the EventBridge API, I’ve got a practical solution for you.

Introduction

The code for this article can be found here:

https://github.com/Crockwell-Solutions/cdk-eventbridge-scheduler

This CDK / Typescript project gets you up and running with all the resources you need to observe your schedules in DynamoDB. Also included is a seeding function to demonstrate the end-to-end workflow.

What Are EventBridge Scheduled Events?

Amazon EventBridge offers two types of schedules:

Recurring (cron-like): These have been around for a while, letting you define periodic event triggers. Previously these were part of CloudWatch. AWS rightly moved this all under the EventBridge services.
One-off schedules: Introduced in 2022, these fire once and optionally delete themselves. They were the missing piece in event-driven systems.

Why One-off Schedules?

One-off schedules are perfect for situations requiring time-sensitive actions:

User notifications: Trigger reminders at specific times based on user activity.
Time-bound processes: Schedule tasks like price changes or temporary discounts to occur at a predefined time.
Temporary API Keys: Generate API keys with a time-bound expiration, using a one-off schedule to trigger their deactivation at the exact expiration time.

For the purposes of this blog, I'm specifically referring to these one-off schedules.

The Problem with Tracking Scheduled Events

EventBridge Scheduler allows you to manage thousands of future events, but it is not easy to get an overview of these events: To know what is planned to fire, or even what has fired in the past. There are some options.

ListSchedules API Limitation: The ListSchedules API returns a maximum of 100 schedules per call. If you’re dealing with millions of events, this becomes impractical. There is also no out-of-the-box way to query these by firing time. You could incorporate the firing time in the schedule name allowing partial filtering using NamePrefix. However, renaming schedules isn’t possible, so updating firing times means deleting and recreating schedules, adding unnecessary complexity for observability. Not ideal.
Custom Solutions: You could just write back event metadata to the event origin. For example, you might want to schedule events for your customers based on their last activity in your platform. Say, schedule an event for 24 hours after they added an item to their basket to send them a reminder. You could just write back information about the scheduled event to the customer record, but what if you don’t want to keep updating customer records. You want to manage this schedule independently?

A completely different approach could be to DynamoDB TTL with Streams instead of EventBridge Scheduled events. You can configure your DynamoDB items with a Time-To-Live (TTL) attribute as the event firing time, allowing you to:

Store schedule metadata in DynamoDB.
Trigger downstream workflows using DynamoDB Streams when TTLs expire.

However, TTL expiration isn’t precise enough for some strict timing requirements. AWS documentation states:

“Items with valid, expired TTL attributes may be deleted by the system at any time, typically within a few days of their expiration.”

In my experience, it performs much better than that, but the point being, you can’t rely on the firing time.

A Robust Solution: CloudTrail and DynamoDB

For a scalable and reliable way to track schedules, we can combine CloudTrail and DynamoDB:

Monitor Scheduler Events with CloudTrail

Enable CloudTrail logging for your AWS environment. It will capture events such as:
- Schedule creation
- Updates (e.g. firing time changes)
- Deletion
Capture Events in a DynamoDB Table

Use EventBridge Rules to filter CloudTrail logs and write relevant events into a DynamoDB table.

Benefits of This Approach

Comprehensive Tracking: Easily query schedules and their firing times, even for millions of events.
Historical Data: Maintain a record of schedules, even after they’ve fired and have deleted themselves in EventBridge.
Resilience: With some further work, you could also store the payloads of the EventBridge Scheduled Events which could allow you to build a disaster recovery system in the event that AWS were to lose your Schedules!

The Architecture

At the core of this architecture is CloudTrail. We leverage monitoring of the EventBridge Schedule API calls:

Independently to the core logic of the schedule creation, our service does the following steps:

CloudTrail picks up the EventBridge API events and they are published on the main Event Bus
An EventBridge rule filters for these events and pushes them to an SQS queue
The queue will batch the messages and send them to the ScheduleMonitorFunction. This Lambda will process the messages and create/update relevant records in DynamoDb

A queue is used during Step 2 because CloudTrail will not guarantee the order of events that are delivered. We need to ensure that changes are processed in order, so SQS will re-batch the messages, which are then sorted by the eventTime as they are processed in the ScheduleMonitorFunction Lambda Function.

To see how this works, deploy the demo repo and run the SeedSchedulesFunction. This creates three scheduled events, waits a few seconds then modifies one of them and deleted one of them. This demonstrate the functionality of the solution.

aws lambda invoke --function-name SeedSchedulesFunction outfile.txt

Event metadata is stored in DynamoDB using the following structure:

Primary Key: PK = SCHEDULEGROUP#SCHEDULENAME
Global Secondary Index (GSI): GSI1 = groupName, fireTime

This design lets you query events by group and firing time, enabling efficient lookups for upcoming schedules within specific time windows:

Items are stored with a Time to Live (TTL) so they will expire and be removed from DynamoDB 30 days after the firing time.

Limitations and Considerations

Fired Events: This solution won’t explicitly track fired events, but since AWS guarantees “at least once delivery,” it’s safe to assume fired schedules have actually been fired.
Storage Costs: DynamoDB storage and CloudTrail logs will incur additional costs, so optimise your queries and retention policies.

Next Steps

This pattern is extensible for further uses. Options that could be considered include:

Analytics: Push data to Redshift, S3, or Athena for advanced querying and analytics.
Payload Recovery: Store payloads of scheduled events for disaster recovery or debugging.

About Me

I’m Ian, an AWS Serverless Specialist and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.

Let's connect on linkedIn, or find out more about my work at Crockwell Solutions

Effortless Debugging: AWS CDK TypeScript Projects in VSCode

Ian — Sun, 27 Oct 2024 23:29:15 +0000

Debugging Lambda functions should be easy, right? You want to debug as you develop, using a standard editor (VSCode), a popular IaC framework (AWS CDK), and a widely adopted language (TypeScript).

But it is never quite that easy.

One of the main challenges to wider serverless adoption is the developer experience itself. Setting up an effective debugging environment for Lambda functions can be challenging, especially with the variety of tools, frameworks, and configurations available.

If you're using the same stack as I am, this guide will help you set up a smooth, flexible debugging environment so you can iterate through Lambda functions like a pro.

Introduction

The code for this article can be found here:

https://github.com/Crockwell-Solutions/cdk-lambda-debugger

The starter project provides a CDK configuration with an example resource stack (API Gateway -> Lambda -> SQS). The Lambda function is ready for debugging using both AWS SAM and Lambda Live Debugger. With this setup, you can iterate quickly through Lambda changes without any deployment delays.

Debugging Goals

When I built this, I wanted to simplify Lambda function debugging for a better developer experience. My main requirements were:

Quick debugging (<5 seconds): Debugging has got to be quick. I don’t want to run cdk synth every time I debug: Within a large project with multiple stacks and functions, this takes too long.
No deployment or hot-swap requirement: I want to debug without deploying each time.
Support for ephemeral environment: Within a team, I need the setup to support multiple developer environments. I want it to work with SSO and I don't want to commit local environment variables.
Local debugging with remote resources: I don't want to emulate resources locally, just the Lambda runtime.

This makes things a little trickier. There are plenty of options out there, but I really just needed a starter project with everything configured. I couldn't find one, so I've made one.

How it works

I want the most flexible developer experience, so the sample project uses two different debugging options.

Run function locally, debug locally: Using AWS SAM
Run function remotely, debug locally: Using Lambda Live Debugger

There is value in having both options available within a project as the SAM setup provides a bit more flexibility during early development for payloads and environment variables. Lambda Live Debugger provides a more representative runtime environment once you have your Lambda function is slightly more mature.

Getting Started

To get started with the sample project, clone the sample project and follow the installation instruction in the README.

After deployment, configure the required environment variables in a local.env.json file. Here’s an example of what that file might look like:

{
  "SQS_QUEUE_URL": "https://sqs.region.amazonaws.com/123456789012/LambdaDebuggerQueue"
}

This file isn’t committed to version control, so each developer can set their own local environment variables. The local.env.json file is referenced from launch.json in VSCode using the SAM CLI argument --container-env-vars, which does then allow for committing and sharing of the launch.json.

{
  "type": "aws-sam",
  "request": "direct-invoke",
  "name": "Lambda Function - SAM Debugger",
  "preLaunchTask": "Build",
  "invokeTarget": {
    "target": "code",
    "architecture": "arm64",
    "projectRoot": "dist",
    "lambdaHandler": "lambda-debugging-function.handler"
  },
  "aws": {
    "region": "region"
  },
  "lambda": {
    "runtime": "nodejs20.x",
    "payload": {
      "path": "${workspaceFolder}/test/test-payload.json"
    }
  },
  "sam": {
    "localArguments": [
      "--container-env-vars",
      "${workspaceFolder}/local.env.json"
    ]
  }
}

The project includes a reusable CustomLambda construct based on the CDK NodejsFunction, bundling with ESBuild and including Lambda Powertools. This bundling is handled as a preLaunchTask in launch.json, avoiding the need to perform a cdk synth. This saves crucial seconds when initiating the debug run.

AWS SAM Debugging

This uses AWS SAM to invoke the Lambda function locally. Select Lambda Function - SAM Debugger from the Run and Debug menu in VSCode and hit F5. You should see the logging output.

The debugger launch configuration triggers a build phase to bundle the code with ESBuild. Bundled code is saved to the dist folder and run locally, removing the need to synth the CDK project. You can set breakpoints in src/lambda-debugging-function.ts, and the debugger will respect source maps to link to the original code.

Lambda Live Debugging

Lambda Live Debugger is an alternative debugging option and offers the added benefit of validating permissions set up in CDK by running the Lambda functions remotely. This uses the Lambda Live Debugger project led by the industry hero Marko at Serverless Life.

Deploy Lambda Live Debugger resources to your AWS account with npx lld.
Set any custom debugger options in lldebugger.config.ts.
In VSCode, select Lambda Function - Lambda Live Debugger from the Run and Debug menu, then press F5.

Lambda Live Debugger will work its magic and deploy out the required resources to your account. Now, to initiate the debug environment, you need to invoke the remote Lambda function. For example, you can add your breakpoint to your local code in src/lambda-debugging-function.ts and then invoke the Lambda function through the API with a call such as:

curl --request POST --header "Content-Type: application/json" --data '{"message": "This is a test message"}' https://xxxxxxxx.execute-api.eu-west-1.amazonaws.com/api/data

In case of code changes, Lambda Live Debugger automatically reloads the updated code without the need to redeploy or restart the debugger. Magic!

When you’re done, remove the debugger resources with npx lld -r.

Wrapping Up

This is still an evolving area, new tools and best practice changes all the time.

I initially used Serverless Framework (v3) for debugging, but with its deprecation and the new signup requirement in v4, I’ve moved to the setup in this article. Running Lambda functions locally always risks slight discrepancies with deployed environments (especially in terms of permissions), but this approach with AWS SAM and Lambda Live Debugger balances speed with reliability.

About Me

I’m Ian, an AWS Serverless Specialist and AWS Certified Cloud Architect based in the UK. I work as an independent consultant, having worked across multiple sectors, with a passion for Aviation.

Let's connect on linkedIn, or find out more about my work at Crockwell Solutions