With the rise of serverless infrastructure tools like AWS CDK, AWS SAM or Serverless Framework more and more developers have immense power at their hands. With the robust abstraction layer over the AWS CloudFormation at our disposal, we can prototype, develop and ship serverless apps faster.
Unfortunately, in using those abstractions, the knowledge of how the underlying service operates might be forgotten or never acquired in the first place. While I believe this precedent is not harmful in a grand scheme of things, on some occasions, a deep understanding of how pieces fit together can help us built more robust and reliable services.
Today, let us focus on one such topic that might get lost as it's most likely entirely abstracted away by the framework you are using - the Event Source Mapping.
What is the Event Source Mapping
Every time I learn something new about an AWS service, especially a service that I used for a long time, I'm met with the realization that, in reality, I know very little about it.
For a long time, while working with AWS Lambda and AWS SQS, integrating the services in various ways, I never thought about what makes those two work together - I did not have to.
My perspective shifted drastically after building (and augmenting) the popular circuit breaker pattern.
Here is how my version of the pattern looked like:
To make it all work, I had to dynamically disable the integration between Amazon SQS and AWS Lambda to make my architecture work. There is no point in invoking the Lambda function if the 3rd party endpoint is unavailable.
After a bit of searching, I stumbled upon this AWS CLI command and this AWS documentation page.
Reading the documentation, my mental model on how AWS Lambda integrates with Amazon SQS greatly improved. In the context of AWS Lambda, the Event Source Mapping refers to an AWS Lambda resource that reads from an event source and invokes a Lambda function.
Suddenly, what Anahit and Jeremy were talking about in episode #92 of Serverless Chats made sense, how tracking the iterator when reading from Amazon DynamoDB and Amazon Kinesis streams works made sense. You get the idea.
The Event Source Mapping is the gluing piece for many integrations. If you are using DynamoDB streams with Lambda, the Event Source Mapping is involved. The same rule applies for Kinesis integration with Lambda. Check out the documentation for the complete list of services and integrations.
Back to the Circuit Breaker pattern
With my newfound knowledge, here are the adjustments I made to my architecture.
The first change(1) is the addition of a Lambda function, which responsibility is to toggle the Event Source Mapping depending on the state of the state machine.
The second(2) is probing the 3rd party endpoint to check if it's available. This is done by invoking the "main" Lambda function directly.
Depending on the response I get back, I will either close the circuit or wait a bit and try again.
Looking back, it would even be possible to skip the "invoker" Lambda function altogether by using a direct integration between AWS Step Functions and AWS Lambda.
While the architecture is not perfect, and I do not feel comfortable recommending it for production usage, I believe it's a viable alternative to what Jeremy proposed. Maybe it could be used as a starting point for your architecture?
Summary
AWS landscape is vast. I sometimes find it hard to keep track of all the changes to the AWS services relevant to my work. That is why I'm a big advocate for "just in time learning" if possible and diving deep from time to time.
I hope that this article was just that, a nudge to dive a bit deeper into the architecture of your serverless app. You never know what kind of new service or resource you might discover.
If you are hungry for more, I post some of my serverless discoveries on Twitter - @wm_matuszewski.
Thank you for your time.
Latest comments (0)