DEV Community: Matt Morgan

Create Stateful Serverless Workflows with AWS Step Functions and JSONata

Matt Morgan — Mon, 31 Mar 2025 13:44:34 +0000

As an avid user of AWS Step Functions, I've been pleased by several excellent releases over the past few years, including Distributed Map, Express Workflows, Intrinsic functions, TestState, redrive, service integrations, and so many others. Those are all fantastic releases, but in my humble opinion, none of them are as big of a deal as the introduction of JSONata expressions. AWS announced this game-changing feature a week before re:Invent 2024.

JSONata

JSONata is a language for querying and transforming JSON data developed by Andrew Coleman of IBM. I had not heard of or worked with JSONata before its inclusion in Step Functions, but it seems like a perfect fit.

I haven't used it yet outside of Step Functions, but JSONata seems quite remarkable in its own right, and I can imagine using it in other contexts. Check out the docs at https://docs.jsonata.org/overview.

JSONata in Step Functions

JSONata simplifies how data is passed from step to step over the existing JSONPath approach. This is a very good thing because the developer experience of Step Functions hasn't always been the greatest. Even frequent users of this service get lost in the flow of InputPath, Parameters, Task Result, ResultSelector, ResultPath, and OutputPath. Wait, which one do I use to add a value to the state input? AWS released the Data flow simulator to help navigate this flow, but in some ways, all that did was drive home the point that this can be complex and hard to use.

Passing Data

When we use JSONata with Step Functions, we do away with the above flow and instead work with Parameters, Output, and one new directive, Assign. Because JSONata lets us create complex expressions, we can use Output to achieve what used to require ResultSelector, ResultPath, and OutputPath, a significant simplification.

The Assign Directive

As good as that is, Assign pushes JSONata support to the top of an impressive feature set. Assign lets us create scoped variables that can be read in later steps! If you've ever had to pass a value from step 2 to step 7 to be read, you're probably out of your seat and cheering. Note that these variables will be scoped when you use Assign in steps within a Map or Parallel state. The Map or Parallel state can use Assign to make values available later in the state machine execution.

Comparing JSONPath with JSONata

Let's look at a simple example. I refactored the state machine I used in this 2019 blogpost. It demonstrates a simple workflow that moves through different states, each implemented by a Lambda Function.

Five years ago, we had no choice but to pass data along from state to state. We could refer back to the original input using $$.Execution.Input, but anything derived by a step must be passed from step to step (or stored in a database). So while my simple state machine example works fine, it introduces an opportunity for weird bugs. I'm passing the CaseId from step to step. It should be understood I'm working on the same case all the way through. What if I pass "001" to the "Assign Case" step and then return a case number of "ABC123"? My step definitions may create a "garbage-in, garbage-out" machine.

{
  "Next": "Assign Case",
  "Type": "Task",
  "OutputPath": "$.Payload",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Parameters": {
    "FunctionName": "arn:aws:lambda:us-east-1:123456890:function:assign-case",
    "Payload.$": "$"
  }
}

{
  "Next": "Work On Case",
  "Type": "Task",
  "OutputPath": "$.Payload",
  "Resource": "arn:aws:states:::lambda:invoke",
  "Parameters": {
    "FunctionName": "arn:aws:lambda:us-east-1:123456890:function:assign-case",
    "Payload.$": "$"
  }
}

It's easy enough to convert these to JSONata steps and start seeing the value of Assign. We set QueryLanguage to JSONata and now we can use the Assign directive.

{
  "QueryLanguage": "JSONata",
  "Next": "Assign Case",
  "Type": "Task",
  "Arguments": {
    "FunctionName": "arn:aws:lambda:us-east-1:1234567890:function:open-case",
    "Payload": "{% $states.input %}"
  },
  "Assign": {
    "caseId": "{% $states.input.inputCaseID %}",
    "message": "{% $states.result.Payload.message %}",
    "status": "{% $states.result.Payload.status %}"
  },
  "Resource": "arn:aws:states:::lambda:invoke"
}

{
  "QueryLanguage": "JSONata",
  "Next": "Work On Case",
  "Type": "Task",
  "Arguments": {
    "FunctionName": "arn:aws:lambda:us-east-1:1234567890:function:assign-case",
    "Payload": {
      "caseId": "{% $caseId %}",
      "message": "{% $message %}",
      "status": "{% $status %}"
    }
  },
  "Assign": {
    "message": "{% $states.result.Payload.message %}",
    "status": "{% $states.result.Payload.status %}"
  },
  "Resource": "arn:aws:states:::lambda:invoke"
}

The Assign Case step now sets the message and status properties, but the caseId remains constant throughout the state machine execution.

JSONata in the AWS Console

Still not convinced? The console experience should close the deal for you. We get a new tab for every JSONata step: "Variables" that shows us any variables the step used and any it assigned.

But that's not all. We also get to view the history of each variable and the assignments made during the state machine execution.

This is impressively useful for debugging complex workflows. I can't think of another tool that records the state of each variable and lets me view it after execution. Just think of how many console.log statements this will save you!

Migrating to JSONata

Making the switch is relatively easy, although more complex workflows require more effort. I have had success doing this by hand. I spent a few hours converting a very complex and hard-to-debug workflow to use JSONata and immediately reaped the benefits of greater observability and maintainability. Although that isn't code I can share, readers might be interested in the diff that upgraded my sample project.

But we don't need to do it by hand! The console now includes a tool we can use to help with the migration.

You get completion right in the tool.

You can also build out powerful Assign expressions here.

It's possible to mix JSONPath and JSONata expressions in the same state machine execution. I wouldn't intentionally do this, but it's useful to be able to refactor one state at a time and test to ensure the workflow is still functional.

What Else Does JSONata Offer?

I feel like we've barely scratched the surface. Predicate queries look very powerful, and now we can do this in an Assign step instead of writing an ES6 reduce callback. JSONata also features an impressive list of built-in functions.

One Last Thing

Think this is cool, but not fully sold on Step Functions? Let me convince you.

Presenting AWS Speakers Directory, an AI Hackathon Project

Matt Morgan — Thu, 29 Jun 2023 11:37:19 +0000

Community Speakers Directory

Hackathons are an interesting phenomenon. We work hard for pay all day and then we continue working hard for free in the evenings and weekends instead of watching TV, hanging with friends and family, pursuing other hobbies and crafts, and sometimes in lieu of sleeping! Why? We each have our own motiviations, but for me, the temptation of collaborating with brilliant folks like Danielle, Johannes, and Julian is too powerful to ignore. A secondary, yet still compelling, motivation is to get hands-on experience with technologies heretofore missing from my toolbox. A distant third is whatever prizes may be offered.

Johannes picked the venue and organized the team like a seasoned professional planning a heist. The job was to create an AI-powered tool using the Transformers Agent framework from Hugging Face. This was a daunting task as none of the four of us have a great deal of experience with AI or ML models, but we did have a pretty good concept of what we could build, courtesy of Johannes. We would build a directory of speakers that can present at community events and use AI to enhance the content and provide a recommendation model to better match talks to events.

We further challenged ourselves by agreeing to:

Leverage CodeCatalyst as a code repository, collaboration tool, and CI/CD pipeline.
Build our website using Flutter.
Build out the backend using AppSync's new Merged API capability.
Deploy our application to sandbox, test, and production environments in a multi-account AWS organization managed by Control Tower.
Write our infrastructure-as-code using AWS CDK in a multi-stack application.
Secure our application using Amazon Cognito and the Amplify SDK.
A single table DynamoDB design.

We each brought expertise to the project that was indespensible to its success. Of course Johannes had the plan and brought the group together. He also organized meetings and make sure we were on task without pushing too hard (hey, it's just a hackathon). But he also wrote a lot of the frontend code and a large part of the AppSync resolvers. Amazing!

Danielle focused on how we can leverage Transformers Agent in Python Lambda functions. She even recorded a video explaining some of the fundamentals so that Johannes and I were able to go from zero to contributing in no time flat. Safe to say we were at risk of not meeting the requirements of the challenge without Danielle's contribution.

Julian designed our overall CDK architecture, implemented the Cognito flows, did a lot of additional Flutter work and also built out the Merge API with L1 constructs, a feat that deserves a blogpost of its own.

My contribution was to provide the AWS Organization, roles that hopefully didn't get in the way too much, and a lot of CDK optimization and fine-tuning. I designed our DynamoDB data model, wrote a bunch of the queries, and got to do my part on the AI bits.

It wasn't long into our project that Johannes and Danielle were honored as AWS Heroes. The hackathon organizers were kind enough to not kick them out of this Community Builders-organized event. Big congrats to Danielle and Johannes!

What follows is a deeper dive into some of the challenges we faced along the way and how we overcame them.

What We Built

Our ambition is to build the AWS User Group Speaker Directory, making it easy to find speakers for remote or in-person events for AWS User Groups. This tool is powered by Transformers Agent to enhance talk profiles, normalize tagging, and make matching speakers and talks to events simple. The Speakers Directory will reduce the burden of organizing events and raise up more voices in the community.

Our application consists of a web layer, a GraphQL API powered by AppSync and a DynamoDB table. This part of the application handles user flows, authentication and authorization with Cognito, and storage of event and talk data. Web assets (our Flutter application) are stored in S3 and served over CloudFront. We have a custom domain managed by Route 53. All of this is composed with the AWS Cloud Development Kit.

CodeCatalyst

Johannes has been a big booster of CodeCatalyst, so it was no surprise he wanted to use it for this project. One cool thing I learned pretty early on is that it supports GitHub Actions out of the box. You can just use them. This is a pretty great feature. It was also nice how easy it was to set up with a deployment role and the tool automatically generated an appropriate role for using tools like CDK or SAM and asked if I wanted to use it. As far as I'm aware, this is the first time AWS has gone out of their way to generate a role that could be seen as suitable for such a CI/CD pipeline. That alone is worth the price of admission as I'm sure many long hours have been spent trying to design the ideal role and many pipelines wind up with permissions too broad!

Other than that, the user experience of CodeCatalyst still leaves something to be desired. I can comment on a pull request, but the author of the PR cannot reply to my comment. Navigating between pipelines, issues and repositories feels a bit sluggish and it's just too many clicks to open a pull request. I do like the idea of an integrated tool that plays well with AWS. I can imagine getting a lot of value from some deeper integrations, such as EventBridge rules firing on a PR. I don't know if that's on the team's roadmap, but it probably should be!

Flutter

I think Johannes is also the reason we picked Flutter for our web tier and Julian also seemed to have prior experience. I've known of Dart for years, but never used it. We had some rough going in the early days as we tried to shoehorn the Flutter build into a CDK asset bundle using Docker. The dang thing went out of memory and drove me crazy for several days before I gave up and went for a build outside of Docker. In the end, it wasn't a Flutter problem, but a Docker problem and some of the libs not working with my M1 Mac. The next challenge was getting CDK to correctly cache builds so it wouldn't make us wait out a Flutter build every time we wanted to deploy the application. It took some doing, but ultimately cleaning up the prior build artifacts just before starting a new build proved to be the ticket and we got asset hashing working well.

Beyond that, Flutter's component-based system let Julian and Johannes build quickly, the Amplify support seemed good and we have the possibility of targeting other platforms.

AppSync / Merged API

AppSync is a technology I haven't used much, but the rest of the team seemed familiar. We implemented JavaScript resolvers (transpiled from TypeScript). It's a bit funny to me how like and unlike Lambda this approach is, but overall it worked pretty well. Using TypeScript let us have more reusable code than we could've with plain js and no build process since (unlike Lambda) JS resolvers must be a single .js or .mjs file.

As a bit of a GraphQL noob, I found JS resolvers to be very easy to use and are a good mental model for working with this kind of technology.

AWS Organization

I had set up a Community Builders AWS Organization recently using superwerker. My intent is for this organization to show the best practices for setting up a multi-account organization with Organizational Units representing sandbox, test and production environments, scoped developer roles and limited production access. I think the developer roles need a little work, but the group was able to be productive in this environment, so that's a win.

Image Generation

We leverage Transformers Agent to generate an image to accompany a talk. This was a bit tricky to get going because the example I was working from uses gradio_tools which seems like a great library for LLM agents using Python, however it requires Python 3.8 or greater and the Docker image we were using was last built a year ago on Ubuntu 18 which uses Python 3.6. This is a very frustrating class of problem to have and I spent about a day trying to find a substitute image to work with or (ugh) upgrade Python in our base image. Ultimately I rebuilt the same image using Ubuntu 22 and Python 3.9 and that worked out for us, but it was a bit of a journey to get there.

Once we had an environment that allowed gradio_tools, the next challenge was that our image had ballooned to 5 GB with all the necessary dependencies, greatly slowly down development and deployment. And what was worse was our Lambda function would go out of memory before completing. We discussed moving the function to Fargate, a detour we weren't looking forward to. Johannes noted that Transformers Agent supports remote execution and I was able to get that working. This makes Lambda an ideal runtime, which definitely matches with our sensibilities!

We used this capability to generate an image based on the talk abstract after a new talk is created on the site. Because the image generation can take about a minute, we leveraged DynamoDB Streams to trigger an asynchronous workflow to generate the image and add it to a talk.

AI-assisted Tagging and Recommendations

We found another, perhaps a more practical, use of Transformers Agent, the ability to pipeline a model like BART which can then be used to score labels as to whether or not they may apply to a text sequence. We reasoned this could be used to help classify our talks by assigning tags that fit them. The tags are then fed into a recommendation engine that will help determine whether or not a talk is appropraite for a specific event.

Our application allows both talks and events to receive tags when they are created. This establishes a baseline of tags. We might see values like "serverless", "devops", and "SAM". Given the baseline of common tags, we can feed them to the classifier and score them. Tags with high scores will be added to the talk or event. Then, if an event organizer wants to get a list of possible talks for the event, we have a real basis for comparison. Talks with like tags are fetched and ranked by number of matched tags.

Using pipeline made it fairly easy to get tag recommendations.

oracle = pipeline(model="facebook/bart-large-mnli")

def handler(event, context):
    tags = get_tags()

    candidate_labels = [t["tagName"]["S"].lower() for t in tags]
    image = event["Records"][0]["dynamodb"]["NewImage"]

    # Use both the title and the description of the talk to identify tags
    identifiedTags = oracle(
        f'{image["title"]["S"].lower()} {image["description"]["S"].lower()}',
        candidate_labels=candidate_labels,
    )

    labels = identifiedTags["labels"]
    scores = identifiedTags["scores"]

    selectedTags = [t for index, t in enumerate(labels) if scores[index] > 0.3]

    save_tags(image, selectedTags)

    response = {"statusCode": 200, "body": identifiedTags}
    return response

This code responds to a DynamoDB Stream, gets a list of tags active in the system, then takes the title and description of the newly-added talk and scores the tags against the text we fed it. Based on our (limited) testing, a score of 0.3 or higher seemed to work out pretty well and we'd then save any of those tags to the talk.

The logic behind recommendations is fairly simple. Based on the tags assigned to an event, an organizer can get a list of recommended talks. We use the event tags to look for talks with similar tags and score them based on the number of matches.

DynamoDB Single Table Design

We decided to use the recommended approach of a single-table design for our database. This is something I have prior experience with, but I feel I've yet to master. However there is a process to follow! To begin with, it's best to craft an ERD so that we can understand what our entities are and how they relate to one another.

From there, we should document expected access patterns and devise keys based on those.

It's inevitable that our expectations won't match reality 100% and when that happens, it's important to return to fundamentals and rework the design. I'd say that our keys changed pretty dramatically a handful of times over the course of the project as new access patterns emerged.

One thing that's challenging is when we realize we need to get all of a kind of thing. How should we do that? A scan with a filter condition? Only if absolutely necessary!

Looking at how we modeled Tags, they are part of an overloaded partition key so that Tags are separate entities assigned to our main domain objects of Talks and Events. Querying a Talk entity with its partition key only will also return its tag entities. Then there's a Global Secondary Index that lets us query by tag and get relevant Talks and Events.

This approach works well when we want to get a list of Talks that have the serverless tag associated with them. But when it came to implementing our recommendation engine, we wanted to be able to provide the classifier with a unique list of existing tags. The access pattern doesn't support this! In fact, the only way we could've implemented this with the current design is to do a full table scan and then filter down to a unique list of tags. I know it's a hackathon, but we just couldn't be satisfied with an approach like that.

So we introduced a new entity with a partition key of just tag. A paginated query on that key will return all the tags. We implemented a Dynamo Stream such that whenever we store a new talk, any tags that were added to it will be evaluated and added to our tags entity. That code, written in TypeScript, uses several great features of the DynamoDB API.

const command = new UpdateCommand({
  ExpressionAttributeNames: {
    '#quantity': 'quantity', '#tagName': 'tagName', '#_et': '_et', '#_ct': '_ct', '#_md': '_md',
  },
  ExpressionAttributeValues: {
    ':one': 1,
    ':tagName': tagName,
    ':timestamp': new Date().toISOString(),
    ':zero': 0,
    ':_et': 'TAG_COUNT',
  },
  Key: {
    pk: 'tag', sk: `tag#${tagName}`,
  },
  TableName: tableName,
  UpdateExpression: `SET #quantity = :one + if_not_exists(#quantity, :zero),
                          #tagName = :tagName,
                          #_et = :_et,
                          #_ct = if_not_exists(#_ct, :timestamp),
                          #_md = :timestamp
                      `,
});

await docClient.send(command);

Although it wasn't an explicit requirement, it seemed like a good idea to be able to track how many times each tag is used. Thanks to the :one + if_not_exists(#quantity, :zero) expression, we can either increment the count of an existing tag or set the count of a new tag to 1. Although this is an Update command, DynamoDB APIs are idempotent so if the key doesn't exist, a new item will be created. if_not_exists is also used here to ensire the creation date (_ct) is only set if the item is new.

These tag summaries (entity type TAG_COUNT) are later queried for use in the summarization ML operation. That code is written in Python. It simply queries on the tag partition key and loops to make sure we get all the items on that key as DynamoDB queries are paginated.

def get_tags():
    response = client.query(
        ExpressionAttributeNames={"#pk": "pk"},
        ExpressionAttributeValues={":tag": {"S": "tag"}},
        KeyConditionExpression="#pk = :tag",
        TableName=tableName,
    )

    tags = response["Items"]

    # Need the loop to get all tags.
    while "LastEvaluatedKey" in response:
        response = client.query(
            ExclusiveStartKey=response["LastEvaluatedKey"],
            ExpressionAttributeNames={"#pk": "pk"},
            ExpressionAttributeValues={":tag": {"S": "tag"}},
            KeyConditionExpression="#pk = :tag",
            TableName=tableName,
        )
        tags.update(response["Items"])

    return tags

Roadmap

Hackathon projects always leave plenty of things undone! We think this project could be a real asset to the community and so we have brainstormed a number of capabilities that could be added going forward. We invite the community to contribute and develop skills with ML, AppSync, CDK, Flutter, and more.

UX Improvements
Update entities
Improve recommendation engine with a live model
Text2Speech
Notification system
Recommendations include location/whether virtual
Allow users to include their own images vs using AI-generated
QoL devexp (builds are still slow)
Integration tests
Security tests and verification in pipeline
Speaker, talk, and event reviews / 5-star system
Import from sessionize/et al
Social login

Conclusion

This was an ambitious project and a lot of work during hours that we might otherwise be doing something else, but it's amazing working with Danielle, Julian, and Johannes and I know this won't be our last collaboration. Personally I learned a ton and got exposed to quite a few technologies I hadn't laid hands on before. As this project was focused on ML and Transformers Agent, we should give special attention there. I feel this API is fairly easy to use and it's an innovative way to expose ML models to developers. That said, it was clear to me over the course of the hackathon that data science is a real discipline and not one I've practiced enough to assert that I'm getting the most from these models and predictions.

Lowering the bar to entry to a deep and complex technology can only be a good thing for the community. It's a lot to ask of someone to master all the technologies outlined in this blogpost and that's coming from a team that has more than half a century of collective experience! Despite being seasoned coders, we still spent valuable hackathon time asking "do we need a VPC?", "how does Sagemaker work?", and "why is this thing so slow?" It's our hope that our contribution will help others to solve these problems and more, and to build great things.

AWS Users, Roles, and Identity Center Demystified

Matt Morgan — Mon, 12 Jun 2023 12:46:50 +0000

I had the privilege and pleasure of speaking in the developer lounge at DC Summit this year. I chose a topic that I've frequently perceived as a stumbling block in a developer's journey to serverless and cloud: Identity and Access Management or IAM.

The gist of my talk is that I introduce some of the basic concepts which are critical to understanding IAM and then show a few practical ways to get hands-on experience. In particular I draw the distinction between IAM Users and Roles. I also had a section on IAM Identity Center, touched on OIDC and finished up talking about how frameworks can help us understand and find success using service roles.

After my talk, I had some questions from a listener and it quickly dawned on me that I had failed to explain that Identity Center Users are not the same as IAM Users. By omitting this critical fact, I'd not cleared up things as much as I'd hoped I could.

So let's try this again. Before I get to how overloaded "user" is, it's a good idea to define a few core concepts in IAM.

IAM Policies

Policies define the permissions that are being granted or denied. While they do nothing by themselves, they can be attached to all types of Principals (see below). Policies define the action or actions will either be allowed or denied and the resources which are governed by the Policy. If two Policies are in conflict (one says Allow and one Deny), then access is denied.

Here's a sample Policy from my talk:



{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "sqs:SendMessage*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:sqs:us-east-1:123456789012:my-queue"
    }
  ]
}

This Policy allows the Principal to which it is attached to send messages to the referenced queue.

IAM Principals

An IAM Principal can refer to a human user or a workload that requires credentialed access to AWS resources. There are two main types of Principals in AWS in addition to root users. The primary difference between types of Principals is whether the credentials they provide are permanent or temporary.

Root Users

Root users are a special type of IAM User that carry additional permissions such as the ability to close the account. Each account has exactly one root user. The best practice is to use that root user to set up other ways to access the account, then stop using it.

IAM Users

We've already established that we aren't only talking about human users but it bears repeating that an IAM User is not necessarily a human user. IAM Users are frequently assigned to machine workflows. For example, a Jenkins server may have an IAM User which allows it to make aws cli requests in order to deploy applications. Or a web application may have an IAM User which allows it to store objects in S3. Humans may be assigned IAM Users which allow them to access the AWS console and/or execute cli or sdk requests to AWS services.

The important thing to keep in mind is that an IAM User is a long-lived or permanent credential. It lasts until revoked. IAM Users can provide one or more access keys, often in the format of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This is effectively a username and password combo for whatever resources can be managed by the IAM User's attached Policies.

IAM Users are still in common use, but they are no longer a best practice. The problem with IAM Users is access keys get passed around, checked into GitHub and exposed as environment variables. Since they don't expire, the risk of exposure is ever-present. It's not possible to work with IAM Users without facing this risk.

Not convinced? I was a bit of a hard sell, myself. I operated my personal account on the root user for more than a decade, then switched to an IAM User. Recently I locked down my root user, deleted my IAM User and switched entirely to roles. Why? In addition to wanting to learn how to secure my account properly, I also came to understand that using Roles is simply a better developer experience. I'll go into detail on my account setup in a future post.

Roles

Roles are used to grant temporary credentials to both human and machine users. This sounds like it might be complicated or annoying, but actually Roles are a lot easier to work with than Users.

Human users using Roles can leverage IAM Identity Center (formerly AWS SSO) which offers a pretty good experience, whether we're federating from Active Directory (a popular choice for enterprises) or managing users within Identity Center (fine for individuals or small team). We get an easy console sign-in experience and similarly frictionless command line access.

Roles are the simplest way to integrate services in IAM. We can grant a Lambda Function the ability to send messages to SQS. We won't need to provide an access key at runtime. The AWS SDK will automatically handle fetching credentials and refreshing them when they expire. The credentials will be scoped to the Policy that is attached to the Role we have given our Lambda Function. This may be complex to understand, but it's easy enough to operate and can largely be handled by a framework.

When it comes to outside access, whether an on-prem server or a third party CI/CD or monitoring service, we still have the option of using Roles by means of IAM Roles Anywhere and OpenID Connect (OIDC). These very similar capabilities allow us to establish a trust relationship between the 3rd party and AWS such that the 3rd party can assume a Role.

IAM User vs Identity Center User

IAM Identity Center is not a Principal like IAM Users and Roles. Instead it's a way to connect a human user to a Role, which can then be used to access the AWS console or services via the cli or sdk. The users that are managed in IAM Identity Center are not IAM Users. This is the point that I'd failed to convey to my listener.

Identity Center provides support for inviting users (again not IAM Users) or federating from an Active Directory or SAML provider. Users can be given permission sets or added to groups that have permission sets. Best of all, Identity Center works at the AWS Organization level, so everything is in one place, even if your Organization spans many AWS accounts and regions.

Comparing Users and Roles

I'm making the claim that we should use Roles, not only because they are more secure, but also because they provide a better developer experience. Let's walk through a few scenarios to see if that holds up.

Console Access

Human access to the AWS console is a basic use case that must be supported.

With an IAM User

Logging into the AWS console with an IAM User requires you to enter your Account ID, IAM user name, and Password, then (hopefully) enter a two-factor authentication code.

Account ID required to sign in
Multi-account not supported
Federation not supported

IAM Users don't support federation so you can't use Active Directory or a SAML provider to sign in. If a human with access to an IAM User leaves an organization, the organization must take care to delete or disable the IAM User since removal from corporate directories will not automatically disable console access. This may lead organizations to limit console access when it's something that can benefit any developer who is developing in the cloud.

IAM Users don't scale well to multi-account scenarios. You have the option of creating separate users in each account or you can allow IAM Users to assume Roles in other accounts, at which point you're much better off using IAM Identity Center.

With IAM Identity Center (and Roles)

Landing page with custom URL
Multi-account, multi-role support
Federation supported

IAM Identity Center allows for a much better sign-in experience and supports multi-account out of the box. It also supports federation so you can give an organization-wide single sign-on experience.

When signing in with IAM Identity Center, users can start at a custom, company-branded URL, choose the account they wish to access and choose the appropriate role for the task at hand.

Role and user management are handled in one place, even in multi-account scenarios. Administrators can create groups and permission sets to manage access. There are a couple of rough edges to Identity Center. For example, when viewing a user in the console, you can tell what groups they are a part of, but you can't easily tell if any additional permission sets have been assigned. That's only visible at the account level. This limitation in mind, it's still a far better experience than trying to manage IAM Users across multiple accounts.

Command-Line Access

Developers frequently require command-line access to AWS resources using tools like the AWS CLI, AWS CDK, or AWS SAM. This can be done with IAM Users or Roles.

With an IAM User

Long-lived access key
Separate access key needed for each account/role
Credentials stored on developer workstation
Credentials not tied to an identity source (no federation)

To use an IAM User at the command line, we must generate an access key and then place it in a credentials file which is normally inside a hidden directory on the user's filesystem. Although developers are recommended to store these credentials well outside a project path, they are occasionally committed to GitHub with disastrous results.

Developers also have the option of setting an access key and secret as environment variables, but this has the drawback of it being more burdensome to switch accounts and roles.

With IAM Identity Center (and Roles)

Short-lived credentials
Easy to set up multi-account, multi-role
Developer need not handle credentials
Credentials can be tied to a federated identity source

IAM Identity Center provides command line capabilities. While the short-lived credentials may be cached on a developer machine for speed, they don't need to be handled by any human so they are less likely to be mishandled, shared, or committed to a repository.

The experience of using IAM Identity Center is seamless and not disruptive while being one of the most secure ways to access AWS. I'll detail how I do this in a future post.

Third Party Machine Access

This could be an on-prem application, a SaaS partner, or even an application running in AWS that isn't using Roles.

With an IAM User

Long-lived access key which must be handled by humans and stored somewhere
Credential can potentially be exfiltrated from a compromised partner or runtime
Traceability is poor - it can be unclear which applications or partners are using which access keys

This scenario is very similar to the developer cli access. We will have an access key and it must be stored somewhere. Usually the key is stored either in a credentials file on disk or as an environment variable. Either way, there's a risk of exposure of this long-lived credential.

The access key will need to be handled by humans at some point which is how they wind up on sticky notes, laptops, and other places they shouldn't be. Rotating access keys regularly helps with this problem, but that often involves manual work.

With Roles

No copy-pasting of access keys
Short-term credentials
No credential need to be stored on the third party

To use Roles with a third party, we'll need to either use OpenID Connect or IAM Roles Anywhere. These options will again allow us to use short-term credentials. Most significantly, we can use 3rd party or on-prem applications without storing any access keys or other secrets. Instead we'll establish a trust relationship using certificates or ssh keys, similar to how developers around the world push code to their version control.

Service Access

Developers working in serverless and managed services will frequently need to create roles to allow intra-service communication.

With an IAM User

Long-lived access key which must be handled by humans and stored somewhere
Credential can potentially be exfiltrated from a compromised partner or runtime
Traceability is poor - it can be unclear which applications or partners are using which access keys

It's not that common to see developers trying to put access keys into Lambda function. Many kinds of integrations, such as an SQS queue subscribing to an SNS topic, have no IAM User option at all. Any compute layer such as Lambda, Fargate, or EC2 can use IAM Users but all of these can support roles.

In addition to security concerns, access keys must be stored somewhere to be provided to an application. Often they are injected during a CI/CD pipeline. This wouldn't be necessary using Roles.

With a Role

Short-lived credentials with refreshes handled by the SDK
No copy-pasting access keys
Framework support
It's your only option in many scenarios!

As previously stated, Roles are the most common way to manage these interactions and are the only option in many cases. Developers who are using frameworks like the AWS Cloud Development Kit or the AWS Serverless Application Model can take advantage of framework features that make Role management relatively easy. The Roles and Policies can be created and assigned using these frameworks. Here are a couple of examples of how that works:

AWS CDK



const queue = new Queue(this, 'MyQueue');

const function = new Function(this, 'MyFunction', { 
  // more args 
});

queue.grantSendMessages(function);

This code will create a function role and assign it a policy that allows it to send messages to the queue that was created.

AWS SAM



Resources:
  MyQueue:
    Type: AWS::SQS::Queue
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      # additional properties
      Policies:
        - AWSLambdaBasicExecutionRole
        - SQSSendMessagePolicy:
          QueueName: !GetAtt MyQueue.QueueName

This is functionally equivalent to the CDK code.

Not using Lambda? Both EC2 and Fargate also support Roles.

Conclusion

In this post, I've defined IAM Policies, Principals, Users, and Roles. I've described IAM Identity Center and tried to explain how the users in Identity Center are not IAM Users.

Beyond that, I made the case that switching from IAM Users to Roles delivers a superior developer experience in addition to being more secure.

The best security tools are the ones that make the right thing the easiest thing. IAM Roles and Identity Center succeed in delivering not just a safer experience, but also a better developer experience.

I hope this post has helped to clear up some of the core concepts in IAM. I invite readers to add questions in the comments if any of these concepts remain unclear.

Avoiding the Serverless Workflow Antipattern

Matt Morgan — Mon, 27 Feb 2023 12:16:45 +0000

A common pattern for teams moving from cron-based jobs to event-driven architectures is to start using AWS Lambda as the compute engine for asynchronous event handlers. This can work out very well for simple event handlers, but I've observed a common antipattern that can appear when workflows grow more complex. Sooner or later workflows with multiple steps enter the picture and we'll need to be careful in our approach lest the complexity get the best of us.

Asynchronous Event Handler

Using Lambda as an event handler can be a pretty good plan. A large number of AWS services can target Lambda, making integrations easy. For third parties, you can use a FURL to create a low-friction webhook. Lambda of course has a pricing model that is ideal for irregularly-scheduled events. It's relatively easy to compose and deploy a Lambda function.

Fan-Out with SQS

This simple architecture can cover a lot of use cases, but not all of them. What happens when complexity grows? The main driver I've seen for adopting more complex architectures is when the job won't complete within 15 minutes, but there can be other reasons for doing this as well, such as asynchronously communicating with some other system in the middle of the job.

Let's say our job is running long and we decide to adopt a fan-out pattern. We can achieve this by putting messages in a queue using SQS and subscribing another Lambda function to those messages. This will give us the opportunity for batching and throttling to make sure our workflow can run in parallel, but not overwhelm other resources.

Fan-Out with Job State Tracking

Okay, now our workload is broken down some, but we have a new problem. How do we know when the job has finished? How do we know the success/failure rate of each step in the job? What if we only want one instance of our job to run at a time? We can't prevent a new job from kicking off if we don't know whether or not the old one has finished.

SQS is a great service, but it's not a workflow engine! It won't maintain job state for us, so if that's important, it will be up to us to implement it. We could store the current job state in a DynamoDB table, writing the number of expected downstream jobs and tallying them as we go, maybe including another step at the end to give us a summary report. Okay, now our architecture looks like this:

To achieve this architecture, each of our Lambda Functions is now responsible for job tracking. We've more or less had to write our own workflow engine. We'll own that code and need to maintain it. This is an antipattern that violates the promise of managed services, misuses SQS, and forces us to add boilerplate, workflow-aware code to every Lambda Function.

Enter Step Functions

AWS Step Functions is a great tool to solve this problem. Step Functions allow us to focus on our business logic while workflow and job state are handled by a managed service. We can query the service or check out the console visualization to find out whether or not a job is in progress or finished.

All the problems outlined above are solved by building the workflow with Step Functions. In the provided example, we still have the Job Start function, which perhaps performs an initial database query to drive the rest of the job, or it could query the Step Functions service to see if another execution is underway (for use cases where we want to limit concurrency). If none of that applies to our workflow, we can skip this step! Likewise we could skip the summary step since Step Functions will be able to provide the output of the work that was done by each of our workers and as such a summary step would only be needed if we wanted to format a human-readable report.

Learning Step Functions

Step Functions can be complex and challenging to learn. They are described using Amazon States Language. This complexity can drive developers to avoid this service, leading to subpar architectures. I acknowledge there's a lot to learn here, but it's well worth it! And it doesn't need to be an uphill battle.

The Workflow Studio is a great way to get get productive fast. You may also be interested in an article I wrote about learning to write Step Functions using AWS CDK. If that doesn't do it for you, there are a number of other free resources on the web that can put you on the right path.

You Also Get...

I left off a lot of the great features of Step Functions by focusing on the comparison to an SQS-driven workflow. Step Functions also support:

synchronous express workflows
native integrations (Lambda-less)
asynchronous invocation and callback with Task Tokens
time-based Wait steps
Intrinsic Functions (again Lambda-less)
automatic Retry

Know and Plan Your Architecture

The best path to success in cloud is to understand our objectives and know which services will help us to achieve them. Sometimes all we need is a single Lambda handler. If "fire and forget" is good enough, then an SQS-based fan-out is probably fine. Still, the use-cases for more complex workflows is indelible and more often than not, we'll need the ability to audit them. Step Functions is a great tool for achieving that aim.

More on Step Functions

As mentioned above, there's a lot of great articles out there on Step Functions to help in your journey. Here are a few of my favorites - and please feel free to add to the list in comments below!

Appreciation for Community Managers

Matt Morgan — Tue, 20 Dec 2022 12:42:48 +0000

I had the privilege of joining the inaugural class of AWS Community Builders back in 2020. I've put a lot into the program in these two years and change, but it's fair to say I've gotten more out. So when @lockhead asked me to help out with an appreciation post for our community management team, I of course said yes! The format that follows is the same as the related posts.

What surprises you most about the community builders program?

I'm impressed with the way the program keeps growing in numbers but somehow maintains the intimacy of a conversation. Just one example of this is the Community Builders slack. There's always something interesting happening there, but it never feels overwhelming or intimidating. Of course Community Builders connect all over social media, in the workplace, at conferences and anywhere else we dare!

What’s your background and your experience with AWS?

It's long and varied :D

I started writing web apps in the late 1990s. I registered my personal AWS account back in 2012 or 2013. I started using AWS heavily in my day job in 2017 and I became a serious hardcore serverless cultist in 2019. Still going strong!

What’s the biggest benefit you see from the program?

Connecting with other builders is a huge benefit. Having worked in a few different companies and having had a peek inside a few more, it's surprising how many of the same problems we're all solving. Taking on challenges is easier with a team and Community Builders are generous of their time to help others in their journeys.

What’s the next swag item that you would like to get?

I know one of the Community Builders had a custom phone case made. This seems like a great idea!

What are you eating for dinner today? Share the recipe!

Heat olive oil in a pan. Get it very hot! Add some green beans and sprinkle with salt and pepper. Toss them vigorously and watch the flames leap. Serve them crisp and hot.

Is there anything else you would like to say about the community builders program in 2022?

This is the best tech community I've been a part of. I'm making lifelong friends, helping others to grow, growing myself, participating in great events, and learning a lot. The program provides me with experiences and activity when I want it and is patient when I get busy with my day job.

I highly recommend applying to be an AWS Community Builder, and if you are not accepted, try again in six months. I know many Community Builders who were accepted on subsequent application. I invite readers to connect with me if you want to know more.

Lambda Powertools TypeScript is Generally Available

Matt Morgan — Tue, 19 Jul 2022 16:03:19 +0000

I gave a first look at Lambda Powertools TypeScript back in January of 2022. I was pretty excited for the library at the time, but it came with an admonishment that it wasn't ready for production use. Well, the general availability announcement dropped July 15 so it's time for another look.

What's Changed?
ES Modules Support
Comparisons
Logger
Metrics
Tracer
Roadmap
Conclusion

What's Changed

So what's changed in the beta? A glance at the CHANGELOG suggests the answer is that not very much has changed in six months. Lambda Powertools TypeScript still supports class decorators, middy, and a manual API. It still covers the core capabilities of logging, metrics and tracing, and no new capabilities have been added. Outside of some bug fixes and optimization, this is still very much the library I previewed back in January.

ES Modules Support

One change I might've liked to see is support for ES Modules. Interest in ES Modules has been growing quickly in the serverless community, driven largely by the desire to use by Top-Level Await.

Lambda Powertools TypeScript can't be used directly as an ES Modules dependency, but there is an issue open, so please consider adding your +1. For now, it's possible to get by with a require shim or via cjs tricks but it would be great to see native support for ES Modules in Lambda Powertools TypeScript.

Comparisons

Given that I've already gone over these modules in the prior post, I thought I'd instead compare the Lambda Powertools TypeScript modules to similar solutions. I'm looking at the API, the bundled script size, cold start and execution time. To gather metrics, I've written a little app using Step Functions that can run many instances of a function in parallel and capture the metrics.

My benchmarking tool will run each function 50 times aiming to achieve a 20% cold start rate. Code samples are available on GitHub.

Logger

The simplest way to log to CloudWatch from AWS Lambda is with console.log. Doing so adds no bloat to your function, there's no dependency management and writes to CloudWatch are asynchronous, so the operation is non-blocking. Many developers use libraries to guarantee logs are in a structured format and to control the verbosity of logging.

That said, we can use console as a baseline due to its simplicity. If we want a function that simply writes some unstructured logs, we can do something like this.

import type {
  APIGatewayProxyEventV2,
  APIGatewayProxyResultV2,
  Context,
} from 'aws-lambda';

export const handler = async (
  event: APIGatewayProxyEventV2,
  context: Context
): Promise<APIGatewayProxyResultV2> => {
  console.log('event: ', event);
  console.log('context: ', context);
  return { statusCode: 200 };
};

Logging out the event yields the stringified context object:

2022-07-19T12:10:45.503Z    2abe532e-2b26-46b5-9a65-884363160556    INFO    context:  {
  callbackWaitsForEmptyEventLoop: [Getter/Setter],
  succeed: [Function (anonymous)],
  fail: [Function (anonymous)],
  done: [Function (anonymous)],
  functionVersion: '$LATEST',
  functionName: 'LoggerConsole',
  memoryLimitInMB: '128',
  logGroupName: '/aws/lambda/LoggerConsole',
  logStreamName: '2022/07/19/[$LATEST]384dbd25ffeb4af49bc22c2ac4f333df',
  clientContext: undefined,
  identity: undefined,
  invokedFunctionArn: 'arn:aws:lambda:us-east-1:123456790:function:LoggerConsole',
  awsRequestId: '2abe532e-2b26-46b5-9a65-884363160556',
  getRemainingTimeInMillis: [Function: getRemainingTimeInMillis]
}

This is pretty noisy and having those succeed, fail and done methods doesn't offer a lot of value.

With Powertools we can inject a more useful context into log messages.

import { Logger } from '@aws-lambda-powertools/logger';

import type { LambdaInterface } from '@aws-lambda-powertools/commons';
import type { APIGatewayProxyEventV2, Context } from 'aws-lambda';

const logger = new Logger();

class Lambda implements LambdaInterface {
  @logger.injectLambdaContext({ logEvent: true })
  public async handler(
    _event: APIGatewayProxyEventV2,
    _context: Context
  ): Promise<void> {
    logger.info('Here is some info!');
  }
}

export const myFunction = new Lambda();
export const handler = myFunction.handler;

Now we get structured logs.

{
    "cold_start": false,
    "function_arn": "arn:aws:lambda:us-east-1:123456790:function:LoggerPowertools",
    "function_memory_size": 128,
    "function_name": "LoggerPowertools",
    "function_request_id": "6eeaa0c9-e58f-45a7-bed2-a4b9d7e65d7e",
    "level": "INFO",
    "message": "Here is some info!",
    "service": "service_undefined",
    "timestamp": "2022-07-19T12:09:28.537Z",
    "xray_trace_id": "1-62d69ef6-dfdbc4be0f59a76c57c52cf8"
}

This is going to be much easier to search and doesn't include useless stringified methods. Plus we get a cold_start boolean tossed in.

For the sake of comparison, I threw in another implementation of the function using the popular and enduring winston library. Let's see how they did.

Function	Avg Cold Start	Avg Duration	Code Size
LoggerConsole	137.24	1.63	771
LoggerPowertools	157.58	1.86	78550
LoggerWinston	184.17	1.62	233142

The no-dependencies version is always going to be the fastest. Powertools adds around 78kb while winston is much heavier at 232kb. In either case we're not adding a lot of latency, but Powertools is smaller and thus faster and it gives us that cold_start metric.

Metrics

Often when it comes to metrics, we think about CPU, latency and other operational metrics and AWS services usually provide those out of the box. This kind of thinking can be flawed when we end up having to use 3rd parties such as google analytics to infer critical business events. A simpler solution is to have the application emit a metric when a business event (say a customer signup) occurs. We have a few options for doing this: We can use aws-sdk, we can use the aws-embedded-metrics lib and now we can use Powertools Metrics. Which is the best? Let's see.

To baseline this, let's use a function that doesn't emit metrics.

import type { APIGatewayProxyEventV2, Context } from 'aws-lambda';

export const handler = async (
  _event: APIGatewayProxyEventV2,
  _context: Context
): Promise<void> => {
  const workflowSuccess = Math.random() > 0.5;
  if (workflowSuccess) {
    console.log('The workflow was successful!');
  } else {
    console.log('The workflow failed.');
  }
};

To understand whether or not our workflow is successful, we'll need to query logs. Ugh!

Let's try emitting metrics using the @aws-sdk/client-cloudwatch library from aws-sdk-v3.

import {
  CloudWatchClient,
  MetricDatum,
  PutMetricDataCommand,
} from '@aws-sdk/client-cloudwatch';

import type { APIGatewayProxyEventV2, Context } from 'aws-lambda';

const client = new CloudWatchClient({});

export const handler = async (
  _event: APIGatewayProxyEventV2,
  _context: Context
): Promise<void> => {
  const workflowSuccess = Math.random() > 0.5;
  let metric: MetricDatum;
  if (workflowSuccess) {
    console.log('The workflow was successful!');
    metric = { MetricName: 'WorkflowSuccess', Value: 1, Unit: 'Count' };
  } else {
    console.log('The workflow failed.');
    metric = { MetricName: 'WorkflowFailure', Value: 1, Unit: 'Count' };
  }
  const command = new PutMetricDataCommand({
    MetricData: [metric],
    Namespace: 'SdkV3Metrics',
  });
  await client.send(command);
};

Now we have some nice metrics in CloudWatch!

The downside of using aws-sdk for this is that it relies on API calls and is somewhat slow. We can try to achieve the same thing using aws-embedded-metrics. How is this different from Cloudwatch custom metrics? It's better and fellow Community Builder Vishnu Prassad can tell you why.

import { createMetricsLogger, Unit } from 'aws-embedded-metrics';

import type { APIGatewayProxyEventV2, Context } from 'aws-lambda';

export const handler = async (
  _event: APIGatewayProxyEventV2,
  _context: Context
): Promise<void> => {
  const workflowSuccess = Math.random() > 0.5;
  const metrics = createMetricsLogger();
  metrics.putDimensions({ Service: 'EMF' });
  if (workflowSuccess) {
    metrics.putMetric('WorkflowSuccess', 1, Unit.Count);
  } else {
    metrics.putMetric('WorkflowFailure', 1, Unit.Count);
  }
  await metrics.flush();
};

In addition to having the advantage of EMF, the code is a bit more concise.

import { LambdaInterface } from '@aws-lambda-powertools/commons';
import { Metrics, MetricUnits } from '@aws-lambda-powertools/metrics';

import type { APIGatewayProxyEventV2, Context } from 'aws-lambda';

const metrics = new Metrics({ namespace: 'Workflow' });

class Lambda implements LambdaInterface {
  @metrics.logMetrics()
  public async handler(
    _event: APIGatewayProxyEventV2,
    _context: Context
  ): Promise<void> {
    const workflowSuccess = Math.random() > 0.5;
    if (workflowSuccess) {
      metrics.addMetric('WorkflowSuccess', MetricUnits.Count, 1);
    } else {
      metrics.addMetric('WorkflowFailure', MetricUnits.Count, 1);
    }
  }
}

export const myFunction = new Lambda();
export const handler = myFunction.handler;

The Powertools version is just a tad more verbose due to the need to use class decorators, but still not bad. In the end we'll care more about performance, so let's run those numbers.

Function	Avg Cold Start	Avg Duration	Code Size
MetricsNone	135.25	0.93	826
MetricsEMF	146.75	1.33	29456
MetricsSDKV3	225.37	32	257026
MetricsPowertools	141.3	1.33	7942

I might've thought the Powertools implementation here would wrap aws-embedded-metrics, but obviously not! Although the actual performance benefit of Powertools over aws-embedded-metrics is marginal, you really have to appreciate how they've kept the size down.

Tracer

In the case of the Tracer module, it actually does wrap the aws-xray-sdk. So why would we use Tracer instead? If the API is nicer and it's not adding much in the way of latency, it could be worth it. Here's an example of using aws-xray-sdk. In this case the Lambda Function is tracing a separate function as well as an SDK call to (somewhat uselessly) get function properties.

import {
  GetFunctionCommand,
  GetFunctionCommandOutput,
  LambdaClient,
} from '@aws-sdk/client-lambda';
import { captureAsyncFunc, captureAWSv3Client } from 'aws-xray-sdk-core';

import type { Context } from 'aws-lambda';
const client = new LambdaClient({});

captureAWSv3Client(client);

const getFunction = async (
  context: Context
): Promise<GetFunctionCommandOutput> => {
  const command = new GetFunctionCommand({
    FunctionName: context.functionName,
  });
  return client.send(command);
};

export const handler = (
  _event: unknown,
  context: Context
): Promise<GetFunctionCommandOutput> =>
  captureAsyncFunc('methodWithCustomTrace', async (subsegment) => {
    const fn = await getFunction(context);
    subsegment?.close();
    return fn;
  });

That captureAsyncFunc part is a bit awkward. How does Powertools do this?

import { LambdaInterface } from '@aws-lambda-powertools/commons';
import { Tracer } from '@aws-lambda-powertools/tracer';
import {
  GetFunctionCommand,
  GetFunctionCommandOutput,
  LambdaClient,
} from '@aws-sdk/client-lambda';

import type { Context } from 'aws-lambda';

const client = new LambdaClient({});

const tracer = new Tracer({ serviceName: 'powertoolsTracer' });
tracer.captureAWSv3Client(client);

class Lambda implements LambdaInterface {
  @tracer.captureMethod()
  public async methodWithCustomTrace(
    context: Context
  ): Promise<GetFunctionCommandOutput> {
    const command = new GetFunctionCommand({
      FunctionName: context.functionName,
    });
    return client.send(command);
  }

  @tracer.captureLambdaHandler()
  public async handler(
    _event: unknown,
    context: Context
  ): Promise<GetFunctionCommandOutput> {
    return this.methodWithCustomTrace(context);
  }
}

export const handlerClass = new Lambda();
export const handler = handlerClass.handler;

I'm really starting to warm to class decorators for cases like this. Function decorators would be better, but as previously discussed, they don't currently exist in TypeScript.

Function	Avg Cold Start	Avg Duration	Code Size
TracerXRay	266.91	50.56	410514
TracerPowertools	265.42	48.25	417980

Checking the performance numbers, Powertools is extremely light, still weighing in at around 7kb and barely impacting performance at all. In this case, Powertools is slightly faster, but I suspect over the long run it's costing a couple of ms, so worth it for the devexp.

Roadmap

One final thing to consider when deciding whether to adopt Lambda Powertools is what features may be in store in the future. The more mature libraries Lambda Powertools Python and Lambda Powertools Java include a number of useful utilities that we may like to see in Lambda Powertools TypeScript. The overall Lambda Powertools Roadmap doesn't tell us very much other than pending dotnet and golang libraries, but we can drill down into TypeScript-specific issues and glimpse the immediate future. It looks like stability is still the main priority, but there are some interesting items like RFC: Testing Factories for AWS Data Objects in the leftmost column.

Community engagement and voting will no doubt help to drive the Powertools roadmap.

Conclusion

To me, this is a no-brainer. One of the challenges in writing Lambda is that many of our dependencies were not designed for Lambda. This library obviously was and the team obviously took care to deliver maximum value in a minimal package. These core utilities promote best practices in a way that is accessible and easy to use. Whether you're writing in TypeScript or JavaScript, you can enjoy good IDE support and a high-level API to implement logging, metrics and tracing in Lambda.

COVER

How to Use Source Maps in TypeScript Lambda Functions (with Benchmarks)

Matt Morgan — Mon, 24 Jan 2022 11:50:51 +0000

TypeScript is a popular language for developers of all kinds and it's made its mark on the serverless space. Most of the major Lambda frameworks now have solid TypeScript support. The days of struggling with webpack configurations are mostly behind us.

Stack Traces
Emitting Source Maps
Source Map Support
CDK Example
- Serverless Stack
SAM Example
- Architect
Serverless Framework Example
Benchmarks
- Not Minified without Source Maps
- Minified without Source Maps
- Minified with Source Maps
- Error Minified without Source Maps
- Error Minified with Source Maps
Conclusion

Stack Traces

If we're already transpiling our code to convert the TypeScript source to Lambda-friendly JavaScript, we might as well go ahead and minify and tree-shake the code as well. Smaller bundles can make deployments faster and may even help with cold start and execution time. However, this can make debugging difficult when we start seeing stack traces that look like this:

{
    "errorType": "SyntaxError",
    "errorMessage": "Unexpected end of JSON input",
    "stack": [
        "SyntaxError: Unexpected end of JSON input",
        "    at JSON.parse (<anonymous>)",
        "    at VA (/var/task/index.js:25:69708)",
        "    at Runtime.R8 [as handler] (/var/task/index.js:25:69808)",
        "    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
    ]
}

The error message here lets us know that we're failing to parse a JSON string, but the stack trace itself is useless for finding the line where the code is failing. We'll have no choice but to look through our code and hope to find the error. We might be able to do a text search for JSON.parse but if that is happening in one of our dependencies, searching won't work. What's next? Add a bunch of log statements to the code? No! If we use Source Maps, we can get more useful stack traces:

{
    "errorType": "SyntaxError",
    "errorMessage": "Unexpected end of JSON input",
    "stack": [
        "SyntaxError: Unexpected end of JSON input",
        "    at JSON.parse (<anonymous>)",
        "    at VA (/fns/db.ts:39:8)",
        "    at Runtime.R8 (/fns/list.ts:6:24)",
        "    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
    ]
}

Now we can see the stack includes a call from line 6 of list.ts which finally fails on line 39 of db.ts.

To enable Source Maps in our application, we need to tell our build tool to emit Source Maps and we need to enable Source Map support in our runtime.

Emitting Source Maps

Emitting Source Maps is very easy with esbuild. We simply set the boolean property in our configuration. Now when we run the build, we'll get an index.js.map file as well as our index.js. This file must be uploaded to the Lambda service. We'll see how to do that with AWS CDK, AWS SAM and the Serverless Framework a bit later in this article.

Source Map Support

Having the index.js.map file in our Lambda runtime isn't sufficient to enable Source Maps. We also have to make sure the runtime knows to make use of them. Fortunately this is very easy ever since Node.js version 12.12.0. We just have to set the --enable-source-maps command line option. Command line options can be set in AWS Lambda by setting the NODE_OPTIONS environment variable. At the time of this writing, AWS Lambda supports Node.js versions 12 and 14. AWS does not publish the minor versions in use in Lambda, but we can discover it by logging out process.version in a function. As of late January, 2022, the Node.js versions in use in Lambda in us-east-1 are v12.22.7 and v14.18.1 so we'll have no trouble using Source Maps.

If we needed to enable Source Maps in a runtime that doesn't support the native version, we could always use Source Map Support.

CDK Example

All example code is available on GitHub.

AWS CDK is my preferred tool for writing and deploying serverless applications, in part because of the aws-lambda-nodejs construct. This construct makes it very easy to work with TypeScript. It wraps esbuild and exposes options. It also supports setting environment variables.

When I'm working with multiple Lambda functions, I often find it helpful to create a single props object that then gets shared among multiple functions.

const lambdaProps = {
  architecture: Architecture.ARM_64,
  bundling: { minify: true, sourceMap: true },
  environment: {
    NODE_OPTIONS: '--enable-source-maps',
  },
  logRetention: RetentionDays.ONE_DAY,
  runtime: Runtime.NODEJS_14_X,
  memorySize: 512,
  timeout: Duration.minutes(1),
};

new NodejsFunction(this, 'FuncOne', {
  ...lambdaProps,
  entry: `${__dirname}/../fns/one.ts`,
});

new NodejsFunction(this, 'FuncTwo', {
  ...lambdaProps,
  entry: `${__dirname}/../fns/two.ts`,
});

As we can see, it's very simple to enable Source Maps when already using AWS CDK and NodejsFunction.

Serverless Stack

Serverless Stack is a very cool value add that builds on top of AWS CDK delivering an awesome developer experience and dashboard. SST's own version of NodejsFunction, Function automatically emits source maps. Their use can be enabled simply by setting NODE_OPTIONS as described.

SAM Example

SAM support for TypeScript has been lagging for some time, but a pull request was just merged that should change that. It looks like we'll be able to add an aws_sam key in the package.json file to enable building within the SAM engine as part of sam build. Although this PR has been merged to aws-lambda-builders, the engine behind sam build, it will still need to be added to aws-sam-cli and released (to much fanfare, one expects) before it can be used with SAM.

Meanwhile - or if we're considering other options for deploying our functions - we can add an extra build step. We'll create an esbuild.ts file that transpiles the functions and then point our SAM template.yaml file at the output of that step.

import { build, BuildOptions } from 'esbuild';

const buildOptions: BuildOptions = {
  bundle: true,
  entryPoints: {
    ['create/index']: `${__dirname}/../fns/create.ts`,
    ['delete/index']: `${__dirname}/../fns/delete.ts`,
    ['list/index']: `${__dirname}/../fns/list.ts`,
  },
  minify: true,
  outbase: 'fns',
  outdir: 'sam/build',
  platform: 'node',
  sourcemap: true,
};

build(buildOptions);

SAM templates will take everything in the CodeUri so the above code will transpile a TypeScript file at fns/list.ts and output sam/build/list/index.js and sam/build/list/index.js.map. By setting CodeUri: sam/build/list, SAM will package the two files and upload them to Lambda.

Now we just need to set the environment variable. This is easy enough to do in a SAM template. We can set it as a global so it only needs to be in the template once

Globals:
  Function:
    Environment:
      Variables:
        NODE_OPTIONS: '--enable-source-maps'

In order to make sure we always build before deploying, we can add some npm scripts.

"scripts": {
  "build:lambda": "npm run clean && ts-node --files sam/esbuild.ts",
  "clean": "rimraf cdk.out sam/build",
  "deploy:sam": "npm run build:lambda && sam deploy --template template.yaml",
  "destroy:sam": "sam delete"
}

This works well enough, but does take some extra effort. SAM users are no doubt eagerly awaiting better TypeScript support.

Architect

Alternately, use Architect. Architect is a 3rd party developer experience that builds on top of AWS SAM. Architect includes a TypeScript plugin.

Serverless Framework Example

The Serverless Framework is known for its plugin system. serverless-esbuild brings bundling and all the options we need to support Source Maps in TypeScript into the sls package and sls deploy commands.

The plugin is configured in our serverless.yml file.

custom:
  esbuild:
    bundle: true
    minify: true
    sourcemap: true

And then we just point our functions at our TypeScript handlers.

functions:
  create:
    handler: fns/create.handler

Much like SAM, Serverless lets us set global environment variables for our functions.

provider:
  name: aws
  lambdaHashingVersion: 20201221
  runtime: nodejs14.x
  environment:
    NODE_OPTIONS: '--enable-source-maps'

Much like AWS CDK, this is a good experience for TypeScript developers. Those already using Serverless Framework should have an easy time adding Source Maps to their applications.

Benchmarks

There's a lot of guidance against using Source Maps in production because of a supposed negative performance impact. Let's measure the admittedly-simple list function and see if minification or the use of source maps has any noticeable effect.

I used autocannon to test the function at 100 concurrent executions for 30 seconds. I also used Lambda Power Tuning to find the ideal memory configuration, which proved to be 512MB. All the results are available.

Not Minified without Source Maps

The unminified function is 1.2MB in size. This is mostly from @aws-sdk/client-dynamodb and @aws-sdk/lib-dynamodb. Whether sticking with SDK v2 might be better performance would be a good topic for another post. This function is over one MB, despite a small amount of custom code.

Running the test, the function has an average execution of 46.99ms and a max of 914ms. 99% of my requests are at or below 90ms.

Minified without Source Maps

Minifying the function drops the size to 534.8kb, less than half the size. My test showed an average response of 47.83ms, a max of 1010ms and 90ms at the 99th percentile. This is slightly worse, but not statistically significant. I expect if I ran these tests over and over again, I would see that minifying the code has no real effect on performance. This is not a surprise. 1.2MB is still fairly small and I don't expect to see much in the way of added latency at that size.

Minified with Source Maps

The minified function is still 534.8kb, of course, but the Source Map is 1.5MB so this will be the largest upload, not that ~2MB is a lot or will significantly slow down our deployments.

The average response time for this test is 46.52ms, max is 968mb and the 99th percentile is 82ms. This is the best result so far, but not statistically significant. I would say this is truly a three-way tie which tells us that adding Source Maps to this function did not increase latency.

That's because of the native support in Node.js! Since no stack traces were emitted, the Source Maps were never referenced. It's possible if we had to rely on a library for this capability, we wouldn't have the same outcome.

Error Minified without Source Maps

Will triggering error conditions make a difference? Let's see. I ran the same test but this time the function has that JSON.parse error in it. This happens before the call to DynamoDB, so we can expect it to be a little faster than the successful function. We get 37.86ms as the average, 1004ms as the max and 58ms at 99%. It's impressive the call to DynamoDB only seems to add about 10ms of latency!

Error Minified with Source Maps

Enabling Source Maps for the errors does impact performance. Our average has dropped to 97.54ms, max at 1129ms and 99% is 243. This is a significant increase. Source Maps do impact latency when an error occurs. This makes sense and confirms the idea that the Source Maps are only referenced when an error occurs - but now that Source Map must be parsed and that takes time.

Conclusion

Use Source Maps in production! In my view, if you are getting so many errors that the performance hit from Source Maps is impacting your bottom line, you should probably go and fix those errors. It's very easy to implement Source Maps in a variety of popular Lambda frameworks and they don't impact successful execution. When functions do fail, the useful stack trace is going to be worth the added latency. Developer hours are always going to be more expensive than a few milliseconds of Lambda execution.

COVER

First Look at Lambda Powertools TypeScript

Matt Morgan — Mon, 10 Jan 2022 12:41:01 +0000

AWS Senior Solutions Architect Sara Gerion announced on January 5, 2022 that Lambda Powertools TypeScript had entered public beta. Lambda Powertools is an AWS-sponsored open source project with the goal of improving developer experience and adoption of best practices when using AWS Lambda. Lambda Powertools TypeScript joins Java and Python Lambda Powertools libraries.

Node.js Tooling for Lambda
Decorators
No Decorators
Capabilities
Tracer
Logger
Metrics
Package Size
Conclusion

Node.js Tooling for Lambda

I'm a big fan of TypeScript and in fact co-authored a book about it. I don't find myself using Java or Python much, so while I've been interested in Lambda Powertools, I never tried it out until now. Lambda Powertools TypeScript joins middy and DAZN Lambda Powertools in the Lambda tooling space for the Node.js runtime. Two things that differentiate Lambda Powertools TypeScript from comparable libraries are it is sponsored by AWS and it supports decorators.

Lambda Powertools TypeScript supports both v2 and v3 of the AWS SDK for JavaScript and examples are given for both versions.

Decorators

Opinions vary on decorators, but I find them to be useful abstractions and have used them heavily in object-oriented TypeScript. However, there's a fairly major limitation in TypeScript and that is that we can put decorators on class methods, but not functions. This means, unfortunately, code that looks like this is out of reach for now.

// This doesn't work :(
@doSomethingGreat()
export const handler = async () => {
  ...
};

It's a pity because that would be an excellent developer experience. To attach the doSomethingGreat decorator to our handler, we need to write something like this instead.

class Lambda {
  @doSomethingGreat()
  public async handler() {
    ...
  }
}

const handlerClass = new Lambda();
export const handler = handlerClass.handler;

It's five extra lines - perhaps not a big deal. In any case, there's not a lot the Powertools team can do as it may well be years before decorators are supported for functions. Keep in mind if we do choose to use decorators and classes in Lambda, we need to be careful around the this reference.

Decorators and TypeScript aren't supported out of the box in Lambda (without using deno) so we'll also need a transpilation step if we go this route. Fortunately this is a mostly solved problem for AWS CDK, AWS SAM and Serverless Framework users. If you want or need to roll your own, esbuild is a great place to start and seems to be the bundler of choice for this purpose.

No Decorators

We may wish to opt out of using classes or transpilers. Lambda Powertools TypeScript can be used without decorators and in fact without TypeScript. We could use the library with vanilla Node.js.

The documentation for Lambda Powertools TypeScript is pretty good and provides several examples with even more on GitHub.

Capabilities

All three versions of Lambda Powertools include Metrics, Logger and Tracer as core utilities. Lambda Powertools Python includes an Event Handler and several other useful utilities supporting Batch Processing, Idempotency, Validation and more.

Each version of Lambda Powertools is developed independently and the capabilities are custom-tailored to the distinct needs of the different runtimes. This is in contrast to AWS CDK using jsii to publish the same constructs to multiple runtimes. Although it may be frustrating to wait for some of these capabilities, this is likely the right approach as it would just add to the complexity to think about generic code that compiles to something that can decorate custom code supported in multiple runtimes.

To test this out, I've implemented all three of the Lambda Powertools TypeScript utilities in one of my sample projects. I chose my CDK Async Testing project because it contains several Lambda functions and includes asynchronous workflows via EventBridge and Step Functions.

If you want to see my instrumented code, it is available in this branch.

Tracer

In order to instrument my functions with the Tracer, I needed to rewrite them as classes. I chose to use decorators because I'm really undecided about whether to start using classes everywhere, so I need to get a feel for it. Starting off with collect.ts, here's the function as it was originally at eleven lines of code.

import { Payment } from '../models/payment';

export const handler = async (input: {
  Payload: { Payment: Payment };
}): Promise<{ Status: number; Payment: Payment }> => {
  const min = 0;
  const max = 1;
  const Status = Math.floor(Math.random() * (max - min + 1)) + min;

  return { Status, Payment: input.Payload.Payment };
};

And here's the refactor to include the Tracer.

import { Tracer } from '@aws-lambda-powertools/tracer';

import type { LambdaInterface } from '@aws-lambda-powertools/commons';
import type { Context } from 'aws-lambda';

import type { Payment } from '../models/payment';

const tracer = new Tracer({ serviceName: 'paymentCollections' });

class Lambda implements LambdaInterface {
  @tracer.captureLambdaHandler()
  public async handler(
    input: { Payload: { Payment: Payment } },
    _context: Context,
  ): Promise<{ Status: number; Payment: Payment }> {
    const min = 0;
    const max = 1;
    const Status = Math.floor(Math.random() * (max - min + 1)) + min;

    return { Status, Payment: input.Payload.Payment };
  }
}

const handlerClass = new Lambda();
export const handler = handlerClass.handler;

The function is now 25 lines of code, which isn't horrible. It ends up at 33 lines after adding in the Metrics and Logger utilities. Of course it's expected that my function will be larger as it has more capabilities now. We should see the increase as additive, not multiplicative. My 11 line function became 25 lines. Had it originally been 111 lines, it would've grown to 125, not doubled.

So what do we get for that? The Tracer module wraps the AWS X-Ray SDK (as a transitive dependency). It doesn't really add any net new capabilities, but makes the SDK easier to work with. In my experience, that SDK is a bit of a bear so this may be well worth it. We can decorate class methods to introduce new trace segments in a single line of code. We can also use the imperative form to add new traces where we see fit. We can capture AWS clients, but that simply exposes the X-Ray SDK.

One thing that isn't solved here (yet?) is the oddness of working with the X-Ray SDK and DynamoDB DocumentClient. When using DocumentClient with X-Ray, we need to do a workaround as the SDK needs access to the DocumentClient service property, but that's not exposed on the type. Here's how I've solved that problem in the past.

import DynamoDB, { DocumentClient } from 'aws-sdk/clients/dynamodb';
import { captureAWSClient } from 'aws-xray-sdk-core';

const client = new DocumentClient();
captureAWSClient((client as DocumentClient & { service: DynamoDB }).service);

UPDATE! This was solved in release 0.5.0! The above code can now be written as:

import { DocumentClient } from 'aws-sdk/clients/dynamodb';
import { captureAWSClient } from 'aws-xray-sdk-core';

const client = new DocumentClient();
captureAWSClient(client);

Very much appreciate the quick turnaround on this improvement! Now that I have better DX, here's what my instrumented app looks like.

original text
And now with Lambda Powertools TypeScript.

import DynamoDB, { DocumentClient } from 'aws-sdk/clients/dynamodb';
import { Tracer } from '@aws-lambda-powertools/tracer';

const tracer = new Tracer({ serviceName: 'paymentCollections' });
const client = new DocumentClient();
tracer.captureAWSClient((client as DocumentClient & { service: DynamoDB }).service);

So basically the same thing. The advantage of this utility will really only come into play vs just using the X-Ray SDK if we're adding several segments to our code. I don't think I'm unlocking that in my sample application, but I did enable tracing in all of my functions and my state machine and the result is pretty good.

Not only do I get this great service map, I also get detailed traces.

The Tracer module is adding the ## index.handler part seen on these screenshots. I'll want to add more traces to really take advantage of this tool. Overall getting these detailed traces of my entire application through all the functions and services is quite impressive and useful. X-Ray is doing most of the work, but a better developer experience means we'll wind up with more applications instrumented properly and that's certainly a good thing.

Getting ahead of myself slightly here, but traces also include logs and logs for the instrumented segments will be found alongside the traces in CloudWatch.

Again, this is great. I'm able to write my application in a distributed way that uses single-purpose functions, but see the whole picture when it executes.

X-Ray can be a fairly cheap service to use provided we remember to set a sampling rate on high-throughput applications. Pricing seems to be based on trace so adding additional segments to a trace doesn't add to the cost.

Logger

The Logger utility is a drop-in replacement for any logger, including console. The value add Logger brings is the ability to inject the Lambda context into each logging message. When we annotate our handler method with @logger.injectLambdaContext(), then use logger.info, we'll see log messages that look like this.

{
  "cold_start": true,
  "function_arn": "arn:aws:lambda:us-east-1:1234567890:function:openCollection",
  "function_memory_size": 128,
  "function_name": "openCollection",
  "function_request_id": "df21844d-f4b6-49b4-b246-da54183ce5cf",
  "level": "INFO",
  "message": "Payment set to collection",
  "service": "paymentCollections",
  "timestamp": "2022-01-09T18:46:20.622Z"
}

If we plan to ingest logs into a search index or even if we just want to use CloudWatch Logs Insights, this can be really handy as the structure will help us to search and filter log messages. On the other hand, if we're just going through a few log messages, this can be a bit noisy. We should keep in mind that any log service (including CloudWatch) is going to bill on volume and extremely verbose logs can be expensive.

With that in mind, there are a lot of good features for the Logger utility and we can structure our logs however we like. Additionally Logger includes a sampling rate feature which can be used to keep costs down.

By default logger methods take one or more arguments. The first argument needs to be a string or object with a message key. I noticed that if I gave a string as a subsequent argument, the string would be converted into a character array and printed like that, so that's something to watch out for.

Metrics

The Metrics utility is used to publish custom CloudWatch metrics. Although Lambda automatically publishes a number of useful metrics like latency, concurrent executions, throttling, etc., custom metrics give us the opportunity to complete the observability story by adding relevant business events to our metrics.

Tracking reliability is important, but it does not tell the whole story! If anything, custom metrics should be the most important ones. How many customers signed up this week? How many of them were able to complete valuable workflows? The answers to these questions lie in our code and if we emit custom metrics, they can also be in our dashboards.

Custom metrics have a pricing structure which can be expensive. Embedded Metrics Format can help manage the cost and is supported by Lambda Powertools TypeScript. Again, the docs here are pretty good, so no need for me to break it down. Instead let's look at the experience. I've added a custom metric of "collectionSuccess" to my collectionSuccess function. In my hypothetical app, some payments wind up in collections and here I'm marking whether or not the collection resulted in a payment.

import { Logger } from '@aws-lambda-powertools/logger';
import { Metrics, MetricUnits } from '@aws-lambda-powertools/metrics';
import { Tracer } from '@aws-lambda-powertools/tracer';

import { PaymentEntity, PaymentStatus } from '../models/payment';

import type { LambdaInterface } from '@aws-lambda-powertools/commons';
import type { Context } from 'aws-lambda';
import type { Payment } from '../models/payment';

const powerToolsConfig = { namespace: 'payments', serviceName: 'paymentCollections' };
const logger = new Logger(powerToolsConfig);
const metrics = new Metrics(powerToolsConfig);
const tracer = new Tracer(powerToolsConfig);

class Lambda implements LambdaInterface {
  @logger.injectLambdaContext()
  @metrics.logMetrics({ captureColdStartMetric: true })
  @tracer.captureLambdaHandler()
  public async handler(input: { Payload: { Payment: Payment } }, _context: Context): Promise<void> {
    try {
      await PaymentEntity.update({ id: input.Payload.Payment.id, status: PaymentStatus.COLLECTION_SUCCESS });
      metrics.addMetric('collectionSuccess', MetricUnits.Count, 1);
    } catch (e) {
      logger.error('Failed to record collection success', e);
      throw e;
    }
  }
}

const handlerClass = new Lambda();
export const handler = handlerClass.handler;

Adding @metrics.logMetrics will cause any metrics we emit to also be logged to CloudWatch. We may or may not want that (keeping costs in mind again). To add a custom metric, we simply use metrics.addMetric.

I instrumented my app to emit metrics for cold starts as well as some custom metrics that describe important events in my application, such as successful and failed payments and collections. Because the point of my application is to demonstrate an integration test, I also instrumented custom metrics that indicate when the test is run.

These metrics can be found in CloudWatch Metrics, placed in a dashboard or exported via API to a 3rd party tool.

Package Size

Adding all of the utilities to my project seemed to add about 600kb unminified or 200kb to minified bundles. Given the value and the need to chain some dependencies into the AWS SDK or X-Ray SDK, this seems quite reasonable and the team has done a good job of staying with their Keep it lean tenent.

Conclusion

Lambda Powertools does an excellent job putting focus on the kinds of utilities developers really need to improve their applications and follow best practices. The core modules here all focus on observability and that emphasis is needed and appreciated. The team has done a good job developing an API that will be attractive to developers who want to use decorators as well as those who don't.

I eagerly await the general availability of this library and will follow the roadmap to see what's coming up and how I can get involved.

COVER

Testing the Async Cloud with AWS CDK

Matt Morgan — Mon, 13 Sep 2021 11:30:28 +0000

One of the best optimizations we can make when adopting cloud and serverless technologies is to take advantage of asynchronous processing. Services like EventBridge, Step Functions, SNS and SQS can form the backbone of highly scalable, reliable, cost-effective and performant job processing. Cloud services like these mean we no longer have to worry about a single process going out of memory or a spike in traffic causing critical business functions to fail when we need them most.

Adopting asynchronous architectures introduces new challenges around test automation. Nearly all automation testing is built around the idea of call and response synchronous processing. We call the web server and test the html string returned. We POST to the /widgets endpoint, receive a 200 OK response and then can GET our new widget. Tools like Selenium and Cypress aren't suited to workflows that involve asynchronous background processing. At best, you can fire an event and then poll for the eventual desired result.

While the idea of testing frameworks for asynchronous processing hasn't really hit the mainstream, there are some great articles out there. I highly recommend following the work of Sarah Hamilton and Paul Swail and their approaches to solving this problem using libraries like aws-testing-library and sls-test-tools.

These approaches share some of the same drawbacks:

They rely on an external credential or key to access the AWS account in question.
They require a wait or sleep step to allow AWS time to process the asynchronous event.
There's no real CI/CD integration happening. A failing test won't cause a deployment to roll back.

Read enough already? Check out my source code.

External Credentials
Don't Sleep
Automatic Rollback
Custom Resource Provider Framework for Test
Payment Collections App
Test Event
Complete Event
Provider
Test Run
Bonus: EventBridgeWebSocket
Infrastructure Testing
Next Steps

External Credentials

When writing tests using something like aws-testing-library or sls-test-tools, we must have credentials to an AWS account that at least lets us send a few events and subscribe to or query the results in order to perform assertions. Depending on the stance of our organization about cloud access, this could be completely fine or it could be a never-gonna-happen dealbreaker. Often this kind of approach will be OK for a development environment, but it could be unlikely to fly in production.

We may be restricted to running such tests in a CI/CD pipeline. That could get us past a security review, but it might also slow the feedback loop of writing tests, resulting in fewer overall tests written.

The safest way to solve this problem is to remove external access from the equation. If we run our tests from a resource within the AWS account, such as a Lambda function, then while we still need normal IAM permissions, we don't need to create any additional external roles. Tests can be run in production without violating the least privilege principle of giving our deployment role the minimum grants required to create the resources we need.

Don't Sleep

Sleep or wait statements are the bane of test automation. When a test fails, can we "fix" it by increasing the timeout? Does doing so reflect the normal operation of using cloud or have we introduced some instability to our system that is now being normalized by making the tests take longer?

In the end, we have to pin our sleep statement to the longest our process can possibly take or decide to be tolerant of test failures. We can mitigate this problem by using polling instead of sleep. To be fair, the solutions given by Sarah and Paul don't preclude polling, but you are on your own to implement it.

Automatic Rollback

Once we have a really good automation suite, we should wish to connect it to our CI/CD process such that if the tests fail, we roll back the deployment. Assuming we are using AWS and CloudFormation, the only lever we really have to pull there is to attach an alarm to a test suite. This could be achieved by having the test publish custom metrics that trigger the alarm and a potential rollback. This would be yet another permission needed to our test runner role and isn't something natively supported by any testing library.

Custom Resource Provider Framework for Test

This is not the first time I've written about Provider Framework. I think the CDK implementation of CloudFormation Custom Resources is one of the coolest things I've come across recently. I'm constantly impressed by the things I can do with it. Let's examine how it solves some of the problems I've outlined above.

Our tests will be executed by Lambda functions instead of via some external tool. These functions will still need normal IAM roles to access the resources needed to perform the test, but the functions themselves are not accessible to any resource or role external from the account. This means we can run tests in production without risking any data exposure.

We get built-in polling in the isComplete step. This is a Lambda function that can be executed at queryInterval that can perform some sort of assertion and then return a boolean value indicating whether the test is complete or we should continue polling.

Provider Framework also has built-in error-handling. Throwing any error will pass a "FAILED" response to CloudFormation, which in turn will trigger a rollback.

These advantages are balanced by there not being a good assertion library ready to use in a Lambda function and by the potential for a poor developer experience if we find ourselves debugging the custom resource itself and having to perform frequent deployments to write a test. I think those problems can be solved with some additional tooling, but nothing exists to date.

Payment Collections App

So let's see a test in action. The design of this application is that we receive an asynchronous notification from our payment provider indicating whether or not a payment was successful. If the payment was successful, we update our database to indicate that. Otherwise we trigger a collections workflow.

Payment events (success or failure) come to us via EventBridge. Our Collections workflow is managed by Step Functions. The record of payment and current status is stored in DynamoDB.

The scenarios we'd like to test are:

Receive a successful payment event and record it in our table.
Receive a failed payment event and kick off the collections workflow, run the collection, and record the eventual result in our table.

Test Event

To write the test we start with a Lambda function that will initiate the events we want to test. In this case, we're going to send two events to our event bus. Here is a TypeScript function that will do exactly that.

import EventBridge from 'aws-sdk/clients/eventbridge';

import type { CloudFormationCustomResourceEvent } from 'aws-lambda';
const eb = new EventBridge();

export const handler = async (event: CloudFormationCustomResourceEvent): Promise<void> => {
  if (event.RequestType === 'Delete') {
    return;
  }
  try {
    const { Version } = event.ResourceProperties;

    const events = ['success', 'failure'].map((status) =>
      eb
        .putEvents({
          Entries: [
            {
              EventBusName: process.env.BUS_NAME,
              Source: 'payments',
              DetailType: status,
              Time: new Date(),
              Detail: JSON.stringify({ id: `${Version}-${status}` }),
            },
          ],
        })
        .promise(),
    );

    await Promise.all(events);
  } catch (e) {
    console.error(e);
    throw new Error('Integration Test failed!');
  }
};

We're able to take advantage of the CloudFormationCustomResourceEvent type from the @types/aws-lambda package. Because Custom Resources support the full lifecycle of CloudFormation, we need to evaluate "Delete" as a no-op. We don't want the test to run if the stack is being deleted, as it would obviously fail since the necessary resources won't exist.

The rest of the function simply uses putEvents to propagate two test events, a success and a failure. Note that we are pulling the Version from the Custom Resource and passing that as the event detail.

Complete Event

The complete event will be called at the interval specified in our code (defaulting to 5 seconds) until it returns a positive result, throws an error or totalTimeout (default 30 minutes and max of two hours) elapses. As before, if this is a stack deletion event, we want the test to just pass as our business logic shouldn't be under test. If we're undergoing a create or update event, then we'll want to query the database to see if the job is finished yet. We are using the same Version from the original Custom Resource to make sure we can query the same item in the table.

import { CloudFormationCustomResourceEvent } from 'aws-lambda';

import { PaymentEntity, PaymentStatus } from '../models/payment';

export const handler = async (
  event: CloudFormationCustomResourceEvent,
): Promise<{ Data?: { Result: string }; IsComplete: boolean }> => {
  if (event.RequestType === 'Delete') {
    return { IsComplete: true };
  }
  const { Version } = event.ResourceProperties;
  try {
    // Query the DynamoDB table to get the meta record. If it has some processed records
    // and the processed count is equal to validated count, the test passes.
    const successResponse = (await PaymentEntity.get({ id: `${Version}-success` })).Item || {};
    const failureResponse = (await PaymentEntity.get({ id: `${Version}-failure` })).Item || {};
    console.log('Success Response: ', successResponse.status);
    console.log('Failure Response: ', failureResponse.status);
    const IsComplete =
      successResponse.status === PaymentStatus.SUCCESS &&
      [PaymentStatus.COLLECTION_FAILURE, PaymentStatus.COLLECTION_SUCCESS].includes(failureResponse.status);
    return IsComplete
      ? {
          Data: {
            Result: `Payment ${Version}-success finished with status ${successResponse.status} and payment ${Version}-failure finished with status ${failureResponse.status}`,
          },
          IsComplete,
        }
      : { IsComplete };
  } catch (e) {
    console.error(e);
    throw e;
  }
};

This function must return an object with IsComplete and a boolean value. If IsComplete is true, then it may also include a Data attribute with a JSON payload. In this case I've defined that as { Result: string }. My intent is to print that string in the console to provide some detail on the test.

Note that we lack the convenience of a nice assertion library, but this example is simple enough to get by with imperative code.

Provider

We need to write a little CDK code to make all this work. In this case, I wrote the entire integration test as a nested stack, which gives it a nice isolation from the actual application. In addition to just organizing our code an potentially avoiding stack limits, this also lets us hedge our best a little by making it easy to disable the nested stack in the event of a test failure blocking a critical deployment.

The stack creates the functions and grants necessary permissions to put events to EventBridge and query DynamoDB. Over the course of the test, we'll see a few events come across EventBridge and also trigger Step Functions. Tests to trigger other asynchronous workflows using SNS or SQS can be done in a similar fashion. Adding the functions to Provider Framework only takes a few lines of code.

const intTestProvider = new Provider(stack, 'IntTestProvider', {
  isCompleteHandler,
  logRetention: RetentionDays.ONE_DAY,
  onEventHandler,
  totalTimeout: Duration.minutes(1),
});

this.testResource = new CustomResource(stack, 'IntTestResource', {
  properties: { Version: new Date().getTime().toString() },
  serviceToken: intTestProvider.serviceToken,
});

We pass in our two handlers to the Provider construct, then create a CustomResource that uses the service token from the Provider. We are also establishing that Version property which will be the current timestamp in milliseconds as a string. This can serve as a unique key, assuming two tests don't kick off in the same millisecond. If we think that might happen, then using some kind of uuid generator would be more appropriate.

It's important that we include some kind of unique value here because if we don't then update events will not detect any change and will not run our test! Setting some kind of unique value here is critical.

Finally the testResource is stored as a member of the IntegrationTestStack so that we can access the output and print it to the console from the main stack.

new CfnOutput(this, 'IntTestResult', { value: intTestStack.testResource.getAttString('Result') });

Test Run

With all that done, we can now deploy our application as normal, whether that's a cdk deploy from our laptop or something more sophisticated like a CI/CD pipeline.

Outputs:
payments-app-stack.IntTestResult = Payment 1631477097557-success finished with status SUCCESS and payment 1631477097557-failure finished with status COLLECTION_FAILURE

We can check out the visualization of a few step function runs.

Bonus: EventBridgeWebSocket

I didn't want to pass up the advantage to give a quick look to David Boyne's EventBridgeWebSocket construct. With just a few lines of code, I'm able to actively monitor EventBridge traffic while my test runs!

new EventBridgeWebSocket(stack, 'sockets', {
  bus: eventBus.eventBusName,
  eventPattern: {
    source: ['payments'],
  },
  stage: 'dev',
});

There's really nothing to this. I just have to provide the bus name and an optional pattern. Now using websocat, I get output like this:

% websocat wss://MY_API_ID.execute-api.us-east-1.amazonaws.com/dev
{"version":"0","id":"e8d8989a-c870-85b2-9f56-aa7867701eae","detail-type":"success","source":"payments","account":"MY_ACCOUNT_ID","time":"2021-09-12T20:22:03Z","region":"us-east-1","resources":[],"detail":{"id":"1631478043585-success"}}
{"version":"0","id":"34f6d36b-0cda-65e2-9851-194a9c1be01d","detail-type":"failure","source":"payments","account":"MY_ACCOUNT_ID","time":"2021-09-12T20:22:04Z","region":"us-east-1","resources":[],"detail":{"id":"1631478043585-failure"}}
{"version":"0","id":"8444d579-c553-4c91-e07a-9de3f9cc17c0","detail-type":"collections","source":"payments","account":"MY_ACCOUNT_ID","time":"2021-09-12T20:22:06Z","region":"us-east-1","resources":[],"detail":{"id":"1631478043585-failure"}}

This isn't explicitly a testing tool of course, but can be of great help when debugging issues and the cost of entry is amazingly low. Just make sure you don't implement this in production, at least without adding an authorizer to the WebSocketApi.

Infrastructure Testing

Now that we've established Provider Framework as a good basis for testing, are there other applications? Yes! Instead of writing a test against our application, we could use aws-sdk to perform assertions against our infrastructure. This has the same advantages outlined above:

We don't need external keys to perform these assertions.
Using addDependency can guarantee the test only runs after the resources are created or updated.
If the test fails, the deployment will automatically roll back.

It's worth mentioning that AWS CDK already has an RFC for integration testing so we might end up with something even better. In the meantime, if you are serious about integration testing, time to give AWS CDK and Provider Framework a look!

Next Steps

The weakness of this approach is the imperative coding and lack of convenience methods exposed by libraries like aws-testing-library and sls-test-tools offer. It definitely feels a lot more ergonomic to write expect({...}).toHaveLog(expectedLog) than to query a database and try to test properties on the returned item. I'm not sure if a library like jest is really suited to running in Lambda, but there's definitely room for innovation on a better assertion engine here.

That said, I think this approach is strong enough on its own. I've been using just such a test for about three months now and find it to be very reliable and a good way to guarantee quality delivery.

COVER IMAGE

Your Complete API Gateway and CORS Guide

Matt Morgan — Tue, 20 Apr 2021 12:29:35 +0000

I recently spent some quality time with some colleagues who were implementing a web app using API Gateway. It's really a very useful service, but navigating it can be challenging due to the sheer number of options available. We wound up grinding out a solution and I wanted to share my learnings, not just the "how" but also the "why" in hopes it'll help others come to some of these decisions more easily.

I will probably make this part of a series, but for this article, I want to go into the ways to enable Cross-Origin Resource Sharing or CORS in API Gateway. It's not as intuitive as it could be and depending on the sort of integration you want to use, you'll need a different implementation.

AWS CDK is my infrastructure-as-code tool of choice. I'll provided examples in TypeScript, but also show some console screenshots and walk through some generated CloudFormation. Examples are available here.

API Gateway
CORS
HTTP API vs REST API
Custom Integration with REST API
Proxy Integration with REST API
Proxy Integration with HTTP API
REST API Service Integrations
HTTP API Service Integrations
Conclusion

API Gateway

If you're in the AWS ecosystem, you have several choices for granting external users access to your application. API Gateway is a full-featured serverless option. It is often compared to Application Load Balancer. ALB has fewer features and is not serverless, but may be cheaper for high-throughput applications. There are some good resources out there that go into this topic at depth, so I won't.

API Gateway is often used for invoking Lambda functions, but can be connected to many other AWS services as well as HTTP integrations. API Gateway offers support for request validation, throttling, transformation and various authorization mechanisms. Taking full advantage of API Gateway can do a lot to offset the higher price point but there can be a high cognitive load in doing so.

CORS

CORS is a security mechanism supported by all major web browsers. If you are dealing with web apps, you are going to contend with CORS one way or another. I have worked in environments where the CORS headers were added by some kind of ingress gateway and I've worked in environments where the headers had to be set explicitly by the applications. API Gateway supports both models, as we shall see.

I urge you to read up on best practices and make the correct choices for your application. Some of my examples use wildcards (*) for allowed domains. You'll typically want something more restrictive than that in a web application.

In brief, what we're going to need to do to support CORS is to add an HTTP OPTIONS method for each of our route and then set the Access-Control-Allow-Origin header on our responses. There are other CORS headers that can be optionally set, such as Access-Control-Allow-Credentials, Access-Control-Allow-Headers, Access-Control-Allow-Methods and Access-Control-Max-Age. Setting these headers works in exactly the same way, so I'll just focus on Access-Control-Allow-Origin.

HTTP API vs REST API

Before we can begin any API Gateway implementation, we need to decide which API implementation we're using. AWS API Gateway first launched in 2015 and has gained features steadily since, as AWS services tend to do. In 2019, AWS announced HTTP API as a lower-cost alternative to original, which became known as REST API.

Just to editorialize for a moment, while it's not hard to keep them straight, these are bad names, since both implementations of the service use HTTP and REST and as such, the names are not at all differentiating. Oh well, never mind, AWS is known for having this problem.

To add further confusion, HTTP API isn't Version 2 of API Gateway, but there is a version 2 of the spec. Payload format version 2.0 was initially rolled out for WebSocket APIs and is now available for HTTP API but not REST API. If you want to understand the differences, the official docs do a good job of giving us a side-by-side comparison. It's clear the intent of version 2.0 is to provide a simpler format, but you should be aware of the differences if you are used to payload version 1.0

So the three API implementations provided by API Gateway are REST API (payload format 1.0), HTTP API (choose either) and Websocket API (payload format 2.0). I won't provide any information on CORS headers for WebSocket API as it isn't part of the WebSocket spec.

The official documentation explains the feature differences between HTTP API and REST API and goes well beyond the scope of this article. I will mention that HTTP API is cheaper than REST API and thus generally the guidance is to choose it if it has all the features you need.

Proxy Integration with REST API

What, you thought you'd exhausted the decision tree? We're just getting started. There's a huge section just on the different integration types for REST API alone. Here we're going to be dealing with just one of those integration patterns, the proxy integration.

Most of the API Gateway/Lambda stacks I've seen have used proxy integrations. In this integration, we pass the request object to the function handler and construct a response object in the function as well, which is then passed back via API Gateway. That doesn't mean we're talking only about lambdaliths here! In fact, in Matt Coulter's excellent article on common Lambda patterns, he implements all three patterns using proxy integrations.

To enable CORS in a proxy integration, we need to do two things:

Return a response to an OPTIONS request for each route we want to enable for CORS.
Return valid CORS headers from our Lambda function.

AWS CDK gives us a nice shortcut for setting those OPTIONS responses. We can use defaultCorsPreflightOptions in our RestApiProps. That might look something like this:



    const restApi = new RestApi(this, 'ProxyCorsRestApi', {
      defaultCorsPreflightOptions: {
        allowOrigins: Cors.ALL_ORIGINS,
      },
    });

Setting this produces the following CloudFormation:



  ProxyCorsRestApiOPTIONSEF3C92C2:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: OPTIONS
      ResourceId:
        Fn::GetAtt:
          - ProxyCorsRestApiB964F16B
          - RootResourceId
      RestApiId:
        Ref: ProxyCorsRestApiB964F16B
      AuthorizationType: NONE
      Integration:
        IntegrationResponses:
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent'"
              method.response.header.Access-Control-Allow-Origin: "'*'"
              method.response.header.Access-Control-Allow-Methods: "'OPTIONS,GET,PUT,POST,DELETE,PATCH,HEAD'"
            StatusCode: '204'
        RequestTemplates:
          application/json: '{ statusCode: 200 }'
        Type: MOCK
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Headers: true
            method.response.header.Access-Control-Allow-Origin: true
            method.response.header.Access-Control-Allow-Methods: true
          StatusCode: '204'

And in a nutshell, this is why I like CDK! AWS SAM also lets you cut down on your yaml quite a bit. I definitely prefer one of these options over raw CloudFormation, but any of them should do the job.

We also need to create an integration. The CDK for that is quite nice:



    const proxyIntegration = new LambdaIntegration(proxyFn);

    restApi.root.addMethod('get', proxyIntegration);

This will produce CloudFormation:



  ProxyCorsRestApiget33C342A8:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId:
        Fn::GetAtt:
          - ProxyCorsRestApiB964F16B
          - RootResourceId
      RestApiId:
        Ref: ProxyCorsRestApiB964F16B
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri:
          Fn::Join:
            - ''
            - - 'arn:'
              - Ref: AWS::Partition
              - :apigateway:us-east-1:lambda:path/2015-03-31/functions/
              - Fn::GetAtt:
                  - RestProxyFn04E83FA9
                  - Arn
              - /invocations

This is verbose in comparison to the CDK, but it helps us understand what exactly is happening here. In order to invoke the Lambda service, API Gateway is making an HTTP POST on the appropriate /invocations endpoint. Good to know, but I'd definitely take a SAM template over this if forced into yaml.

Setting the headers is done in the actual Lambda function. Here's a simple TypeScript example:



import { APIGatewayProxyResult } from 'aws-lambda';

export const handler = async (): Promise<APIGatewayProxyResult> => {
  return {
    body: JSON.stringify({ state: 'ok' }),
    headers: {
      'Access-Control-Allow-Origin': '*',
    },
    statusCode: 200,
  };
};

Pretty straightforward in that we just return an object with the headers set, but it must be done in this exact format in order for API Gateway to map the response correctly - mapping still occurs in a proxy integration, you just don't have direct control over it.

Let's give our integrated route a look in the console.

Notice the "Integration Response" section is grayed out. Most of the other areas here don't have a lot going on. This integration pattern puts most of the work on Lambda and remains quite simple here in API Gateway.

Let's see if we can make that OPTIONS request the web browser will need. OPTIONS requests typically return an HTTP 204 response if successful.



% curl -i -X OPTIONS https://gzpd0b4wle.execute-api.us-east-1.amazonaws.com/prod/
HTTP/2 204
date: Tue, 20 Apr 2021 12:11:46 GMT
x-amzn-requestid: f409a183-4bed-4f3c-8a2b-d26b77665a99
access-control-allow-origin: *
access-control-allow-headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent
x-amz-apigw-id: eFO4aEnZIAMFkYw=
access-control-allow-methods: OPTIONS,GET,PUT,POST,DELETE,PATCH,HEAD
x-cache: Miss from cloudfront
via: 1.1 b051e9c33308597b659c33b8999b521d.cloudfront.net (CloudFront)
x-amz-cf-pop: IAD89-C2
x-amz-cf-id: ubVksQvwI3Xd7O4cli00MKp6QMnVpu6ofh2aaS7Ui7Mc3fAx-7Oaqw==

And then call the GET endpoint to verify the header is present.



% curl -i https://gzpd0b4wle.execute-api.us-east-1.amazonaws.com/prod/
HTTP/2 200
content-type: application/json
content-length: 14
date: Tue, 20 Apr 2021 12:12:16 GMT
x-amzn-requestid: 8335bb7d-bfd0-4814-9b16-cbbef3fad49f
access-control-allow-origin: *
x-amz-apigw-id: eFO88GedIAMFcMw=
x-amzn-trace-id: Root=1-607ec51f-791687c52f7630d700b81a35;Sampled=0
x-cache: Miss from cloudfront
via: 1.1 27eb501c8caff149895f88cac34554af.cloudfront.net (CloudFront)
x-amz-cf-pop: IAD89-C2
x-amz-cf-id: b-o4OnglA0hpW8ij-dGCGU1wnZjUyY6W43hJ2IDs46fZJjuQJ3r4kw==

{"state":"ok"}

So it's relatively easy to set up CORS for this pattern, but the downside is you have to add the headers to each function, which could produce a lot of boilerplate in larger stacks. I would definitely seek to add some kind of middleware solution when adding CORS headers across many functions.

Custom Integration with REST API

The next pattern is a bit more complicated.

A custom integration abstracts the HTTP request and response away from our Lambda function. We will have to map any parts of the request we want our function to see and then we'll have to map its response to something API Gateway can send back to the client. This pattern maximizes the work for API Gateway to do and minimizes how much we do in Lambda. Mappings are done using Velocity Template Language (VTL).

Why choose this integration pattern? VTL isn't exactly developer-friendly, but it's powerful. Additionally, API Gateway bills per request while Lambda bills per millisecond spent in startup and execution. If you have transformations that take meaningful amounts of time on a high-throughput API, using a custom integration could save you some money.

The other reason to think about a custom integration is it opens the door to non-Lambda integrations that use the same templates. Consider an API Gateway that serves some requests to Lambda, because some compute layer is needed for them, and integrates other requests directly with DynamoDB or S3 because they are simple enough to skip the compute layer. Such an integration would need to use mapping templates for the DynamoDB or S3 integrations already, so why not use them for Lambda as well?

Perhaps that's an edge case, but it's worth considering how best to shave your AWS bill. Service integrations and VTL could mean major savings, but only if the pattern doesn't increase development complexity by too much.

To add CORS to a custom integration we will need three things:

Return a response to an OPTIONS request for each route as before.
Add the desired headers to our integration responses.
Map the desired headers from our integration responses into our method responses.

The first part works just the same as it does in proxy integrations:



    const restApi = new RestApi(this, 'CustomCorsRestApi', {
      defaultCorsPreflightOptions: {
        allowOrigins: Cors.ALL_ORIGINS,
      },
    });

Now we need to add the header to our integration response which means it's going to be in the CDK code.



    const customIntegration = new LambdaIntegration(customFn, {
      integrationResponses: [
        {
          responseParameters: {
            'method.response.header.Access-Control-Allow-Origin': "'*'",
          },
          responseTemplates: {
            'application/json': JSON.stringify({
              message: '$util.parseJson($input.body)',
              state: 'ok',
            }),
          },
          statusCode: '200',
        },
      ],
      passthroughBehavior: PassthroughBehavior.NEVER,
      proxy: false,
      requestTemplates: {
        'application/json': JSON.stringify({
          input: 'this is the input',
        }),
      },
    });

This code contains a pair of request/response templates, which is one way to do custom integrations. The important part here is the responseParameter which will set the Access-Control-Allow-Origin header to a wildcard value. Note also I need to set proxy: false. CDK considers a proxy integration to be the default integration.

The third thing we'll need is to set responseParameters in the method's methodResponses.



    restApi.root.addMethod('get', customIntegration, {
      methodResponses: [
        {
          statusCode: '200',
          responseParameters: {
            'method.response.header.Access-Control-Allow-Origin': true,
          },
        },
      ],
    });

The boolean value here is only whether or not the parameter is required and this would still work with a value of false but it will not work if this mapping isn't in place. Note that for the sake of brevity, I've only mapped a 200 OK response. In a real application, we'd need to handle error codes in both the request and response templates and mappings. If we fail to do this, we're likely to get an error directly from the API Gateway service that will be challenging to debug.

Now that this code is in place, we'll see the OPTIONS request in our CloudFormation template, as with the proxy integration and we'll also be able to see the mappings in the GET method we've added:



  CustomCorsRestApiget57AF1EA7:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId:
        Fn::GetAtt:
          - CustomCorsRestApi3D2B9919
          - RootResourceId
      RestApiId:
        Ref: CustomCorsRestApi3D2B9919
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        IntegrationResponses:
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: "'*'"
            ResponseTemplates:
              application/json: '{"message":{"output":"$util.parseJson($input.body)"},"state":"ok"}'
            StatusCode: "200"
        PassthroughBehavior: NEVER
        RequestTemplates:
          application/json: '{"input":"this is the input"}'
        Type: AWS
        Uri:
          Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :apigateway:us-east-1:lambda:path/2015-03-31/functions/
              - Fn::GetAtt:
                  - CustomFnFE03B841
                  - Arn
              - /invocations
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: "200"

The upside of our custom integration is our function can basically be anything we like. We aren't bound by any particular API, but we do need to map to our templates. Here's my sample function:



interface inputEvent {
  input: string;
}

export const handler = async (event: inputEvent): Promise<string> => {
  return event.input;
};

I defined an interface that maps to my request template and returns a string which is output in the response template. There's a lot that can be done here but that will have to be the subject of other investigations.

The console view is a bit more relevant now. If we drill down into that Integration Response, we can find the header mappings setting CORS headers for as well as the request template.

Let's run the same test again for OPTIONS.



% curl -i -X OPTIONS https://153hn2leyd.execute-api.us-east-1.amazonaws.com/prod/
HTTP/2 204
date: Tue, 20 Apr 2021 12:13:41 GMT
x-amzn-requestid: 4e7b8882-56d6-497a-8850-cd4b3a274968
access-control-allow-origin: *
access-control-allow-headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token,X-Amz-User-Agent
x-amz-apigw-id: eFPKaHsioAMFliA=
access-control-allow-methods: OPTIONS,GET,PUT,POST,DELETE,PATCH,HEAD
x-cache: Miss from cloudfront
via: 1.1 c84ecfd128e1f4c41a53a2b42410f3b8.cloudfront.net (CloudFront)
x-amz-cf-pop: IAD89-C3
x-amz-cf-id: 8adT0FmbEfTm3wxzoK6B5ZgM2sHoxPpML2F2wSJU-X6UUjfZCI0nDw==

And on the GET endpoint.



% curl -i https://153hn2leyd.execute-api.us-east-1.amazonaws.com/prod/
HTTP/2 200
content-type: application/json
content-length: 55
date: Tue, 20 Apr 2021 12:13:46 GMT
x-amzn-requestid: bf6d4d09-cf3a-4772-bb19-fb2b0119e0ec
access-control-allow-origin: *
x-amz-apigw-id: eFPLIF_ToAMFTbQ=
x-amzn-trace-id: Root=1-607ec57a-1e08f2330043a36640aa8aac;Sampled=0
x-cache: Miss from cloudfront
via: 1.1 ba82151bf51e4c722c5305c983d8b71e.cloudfront.net (CloudFront)
x-amz-cf-pop: IAD89-C3
x-amz-cf-id: dPiy6f4MhSYlvUz35vmWTFqiAWfGynZw0X_oKvrdWSWZA-nLUxWjlA==

{"message":{"output":"this is the input"},"state":"ok"}

The header is set correctly!

Proxy Integration with HTTP API

It's pretty obvious that one of the goals in developing HTTP API was to simplify. This is exemplified in the fact that HTTP API only supports proxy integrations with Lambda.

That said, setting CORS for a proxy integration works differently in HTTP API than it does in REST API. Anybody transitioning from REST API to HTTP API is likely to get caught up by the change. Let's walk through the CDK code:



    const httpApi = new HttpApi(this, 'ProxyCorsHttpApi', {
      corsPreflight: { allowMethods: [CorsHttpMethod.ANY], allowOrigins: ['*'] },
    });

This looks a lot like how it's done in REST API, however when we look at the generated CloudFormation, there's a sizable difference.



  ProxyCorsHttpApiC5DA21B9:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      CorsConfiguration:
        AllowMethods:
          - "*"
        AllowOrigins:
          - "*"
      Name: ProxyCorsHttpApi
      ProtocolType: HTTP

That's it! You do not have to manually set OPTIONS requests or map headers. Instead the API Gateway service reads the CORS configuration and manages all of this for you. If you are describing your API Gateway in CloudFormation, you're definitely going to appreciate this innovation.

Likewise the AWS Console for API Gateway has a specific section for configuring CORS.

Just for the sake of knowledge, I tried and you can actually skip all of this and return headers from your Lambda function. That works, but then if you configure CORS in API Gateway, it'll overwrite the headers your function returns.

So let's make an OPTIONS request to our endpoint.



% curl -i -X OPTIONS https://6as8srxaw2.execute-api.us-east-1.amazonaws.com
HTTP/2 204
date: Tue, 20 Apr 2021 12:15:20 GMT
apigw-requestid: eFPZ2jtpoAMEVjQ=

Now for the trick. Everything is looking good, so I'll call my endpoint.



% curl -i https://6as8srxaw2.execute-api.us-east-1.amazonaws.com
HTTP/2 200
date: Tue, 20 Apr 2021 12:15:27 GMT
content-type: text/plain; charset=utf-8
content-length: 14
apigw-requestid: eFPa1hVpIAMEVRw=

{"state":"ok"}

No CORS header! What happened? It turns out that unlike REST API, HTTP API will only set the header if you pass in an Origin header on the request. Let's try that again.



% curl -i -H "Origin: https://mysite.com" https://6as8srxaw2.execute-api.us-east-1.amazonaws.com
HTTP/2 200
date: Tue, 20 Apr 2021 12:16:19 GMT
content-type: text/plain; charset=utf-8
content-length: 14
access-control-allow-origin: *
apigw-requestid: eFPjFitBIAMEVIw=

{"state":"ok"}

This is probably fine and correct behavior, but I'm certain it has caused stress and annoyance from developers trying to build on HTTP API. The official docs do mention CORS headers returned from Lambda will be ignored, but do not mention that the request must contain an Origin header for CORS to work.

REST API Service Integrations

As mentioned above, service integrations (meaning API Gateway invokes an AWS service directly without Lambda), follow the same pattern as custom integrations. You will need to take the same steps to enable CORS. Although I don't talk about CORS, I do have an article on integrating DynamoDB with REST API. There isn't a lot of information in the official docs on how to build out service integrations or even what services can be integrated with in this way, but think of it as API Gateway being configured to make SDK calls and you'll be somewhere in the vicinity of figuring this out.

HTTP API Service Integrations

Continuing the pattern we've seen throughout, HTTP API service integrations are simpler, but more limited. The docs limit to just five different services you can integrate with at the time of this writing. The good news if you want to use one of these integrations and need to support CORS is you'll do the exact same thing you needed to do for an HTTP API Lambda proxy integration.

Conclusion

It's not too hard to navigate any one of these solutions, but it might be hard to keep them straight and the documentation isn't always clear. It's helpful to be able to correlate the implementation with the API choices you're making. Once we have that down, we can stop the churn and appreciate the gains we get from this service.

I'm interested in covering authorization and request/response validation in future articles. Feel free to drop a comment if there's any topic you'd like to see explored at depth!

COVER: Published by Actualités Théâtrales J.M. (Paris) ca. 1880–1890, Public domain, via Wikimedia Commons

AWS CDK - Fullstack Polyglot with Asset Bundling

Matt Morgan — Wed, 24 Mar 2021 11:24:37 +0000

This is my third article that deals with asset bundling in AWS CDK. For more context, you may also be interested in:

Asset management is one of the better features of CDK. We provide a path on the local filesystem and CDK will automatically stage the files in S3 and produce valid CloudFormation to use the files in Lambda, an S3 website, or any other use I may have for assets. The ability to bundle application code with infrastructure and push them together is a superpower and beats the heck out of anything that requires an s3 sync as an extra step.

Even better is when we let CDK handle the bundling internally. Not only does this make for easy one-step builds, but we can also set rules for managing change. We can tell CDK how to diff our output, thereby ensuring updates are only made when necessary. This will save enormous amounts of time on larger stacks.

I've covered use cases for TypeScript applications in Lambda and S3 websites in the articles linked above. In this article I'll explore how to add a function written in Go to a TypeScript CDK stack and gain all the same benefits we enjoy using a high-level construct like aws-lambda-nodejs.

Why Go?
Asset Hash
Local Bundling
Docker Bundling
BYO Dockerfile
Conclusion

Why Go?

I'm going to focus on running a build for a Lambda function written in Go in this post, but many of the same techniques could be leveraged for any kind of asset bundling. Go is a language that compiles to a binary while TypeScript compiles to JavaScript, so we're dipping our toes in very different worlds here. I've only recently begun to properly learn Go (thanks Cloud Academy), so I've got a mix of googled hacks and assumed best practices. That's good enough to bundle into a CDK project.

I used Go for a recent exploration of Open Policy Agent in my rego.fyi website. I'm going to show snippets from that app. The full repo can be found here.

Asset Hash

Let's start with the best part first. The AssetOptions interface has three optional properties: assetHash, assetHashType and bundling. In fact, giving the asset options is optional. We can simply write:

Code.fromAsset('/path/to/my/stuff');

This will add the specified path as an asset. It isn't modified at all and the input path will be used to detect changes. The above code can be part of a Lambda function. Example:

import { Code, Function as LambdaFunction, Runtime } from '@aws-cdk/aws-lambda';
import { join } from 'path';

new LambdaFunction(this, 'myFn', {
  code: Code.fromAsset(join(__dirname, 'handler')),
  runtime: Runtime.NODEJS_14_X,
  handler: 'index.handler'
});

or it could just be an asset you want uploaded to S3:

import { Asset } from '@aws-cdk/aws-s3-assets';
import { join } from 'path';

new Asset(this, 'MyAsset', {
  path: join(__dirname, 'myfile.txt')
});

There's obviously a difference here. Code.fromAsset takes the path as the first arg, then has an optional AssetOptions argument while new Asset is declared as a standalone construct that requires a scope and an id. The Asset construct takes a required third argument of AssetProps which extends AssetOptions but adds the required path property.

But wait there's more! You might be bundling for an S3 website.

import { BucketDeployment, Source } from '@aws-cdk/aws-s3-deployment';
import { join } from 'path';

new BucketDeployment(stack, 'DeployWebsite', {
  destinationBucket: new Bucket(stack, 'WebsiteBucket', {
    autoDeleteObjects: true,
    publicReadAccess: true,
    removalPolicy: RemovalPolicy.DESTROY,
    websiteIndexDocument: 'index.html',
  }),
  distribution,
  distributionPaths: ['/*'],
  sources: [Source.asset(join(__dirname, '../path/to/asset'))],
});

Or maybe you're building a Docker image?

import { DockerImageAsset } from '@aws-cdk/aws-ecr-assets';
import { join } from 'path';

new DockerImageAsset(this, 'MyBuildImage', {
  directory: join(__dirname, 'my-image')
});

Seems confusing? It's actually not that bad. We have a common interface used for building Lambda functions, miscellaneous assets, S3 websites, Docker images and anything else you may wish to transform in a build pipeline. Now that that's clear, let's dig into that AssetOptions interface.

The assetHash property would be appropriate if you wanted to generate your own hash instead of letting CDK manage that for you. Most of the time you won't want to use this, but it's good it's there if you need it.

I'm more interested in assetHashType. We get a couple of options to consider: OUTPUT and SOURCE. There is also BUNDLE, deprecated by OUTPUT, and CUSTOM, which just means assetHash is provided. Between OUTPUT and SOURCE, SOURCE is the default, but I think I prefer OUTPUT. If my inputs change (perhaps a library bump) but my output doesn't (that library bump had no impact on the bundled code), then do I really want to deploy new code?

I'll go into another reason for preferring OUTPUT later in the article. The takeaway here is we have a lot of flexibility to hash the input, output or provide our own hash. That's great because controlling this feature will make deployments faster. We'll spend less time staring at progress bars and we'll also be able to respond more quickly to operational issues. This will speed up our dev cycle as well. It's a great feeling running a stack with 11 Lambda Functions, making a change to just one of them and then having a very quick deployment as CDK identifies the one that changed and deploys an update only for that one.

Local Bundling

Let's move onto the final optional property of AssetOptions, bundling. We're going to need to use this if we want to run any kind of build or compilation process on our code. We could run that as a separate process and then use the output of that process as our asset, but doing it as part of the CDK build is much better.

The default for bundling is to use Docker, even if you aren't aiming to produce an image. This makes sense for some purposes and will cause problems for others. If our build environment runs a different operating system than our production environment and we are compiling binaries, we may need Docker builds. On the other hand, we might be building in an environment that doesn't run Docker. We could even be building in a Docker environment where spawning a new container from the current one (Docker-in-Docker) is not allowed!

Let's start from a place where we do not want to use Docker. Unfortunately we can't just leave Docker out entirely. The image property of bundling is required. Instead let's just let the user know that we aren't supporting a Docker build.

new LambdaFunction(stack, 'MyFn', {
    code: Code.fromAsset(myPath, {
      assetHashType: AssetHashType.OUTPUT,
      bundling: {
        command: ['sh', '-c', 'echo "Docker build not supported. Please install go."'],
        image: DockerImage.fromRegistry('alpine'),
      },
    }),
    handler: 'main',
    runtime: Runtime.GO_1_X,
  });

If local bundling fails, now we at least have a meaningful error message to the user. Of course the above code will fail every build, so we have more work to do. Let's add local bundling. The local property of bundling must implement ILocalBundling, an object with one method, tryBundle that receives the outputDir as an argument and must return a boolean type. If tryBundle returns true and some code artifact is found in outputDir, then CDK will see the build as successful and skip the Docker build. tryBundle is synchronous and will not resolve promises so we need a strategy for running the build synchronously and accurately reporting whether or not it was successful.

To synchronously run our build, we'll use execSync from the nodejs child_process module. tryBundle should implement try/catch to ensure it returns a boolean about the state of our build. It's common practice to execute some kind of test to see if the runtime supports our build. We could use go version to see if our runtime can support a build in Go.

import { Code, Function as LambdaFunction, Runtime } from '@aws-cdk/aws-lambda';
import { AssetHashType, DockerImage, Stack } from '@aws-cdk/core';
import { execSync, ExecSyncOptions } from 'child_process';
import { join } from 'path';

const execOptions: ExecSyncOptions = { stdio: ['ignore', process.stderr, 'inherit'] };
const goPath = join(__dirname, '..');

new LambdaFunction(stack, 'MyFn', {
    code: Code.fromAsset(goPath, {
      assetHashType: AssetHashType.OUTPUT,
      bundling: {
        command: ['sh', '-c', 'echo "Docker build not supported. Please install go."'],
        image: DockerImage.fromRegistry('alpine'),
      },
      local: {
        tryBundle(outputDir: string) {
          try {
            execSync('go version', execOptions);
          } catch {
            return false;
          }
          execSync(`GOARCH=amd64 GOOS=linux go build -ldflags="-s -w" -o ${join(outputDir, 'main')}`, {
            ...execOptions,
            cwd: join(goPath, 'fns/go'),
          });
          return true;
        }
      }
    }),
    handler: 'main',
    runtime: Runtime.GO_1_X,
  });

I've included some options for standard output from execSync that work well for me on my terminal. For this project, I've structured things in such a way that my go.mod and go.sum are in the root of my project and my go source is in fns/go. Because of that, I need to set the path of the build to the project root, then run the build command in fns/go.

For the uninitiated, go.mod and go.sum are for dependency management using go modules. This is not completely unlike package.json and package-lock.json. I'm not crystal clear on best practices myself, but it does seem like these belong in the root. It is also typical to put a main.go file in the root of a project, but that felt wrong for a polyglot project. By putting my dependency files in the root and my function code in directory parallel to fns/ts (for TypeScript), I'm able to run my tests from the root of the project with go test ./..., while organizing my project in a way that makes sense to me.

This is why setting AssetHashType to OUTPUT is so important, because I'm actually setting my source to the root of the project and I do not want to use the entire project as change detection for this one function.

In summation, I've organized my project in a way that makes sense to me and seems efficient. Is it the best practice? You tell me. I spent some time searching for answers and couldn't come up with one.

A couple of other things to mention here about the build. Using local bundling, I very well may be running a build in an environment (MacOS in this case) that is different than my execution environment (Lambda, which is Amazon Linux). If I don't run a build that targets my environment, it's not going to work. Fortunately Go has me covered and by adding the GOARCH=amd64 GOOS=linux flags to my build, I'll target the right architecture and OS. As for ldflags, I admit that's a copy-paste. I read the docs and I'm still not sure why I'm doing it! Maybe it'll dawn on me some day.

Finally, I do have the option of making tryBundle a little more concise. Since execSync will correctly throw an error if my build fails, I could skip go version and just write:

tryBundle(outputDir: string) {
  try {
    execSync(`GOARCH=amd64 GOOS=linux go build -ldflags="-s -w" -o ${join(outputDir, 'main')}`, {
      ...execOptions,
      cwd: join(goPath, 'fns/go'),
    });
    return true;
  } catch {
    return false;
  }
},

It's just a matter of taste here. Do you like the go version being output?

Docker Bundling

Docker bundling could well be appropriate if we don't want to install Go on the build system. Using Docker builds might allow a polyglot app to be checked out and worked on by developers who don't use every language in the app. It might also work out well for apps written in languages that have many different versions available that could be on contributors' workstations. The Docker build could pin us to specific languages and versions. Let's see how that looks.

We have already provided the command and image arguments to shy users away from Docker. Now we'll flip it around and use those commands to run a Docker build. First we need a base image other than alpine. CDK is able to provide a suggestion here. The Runtime constant exposes a bundlingDockerImage for each available runtime.

bundling: {
  command: ['sh', '-c', 'echo "Need a command"'],
  image: Runtime.GO_1_X.bundlingDockerImage,
}

For the NodeJS, Python, Java and some DotNet runtimes, this image will point to amazon/aws-sam-cli-build-image-* images, that are now maintained by the AWS SAM project. For whatever reason, the Go image still (at the time of this writing) uses lambci/lambda:build-go1.x from Michael Hart's lambci project that originally inspired SAM. This image is perfectly good for my purposes, but by using a CDK constant, I do run the risk that some day this changes to another image, so I should keep an eye on that.

The way the Docker build works here is CDK will automatically map a volume of my build path (specified as the first argument of fromAsset) to /asset-input in the container. My build is expected to produce output to /asset-output in order to be considered successful.

As I noted above, my dependency files (go.mod and go.sum) are in the root of my project, so I need to set the input to the root of my project and the entire project is shared as volume. This is a little inefficient, but still relatively fast. The entire build is taking less than 20 seconds on my MBP.

To make this work, we'll set the user and workingDirectory properties. For workingDirectory, we can map right into the directory with the go files /asset-input/fns/go and save ourselves a clumsy cd in the command. We'll also need to specify the user as root in order to create the go cache. For those experienced in Docker, this will stick out as a problem, but running a build as root isn't a big deal so long as we aren't running a production workload as root.

Finally we set the command to basically the same thing we had for local bundling and now we have a working build!

bundling: {
  command: ['sh', '-c', 'GOARCH=amd64 GOOS=linux go build -ldflags="-s -w" -o /asset-output/main'],
  image: Runtime.GO_1_X.bundlingDockerImage,
  user: 'root',
  workingDirectory: '/asset-input/fns/go',
},

Specifying the architecture and OS may not be necessary since we should be building in a Lambda-friendly environment, but it also doesn't hurt. Notice the build outputs directly to /asset-output/main. From there, CDK picks up the artifact and stages it for upload. You'll be able to see it under cdk.out.

BYO Dockerfile

If you're at all picky about your Docker builds, you probably want to provide your own Dockerfile. Of course you can! This is most emphatically not an article on Docker best practices, but I took at shot at building a Dockerfile that ticks at least most of the boxes.

FROM lambci/lambda:build-go1.x

ENV APP_USER app
ENV APP_HOME /usr/app
ENV ASSET_DIR /asset
ENV GOPATH $APP_HOME/go
ENV GOARCH=amd64 GOOS=linux 
RUN groupadd $APP_USER && useradd -m -g $APP_USER -l $APP_USER
RUN mkdir -p $APP_HOME && chown -R $APP_USER:$APP_USER $APP_HOME
RUN mkdir -p $ASSET_DIR && chown -R $APP_USER:$APP_USER $ASSET_DIR
WORKDIR $APP_HOME
USER $APP_USER

COPY go.mod go.sum ./

RUN go mod download && go mod verify

COPY /fns/go/authorizer.go ./

RUN go build -ldflags="-s -w" -o $ASSET_DIR/main

I'm using a non-root user, caching dependencies and explicitly using COPY instead of sharing the volume. Is that perfect? The point is you can build your own according to whatever practices you follow and CDK isn't going to silently share volumes or try to run as a particular user.

Implementing this Dockerfile is pretty easy too. We can simply do this:

new LambdaFunction(stack, 'AuthZFun', {
  code: Code.fromDockerBuild(join(__dirname, '..')),
  handler: 'main',
  runtime: Runtime.GO_1_X,
});

One gotcha here is that you must use an absolute path to your Dockerfile. I don't know why this is. It seems like a bug, as Docker can certainly build off a relative path, but when I try it my build fails with GlobIgnoreStrategy expects an absolute file path. Just use the full path and you'll be fine.

What about combining local bundling and a fallback to a Dockerfile? Sure, we can support that!

new LambdaFunction(stack, 'AuthZFun', {
  code: Code.fromAsset(goPath, {
    assetHashType: AssetHashType.OUTPUT,
    bundling: {
      command: ['sh', '-c', 'cp /asset/main /asset-output/main'],
      image: DockerImage.fromBuild(join(__dirname, '..')),
      local: {
        tryBundle(outputDir: string) {
          try {
            execSync('go version', execOptions);
          } catch /* istanbul ignore next */ {
            return false;
          }
          execSync(`GOARCH=amd64 GOOS=linux go build -ldflags="-s -w" -o ${join(outputDir, 'main')}`, {
            ...execOptions,
            cwd: join(goPath, 'fns/go'),
          });
          return true;
        },
      },
      workingDirectory: '/usr/app',
    },
  }),
  handler: 'main',
  runtime: Runtime.GO_1_X,
});

Using the same Dockerfile, we now need to set our workingDirectory to /usr/app, as specified by the Dockerfile. Otherwise, it'll default to /asset-input. Unfortunately this method will still do the (here unneeded and unnecessary) volume share, but we can dead-end that by ignoring the share in our Docker build. Note that we do need this command to copy our build from /asset/main to /asset-output/main. For one thing, the output dir isn't configurable in this build and for another, because the output is being delivered via another Docker share, just running the build won't do anything, even if the we output to /asset-output/main during the Docker build. That's because the volume share doesn't happen until the built image is run with that command.

Conclusion

All this said and done, I still prefer local builds and focus on those in my own work, but it's definitely useful to understand the other builds that are available here. I had a tough time figuring out some of the ins and outs of managing Docker builds.

I've said before that I find the API for bundling a little less intuitive than it needs to be, but the output is excellent. I can make a change to my stack, say, modifying the API Gateway throttlingRateLimit. Then when I deploy, I'm treated to something like this:

regofyi: creating CloudFormation changeset...
[··························································] (0/2)

8:43:31 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack           | regofyi

Even though all my code was re-bundled, CDK correctly determined the build output hadn't changed, so it wasn't included in the changeset and I wound up with a really simple deployment.

COVER IMAGE

rego.fyi: A Study in Serverless Authorization with Open Policy Agent

Matt Morgan — Mon, 08 Mar 2021 12:12:58 +0000

Open Policy Agent is a decision engine built on a declarative language called Rego. OPA is pronounced "oh-pa", but to na kang setóp da mesach! so go ahead and say "oh-pee-ayy" because I know you want to. OPA is general-purpose and written in Go. It allows the creation and compilation of policies written in Rego which can then be hydrated with some kind of data source and then compared against an input to produce a result. OPA docs, linked above, are excellent so check them out for more information.

tldr
Why OPA?
Serverless OPA
rego.fyi
Rego Policy
OPA Authorizer
Web App
AWS CDK
Conclusion

tl;dr

The code is here!
rego.fyi

Why OPA?

The paradigm presented by OPA is compelling when applied to the use case of multi-tenant SaaS applications. Such applications have complex authorization rules which may include ensuring a user acts within their tenant or personal data, ensuring a user has the right role or permission, ensuring the user belongs to a tenant that is subscribed to the service being provided and many others. Often these rules are implemented in imperative logic throughout the application. A check for subscription may be implemented in middleware while limiting the scope of tenant access is often found in the WHERE clause of a SQL query. Spreading authorization concerns throughout an application makes it very hard to audit and understand what rules the application are actually enforcing and it makes it easy for bugs to creep in.

The more complex the application, the more authorization becomes a problem that needs a single solution. This is the problem OPA can solve. It's enticing to think about authorizing a microservice architecture with OPA. This would allow developers to focus on the problem the service needs to solve and share a common authorization abstraction.

Serverless OPA

The usual way to implement OPA for microservices is to stand up an authorization service implementing OPA and have other services invoke it over http using some of the published middleware. Even the OpenFaaS version written in Go depends on a standalone authorization service. I wanted to see if I could use OPA in a 100% serverless environment, making policy decisions in an API Gateway request authorizer without the overhead of additional http requests or the need to run a separate service. I found inspiration in this excellent sls-lambda-opa repo.

I think of this solution as a layered architecture, where the bottom layer is the authorizer implementing the OPA library, capable of compiling Rego policies. On top of that is the actual policy that states I want to compare a claim like permissions or subscriptions or I'm interested in the HTTP resource and method. Above that is the service or endpoint-specific data that states the actual resources, methods and subscriptions that will be evaluated. Then finally we have the user's session or context, delivered in a JSON Web Token (or JWT).

OPA sandwich?

Each of these layers can be decoupled from the others. An authorizer function implementation might be used across several services with the same policy but different data hydrating the policy.

rego.fyi

My first attempt at putting all this together was a fairly terrible demo that needed a REST client and the copying of tokens. I had the urge to build a little fullstack app and so I came up with rego.fyi. In some ways, it's a less impressive take on The Rego Playground, but mine is serverless and built on the kind of architecture I want to work with. I'll also caveat that I am absolute trash when it comes to visual design. I'm just awful at it. Respect to those who are good, but I'm not one of you.

The architecture of my app is a little different than what I'd envision using in production. I wanted to be able to experiment with different policies, so the policies, along with data and the user input are all sent to be evaluated by my authorizer function. The authorizer compiles the policy in real time, makes the policy decision based on the data and user input, then returns an IAM policy document specifying whether my Lambda function can be invoked, as appropriate.

In a real application, I likely wouldn't want to compile the policy on request, but instead compile it once on startup. I don't have the expectation of needing to change policies on the fly, though if I did, the policy could be loaded from S3 or a database. What I would probably do in a real application is load the policy from a Lambda Layer.

Rego Policy

I am by no means an expert on the rego language and won't give an overview here when there are already useful docs. I did manage to put together a workable policy for my experiment.

package policy

import data.requests
import data.permissions
import data.subscriptions

default allow = false

allow {
    check_policy[input]
}

check_policy[input] {
    r = requests[_]
    some i; match_with_wildcard(permissions, input.permissions[i])
    some j; match_with_wildcard(subscriptions, input.subscriptions[j])
    match_with_wildcard(r.methods, input.method)
    match_with_wildcard(r.resources, input.resource)
}

match_with_wildcard(allowed, value) {
    allowed[_] = "*"
}
match_with_wildcard(allowed, value) {
    allowed[_] = value
}

The syntax of this policy is explained well in the docs, but just to call out a few things, everything in check_policy can be considered an AND comparison while the duplicative call signature of match_with_wildcard makes it an OR comparison. some i; match_with_wildcard(permissions, input.permissions[i]) is a fairly elegant one-liner that makes sure one item in the left-side array matches at least one item in the right-side array.

Note the policy defines fields I care about, namely the HTTP method and resource as well as my custom claims of permissions and subscriptions. This policy doesn't include anything about the values that should be compared to, but does describe the shape of the data and how it should be compared. In order to give those values, we need a data file.

{
  "requests": [{ "methods": ["GET"], "resources": ["/orders"] }],
  "permissions": ["start_order", "view_invoice"],
  "subscriptions": ["newsletter"]
}

This data file could apply to one API while another one with permissions like cuddle_hedgehogs or introspect_navel protects another one. That's the power of this layered approach! But the best part is rego ships with a testing framework.

The best way to experience a good separation of concerns is with some solid unit tests. They are quite easy to write.

package policy

test_get_allowed {
    allow with input as {"permissions":["start_order"], "resource": "/orders", "method":"GET", "subscriptions":["newsletter"]}
}

test_get_wrong_subcription_denied {
    not allow with input as {"permissions":["start_order"], "resource": "/orders", "method":"GET", "subscriptions":["pizza_of_the_month"]}
}

test_get_wrong_permission_denied {
    not allow with input as {"permissions":["change_password"], "resource": "/orders", "method":"GET", "subscriptions":["newsletter"]}
}

I'm providing the different user inputs and expecting them to either be allowed or not. Rego also gives me test coverage out of the box!

% opa test . -c
{
  "files": {
    "policy.rego": {
      "covered": [
        {
          "start": {
            "row": 7
          },
          "end": {
            "row": 7
          }
        },
        {
          "start": {
            "row": 9
          },
          "end": {
            "row": 10
          }
        },
        {
          "start": {
            "row": 13
          },
          "end": {
            "row": 18
          }
        },
        {
          "start": {
            "row": 22
          },
          "end": {
            "row": 22
          }
        },
        {
          "start": {
            "row": 24
          },
          "end": {
            "row": 25
          }
        }
      ],
      "not_covered": [
        {
          "start": {
            "row": 21
          },
          "end": {
            "row": 21
          }
        }
      ],
      "coverage": 92.3
    },
    "policy_test.rego": {
      "covered": [
        {
          "start": {
            "row": 3
          },
          "end": {
            "row": 4
          }
        },
        {
          "start": {
            "row": 7
          },
          "end": {
            "row": 8
          }
        },
        {
          "start": {
            "row": 11
          },
          "end": {
            "row": 12
          }
        }
      ],
      "coverage": 100
    }
  },
  "coverage": 94.75
}

Okay, that's a bit verbose and looks like I need another test, but still really useful. There's a --format=pretty option, but it doesn't seem to do anything. Perhaps it's a WIP.

Anyway, this is great! Our old applications with some of the authorization logic in middleware, some in SQL and some in between just can't compete with the ability to unit test the policy logic separate from any application code.

OPA Authorizer

The only difference between a RequestAuthorizer and a TokenAuthorizer is the TokenAuthorizer only sees the specified token while a RequestAuthorizer sees the entire request. This is a better fit, since I want to look at things like the HTTP method and path.

My authorizer function needs to:

Unpack my "token" (which really consists of the policy, data and user token, base64 encoded for demo purposes)
Compile the policy with the provided data.
Compare the user input and request to the policy.
Return an appropriate IAM policy to allow or deny access to the function handler.

This authorizer needs to be written in Go because the supporting OPA libraries are only available to the Go runtime. I normally write TypeScript, but this was my second try at Go and I think I did okay, thanks to countless examples and tutorials across the Internet. In fact, it's fair to say that imitation is a sincere form of flattery.

This was also the first time I've used Go with Lambda and I must say, this might be addicting.

Millisecond billing, yeah!

I'm doing around 50ms with cold starts and single-digits otherwise, even though I'm doing all of those things (unpacking, compiling, deciding, generating) on every request. If the policy is compiled at start time, it's even faster.

Web App

My poor design skills notwithstanding, I'm reasonably good at programming in React. I built off a previous effort to do a simple fullstack non-CRA* React app with esbuild. I think it works pretty well.

In order to keep my app from looking like complete trash, I used Material-UI from Google for the components and that was pretty easy. I also delved into Testing Library and I find it quite nice and loved that I could write tests without having to render anything shallowly. I used React Context for state and I like that a lot better than working with Redux. You can check my code or query about this in the comments as I'm not going to do into great detail here, but as someone who doesn't program in React every day, it's nice checking in and seeing these innovations.

The app is just one page with no routing. When thinking about how to do this app, I actually thought about trying to figure out some kind of client-side JWT or perhaps do a round trip to a backend to get a user token. Ultimately I decided that cryptographic signing is beyond the scope of what I wanted to do in this app, so all it does is stringify the various form fields, base64 encode that string and finally pass the whole thing as an Authentication header. This allows me to have a decoding piece in my authorizer but of course it's in no way secure.

AWS CDK

I love working with CDK. My post on CDK S3 websites covers most of topics relating to asset bundling, but it's worth remarking on here. My CDK app handles all the asset bundling of my React web app (written in TypeScript), my authorizer function (written in Go) and my function handler (written in TypeScript). Because of the way asset bundling works in CDK, I can safely cdk deploy and only the parts of the application that have changed will deploy. This is great for a fullstack application and makes deployments very fast. I'll dig into this topic a bit more in a future post.

I also was able to get a unified npm test command that runs all the tests for:

CDK infrastructure as code written in TypeScript
React web app written in TypeScript
Lambda function handler written in TypeScript
Authorizer function written in Go
Policy written in Rego

% npm t

> cdk-esbuild-s3-website@0.0.1 pretest /Users/mattmorgan/mine/rego.fyi
> npm run lint


> cdk-esbuild-s3-website@0.0.1 lint /Users/mattmorgan/mine/rego.fyi
> eslint . --ext=.js,.ts


> cdk-esbuild-s3-website@0.0.1 test /Users/mattmorgan/mine/rego.fyi
> npm run test:opa && npm run test:go && npm run test:ts


> cdk-esbuild-s3-website@0.0.1 test:opa /Users/mattmorgan/mine/rego.fyi
> opa test ./opa

PASS: 3/3

> cdk-esbuild-s3-website@0.0.1 test:go /Users/mattmorgan/mine/rego.fyi
> go test ./...

ok      _/Users/mattmorgan/mine/rego.fyi/fns/go 1.222s

> cdk-esbuild-s3-website@0.0.1 test:ts /Users/mattmorgan/mine/rego.fyi
> jest --coverage --silent

 PASS   node  fns/ts/lambdalith.spec.ts
 PASS   dom  ui/providers/PayloadsProvider.spec.tsx
Bundling asset Default/AuthZFun/Code/Stage...
go version go1.16 darwin/amd64
Bundling asset WebTestStack/DeployWebsite/Asset1/Stage...
Bundling asset ApiTestStack/AuthZFun/Code/Stage...
0.8.56
go version go1.16 darwin/amd64
Bundling asset TestStack/DeployWebsite/Asset1/Stage...
0.8.56

> cdk-esbuild-s3-website@0.0.1 build /Users/mattmorgan/mine/rego.fyi
> npm run clean && npm run build:website


> cdk-esbuild-s3-website@0.0.1 build /Users/mattmorgan/mine/rego.fyi
> npm run clean && npm run build:website

 PASS   dom  ui/components/Header.spec.tsx
Bundling asset Default/LambdalithFn/Code/Stage...
 PASS   dom  ui/components/TextArea.spec.tsx

> cdk-esbuild-s3-website@0.0.1 clean /Users/mattmorgan/mine/rego.fyi
> rimraf cdk.out coverage website/js


> cdk-esbuild-s3-website@0.0.1 clean /Users/mattmorgan/mine/rego.fyi
> rimraf cdk.out coverage website/js

Bundling asset ApiTestStack/LambdalithFn/Code/Stage...
 PASS   node  cdk/lambda.spec.ts
 PASS   dom  ui/components/RequestControl.spec.tsx

> cdk-esbuild-s3-website@0.0.1 build:website /Users/mattmorgan/mine/rego.fyi
> NODE_ENV=production ts-node --files esbuild.ts build

 PASS   dom  ui/App.spec.tsx

> cdk-esbuild-s3-website@0.0.1 build:website /Users/mattmorgan/mine/rego.fyi
> NODE_ENV=production ts-node --files esbuild.ts build

 PASS   node  cdk/restApi.spec.ts
Running build...
Running build...
Bundling asset TestStack/AuthZFun/Code/Stage...
 PASS   node  cdk/website.spec.ts (7.308 s)
go version go1.16 darwin/amd64
Bundling asset TestStack/LambdalithFn/Code/Stage...
 PASS   node  cdk/rego.fyi-stack.spec.ts (8.261 s)
-----------------------|---------|----------|---------|---------|-------------------
File                   | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
-----------------------|---------|----------|---------|---------|-------------------
All files              |     100 |      100 |     100 |     100 |
 cdk                   |     100 |      100 |     100 |     100 |
  getCFAndZone.ts      |     100 |      100 |     100 |     100 |
  lambda.ts            |     100 |      100 |     100 |     100 |
  rego.fyi-stack.ts    |     100 |      100 |     100 |     100 |
  restApi.ts           |     100 |      100 |     100 |     100 |
  website.ts           |     100 |      100 |     100 |     100 |
 fns/ts                |     100 |      100 |     100 |     100 |
  lambdalith.ts        |     100 |      100 |     100 |     100 |
 opa                   |     100 |      100 |     100 |     100 |
  policy.rego          |     100 |      100 |     100 |     100 |
 ui                    |     100 |      100 |     100 |     100 |
  App.tsx              |     100 |      100 |     100 |     100 |
 ui/components         |     100 |      100 |     100 |     100 |
  Header.tsx           |     100 |      100 |     100 |     100 |
  RequestControl.tsx   |     100 |      100 |     100 |     100 |
  Sidebar.tsx          |     100 |      100 |     100 |     100 |
  TextArea.tsx         |     100 |      100 |     100 |     100 |
 ui/pages              |     100 |      100 |     100 |     100 |
  Rego.tsx             |     100 |      100 |     100 |     100 |
 ui/providers          |     100 |      100 |     100 |     100 |
  PayloadsProvider.tsx |     100 |      100 |     100 |     100 |
-----------------------|---------|----------|---------|---------|-------------------

Test Suites: 10 passed, 10 total
Tests:       27 passed, 27 total
Snapshots:   6 passed, 6 total
Time:        8.804 s, estimated 9 s

Thanks to jest projects, I'm able to run my .tsx tests with the jsdom test environment and the .ts tests with node. Fullstack testing!

Conclusion

My success at getting this working as a RequestAuthorizer makes a very compelling case for introducing OPA into serverless projects. We can join the delegation of complex authorization rules with API Gateway request validation and find ourselves in a world where our function handlers are very simple - or maybe even skip them entirely.

COVER IMAGE

DEV Community: Matt Morgan

Create Stateful Serverless Workflows with AWS Step Functions and JSONata

JSONata

JSONata in Step Functions

Passing Data

The Assign Directive

Comparing JSONPath with JSONata

JSONata in the AWS Console

Migrating to JSONata

What Else Does JSONata Offer?

One Last Thing

Presenting AWS Speakers Directory, an AI Hackathon Project

Related Posts

Community Speakers Directory

What We Built

CodeCatalyst

Flutter

AppSync / Merged API

AWS Organization

Image Generation

AI-assisted Tagging and Recommendations

DynamoDB Single Table Design

Roadmap

Conclusion

AWS Users, Roles, and Identity Center Demystified

IAM Policies

IAM Principals

Root Users

IAM Users

Roles

IAM User vs Identity Center User

Comparing Users and Roles

Console Access

With an IAM User

With IAM Identity Center (and Roles)

Command-Line Access

With an IAM User

With IAM Identity Center (and Roles)

Third Party Machine Access

With an IAM User

With Roles

Service Access

With an IAM User

With a Role

Conclusion

Avoiding the Serverless Workflow Antipattern

Asynchronous Event Handler

Fan-Out with SQS

Fan-Out with Job State Tracking

Enter Step Functions

Learning Step Functions

You Also Get...

Know and Plan Your Architecture

More on Step Functions

Appreciation for Community Managers

What surprises you most about the community builders program?

What’s your background and your experience with AWS?

What’s the biggest benefit you see from the program?

What’s the next swag item that you would like to get?

What are you eating for dinner today? Share the recipe!

Is there anything else you would like to say about the community builders program in 2022?

Lambda Powertools TypeScript is Generally Available

Table of Contents

What's Changed

ES Modules Support

Comparisons

Logger

Metrics

Tracer

Roadmap

Conclusion

How to Use Source Maps in TypeScript Lambda Functions (with Benchmarks)

Table of Contents

Stack Traces

Emitting Source Maps

Source Map Support

CDK Example

Serverless Stack

SAM Example

Architect