DEV Community: Andres Moreno

Step Functions without ASL? Welcome Lambda Durable Functions

Andres Moreno — Wed, 17 Dec 2025 00:05:00 +0000

During re:Invent, AWS announced a new feature within AWS Lambda called durable functions. These are the same Lambda functions we all love, but they let you run multi-step workflows by keeping checkpoints and state. What does this mean? You can run similar functionality to what we’ve typically used AWS Step Functions for. But instead of using Amazon State Language, you can use familiar code and dependencies.

Terms and Concepts

Let’s go over a few concepts around durable functions.

Durable Function - It’s a regular Lambda function that can pause and resume by making use of a checkpoint and replay mechanism.
Checkpoint - When the function executes a step or needs to wait for a callback, it will add a checkpoint that persists the current state and stops its execution.
Replay - Once the workflow is ready to resume execution, it will pick up from the last checkpoint rather than running the whole workflow again.
Durable Execution - The complete lifecycle of a durable function. From the moment it is triggered until it completes all defined steps and is closed.
Durable Context - The context that is provided to the Lambda handler. This contains the methods for the durable operations.
Durable Operations - These are the operations that allow us to create a workflow within a Lambda function. Each step has built-in retries and automatically creates checkpoints when it runs. Let’s go over each of the operations at a high level.
- Steps - Run business logic and are defined by using the context.step() operator.
- Wait States - Planned pauses that cause the function to stop running until they are resumed. This can be used to wait for a specific amount of time, for an external callback, or for other specific conditions. These are defined using context.wait() , context.waitForCallback() or context.waitForCondition().
- Parallel - There will be situations where you want to optimize for speed and run things concurrently to reduce execution time. The parallel operation is used for that, and is defined by using the context.parallel() operator.
- Iterations - To process an array of items in a loop, you will have to use the context.map() operation.
- Invoke other Lambda functions - You can invoke external Lambda functions from within a workflow. To do this, you will call the context.invoke() operator.
- Nested Operations - To run operations as a child of another operation, you will call context.runInChildContext().

How to create a Lambda durable function

This is probably the best thing about all of this. Most of the setup is done the same way as a regular Lambda function, BECAUSE IT’S THE SAME RESOURCE!. To enable a durable context for the function, we need to make one change to our IaC and another to the code handler. Let’s go over these.

Infrastructure Setup

To set it up in SAM, all you need to do is set the DurableConfig object.

  ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      CodeUri: workflows/example/
      DurableConfig:
        ExecutionTimeout: 10
        RetentionPeriodInDays: 1

Let’s go over these two properties in more detail:

ExecutionTimeout - This is the total time, in seconds, for a full durable execution. This can be up to a full year.
RetentionPeriodInDays - The number of days the durable execution history is retained after it is closed.

Code Handler Setup

To make this simpler for us, AWS has provided a new SDK called @aws/durable-execution-sdk-js. This contains everything that you’ll need to build workflows in a Lambda function. But right now, the only thing we need to have a durable context is to wrap our handler with the withDurableExecution function.
Yes, that is all!

import { withDurableExecution } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(async (event, context) => {
  // Your handler code
})

With those things in place, we can now define the workflow.

Durable Operations

Now I want to go through the different durable operations in detail and show them in a working example. If you want to follow along, I have everything defined and deployable in this GitHub location.

Step

At its most basic level, we have the step. This will create a checkpoint after it’s run, and you can define any logic you want within the step. I recommend splitting the actual business logic outside of the workflow definition to keep it clean and readable. This makes it easy to follow when there are issues with your workflow, and you can find the step that broke.

  // Step 1: Initial step operation - process input data
  const workItems = await context.step('processInputData', async () => {
    return processData(event.inputData);
  });

Human in the loop

Whenever you want to wait for a human action, you will use the waitForCallback operation.

  // Step 2: Wait for callback operation - pause for external event
  // Wait for external callback with timeout handling
  const callbackResult = await context.waitForCallback(
    "wait-for-external-callback",
    async (callbackId, ctx) => {
      // Submit callback ID to external system (simulated)
      ctx.logger.info(`Callback ID ${callbackId} submitted to external system`);
      // In real implementation, this would call an external API
      // await submitToExternalAPI(callbackId);
    },
    { timeout: { minutes: 60 } } // 1 hour timeout
  );

This will give us a callbackId that we can use to resume our workflow. When testing, you can resume the workflow from the AWS Console, but in real life, you will need to do this programmatically. Thankfully, the SDK provides the SendDurableExecutionCallbackSuccessCommand and SendDurableExecutionCallbackFailureCommand in the Lambda client, which can be used to resume the workflow. You can also specify a timeout so the workflow does not wait forever.

Once the command is sent the workflow will resume and continue processing.

Wait

If you ever need to wait a specific amount of time, you can use the wait operation, which lets you specify the amount of time you want to wait.

  // Step 3: Simple wait operation - demonstrate time-based wait
  await context.wait({ seconds: 5 }); // Wait for 5 seconds

Parallel Processing

When you have several operations that can be handled independently, you might want to run them in parallel to reduce the total processing time. That’s where the parallel operation comes in.

  // Step 4: Parallel operations - process multiple work streams concurrently
  const parallelResults = await context.parallel([
    async (ctx) => ctx.step('parallelTask1', async () => {
      return await performDataValidation(workItemsCount);
    }),
    async (ctx) => ctx.step('parallelTask2', async () => {
      return await performDataEnrichment(workItemsCount);
    }),
    async (ctx) => ctx.step('parallelTask3', async () => {
      return await performQualityCheck();
    })
  ]);

This will attempt to run all three operations in parallel. If you are worried about putting too much load on another service, you can set concurrency limits so that not everything is processed at once.

Iterate Arrays

I always say that programming is all about ifs and fors. So how do you handle the for situation when working with durable functions? We have the map operator!

  // Step 5: Map operation - iterate over collection with checkpoints
  const mapResults = await context.map(workItems, async (ctx, item, index) => {
    return await ctx.step(`processItem-${index}`, async () => {
      const { processedItem, processingTime } = processWorkItem(item, index);

      // Simulate processing time based on priority
      await new Promise(resolve => setTimeout(resolve, processingTime));

      return processedItem;
    });
  });

You might be asking yourself, Why not just use a regular for loop? The answer is CHECKPOINTS. By using the map operation, you get the benefit of having Lambda manage the checkpoint and the current state of where things left off. This helps immensely if it didn’t process the array completely.

Wait for condition

There is also a special wait operation that waits for a specific condition to be met. For this, you will use the waitForCondition operation. You can think of this operation as a do/while, where it will keep iterating until the while condition is met.

// Step 6: Wait for condition - poll until external system is ready
  const conditionResult = await context.waitForCondition(
    async (state, ctx) => {
      const readinessCheck = await checkSystemReadiness();
      return {
        ...state,
        ready: readinessCheck.ready
      };
    },
    {
      initialState: {
        ready: false,
      },
      waitStrategy: (state) =>
        state.ready
          ? { shouldContinue: false }
          : { shouldContinue: true, delay: { seconds: 3 } }
    }
  );

This has two parameters:

WaitForConditionCheckFunc - Function that will check the current state and return the updated state.
WaitForConditionConfig - Holds the configuration for the initial state and the wait strategy used to determine when it should continue and how long to delay before triggering WaitForConditionCheckFunc again.

Invoke Lambda Function

As much as you try to avoid it, you will have a reason to invoke a separate Lambda function that does its own processing. This will be done by using the invoke operation. All you need to do is give it the function’s ARN and the expected payload.

  // Step 7: Invoke another Lambda function
  const invokePayload = createInvokePayload(workItems.length, context.executionId);
  const invokeResult = await context.invoke(
    'invoke-hello-world',
    process.env.HELLO_WORLD_FUNCTION_ARN,
    invokePayload
  );

Child contexts

The runInChildContext operation creates an isolated execution context for a group of operations. These have their own checkpoint logging and can have multiple steps, waits and other operations. This is treated as a single unit for retry and recovery. Use these when you want to organize complex workflows, implement sub-workflows or isolate operations that should retry together.

  // Step 8: Run operations in child context for isolation
  const childContextResult = await context.runInChildContext('isolated-operations', async (childCtx) => {
    // These operations run in isolation with their own checkpoint log
    const metadata = await childCtx.step('processMetadata', async () => {
      return await processMetadataInChild(context.executionId, workItems.length);
    });

    const validation = await childCtx.step('validateConfiguration', async () => {
      return await validateConfigurationInChild();
    });

    return {
      metadata,
      validation,
      childExecutionId: childCtx.executionId,
      completedAt: Date.now()
    };
  });

Invoking & Troubleshooting

Once your durable function is deployed, you will want to invoke the function. Well, surprise, surprise, this is done exactly the same way you invoke any Lambda function, which means you can use all your regular event sources like API Gateway, EventBridge, and SQS. All of what I’m about to do can be done using the CLI, but for demo reasons, I will be using the console so we can get better visualizations.

When the Lambda is deployed with the DurableConfig you will see a new Durable executions tab in the Lambda console.

To create a new execution, I invoked the function using the standard testing tool in the Lambda console.

Now, let’s look at the details of the execution.

This view should look very similar to what we’ve seen in AWS Step Functions. It doesn’t have the nice workflow diagram, though. There are two sections here:

Durable operations - here we will see all the operations that have been executed. In our example above, only two items appear because it has the human-in-the-loop step and is waiting for someone to respond with the callback ID. We will do that in a second.
Event history - a more verbose list of what has happened. This is where you can track errors that occurred in a specific step and get a broader view of everything that was executed.

Now, let me send the success message for the callback. I will be doing this directly from the console, as shown below. If you are looking for a way to do this programmatically, I’ve included a script here. This script will take in the callback ID and send a success message to continue the flow.

This will now continue and execute all of the steps we have defined in our workflow.

You’ll notice that some operations have explicit names and others have random IDs. The reason is that I did not explicitly specify a name for all the operations, so it will generate a random one for us.

Let’s walk through the operations.

WaitForCallback - The duration was 9 minutes and 43 seconds. That is how long it took to write a few of the paragraphs above, and then press the “Send success” button.
Wait - As expected, this took the 5 seconds we had configured it for.
Parallel - This operation groups all sub-operations under it, making them easier to track.

Map - Similarly to the parallel step, it will group all of the iterations under it.

WaitForCondition - The special emphasis here is that it has a 2 in Retries. This means it had to iterate twice before the condition was met.
ChainedInvoke - Not too much to see for this step. It simply invoked the Lambda function and continued on.
RunInChildContext - This one also groups all the child operations under it.

This allows you to see the different operations and how they look in the console. You can get this information using the CLI or the SDK, and create your own visualizations if necessary.

Wrap Up

Woah! That was a lot! We went through the setup to enable and deploy a durable function, as well as how to build a workflow that uses all of the currently supported durable operations.

The fact that I am now able to create long-running workflows where I can include any npm dependency directly without having to explicitly call external Lambda functions is a huge win compared to what we've been doing with Step Functions.

I am going to keep playing around with this and will be doing a side-by-side comparison with AWS Step Functions. Stay tuned!

Until next time!

Andres Moreno

Log buffering with Lambda Powertools

Andres Moreno — Fri, 28 Mar 2025 00:05:00 +0000

The Lambda Powertools team released a new feature that allows your Lambda functions to buffer logs. That sounded cool, but I didn't understand how it would work or how logs would show in CloudWatch. I decided to try it out and provide a visual example of how it looks with different configuration options.

What is log buffering?

This is what the Lambda Powertools documentation says:

Log buffering enables you to buffer logs for a specific request or invocation. Enable log buffering by passing logBufferOptions when initializing a Logger instance. You can buffer logs at the WARNING, INFO, DEBUG, or TRACE level, and flush them automatically on error or manually as needed.

The three main terms from that explanation are buffering, level, and flush. Let's explain each of these in the context of log buffering.

Buffering—This means that the Lambda function will hold the logs in memory without sending them to CloudWatch unless necessary.
Level - The log level at which you want to start buffering the logs. You can choose which level to begin buffering at. For example, if you configure the log buffer to do it at the INFO level, it will buffer those logs and any log levels with a lower value. In this case, it will also buffer TRACE and DEBUG. Below are all the supported log levels and their numeric value.

Level	Numeric Value
TRACE	6
DEBUG	8
INFO	12
WARN	16
ERROR	20
CRITICAL	24
SILENT	28

Flush - This means the buffered logs will get sent to CloudWatch. You can tell Lambda to send the logs to troubleshoot in case of errors in different ways.

Why buffer logs?

At this point, you might ask yourself, why do I want to buffer logs? The reason might not be immediately apparent. Logs are usually only needed when your application hits an error. This means that we are constantly logging things for all the successful requests and producing logs that no one will see. By enabling the logging buffer, you will only print the logs whenever a specific scenario or error is hit, allowing you to reduce the number of logs produced, the noise and, even better, your logging costs.

Configuration options

There are a few pieces of configuration for buffer logging that can be set when you initialize a logger to handle things according to your needs.

enabled—this should be straightforward. Log buffering is turned off by default; in order to start using it, this parameter will need to be set to true.
maxBytes—this is the max amount of bytes the buffer will hold in memory. If you hit the max limit, the buffer will start dropping older logs to accommodate the new ones. This configuration allows you to protect the Lambda function from running out of memory. The logger will be nice enough to let you know it did this whenever the logs are flushed. The default value for this is 20480 bytes.
bufferAtVerbosity—this is where you will configure at which log level you want to start buffering. Whatever value you configure will be buffered, as will any level with a lower numeric value. The default log level set is DEBUG.
flushOnErrorLog—This is a boolean value that tells the buffer to log whenever it encounters a logger.error call. When writing Lambda functions, we typically have a try/catch statement where we log any errors that get thrown. This helps automatically flush the logs when an error is caught and logged. The default value is true.

But now the question is, what happens if I don't catch and log an error? Will I not be able to see the buffered logs? Thankfully, the Lambda Powertools team thought about this, and they allow you to flush the logs in two other ways:

flushBufferOnUncaughtError—This option is only allowed when injecting the Lambda context into the logger. This property will flush the logs whenever the Lambda function encounters any error and is no explicitly handled by your code.
flushBuffer function—You might want to flush the logs whenever a specific code path is hit or other scenarios that are specific for your use case. To do this, the logger has a new function called flushBuffer that will manually send all of the buffered logs into CloudWatch.

Let's see this in action!

To enable this feature, all we have to do is set the enabled property to true in the logBufferOptions object for the Logger constructor, as shown below:

export const logger = new Logger({
  logBufferOptions: {
    enabled: true
 }
});

Now, to test, we will add all the different types of logs to our Lambda handler like this:

    logger.trace('Trace Log');
    logger.debug('Debug Log');
    logger.info('Info Log');
    logger.warn('Warn Log');
    logger.error('Error Log');
    logger.critical('Critical Log');

If we run the Lambda function and view the logs produced in CloudWatch, we get this:

The results were unexpected, but then I looked at the default values for the logger options to try and understand this better. There are two default values to keep in mind for this example: flushOnErrorLog is enabled by default, and the buffer's default log level is DEBUG. With this, let's understand why the logs show in this specific order:

The TRACE and DEBUG logs are added to the buffer because of the log level setting and are not shown because of this.
The INFO and WARN logs are printed as is since they are not included in the buffer because of the log level setting.
The TRACE and DEBUG logs are now shown because we hit the ERROR log, which means the logs are flushed before the ERROR log is printed.
Finally, the ERROR and CRITICAL logs are printed.

If we were to run the same test but, in this scenario, don't log an error, we would really appreciate the value of the log buffer.

From the image above, we can see that only the non-buffered logs made it to CloudWatch, meaning that if there is no error, we won't see or get charged for the buffered logs.

Now, let's configure the logger to buffer at the WARN level.

export const logger = new Logger({
  logBufferOptions: {
    enabled: true,
    bufferAtVerbosity: 'WARN'
 }
});

As you can se, now we get fewer logs if we run our Lambda function because we are buffering at a higher level.

For our next test, let's intentionally overload our buffer to make it hit the maximum size limit. For this example, we are going to manually flush the logs by calling the function. This means we'll need to turn off the flushOnErrorLog property. In addition to that, we are going to set the maxBytes property to 1000, to make the test easier.

export const logger = new Logger({
  logBufferOptions: {
    enabled: true,
    bufferAtVerbosity: 'WARN',
    flushOnErrorLog: false,
    maxBytes: 1000
 }
});

And for the logging code, we'll do a simple for loop and manually flush at the end.

for (let index = 0; index < 500; index++) {
  logger.trace('Trace Log');
  logger.debug('Debug Log');
  logger.info('Info Log');
  logger.warn('Warn Log');
  logger.error('Error Log');
}

logger.flushBuffer();

When we run this, Lambda Powertools will let us know that we've exceeded the buffer's size and that some of the logs were lost to make room for the newer ones.

What happens if a single log exceeds the buffer's size? Well, let's try it. To do this, I will set the maxBytes property to 1 byte.

export const logger = new Logger({
  logBufferOptions: {
    enabled: true,
    bufferAtVerbosity: 'WARN',
    flushOnErrorLog: false,
    maxBytes: 1
 }
});

If we run the same code as above, we will see a message telling us the log could not be buffered because it was too big.

Wrap up

I'm very excited to start using this in projects and see how the CloudWatch pricing can be impacted. The best part is that Lambda Powertools is taking care of most of the heavy lifting, and all we have to do is configure a few properties to have it behave the way we want it to. I really want to thank the Lambda Powertools team, as they keep impressing me with all the great functionality they deliver that keep making my life easier.

Let me know what you think about this feature!

Until next time!

Andres Moreno

Using Amazon Cognito with the user-password flow

Andres Moreno — Wed, 31 Jul 2024 00:00:00 +0000

On my post called Secure API Gateway with Amazon Cognito using SAM I talked about different Auth terms and walked through a setup to use the Client Credentials Flow, but Cognito recently introduced pricing changes for machine-to-machine authentication that will make this cost us and my main goal is to do this while staying in the free tier for personal projects that will not be generating any income. That is why in this post I am going to setup Amazon Cognito using a different flow called user password-based authentication. With this type of authentication we are charged based on the Monthly Active Users (MAUs) and AWS gives you the first 50,000 MAUs for free, and in my case this will usually stay at 1 per project, so I should be fine.

How does the user-password flow work?

Initially I thought I could use Basic Auth where you provide the encoded username and password in the Authorization header but that is not the case. In the image below I have a picture that shows the interactions that happen to get an authenticated request using this flow.

Let's dive deeper into each interaction

Caller makes a request with the username and password to your API.
API makes a call to Cognito to get the Access, Id and refresh tokens and returns it to the user
User provides Id token in Authorization header with each API call.
API verifies token with Cognito to make sure it's valid.

Setting it all up with SAM

We are going to stick with the same architecture for the Amazon Cognito resources to simplify each APIs configuration and to be able to manage users from a single location.

Now let's see how each of these pieces are set up using SAM.

Auth Stack

This stack has the resources that will be consumed by our API stacks.
Full auth stack setup can be found in this GitHub repository

1. Updates to the User Pool

Unlike our M2M setup that didn't need any properties for the user pool, for this type of authentication we need to setup properties that are specifying the attributes to be hosted for the user such as email, names, etc.

  CognitoUserPool:
    Type: AWS::Cognito::UserPool
+   Properties:
+     UsernameAttributes:
+       - email
+     AutoVerifiedAttributes:
+       - email
+     VerificationMessageTemplate:
+       DefaultEmailOption: CONFIRM_WITH_LINK
+     EmailConfiguration:
+       EmailSendingAccount: COGNITO_DEFAULT

The user pool attributes are:

UsernameAttributes - this is specifying what you allow as a user name. The options here are email or phone_number.
AutoVerifiedAttributes - attributes that you allow to be verified automatically by Cognito. For example we are using email this means Cognito will automatically send a verification email to the user. If we didn't set this attribute an administrator would have to manually verify users in Cognito.
VerificationMessageTemplate - This is where you can set your own template for the email that will get sent to users to verify. For this example we are using a default option provided by Cognito where they will confirm by clicking a link. The other option is CONFIRM_WITH_CODE where a user will receive a code and they have to enter it manually to verify.
EmailConfiguration - This property allows us to setup the configuration for the sender email for the verification or any other communications happening from Cognito. In our case we are using COGNITO_DEFAULT which reduces the amount of setup we need to get an Amazon SES verified email. The COGNITO_DEFAULT has some limits that you will need to consider if you are using it.

API Stack

Our API Stack is simplified when using this flow. Why? We do not need to create a resource server since we will not be using OAuth capabilities.

1. Delete UserPoolResourceServer

As mentioned before, we don't need this resource anymore. So let's get rid of it from our stack by removing it from the template.

2. User Pool Client Updates

Below is the definition for our user pool client.

  CognitoUserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref CognitoUserPoolId
-     GenerateSecret: true
-     AllowedOAuthFlows:
-       - client_credentials
-     AllowedOAuthScopes:
-       - layerless-esbuild/api
-     AllowedOAuthFlowsUserPoolClient: true
+     SupportedIdentityProviders:
+       - COGNITO
+     ExplicitAuthFlows:
+       - ALLOW_USER_PASSWORD_AUTH
+       - ALLOW_REFRESH_TOKEN_AUTH

A few changes need to be done in order for this authentication flow to work.

GenerateSecret - We first get rid of this property since it is not needed for this flow.
AllowedOAuthFlows - When using the User/Password auth flow we do not need to set this property.
AllowedOAuthScopes - Since we are not using an OAuth flow we do not need to set up scopes.
SupportedIdentityProviders - We are going to use COGNITO as our only provider. You can configure different identity providers to simplify your users sign in by using Google, Facebook or any of the supported providers.
ExplicitAuthFlows: this is where we will configure the ALLOW_USER_PASSWORD_AUTH which will allow us to authenticate using a username and a password. I've added ALLOW_REFRESH_TOKEN_AUTH since it's required but we will not be going to do token refreshes in this post.

4. Updates to API Gateway

The only thing that needs to change in the API Gateway is the removal of the AuthorizationScopes. For this flow this is not needed.

  Auth:
    DefaultAuthorizer: ClientCognitoAuthorizer
    Authorizers:
      ClientCognitoAuthorizer:
        UserPoolArn: !Ref CognitoUserPoolArn
-       AuthorizationScopes:
-         - layerless-esbuild/echo

THAT'S IT!! We have now successfully setup our API to authenticate using the user-password flow.

You can find the full working example in this GitHub repository

Testing with Postman

1. Grab Client Id

We only need the client id when authenticating with the user-password flow, this is because we are going to enter a username and password to authenticate and that is where the token will be coming from. So we will get this value from the console by going into our new application client.

2. Requesting auth tokens

To request the tokens we need to call Amazon Cognito specifying we are doing an InitiateAuth command.
This will require us to make a POST call to https://cognito-idp.us-east-1.amazonaws.com. The region will change depending on where you created your User Pool. We specify the command by adding the X-Amz-Target, we also need to specify the Content-Type since it's an AWS specific type. Below is a screenshot that shows how the headers should look.

The body requires the following parameters:

AuthFlow - This is where we specify that we want to use the user-password flow by setting a value of USER_PASSWORD_AUTH
ClientId - We will set the value to the one we grabbed in step 1.
AuthParameters - in the object we will provide the USERNAME and the PASSWORD of our user.

When you send this request you should get a response back with the AccessToken, IdToken, RefreshToken, TokenType and ExpiresIn. This also returns the ChallengeParameters but this flow does not require any challenge responses to be generated. (I've edited the full response from the picture to not expose the full keys).

5. Make request

Grabbing the IdToken from the response we got from the request above we can now take that and use it to authorize the request to the API endpoint as seen below.

Authenticating automation calls

Whenever I change authentication methods I get stuck setting it all up for my test automation. This requires us to do the same thing we did in Postman but programmatically. I am doing this by running a small NodeJS script that will make the call as shown below.

const initiateAuthResponse = await axios({
    url: process.env.COGNITO_URL,
    method: 'POST',
    headers: {
      'Content-Type': 'application/x-amz-json-1.1',
      'X-Amz-Target': 'AWSCognitoIdentityProviderService.InitiateAuth'
    },
    data: JSON.stringify({
      ClientId: process.env.CLIENT_ID,
      AuthFlow: 'USER_PASSWORD_AUTH',
      AuthParameters: {
        USERNAME: process.env.USERNAME,
        PASSWORD: process.env.PASSWORD
      }
    })
  });


  const token = initiateAuthResponse.data.AuthenticationResult.IdToken;

If you look at the code above it should seem very familiar to what we did in Postman. Making a POST request with the necessary headers and body. Once you get the response you can do whatever you need with the tokens.

I've provided 2 examples on how to handle the user credentials for this script to use within a CI/CD pipeline.

Create user programmatically - In this example we are creating a Cognito user programmatically and using the credentials to authenticate. We do this by setting the values as environment variables using the >> $GITHUB_ENV command. At the end of the run we delete the user so we don't have an endless amount of orphaned automation users.

  test-api-with-user-password-auth-inline-create-user:
    name: Run Portman With USER_PASSWORD_AUTH - Inline Create User
    runs-on: ubuntu-latest
    environment: ${{ inputs.ENVIRONMENT }}
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-region: us-east-1
          role-to-assume: ${{ secrets.PIPELINE_EXECUTION_ROLE }}
          role-session-name: create-test-user
          role-duration-seconds: 3600
          role-skip-session-tagging: true

      - name: Create Test User
        run: |
          username=$(uuidgen)@andmore.dev
          password=$(uuidgen)G1%
          echo "USERNAME=$username" >> $GITHUB_ENV;
          echo "PASSWORD=$password" >> $GITHUB_ENV;
          aws cognito-idp admin-create-user --user-pool-id ${{ inputs.COGNITO_USER_POOL_ID }} --username $username --message-action SUPPRESS
          aws cognito-idp admin-set-user-password --user-pool-id ${{ inputs.COGNITO_USER_POOL_ID }} --username $username  --password $password --permanent

      - name: Test API
        env:
          COGNITO_URL: ${{ secrets.COGNITO_URL }}
          CLIENT_ID: ${{ secrets.COGNITO_CLIENT_ID }}
        run: |
          npm ci

          node ./portman/get-auth-token/user-password-auth.mjs
          npx @apideck/portman --cliOptionsFile portman/portman-cli.json --baseUrl ${{ inputs.BASE_URL }}

      - name: Delete Test User
        run: |
          aws cognito-idp admin-delete-user --user-pool-id ${{ inputs.COGNITO_USER_POOL_ID }} --username $USERNAME

Use GitHub Secrets - This is one of the simpler routes but requires to have a user already set up. All you need to do is store the credentials in GitHub secrets and reference them in the workflow.

  test-api-with-user-password-auth-github-secrets:
    name: Run Portman With USER_PASSWORD_AUTH - Load GitHub Secrets
    runs-on: ubuntu-latest
    environment: ${{ inputs.ENVIRONMENT }}
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-region: us-east-1
          role-to-assume: ${{ secrets.PIPELINE_EXECUTION_ROLE }}
          role-session-name: create-test-user
          role-duration-seconds: 3600
          role-skip-session-tagging: true

      - name: Test API
        env:
          COGNITO_URL: ${{ secrets.COGNITO_URL }}
          CLIENT_ID: ${{ secrets.COGNITO_CLIENT_ID }}
          USERNAME: ${{ secrets.COGNITO_USERNAME }}
          PASSWORD: ${{ secrets.COGNITO_PASSWORD }}
        run: |
          npm ci

          node ./portman/get-auth-token/user-password-auth.mjs
          npx @apideck/portman --cliOptionsFile portman/portman-cli.json --baseUrl ${{ inputs.BASE_URL }}

There are two considerations to take into account here. If you are going to have a user per repository it might become a burden to maintain these users as your applications and repositories grow. On the other hand, if you share a single user with all your repos you might be introducing security vulnerabilities since it will be easier to have this password compromised.

Wrap Up

In this post we were able to update our authentication to use the user-password flow instead of M2M for our APIs, this allows us to stay within the Cognito free tier. We were able to verify that we can still authenticate with Postman and our test automations. Hopefully this shows the flexibility there is with Cognito and how you can configure it differently to satisfy your use cases. I will be exploring the other authentication flows so you can easily choose the one that makes more sense for you.

Creating users in Amazon Cognito programmatically

Andres Moreno — Thu, 11 Jul 2024 00:00:00 +0000

When you have CI/CD pipelines that run automated tests against your APIs you might need to dynamically create users in Amazon Cognito to run them. If that is the case you are in the right place. In this post we'll be going over what you need to do to create a valid user in Cognito to be used by your automation.

Create user in Cognito

The first thing we need to do is to create a user in our Cognito User Pool. Since this is for automation purposes I am creating users using the andmore.dev domain that I own. I want these users scoped to the specific repository that is running the CI/CD pipeline, so I'm generating a unique user with a UUID.

username=$(uuidgen)@andmore.dev
aws cognito-idp admin-create-user --user-pool-id [YOUR-USERPOOL-ID] --username $username --message-action SUPPRESS

In the commands above, I am generating the username with a new uuid. Since I have email forwarding enabled for my domain, each new user will send me an email. To avoid this side effect I have added the --message-actions property as SUPPRESS, this will disable any messages from being sent.

This is all we need right? I thought this too, but there is more. The user is created with a temporary password and is not usable yet. Since we are doing this for an automated process, we can't really go through the flow of opening the email, signing in and changing the password. So let's see what else we need to do.

Verifying user

Cognito offers a command that allows admins to set a users password, bypassing the manual steps mentioned above.

password=$(uuidgen)G1%
aws cognito-idp admin-set-user-password --user-pool-id [YOUR-USERPOOL-ID] --username $username  --password $password --permanent

We first need to generate a password, to keep it simple I'm doing it by creating a new uuid and appending a capital letter, a number and a special character to satisfy the password policy set in my user pool. Now we can provide this to the admin-set-user-password command and make sure we pass in the --permanent flag, since by default it will set the password as temporary leaving us in the same spot as before.

Now the user is all setup to be able to authenticate using any of the flows you have setup for your Cognito User Pool Client.

Delete the user

You probably don't want to keep all of these users once your pipeline is done. To delete the user you will need to call the admin-delete-user command with the generated username.

aws cognito-idp admin-delete-user --user-pool-id [YOUR-USERPOOL-ID] --username $username

Security

To be able to run these commands you'll need permissions to run these for your Cognito user pool. Below is an example IAM policy that will give you the necessary permissions to run all three commands.

{
  "Effect": "Allow",
  "Action": [
      "cognito-idp:AdminCreateUser",
      "cognito-idp:AdminSetUserPassword",
      "cognito-idp:AdminDeleteUser",
  ],
  "Resource": [
      "YOUR-USERPOOL-ARN",
  ]
}

Make sure to restrict the access to this policy since with this level of access people could create and authenticate with them. I am adding these permissions to my pipeline execution role that allows GitHub to assume this role for a specific repository by setting the trust relationship of the IAM role as follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "[YOUR-OIDC-PROVIDER-ARN]"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "ForAllValues:StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:[YOUR-ORG]/[YOUR-REPOSITORY]:ref:refs/heads/[ALLOWED-BRANCH]",
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

The token.actions.githubusercontent.com:sub allows an array and wildcards so if you have a specific format for branching, you can play around with the condition properties to meet your needs while still being protected.

Wrap up

We were able to create and verify a user for our automation, as well as clean up after we are done so we are not left with orphaned users all over the place.
I hope this is helpful to you. Cognito documentation is really not that straightforward and can be hard to find what you are looking for, this is why I'm trying to provide tutorials around Cognito, so stay tuned since I have more content lined up around this topic.

Secure API Gateway with Amazon Cognito using SAM

Andres Moreno — Wed, 08 May 2024 00:00:00 +0000

Backstory

I create a lot of APIs, these are for blog posts, for playing around with new functionality or tools that I've created for myself. All of these have been created without authentication in place. Not securing APIs can create data exposures for you, but it can also pose a financial risk to your accounts if a malicious user gets your endpoints. This is why I want to secure any API I create but I want this to be with minimal setup so that it's simple to replicate many times.

We first need to understand a few concepts around what we're setting up.

What is Cognito?

Amazon Cognito is a service provided by AWS that allows you to add authentication to your applications or services. It integrates natively with API Gateway to secure each endpoint.

Cognito has multiple layers where you can apply different types of configurations, this gives us the flexibility to get things setup for different use cases.

User Pool -
A Cognito user pool is the backbone to everything in Cognito. This is an OpenID Connect identity provider which contains the user directory to authenticate and authorize users.
User Pool Domain -
The user pool domain is used to give the authentication url a better name for users and applications to authenticate with.
Resource Server -
An OAuth 2.0 API server that validates that an access token contains the scopes that authorize the requested endpoint in the API.
User Pool Client -
A user pool client is a configuration within a user pool that directly interacts with your application that will be authenticating using Cognito.

Authentication Flows

There are several authentication flows that you can use for your applications. In this post from Auth0 you can get a better understanding about which flow is better for your use case.

Since I am setting up very basic authentication to be able to test my API using Postman I will use the Client Credentials Flow (CCF) to allow us to authenticate our requests by getting an access token by sending a client id and a client secret to Cognito. This token is a JSON Web Token (JWT). The CCF is recommended when working with Machine-to-Machine (M2M) communication like CLIs, APIs, etc.

Setting it all up with SAM

I don't want to have a Cognito User Pool per API created, to simplify this I will have a single Auth Stack that will contain the User Pool and the User Pool Domain resources. We will then share the data from this stack to our API stacks to be able to create the Resource Server and User Pool Clients separately. Below is a picture that shows this structure.

Now let's see how each of these pieces are set up using SAM.

Auth Stack

In this stack we are going to define the resources that will be consumed by other APIs to authenticate.

You can find my full setup for this stack in this GitHub repository

1. User Pool

The User Pool doesn't require a lot of configuration when doing CCF. You are able to add more restrictions and configuration but as mentioned before, we are trying to keep it simple for now.

  CognitoUserPool:
    Type: AWS::Cognito::UserPool

2. User Pool Domain

In my setup I'm using andmoredev as my domain which makes our authentication base url look like https://andmoredev.auth.us-east-1.amazoncognito.com. You can setup up your own custom domain by setting up the CustomDomainConfig.

  CognitoUserPoolDomain:
    Type: AWS::Cognito::UserPoolDomain
    Properties:
      UserPoolId: !Ref CognitoUserPool
      Domain: andmoredev

We need to make sure other stacks can get access to the User Pool Id and ARN to be able to create the resource server and user pool client. To do this we are going to use SSM Parameters.

  CognitoUserPoolIdParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub /${AWS::StackName}/CognitoUserPoolId
      Type: String
      Value: !Ref CognitoUserPool

  CognitoUserPoolArnParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub /${AWS::StackName}/CognitoUserPoolArn
      Type: String
      Value: !GetAtt CognitoUserPool.Arn

Now we can go and add authentication to our API stack using this user pool.

API Stack

I will be adding authentication to an existing API in this GitHub repository. To look at the specific changes I made to get authentication working, you can look at the PR where I did this.

1. Consuming Auth Stack resources

We first need to get the user pool Id and ARN from SSM by adding them into the Parameters section of our template.

Parameters:
  CognitoUserPoolId:
    Type: 'AWS::SSM::Parameter::Value<String>'
    Default: '/andmoredev-auth/CognitoUserPoolId'

  CognitoUserPoolArn:
    Type: 'AWS::SSM::Parameter::Value<String>'
    Default: '/andmoredev-auth/CognitoUserPoolArn'

2. Resource Server

We are creating a resource server with one scope that will be used to give access to all endpoints in our API. I will not go into more details on more advanced scope design in this post.

  LayerlessESBuildResourceServer:
    Type: AWS::Cognito::UserPoolResourceServer
    Properties:
      UserPoolId: !Ref CognitoUserPoolId
      Identifier: layerless-esbuild
      Name: Layerless ESBuild
      Scopes:
        - ScopeName: api
          ScopeDescription: Allow api access

3. User Pool Client

Below is the definition for our user pool client.

  CognitoTestAutomationClient:
    Type: AWS::Cognito::UserPoolClient
    DependsOn:
      - PostmanResourceServer
    Properties:
      UserPoolId: !Ref CognitoUserPoolId
      GenerateSecret: true
      AllowedOAuthFlows:
        - client_credentials
      AllowedOAuthScopes:
        - layerless-esbuild/api
      AllowedOAuthFlowsUserPoolClient: true

Let's talk about the properties we've here.

UserPoolId - reference to the id of the user pool created in our Auth Stack.
GenerateSecret - this is necessary for us to use with client credentials flow.
AllowedOAuthFlows - we are telling Cognito that this client will only allow CCF
AllowedOAuthScopes - We need to make sure this array contains scopes that have been defined in our resource server.
AllowedOAuthFlowsUserPoolClient - This is what enables us to use standard OAuth functionality with our user pool client.

I've seen deployment scenarios where the resource server gets deployed after the client and we get an error saying the scope does not exist, this is the reason I am explicitly adding the DependsOn for this resource.

4. Hooking it all up to API Gateway

To tell our API Gateway to authenticate using our new cognito user pool we need to add the Auth property, it will look something like this.

  Auth:
    DefaultAuthorizer: ClientCognitoAuthorizer
    Authorizers:
      ClientCognitoAuthorizer:
        UserPoolArn: !Ref CognitoUserPoolArn
        AuthorizationScopes:
          - layerless-esbuild/echo

We are consuming the user pool ARN from the Auth Stack and allowing the scope that we've created in our resource server.

THAT IS ALL! Now we can deploy this stack to be able to test it.

Testing it with Postman

1. Grab authentication data

To test this we need to go into the AWS console and grab the Client Id and Client Secret that were generated. These are located in the Cognito service by selecting the new user pool and going in the App integration section. In the bottom you will see your new app client, once you open it you will see something like the image below. You can copy the Client ID and Client secret from here, we will use these values in the next steps.

Please handle these values with care, if compromised someone could gain access to your APIs and do malicious things.

2. Test un-authenticated request

To verify our API is secure we will first run an unauthenticated request. To do this we will call our endpoint without setting anything for authentication, when we send this request we should get a 401 - Unauthorized response as shown below.

3. Set authentication data in Postman

With the values we will now use a new Postman feature called Vaults that allows us to securely store sensitive data. To do this we will go into the Vault section in the bottom of the window and add our secrets.

4. Setup request authentication

Now back in the Postman request we can set the Authorization configuration. There are several parameters to set here.

Grant type - This will have a value of Client Credentials specifying that we will be using the client credentials flow.
Access Token URL - The value for your specific user pool will vary depending on your configured user pool domain. It will look something like this https://[your-domain].auth.us-east-1.amazoncognito.com/oauth2/token
Client ID - We will grab this from the vault by setting a value of {{vault:clientId}}
Client Secret - Also from the vault with a value of {{value:clientSecert}}
Scope - This will be based on what was set on the resource server. From our example it is layerless-esbuild/echo.

5. Get the JWT

We can now get a new access token by going to the bottom of the Authorization section and pressing the Get New Access Token button. If successful you will receive a prompt where you can click a button that says Use Token. By pressing that button you will get the value presented in the Access Token and it will be used in the Authorization header of your request.

If we send the request now we will get a successful response.

Wrap Up

Sadly security is often left as an afterthought, just like it happened to me with all the APIs I've created. In this post we were able to understand a few of the concepts relating to authentication and the resources necessary to set this up in AWS with Amazon Cognito. We also tested that our API is now secure and how we can get a token to authenticate against it.

I hope this allows people to add a basic layer of security to their APIs so that we make malicious users work a little bit more.

Using YAML anchors and aliases in a SAM template

Andres Moreno — Sun, 31 Mar 2024 00:00:00 +0000

Last month I wrote a post about getting rid of Lambda Layers by using ESBuild. What I quickly learned is that the Metadata attribute has to be copied and pasted for EVERY Lambda function in your stack. I tried using the Global section in the SAM template and it turns out it's not supported. I started thinking about how I could reuse the same configuration across my template and found that YAML already has a functionality that does this called YAML Anchors and Aliases. In this post I will go through what YAML aliases and anchors are and how we can use them in SAM.

What are YAML anchors and aliases?

You can think of anchors as assigning a value to a variable to be used in other places. The way you define an anchor is by adding an & on an entity with the name of the anchor.

personName:
  type: object
  properties: &person-name-properties
    firstName:
      type: string
    lastName:
      type: string

In the example above we have created an anchor for the properties attribute of the personName schema with the name person-name-properties. Later in the file you can reference the anchor with an alias by using the * symbol and the anchor name.

person:
  type: object
  properties:
    name:
      type: object
      properties: *person-name-properties

When the YAML is proccessed the person object will look like this.

person:
  type: object
  properties:
    name: 
      type: object
      properties:
        firstName:
          type: string
        lastName:
          type: string

YAML allows you to override and/or extend properties to be able to get it into the correct structure. To update the name object to have a minLength for the firstName and include a new attribute called middleName we would do something like this.

person:
  type: object
  properties:
    name:
      type: object
      properties: 
        <<: *person-name-properties
        firstName:
          type: string
          minLength: 3
        middleName:
          type: string

If you look at the example above we are now using the << attribute which essentially deconstructs whats defined in the anchor and allows you to overwrite something by adding the same attribute or add new ones in line. If we process the YAML above we would get the result below.

person:
  type: object
  properties:
    name: 
      type: object
      properties:
        firstName:
          type: string
          minLength: 3
        lastName:
          type: string
        middleName:
          type: string

With this functionality we can define our base Metadata attributes for ESBuild and use them for our template right? .... right?

Why are YAML anchors not as straightforward with SAM?

SAM is strict in what it accepts in the template.yaml when doing a deployment, you will not see these errors when doing sam build though, so be careful to make sure your template is actually deployable. Here is an example where I'm defining the esbuild config metadata at the root of my template and consuming it in a function

AWSTemplateFormatVersion: 2010-09-09
Description: Layerless ESBuild Example
Transform:
  - AWS::Serverless-2016-10-31

esbuild: &esbuild
  BuildMethod: esbuild
  BuildProperties:
    Format: esm
    Minify: false
    OutExtension:
      - .js=.mjs
    Target: es2020
    Sourcemap: false
    EntryPoints:
      - index.mjs
    Banner:
      - js=import { createRequire } from 'module'; const require = createRequire(import.meta.url);

Resources:
  EchoFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/functions/echo
      Runtime: nodejs20.x
      Handler: index.handler
    Metadata: *esbuild

The built template after running sam build looks like this

AWSTemplateFormatVersion: '2010-09-09'
Description: Layerless ESBuild Example
Transform:
- AWS::Serverless-2016-10-31
esbuild:
  BuildMethod: esbuild
  BuildProperties:
    Format: esm
    Minify: false
    OutExtension:
    - .js=.mjs
    Target: es2020
    Sourcemap: false
    EntryPoints:
    - index.mjs
    Banner:
    - js=import { createRequire } from 'module'; const require = createRequire(import.meta.url);
Resources:
  EchoFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: EchoFunction
      Runtime: nodejs20.x
      Handler: index.handler
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Banner:
        - js=import { createRequire } from 'module'; const require = createRequire(import.meta.url);
        EntryPoints:
        - index.mjs
        Format: esm
        Minify: false
        OutExtension:
        - .js=.mjs
        Sourcemap: false
        Target: es2020
      SamResourceId: EchoFunction

Doesn't look bad right? It looks like the anchor got referenced and placed correctly in my Lambda function.But when we run sam deploy we'll get this error.

Status: FAILED. Reason: Invalid template property or properties [esbuild]

It doesn't like the esbuild property we've defined since it's not part of the SAM template schema.

Working around the limitation

To avoid getting an error on deployment we can rename the property to Metadata instead. This is an accepted property in SAM that will build and deploy successfully. Below you can see how I renamed esbuild to Metadata

AWSTemplateFormatVersion: 2010-09-09
Description: Layerless ESBuild Example
Transform:
  - AWS::Serverless-2016-10-31

Metadata: &esbuild
  BuildMethod: esbuild
  BuildProperties:
    Format: esm
    Minify: false
    OutExtension:
      - .js=.mjs
    Target: es2020
    Sourcemap: false
    EntryPoints:
      - index.mjs
    Banner:
      - js=import { createRequire } from 'module'; const require = createRequire(import.meta.url);

Resources:
  EchoFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/functions/echo
      Runtime: nodejs20.x
      Handler: index.handler
    Metadata: *esbuild

This is great! We can have several Lambda functions that share the same ESBuild configuration. But what happens when we want to define multiple anchors? If we were to add a second Metadata attribute we will now have invalid YAML since it doesn't accept duplicate properties names at the same level. The funny thing is this actually builds and deploys correctly so if you have no linting in your processes this might work for you. But there is a better way we can define multiple anchors without sacrificing our linting process.

Instead of making the Metadata property the anchor we can add multiple anchors under the Metadata property. In the example below I am going to add a second anchor that contains only the BuildProperties and I'll update the base ESBuild config to use it.

AWSTemplateFormatVersion: 2010-09-09
Description: Layerless ESBuild Example
Transform:
  - AWS::Serverless-2016-10-31

Metadata:
  esbuild-properties: &esbuild-properties
    Format: esm
    Minify: false
    OutExtension:
      - .js=.mjs
    Target: es2020
    Sourcemap: false
    EntryPoints:
      - index.mjs
    Banner:
      - js=import { createRequire } from 'module'; const require = createRequire(import.meta.url);

  esbuild: &esbuild
    BuildMethod: esbuild
    BuildProperties: *esbuild-properties

Resources:
  EchoFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/functions/echo
      Runtime: nodejs20.x
      Handler: index.handler
    Metadata: *esbuild

We've done it! We can now define reusable YAML objects in our SAM templates. If we wanted to extend or overwrite any properties we can use the << attribute.

Resources:
  EchoFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/functions/echo
      Runtime: nodejs20.x
      Handler: index.handler
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        <<: *esbuild-properties
        Minify: true
        External:
          - '@aws-sdk/*'

Wrap up

By using standard YAML functionality and repurposing the SAM Metadata property we were able to successfully setup reusable snippets of YAML for our SAM templates. This allows us to greatly reduce the size of the template we are maintaining, it also reduces the risk of errors by giving standard settings to be used throughout the whole file.

Drop the layers, bundle up with ESBuild instead

Andres Moreno — Wed, 07 Feb 2024 00:00:00 +0000

I've seen a lot of posts around the problems that Lambda Layers bring, a very good one is called You shouldn't use Lambda Layers by AJ Stuyvenberg. In this post AJ explains the myths and cons of using Lambda Layers. What is not easy to find is clear examples on how to actually get rid of Lambda Layers by using a bundler. In this post we will go through a structure and configuration that allows us to remove Layers by using ESBuild to bundle the dependencies and shared code for your functions.

Backstory

I've been using Lambda Layers for a while as a way to share dependencies and code between different Lambda functions. This was done as a way to keep the code editable from the AWS console and reduce the amount of times we include an npm dependency in the package.json of functions. We had other problems than what AJ mentioned in his post, we noticed that once the project got to a certain size, deployments are a pain. Every deployment creates a new version of the layer which then creates an update for every single Lambda function that's consuming the layer. And this is why I started looking into moving away from layers and instead bundling with ESBuild.

Link to complete example on GitHub

Configuring ESBuild for Lambda Functions in SAM

The first thing we need to do, is to add the configuraton to let SAM know that we want to build our function using ESBuild. This is done using the 'Metadata' attributes as seen below.

Metadata:
  BuildMethod: esbuild
  BuildProperties:
    Format: esm
    OutExtension:
      - .js=.mjs
    Target: es2020
    Minify: false
    Sourcemap: true
    EntryPoints:
      - index.mjs
    Banner:
      - js=import { createRequire } from 'module'; const require = createRequire(import.meta.url);
    External:
      - '@aws-sdk/client-secrets-manager'

Let's break these down:

BuildMethod - This is where we tell it to use esbuild to bundle the Lambda function code.
Format - Since we are working with ECMAScript Modules we will want our format to be esm.
OutExtension - The artifact that gets outputted by esbuild has a .js extension, since we have specified esm for the format, we need to output as .mjs which is how NodeJS knows that it is working with ESM.
Target - Used to specify the ECMAScript version to use. Here we will be using the default value of es2020.
Minify - this can help reduce the bundle size even more but it makes the code unreadable. For my example I will keep it as false so I can still read the code of my deployed function in the AWS console.
Sourcemap - Attribute to specify if you want to include the sourcemap in your function. Including this allows for a better troubleshooting experience. This is because you will be able to get the stack trace and line numbers from the original code and not the bundled, the downside is that this may have performance impact on your function since it makes it bigger.
EntryPoints - specify the entry points for the application.
Banner - this is not documented by AWS. This property allows us to workaround an issue that ESBuild has with depenedencies that are using require. Link to GitHub issue with the solution.
External - Used to exclude specific packages from the bundle. In this case we are importing the getSecret action from the @aws-lambda-powertools/parameters/secrets package. This package requires the @aws-sdk/client-secrets-manager as a dependency, since we know this is already included in the NodeJS 20.X runtime, we exclude it so we use the one included instead of us adding it.

If you have a dependency that needs to be excluded for all functions my recommendation would be to put that in the devDependencies instead.

When we run sam build ESBuild will take care of bundling the shared modules that are being imported in the function from the src/shared directory. It will also look at any other dependencies and include them from the node_modules.

Dependency management improvements

There are many ways to structure your project, the one shown in my example is by far the simplest one I could find to reduce the effort of managing npm dependency versions.
How am I doing this? I have a single package.json that includes all the dev and non dev dependencies, when bundling with ESBuild it will take care of only including the dependencies needed and ignore the rest. By keeping a single package.json, we can reduce the amount of pull requests that a tool like Dependabot would open saving a lot of hours of reviewing, testing and merging.

Configuration for Common JS

The use of ESBuild is not restricted to ESM only, if you are using Common JS the configuration is very similar, we need to exclude a few of the attributes. Below is an example for the same function in Common JS

Metadata:
  BuildMethod: esbuild
  BuildProperties:
    Minify: false
    Target: es2020
    Sourcemap: true
    EntryPoints:
      - index.js
    External:
      - '@aws-sdk/client-secrets-manager'

If you are interested in migrating from Common JS to ESM, I wrote an article on How to update Lambda functions from Common JS to ECMAScript

Wrap Up

We learned how you can configure ESBuild to bunlde your functions code instead of using Layers. We were also able to keep a single source for the npm dependencies of our project which reduces the ongoing maintenance. I hope this helps you easily get setup with all your future projects.

Call APIs from Step Functions using SAM

Andres Moreno — Tue, 23 Jan 2024 23:31:44 +0000

Benoit Boure wrote an article last week on Calling External Endpoints With Step Functions and the CDK, but if you know me I love SAM and wanted to show the same setup but instead of using CDK we will be using SAM.

I will not go into the details of how it works since Benoit did a great job of explaining that in the post mentioned above.

We'll be using the OpenWeather API to get the temperature of a place and check if we need to wear a warm jacket, light jacket or no jacket. Below is the state machine workflow diagram that we will be executing.

Link to complete example on GitHub

Steps to follow

1. Create EventBridge Connection

The HTTP task uses an EventBridge Connection to authenticate the request. For the OpenWeather API we need to provide the API Key as a query parameter and not as an Authorization header as you would typically do on other APIs. To accomplish this we add the InvocationHttParameters object where we add the query string parameter to the request with the name appid.

OpenWeatherConnection:
  Type: AWS::Events::Connection
  Properties:
    Name: 'openweather-connection'
    AuthorizationType: API_KEY
    AuthParameters:
      ApiKeyAuthParameters:
        ApiKeyName: Authorization
        ApiKeyValue: !Ref OpenWeatherAPIKey
      InvocationHttpParameters:
        QueryStringParameters:
          - IsValueSecret: true
            Key: appid
            Value: !Ref OpenWeatherAPIKey

2. Create Step Function Execution Policy

There are several permissions needed to execute an endpoint using the HTTP task.

It needs access to the secret that the EventBridge Connection creates in Secrets Manager.
Permission to retrieve the connection credentials from the EventBridge Connection.
Permission to execute the InvokeHTTPEndpoint. In this case the resource needs to be the ARN of the state machine that is executing it. To avoid a cyclical dependency in SAM we are parsing the ARN ourselves. I'm adding a wildcard at the end to account for the unique identifier that AWS adds to the end of generated resource names. We are also restricting the HTTP Invocation to only allow a GET to the OpenWeather API.

Policies:
  - Version: 2012-10-17
    Statement:
      - Effect: Allow
        Action:
          - secretsmanager:DescribeSecret
          - secretsmanager:GetSecretValue
        Resource: !GetAtt OpenWeatherConnection.SecretArn
      - Effect: Allow
        Action:
          - events:RetrieveConnectionCredentials
        Resource: !GetAtt OpenWeatherConnection.Arn
      - Effect: Allow
        Action:
          - states:InvokeHTTPEndpoint
        Resource: !Sub arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:ShouldIWearAJacketStateMachine*
        Condition:
          StringEquals:
            states:HTTPMethod: GET
          StringLike:
            states:HTTPEndpoint: !Ref OpenWeatherBaseUrl

3. Calling endpoint from the Step Function

With all that in place we can now add the step to our state machine as shown below.

We are specifying the HTTP method as GET, the connection ARN to our EventBridge Connection as the authentication, OpenWeather base URL and we are appending query parameters to get the temperature of the location and the units we want.

"Get Temperature": {
  "Type": "Task",
  "Resource": "arn:aws:states:::http:invoke",
  "Parameters": {
    "Method": "GET",
    "Authentication": {
      "ConnectionArn": "${OpenWeatherConnectionArn}"
    },
    "ApiEndpoint": "${OpenWeatherBaseUrl}",
    "QueryParameters": {
      "lat.$": "$.latitude",
      "lon.$": "$.longitude",
      "units": "imperial"
    }
  },
  "ResultSelector": {
    "temperature.$": "$.ResponseBody.main.temp"
  }
}

Wrap Up

As you can see it is not that complicated to replace any Lambda Functions you have to call third party endpoints. The permissions can get a little bit complicated but once you understand them you can repeat them for other requests you need to do.

How to update Lambda Functions from Common JS to ECMAScript

Andres Moreno — Mon, 11 Dec 2023 00:00:00 +0000

Version 5.0.0 of the @middy/core package no longer supports Common JS in favor of ECMAScript Modules (ESM). ESM improves performance by 2X as mentioned in this post. Seeing this change prompted me to investigate what needs to be updated to get Lambda functions to be ESM to not fall behind on my dependency versions. In this post, we'll go through the 3 steps needed to go from Common JS to ESM.

1. Designate Lambda functions as ECMA Script Modules

There are two ways you can designate a Lambda function as an ECMA Script Module.

Update package.json

You can add the type attribute with a value of module to the package.json as shown below.

Update file extension

You can also set the extension of the files to be .mjs instead of .js or .cjs

2. Update how you include dependencies

In Common JS dependencies are included using the require keyword, in ECMAScript with the import keyword. Below we can see the code change needed to include dependencies in ESM.

3. Update how the handler and other functions are exported

The way you export functions in ESM is different than how it is done for Common JS. The export will have to change from exports.functionName to export const functionName as shown below.

Last words

With these three changes you are now done, these are simple steps to take but might get more complicated depending on your project structure. If you are applying this to a more complicated project I recommend having test automation (unit tests, contract tests, etc) to verify no bugs are getting introduced.

Speed up new serverless application development with customized SAM templates

Andres Moreno — Sat, 01 Jul 2023 00:00:00 +0000

As you start setting patterns and best practices within your projects at some point you will want to share these to simplify the lives of other developers.

SAM allows you to initialize projects by using the init command and selecting a template for your project. SAM has built-in templates to generate simple Hello-World starter projects (you can find the list of templates here). These will help you get started and get familiar with how to structure your serverless applications but as your applications grow you will want to add your own flavor to these. This is why SAM provides the ability to use custom templates to kick start applications that include your own designs.

How does `sam init` work?

We first need to understand how the init command works.

sam init uses cookiecutter to prompt and resolve all the values for a template.

You will first select if you want to use an existing AWS template or if you want to use a custom template. If you choose the latter you will have to provide the location of the template.

Once you have a template selected it will use cookiecutter to prompt the user for all the necessary information to do the proper replacements in the files.

Creating a custom template

Since SAM uses cookiecutter to generate a project you can use any of the capabilities described in their documentation

1. Create project structure

You give your project the necessary folder structure, as well as include any files needed within these folders. Usually you already have a project in place that you want to model from when creating a template, you can simply copy all of the files and folders from that existing project into your template project.

2. Identify replaceable values

Now you need to identify what values you need to be entered by the person to initialize the project. These are things that will vary from project to project. For example, if you are working with a serverless API you could provide values for DynamoDB table name, API stage name, stack name, and Lambda Function runtime.

With the attributes identified you will now replace what you currently have with something that looks like this {{ cookiecutter.project_name }}, this is what cookiecutter will be scanning for to be able to replace the values with the users input. This format can be applied to any spot within a file as well as for any folder or file names.

In the screenshot below you can see how I'm using cookiecutter replacement format for a folder name as well as for a string within a file.

3. Setup cookiecutter metadata

Cookiecutter needs to know which attributes to prompt the user for to be able to use the values as replacements in the template. This is done in a cookiecutter.json file at the root of the template project. This is a JSON file that contains all the attributes identified in step #2, the file will look something like this:

{
  "outputDir": "",
  "service_name": "",
  "dynamodb_table_name": ""
}

Using the custom template

All you have to do now is run the sam init command, when asked what the template source is you will choose Custom Template Location. You have several options here that depend on where your template is currently located.

In my example below I used git, where I specify the repository url where my template is located:

Summary

We learned how to create a custom template for your serverless applications in 3 simple steps. Having your own template is valuable when your team starts growing and there are new applications regularly getting created. Reducing the amount of time developers spend creating folder structures and adding boilerplate functionality to an application is great so the time focused can be spent providing business value.

How to Run the JavaScript AWS SDK Locally

Andres Moreno — Tue, 31 Jan 2023 13:29:35 +0000

When using Access Keys I got used to setting the default profile in my credentials file and it worked just fine. When I began having to manage more accounts and switched to SSO, updating the default profile becomes more of a pain. I will show you a couple of ways to authenticate the SDK using named profiles.

I will be showing how to authenticate the AWS SDK against a specific AWS Account using the following methods:

Setting environment variables
Getting credentials in the code

Setting environment variables

The easiest way you can run the AWS SDK from your local machine is by setting the AWS_PROFILE environment variable (if not set it will default to the default profile).

In your terminal you can set the AWS_PROFILE environment variable so the SDK can find the profile from the ~/.aws/config file and use those values. When using AWS SSO you will need to log in by running aws sso login --profile my-profile.
Look at this Ben Kehoe's post for an explanation on how this command works.

How do you set environment variables?

Linux/macOS

$ export AWS_PROFILE=my-profile
$ export AWS_REGION=us-east-1

Powershell

PS C:\> $Env:AWS_PROFILE="my-profile"
PS C:\> $Env:AWS_REGION="us-east-1"

Windows Command Prompt

C:\> setx AWS_PROFILE my-profile
C:\> setx AWS_REGION us-east-1

You can now use any client by initializing it as shown below.

  const { SecretsManagerClient } = require('@aws-sdk/client-secrets-manager');
  const secretsManagerClient = new SecretsManagerClient();

There are more environment variables that you can set in you terminal to accomplish different things, here is a list of supported environment variables.

Getting credentials in code

Another way to provide the credentials to the SDK is by using the credentials-provider class. This requires your code to be aware of any parameters you need to set.

To make this work the same way as the environment variable route you will need to provide the profile as an input in your code.

Once you are signed in to your SSO with aws sso login --profile my-profile, you can get the credentials using the credential-providers package as shown below (If using the default profile you can start the client the same way than with environment variables)

  const { SecretsManagerClient } = require('@aws-sdk/client-secrets-manager');
  const { fromSSO } = require('@aws-sdk/credential-providers');
  const credentials = await fromSSO({
    profile: inputtedProfile
  })();

With the AwsCredentialIdentityProvider object you can initialize any client by providing the credentials and the region as shown below:

  const secretsManagerClient = new SecretsManagerClient({
    credentials: credentials,
    region: inputtedRegion
  });

Conclusion

Running the SDK from your local machine is very important and valuable to be able to provide tooling and improve your developer experience.

We went through some of the ways you can authenticate code that is being executed locally when you have several profiles .
There are other ways you can authenticate, so if these options do not work for you dueto organization security policies or any other reason, I recommend looking into the documentation or reach out to the community to find which one fits you best.
Twitter AWS Community
AWS Developers Slack Workspace

Add Authentication to Portman API tests

Andres Moreno — Fri, 25 Mar 2022 01:22:23 +0000

When creating an API you want to have security to avoid bad usage and incurring cost by unintended use. Whenever we add a security layer to our APIs any automation that we have around it automatically becomes more complex. Portman provides ways to configure authentication to be used when it runs. In this post we will configuring Portman to successfully run against our secure API.

We will be using the API that we already added Portman configuration to in the Get better API testing by using Portman post. First we need to add security to the API so it only allow access to calls made with a valid API key, after that we can configure Portman to use that API Key and successfully make all the requests needed.

Here is the source code for the end result. If you want to follow along clone/download this commit.

Add Security to API Gateway

To add API Key authentication to the API we will be adding the Auth property in the API resource in the SAM template as shown below:

ProductsAPI:
    Type: AWS::Serverless::Api
    Properties:
        StageName: Lambdaless
        Auth:
          ApiKeyRequired: true
          UsagePlan:
            CreateUsagePlan: PER_API
            UsagePlanName: ProductsAPIUsagePlan
        DefinitionBody:
          'Fn::Transform':
            Name: AWS::Include
            Parameters:
              Location: ./products-openapi.yaml

We will also be adding the ApiKeyId in the Outputs section to help us retrieve the API Key later

ApiKeyId:
    Description: API Key ApiKeyId
    Value: !Ref ProductsAPIApiKey

Once the API is deployed with these changes, it will now require an API Key to be passed in the header. This will cause the Portman tests to fail with a 403 since it is not providing the header yet. I will not go into any more details around how security works for API Gateway since it is not in the scope of this article.

Get the API Key

There are two ways you can get the API Key that was generated:

Go to the console and retrieve it from the API Gateway service
With the CLI. Below I will be explaining how to do it from the CLI.

First we need to get the API Id. We can do this by executing the following command:

aws cloudformation describe-stacks --stack-name products-service

The response for this command comes back with the Outputs defined in the SAM template where we have included the ApiKeyId. Copy the value presented in the OutputValue.

Now we need to run following command replacing the --api-key value with the value copied from the previous step to get the API Key that we can include in our headers:

aws apigateway get-api-key --api-key REPLACE_WITH_APIID --include-value

The API Key will be contained in the "value" attribute in the response. Copy this since we will be using it for the portman configuration in a bit.

Add security definitions in the OpenAPI

If you try to run Portman right now you will get 403 Forbidden responses because Portman is not providing an API key to the request. Portman uses your OpenAPI definition to generate and run the tests, if we do not define the securitySchemes it will not know to apply them. So first we need to setup the securitySchemes supported by our API which in our case is only API Key, to do this we will add the following YAML under the components section of our OpenAPI spec.

 securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-KEY

OpenAPI supports several authenticaton schemes if you wish to learn more about these please look at the OpenAPI documentation here.
Portman currently suports the API Key, HTTP basic auth and HTTP bearer token security schemes.

In OpenAPI you can apply a different security scheme per endpoint, in our case we want to apply the same one across the whole API. To do this we will be adding the root level security property to define it as shown below.

security:
  - ApiKeyAuth: []

If we run Portman again we will still get a 403 response since we are still not telling Portman what API Key to use.

Configure Portman to use an API Key

Portman allows you to configure securityOverwrites in the globals section of your Pormtan config file.
We will be updating ours to include an overwrite for the apiKey.

{
  "securityOverwrites": {
    "apiKey": {
      "value": "INSERT THE API KEY HERE"
    }
  }
}

Now if we run Portman we wll get a successful run just as we did prior to adding security to the API.

Recap

In this post we successfully tested a secure API using Portman by

Adding security to the API
Updating the OpenAPI spec to define the type of authentication to be used
Configured Portman to successfully include an API key in the header of every request to be able to automate our API testing for secure APIs