DEV Community: Ali Haydar

Exploring the Inner Workings of SST Live Lambda

Ali Haydar — Sun, 25 Feb 2024 06:10:31 +0000

In the evolving world of serverless architecture, the developer experience is a commonly debated topic. In particular, the challenges associated with testing locally and obtaining a fast feedback loop.

Nowadays, we have a few technologies that make it convenient to develop serverless applications locally, enabling a swift integration between your local environment and the cloud (e.g. SAM Sync, CDK Watch and SST Live Lambda).

In a previous article, I demonstrated how to use the SST Live Lambda with a small project that uses a deployed endpoint in AWS API Gateway to invoke a locally developed Lambda.

I wanted to know how this works. Through this post, I aim to delve into the specifics of the underlying architecture of the SST Live Lambda functionality. For the official docs, you can find them here.

Requests are proxied to the local machine

SST docs state that the requests are proxied to the local machine, which allows SST to run the local version of the function with the event, context and credentials of the remote Lambda function. The communication between local and remote happens using AWS IoT over WebSocket.

Let's check the details!

Create a new SST project: ```

npx create-sst@latest
? Project name demo-sst
✔ Copied template files
Next steps:

cd demo-sst
npm install (or pnpm install, or yarn)
npm run dev


* Run `npm install` to install the dependencies
* Update the `stacks` directory with the resources you'd like to provision
* Run `npm run dev` to start the project

Two things happen When you run `npm run dev`:

### SST creates a bootstrap stack in CloudFormation

The stack creates resources that are needed in the deployment process of your app (more details [here](https://docs.sst.dev/advanced/bootstrapping)).

Note that each app needs to be bootstrapped once, so the next time we run `npm run dev` this step won't be executed.

As this article is focused on the Live Lambda, we will only focus on the bootstrap script a little. In summary, the following resources were created:

- An S3 bucket that stores critical information about the apps (e.g. prod/dev mode, the config needed to store secrets and variables, etc.).
- A Lambda function that automatically deletes the S3 bucket objects when they are no longer needed
- A lambda function that handles the collection and uploading of metadata to the S3 bucket
- An EventBridge rule that triggers the MetadataHandler Lambda function

### SST Deploys Your Application Resources

SST deploys the application resources as defined in the `stacks` directory, enabling the usage of the Live Lambda feature.

Let's look at some details beyond what's mentioned in the SST documentation.

I deployed a simple stack that includes API Gateway and a Lambda function:

import { StackContext, Api } from 'sst/constructs';

export function API({ stack }: StackContext) {
const api = new Api(stack, 'api', {
routes: {
'GET /': 'packages/functions/src/lambda.handler',
},
});

stack.addOutputs({
ApiEndpoint: api.url,
});
}


You can see the resources in CloudFormation and verify that the API Gateway and the Lambda function were created successfully.

First, the deployment will include all the resources defined in the `stacks` directory but will stub the lambda function by one defined by SST (not the real lambda function having your code). The diagram would look as follows: 
![SST Live Lambda Deployment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dsbdi5yk381ioqqesfow.png)

- The "Deployed Stack" is what you defined but with a stub lambda function
- The AWS account is configured with a default IoT endpoint in each region. SST will use that endpoint based on the region configured `sst.config.ts`
- SST starts a local WebSocket client and connects to that IoT endpoint.

Once the stack is deployed, we're outputting the API Gateway Endpoint that we can request. Send a GET request to this endpoint (e.g. `curl https://ser55ggwrf.execute-api.us-east-1.amazonaws.com`). Below is how the flow works:
![SST Live Lambda Invocation](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i9xrv2b9ure21toqwf1w.png)

There are still more details to be discovered, especially on how the comms happen between the IoT endpoint and the stub Lambda, and the response relayed back to that Lambda.

Lambda to S3: Better Reliability in High-Volume Scenarios

Ali Haydar — Tue, 06 Feb 2024 04:51:29 +0000

I have an API Gateway endpoint that routes requests to a lambda function that writes data into an S3 bucket.

In high-volume scenarios, when managing a significant influx of requests, there is an increased risk of failures when writing data to S3. This could be due to various factors such as network issues, S3 service disruptions, concurrent write conflicts, or exceeding capacity limits (Amazon S3 supports request rates of up to 3500 Put requests per second to a single prefix. In some scenarios, rapid concurrent Put requests to the same key can result in a 503 response.). Therefore, addressing and mitigating these potential failure points is essential to ensure the reliability of data writes to S3 in such situations.

An example where this might occur is having a webhook that hits your endpoint when a change happens in an external system. I've experienced this in tools such as Slack, Github, Jira, etc. Considering the large volume of data generated by these tools, using your endpoint frequently at a larger scale could pose challenges.

Settle in with your favourite beverage as we're about to explore some details together; this post is set to be a bit lengthier than usual.

Build the project

We will start by building a project with SST that provisions an API Gateway, a Lambda, and an S3 bucket. Once implemented, we'll look into testing for concurrent write conflicts or exceeding capacity limits.

The architecture looks as follows:

Run the following to create the project locally:

npx create-sst@latest
Ok to proceed? (y) y
? Project name s3-writes-reliability
✔ Copied template files

Next steps:
- cd s3-writes-reliability
- npm install (or pnpm install, or yarn)
- npm run dev

Delete the unnecessary infrastructure under stacks and packages. The infrastructure lives under stacks and the actual Lambda code lives under packages.

Create the following stack:

import { StackContext, Api, Bucket, Config } from "sst/constructs";
import * as iam from "aws-cdk-lib/aws-iam";

export function API({ stack }: StackContext) {

  const bucket = new Bucket(stack, "Uploads");

  const api = new Api(stack, "api", {
    routes: {
      "POST /todo": "packages/functions/src/lambda.handler",
    },
  });

  const BUCKET_NAME = new Config.Parameter(stack, "BUCKET_NAME", {
    value: bucket.bucketName,
  });

  api.attachPermissions([
    new iam.PolicyStatement({
      actions: [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:ListBucket",
        "s3:GetObject",
      ],
      effect: iam.Effect.ALLOW,
      resources: [
        bucket.bucketArn + "/*",
      ],
    }),
  ]);

  api.bindToRoute("POST /todo", [BUCKET_NAME, bucket]);

  stack.addOutputs({
    ApiEndpoint: api.url,
  });

}

This stack will provision the API Gateway endpoint, the lambda function and the S3 bucket to which we will write.

Now let's update the lambda function under packages/functions/src/lambda.ts:

import { ApiHandler } from "sst/node/api";
import { Config } from "sst/node/config";
import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({});

export const handler = ApiHandler(async (_evt) => {

  const params = {
    Body: _evt.body,
    Bucket: Config.BUCKET_NAME,
    Key: Math.random().toString(),
   };

  const response  = await client.send(new PutObjectCommand(params));

  return {
    statusCode: 200,
    body: JSON.stringify(response)
  };
});

We built an API endpoint that invokes a Lambda, which in turn puts an object in S3.

Deliberate Error Creation

We will explore the S3 Capacity limit (3500 requests/second), looking to hit that limit. This exercise's purpose is to mirror real-life scenarios where a Lambda function might be invoked at scale, allowing us to explore the behaviour and performance of the system under heavy load.

First attempt - lambda making lots of requests - failed

To achieve this, I first tried to modify the Lambda function to simulate the generation of this significant volume of requests. I changed the lambda function to the following:

import { ApiHandler } from "sst/node/api";
import { Config } from "sst/node/config";
import { PutObjectCommand, S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({});

async function uploadObject(key:any, data: any) {
  const params = {
    Bucket: Config.BUCKET_NAME,
    Key: key,
    Body: data
  };

  try {
    await client.send(new PutObjectCommand(params));
    console.log(`Successfully uploaded: ${key}`);
  } catch (error) {
    console.error(`Error uploading ${key}: ${error}`);
  }
}



export const handler = ApiHandler(async (_evt) => {
  try {

    const promises = [];

    for (let i = 1; i <= 5000; i++) {
      promises.push(uploadObject(String(i), _evt.body));
    }
    await Promise.all(promises);
  } catch (error) {
    console.error("Unhandled error in handler:", error);

    return {
      statusCode: 500,
      body: JSON.stringify({ error: "Internal Server Error", details: error.message }),
    };
  }
});

Afterwards, I used curl to invoke the function (Change the API URL to point to the endpoint that SST outputted upon starting the project):

curl -X POST -H "Content-Type: application/json" -d '{"key1": "value1", "key2": "value2"}' https://sdfsdfsdf.execute-api.ap-southeast-2.amazonaws.com/todo

I didn't get any 503 Slow Down from S3, indicating that the number of requests is high - that's because the script above won't run within a single second.

Second attempt - loading the API endpoint - successful
The second attempt invokes the Lambda function with lots of simultaneous requests. This would spin up multiple Lambda instances to cater for the load, and each instance would hit S3 at the same time. Let's see if that'll work.

I'll use Apache JMeter to do this experiment:

This test resulted in many errors related to the Lambda function being throttled due to exceeding the rate limit "errorType":"ThrottlingException","errorMessage":"Rate exceeded".

I've changed the Jmeter threads to 1000 to match the limit of 1000 concurrent executions allowed by Lambda in a single region. At the same time, I kept using promise.all in the Lambda code to achieve parallel requests to S3. With this combination, I managed to get a few SlowDown: Please reduce your request rate. errors.

Using Jmeter is a better approach to simulate a real case scenario, such as having a Slack or Github webhook that is hitting the endpoint.

Build a more reliable system

We managed to get the S3 service to return an error that we're exceeding the rate limit. How can we build a more reliable architecture that would help us avoid this kind of error?

We have a few options here:

Distribute objects across multiple S3 bucket prefixes as the rate limit is set by prefix (this means changing the behaviour of the application, which might not be something you're planning to do)
Use a retry mechanism - this could be handled on the Lambda side, where we retry submitting requests that return a 503 error. This is a great solution, but it depends on your traffic and whether postponing the request would work. It's worth testing this solution, especially since it doesn't require a big effort.
Alleviate the load on S3 by introducing a queue between your APIGW and Lambda function. We'll implement this solution.

The architecture looks as follows:

In this diagram, SQS serves as a buffer between APIGW and Lambda. Instead of directly processing incoming requests, we enqueue them in an SQS queue. This decouples the incoming request rate from the rate at which Lambda and S3 can process the requests, which smoothes out spikes in the incoming request rate, reducing the immediate load on S3 and providing a more controlled flow of data. If there are temporary issues with S3 or Lambda, messages can be retained in the SQS queue and retried, providing a more resilient system.

Let's create this additional infrastructure. Modify the previously created stack as follows:

  const queue = new Queue(stack, "Queue", {
    consumer: {
      function: {
        handler: "packages/functions/src/lambda.handler",
        timeout:10,
        environment:{ BUCKET_NAME: bucket.bucketName },
      }
    }
  });

  queue.attachPermissions([
    new iam.PolicyStatement({
      actions: [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:ListBucket",
        "s3:GetObject",
      ],
      effect: iam.Effect.ALLOW,
      resources: [
        bucket.bucketArn + "/*",
      ],
    }),
  ]);

This creates a queue and repurposes the Lambda we created to poll that queue. In addition, we attached the permission to put objects in S3 to the consumer Lambda.

Now, let's update the API route we have to write into the queue rather than triggering the Lambda function:

 const api = new Api(stack, "api", {
    routes: {
      "POST /": {
        type: "aws",
        cdk: {
          integration: {
            subtype: HttpIntegrationSubtype.SQS_SEND_MESSAGE,
            parameterMapping: ParameterMapping.fromObject({
             QueueUrl: MappingValue.custom(queue.queueUrl),
             MessageBody: MappingValue.custom("$request.body"),
            }),
          }
        }
      }
    },
  });

Once done, let's update the Lambda function. We need to update the event to be of type SQSEvent, then pass the SQS Record body to the uplaodObject function:

      const promises = _evt.Records.map(record => uploadObject(record.messageId,record.body));
      await Promise.all(promises);

Is that enough to alleviate the load from S3 so that we stop having request rate errors? Let's try it again using Jmeter.

Unfortunately, I still encountered a few | + Error uploading 9: SlowDown: Please reduce your request rate. errors. Setting the eventSource batchSize to 5 seems to have fixed the issue of the rate limit on S3. It might not work where requests are being more aggressively sent. In this case, we can still set a deliveryDelay parameter on the queue, which postpones the delivery of new messages for some time.

Conclusion

In addressing challenges related to high-volume data writes on S3, we’ve proposed a more reliable architecture involving queues. However, this solution isn’t one-size-fits-all. The queue solution has its own problems to consider — how to handle failed uploads, and should a dead letter queue be considered? Is the message size limit in SQS a problem?

Building a more reliable system is an iterative process, and this article provides a foundation for enhancing the robustness of serverless architectures in handling substantial data loads. Ultimately, developers are encouraged to adapt and refine these solutions based on their specific use cases.

Finally, I wouldn’t recommend rushing into building this kind of solution from the first day, unless you’re dealing with some serious data traffic. As with all software development, it’s important to be pragmatic and iteratively build your system blocks as you need.

How do you enhance the resilience of your system? Share your thoughts in the comments.

Exploring Local Development with SST for Serverless Applications

Ali Haydar — Fri, 29 Dec 2023 06:41:22 +0000

A couple of years ago, I wrote an article on freeCodeCamp about testing serverless applications in AWS. In that article, I explained the technical aspects of testing a serverless application formed of an API Gateway, a Lambda function and a DynamoDB table. I talk about how to test the function using unit tests, the DB with a bash script, and the overall app (i.e. the API endpoint) with e2e tests. However, there's an important piece that I missed in that article. Where does all of the testing happen? I assumed that testing occurs on the cloud, except for unit tests.

A compelling question frequently arises in my enthusiastic discussions about serverless technology: How can we conduct local testing?

I typically encourage the usage of mocks, but I think it's more suitable for testing in isolation (e.g. testing a single component of your app, mocking external services it integrates with). I am not a fan of Emulators; my experience with these tools that mimic a cloud service has been brittle, mainly due to setup complexity and parity with the real service used.

A few years ago, I failed a technical interview because I did not offer a way to start and test a serverless app in a local environment. There's a need for a mindset change when working with serverless resources; some might disagree. That is going to be a discussion for a different day. For this post, I wanted to explore how to bridge the gap between cloud-based testing and the convenience of local development.

Photo by Ben Mack

I read about SST (Serverless Stack) and the live Lambda feature it offers. The SST Docs mention:

Live Lambda Development or Live Lambda is feature of SST that allows you to debug and test your Lambda functions locally, while being invoked remotely by resources in AWS. It works by proxying requests from your AWS account to your local machine.

Let's explore the Live Lambda feature of SST. This will be an opportunity to set up an app with SST and evaluate the developer experience.

Building an app with SST

SST relies on AWS CDK to write the infrastructure code using a set of programming languages (We'll use TypeScript in our example). The CDK converts that code into CloudFormation templates behind the scenes.

SST offers a few constructs built on top of AWS CDK:

sst build runs cdk synth internally. This converts the code to CloudFormation templates
pnpm sst dev or pnpm sst deploy runs cdk deploy. This submits the templates to CloudFormation, which creates the stacks and their defined resources.

Let's create an SST project:

npx create-sst@latest
? Project name demo-sst
✔ Copied template files

Next steps:
- cd demo-sst
- npm install (or pnpm install, or yarn)
- npm run dev

The infrastructure code is listed under /stacks, and the application code is listed under /packages. Notice that under MyStack.ts, an event bus and a few API routes are defined. Feel free to remove these, keeping the following only:

import { StackContext, Api } from 'sst/constructs';

export function API({ stack }: StackContext) {
  const api = new Api(stack, 'api', {
    routes: {
      'GET /': 'packages/functions/src/lambda.handler',
    },
  });

  stack.addOutputs({
    ApiEndpoint: api.url,
  });
}

This code defines a GET API endpoint that will invoke the lambda function.

Feel free to delete the folders and files that aren't needed under packages.

Once the app is started and deployed, you will get the API URL, which we specified as an output in the /stacks/MyStack.ts file. Requesting this API will return the value define in the /packages/functions/src.lambda.ts file. Change the returned value there and submit a new request to the API endpoint. Notice how you got the new value.

This is the SST Live Lambda. Without the long loop of deploying your changes to AWS, you get to execute your local Lambda function from the API Gateway already deployed in AWS.

SST mainly creates a WebSocket and proxies requests from Lambda functions to your local machine. You can find more details in the SST docs.
Here's how it looks like:

How is this different from running your Lambda using a NodeJS worker?

Even though the functional behaviour of your Lambda code might be similar (i.e. experiencing similar behaviour of the Lambda function), running a Live Lambda offers a fully integrated environment where the Lambda configuration is as close to production since the whole environment is already running in the cloud. That provides better confidence over a local setup where you rely on AWS creds to do the execution. The integration allows you to test integrations with other AWS services such as API Gateway, Event Bridge, S3, etc. This offers a great win over setting up emulators on your local.

The next step would be to build a larger-scale project that integrates with services from AWS, such as S3, SQS, Event-Bridge, etc. How do you build a serverless app? and what's your dev experience like?

Avoiding Data Overwrites in DynamoDB

Ali Haydar — Mon, 18 Dec 2023 21:30:00 +0000

In traditional database management systems, such as MySQL, there is a separation between the "INSERT" and the "UPDATE" operations. The "INSERT" statement is used to add records to a table, and the "UPDATE" statement is used to update existing records. If, for example, you try to insert a new record with an existing primary key, you would get an error.

On the other hand, DynamoDB uses a single operation PutItem to both insert or update existing records. When the PutItem operation is used with an existing primary key on the DynamoDB table, it will overwrite the data on that record.

Photo by Alex Green on Pexels

If you're not familiar with it, DynamoDB is a NoSQL database service provided by AWS, known for its seamless scalability, high performance, and simplified data model.

Is `PutItem` a risky operation?

The PutItem behaviour, where it overwrites existing data, might be considered risky in some instances if the developers aren't careful about how to update existing records. That might be simply a bug or an oversight of a certain scenario, but could as well be a result of an incorrect DynamoDB table design to start with.

Of course, DynamoDB offers conditional expression to help mitigate the data overwrite risk. Without conditional expression, we'd need to add the logic in our code to check whether an item exists or not, and that's costly. This also comes with the risk of race conditions if other requests are trying to update the same item.

Example where data is overwritten

Let's see this as an example. Assume we have an "Employees" table designed as follows:

When creating a new employee, we'd want to ensure we don't overwrite an existing user. Below we write the code using Terraform and JavaScript.

For simplicity, I'll post the Terraform code in a single file main.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
variable "aws_region" {
  type    = string
  default = "ap-southeast-2"
}

provider "aws" {
  region = var.aws_region
}

resource "aws_dynamodb_table" "employees_table" {
  name         = "employees"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "username"
  attribute {
    name = "username"
    type = "S"
  }
}

# IAM Role
resource "aws_iam_role" "add_employee_lambda_role" {
  name               = "add_employee_lambda_role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

# IAM Policy that allows interaction with DynamoDB
resource "aws_iam_policy" "add_employee_lambda_policy" {
  name        = "add_employee_lambda_policy"
  description = "Allow lambda to access dynamodb"
  policy      = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "dynamodb:PutItem",
        "dynamodb:DeleteItem",
        "dynamodb:GetItem",
        "dynamodb:Scan",
        "dynamodb:Query",
        "dynamodb:UpdateItem"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

# Attach IAM Policy to IAM Role
resource "aws_iam_role_policy_attachment" "add_employee_lambda_role_policy_attachment" {
  role       = aws_iam_role.add_employee_lambda_role.name
  policy_arn = aws_iam_policy.add_employee_lambda_policy.arn
}

# data archive for lambda function
data "archive_file" "add_employee_lambda_zip" {
  type        = "zip"
  source_dir  = "${path.module}/../src"
  output_path = "${path.module}/add_employee.zip"
}

# Add employee lambda function
resource "aws_lambda_function" "add_employee_lambda" {
  filename         = data.archive_file.add_employee_lambda_zip.output_path
  function_name    = "add_employee_lambda"
  role             = aws_iam_role.add_employee_lambda_role.arn
  handler          = "add_employee.handler"
  source_code_hash = data.archive_file.add_employee_lambda_zip.output_base64sha256
  runtime          = "nodejs18.x"
  publish          = true
  environment {
    variables = {
      DYNAMODB_TABLE = aws_dynamodb_table.employees_table.name
    }
  }
}

Next, we'll write a simple JS Lambda function to put items into DynamoDB:

import {
  DynamoDBClient,
  PutItemCommand,
} from '@aws-sdk/client-dynamodb';
import { marshall } from '@aws-sdk/util-dynamodb';

const tableName = process.env.DYNAMODB_TABLE;
const client = new DynamoDBClient({});

export const handler = async (event) => {
  const { username, name, department, jobTitle } = event;

  const params = {
    TableName: tableName,
    Item: marshall({
      username,
      name,
      department,
      jobTitle,
    }),
  };
  const command = new PutItemCommand(params);
  const response = await client.send(command);
  console.log(response);
  return response;
};

Deploy this infrastructure (terraform apply), then navigate to the Lambda function in AWS Console and test the lambda with the following event object:

{
  "username": "ali",
  "name": "Ali Haydar",
  "department": "Engineering",
  "jobTitle": "Platform Lead"
}

Verify the item got added to DynamoDB.

Assume a new employee with the same first name joined the company, and the admin just went ahead with the following employee data (let's use the Lambda test functionality to add the new employee - of course, in a real-life scenario there would be a UI for that):

{
  "username": "ali",
  "name": "Ali Wong",
  "department": "Acting",
  "jobTitle": "Stand-up comedian and acress"
}

A few days later, the first employee "ali" got promoted to Chief Finance Officer, so it's time to update this info in the system. When searching for "ali", only "Ali Wong" was found. Where's our intended user?

That was an example where an oversight and a poor design of the table caused a bug leading to data loss.

example with conditional expression

To protect against this kind of mistake, we modify our code slightly, adding a conditional expression to the params:

ConditionExpression: 'attribute_not_exists(username)',

Deploy the change.

Try to update the existing "Ali Wong" record with the following data:

{
  "username": "ali",
  "name": "Ali Haydar",
  "department": "Engineering",
  "jobTitle": "Platform Lead"
}

Notice the error:

"errorType": "ConditionalCheckFailedException",
"errorMessage": "The conditional request failed",

The conditional expression checks for the non-existence of the username before allowing the update.

The conditional expressions can be useful in multiple other use cases. Have a look at the docs.

I've seen patterns in relational databases where an operation behaves similarly to PutItem in DynamoDB. Mostly, these were code-built functions that handle the logic of "Insert" or "Update". In some cases, we referred to them as "upsert" operations. The concept is often implemented using statements like INSERT ON DUPLICATE KEY UPDATE in MySQL.

In the example above, even though there is no direct impact of the table design; using a username as a primary key makes it more prone to data overwrite, especially if that's a manually entered username. In addition, it does limit the access patterns to the table, but that's a discussion for another day. One way to improve this might be to use a composite key of username and department for example. What do you think?

How to enhance your Lambda function performance with memory configuration?

Ali Haydar — Mon, 21 Aug 2023 22:30:00 +0000

I have a lambda function that's taking a few seconds to execute, and that's not related to a cold start. It's the code in the handler function that's taking time, and that's because it's CPU and memory intensive.

Performance problems related to cold start can be enhanced through provisioned concurrency and managing the "Init" code. You can read about this in my previous articles:

We know the performance is related to CPU and Memory, so the solution might be as simple as increasing the memory. Isn't it?

Let's do a small demonstration comparing the execution of the Lambda with different memory configurations. It's worth noting that we cannot explicitly control the Lambda CPU; it's automatically increased as we increase the memory. From the AWS docs:

The amount of memory also determines the amount of virtual CPU available to a function. Adding more memory proportionally increases the amount of CPU, increasing the overall computational power available. If a function is CPU-, network- or memory-bound, then changing the memory setting can dramatically improve its performance.

Testing the performance of a Lambda function

For this demo let's use the following Lambda function that I generated with the help of ChatGPT:

export const handler = async (event) => {
    const matrixSize = 100; // Size of the square matrix
    const matrix = generateMatrix(matrixSize);
    const result = performMatrixMultiplication(matrix);

    return result;
};

function generateMatrix(size) {
    const matrix = [];
    for (let i = 0; i < size; i++) {
        const row = [];
        for (let j = 0; j < size; j++) {
            row.push(Math.random());
        }
        matrix.push(row);
    }
    return matrix;
}

function performMatrixMultiplication(matrix) {
    const size = matrix.length;
    const result = new Array(size);

    for (let i = 0; i < size; i++) {
        result[i] = new Array(size);
        for (let j = 0; j < size; j++) {
            let sum = 0;
            for (let k = 0; k < size; k++) {
                sum += matrix[i][k] * matrix[k][j];
            }
            result[i][j] = sum;
        }
    }

    return result;
}

The Lambda function generates a large square matrix, performs matrix multiplication, and returns the resulting matrix. That's an example of a CPU and memory-intensive function.

Lambda with 128 MB memory

Create this function manually in the AWS console, selecting NodeJs as the runtime, and keep the rest of the configuration to the default. This will create the Lambda with a 128 MB memory.
Execute the function by navigating to the "Test" tab and clicking the "Test" button.

Notice the results:

Duration: 298.55 ms Billed Duration: 299 ms Memory Size: 128 MB Max Memory Used: 72 MB  Init Duration: 160.36 ms

We got a duration of 298.55 ms.

Lambda with 256 MB memory

In the configuration tab, update the Lambda memory to 256 MB
Test the lambda function

Notice the results:

Duration: 133.94 ms Billed Duration: 134 ms Memory Size: 255 MB Max Memory Used: 73 MB  Init Duration: 162.88 ms

We got a duration of 133.94 ms by doubling the memory.

Lambda with 512 MB memory

In the configuration tab, update the Lambda memory to 512 MB
Test the lambda function

Notice the results:

Duration: 51.12 ms  Billed Duration: 52 ms  Memory Size: 512 MB Max Memory Used: 73 MB  Init Duration: 144.95 ms

We got a duration of 51.12 ms by doubling the memory.

Lambda with 1024 MB memory

In the configuration tab, update the Lambda memory to 1024 MB
Test the lambda function

Notice the results:

Duration: 30.42 ms  Billed Duration: 31 ms  Memory Size: 1024 MB  Max Memory Used: 73 MB  Init Duration: 163.51 ms

We got a duration of 30.42 ms by doubling the memory

Shall we continue to increase the memory?

We have demonstrated that more memory means faster execution.

At some point, the increased memory, which comes with an increased CPU won't cause better performance. But for now, let's accept that a 30.42 ms execution time is reasonable. It is worth noting that the Memory used in all execution was 73MB, so it's the CPU that was improving with the memory upgrades.

How to choose a memory config?

As with any software configuration, there are some trade-offs. In this case, it's the cost of Execution.

Lambda is priced based on number of requests and the duration of requests. In addition to that, the price of request duration is dependent on the memory associated with the Lambda function. From the AWS Docs:


Memory (MB) Price per 1ms
128           $0.0000000021
512           $0.0000000083
1024        $0.0000000167
1536        $0.0000000250
2048        $0.0000000333
3072        $0.0000000500
4096        $0.0000000667
5120        $0.0000000833
6144        $0.0000001000
7168        $0.0000001167
8192        $0.0000001333
9216        $0.0000001500
10240       $0.0000001667

So how can we find a good balance between memory and cost?

The aws lambda power tuning tool helps optimise the Lambda performance and cost in a data-driven manner. Let's try it out:

There are multiple ways to deploy the application into the AWS account. We'll deploy it from the AWS Serverless Application Repository as that's the simplest option.
The deployment of this serverless application will create a step function starting with the name powerTuningStateMachine
Start the execution of the step function and pass the following JSON:

{
    "lambdaARN": "your-lambda-function-arn",
    "powerValues": [128, 256, 512, 1024, 1536, 2048, 3008],
    "num": 50,
    "payload": {},
    "parallelInvocation": true,
    "strategy": "cost"
}

Once executed this will generate the following output in the "Execution input and output" tab of the step function:

{
  "power": 512,
  "cost": 1.428e-7,
  "duration": 16.566666666666666,
  "stateMachine": {
    "executionCost": 0.00033,
    "lambdaCost": 0.00011801685000000001,
    "visualization": "https://lambda-power-tuning.show/XXXX"
  }
}

The visualisation looks as follows:

This graph shows that memory of 512 MB would offer a good balance of cost and performance.

Thanks for reading this far. Did you like this article, do you have feedback or would like to further discuss the topic? Please reach out on Twitter or LinkedIn.

Effortlessly Juggling Multiple Concurrent Requests in AWS Lambda

Ali Haydar — Mon, 26 Jun 2023 16:58:30 +0000

Have you ever tried to handle a ton of requests at once with AWS Lambda? What if you had a critical operation that required speed, scaling the performance of a request down to single-digit milliseconds?

In a previous article, I discussed how to speed up your Lambda function, touching on the concept of a 'cold start' and how to minimize it by moving some non-critical code outside the Lambda handler, such as establishing a DB connection.

When we send a request to execute a Lambda function, the service initialises an execution environment. That's a virtual machine (MicroVM) dedicated to each concurrent invocation of a Lambda function. Within this VM, there's a kernel, a runtime (e.g. Node.js), function code, and extensions code. The cold start period includes creating this environment, initialising the runtime and extensions, and downloading the code. It also includes the execution of the 'Init' code (the code you write before the handler function).

With multiple consecutive executions, the Lambda service would reuse a previously created execution environment, and only execute the code within the handler function (that's what we call warm start). That's fascinating and would be sufficient in lots of cases. However, in some instances, we need to execute a single Lambda function concurrently. That would cause the creation of a new execution environment for each execution, which means that we cannot leverage the warm Lambda, and would need to experience the cold start for each invocation.

The solution in this case is to set up provisioned concurrency. Let's test it.

Set up the project

We will build a simple project consisting of 2 Lambda functions, one orchestrator and one worker that will be invoked concurrently from the orchestrator Lambda.

Let's create the infrastructure in Terraform:

Create the IAM policies and roles that will be used by the Lambda functions

  locals {
    policy_document_cloudwatch = {
      Version = "2012-10-17"
      Statement = [
        {
          Effect = "Allow"
          Action = [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
          ]
          Resource = "arn:aws:logs:*:*:*"
        }
      ]
    }
    policy_document_lambda_invoke = {
      Version = "2012-10-17"
      Statement = [
        {
          Effect   = "Allow"
          Action   = ["lambda:InvokeFunction"]
          Resource = ["arn:aws:lambda:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:function:*"]
        }
      ]
    }
  }

  data "aws_iam_policy_document" "lambda_assume_role" {
    statement {
      actions = ["sts:AssumeRole"]
      principals {
        type        = "Service"
        identifiers = ["lambda.amazonaws.com"]
      }
    }
  }

  resource "aws_iam_role" "worker_lambda_execution_role" {
    name               = "worker_lambda-exec-role"
    assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
  }


  resource "aws_iam_role_policy_attachment" "lambda_policies_attachment_cloudwatch" {
    role       = aws_iam_role.worker_lambda_execution_role.name
    policy_arn = aws_iam_policy.lambda_policies_cloudwatch.arn
  }

  resource "aws_iam_policy" "lambda_policies_cloudwatch" {
    name        = "lambda_logging_cloudwatch_access"
    description = "lambda logs in CloudWatch"
    policy = jsonencode({
      Version   = "2012-10-17"
      Statement = local.policy_document_cloudwatch.Statement

    })
  }


  resource "aws_iam_role" "orchestrator_lambda_execution_role" {
    name               = "orchestrator-lambda-exec-role"
    assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
  }


  resource "aws_iam_role_policy_attachment" "lambda_policies_attachment_cloudwatch_and_lambda_invocation" {
    role       = aws_iam_role.orchestrator_lambda_execution_role.name
    policy_arn = aws_iam_policy.lambda_policies_cloudwatch_and_lambda_invocation.arn
  }


  resource "aws_iam_policy" "lambda_policies_cloudwatch_and_lambda_invocation" {
    name        = "lambda_logging_cloudwatch_access_and_lambda_invocation"
    description = "lambda logs in CloudWatch and worker lambda invocation"
    policy = jsonencode({
      Version = "2012-10-17"
      Statement = concat(
        local.policy_document_cloudwatch.Statement,
        local.policy_document_lambda_invoke.Statement
      )
    })
  }

Create a lambda.tf file configuring the lambda functions

  resource "aws_lambda_function" "orchestrator_lambda" {
    function_name = "orchestrator_lambda"

    handler = "orchestrator_lambda.handler"
    runtime = "nodejs18.x"

    filename = "orchestrator_lambda_function.zip"

    source_code_hash = filebase64sha256("orchestrator_lambda_function.zip")

    role = aws_iam_role.orchestrator_lambda_execution_role.arn

    depends_on = [
      aws_iam_role_policy_attachment.lambda_policies_attachment_cloudwatch_and_lambda_invocation
    ]

    environment {
      variables = {
        FUNCTION_NAME = aws_lambda_function.worker_lambda.function_name
      }
    }
  }


  resource "aws_lambda_function" "worker_lambda" {
    function_name = "worker_lambda"

    handler = "worker_lambda.handler"
    runtime = "nodejs18.x"

    filename = "worker_lambda_function.zip"

    source_code_hash = filebase64sha256("worker_lambda_function.zip")

    role = aws_iam_role.worker_lambda_execution_role.arn

    depends_on = [
      aws_iam_role_policy_attachment.lambda_policies_attachment_cloudwatch
    ]
  }

We have yet to define any concurrency. We will get to this later.

Now let's implement our Lambda functions:

Create a new orchestrator_lambda.js file

  import { InvokeCommand, LambdaClient, LogType } from "@aws-sdk/client-lambda";

  const invokeLambda = async (payload) => {
    const client = new LambdaClient();
    const command = new InvokeCommand({
      FunctionName: process.env.FUNCTION_NAME,
      LogType: LogType.Tail,
      Payload: JSON.stringify(payload),
    });
    await client.send(command);

    console.log("Lambda function invoked!");
  };

  export const handler = async (event, context) => {
    const promises = [];

    for (let i = 0; i < 10; i++) {
      const payload = { invocation: i + 1 };
      promises.push(invokeLambda(payload));
    }

    await Promise.all(promises);
  };

Create a new worker_lambda.js file

  export const handler = async (event, context) => {
    console.log("here is the event received: ", event);
  };

Test the concurrent Lambda execution before and after setting up the provisioned concurrency

Before Provisioned Concurrency

In the AWS Console, navigate to the "Orchestrator Lambda" and click the "Test" button. This will invoke the "Worker Lambda" 10 times.

Have a look at the CloudWatch logs of the "Worker Lambda":

Notice the "Worker Lambda" got invoked 10 times concurrently. Each time starting a new execution environment and initialising it. That "cold start" duration shows as the "Init Duration" in the previous screenshot; there are ~ 160ms used for "Init" in every invocation. What if we can save this time?

After Provisioned Concurrency

Let's set up provisioned concurrency for the "Worker Lambda". This will allocate a specified number of execution environments for the Lambda function, which would ensure rapid function start-up, skipping the "Init" part. The Provisioned concurrency works only with a Lambda alias (a pointer to a specific Lambda version) or Lambda version, so we'll set this up as well.

In your lambda.tf file, under the worker_lambda resource, add the publish property, which will create a Lambda version:

  publish       = true

Add the Lambda alias, which will point to the created version of the Lambda

  resource "aws_lambda_alias" "worker_lambda_alias" {
    name             = "worker_lambda_alias"
    function_name    = aws_lambda_function.worker_lambda.function_name
    function_version = aws_lambda_function.worker_lambda.version // This is the latest published version
  }

Add the provisioned concurrency configuration

  resource "aws_lambda_provisioned_concurrency_config" "worker_lambda_provisioned_concurrency" {
    function_name                     = aws_lambda_alias.worker_lambda_alias.function_name
    provisioned_concurrent_executions = 10
    qualifier                         = aws_lambda_alias.worker_lambda_alias.name
  }

On the "Orchestrator Lambda" config, pass the alias as an environment variable. So the environment tag would look as follows:

  environment {
    variables = {
      WORKER_FUNCTION_NAME = aws_lambda_function.worker_lambda.function_name
      WORLER_ALIAS         = aws_lambda_alias.worker_lambda_alias.name
    }
  }

Update the Lambda "Orchestrator Lambda" code to hit the alias, in the orchestrator_lambda.js:

  const command = new InvokeCommand({
    FunctionName: `${process.env.WORKER_FUNCTION_NAME}:${process.env.WORKER_ALIAS}`,
    LogType: LogType.Tail,
    Payload: JSON.stringify(payload),
  });

After you deploy these changes, navigate to the AWS Console and click the "Test" button in the "Orchestrator" Lambda. This will invoke the "Worker Lambda" 10 times.

Have a look at the CloudWatch logs of the "Worker Lambda":

Notice that we no longer have an "Init" duration, as we have 10 provisioned Worker Lambda functions. If we submitted 11 concurrent requests, then one worker lambda will have an "Init" duration.

It's worth noting that, even with provisioned concurrency, you might still experience some instances having "Init" duration if you invoke the function immediately after deployment.

Provisioned concurrency comes with a cost as we have Lambda instances running all the time. One thing to think of is how would you balance the cost and performance benefits of using provisioned concurrency. I'd love to hear your thoughts.

Prevent API overload with rate limiting in AWS

Ali Haydar — Tue, 11 Apr 2023 21:01:26 +0000

In the early days of the covid pandemic, and for a significant period, when we went to the supermarket, we had to wait in a queue for our turn to enter the supermarket. Only a certain number of customers could shop together at a particular time. The intention was to keep a physical distance between individuals to limit the spread of the virus, hence keeping people safe and limiting the load on the health system.

That's similar to what we do in Software when we rate limit our APIs. We do it for multiple reasons, including preventing abuse (e.g. malicious user overwhelming the system with too many requests), managing downstream services and resources (e.g. database) or simply using rate limit as a way to monetize the APIs (e.g. enabling free users to use the APIs less frequently than paid users).

This post will build an API using AWS API Gateway and explore how to rate-limit calls to our endpoints. API Gateway is a managed service from AWS that helps you publish, manage and monitor your APIs. We will use Terraform to set up our infrastructure.

We deploy APIs to stages in API Gateway, where each stage has a single deployment. You could have one stage for development, one for testing and one for production. Each stage has its URL and settings.

We will build an endpoint that returns the number of people in the supermarket. It will be of this shape: GET /.../prod/customers

Build the API

In Terraform:

Create the REST API

  resource "aws_api_gateway_rest_api" "api" {
    name        = "SupermarketCustomers"
    description = "Tracks the number of customers that enter the supermarket"
    endpoint_configuration {
      types = ["REGIONAL"]
    }
  }

Create the "Customers" resource. That will be the first endpoint:

  resource "aws_api_gateway_resource" "customers_resource" {
    rest_api_id = aws_api_gateway_rest_api.api.id
    parent_id   = aws_api_gateway_rest_api.api.root_resource_id
    path_part   = "customers"
  }

Create the method for this resource - GET in this case:

  resource "aws_api_gateway_method" "get_method" {
    rest_api_id   = aws_api_gateway_rest_api.api.id
    resource_id   = aws_api_gateway_resource.customers_resource.id
    http_method   = "GET"
    authorization = "NONE"
  }

Notice the http_method in this case, and the authorization set to NONE, as we'd want this API to be accessible to everyone (some people would like to check the number of customers in the supermarket from home)

Until now, we have configured the API and method. Now we must move to the integration part, which covers interacting with the backend. Specifying an AWS service, HTTP Backend, or MOCK is possible. In this example, we will use MOCK to mock the backend and hardcode the response to return on the API Gateway.

  resource "aws_api_gateway_integration" "mock_backend" {
    http_method = aws_api_gateway_method.get_method.http_method
    resource_id = aws_api_gateway_resource.customers_resource.id
    rest_api_id = aws_api_gateway_rest_api.api.id
    type        = "MOCK"
    request_templates = {
      "application/json" = jsonencode(
        {
          statusCode = 200
        }
      )
    }
  }

Define the integration response and method response:

  resource "aws_api_gateway_method_response" "response_200" {
    rest_api_id = aws_api_gateway_rest_api.api.id
    resource_id = aws_api_gateway_resource.customers_resource.id
    http_method = aws_api_gateway_method.get_method.http_method
    status_code = 200
  }

  resource "aws_api_gateway_integration_response" "mock_backend_response" {
    rest_api_id = aws_api_gateway_rest_api.api.id
    resource_id = aws_api_gateway_resource.customers_resource.id
    http_method = aws_api_gateway_method.get_method.http_method
    status_code = aws_api_gateway_method_response.response_200.status_code

    # Transforms the backend JSON response to XML
    response_templates = {
      "application/json" = <<EOF
    {
      "message": "hello from the mocked backend"
    }
  EOF
    }
  }

As mentioned above, changes in API Gateway won't apply without a deployment, which needs to happen to a Stage:

  resource "aws_api_gateway_deployment" "api_deployment" {
    rest_api_id = aws_api_gateway_rest_api.api.id
    lifecycle {
      create_before_destroy = true
    }
  }

  resource "aws_api_gateway_stage" "prod" {
    deployment_id = aws_api_gateway_deployment.api_deployment.id
    rest_api_id   = aws_api_gateway_rest_api.api.id
    stage_name    = "prod"
  }

Deploy this configuration to your AWS account. You should be able to see your API, resource and method as per the following screenshot:

Click the "Test" button to get a response like the following:

Rate & Burst Limits

AWS API Gateway has a default set of 10,000 requests per second limit per region within an AWS account, with a burst of 5000 requests. That's across all APIs of different types (e.g. REST, WebSocket).

AWS uses the Token Bucket Algorithm to throttle requests. Each request is a token in the bucket. The burst limit is the maximum number of concurrent tokens consumed simultaneously. The rate limit is the speed at which the bucket is refilled with new tokens. You can also look at the rate limit as the maximum number of tokens that can be used within one second (or a period).

Assume we set a rate limit of 10 requests per second and a burst rate of 15 requests. That means that ten requests are added to the bucket every second up to the maximum of 15 requests. If we consume 15 requests per minute, the usage rate will be bigger than the refill rate, which means we'll consume the requests in our bucket faster. So in 2 seconds, we will start throttling requests returning a 429: Too Many Requests error. Here are some details:

In 1 second, we have 15 requests in the bucket; we consumed 15 requests and refilled 10. That leaves ten requests in the bucket
In 2 seconds, we have ten requests in the bucket; we consumed 15 requests and refilled 10. 5 of the requests consumed will return an error, as there are no more available requests in the bucket.

However, suppose we're consuming requests at the same speed as the refill rate (rate limit of 10 requests per second). In that case, we will always be able to serve these requests without a problem, as the burst rate (maximum number of requests simultaneously) will never be reached.

Setting the rate and burst limits properly based on the expected traffic levels and available resources is essential to ensure reliable service.

This is how we set the rates in Terraform at the stage and route level:

resource "aws_api_gateway_method_settings" "get_method_settings" {
  rest_api_id = aws_api_gateway_rest_api.api.id
  stage_name  = aws_api_gateway_stage.prod.stage_name
  method_path = "*/*"

  settings {
    throttling_burst_limit = 1
    throttling_rate_limit  = 2
  }
}

After you deploy this change, Invoke the endpoint URL (You can find the URL under stages -> prod -> / -> /customers -> GET). Notice the "Too many requests" error when invoked more than a single time per second.

With this approach, a single customer might overuse the API, which would cause throttling for other customers. It is possible to control this with usage plans. This requires that each customer has an API key, which we haven't set up in our example.

Have you implemented rate limiting in the past? What would you do differently?

Thanks for reading this far. Did you like this article, and do you think others might find it helpful? Feel free to share it on Twitter or LinkedIn.

How to securely expose your local app to the internet using EC2?

Ali Haydar — Tue, 31 Jan 2023 21:35:00 +0000

Are you developing an app and want others to access it before it's available in the cloud? Or are you integrating with a third-party tool and wish to enable access to your local app for a better development experience? This article is for you.

There are many use cases where you might need to expose your local app (the app you are developing on your local machine) to the internet. That could be a site, an API or a chatbot. I've worked on a few of these cases, most recently a side project developing a Slack application.

When integrating with Slack, you need to provide Slack with a redirection URL so that the user goes back to your site after granting the necessary permissions (this is common when using OAuth).

As you are working on a local machine, the URL should point to your localhost, which has a private IP. Your home/office router will have a public IP assigned by the internet provider, but any device behind that router will be given a private IP. So how can we expose our app to the internet?

How to expose your app to the internet?

Note that this setup might incur some costs.

First, I used ngrok, which creates a tunnel between my local machine and servers that are already exposed to the internet - So your requests to the ngrok server get forwarded to your local app. That's convenient and useful. However, the URL of that server changes every time you connect, which means I need to update my app config every time I connect.

I wanted something more permanent and maintainable, where I start my app, add my changes and test. One option was to subscribe to the ngrok paid services, where you can get a permanent URL. I need this for a side project and want it to be cost-effective, so I decided to implement a basic solution myself.

To achieve this, we'll create an EC2 instance that allows ingress access from the internet, and install an nginx server that acts as a reverse proxy to forward requests to this server to my app running on my local machine, using SSH tunnelling.

Let's build it with Terraform.

Create an EC2 instance

We will use the t2.micro instance as it's covered in the Free Tier for 12 months.

First, we will create a Security Group, which will be attached to the EC2 instance, to allow ingress HTTP and ssh access to the machine:

resource "aws_security_group" "allow_http_ssh_and_http_on_80" {
  name        = "allow_http"
  description = "Allow http inbound traffic"


  ingress {
    description = "http"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]

  }
  ingress {
    description = "ssh"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]

  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }


  tags = {
    Name = "allow_http_ssh"
  }
}

We need a key pair to SSH in the EC2 instance, so I used one that I had already created using the Terraform data source:

data "aws_key_pair" "ec2_instance_key_pair" {
  key_name = "ec2-instances"
}

You could create a new key pair using the aws_key_pair resource.

Next, we will create an EC2 instance - I'll install and set up nginx on this instance as user data. That's the script that runs after the instance starts.


resource "aws_instance" "port_forwarding_server" {
  ami                    = "ami-06bb074d1e196d0d4"
  instance_type          = "t2.micro"
  key_name               = data.aws_key_pair.ec2_instance_key_pair.key_name
  vpc_security_group_ids = [aws_security_group.allow_http_ssh_and_http_on_80.id]

  user_data = <<EOF
#!/bin/bash
echo "installing nginx"
sudo amazon-linux-extras install nginx1 -y

echo "updating nginx config for reverse proxy"
echo "
user nginx;
worker_processes auto;
include /usr/share/nginx/modules/*.conf;
events {
    worker_connections 1024;
}
http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    error_log /dev/null;
    access_log /dev/null;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    upstream express_server {
        server 127.0.0.1:8080;
        keepalive 64;
    }
    server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;
        location / {
            proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
            proxy_set_header X-Real-IP \$remote_addr;
            proxy_set_header Host \$http_host;
            proxy_set_header Upgrade \$http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_http_version 1.1;
            proxy_pass http://express_server/;
            proxy_redirect off;
            proxy_read_timeout 240s;
        }
    }
}" | sudo tee /etc/nginx/nginx.conf
## Starting Nginx Services
sudo chkconfig nginx on
sudo service nginx start
sudo service nginx restart
EOF
  tags = {
    Name = "local-dev-tunneling-server"
  }
}

This server will receive requests on port 80 and forwards them to the localhost server on port 8080.
The only thing that's left to do is to start your local app and start a remote ssh port forwarding session by running the following command:

ssh -i ~/.ssh/ec2-instances.pem -R 8080:localhost:8080 ec2-user@<public-ipv4-dns>

You can copy the instance Public IPv4 DNS from the AWS console.

Now we need to have the site secure with SSL/TLS. So we can either add a load balancer and associate it with a certificate from AWS ACM or directly create a certificate on the instance. Let's do the latter using OpenSSL.

First, enable ingress access of HTTPS on port 443 on the security group

    ingress {
      description = "https"
      from_port = 443
      to_port   = 443
      protocol  = "tcp"
      cidr_blocks = [
        "0.0.0.0/0"
      ]
    }

Update the startup script to create the certificate and use it in the nginx config. Update the user_data tag in Terraform to:

    user_data = <<EOF
  #!/bin/bash
  echo "installing nginx"
  sudo amazon-linux-extras install nginx1 -y

  echo "create a cert"
  sudo mkdir /etc/ssl/private
  sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

  echo "updating nginx config for reverse proxy"
  echo "
  user nginx;
  worker_processes auto;
  include /usr/share/nginx/modules/*.conf;
  events {
      worker_connections 1024;
  }
  http {
      sendfile on;
      tcp_nopush on;
      tcp_nodelay on;
      keepalive_timeout 65;
      types_hash_max_size 2048;
      error_log /dev/null;
      access_log /dev/null;
      include /etc/nginx/mime.types;
      default_type application/octet-stream;
      upstream express_server {
          server 127.0.0.1:8080;
          keepalive 64;
      }
      server {
          listen 80 default_server;
          listen [::]:80 default_server;
          listen 443 ssl;
          ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
          ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
          server_name _;
          location / {
              proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
              proxy_set_header X-Real-IP \$remote_addr;
              proxy_set_header Host \$http_host;
              proxy_set_header Upgrade \$http_upgrade;
              proxy_set_header Connection "upgrade";
              proxy_http_version 1.1;
              proxy_pass http://express_server/;
              proxy_redirect off;
              proxy_read_timeout 240s;
          }
      }
  }" | sudo tee /etc/nginx/nginx.conf
  ## Starting Nginx Services
  sudo chkconfig nginx on
  sudo service nginx start
  sudo service nginx restart
  EOF

Now navigate to the Public IPv4 DNS provided by EC2 in your browser, which should direct the user to your local app on HTTPS.
How could you further improve the experience? I'd like to hear your thoughts.

Thanks for reading this far. Did you like this article, and do you think others might find it useful? Feel free to share it on Twitter or LinkedIn.

Automate infrastructure for manually created resources in AWS

Ali Haydar — Sun, 18 Dec 2022 20:09:33 +0000

Unless you've been working on greenfield projects all of the time in the past few years, you have likely encountered scenarios where AWS resources are provisioned manually. So, you might have a few EC2 instances, Lambda functions, and databases created manually from the Web console.

You just encountered a case where you want to modify or extend that infrastructure, and you might be thinking about whether to update it manually from the console. Still, you get that feeling of unease and frustration to update these resources, especially if they are in a production account.
You might opt to do this manually due to time (or other) constraints, and you ping your colleague and pair on doing the update because it is always good to have another pair of eyes when touching production resources. That's fine. However, if you prefer to turn your infrastructure to code with Terraform, here's how you do it.

In this article, we will build a Lambda manually from the AWS console and import it to Terraform.

Build your Lambda Manually

We will create a Lambda function from the AWS Console in this step.

Write your Terraform Config

We want to change the Lambda function but don't want to do it from the console. So, we will create our Terraform configuration.

I am using Terraform Cloud, so I will create a workspace and call it terraform-import. However, you can run the same config from your local machine.

Create a folder to contain your infrastructure code
Create a new file called main.tf to specify the AWS provider

  terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }

  required_version = ">= 1.1.0"
  cloud {
    organization = "MyAwesomeOrganisation"

    workspaces {
      name = "terraform-import"
    }
  }
  }
  provider "aws" {
  region = "us-east-1"
  }

Run terraform login if you're using Terraform cloud. Otherwise, skip this step
Run terraform init to initialise the workspace
Run terraform plan - you will get the following output: No changes. Your infrastructure matches the configuration.

Now we need to create the configuration using Terraform import.

Create a lambda.tf file


  resource "aws_lambda_function" "my_awesome_lambda" {
  function_name = "MyAwesomeFunction"
  }

Run terraform import aws_lambda_function.my_awesome_lambda MyAwesomeFunction

This command will generate a state file that would map the resource in AWS in the terraform state. If you're using Terraform Cloud, Navigate to the states file in the left panel under your workspace. If you're working on your local, notice a new file generated, terraform.tfstate.

Have a look at the state and see what's different.

Based on the state file, we need to update the lambda config so that we get No Changes when we run terraform apply the next time. We should not have a resource deletion or replacement. That's the goal.

When I ran terraform plan first, I got an error saying that the role argument was required. So I updated my lambda.tf file to become as follows:

resource "aws_lambda_function" "my_awesome_lambda" {
  function_name = "MyAwesomeFunction"
  role    = "arn:aws:iam::24886324234235555:role/service-role/MyAwesomeFunction-role-r09h0n7l"
}

I got the role from the state file, the Lambda execution role.

Run terraform plan again. The following error I got is: Error: handler and runtime must be set when PackageType is Zip. So I updated my Lambda as follows:

resource "aws_lambda_function" "my_awesome_lambda" {
  function_name = "MyAwesomeFunction"

  handler = "index.handler"
  runtime = "nodejs16.x"
  role    = "arn:aws:iam::248869629908:role/service-role/MyAwesomeFunction-role-r09h0n7l"

}

Run terraform plan again. That's the result I got:

  # aws_lambda_function.my_awesome_lambda will be updated in-place
  ~ resource "aws_lambda_function" "my_awesome_lambda" {
        id                             = "MyAwesomeFunction"
      + publish                        = false
      ~ runtime                        = "nodejs18.x" -> "nodejs16.x"
        tags                           = {}
        # (18 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

That means it will change my Lambda from NodeJs 18 to NodeJs 16. I don't want that, so I will update my lambda.tf to use Node 18 runtime.

Run terraform plan again. You should get the following message: No changes. Your infrastructure matches the configuration.

Now you have your infrastructure documented, so you can update it, get it reviewed, and add it to the version control with peace of mind.

We can improve this. So, for example, instead of typing the arn of the Lambda execution role, we can also automate this with Terraform. Try this out.

Update the lambda function

Create a new folder, src and an index.mjs file. Add the following code:

export const handler = async (event) => {
  console.log("This lambda was updated using Terraform");

  const response = {
    statusCode: 200,
    body: JSON.stringify("Hello from Lambda!"),
  };
  return response;
};

Package your Lambda by running the following:

cd ../src

zip -r awesome_lambda.zip .

cp lambda_function.zip ../infrastructure/

This will package the Lambda into a zip file and places it under the infrastructure directory.

Add the following line to the lambda resource:

  filename         = "${path.module}/awesome_lambda.zip"

Run terraform plan. You will get the following changes:

Terraform will perform the following output:

  # aws_lambda_function.my_awesome_lambda will be updated in-place
  ~ resource "aws_lambda_function" "my_awesome_lambda" {
      + filename                       = "./awesome_lambda.zip"
        id                             = "MyAwesomeFunction"
      ~ last_modified                  = "2022-12-09T18:30:44.000+0000" -> (known after apply)
      + publish                        = false
        tags                           = {}

Run terraform apply. Open the Lambda in your AWS console, and notice how the code has changed.

Run terraform destroy to destroy your infrastructure.

Do you have a better way to automate your legacy infrastructure? I'd like to hear your thoughts.

Thanks for reading this far. Did you like this article, and do you think others might find it useful? Feel free to share it on Twitter or LinkedIn.

Optimise your AWS Lambda performance with NodeJS top-level await

Ali Haydar — Thu, 01 Dec 2022 18:30:10 +0000

A few months ago, I wrote an article about how to speed up your Lambda function, which explores cold starts a bit and talks about the init code phase and writing code outside the Lambda handler function.

I got a few comments asking how to call an async function outside the Lambda handler. There are a few use cases where this could be useful. For example, when retrieving a password from AWS Secrets Manager, loading a file from S3, or opening a connection to RDS database, it might not be required to do this operation upon every function invocation. So it would be great to run this operation during the function init.

const {
  SecretsManagerClient,
  GetSecretValueCommand,
} = require("@aws-sdk/client-secrets-manager");

const client = new SecretsManagerClient();
const input = {
  SecretId: "<SECRET_MANAGER_RESOURCE_ID>",
};

const command = new GetSecretValueCommand(input);
const response = await client.send(command); // This will throw an error

exports.handler = async (event) => {
    // TODO use the returned response
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

This will throw the following error:

  "errorType": "Runtime.UserCodeSyntaxError",
  "errorMessage": "SyntaxError: await is only valid in async function",

The only way we used to have to keep an optimal performance was to move const response = await client.send(command); into the Lambda handler. This implementation is Ok but not the best, because it will be executed upon every Lambda invocation, and it doesn't help with the readability of the code.

However, NodeJS introduced Top-Level await as an experimental feature in version 14, then it became generally available in version 16.

That's great. How can we use it? We need to change our code to use ES modules. You can do this by either changing the extension of your JS file to have the .mjs file extension or simply adding the "type": "module" to your package.json file.

The code would look as follows:

import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient();
const input = {
  SecretId: "<SECRET_MANAGER_RESOURCE_ID>",
};

const command = new GetSecretValueCommand(input);
const response = await client.send(command); // This will work well

export const handler = async (event) => {
    // TODO use the returned response
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

Do you know of any other valuable tips with Lambda and NodeJS? I'd love to hear from you.

Thanks for reading this far. Did you like this article, and do you think others might find it useful? Feel free to share it on Twitter or LinkedIn.

How to store (or Not) your DB password in AWS

Ali Haydar — Tue, 22 Nov 2022 16:24:04 +0000

If you've worked with databases, you likely have given some thought to where to store your DB credentials. Should they live on the server along with your application? Should they be in environment variables? Should they be in a vault, and you query them with every request?

In this article, we will create a simple application that uses RDS MySQL as its storage and use AWS Lambda to run our code. The app will retrieve a list of users from the database. We will use a single table and a single Lambda function.

We will go through multiple iterations covering credentials management in different ways, starting from incorrect practices towards correctly addressing secrets in a production system.

As part of this process, let's build our infrastructure using Terraform because it's fun.

The resources provisioned throughout this post might incur some cost

Please do not use this code without understanding what it does. The database is public in the snippets below when it is essential to have it secured in a VPC in a real-life scenario.

Build the app with credentials as env variables or in plain text

In this first iteration, we will generate credentials to access the DB and use them to connect from Lambda. Of course it's not a good idea to have the credentials stored in open text, but we'll start with it and evolve it to be secure throughout the article.

Create an "infrastructure" folder in your project where we will place the Infrastructure code.

Create the database

First, let us create a MySQL database; in a rds.tf file:

module "db" {
  source = "terraform-aws-modules/rds/aws"

  identifier = "demodb"

  engine            = "mysql"
  engine_version    = "8.0.30"
  instance_class    = "db.t3.micro"
  allocated_storage = 5

  db_name  = "demodb"
  username = "user"
  port     = "3306"

  family = "mysql8.0"

  # DB option group
  major_engine_version = "8.0"

  publicly_accessible = true

  skip_final_snapshot = true
}

output "db_instance_password" {
  sensitive = true
  value     = module.DB.db_instance_password
}

I am using pre-built modules of Terraform for the database resources rather than building them from scratch (https://registry.terraform.io/modules/terraform-aws-modules/rds/aws/latest).

Create your infrastructure by running the following command: terraform apply -var-file=variables.tfvars. When we create the database using this module, it will generate a random password as an output (that's the password of the master database user). The output bloc will store the generated password in the Terraform state - By running terraform output -json, you get a JSON object with that password:

{
  "db_instance_password": {
    "sensitive": true,
    "type": "string",
    "value": "pgO5Wle0vXrJX2CL"
  }
}

That's NOT a good way to display and store a password, as it's visible to anyone who might have access to your terraform state (in our case, it's a local file that got generated when we apply the infrastructure - terraform.tfstate).

You might have noticed that I have publicly_accessible=true in my configuration. Don't do this in production. That's my lazy workaround to access the DB from my local machine to add data and to avoid setting up fully fleshed-out network & security patterns. It's also better as you create your table through code. For this example, we'll connect to the instance manually and run the following scripts:

CREATE TABLE users (
  id INTEGER,
  first_name VARCHAR(40),
  last_name VARCHAR(40),
  user_type ENUM ('admin', 'finance', 'dev')
);

INSERT INTO users (id, first_name, last_name, user_type)
VALUES
(1, 'Pam', 'Beesly', 'finance'),
(1, 'Dwight', 'Schrute', 'admin'),
(1, 'Michael', 'Scott', 'finance');

Create a Lambda function

Create a new file under "infrastructure" - call it lambda.tf and update it with the following config:

data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "lambda_execution_role" {
  name               = "lambda-get-users-exec-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}

resource "aws_lambda_function" "get_users_lambda" {
  function_name = "get-users-lambda"

  handler = "index.handler"
  runtime = "nodejs16.x"

  filename = "lambda_function.zip"

  source_code_hash = filebase64sha256("lambda_function.zip")

  role = aws_iam_role.lambda_execution_role.arn

  environment {
    variables = {
      "RDS_HOSTNAME" = "demodb.cvwlm7q0gjdn.ap-southeast-2.rds.amazonaws.com"
      "RDS_USERNAME" = "user"
      "RDS_PASSWORD" = "pgO5Wle0vXrJX2CL"
      "RDS_PORT"     = "3306"
      "RDS_DATABASE" = "demodb"
    }
  }
}

Did you notice that we included the credentials as environment variables? That is NOT a good practice, as we now have the password in the Terraform state, the Terraform code, and the Lambda configuration (anyone with access to these systems can see the credentials). We will change this in future iterations.

Create a new src folder at the same level of infrastructure.

Run npm init inside the folder
Run npm i mysql to install the package that enables us to connect and query the database
Add an index.js file with the following content:

const MySQL = require("mysql");
const con = MySQL.createConnection({
  host: process.env.RDS_HOSTNAME,
  user: process.env.RDS_USERNAME,
  password: process.env.RDS_PASSWORD,
  port: process.env.RDS_PORT,
  database: process.env.RDS_DATABASE,
});

exports.handler = async (event) => {
  const SQL = "SELECT * FROM users";
  return new Promise((resolve, reject) => {
    con.query(SQL, function (err, result) {
      if (err) throw err;
      console.log("========Executing Query=======");
      const response = {
        statusCode: 200,
        body: JSON.stringify(result),
      };
      resolve(response);
    });
  });
};

After running terraform apply again, you should see your new Lambda function provisioned from your AWS Console (check the configuration).

Open the created lambda function from the AWS console and click the "Test" button. Notice the duration taken to execute the function:

  Duration: 237.95 ms Billed Duration: 238 ms Memory Size: 128 MB Max Memory Used: 70 MB  Init Duration: 253.66 ms

We will use these values to benchmark against other solutions to better understand the tradeoffs.

Build the app with credentials stored in an SSM Parameter Store

It doesn't require much thought to identify the risk involved in keeping secrets as plain text, even if your code is private and your AWS account is not accessible to everyone. It's easy to make a mistake or keep a door open that would be an open invitation to hackers.

Alright, we generated the password during the infrastructure provisioning, and we retrieved this password by running terraform output -json. Why don't we store it in SSM Parameter Store and retrieve it as needed - this way, we don't need to put it in code, and no one will see it as an environment variable.

We have a few changes to make:

Create a new SSM file under the "infrastructure folder:

  resource "aws_ssm_parameter" "rds_password" {
    name  = "RDS_PASSWORD"
    type  = "SecureString"
    value = "placeholder"
  }

Run terraform apply

This will create a secure parameter in the SSM Parameter store. In real life, it might be a good idea to use all of the environment variables in the Parameter Store - You could place them under a path and query all of them with a single request from within your Lambda. For now, I will only store the password.

In AWS Console, navigate to Systems Manager -> Parameter Store, and fill in your password instead of the "placeholder" value we left in ssm.tf.

Let's remove this value from the Lambda environment variables in lambda.tf. Then let's update our Lambda code to retrieve the value from SSM.

Run npm install @aws-sdk/client-ssm
Update the index.js file as follows:

  import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";
  import mysql from "mysql";

  const ssmClient = new SSMClient();

  const input = {
    Name: "RDS_PASSWORD",
    WithDecryption: true,
  };
  const command = new GetParameterCommand(input);
  const rdsPassword = await ssmClient.send(command);

  const con = mysql.createConnection({
    host: process.env.RDS_HOSTNAME,
    user: process.env.RDS_USERNAME,
    password: rdsPassword.Parameter.Value,
    port: process.env.RDS_PORT,
    database: process.env.RDS_DATABASE,
  });

  export const handler = async (event) => {
    const sql = "SELECT * FROM users";
    return new Promise((resolve, reject) => {
      con.query(sql, function (err, result) {
        if (err) throw err;
        console.log("========Executing Query=======");
        const response = {
          statusCode: 200,
          body: JSON.stringify(result),
        };
        resolve(response);
      });
    });
  };

The duration taken to execute the function is:

Duration: 104.29 ms Billed Duration: 105 ms Memory Size: 128 MB Max Memory Used: 93 MB Init Duration: 779.47 ms

Notice how the "init" duration is now 779.47 ms compared to when we had the password in the environment variable (253.66 ms). That's because we have more code running in the function init to connect to SSM and get the password.

That's a bit slower, but it's way more secure than the first instance.

Build the app with credentials stored in AWS Secrets Manager

AWS Secrets Manager shares a few functionalities with AWS Parameter Store. The difference is that Secrets Manager supports secrets rotation and integrates with RDS - That's a very significant security improvement to what we had earlier.

First, let us create the Secrets Manager resource. Create a new file secrets_manager.tf under the "infrastructure" folder:


resource "aws_secretsmanager_secret" "rds_secret" {
name = "secret-manager-rds"
description = "secret for RDS"
}

resource "aws_secretsmanager_secret_version" "rds_credentials" {
secret_id = aws_secretsmanager_secret.rds_secret.id
secret_string = "placeholder"
}

I avoid manually opening the AWS console and pasting the secrets in real life. It's better to have the secret automatically generated and passed in Terraform without any manual intervention. Here's an example: https://stackoverflow.com/a/68200795/1263668 - for now, we'll use the console for convenience.

After running terraform apply, you can see the newly created secret in your AWS console -> Secrets Manager.

Click "Retrieve secret value"
Click the "Edit" button and enter the value of your password instead of the placeholder

Next, let us delete the SSM Parameter Store parameters we created (Delete ssm.tf) and update the Lambda to use Secrets Manager instead.

Update the lambda execution role to permit it to interact with Secrets Manager. In the iam.tf file:


  resource "aws_iam_policy" "lambda_policies" {
  name = "lambda_secrets_manager_access"
  description = "lambda access to secrets manager"
  policy = data.aws_iam_policy_document.lambda_policies_document.json
  }

  data "aws_iam_policy_document" "lambda_policies_document" {
  statement {
  actions = [
  "secretsmanager:GetSecretValue"
  ]
  resources = [aws_secretsmanager_secret.rds_secret.arn]
  }
  }

In the lambda.tf file, add the following environment variable:


  "SECRET_MANAGER_RESOURCE_ID" : aws_secretsmanager_secret.rds_secret.id

Update the lambda code in index.js- Update the code before the handler function to the following:


  import {
  SecretsManagerClient,
  GetSecretValueCommand,
  } from "@aws-sdk/client-secrets-manager";
  import MySQL from "mysql";

  const client = new SecretsManagerClient();

  const input = {
  SecretId: process.env.SECRET_MANAGER_RESOURCE_ID,
  };

  const command = new GetSecretValueCommand(input);
  const rdsPassword = await client.send(command);

  const con = MySQL.createConnection({
  host: process.env.RDS_HOSTNAME,
  user: process.env.RDS_USERNAME,
  password: rdsPassword.SecretString,
  port: process.env.RDS_PORT,
  database: process.env.RDS_DATABASE,
  });

Test the Lambda function.

The duration taken to execute the function is (the init duration is slightly faster than with SSM parameter store but still slower than having the secret passed as an env variable):


Duration: 109.62 ms Billed Duration: 110 ms Memory Size: 128 MB Max Memory Used: 86 MB Init Duration: 597.35 ms

Rotate the secrets in AWS Secrets Manager

So far, there isn't much difference between "SSM Parameter Store" and "Secrets Manager". However, AWS Secrets Manager offers a feature to rotate secrets. How awesome is that! - AWS Secrets Manager uses a Lambda function to rotate the secret and sync it to your database in RDS.

Unfortunately, it's more complicated than selecting a checkbox to enable the secret rotation. There is some configuration to change, including creating the rotator lambda.

Create a new Lambda function that will handle the rotation - here is a template https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/master/SecretsManagerRDSMySQLRotationMultiUser/lambda_function.py - I created it in a separate folder within the same repository. If you are like me, new to using Python, you could use this helpful guide:https://docs.aws.amazon.com/lambda/latest/dg/python-package.html.
Add the following to the lambda.tf file:


  resource "aws_lambda_function" "secret_rotator" {
  function_name = "secret-rotator-lambda"

      handler = "rotate.lambda_handler"
      runtime = "nodejs16.x"

      filename = "lambda_rotator_function.zip"

      source_code_hash = filebase64sha256("lambda_rotator_function.zip")

      role = aws_iam_role.lambda_rotator_execution_role.arn

      depends_on = [
        aws_iam_role_policy_attachment.lambda_rotator_policies_attachment
      ]

  }

In the iam.tf file add the following - we need to give the rotator lambda access to rotate the secret and permit the Secrets Manager to invoke the Lambda


  data "aws_iam_policy_document" "secrets_access_policy_document" {
  statement {

        actions = [
          "secretsmanager:DescribeSecret",
          "secretsmanager:GetSecretValue",
          "secretsmanager:PutSecretValue",
          "secretsmanager:UpdateSecretVersionStage",
        ]
        resources = [aws_secretsmanager_secret.rds_secret.arn]

  }
  statement {

        actions = [
          "secretsmanager:GetRandomPassword"
        ]
        resources = ["*"]

  }
  statement {
  actions = [
  "logs:CreateLogGroup",
  "logs:CreateLogStream",
  "logs:PutLogEvents"
  ]
  resources = [
  "arn:aws:logs:*:*:*",
  ]
  }
  }
  resource "aws_iam_policy" "secrets_access_policy" {
  name = "secrets-access-policy"
  policy = data.aws_iam_policy_document.secrets_access_policy_document.json
  }

  resource "aws_iam_role_policy_attachment" "lambda_rotator_policies_attachment" {
  role = aws_iam_role.lambda_rotator_execution_role.name
  policy_arn = aws_iam_policy.secrets_access_policy.arn
  }

  resource "aws_lambda_permission" "allow_secrets_manager" {
  statement_id = "AllowExecutionFromSecretsManager"
  action = "lambda:InvokeFunction"
  function_name = aws_lambda_function.secret_rotator.function_name
  principal = "secretsmanager.amazonaws.com"
  source_arn = aws_secretsmanager_secret.rds_secret.arn
  }

Update the "Get Users" lambda to read all the info from the secrets manager rather than the env variables. In the index.js file, change the code just before the handler function to:


  const response = await client.send(command);

  const secret = JSON.parse(response.SecretString);

  const con = mysql.createConnection({
  host: secret.host,
  user: secret.username,
  password: secret.password,
  port: secret.port,
  database: secret.dbname,
  });

Remove the above variables from the lambda.tf file
Apply your changes (repackage your Lambda, then run terraform apply)
Navigate to the AWS Console, and notice how the Rotation Configuration section is updated:
Upon the first deployment, the secret should rotate, and you can see this in the CloudWatch logs - You can also see the new secret in Secrets Manager

That's way more secure as we can rotate the secret often; still, we have a master password that we use, and we need to keep it away from casual access.

Connect to the database without Secrets

Another way to connect to the RDS database is to use IAM user or role credentials and an authentication token instead of a username/password.

Before getting into the implementation, the max number of connections per second cannot exceed 200 authentication connections imposed by IAM. Look at the limitations and recommendations in the AWS Docs.

Now let's get to the implementation:

First, we need to activate IAM Database Authentication. We will do this in Terraform. in the rds.tf file, add the following attribute: iam_database_authentication_enabled = true.

Second, we will create a database that uses AWS Auth token: CREATE USER lambda_user IDENTIFIED WITH AWSAuthenticationPlugin as 'RDS';

Next, we need to create an IAM policy that allows connecting to the DB as the created user and attach it to the execution role of the Lambda:


data "aws_iam_policy_document" "lambda_rds_connect_policy_document" {
  statement {
    actions = [
      "rds-db:connect"
    ]
    resources = ["arn:aws:rds-db:${var.region}:${data.aws_caller_identity.current.account_id}:dbuser:${module.db.db_instance_resource_id}/lambda_user"]
  }
}

resource "aws_iam_policy" "lambda_rds_connect_policy" {
  name        = "lambda_connect_to_rds"
  description = "lambda access to connect to rds"
  policy      = data.aws_iam_policy_document.lambda_rds_connect_policy_document.json
}

resource "aws_iam_role_policy_attachment" "lambda_policies_attachment_db_connect_policy" {
  role       = aws_iam_role.lambda_execution_role.name
  policy_arn = aws_iam_policy.lambda_rds_connect_policy.arn
}

Note here that we are using db_instance_resource_id here - I mistakenly used db_instance_id first, which cost me a couple of hours of investigation.

The next step would be to modify the Lambda code. Update the index.js file before the handler function, as follows:

import {
  SecretsManagerClient,
  GetSecretValueCommand,
} from "@aws-sdk/client-secrets-manager";
import mysql from "mysql2";
import { Signer } from "@aws-sdk/rds-signer";

const client = new SecretsManagerClient();

const input = {
  SecretId: process.env.SECRET_MANAGER_RESOURCE_ID,
};

const command = new GetSecretValueCommand(input);
const response = await client.send(command);

const secret = JSON.parse(response.SecretString);

const signer = new Signer({
  hostname: secret.host,
  port: secret.port,
  username: "lambda_user",
});

const token = await signer.getAuthToken();

const con = mysql.createConnection({
  host: secret.host,
  user: "lambda_user",
  password: token,
  port: secret.port,
  database: secret.dbname,
  ssl: "Amazon RDS",
});

Note that we changed the MySQL client we are using in our Lambda from mysql to mysql2, as mysql does not support this authentication method. So make sure to run npm i mysql2.
Using the mysql package would throw this error: "ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client".

I hope this article was helpful, and I would love to hear your thoughts. In summary:

Don't use env variables or config to store your credentials
Using Parameter Store is a good first option if you would like to secure your credentials
The best option is using Secrets Manager with password rotation, in my opinion, especially if you're using RDS
IAM Authentication is the most secure, in my opinion, but has a few limitations. So good to consider them before using it in a production environment.

Thanks for reading this far. Did you like this article, and do you think others might find it useful? Feel free to share it on Twitter or LinkedIn.

How to query a large file in S3

Ali Haydar — Mon, 22 Aug 2022 16:48:00 +0000

S3 (Simple Storage Service) is a top-rated service for data storage in AWS. It offers high durability, availability and scalability. I've used S3 for various use cases, including storing many files, backups of databases, analytics, data archiving and static site hosting.

A few months ago, I encountered a case where we needed to query data from a large JSON file in S3. We went with the usual approach of getting the object in our Lambda, filtering the JSON object for the data we need in code. So we loaded the file content in memory and filtered it. As this file grew significantly, the Lambda started timing out.

We often need to retrieve the whole file from S3, but if it is large (e.g. 2TB), it will slow down your app. If you are transferring this file Out from Amazon S3 directly to the internet (e.g. if you host your app outside of AWS or you're downloading this file), that's a significant cost increase.

What can we do?

AWS provides a feature in S3 called "S3 Select", where the user can query parts of an object using an SQL-like statement. This is good because the filtering happens in S3 before it reaches your application.
S3 Select works on multiple file types, including CSV, JSON and Parquet.

Please show us the code...

Before getting into the code, the below diagrams could help us visualize how this works.
Get Object

S3 Select

For the rest of this post, we will use Terraform to set up an S3 bucket and a lambda. We will first retrieve an entire object in our Lambda, then change it to query parts of the object using "S3 Select".

If you're impatient, feel free to skip to the "S3 Select" section.

Terraform

I will split the Terraform configuration into multiple files - Feel free to adjust this as you deem appropriate - in an "infrastructure" folder:

Create a provider.tf to include AWS as a provider



  provider "aws" {
    profile = "default"
    region  = "ap-southeast-2"
  }

Create a test.json file and add it to an S3 bucket

Assume we have a JSON file called 'test.json' and has the following content (copied from MDN) - we will keep the size of the file small for this example:



{
  "squadName": "Super hero squad",
  "homeTown": "Metro City",
  "formed": 2016,
  "secretBase": "Super tower",
  "active": true,
  "members": [
    {
      "name": "Molecule Man",
      "age": 29,
      "secretIdentity": "Dan Jukes",
      "powers": [
        "Radiation resistance",
        "Turning tiny",
        "Radiation blast"
      ]
    },
    {
      "name": "Madame Uppercut",
      "age": 39,
      "secretIdentity": "Jane Wilson",
      "powers": [
        "Million tonne punch",
        "Damage resistance",
        "Superhuman reflexes"
      ]
    },
    {
      "name": "Eternal Flame",
      "age": 1000000,
      "secretIdentity": "Unknown",
      "powers": [
        "Immortality",
        "Heat Immunity",
        "Inferno",
        "Teleportation",
        "Interdimensional travel"
      ]
    }
  ]
}


  - Add the following content to the `s3.tf` file

    ```


      resource "aws_s3_bucket" "s3-select" {
      bucket = var.bucketName
      }

      resource "aws_s3_bucket_policy" "outoffocus-policy" {
      bucket = aws_s3_bucket.s3-select.id
      policy = templatefile("s3-policy.json", { bucket = var.bucketName })
      }

      resource "aws_s3_object" "outoffocus-index" {
      bucket = aws_s3_bucket.s3-select.id
      key    = "test.json"
      source = "./test.json"
      acl    = "public-read"
      }

Create the IAM roles and permissions needed to get the Lambda to interact with the S3 bucket. In a iam.tf file:



  resource "aws_iam_role" "s3_select_lambda_execution_role" {
  name               = "s3_select_lambda_execution_role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
  }

  data "aws_iam_policy_document" "lambda_assume_role" {
    statement {
      actions = ["sts:AssumeRole"]
      principals {
        type        = "Service"
        identifiers = ["lambda.amazonaws.com"]
      }
    }
  }

  resource "aws_iam_role_policy_attachment" "lambda_policies_attachment" {
    role       = aws_iam_role.s3_select_lambda_execution_role.name
    policy_arn = aws_iam_policy.lambda_policies.arn
  }

  resource "aws_iam_policy" "lambda_policies" {
    name        = "s3-select-lambda_logging_cloudwatch_access"
    description = "lambda logs in CloudWatch"
    policy      = data.aws_iam_policy_document.lambda_policies_document.json
  }

  data "aws_iam_policy_document" "lambda_policies_document" {
    statement {
      actions = [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
      resources = [
        "arn:aws:logs:*:*:*",
      ]
    }
    statement {
      actions = [
        "s3:GetObject"
      ]
      resources = [
        "arn:aws:s3:::${var.bucketName}/*"
      ]
    }
  }

Create a lambda.tf file that would include the following configuration:



  resource "aws_lambda_function" "s3-select" {
  function_name = "s3-select-lambda"

  handler = "index.handler"
  runtime = "nodejs16.x"

  filename = "lambda_function.zip"

  source_code_hash = filebase64sha256("lambda_function.zip")

  role = aws_iam_role.s3_select_lambda_execution_role.arn

  depends_on = [
    aws_iam_role_policy_attachment.lambda_policies_attachment
  ]
  }

Finally, create a variables.tf file to include the bucketName variable used earlier ```

variable "bucketName" {
default = "your-unique-s3-bucket-name"
type = string
}


### Lambda Code

Create a new npm project by running `npm init` and following the steps in your terminal. Create a "src" folder with an index.js file in it:

const { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");
const consumers = require("stream/consumers");

const params = {
Bucket: "your-unique-s3-bucket-name",
Key: "test.json",
};

exports.handler = async (event) => {
const client = new S3Client({});
const command = new GetObjectCommand(params);
const response = await client.send(command);

const objectText = await consumers.text(response.Body);

console.log("here is the returned object", objectText);
return {
statusCode: 200,
body: JSON.stringify("Hello from Lambda!"),
};
};


Of course, this is not optimal (we could use TS and pass the params as env variables, etc.), but this should be sufficient for this blog post.

To package your Lambda into a zip file and copy it to the infrastructure folder, run the following commands:

zip -r lambda_function.zip ./.js ./.json ./node_modules/*

cp lambda_function.zip ../infrastructure/


Once done, deploy your code to AWS by running `terraform init` then `terraform apply` in your "infrastructure" folder.

### S3 Select

There's nothing you would need to do to enable S3 Select. It's an API call where we pass the SQL-Like statement to S3, and everything happens auto-magically from there.

Let's start with writing the SQL statement. Consider we want to return the "Molecule Man" age in the "members" array. The query looks as follows:

SELECT s.age FROM s3object[].members[] s WHERE s.name = 'Molecule Man'


The query isn't precisely SQL, but the syntax is very similar. For more details, check this [page](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference-select.html). I also find it helpful to try the query in the AWS console. To do that:

- Select the object in the S3 bucket
- Click Action -> S3 Select
- Write the query

![AWS Console - S3 Select](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cs9mwnzhgwz8bfyyfax3.png)

The new Lambda function code looks as follows:

const { S3Client, SelectObjectContentCommand } = require("@aws-sdk/client-s3");
const consumers = require("stream/consumers");

const params = {
Bucket: "your-unique-s3-bucket-name",
Key: "test.json",
ExpressionType: "SQL",
Expression:
"SELECT s.age FROM s3object[].members[] s WHERE s.name = 'Molecule Man'",
InputSerialization: {
JSON: {
Type: "DOCUMENT",
},
},
OutputSerialization: {
JSON: {},
},
};

exports.handler = async (event) => {
const client = new S3Client({});
const command = new SelectObjectContentCommand(params);
const response = await client.send(command);

const objectText = await consumers.text(response.Body);

console.log("here is the returned object", objectText);
return {
statusCode: 200,
body: JSON.stringify("Hello from Lambda!"),
};
};


The "InputSerialization" property defines the data format in our query object. The JSON Type could be a "DOCUMENT|LINE" to specify whether the whole file is a JSON object or each line is a JSON object.

The "OutputSerialization" property specifies how the output object is JSON format. We could define a "RecordDelimiter" property inside it, which would be used to separate the output records. Keeping it empty would default to using the newline character ("\n").

For more info, have a look at https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#selectObjectContent-property.

I hope this was helpful. Do you have a different method to query big objects in S3? I'd love to hear from you.

---
Thanks for reading this far. Did you like this article and you think others might find it useful? Feel free to share it on [Twitter](twitter.com/intent/tweet?url=https%3A%2F%2Fdev.to%2Faws-builders%2Fhow-to-query-a-large-file-in-s3-4doa%0A%0A%23aws%20%23awscommunity%20%23awscommunitybuilder%20%23softwaredeveloment&text=Check%20this%20article%20by%20%40Alee_Haydar%20on%20how%20to%20query%20large%20files%20in%20S3%3A%20%F0%9F%91%89) or [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https://dev.to/aws-builders/how-to-query-a-large-file-in-s3-4doa).

DEV Community: Ali Haydar

Exploring the Inner Workings of SST Live Lambda

Requests are proxied to the local machine

Lambda to S3: Better Reliability in High-Volume Scenarios

Build the project

Deliberate Error Creation

Build a more reliable system

Conclusion

Exploring Local Development with SST for Serverless Applications

Building an app with SST

How is this different from running your Lambda using a NodeJS worker?

Avoiding Data Overwrites in DynamoDB

Is PutItem a risky operation?

Example where data is overwritten

example with conditional expression

How to enhance your Lambda function performance with memory configuration?

Testing the performance of a Lambda function

Lambda with 128 MB memory

Lambda with 256 MB memory

Lambda with 512 MB memory

Lambda with 1024 MB memory

Shall we continue to increase the memory?

How to choose a memory config?

Effortlessly Juggling Multiple Concurrent Requests in AWS Lambda

Set up the project

Test the concurrent Lambda execution before and after setting up the provisioned concurrency

Before Provisioned Concurrency

After Provisioned Concurrency

Prevent API overload with rate limiting in AWS

Build the API

Rate & Burst Limits

How to securely expose your local app to the internet using EC2?

How to expose your app to the internet?

Create an EC2 instance

Automate infrastructure for manually created resources in AWS

Build your Lambda Manually

Write your Terraform Config

Update the lambda function

Optimise your AWS Lambda performance with NodeJS top-level await

How to store (or Not) your DB password in AWS

Build the app with credentials as env variables or in plain text

Create the database

Create a Lambda function

Build the app with credentials stored in an SSM Parameter Store

Build the app with credentials stored in AWS Secrets Manager

Rotate the secrets in AWS Secrets Manager

Connect to the database without Secrets

How to query a large file in S3

What can we do?

Terraform

Is `PutItem` a risky operation?