Build AWS Lambda Layers with AWS CDK

AWS Lambda layers were a game changer for me. When used properly, they can reduce deployment time dramatically and, in turn, reduce time to production in Serverless services with multiple AWS Lambda functions.

However, it’s often a feature that’s misunderstood and eventually not used.

This blog post covers AWS Lambda layers basics and how to build them with Docker & AWS CDK in Python.

A complete & working code can be found at my open source template project: https://github.com/ran-isenberg/aws-lambda-handler-cookbook

This blog was originally published on my website “Ran The Builder” https://www.ranthebuilder.cloud/

What is a Lambda Layer

AWS Lambda layers provide a convenient way to package libraries and other dependencies you can use with multiple Lambda functions.

A layer is a .zip file archive that can contain additional code or data. A layer can contain libraries, a custom runtime, data, or configuration files. Layers promote code sharing and separation of responsibilities so that you can iterate faster on writing business logic — AWS

Build a Layer with AWS CDK & Docker in Python with Poetry

Let's assume that our Serverless service project uses 'poetry' for dependency management and that the pyproject.toml file looks like this:

Our Goal

We want to bundle AWS Lambda Powertools, mypy-boto3-dynamodb other libraries under the [tools.poetry.dependencies] section into a Lambda layer to be consumed by all our lambda functions and deployed with AWS CDK.

Assumptions

1. You installed Docker.
2. You installed AWS CDK.
3. You use poetry as your Python dependency manager.
4. You use just one general pyproject.toml file for all the AWS Lambda functions. It includes all the requirements of all functions.

Prepare a .build Folder for CDK Deploy

Before we write any CDK code, we need to prepare the layer build folder.
We will store all the required build artifacts in a new folder: the '.build' folder, which will not be part of the code base. We will keep a folder for the Lambda layers artifacts and another folder that contains the service Lambda function code, ready to be packaged by AWS CDK.
Let's start with the layer.
First, we need to create a requirements.txt file from our pyproject.toml that will be used to create the layer itself. In this example, we use poetry, but pipenv is a valid option too.

We generate a requirements.txt from the [tool.poetry.dependencies] section of the toml file.
Unlike the [tool.poetry.dev-dependencies] section in the toml, the libraries in the 'tool.poetry.dependencies' section are the libraries that the Lambda functions in the project require in runtime and must be uploaded to AWS.
Run the following commands to create the .build/common_layer folder and to generate the layer requirements.txt file inside it. As mentioned above, these files are not part of the code base and we use the .build folder to store custom build & deploy artifacts.
The poetry command will export the non development dependencies to the requirements.txt file.

Which produces the following requirements.txt file in the '.build/common_layer' folder. The name 'common_layer' represents the layer purpose, the common usage Lambda layer in our service that all functions use.

Now, we prepare the Lambda function build folder. We can do this by running the following commands:

In our project, the Lambda function handlers code reside in the project's root folder under the 'service' folder. Before we deploy the Lambda functions to AWS, we copy the 'service' folder to the '.build/lambdas' folder, which will pack as a zip and upload to AWS S3 by AWS CDK.
The CDK code that creates the Lambda function will take the handler code from the '.build/lambdas' folder.
We do this to have the same import hierarchy in both AWS CDK and the Lambda function container environment after its deployment to AWS and to prevent runtime import issues such as:

This error occurs when building the function without copying it first to .build/lambdas but copying the code directly from the 'service' folder on the root level.

AWS CDK Code

Now that everything is in place let's write a CDK construct that creates a Lambda layer based on the prebuilt requirements.txt file and a Lambda function in Python 3.9 that uses the layer.
AWS CDK will spin up a Lambda layer builder container using Docker.
The container will look for a requirements.txt file, download all required libraries, and zip them.
Once the layer object is created, you can add it to any Lambda function.

---

The AWS CDK construct 'MyConstruct' creates:

1. Basic Lambda role (lines 17–24).
2. Lambda layer (lines 27–34). CDK looks for a requirements.txt file in the entry folder ('.build/common_layer') and uses Python 3.9 as runtime compatibility.
3. Lambda function (lines 36–46). The function uses the layer in line 45 that it receives as an argument. The handler code and other modules are taken from the code folder '.build/lambdas.'

---

Deploy Time!

Now we can run CDK deploy and sit back while the resources are created.
We run CDK deploy command (and point it to the app.py file in the project) combined with the previous commands that create the .build folders.

In the AWS Lambda handler cookbook Serverless template project, we use a makefile to abstract away such details. Running 'make deploy' will run these three command together.
See the makefile here.

Use External ARN Layer

If you wish to use an externally built Lambda layer you can swap the '_build_common_layer' implementation to the following:

---

In this case we build a public layer built by the amazing AWS Lambda Powertools team. Notice how we select a specific version (17) as part of the ARN.

---