Manage Python dependencies in Serverless projects with serverless-layers plugin

Struggling to properly package your Python dependencies into your functions? In this post I’ll take a look at serverless-layers, a Serverless Framework plugin that can do it for you!

Sample project

Given a simple project that has the following directory structure:

serverless-layers-example/
├── requirements.txt
├── serverless.yml
└── src
    ├── __init__.py
    └── functions
        ├── __init__.py
        └── example_function
            ├── __init__.py
            └── handler.py

Where serverless.yml contains the following:

service: serverless-layers-example

provider:
  name: aws
  runtime: python3.8

functions:
  example_function:
    handler: src/functions/example_function/handler.handle_event

The example_function is a simple function that calls an external API (cat-facts 🐱) using the requests library and returns the response in the body.

import json

import requests

FACTS_URL = "https://cat-fact.herokuapp.com/facts"

def get_fact():
    r = requests.get(FACTS_URL, timeout=5)

    return r.json()[0]["text"]

def handle_event(event, context):
    fact = get_fact()

    body = {
        "message": fact
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response

The requirements.txt file contains the following (generated by running pip freeze > requirements.txt after running pip install requests):

certifi==2020.12.5
chardet==4.0.0
idna==2.10
requests==2.25.1
urllib3==1.26.2

After deploying the function with sls deploy and invoking the function with sls invoke -f example_function, you’ll be greeted with the following error:

{
    "errorMessage": "Unable to import module 'src/functions/example_function/handler': No module named 'requests'",
    "errorType": "Runtime.ImportModuleError"
}

Let’s see how serverless-layers can help us solve the above.

serverless-layers

One of the first hits when searching for solutions to manage Python dependencies in Serverless projects is the popular serverless-python-requirements plugin. However serverless-layers is a plugin that can work just as well.

As the name suggests, serverless-layers utilizes Lambda layers to manage functions dependencies. It automatically attaches a layer to each function defined in your serverless.yaml containing the dependencies.

Let’s give it a try!

Install

• Create a package.json by running npm init (If not already exists)
• Install the plugin npm i -D serverless-layers
• Add the plugin to the plugins section in serverless.yml
• Because we haven’t specified the deploymentBucket in the provider section, we have to configure the layersDeploymentBucket in the plugin configuration in the custom section and make sure the bucket exists.

service: serverless-layers-example

provider:
  name: aws
  runtime: python3.8

plugins:
  - serverless-layers

custom:
  serverless-layers:
    layersDeploymentBucket: <bucketName>

functions:
  example_function:
    handler: src/functions/example_function/handler.handle_event

Redeploy the function

Invoke the function again with sls invoke -f example_function. You should now see the response as expected.

What happened

1. The plugin first checks if the layer for this project and stage already exists by checking if a requirements.txt exists at <bucketName>/serverless/<projectName>/<stage>/layers/requirements.txt.

2. If not, it downloads the dependencies as specified in requirements.txt to .serverless/layers/python/

3. This directory is zipped and uploaded to the bucket. The resulting zipfile can be found the .serverless directory and with the example config as above has the name <projectName>-<stage>-<runtime>-default.zip. You could inspect the contents of this zipfile by running zipinfo <zipfileName>.

4. A Lambda layer is created
   

5. The requirements.txt that the layer is based on is uploaded to the bucket
   

6. The layer gets attached to the function. This is visible in the cloudformation-template-update-stack.json file in the .serverless directory.
   

"ExampleUnderscorefunctionLambdaFunction": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Handler": "src/functions/example_function/handler.handle_event",
    "Runtime": "python3.8",
    "FunctionName": "serverless-layers-example-dev-example_function",
    "Layers": [
      "arn:aws:lambda:us-east-1:************:layer:serverless-layers-example-dev-python-default:1"
    ]
  },      
}

Adding/updating dependencies

When adding or updating the dependencies the same process as described above gets triggered, but only if the requirements.txt has actually changed, compared to the one stored in the bucket next to the active layer.

Example of the output when changes are detected:

Example of the output when no changes are detected:

Caveats

• At the time of writing the plugin does only perform a simple check to see if dependencies changed. Changing the order of the listed dependencies already triggers an update of the Lambda layer.

Conclusion

serverless-layers is a handy plugin that can help you with managing Python dependencies using Lambda layers. It takes away the operational overhead of managing Lambda layers and because the dependencies are no longer packaged with the Lambda function itself it drastically reduces its size and time it takes to deploy new versions of your code.

It might even improve cold-start times (check out This is all you need to know about Lambda cold starts#Where the dependency is loaded from matters), but I have not tested that.

serverless-layers currently only supports dependencies defined through requirements.txt. There is no support for other Python dependency management tools such as Pipenv/Pipfile or Poetry.