Creating AWS Lambda Layers for Python Runtime: A Complete Guide

Managing dependencies for AWS Lambda functions can be challenging, especially when using Python packages like NumPy, pandas, or requests that contain compiled binaries. This guide provides a comprehensive approach to creating AWS Lambda layers, ensuring compatibility with Lambda’s Linux-based runtime, even when developing on non-Linux systems.

What Are AWS Lambda Layers?

AWS Lambda layers are .zip archives that let you share libraries, custom runtimes, or other dependencies across functions. They simplify updates, promote modularity, and reduce deployment sizes.

Common Challenges

Python packages with compiled code may not work out-of-the-box on AWS Lambda, especially if you’re developing on Windows or macOS. Errors like “Unable to import module” often arise because pip installs packages for your local architecture rather than Lambda’s Linux environment.

To address this, you need to build your deployment package or layer in a way that is compatible with AWS Lambda’s Linux runtime.

Step-by-Step Guide to Creating AWS Lambda Layers

1. Prepare Your Environment

• Ensure the latest Python version (or the runtime version of your Lambda function) is installed.

• Check your pip version and upgrade it if it’s older than 19.3.0:
  

    py -m pip install --upgrade pip

2. Create the Directory Structure

• Create a working directory for your Lambda layer:

    mkdir -p layer/python

• Open the directory in your IDE (e.g., Visual Studio Code).

3. Install Dependencies

To ensure compatibility, use pip’s --platform parameter to target AWS Lambda’s Linux environment:

    pip install `  
        --platform manylinux2014_x86_64 `  
        --target=layer/python `  
        --implementation cp `  
        --python-version 3.12 `  
        --only-binary=:all: --upgrade `  
        pandas numpy requests

This installs the dependencies directly into the layer/python directory, ready for packaging.

Note: Ensure your installed packages don’t exceed Lambda’s 50 MB limit for direct uploads. You’ll need to upload the .zip file to an S3 bucket for larger packages.

4. Package the Layer

Navigate to the layer directory and create a .zip archive:

    cd layer
    zip -r ../python_layer.zip .

5. Deploy the Layer to AWS Lambda

• Log in to the AWS Management Console.

• Go to Lambda > Layers and click Create Layer.

• Upload the python_layer.zip file, select the compatible runtime (e.g., Python 3.12), and save the layer.

6. Attach the Layer to Your Lambda Function

• Open your Lambda function in the AWS Console.

• Under the Layers section, click Add a layer.

• Select your custom layer, specify the version, and attach it.

Test Your Lambda Function

Here’s an example Lambda function to verify the layer works:

    import pandas as pd
    import numpy as np
    import requests

    def lambda_handler(event, context):
        return {
            "statusCode": 200,
            "body": {
                "pandas_version": pd.__version__,
                "numpy_version": np.__version__,
                "response_status": requests.get("https://example.com").status_code
            }
        }

Deploy the function, add the layer, and invoke it to confirm the dependencies are working correctly.

Some important Tips :

• Cross-Platform Compatibility: This process works across Windows, macOS, and Linux.

• Avoid Empty Directories: Before zipping, ensure no unnecessary or empty directories remain in the layer folder.

• File Size Check: Always check your .zip file size before uploading to Lambda (use S3 for files larger than 50 MB).

• Local Testing: Use Docker to emulate the Lambda environment.

Conclusion

By following these steps, you can efficiently create and deploy AWS Lambda layers for Python runtimes. This approach ensures compatibility with AWS Lambda’s Linux-based environment while promoting reusability and reducing deployment sizes.

If you’re working with larger packages or need alternative methods (e.g., Docker and Amazon ECR), check out additional resources for advanced deployment techniques.

References

• Working with .zip file archives for Python Lambda functions

• Working with layers for Python Lambda functions

• Managing Lambda dependencies with layers

• Lambda Package Exceeds 60MB: Solutions for Large Dependencies?