How to create a Python Lambda Layer?

In this article I intend to explain what a Lambda Layer is, the benefits of it, and lastly, how to create a Layer for a Python Lambda function.

What is a Lambda Layer?

A Lambda Layer is an isolated zip file that contains libraries, packages and/or application code that is shareable between your Lambda functions. A common use case for a Lambda Layer is, for example your company has two separate Python Web Scrapers. Both functions use pandas to do some sort of data processing. Since they both use the same Python packages, the smart solution is to create a Lambda Layer, so both functions can access the Layer. Below is a visual from AWS that shows a solution with Lambda Layers vs without Lambda Layers.

Benefits of Lambda Layers

• Reduces the size of uploaded deployment archives.
• Increases the speed of deployment.

Creating a Python Lambda Layer

We will create the layer using an EC2 Instance that will be provisioned by IaC (CloudFormation). I know what you are thinking. Why do we need an EC2 Instance to simply create a Lambda Layer? Why not just do it locally? The architecture for a Python Lambda Layer requires a Linux OS. Doing it on a Linux EC2 Instance helps us meet the following requirements:

1. AWS Lambda runs on a Linux machine
2. Any Python Packages added to the Layer need to be compiled with the correct architecture (Linux x86_64).

You can find the complete CloudFormation template on my GitHub.

Create stack parameters

---
Parameters:
  InstanceName:
    Type: String
    Description: Enter the name of your instance.
    Default: LinuxInstance

The only parameter we will be using is for our instance name.

Create IAM Role & Instance Profile

# Instance Profile
  InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      InstanceProfileName: !Join ["_", [!Ref InstanceName, "Profile"]]
      Path: /
      Roles:
        - !Ref InstanceRole

  # Instance Role
  InstanceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Join ["_", [!Ref InstanceName, "EC2PublishLambdaLayer"]]
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: !Join ["_", [!Ref InstanceName, "Policy"]]
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: "lambda:PublishLayerVersion"
                Resource: "*"

NOTE: The Policy Document contains the action lambda:PublishLayerVersion, without this policy we will not be able to publish the Lambda Layer from within the EC2 Instance.

Create EC2 Instance

 # EC2 Instance
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-09d3b3274b6c5d4aa # Static AMI ID chosen from List of AMIs in the console
      InstanceType: t2.micro
      IamInstanceProfile: !Ref InstanceProfile
      SecurityGroups:
        - !Ref SSHSecurityGroup
      # Install Python, Pandas and Publish Lambda Layer
      UserData:
        Fn::Base64: |
          #!/bin/bash 
          sudo amazon-linux-extras install python3.8
          curl -O https://bootstrap.pypa.io/get-pip.py
          python3.8 get-pip.py --user
          mkdir -p home/ec2-user/python
          cd home/ec2-user
          python3.8 -m pip install pandas -t python/
          zip -r layer.zip python
          aws lambda publish-layer-version --layer-name pandas-layer --zip-file fileb://layer.zip --compatible-runtimes python3.8 --region us-east-1
      Tags:
        - Key: Name
          Value: !Ref InstanceName

If you look at the UserData property, we are passing bash scripts. This is where the magic happens. We defined scripts to install Python and pandas. We then zip the directory that has the Python Package and lastly publish the Lambda Layer to AWS.

Upload CF Stack

To upload your CF stack, use the command below.

aws cloudformation deploy 
--template-file <path_to_file> 
--stack-name <stack_name> 
--region us-east-1 
--capabilities CAPABILITY_NAMED_IAM

If the above command is successful, you will get an output in your terminal saying "Successfully created/updated stack - ". The stack should take a few minutes to provision our EC2 Instance and execute our scripts. Lastly, navigate to Lambda Layers in the AWS console and you should see the Layer that was created.

Clean Up

To clean up all the resources that were created simply delete the CloudFormation stack.

aws cloudformation delete-stack 
--stack-name <stack_name>

I hope you found this article useful. Stay curious, keep learning, and keep building. Thank you.

Comments

[Avinash Dalvi]

Good article. Small correction in title required. Lamda should Lambda.

[Gilbert Young Jr]

Appreciate it bro @avinashdalvi_ . Good looking out on the typo.

[Gilbert Young Jr]

Thank you @rishitharamesh1