Lambda with Oracle Layers

Overview

Recently, I came across a need to finally use Lambda in AWS. I was looking to query an Oracle database and export the Oracle table to a CSV file. While using Python I realized the Oracle client is not available in Lambda so I needed a way to add libraries and dependancies in order to execute my code. Lambda layers allows you to do this. You can add several layers to your Lambda function. This article documents how to create the Oracle Layer for Python in Lambda with the required libraries.

The main limitation that I discovered using Lambda layers is the size limit of 250 megs for all your code and libraries. Fortunately, I was just under this limitation but there are several ways to get around this if you run into this problem. One is to break your code into various micro services. The other is to to use EFS and place all of your libraries in the EFS volume and mount the volume on your Lambda function.

Building the Oracle Layer

Optional, but preferred, commission an EC2 instance so you can start from a clean environment.

Verify the Python version

python3 --version

Create your project directory

mkdir project

Create a directory called python

mkdir python

Make library directory

mkdir lib

Install cx_Oracle Python library

pip3 install cx_Oracle -t python/

Download the latest Oracle Client from https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html

wget https://download.oracle.com/otn_software/linux/instantclient/218000/instantclient-basic-linux.x64-21.8.0.0.0dbru.zip -O oracli.zip

Unzip the Oracle Client into the library folder

unzip -j oracli.zip -d lib/

Ensure Libaio is insalled on the EC2 instance (it should be)

sudo yum install libaio -y

Copy Libaio to library

cp /lib64/libaio.so.1 lib/libaio.so.1

Zip up all your libraries to be used for the Lambda Layer

zip -y -r oracletable.zip python/ lib/

Copy your zipped library file to a S3 bucket so that the Lambda function can access it.

Adding Layer into Lambda

• Copy the url to be use when creating the layer
• Go to lambda and create a layer using AWS Lambda Layer documentation.
• Create the Lambda function using AWS Lambda Documentation.
• Add your code
• Add the layer your created
• Run the Lambda function
• Using CloudFormations

Below is an example of the CloudFormation code to create a Lambda Layer and Lambda function.

Note: Amazon Python Power Tools Layer documentation is at https://awslabs.github.io/aws-lambda-powertools-python/2.6.0/

Role:
  Type: 'AWS::IAM::Role'
  Properties:
    RoleName: lambda-role-name
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
              - s3.amazonaws.com
          Action:
            - 'sts:AssumeRole'
    Path: /
    Policies:
      - PolicyName: lambda-policy
        PolicyDocument:
          Version: "2012-10-17"
          Statement:
# Add access to secrets manager to give access to Lambda function to use
            - Effect: Allow
              Action: 
                - secretsmanager:GetResourcePolicy
                - secretsmanager:GetSecretValue
              Resource: 'arn:aws:secretsmanager:<region>:<account>:secret:<secretname>'
            - Effect: "Allow"
              Action:
                - s3:ListBucket
              Resource:
                - !Sub 'arn:aws:s3:::<S3 Bucket>'
            - Effect: "Allow"
              Action:
                - s3:GetObject
                - s3:PutObject
                - s3:PutObjectAcl
              Resource:
                - !Sub 'arn:aws:s3:::<S3 Bucket>/*'
                - !Sub 'arn:aws:s3:::<S3 Bucket>/*'
    ManagedPolicyArns:
      - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
      - 'arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

# Create the Oracle Layer for Python
OrcaleLayer:
  Type: AWS::Lambda::LayerVersion
  Properties:
    CompatibleRuntimes:
      - python3.7
    Content:
      S3Bucket: <S3 Lambda Bucket with code>
      S3Key: <layer ip file>
    Description: "Oracle Layer"
    LayerName: oracle-layer

# Create Lambda Function
OracleFunction:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: oracle-lambda
    Handler: oraclelambda.lambda_handler
    Layers:
# Oracle Client Library Layer
      - !Ref OracleLayer
# Add Python Power Tools
      - 'arn:aws:lambda:us-east-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:13'
    Role:
      Fn::GetAtt:
        - Role
        - Arn
    Environment:
#Pass Secret Name, Target Bucket, and S3 Prefix to the Python code
      Variables:
        SecretName: <secret name>
        TargetBucket: <target bucket>
        S3Prefix: <target prefix>
# Location of the Python Code
    Code:
      S3Bucket: <S3 Lambda Bucket with code>
      S3Key: <pthon code zip file>
    Runtime: python3.7
    Timeout: 90
    TracingConfig:
      Mode: Active
    VpcConfig:
      SecurityGroupIds: <security group id>
        - !Ref SecurityGroupId
      SubnetIds:
        - <subnet 1>
        - <subnet 2>

What’s next?

Currently, the process for updating the layers and code is very manual. I’m looking to find ways to automate getting the CloudFormation templates and Lambda to get updated when new changes are made.