Deploying lambda with custom docker image

Hello everyone,

Scenario: Whenever I want to deploy any Python code or NodeJS code using lambda, the first thing that comes to my mind is, How to deploy their required libraries
Suppose I want to deploy a Python code that will depend on the Pandas library and some other libraries used for Machine Learning. When I combine all these libraries and try to create a Lambda Layer for using them in the code, I get the error size limit exceeded. The maximum limit that we can upload to Lambda Layer is 250MB. If the size is more than that we can't upload them to the Lambda Layer.

Solutions: We can solve this problem in two ways.

• One way is to upload zipping all the libraries including lambda function code and upload that zip file to the lambda

• The second way is to create a docker image and deploy that image to lambda

Will see how to implement the above two ways

Zipping code and uploading it to Lambda:

• First, create a folder and install all the required libraries in that folder. For Python, you can use pip install pandas -t /path/to/folder command to install the Python libraries to a specific folder
• After installing the required libraries, create a file named lambda_function.py and place your code in that file.
• The folder structure should look like below

• Go to Lambda and click on the code. From the right side click on the Upload drop-down button and select .zip file

Zip your folder and upload that zip file here.

• Once it's uploaded, within seconds your code is ready to run.
   

• Make sure that lambda_function.pyfile has the def lambda_handler(event, context)function. Lambda execution starts from there and make sure that the function returns something at the end of the execution so that lambda knows execution is successfully completed.

• But the main problem I faced was, that I had to write the code in the local system, make it zip, and upload it again if I wanted to test my code repeatedly.

Building custom docker image and deploying to Lambda:

Before building the docker image there are certain things we need to take care of. As I said above Lambda execution starts from lambda_handler(event, context) function. So we need to alter the code according to that. One more thing is after the function execution we need to return some data so that lambda knows the execution is done successfully

Will divide this task into 3 small tasks like below

• Building a docker image
• Uploading docker image to ECR(Elastic Container Registry)
• Deploying Lambda function using ECR docker image

Build docker image:

• Once the code is ready create a file named Dockerfile in the root directory of the code 

• Sample Dockerfile code

FROM public.ecr.aws/lambda/python:3.9

COPY source/folder destination/folder ###(copy required files to the image)
RUN pip install -r requirements.txt ###installing required libraries
CMD ["export_to_csv.lambda_handler"] ###(executing python file code name export_to_csv and calling handler function)

• Once you have the Python code and docker file ready let's build the docker. Before building the docker image let's create ECR Repo

Uploading docker image to ECR:

• Visit the ECR service for the AWS search bar
• Click on Get Started or Create Repository, give a name for the repo and click on Create Repository

• Click on the created repository and click on view push commands button

• You can find the commands to build the image and push it to the repository

aws ecr get-login-password --region {regionname} | docker login --username AWS --password-stdin {ECR Url}   #Login into ecr
docker build -t test /code/folder #building docker image from current folder
docker tag test:latest {ECR Url}/{repo}:latest #tagging docker image
docker push {ECR Url}/{repo}:latest #pushing to ECR

Deploying Lamba with docker image:
Now we have a docker image with all dependencies ready with the code. Let's deploy it to Lambda

• Go to Lambda service and click on create function. From the options select container image. Give a name to the function

• Browse the ECR repo and select the uploaded image from the browse image button. Click on the Create function button.

That's it you have deployed a lambda function using a custom docker image. 
You can use AWS Code Pipeline to build the docker image and upload it into ECR whenever code changes are pushed to the code repo.