How to Package Dependency for AWS Lambda with Docker

Introduction:

This guide explains how to package dependencies for your AWS Lambda function. The tutorial demonstrates automating the process of packaging Python dependencies for AWS Lambda layers using Docker.

By the end of this guide, you will have created a Dockerfile. Building this file produces a Docker Image. From this image, you can run a Docker container to generate a .zip file. You can then upload this file to Lambda as a dependency package for your Lambda function.

We will explore two methods:

1. Manual method: involving setting up an EC2 instance

2. Automated method: using Docker containers.

So, let's dive right in! We'll walk you through each step for both the manual and automated approaches in the sections below. Let's get started!

Manual Approach:

This method requires setting up the following:

• Launching: an EC2 instance with Amazon Linux OS.
• Installing: the following:
  • The desired version of Python,
  • Pip, and
  • The external Python library into a target folder.
• Compressing: the library folder.
• Using: scp (secure copy) to transfer it from the remote machine to your local computer.
• Uploading: the compressed folder from step 3 to the Lambda function.

This approach could be ideal if you're fond of hands-on processes. However, for those in search of automation and a method that's free of cloud charge, kindly opt for the second approach.

Automated Approach:

In this method, we'll use Docker containers to streamline the process. We'll explore the files involved in the project and the functions they perform.

1. Dockerfile

This is the Dockerfile, serving as the entry point into our code. We use it to configure the Amazon Linux image as the OS within our container. It orchestrates the setup of all supporting code files, such as install_python.sh and requirements.txt. Each line in the code snippet has a comment to explain its function.

# Use the official Amazon Linux Docker image
FROM amazonlinux:latest

# Update package repositories and install necessary packages
RUN yum update -y

# Set the working directory
WORKDIR /usr/src

# Copy the bash script and requirements file into the container
COPY install_python.sh .
COPY requirements.txt .

# Make the bash script executable
RUN chmod +x install_python.sh

# Run the bash script to install Python
RUN ./install_python.sh

# Install Python dependencies into 'modules' directory
RUN pip3.12 install -r requirements.txt -t /usr/src/modules/

# Zip up the installed dependencies
RUN zip -9r /usr/src/modules.zip /usr/src/modules

# Declare a volume to persist data
VOLUME /usr/src/data

# Move modules.zip to the volume
CMD ["mv", "/usr/src/modules.zip", "/usr/src/data/"]

2. Install_python.sh
This script builds and installs Python from source, along with installing pip. It is called from the Dockerfile.

#!/bin/bash

# Install development tools
yum groupinstall "Development Tools" -y

# Install necessary dependencies
yum install -y gcc bzip2-devel libffi-devel openssl openssl-devel wget

# Set Python version
PYTHON_VERSION="3.12.2"

# Download Python source
wget "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz"

# Extract Python source
tar xzf "Python-${PYTHON_VERSION}.tgz"

# Navigate into the extracted directory
cd "Python-${PYTHON_VERSION}"

# Configure Python build with OpenSSL 1.1
./configure --with-openssl=/usr/lib64/openssl11 --enable-optimizations

# Build and install Python
make && make install

# Download get-pip.py
curl -O "https://bootstrap.pypa.io/get-pip.py"

# Install pip for Python 3.12
python3.12 get-pip.py

# Check pip version
pip3.12 --version

3. Requirements.txt
Our requirements.txt file lists the libraries needed for packaging the project, following a standard practice in Python projects. The content of the file is as follows:

requests

Compile Python Version

To compile a different Python version, you can obtain the link to your preferred version from here. You have to ensure that the link points to a .tgz file. If you select a different version from the one in this post, adjust the following lines in the Dockerfile and install_python.sh.

1. Install_python.py:

# Set Python version
PYTHON_VERSION="3.12.2"

# Install pip for Python 3.12
python3.12 get-pip.py

# Check pip version
pip3.12 --version

The third digit in the version number, 3.12.2, is included in the PYTHON_VERSION, but Python ignores it when running get-pip.py and in pip3.12. This applies to any version of Python we decide to use.

2. Dockerfile

# Install Python dependencies into 'modules' directory

RUN pip3.12 install -r requirements.txt -t /usr/src/modules/

Result:

When we build our Docker image and convert it to a container, the output is a zip file named modules.zip in the same folder as our Dockerfile. Below is the terminal log of successfully runing our Dockerfile.

Code:

Our code is hosted on a GitHub repository. You can access it: here.

Conclusion:

In this guide, we covered two approaches to packaging Python dependencies for AWS Lambda. The first involves using an AWS EC2 instance, installing Amazon Linux OS on it, and building Python from source, along with installing pip, which is then used to install dependencies listed in the requirements.txt file. The second approach streamlines the process using Docker, with the Dockerfile as the entry file.

You can read more in the AWS documentation here.