DEV Community: Adedamola Adedapo

How to Package Dependency for AWS Lambda with Docker

Adedamola Adedapo — Mon, 13 May 2024 18:21:17 +0000

Introduction:

This guide explains how to package dependencies for your AWS Lambda function. The tutorial demonstrates automating the process of packaging Python dependencies for AWS Lambda layers using Docker.

By the end of this guide, you will have created a Dockerfile. Building this file produces a Docker Image. From this image, you can run a Docker container to generate a .zip file. You can then upload this file to Lambda as a dependency package for your Lambda function.

We will explore two methods:

Manual method: involving setting up an EC2 instance
Automated method: using Docker containers.

So, let's dive right in! We'll walk you through each step for both the manual and automated approaches in the sections below. Let's get started!

Manual Approach:

This method requires setting up the following:

Launching: an EC2 instance with Amazon Linux OS.
Installing: the following:
- The desired version of Python,
- Pip, and
- The external Python library into a target folder.
Compressing: the library folder.
Using: scp (secure copy) to transfer it from the remote machine to your local computer.
Uploading: the compressed folder from step 3 to the Lambda function.

This approach could be ideal if you're fond of hands-on processes. However, for those in search of automation and a method that's free of cloud charge, kindly opt for the second approach.

Automated Approach:

In this method, we'll use Docker containers to streamline the process. We'll explore the files involved in the project and the functions they perform.

1. Dockerfile

This is the Dockerfile, serving as the entry point into our code. We use it to configure the Amazon Linux image as the OS within our container. It orchestrates the setup of all supporting code files, such as install_python.sh and requirements.txt. Each line in the code snippet has a comment to explain its function.



# Use the official Amazon Linux Docker image
FROM amazonlinux:latest

# Update package repositories and install necessary packages
RUN yum update -y

# Set the working directory
WORKDIR /usr/src

# Copy the bash script and requirements file into the container
COPY install_python.sh .
COPY requirements.txt .

# Make the bash script executable
RUN chmod +x install_python.sh

# Run the bash script to install Python
RUN ./install_python.sh

# Install Python dependencies into 'modules' directory
RUN pip3.12 install -r requirements.txt -t /usr/src/modules/

# Zip up the installed dependencies
RUN zip -9r /usr/src/modules.zip /usr/src/modules

# Declare a volume to persist data
VOLUME /usr/src/data

# Move modules.zip to the volume
CMD ["mv", "/usr/src/modules.zip", "/usr/src/data/"]

2. Install_python.sh
This script builds and installs Python from source, along with installing pip. It is called from the Dockerfile.



#!/bin/bash

# Install development tools
yum groupinstall "Development Tools" -y

# Install necessary dependencies
yum install -y gcc bzip2-devel libffi-devel openssl openssl-devel wget

# Set Python version
PYTHON_VERSION="3.12.2"

# Download Python source
wget "https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz"

# Extract Python source
tar xzf "Python-${PYTHON_VERSION}.tgz"

# Navigate into the extracted directory
cd "Python-${PYTHON_VERSION}"

# Configure Python build with OpenSSL 1.1
./configure --with-openssl=/usr/lib64/openssl11 --enable-optimizations

# Build and install Python
make && make install

# Download get-pip.py
curl -O "https://bootstrap.pypa.io/get-pip.py"

# Install pip for Python 3.12
python3.12 get-pip.py

# Check pip version
pip3.12 --version

3. Requirements.txt
Our requirements.txt file lists the libraries needed for packaging the project, following a standard practice in Python projects. The content of the file is as follows:



requests

Compile Python Version

To compile a different Python version, you can obtain the link to your preferred version from here. You have to ensure that the link points to a .tgz file. If you select a different version from the one in this post, adjust the following lines in the Dockerfile and install_python.sh.

1. Install_python.py:



# Set Python version
PYTHON_VERSION="3.12.2"

# Install pip for Python 3.12
python3.12 get-pip.py

# Check pip version
pip3.12 --version

The third digit in the version number, 3.12.2, is included in the PYTHON_VERSION, but Python ignores it when running get-pip.py and in pip3.12. This applies to any version of Python we decide to use.

2. Dockerfile



# Install Python dependencies into 'modules' directory

RUN pip3.12 install -r requirements.txt -t /usr/src/modules/

Result:

When we build our Docker image and convert it to a container, the output is a zip file named modules.zip in the same folder as our Dockerfile. Below is the terminal log of successfully runing our Dockerfile.

Code:

Our code is hosted on a GitHub repository. You can access it: here.

Conclusion:

In this guide, we covered two approaches to packaging Python dependencies for AWS Lambda. The first involves using an AWS EC2 instance, installing Amazon Linux OS on it, and building Python from source, along with installing pip, which is then used to install dependencies listed in the requirements.txt file. The second approach streamlines the process using Docker, with the Dockerfile as the entry file.

You can read more in the AWS documentation here.

Provisioning Multiple AWS RDS Instances with Terraform

Adedamola Adedapo — Sat, 23 Mar 2024 19:06:57 +0000

In this post, we'll create multiple RDS instances using Terraform, a cloud provider-agnostic Infrastructure as Code (IAC) tool, primarily used by DevOps teams to automate infrastructure tasks. For a quick introduction to AWS RDS, you can refer to the blog by K21Academy here.

Prerequisites

To complete this tutorial, ensure that the AWS CLI client is installed and configured with the access and secret keys. If you haven't done this yet, follow the instructions in the previous post here to complete this stage.

1. Create Development Folder

Create the folder named RDS and within it, create a file named main.tf.

$ mkdir RDS
$ touch RDS/main.tf

Directory structure:

- RDS/
  - main.tf

2. Provider Configuration

Open main.tf in your favorite editor (if you use vim, say hi in the comment😊) and add the following code:

provider "aws" {
    region = ""
    profile = ""
}

Explanation:

Terraform, being cloud provider agnostic, requires a provider block. The AWS provider authenticates Terraform with your AWS account and exposes the necessary resources for management.
region: Specifies the AWS region where resources will be provisioned or managed.
profile: Specifies the AWS profile name, especially useful if multiple profiles are defined. If not, set it to default.

After filling:

provider "aws" {
    region = "us-west-2"
    profile = "default"
}

3. Instance Template

Below is the Terraform template for creating an RDS instance:

resource "aws_db_instance" "db_instance1" {
  engine                 = "postgres"       # Specify the database engine (PostgreSQL)
  engine_version         = "16.1"           # Specify the PostgreSQL version
  multi_az               = false            # Set to true for Multi-AZ deployment
  identifier             = "xxxxxxxxxx" # Unique identifier for the first DB instance
  username               = "xxxxxxxxxx"     # Username for the master DB user of the first instance
  password               = "xxxxxxxxxx"     # Password for the master DB user of the first instance
  instance_class         = "db.m5d.xlarge"  # Instance type for the first DB instance
  allocated_storage      = xxx              # Allocated storage in GB for the instance
  db_subnet_group_name   = aws_db_subnet_group.database_subnet_group.name
  vpc_security_group_ids = [aws_security_group.database_security_group.id]
  availability_zone      = data.aws_availability_zones.available_zones.names[0]
  db_name                = "xxxxxxxxx"
  skip_final_snapshot    = true
  publicly_accessible    = true             # Allow access from the public internet
}

4. Define Dependencies:

The aws_db_instance is a Terraform block that produces an RDS instance resource. This RDS instance is also a database instance, which is an isolated database environment in the cloud. A DB instance can contain multiple user-created databases depending on preference.

engine: The engine field specifies the type of database instance to be provisioned. Common examples are MySQL, PostgreSQL, and Amazon Aurora. We are experimenting with the PostgreSQL engine.
engine_version: This defines the version of the Postgres database to be used. Note that this version must be within the ranges recognized by AWS. Therefore, check your web console to be sure.
multi_az: Defines if instance the should be provisioned in multiple AWS available zone. This is useful in cases where our customers are in different geographical location, and we will have planned to deployed database to be queried by each customer group in the same location with them. This is mostly useful in a microservice architecture, but our design is a simple one, as a result, we have explicitly set it value to false.
identifier: This is the field that specifies a unique identifier for the db instance.
username: Defines a username for the db instance.
password: Password to be used to access the database instance after it is provisioned. I recommend that you use at least a 12-character-long password. Use a free password generator here.
instance_class: Defines the class of the database instance to be provisioned. For a list of all classes with great insight into usage, visit here.
allocated_storage: Size of the db storage capacity.
db_subnet_group_name: Defines the IP address ranges group to add the database instance to. The Terraform block for creating a subnet is aws_db_subnet_group. Preceding steps are dependent steps to achieve the aim. Follow the steps in the code below to define a subnet.

#step 1: use data source to get all availability zones in region
data "aws_availability_zones" "available_zones" {}

#step 2: create a default subnet in the first AZ if one does not exist
resource "aws_default_subnet" "subnet_az1" {
  availability_zone = data.aws_availability_zones.available_zones.names[0]
}

#step 3: create a default subnet in the second AZ if one does not exist
resource "aws_default_subnet" "subnet_az2" {
  availability_zone = data.aws_availability_zones.available_zones.names[1]
}

#step 4: create the subnet group for the rds instance
resource "aws_db_subnet_group" "database_subnet_group" {
  name        = "database-subnets"
  subnet_ids  = [aws_default_subnet.subnet_az1.id, aws_default_subnet.subnet_az2.id]
  description = "subnets for database instance"

  tags = {
    Name = "database-subnets"
  }
}

vpc_security_group_ids: VPC security group controls the traffic that is allowed to reach and leave the resources within the VPC. This field specifies the ID of the predefined VPC security group to be used. Follow the code below to define one:

#Step 1: create default vpc if one does not exist
resource "aws_default_vpc" "default_vpc" {
  tags = {
    Name = "default vpc"
  }
}

# create security group for the database
resource "aws_security_group" "database_security_group" {
  name        = "database security group"
  description = "enable postgres/aurora access on port 5432"
  vpc_id      = aws_default_vpc.default_vpc.id

  ingress {
    description = "postgre/aurora access"
    from_port   = 5432
    to_port     = 5432
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # Modify this to allow access from specific IPs
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "database security group"
  }
}

availability_zone: Availability zones (AZs) are isolated data centers located within specific regions in which public cloud services originate. Remember that you have defined a region to provision your instances in the provider block. Typically, a region contains at least 2 available zones; this field allows you to specify which of the zones to use. To get all available zones, you use a data block in Terraform as shown below:

# use data source to get all availability zones in region
data "aws_availability_zones" "available_zones" {}

Select the first AZ as shown below:

data.aws_availability_zones.available_zones.names[1]

db_name: Takes a unique name for the db instance.
skip_final_snapshot: Determines whether a final DB snapshot is created before the DB cluster is deleted. If true is specified, no DB snapshot is created.
publicly_accessible: This is a true or false field. If true, it means that your database instance will be accessible from the public internet. However, note that this is also dependent on the defined ingress cidr_block. If it is not set to be accessible from anywhere i.e., 0.0.0.0/0, then only the specified CIDR will have access to the database instance.

5. Create multiple instances

You can create multiple instances by copying and pasting the aws_db_instance block template and making the necessary fields unique to each instance. Alternatively, a more sophisticated approach would involve specifying an array of objects holding unique fields for each instance and iterating through them to create instances dynamically.

You can download an extended template for creating multiple databases here.

6. Running the code

Change your directory into the folder where the multiple_postgres_db.tf file is. Update all the x-redacted field values. Then perform the operation below.

$ terrafrom init
$ terraform plan
$ terraform apply

After running the files successfully, you get the 3 db instance host endpoint output on your command. To connect to the DB instances, assuming you have psql, the Postgres terminal client is installed. Run the following command.

$ psql -U <username> -h <db host endpoint as outputted as applying the .tf script> -p 5432 <db_name>

Thank you 🙏🤗🙌

AWS CLI Setup: IAM User Configuration & Command-Line Access

Adedamola Adedapo — Sat, 16 Mar 2024 20:41:17 +0000

Assuming you've already navigated through the AWS Management Console and set up an IAM user using the AWS web console, as well as generated the essential access and secret keys for programmatic access, you're now ready to configure the AWS Command Line Interface (CLI) for seamless interaction with AWS services.

If you haven't completed these initial steps yet, don't worry! You can follow along with this hassle-free setup video here.

With your IAM user credentials in hand, let's proceed to install and configure the AWS CLI.

1. Installing the AWS CLI

To start, you'll need to install the AWS CLI client on your system. Refer to the detailed installation process outlined in the AWS official documentation here.

Once installed, you can verify that the CLI is accessible from your shell by using the following commands:

$ which aws
/usr/local/bin/aws 
$ aws --version
aws-cli/2.11.21 Python/3.11.3 Darwin/22.5.0 exe/x86_64 prompt/off

2. Configuring the AWS CLI

Now that the AWS CLI is successfully installed, the next step is to configure it with your IAM user credentials. You can do this by running the aws configure command in your terminal and filling in the prompted fields with your access and secret keys.

$ aws configure
AWS Access Key ID [None]: <paste access key> -- generated in step 1 
AWS Secret Access Key [None]: <paste secret key> -- generated in step 1 
Default region name [None]:
Default output format [None]:

Remember, you can always update these configuration values by rerunning the aws configure command.

3. Finding Your Default Region

To determine the appropriate value for the Default region name prompt, navigate to your AWS Management Console. Open the service you intend to access or make API calls to, and in the top-right corner of the screen, click on the second menu item to the left. Find the region that matches your desired location, such as eu-west-1 as in the image below.

Note that you cannot obtain your service region from the IAM service, as it is a global service.

Conclusion

With the AWS CLI now configured, you're all set to interact with AWS services seamlessly from your command line interface.