DEV Community

Hasanul Islam
Hasanul Islam

Posted on • Edited on

1

Serverless Python: Managing Python Dependencies in AWS Lambda Layer

Usage:

 build_layer.sh <layer_name> <path/to/requirements/file> <python-version>
Enter fullscreen mode Exit fullscreen mode

for example,

 scripts/build_layer.sh tensorflow requirements/tensorflow.txt 3.8
Enter fullscreen mode Exit fullscreen mode

where requirements/tensorflow.txt contains:

tensorflow==2.4.0
Enter fullscreen mode Exit fullscreen mode

build_layer.sh

#!/bin/bash
set -eo pipefail

function usage {
    file_name=$0
    echo "usage: ${file_name} <layer_name> <path/to/requirements.txt/file> <python_version>"
    exit 1
}

layer_name=${1}
requirements_file=${2}
python_version=${3}

if [[ -z "${requirements_file}" ]]; then
  requirements_file="requirements.txt"
fi

if [[ -z "${layer_name}" || ! -f "${requirements_file}" ]]; then
  usage
fi

layer_path=".layers/${layer_name}/python/lib/python${python_version}/site-packages"
rm -rf "$layer_path"

echo "Installing packages from ${requirements_file}"
docker run --rm \
           --user $UID \
           --volume "$PWD:/var/task" \
           --workdir /var/task \
           --env CV_PYPI_USERNAME \
           --env CV_PYPI_PASSWORD \
           "lambci/lambda:build-python${python_version}" \
            pip install -Ur "${requirements_file}" --target "${layer_path}"

echo "Removing unnecessary packages from ${layer_path}"
rm -rf "$layer_path"/{botocore,boto3}*

echo "Removing unnecessary files/folders from ${layer_path}"
find "$layer_path" -type f -name "*.pyc" -exec rm -rf {} +
find "$layer_path" -type d -name "__pycache__" -exec rm -rf {} +
find "$layer_path" -type d \( -name "tests" -o -name "test" \)  -exec rm -rf {} +

echo "Layer creation finished at ${layer_path}"
Enter fullscreen mode Exit fullscreen mode

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read full post →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay