DEV Community

Python-T Point
Python-T Point

Posted on • Originally published at pythontpoint.in

🐍 How to package Python Lambda with virtualenv layers

#googlecloud #cloud #devops #python

🚀 Counterintuitive start — Adding a virtualenv layer can increase cold‑start latency, even though the layer shrinks the uploaded zip.

package python lambda with virtualenv layers

Packaging a Python Lambda with a virtualenv layer involves zipping the site‑packages directory, publishing it as a Lambda layer, and referencing that layer in the function configuration. At invocation the Lambda runtime extracts the layer, adds its python/ directory to sys.path, and then loads the handler. The extra extraction and path manipulation introduce a small initialization cost while keeping the deployment artifact smaller.

📑 Table of Contents

  • 🚀 Counterintuitive start — Adding a virtualenv layer can increase cold‑start latency, even though the layer shrinks the uploaded zip.
  • 🛠️ Virtual Environments — Why They Matter
  • 📦 Lambda Layers — How They Fit
  • 🔧 Build Process — Steps to Package
  • 🔗 Creating the venv and installing dependencies
  • 🗂️ Preparing the layer zip
  • ⚙️ Publishing the layer
  • 🚀 Deploy — Attaching Layers and Testing
  • 🔧 Creating the function
  • 🛠️ Updating the function configuration with the layer
  • 🔎 Invoking the function
  • 📊 Comparison — Layer vs. Bundled Deployment
  • 🟩 Final Thoughts
  • ❓ Frequently Asked Questions
  • Can I use multiple layers for a single Lambda function?
  • What runtime does the layer need to match?
  • How do I keep the layer size under the 50 MiB limit?
  • 📚 References & Further Reading

🛠️ Virtual Environments — Why They Matter

A virtual environment isolates a private copy of the interpreter, pip, and all installed packages. This isolation prevents version conflicts and eliminates unnecessary system libraries from the Lambda bundle.

$ python3 -m venv .venv
Created virtual environment in '.venv'.
$ source .venv/bin/activate
(.venv) $ pip -version
pip 23.2.1 from /home/user/.venv/lib/python3.11/site-packages/pip (python 3.11)
Enter fullscreen mode Exit fullscreen mode

What this does: (Also read: ☁️ aws cloudformation vs terraform for python deployments — which one should you use?)

  • python3 -m venv .venv : creates .venv with its own bin, lib, and include directories.
  • source .venv/bin/activate : prepends the venv’s bin to $PATH, ensuring the shell uses the venv‑specific python and pip.
  • pip -version : confirms that pip resolves to the venv installation.

Key point: A virtual environment provides a reproducible dependency set that can be packaged into a Lambda layer without affecting the base runtime.

📦 Lambda Layers — How They Fit

A Lambda layer is a .zip archive that the runtime extracts and adds to the function’s sys.path. Only the python/ sub‑directory is considered, which keeps the layer size minimal. (Also read: 🐍 kubectl exec hangs when running Python scripts — what's going on)

$ mkdir -p layer/python
$ cp -r .venv/lib/python3.11/site-packages/* layer/python/
$ cd layer && zip -r9 ../my_layer.zip . adding: python/ ...
Enter fullscreen mode Exit fullscreen mode

What this does: (Also read: 🚀 Terraform deploy for Python Flask and Docker made easy)

  • mkdir -p layer/python : creates the directory hierarchy required by Lambda.
  • cp -r .venv/lib/python3.11/site-packages/* layer/python/ : copies only the installed packages, omitting binaries and test files that would bloat the layer.
  • zip -r9 ../my_layer.zip . : archives the directory with maximum compression; the -r9 flags preserve the tree structure.

Key point: Including only the python/ folder avoids exceeding the 50 MiB compressed layer limit.

🔧 Build Process — Steps to Package

🔗 Creating the venv and installing dependencies 

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install -r requirements.txt
Collecting requests==2.28.2 Downloading requests-2.28.2-py3-none-any.whl (62 kB)
Collecting boto3==1.26.0 Downloading boto3-1.26.0-py2.py3-none-any.whl (131 kB)
Successfully installed boto3-1.26.0 requests-2.28.2
Enter fullscreen mode Exit fullscreen mode

🗂️ Preparing the layer zip 

$ mkdir -p layer/python
$ cp -r .venv/lib/python3.11/site-packages/* layer/python/
$ cd layer && zip -r9 ../my_layer.zip . adding: python/requests/... adding: python/boto3/...
$ cd ..
Enter fullscreen mode Exit fullscreen mode

⚙️ Publishing the layer 

$ aws lambda publish-layer-version \ -layer-name my-python-deps \ -zip-file fileb://my_layer.zip \ -compatible-runtimes python3.11
{ "LayerVersionArn": "arn:aws:lambda:us-east-1:123456789012:layer:my-python-deps:1", "Version": 1, "Description": "", "CreatedDate": "-09-12T14:23:45.123+0000", "CompatibleRuntimes": [ "python3.11" ]
}
Enter fullscreen mode Exit fullscreen mode

What this does:

  • aws lambda publish-layer-version : uploads the zip and registers it as a reusable layer.
  • - compatible-runtimes python3.11: binds the layer to the Python 3.11 runtime.
  • The returned LayerVersionArn is used when configuring the function.

According to the AWS Lambda documentation, a layer can be shared across up to five functions in the same account, reducing duplication and speeding future deployments.

🚀 Deploy — Attaching Layers and Testing

🔧 Creating the function 

$ mkdir -p src
$ echo 'def handler(event, context):\n import requests\n return {"statusCode": 200, "body": requests.get("https://example.com").text[:100]}' > src/lambda_function.py
$ zip -r function.zip src adding: src/ adding: src/lambda_function.py
Enter fullscreen mode Exit fullscreen mode

🛠️ Updating the function configuration with the layer 

$ aws lambda create-function \ -function-name my-python-lambda \ -runtime python3.11 \ -handler lambda_function.handler \ -zip-file fileb://function.zip \ -role arn:aws:iam::123456789012:role/lambda-exec-role
{ "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:my-python-lambda", "Runtime": "python3.11", "Handler": "lambda_function.handler", "CodeSize": 1234, "Description": "", "Timeout": 3, "MemorySize": 128, "LastModified": "-09-12T14:30:12.000+0000", "CodeSha256": "abcd1234...", "Version": "$LATEST"
}



$ aws lambda update-function-configuration \ -function-name my-python-lambda \ -layers arn:aws:lambda:us-east-1:123456789012:layer:my-python-deps:1
{ "FunctionName": "my-python-lambda", "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:my-python-lambda", "Runtime": "python3.11", "Role": "arn:aws:iam::123456789012:role/lambda-exec-role", "Handler": "lambda_function.handler", "Layers": [ { "Arn": "arn:aws:lambda:us-east-1:123456789012:layer:my-python-deps:1", "CodeSize": 456789 } ], "CodeSize": 1234, "Description": "", "Timeout": 3, "MemorySize": 128, "LastModified": "-09-12T14:31:00.000+0000", "Version": "$LATEST"
}
Enter fullscreen mode Exit fullscreen mode

🔎 Invoking the function 

$ aws lambda invoke \ -function-name my-python-lambda \ -payload '{}' \ response.json
{ "StatusCode": 200, "ExecutedVersion": "$LATEST"
}
$ cat response.json
{"statusCode":200,"body":"...first 100 characters of example.com..."}
Enter fullscreen mode Exit fullscreen mode

What this does:

  • aws lambda create-function : registers the handler code and base configuration.
  • aws lambda update-function-configuration -layers : attaches the published layer, making its packages available at runtime.
  • aws lambda invoke : runs the function synchronously; the response confirms successful execution.

Key point: The function runs with dependencies supplied by the layer, demonstrating a complete end‑to‑end workflow without embedding libraries in the function zip. (More onPythonTPoint tutorials)

📊 Comparison — Layer vs. Bundled Deployment

Aspect Layer Approach Bundled Approach
Deployment size Function zip stays < 1 MiB (layer holds libs) Function zip includes all libs, often > 10 MiB
Cold‑start latency +~50 ms overhead to mount layer 0 ms extra, but larger download time for the zip
Versioning Layer version reusable across many functions Each function must be updated independently
Maximum size limit Layer limit 50 MiB (compressed), function zip 250 MiB Single zip limited to 250 MiB; large dependencies may exceed

Key point: Layers provide modularity and reuse with a modest cold‑start penalty; bundling is simpler but scales poorly as dependency count grows.

Using virtualenv layers isolates dependencies cleanly, allowing code updates without repackaging heavy libraries each time.

🟩 Final Thoughts

The described workflow—creating a virtual environment, zipping its site‑packages into a Lambda layer, and attaching that layer to a function—optimizes build speed, deployment size, and runtime performance. For typical production workloads the slight cold‑start increase is outweighed by the operational benefits of a single reusable layer.

Adopting this pattern means future updates affect only the function code; the layer, which houses all third‑party packages, can be versioned independently. This aligns with immutable infrastructure principles: each artifact (code zip, layer zip) is built once and never mutated in place.

❓ Frequently Asked Questions

Can I use multiple layers for a single Lambda function?

Yes. Lambda supports up to five layers per function. Layers are merged in the order listed; later layers can override files from earlier ones.

What runtime does the layer need to match?

The layer must be published with the same runtime version (e.g., python3.11) that the function uses; otherwise deployment fails.

How do I keep the layer size under the 50 MiB limit?

Exclude unnecessary files (tests, documentation, binaries) before zipping, and consider using --no-deps for packages already present in the base runtime.

💡 Want to practise this hands-on? DigitalOcean gives new accounts $200 free credit for 60 days — enough to spin up a full Linux/Docker/Kubernetes environment at no cost.

📚 Recommended reading: Best DevOps & cloud books on Amazon — from Linux fundamentals to Kubernetes in production, curated for working engineers.

📚 References & Further Reading

  • Official AWS Lambda documentation — detailed guide on layers and runtime behavior: docs.aws.amazon.com
  • Python virtual environments — how venv works and best practices: docs.python.org
  • AWS CLI reference — commands for publishing layers and updating functions: docs.aws.amazon.com

Top comments (0)