DEV Community


Posted on


Running Python 3.11 on AWS Lambda

AWS Lambda functions have completely revolutionized the way I work with (and think about) compute. It's just amazingly convenient to implement a quick function, push it to Lambda, schedule it with EventBridge and let it run for free or for almost free.

However, if you're like me, you want to keep pace with the latest and greatest in the Python world, which, at the time of this writing, is Python 3.11. I love the way it speeds things up, I find the new error messages unbelievably useful and I like that tomllib became part of the standard library.

And, this passion to keep up with the latest and greatest interferes with using AWS Lambda, which still only offers Python 3.7, 3.8 and 3.9. There are rumors that 3.10 will be supported eventually, but 3.11 still seems far away.

Luckily, there's a way around this, and it's fairly easy to implement: create your own Lambda runtime.

There's ample documentation and a wide array of tutorials that show you how to do this (e.g. However, I also like to follow the D.R.Y. (Don't Repeat Yourself) principle, so why not package it as a reusable Docker image and use it in all my Lambda apps?

Step 1. Setting things up

Let's start by creating a new directory for our entire build setup:

mkdir -p python-lambda-runtimes 
cd python-lambda-runtimes
Enter fullscreen mode Exit fullscreen mode

Let's create a Dockerfile in the directory with the following content (we're going to look at what each line does after the code).

# Define custom function directory
ARG FUNCTION_DIR="/var/task/"

FROM python:3.11-slim-buster as build-image

# Include global arg in this stage of the build
RUN mkdir -p ${FUNCTION_DIR}

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
  apt-get install -y \
  g++ \
  make \
  cmake \
  unzip \

# Install the function's dependencies
RUN pip install --target ${FUNCTION_DIR} awslambdaric

FROM python:3.11-slim-buster

# Include global arg in this stage of the build
# Set working directory to function root directory

# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT [ "/usr/local/bin/python", "-m", "awslambdaric" ]
Enter fullscreen mode Exit fullscreen mode

Let's see what's happening here. We initiate a multi-phase build to reduce the size of our final image. On the FROM lines we choose which Python version we want to use for our runtime. If you wanted to work with Python 3.10 then you could simply replace the 3.11 part with 3.10.
As a last step of the first build phase we install awslambdaric, which is the AWS Lambda Runtime Interface Client which makes sure that the Lambda environment is able to communicate with our own code.
In the 2nd stage we simply copy the runtime interface client into our final image.

Step 2: Create a repository in your favorite Docker Container Registry

Since Docker has changed pricing model some time ago I started using AWS Elastic Container Registry (ECR) for my custom, private images. So, let's create a new repo. You can either use the AWS Console if you prefer the GUI, or if you like the CLI, you can simply run this command:

aws ecr create-repository --repository-name python-lambda-runtime
Enter fullscreen mode Exit fullscreen mode

This will create a repository called python-lambda-runtimes. Feel free to replace the name with anything you prefer.

Step 3: Log in to your preferred repo

For example, if you are using ECR you can use this to log in:

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>
Enter fullscreen mode Exit fullscreen mode

Please, replace with the ID of your AWS account.

Step 4: Build and push your image

To build your image and push it to ECR, you can do the following:

docker build -t python-lambda-runtime . && \
docker tag python-lambda-runtime:latest <AWS_ACCOUNT_ID> && \
docker push <AWS_ACCOUNT_ID>
Enter fullscreen mode Exit fullscreen mode

After this, you have everything in place to start using your pre-built image for your Lambda functions. In the next section, we'll do through all the steps you need to take to use it in your SAM template.

Step 5: Use the image in your SAM template

  1. Create a Dockerfile in the directory of your Lambda function. Add the following content:
COPY . /var/task/
RUN chmod -R 0755 .
RUN pip install -r requirements.txt
CMD ["app.lambda_handler"]
Enter fullscreen mode Exit fullscreen mode
  1. Look for the Type: AWS::Serverless::Function part of your template.yaml file. Remove the CodeURI, Handler, and the Runtime lines, and add PackageType: Image instead.
  2. Add a new section with the same indentation as the Type: AWS::Serverless::Function with the following content:
    Dockerfile: Dockerfile
Enter fullscreen mode Exit fullscreen mode

For example, if your function was called TestFunction and lived in the directory test_function of your project root, your Metadata would look like this:

    DockerTag: TestFunction
    DockerContext: ./test_function
    Dockerfile: Dockerfile
Enter fullscreen mode Exit fullscreen mode

If you run sam build, it will create the new container image for you.

If you've already deployed your Lambda function before, you'll have to delete it or deploy the containerized one under a new name. Also, because your samconfig.toml file already contains settings for your previous deployment, it makes sense to rename it to something else, and run sam deploy --guided to re-populate it with your new settings.

Top comments (4)

jcity profile image
Justin Myers

Great post!

Any idea if there is any additional steps to get this to work with API Gateway / Proxy?

gaborschulz profile image

Thanks. This works with API Gateway, I have it in use with it.

rafpironti profile image

Hey, this post is great.
Can you please expain better the step number 6 (SAM template)? I can not understand properly.
Best, Raffaele

gaborschulz profile image

Hi Raffaele,

Sure. About which part would you like more details?

Tired of sifting through your feed?

You can change your feed and see more relevant posts by adding a rating to different tags on DEV. Head here to adjust your weights.