Deploying to AWS lambda has many advantages, covering cost and speed of execution. There are many options for deployment, including through the console, AWS CLI, CloudFormation and AWS SAM. However, the standard Lambda package size has a limit of 50mb zipped and 250mb unzipped, which in some cases can cause issues, especially with some machine learning frameworks, where the dependencies can hit this limit before you've even written any code.
To solve this issue, you can deploy your code inside a docker container, which has an effective limit of up to 10GB.
Part 1 of this solution will go through a number of AWS Services to show how to use AWS SAM to deploy a docker container to AWS Lambda, based on the Hello World Template.
Part 2 will show how to add this to a CodeBuild and CodePipeline and deploy automatically via GitHub.
Part 1 will use:
- AWS SAM
- AWS Lambda
- AWS API Gateway
- Docker
Part 2 will then use:
- Github
- AWS Github App
- AWS CodeStar Connection
- AWS Code Pipeline
- AWS CodeBuild
- AWS CloudFormation
This is what our deployment will look like after both parts are completed:
The first step is to set up AWS Access Keys: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
Then once you have the Access Key and Secret Key, you need to set them up in a terminal, by either setting up a profile, the AWS CLI directly using 'aws configure' or Environment Variables.
The core of this deployment method is to use the AWS SAM CLI. This deployment can be started with the sample hello world application.
You will need the AWS CLI and the AWS SAM CLI:
You can also create a blank application, however some of the template.yaml file from the hello world sample application will be useful, and we'll be using it as the base of this guide.
#Step 1 - Download a sample application
sam init
#Step 2 - Build your application
cd sam-app
sam build
#Step 3 - Deploy your application
sam deploy --guided
This will deploy your Lambda Application and create an API Gateway for you. The guided deploy will have some options to choose, make sure you allow role creation, and at this step, allow deployment without authorisation (which you should set up later if you don't want a public API) as sam will not deploy without it.
You could stop here, but this guide will show you how to turn the Lambda deployment into a Docker Container, and then automate the deployment through the use of Code Pipeline, Code Build and link it to Github.
Within the sam-app folder that is generated, the first file you will need to update is the template.yaml file. The section that creates the Lambda Function is with the type AWS::Serverless::Function.
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Runtime: python3.9
Architectures:
- x86_64
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
This deploys the Lambda as a python 3.9 application and zipped up, with the x86_64 architecture.
Containerising the Lambda
To change this to use docker we need to make some changes, and define an external docker_file. The main changes are the PackageType, ImageConfig and Metadata. We also remove the CodeUri, Handler and runtime as these are defined in the docker file.
We've also added a HelloWorldAPI section for the API, though one is implicitly defined. If you leave this out you will see some warnings about reserved keywords. This section is also where you can add Cors later on if needed.
Resources:
HelloWorldAPI:
Type: AWS::Serverless::Api
Properties:
Name: Hello World API
StageName: Prod
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
PackageType: Image
ImageConfig:
Command: ["app.lambda_handler"]
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
RestApiId: !Ref HelloWorldAPI
Path: /hello
Method: get
Metadata:
Dockerfile: hello_world_docker
DockerContext: .
DockerTag: v1
We then create a docker file that creates our docker container.
FROM public.ecr.aws/lambda/python:3.9
COPY hello_world/. ./
RUN python3.9 -m pip install -r requirements.txt
CMD ["app.lambda_handler"]
This uses the Python 3.9 image from the Elastic Container Repository, then copies our code into the home directory of the container, runs python to install the requirements.txt (which just contains the requests module for now) and then sets the lambda_handler to be the entry point.
We then need to build the container with sam:
sam build --use-container -t template.yaml
You can then deploy the container to Lambda using:
sam deploy --stack-name sam-app --resolve-s3 --capabilities CAPABILITY_IAM --resolve-image-repos
(you may need to remove any stored config file from any previous sam build, likely to be called samconfig.toml or it may override, especially the s3 parameters).
This will deploy your lambda to a stack named sam-app, work out the s3 bucket to use for deployment, give permission to create IAM roles for deployment, and work out which ECR Repository to use.
With this deployed, you can go to the API Gateway URL mentioned in the output of the sam deploy command. You should see:
{"message": "hello world"}
Part 2 will create these commands inside a CodeBuild environment, allowing a Code Pipeline to take the code from a GitHub repository, then use the SAM CLI inside CodeBuild to build the container and deploy it automatically.
You can also use the aws cloudformation describe-stacks command inside CodeBuild to get the API generated, and use command line tools like sed to then replace this URL into other files, such as a front end (which could then also be deployed by CodeBuild, or another pipeline).
Part 2 coming soon.
Top comments (2)
Part 2 ever going to happen?
Yeah, I will have part 2 up soon, likely in the next week or two. Ended up writing most of it, then getting caught up with other work.