This is Part 1 of a two-part post. In this part, I will cover provisioning the lambda docker image, and in the next part, I will cover interacting with it.
Covered:
- Declare Lambda Docker Image in CDK
- Importing local modules to main docker script
- Writing the Dockerfile
- Reading/Writing to Docker Image filesystem
- Testing locally
Prerequisites
- Docker installed
- CDK installed and set up correctly
Prior experience with Docker would be helpful, although I managed to figure this out and it was my first time dealing with Docker so you should be fine.
Directory structure
cdk-project
|-- lambdas/
|-- docker_func/
|-- app/
|-- __init__.py
|-- components/
|-- __init__.py
|-- com1.py
|-- com2.py
|-- utils/
|-- __init__.py
|-- ut1.py
|-- main.py
|-- output.json # Optional
|-- Dockerfile
|-- README.md
|-- requirements.txt
|-- project/
|-- project_stack.py
Process
Step by step process based on the above project structure. The file names I’ll be using in these steps directly relate to the files in the above Directory Structure.
Note: Starting in the root directory of your CDK project ../#/cdk-project>
1 - Declare the docker using CDK
# ../project/project_stack.py
docker_lambda = _lambda.DockerImageFunction(self, 'DockerLambda',
code=_lambda.DockerImageCode.from_image_asset(
'lambdas/docker_func'
),
timeout=Duration.seconds(30), # Default is only 3 seconds
memory_size=1000, # If your docker code is pretty complex
environment={
"BUCKET_NAME": bucket.bucket_name # Optional
}
)
2 - In a new terminal navigate to top-level of docker function
cd lambdas/docker_func
3 - Provision some basic components/util classes
# app/components/com1.pyclass Com1():
def __init__(self):
pass
###
# app/components/com2.pyclass Com2():
def __init__(self):
pass
###
# app/utils/ut1.pyclass Ut1():
def __init__(self):
pass
4 - Set components and utils to be treated as modules by adding __init__.py to each folder and importing the classes specified in #3.
# app/components/__init__.py
from components.com1 import *
from componented.com2 import *
###
# app/utils/__init__.py
from utils.ut1 import *
Warning
If you skip this step you will not be able to import these modules in the way that you think. When docker packages up your files it moves them all around which breaks relative imports (even though it may work locally)
5 - Write the dockerfile
FROM public.ecr.aws/lambda/python:3.8
# Copy function code
COPY app/main.py ${LAMBDA_TASK_ROOT}
# Add local dependencies
ADD app/components components
ADD app/utils utils
ADD app/template.docx template.docx
# Install the function's dependencies using file requirements.txt
# from your project folder.COPY requirements.txt .
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
CMD [ "main.handler" ]
Info
I'm usingmain.pyin this example, note all occurances and change if needed
In the Add local dependencies section of the Dockerfile there’s some important details.
These ADD commands tell docker that when it is packaging up your files it needs to grab the contents of source and place it in target when done.
eg. ADD source target > ADD app/components components
This means that once our app is built, we can access the files inside components via
from components.com1 import Com1
The last line CMD ["main.handler"] sets the docker entry point to the function handler in main.py:
# main.py
def handler(event, context):
pass
If you don’t want to call your file main.py then just change all instances where I use it to your preferred name.
6 - Deploy your changes. It should take a while to upload initially. I mean a long while. Like 7–10 minutes.
Writing to the local file system (optional)
If you try to write a file you will get:
[ERROR] OSError: [Errno 30] Read-only file ststel: '/path'
This is because the only writable directory with any AWS Lambda function is the /tmp folder.
If you need to write a document, do it like below:
document.save('/tmp/report.docx') # example using python-docx
Testing locally
Uploading the docker image every time you make a change is possibly the fastest way to transform your brain from goo to soup. Don’t do that. Do this instead.
Say you’ve got a Docker Image that uses the python-docx library (not important, it’s just a good example). Your script loads a template.docx file and saves the resulting output.docx file.
To test locally we use python’s if __name__ == "__main__": magic to provide dummy data to our handler function. This method will ONLY run when we manually run the script.
# main.py
def handler(payload, context, prefix='/tmp/'):
# ... code
document.save(prefix+"output.json")
# ... more code
return
if __name__ == "__main__":
with open("input.json") as f: # input.json sample event input
handler(json.load(f), None, prefix='')
Note in the above code I’m using a prefix argument to my handler() function. This is to control where the output file is written. For my local testing, I don’t want the file to be written to the /tmp directory because that will be a reserved folder once I deploy.
To run the file locally you will need to be in the app folder (referencing the directory structure from the start).
C:\<path>\cdk-project\lambdas\docker_func\app> python main.py
That should pretty much cover it. The above steps are all I needed to do to get my lambda working. For reference what it does is:
- One lambda function preps a json object and sends it to the Docker Image.
- Docker Image parses the json data, imports 3rd party libraries, reads and populates template documents, saves the output to docker filesystem, uploads the result to S3, and finally returns the S3 object URI to the Lambda that invoked it.
Hopefully this is helpful to anyone who reads. Thanks.
Top comments (0)