This article is concise material from "AWS Dev Hour"
https://www.twitch.tv/videos/892197005 lasts just over an hour, this article will go through the main notes and has a separate working GitHub I'm building into.
https://github.com/DarrenBro/modernApp-AWS-CDK
There were however some issues I came across not covered by the video that I've included fixes for or tasks you should complete on your end.
Node Version 15+ had issues with deploying lambda assets https://github.com/aws/aws-cdk/issues/12536
Fix ->Downgrading node and deleting cdk.out/.cache should stop this issue from occurring.
https://stackoverflow.com/questions/8191459/how-do-i-update-node-js
If you continue without completing steps not shown in the video for getting pillow(PIL) into your directory (steps shown below title Pillow) you'll get the following CloudWatch log error;
Fix -> I've included the necessary extracted zip file and put it in the project under the folder "reklayer".
The video misses out showing the creation of the lambda layer. This allows the lambda to import 'PIL' from your attached zip file you created.
Fix -> I've attached code snippets that need included in cdkMainStacks.ts.
Otherwise you'll get the below error in your CloudWatch logs.
'index' is referring to the main body of the rekognition lambdaI've flooded cdkMainStacks.ts with comments better explaining different components. This file is the main area that most of the code will get added.
For me this has been a great way to get more hands on with CDK with good progression intervals and building it out yourself is the best way to get familiar with any tech, plus finding solutions for these errors has helped commit it more to memory.
Tech Stack
aws-cli
https://docs.aws.amazon.com/cli/latest/userguide/install-macos.html
Node.js to run the project (Version 10.3.0–14.15.3)
https://nodejs.org/en/download/
IDE that can run any of these languages;
Typescript, Javascript, Java, C# or Python
(I'm using IntelliJ with Typescript)
AWS CDK toolkit and bootstrap
Details of the App
Built using CDK
User will be able to upload a photo through a react JS UI, stored in s3 (upcoming sessions).
It will then trigger a lambda upon the s3 action and store it in dynamoDB (which stores metadata and labels, done this session)
We can then query(scan) the db to check labels for the image have been added after going through rekognition service.
All graphics are taken from the AWS run through video.
Session 1 will consist of;
Installing prerequisites
Explaining what AWS CDK is
Creating a new project and getting dependencies installed
Building 3 components -> s3 / lambda & dynamo db
Copying an image to scan and scanning the table
What is AWS CDK?
A framework we can use to declare our AWS resources
Sits on top of Cloud Formation
(CDK produces the CFT Template after synthesise "cdk synth")
Simplifies the process of building out these resource templates
This is what an application looks like in the CDK, starting with the App, and broken into stacks and constructs(which is what will be built later on).
Steps to get started;
"mkdir cdk-project"
"cdk init" (shows templates available)
"cdk init app -l=typescript" (Go with typescript)
Install aws-iam "npm i @aws-cdk/aws-iam"
Notes
app initialised in cdk-projects.ts
cdk.json tells cdk toolkit how to run the project
Pillow
The AWS Lambda function uses the Pillow library for the generation of thumbnail images. This library needs to be added into our project so that we can allow the CDK to package it and create an AWS Lambda Layer for us. To do this, you can use the following steps.
Launch an Amazon EC2 Instance (t2-micro) using the Amazon Linux 2 AMI
SSH into your instance and run the following commands:
sudo yum install -y python3-pip python3 python3-setuptools
python3 -m venv my_app/env
source ~/my_app/env/bin/activate
cd my_app/env
pip3 install pillow
cd /home/ec2-user/my_app/env/lib/python3.7/site-packages
mkdir python && cp -R ./PIL ./python && cp -R ./Pillow-8.1.0.dist-info ./python && cp -R ./Pillow.libs ./python && zip -r pillow.zip ./python
Copy the resulting archive 'pillow.zip' to your development environment (we used an Amazon S3 bucket for this)
Extract the archive into the 'reklayer' folder in your project directory
Your project structure should look something like this:
project-root/reklayer/python/PIL
project-root/reklayer/python/Pillow-8.1.0.dist-info
project-root/reklayer/python/Pillow.libs
Remove the python.zip file to clean up
Terminate the Amazon EC2 Instance that you created to build the archive
1st component to build is s3
npm i @aws-cdk/aws-s3
In cdkMainStack.ts we'll be adding;
import s3 = require('@aws-cdk/aws-s3');
const imageBucket = new s3.Bucket(this, imageBucketName, {
removalPolicy: cdk.RemovalPolicy.DESTROY
})
new cdk.CfnOutput(this, 'imageBucket', {value: imageBucket.bucketName});
A quick note on the constructs here.
So for s3 code above it's the same principle.
2nd component to build is dynamoDB for storing the image labels
npm i @aws-cdk/aws-dynamodb
const imageTable = new dynamodb.Table(this, 'ImageLabels', {
partitionKey: {name: 'image', type: dynamodb.AttributeType.STRING},
removalPolicy: cdk.RemovalPolicy.DESTROY
});
new cdk.CfnOutput(this, 'cdkTable', {value: imageTable.tableName});
3rd component to build is Lambda layer and Lambda
Layer needed to import the PIL library when rekognition lambda is executed.
const layer = new lambda.LayerVersion(this, 'pil', {
code: lambda.Code.fromAsset('reklayer'),
compatibleRuntimes: [lambda.Runtime.PYTHON_3_7],
license: 'Apache-2.0',
description: 'A layer to enable the PIL library in our Rekognition Lambda',
});
Lambda's job is to pull an image from the s3 bucket, take the image and send it to the rekognition service to do the image detection.
npm i @aws-cdk/aws-lambda @aws-cdk/aws-lambda-event-sources
const rekognitionLambdaFunc = new lambda.Function(this, 'rekognitionFunction', {
code: lambda.Code.fromAsset('rekognitionlambda'),
runtime: lambda.Runtime.PYTHON_3_7,
handler: 'index.handler',
timeout: Duration.seconds(30),
memorySize: 1024,
layers: [layer],
environment: {
"TABLE": imageTable.tableName,
"BUCKET": imageBucket.bucketName,
"RESIZEDBUCKET": resizedBucket.bucketName
}
});
Last part to add is Permissions and Roles
To trigger Lambda when object(image) is created in S3.
rekognitionLambdaFunc.addEventSource(new event_sources.S3EventSource(imageBucket, {events: [s3.EventType.OBJECT_CREATED]}))
Permission to read from s3.
imageBucket.grantRead(rekognitionLambdaFunc);
resizedBucket.grantPut(rekognitionLambdaFunc);
Permission to allow the result of rekognition service from the sent image to be stored in dynamoDB.
imageTable.grantWriteData(rekognitionLambdaFunc);
Permission policy to allow label detection from rekognition across all resources.
rekognitionLambdaFunc.addToRolePolicy(new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
// permission policy to allow label detection from rekognition across all resources
actions: ['rekognition:DetectLabels'],
resources: ['*']
}))
And make sure you've updated your bucket name to the same as the stack. (Would recommend changing prefix to your name, but stack with add a unique suffix ID value in template after synthesising).
To create your template for your stacks run:
cdk synth
A set of files in a folder called "cdk.out" will be produced with assets and your json template file.
You can re-run cdk synth to populate any code updates.
synth will manufacture a stack (our components) down into fully formed json yaml template (CloudFormation template) which will include all the resources we added above.
When you see the generated template it will already be quite large. Already cdk has saved us a bunch of effort!
To deploy the resources that we just created run the below command.
cdk deploy
After seeing the below you just want to hit 'y'
This should take around a few minutes to complete.
Result of "cdk deploy" -> ✅ cdkMainStack
Output example
cdkMainStack.cdkTable = cdkMainStack-ImageLabelsE524135D-104WIEO86Q2JP
cdkMainStack.imageBucket = cdkmainstack-dbrocdkimagebucketb661dc68-uok6ax6q62sh
You can also view your CF stacks in AWS console straight away, pretty nice!
So that's your s3, lambda and dynamo db all created for you.
A quick note on the lambda. For the clients below:
As this lambda is being invoked we don't have to re-instantiate these above clients every time the lambda is run, it can stay 'hot' and this helps with performance.
Test the image upload (now that s3 resources have been created using cdk deploy)
You'll need to use your unique bucket name, e.g.
aws s3 cp testimage.jpg s3://cdkmainstack-dbrocdkimagebucketb661dc68-uok6ax6q62sh
Logs
Check CloudWatch logs for events and for any errors
In log group "/aws/lambda/cdkMainStack……" -> click latest log stream to see the image being processed.
Or you can check your new dynamoDB table in the console.
These labels are from an image of Stargate's Atlantis' wormhole after it was sent through AWS Rekognition
Or you can scan the dynamoDB table, using the output name from the cdk synth. e.g.
aws dynamodb scan --table-name cdkMainStack-ImageLabelsE524135D-104WIEO86Q2JP
And you should see a single output of your image in the terminal
To get rid of all resources and stop any future running costs run:
cdk destroy
(which is also why we added to dynamoDB)
removalPolicy: cdk.RemovalPolicy.DESTROY
However, even with this policy added to S3 and using cdk destroy the formation will have a status "DELETE_FAILED"
Without the policy added you will see that the CloudFormation stack is deleted successfully but the S3 bucket remains. Why?
By default, the Construct that comes from the S3 package, has a default prop called removalPolicy: cdk.RemovalPolicy.RETAIN.
(CloudFormation does not destroy buckets that are not empty).
Option 1: Manually clean the bucket contents before destroying the stack
You can do this from the AWS S3 user interface or through the command line, using the AWS CLI:
Cleanup bucket contents without removing the bucket itself;
aws s3 rm s3://bucket-name --recursive
Then run;
cdk destroy
Then the cdk destroy will proceed without errors. However, this can quickly become a tedious activity if your stacks contain multiple S3 buckets or you use stacks as a temporary resource so some automation would help (option 2).
Option 2: Automatically clear bucket contents and delete the bucket
A 3rd party package called @mobileposse/auto-delete-bucket provides a custom CDK construct that wraps around the standard S3 construct and internally uses the CloudFormation Custom Resources framework, to trigger an automated bucket contents cleanup when a stack destroy is triggered.
Install the package:
npm i @mobileposse/auto-delete-bucket
Use the new CDK construct instead of the standard one:
import { AutoDeleteBucket } from '@mobileposse/auto-delete-bucket'
const bucket = new AutoDeleteBucket(this, 'my-data-bucket')
Summary
Add components / build / deploy
cdk synth
cdk deploy
(If you makes any changes, you can run another 'cdk deploy')
Take note of outputs (example)
cdkMainStack.cdkTable = cdkMainStack-ImageLabels
cdkMainStack.imageBucket = cdkmainstack-dbrocdkimagebucket
Copy test image
aws s3 cp testimage.jpg s3://{bucketName}
Check table has been populated, if not check logs
aws dynamodb scan --table-name {tableName}
Cleanup tasks
aws s3 rm s3://{bucketName} --recursive
cdk destroy
That's all for this session, take care!
Top comments (0)