A year late, I touched Amazon EventBridge Pipes,
I made this for verification purposes, using Amazon EventBridge Pipes to launch an ECS Task.
I'm a newbie to ECS, so this is the first time for me to make it properly.
Why I made it.
I'm a person who usually uses AWS Lambda to build things,
I usually use AWS Lambda to create various things, but for my work, I had to create a heavy process (batch, yes...) which is a case where containers would be the best choice. I was asked to create a heavy process (batch, yes...).
How do we start the ECS Task?
After finishing the development of the application locally, we started to consider what kind of configuration we should use. And then we started to consider,
The problem was how to start the ECS Task.
The way to do it is,
- Call AWS Lambda from SQS to launch ECS Task
- ECS Task triggered by Step Functions
In terms of processing, we decided to use SQS triggers. And then, we decided to use SQS to AWS Lambda to launch ECS Task,
Then I thought, Call AWS Lambda from SQS and launch ECS Task?
But I thought that it would be a bit odd to call Lambda just for the purpose of launching the task (my favorite AWS service is AWS Lambda!)... I was thinking..,
I found out that I can use Amazon EventBridge Pipes to launch ECS Tasks, so I decided to build it using Amazon EventBridge Pipes (sorry for being a year late).
I also decided to build it with AWS CDK v2.
About Amazon EventBridge Pipes
Learn more about Amazon EventBridge Pipes.
https://aws.amazon.com/eventbridge/pipes/
Created this time
The following is available to the public.
https://github.com/Kenichiro-Wada/cdk-eventbridge-pipes-ecs-tasks
The required items, etc. are listed in the README.
AWS CDK operation is confirmed with Cloud9 as of 2023/12/21.
In building it, we refer to the following site.
https://dev.classmethod.jp/articles/cdk-ecs-ecr/
(Sorry, Japanese Only)
We would like to take this opportunity to thank you!
A cautionary tale if you try to run it.
The sample code, needs to be rewritten in two places, please make the changes after checkout.
About VPC
In this requirement, there is a requirement to use an existing VPC, so the sample also specifies the ID, etc. of the existing VPC and imports and uses the information of a Public subnet. (Public Subnet is not in the requirement, but the sample is set to Public.)
Please change the VPC ID to match your environment.
If you want to create a new VPC together..., please refer to the official documentation.
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.Vpc.html
You can create a NAT Gateway by specifying the name and CIDR, but you need to be careful about this because it will create a NAT Gateway for each AZ.
If you want to launch the container itself on a private subnet, you will need a NAT Gateway (NAT instance) or VPC endpoint.
https://zenn.dev/yoshinori_satoh/articles/ecs-fargate-vpc-endpoint
(Sorry Japanese only)
Notes on exporting Pipes logs to Cloudwatch Logs
I am including this because there was an update in November 2023 to support log output to Cloudwatch Logs, etc.
When I first made it, I got an error, what? Not supported? I threw an issue in a mad rush, but it was nothing, I was using an older version of Typo and aws-cdk-lib, so be careful with Typo. And I'm really sorry.
I've just solved the problem and set it up,
The level is set to Optional in the PipeLogConfigurationProperty, but even if it is not specified, it does not cause an error in the cdk diff, but it causes an error at runtime.
Since it is already specified on the sample code, it should not be an error.
I am waiting for Pipes to support L2 for now, because I hope it will set this area well even if it is not set in L2...
Summary
Since the beginning of 2023, I had been thinking that I had to try it! I finally got to try a service I've been meaning to try since the beginning of 2023.
EventBridge Pipes, it's quite useful. I didn't try the filter and enhancement this time, but I will try it when I get a chance.
I would also like to connect other processes.
As for the containers, this is the first time I've touched them properly except for the ones I inherited,
On the other hand, it is quite fun. On the other hand, there are still many things I don't understand, so I would like to continue my study.
At any rate, based on the verification results, I have finished building a batch (yes, it's a batch) that is necessary for business and am testing it to great acclaim. I am sure it will work.
Special Thanks
Introduction to AWS Container Design and Construction "Authentic" (Japanese book)
AWS CDK stack for building Amazon EventBridge Pipes (SQS to CloudWatch Logs)
Implementing ECS and ECR Container Configuration with CDK (for the second time)(Japanese Only)
Tweet this and that with EventBridge Pipes - I want to "|" with you... Postscript - (Japanese Only)
Top comments (0)