DEV Community: okamos

An Essential CloudFormation Template for robust Scheduled Tasks in ECS on EC2

okamos — Mon, 26 Jun 2023 13:58:43 +0000

Introduction

This is a true story.

One day

Me: Okay, the batch program is complete. This batch processing is super important, so it needs to work flawlessly.

Me: If it doesn't run...

Me: No use thinking about it now.

Me: Registering it with ECS's scheduled tasks..., Done!

Me: Even if the program fails, I'm monitoring for program errors and task failures, so it should be fine!

Few months later

Boss: Hey! The batch program seems to be not working!!

Me: Wha...?

Me: I haven't received any error or task failure notifications!

Boss: It's doesn't matter! Fix it!!

Me: Yes, sir! For now, I have manually run the task for today!!

After work

Me: I wonder why it didn't run. Let's take a look at CloudTrail...ECS tasks are launched via the RunTask API, so let's search for the event name RunTask...

Me: There it is. And the reason for the failure is...?

"failures": [
    {
        "arn": "arn:aws:ecs:ap-northeast-1:xxx:container-instance/xxx",
        "reason": "AGENT"
    }
]

Me: The reason is...AGENT?

Motivation

Ever heard of the saying, 'You can't stop the waves, but you can learn to surf'?
Well, in my case, the 'waves' were task failures in an ECS on EC2 environment and 'learning to surf' involved a lot of head-scratching, frantic Googling, and, ultimately, a date with CloudFormation Templates and Step Functions.
This post is a chronicle of my 'surfing' lessons: the struggles I faced and how I eventually tamed the 'waves'.

The Root Cause

In an ECS on EC2 environment, there are three patterns of failure for scheduled tasks:

Failure due to a program: This happens when a task fails due to an error in the program. If there is a stack trace in the standard output of the task indicating a program error, this is the case.
Failure due to task settings: This represents situations like insufficient memory. If there is not enough memory to launch the ECS task and run the program, the task will terminate prematurely.
Failure due to an inability to launch the task: This is the problem this article is addressing. It occurs when the connection with the ECS Agent is lost, or when there isn't enough memory left on the EC2 itself to launch the task. This case is difficult to detect because the task ends without running.

As we focus on the third type of failure in this article, let's delve deeper into its root cause.

On an EC2 running as a container instance, the ECS Agent is constantly active in order to communicate with ECS. However, this Agent is updated once every few hours, and if this timing overlaps with the launch of an ECS task, the operation results in an error. This is not due to a program error or an abnormal termination of the ECS task, but rather because it can't even board the EC2 in the first place.

So, what's the best way to counter this issue?"

The Solution

Firstly, one possible solution is to interpose StepFunctions.
This method is brilliant, as it allows for retries with StepFunctions via RunTask, and the changes are minimal. (While retries are possible with EventBridge Rules, it appears that it won't detect this particular case.)

However, if many tasks are in operation, the cost of migrating all of them can be a bit high.

Therefore, I thought about creating an infrastructure capable of detecting a failure to launch and retrying, and ended up creating the following CloudFormation Template 🎉

iisyos / ecs_rerun_cfn

ecs_rerun_cfn

This repository contains a CloudFormation stack to rerun ECS tasks on EC2 instances if they fail before reaching the EC2 instance. This typically occurs when the ECS Agent running on the EC2 instance is disconnected. You can refer to API failure reasons for a comprehensive list of reasons why the ECS API may fail, particularly the RunTask or StartTask actions.

How it works

The stack creates the following resources:

EventBridge (and rules)
Step Functions
SNS Topic
Lambda Function

And some IAM roles and policies.

When an ECS API failure occurs, EventBridge catches the error and triggers the Step Functions. The Step Functions then notify the SNS Topic of the error, which triggers a Lambda Function to send a message to Slack. If the reason for the failure is AGENT, the task is retried up to three times.

How to use

To create the stack, use the quick-create links…

View on GitHub

(Right-side resources of the architecture diagram will be created. The repository also includes Terraform for creating left-side resources.)

This template detects the failure of ECS task launch using EventBridge and uses Step Functions to retry the task.

Unlike the previously mentioned infrastructure, you don't need to migrate all existing ECS tasks to this template.
It attempts to retry only when the launch of a task fails.

Quick create link is here 💁‍♂️

https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/create/review?templateURL=https://s3.ap-northeast-1.amazonaws.com/cd-template-okamos/ecs_rerun_cfn/main.yml

Template Description

As shown in the architecture diagram, the Template creates the following resources:

EventBridge (and rules)
Step Functions
SNS Topic
Lambda Function
(And associated IAM Roles and Policies.)

EventBridge

Firstly, we need to detect failure events. That's where EventBridge comes in. Here's a rule I created:

{
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.ecs"],
  "detail": {
    "responseElements": {
      "failures": {
        "reason": [{
          "exists": true
        }]
      }
    },
    "requestParameters": {
      "startedBy": [{
        "anything-but": "AWS Step Functions"
      }]
    },
    "eventSource": ["ecs.amazonaws.com"],
    "eventName": ["RunTask"]
  }
}

I'll provide a simple explanation of this rule. 'anything-but' means negation, implying that ECS task events kicked by StepFunctions are excluded to avoid the risk of an infinite loop. In fact, I experienced an infinite loop during testing and had to hurriedly delete the resources...

'exists' is a condition of existence, meaning in this case, it triggers when 'failures' are not an empty array.

For more details, refer to the official document.

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html

Step Functions

SNS collaborates with Lambda and is used for Slack notifications. Check Reason verifies the cause of failure, and if it's AGENT, it attempts a retry.

Conclusion

Batch programs play a crucial role in many applications. When their execution is interrupted, it can lead to major problems. In this article, we proposed a solution for when task execution on Amazon ECS fails due to an AGENT error.

As a solution, we created a CloudFormation template to detect ECS launch failures and retry. We used EventBridge to detect failure events and Step Functions to restart tasks.

Moreover, this method does not require migrating all existing ECS tasks, and it only attempts to retry when a task's launch fails, making it an easy-to-implement external resource.

I hope this article will help you achieve reliable batch programs on Amazon ECS.😉

Shipping Environment variables from AWS to Heroku

okamos — Mon, 08 May 2023 13:09:48 +0000

What I built

I built a GitHub Action that sets Heroku environment variables from the AWS SSM Parameter Store. This Action was created to efficiently synchronize environment variables when using both AWS and Heroku.

Category Submission:

DIY Deployments

App Link

iisyos / ssm-to-heroku-env-action

ssm-to-heroku-env-action

GitHub Action that sets Heroku environment variables from AWS SSM Parameter Store This Action was created to efficiently synchronize when using the same environment variables on AWS and Heroku.

Inputs

Input	Required	Default	Info
path	Yes	N/A	The path to the environment variable in the AWS SSM Parameter Store that you want to set in Heroku
email	Yes	N/A	Heroku email
token	Yes	N/A	Heroku token
app	Yes	N/A	Heroku app name

And you need to set AWS credentials by using like AWS Credentials for GitHub Actions.

Description

First of all, this actions retrieve environment variables from AWS SSM Parameter Store specifying path input. This mean when you set like under CLI command, you can retrieve under '/test/'　recursively if you specified path as '/test/'.

aws ssm put-parameter \
    --name "/test/VALUE1" \
    --value "test1" \
    --type String \
    --region "ap-northeast-3"

Authentication…

View on GitHub

marketplace

Screenshots

N/A

Description

This action retrieves environment variables from the AWS SSM Parameter Store using the specified path input. For example, if you have set a parameter with the following CLI command and specified the path as '/test/', the action will retrieve the parameter stored under '/test/'.

aws ssm put-parameter \
    --name "/test/VALUE1" \
    --value "test1" \
    --type String \
    --region "ap-northeast-3"

Authentication with Heroku is done by writing the email and　token　in the .netrc file, as described in Heroku CLI Authentication. You can get your token from here. Additionally, you need to set your app name in the app　input.

Link to Source Code

repository

Permissive License

MIT

Background (What made you decide to build this particular app? What inspired you?)

Honestly, it was mainly about money. 🥺
Initially, I had the application running on AWS, but the cost of maintaining it was a bit high, so I decided to switch to Heroku. However, there was still a possibility that I would use AWS in the future, so I stopped the resources without deleting them. As the application scales, the number of environment variables will grow, and managing them becomes a challenge. Since they were originally managed in the AWS SSM Parameter Store, migrating them would be cumbersome. Therefore, I decided to create an infrastructure with AWS as the master and Heroku as a reference.

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

I learned how to authenticate Heroku by writing emails and tokens in the .netrc file and the basics of AWS CLI.
I also learned the application of Interface and Class in TypeScript.
It was difficult to implement a generic interface.

#GitHubHack23: StarTweet

okamos — Wed, 03 May 2023 13:05:31 +0000

What I built

StarTweet is a GitHub Action that sends a thank you tweet to users who star your GitHub repository.

Category Submission:

Wacky Wildcards

App Link

repository
marketplace

Screenshots

Description

This action sends a thank you tweet to users who star your GitHub repository. It aims to appreciate and engage with your community on Twitter.

Link to Source Code

https://github.com/iisyos/star_tweet

Permissive License

MIT License

Background (What made you decide to build this particular app? What inspired you?)

I built this GitHub Action to create a more engaging and personal experience for people who support your projects. When users star my repositories, I wanted to show them my appreciation and start a conversation on Twitter.
I hope that this action will encourage communication among engineers around the world.😉

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

I use this template (https://github.com/actions/typescript-action).It' easy to start GitHub Action for TS engineer.
While building this GitHub Action, I learned how to work with the Twitter API and how to effectively use webhooks to create automated workflows.

Additional Resources/Info

Simple Action: YAML Replacement

okamos — Wed, 03 May 2023 02:26:28 +0000

What I built

I build a GitHub Action called "YAML Replacement" that allows you yto replace specific values in a YAML file.

Category Submission:

Wacky Wildcards

App Link

Repository
Marketplace

Screenshots

N/A

Description

When you want to update a file cached in the browser, you change the URL of the file, like this:

<script src="production.min.js?ver=1"></script>

<script src="production.min.js?ver=2"></script>

The tails of url is often managed in YAML file. So when you deploy application, you need update the phrase 1 to 2. This action update the phrase to current date formatted YYYYMMDDhhmm.

The version number at the end of the URL is often managed in a YAML file. So, when you deploy your application, you need to update the version number from 1 to 2. This action updates the version number to the current date formatted as YYYYMMDDhhmm.

Link to Source Code

You can find the source code for the YAML Replacement action on GitHub: https://github.com/iisyos/yaml-replacement-action

Permissive License

This action is released under the MIT License.

Background (What made you decide to build this particular app? What inspired you?)

I decided to build GitHub Action because I often encountered scenarios where I needed to update the version number in a YAML file when deploying applications for updating browser cached files or just maintaining versions. Manually updating the version number was time-consuming and error-prone, so I wanted to automate this process to improve efficiency and accuracy.

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

I built the YAML Replacement action using JavaScript and the GitHub Actions framework. The action takes inputs such as the path to the YAML file, the target key of the value to be replaced, and an optional flag indicating whether the changes should be pushed.

During the execution of the action, it reads the YAML file, identifies the target key, and replaces the corresponding value with the current date formatted as YYYYMMDDhhmm. If the "need-push" flag is set to true, the action commits and pushes the changes to the repository.

Along the way, I gained a deeper understanding of GitHub Actions, because it was my first time building actions😉.

Additional Resources/Info

N/A