DEV Community

Cover image for An Essential CloudFormation Template for robust Scheduled Tasks in ECS on EC2
okamos
okamos

Posted on

An Essential CloudFormation Template for robust Scheduled Tasks in ECS on EC2

Introduction

This is a true story.

One day

Me: Okay, the batch program is complete. This batch processing is super important, so it needs to work flawlessly.

Me: If it doesn't run...

Me: No use thinking about it now.

Me: Registering it with ECS's scheduled tasks..., Done!

Me: Even if the program fails, I'm monitoring for program errors and task failures, so it should be fine!

Few months later

Boss: Hey! The batch program seems to be not working!!

Image description

Me: Wha...?

Image description

Me: I haven't received any error or task failure notifications!

Boss: It's doesn't matter! Fix it!!

Me: Yes, sir! For now, I have manually run the task for today!!

After work

Me: I wonder why it didn't run. Let's take a look at CloudTrail...ECS tasks are launched via the RunTask API, so let's search for the event name RunTask...

Me: There it is. And the reason for the failure is...?

"failures": [
    {
        "arn": "arn:aws:ecs:ap-northeast-1:xxx:container-instance/xxx",
        "reason": "AGENT"
    }
]
Enter fullscreen mode Exit fullscreen mode

Me: The reason is...AGENT?

Motivation

Ever heard of the saying, 'You can't stop the waves, but you can learn to surf'?
Well, in my case, the 'waves' were task failures in an ECS on EC2 environment and 'learning to surf' involved a lot of head-scratching, frantic Googling, and, ultimately, a date with CloudFormation Templates and Step Functions.
This post is a chronicle of my 'surfing' lessons: the struggles I faced and how I eventually tamed the 'waves'.

The Root Cause

In an ECS on EC2 environment, there are three patterns of failure for scheduled tasks:

  1. Failure due to a program: This happens when a task fails due to an error in the program. If there is a stack trace in the standard output of the task indicating a program error, this is the case.

  2. Failure due to task settings: This represents situations like insufficient memory. If there is not enough memory to launch the ECS task and run the program, the task will terminate prematurely.

  3. Failure due to an inability to launch the task: This is the problem this article is addressing. It occurs when the connection with the ECS Agent is lost, or when there isn't enough memory left on the EC2 itself to launch the task. This case is difficult to detect because the task ends without running.

As we focus on the third type of failure in this article, let's delve deeper into its root cause.

On an EC2 running as a container instance, the ECS Agent is constantly active in order to communicate with ECS. However, this Agent is updated once every few hours, and if this timing overlaps with the launch of an ECS task, the operation results in an error. This is not due to a program error or an abnormal termination of the ECS task, but rather because it can't even board the EC2 in the first place.

So, what's the best way to counter this issue?"

The Solution

Firstly, one possible solution is to interpose StepFunctions.
This method is brilliant, as it allows for retries with StepFunctions via RunTask, and the changes are minimal. (While retries are possible with EventBridge Rules, it appears that it won't detect this particular case.)

However, if many tasks are in operation, the cost of migrating all of them can be a bit high.

Therefore, I thought about creating an infrastructure capable of detecting a failure to launch and retrying, and ended up creating the following CloudFormation Template πŸŽ‰

ecs_rerun_cfn

This repository contains a CloudFormation stack to rerun ECS tasks on EC2 instances if they fail before reaching the EC2 instance. This typically occurs when the ECS Agent running on the EC2 instance is disconnected. You can refer to API failure reasons for a comprehensive list of reasons why the ECS API may fail, particularly the RunTask or StartTask actions.

How it works

The stack creates the following resources:

  1. EventBridge (and rules)
  2. Step Functions
  3. SNS Topic
  4. Lambda Function

And some IAM roles and policies.

When an ECS API failure occurs, EventBridge catches the error and triggers the Step Functions. The Step Functions then notify the SNS Topic of the error, which triggers a Lambda Function to send a message to Slack. If the reason for the failure is AGENT, the task is retried up to three times.

How to use

To create the stack, use the quick-create links…

Image description

(Right-side resources of the architecture diagram will be created. The repository also includes Terraform for creating left-side resources.)

This template detects the failure of ECS task launch using EventBridge and uses Step Functions to retry the task.

Unlike the previously mentioned infrastructure, you don't need to migrate all existing ECS tasks to this template.
It attempts to retry only when the launch of a task fails.

Quick create link is here πŸ’β€β™‚οΈ

https://console.aws.amazon.com/cloudformation/home?region=ap-northeast-1#/stacks/create/review?templateURL=https://s3.ap-northeast-1.amazonaws.com/cd-template-okamos/ecs_rerun_cfn/main.yml

Template Description

As shown in the architecture diagram, the Template creates the following resources:

  • EventBridge (and rules)
  • Step Functions
  • SNS Topic
  • Lambda Function
  • (And associated IAM Roles and Policies.)

EventBridge

Firstly, we need to detect failure events. That's where EventBridge comes in. Here's a rule I created:

{
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.ecs"],
  "detail": {
    "responseElements": {
      "failures": {
        "reason": [{
          "exists": true
        }]
      }
    },
    "requestParameters": {
      "startedBy": [{
        "anything-but": "AWS Step Functions"
      }]
    },
    "eventSource": ["ecs.amazonaws.com"],
    "eventName": ["RunTask"]
  }
}

Enter fullscreen mode Exit fullscreen mode

I'll provide a simple explanation of this rule. 'anything-but' means negation, implying that ECS task events kicked by StepFunctions are excluded to avoid the risk of an infinite loop. In fact, I experienced an infinite loop during testing and had to hurriedly delete the resources...

'exists' is a condition of existence, meaning in this case, it triggers when 'failures' are not an empty array.

For more details, refer to the official document.

https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html

Step Functions

Image description

SNS collaborates with Lambda and is used for Slack notifications. Check Reason verifies the cause of failure, and if it's AGENT, it attempts a retry.

Conclusion

Batch programs play a crucial role in many applications. When their execution is interrupted, it can lead to major problems. In this article, we proposed a solution for when task execution on Amazon ECS fails due to an AGENT error.

As a solution, we created a CloudFormation template to detect ECS launch failures and retry. We used EventBridge to detect failure events and Step Functions to restart tasks.

Moreover, this method does not require migrating all existing ECS tasks, and it only attempts to retry when a task's launch fails, making it an easy-to-implement external resource.

I hope this article will help you achieve reliable batch programs on Amazon ECS.πŸ˜‰

Top comments (0)