Mastering ECS Task Scheduling: Effective Strategies to Reduce Costs

Introduction

As we moved our service deployment from Lambda to ECS on Fargate, we aimed to minimize costs outside of the production environment. We implemented several cost-saving measures:

• Configuring vCPU and memory to the minimum viable settings (0.25 vCPU, 0.5 GB)
• Utilizing Fargate Spot for cost efficiency

These settings were the extent of what we could optimize on the ECS side. However, considering that costs accrue per hour based on vCPU and memory usage, we developed a system using Lambda and EventBridge to stop ECS tasks outside of business hours.

I have detailed the Lambda function code and EventBridge Terraform configuration below, so you can easily replicate and utilize them to reduce costs.

AWS Fargate Pricing

Lambda Function

The integration with EventBridge allows for scheduled execution, making Lambda an ideal environment for stopping and starting ECS tasks.

Considering the Single Responsibility Principle, I initially thought about separating the stop and start processes into different functions. However, since the code does not change frequently and to keep management simple, I decided to consolidate them into a single function.

In some cases, tasks need to be stopped or started manually rather than on a schedule, so I included the action parameter to select between scheduled and manual modes.

Below is the complete code, which you can copy and paste by simply changing the ECS cluster and service names specified in clusters_services_scheduled and clusters_services_manual.

import boto3
from botocore.exceptions import ClientError

def update_service(cluster_name, service_name, desired_count):
    client = boto3.client('ecs')
    application_autoscaling_client = boto3.client('application-autoscaling')

    # Control the minimum number of tasks in the AutoScaling policy
    scalable_targets = application_autoscaling_client.describe_scalable_targets(
        ServiceNamespace='ecs',
        ResourceIds=[f'service/{cluster_name}/{service_name}'],
        ScalableDimension='ecs:service:DesiredCount'
    )['ScalableTargets']

    for scalable_target in scalable_targets:
        application_autoscaling_client.register_scalable_target(
            ServiceNamespace='ecs',
            ResourceId=f'service/{cluster_name}/{service_name}',
            ScalableDimension='ecs:service:DesiredCount',
            MinCapacity=0 if desired_count == 0 else 1,
            MaxCapacity=scalable_target['MaxCapacity']
        )

    # Update the service
    service_update_result = client.update_service(
        cluster=cluster_name,
        service=service_name,
        desiredCount=desired_count
    )
    print(service_update_result)

def lambda_handler(event, context):
    try:
        client = boto3.client('ecs')
        elbv2_client = boto3.client('elbv2')

        action = event.get('action')  # 'stop' or 'start'
        environment = event.get('environment')  # 'scheduled' or 'manual'

        clusters_services_scheduled = [
            ('example-cluster', 'example-service-stg')
        ]

        clusters_services_manual = [
            ('example-cluster', 'example-service-dev')
        ]

        if environment == 'scheduled':
            clusters_services = clusters_services_scheduled
        elif environment == 'manual':
            clusters_services = clusters_services_manual
        else:
            raise ValueError("Invalid environment specified")

        desired_count = 0 if action == 'stop' else 1

        for cluster_name, service_name in clusters_services:
            update_service(cluster_name, service_name, desired_count)

        if action == 'start':
            for cluster_name, service_name in clusters_services:
                # Retrieve new task IDs and register them with the target group
                tasks = client.list_tasks(
                    cluster=cluster_name,
                    serviceName=service_name
                )['taskArns']

                task_descriptions = client.describe_tasks(
                    cluster=cluster_name,
                    tasks=tasks
                )['tasks']

                # Retrieve target group information associated with the service
                load_balancers = client.describe_services(
                    cluster=cluster_name,
                    services=[service_name]
                )['services'][0]['loadBalancers']

                for load_balancer in load_balancers:
                    target_group_arn = load_balancer['targetGroupArn']

                    for task in task_descriptions:
                        task_id = task['taskArn'].split('/')[-1]
                        elbv2_client.register_targets(
                            TargetGroupArn=target_group_arn,
                            Targets=[{'Id': task_id}]
                        )

    except ClientError as e:
        print(f"Exception: {e}")
    except ValueError as e:
        print(f"Exception: {e}")

Code Explanation

Task Count Control in Services

The task count of ECS services is updated within the update_service function:

    service_update_result = client.update_service(
        cluster=cluster_name,
        service_name=service_name,
        desiredCount=desired_count
    )

This section uses the update_service method to dynamically update the desiredCount for the specified cluster and service. The desiredCount determines the number of tasks aimed for the service, facilitating the stop or start operations.

Scaling Settings Control

The update_service function adjusts the number of instances in an ECS service and sets the MinCapacity in the Auto Scaling policy to either 0 or 1:

def update_service(cluster_name, service_name, desired_count):
    client = boto3.client('ecs')
    application_autoscaling_client = boto3.client('application-autoscaling')
    ...

Even if the service's task count is set to 0, if MinCapacity remains at 1, the system will attempt to launch tasks repeatedly. Therefore, this setting is crucial if Auto Scaling is configured.

Task Launch and Target Group Registration

If you're integrating ELB (ALB) with ECS, when tasks are launched, you need to retrieve new task IDs and register them with the ALB's target group:

if action == 'start':
    for cluster_name, service_name in clusters_services:
        # Retrieve new task IDs and register them with the target group
        tasks = client.list_tasks(cluster=cluster_name, serviceName=service_name)['taskArns']
        ...

Even during task shutdown, the ECS service remains linked to the target group, allowing you to retrieve the target group in the following section:

load_balancers = client.describe_services(
    cluster=cluster_name,
    services=[service_name]
)['services'][0]['loadBalancers']

for load_balancer in load_balancers:
    target_group_arn = load_balancer['targetGroupArn']

Terraform Code Example

The configuration for deploying the Lambda function via a ZIP file is as follows:

resource "aws_lambda_function" "ecs_task_scheduler" {
  function_name    = "ecs-task-scheduler"
  s3_bucket        = var.s3_bucket_lambda_functions_storage_bucket
  s3_key           = "ecs-task-scheduler.zip"
  handler          = "app.lambda_handler"
  runtime          = "python3.12"
  role             = var.iam_role_ecs_task_scheduler_lambda_exec_role_arn
  timeout          = 300 # 5 minutes
}

Place the Dockerfile and build.sh in the same directory as app.py. Run ./build.sh to create the ecs-task-scheduler.zip. Upload this ZIP file to the designated S3 bucket and execute terraform apply to complete the deployment.

FROM public.ecr.aws/lambda/python:3.12

# Install Python dependencies
COPY requirements.txt /var/task/
RUN pip install -r /var/task/requirements.txt --target /var/task

# Copy the Lambda function code
COPY app.py /var/task/

# Set the working directory
WORKDIR /var/task

# Set the CMD to your handler
CMD ["app.lambda_handler"]

#!/bin/bash

# Build the Docker image
docker build -t ecs-task-scheduler-build.

# Create a container from the image
container_id=$(docker create ecs-task-scheduler-build)

# Copy the contents of the container to a local directory
docker cp $container_id:/var/task ./package

# Clean up
docker rm $container_id

# Zip the contents of the local directory
cd package
zip -r ../ecs-task-scheduler.zip .
cd ..

# Clean up
rm -rf package

Manual Execution Method

To manually execute the Lambda function, use the testing tab in the console. Paste the JSON formatted request into the test event body and press the test button to execute the function.

{
  "action": "start",
  "environment": "manual"
}

EventBridge

Use the Cron expression in EventBridge to schedule the automatic execution of the Lambda function on specific days and times. This feature allows you to build a schedule to stop ECS tasks during weekday nights and all weekend.

Weekday Schedule (22:00 - 5:00 Stop)

• Stop: Weekdays at 22:00 (UTC 13:00)
• Start: Weekdays at 5:00 (UTC 20:00)

Weekend Schedule (All Day Stop)

• Stop: Saturday at 00:00 (UTC 15:00 previous day)
• Start: Monday at 5:00 (UTC 20:00)

Below is an example of Terraform code to set these schedules. Define the EventBridge rules and set the appropriate Lambda function as the target. Note that time settings are in UTC, so adjust according to your region.

Weekday Schedule

# Weekday stop schedule (every day at 22:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekday_stop_tasks_schedule" {
  name                = "ECSWeekdayStopTasksSchedule"
  description         = "Schedule to stop ECS tasks on weekdays at 22:00 JST"
  schedule_expression = "cron(0 13 ? * MON-FRI *)"  # Weekdays at 22:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekday_stop_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekday_stop_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekdayStop"
  arn       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "stop"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekday_stop_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekdayStop"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekday_stop_tasks_schedule.arn
}

# Weekday start schedule (every day at 5:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekday_start_tasks_schedule" {
  name                = "ECSWeekdayStartTasksSchedule"
  description         = "Schedule to start ECS tasks on weekdays at 5:00 JST"
  schedule_expression = "cron(0 20 ? * MON-FRI *)"  # Weekdays at 5:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekday_start_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekday_start_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekdayStart"
  arn       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "start"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekday_start_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekdayStart"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekday_start_tasks_schedule.arn
}

Weekend Schedule

# Saturday stop schedule (at 0:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekend_stop_tasks_schedule" {
  name                = "ECSWeekendStopTasksSchedule"
  description         = "Schedule to stop ECS tasks on Saturday at 00:00 JST"
  schedule_expression = "cron(0 15 ? * SAT *)"  # Saturday at 0:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekend_stop_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekend_stop_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekendStop"
  arn       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "stop"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekend_stop_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekendStop"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekend_stop_tasks_schedule.arn
}

# Monday start schedule (at 5:00 JST)
resource "aws_cloudwatch_event_rule" "ecs_weekend_start_tasks_schedule" {
  name                = "ECSWeekendStartTasksSchedule"
  description         = "Schedule to start ECS tasks on Monday at 05:00 JST"
  schedule_expression = "cron(0 20 ? * MON *)"  # Monday at 5:00 JST
}

resource "aws_cloudwatch_event_target" "ecs_weekend_start_tasks_target" {
  rule      = aws_cloudwatch_event_rule.ecs_weekend_start_tasks_schedule.name
  target_id = "ecsTaskSchedulerWeekendStart"
  ar

n       = var.lambda_function_ecs_task_scheduler_arn

  input = jsonencode({
    action      = "start"
    environment = "scheduled"
  })
}

resource "aws_lambda_permission" "ecs_weekend_start_tasks_allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvokeLambdaWeekendStart"
  action        = "lambda:InvokeFunction"
  function_name = var.lambda_function_ecs_task_scheduler_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.ecs_weekend_start_tasks_schedule.arn
}

IAM Role

Here's an example of Terraform code to define the execution role for the Lambda function scheduling ECS tasks. This role enables the Lambda function to call APIs from ECS and related AWS services.

• ECS Service Management: Update services, list tasks, and fetch detailed information about tasks and services.
• Auto Scaling Management: Register and deregister scalable targets, retrieve information about scalable targets.
• Elastic Load Balancing (ELB) Management: Register and deregister targets, fetch detailed information about target groups and listeners.
• Log Management: Create log groups and streams, post log events.

resource "aws_iam_policy" "ecs_task_scheduler_policy" {
  name        = "ecs-task-scheduler-policy"
  description = "Policy for ECS task scheduler Lambda function"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "ecs:UpdateService",
          "ecs:ListTasks",
          "ecs:DescribeTasks",
          "ecs:DescribeServices"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "application-autoscaling:RegisterScalableTarget",
          "application-autoscaling:DeregisterScalableTarget",
          "application-autoscaling:DescribeScalableTargets"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "elasticloadbalancing:RegisterTargets",
          "elasticloadbalancing:DeregisterTargets",
          "elasticloadbalancing:DescribeTargetGroups",
          "elasticloadbalancing:DescribeListeners",
          "elasticloadbalancing:DescribeRules"
        ],
        Resource = "*"
      },
      {
        Effect   = "Allow",
        Action   = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents"
        ],
        Resource = "arn:aws:logs:*:*:*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_scheduler_policy_attach" {
  role       = aws_iam_role.ecs_task_scheduler_lambda_exec_role.name
  policy_arn = aws_iam_policy.ecs_task_scheduler_policy.arn
}

Conclusion

As we continue to aim for further cost reductions, the next step will be to introduce Compute Savings Plans.

Compute Savings Plans Pricing

Comments

[Kishor Kumbhar]

Great post !!

May i know how its different from sheduling with native option available with ASG scaling.

[Atsushi Suzuki]

Thank you for your appreciation!

The method described in the article differs from ASG scaling scheduling primarily because it targets ECS services directly and uses AWS Lambda for more granular control. While Auto Scaling Groups (ASG) scheduling allows you to define scaling actions based on time schedules, it operates at the EC2 level, impacting all services running on those instances.

In contrast, the approach using Lambda and EventBridge provides specific control over ECS tasks, allowing us to start or stop individual services as needed without affecting others.

[Kishor Kumbhar]

Understood,

So this solution will scale-in the minimum tasks running in service to zero, resulting graceful termination of tasks then eventually ec2 resources, is it ?

If you are running Stateful services yea this makes sense. But for stateless services scaling policies with asg scheduling works same imo.