Introduction
Amazon EventBridge Scheduler, introduced by AWS in November 2022, is a powerful serverless scheduler that allows you to create, run, and manage tasks across various AWS services from one central, managed platform. This highly scalable service enables you to schedule millions of tasks, invoking any AWS service as a target, without the need for provisioning or managing underlying infrastructure. In this blog post, we will explore how to use EventBridge Scheduler to automate AWS Batch jobs, ensuring smooth and efficient task execution.
What is Amazon EventBridge Scheduler?
Amazon EventBridge Scheduler is a serverless scheduler introduced by AWS to facilitate the easy creation, execution, and management of tasks across multiple AWS services. With support for over 200 services and more than 6,000 APIs, the scheduler allows you to configure schedules with a minimum granularity of one minute, ensuring precise control over task execution. It also offers at-least-once event delivery to targets, along with the ability to set delivery windows, retries, retention times, and dead letter queues.
How does EventBridge Scheduler work?
EventBridge Scheduler enables you to create one-time or recurring schedules, making it flexible enough to handle various use cases. Whether you need to trigger a task just once or run it periodically, EventBridge Scheduler has you covered. The scheduler requires no event bus, streamlining the process of setting up schedules and reducing complexity. Here are some key features of EventBridge Scheduler:
- Quota on schedules: Up to 1 million schedules per AWS account.
- Event invocation throughput: Capable of supporting throughput in thousands of transactions per second (TPS).
- Targets: Over 270 services and more than 6,000 API actions with AWS SDK targets.
- Time expressions and time-zones: Supports
at(),cron(), andrate()expressions in all time-zones and with Daylight Saving Time (DST) adjustments. - One-time schedules: Allows the creation of single, one-off schedules with precise execution times.
- Time window schedules: Enables the configuration of schedules with a time window for more flexibility.
- Rule quota consumption: No rule quota consumption; soft limit of 1 million schedules.
Scheduling AWS Batch Jobs with EventBridge Scheduler
Now, let's dive into the process of automating AWS Batch jobs using Amazon EventBridge Scheduler. Below are the steps to achieve this:
Creating the Required Policies
To get started, we need to create a policy that grants the necessary permissions for the Scheduler to execute our AWS Batch job. The policy should include permissions for "batch:SubmitJob", "batch:DescribeJobDefinitions", and "batch:DescribeJobQueues".
# CloudFormation code snippet for creating the BatchSubmitPolicy
BatchSubmitPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
Description: Policy to allow Scheduler to execute AWS Batch jobs
ManagedPolicyName: YourBatchSubmitPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: 'Allow'
Action:
- "batch:SubmitJob"
- "batch:DescribeJobDefinitions"
- "batch:DescribeJobQueues"
Resource:
- YourBatchJobDefinitionARN
- YourBatchJobQueueARN
Creating the required role
Next, we create an IAM role that the Scheduler will assume to execute the tasks. This role will have the previously created BatchSubmitPolicy attached.
# CloudFormation code snippet for creating the EventBridgeRole
EventBridgeRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: 'Allow'
Principal:
Service:
- 'scheduler.amazonaws.com'
Action: 'sts:AssumeRole'
Path: "/"
ManagedPolicyArns:
- !Ref BatchSubmitPolicy
Description: EventBridge role for AWS Batch job execution
RoleName: YourEventBridgeRole
Configuring the Schedule
Now we create the EventBridge schedule, specifying the name, description, and the desired ScheduleExpression. In this example, we use a rate expression to trigger the job every 8 hours.
# CloudFormation code snippet for creating the EventSchedule
EventSchedule:
Type: AWS::Scheduler::Schedule
Properties:
Description: 'EventBridge schedule to trigger the AWS Batch job'
FlexibleTimeWindow:
Mode: "OFF"
Name: YourBatchExecSchedule
ScheduleExpression: 'rate(8 hours)'
State: 'ENABLED'
Defining the Target
The TargetArn needs to be defined in a specific format, and in this case, we'll set it to invoke the "batch:SubmitJob" API action of AWS Batch.
# CloudFormation code snippet for defining the Target
Target:
Arn: arn:aws:scheduler:::aws-sdk:batch:submitJob
RoleArn: !GetAtt EventBridgeRole.Arn
Input: !Sub |
{
"JobName": "scheduled-batch-job",
"JobDefinition": "YourBatchJobDefinitionARN",
"JobQueue": "YourBatchJobQueueARN",
"ContainerOverrides": {
"Environment": [
{
"Name": "LoggingLevel",
"Value": "${LoggingLevel}"
},
{
"Name": "Prefix",
"Value": "${Prefix}"
}
]
}
}
Complete CloudFormation Code
Here's the complete CloudFormation template to create the EventBridge schedule for your AWS Batch job.
AWSTemplateFormatVersion: "2010-09-09"
Description: CloudFormation template to create an EventBridge schedule for AWS Batch jobs
Parameters:
LoggingLevel:
Type: String
AllowedValues: [NOTSET, DEBUG, INFO, WARNING, ERROR, CRITICAL]
Default: "INFO"
Prefix:
Type: String
Default: "prefix-value"
Resources:
BatchSubmitPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
Description: Policy to allow Scheduler to execute AWS Batch jobs
ManagedPolicyName: YourBatchSubmitPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: 'Allow'
Action:
- "batch:SubmitJob"
- "batch:DescribeJobDefinitions"
- "batch:DescribeJobQueues"
Resource:
- YourBatchJobDefinitionARN
- YourBatchJobQueueARN
EventBridgeRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: 'Allow'
Principal:
Service:
- 'scheduler.amazonaws.com'
Action: 'sts:AssumeRole'
Path: "/"
ManagedPolicyArns:
- !Ref BatchSubmitPolicy
Description: EventBridge role for AWS Batch job execution
RoleName: YourEventBridgeRole
EventSchedule:
Type: AWS::Scheduler::Schedule
Properties:
Description: 'EventBridge schedule to trigger the AWS Batch job'
FlexibleTimeWindow:
Mode: "OFF"
Name: YourBatchExecSchedule
ScheduleExpression: 'rate(8 hours)'
State: 'ENABLED'
Target:
Arn: arn:aws:scheduler:::aws-sdk:batch:submitJob
RoleArn: !GetAtt EventBridgeRole.Arn
Input: !Sub |
{
"JobName": "scheduled-batch-job",
"JobDefinition": "YourBatchJobDefinitionARN",
"JobQueue": "YourBatchJobQueueARN",
"ContainerOverrides": {
"Environment": [
{
"Name": "LoggingLevel",
"Value": "${LoggingLevel}"
},
{
"Name": "Prefix",
"Value": "${Prefix}"
}
]
}
}
Conclusion
Amazon EventBridge Scheduler offers a powerful and flexible solution for automating tasks across various AWS services. In this blog post, we explored how to use EventBridge Scheduler to schedule and trigger AWS Batch jobs. By following the steps outlined in this guide, you can efficiently manage your tasks and improve operational efficiency.
Top comments (1)
How to use !Ref arn value in
"JobName": "scheduled-batch-job",
"JobDefinition": "YourBatchJobDefinitionARN",
"JobQueue": "YourBatchJobQueueARN",