DEV Community: Angel Pizarro

Planning for deprecation of dependencies

Angel Pizarro — Wed, 10 Apr 2024 19:55:06 +0000

As time marches on, so does technology. Either you adopt to the changes as they approach, or wake up to A Bad Day at work. That's what happened to one of my customers way back in 2013 when they found they could not connect RDP to their fleet of >1000 Windows instances because they were not updating the images, even though the breaking change was communicated for 3 years prior. Restarting the instances with a user data Powershell script to update the instances fixed the issue, but it took a few days of development and testing before services were back online.

Don't have a bad day and take a look at your stack for possible deprecation!

Here are three examples that you may want to pay attention to if you work on AWS:

The Amazon EC2 Instance Metadata Service will be switching to IMDSv2 for new instances this summer. You can read more about that in this blog post. While you can still turn on v1, there are good security reasons to migrate. Read the docs on how to transition from v1 to v2 today!
The AWS Go SDK v1 will enter maintenance mode and be unsupported effective June 31, 2024. Time to update all those Terraform scripts!
While Amazon Linux 2 End-Of-Life was extended to 2025, you should really start migrating to AL2023 as soon as possible. There are lot of fundamental differences that have a high chance of disrupting your applications if you do not test them. Read more about that the differences in the AL2 and AL2023 in the documentation and make plans to migrate within the next 6 months.

These are just a couple of the examples I had conversations about in the last few weeks, there are more. It's worth repeating:

Don't have a bad day and take a look at your stack for possible deprecation!

Why you may want to use AWS Batch on top of ECS or EKS

Angel Pizarro — Wed, 20 Mar 2024 13:45:16 +0000

Since AWS Batch is an overlay on top of other AWS container orchestration services (ECS and EKS), we sometimes get the question about why use it at all.

Here are some reasons you may want to consider AWS Batch for handling batch-style and asynchronous workloads:

The job queue. Having a place that actually holds all of your tasks and handles API communications with ECS is actually a large value add.
Fair share scheduling - in case you have mixed workloads with different priorities or SLA, a fair share job queue will allow you to specify the order of placement of jobs over time. See this blog post for more information.
Array jobs - a single API request for up to 10K jobs using the same definition. As mentioned Step Functions has a Map function, but underneath this would submit a single Batch job or ECS task for each map index, and you may reach API limits. The Batch array job is specifically to handle the throughput of submitting tasks to allow for exponential back off and error handling with ECS runtask.
Smart scaling of EC2 resources - Batch creates an EC2 autoscale group for the instances, but it is not as simple as that. Batch managed scaling will send specific instructions to the ASG about which instances to launch based on the jobs in the queue. It also does some nice scale down as you burn down your queued jobs to pack more jobs on fewer instances at the tail end, so the resources scale down faster.
Job retry - you can set different retry conditions based on the exit code of your job. For example if your job fails due to a runtime error, don't retry since you know it will fail. But if a job fails due to a Spot reclamation event, then retry the job.

The above is not a complete list, but just some highlights. The following are some things about Batch to be aware of if you are thinking of using it for your workloads:

It is tuned for jobs that are minimum 3 to 5 minutes wall-clock runtime. If your individual work items are < 1 minute, you should pack multiple work items into a single job to increase the run time. Example: "process these 10 files in S3".
Sometimes a job at the head of the queue will block other jobs from running. There are a few reasons this may happen, such as an instance type being unavailable. Batch just added blocked job queue CloudWatch Events so you can react to different blocking conditions. See this blog post for more information.
Batch is not designed for realtime or interactive responses - This is related to the job runtime tuning. Batch is a unique batch system in the sense that it has both scheduler and scaling logic that work together. Other job schedulers assume either a static compute resource at the time they make a placement decision, or agents are at the ready to accept work. The implication here is that Batch does a cycle of assessing the job queue, place jobs that can be placed, scale resources for what is in the queue. The challenge here is that you don't want to over-scale. Since Batch has no insight into your jobs or how long they will take, it makes a call about what to bring up that will most cost effectively burn down the queue and then waits to see the result before making another scaling call. That wait period is key for cost optimization but it has the drawback that it is suboptimal for realtime and interactive work. Could you make it work for these use case? Maybe but Batch was not designed for this and there are better AWS services and open source projects that you should turn to first for these requirements.

Again, not a complete list, but these represent some of the common challenges I've seen for new users.

Hope this list helps, and if you have any questions, leave a comment below.

Connecting to the underlying EC2 instance that is running your AWS Batch job

Angel Pizarro — Thu, 18 Jan 2024 15:47:06 +0000

AWS Batch is a great service for submitting asynchronous and background work to. It's a managed service that adds job queue and compute scaling functionality to AWS container orchestration services - ECS and EKS for these types of workloads. AWS Batch is optimized for jobs that are at least a few minutes long, and I've run days-long processes on it.

Sometimes you need to connect to the underlying EC2 instance to debug, or inspect the outputs of running containers, but the underlying container instance ID is not directly available to you using the Batch API. For processes running on ECS I wrote a small Python script that queries for the underlying instance ID of the job using boto3 (and a little regex) and then prints out a CLI command to connect to the instance using a AWS Systems Manager session. The script takes in an AWS Batch job ID as a required parameter.

#!/usr/bin/env python 
import boto3, regex

from argparse import ArgumentParser

# get a job id from the command line
parser = ArgumentParser()
parser.add_argument("job_id", help="The AWS Batch job ID to get the EC2 instance ID on which it ran.")
args = parser.parse_args()

# Get a Boto session 
session = boto3.Session()
# Get a client for AWS Batch 
batch_client = session.client('batch')
# Get a client for AWS ECS
ecs_client = session.client('ecs')

# Describe a batch job
job_description = batch_client.describe_jobs(jobs=[args.job_id])

container_instance_arn = job_description["jobs"][0]["container"]["containerInstanceArn"]

# regex for pulling out the ECS cluster ID and container instance ID from a container instance ARN 
regex_pattern = r"arn:aws:ecs:(?P<region>.*):(?P<account_id>.*):container-instance/(?P<cluster_id>.*)/(?P<container_instance_id>.*)"
match = regex.match(regex_pattern, container_instance_arn)
cluster_id = match.group("cluster_id")
container_instance_id = match.group("container_instance_id")

# Describe a container instance and get the instance ID
container_instance_description = ecs_client.describe_container_instances(cluster=cluster_id, containerInstances=[container_instance_id])

ec2_instance_id = container_instance_description["containerInstances"][0]["ec2InstanceId"]

print("To connect to the EC2 instance use the AWS CLI like so:")
print("aws ec2 connect-to-instance --instance-id " + ec2_instance_id)

If you need to find and connect to the underlying EC2 Instance for an AWS Batch job, I hope this script helps.

Updated Jan 22, 2024 to use a boto3.Session as per advice from this post which I agree with.