Auto-Stop EC2 instances when they finish a task

Sorry serverless folks, but AWS Lambda isn't the answer to every problem!

• Lambda functions run for a maximum of 15 minutes
• They only have 512MB of temporary disk space
• Packages can only be up to 250MB (unzipped)

For situations that don't fit these limitations, there's always Amazon EC2! (Containers with Fargate are another option.)

Here's some tips on how to use Amazon EC2 instances for batch work including running scripts at startup and shutting down when finished.

Running a script at every Start

Most people know about the ability to pass a script via User Data that will executed when an EC2 instance is first booted.

• Running Commands on Your Linux Instance at Launch
• Running Commands on Your Windows Instance at Launch

Here's a bit of history... AWS didn't come up with the idea of passing a script to the instance. User Data was simply there to pass some information that was then accessible to applications running on the instance (eg "Here's the password for the database"). It was Canonical that came up with that concept and created Cloud-Init. They still maintain it today:

Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.

A script passed through User Data is executed the first time an instance is booted. (Or, to be more accurate, once per Instance ID.) This makes it great for installing software and configuring systems, since it only runs once.

But... what if you want to run a script every time the instance starts? Fortunately, Cloud-Init can also run scripts:

• On every boot
• On the next boot only

So, if you're going to use an Amazon EC2 Linux instance to run a batch job on startup, simply install the script in this directory:

/var/lib/cloud/scripts/per-boot/

Each time the instance is booted (started), the script will run. This is a great way to trigger a batch process on the instance. The script could then retrieve information to be processed from a database, Amazon SQS queue or even the instance User Data. (For example, you could pass an S3 filename through User Data and the script could download and process the file. Please note that User Data can only be updated when an instance is in the Stopped state.)

Shutting Down when you're Finished

Once your batch job has completed, what's the easiest way to shutdown the instance?

Yes, you could call the StopInstances() command, passing the instance ID. However, an easier way is to simply issue the operating system shutdown command:

sudo shutdown now -h

(The -h means 'halt', which tells the virtual hardware to turn itself off. If you don't include it, then the operating system turns off but the virtual computer keeps running.)

Shutting down the instance from the operating system takes advantage of the Shutdown Behavior parameter that tells EC2 what to do when the operating system shuts down the computer. It can be set to either Stop or Terminate. The default is Stop, which means the instance will be turned off, but can turn on again afterwards.

A Warning!

If something goes wrong with the batch script, it might shutdown the instance before you can login and debug the situation. Then, if you start the instance again, it will once again fail and shutdown.

I suggest you add a circuit breaker in the code so you can tell the code not to shutdown. For example, the script could check if a particular tag has been added to the instance. If so, skip the shutdown step.

Photo credits: Production cookie in factory | Vic | Flickr

Comments

[Andrew Brown 🇨🇦]

I guess the question is why would you want to use EC2 instances over Fargate.
I suppose learning and configuring docker is a factor.
Another one could cost. EC2 could be more cost-effective than Fargate.

Either way, it's good to explore different ways to do things.

[Robert Moskal]

My problem is that fargate tasks are limited to 30GB of memory, but I need to run a really huge R job.

[jrotenstein]

Update: The method for running scripts on "every boot" appears to have changed with Amazon Linux 2. You might need to find the newer method for starting the script on every boot.

[Devang Hingu]

Good article @aws_john
actually i want to start my instance when i got traffic more than 200 in SQS. is that possbile?

[Denny Chandra]

Thanks for the article, John. I've got a use case at work needs your solution. Do you happened to have the code in production or POC that I can have a look?