DEV Community: John Rotenstein

Simple EC2 Stopinator in Lambda

John Rotenstein — Mon, 25 May 2020 09:46:23 +0000

A Stopinator is the general term applied to a utility that can start/stop Amazon EC2 instances on a particular schedule or in particular circumstances.

There are quite a few Stopinators available, such as the Instance Scheduler from AWS. However these Stopinators tend to be quite complex, involving CloudFormation templates and databases.

Therefore, I set myself a challenge to create a Simple Lambda Stopinator to help people in situations such as this StackOverflow question asking how to send a notification when an EC2 instance is still running and then eventually terminate it after a given period. How simple? Just paste some code into a Lambda function and you're done!

I have, in fact, written two stopinators:

Type 1: Run this Lambda function once per night to stop/terminate tagged Amazon EC2 instances
Type 2: Run this Lambda function throughout the day to notify/stop/terminate instances after a given period of time

You can find them in: GitHub: Simple Lambda Stopinators

(I hope to also write another Stopinator with start/stop scheduling, but that gets a bit more complicated with weekdays/weekends and timezones.)

Here's an explanation of the Stopinators and how to use them.

Type 1 Stopinator: Stop/Terminate instances at night

I originally wrote this Stopinator when I was delivering AWS training courses. I would launch Amazon EC2 instances for demonstrations during the day and would often forget to terminate them. So, I wanted a script that could identify instances tagged as 'temporary' and terminate them at night, so things were clean for teaching the next day.

However, I didn't always remember to tag the temporary instances, so I also added functionality that would stop unidentified instances at night rather than terminating them (just in case I still needed them). This then also required the ability to mark instances that the Stopinator should 'ignore' because I wanted them to keep running.

This is how the Type 1 Stopinator works:

The Lambda function should be scheduled to run each night (eg 7pm) using Amazon CloudWatch Events
It looks at every running instance in every region (or just the regions you specify)
If the instance has an Auto-Stop = Terminate tag, it terminates the instance
If the instance has an Auto-Stop = Ignore tag, it does nothing
Otherwise, it stops the instance

That's it — simple, eh‽ (Yes, that's an interrobang.)

To install it:

Create an IAM Role and add an in-line policy with permissions from the Stopinator IAM Role
Create a new AWS Lambda Python function that uses the IAM Role
Paste in the Type 1 Stopinator code
Go to Amazon CloudWatch, click Events and add a Scedule with a cron expression that triggers the Lambda function each night. Be careful — AWS uses the UTC timezone, so you'll need to adjust for your particular timezone.

By default, it will automatically stop any untagged instances. If you want to terminate instances instead of stopping them, add an Auto-Stop = Terminate tag (that is, Key = Auto-Stop, Value = Terminate). If there are instances you don't want it to touch, use Auto-Stop = Ignore.

You can also edit this section to control which regions are checked:

# Provide regions here (eg ['us-west-2', 'ap-southeast-2'], or use [] for all regions
regions_to_check = []

That's it. So simple! Enjoy.

Type 2 Stopinator: Stop/Terminate/Notify instances after a given duration

This Stopinator was inspired by a StackOverflow question that was asking for a way to send notifications about a running instance, and then eventually turn it off.

This AWS Lambda function needs to be scheduled to run at regular intervals, so that it continually checks whether instances have exceeded their duration. The more often it runs, the closer it will match the desired running duration.

This is how the Type 2 Stopinator works:

The Lambda function should be scheduled to run throughout the day (eg every 5 minutes) using Amazon CloudWatch Events
It looks at every running instance in every region (or just the regions you specify) and checks for one of these tags (in this order):
- Terminate-After
- Stop-After
- Notify-After (or any tag that starts with Notify-After, such as Notify-After1, Notify-After2, etc)
It checks the duration supplied in the tag value (eg 30m, 1.5h, 24h)
If the instance has been running longer than the given duration, it will terminate/stop the instance or send a notification to an Amazon SNS topic.

To install it:

Create an IAM Role and add an in-line policy with permissions from the Stopinator IAM Role
Create a new AWS Lambda Python function that uses the IAM Role
Paste in the Type 2 Stopinator code
Go to Amazon CloudWatch, click Events and add a Schedule at a "Fixed rate of 5 minutes" (or whatever frequency you wish) that triggers the Lambda function
Optional: Create an Amazon SNS topic and insert the ARN in this line at the top of the Lambda function:

# To send a notification, insert SNS Topic ARN here
SNS_TOPIC_ARN = ''

Then, subscribe to the Amazon SNS topic to receive notifications

To remember whether a notification has been sent, the Stopinator will remove the Notify-After tag once it has been used. This allows multiple notifications to be tracked by supplying multiple tags that start with Notify-After in the tag key.

You can also edit this section to control which regions are checked:

# Provide regions here (eg ['us-west-2', 'ap-southeast-2'], or use [] for all regions
regions_to_check = []

If you find any problems with the Stopinators, feel free to add an issue to the GitHub repository.

If they have been useful for you, feel free to add a comment below. Enjoy!

S3 Pre-signed URLs for Requester Pays buckets

John Rotenstein — Tue, 14 Apr 2020 04:37:38 +0000

Ever heard of a Requester Pays Amazon S3 bucket? Great!

Ever heard of an Amazon Pre-signed URL? Great!

Question: Can you combine them together?

If you're interested to know, read on...

Requester Pays buckets

First, let's cover some terminology. Imagine you are NASA and you have some very detailed scientific pictures of the Earth. NASA would like to make these pictures available to everybody (since it is financed by the government) but bandwidth is costly. If lots of people downloaded the images, the Data Transfer costs could be eye-watering.

Enter Requester Pays buckets, which allow you to avoid paying Data Transfer charges by transferring that cost to the person requesting the photos. Such buckets cannot provide 'anonymous access' because AWS needs to know who is requesting the objects (so they can be charged for the Data Transfer component). Thus, it's not possible to simply put the URL of a photo in a browser to access it (eg http://my-bucket/photo.jpg). Instead, the request must come from an IAM Role or IAM User that the related AWS account can be appropriately charged.

Pre-signed URLs

Second, we have the concept of a pre-signed URL. The best way to explain this is by way of an example.

Imagine that you have a photo-sharing website using Amazon S3 and you want to keep users' photos private. A user should be able to login and view their own pictures, but the pictures should not be publicly accessible. In addition, you want the ability for users to share selected photos with other users.

In such a scenario, it would not be easy to create an IAM policy that grants access to a user's own photos and also those that have been shared with them, since there could be hundreds of shared photos. Instead, the ability to access photos in S3 would work like this:

A user authenticates to the application
A user requests a photo, or they navigate to a page that displays photos on the web page (via an <img> tag)
The application then checks whether they are authorized to view the image.
If they are permitted access, the application generates a pre-signed URL that grants time-limited access to the object in Amazon S3
The user's browser then uses that URL to request access to the object and Amazon S3 grants access to the object

A pre-signed URL looks like this:

https://my-bucket.s3.ap-southeast-2.amazonaws.com/foo.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXXXX%2F20200414%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20200414T034309Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=e89331d191...

The pre-signed URL contains:

A URL to the actual object in Amazon S3
The time that the URL expires
The authorized user's Access Key (which is okay to be public)
A hash signature (based on the authorized user's Secret Access Key)

When Amazon S3 receives such a URL, it recalculates the signature and checks that the time has not expired. If everything seems okay, it grants access to the object in Amazon S3. If anything in the URL has been changed or falsified (such as the expiry time), the request will fail.

A pre-signed URL can be created in a number of ways.

Using the AWS CLI:

aws s3 presign s3://my-bucket.s3.amazonaws.com/foo.jpg

Or, using the boto3 AWS SDK in Python:

s3_client = boto3.client('s3')

response = s3_client.generate_presigned_url(
    'get_object',
    Params={
        'Bucket': bucket_name,
        'Key': object_name
    },
    ExpiresIn=expiration
)

Combining Request Pays + Pre-Signed URLs

Here's the fun bit.

A question on StackOverflow had a scenario where:

Account A is providing a requester-pays bucket (Bucket-A)
An application in Account B wants to allow users to view/access an object in Bucket-A
These users do not have their own AWS account, so the application will need to generate a pre-signed URL (which will send the Data Transfer costs to Account B)

However, they were getting an AccessDenied error.

The reason for this error turned out to be that requests going to a Requester-Pays bucket needs an acknowledgement that the requester is willing to pay for the Data Transfer costs (otherwise they might access a URL without knowing it would cost them more).

To do this, a URL to a requester-pays bucket requires an additional header in the request:

http://...&x-amz-request-payer=requester

However, adding this to a pre-signed URL results in a SignatureDoesNotMatch error because the request URL no longer matches the signature that was originally generated.

The solution, it turned out, was to specify this additional parameter within the signing request:

s3_client = boto3.client('s3')

response = s3_client.generate_presigned_url(
    'get_object',
    Params={
        'Bucket': bucket_name,
        'Key': object_name,
        'RequestPayer': 'requester'  // Additional parameter
    },
    ExpiresIn=expiration
)

Resulting flow

The final flow for this solution is:

User wishes to access object in Requester Pays bucket
Application generates a pre-signed URL (since anonymous access is not permitted), but also specifies the additional parameter to accept charges
The pre-signed URL is sent to the user's web browser
The web browser requests the object from Amazon S3
Amazon S3 verifies the signature and expiry time on the pre-signed URL
Amazon S3 then provides the contents of the object and charges the Data Transfer portion to the AWS Account that created the pre-signed URL

It's not Rocket Science, but does help the Rocket Scientists.

Auto-Stop EC2 instances when they finish a task

John Rotenstein — Fri, 14 Feb 2020 06:26:28 +0000

Sorry serverless folks, but AWS Lambda isn't the answer to every problem!

Lambda functions run for a maximum of 15 minutes
They only have 512MB of temporary disk space
Packages can only be up to 250MB (unzipped)

For situations that don't fit these limitations, there's always Amazon EC2! (Containers with Fargate are another option.)

Here's some tips on how to use Amazon EC2 instances for batch work including running scripts at startup and shutting down when finished.

Running a script at every Start

Most people know about the ability to pass a script via User Data that will executed when an EC2 instance is first booted.

Here's a bit of history... AWS didn't come up with the idea of passing a script to the instance. User Data was simply there to pass some information that was then accessible to applications running on the instance (eg "Here's the password for the database"). It was Canonical that came up with that concept and created Cloud-Init. They still maintain it today:

Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal installations.

A script passed through User Data is executed the first time an instance is booted. (Or, to be more accurate, once per Instance ID.) This makes it great for installing software and configuring systems, since it only runs once.

But... what if you want to run a script every time the instance starts? Fortunately, Cloud-Init can also run scripts:

On every boot
On the next boot only

So, if you're going to use an Amazon EC2 Linux instance to run a batch job on startup, simply install the script in this directory:

/var/lib/cloud/scripts/per-boot/

Each time the instance is booted (started), the script will run. This is a great way to trigger a batch process on the instance. The script could then retrieve information to be processed from a database, Amazon SQS queue or even the instance User Data. (For example, you could pass an S3 filename through User Data and the script could download and process the file. Please note that User Data can only be updated when an instance is in the Stopped state.)

Shutting Down when you're Finished

Once your batch job has completed, what's the easiest way to shutdown the instance?

Yes, you could call the StopInstances() command, passing the instance ID. However, an easier way is to simply issue the operating system shutdown command:

sudo shutdown now -h

(The -h means 'halt', which tells the virtual hardware to turn itself off. If you don't include it, then the operating system turns off but the virtual computer keeps running.)

Shutting down the instance from the operating system takes advantage of the Shutdown Behavior parameter that tells EC2 what to do when the operating system shuts down the computer. It can be set to either Stop or Terminate. The default is Stop, which means the instance will be turned off, but can turn on again afterwards.

A Warning!

If something goes wrong with the batch script, it might shutdown the instance before you can login and debug the situation. Then, if you start the instance again, it will once again fail and shutdown.

I suggest you add a circuit breaker in the code so you can tell the code not to shutdown. For example, the script could check if a particular tag has been added to the instance. If so, skip the shutdown step.

Photo credits: Production cookie in factory | Vic | Flickr

IAM Policy to list an S3 bucket, except for the top-level (root) of the bucket

John Rotenstein — Wed, 12 Feb 2020 23:37:34 +0000

I saw a question on StackOverflow (Allow S3 bucket top-level listing to specific users only) that posed an interesting challenge:

Allow a user to list the contents of an Amazon S3 bucket
But... Don't let them list the top-level of the bucket

I'm always up for a challenge!

IAM Policy Variables

The question reminded me a bit of IAM Policy Elements: Variables and Tags, which allows the construction of an IAM Policy that inserts a username in appropriate spots:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": ["s3:ListBucket"],
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::mybucket"],
      "Condition": {"StringLike": {"s3:prefix": ["${aws:username}/*"]}}
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": ["arn:aws:s3:::mybucket/${aws:username}/*"]
    }
  ]
}

This policy says:

Let the user list/get/put objects
But only in a folder that matches my IAM User username

That's close, but sort of opposite of "list everything except the top-level".

Playing with the ARN

Next, I tried playing with the ARN to see whether I could grant permission for any subfolder within a bucket:

arn:aws:s3:::BUCKET-NAME/*/*

Granting permission for that ARN is saying "Only for a path within the bucket that contains a slash". Thus, it should work for sub-folders but not the top-level.

Unfortunately, the s3:ListBucket permission applies to the bucket itself rather than the contents of the bucket. When granting s3:ListBucket, you must provide the ARN of the bucket without using /*.

Playing with the Prefix

This led me to the idea using the Prefix to specify which folders can be used. The excellent Grant IAM User Access to a Folder in an Amazon S3 Bucket article from AWS Support shows some examples of how to use prefixes, such as:

{
     "Action": ["s3:ListBucket"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::BUCKET-NAME"],
     "Condition":{"StringEquals":{"s3:prefix":["","media"]}}
}

This permits listing of the bucket, but only the top-level folder, or the media folder.

This is getting closer!

I then tried applying negative logic with StringNotEquals, stating that listing was permitted if the prefix was not empty:

{
     "Action": ["s3:ListBucket"],
     "Effect": "Allow",
     "Resource": ["arn:aws:s3:::BUCKET-NAME"],
     "Condition":{"StringNotEquals":{"s3:prefix":""}}
}

This correctly denied my listing the root directory, but permitted listing a sub-folder. However, it also allowed me to do this:

aws s3 ls s3://BUCKET-NAME/a

This command would list all objects that started with a. This was permitted because the prefix was not empty. Therefore, the solution is invalidated because people could try every letter/number as a prefix to view the contents of the top-level of the bucket.

The Solution

More playing led me to this solution:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKET-NAME",
            "Condition": {"StringLike": {"s3:prefix": ["*/*"]}}
        }
    ]
}

This says that listing the bucket is permitted if the Prefix contains a slash. This prevents listing of the top-level of the bucket and allows any sub-folder to be listed.

It also requires that at least one slash exists in the path, so this will work:

aws s3 ls s3://BUCKET-NAME/folder/

However, this will not work since it does not contain a slash:

aws s3 ls s3://BUCKET-NAME/folder

You'll also find my solution on StackOverflow: Allow S3 bucket top-level listing to specific users only

Amazon Route 53: How to automatically update IP addresses without using Elastic IPs

John Rotenstein — Thu, 06 Feb 2020 03:56:09 +0000

Here's a handy way to automatically update an A-Record in Amazon Route 53 whenever an EC2 instance changes IP address.

Scenario: You have a domain name in Amazon Route 53 pointing to an Amazon EC2 instance. However, if the instance is stopped and started, its public IP address changes. This breaks the A-Record since it is pointing to the wrong IP address.

"Wait!" you might say, "Why not simply use an Elastic IP address?"

(An Elastic IP address is a static IP address that can be assigned to resources in an Amazon VPC. The IP address stays the same until you return the Elastic IP address to AWS.)

Yes, using an Elastic IP address on the instance will prevent the A-Record from breaking. However, AWS gives a default limit of 5 Elastic IP addresses per region. You can request a limit increase, but what if you need lots of them?

As an example, a service might provide a separate Amazon EC2 instance for each customer, to ensure secure separation of data. Each customer is given a domain name to access their server (acme.example.com). The required number of Elastic IP addresses could be huge.

Fortunately, there is a fairly simple way to achieve the objective without needing any Elastic IP addresses.

Architecture

The architecture is quite simple:

When the EC2 instance starts, it should get its new public IP address and update its own record in Route 53
The DNS name to update is stored in a Tag on the EC2 instance
The script should execute every time the EC2 instance starts (that is, every time it starts, not just the first boot)

Implementation

First, there should be a Record Set in Amazon Route 53 that defines the existing domain name.

It is currently pointing to the public IP address of an EC2 instance.

Next, add some tags to the EC2 instance that will be used by the script:

DNS Name: The DNS Name to associate with the instance
Hosted Zone ID: Uniquely identifies the Zone record in Route 53 that needs to be updated (get it from your Route 53 Hosted Zone record)

Whenever the EC2 instance starts, it will run a script that will:

Grab the information from the above tags
Retrieve the instance's current public IP address
Update the Route 53 record set with the new IP address

#!/bin/bash
# Extract information about the Instance
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id/)
AZ=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone/)
MY_IP=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4/)

# Extract tags associated with instance
ZONE_TAG=$(aws ec2 describe-tags --region ${AZ::-1} --filters "Name=resource-id,Values=${INSTANCE_ID}" --query 'Tags[?Key==`AUTO_DNS_ZONE`].Value' --output text)
NAME_TAG=$(aws ec2 describe-tags --region ${AZ::-1} --filters "Name=resource-id,Values=${INSTANCE_ID}" --query 'Tags[?Key==`AUTO_DNS_NAME`].Value' --output text)

# Update Route 53 Record Set based on the Name tag to the current Public IP address of the Instance
aws route53 change-resource-record-sets --hosted-zone-id $ZONE_TAG --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"'$NAME_TAG'","Type":"A","TTL":300,"ResourceRecords":[{"Value":"'$MY_IP'"}]}}]}'

To execute the script automatically each time the instance starts (as opposed to User Data scripts that only run on the first boot), put the above script in this directory:

/var/lib/cloud/scripts/per-boot/

Finally, the EC2 instance will need an IAM Role assigned that has permission to run the above commands:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeTags",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/HOSTED-ZONE-ID"
        }
    ]
}

How to test

To test the script, simply Stop the instance then Start it again.

This will result in a new public IP address being assigned to the instance. The script will call Amazon Route 53 to update the record set. This might take a minute to update.

Then, return to Route 53 and look at the IP address assigned to the A-Record. It should be updated with the new IP address.