AWS Cost Optimization : Automatic Detect Your Unused EBS and Delete It

What is cost optimization in AWS?

Its primary objective is to achieve the lowest possible cost for the system or workload while operating in the AWS environment. You should try to minimise expenditures while taking into consideration the needs of your account. However, you shouldn't do this at the expense of performance, security, or dependability.

It is crucial to fully grasp the value of AWS, as well as measure and efficiently manage your AWS consumption and expenses, as you transfer workloads to AWS and grow your use of different AWS services. This is especially important when you increase your use of AWS services.

Cost Optimization Use-Case

A lack of awareness of the EBS volume lifetime results in additional expenditures for underused and neglected resources. Unexpected costs on an AWS account may result from Elastic Block Storage (EBS) volumes that aren't associated with an EC2 instance or used. Some EBS volumes may continue to exist after an EC2 instance is shut down. You are paying for EBS volumes in AWS accounts even though they are not associated.  Following the below steps will allow you to save cloud charges and avoid wasted resources by removing an EBS volume that was accidentally left unattached.

Steps

Prerequisite

EBS Volume
We purposely create another 2 new AWS EBS that not attach to any EC2 instance. We use Boto3 ; which Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python which is maintained and published by Amazon Web Services. Boto3 allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2 as well as provision AWS Services from Boto3. You need to configure your local environment with your AWS Access Key ID and Access Secret Key by execute below command:

aws configure

Once you have configured your AWS account and installed the Boto3 SDK, then you may execute the Pyhton code below to create a new EBS volume for our use-case.

### Author:Najib Radzuan
### CreatedDate:23 Dec 2022
### Purpose: Create New EBS volumes
### Requirements: Boto3(EC2)

import boto3
ec2 = boto3.client('ec2')
response = ec2.create_volume(
    AvailabilityZone='ap-southeast-1a',
    Size=20,
    VolumeType='gp2',
    TagSpecifications=[
        {
            'ResourceType': 'volume',
            'Tags': [
                {
                    'Key': 'Name',
                    'Value': 'Unused_Vol1'
                },
            ]
        },
    ],
)

We had created 2 new EBS volumes and it not attach to any EC2 instance as shown below.

AWS IAM Role

Step 1
We need to create a new AWS Simple Notification Service (SNS) for our use-case notification whenever the AWS Lambda function detected unused EBS and deleted it. Enter the following input for our AWS SNS Topic:

• Type: Standard
• Name: Notify-Unused-EBS-Volume
• Display: Notify-Unused-EBS-Volume

Let the rest of the configuration to be default values.

Next, we to set/create our subscription of the newly created SNS Topic.

• Topic ARN: [Newly created AWS SNS Topic ARN]
• Protocol: Email
• Endpoint: [your subscriber email]

Let the rest to be default values.

Then, we need confirmation from the subscriber we set above via email below:

Once the subscriber clicked the "Confirmation subscription", they we redierected below page that said they are subscribed to the AWS SNS Topic:

Finally, we have new subscriber to our AWS SNS Topic:

Don't forget to copy and paste the AWS SNS Topic ARN somewhere, since we going to use it in AWS Lambda Function code later for SNS notification.

Step 2

Create a new AWS Lamda Function and you can enter below configuration for our new AWS Lambda Funtion:

• Author from scratch
• Function name: do4m-unused-volume
• Runtime : Pyhton 3.9
• Architecture: x86_64

Left the rest configuration to be default values.

Copy and paste below code into your Lambda Function code. There is 2 parts in this code:

1) Boto3

• We use Boto3 EC2 module to find all unsed EBS volumnes in the region we set in the code.

• We put the detected unused EBS ID in "Attachement" for SNS notification usage.

• Once we get all the unused EBS IDs, then we delete EBS volume that had mark with “Available” status.

2) AWS SNS

• We get from "Attachment" unused EBS IDs and we send out the SNS email notification to our subscriber(s) that we set in the previous step.

### Author:Najib Radzuan
### CreatedDate:23 Dec 2022
### Purpose: Detect all unused EBS volumes under selected region
### Requirements: Boto3(EC2),SNS Arn

import boto3
ec2 = boto3.client('ec2')
sns_client = boto3.client('sns')
volumes = ec2.describe_volumes()
ec2 = boto3.resource('ec2',region_name='ap-southeast-1')

def lambda_handler(event, context):
    #Get All Unused Volume
    unused_volumes = []
    for vol in volumes['Volumes']:
        if len(vol['Attachments']) == 0:
            vol1 = ("-----Unused Volume ID = {}------".format(vol['VolumeId']))
            unused_volumes.append(vol1)

    #Delete Volume if Unused
    for vol in ec2.volumes.all():
        if  vol.state=='available':
                vid=vol.id
                v=ec2.Volume(vol.id)
                v.delete()
                print ('Deleted ' +vid)
        ## If we use tagging as ##
        # continue
        # for tag in vol.tags:
        #     if tag['Key'] == 'Name':
        #         value = tag['Value']
        #         if value != 'DND' and vol.state == 'available':
        #             vid = vol.id
        #             v = ec2.Volume(vol.id)
        #             v.delete()
        #             print('Deleted ' + vid)

    #email
    sns_client.publish(
        TopicArn='arn:aws:sns:ap-southeast-1:627315336549:Notify-Unused-EBS-Volume',
        Subject='Warning - Unused Volume List',
        Message=str(unused_volumes)
    )
    return "success"

Step 3
The final step we going to set our AWS EventBridge Rule for schedule to trigger our AWS Lambda Function we created in the previous step. Go to AWS EventBridge->Create New Rule; we enter below input.

• Name: do4m-unused-ebs-rule
• Description: do4m-unused-ebs-rule
• Event Bus: default
• Rule Type: Schedule Proceed with "Continue to create rule"

We set our EventBridge Schedule Pattern. We set our Cron Expression as below. For this use-case; I created schedule patter for every Saturday at 8 AM it will triggered Lambda and it happen every month every year. You may change to your requirement and you can refer the cron-expression here.

Next, we choose our Target for EventBridge Rule as invoke the AWS Lambda. We select the AWS Lambda Function we created in the previous step as our Lambda Function below.

Lastly, we created our AWS EventBridge Rule for Scheduler.

Note
For sample I've changed to trigger our AWS Lambda function every 5 minutes to get fast result.

The Results

We can monitor whether our EventBridge Rule is work by go to CloudWatch Log group we created via AWS Lambda Function. As we can see CloudWatch log record below it detected the unused EBS volumes and deleted it.

Our EBS Volume is deleted and only left the "In-Use" EBS volume in AWS Console.

We also get SNS notification that our AWS Lambda Function detected the unused and deleted it.

Summary

In this post we learn how to do AWS Cost Optimization by auto detect your unused EBS volume via AWS Lambda Function which it triggered by AWS EventBridge Rule. Lastly we get AWS SNS notifition email whenever Lambda Function found unused EBS volume.