Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

aws-auto-cleanup

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

Functional Requirements

• Reduce operational run-time of resources used for a testing\development, or for temporary spin-up of resources
• Reduce cost\operational expenses
• Ability to whitelist AWS resources that need to be retained

Operating Cost

< $2.00/mo for the following:

• AWS::Events::Rule
• AWS::Lambda::Function
• AWS::Lambda::Permission
• AWS::Logs::LogGroup
• AWS::IAM::Role
• AWS::S3::Bucket
• AWS::S3::Bucket
• AWS::DynamoDB::Table
• AWS::DynamoDB::Table

Node	Type	Unit	Usage	Total
auto-cleanup-settings-prod (DynamoDB-Table)	provisioned read	$0.00013 per hour for units of read capacity	1 units per month	$0.10
auto-cleanup-settings-prod (DynamoDB-Table)	provisioned write	$0.00065 per hour for units of write capacity	1 units per month	$0.48
auto-cleanup-settings-prod (DynamoDB-Table)	storage	$0.25 per GB-month	1 GB	$0.25
auto-cleanup-whitelist-prod (DynamoDB-Table)	provisioned read	$0.00013 per hour for units of read capacity	1 units per month	$0.10
auto-cleanup-whitelist-prod (DynamoDB-Table)	provisioned write	$0.00065 per hour for units of write capacity	1 units per month	$0.48
auto-cleanup-whitelist-prod (DynamoDB-Table)	storage	$0.25 per GB-month	1 GB	$0.25
auto-cleanup-prod (Lambda-Function)	1000	128MB * AvgTime * Invocations per month (Usage)	10000 invocations	$0.02
ServerlessDeploymentBucket (S3-Bucket)	Standard	$0.023 per GB - first 50 TB / month of storage used	10 GB	$0.23
auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket)	Standard	$0.023 per GB - first 50 TB / month of storage used	10 GB	$0.23
Dynamo Storage	Discount	First 25GB Free (-$0.25 per GB)	-2	($0.50)
Dynamo Backup	Point In Time Recovery	$0.20 per GB-month (Continuous backups)	0	$0.00
Dynamo Backup	OnDemand	$0.10 per GB-month (On Demand)	0	$0.00
Dynamo Network Outbound Traffic	Transfer	$0.09 per GB (Data Transfer Out)	1	$0.09
Lambda Service	Number of invocations	Invocation call for a Lambda function	10000	$0.00

Diagram

Stack

1. Serverless Stack Framework Core: 2.1.1 Plugin: 4.0.4 SDK: 2.3.2 Components: 3.1.3
2. AWS CLI
3. NPM

Preferred configuration

After you clone the aws-auto-cleanup repository in the next step, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:

1. Change the serverless.yml file line 4, to your company name or project name.

2. Within the auto_cleanup/data/auto-cleanup-settings.json file, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:

       "ttl": {
         "N": "120"
       }
   

Deploy

Following the directions listed on the README, over at the aws-auto-cleanup Github page!

Useful commands

serverless deploy --region region-example --aws-profile profile-example

serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Outputs

• Under CloudWatch, you'll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):

/aws/lambda/auto-cleanup-prod

With a log stream of the latest Serverless invocation (or scheduled Lambda execution):

| 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849268 | [INFO] S3 Bucket '46346-artifacts' was created 505
days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket '46346-cpds-infra' was created
494 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket '46346-sree52-testbkt' was
created 430 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-myservice-bucket' was
created 158 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-poc-deployment-bucket'
was created 166 days ago and has been deleted. (s3_cleanup.py,
buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket 's3-to-sns' was created 264 days
ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket
'serverless-telegram-bot-serverlessdeploymentbuck-46346' was created
236 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'swagger-bucket-1' was created
199 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'terraform-bucket-lokesh' was
created 243 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849270 | [INFO] S3 Bucket
'twilio-voice-test-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

Performing clean-up

1. Change the dry_run item in the DynamoDB table (auto-cleanup-settings-prod - if you kept the default settings name) to false to perform clean-up:

   { "dry_run" : { "BOOL" : false }}

   1. Invoke using:

serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Recap

aws-auto-cleanup helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!