loading...

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

dietertroy profile image Troy ・4 min read

aws-auto-cleanup

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

Functional Requirements

  • Reduce operational run-time of resources used for a testing\development, or for temporary spin-up of resources
  • Reduce cost\operational expenses
  • Ability to whitelist AWS resources that need to be retained

Operating Cost

< $2.00/mo for the following:

  • AWS::Events::Rule
  • AWS::Lambda::Function
  • AWS::Lambda::Permission
  • AWS::Logs::LogGroup
  • AWS::IAM::Role
  • AWS::S3::Bucket
  • AWS::S3::Bucket
  • AWS::DynamoDB::Table
  • AWS::DynamoDB::Table
Node Type Unit Usage Total
auto-cleanup-settings-prod (DynamoDB-Table) provisioned read $0.00013 per hour for units of read capacity 1 units per month $0.10
auto-cleanup-settings-prod (DynamoDB-Table) provisioned write $0.00065 per hour for units of write capacity 1 units per month $0.48
auto-cleanup-settings-prod (DynamoDB-Table) storage $0.25 per GB-month 1 GB $0.25
auto-cleanup-whitelist-prod (DynamoDB-Table) provisioned read $0.00013 per hour for units of read capacity 1 units per month $0.10
auto-cleanup-whitelist-prod (DynamoDB-Table) provisioned write $0.00065 per hour for units of write capacity 1 units per month $0.48
auto-cleanup-whitelist-prod (DynamoDB-Table) storage $0.25 per GB-month 1 GB $0.25
auto-cleanup-prod (Lambda-Function) 1000 128MB * AvgTime * Invocations per month (Usage) 10000 invocations $0.02
ServerlessDeploymentBucket (S3-Bucket) Standard $0.023 per GB - first 50 TB / month of storage used 10 GB $0.23
auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket) Standard $0.023 per GB - first 50 TB / month of storage used 10 GB $0.23
Dynamo Storage Discount First 25GB Free (-$0.25 per GB) -2 ($0.50)
Dynamo Backup Point In Time Recovery $0.20 per GB-month (Continuous backups) 0 $0.00
Dynamo Backup OnDemand $0.10 per GB-month (On Demand) 0 $0.00
Dynamo Network Outbound Traffic Transfer $0.09 per GB (Data Transfer Out) 1 $0.09
Lambda Service Number of invocations Invocation call for a Lambda function 10000 $0.00

Diagram

diagram

Stack

  1. Serverless Stack Framework Core: 2.1.1 Plugin: 4.0.4 SDK: 2.3.2 Components: 3.1.3
  2. AWS CLI
  3. NPM

Preferred configuration

After you clone the aws-auto-cleanup repository in the next step, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:

  1. Change the serverless.yml file line 4, to your company name or project name.
  2. Within the auto_cleanup/data/auto-cleanup-settings.json file, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:

        "ttl": {
          "N": "120"
        }
    

Deploy

Following the directions listed on the README, over at the aws-auto-cleanup Github page!

Useful commands

serverless deploy --region region-example --aws-profile profile-example
serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event
Enter fullscreen mode Exit fullscreen mode




Outputs

  • Under CloudWatch, you'll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):

/aws/lambda/auto-cleanup-prod

With a log stream of the latest Serverless invocation (or scheduled Lambda execution):

| 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849268 | [INFO] S3 Bucket '46346-artifacts' was created 505
days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket '46346-cpds-infra' was created
494 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket '46346-sree52-testbkt' was
created 430 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-myservice-bucket' was
created 158 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-poc-deployment-bucket'
was created 166 days ago and has been deleted. (s3_cleanup.py,
buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket 's3-to-sns' was created 264 days
ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket
'serverless-telegram-bot-serverlessdeploymentbuck-46346' was created
236 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'swagger-bucket-1' was created
199 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'terraform-bucket-lokesh' was
created 243 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849270 | [INFO] S3 Bucket
'twilio-voice-test-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

Performing clean-up

  1. Change the dry_run item in the DynamoDB table (auto-cleanup-settings-prod - if you kept the default settings name) to false to perform clean-up:

    { "dry_run" : { "BOOL" : false }}

    1. Invoke using:

serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Recap

aws-auto-cleanup helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!

Posted on by:

dietertroy profile

Troy

@dietertroy

Global #AWS Solutions Architect for Perficient, AWS Partner Network Ambassador

Discussion

pic
Editor guide