DEV Community 👩‍💻👨‍💻

Troy
Troy

Posted on

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

aws-auto-cleanup

Using aws-auto-cleanup to keep an AWS test account neat & tidy (and minimal cost!)

Functional Requirements

  • Reduce operational run-time of resources used for a testing\development, or for temporary spin-up of resources
  • Reduce cost\operational expenses
  • Ability to whitelist AWS resources that need to be retained

Operating Cost

< $2.00/mo for the following:

  • AWS::Events::Rule
  • AWS::Lambda::Function
  • AWS::Lambda::Permission
  • AWS::Logs::LogGroup
  • AWS::IAM::Role
  • AWS::S3::Bucket
  • AWS::S3::Bucket
  • AWS::DynamoDB::Table
  • AWS::DynamoDB::Table
Node Type Unit Usage Total
auto-cleanup-settings-prod (DynamoDB-Table) provisioned read $0.00013 per hour for units of read capacity 1 units per month $0.10
auto-cleanup-settings-prod (DynamoDB-Table) provisioned write $0.00065 per hour for units of write capacity 1 units per month $0.48
auto-cleanup-settings-prod (DynamoDB-Table) storage $0.25 per GB-month 1 GB $0.25
auto-cleanup-whitelist-prod (DynamoDB-Table) provisioned read $0.00013 per hour for units of read capacity 1 units per month $0.10
auto-cleanup-whitelist-prod (DynamoDB-Table) provisioned write $0.00065 per hour for units of write capacity 1 units per month $0.48
auto-cleanup-whitelist-prod (DynamoDB-Table) storage $0.25 per GB-month 1 GB $0.25
auto-cleanup-prod (Lambda-Function) 1000 128MB * AvgTime * Invocations per month (Usage) 10000 invocations $0.02
ServerlessDeploymentBucket (S3-Bucket) Standard $0.023 per GB - first 50 TB / month of storage used 10 GB $0.23
auto-cleanup-prod-resourcetreebucket-troydieter (S3-Bucket) Standard $0.023 per GB - first 50 TB / month of storage used 10 GB $0.23
Dynamo Storage Discount First 25GB Free (-$0.25 per GB) -2 ($0.50)
Dynamo Backup Point In Time Recovery $0.20 per GB-month (Continuous backups) 0 $0.00
Dynamo Backup OnDemand $0.10 per GB-month (On Demand) 0 $0.00
Dynamo Network Outbound Traffic Transfer $0.09 per GB (Data Transfer Out) 1 $0.09
Lambda Service Number of invocations Invocation call for a Lambda function 10000 $0.00

Diagram

diagram

Stack

  1. Serverless Stack Framework Core: 2.1.1 Plugin: 4.0.4 SDK: 2.3.2 Components: 3.1.3
  2. AWS CLI
  3. NPM

Preferred configuration

After you clone the aws-auto-cleanup repository in the next step, you will have the opportunity to change the parameters of the serverless-stack deployment. A few notable changes:

  1. Change the serverless.yml file line 4, to your company name or project name.
  2. Within the auto_cleanup/data/auto-cleanup-settings.json file, you will find the default parameters for the function. I have (obviously) found that the TTL (time-to-live) of 7 days to be too low. I have raised that to 120 days. Example:

        "ttl": {
          "N": "120"
        }
    

Deploy

Following the directions listed on the README, over at the aws-auto-cleanup Github page!

Useful commands

serverless deploy --region region-example --aws-profile profile-example
serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event
Enter fullscreen mode Exit fullscreen mode




Outputs

  • Under CloudWatch, you'll see under Logs > Log Groups the following log group (if you kept the default Lambda function name):

/aws/lambda/auto-cleanup-prod

With a log stream of the latest Serverless invocation (or scheduled Lambda execution):

| 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-dev-serverlessdeploymentbucke-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'outbound-email-send-hand-serverlessdeploymentbuck-46346' was created
234 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849212 | [INFO] S3 Bucket
'46346-ai-chat-bot-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849268 | [INFO] S3 Bucket '46346-artifacts' was created 505
days ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket '46346-cpds-infra' was created
494 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket '46346-sree52-testbkt' was
created 430 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-myservice-bucket' was
created 158 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849269 | [INFO] S3 Bucket 'python-poc-deployment-bucket'
was created 166 days ago and has been deleted. (s3_cleanup.py,
buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket 's3-to-sns' was created 264 days
ago and has been deleted. (s3_cleanup.py, buckets(), line 168)

| | 1600638849269 | [INFO] S3 Bucket
'serverless-telegram-bot-serverlessdeploymentbuck-46346' was created
236 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'swagger-bucket-1' was created
199 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

| | 1600638849269 | [INFO] S3 Bucket 'terraform-bucket-lokesh' was
created 243 days ago and has been deleted. (s3_cleanup.py, buckets(),
line 168)

| | 1600638849270 | [INFO] S3 Bucket
'twilio-voice-test-dev-serverlessdeploymentbucket-46346' was created
230 days ago and has been deleted. (s3_cleanup.py, buckets(), line
168)

Performing clean-up

  1. Change the dry_run item in the DynamoDB table (auto-cleanup-settings-prod - if you kept the default settings name) to false to perform clean-up:

    { "dry_run" : { "BOOL" : false }}

    1. Invoke using:

serverless invoke --function AutoCleanup --region region-example --aws-profile profile-example --type Event

Recap

aws-auto-cleanup helps to keep multi-region AWS accounts clean. Resources are either white-listed, or deleted after specified amount of days. This reduces monthly expenses and conflicting resources within the same region!

Top comments (0)

Want to rep DEV and be comfy at the same time?

Check out our classic DEV shirt — available in multiple colors.