CloudCycle - Set lifecycle for your cloud resources to avoid surprising costs

The Problem

I have recently forgot to Turn off / Terminate two ec2 instances running m5x.large and t3.medium that runs about 1 month and I just received a notification of my AWS monthly bill and shocked me!

At the first place I thought my account was hacked, But turns out that I just left two AWS EC2 Instances running. 🥲😔

The Solution (CloudCycle)

So, that's why I came up with a basic solution and share it. So now, whether I forgot to turn it off / terminate a running AWS Resources, I don't have to worry about it anymore, if I have properly set the desired lifecycle of my cloud resources.

I have created a lambda function which get's executed every 15 minutes to check if the supported resources are due to termination or not.

Now, if you or your team forgot to terminate cloud resources, the CloudCycle will do the job for you.

Tools & Language used

1. Golang - Since lambda function bills will be based on the duration of execution. At first, I developed this by using python and I have realized that I need to consider it's performance and execution time.

2. Terraform - Used only for deployment and examples.

Architecture

This setup will utilized the schedule expression of event bridge, In which event bridge get's executed every 15 minutes. In doing so, lambda function get validate if the supported resources is valid for termination.

But why termination? Instead of turning it off?

Of course there's a free tool available called Cloud-Custodian. But still, sometimes turning it off isn't enough and it can lead to lots of unused resources and can still occur minimal cost.

Sample Code for EC2 Instance Service

Below are the sample codes for EC2 Instance service.

package services

import (
    "context"
    "github.com/aws/aws-sdk-go-v2/service/ec2"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/service/ec2/types"
    "github.com/redopsbay/cloudcycle/internal"
    "github.com/redopsbay/cloudcycle/internal/schedule"
    "fmt"
)

type EC2 struct {
    InstanceId         string
    CloudCycle         string
    MarkForTermination bool
}

type EC2Instances struct {
    Instances []EC2
}

func GetEC2Instances(ctx context.Context, client *ec2.Client) ([]types.Reservation, error) {
    filters := []types.Filter{{
        Name:   aws.String("tag-key"),
        Values: []string{internal.TagKey},
    },
    }

    DescribeInputs := ec2.DescribeInstancesInput{
        Filters: filters,
    }

    instances, err := client.DescribeInstances(ctx, &DescribeInputs)

    if err != nil {
        panic(err)
        return []types.Reservation{}, err
    }

    return instances.Reservations, nil
}

func MarkInstancesForTermination(reservations []types.Reservation) (EC2Instances, error) {
    var instances EC2Instances

    for _, reservation := range reservations {
        for _, instance := range reservation.Instances {
            for _, tag := range instance.Tags {
                if *tag.Key == internal.TagKey {
                    Lifecycle, err := schedule.GetLifeCycle(instance.LaunchTime, *tag.Value)
                    if err != nil {
                        return EC2Instances{}, err
                    }

                    if schedule.ValidForTermination(Lifecycle) {
                        ec2instance := EC2{
                            InstanceId:         *instance.InstanceId,
                            CloudCycle:         *tag.Value,
                            MarkForTermination: true,
                        }
                        instances.Instances = append(instances.Instances, ec2instance)

                    } else {
                        ec2instance := EC2{
                            InstanceId:         *instance.InstanceId,
                            CloudCycle:         *tag.Value,
                            MarkForTermination: false,
                        }
                        instances.Instances = append(instances.Instances, ec2instance)
                    }
                }
            }
        }
    }
    return instances, nil
}

func StartEC2InstanceTermination(ctx context.Context, client *ec2.Client) error {
    var instanceIds []string

    reservations, err := GetEC2Instances(ctx, client)
    if err != nil {
        fmt.Println("Unable to get instances.")
        return err
    }

    instances, err := MarkInstancesForTermination(reservations)
    if err != nil {
        fmt.Println("Unable to mark instances for termination.")
        return err
    }

    for _, instance := range instances.Instances {
        if instance.MarkForTermination {
            instanceIds = append(instanceIds, instance.InstanceId)
            fmt.Printf("\nInstanceID: %s, ForTermination: %t, CloudCycle: %s\n",
                instance.InstanceId,
                instance.MarkForTermination,
                instance.CloudCycle)
        }
    }

    TerminatedOutput, err := client.TerminateInstances(ctx, &ec2.TerminateInstancesInput{
        InstanceIds: instanceIds,
    })

    for _, state := range TerminatedOutput.TerminatingInstances {
        if *state.CurrentState.Code == 0 {
            fmt.Printf("InstanceID: %s, State: Pending for Termination", *state.InstanceId)
        } else if *state.CurrentState.Code == 32 {
            fmt.Printf("InstanceID: %s, State: Shutting down", *state.InstanceId)
        } else if *state.CurrentState.Code == 48 {
            fmt.Printf("InstanceID: %s, State: Shutting down", *state.InstanceId)
        } else if *state.CurrentState.Code == 16 {
            fmt.Printf("InstanceID: %s, State: Still running", *state.InstanceId)
        } else if *state.CurrentState.Code == 64 {
            fmt.Printf("InstanceID: %s, State: Stopping", *state.InstanceId)
        } else if *state.CurrentState.Code == 80 {
            fmt.Printf("InstanceID: %s, State: Stopped", *state.InstanceId)
        } else {
            fmt.Printf("InstanceID: %s, State: Unknown", *state.InstanceId)
        }
    }

    return nil

}

Objective

My objective with CloudCycle is to automatically cleanup supported resources based on the specified duration or lifecycle through resource tagging, And to support the commonly used resources that causes AWS Bills to grow even though the cloud resources is not needed anymore.

No more story telling!

For complete documentation and project link, you can proceed directly to my GitHub repo below.

https://github.com/redopsbay/cloudcycle

This repo is open for contributors!!! Some documents and cloud resources are currently WORK-IN-PROGRESS

Usage

Just specify that tag and set your desired lifecycle for supported resources with CloudCycle Key. Below are the supported duration.

Suffixes	Detail	Sample Value
m	Minutes	60m
h	Hours	2h
d	Days	7d

How does it works?

CloudCycle will get all the supported resources with a tagged key CloudCycle and it will simply compare the current time vs launch time of the supported resources with the specified key/value pair resource tag if the supported resources are valid for termination.

Below are the sample terraform code.

Specify ec2 lifecycle by 24 hours from it's launch date.

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"

  tags = {
    CloudCycle = "1d" // This ec2 instance will be terminated within 24 hours from it's launch date.
  }
}

Sample Terraform Deployment Usage

For deployment, you can refer to the github repo Deployment Page

Benefit's

Sometimes it's better to let go than stay strong. Leaving up unused resources will incur costs.

And lastly, No nightmare's, No poverty! 😂🤣

Reference

• https://github.com/redopsbay/cloudcycle