DEV Community

Arjun Mullick
Arjun Mullick

Posted on

Monitoring and Analyzing Cloud Storage Costs

TL;DR: Use of Python scripts and AWS CLI tools to automatically monitor and analyze cloud storage costs, optimize spending on AWS S3 by tracking usage and identifying savings opportunities.

Abstract
Managing cloud storage costs is crucial for any organization using AWS S3. By leveraging Python scripts and AWS Command Line Interface (CLI) tools, users can automate the monitoring and analysis of storage expenses, track usage patterns, and quickly identify cost-saving opportunities. This article explains, in simple terms, how to set up these tools, provides practical code examples, and highlights best practices for ongoing cloud cost optimization.

Image intro

Introduction
Cloud storage is powerful, but costs can add up quickly if you’re not paying attention. AWS S3 offers a range of storage classes and features — like Intelligent-Tiering and lifecycle policies — to help you optimize costs automatically. However, to truly control your spending, you need to monitor your usage, analyze where your money is going, and adjust your storage strategies as your data grows and changes. By combining the AWS CLI and Python scripts, you can automate this process, making it easier to keep your cloud bills in check and spot trends before they become expensive problems. Learn more about S3 Intelligent-Tiering and cost optimization.

Prerequisites

  • AWS account with S3 and billing permissions.
  • Python 3.x installed.
  • Boto3 (the AWS SDK for Python) installed.
  • AWS CLI installed and configured with your credentials.
  • Basic familiarity with running command-line commands and Python scripts.

If you’re new to AWS CLI, here’s a getting started guide.

Setting Up Your Environment
First, make sure you have the AWS CLI and Boto3 installed:

pip install boto3
pip install awscli
Enter fullscreen mode Exit fullscreen mode

Configure your AWS credentials if you haven’t already:

aws configure
Enter fullscreen mode Exit fullscreen mode

You’ll be prompted for your AWS Access Key, Secret Key, region, and output format.

Using AWS CLI to Get Storage Cost and Usage Data
The AWS CLI allows you to quickly retrieve information about your S3 buckets and storage usage. For example, to see the total size and object count for a bucket:

aws s3 ls s3://your-bucket-name --recursive --human-readable --summarize
Enter fullscreen mode Exit fullscreen mode

This command will print the total number of objects and the total size of your bucket, helping you understand your storage footprint.

To get more detailed cost information, you can use the AWS Cost Explorer via CLI:

aws ce get-cost-and-usage \
  --time-period Start=2024-07-01,End=2024-07-31 \
  --granularity MONTHLY \
  --metrics "UnblendedCost" \
  --filter file://filter.json
Enter fullscreen mode Exit fullscreen mode

This command returns your AWS costs for the specified period. You can further filter by service (e.g., S3) in the filter.json file.

See AWS documentation for more on Cost Explorer.

Automating Cost Analysis with Python
Python scripts can help you automate cost analysis, generate reports, and even alert you to unusual spending.

Here’s a simple example using Boto3 to list all your S3 buckets and their sizes:

import boto3
s3 = boto3.client('s3')
response = s3.list_buckets()
for bucket in response['Buckets']:
    bucket_name = bucket['Name']
    size = 0
    objects = s3.list_objects_v2(Bucket=bucket_name)
    if 'Contents' in objects:
        for obj in objects['Contents']:
            size += obj['Size']
    print(f"Bucket: {bucket_name}, Size: {size / (1024**3):.2f} GB")
Enter fullscreen mode Exit fullscreen mode

For more advanced analysis, you can use the boto3 Cost Explorer client to pull cost data and analyze trends:

import boto3
import datetime
client = boto3.client('ce')
end = datetime.date.today()
start = end.replace(day=1)
response = client.get_cost_and_usage(
    TimePeriod={'Start': str(start), 'End': str(end)},
    Granularity='MONTHLY',
    Metrics=['UnblendedCost'],
    Filter={
        "Dimensions": {
            "Key": "SERVICE",
            "Values": ["Amazon Simple Storage Service"]
        }
    }
)
print(response)
Enter fullscreen mode Exit fullscreen mode

This script fetches your S3 storage costs for the current month.

Scheduling and Automating Reports
You can schedule these scripts to run daily, weekly, or monthly using cron jobs (Linux/Mac) or Task Scheduler (Windows). This way, you’ll always have up-to-date cost and usage reports without manual intervention.

Interpreting the Results
Look for buckets with rapid growth: These may need lifecycle policies or intelligent tiering.
Identify rarely accessed data: Move it to a cheaper storage class.
Spot unusual spikes: Investigate for accidental uploads or misconfigured applications.
AWS S3 Intelligent-Tiering can help automate some of these optimizations, but you still need to monitor and adjust as your data and usage patterns change. How S3 Intelligent-Tiering works.

Best Practices

  • Set up regular monitoring: Don’t wait for a surprise bill.
  • Use tags: Tag your buckets and objects by project or department to track who is generating costs.
  • Combine automation with AWS features: Use lifecycle policies and Intelligent-Tiering alongside your monitoring scripts for maximum savings.
  • Review AWS billing alerts: Set up alerts for when you approach budget thresholds.

Automate S3 Lifecycle rules at scale.

Conclusion
By combining AWS CLI tools and Python scripts, you can automate the monitoring and analysis of your cloud storage costs, making it much easier to manage your AWS S3 spending. These tools help you understand where your money is going, identify savings opportunities, and keep your cloud storage efficient and cost-effective.

References:

  1. Using S3 Intelligent-Tiering
  2. How S3 Intelligent-Tiering works
  3. Automate S3 Lifecycle rules at scale

Top comments (0)