Automate AWS CloudWatch Log Retention with Bash: Save Costs & Stay Compliant

🔹 Introduction :

Managing CloudWatch log groups is a critical part of maintaining operational efficiency and cost control in AWS. However, it's easy to overlook retention settings — especially when log groups are created automatically by various AWS services. Without a defined retention period, logs accumulate indefinitely, leading to increased storage costs and unnecessary clutter.

In this blog, I’ll walk through streamlined approach to automatically detect CloudWatch log groups without a retention policy, update them to a 30-day retention period, and generate an HTML report delivered straight to your inbox.

The solution is powered by a simple Bash script that leverages the AWS CLI and standard Linux utilities — making it easy to integrate into any DevOps workflow.

Whether you're a cloud engineer trying to stay compliant or just looking to reduce AWS costs, this automated approach will save time, improve visibility, and ensure consistent log management across your environment.

🔹 Challenges Faced in Manual Process:
Manually managing log retention policies in AWS is like trying to clean every file cabinet in a skyscraper—painful, slow, and error-prone. Some of the common problems:

❌ You can't visually identify which logs lack retention
❌ You have to click through each log group in the AWS Console
❌ There’s no built-in notification when retention is missing
❌ Risk of accumulating terabytes of unused logs

So I thought — “Why not automate the boring stuff?”

🔹 Benefits of Automating CloudWatch Retention Updates
Automating retention policies brings a whole bouquet of benefits:

🌟 Cost Control – Say goodbye to ever-growing log storage bills
🔍 Audit Friendly – Track what's changed, when, and how
📧 Proactive Alerting – Get email summaries with detailed tables
🧹 Cleaner Environment – Consistent retention policies = better hygiene
⏱️ Time Saved – No more manual clicking or forgetfulness

🔹 Prerequisites
Before I dive in, make sure you have the following:

1. - An AWS account with access to CloudWatch
2. - IAM permissions to read and update log groups
3. - AWS CLI configured on your machine
4. - Bash shell environment (Linux or macOS)
5. - Tools like jq, sendmail, mailutils installed

🔹 Step 1: Install AWS CLI
If you haven’t installed the AWS CLI yet, follow the steps below:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install


Then configure your credentials:
aws configure

🔹 Step 2: Install Dependencies
You’ll also need jq and sendmail for parsing and email delivery:

sudo apt install jq mailutils -y

🔹 Step 3: Create IAM Policy as per below , attached to IAM role and assign that role to EC2 instance.

You’ll need the following IAM permissions to make it work:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DescribeLogGroups",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": "*"
    },
    {
      "Sid": "PutRetentionPolicy",
      "Effect": "Allow",
      "Action": "logs:PutRetentionPolicy",
      "Resource": "*"
    },
    {
      "Sid": "CloudWatchMetricsAccess",
      "Effect": "Allow",
      "Action": "cloudwatch:GetMetricStatistics",
      "Resource": "*"
    }
  ]
}

Permissions include:

• logs:DescribeLogGroups
• logs:DescribeLogStreams
• logs:PutRetentionPolicy
• cloudwatch:GetMetricStatistics

🔹 Step 4: Clone the GitHub Repository
Instead of writing the script manually, you can simply clone the prebuilt GitHub repository that includes the script, required IAM policy, and a README.
git clone https://github.com/alokshanhbti/cloudwatch-retention-update.git
cd cloudwatch-retention-update

Inside the folder, you’ll find:

1. cloudwatch-retention-update.sh – The automation script
2. iam-policy.json – IAM policy required for permissions
3. README.md – Full documentation and usage instructions

🔹 Step 5: Make the Script Executable
After saving the script, make it executable with:
chmod +x cloudwatch-retention-update.sh

🔹 Step 6: Run the Script
Simply execute:
./cloudwatch-retention-update.sh

The script will log activity to a file, apply changes, and email the report to the address you specify.

🔹 Step 7: Script Flow
Here’s how the script works behind the scenes:

🔍 Scan CloudWatch for log groups with no retention
🧠 Fetch metadata: log group name, retention, last event, service name, and storage
✍️ Update retention to 30 days using put-retention-policy
📨 Generate HTML email with two colorful tables:
Before update
After update
📬 Send email via sendmail with all details

🔹 Step 8: Screen shots of email and logs

Email part Before update :

Email part After update :

Logs :

🔹 Conclusion

Automating CloudWatch log retention is a simple yet highly effective way to maintain a clean, cost-efficient, and compliant cloud environment. With this Bash script, you can easily identify log groups without retention settings, apply a consistent 30-day policy, and receive a well-formatted email report — all with minimal effort and zero manual intervention.

This solution not only improves visibility and governance but also frees up your time to focus on higher-value tasks.

Thank you for reading!
If this script helps improve your cloud hygiene, feel free to share it with your team or contribute to the project.

📂 Access the GitHub Repository Here:

alokshanhbti / cloudwatch-retention-update

cloudwatch-retention-update repo that audits AWS CloudWatch log groups with no retention period and updates them to 30-day retention and sends email

📊 CloudWatch Log Retention Manager

cloudwatch-retention-update.sh is a Bash script that audits AWS CloudWatch log groups with no retention period set, updates them to a 30-day retention, and sends a HTML email report containing color-coded tables.

---

🔧 Features

✅ Identifies log groups without retention
✅ Fetches last log date, associated AWS service, and storage usage (in GB)
✅ Applies a 30-day retention policy
✅ Sends an HTML email via sendmail with:

• 📋 Before Update Table
• ✅ After Update Table

---

📁 Script Overview

• 📂 Log Group Scan — Uses aws logs describe-log-groups and jq to filter targets
• ⏳ Retention Status — Detects null retention policies
• 📅 Last Log Timestamp — Uses describe-log-streams
• 💾 Storage Usage (GB) — Uses cloudwatch:GetMetricStatistics for StoredBytes
• 📧 HTML Email Report — Sends two HTML tables (before & after) with colors

---

🚀 Usage

Step 1: Make it executable

chmod +x cloudwatch-retention-update.sh

Step

…

View on GitHub

Happy automating! 🚀