How We Archived Over 14TB of S3 Logs and Cut Storage Costs by 95%

#s3 #cloud #cloudsecurity #aws

Context

During an internal audit aimed at optimizing our cloud infrastructure costs, we discovered that we had over 14TB of AWS WAF logs sitting in an S3 bucket — with no lifecycle policy configured.

These logs had accumulated over several years and were stored in the S3 Standard storage class, which generated a significant monthly cost — especially considering these logs were rarely accessed.

To make things worse, new logs were being added daily , and our S3 costs were growing unchecked.

The Challenge

We were facing three main constraints:

We couldn’t delete the logs due to compliance requirements.
The files were very small (under 40 MB), making them too expensive to transition to a colder storage class like Glacier or Deep Archive using standard tools.
Downloading logs locally to compress and re-upload wasn’t an option: AWS S3 data egress costs exceed $1000 per TB , making this strategy unfeasible at scale.

On top of that, we were spending nearly $350/month just on S3 storage — an unnecessary cost for infrequently accessed data.

The Solution

After evaluating multiple options, we implemented a simple but effective strategy :

We launched an EC2 instance inside our VPC, with an EBS volume sized for the largest log month.
We attached an S3 VPC endpoint, ensuring all traffic remained within the AWS network to avoid data transfer charges.
We wrote a lightweight Bash script to:
List and download all logs from a specific month for a given Web ACL.
Bundle them (using tar, since logs were already gzipped) into larger archive files.
Upload those archives back to the same bucket — but this time, directly using the S3 Glacier Deep Archive storage class.

Here’s the script:

#!/bin/bash

BUCKET="aws-waf-logs"
UUID="1xxxxxxx-dxxx-4xxx-8xxx-axxxxxxxxxx" #Only if you use firewall manager to manage WAF Logs
ORIGEN="us-east-1"
YEAR="2023"
TMP_DIR="./waf-archive"

mkdir -p "$TMP_DIR"

# List account_ids
aws s3 ls "s3://$BUCKET/$UUID/AWSLogs/" | awk '{print $2}' | sed 's#/##' | while read -r ACCOUNT_ID; do
  echo "Account ID: $ACCOUNT_ID"

  # List WebACLs
  aws s3 ls "s3://$BUCKET/$UUID/AWSLogs/$ACCOUNT_ID/WAFLogs/$ORIGEN/" | awk '{print $2}' | sed 's#/##' | while read -r WEB_ACL; do
    echo "WebACL: $WEB_ACL"

    # Listar months of the yeat
    aws s3 ls "s3://$BUCKET/$UUID/AWSLogs/$ACCOUNT_ID/WAFLogs/$ORIGEN/$WEB_ACL/$YEAR/" | awk '{print $2}' | sed 's#/##' | while read -r MONTH; do
      echo "Mes: $MONTH"

      S3_PREFIX="$UUID/AWSLogs/$ACCOUNT_ID/WAFLogs/$ORIGEN/$WEB_ACL/$YEAR/$MONTH/"
      TAR_NAME="${YEAR}_${MONTH}.tar"
      TAR_PATH="$TMP_DIR/$TAR_NAME"
      #Here you can choose another bucket to archive the logs, or create another folder in the same bucket
      ARCHIVE_KEY="archived_logs/$ACCOUNT_ID/$WEB_ACL/$TAR_NAME"

      echo "Downloading from: s3://$BUCKET/$S3_PREFIX"
      mkdir -p "$TMP_DIR/files"
      aws s3 cp --recursive "s3://$BUCKET/$S3_PREFIX" "$TMP_DIR/files"

      echo "Packaging: $TAR_PATH"
      tar -cf "$TAR_PATH" -C "$TMP_DIR/files" .

      # Change $BUCKET if you use another bucket
      echo "Uploading to: s3://$BUCKET/$ARCHIVE_KEY"
      aws s3 cp "$TAR_PATH" "s3://$BUCKET/$ARCHIVE_KEY" --storage-class DEEP_ARCHIVE

      echo "Removing temporal files"
      rm -rf "$TMP_DIR/files"
      rm -f "$TAR_PATH"

    done
  done
done

rmdir "$TMP_DIR"

echo "Archived complete for $YEAR"

This strategy allowed us to retain all logs , fully meeting our compliance requirements , while implementing a financially sustainable retention policy.

Results

We went from paying nearly $350/month in S3 Standard to under $15/month using S3 Glacier Deep Archive — a 95.6% reduction in storage costs.

Key advantages:

The entire process ran inside AWS , so there were no egress charges.
Monthly log processing was automated and scalable across multiple Web ACLs.
The total EC2 cost for processing was around $10 , compared to over $600 if we had used S3’s native transition feature — even under optimistic assumptions (e.g., every object being ~75MB).

Lessons Learned

Don’t overlook lifecycle policies — buckets storing large volumes of data over time can lead to massive and unnecessary costs.
Bundling small files into larger archives greatly reduces storage costs, especially when paired with colder S3 storage classes.
Keep data movement within the AWS network whenever possible — this avoids transfer charges and speeds up processing.

Recommendations

Configure lifecycle policies from day one — it’s easy to ignore logs until they become a financial burden.
If you need to retain data long-term with little or no access , S3 Glacier Deep Archive is an excellent option (just account for longer retrieval times).

DEV Community