DEV Community: Agustin De Mozzi

How to Use AWS Security Agent as a Free Vulnerability Scanner for Your Project

Agustin De Mozzi — Thu, 27 Aug 2026 15:29:53 +0000

AWS Security Agent

If you’re building a side project or an early-stage startup, security is probably not at the top of your priority list. You’re focused on shipping features, fixing bugs, and getting users. But vulnerabilities don’t care about your roadmap.

The good news is that AWS Security Agent offers code review and custom security requirements at no cost. You get up to 1,000 PR code reviews per month and 20 custom security requirements per account — more than enough for any personal project or small team. The agent automatically scans every pull request against your security rules and posts findings directly as PR comments.

In this tutorial, I’ll walk you through how to set it up and, more importantly, how to write custom security requirements tailored to your specific project. We’ll use a Node.js e-commerce application as an example.

What You’ll Get

Automated security scanning on every pull request (up to 1,000/month)
Custom security rules that match your project’s specific needs (up to 20 per account)
Findings posted directly as comments in your PRs
Zero cost for code review capabilities (see FAQ)

Prerequisites

Before we start, make sure you have:

An AWS account — If you don’t have one, create a free AWS account here. You’ll need a valid credit card for verification, but you won’t be charged for the code review features we’ll use in this tutorial.
A GitHub or Bitbucket Cloud account with at least one repository you want to scan.
Admin access to your GitHub organization (or user account ownership)

Step 1: Create an Agent Space

An Agent Space is a dedicated workspace where AWS Security Agent organizes all security assessments for a specific application. Think of it as a container for your project’s security configuration, reviews, and findings.

Sign in to the AWS Console and navigate to AWS Security Agent.
Go to the Agent Spaces page.
Click Create Agent Space.
Enter a name that identifies your project (e.g., “ecommerce-space”).
Optionally add a description like “Node.js e-commerce application — payment processing and user management”.
Click Create.

Agent space creation

For more details, see the official documentation on creating an Agent Space.

Step 2: Connect Your Repository

Once your Agent Space is ready, connect it to your code repository so the agent can scan your pull requests. In this example we’ll use GitHub, but AWS Security Agent also supports Bitbucket Cloud, GitLab, and GitHub Enterprise Server — see the full integrations documentation for other providers.

In the AWS Security Agent console, navigate to Integrations.
Click Add integration and select GitHub.
Click Open AWS Security Agent in GitHub — you’ll be redirected to GitHub.
Select the organization or user account where your repository lives.
Choose which repositories AWS Security Agent can access (you can grant access to all repos or select specific ones).
Click Install and authorize in GitHub.
Back in the AWS Console, enter a Registration name (e.g., “ecommerce_repo”), select the account type, and click Connect.

After connecting, navigate to your Agent Space, enable code review for the connected repositories, and you’re set. Every new PR will be automatically scanned.

For the full step-by-step guide, see Connect AWS Security Agent to GitHub repositories.

Step 3: Understand the Managed Security Packs

Before writing custom requirements, it’s worth knowing that AWS Security Agent already comes with managed security requirement packs based on industry standards and best practices. These are enabled with a single click and cover common vulnerabilities out of the box.

You should enable the managed packs that are relevant to your project as a baseline. They’re maintained by AWS and updated automatically.

Managed Security Requirements Packs

The custom requirements we’ll create next are meant to complement these packs with rules that are specific to your business logic — things that a generic security scanner would never catch.

Step 4: Create a Custom Security Requirement

Custom security requirements let you teach AWS Security Agent what matters specifically for your project — the business logic that generic scanners can’t know about. You have 20 slots, so use them wisely.

Before jumping into the console, let’s think about what to write and how to write it well.

How to Think About Custom Requirements

The key question is: what are the security rules that are specific to MY project that a generic scanner wouldn’t know about?

For our Node.js e-commerce example, we need to think about:

What sensitive data do we handle? (user info, payment data, order history)
What are the critical business flows? (checkout, authentication, inventory)
What resources belong to specific users? (orders, carts, addresses, invoices)

The best custom requirements target business logic that a generic scanner would never catch. They also need to provide ongoing value — not just a one-time check, but something that gets re-evaluated every time new code is introduced.

How to Write a Good Requirement

Here’s a framework you can follow. For the official guidelines, see Best practices for defining security requirements.

Focus on recurring risks — The best custom requirements catch issues that can be reintroduced with every new feature. Ownership validation, input sanitization at specific boundaries, or business rule enforcement are great candidates because they apply to every new endpoint, not just existing ones.
Cover your business logic — Generic scanners know about SQL injection and XSS. They don’t know that in YOUR app, orders have owners, prices should never come from the frontend, or that specific workflows require specific validations.
Be specific in compliance criteria — The more precise you are about what “compliant” and “non-compliant” look like, the better the agent will perform. Include concrete examples, field names, and patterns from your actual codebase.
Keep applicability focused — Clearly state when the requirement should NOT apply to avoid false positives. If a requirement only applies to route handlers that accept resource IDs, say so.
Let managed packs handle the generic stuff — Don’t waste your 20 custom requirement slots on things like “validate JWT tokens” or “don’t hardcode secrets.” The managed packs already cover those. Use your custom requirements for what only YOU know about your application.

Example: Enforce Resource Ownership Validation on All User-Scoped Endpoints

Why this matters: In an e-commerce app, resources like orders, carts, and addresses belong to specific users. Every time a new endpoint is created that accesses one of these resources by ID, there’s a risk of introducing an IDOR (Insecure Direct Object Reference) vulnerability — where any authenticated user can access another user’s data simply by guessing or enumerating IDs.

This is not a one-time fix. Every new endpoint that touches a user-owned resource needs this check. It’s the kind of thing that slips through when a developer is in a rush to ship a feature — and that’s exactly why it makes a great custom requirement. The agent will re-evaluate it on every PR that adds or modifies route handlers.

Custom Security Requirements Fields

Loading It Into the Console

Now that we know what we want to enforce, let’s create it in AWS Security Agent:

In the AWS Security Agent console, go to Security requirements.
Select the Custom security requirements packs tab.
Click Create security requirements pack , give it a name (e.g., “ecommerce-packs”) and an optional description, then click Create security requirement pack.
Open the pack you just created. In the Pack security requirements section, click Add requirement manually.
Fill in the fields with the values from the table above: name, description, applicability, compliance criteria, and optionally remediation guidance.
Click save.

Custom Security Requirement

Once created, you’ll see it enabled on the security pack:

Security Requirement Pack

Seeing It in Action

To test this setup, I created a simple Node.js e-commerce API with endpoints for orders, cart, and user profiles. All the existing endpoints correctly validate resource ownership using an authorizeOwnership middleware. Then I opened a PR that adds a new invoice endpoint — but this time, "forgot" to add the ownership check.

Here’s what happened: the Security Agent reviewed the PR and flagged the exact vulnerability within minutes.

The finding clearly explains:

What’s wrong: the invoice endpoint loads an order by ID without verifying it belongs to the authenticated user
The impact: any logged-in user can access another customer’s order details, including PII like shipping addresses
How to fix it: apply the authorizeOwnership middleware or scope the query with the user's ID

Security Agent PR Review

This is the custom requirement doing its job. A generic scanner might flag this as a general authorization issue, but because we defined the specific pattern our app uses (the authorizeOwnership middleware, the user field on resources), the finding is precise and actionable.

Bonus: What the Managed Packs Found

Here’s the thing — the custom requirement wasn’t the only thing that caught issues. The managed security packs (which require zero configuration) independently found 6 additional vulnerabilities in the same PR:

JWT verification without an algorithm allowlist (HIGH)
A non-atomic checkout flow vulnerable to race conditions (HIGH)
Missing input validation on cart endpoints (MEDIUM)
NoSQL query injection via unvalidated ObjectIds (MEDIUM)
Unreachable order cancellation logic due to a state machine contradiction (MEDIUM)
Cart item deduplication bypass via case-sensitive ID comparison (MEDIUM)

Security Agent PR Review Sumamry

That’s 7 total findings from a single PR review — 1 from your custom requirement and 6 from the managed packs — all without spending a cent.

Wrapping Up

With this setup, you now have an automated security review pipeline that:

Scans every pull request against both AWS-managed security standards and your custom rules
Catches project-specific vulnerabilities that generic tools would miss
Costs nothing for code review features (you only pay if you use penetration testing)
Requires no maintenance — the managed packs are updated by AWS, and your custom requirements stay active until you change them

For a solo developer or a small team, this is a significant improvement over having no security review at all. The 1,000 PR reviews per month and 20 custom requirements are more than enough to cover most early-stage projects.

The most valuable part is the custom requirements. Take 30 minutes to think about what’s specific to your project — the data you handle, the services you connect to, the business rules that matter — and translate those into clear requirements. That’s 30 minutes that will save you from shipping vulnerabilities to production.

Useful links:

How to Analyze AWS WAF Count Rules and Generate Automated Reports

Agustin De Mozzi — Fri, 26 Jun 2026 18:02:40 +0000

The Problem

If you manage AWS WAF at scale — across multiple accounts, multiple WebACLs, and dozens of protected resources — you know the pain. AWS WAF logs are JSON blobs sitting in S3. The console gives you basic metrics, but when you need to answer questions like:

“What happens if we switch these rules from Count to Block?”
“Are we blocking legitimate traffic?”
“What are the top attack patterns hitting our applications?”

…you’re left manually querying Athena, exporting CSVs, and building spreadsheets. For every single WebACL. Every time.

I built WAF Report Builder to solve this. It’s a Python CLI tool that takes your WAF logs from S3, runs the analysis through Athena, and generates a self-contained HTML report you can share with your team, your clients, or your CISO — no AWS console access required.

The repo is available at: github.com/agudemozzi/aws-waf-report-builder

If you’re managing AWS WAF at scale and tired of manual log analysis, give it a try. PRs and feedback welcome.

How you can generate reports for AWS WAF Count Rules with this tool?

The tool operates in two modes:

Count Mode (Pre-enforcement)

This is the mode you use before activating rules. AWS WAF allows you to set managed rules to Count mode with an override action of Block. This means the rule evaluates traffic and logs what would have been blocked, without actually blocking anything.

The report shows you exactly what traffic those rules are flagging:

Top source IPs
Top targeted URIs
Geographic distribution (with an interactive map)
Which rules are triggering the most, broken down by protected resource
Detailed drill-down: for each rule, what URIs and User-Agents are being flagged

This gives you the confidence to flip the switch — or identifies rules that might generate false positives.

Block Mode (Active Enforcement)

Once rules are active, you need visibility into what’s actually being blocked. Are you stopping real attacks? Are you accidentally blocking legitimate users?

Same report structure, but filtered to requests that were effectively terminated by a WAF rule.

The Report

The output is a single .html file that opens in any browser. No server needed, no dependencies for the viewer:

Press enter or click to view image in full size

Each report includes:

Summary section explaining what the report covers, the analysis period, and the account/WebACL context
Top IPs — immediately shows if there’s a single source hammering your application
Geographic map — choropleth visualization of request origins
Top URIs — what paths are being targeted
Top rules per resource — separate tables for each CloudFront distribution, ALB or API Gateway, so you can see which resource is getting the most rule activity
Rule drill-down — expandable accordions showing URI + User-Agent combinations for each rule, with full horizontal scrolling for long values. Rules involving query strings automatically include the Args column.

Press enter or click to view image in full size

Why Not Just Use the AWS Console?

A few reasons this is better for consulting and multi-account environments:

Shareability : The HTML file can be emailed, attached to a ticket, or embedded in a presentation. No AWS access needed for the recipient.
Multi-account : When you manage 20+ accounts with Firewall Manager, you need reports per WebACL. This tool processes one S3 path at a time, or multiple in batch.
Pre-enforcement confidence : The Count mode analysis is something the console doesn’t surface well. You get a clear picture of impact before changing rule actions.
Audit trail : Generated reports serve as point-in-time snapshots. Run them monthly and you have a security posture history.
Customizable : It’s Python + SQL + HTML. Need an extra metric? Add a query and a template section.

What’s Next

Some ideas I’m considering:

Timeline analysis : Adding a time-series view to spot attack spikes
CSV input mode : For environments where you can’t run Athena directly but can export results
Comparative reports : Diff between two periods to see if rule tuning had an effect

Agustín De Mozzi

Centralizing AWS WAF Logs with Kinesis Firehose and S3

Agustin De Mozzi — Sun, 14 Sep 2025 05:05:35 +0000

When it comes to cloud security, logs are very important. Having them centralized and well-structured is key for auditing, detecting incidents, and generating meaningful reports.

In this article, we’ll walk through how to configure AWS WAF logging to a centralized S3 bucket using Amazon Kinesis Data Firehose , leveraging dynamic partitioning to keep logs organized by account, Web ACL, and date.

Why Firehose Instead of Direct S3 Logging?

At first glance, sending logs directly from WAF to S3 seems like the simplest solution. However, costs and flexibility tell a different story.

Direct S3 logging from WAF :

AmazonCloudWatch USE1-S3-Egress-Bytes costs $0.25 per GB (first 10TB).

Via Firehose :

Amazon Kinesis Firehose Dynamic Partitioning → $0.02 per GB
Amazon Kinesis Firehose PutRecordBatch → $0.029 per GB
Total: $0.049 per GB → ~80% cheaper.

On top of that, Firehose offers extra benefits:

Process logs in-transit with Lambda.
Dynamically partition logs in S3.
Custom compress data before storing.

Architecture

This solution works whether you manage Web ACLs through Firewall Manager or deploy them individually across multiple AWS accounts.

Step-by-Step Setup

1. Create the Lambda Function

The following Lambda extracts account ID and WebACL name from the WAF log and sets partition keys. You have to create this function in the same account where Firewall Manager is configured, or in the same account where the Web ACL is created:

import json
import base64
from datetime import datetime

def lambda_handler(event, context):
    output = []
    for record in event['records']:
        payload = base64.b64decode(record['data'])
        log = json.loads(payload)
        webacl_arn = log.get("webaclId", "")
        account_id = "UnknownAccount"
        webacl_name = "UnknownWebACL"
        try:
            arn_parts = webacl_arn.split(":")
            if len(arn_parts) > 4:
                account_id = arn_parts[4] # Account ID
            resource_parts = webacl_arn.split("/")
            if len(resource_parts) > 2:
                webacl_name = resource_parts[-2] # WebACL name
        except Exception:
            pass
        now = datetime.utcnow()
        year = str(now.year)
        month = f"{now.month:02d}"
        day = f"{now.day:02d}"
        record['result'] = 'Ok'
        record['metadata'] = {
            "partitionKeys": {
                "accountId": account_id,
                "webaclName": webacl_name,
                "year": year,
                "month": month,
                "day": day
            }
        }
        output.append(record)
    return {'records': output}

With this setup, logs will be stored in S3 like:

s3://aws-waf-logs-your-bucket///year=2025/month=09/day=09/

2. Allow Firehose to Invoke the Lambda

Update the Lambda resource policy so that firehose.amazonaws.com can invoke it.

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "Inovke",
      "Effect": "Allow",
      "Principal": {
        "Service": "kinesis.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-1:<AccountId>:function:<your-lambda-name>",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:firehose:us-east-1:<AccountId>:deliverystream/<your-firehose>"
        }
      }
    },
    {
      "Sid": "GetConfig",
      "Effect": "Allow",
      "Principal": {
        "Service": "kinesis.amazonaws.com"
      },
      "Action": "lambda:GetFunctionConfiguration",
      "Resource": "arn:aws:lambda:us-east-1:<AccountId>:function:<your-lambda-name>",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:firehose:us-east-1:<AccountId>:deliverystream/<your-firehose>"
        }
      }
    }
  ]
}

Note that the ”AWS:SourceArn” refers to a resource that does not exist yet, but it will be created in the next steps

3. Create the IAM Role for Firehose

The role needs:

Trust policy for Firehose.
Permissions for the S3 bucket.
Permissions to invoke the Lambda function.

Trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket",
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket/*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "lambda:InvokeFunction",
        "lambda:GetFunctionConfiguration"
      ],
      "Resource": "arn:aws:lambda:us-east-1:<AccountID>:function:<your-lambda-name>:*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:<AccountID>:log-group:/aws/kinesisfirehose/waflogs:log-stream:*"
      ],
      "Effect": "Allow"
    }
  ]
}

4. Create the Destination S3 Bucket in the logs account

The bucket must start with the prefix aws-waf-logs- , and have the following cross-account bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountFirehosePut",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<AccountID>:role/<Role created previously>"
        ]
      },
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket",
        "arn:aws:s3:::aws-waf-logs-your-destination-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<AccountID>:role/<Role created previously>"
        ]
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::aws-waf-logs-your-destination-bucket/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control"
        }
      }
    }
  ]
}

5. Create the Firehose Delivery Stream

This step can only be done via CLI or SDK , as the console doesn’t allow choosing a bucket from another account. The delivery stream name must start with the prefix “aws-waf-logs-”

Example cli command:

aws firehose create-delivery-stream \
  --delivery-stream-name aws-waf-logs-<your-name>\
  --delivery-stream-type DirectPut \
  --extended-s3-destination-configuration '{
    "RoleARN": "arn:aws:iam::<AccountId>:role/<Role created previously>",
    "BucketARN": "arn:aws:s3:::aws-waf-logs-your-destination-bucket",
    "Prefix": "!{partitionKeyFromLambda:accountId}/!{partitionKeyFromLambda:webaclName}/!{partitionKeyFromLambda:year}/!{partitionKeyFromLambda:month}/!{partitionKeyFromLambda:day}/",
    "ErrorOutputPrefix": "processing-failed/",
    "BufferingHints": {
      "SizeInMBs": 128,
      "IntervalInSeconds": 300
    },
    "CompressionFormat": "GZIP",
    "ProcessingConfiguration": {
      "Enabled": true,
      "Processors": [
        {
          "Type": "Lambda",
          "Parameters": [
            {
              "ParameterName": "LambdaArn",
              "ParameterValue": "arn:aws:lambda:us-east-1:<AccountId>:function:<your-lambda-name>:$LATEST"
            },
            {
              "ParameterName": "NumberOfRetries",
              "ParameterValue": "3"
            },
            {
              "ParameterName": "RoleArn",
              "ParameterValue": "arn:aws:iam::<AccountId>:role/<Role created previously>"
            },
            {
              "ParameterName": "BufferSizeInMBs", //Change this base on your need
              "ParameterValue": "1"
            },
            {
              "ParameterName": "BufferIntervalInSeconds",
              "ParameterValue": "300"
            }
          ]
        }
      ]
    },
    "DynamicPartitioningConfiguration": {
      "RetryOptions": {
        "DurationInSeconds": 300
      },
      "Enabled": true
    },
    "CustomTimeZone": "UTC"
  }'

6. Configure WAF or Firewall Manager

Finally, update your Web ACLs (or Firewall Manager policies) to use the delivery stream as the log destination.

In a Web ACL:

setting this up you’ll:

Save up to 80% in logging costs compared to direct S3 delivery.
Keep logs centralized and neatly organized by account and WebACL.
Simplify analysis with Athena, OpenSearch, or SIEMs.

This approach is scalable and works seamlessly across multi-account environments managed with Firewall Manager, as well as standalone Web ACLs.

How We Archived Over 14TB of S3 Logs and Cut Storage Costs by 95%

Agustin De Mozzi — Wed, 30 Jul 2025 14:16:57 +0000

Context

During an internal audit aimed at optimizing our cloud infrastructure costs, we discovered that we had over 14TB of AWS WAF logs sitting in an S3 bucket — with no lifecycle policy configured.

These logs had accumulated over several years and were stored in the S3 Standard storage class, which generated a significant monthly cost — especially considering these logs were rarely accessed.

To make things worse, new logs were being added daily , and our S3 costs were growing unchecked.

The Challenge

We were facing three main constraints:

We couldn’t delete the logs due to compliance requirements.
The files were very small (under 40 MB), making them too expensive to transition to a colder storage class like Glacier or Deep Archive using standard tools.
Downloading logs locally to compress and re-upload wasn’t an option: AWS S3 data egress costs exceed $1000 per TB , making this strategy unfeasible at scale.

On top of that, we were spending nearly $350/month just on S3 storage — an unnecessary cost for infrequently accessed data.

The Solution

After evaluating multiple options, we implemented a simple but effective strategy :

We launched an EC2 instance inside our VPC, with an EBS volume sized for the largest log month.
We attached an S3 VPC endpoint, ensuring all traffic remained within the AWS network to avoid data transfer charges.
We wrote a lightweight Bash script to:
List and download all logs from a specific month for a given Web ACL.
Bundle them (using tar, since logs were already gzipped) into larger archive files.
Upload those archives back to the same bucket — but this time, directly using the S3 Glacier Deep Archive storage class.

Here’s the script:

#!/bin/bash

BUCKET="aws-waf-logs"
UUID="1xxxxxxx-dxxx-4xxx-8xxx-axxxxxxxxxx" #Only if you use firewall manager to manage WAF Logs
ORIGEN="us-east-1"
YEAR="2023"
TMP_DIR="./waf-archive"

mkdir -p "$TMP_DIR"

# List account_ids
aws s3 ls "s3://$BUCKET/$UUID/AWSLogs/" | awk '{print $2}' | sed 's#/##' | while read -r ACCOUNT_ID; do
  echo "Account ID: $ACCOUNT_ID"

  # List WebACLs
  aws s3 ls "s3://$BUCKET/$UUID/AWSLogs/$ACCOUNT_ID/WAFLogs/$ORIGEN/" | awk '{print $2}' | sed 's#/##' | while read -r WEB_ACL; do
    echo "WebACL: $WEB_ACL"

    # Listar months of the yeat
    aws s3 ls "s3://$BUCKET/$UUID/AWSLogs/$ACCOUNT_ID/WAFLogs/$ORIGEN/$WEB_ACL/$YEAR/" | awk '{print $2}' | sed 's#/##' | while read -r MONTH; do
      echo "Mes: $MONTH"

      S3_PREFIX="$UUID/AWSLogs/$ACCOUNT_ID/WAFLogs/$ORIGEN/$WEB_ACL/$YEAR/$MONTH/"
      TAR_NAME="${YEAR}_${MONTH}.tar"
      TAR_PATH="$TMP_DIR/$TAR_NAME"
      #Here you can choose another bucket to archive the logs, or create another folder in the same bucket
      ARCHIVE_KEY="archived_logs/$ACCOUNT_ID/$WEB_ACL/$TAR_NAME"

      echo "Downloading from: s3://$BUCKET/$S3_PREFIX"
      mkdir -p "$TMP_DIR/files"
      aws s3 cp --recursive "s3://$BUCKET/$S3_PREFIX" "$TMP_DIR/files"

      echo "Packaging: $TAR_PATH"
      tar -cf "$TAR_PATH" -C "$TMP_DIR/files" .

      # Change $BUCKET if you use another bucket
      echo "Uploading to: s3://$BUCKET/$ARCHIVE_KEY"
      aws s3 cp "$TAR_PATH" "s3://$BUCKET/$ARCHIVE_KEY" --storage-class DEEP_ARCHIVE

      echo "Removing temporal files"
      rm -rf "$TMP_DIR/files"
      rm -f "$TAR_PATH"

    done
  done
done

rmdir "$TMP_DIR"

echo "Archived complete for $YEAR"

This strategy allowed us to retain all logs , fully meeting our compliance requirements , while implementing a financially sustainable retention policy.

Results

We went from paying nearly $350/month in S3 Standard to under $15/month using S3 Glacier Deep Archive — a 95.6% reduction in storage costs.

Key advantages:

The entire process ran inside AWS , so there were no egress charges.
Monthly log processing was automated and scalable across multiple Web ACLs.
The total EC2 cost for processing was around $10 , compared to over $600 if we had used S3’s native transition feature — even under optimistic assumptions (e.g., every object being ~75MB).

Lessons Learned

Don’t overlook lifecycle policies — buckets storing large volumes of data over time can lead to massive and unnecessary costs.
Bundling small files into larger archives greatly reduces storage costs, especially when paired with colder S3 storage classes.
Keep data movement within the AWS network whenever possible — this avoids transfer charges and speeds up processing.

Recommendations

Configure lifecycle policies from day one — it’s easy to ignore logs until they become a financial burden.
If you need to retain data long-term with little or no access , S3 Glacier Deep Archive is an excellent option (just account for longer retrieval times).