DEV Community

Cover image for A $200,000 AWS Bill Nobody Noticed — Fixed in 5 Minutes
Bharath Nadar
Bharath Nadar

Posted on

A $200,000 AWS Bill Nobody Noticed — Fixed in 5 Minutes

#aws #finops #cloudtrail #devops

We had a $200,000 problem hiding in plain sight.

Not a breach. Not a runaway workload. A single config field set years ago that nobody had revisited as the company scaled.

The Numbers That Didn't Add Up

Late 2025. Reviewing AWS Cost Explorer. CloudTrail — a line item that used to sit quietly around $1,000 a month — had crept to $10,000 in one prod account and $30,000 at the org level.

It never triggered an alert because it didn't spike. It crept.

Around the same time, other costs had come down — reserved instance discounts, some workloads rightsized. The overall bill looked reasonable. CloudTrail was hiding in the noise.

By the time I caught it: roughly $200,000 spent on audit logs nobody had asked for.

What Was Causing It

Breaking down CloudTrail by usage type in Cost Explorer showed the bill was driven entirely by data event recording — 8.98 billion S3 events in us-east-1 alone in December, at $0.10 per 100,000 events.

The org trail config explained why:

{
  "ReadWriteType": "All",
  "DataResources": [
    { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::"] }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Every S3 API call. Every read, every write. Every bucket. Every account in the org.

S3 reads account for 85–95% of all S3 data events in a production environment. ML pipelines, CDN origins, build artifact stores — all logging every single fetch. Billions of them. Every month.

Does Compliance Actually Require This?

Before touching anything, I checked.

Framework S3 Read Logging Required?
SOC 2 Type II No
ISO 27001 No
GDPR No
CIS AWS Benchmark No

Nobody needs to know a Lambda function fetched a static asset at 3am. They do need to know if someone deleted a production backup.

The Fix

Phase 1 — Switch to WriteOnly 

aws cloudtrail put-event-selectors \
  --trail-name <your-org-trail-arn> \
  --event-selectors '[{"ReadWriteType":"WriteOnly","IncludeManagementEvents":true,"DataResources":[{"Type":"AWS::S3::Object","Values":["arn:aws:s3:::"]}]}]' \
  --region us-east-1
Enter fullscreen mode Exit fullscreen mode

"All" → "WriteOnly".

One word. ~$300,000 in annual savings.

Phase 2 — Scope to Sensitive Buckets Only

Switching to WriteOnly eliminates ~90% of the cost. The next step is replacing the wildcard with only buckets that genuinely need auditing — PII, credentials, or compliance data.

{
  "ReadWriteType": "WriteOnly",
  "DataResources": [
    {
      "Type": "AWS::S3::Object",
      "Values": [
        "arn:aws:s3:::your-pii-bucket/",
        "arn:aws:s3:::your-secrets-bucket/"
      ]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

This brings data event costs close to zero while keeping full audit coverage where it actually matters.

At org scale, tag sensitive buckets (security-scan: enabled) and use that tag to drive both CloudTrail scoping and security scanner configuration.

The Real Lesson

A security tool was onboarded and as part of its setup, full S3 data event logging was recommended — a legitimate recommendation for visibility. The team followed it. Nobody flagged the cost implication at scale.

When a security team introduces a new tool, the wider team deserves to know:

  • What config changes does it recommend — and what do those cost at your scale?
  • What AWS permissions does it need?
  • Who owns the monthly bill for this tool's footprint?

Security and FinOps need to be in the same conversation when a new tool lands.

Not because security is the enemy of cost — but because the most expensive line on your bill might be a setting a well-intentioned engineer enabled six months ago and never revisited.

The config that cost $1,000 a month in 2024 cost $360,000 a year in 2025. The infrastructure scaled. The config didn't.

Top comments (0)