One Default CloudTrail Setting Cost Us $200,000 — Fixed in 5 Minutes

We had a $200,000 problem hiding in plain sight.

Not a breach. Not a runaway workload. Not even the security tool we had just onboarded. A single default setting, accepted during that tool's setup, that nobody revisited as the company scaled.

The Numbers That Didn't Add Up

Late 2025. Reviewing AWS Cost Explorer. CloudTrail — a line item that used to sit quietly around $1,000 a month — had crept to $10,000 in one prod account and $30,000 at the org level.

It never triggered an alert because it didn't spike. It crept.

Around the same time, other costs had come down — reserved instance discounts, some workloads rightsized. The overall bill looked reasonable. CloudTrail was hiding in the noise.

By the time I caught it: roughly $200,000 spent on audit logs nobody had asked for.

What Was Causing It

Breaking down CloudTrail by usage type in Cost Explorer showed the bill was driven entirely by data event recording — 8.98 billion S3 events in us-east-1 alone in December, at $0.10 per 100,000 events.

The org trail config explained why:

{
  "ReadWriteType": "All",
  "DataResources": [
    { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::"] }
  ]
}

Every S3 API call. Every read, every write. Every bucket. Every account in the org.

S3 reads account for 85–95% of all S3 data events in a production environment. ML pipelines, CDN origins, build artifact stores — all logging every single fetch. Billions of them. Every month.

Where That Config Came From

One of the onboarding steps for a new security tool was to enable CloudTrail S3 data events. The setup offered a default: log read and write events for all S3 buckets. An engineer accepted it. Completely reasonable — it was the default, and a security tool was asking for it.

That's the trap. The security tool wasn't reading our buckets. Our own workloads were, the same way they always had. The difference was that CloudTrail now recorded every one of those GetObject and HeadObject calls as an audit event, at $0.10 per 100,000. As traffic grew, so did the bill.

Does Compliance Actually Require This?

Before touching anything, I checked.

Framework	S3 Read Logging Required?
SOC 2 Type II	No
ISO 27001	No
GDPR	No
CIS AWS Benchmark	Recommended (Level 2), not mandatory

The CIS benchmark does recommend object-level logging for both reads and writes. That's often why security tools ask for it — it's an easy check to pass. But passing a check isn't the same as needing the data for every bucket.

Read auditing earns its cost on buckets that hold customer data, PII, or credentials — there you want to know who accessed what. For everything else, writes and deletes are what matter.

Nobody needs to know a Lambda function fetched a static asset at 3am. They do need to know if someone deleted a production backup.

The Fix

Phase 1 — Switch to WriteOnly

aws cloudtrail put-event-selectors \
  --trail-name <your-org-trail-arn> \
  --event-selectors '[{"ReadWriteType":"WriteOnly","IncludeManagementEvents":true,"DataResources":[{"Type":"AWS::S3::Object","Values":["arn:aws:s3:::"]}]}]' \
  --region us-east-1

"All" → "WriteOnly".

One word. ~$300,000 in annual savings.

Phase 2 — Add Reads Back Only Where They Matter

Switching to WriteOnly eliminates ~90% of the cost. The next step is turning read logging back on — but only for buckets that genuinely need it: customer data, PII, credentials, or compliance data.

[
  {
    "ReadWriteType": "WriteOnly",
    "IncludeManagementEvents": true,
    "DataResources": [
      { "Type": "AWS::S3::Object", "Values": ["arn:aws:s3:::"] }
    ]
  },
  {
    "ReadWriteType": "ReadOnly",
    "IncludeManagementEvents": false,
    "DataResources": [
      {
        "Type": "AWS::S3::Object",
        "Values": [
          "arn:aws:s3:::your-pii-bucket/",
          "arn:aws:s3:::your-customer-data-bucket/"
        ]
      }
    ]
  }
]

Writes are audited everywhere. Reads are audited only on the handful of buckets where access itself is sensitive. Full coverage where it matters, at a fraction of the cost.

At org scale, tag sensitive buckets (security-scan: enabled) and use that tag to drive both CloudTrail scoping and security scanner configuration.

Phase 3 — Make Sure It Can't Creep Again

The original problem wasn't the spend — it was that nobody saw it grow. Set an AWS Budget or Cost Anomaly Detection monitor on the CloudTrail service so a slow climb gets flagged before it becomes a six-figure number.

The Real Lesson

The security tool wasn't the problem. Neither was the engineer who set it up. The problem was a CloudTrail default — log every read and every write on every bucket — accepted as part of a setup checklist without anyone asking what it would cost.

Read logging is valuable on buckets that hold customer data, PII, or credentials. On everything else, it's billions of events nobody will ever look at.

When a new tool asks you to enable CloudTrail data events as part of its setup, check what the default actually covers:

• Read, write, or both? Writes are usually enough.
• All buckets, or only the ones that hold sensitive data?
• What does that cost at your request volume? At $0.10 per 100,000 events, a busy bucket adds up fast.
• Who owns the monthly bill for this tool's footprint?

Security and FinOps need to be in the same conversation when a new tool lands.

Not because security is the enemy of cost — but because the most expensive line on your bill might be a default checkbox a well-intentioned engineer left ticked during a tool rollout.

The config that cost $1,000 a month in 2024 cost $30,000 a month in 2025. Our traffic scaled. The default didn't care.