Originally published on kuryzhev.cloud
Your versioned S3 bucket is quietly billing you for millions of noncurrent objects that no S3 lifecycle policy was ever written to catch. We found this out the hard way during a routine Cost Explorer review — a "small" logging bucket was sitting at 40TB of noncurrent versions nobody knew existed. If you're running compliance workloads, backups, or high-volume logging on S3, these are the rules I now treat as non-negotiable.
Split lifecycle rules by prefix and tag, never one rule per bucket
A single broad rule with no prefix filter is a blast radius waiting to happen. I've seen a rule meant for logs/ accidentally match logs-backup/ because someone typed a prefix without the trailing slash, and it deleted three weeks of audit data before anyone noticed. Filters can combine one prefix with a tag set using an and block, so use that instead of loose wildcards whenever object count matters.
Map rule IDs 1:1 to a naming convention — logs/, backups/daily/, compliance/retain/ — so anyone reading the Terraform state knows exactly what a rule targets without cross-referencing the console. Also know your ceiling: AWS allows up to 1,000 lifecycle rules per bucket. If you're running a multi-tenant bucket and approaching that limit, it's a sign you should split into separate buckets, not cram more and conditions into one rule.
Use Object Lock in Compliance mode for anything with a retention mandate
Object Lock and a plain S3 lifecycle policy solve different problems — expiration schedules storage cost, Object Lock enforces legal retention that nobody, including root, can override. Watch out: Object Lock can only be enabled at bucket creation time. There's no retrofit; if you forgot it on a bucket that's already holding compliance data, you're migrating objects to a new bucket, not flipping a setting.
Governance mode lets privileged users shorten or remove a retention period — fine for internal policy, useless for an audit. Compliance mode is the only one that's actually audit-safe, since no principal can undo it before the retention date. It also requires versioning to be enabled, and it interacts directly with expiration rules: a locked version blocks deletion even if an expiration rule fires against it, so a lifecycle rule silently no-ops on protected objects rather than erroring.
Transition logs aggressively, expire even more aggressively
Logs are high-volume and low-value per object — treat them differently from backups from day one. A tiering path like STANDARD → STANDARD_IA at 30 days → GLACIER_IR at 90 days → expire at 400 days works well for most CloudTrail and ALB access log volumes, but tune the numbers to your actual query patterns, not a copy-pasted default.
Gotcha: STANDARD_IA and Glacier/Glacier IR carry minimum billable storage durations — 30 days for IA, 90 days for Glacier — so transitioning objects that get deleted or re-uploaded before that window closes triggers a prorated early-deletion charge. For logs with a lifespan under 30 days, plain expiration is often cheaper than any transition at all. Model the cost before applying a tiering rule blindly, and give log types their own prefixes so a policy tuned for CloudTrail doesn't accidentally reshape retention for application logs with a different mandate.
# terraform: s3-lifecycle-governance.tf
# Requires: aws provider ~> 5.0, versioning + Object Lock pre-enabled on bucket
resource "aws_s3_bucket_versioning" "compliance" {
bucket = aws_s3_bucket.compliance.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_lifecycle_configuration" "compliance" {
bucket = aws_s3_bucket.compliance.id
# Rule 1: application logs — aggressive tiering + expiration
rule {
id = "logs-tiering"
status = "Enabled"
filter {
prefix = "logs/"
}
transition {
days = 30
storage_class = "STANDARD_IA"
}
transition {
days = 90
storage_class = "GLACIER_IR"
}
expiration {
days = 400
}
# Orphaned multipart uploads from log shippers — always include this
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
}
# Rule 2: backups — longer retention, slower tiering
rule {
id = "backups-retention"
status = "Enabled"
filter {
and {
prefix = "backups/"
tags = {
"retention" = "long"
}
}
}
transition {
days = 60
storage_class = "GLACIER"
}
expiration {
days = 1825 # 5 years, adjust to compliance mandate
}
}
# Rule 3: noncurrent versions — closes the "silent growth" gap
rule {
id = "noncurrent-cleanup"
status = "Enabled"
filter {
prefix = "" # applies bucket-wide
}
noncurrent_version_transition {
noncurrent_days = 30
storage_class = "GLACIER_IR"
}
noncurrent_version_expiration {
noncurrent_days = 180
}
}
}
Watch noncurrent version growth — it's the silent cost and compliance trap
Every PUT against a versioned bucket creates a new version, and without an explicit noncurrent_version_expiration block, that storage grows unbounded and completely invisible in the console UI. This is exactly the mistake that caused our 40TB surprise — versioning had been enabled for durability years earlier, and nobody ever added the matching cleanup rule. It only surfaced when Cost Explorer flagged an unexplained S3 spend jump.
Add noncurrent_version_transition and noncurrent_version_expiration as their own block, separate from your current-version rules — see Rule 3 in the Terraform above. One more subtlety worth knowing: locked versions are correctly skipped by expiration, but unlocked noncurrent versions in the same bucket still get deleted on schedule, which can create gaps in your retained history if you assumed Object Lock covered everything.
Always add an abort-incomplete-multipart-upload rule
Orphaned multipart uploads are pure waste, and they're invisible unless you go looking. A backup tool or CI job that gets interrupted mid-upload leaves parts sitting in S3, billed at full STANDARD rate, indefinitely, with no expiration ever applied by default.
Set abort_incomplete_multipart_upload { days_after_initiation = 7 } on every single bucket — no exceptions, no "we'll add it later." Verify it's actually catching things periodically, since lifecycle evaluation isn't instant; it runs roughly once a day, so expect up to 24-48 hours of lag between a rule change and it taking effect.
# Validate what's actually configured before assuming it's correct
aws s3api get-bucket-lifecycle-configuration \
--bucket compliance-prod-logs \
--query 'Rules[].{ID:ID,Status:Status,Filter:Filter,Expiration:Expiration}' \
--output table
# Check for orphaned multipart uploads lifecycle hasn't caught yet
aws s3api list-multipart-uploads --bucket compliance-prod-logs \
--query 'Uploads[].{Key:Key,Initiated:Initiated}' --output table
# Example failure when a rule has a filter but no action — silently no-ops
# {
# "Error": {
# "Code": "InvalidArgument",
# "Message": "Found rule without expiry or transition"
# }
# }
# Audit noncurrent version bytes manually (or use S3 Storage Lens for this at scale)
aws s3api list-object-versions --bucket compliance-prod-logs \
--query 'Versions[?IsLatest==`false`].[Key,Size]' --output text | \
awk '{sum+=$2} END {print "Noncurrent bytes:", sum}'
Lifecycle rules are not access control — pair them with deny policies
An S3 lifecycle policy expires objects on a schedule; it does nothing to stop someone from deleting them manually five minutes earlier. For compliance buckets, add an explicit bucket policy Deny on s3:DeleteObject and s3:DeleteObjectVersion for every principal outside a documented break-glass role, and back that up with SCP-level guardrails at the OU level so a bucket policy edit can't quietly undo the protection.
Security note worth flagging: lifecycle-driven deletions don't appear in CloudTrail as a delete action by a principal — they show up as "Lifecycle Expiration" events instead. If your audit tooling only watches for DeleteObject API calls, expirations will slip through unnoticed. Plan for that by enabling S3 server access logs or an EventBridge rule for object expiration events if you need a complete audit trail. We cover the Terraform side of locking down state and access patterns like this in more depth in our Terraform security notes.
Validate before you ship — dry-run and inventory-audit every rule change
There's no native dry-run flag for S3 lifecycle configuration, so test scope changes on a cloned prefix in a non-production bucket first. Run aws s3api get-bucket-lifecycle-configuration before and after a change and diff the output — it's tedious, but it's the only way to confirm the JSON you shipped is the JSON that's actually active, since Terraform state and reality drift more often than people admit.
Gotcha that catches almost everyone eventually: lifecycle transitions and expirations evaluate against an object's creation date, not its last-modified date. Restoring an object from Glacier or re-uploading it resets "last accessed" but not creation date, which means it can transition again on a schedule you didn't expect. Enable S3 Inventory reports to track age and storage-class distribution over time, and if you're managing this across many accounts, S3 Storage Lens's free tier will surface lifecycle rule counts and noncurrent version bytes per bucket without writing a single custom script.
None of this is exotic — it's the same handful of edge cases repeating across every account we've audited. Get the S3 lifecycle policy basics right per bucket (prefix scoping, noncurrent cleanup, multipart abort, Object Lock where it's actually required), and the rest is just tuning tiering windows to match how your data is actually used. For the full lifecycle configuration reference, AWS's own S3 Object Lifecycle Management docs and the Object Lock guide are worth bookmarking — they're more precise on edge cases than most blog posts, including this one.
Top comments (0)