Object Storage Lifecycle Policies: Cutting Storage and Egress Cost Without Losing Data You Need

An object storage bill has four line items that behave nothing alike: storage per GB-month, request counts, retrieval fees, and data transfer out. Lifecycle policies move the first one. They can make the other three worse if you write them from intuition rather than from your actual object inventory.

The failure mode we see most often is a team that reads about Glacier Deep Archive at roughly a fortieth the price of S3 Standard, writes a blanket "transition everything after 30 days" rule, and watches the next invoice go up — because the bucket holds forty million thumbnails averaging 40 KB, and per-object transition requests cost more than the storage those objects were consuming.

Below is the order of operations that avoids that: measure, then compute break-even, then handle egress separately, then roll out with an undo path.

Read the bill before you write a rule

Total bucket size is the least useful number you have. Three others decide whether lifecycle rules will help:

1. Object count and size distribution. Savings scale with bytes; transition costs scale with object count. A bucket of 500,000 objects averaging 20 MB behaves completely differently from 200 million objects averaging 50 KB, even at the same 10 TB.
2. The age curve of reads, not writes. Objects that are never read after 30 days are archive candidates. Objects read once a quarter are usually cheaper left in a hot class, because retrieval fees dwarf the storage delta.
3. What is in the bucket that you cannot see. ListObjectsV2 does not show incomplete multipart uploads or, by default, noncurrent versions. Both are billed.

On AWS, S3 Storage Lens gives you object-count and size-distribution metrics per prefix, and an S3 Inventory report (Parquet, delivered daily) gives you the raw manifest you can query in Athena or DuckDB. GCS has Storage Insights inventory reports. On Cloudflare R2 and Backblaze B2 you generally list into your own manifest and analyze it yourself.

Failed or abandoned multipart uploads leave parts that are billed at the full storage rate but do not appear in normal object listings. Buckets fed by flaky CI jobs or mobile uploaders can accumulate terabytes of them over a couple of years. An AbortIncompleteMultipartUpload rule with DaysAfterInitiation: 7 costs nothing, deletes no completed object, and is frequently the single highest-return line in an entire lifecycle policy. Run aws s3api list-multipart-uploads --bucket <name> before you write anything else.

Compute the break-even before you transition anything

Here are the S3 list prices that drive the math (us-east-1, at the time of writing — verify against the current pricing page, these change):

Three charges sit outside that table and decide most of the outcome:

• Transition requests. Moving an object to Standard-IA or Glacier Instant Retrieval costs about $0.01 per 1,000 objects. Moving to Glacier Flexible Retrieval or Deep Archive costs about $0.05 per 1,000 — five times more, per object, regardless of size.
• Minimum billable object size. Standard-IA and Glacier Instant Retrieval bill every object as at least 128 KB. A 10 KB object in Standard-IA is billed as 128 KB, which makes it more expensive than it was in Standard.
• Per-object metadata overhead in the Glacier classes. Glacier Flexible Retrieval and Deep Archive add roughly 40 KB of billed overhead per object — about 32 KB at the archive rate plus 8 KB at the Standard rate for the name index.

Run the arithmetic on one object before you run it on a hundred million. Take a 128 KB object moving from Standard to Deep Archive. The transition costs $0.00005. The raw storage saving is about $0.022 per GB-month, so on 0.000125 GB that is roughly $0.0000027 per month — before you add the 40 KB overhead, which eats most of what is left. Payback lands somewhere past the decade mark. The same transition on a 500 MB object pays for itself in under a day.

The practical rule: put a size floor on every transition. S3 lifecycle filters support ObjectSizeGreaterThan, so gate archive transitions at 128 KB minimum and, for the Glacier classes, more comfortably at 1 MB. Small objects should be consolidated at write time — packed into archives, or moved into a database — not shuffled between storage classes.

Minimum duration is the second trap. An object transitioned to Standard-IA on day 30 and deleted on day 45 is still billed for 30 days in Standard-IA. If your data has a 60-day total lifespan, a transition at day 30 buys you almost nothing and adds a request charge. Expire it directly instead.

Egress is a separate bill, and lifecycle will not touch it

No storage class changes what you pay to move bytes to the internet. AWS internet egress starts around $0.09/GB in the first tier — an order of magnitude above the monthly cost of storing that same gigabyte in Standard. If transfer out dominates your bill, lifecycle rules are the wrong lever entirely. The levers that work:

• Cache hit ratio. Serving through CloudFront, Cloudflare, or Fastly turns repeated origin reads into cache hits. Measure hit ratio before optimizing anything else.
• Keep reads in-region and off the NAT gateway. Traffic from a private subnet to S3 through a NAT gateway pays roughly $0.045/GB in processing charges on top of everything else. An S3 gateway VPC endpoint removes that and costs nothing.
• Providers that price egress at zero. Cloudflare R2 charges about $0.015/GB-month with no egress fee. Backblaze B2 is about $0.006/GB-month with free egress up to three times your average monthly stored data. For a public download bucket, that difference can be larger than every lifecycle optimization combined.

Retrieval fees deserve the same scrutiny. Glacier Instant Retrieval storage is $0.004/GB-month, but reading that gigabyte back costs $0.03 — more than seven months of storage. Divide monthly bytes read by bytes stored for the prefix. If that ratio is above a few percent, archiving that prefix loses money.

Roll it out with an undo path

Lifecycle has no dry-run mode, so build one: query your inventory manifest for the exact object count and byte total each rule would match, and check the number against what you expect before applying anything.

Then keep expiration rules in separate lifecycle rules from transition rules. They fail differently, and you want to be able to disable deletion without unwinding your archiving. Ship expiration scoped to one prefix first, watch it for a full billing cycle, then widen the filter.

In a versioning-enabled bucket, an Expiration action only writes a delete marker. The object data survives as a noncurrent version and keeps being billed. You need NoncurrentVersionExpiration (plus ExpiredObjectDeleteMarker to sweep the markers) before storage actually drops. Teams routinely apply an expiration policy, see no change in the bill, and conclude lifecycle does not work. Used deliberately, this is a feature: a NoncurrentVersionExpiration of 30 days gives you a 30-day undo window on an over-broad rule.

Two operational notes for the rollout: lifecycle evaluation is asynchronous and runs roughly once a day, so nothing happens the moment you apply a policy — do not assume the rule is broken at hour two. And expirations are silent. Wire a bucket notification on delete events, or track object count in Storage Lens, so a mis-scoped filter surfaces in a dashboard rather than in a restore request six weeks later.

For data with a legal or compliance retention requirement, lifecycle rules are the wrong protection layer — a policy edit removes them. Use Object Lock in compliance mode, which no credential in the account can override for the retention period.

---

Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.