Create an S3 Lifecycle Policy with PowerShell

First, I’m a big believer in doing infrastructure as code.

Using the AWS SDK with any library is great, but for things like S3 I’d highly recommend you use a Terraform module such as Cloudposse terraform-aws-s3-bucket module. Everything Cloudposse produces has great quality, flexibility with naming conventions, and more.

Now that this disclaimer is out of the way, I’ve run into scenarios where you can have a bucket with a large amount of data such as databases which would be good to do some cleanup on before you migrate to newly managed backups.

In my case, I’ve run into 50TB of old backups due to tooling issues that prevented cleanup from being successful. The backup tooling stored a sqlite database in one subdirectory and in another directory the actual backups.

I preferred at this point to only perform the lifecycle cleanup on the backup files, while leaving the sqlite file alone. (side note: i always feel strange typing sqlite, like I’m skipping an l 😁).

Here’s an example of how to do this from the AWS PowerShell docs.

I modified this example to support providing multiple key prefixes. What wasn’t quite clear when I did this the need to create the entire lifecycle policy collection as a single object and pass this to the command.

If you try to run a loop and create one lifecycle policy for each Write-S3LifecycleConfiguration command, it only kept what last ran. Instead, ensure you create the entire object as shown in the example, and then you’ll be able to have multiple lifecycle policies get attached to your bucket.

Good luck!

	Import-Module AWS.Tools.S3
	Set-AWSCredential -ProfileName foo
	$Region = 'eu-west-1'
	$TargetBucket = 'my-big-bucket'
	$Prefix = @(
	'keyprefix1'
	'keyprefix2'
	'keyprefix3'
	)
	$Rules = @()
	foreach ($p in $prefix)
	{
	$Filter = "$p/MySubKeyToFilterOn"
	$NewRule = [Amazon.S3.Model.LifecycleRule]@{
	Expiration = @{
	Days = 60
	}
	Id = 'expire-current-60-{0}' -f $p.replace('_', '-').ToLower()
	Filter = @{
	LifecycleFilterPredicate = [Amazon.S3.Model.LifecycleAndOperator]@{
	Operands = @(
	[Amazon.S3.Model.LifecyclePrefixPredicate] @{
	'Prefix' = $Filter
	}
	[Amazon.S3.Model.LifecycleTagPredicate] @{
	'Tag' = @{
	'Key' = 'archived'
	'Value' = 'yes'
	}
	}
	)
	}
	}
	Status = 'Enabled'
	NoncurrentVersionExpiration = @{
	NoncurrentDays = 90
	}
	}
	$Rules += $NewRule
	}
	Write-S3LifecycleConfiguration -BucketName $TargetBucket -Configuration_Rule $Rules -Region $Region -Verbose

view raw aws-s3-lifecycle-policy-creation.ps1 hosted with ❤ by GitHub

#tech #development #aws #powershell #devops