DEV Community

Danny Chan for AWS Community Builders

Posted on

3 2 1 1 1

Unlock the Power of Enhanced Filtering with S3 Batch Operations

What is Amazon S3 Batch Operations?

  • It allows you to perform batch operations on large sets of objects in Amazon S3.
  • You can select objects based on prefixes, tags, and metadata.
  • Actions that can be performed include copying, archiving, applying access controls, and adding object tags.
  • The goal is to simplify and automate repetitive tasks.


 

Any news on Batch Operations?

  • You can now manage objects within an entire bucket, based on prefix, suffix, creation date, or storage class.
  • You can quickly apply the operation to all the matching objects.


 

Benefit of Batch Operations

  • You can easily perform one-time or batch workloads:
  • Copying objects between staging and production buckets
  • Invoking an AWS Lambda function to convert file types
  • Restoring archived backups from S3 Glacier storage


 

Create a replication rule

Set the filtering: file name prefix

Image description

Image description

Image description

Image description


 

Create a CSV inventory configuration for batch operation

Manifest.csv contains all objects' names for the batch operation

Image description

Image description

Image description

Image description

Image description

Image description

generative-ai-cities,aws_team_1.png
generative-ai-cities,aws_team_2.png
generative-ai-cities,aws_team_3.png
generative-ai-cities,aws_team_1_large.png
generative-ai-cities,aws_team_2_large.png
generative-ai-cities,aws_team_3_large.png
Enter fullscreen mode Exit fullscreen mode


 

Create an S3 batch operation using a CSV inventory configuration

Select "copy" to replicate objects to the destination bucket.

Image description

Image description

Image description

Image description


 

Create an S3 batch operation using an S3 replication configuration

Filter objects by creation date
You can specify filters to reduce the scope of replicated objects.
These filters work in conjunction with existing filters in your replication configuration.
If no filters are specified, all objects defined by the replication configuration will be replicated.

Image description

Image description

Image description

Image description

Image description

Image description


 

Full visibility

Monitor the running time and percentage of objects completed.
Receive a detailed completion report with the status of each object.

Image description

Image description

Image description

Image description

Image description

Image description


 

Reference

Amazon S3 Batch Operations now manages buckets or prefixes in a single step

Replication configuration

Replication configuration hand-on

Do your career a big favor. Join DEV. (The website you're on right now)

It takes one minute, it's free, and is worth it for your career.

Get started

Community matters

Top comments (2)

Collapse
 
kennc profile image
Kenn C •

Thanks for your valuable sharing. It really helpful for me to do machine learning though S3 Batch Operations.

Collapse
 
danc profile image
Danny Chan •

Happy learning 😀

Create a simple OTP system with AWS Serverless cover image

Create a simple OTP system with AWS Serverless

Implement a One Time Password (OTP) system with AWS Serverless services including Lambda, API Gateway, DynamoDB, Simple Email Service (SES), and Amplify Web Hosting using VueJS for the frontend.

Read full post

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay