Cross-Region Bucket Replication of Existing Data Using Amazon S3 Batch Operation

“ I have checked the documents of AWS to do cross-region bucket replication of existing data using amazon s3 batch operation. Pricing of S3 bucket as per storage size and S3 batch job replication as per $0.25 per job.”

An S3 Batch Operations is a data management functionality in Amazon S3 that allows you to handle billions of items at scale with only a few clicks in the Amazon S3 Management Console or a single API request. With S3 Batch Operations, you can make changes to object metadata and properties, as well as perform other storage management tasks such as copying or replicating objects between buckets, replacing object-tag sets, modifying access controls and restoring archived objects from S3 Glacier.

A single job can execute a specific operation on billions of objects carrying exabytes of data. It manages retries, tracks progress, sends notifications, generates completion reports and delivers events to AWS Cloudtrail for all changes made and tasks executed.

In this post, you will get to know how to do cross-region bucket replication of existing data using amazon s3 batch operation. Here I have created a s3 bucket, IAM role, S3 replication rule and s3 batch operation.

Prerequisites

You’ll need an Amazon Simple Storage Service for this post. Getting started with Amazon Simple Storage Service provides instructions on how to create a bucket in simple storage service.

Architecture Overview

The architecture diagram shows the overall deployment architecture with data flow, amazon s3 and iam role.

Solution overview

The blog post consist of the following phases:

1. Create of IAM Replication Role and Replication Rule on Source Bucket
2. Create of IAM Batch Role and Replication Job
3. Output of S3 Batch Operation

I have s3 buckets as below →

Phase 1: Create of IAM Replication Role and Replication Rule on Source Bucket

1. Open the IAM console and create a replication role with required permission. Also open the source bucket, create a replication rule under the management section.

Phase 2: Create of IAM Batch Role and Replication Job

1. Open the IAM console and create a batch role with required permission. Also open the source bucket, create a replication job.

Phase 3: Output of S3 Batch Operation

Clean-up

Delete S3 Bucket and IAM Role.

Pricing

I review the pricing and estimated cost of this example.

Cost of S3 in N. Virginia = $0.232

Cost of S3 in Oregon = $0.232

S3 Batch Replication for Job = $0.25

Total Cost = $0.714

Summary

In this post, I showed “cross-region bucket replication of existing data using amazon s3 batch operation”.

For more details on Amazon S3 Batch Operations, Checkout Get started Amazon S3 Batch Operations, open the Amazon S3 Batch Operations console. To learn more, read the Amazon S3 Batch Operations documentation.

For more details on IAM, Checkout Get started IAM, open the IAM console. To learn more, read the IAM documentation.

Thanks for reading!

Connect with me: Linkedin